Diverse Projection Ensembles for Distributional Reinforcement Learning
Authors: Moritz Akiya Zanger
Wendelin Boehmer
Matthijs T. J. Spaan
Venue: ICLR 2024 poster (2024)
Keywords: Projections
Epistemic Uncertainty
Ensembles
Exploration
Reinforcement Learning
Distributional Reinforcement Learning
Machine Learning
Artificial Intelligence
Primary area: reinforcement learning
TL;DR: This paper introduces ensembles for distributional reinforcement learning that use different projections to drive diversity and exploration.

Abstract: In contrast to classical reinforcement learning, distributional RL algorithms aim to learn the distribution of returns rather than their expected value. Since the nature of the return distribution is generally unknown a priori or arbitrarily complex, a common approach finds approximations within a set of representable, parametric distributions. Typically, this involves a projection of the unconstrained distribution onto the set of simplified distributions. We argue that this projection step entails a strong inductive bias when coupled with neural networks and gradient descent, thereby profoundly impacting the generalization behavior of learned models. In order to facilitate reliable uncertainty estimation through diversity, this work studies the combination of several different projections and representations in a distributional ensemble. We establish theoretical properties of such projection ensembles and derive an algorithm that uses ensemble disagreement, measured by the average $1$-Wasserstein distance, as a bonus for deep exploration. We evaluate our algorithm on the behavior suite benchmark and find that diverse projection ensembles lead to significant performance improvements over existing methods on a wide variety of tasks with the most pronounced gains in directed exploration problems.

Published as a conference paper at ICLR 2024
DIVERSE PROJECTION ENSEMBLES FOR DISTRIBUTIONAL REINFORCEMENT LEARNING
Moritz A. Zanger Wendelin Böhmer Delft University of Technology. The Netherlands {m.a. a. zanger, j. bohmer, m. J.t.j.spaan Studel ft..l...nnn
Matthijs T. Spaan
ABSTRACT
In contrast to classical reinforcement learning (RL), distributional RL algorithms aim to learn the distribution of returns rather than their expected value. Since the nature of the return distribution is generally unknown a priori or arbitrarily complex, a common approach finds approximations within set of representable, parametric distributions. Typically, this involves projection of the unconstrained distribution onto the set of simplified distributions. We argue that this projection step entails strong inductive bias when coupled with neural networks and gradient descent, thereby profoundly impacting the generalization behavior of learned models. In order to facilitate reliable uncertainty estimation through diversity, we study the combination of several different projections and representations a distributional ensemble. We establish theoretical properties of such projection ensembles and derive an algorithm that uses ensemble disagreement, measured by the average I-Wasserstein distance, bonus for deep exploration. We evaluate our algorithm on the behavior suite benchmark and VizDoom and find that diverse projection ensembles lead significant performance improvements over existing methods on a variety of tasks with the most pronounced gains in directed exploration problems.
INTRODUCTION
In reinforcement learning (RL), agents interact with an unknown environment, aiming to acquire poli- cies that yield high cumulative rewards. In pursuit of this objective, agents must engage in a trade-off between information gain and reward maximization, a dilemma known as the exploration/exploitatio trade-off. In the context of model-free RL, many algorithms designed address this problem effi- ciently rely on form of the optimism in the face of uncertainty principle (Auer 2002) where agents act according to upper confidence bounds of value estimates. When using high-capacity function approximators (e.g., neural networks) the derivation of such confidence bounds is non-trivial. One popular approach fits an ensemble of approximations to a finite set of observations (Dietterich) 20000] Lakshminaray et al. 2017). Based the intuition that set of parametric solutions explains observed data equally well but provides diverse predictions for unobserved data, deep ensembles have shown particularly successful at quantifying uncertainty for novel inputs. An exploring agent may, for example, seek to reduce this kind of uncertainty by visiting unseen state-action regions suffi- ciently often, until ensemble members converge to almost equal predictions. This notion of reducible uncertainty is also known as epistemic uncertainty (Hora 1996 Der Kiureghian and Ditlevsen) 2009).
A concept somewhat orthogonal to epistemic uncertainty is aleatoric uncertainty, that is the uncer- tainty associated with the inherent irreducible randomness of an event. The latter is the subject of the recently popular distributional branch of RL (Bellemare et al., 2017). which aims to approximate the distribution of returns, as opposed to its mean. While distributional RL. naturally lends itself to risk-sensitive learning, several results show significant improvements over classical RL even when distributions are used only to recover the mean (Bellemare et al. 2017 Dabney et al.) 2018b) Rowland et al. 2019 Yang et al.) 2019 Nguyen-Tane et al.) (02 1). In general, the probability distribution of the random return may be arbitrarily complex and difficult to represent, prompting many recent advancements to rely on novel methods to project the unconstrained return distribution onto set of representable distributions.
In this paper, we study the combination of different projections and representations in an ensemble of distributional value learners. In this setting, agents who seek to explore previously unseen states and

---

Published as a conference paper at ICLR 2024
actions recognize such novel, out-of distiiibbbiitii inputs by the diversity of member predictions: through learning, these predictions align with labels in frequently visited states and actions, while novel regions lead to disagreement. For this, the individual predictions for unseen inputs, hereafter also referred to as generalization behavior, are required to be sufficienlly diverse. We argue that the projection step in distributional RL imposes an inductive bias that leads such diverse generalization behaviors when joined with neural function approximation. We thus deem distributional projections instrumental to the construction of diverse ensembles, capable of effective separation of epistemic and aleatoric uncertainty. To illustrate the effect of the projection step in the function approximation setting, Fig. [hooss a toy regression problem where the predictive distributions differ visibly for inputs x not densely covered by training data depending on the choice of projection.
Our main contributions are as follows:
(1) We introduce distributional projection ensembles and analyze their properties theoretically. In our set- ting, each model is iteratively updated toward the projected mixture over ensemble return distributions. We describe such use of distributional ensembles for- mally through a projection mixture operator and es- tablish several of its properties, including contractiv- ity and residual approximation errors.
Categocical
Quantile
0.0
-0.1
Toy ID-regression: Black dots are training data with inputs x: and labels y. Two models have been trained to predict the distri- bution p(y]z) using a categorical projection (L.h.s.) and a quantile projection (r.h.s.). We plot contour lines for the [0.1, 0.9] quantiles of the predictive distributions over the interval æ E [-1.5, 1.5].
Figure I:
(2) When using shared distributional temporal differ- ence (TD) targets, ensemble disagreement is biased represent distributional TD errors rather than errors w.r.t. the true return distribution. To this end, we de- rive a propagation scheme for epistemic uncertainty that relates absolute deviations from the true value function to distributional TD errors. This insight allows us to devise an optimism-based exploration algorithm that leverages a learned bonus for directed exploration.
(3) We implement these algorithmic elements in deep RL setting and evaluate the resulting agent on the behavior suite (Osband et al., 2020), a benchmark collection of 468 environments, and a set of hard exploration problems in the visual domain VizDoom (Kempka et al., (016). Our experiments show that projection ensembles aid reliable uncertainty estimation and exploration, outperforming baselines most tasks, even when compared to significantly larger ensemble sizes.
RELATED WORK
Our work builds swiftly growing body of literature in distributional RL (Morimura et al. 2010 [Bllemare et al. 2017). In particular, several of our theoretical results rely on works by Row land et al. (2018) and Dabney et al. (2018b), who first provided contraction properties with categorical and quantile projections in distributional RL respectively. Numerous recently proposed algorithms (Dabney al. 2018a| Rowland et al.) 2019 Yang et al. 2019, Nguyen-Tang Tang al. 2021) based on novel representations and projections, typically with an increased capacity to represent complex distributions. In contrast our approach, however, these methods have no built-in functionality to estimate epistemic uncertainty. To the best of our knowledge, our work is the first to study the combination of different projection operators and representations in the context of distributional RL.
Several works, however, have applied ensemble techniques to distributional approaches. For example, Clements al. (2019), Eriksson et al. (2022), and Hoel al. (2023) use ensembles of distributional models to derive alcatoric and epistemic risk measures. Lindenberg et al. (2020) use an ensemble of agents in independent environment instances based on categorical models to drive performance and stability. Jiang et al.) (2024) leverage quantile-based ensembles to drive exploration in contextual MDPs, while Nikolov et al. (2019) combine a deterministic Q-ensemble with distributional categor- ical model for information- directtd sampling. In broader sense, the use of deep ensembles for value estimation and exploration is widespread (Osband et al., 2016 2019 Flennerhag et al. 2020 Fellows et al. 2021 Chen et al., 2017). A notable distinction between such algorithms is whether ensemble members are trained independently or whether joint TD backups are used. Our work falls into the latter category which typically requires propagation mechanism to estimate value uncertainty rather
2

---

Published as a conference paper at ICLR 2024
than uncertainty in TD targets (Janz al. 2019 Fellows et al. 2021 Moerland et al. 2017). Our proposed propagation scheme establishes a temporal consistency between distributional TD errors and emrors w.r.t the true return distribution. In contrast the related uncertainty Bellman equation (O'Donoghue et al. 2018), our approach applies to the distributional setting and devises uncertainty propagation from the perspective of error decomposition, rather than posterior variance.
BACKGROUND
Throughout this work, we consider a finite Markov Decision Process (MDP) (Bellman 1957 of the tuple (S, A, R, 7, P. μ1) as the default problem framework, where S is the finite state space, A is the finite action space, R: S + P (R) is the immediate reward distribution, y E [0. 1] is the discount (S) is the transition kernel, and μ (S)) is the start state distribution. Here, we write (x') ) to indicate the space of probability distributions defined over some space X. Given a state S₃ at time agents draw an action A, from stochastic policy S + (AA) to be presented the random immediate reward RG ~ Re, As) and the successor state S ~ P(-Ss, At). Under policy и and transition kernel P. the discounted return is a random variable given by the discounted cumulative sum of random rewards according to 2 as = Ei 2 Re, where S. = s. Ao = Note that our notation will generally use uppercase letters to indicate random variables. Furthermore, we write D(Z" as) E (R) to denote the distribution of the random variable (*(s,a). that is a state-action-depe deepeded distribution residing in the space of probability distributions ((R). For explicit referrals, we label this distribution 17 (8, a) D (2* (s,a)) The expected value of (s, a)
factor, P: S A
is known as the acte-nction value (T, a) = E9 - s, a)] and adheres to a temporal consistency condition described by the Bellman equation (Bel man 1957)
Q" (s, a) = Ep,, RR + Q (S) A1)[S = A, = a],
(1)
where Er,,, indicates that successor states and actions are drawn from P and respectively. Moreover, the Bellman operator 7 (s, a) := Ep,,, [Ro + (Q(S), A1) = s, Ao = a) has the unique fixed point Q" (s, a).
3.1 DISTRIBUTIONAL REINFORCEMENT LEARNING
The distributional Bellman operator T (Bellemare et al. 2017) is a probabilistic generalization of T and considers return distributions rather than their expectation. For notational convenience, first define pa to be the transition operator according to
P" (= _ A1),
where
S1 P(-SS = A. = a), A (-1S1), (2)
and 2 indicates equality in distributional law (White 1988). In this setting, the distributional Bellman operator is defined as
T 7*Z(s, _ Ro + +> (s,
Similar to the classical Bellman operator, the distributional counterpart T : (R)SAA ((R)SxA has the unique fixed point T 2 Zz, that is the true return distribution z In the context of iterative algorithms, we will also refer to the identity T Z(s, a) as a bootstrap of the distribution Z(s, a). For the analysis of many properties of T is helpful to define a distance metric over the space of return distributions P (R)SAA Here, the supremum p- Wasserstein metric up : P (R)SxA (R) - [0, oo] has proven particularly useful. In the univariate case, up is given by
(p(u,1/)= = sup (1 - 1000---7l.. (T)
s.aESxA
where E (1) v. v are any two state-action return distributions, and Fo(s.a) : + [0, 1] the cumulative distribution function (CDF) of v(ss, a). For notational brevity, we will use the notation Wp (v((s, a), (s(ss, a)) w.,(,, (u,,''(ss a) for the p-sasserstein distance between distributions v, v/, evaluated at (s, a). One of the central insights of previous works in distributional RL is that the operator T is a 7-contraction in Wp (Bellemare et al. (017), meaning that we have wp (TT T"v/) < x (v, v'), property that allows us (in principle) to construct convergent value iteration schemes in the distributional setting.
3.2 CATEGORICAL AND QUANTILE DISTRIBUTIONAL RL
In general, we can not represent arbitrary probability distributions in PR) and instead resort to parametric models capable of representing subset F of 3 (R)). Following Bellemare al. 2023
3

---

Published as a conference paper at ICLR 2024
refer to F as a representation and define it to be the set of parametric distributions Pe with # = (Po E (R) : E θ). Furthermore, we define the projection operator II: (R) + & to be mapping from the space of probability distributions (RR to the representation P Recently, two particular choices for representation and projection have proven highly performant in deep RI.: the categorical and quantile model.
The categorical representation (Bellemare et al., 2017 Rowland et al.
2018) assumes a weighted
mixture of K Dirac deltas dz. with support at evenly spaced locations za [=1, The
categorical representation is then given by &c = 1 0rd=e 00x 2 O, > k= 0g = 13. The corresponding categorical projection operator Il maps distribution from 3
(R) to distribution in Fc by assigning probability mass inversely proportional to the distance to the closest 2 in the support [21, ..... 2 for every point in the support of v. For example, for single Dirac distribution 6² and assuming E S x S 1 the projection is given by
S++1 S2z
Rk+1-SE 62L11"
(5)
ズk+1-5t
The corner cases are defined such that I1 = 6z, Vx S 21 and Ic = 62x Va ZK. It is
straightforward to extend the above projection step to finite mixtures of Dirac distributions through Il Ex prde. >n pxllcos.. The full definition of the projection IIc is deferred to Appendix A.5
The quantile representation (Dabney et al. 2018b), like the categorical representation, comprises mixture distributions of Dirac deltas dog (z), but in contrast parametrizes their locations rather than probabilities. This yields the representation .0o {Σ 1o2 f50, (2)]0 RR. For some distribution v E (RR), the quantile projection T is mixture of K Dirac delta distributions with the particular choice of locations that minimizes the 1-Wasserstein distance between v E (R) and the projection lQ E FQ. The parametrization O2 with minimal 1-Wasserstein distance is given by the evaluation of the inverse of the CDF, F5!, at midpoint quantiles т 2k-1 k [1, K). s.t. 0g- F. 26- Equivalently, 0g is the minimizer of the quantile regression loss (QR) (Koenker and Haooock 2001). which is more amenable to gradient- based optimization. The loss is given by
(o v) = TE (P=, (Z 0.))];
where (2) = uur - 1 (nco) (u) is an error function that assigns asymmetric weight to over- or underestimation errors and denotes the indicator function.
4 EXPLORATION WITH DISTRIBUTIONAL PROJECTION ENSEMBLES
This paper is foremost concerned with leveraging ensembles with diverse generalization behaviors induced by different representations and projection operators. To introduce the concept of distri- butional projection ensembles and their properties, we describe the main components in a formal setting that foregoes sample-based stochastic approximation and function approximation, before moving to a more practical deep RL setting in Scction We begin by outlining the projection mixture operator and its contraction properties. While this does not inform exploration algorithm in its own right, it lays a solid algorithmic foundation for the subsequently derived exploration frame- work. Consider each associated with a representation R and a projection operator II; In this setting, we assume that each member distribution (), ) E Fi is an element of the associated representation F, and the projection operator II; : (R) + F, maps any distribution v E (RR to 3, such that I1 E F,, The set of representable uniform mixture distributions over E is then given by FE = {Mæ(s,a) a) ( a) = M E. (s a),,), (,(s,a) E F, i [1..... MM}. We can now define a central object in this paper, the projection mixture operator 2M (R) - F E, as follows:
ensemble E {n,(s, a) i E [1.....M] of M member distributions (,(s, a),
aum(ss a) = Ni Σ Ma((s,a)
Joining Qu with the distributional Bellman operator T yields the combined operator 22 T Fig. 2
illustrates the intuition behind the operator 7 the distributional Bellman operator T is applied to a return distribution y (Fig. 2 and b), then projects the resulting distribution with the individual projection operators II, onto M different representations 7) = IT 7) E F, (Fig. 2e and d), and finally recombines the ensemble members into a mixture model in FE (Fig. e). In connection with iterative algorithms, we are often interested in the contractivity of the combined operator 22r7 " to establish convergence, Proposition deelnneatss conditions under which we can combine individual projections II, such that the resulting combined operator Uay T is a contraction mapping.
4

---

Published as a conference paper at ICLR 2024
a)
b)
Tm
c) ITT
d) IlTT
Dai T
Figure 2: Illustration of the projection mixture operator with quantile and categorical projections.
Proposition 1 Lef IT, i [1...., M] be projection operators II; (R) + F; mapping from the space of probability distributions (R) to representations F; and denote the projection mixture operator Q2 9 (RR) + #E as defined in Eq. 7 Furthermore, assume that for some E [1, oo) each projection 11, is bounded in the p- p-Wasserstein metric the sense that for any two return distributions n. nf have Wip (II,, Iar) ) (s. a) I coup (1, (/,7 (s, a) for a constant C-- Then, the combined operator QMTT" is bounded in the supremum p- Wasserstein distance Wp by
Wp TT Sur 1/) pp;,0p():)7)
and is accordingly a contraction so long as E,7 < 1, where Ep = (EM M 1.
The proof is deferred to Appendix The contraction condition in Proposition Dis naturally satisfied for example if all projections TI, are non-expansions in joint metric Wpp It is, however, more permissive in the sense that only requires the joint modulus Cp to limited, allowing for expanding operators in the ensemble for finite p. A contracting combined operator 22 allows us to formulate a simple convergent iteration scheme where sequence of steps k. ensemble members are moved toward the projected mixture distribution according to f II, 7 yielding the (A: + 1)-th mixture distribution M,, = M Ei= fH,4+1 This procedure can be compactly expressed by
1
1E,,++ = 2MT iE for [0. 1,.
and has unique fixed point which we denote TE = 1)
4.1 FROM DISTRIBUTIONAL APPROXIMATIONS TO OPTIMISTIC BOUNDS
We proceed to describe how distributional projection ensembles be leveraged for exploration. Our setting considers exploration strategies based on the upper-confideece-boet -ound (UCB) algorithm (Auer 2002). In the context of model-free RL, provably efficient algorithms often rely on the construction of a bound, that overestimates the true state-action value with high probability (Jin et al. 2018 2020). In other words, we are interested in finding an optimistic value Q+ (s, a) such that Q (s, 2 (s. with high probability. To this end, Proposition 2 relates an estimate (ss a) the true value (s) through a distributional error term.
Proposition 2 Let ((, E|2(s, a)] be a state-action value estimate where (1, a) fi(s, a) random variable distributed according turn distribution (" (8, a). Further, denote Q" (²{ssa) = E (8, a)) the true state-action, where z (s. 17" (F, (s,a). a). We have that QT a) is bounded from above by (s. + w1 (fi, (fi,7") ()) () 2 (s. ((s,)) where wi is the 1-Wasserstein distance metric.
an estimate ij(s, a) of the true state-action re-
The proof follows from the definition of the Wasserstein distance and is given in Appendix A Proposition 2 jimplis that, for given distributional estimate n)ss, a), upper bound on (T, a) by adding a bonus of the 1- I-Wasserstein distance between an estimate i) a) and the true return distribution 1/" (s, a). which we define bzd b²*(sa) wi (f), 17 (s, a) in the following. By adopting an optimistic action-selection with this guaranteed upper bound on Q" a) according
construct optimistic
= arg max[ a) + 6" a)),
aĒ A
we maintain that the resulting policy inherits efficient exploration properties of known optimism- based exploration methods. Note that in convergent iteration scheme, we should expect the bonus 6nd a) to almost vanish in the limit of infinite iterations. We thus refer to 6" (s, a)) as measure of the epistemic uncertainty of the estimate 7j(8, a).
4.2 PROPAGATION OF EPISTEMIC UNCERTAINTY THROUGH DISTRIBUTIONAL ERRORS
By Proposition?2 2 an optimistic policy for efficient exploration can be derived from the distributional error 6dd (s, a). However, since we do not assume knowledge of the true return distribution ( a),
5

---

Published as a conference paper at ICLR 2024
this error term requires estimation. The primary purpose of this section is to establish such an estimator by propagating distributional TD errors. This is necessary because the use of TT backups prohibits consistent uncertainty quantification in values (described extensively in the Bayesian setting for example by Fellows et al. (021). The issue is particularly easy to see by considering the backup in a single (s,a) tuple: even if every estimate fjc(s, a) in an ensemble fits the backup T" a) accurately, this does not imply fj a) = 17" (s, a) as the TD backup may have been incorrect. Even a well-behaved ensemble (in the sense that its disagreement reliably measures prediction errors) in this case quantifies errors w.r.t. the bootstrapped target f2 T f)e a), rather than the true return distribution 17" (s, a).
To establish a bonus estimate that allows for optimistic action selection in the spirit of Proposition 2 we now derive propagation scheme for epistemic uncertainty in the distributional setting. More specifically, we find that an upper bound on the bonus b" (s, a) satisfies a temporal consistency condition, similar to the Bellman equations, that relates the total distributional error w1 (E (s, to a one-step error w1 (f). Qx (axTTff)(s (s, a) that is more amenable to estimation.
Theorem
Let ii(s,)) E (R) be an estimate of the true return distribution nf (s, a) (R),
and denote the projection mixture operator Ωu : (R) + FE with members II, and bounding moduli e and Ep as defined in Proposition [ Furthermore, assume SyyT" is contraction mapping with fixed point T/g. We then have for all (5,a) ESSxX A
wi (f, (s, a) S wi (M. (s, + w
(f, 12 (S) An) s, Ao = a).
where S1 ~ P( = s, Au = a) and A1 ~ (-1S1)
The proof is given in Appendix and exploits the triangle inequality property of the Wasserstein distance. It may be worth noting that Theorem 3 is a general result that is not restricted to the use of projection ensembles. is, however, natural complement to the iteration described in Eq. in that it allows us to reconcile the benefits of bootstrapping diverse ensemble mixtures with optimistic action selection for directed exploration. To this end, we devise a separate iteration procedure aimed at finding an approximate upper bound on w1 (f, ng)) (s, Denoting the k-th iterate of the bonus estimate (s, a), we have by Theorem 3 that the iteration
b (8, a) = wi (ū,, TT) a) + pr [bx (Si, Ai)|So - =, Ao = a] V(s, E x A, converges to an upper bound on w1 (fi, (s. a I Notably, this iteration requires only local error estimate w1 (ñ) TT) (s, a) and is more amenable to estimation through our ensemble.
We conclude this section with the remark that the use of projection ensembles may clash with the intuition that epistemic uncertainty should vanish in convergence. This is because each member inherits irreducible approximation errors from the projections II, In Appendix A we provide general bounds for these errors and show that residual errors can be controlled through the number of atoms K in the specific example of an ensemble based on the quantile and categorical projections.
DEEP DISTRIBUTIONAL RL WITH PROJECTION ENSEMBLES
Section 4 has introduced the concept of projection ensembles in a formal setting. In this section, we aim to transcribe the previously derived algorithmic components into a deep RL algorithm that departs from several of the previous assumptions. Specifically, this includes 1) control with a greedy policy, 2) sample-based stochastic approximation, 3) nonlinear function approximation, and 4) gradient-based optimization. While this sets the following section apart from the theoretical setting considered in Section 4 we hypothesize that diverse projection ensembles bring to bear several advantages in this scenario. The underlying idea is that distributional projections and the functional constraints they entail offer an effective tool to impose diverse generalization behaviors on an ensemble, yielding a more reliable tool for out-of-distribution sample detection. In particular, we implement the above-described algorithm with a neural ensemble comprising the models of the two popular deep RL algorithms quantile regression deep Q network (QR-DQN) (Dabney et al. 2018b) and C51 (Bellemare al. 2017)
'To see the convergence, note that the sequence is equivalent to iteration with 7' in MDP with the deterministic immediate reward wi (ñ, f2r f)) (s.

---

Published as a conference paper at ICLR 2024
5.1 DEEP QUANTILE AND CATEGORICAL PROJECTION ENSEMBLES FOR EXPLORATION
In this section, we propose Projection Ensemble DQN (PE-DQN), a deep RL algorithm that combines the quantile and categorical projections (Dabney al. 201 Bellemare et al., 2017) into a diverse ensemble to drive exploration and learning stability. Our parametric model consists of the mixture distribution 7 parametrized by 0. We construct T as an equal mixture between a quantile and categorical representation, each parametrized through a NN with K output logits where we use the notation 8 to mean the k-th logit of the network parametrized by the parameters 8 of the i-th model in the ensemble. We consider sample transition (s, a, s', a') where a' is chosen greedily according to Ez-ng, .ar) [Z]. Dependencies on (s, a) are hereafter dropped for conciseness by writing & = (s, a) and l ( a').
Projection losses. Next, we assume that bootstrapped return distributions are generated by a set of delayed parameters 0, as common (Mnih et al., 2015). The stochastic (sampled) version of the distributional Bellman operator T", applied to the target ensemble's 's mixture distribution 77, yields
Th Σ E
(10)
Instead of applying the projection mixture Da analytically, as done in Section 4 the parametric estimates 7E are moved incrementally towards a projected target distribution through gradient descent on a loss function. In the quantile representation, we augment the classical quantile regression loss (Koenker and Hallock 2001) with an importance-sampling ratio Kp(0))) to correct for the non- For set of fixed quantiles Tke the uniformity of atoms from the bootstrapped distribution T" loss L is given by
L1 (10, T
Σ Kp(01,)))))))
(prn( + (2(()) -11)).
(11)
The categorical model minimizes the Kullback-Leibler (KL) divergence between the projected bootstrap distribution II T"NEE and estimate 7e2. The corresponding loss is given by
L2 (102, IeTt
DxL (Ic )
(12)
As T" a is a mixture of Dirac distributions, the definition of the projection Ic according to Eq. can be applied straightforwardly to obtain the projected bootstrap distribution Iee i "1.6 "
Uncertainty Propagation. We aim to estimate state-action dependent bonus (s a) in the spirit of Theorem 3 and the subsequently derived iteration with set of parameters Φb For this, we estimate the local error estimate wi (NE., Qas 7 ne,e) a) as the average ensemble disagreement Wavg (s, a) (M 1)) Eij== wi (110, 7o, (s, a). The bonus (s a) can then be learned in the same fashion regular value function with the local uncertainty estimate Wavy a) as an intrinsic reward. This yields the exploratory action-selection rule
0g arg max (Ezz~ng.ets.a) [2] bb a)), acA
(13)
where / is a hyperparameter to control the policy's drive towards exploratory actions. Further details on our implementation and an illustration of the difference between local error estimates and bonus estimates practice are given in Appendix B.2 2 and Appendix B.3
6 EXPERIMENTAL RESULTS
Our experiments are designed to provide us with better understanding of how PE-DQN operates, in comparison to related algorithms as well as in relation to its algorithmic elements. To this end, we aimed to keep codebases and hyperparameters between all implementations equal up to algorithm- specific parameters, which we optimized with grid search on a selected subsets of problems. Further details regarding the experimental design and implementations are provided in Appendix B
We outline our choice of baselines briefly: Bootstrapped DQN with prior functions (BDQN+P) (Osband et al.] 2019) approximates posterior sampling of parametric value function by combining statistical bootstrapping with additive prior functions an ensemble of DQN agents. Information-
directed sampling (IDS-C51) (Nikolov et al. 2019) builds the BDQN+P architecture but acts
7

---

Published as a conference paper at ICLR 2024
according to information- ratio for which Nikolov et al. (2019) estimate aleatoric uncertainty (noise) with the categorical CS1 model. In contrast, Decaying Left- Truncated Variance (DLTV) QR-DQN (Mavrin et al., 2019) uses a distributional value approximation based on the quantile representation and follows a decaying exploration bonus of the left-truncated variance.
6.1 Do DIFFERENT PROJECTIONS LEAD TO DIFFERENT GENERALIZATION BEHAVIOR?
staues (nn (d) Ep.
First, we examine empirically the influence of the projec- tion step deep distributional RL on generalization be- haviors. For this, we probe the influence of the quantile and categorical projections on generalization through an experiment that evaluates exploration in reward-free setting. Specifically, we equip agents with an action- selection rule that maximizes a particular statistic S[Z] of the predictive distribution fj(s, a) according to
Categonical Quaatill
0.5
Mean
Skew
Kunt
Statitic
Figure Deep-sea exploration with dif- ferent statistics. Higher means more ex- ploration. Bars represent medians and in- terquartile ranges of 30 seeds.
arg max (S[Z)) Z f((s,
acA
The underlying idea is that this selection rule leads to exploration of novel state-action regions only high values of the statistic are correlated with high epistemic uncertainty. For example, if we choose quantile representation with $ / to be the variance of the distribution, we recover basic form of the exploration algorithm DLTV-QR (Mavrin for the first four statistical moments on the deep exploration benchmark deep sea with size 50. Except for the mean (the greedy policy), the choice of projection influences significantly whether the statistic-man maxitiiiiiiiiiig policy leads to more exploration, implying that the generalization behaviour of the 2nd to 4th moment of the predictive distributions is shaped distinctly by the employed projection.
al., (019). Fig. 3 shows the results of this study
6.2 THE BEHAVIOUR SUITE
In order to assess the learning process of agents in various aspects on a wide range of tasks, we evaluate PE-DQN on the behavior suite (bsuite) (Osband et al., 2020), battery of benchmark problems constructed to assess key properties of RL algorithms. The suite consists of 23 tasks with up to 22 variations in size seed, totaling 468 environments.
Comparative evaluation. Fig. (a) shows the results of the entire bsuite experiment, summarized in seven core capabilitiess These capability scores are computed as proposed by Osband al. (2020) and follow handerafted scoring function per environment. For example, exploration capability scored by the average regret the sparse-reward environments deep sea, stochastic deep sea, and cartpole swingup. The full set of results is provided in Appendix B Perhaps unsurprisingly. PE-DQN has its strongest performance in the exploration category but we find that it improves upon baselines in several more categories. Note here that PE-DQN uses substantially fewer models than the baselines, with total of 4 distributional models compared to the 20 DQN models used in the ensembles of both BDQN+P and IDS, where the latter requires an additional C51 model.
6.3 THE DEEP-SEA ENVIRONMENT AND ABLATIONS
Deep sea is hard exploration problem in the behavior suite and has recently gained popularity an exploration benchmark (Osband et al. 2019 Janz et al. 2019 Flennerhag et al. 2020). It is a sparse reward environment where agents can reach the only rewarding state the bottom right of an N N grid through a unique sequence of actions in an exponentially growing trajectory space. We ran an additional experiment on deep sea with grid sizes up to 100; double the maximal size in the behavior suite. Fig. (b) shows a summary of this experiment where we evaluated episodic regret, that is the number of non-rewarding episodes with a maximum budget of 10000 episodes. PE-DQN scales more gracefully to larger sizes of the problem than the baselines, reducing the median regret by roughly half. The r.h.s. plot in Fig. (b) shows the results of ablation studies designed to provide a more nuanced view of PE-DQN's 's performance; the baselines labeled PE-DONIQR/QRI and PE-DQNIC5 I/C51] use the same bonus estimation step as PE-DQN except that ensemble members consist of equivalent models with the same projections and representations. Conversely, PE-DQN [Ind.] uses PE- PE-DQN's diverse projection ensemble and employs an optimistic oction-selectio directly with the ensemble disagreement Wavg (s,a) a) but trains models independently and accordingly does not make use of an
8

---

Published as a conference paper at ICLR 2024
Exploratio
Crodit Assignment
Hoseline Comparisoos
Ablaticn Studies
Ep regres (in 1e3) @ 10000 Fip
10.0
9.0
IDS CSI
8.0
7.0
5.0
DLIV-QR
4.0
PF-DIO (QR/QR]
Memury
3.0
2.0
1.0
Scale
PE-DQN [[nd.]
Nobe
10
25
do
55
70 85
100
10
25
40
55
70
100
Deep-sea
N
Figure 4: (a) Summary of bsuite experiments. Wide is better. (b) Median episodic regret for deep sea sizes up to 100. Low is better. Shaded regions are the interquartile range of 10 seeds.
Environment
Dasse
Sparse
Very Sparse
110
Episodic relurns
DOJN
0.5
DETV-QR
BDQN+P
0.0
PE-DON
0.5
1.0
0.0
0.5
1.0
0.5
1.0
1.5
MyWayHome (a)
(b) Environment interactions
10")
Figure 5: (a) Visual observation in the VizDoom environment (Kempka et al. 2016). (b) Mean learning curves different variations the My WayyHome VizDoom environment. Shaded regions are 90% Student's confidence intervals from 10 seeds.
uncertainty propagation scheme in the spirit of Theorem 3 Both components lead to pronounced difference in exploration capability and rendered indispensable PR-DQN's overall performance.
6.4 THE VIZDOOM ENVIRONMENT
We investigate PE-DQN's behavior in a high-dimensional visual domain. The VizDoom environment MyWayHome (empka al., 2016) tasks agents with finding (rewarding) object by navigating in a maze-like map with ego- geo-peespeeetve pixel observations as seen in Fig. 5) (a). Following work by Pathak et al. (2017), run three variations of this experiment where the reward sparsity increased by spawning the player further away from the goal object. Learning curves for all algorithms are shown Fig. (b). Among the tested algorithms, only PE-DQN finds the object across all 10 seeds in all environments, indicating particularly reliable novelty detection. Interestingly, the sparse domain proved harder to baseline algorithms which we attribute the "forkedness" the associated map (see Appendix B This result moreover shows that diverse projection ensembles scale gracefully to high-dimensional domains while using significantly fewer models than the ensemble- based baselines.
7 CONCLUSION
In this work, we have introduced projection ensembles for distributional RL, method combining models based on different parametric representations and projections of return distributions. We provided a theoretical analysis that establishes convergence conditions and bounds on residual approximation errors that apply to general compositions of such projection ensembles. Furthermore, we introduced general propagation method that reconeiles one-step distributional TD errors with optimism- based exploration. PE-DQN, deep RL algorithm, empirically demonstrates the efficacy of diverse projection ensembles on exploration tasks and showed performance improvements on wide range of tasks. We believe our work opens up number of promising avenues for future research. For example, we have only considered the use of uniform mixtures over distributional ensembles in this work. A continuation of this approach may aim to diverse collection of models less conservatively, aiming to exploit the strengths of particular models in specific regions of the state-action space.

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGEMENTS
We thank Max Weltevrede, Pascal van der Vaart, Miguel Suau, and Yaniv Oren for fruitful discussions and remarks. The project has received funding from the EU Horizon 2020 programme under grant number 964505 (Epistemic AI). The computational resources for empirical work were provided by the Delft High Performance Computing Centre (DHPC) and the Delft Artificial Intelligence Cluster (DAIC).
REFERENCES
T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2019.
Auer. Using confidence bounds for sxploitation-exploraiiii trade-offs. Journal of machine learning research, 3, 2002.
M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective reinforcement learning. In International conference on machine learning. PMLR, 2017.
M. G. Bellemare, W. Dabney, and M. Rowland. Distributional reinforcement learning. MIT Press, 2023.
R. Bellman. A Markovian decision process. Journal of mathematics and mechanics, 6, 1957.
Y. Burda, H. Edwards, D. Pathak, A. J. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity-driven learning. In International Conference on Learning Representations, ICLR, 2019a.
Y. Burda, H. Edwards, Storkey, and O. Klimov. Exploration by random network distillation. In International conference on learning representations, ICLR, 2019b.
R. Chen, S. Sidor, P. Abbeel, and J. Schulman. UCB exploration via Q-ensembles. arXiv preprint arXiv:1706.01500 2017.
W. R. Clements, B. Van Delfi, B.-M. Robaglia, R. B. Slaoui, and S. Toth. Estimating risk and uncertainty in deep reinforcement learning. arXiv preprint arXiv:1905. 0965009880. 2019.
W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning. PMLR, 2018a.
W. Dabney, M. Rowland, M. Bellemare, and Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018b.
Delft Artificial Intelligence Cluster (DAIC), 2024.
Delft High Performance Computing Centre (DHPC). DelftBlue Supercomputer (Phase 1), 2022.
A. Der Kiureghian and O. Ditlevsen. Aleatory epistemic? Does it matter? Structural safery, 31, 2009.
T. G. Dietterich. Ensemble methods in machine learning. In Multiple classifier systems: First international workshop, MCS. Springer, 2000.
H. Eriksson, D. Basu, M. Alibcigi, and C. Dimitrakakis. Sentinel: based distributional reinforcement learning. In Uncertainty in artificial intelligence. PMLR, 2022.
Taming uncertainty with ensemble
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-RI. with importance weighted actor-learner architectures. International conference on machine learning. PMLR, 2018.
M. Fellows. K. Hartikainen, and S. Whiteson. Bayesian Bellman operators. Advances in neural
information processing systems, 34, 2021.
10

---

Published as a conference paper at ICLR 2024
S. Flennerhag, J. X. Wang. P. Sprechmann, F. Visin, A. Galashov, S. Kapturowski, D. L. Borsa, N. Heess, A. Barreto, and R. Pascanu. Temporal difference uncertainties as a signal for exploration. arXiv preprint arXhv:2010.02255. 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level per- formance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, 2015.
C.-J. Hoel, K. Wolff, and L. Laine. Ensemble quantile networks: Uncertainty-aware reinforcement learning with applications in autonomous driving. IEEE Transactions on intelligent transportation systems, 2023.
S. C. Hora. Alcatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability engineering & system safety, 54, 1996.
O. lbe. Fundamentals of Applied Probability and Random Processes. Elsevier Science, 2014.
D. Janz, J. Hron, P. Mazur, K. Hofmann, J. M. Hernández-Lobato, and S. Tschiatschek. Successor uncertainties: Exploration and uncertainty in temporal difference learning. Advances in neural information processing systems, 32, 2019.
Y. Jiang, J. Z. Kolter, and R. Raileanu. On the importance of exploration for generalization in reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
C. Jin, Z. Allen-Zhu, S. Bubeck, and I. Jordan. Is Q-learning provably efficient? Advances in neural information processing systems, 31, 2018.
C. Jin, Z. Yang. Z. Wang, and M. I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Proceedings of Thirty Third Conference on Learning Theory, volume 125. PMLR, 2020.
M. Kempka, M. Wydmuch, G. Runc, J. Toczek, and W. Jaskowski. ViZDoom: A Doom-based based Al research platform for visual reinforcement learning. In IEEE Conference on computational intelligence and games. IEEE, 2016.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, International conference on learning representations, ICLR, 2015.
R. Koenker and K. Hallock. Quantile regression. Journal of economic perspectives, 15, 2001.
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems. 30, 2017.
Lindenberg, J. Nordqvist, and K.-O. Lindahl. Distributional reinforcement learning with ensembles. Algorithmms, 13, 2020.
E. Mariucci and M. Reiß. Wasserstein and total variation distance between marginals of Lévy processes. Electronic journal of statistics, 12, 2018.
B. Mavrin, H. Yao, Kong, K. Wu, and Y. Yu. Distributional reinforcement learning for efficient exploration. In International conference on machine learning. PMLR, May 2019.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518, 2015.
T. M. Moerland, J. Broekens, and C. M. Jonker. Efficient exploration with double uncertain value networks, arXiv: [cs, stat], Nov. 2017.
Morimura, M. Sugiyama. H. Kashima, H. Hachiya, and T. Tanaka. Nonparametric return distribu- tion approximation for reinforcement learning, In International conference on machine learning. PMLR, 2010.
Nguyen- Tang. S. Gupta, and S. Venkatesh. Distributional reinforcement learning via moment matching. Proceedings of the AAAI conference artificial intelligence, 35, May 2021.
11

---

Published as a conference paper at ICLR 2024
N. Nikolov, J. Kirschner, F. Berkenkamp, and A. Krause. Information-directed exploration for deep reinforcement learning. In International conference on learning representations, ICLR, 2019.
I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN. Advances in neural information processing systems, 29, 2016.
I. Osband, B. Van Roy, Russo, Z. Wen, et al. Deep exploration via randomized value functions. Journal of machine learning research, 20, 2019.
I. Osband, Y. Doron, M. Hessel, J. Aslanides, E. Sezener, A. Saraiva, K. McKinney, T. Lattimore, C. Szepesvári, S. Singh, B. V. Roy, R. Sutton, D. Silver, and H. van Hasselt. Behaviour suite for reinforcement learning. In International conference learning representations, ICLR, 2020.
B. O' O'Donoghue, I. Osband, R. Munos, and V. Mnih. The uncertainty Bellman equation and explo- ration. In International conference on machine learning. PMLR, 2018.
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self sulf-super prediction. In International conference on muchine learning. PMLR, 2017.
M. Rowland, M. Bellemare, W. Dabney, R. Munos, and W. Tch. An analysis of categorical distributional reinforcement learning. In International conference artificial intelligence and statistics. PMLR, 2018.
M. Rowland, R. Dadashi, S. Kumar, R. Munos, M. G. Bellemare, and W. Dabney, Statistics and samples in distributional reinforcement learning. In International conference on machine learning. PMLR, 2019.
T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. In Y. Bengio and Y. LeCun, editors, International conference on learning representations, ICLR, 2016.
D. Schmidt and T. Schmied. Fast and data- training of rainbow: An experimental study on Atari. arXiv preprint 1rXiv:2111.10241 2021.
D. J. White. Mean, variance, and probabilistic criteria in finite Markov decision processes: A review. Journal of optimization theory and applications, 56, 1988.
D. Yang, L. Zhao, Z. Lin, T. Qin, J. Bian, and T.-Y. Liu. Fully parameterized quantile function for distributional reinforcement learning. Advances neural information processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2024
A
APPENDIX
This section provides proofs for the theoretical claims and establishes further results on the residual approximation error incurred by our method.
A.1 PROOF OF PROPOSITION I
Before stating supporting lemmas and proof's of the results in Section 4 we recall several basic prop- erties of the p- Wasserstein distances which we will find useful in the subsequent proofs. Derivations of these properties can for example be found in an overview by Mariucci and Reils (2018).
P.1 The p- Wasserstein distances satisfy the triangle inequality, that is (x, 2) + wy(Z, Y) (X((X, Y) S (p(X,
P.2 For random variables X and Y and an auxiliary variable Z independent of X and Y, the p- Wasserstein metric satisfies the inequality
wp(X + Z,Y + 2) S (X,
P.3 For a real-valued scalar a R. have
up(aX, aX) = lalw>(X,X)
Lemma Let ΣE= tv,, v/ E. 1/ be two mixture distributions v, v.v/ E (R)). Furthermore denote Wil(V, v)) the p- p-Wasserstein metric between and v'. Then wo satisfies
uhfu, v') S M _ Eej(us vt).
Proof. The Wasserstein distance in its general form is expressed in terms of couplings between the probability measures v and according to
(p(,, = inf mEI(eee))
((..))~all - //2///2
where [(v, v/) is the set of all couplings between v and v', i.e. joint distributions on (R?) with marginals and v/. Now suppose for each we have coupling ps (a,, y) E [(vi, 14) such that
B(z.))np [ - wl'l = inf ((an)~=le y/' wp((,,
nET(s,,)))
Since by definition pı(z,y) is a coupling of vs and v/, the mixture of couplings pa M
Mr 2) is then valid coupling of and v', as J pllz v)dy v(x) and paæ', y)dz' can thus write
inf pellow
Beaws~all yl
B(ay))all y11"]
Σ
M E Beeniinille - vl
Σ wuf(uun
Proposition 1 Let II,, i [L..,, M] be projection operators 11, : PR) + F. mapping from the space of probability distributions F (R) to representations F; and denote the projection mixture operator Jas P (RR) + FE as defined in Eq. Furthermore, assume that for some p E [1, 20) each projection II, is bounded in the p- --Wassrsstein metric in the sense that for any return distributions n. Tf we have wo (IIm,, Hay)) w,(lI,m
(s, a) I w. (7, 7/ (s, a) for a constant c. Then, the combined operator 2xTT is bounded in the supremum p- Wasserstein distance wp by
#p(0 TT 1), 22s T=11') S (1,10111
and is accordingly a contraction long 7 < 1, where Cp
(EM M
Ei=l 17 L/V.
13

---

Published as a conference paper at ICLR 2024
Proof. Due the assumption of the proposition, we have (p I1,1/) & cxpy(u,)).
With
Lemma and the y-contractivity of T", it follows that
(f (([@MT"nn.. 9, 1M J. Σ I.
Σ (.T''
- M
Σ
T 17. I1,
$ M
11
= ,n m/) E.
The state then finally follows by taking the p-th root, yielding the joint modulus Ep = -EE M 1/D.
A.2 PROOF OF PROPOSITION 2
Proposition 2 Let (s, B[2(s, a)] be a state-action value estimate where (s, a) i a) is a random variable distributed according to an estimate i)(s, a) of the true state-action re- turn distribution (s, a). Further, denote Q"
E (s, a)) the true state-action, where 7 (8 a) ~ 17 (s, a). We have that Q" (s, a) is bounded from above by (s,,a) + w (f, (")n")) ) (s,a) a) 2 (s, V(s,a) ESx,
where wi is the 1- L-Wasserstein distance metric.
Proof. We begin by stating a property that relates the expected value E[X] the CDF of X under the condition that the expectation E[X] is well-defined and finite. The property is an extension to the property of the expectation of nonnegative variables which itself is a consequence of Fubini's Theorem (see for example lbe 2014 for this). Let X ~ and write F, for the CDF of v. then:
[[
(1 (1- (1=
Fx)ddr.
Now, suppose an auxiliary variable X' is distributed according to the law v. It then follows that
(zv(z)- -F((=)dxx 1 (Fa dx)
E E[X]
[[
Fv( Fu(x)dx
(x) (x)
where the last step was obtained by a change of variables in the definition of the 1-Wasserstein distance:
wi (v,1)) = [ F, Fr (T) F (T)
Fx(z) Fzr (z)
The result of Proposition 2 is obtained by rearranging.
A.3 PROOF OF THBOREM 3
Before stating the proof of Theorem 3 we formalize the notion of pushforward distribution which will useful in more explicit description of the distributional Bellman operator 7 Our notation here follows the detailed exposition by Bellemare et al. (0233)
14

---

Published as a conference paper at ICLR 2024
Definition 5 For function f R. - R. and a random variable Z with distribution D(Z), V E (R), the pushforward distribution f E (RR) of through f is defined as
1 (f (B)), AB ((R).
where B are the Borel subsets of R.
Equivalently to Definition 5 we may write sµv (2). By defining bootstrap transformation br, IR R with b,,y + ty,, we state explicit definition of the distributional Bellman operator T according to Definition
Definition (Distributional Bellman Operator (Bellemare al. 2017) J The distributional Bellman operator T ((R)SXA (RR))xx is given by (s) (bRo.))
) ( A = Ao =a
where P(-180 = Ao = =a), A (-1Si).
Lemma Let ( E (R) be the pushforward distribution of ( through ben IR + R. Then we have two distributions v, v' and the 1-Wasserstein distance wi that w/ (Brr) vV, ( = -yun (v, v/).
Proof. The proof follows from the definition of the 1- I-Wasserstein distance. Let Z ~ v and Z' ~ v be two independent random variables, then
wi ((bn)) uV (br)) wi (D(r + D(),
P-1 .. (Punnau(T))
af u (T) dr
Theorem 3 Let ij(s,a) E (K) be estimate of the true return distribution nf (s, a) (R), and denote the projection mixture operator Qy : (R) + f with members II, and bounding moduli c, and Cp defined in Proposition 1. Furthermore, assume 2MrT" is contraction mapping with fixed point ME. We then have for all (s,a) E x A
(s, + + me) where S₁ P(-1S = s, Ao = a) and A1 7-1S2).
wi (f) n (s,a) S wi
( Ai) = s, = =a
Proof. Since (8, a) is the fixed point of the combined operator 2MT"" we have that OM T (s. = n% (s, a). From the triangle inequality follows that wi (ń, 73) (s, a) wi Q Furthermore, for the second term on the r.h.s. in Eq. 14 the following holds:
Rus T 1)) (s, a) wn (2x T. T
TT) (8,
(14)
wi (RMT Aii TT (s,
wi
11
I m (s,a)
I J Σ qwn (T
T (8,a)
m (s, a).
Under slight abuse of the assumptions in Section we here consider an immediate reward distribution with finite support R to simplify the following derivation. In this case, we can write out the expectation in Definition Blas
(T*))(s.a)= Σ' Σ Pr(Ro = Ai = = - s'180 Ao a) ((br,))) Mi'', fisssaa)), aa)),
PER FERN'E8n'EA
where Pr(-) is the joint probability distribution given by the transition kernel P(-|s, a). the immediate reward distribution R(- a), and the policy π x--1S).. Thus, by Lemma 4 and Lemma it follows that
T (T i, 7 mit (8)
< E E W1 (bR.. (S,, A1), (bRo.>) (T (
A1))
s, = =]
L w1 (7) 7/ (S A1) So s, Ao = a)
where S₁ P(-1S. 8, Ao a) and A (-|S"). The proof is completed by rearranging.
15

---

Published as a conference paper at ICLR 2024
A.4 RESIDUAL EPISTEMIC UNCERTAINTY
Due to limitation to inite-dimensional representations and the of varying projections, algorithm incurs residual approximation errors which may not vanish even convergence. In the context of epistemic uncertainty quantification, this is unfortunate it can frustrate exploration or lead to overconfident predictions. Specifically, the undesired properties are twofold: 1) Even in convergence, the fixed point T does not equal the return distribution (bias). 2) Even in the fixed point ne. the ensemble disagreement Wavg does not vanish. Often, however, we may be able to upper bound and control the error incurred due to the projections II, In this case, Propositions 8 and provide upper bounds on both types of errors function of bounded projection errors.
Proposition 8 Let Ωm be a projection mixture operator with individual projections II, defined as in Eq. 77) Further, assume each projection II, is upper bounded by w. v) I d, for some p [1, oo). Then, the p- p-Wasserstein distance between the fixed point 7E (s,a) = f2 (s.a)
and the true return distribution (8, = TT=1" (8, u) satisfies
M I M
dp I-EØT V(s,a) ESXA,
where
dp =
Proof. To show the desired property, will make use of Proposition D the dependency on (s, a) in this section for brevity. It follows then from the triangle inquuality that Wp (1E n² "²" wp(a 7 T "E Aasn")
and Lemma We omitted
(
=w T TE
T Sts 7 [)" + (SMmT
&p 11) + up(
(1 M
Σ ,77"
S Scp 77
G Σ (I,," . ))
Per the assumption of Proposition 8 and by rearranging we obtain the desired result.
Proposition 9 Let Wavg be the average ensemble disagreement given by MMM Zijj=1 Wp (h ît) and assume individual projections II are bounded by wp (I,,, v) S d, For an ensemble E whose mixture distribution equals exactly the fixed point mi (s. = T mg (s, a), the average ensemble disagreement Wang satisfies the inequality
M M Σ
Wavg (s. S
2M d ((s.0) M-1
where d
d2
Proof. the fixed point (s, a) QN a), the distributional error estimated by Wavg (s. a) does vanish, unlike the ground truth error wi (nt 2M 7T (s, = O. The shown property upper bounds this mismatch and is direct consequence of the assumption (IT,,, v) < d, which postulates an upper bound on the error introduced by the projection II; in terms of the p- Wasserstein distance. The average disagreement is given by
Wavg (8,a) MCM
(p i (s) 4==1
The proof is given by applying the triangle inequality and the assumption of the proposition with (p ( = wp (T (mng))
< w. (Un TE) + wp 1,
< d; + +
Plugging in and rearranging yields the desired result.
Lemma [Projection error of the categorical projection (Rowland et al. 2018 ] For any distribu- tion v E P ([zmin Emax)) with support on the interval [zminy Zmax. and categorical projection as
defined in Eq. (5) with K atoms 2ạ E [21, 2x] S.I. 21 2 Zemin and ž S Zmax. the error incurred by the projection Ie is upper bounded in the 1- Wasserstein distance by the identity
wi wi (ToC,, S sup (IK+1 - =x
ISkSK
16

---

Published as a conference paper at ICLR 2024
Proof (restated). The proof uses the duality between the 1- 1-Wasserstein distance and the 1-Cramér distance stating
41,,v')= - / [F((x)
/ (t) - w1(,v'
For (x)
and be obtained by change of variables. The 1 formulation simplifies the analysis of the categorical projection, yielding
W1 (oovvv))
FFlou()) - Fu(x))dr
(2k+1 =x)]F(1ou(E)) (z)2)]
x
Σ k=
(#t+1 2xx 2x) - Fe(=x)
$ sup (#++1 -x)]
Σ
F
IskSk
< sup (2k+1 -xx]l-
IskK
Lemma 11 [Projection error of the quantile projection (Dabney et al. 2018b For any distribu-
tion v E (2min, with support on the interval (zuin; 2max] and a quantile projection defined
according Eq. (6) with K equally weighted locations 0g E [0,,... 0k) the error incurred by the projection 1lq is bounded in the I-Wasserstein distance by the identity
Zmax
Zmin
W1 (IQV,
K
Proof (restated). The projection 11 is given by
Tg = E FF (ra)
where Tы.
2k-
2K
The desired identity wi (IlQ v) is accordingly given by the continuous integral
wi (1 /. (T) F,
and can be rewritten terms of sum of piecewise expectations
wi (1Qv v) From this, it follows that
Σ E IIX F, 2K
(2-1(22-1))
w1 (IlQ,, v)
s (F) (0))
Zmax Zunin
K
Let (s, a) be the fixed point return distribution for an ensemble of the categorical Corollary 12
and quantile projections with the mixture operator Q 1/2 11//llgnss.. a) + 1/2 (l//ll a). Furthermore, suppose the return distribution Ti (s, a) has bounded support on the interval (Rmax R1 7) where Rmax and Rmin denote the maximum and minimum immediate reward of the MDP. The average ensemble disagreement Wovg (s, a) is then bounded by
4( A(Rmax Rmmnn)
Wavg (5, a) S
(1-7)K
Proof. The result follows straightforwardly from Proposition 9 and Lemmas 10
17

---

Published as a conference paper at ICLR 2024
Table
Hyperparameter search space for bsuite
Hyperparameter
Values
Neural net architecture
[[64, 64], [128, 128), [12]]
Learning rate
[ 10-6 1 10 ,,5 10 1 10-3)
Prior function scale
5.0, 20.0]
Heads K
[51, 101]
Initial bonus 3
[0.5, 5.0, 50.0]
Table 2: Hyperparameter search space for VizDoom
Hyperparameter
Values
Learning rate
[1.25 x 10 2.5 x 10 3.7.5 x 10
10 6.25 10 5,7. x 10 10
Prior function scale Initial bonus
[1.0,30, 5.0]
[0.05, 0.1, 0.5, 5.0]
A.5 THE CATEGORICAL PROJECTION
The full definition of the categorical (or also Cramér) projection stated by Rowland et al. 2018 is given below.
Definition 13 [Categorical projection (Rowland et al. 2018) ] For set fixed locations ...... 2K be the hat function centered around zę for
where 21 22 <... < 2K, let lz.
0, 1]
k == 1,.... K given by
21+1
for x. E [Ek, #k+1]
and 1 < k < K,
for x E [2k=1,2k]
and 1 < k S K,
hzg (x)
for x S 21
and k = 1,
for x 2 2K otherwise.
and l: K,
Furthermore, let the categorical representation Fo be defined as a finite mixture of Dirac deltas 0% 13. The categorical projection operator IIc (R) + Fc of distribution P (TR) then defined
Fc = (Ex,,0 0rg=z 10x 2 0, Ek-
8 Bumolbba (w:))2x
Nev Σ h-1
B EXPERIMENTAL DETAILS
We provide a detailed exposition of our experimental setup, including the hyperparameter search procedure, hyperparameter settings, algorithmic details, and the full bsuite experimental results.
B.1 HYPERPARAMETER SETTINGS
In our experiments, we aimed to keep most hyperparameters between different implementations equal to maintain comparability between the analyzed methods. Algorith hyperparameters were optimized over a search space of hyperparameters using Optuna (Akiba al. 2019). The total search space for bsuite and VizDoom parameter only applies to distributional algorithms. C51 requires us to define return ranges, which defined manually and can be found in the online code repository. All algorithms use the Adam optimizer (Kingma and Ba 2015).
given in Table I and Table respectively. where the Heads K
Bsuite. For bsuite, the hyperparameter search was conducted on a subselection of environments of the bsuite, shown in Table 3 For each environment, evaluate of hyperparameters by
18

---

Published as a conference paper at ICLR 2024
Table 3: Hyperparameter search environments
Environment ID
Horizon in no, of episodes
Scoring function f
dcep_sea/20 sea/20
500
E(e,0) visited (s, (..0) I visited (s, a)
deep_sea_stor 2toosssssst
1500
mountain_car/
100
means of a scoring function. A particular set hyperparameters evaluated every T/5 episodes with a maximum training horizon of 7 episodes. The "continuous" scoring functions make the hyperparameter search more amenable to pruning, for which we use the median pruner of Optuna, reducing the computational burden of the combinatorial search space significantly.
Here, [ss,) I visited (s, a) is the count of visited state-action tuples and (-1) is simply the negative number of total environment interactions. For every hyperparameter configuration Çi. the scores f(G) are calibrated to facilitate a meaningful comparison between different environments. The calibrated score function
is given by
fe(C:) = exp (0.693 f(()) Ms -(s))
(15)
sup; f(G)) PC
where jlç is the average score of all hyperparameter configurations Pk Σ 1/Nf(c.). and sup; f({)) is the maximal score achieved. The calibration function in Eq. (T5) was chosen heuristically to have an intuitive interpretation: it assigns a score of the best-performing hyperparameter configuration, 0.5 to configurations that achieve exact average performance, and decays exponentially according to score. The final score assigned to a hyperparameter configuration C is the sum of all scores of the tested environments. Table shows the full set of hyperparameters used for every algorithm.
VizDoom. For the VizDoom domain, the hyperparameter search was conducted on the My WaytlomeeSursee-u variation with training budget of million frames, where final config- urations were chosen by achieved return at the end of training. Due to the sparsity of the prob- lem, did not make use of pruning algorithm. The specific difference between the different variations of the VizDoom environment MyWay Home are shown in Fig. 6 where the sparsity of the problem increased by changing the agents spawning location to room further from the goal position. The network architecture is based large extent the rainbow network pro- posed by [Schmidt and Schmied (202I) who in turn base their architecture on IMPALA (Espe- holt et al.] 2018). The specific algorithm contig- uration for VizDoom given in Table 3 with schematic of the network architecture shown in Fig- g T our preprocessing pipeline used for the Vozmoom environments.
Room 13 sparse
O
Goal
Room 17
very sparse
Figure 6: Map for the VizDoom MyWayHome en- vironment. Agents spawned the sparse and very sparse locations to vary the exploration diffi- culty.
B.2 IMPLEMENTATION DETAILS
Parametric model. Our parametric model is mixture distribution TJE parametrized by 0. We con- struct 77.9 as an equal mixture between a quantile and a categorical representation, each parametrized through NN with K output logits where use the notation 0% to mean the k-th logit of the network parametrized by the parameters 0; of the i-th model in the ensemble, We consider a sample transition (s, a, 8', a') where a' is chosen greedily according E. Ez [Z]. Dependencies
19

---

Published as a conference paper at ICLR 2024
Table 4: Hyperparameter settings bsuite
Hyperparameter
BDQN+P
DLTV
IDS
PE-DQN
Net architecture
[64, 64]
[512]
[64, 64 [512]
[512]
Adam Learning rate
10-3
10
10-3 10-
5 x 10-4
Prior function scale
5.0
20.0
20.0/5.0
20.0
Heads K
101
1/101
101/101
Ensemble size
20
1
20/1
2/2
Initial bonus Bini
n/a
5.0
5.0
5.0
Final bonus Briaa
n/a
n/a
5.0
5.0
Bonus decay (in eps)
n/a
10 /Noo/ssssdsss episodos
0.33 Nopisodss
0.33 X Nepisodes
Discount
0.99
Buffer size
10, 000
Adam epsilon
0.001 /bacch size
Initialization
He truncated normal He et al. 2015
Update frequency
Target update step size
1.0
Target update frequency
4
Batch size
128
Table Hyperparameter settings VizDoom
Hyperparameter
BDQN+P
DLTV
IDS
PE-DQN
Adam Learning rate
2.5 x 10-5
7.5 x 10-5
2.5 x 10-5
6.25 x 10-5
Prior function scale
1.0
3.0
1.0
3.0
Heads K
1
101
1/ 10101
101/101
Ensemble size
10
10/1
2/2
Initial bonus 8mn
n/a
0.5
0.1
5.0
Final bonus Fnnal
n/a
n/a
0.01
0.01
Bonus decay (in frames)
n/a Huber
103 N QR-Huber
0.33 Nirames
0.33 X Nitames
Loss function
Huber/C51
QR-Huber/C551
Initial e in e-greedy
1.0
Final 6 in e-greedy
0.01
decay time
500, 000
Training starts
100, 000
Discount
0.997
Buffer size
1, 000, 000 512
Batch size
Parallel Envs
32
Adam epsilon
0.005 /bthh/ size
Initialization
He uniform (He et 2015
Gradient clip norm
TV
Regularization
spectral normalization
Double DQN
Yes 1
Update frequency
Target update step size
1.0
Target update frequency
8000
PER 30
0.45
n-step returns
10
20

---

Published as a conference paper at ICLR 2024
Table 6: VizDoom Preprocessing
Parameter
Value
Grayscale
Yes
Frame-skipping
No
Frame-stacking
Resolution
42 x 42 2100
Max. Episode Length
on (s,a) are dropped for conciseness by writing 0u((s,)) a) = Ojk. and (s(,''a)) = O2k The full mixture model 11E,9 is then given by
1/F,0 Σ
Ad-2
with p(OL))= =(01k)=00kk (p(0))0(www)
(16)
p(0bs)=o(0aa) ((0_=))))))))())))_=_=)))))))))) =(00=)==k:
where (x;) eth /E//E, e is the softmax transfer function. Consequently, this representation
comprises total of 2K atoms, K of which parametrize locations in the quantile model, and the remaining K parametrizing probabilities in the categorical representation. The losses used for each projection method are as provided the main text.
Distributional estimation of bonuses. For the parametric bonus estimate (s a) we use the same procedure for learning distributional projection ensemble as with extrinsic rewards. Note that it is necessary for our method learn a distributional estimate of the bonus but find that diverse projection ensembles are good value learners in general and simply reuse the existent function approximation machinery for an intrinsic reward instead of the extrinsic reward. We thus have model of parameters é trained with an alternate tuple (s, Wavg s', d'),, where replaced the immediate reward with the ensemble disagreement Wavg and a' is an exploratory action chosen according to the rule
a' = arg max (Ez~ [2] + (s, a)), where (s(s,a) = E B (B.a)
(17)
aEAA
Here, 3 is a hyperparameter to control the policy's drive towards exploratory actions.
Pseudocode. We provide pseudocode for a basic version of PE-DQN where we have simplified details such as the previously described distributional estimation of by a), prioritized replay, double Q-lcarning, and prior functions for clarity.
Randomized prior functions added all baselines and PE-DQN. Specifically, we add the output fixed, randomly initialized neural network of the same architecture as the main net, scaled by hyperparameter, the main network's 's logits. In the case of C51, the prior function is added pre softmax. To the best of our knowledge, DLTV-QR does not use prior functions in its original formulation but find it to be crucial in improving exploration performance. Fig. (b) shows experiment assessing the exploration performance of DI.TV-QR with randomized prior functions and prior scale 20 (DLTV [rpf20]) compared to the vanilla implementation without priors (DLTV (rpf0J).
information-gall gain in our IDS implementation for bsuite is computed in a slightly modified way compared to the vanilla version. Nikolov al. (2019) compute the information gain function [(s, a) with
(x
T(s,a) = log
62,
where (s) is the empirical variance of BDQN+P predictions, = 1 x 10-5 is zero-division protection, and p²(ss, , a) the clipped action- spacce normalized return variance
Var ((s,a)
p(s, a)² - max
0.25
(18)
A E.EA Var(Z(s,a))
21

---

Published as a conference paper at ICLR 2024
Algorithm PE-DQN
quantile model with parameters 0,, target parameters 01, and K heads
2: categorical model with parameters 02, target parameters 02, K heads, and grid ;
3: bonus estimation model with parameters Φ and target parameters Ø
4: exploration parameter 3, learning rate
5: initialize Buffer
6: sample initial state so
7: for = 0,..,,
.o....,T T do predict locations [011, 01k ($2, a) and probabilities [021, ..... 022 a)
8:
9:
Q(st, a) == - 01x((1 81, a) A + (2),
10:
predict bonus (8t,
11:
a, + arg max.aaxaa (s,,, a) + (bb(st,
12:
for j O.... Nurinsttps do
13:
sample transition tuple (sj, aj ry 55)) ~ B
14:
predict locations [f11..... k (sy, az) and probabilities [021..... 02x((,,,,)),,
15:
predict target locations 10n..... 01K ( a) and probabilities (021.... 02x (2x](s').a)
16:
(( == Ek (5, 1/4 + (s) a)
17:
aj + arg max. (Q(ss, })}
mixture target 1/1s 1
2 EK
18 10 + 12 (e ory
19:
quantile loss (01, 1/1))
20:
categorical loss L2 + Lc Thss)
K
wasserstein distance Tiner + wi (2 1 d0
Σ (2) a;))
22:
bonus estimation target + rintr + ,bg(b), al)
23:
bonus estimation loss La < (s) az): B/
24:
02, 65 + [0,, 02, 6)2T Vo (1 + 1a la)
25:
end for
26:
execute a, and store (s,, ae, Te, 8c+1) in
27: end for
22

---

Published as a conference paper at ICLR 2024
Table 7: VizDoom wall clock time comparisons
Environment
BDQN+P
DLTV
IDS
PE-DQN
My WayHome - Dense
14h 35m
14h 22m
16h 49m
17h 3m
My WayHome- Sparse
14h 29m
13h 49m
16h 1 11m
16h I 11m
My Myyee Very Sparse
21h 27m
21h 12m
23h 3m
23h 3m
Var (s, a)) here is the variance of the distributional estimate provided by C51. We replace the clipping in Eq. (18) by adding a small constant 61 1 x 10 to Var(Z a)) s.t.
Var (s. a)) +
ps a)²
61 [Ai EoEA Var Var(Z(s.a)
Fig. 7 (b) shows the effect of clipping in the vanilla version (IDS-C5) [clip]) compared to our variation (IDS-C51 [noclip]) on the deep sea environment.
Intrinsic reward priors are computational method we implement with PE-DQN, which leverages the fact that we can compute the one-step uncertainty estimate (ay a) deterministically from parametric ensemble given state-action tuple. This obviates the need to learn it explicitly in
the bonus estimation step. We thus add Wavg (5, a) automatically to the forward pass of the bonus estimator ba(s, (8, a) sort of "prior" mechanism according
(s(s a) == 1% (s, a) + (s, a),
where 15th is the raw output of the bonus estimator NN of parameters 6. In the VizDoom environment, follow the default pipeline suggested by Burda al. (2019b) and subsequent works (Burda et al., 2019a) that normalize intrinsic rewards by running estimate of its marginal standard deviation.
Bonus decay is the decaying of the exploratory bonus during action selection. It is well-known that the factor 3 is a sensitive parameter for UCB-type exploration algorithms, enabling efficient exploration when chosen correctly but simultaneously preventing proper convergence when chosen wrongly. Due to the variety of tasks included in the bsuite and VizDoom, we opted for a fixed schedule by which (3 is linearly decayed to 0.0 over one third of the total training horizon. In the bsuite experiments, we apply this schedule to all tested baselines where applicable and chose the initial 3mi value according to the hyperparameter search. Since the decay rate central part of the DLTV algorithm, we here do not use our linearly deacying schedule but adopt the original decay rate of B = 30 /lg(at)) ol where a is scaling parameter.
Ensembles and their size are a central parameter in IDS and BDQN+P. For the bsuite experiments, we used size of 20 as in the implementation by Osband et al. (2020) who find that increasing the ensemble size beyond 20 did not lead to significant performance improvements on the bsuite. Fig. 7 (a) shows comparison of the influence of ensemble size in BDQN+P compared to PE-DQN. For VizDoom, we used 10 models in accordance with Nikolov et al. (019) for their Atari experiments. Here, we follow the original implementations and let the ensembles used in BDQN+P and IDS-C51 (and also PE-DQN) share a network body for fcature extraction to save computation.
Replay buffer In the VizDoom environment, all our algorithms make use of prioritized experience replay (Schaul et al. 2016).
The computational resources used conduct the bsuite experiments High Performance Computing Centre (DHPC) and Delft Artificial Intelligence Cluster (DAIC). We deployed bsuite environments in 16 parallel jobs to be executed GPUs, 16 Intel XEON E5-6248R 24C 3.0GHz CPUs, and 64GB of memory in total. In this setup, the execution of one seed on the entire suite experiment took approximately 38 hours for DLTV. 72 hours for PE-DQN, and 80 hours for IDS. Due to the narrower network architecture of BDQN+P, we in this case parallelized environments over 64 Intel XEON E5-6248R 24C 3.0GHZ CPUs, taking approximately 76 hours wall-clock time for the entire suite. In the VizDoom environments, we deployed 32 parallel environments for each agent on the same hardware. In this case, computation for 10 x 106 took approximately 24 hours per seed per environment and did not differ significantly between any of the tested methods. Table shows the average wall clock time for the VizDoom experiments.
supplied by Delft
8 NVIDIA Tesla VooS 32GB
23

---

Published as a conference paper at ICLR 2024
Explonaion
visied g) (in 1c.3)
Crodit Assignment
Generalization
-20
75 u.c.
A: PE-DON
CSI (socip) Ous
1.0
Memory
BDQN-P(30)
mS--51 Vonita
C. BDQN-PI71
DCEV-OK (mptin) Dure
0.5
BION
DLTV-QR (ufit) Vanils
BDQNHRY2)
Noise
0.0
1'0
2.0
3.0
No. opisodes 1e3)
Figure 7: (a) Summary of bsuite experiments. Comparison between BDQN+P with different ensemble sizes and PE-DQN (total ensemble size 4). (b) Deep sea comparison between our implementations and vanilla implementations of baseline algorithms. Shown are median state-action visitation counts over number of episodes on the deep sea environment with size 50, Shaded regions represent the interquartile range of 10 seeds. Higher is better.
Encemble
(.4]
X x [4]
x [-]]
ReLU
Residual Block
ReLU
Coux,
ReL.U
Residual
Conv.
x3
Residual Block
[82, 64, 64) ch.
Max 3 x stridd
stride
/255
Figure 8: Schematic of the architecture used for VizDoom environments. Based on the architecture used by Espeholt al. (2018).
24

---

Published as a conference paper at ICLR 2024
|=1000
1=4000
(=8000
I=16000
t=32000
-0.1)
1.6
Invern (s.0 evons
1.0
25
1/(N
Eas disagnement
Wang (5. a)
0.10
25
0.05
50
0.00
bonns estaaate
(9 ()
0.2
25
50
25
50
25
50
Figure 9: comparison of inverse counts (top row), ensemble disagreement (mid row), and bonus estimates (bottom row) on the deep sea environment. t indicates total environment interactions. Each image depicts the state-space deep sea, where only the lower triangle (including the diagonal) is reachable. For each state, the plotted values indicate the maximum of two actions. At / = 32000, the agent has discovered the goal-state at the bottom right.
B.3 ADDITIONAL EXPERIMENTAL RESULTS
Fig. illustrates comparison of the uncertainty estimates used in PE-DQN for the deep environment, Every plot shows the entire state-space of the deep sea environment. In deep sea, the agent starts at the top left entry in matrix and, depending his action, moves the left or right column while descending one row. The upper right triangular matrix above the diagonal is thus not reachable to the agent. The goal, i.e., the rewarding final state is located at the bottom right of the matrix.
For different time steps (total environment interactions) during training, evaluate the entire state-space and compare three quantities:
Inverse counts are the inverse of visitations each state-action (o..))oo.1 For every state, we plot the maximum of both actions.
Ensemble disagreement defined (s, ( 1)) Σmy=1 wi (7e, n6, (s, a).
1/(M
For every state, plot the maximum of both actions.
Bonus estimates by a) as defined in Section 5.1 For every state, we plot the maximum of both actions.
In the top row, the agent has explored increasing fraction of the state space with increasing time. The number of states with high inverse counts thus decreases. The ensemble disagreement Wave (s, a) behaves similarly inverse counts, result in line with the notion that Waeg a) serves
estimate
TE, of the local TD error wi (ME... e, Sny T"TE, (s, a). which is expected to decrease with number of visits. In contrast to this, expect bonus estimates bø(s, a) quantify errors w.r.t the true value, that wi (f), n7" (s, a). As result, bs(s, a) should not, for example, vanish prematurely for the initial state at the top left, even after many visitations, since its value only be assessed upon having explored the entire state space. The bottom row of Fig. Dis closely in line with this intuition. At / 32000, the agent has discovered the reward the bottom right.
25

---

Published as a conference paper at ICLR 2024
bandit
bandit noise
handit scale
cartpole
1.0
1.0
1.00
www
0.6
0.50
0.4
200
0
0
5000
10000
5000
10000
5000
10000
500
1000
caripole_noiss noise
parpole_scale
portpolc_swinguue
catch
600
400
1.0
0.5
0.0
200
500
1000
1000
500
1000
5000
10000
catch_none noise
catch_stee scale
deep_
deep sea, cra_stochastic
0.5
0.0
0.0
5000
10000
5000
10000
10000
discounting chain
memory_len
memory_siee
0.2
1.03
0.5
1.02
0.00
0.0
1.01
-0.2
-0.05
500
1000
5000
10000
5000
10000
5000
10000
mmist_non _ooiss
mmnist_scale
mountain
mountain car_ noise
0.5
0.5
200
0.0
500
-0.5
-0.6
- 750
1000
5000
10000
5000
10000
500
1000
500
1000
mountain car scale
umbrella_disten
umbrella__engee
400
MY
0.5
IDS-C51
PE-DQN
600
800
0.0
0.0
DLTV-QR
0
500
1000
5000
10000
5000
10000
Figure 10: Averaged episodic return for all 23 bsuite tasks.
B.4 FULL RESULTS OF BSUITE EXPERIMENTS
Fig. shows the averaged undiscounted episodic return for all bsuite tasks. Each curve represents the average over approximately 20 variations of the same task (Osband et al.] (2020) provide a detailed account of the task variations) where results were taken from separate evaluation episode using a greedy action-selection rule. In the "scale" environments, evaluation results were rescaled to the original reward range maintain sensible average. Bold titles indicate environments tagged as hard exploration tasks.
26


Reviews:
[{"review_id": "MWnuMAonBW", "reviewer": "Reviewer_1pzE", "summary": "This paper proposes a simple ensemble method for distributional RL. The authors discuss some basic theoretical properties of this ensemble and experimentally test its performance.", "questions": "Apart from the multiple questions I asked in the weaknesses section, what is the use of Theorems and Propositions in Section-4, when they play no role in the sections ahead. If they are not empirically useful results, but simply theoretical curiosities, I still don't know what I learned from them. They seem simple manipulations of the various definitions without an end goal in sight.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper presents a neat and simple idea -- creating a distributional RL algorithm whose return distribution function is an average of multiple return distribution functions. This idea could be easily applied by many practitioners to improve performance. The authors have demonstrated that the ensemble performs better (as with all RL algorithms, the true test of whether it is indeed actually better will be if it is adopted by more people) on bsuite, deep sea environment and VizDoom environment as compared to baselines.", "weaknesses": "**Originality**: The paper states that \"our work is the first to study the combination of different projection operators and representations in the context of distributional RL\", but consider the paper [_Distributional Reinforcement Learning with Ensembles_ by Lindenberg, Nordqvist and Lindahl](https://arxiv.org/abs/2003.10903) from 2020. Specifically, equation (7) in that paper has the exact same form as the formulation of $\\eta_M$ in the paper under review.\n\n**Poor writing**: The Achilles heel of this paper is the muddled presentation of ideas. Let me elaborate\n1. There are many places where **imprecise writing** makes the presentation difficult to parse. For example, in the abstract it is stated that \"profounding impacting the _generalization behavior_ of learned models\". What does generalization behavior here mean? It is [known](https://arxiv.org/abs/2111.09794) that generalization in the context of RL can have many different meanings. As another example, in the introduction it is stated that \"the space of potential return distributions is infinite-dimensional\". The space of probability measures (as opposed to finite signed measures) is _not_ a vector space, and so how do the authors define a dimension in this context? It couldn't be the cardinality of the set because then the set of representable distributions will also have infinite dimensions. Or do the authors mean parametric vs nonparametric, where the dimension is defined on the parameter space? As another example, Figure 1 is poorly explained -- what are the x and y axis, how does it convey the notion of \"distinct generalization signature\" as stated in the paper? As another example, in section 3, the notation is inconsistent. $S_t$ and $A_t$ are used to denote samples or random variables? If samples, then it is inconsistent with what is stated next that capital letters denote random variables. If random variables, then what does $\\mathcal{R}(\\cdot \\mid S_t, A_t)$ or $\\mathcal{R}(S_t, A_t)$ (two ways to stating the same thing?) even mean? $\\mathcal{D}^\\pi(s,a)$ is in $\\mathscr{P}(\\mathbb R)$ and not in $\\mathscr{P}(\\mathbb R)^{\\mathcal S \\times \\mathcal A}$. In the definition of a pushforward measure in the appendix, it is stated that $B \\in \\mathbb R$, which should be $B \\in \\mathcal{B}(\\mathbb R)$ where $\\mathcal{B}(\\mathbb R)$ denotes the Borel subsets of $\\mathbb R$.\n2. There are many places where the presentation is **incomplete or even wrong**. The notion of inverse CDF is not defined anywhere despite it playing a critical role in the paper. When defining the supremum $p$-Wasserstein metric, the authors let $p \\in [0, \\infty)$. I am pretty sure that this is incorrect and should be $p \\in [1, \\infty)$. As a simple test, think what happens when $p=0$. See the monograph _One-Dimensional Empirical Measures, Order Statistics, and Kantorovich Transport Distances_ by Bobkov and Ledoux for more details.  The reference of Rösler 1992 for distributional RL makes no sense -- yes it studies related math, but it is not on distributional RL! It is stated in section 3.2 that $\\Pi_C$ is a projection defined on $\\mathscr{P}(\\mathbb R)$, but the complete definition is never even given. It is defined on finite mixtures of Dirac measures, but what about countable mixtures of Dirac measures, or non-atomic measures? In section 4, it is stated that $\\eta_M$ has support over $\\mathscr{F}_M$. I believe this is incorrect. Consider $\\mathscr{F}_1 = \\{Z\\}$ and $\\mathscr{F}_2 = \\{-Z\\}$, where $Z$ is, say, some random variable having Gaussian distribution. Then $\\eta_M = \\delta_0 \\notin \\mathscr{F}_M$.\n\nI really wanted to like this paper -- the idea is simple and potentially useful. But the poor presentation forces me to not recommend acceptance of this paper, unless significantly rewritten. I wasted many hours disentangling, rather than learning, the ideas in the paper, and I wouldn't wish that on potential readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JLiMKWHTuD", "reviewer": "Reviewer_a3AN", "summary": "This paper considers the use of using ensembles in distributional reinforcement learning. The paper first develops the theoretical framework for using different projection operators in an ensemble and the conditions under which the ensemble still forms a contraction mapping. Then the paper proposes an upper bound on the Q-value estimated by the ensemble via a 1-Wasserstein distance between the learned return distribution, $\\hat{\\eta}(s, a)$, and the true return distribution $\\eta^\\pi(s,a)$, $b^\\pi(s,a) = w_1(\\hat{\\eta}, \\eta^\\pi)(s,a)$. This quantity can be seen as a form of epistemic uncertainty since it would vanish if the learned return distribution is equal to the true return distribution (Proposition 1&2). Since $\\eta^\\pi(s,a)$ is unknown and uncertainty estimation with bootstrapping can incur bias, the paper then proposes a bellman-like estimate for $b^\\pi(s,a)$ that allows it to be estimated recursively using local information (theorem 3).\n\nIn the second part of the paper, the paper applies this framework to DQN. The algorithm PE-DQN consists of three components: a QR-DQN ($\\theta_1$), a C51 ($\\theta_2$), and an uncertainty module ($b_\\phi$). During training, $\\theta_1$ and $\\theta_2$ are updated with the respective distributional RL algorithm, and $b_\\phi$ is updated with the uncertainty update from theorem 1. During action selection, the $\\theta_1$ and $\\theta_2$ are ensembled to estimate the Q-value, and $b_\\phi$ is used to estimate the exploration bonus.\n\nExperiments show that PE-DQN is able to outperform relevant baselines in terms of exploration, generalization and credit assignment in bsuite. The paper then shows that in the VizDoom environments with different reward-sparsity levels, PE-DQN is able to match the performance of relevant baselines in the dense reward setting and outperform the baselines in the sparse reward setting.", "questions": "- I wonder if using different architectures or different ensemble weights for the ensemble would further improve the performance. Perhaps there is even a way where one can adjust the weights to get a tighter estimate of the uncertainty.\n\n- How does this propagation scheme compare to other schemes like UBE [4]?\n\n## Reference\n\n[1] On the Importance of Exploration for Generalization in Reinforcement Learning. Jiang et al.\n\n[2] Distributional Reinforcement Learning for Efficient Exploration. Marvin et al.\n\n[3] Estimating risk and uncertainty in deep reinforcement learning. Clement et al.\n\n[4] The Uncertainty Bellman Equation and Exploration. O'Donoghue. et al.\n\n[5] Long-Term Visitation Value for Deep Exploration in Sparse Reward Reinforcement Learning. Parisi et al.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Personally, I really like the ideas proposed in the paper. \n\n-  I find the theoretical motivation compelling and the derivation is straightforward and elegant. The use of 1-Wasserstein between different ensembles provides a nice alternative interpretation of the variance approach many existing works use.\n- The proposed uncertainty propagation allows the uncertainty estimation to take in future uncertainty, which many similar methods cannot do, and also addresses some shortcomings of existing methods that do propagate future uncertainties (e.g., the uncertainty bellman equation).\n- Empirically, combining the different approaches of distributional RL also makes a lot of sense. Since they have very different inductive biases, it's less likely that they would collapse to the same prediction before the return estimate converges to the true one (in principle).\n-  The experiments support the theory reasonably well.\n\nOverall, I think this paper is a nice addition to the distributional RL literature and I am not aware of similar works.", "weaknesses": "- The performance improvement over existing methods is relatively small. It would be perhaps ideal to find scenarios where the performance of the PE-DQN is more significant than existing methods. One task that I can think of is Procgen. Recently, [1] showed that incorporating epistemic uncertainty can significantly improve the performance of Value-based methods on Procgen but the method in [1] does not incorporate future uncertainty and only uses an ensemble of QR-DQNs. I would expect PE-DQN to be much better in this case.\n\n- I would like to see a comparison of the wall clock time to get a better understanding of the computational cost.\n\n- I would like to see a sensitive analysis of $\\beta$.\n\n- There are several additional works that use ensembles of distributional RL models to do exploration [1, 2, 3]. This paper's contribution is unique (to the best of my knowledge) but I feel it would be better to discuss these works to provide a more comprehensible discussion of what has already been done and what shortcomings this work addresses. [5] also proposes a similar idea but in the tabular setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CCdDNhsqwT", "reviewer": "Reviewer_9dG6", "summary": "The paper proposed a novel distributional RL method with improved exploration efficiency. The algorithm maintains an ensemble of return distribution models, each associated with a representation and a projection operator bounded in the p-Wasserstein metric. The ensemble is updated towards a shared TD target computed by first applying the distributional Bellman operator to a return distribution, and then projecting it to individual representations using corresponding projection operator to constitute a new mixture model, an operator the authors named the projection mixture operator. They have demonstrated the contraction property of the proposed projection mixture operator. They then move on to propose to use the W1 distance between true and estimated return distributions as an exploration bonus to construct a provably optimistic Q-value for action selection. Without access to the true return distribution, the authors approximate the exploration bonus with its upper bound which can be estimated incrementally. When the ensemble consists of categorical and quantile projections, the authors provided loss functions on which the model parameters can be updated through gradient descent. Experiments on a variety of tasks have shown improved exploration and overall performance compared to several ensemble / distributional RL benchmarks.", "questions": "1. is each individual ensemble member constrained to a mixture representation of the return distribution? If so, how is your approach different from a super mixture model containing num_members * num_atoms_per_mixture atoms without splitting into individual mixtures?\n2. the atoms in a quantile representation can be thought of with equal weights whereas those in a categorical representation cannot. Would you not need to make this distinction in Eq. 7?\n3. the local estimate of the exploration bonus $w_1 (\\hat{\\eta}, \\Omega_M\\mathcal{T}^\\pi\\hat{\\eta})$ seems to be measuring the distance between the ensemble return distribution as a whole and its backup target, I failed to see how it may be estimated as the model disagreement among ensemble members (page 7)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. proposed a novel approach to deep exploration in distributional RL\n2. the paper is well written in general with clear and stringent logic\n3. has systematically shown that an upper confidence bound of the Q-value can be constructed from the epistemic uncertainty of return distribution estimation in terms of W1, rather than merely treating the latter as a curiosity bonus", "weaknesses": "1. since the proposed algorithm is an ensemble on distributional RL, comparisons only with either distributional RL or ensemble on scalar RL but not both seem inadquate. Perhaps worth comparing to some of the works you mentioned in Related Work.\n2. the illustration for Fig.1 and Fig.2 can be expanded", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FksGCKB86F", "reviewer": "Reviewer_fJQm", "summary": "This paper proposes projection ensembles method in distributional reinforcement learning for generalization and suggests optimistic bound based on 1-wasserstein distance to promote deep exploration. The authors provide some theoretical results for the proposed Bellman operator. The authors also provide a practical algorithm, PE-DQN and demonstrate its performance on deepmind b-suite benchmark.", "questions": "•\tMinor Comments\no\tSection 5.1, paragraph of Uncertainty Propagation : ~exploratory actions.Further details ~ → ~exploratory actions. Further details ~\n\no\tIn Section 6.1, $\\mathbb{S[Z]}$ should be the mean-variance measure to recover DLTV-QR, not just the variance.\n\no\tIn Section 6.2, “the bsuite consists of 22 tasks” seems incorrect as Fig 9 shows 23 bsuite tasks.\n\no\tAppendix B.1, Table 4: DLTV has a fixed decaying schedule, $\\sqrt{\\logt /t}$. How can $\\beta_{final}$ be defined?\n\no\tAppendix B.2, Table 5: Initial bonus $\\beta_{init}$ is written twice.\n\no\tAppendix B.2, Equation (17) : What does $\\eta_{M,\\phi}$ refer to? The expression seems to be ambiguous.\n\n•\t- In the paper, an upper bound is obtained by Proposition 2, but it seems insufficient to justify the proposed bonus $b_{\\phi}$ as an epistemic uncertainty estimate. Since the confidence bound in Proposition 2 is just an expression naturally induced by the 1-Wasserstein distance, it seems invalid from a regret analysis perspective. Furthermore, the authors again approximate the total distributional error by the one-step TD distributional error, which seems to have no significant theoretical connection. Of course, I think it is quite difficult to describe this theoretically, so I suggest to plot the experimental results of the bonus estimate $\\hat{b}_{k+1}(s,a)$ or $\\mathbb{E}_{s,a}[\\hat{b}_{k+1}(s,a)]$ in each environment.\n\n•\tIn Figure 4, is PE-DQN [Ind.] without uncertainty propagation equivalent to the C51/QR mixture model with greedy action selection? If so, the performance improvement of PE-DQN seems to be mainly due to the introduction of bonus estimates. Based on the experimental results in the current paper, it is difficult to confirm whether the elimination of the inductive bias by projection contributes to the performance improvement.\n\n\n•\tProjection ensembles seem to be an attempt to relax the priors of the model as in IQN[1] and FQF[2]. Can the authors explain the difference from those?\n\n[1] Dabney, Will, et al. \"Implicit quantile networks for distributional reinforcement learning.\" International conference on machine learning. PMLR, 2018.\n\n[2] Yang, Derek, et al. \"Fully parameterized quantile function for distributional reinforcement learning.\" Advances in neural information processing systems 32 (2019).", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "•\tThe paper is well-written in general, and easy to follow. The authors provide sufficient background in distributional RL. Their theoretical results were clearly illustrated and the proof appears to be valid. The experimental results of the PE-DQN appear to be promising.", "weaknesses": "•\tThe description of the motivation and related work appears to be rather insufficient.\n\n•\tTechnical novelty and its contribution seems weak. While the paper focuses on the exploration issue in distributional RL, its theoretical analysis only shows the contractivity and upper bound for proposed Bellman operator.\n\n\n•\tIt is a bit odd that the benchmark for comparison does not include the C51, QR-DQN, which employes an $\\epsilon$-greedy scheduling method. It is also difficult to verify that the baseline is properly reproduced, since the experiment was not conducted in the standard Atari benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)