The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learning
Authors: Yunhao Tang
Remi Munos
Mark Rowland
Bernardo Avila Pires
Will Dabney
Marc G Bellemare
Venue: NeurIPS 2022 Accept (2022)
Keywords: Distributional Reinforcement Learning
Off-policy Learning

Abstract: We study the multi-step off-policy learning approach to distributional RL. Despite the apparent similarity between value-based RL and distributional RL, our study reveals intriguing and fundamental differences between the two cases in the multi-step setting. We identify a novel notion of path-dependent distributional TD error, which is indispensable for principled multi-step distributional RL. The distinction from the value-based case bears important implications on concepts such as backward-view algorithms. Our work provides the first theoretical guarantees on multi-step off-policy distributional RL algorithms, including results that apply to the small number of existing approaches to multi-step distributional RL. In addition, we derive a novel algorithm, Quantile Regression-Retrace, which leads to a deep RL agent QR-DQN-Retrace that shows empirical improvements over QR-DQN on the Atari-57 benchmark. Collectively, we shed light on how unique challenges in multi-step distributional RL can be addressed both in theory and practice.

The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learning
Yunhao Tang
Mark Rowland
Rémi Munos DeepMind munos@deepmind. com
DeepMind robiatyh@ddepmind.com com
DeepMind markrowlland&deeppinn. com
Bernardo Ávila Pires
Will Dabney DeepMind vddaneeyddeepinnd.com com
Marc G. Bellemare
DeepMind bavilapireesdeepmind.ccm com
Google Brain be bellemare@google.com com
Abstract
We study the multi-step off-policy learning approach distributional RL. Despite the apparent similarity between value-based RI. and distributional RI., our study reveals intriguing and fundamental differences between the two cases in the multi- step setting. We identify a novel notion of path-dependent distributional TD error, which is indispensable for principled multi-step distributional RL. The distinction from the value-based case bears important implications on concepts such backward-vies algorithms. Our work provides the first theoretical guarantees multi-step off policy distributional RL algorithms, including results that apply to the small number of existing approaches we derive a novel algorithm, Quantile Eegression-Retracee which leads to a deep RI agent QR-DON-Retrace that shows empirical improvements over QR-DQN the Atari-57 benchmark. Collectively, we shed light on how unique challenges in multi-step distributional RL can be addressed both in theory and practice.
multi-step distributional RL. In addition,
Introduction
The return Eie r² RR is fundamental concept in reinforcement learning (RL). In general, the return a random variable, whose distribution captures important information such as the stochasticity in future events. While the classic view of value-based RL typically focuses on the expected return [1-3], learning the full return distribution of both theoretical and practical importance [4-10].
To design efficient algorithms for learning return distributions, a natural idea is to construct distri- butional equivalents of existing multi-step off off-policy value-basee algorithms. In value-based RL, multi-step learning tends to propagate useful information more efficiently and off-policy learning is ubiquitous in modern RL systems. Meanwhile, the return distribution shares inherent commonalities with the expected return, thanks to the close connection between the distributional Bellman equation [4-6, 10] and the celebrated value-based Bellman equation [2]. The Bellman equation foundational to value-based RL algorithms, including many multi-step off off-policy methods [11-14]. Due to the apparent similarity between distributional and value-based Bellman equations, should expect key
value-based concepts and algorithms to seamlessly transfer to distributional learning?
Our study indicates that the answer is There critical differences between distributional and value-based RI., which requires a distinct treatment of multi-step learning. Indeed, thanks to the focus on expected returns, the value-hased setup offers many unique conceptual and computational simplifications in algorithmic design. However, we find that such simplifications do not hold for distributional learning. Multi-step distributional RL requires deeper look the connections between fundamental concepts such as n-step returns, TD errors and importance weights for off-policy learning. To this end, make the following conceptual, theoretical and algorithmic contributions:
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

m(I, a)
Â 0:0
C1 A0.1 0:1
+
Figure 1: Illustration of a multi-step distributional RL target, constructed sum of the initial distribution (left) and weighted distributional TD errors 0 6j 0. across multiple time steps (middle and right); see Section 3 for further details and notation. In general, distributional TD errors are signed measures, reflected by the downwards probability mass; they are also scaled by trace coefficients to correct for off-policy discrepancies between target and behavior policy.
Distributional TD error. We demonstrate the emergence of a novel notion of path-dependen distributional TD error (Section 4). Intriguingly, as the name suggests, path-dependent distributional TD errors are puth-dependent, i.e., distributional TD errors at time / depend on the sequence of immediate rewards (R This differs from value-based TD errors, which are path-independent. We will show that the path-dependency property is not artifact, but rather fundamental property of distributional learning. We show numerically that naively constructing certain path-ind aath-independent distributional TD errors does not produce convergent algorithms. The deth-deenndency property also has conceptual and computational impacts on forward-vien estimates and backward-viert algorithms.
Theory of multi-step distributional RL. We derive distributional Retrace, novel and generic multi-step off-policy operator for distributional learning. We prove that distributional Retrace is contractive and has the target return distribution as its fixed point. Distributional Retrace interpolates between the one-step distributional Bellman operator [6] and Monte Monte-Carlo (MC) estimation with importance weighting [15], trading-of the strengths from the two extremes.
Approximate multi-step distributional RL. Finally, we derive Quantile Segression-Retracee novel algorithm combining distributional Retrace with quantile representations of distributions [16] (Section 5). One major technical challenge is to define the quantile regression (QR) loss against signed measures, which are unavoidable in sample-based settings. We bypass the issue of ill-defined QR loss and derive unbiased stochastic estimates to the QR loss gradient. This leads up to QR-DQN-Retrace, a deep RI. agent with performance improvements over QR-DQN on Atari-57 games.
In Figure 1, we illustrate how the back-up target is computed for multi-step distributional RL. In summary, we take our findings to demonstrate how the set of unique challenges presented by multi- step distributional RL can be addressed both theoretically and empirically. Our study also opens up many exciting research pathways in this domain, paving the way for future investigations.
2 Background
Consider a Markov decision process (MDP) represented as the tuple (X, PR. P,v) where X' is the state space, A the action space, PR: X' x A (g)) the reward kernel (with A finite set of possible rewards), P X x A (2') the transition kernel and 7 E [0, 1) the discount factor. In general, we use S (A) denote distribution over set A. We assume the reward take finite set of values mainly because it is notationally simpler to present results; it is straightforward to extend our results to the general case. Let X (AA) be fixed policy. We use (Xe, A., R2 ) a to denote a random trajectory sampled from n,, such that A, x())X,), R₂ ~ Pn(-I Ae), ~ Ee y R, as the random return, obtained by following x starting from (x,a). a). The Q-function Q" (z, = E[G" (z, т. For convenience, we also adopt the vector notation Q E X Define the one-step value-based
P(- A.). Define GH (xx
(x,a) a)] is defined as the expected return under policy
Bellman operator 7 XXX ب R such that 7 ("Q(z,a) a) = E[Ro + (X1, (X),A7) Xoo a). The Q-function Q" satisfies Q" = T"Q"
Jr An = a) where Q(X,, A7) = E. and is also the unique fixed point of TT.

---

2.1 Distributional reinforcement learning
In general, the return G"(z, a) is a random variable and we define its distribution as 1," a) = , (Z, a)). The return distribution satisfies the distributional Bellman equation [4-6, 17, 10],
11 (a. a) E, (bbR..) (X) Xo x, Au =
where (brr)# ( - (R)) is the pushforward operation defined through the function Dyn (2) r+ [17]. For convenience, we adopt the notation 17" ( E. (a)(x)) (X,,a) Throughout the paper, we focus on the space of distributions with bounded support P b0 (R).
Let 1j P (I)) x be any distribution vector, we define the distributional Bellman operator
T 9 ( (R) xx)(R XxA
(R)xxA as follows [17, 10).
Tn((z,)) = E
X x, Ao aa
Let rf" be the collection of return distributions under æ, the distributional Bellman equation can then be rewritten as 1/" ," The distributional Bellman operator 7 >-contractive under the supremum p- p-Wasserstein distance [16, 10], so that 1y" is the unique fixed point of Tx, See Appendix B for details of the distance metrics.
2.2 Multi-step off-policy value-based learning
We provide a brief background on the value-based multi-step off-policy setting as reference for the distributional case discussed below. In off-policy learning, the data is generated under behavior policy μ., which potentially differs from target policy и. The aim is to evaluate the target Q-function Q As a standard assumption, we require (upp(r(-(z)) C supp(u(-)a) [x)), Var E X. Let Pe (2) [[A,, Xx) be the step-wise importance sampling (IS) ratio time step t. Step-wiss wise IS ratios are critical in correcting for the off-py discrepancy between π and ja.
Let E [0, pt] be a time-dependent trace coefficient. We denote C₁ = C .. cr and define C1:0 = 1 by convention. Consider generic form of the return-based off-policy operator RE as [13],
R=*"((x,a) = Q(x
Q((r,))))))))))) a) + Ep Σ Gan't (Re + 7Q (X+11, A4+1 Q(X,, At))
(3)
87 value based TT error
In the above and below, we omit the notation conditioning on X, = x, Ag = a for conciseness. The general form of RT encompasses many important special cases: when on-policy and c X, it recovers the Q-function variant of TD(A) [2, 12]; when e = À min(E, Pt). ), it recovers a specific form of Retrace [13]; when o - Pt. it recovers the importance sampling (IS) operator. The back-up target is computed as mixture over TD errors 67. each calculated from the one-step transition data. We also define the discounted TD error 07 7 88 which can be interpreted as the difference between n-step returns from two time steps and + 1, as we discuss in Section 4. As we will detail, the property of 55 marks a significant difference from the distributional RL setting.
By design, RT has Q" as the unique fixed point. Multi-step updates make use of rewards from multiple time steps, propagating learning signals more efficiently. This is reflected by the fact that RT. is 8-contractive with B E [0 [13] and often contracts Q faster than the one-step Bellman operator 7 Our goal to design distributional equivalents multi-step off- off-policy operators, which can lead to concrete algorithms with sample-basee based learning.
3 Multi-step off- off-policy distributional reinforcement learning
We now present the core theoretical results relating to multi-step distributional operators. In general, the aim is to evaluate the target distribution r7" with access to oof-policy data generated under p.
Below, we use Ge = Ei y s-t R, to denote the partial sum of discounted rewards between two time steps t' < t. We define the generic form of multi-step off- off-policy distributional operator RM such that for any n E P (B) its back-up target (,("",(z, a) is computed as
71, a) + E
Σ t=0
Gia (bGorget1) # (Xe+1, A - (bo ind ) (X As)
(4)
A Multisstep Distributional TD
3

---

As an effort to simplify the naming, we call RT the distributional Retrace operator. Distributional Retrace only requires C E [0, Pr] and and represents a large family of distributional operators. Throughout, we will heavily adopt the pushforward notations. This is mainly because instead of directly working with the random variable G we find it much more convenient to express various important multi-step operations with pushfoward notations.
The back up target R (= a) is written as weighted sum of the TD errors A which we extensively discuss in Section 4. Though the form of RT seems to bear certain similarities to the galue-based operator in Equation (3), the critical differences lie in subtle definitions of the distributional TD errors Ahe and where to place the traces C1at for off- pollcc corrections. We resume to unpack the insights entailed by the design of the operator in Section 4.
distributional
Below, we first present theoretical properties of the distributional Retrace operator. We start with a key property which underlies many ensuing theoretical results. Given a fixed n-step reward sequence 70m-1 and a fixed state-action pair (x. a) E X' A, we call pushfoward distributions of the form (b (E xruin)) (75, a) the n-step target distributions. Our result shows that the back-up target of Retrace is a convex combination of n-step target distributions with varying values of n.
Lemma 3.1. (Convex combination) The Retrace back-up target is convex combination of n- step target distributions. Formally, there exists an index set [ar, a) such that R R=*'n(rr a) (x,(( Wa7p where w; 2 0, Eie((r,a) W; = 1 and (M)))T().)) are n,,return target distributions.
Since RT E go (R) XxA, we can measure the contraction of RT under probability metries.
Proposition 3.2. (Contraction) R is 8-contractive under supremum p- --Wasserstein distance, where 3 = max E Ey [c1...C-11(1 c)] S.
The contraction rate of the distributional Retrace operator is determined by its effective horizon. At one extreme, when c O, the effective horizon is 1 and 9 7. in which case Retrace recovers the one-step operator. At the other extreme, when C Pt. the effective horizon is infinite which gives 3=0 0. This latter case can be understood as correcting for all the off off-policy discrepancies with IS, which is very efficient in expectation but incurs high variance under sample-base approximations. Proposition 3.2 also implies that the distributional Retrace operator has unique fixed point.
Proposition 3.3. (Unique fixed point) RT has 17" as the unique fixed point in P (R) XXA.
The above result suggests that starting with 7/ E 3 (IR) (IR) the recursion M]++1 RT.P produces iterates (n)) PRo which converge 17 in W a rate of 0(3*).
Understanding multi-step distributional reinforcement learning
Now, we pause and take a closer look at the construction of the distributional Retrace operator. We present number of insights that distinguish distributional learning from value-based learning.
4.1 Path-dependent TD error
The value-based Retrace back-up target can be written mixture of value-based TD errors. To better parse the distributional Retrace operator and draw comparison to the value-based setting, we seek to rewrite the distributional back-up target R RT,"n((, a) into weighted sum of some notion of distributional TD errors. To this end, we start with a natural analogy to the balue-based TD error.
Definition 4.1. (Distributional TD error) Given a transition (X,, A., Re, X define the associ- ated distributional TD error as A A Ae, Re, X.41) = (bm..) #1 (X.11. A7 At).
When the context is clear, we also adopt the concise notation A = ( A.. X.+1). By construction, distributional TD errors are signed measures with zero total mass [10]. The distributional TD error is natural counterpart to the value-based TD error, because they both stem directly from the corresponding one-step Bellman operators. However, unlike in value-based RL, where TD emors alone suffice to specify the multi-step learning operator (Equation (3)), in distributional RI this is not enough. We introduce the path-dependent distributional TD error, which serves as the building block to distributional Retrace.

---

Definition 4.2. (Path-dependent distributional TD error) Given a trajectory (X,, Ro) define the path-dependent distributional TD eπror at time follows,
A Oct = (bGo.-inn A
Path-dependent distributional TD errors are defined as pushforward measures from AT, where the pushforward operations depend on Go.--1. This equips ÅП₃ with an intriguing property, path- dependency. Concretely, this means that the path wath-dependent distributional TD error depends the sequence of rewards (R,)² leading up to step t. With the above definitions, we can finally rewrite the back-up target of distributional Retrace as weighted sum of path-dependen distributional
We now illustrate the difference between TD errors R [T."m((z a) = 71,, a) + Ep i' A.a) value-based and distributional TD errors.
Comparison with value-based TD equivalents. The value-based equivalent to the distributional TD error is the discounted value-basee TD error 07 = 6'07 which we briefly mentioned in Section To see why, note that discounted value-based TD errors allow to rewrite the value- based Retrace back-up target as R==Q((x, a) Q(x, a) + E IEr= C1:26]]. For direct comparison between the two settings, we rewrite both AT, and 67 the difference between two n-step predictions evaluated at two time steps and f ++ 1,
(Xi+1, A A411 (DCo.-1.n)) ((x,, Ar), 57 (Goz + + (X++1, A7+1)) - +
A 0:1 (bG.
(Distributional)
(Value-based)
The above rewriting attributes the path-dependency to the fact that the n-step distributional prediction (bGmn)) 7) (Xe+1, A7+1) is non-linear in Guan Indeed, in the value-basee setting, because Go. Go.-1 + 7 Rg the partial sum of rewards Go cancels out common term. This leaves the discounted
error 8g path-independent. In other words, the computation 87 does not depend on past rewards (R.) )==) In contrast, in the distributional setting, the pushforward operations are non-linear in the partial sum of rewards Goa-1- As result, Go 1 does not cancel out in the definition of A making the path-dependent TD error Âg. depend on the past rewards (R.)==)-
The path-dependent property is not an artifact of the distributional Retrace operator RT instead, is an indispensable element for convergent multi-step distributional learning in general. We show this by empirically verifying that multi-step learning operators based on alternative definitions of math-independett distributional TD errors are non-convergent even for simple problems.
Numerically non-convergent path-independent operators. Consider the path-independent distributional TD error A, == (bo,,*) # A.. We arrived at this definition by dropping the path- dependent term Go.-1 in the pushforward of A Such definition seems appealing because when = 17", the error zero in expectation E,, A. A 0. This implies that we can construct multi-stey operator by a weighted sum of the alternative path-independent TD error R," n)(,, a) = (7,,, a) + Ty,, Σt= A.
Logzo Ereers
R
T
By construction, RM has 17"
R
as one fixed point.
Iterations
We provide very simple counterexample which R. , is not contractive: consider an MDP with one state and one action. The state transitions back itself with deterministic reward R. = 1. When the discount factor is = 0.5, 17" is Dirac distribution centered
Figure 2: Non-convergent example: comparing L,, 'jo,?"")) 17"T) across it- erations. We plot 10 randomly ini- tialized runs. Note (R. )*m does not converge to sy" while others do.
We consider the simple case ei #1 and 6 0, 1I 2 2. We use the Lp distance to measure the convergence of the distribution iterates [10]. Figure 2 shows that )* does not converge 17", while the one-step Bellman operator 7 and distributional Retrace RM are convergent.
In Appendix C, we discuss yet another alternative to A designed to path-independent ''A Though the resulting multi-step operator still has 17" one fixed point, show numerically that
5

---

is not contractive on the same simple example. These results demonstrate that naively removing the path-dependency might lead to non-convergent multi-step operators.
4.2 Backward-view of distributional multi-step learning
To highlight the difference between distributional and value-based multi-step learning, we discuss the impact that path-dependent distributional TD errors have the backward- view distributional algorithm. Thus far, distributional back-up targets are expressed in the forward-view, i... the back-up target at time t is calculated view algorithms, unless truncated, wait until the episode finishes to carry out the update, which might be undesirable when the problem is non-episodic or has very long horizon.
function of future transition tuples (X A., R.)ast- The forward-
In the backward-view, when encountering a distributional TD error 4, the algorithm carries out updates for all predictions at time 5' [2]. To this end, the algorithm needs maintain additional partial return traces, i.e., the partial sum of rewards Gra, in order to calculate the path-dependent
TD error AÇ. Unlike the malue-based state-dependent eligibility traces [2, 18], partial return traces are dependennn. This implies that in an episode of steps, value-based backward- -iew algorithms require memory of size min( (())) while the distributional algorithms requires (TT).
In addition to the added memory complexity, the incremental updates of distributional algorithms are also much more complicated due to the path-dependent TD errors. We remark that the path- independent nature value-based TD errors greatly simplify the value-based backward view algo- rithm. For more detailed discussion, see Appendix D.
4.3 Importance sampling for multi-step distributional RL
In our initial derivation, arrived at RT through the application of importance sampling (IS) in different way from the value-based setting. We now highlight the subtle differences and caveats.
For fixed n 2 consider the trace coefficient et = p.Tl(t n). The back-up target of the resulting Retrace operator reduces to Ey [Pi:-1 (DGomn...7)) # (X AT) This can be seen as applying IS the n-step prediction (bGom- # 7) (X,:; A7). As caveat, note that an appealing alternative approach is to apply IS to G producing the estimate (bpL. 1 1.7") # " (Xn, AA). This latter estimate does not properly correct for the off-policy discrepancy between и and J.. To see why, note that applying the IS ratio to Gon instead of to the probability of its occurrence, is an artifact of value-based RL because the expected return linear in Go., [11]. In general for distributional RL, one should importance weigh the measures instead of sum of rewards.
5 Approximate multi-step distributional reinforcement learning algorithm
We now discuss how the distributional Retrace operator combines with parametric distributions, using the construction of the novel Quantile Regression-Retee
algorithm as a practical example. We focus on the quantile representation because it entails the best empirical performance of large-scale distributional RL [16, 19]. Speficially, we present an application of quantile regression with signed measures, which is interesting in its own right. Below, we start with brief background on quantile representations [16], followed by details on the proposed algorithm.
Consider parametric distributions of the form: Σm- 5z, for a fixed m 2 1, where (2+) E R
are a set of parameters indicating the support of the distribution. Let PR denote the family of distribution (R) == ( Σ == 1z E R.. We define the projection Ho (TR) + P (R) as TT arg min (R) Wi (1. v). which projects any distribution onto the space of representable distributions in the parametric class under the W1 distance. With an abuse of notation, we also let lll denote the component-win projection when applied to vectors. See [16, 10] for more details.
Gradient-based learning via quantile regression. We can use quantile regression [20-22] to calculate the projection ITz)) Let (=), 2 E IR denote the CDF of given distribution 7) Let F be the generalized CDF inverse, we define the 7-th quantile as FØ (T) for T E [0, 1). The projection ye [16]. To learn the 7-th quantile for any [0, 1], it suffices to solve the quantile regression problem whose optimal solution is F, (r): ming (n) Ez (Z 0)] where (u) u(T I[u 0)). In practice, we carry out the gradient update 0 +- 0 a V ( to find the optimal solution and learn the quantile 0 ~ FT)(().
To is equivalent to computing
(r.) for 7 E (2i-) (2i-1

---

5.1 Distributional Retrace with quantile representations
Given an input distribution vector 7, we use the distributional Retrace operator to construct the back-up target R= Then, space of representations I Overall, we are interested in the recursive update: start with any 1Jo (B) consider the sequence of distributions generated via 7]++1 IIg R RT direct application of Proposition 3.2 allows us to characterize the convergence of the sequence, following the approach of [10].
the quantile projection to map the back-up target onto the
Theorem 5.1. (Convergence of quantile distributions) The projected distributional Retrace operator
IT is B-contractive under W oo distance in P e As a result, the above 77 converges to limiting distribution 17 in W oo. such that W oo (7i,, m - (8) ( W oo (11. m7)) Further, the quality of the fixed point is characterized W 00 (177: 77) ) S (1 B) W oa (Ilon".
Thanks to the faster contraction rate 3 S Y, the advantage of the projected operator Ro TM is two-fold: (1) the operator often contracts faster to the limiting distribution 1/ than the one-step operator T contracts to its own limiting distribution 77т [16]: (2) the limiting distribution n/ also enjoys better approximation bound to the target distribution. We verify such results in Section 7.
5.2 Quantile Regression-Eee Retrace: distributional Retrace with quantile regression
Below, we use (2, a) to represent the i-th quantile of the distribution at (x; a). Overall, we have tabular quantile representation 7) (z, 0) vy Ei م3.(x,0), V(z, a) X' A. where use the notation 7/= stress the distribution's 's dependency
parameter (z, a). For any given bootstrapping
distribution vector 17 E Po (B) (B)**x, in order to approximate the projected back-up target II R"" with the parameterized quantile distribution 1)2, we solve the set of quantile regression problems for all 1 < i S m, (I, a) E X x A.
min L (tm.a) (R=H a)), where T; (2i 1)/2m. (x..a)
For any fixed (z, a. i), to solve the quantile regression problem, we apply gradient descent on (x, a). In practice, with one sampled trajectory (X., A,, R. = ~ μl the aim to construct an unbiased stochastic gradient estimate of the QR loss (2,(),)) (Ra a)). Below, let b, bG..-1.1 for
simplicity. We start with stochastic estimate D (x) (RT-M a)) for the QR loss,
TT (((()))) (n) +
Σ Cis (Er(te))) (1b+11) 7f (X++1,
T.
(itt,a,) ((x))n(Xx. Ai)):
Since (m.a) (R,,nn(z, a)) is differentiable with 22, a), we use V V2,(n,a) T (2 (R=#n(E, a)) as the stochastic gradient estimate. This gradient estimate is unbiased under mild conditions.
Lemma 5.2. (Unbiased stochastic QR loss gradient estimate) Assume that the trajectory terminates within H < do steps almost surely, then we have E, (. (R=-H1(x, a))l
L (R=-"((z,a)) a)) and E, =V, 2i(m,a) (R**"((x, a)) Vai( Li (R=*n(zza)) 22((..)
(i(,,))
The above stochastic estimate bypasses the challenge that the QR loss is only defined against distributions, whereas sampled back-up targets RT*")((,, a) 7) a) E An are signed measures in general. In Quantile Ciaression-Retraace we use 7), itself as the bootstrapping distribution, such that the algorithm approximates the fixed point iteration 7/= + RT Concretely, we carry out the following sample-based update
2, (x, a) <- (x, a) aV (1r,)
L (()a,)) (R a)), for V 1 m, (x
5.3 Deep reinforcement learning: OR-DON-Retran
We introduce a deep RL implementation of the Quantile Regression-Retie QR-DQN QR-DQN-Retrace where the parametric representation is combined with function approximations [23, 16, 19]. The base agent QR-DQN [23] parameterizes the quantile locations (I, a; w) with the output of a neural network with weights w. Let 17(2, a; w)) QR-DQN-Retrace updates its parameters by stochastic gradient descent on the estimated QR loss, averaged across all m quantile levels w + - a 2 Σm V ( (RT'/"(a a; w)). In practice, the update is further averaged over state-action pairs sampled from replay buffer.
Σ 5 (x:(w)) denote the parameterized distribution.
7

---

Log diannce DOInt
0.00
10
Log dizance Frmm 100d
Log projectdd
==0.0
2.0.0
=-0.5
84
(a) Off-policyness
(b) Trace coefficient Ct. (c) Fixed point quality
(d) Uncorreeted
Figure 3: Tabular experiments to illustrate properties of the distributional Retrace operator: show average results across 10 randomly sampled MDPs. (a) Contraction rate vs. off-policyness: (b) Contraction rate vs. trace coefficient c min(Pt, ž); (c) Fixed point quality vs. trace coefficient cz; (d) The uncorrected operator introduces bias to the fixed point while Retrace is unbiased.
Discussions
Categorical representations. The categorical representation is another commonly used class of parameterized distributions in prior literature [23, 17, 24, 10]. We obtain contractive guarantees for the categorical representation similar Theorem 5.1. As with QR, this leads both improved fixed- point approximations and faster convergence. Further, this leads to a deep RL algorithm C51-Retrace. The actor-critic Reactor agent [25] uses C51-Retrace as a critic training algorithm, although without explicit consideration or analysis of the associated distributional operator. See Appendix E for details. We empirically evaluate the stand-ad alone improvements of C51-RG 151-Retrace over C51 in Section 7.
Uncorrected methods. The uncorrected methods do not correct for the off pefepolccyness and hence obtain a biased fixed point [26-28]. The Rainbow agent [26] combined n-step uncorrected learning with C51, effectively implementing distributional operator whose fixed point differs from ry"
On-policy distributional TD(A). Nam et al. [29] propose SR(A), distributional version of on- policy (D) [30]. operator form, this can be viewed as a special case Equation (4) with μ = n, X; [29] also introduce sample-replacement technique for more efficient implementation.
7
Experiments
We carry number of experiments to validate the theoretical insights and empirical improvements.
7.1
Illustration distributional Retrace properties on tabular MDPs
We verify few important properties of the distributional Retrace operator tabular MDP. The results corroborate the theoretical results from previous sections. Throughout, we use quantile representations with m 100 atoms; obtain similar results for categorical representations. See Appendix for details on the experiment setup. Let Yo be the initial distribution, we carry out dynamic programming with RT and denote 7)-- =RR=-p ) as the htth distribution iterate.
Impact of off off-policyness. We control the level of off poflccolssssss by setting the behavior policy μ to be uniform policy and the target policy to π (1 E)μ + εлa where Ta fixed deterministic policy. Moving from = 0 to e = 1, we transition from on-policy to very off-policy. We use Ip (Tk, 7 to measure the contraction rate to the fixed point. Figure 3 shows that as the behavior becomes more off-policy, the contraction slows down, degrading the efficiency of multi-step learning.
Impact trace coefficient et. Throughout, we set C₁ = min(pt. 2) with 7 to control the effective trace length. With fixed level of off- uff-policyness up the contraction to the fixed point as predicted by Proposition 3.2.
0.5, Figure 3(b) shows that increasing speeds
Quality of fixed point. We next examine how the quality of the fixed point is impacted by by measuring (7k. [nnn" point we take the converged value to be (, , Ig which measures the fixed point quality. Figure 3(c) shows when increases, the fixed point quality improves, in line with the Theorem 5.1. This phenomenon does not arise labular non-distributioaal reinforcement learning, although related phenomena do occur when using function approximation techniques.
proxy to (m.,,"")) As k increases the error flattens, at which

---

slep
15.
Retrace
130
Uncorrected
Moan acores
Modian spors
12.2
Modian scotee
One step
One step
One stop
Retrace
Retrace Uncorrected
Retrace
Uncorrected
Millons training frames
Millions of training frames
Millins training frames
Miltons of training frames
(a) C51
(b) QR-DQN
Figure 4: Deep RL experiments on Atari-57 games for (a) C51 and (b) QR-DQN. We compare the one-step baseline agent against the multi-step variants (Retrace and uncorrected n-step). For all multi-step variants, we use πt For each agent, we calculate the mean and median performance across all games, and plot the mean + standard error across 3 seeds. In almost all settings, multi-step variants provide clear advantage over the one-step baseline algorithm.
Bias of uncorrected methods. Finally, we illustrate critical difference between Retrace and uncorrected n-step methods [26]: the bias to the fixed point. Figure 3(d) shows that uncorrected n-step arrives at a fixed point in between 17" and 17", showing an obvious bias from r7"
7.2 Deep reinforcement learning
We consider the control setting where the target policy π is the greedy policy with respect to the Q-function induced by the parameterized distribution. Because the training data is sampled from replay, the behavior policy is s-greedy with respect to Q-functions induced by previous copies of the parameterized distribution. We evaluate the performance of deep RL agents on 57 Atari games [31]. To ensure fair comparison across games, we compute the human normalized scores for each agent, and compare their evaluated mean and median scores across all 57 games during training.
Deep RL agents. The multi-step agents adopt exactly the hyperparameters the baseline agents. The only difference is the back-up target. For completeness of results, we show the combi- nation of Retrace with both C51 and QR-DQN. For QR-DQN, regression, which is a thresholded variant of the QR loss [16]. Throughout, we use X min(p;, Z) with
the Huber loss for quantile
in [13]. See Appendix F for details. In practice, sampled trajectories truncated length n. We also adapt Retrace to the 7-step case, see Appendix A.
Results. Figure 4 compares one-step baseline, Retrace and uncorrected n-step [26]. For C51, both multi-step methods clearly improve the median performance slightly outperforms uncorrected restep towards the end of learning. For QR-DQN, all multi-step algorithms achieve clear performance gains. Retrace significantly outperforms the uncorrected n-step with the mean performance, while obtaining similar results on the median performance. Overall, distributional Retrace achieves clear improvement over the one-step baselines. The uncorrected n-step method typically takes off faster than Retrace but may to slightly worse performance.
the one-step baseline. Retrace
Finally, note that the value-based setting, uncorrected methods are generally more high-performing than Retrace, potentially due to a favorable trade-off between contraction rate and fixed- point bias [32]. Our results add the benefits of off- off-policy corrections in the control setting.
8 Conclusion
We have identified a number of fundamental conceptual differences between value-based and distri- butional RL in multi-step settings. Central to such differences the novel notion path-dependent distributional TD error, which naturally arises from the multi-step distributional RL problem. Build- ing on this understanding, we have developed the first principled multi-step off- off-policy distributional operator Retrace. We have also developed an approximate distributional RL algorithm, Quantile EegressioonRRetraee which makes distributional Retrace highly competitive in both tabular and high-dimensional setups. This paper also opens up several avenues for future research, such as the interaction between multi-step distributional RI. and signed measures, and the convergence theory of stochastic approximations for multi-step distributional RL.

---

References
[1] D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Athena Scientific, 1996.
[2] Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
[3] Csaba Szepesvári. Algorithms for Reinforcement Learning. Morgan & Claypool Publishers, 2010.
[4] Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametrie return distribution approximation for reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2010.
[5] Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Parametric return density estimation for reinforcement learning. Proceedings of the Conference Uncertainty in Artificial Intelligence, 2010.
[6] Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective rein- forcement learning. Proceedings of the International Conference on Machine Learning, 2017.
[7] Derek Yang. Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Yan Liu. Fully parameterized quantile function for distributional reinforcement learning. Advances in neural information processing systems, 2019.
[8] Cristian Bodnar, Adrian Li, Karol Hausman, Peter Pastor, and Mrinal Kalakrishnan. Quantile QT-OPT for risk-aware vision-based robotic grasping. In Proceedings of Robotics: Science and Systems, 2020.
[9] Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subrama- nian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic. Franziska Eckert, Florian Fuchs, et al. Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature, 602(7896):223-228, 2022.
[10] Mare Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Leurning. MIT Press, 2022. http://wwtt www.diiutiiinaal.......... -rl. org.
[11] Doina Precup, Richard S. Sutton, and Sanjoy Dasgupta. Off-policy temporal-difference learning with function approximation. In Proceedings of the International Conference on Machine Learning, 2001.
[12] Anna Harutyunyan, Mare G. Bellemare, Tom Stepleton, and Rémi Munos. Q(A) with off- poo-cy corrections. In Proceedings of the International Conference Algorithmic Learning Theory, 2016.
[13] Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare G. Bellemare, Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, 2016.
[14] Ashique Rupam Mahmood, Huizhen Yu, and Richard S Sutton. Multi-step off-policy learning without importance sampling ratios. arXiv, 2017.
[15] Yash Chandak, Scott Nickum, Bruno da Silva, Erik Learned-Miller, Emma Brunskill, and Philip S. Thomas. Universal off-policy evaluation. Advances in Neural Information Processing Systems, 2021.
[16] Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018
[17] Mark Rowland, Marc G. Bellemare, Will Dabney, Rémi Munos, and Yee Whye Teh. An analysis of categorical distributional reinforcement learning. Proceedings of the International Conference Artificial Intelligence and Statistics, 2018.
10

---

[18] Hado van Hasselt, Sephora Madjiheurem, Matteo Hessel, David Silver, André Barreto, and Diana Borsa. Expected eligibility traces. Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[19] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2018.
[20] Roger Koenker and Gilbert Bassett Jr. Regression quantiles. Econometrica: Journal of the Econometric Society, pages 33-50, 1978.
[21] Roger Koenker. Quantile Regression. Econometric Society Monographs. Cambridge University Press, 2005.
[22] Roger Koenker, Victor Chernozhukov, Xuming He, and Limin Peng. Handbook of Quantile Regression. CRC Press, 2017.
[23] Mare G. Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The Cramer distance as solution to biased Wasserstein gradients. arXiv preprint arXiv: 10743 2017.
[24] Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Mare G. Bellemare, and Will Dabney, Statistics and samples in distributional reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2019.
[25] Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot, Mare G. Bellemare, and Rémi Munos. The Reactor: A fast and sample-efficient actor-critic agent for reinforcement learning. In Proceedings of the International Conference on Learning Representations, 2018.
[26] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018
[27] Steven Kapturowski, Georg Ostrovski, John Quan, Rémi Munos, and Will Dabney, Recurrent experience replay in distributed reinforcement learning. In Proceedings of the International Conference on Learning Representations, 2019.
[28] Tadashi Kozuno, Yunhao Tang, Mark Rowland, Rémi Munos, Steven Kapturowski, Will Dabney, Michal Valko, and David Abel. Revisiting Peng's Q(2) for modern reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2021.
[29] Daniel W. Nam, Younghoon Kim, and Chan Y. Park. GMAC: distributional perspective on actor-critic framework. In Proceedings of the International Conference on Machine Learning, 2021.
[30] Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine learning. 3(1):9-44, 1988.
[31] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[32] Mark Rowland, Will Dabney, and Rémi Munos. Adaptive trade-offss in off-policy learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2020.
[33] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540);599-333 February 2015.
[34] Walter Rudin. Principles of Mathematical Analysis. McGraw-Hill New York, 1976.
11

---

[35] Charles R. Harris, K. Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Coumaaeaa, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel Smith, Robert Kem. Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre (érard-Marchan Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825)3357-362, 2020.
[36] John D. Hunter. Matplotiib: A 2D graphics environment. Computing in science & engineering, 9003):9--95, 2007.
[37] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. Jax: composable transformations of Python+NumPy programs. 2018.
[38]
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacci, Jonathan Godwin, Chris Jones, Tom Hennigan, Matteo Hessel, Steven Kapturowski, Thomas Keck, Turii Kemaev, Michael King, Lena Martens, Vladimir Mikulik, Tamara Norman, John Quan, George Papa- makarios, Roman Ring. Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojciech Stokowiec, and Fabio Viola. The DeepMind JAX ecosystem. 2010.
[39] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference Learning Representations, 2015.
[40] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320, 2021.
12


Reviews:
[{"review_id": "VOQQNC02muM", "reviewer": "Reviewer_6egT", "summary": "This paper investigates the multi-step off-policy distributional RL algorithms. Firstly, the multi-step distributional TD error is defined and the contraction property is also provided. Building this result, the authors pointed out the fundamental difference between multi-step distributional RL and classic RL by proposing a new notion called path-dependence distributional TD error. Finally, a new unbiased version of the multi-step distributional RL algorithm is proposed, including quantile-regression-Retrace and QR-DQN-Retrace, which is slightly better than the QRDQN and the naïve biased version of the multi-step distributional RL algorithm.", "questions": "1. Path-dependent distributional TD error. From Line 174 to 180, I feel that $G_{0:t-1}$ can still be canceled out in the definition of distributional TD error so that only R_t is left. I may miss some points, but could the authors provide more explanation about that? The more detailed derivation could help readers to have a better understanding of their discrepancy.\n\n2. To demonstrate the key role of path-dependent distributional TD error, the authors only show that the path-independent operator numerically can be non-convergent. My suggestion is that a more rigorous proof of the non-convergent behavior could be more convincing.\n\n3. The advantage of path-dependent distributional TD error needs to be further explored. As multi-step distributional TD may heavily depend on the past rewards, can it be beneficial for the optimization of RL algorithms? Or it can lead to an instability issue? A deeper explanation is welcome in the future version.\n\n4. What is the motivation of the proposed unbiased QR loss in Line 280 and what is the advantage over the naïve biased version in [26]? It seems that the empirical improvement is not significant.\n\n5. The experimental result across all Atari games can be also provided rather than only the mean and median of average rewards. For instance, authors can present a histogram to illustrate the improvement ratio of the proposed algorithm across all Atari games.\n", "limitations": "In the checklist, the authors said they described the limitations of their work. Maybe I missed them, but I have not found any discussion about them in the main context. This paper indeed makes some contribution to the multi-step off-policy distributional RL as claimed in the introduction, but perhaps the main limitation lies in that some results (both theoretically and empirically) are not inspiring or surprising enough that I have not learned too much after carefully reading this paper. For example, Proposition 3.2 may be a straightforward result based on previous results in the one-step case. Moreover, the underlying impact of the path-independent distributional TD error can be deeply investigated. The empirical improvement of the proposed unbiased Retrace algorithm seems to be only slightly better or even worse compared with the uncorrected version, as illustrated in Figure 6.   \n\n\nIn summary, I faithfully thank the pain-taking efforts the authors make in order to make a solid contribution in the multi-step off-policy distributional RL setting, but the limitations may not be addressed easily. My suggestion is that the path-dependence notion can be further probed and the algorithm novelty can be improved to expect better performance. I would be happy to see a better version of this paper in the future.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\n\nTheoretically, the paper provides the first theoretical guarantees on multi-step off-policy distributional RL algorithms (see Proposition 3.2 and 3.3). To the best of my knowledge, the notion of path-dependent distributional TD error is novel. However, combining multi-step distributional RL with quantile regression is straightforward, although a relatively different unbiased version in Line 280 is derived.\n\n\n### Quality  \n\nThe writing quality is good. One minor typo in Line 239, “Speficially” should be “specifically”.\n\nThe technical contribution can be further improved. Generally, investigating the multi-step distributional RL may not be somewhat intriguing as some theoretical results in one-step distributional RL can normally be straightforward to extend into the multi-step version. As such, the theoretical result may not be surprising to me, and the design of the QR-based Retrace algorithm may not be novel enough.\n\n\n###  Clarity  \n\nThe paper is well organized and easy to follow. The key confusing point is the comparison between path-dependent distributional TD error and the value-based TD error, although I understand the authors did make some efforts to clarify it. Please refer to Question for my detailed concern.\n\n###  Significance  \n\nOne potential limitation of this paper is the proposed algorithm may not achieve significant improvement compared with the baseline “Uncorrected” as suggested in Figure 4. It is easy to expect that the multi-step distributional RL can be significantly better than the one-step and thus it is not surprising that “Retrace” and “Uncorrected” are superior to “one step”. However, as far as I can tell, the contribution in the algorithm part is the proposal of an unbiased version of QR loss (Line 280 and Lemma 5.2), but I can not recognize the improvement of the unbiased version in contrast to its biased version (“Uncorrected”).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NXA8RZbdfnZ", "reviewer": "Reviewer_TbeN", "summary": "This paper provides theoretical guarantees on multi-step off-policy distributional RL algorithms and derives a DRL algorithm, QR-DQN-Retrace based on QR-DQN which shows empirical improvements over QR-DQN on Atari.", "questions": "1. Is it possible to demonstrate the difference between multi-step value based algorithms and multi-step DRL algorithms in an empirical way?\n2. As Figure 4 shows, the improvement of \"Retrace\" over \"Uncorrelated\" seems to be not significant.\n3. How about applying the retrace idea on other DRL algorithms such as IQN, FQF.\n4. Can this method ensures the validity of the learned distributions, such as the crossing issues?\nZhou, Fan, Jianing Wang, and Xingdong Feng. \"Non-crossing quantile regression for distributional reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020): 15909-15919.", "limitations": "Although there are some theoretical contributions, the algorithm design is not novel enough, and also the empirical part needs to be improved to support the idea of the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "There are two main contributions of this paper:\n1. This paper systematically discusses the similarity and difference between multi-step value-based RL and multi-step distributional RL. They also provide theoretical guarantees on multi-step off-policy distributional RL algorithms.\n2. This paper derives a novel algorithm, Quantile Regression-Retrace, which leads to a deep RL agent QR-DQN-Retrace that shows empirical improvements over QR-DQN on the Atari-57 benchmark. \n\nThere are some weaknesses of this paper.\n1. Although there are some theoretical contributions, the methodological novelty is not that big from the perspective of algorithm architecture compared to the baseline QR-DQN.\n2. The empirical part is not that solid while some more comparisons between multi-step value based algorithms and multi-step DRL algorithms are helpful. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8iMurE2pyT_", "reviewer": "Reviewer_seSz", "summary": "The authors study a multi-step off-policy learning approach to the distributional RL setting. The essential contributions of the paper may be seen as the theoretical and mathematical frameworks necessary to extend one-step distributional RL to an importance-sampled multi-step algorithm. This extension enables off-policy corrections for distributional RL in the same way that the Retrace algorithm enables off-policy corrections for value-based learning. Through a careful analysis, the authors elucidate the fact that such an extension of a Retrace style off-policy update to the distributional RL case is non-trivial. This fact is primarily owed to the nonlinearity, and repeated application of the pushforward operator used to produce the Bellman recurrence relation on the return distribution. The recurrence relation thus establishes a path-dependent relationship between the return distribution and the distributional TD error. This is in direct contrast to the path-independent relationship present in value-based RL methods.\n\nThe authors go on to substantiate the proposed methods theoretically and empirically. They evaluate their methods on Atari control problems and compare them against one-step distributional algorithms and uncorrected multi-step distributional algorithms. For each experiment, the authors run their algorithms using two different parametric distribution representations: categorical and quantile regressions. The proposed methods generally outperform the compared methods in either the mean or median score.", "questions": "-Could you please detail exactly in which way the pushforward operations are non-linear? I believe it to be non-obvious, considering that they are constructed using an affine measurable map, and both the composition and inverse of an affine map will be affine.\n\n-Are there particular use cases or real-world consequences of the proposed approach that are worth mentioning? For example, are there specific applications that are now enabled by the work?\n", "limitations": "-The authors state in the checklist that they discuss the limitations of their work, although I could not find an explicit statement in the paper. This does not diminish the contributions of the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n-The work is, to the best of my knowledge, novel and significant, as it introduces a non-trivial extension of one-step distributional RL to the multi-step off-policy setting.\n\n-The work provides a theoretical basis for the continued study of multi-step paradigms in off-policy reinforcement learning.\n\n-The authors make clear that the extension provided in the paper deviates from the value-based multi-step paradigm. \n\n-The paper provides adequate clarity of the introduced concepts and is precise with its mathematical notation.\n\n\n\nWeaknesses\n\n-The experimental results are not overly convincing that employing the multi-step distributional RL algorithm provides a significant empirical advantage.\n\n-Minor grammatical mistake in line 357.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept