Interpreting Distributional Reinforcement Learning: A Regularization Perspective
Authors: Ke Sun
Yingnan Zhao
Yi Liu
Enze Shi
Yafei Wang
Xiaodong Yan
Bei Jiang
Linglong Kong
Venue: Submitted to ICLR 2023 (2023)
Keywords: distributional reinforcement learning
regularization
entropy

Abstract: Distributional reinforcement learning~(RL) is a class of state-of-the-art algorithms that estimate the entire distribution of the total return rather than its expected value alone. The theoretical advantages of distributional RL over expectation-based RL remain elusive, despite the remarkable performance of distributional RL. Our work attributes the superiority of distributional RL to its regularization effect stemming from the value distribution information regardless of only its expectation. We decompose the value distribution into its expectation and the remaining distribution part using a variant of the gross error model in robust statistics. Hence, distributional RL has an additional benefit over expectation-based RL thanks to the impact of a \textit{risk-sensitive entropy regularization} within the Neural Fitted Z-Iteration framework. Meanwhile, we investigate the role of the resulting regularization in actor-critic algorithms by bridging the risk-sensitive entropy regularization of distributional RL and the vanilla entropy in maximum entropy RL. It reveals that distributional RL induces an augmented reward function, which promotes a risk-sensitive exploration against the intrinsic uncertainty of the environment. Finally, extensive experiments verify the importance of the regularization effect in distributional RL, as well as the mutual impacts of different entropy regularizations. Our study paves the way towards a better understanding of distributional RL, especially when looked at through a regularization lens.

Under review as a conference paper at ICLR 2023
INTERPRETING DISTRIBUTIONAL REINFORCEMENT LEARNING: A REGULARIZATION PERSPETTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
Distributional reinforcement learning (RL) is a class of state-of-the-art algorithms that estimate the entire distribution of the total return rather than its expected value alone. The theoretical advantages of distributional RL over expectation- based RL remain elusive, despite the remarkable performance of distributional RI. Our work attributes the potential superiority of distributional RI to its reg- ularization effect stemming from the value distribution information regardless only its expectation. We decompose the value distribution into its expectation and the remaining distribution part using a variant of the gross error model robust statistics. Hence, distributional RL has additional benefit over expectation- based RL thanks to the impact of riskesensitive entropy regularization within the Neural Fitted Z-Iteration framework. Meanwhile, resulting regularization in actor-critic algorithms by bridging the risk-sensitive entropy regularization of distributional RL and the vanilla entropy in maximum entropy RL. It reveals that distributional RL induces an augmented reward func- tion, which promotes a risk-sensitive exploration against the intrinsic uncertainty of the environment. Finally, extensive experiments verify the importance of the regularization effect in distributional RL, as well as the mutual impacts of different entropy regularizations. Our study paves the way towards better understanding distributional RL, especially when looked at through a regularization lens.
investigate the role of the
INTRODUCTION
The intrinsic characteristics of classical reinforcement learning (RI.) algorithms, such as temporal- difference (TD) learning (Sutton & Barto 2018) and Q-learning (Watkins & Dayan 1992). are based on the expectation of discounted cumulative rewards that an agent observes while interacting with the environment. In stark contrast to the classical expectation-basee RL, new branch of algorithms called distributional RL estimates the full distribution of total returns and has demonstrated the state- of-the-art art performance in wide range of environments (Bellemare
al.] 2017a Dabney et al.)
2018b a Yang et al.] 2019 Zhou et al.] 2020 Nguyen et al.) 2220; Sun et al. 2022)) ).2 Meanwhile, distributional RL also inherits other benefits in risk-sensitive control (Dabney et al. 2018a), policy exploration settings (Mavrin et al., 2019; Rowland et al. 2019) and robustness (Sun et al.) [021].
Despite the existence of numerous algorithmic variants of distributional RL with remarkable em- pirical success, we still have a poor understanding of what the effectiveness of distributional RL is stemming from and theoretical studies advantages of distributional RL over expectation- RL are still less established. Distributional RL problems was also mapped to a Wasserstein gradient flow problem (Martin et al. 2020), treating the distributional Bellman residual as a potential energy functional. Offline distributional RI. (Ma et al.) 2021) has also been proposed to investigate the effi- cacy of distributional RL in both nest-nl and risk-averse domains. (Lyle et al. 2019) proved in many realizations of tabular and linear approximation settings, distributional RL behaves the same as expectation-bet based RL under the coupling updates method, but diverges in non- mon-linear approxima- tion. Although the explanation from these works is not sufficient yet, the trend is encouraging for recent works towards closing the gap between theory and practice in distributional RL.
In this paper, illuminate the behavior difference of distributional RL over expectation- RL through the lens of regularization to explain its empirical outperformance in most practical envi- ronments. Specificallyy, we simplify distributional RI. into Neural Fitted Z- framework,

---

Under review as a conference paper at ICLR 2023
within which establish equivalence of objective functions between distributional RL and risk-sensitive entropy regularized Neural Fitted Q-Iteration from the perspective of statistics. This result is based two analytical components, i.., oetion-value density function decomposition by leveraging of a variant of gross error model in robust statistics, as well as Kullback-Leibler (KI.) divergence to measure the distribution distance between the current and target value distribution in each Bellman update. Then we establish a connection between the impact of risk-sensitive en- tropy regularization of distributional RL and vanilla entropy in maximum entropy RL, yielding Distribution- Eenionny-ttt-t-eeeee Regullaized Actor Critic algorithm. Empirical results demonstrate the crucial role of sissitenee entropy regularization effect from distributional RL in the potential superiority RL on both Atari games and MuJoCo environments. We also reveal mutual impacts of both risk-sensitive entropy in distributional RL and vanilla entropy in maximum entropy RI., providing more potential research directions in the future.
over expectation-base
PRELIMINARY KNOWLEDGE
In classical RL, an agent interacts with an environment via a Markov decision process (MDP), a 5-tuple AS, R. P), where S and A are the state and action spaces, respectively. P is the environment transition dynamics, R the reward function and y E (0, 1) is the discount factor.
Action-value Function vs value Distribution. Given a policy the discounted sum of future rewards is a random variable (s, a) = Σi= 7² R(st, (t), where 8g = 8, ag = a, 8++1 ~ P(-[st, 0z), and ag ~ TT-1sx). In the control setting, expectation-based RI focuses on the action- value function Q (s,ss), a), the expectation of (s, (s, a), i.e., Q"((s, a) = E. [2 (s) Distributional RL, on the other hand, focuses on the action-value distribution, the full distribution of (s) The density function if exists of action-value distribution is called action-value density function.
Bellman Operators vs Distributional Bellman Operators. For the policy evaluation in expectation-bused RL, the value function is updated via the Bellman operator T"Q(s,a) = [[](s,a]] + +.Ber (s', a')]- We also define Bellman Optimality Operator Qs.)
B[R(s,a)] + 7 max Eg'=p IQ (8', a')]- In distributional RI,, the action-value distribution of Z" a) is updated via the distributional Bellman operator 'I", i.c., I=Z a) 2 R(s, + yZ (s', a'), where s' ~ P(-|s, a) and a' ~ (-Is'). The equality implies that random variables of both sides are equal in distribution. This random-variahle definition of distributional Bellman operator is appealing and easily understood due to its concise form, although its value-distribution definition is more mathematically rigorous (Rowland et al. 2018: Bellemare et al. 2022).
Categorical Distributional RL. Categorical Distributional RL (Bellemare et 2017a) can be viewed as the first successful distributional RL algorithm family that approximates the value dis- tribution 7) by a discrete categorical distribution v) = E P:o.., where 21 S 22 S S 2N is set of fixed supports and (p:)/ are learnable probabilities. The leverage of a heuristic projection operator He (see Appendix [ for more details) as well as the KI. divergence allows the theoretical convergence of categorical distribution RI. under Cramér distance (Rowland et al. 2018).
REGULARIZATION EFFECT OF DISTRIBUTIONAL RL
3.1 DISTRIBUTIONAL RL: NEURAL FITTED Z-ITERATION (NEURAL FZI)
Expectation based RL: Neural Fitted Q-Iteration (Neural FQI.. Neural FQI (Fan et al. 2020 Riedmiller 2005) offers statistical explanation DQN (Mnih et al., 2015), capturing its key features, including experience replay and the target network Qo-. In Neural FQI. we update param- eterized (s, a) in each iteration l in a regression problem:
1 = argmin Q. m
1 Σ 1=1
[y - Q6 (s,,))))
Q8. (5/ a) is fixed within every Turget steps to up- date target network Qu- by letting 8* = 9. The experience buffer induces independent samples ((s,, ri, <()eem)) In ideal case when we neglect the non-convexity and TD approximation
where the target y: r(s,, az) + m
2

---

Under review as a conference paper at ICLR 2023
errors, we have Q/+11 T Φa.; which is exactly the updating rule under Bellman optimality operator (Fan et al. 2020). In the viewpoint of statistics, the optimization problem in Eq. in each iteration is standard supervised and neural network parameterized regression regarding Qo.
Distributional RL: Neural Fitted Z-Iteration (Neural FZI). We interpret distributional RL as a Neural Fitted Z-Iteration owing to the fact that this iteration is by far closest the practical algo- rithms and more interpretable. Analogous to Neural FQI, we can simplify value-based distributional RL algorithms parameterized by Zo into a Neural Fitted Z-Iteration (Neural FZI) as
1
n dy,,,, 28 (5s, a:),
z + argmin
Σ 2 TA
-
(2)
where the target Y, = R(s,, as) + Z (5),, with the policy #2 following the greedy rule Tz (s{) = a E 2 (s/, a']] is fixed within every Tuaget steps to update target network Zg.. dp is divergence between two distributions. Notably, choices of representation for Zg and the metric dp are pivotal for the empirical success of distributional RL algorithms (Sun et al. 2022
3.2 DISTRIBUTIONAL RL: ENTROPY-REGUILAN
NEURAL FQI
Aetion-Value Density Function Decomposition. To separate the impact of additional distribution information from the expectation of 2", we leverage a variant of gross error model from robust statistics (Huber, 2004). which was also similarly used to analyze Label Smoothing (Müller al. 2019) and Knowledge Distillation (Hinton et al. 2015). Particularly, we utilize a histgram (*'(()) with N bins to approximate an arbitrary continuous action-value density function y**((r given a state and action a as the histogram is probably the simplest approach for the density function estimate in the literature of non-parametric statistics. Given a fixed set of supports 20 S 21 S ... S 2N with the equal bin size as A, A, = (2) i 1, I...,NN N 1 with AN [2N-1, ZN], the continuous histogram density function is (z) interval that E 11"'(ss a)] falls into, i.e., E a)] E Ar. We conduct an action-value density function decomposition over (>x( as follows:
(r E A,)AD. Denote Ag as the
N Mlle Ai)A,,
(1-
Σ
(3)
where 5r.0 induces a new histogram Maxx) = Ei= 1xx A.)AA that is used to approximate
continuous density function p(r). (xx) or u(x)) aims at characterizing the impact of action-valus distribution regardless of its expectation E 1 (s. a)) on the performance of distributional RL algo- rithms. e controls the proportion between single-bin histogram 1(x E AE)/ A and fa(z), where we will later show that this single-bin histogram function is linked to Neural FQI. Before diving deeper, we begin by showing that (ixx) is a valid density function under certain e in Proposition [ Proposition Denote pr. E AB) = PE/A. Following the density function decomposition in Eq. ((x) = EN 1II A.)/A is a valid probability density function + 2 PE-
Proof is provided in Appendix B We next show that the histogram density estimator (***(x) enjoys a uniform convergence rate to approximate an arbitrary continuous action-value density function
(z) under the mild condition in Theorem ! Proof provided in Appendix C
Theorem 1. (Appreximation Analysis of fr Suppose (xx) is Lipschitz continuous and the sup- port of a random variable is partitioned by N bins with bin size A. Then
(V log N n
sup 1. (x)
0 +OP
(4)
Intermediate Role of Histogram Density Estimator. Categorical and Quantile-based distribu- tional RL are the two main families that estimate the true action-value density by the discrete cate- gorical distribution and quantile function, respectively. We analyze that histogram density estimate plays intermediate role within these two branches. (1) Connection to Categorical Distributional RL. Although the continuous histogram density estimator is in contrast to the discrete categorical distribution, in Proposition 2 (proof in Appendix D) we reveal that minimizing the KI. divergence
3

---

Under review as a conference paper at ICLR 2023
between the target density function and the histogram density estimate is equivalent to the param- eterized categorical distribution (2) Connection to Quantile-base based Distributional RL. Histogram and quantile functions are "two sides of coin". The histogram estimates the density function by giving each bin an equal amount of information, while the quantile function gives each fraction of data the same amount of information. Based on this insight, we thus argue that our analysis based on the histogram density estimate is largely general and representative in distributional RL families.
Proposition 2. (Equivalence to Categorical Distribution) Suppose the target categorical distribu- tion c Exn p, and the target histogram function h(x) Σ p;1 E A A updating the parameterized categorical distribution co under KL divergence is equivalent to updating the parameterized histogram function hg-
Distributional RL: Entropy-regularize Neural FQI. We apply the decomposition on the tar- get action-value histogram density function and choose KL divergence as dp in Neural FZI. Let H(P, Q) be the cross entropy between two probability measures P and Q, i... H(P.Q) Q) - -SEx P(z) log Q(x) dz. The target histogram density function g.. is decomposed as pr' = (1 - () E Ag)AA A + cī]().. We can derive the following -ngraiy-regularriee form for distribu- tional RL in Proposition 3
Proposition 3. Denote qo (r) as the histogram density function
dunction of Z (s. a) in Neural FZI. Based
on the decomposition in Eq. 3 and KL divergence as dp, Neural FZI in Eq. 2 is simplified as
7 argmin do
log 9o a. (A + aH HH ((())
900 (.a))
(5)
12
where = e/(1 - =) > 0 and Ag represents the interval that E [[ (Zz falls into, i.e., E (5/,
E A/-- pel.az(d)) is the resulting histogram density function in the next state action pair (s/, (x))). Proof is given in Appendix F. In Proposition 4 with proof in Appendix G we further show that minimizing the first term in Eq. 5 "almost" equivalent to minimizing Neural FQI. For the uniformity of notation, we still use s, a in the following analysis instead of St az.
Proposition (Connection between Neural FZI and FQI via Decomposition) In Eq. of Neural FZI, if the function class (Zo- 8 E } is sufficiently large such that it contains the target {Y:) where Y, = R(si, ai) + 7Zz (z())z(($)).. Minimizing the first term in Eq. 5 as A + 0 implies
P(Z (s. a) = T a)) 1.
Interpretation of Proposition 4 Given the fact that 0/+11 TU ideally in Neural FQI (an
ct al. 2020), we have 23= = Q) with probability one under the assumption in Proposition This indicates 2/²++ may take other values instead of its expectation part as E || T but the probability when these events for other values happen is O. This result establishes a theoreti- cal link between Neural FZI regarding the first term in Eq. 5 with Neural FQI.
Interpretation of Proposition 3 Based on the equivalence between the first term of Neural FZI and FQI, we therefore interpret the distributional RL form in Eq. S as regurariere Neural FQI. Thus, the second regularization term (( ()) aims at explaining the behavior difference between distributional RL and expectation- RL. It pushes q for the current state-action pair
to approximate (< for the next state-action pair, which "deducts" the expectation effect from the whole action-value distribution by leveraging of the density function decomposition technique proposed in B In summary, we interpret impacts of these two terms in Eq.5jon the distributional RL optimization as expectation effect and distributional regularization effect, respectively.
Risk-Sensitive Entropy Regularization. We attribute the behavior difference of distributional RL, especially the ability to significantly reduce intrinsic uncertainty of the environment (Mavrin et al., 2019). into the regularization term in Fq.5) According to the literature of risks in RL. (Dabney et al., 2018a), where "risk" refers to the uncertainty over possible outcomes and "risk-sensitive policies" are those which depend upon more than the mean of the outcomes, we hereby call the novel cross entropy regularization for the second term in Eq. 5 as risk-sensitive entropy regularization. This risk-sensitive entropy regularization derived within distributional RL expands the class of policies using information provided by the distribution over returns (i.e. to the class of risk-sensitive poli- cies). It should also be noted that risk-sensitive entropy regularization is indeed "risk-neutral"
4

---

Under review as a conference paper at ICLR 2023
in the sense of convexity or concaveness of utility functions, where our policy is still applying linear utility function U. defined as <(-[s) = arg max, Ez(s,a) Correspondingly, We can additionally vary different distortion risk measures to explicitly lead the policy to being risk-averse
or risk-seeking (Dabney et al. 2018a).
Remark KL divergence. As stated in categorical distributional RL in Section 2 when the cat- egorical distribution is applied after the projection operator IIc, distributional Bellman operator T has the contraction guarantee under Cramér distance (Rowland al. 2018), albeit the use of a non- expansive KL divergence (Morimura et al. 2012). Similarly, our histogram density function with the projection IC equipped with KL divergence also enjoys a contraction property due the equiva- lence between optimizing histogram function and categorical distribution analyzed in Proposition We also summarize favorable properties of KI. divergence in distributional RI. Appendix F
Remark on the Attainability of p²',=((")) In practical distributional RL algorithms, we typically use the bootstrap, e.g., TD learning, to attain the target probability density estimate 7i (z²))² based on Eq. as long as E (s, a}] exists and 6 2 1- PE in Proposition | The leverage of (d²)) and the regularization effect revealed in Eq. of distributional RL de facto establishes a bridge with maximum entropy RL (Williams & Peng 1991). on which we have a deeper analysis in Section 3.3
3.3 CONNECTION WITH MAXIMUM ENTROPY RL
Vanilla Entropy Regularization in Maximum Entropy RL. Maximum entropy RL (Williams & Peng 199T), including Soft Q-Learning (Haarnoja et al. 2017), explicitly optimizes for policies that aim to reach states where they will have high entropy in the future:
J(T) = Σ Beeeae)m (r (st, at) + (-(()))))))
En Ke (a|s;) log Ty (als,) and Pr is the generated distribution following The temperature parameter 33 determines the relative importance of the entropy term against the cumulative rewards, and thus controls the action diversity of the optimal policy learned via Eq, 7 This maximum entropy regularization has various conceptual and practical advantages. Firstly, the learned policy is encouraged to visit states with high entropy in the future, thus promoting the exploration over diverse states (Han & Sung 2021). Secondly, it considerably improves the learn- ing speed (Mei et al.] (020) and therefore is widely used in state-of-thh-ar algorithms, e.g., Soft Actor-Critic (SAC) (Haarnoja et al. 2018). Similar empirical benefits of both distributional RL and maximum entropy RL also encourage us to probe their underlying connection.
where H (To (+Ix,) =-E.
Risk-Sensitive Entropy Regularization in Distributional RL. To make a direct comparison with maximum entropy RL, we need to specifically analyze the impact of the regularization term in Eq 5 and thus we incorporate the risk-sensitive entropy regularization of distributional RL into the policy gradient framework akin to maximum entropy RL. Concretely, we conduct our analysis by showing the convergence of Distribution- Distribution-Entropy- Regularized Policy Iteration (DERPI), which is the coun- terpart for Soft Policy Iteration (Haarnoja et al.) 2018), i.e., the underpinning of SAC algorithm. In Eitton--nnttoyeegulleea
Policy Iteration replaces the vanilla entropy regulariza- tion in Soft Policy Iteration with our risk-sensitive entropy regularization in Eq. 5 from distributional RL. In the policy evaluation step of distribution- ciaribution-etrropy.o reguularree policy iteration, a new soft Q- value, i.c., the expectation of Z (s. a), can be computed iteratively by applying modified Bellman operator T which we call Distribution-Entrop Entroopy-eeguaarizeee Regularized Bellman Operator defined as
principle,
Q (st, az) (st, + > 7 [V (8t4 az)]
(8)
where new soft value function V (s1+1 St, ar) conditioned on 8t, at is defined by
V (1141/52, a.) = |-, IQ (St+1, 0r+1)] (sc+1,0(+1)
+ J(H (1 ",0" )t).
(9)
and f is a continuous increasing function over the cross entropy H. Note that in this specific tabular
setting regarding &t and o,, we particularly use % ft. (**) to approximate the true density function of Z($t, at), and fo to represent the true target value distribution regardless of its expectation,
which can normally be obtained via bootstrap estimate f ( similar in Eq. 5 The f trans- formation over the cross entropy H between S and 40 st.0t (1:) serves as our risk-sensitive entropy
5

---

Under review as a conference paper at ICLR 2023
regularization. As opposed the vanilla entropy regularization in maximum entropy RL that en- courages the policy to explore, our risk-sensitive entropy regularization in distributional RL plays a role of the reward correction augmented reward, and therefore augments the action-value func- tion Qs+,, ae) in the value-based RL and the objective function in policy gradient RI. by additionally incorporating the value distribution knowledge. As we have discussed Neural FZI above in Sec- tion 3.2, which is established on the value-based based RL, we now shift our attention to the properties of our risk risk-sensitive entropy regularization in the framework of policy gradient. In Lemmal] [ we firstly show that our Distribution-Entrrn Rerrlpp--ee Bellman operator T.¹ still inherits the convergence property in the policy evaluation phase.
Lemma 1. DDisribbution-aauropy- Entropy-Regulaiieee Regularized Policy Evaluation) Consider the distribution- entropy-regulare Bellman operator TØ in Eq. 8 and the behavior of expectation of (s a)), i.e., Q(s, a). Assume H(p"r, , -9 _.)"0))= S M for all (st, ai) E x A, where M is a constant. Define Qu+ TFQ², then Q will converge to a corrected Q-value of I as k + 0o with the new objective function defined as
T Σ Bonaarypp [r (se; ae) + 7H (prtine gg ((6")))]]]
('(z)
(10)
In Lemma we reveal that the new objective function for distributional RL can be interpreted as an augmented reward function. Secondly, in the policy improvement for distributional RL, we keep the vanilla policy improvement updating rules according to
arg max Enrwa' [QTH dt)]
TT New
(11)
E'E
Next we can immediately derive new policy iteration algorithm, called Ensrribution-Etrropp Regularized Policy Iteration (DERPI) that alternates between It will provably converge to the policy
cererionnnnnroo--eeguaiaiii pol-
icy evaluation in Eq.8 and the policy improvement in Eq. I with the optimal risk-sensitive entropy among all policies in II as shown in Theorem
Policy Iteration) Assume H( To " S M for all (st, ax) E S x A, where M is a constant. Repeatedly applying distribution-entropp enrooer-e---gellaicce policy evaluation in Eq, [ Band the policy improvement in Eq. 17 the policy converges an optimal policy nt" such that Q" (St, a.) 2 Q² (st, oz) for all II.
Theorem 2.
Please refer to Appendix H for the proof of Lemma D and Theorem 2 According to Theorem 2 it turns out that if we incorporate the risk-sensitive entropy regularization into the policy gradient framework in Eq. TO we are able to design a variant of "soft policy iteration" that can guarantee the convergence to optimal policy. As such, we provide comprehensive comparison between vanilla entropy in maximum entropy RL and risk-sensitive entropy in distributional RL as follows.
Vanilla Entropy Regularization vs Risk-Sensitive Entropy Regularization. (1) Objective function. By comparing two objective function J J(T) in Eq. 7 for maximum entropy RL and J'(x) in Eq. 10 for distribu- tional RL, distributional RL tries to maximize the risk- sensitive entropy regularization w.r.f π. This indicates that the learned policy in distributional RL is encour- aged to visit state and action pairs in the future whose action-value value distributions have higher degree of dis- persion, e.g., variance, in spite of its expectation, thus promoting the risk-sensitive exploration to reduce the intrinsic uncertainty of the environment. An intuitive illustration is provided in Figure I (2) State-action dependent regularization. The vanilla en- tropy H(((--)::) in maximum entropy RL is state-wise, while our risk-sensitive regularization H( 96 (t) is state-action-wize implying that it is a more fine-grained regularization to char- acterize the action-value distribution of (st, az) in the future.
90 (x)
# s0 (x)
(x)
Figure 1: 4 6,0 is encouraged to disperse under the risk-sensitive entropy regular- ization of distributional RL.
3.4 ALGORITHM: ESSTRBUTIONN-ENTROPP-- REGNROPP--R ACTOR-CRITIE (DERAC)
In practice, large continuous domains require us to derive a practical approximation to DERPI. We thus extend DERPI from the tabular setting to the function approximation case, yielding the

---

Under review as a conference paper at ICLR 2023
Distribution-Entrr Entropy-Regulariin Actor-Critic (DERAC) algorithm by using function approxima- tors for both the value distribution go( ae) and the policy (a(aj) st). The key characteristics of DERAC algorithm is that
function approximator to represent the whole value distribution qe Qo(st, ax) = E (4o (+)]
rather than only the value function, and conduct the optimization mainly hased on the value function
Optimize the parameterized value distribution qa- The new value function is originally trained minimize the squared residual error of Eq. 8 Here for desirable interpretation, we impose the zero expectation assumption over the residual, i.e., 7 Qa(s, a) - Qa a) + b with EE [b] = 0. The resulting simplified objective function (0) can be well interpreted as an interpolation between the expectation effect and distributional regularization effect:
(0) Es,a [TT Qo (s. Qo(s,a))
(12)
x (1 2) [T I [go- (s, a)]l - IE [go(s, a)))² + AB.,a [H(p % 16,)),,
where the results is simplified by using a particular increasing function f(H)) = (AZL) // and [0, 1] is the hyperparameter that controls the risk-sensitive regularization effect. Interestingly, when we leverage the whole target density function 570 to approximate the true n² the objective function Eq. 12 can be viewed as an exact interpolation of loss functions between expectation- based RL (the first term) and categorical distributional RL loss (the second term), c.g., C51. Note that for the target TE [go. (s, a)], we use the target value distribution neural network ga- to stabilize the training, which is consistent with the Neural FZI framework analyzed in Section 3.1
Optimize the policy Ts. We optimize T in Eq. based on the Q(s, a) and thus the new objec- tive function (0) can be expressed as 5x (0) E, B.awm m E [go(s, a)]]. The complete DERAC algorithm is described in Algorithm of Appendix
EXPERIMENTS
In the experiment, we firstly verify the regularization effect of distributional RL analyzed in Sec- tion B.2 decomposing the action-value histogram density function via Eq. 5gn both Atari games and MuJoCo environments. Next, we demonstrate the convergence and favorable performance of DERAC algorithm on continual control environments. Finally, an empirical extension to Implicit Quantile Networks (IQN) is provided to reveal mutual impacts of different entropy regularizations.
To demonstrate the value distribution decomposition, we mainly present results on three Atari games, including Breakout. Seaquest, Hero, over 3 seeds and three continuous control MuJoCo environments in OpenAl Gym, including ant, swimmer and bipedalwalkerhardcorrs over seeds. For the extension IQN, we perform experiments eight MuloCo environments.
Environments.
Baselines. To evaluate the risk-sensitive entropy regularization effect of distributional RL, we conduct ablation study on C51 (Bellemare et al.) 2017a) Atari games and distributional SAC (DSAC) (Ma et al. (2020) on MuJoCo environments. The implementation of DERAC algo- rithm is based on distributional SAC (Haarnoja et al. 2018 Ma al. 2020). More implementation details are provided in Appendix!! [
4.1 DISTRIBUTION REGULARIZATION EFFECT OF DISTRIBUTIONAL RL
We demonstrate the rationale of action-value density function decomposition in Eq. and the distri- bution regularization effect analyzed in Eq. based on C51 algorithm equipped with KL divergence. Firstly, it fact that the value distribution decomposition is based on the equivalence between KL divergence and cross entropy owing to the leverage of target network. Hence, we demonstrate that C51 algorithm can still achieve similar results under the cross entropy loss across four Atari games in Figure of Appendix K In both the value-based C51 loss and the critic loss in DSAC with C51, we replace the whole target categorical distribution (xx) in C51 with the derived (x* under different in the cross entropy loss, allowing to investigate the risk-sensitive regularization effect of distributional RL. Concretely, we define as the proportion of probability of the bin that contains the expectation with the mass to transport to other bins. We use to replace for convenience the leverage of can always guarantee the valid density function ğ analyzed in Proposition [ A large proportion probability that transports less mass other bins, corresponds large in Eq.
7

---

Under review as a conference paper at ICLR 2023
Breakout
Seaquest
Hero
5000
DON
DON
Average Return
Average Return
4000
1003
Min mulle 0.50
Mpl nol 0.5)
300
Average
Mar quite 0.35
5000
Map mode
200
100
2000
2003
0.3
0.2
0.6
3.5
0.5
0.0
0.2
64
0.6
0.5
0.0
082
6.5
38
Time Steps (1e7)
Time Steps (1e7)
Time Steps (1e7)
ant
swimmer
bipedallallkeraardcore
4000
Average Return
.5000
Average Retum
100
Average Retum
1.50
qalle-
1000
SAG
DSAC
His fullc 0.11
qulte- 9:31-
60
80
Mits Rulle
Miu Galle
-50
50
WV
BAC
DSAC
-1000
Mij qall--lll---
0.2
3,4
0.5
0.5
1.0
0.
0.6
0.6 0.5
1.0
02
C.4
0.6
5.
To
Time Steps (1e6)
Time Steps (1e6)
Time Steps (1e6)
Figure 2: (First Row) Learning curves of C51 with value distribution decomposition H(p, qa) under different E on three Atari games over 3 seeds. (Second Row) Learning curves of C51 with value distribution decomposition H(H, qo) under different & on three MuloCo environments over 5 seeds.
As shown in Figure 2 when & gradually decreases from 0.8 to 0.1, the learning curves of C51 H(,, qo) tend to degrade from vanilla C51 to DQN across both Atari and MuloCo, although their sensitivity in terms of & may depend on the environment, e.g., bipedalwalkerharddorr. This em- pirical observation corroborates the theoretical results we derive in Section 3.2 suggesting that risk-sensitive entropy regularization pivotal to the success of distributional RL algorithms.
ant
swimmer
bipeddawalkerhardcore
6300
Average Return
Average Return
Avarage Return
200
4300
DEACIOS1)
3000
60
1003
SAC
60
1000
DBA010811
-2000
0.2
0.A.
0.6
0.8
1.0
1.0
Time Steps (1e6)
Time Steps (1e6)
Time Steps (1e8)
Figure 3: Learning curves of DERAC algorithms on three MuloCo environments 5 seeds.
4.2
CONVERGENCE OF DERAC ALGORITHM
We further demonstrate the convergence of DERAC algorithm. Figure showcases that DE- RAC converges and achieves desirable performance on MuloCo environments compared with AC (SAC without vanilla entropy) in the blue line. More importantly, Distribution Entropy-- Regularization (DER) in the red line could be remarkably beneficial for learning on the complex Bipedallaalkerhardcor,, where risk-sensitive exploration significantly improves the performance. is worthwhile know that our goal to introduce DERAC algorithm is not to pursue the empiri- cal superiority of performance, but to corroborate the theoretical convergence of DERAC algorithm and DERPI Theorem 2 addition, as we choose & 0.9 in DERAC algorithm, there exists a distribution information loss, resulting in the learning performance degradation, e.g., on Swimmer. In practice, we can directly deploy distributional SAC to seck for a better performance. We also provide sensitivity analysis of DERAC regarding in Figure of Appendix K
4.3 EXTENSION TO QUANTILE-BASED DISTRIBUTIONAL RI.
Finally, due to the fact that our aforementioned theoretical analysis is closely connected to cate- gorical distributional RL algorithms, e.g., C51, in order to make comprehensive conclusion in
8

---

Under review as a conference paper at ICLR 2023
ant
humanoid
walker2d
bipedalwalkerien
Aycrage Return
5600
AC--IE
6990
ACWVE
5000
250
200
1000
4030
4000
150
5000
100
3000
2000
2000
1000
1000
1000
-100
0.2
0.4
0.5
0.5
1.0
0.2
04
3,6
0.
1.0
0.2
2.4
0.5
0,5
0.2
0.4
0.5
1.0
Time Sugge (1e5)
Timee Steps 1100
Timee (ieze(led)
Tiae Stape (led)
halfcheetah
reacher
swimmer
humanoidstandup
10000
100
190020
Average Return
1000
140000
6000
120000
-40
4000
-390
100000
2000
-125
ACrit
80300
AC-VE
-340
AC-30
AC-25
AC-RE
AC- 12
082
0.6
0.6
1.0
-350
0.2
04
0.A.
08
1.0
0.2 04
0.8
40000
0.2
6.4
0X
1.0
Time Sugel (1:5)
Tiae Steps (1e0)
Tite (tepa(l66)
Tine (lep)
Figure 4: Learning curves of AC, AC+VE, AC+RE and AC+RE+VE over 5 seeds with smooth size 5 across eight MuloCo environments where distributional RL part is based on IQN.
broader distributional RL branches, thus heuristically extend results in guantile-based dis- tributional RL. This investigation is also based on the intermediate role of histogram function we analyzed in Section 3.2 Specifically, careful ablation study is conducted to control the effects of vanilla entropy (VE), risk-sensitive entropy (RE) and their mutual impact. We denote SAC with and without vanilla entropy as AC and AC+ VE, and distributional SAC with and without vanilla entropy as AC+RE+VE and AC+RE, where VE and RE are short for Vanilla Entropy and Risk-sensitive Entropy. For the implementation, we leverage the quantiles generation strategy in IQN (Dabney et al. suggested in Figure f although both vanilla entropy and tisk-sensitive entropy effects may vary for different environments, we make the following conclusions:
2018a in distributional SAC (Ma et al.) (020). (lyper-paramette are listed in Appendix As
(1) Vanilla entropy effect can enhance the performance as easily observed that AC+VE (blue line) outperforms AC (red lines) across most environments except on the humanoid and swimmer. The risk-sensitive entropy effect (RE) from distributional RL is also able to benefit the learning due to the fact that AC+RE (black lines) is more likely to bypass AC (red lines) especially on the complex Bipeal Bapeaa environment (hard for exploration).
(2) The use of both risk-sensitive entropy and vanilla entropy may interfere with each other, e.g., on BikealWalkerrarddorr and Swimmer games, where AC+RE+ VE (orange lines) is significantly inferior to AC+RE (black lines). This may results from the different exploration preference of two regularization effects. SAC encourages the policy to visit states with high entropy to pursue the diversity of states to optimize, while distributional RL promotes the risk-sensitive exploration to visit state and action pairs whose action-value distribution has larger degree of dispersion. We hypothesize that mixing two different exploration directions may lead sub-optimal solutions in certain environments, thus interfering with each other eventually.
DISCUSSIONS AND CONCLUSION
Our regularization interpretation is based on histogram function equipped the KI. divergence, strongly connected with categorical distributional RL. Although the histogram is linked with quan- tile function as well, a direct analysis based on quantile function is also promising in the future.
In this paper, we illuminate the behavior difference of distributional RL over expectation-based RL from the perspective regularization. A risk-sensitive entropy regularization is derived for distributional RL within Neural FZI to explain the potential advantage of distributional RL. We also establish a connection between distributional RL with maximum entropy RL. Our research contributes to deeper understanding of the potential superiority of distributional RL algorithms.

---

Under review as a conference paper at ICLR 2023
Ethics Statement. As our study is related to reveal the regularization effect of distributional RL algorithms, it not involved with any ethics issue in our opinion.
Reproducibility Statement. Our results based on the public implementation released (Ma et al. 2020) with necessary implementation details given in Appendix We also provide the detailed proof in Appendix.
REFERENCES
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. International Conference on Learning Representations, 2017, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. International Conference on Machine Learning (ICML), 2017a
Mare G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The cramer distance as a solution to biased wasserstein gradi- ents. arXiv preprint arXiv:1705. 10743, 2017b.
Marc G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinorrmment Learning, MIT Press, 2022. http:://wwz wwwwwritttttsiiaaaa...o
org
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. International Conference on Machine Learning (ICML), 8a.
Will Dabney, Mark Rowland, Mare G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. Association for the Advancement of Artificial Intelligence (AAAI), 2018b.
Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q- learning. Learning for Dynamics and Control, pp. 486 486-489, 489. PMLR, 2020.
Tuomas Haarnoja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pp. 1352-1361.
PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.055: 2018.
Seungyul Han and Youngchul Sung. A max-min entropy framework for reinforcement learning. Advances in neural information processing systems (NeurIPS). 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 10Xiv:1503.02533 2015.
Peter Huber. Robust Statistics, volume 523. John Wiley Sons, 2004.
Clare Lyle, Marc G Bellemare, and Pablo Samuel Castro. A comparative analysis of expected and distributional reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2019.
Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang. and Qianchuan Zhao. Dsac: Distributional soft actor critic for risk- reinforcement learning. arXiv preprint arXiv:2004.14547, 2020.
Yecheng Jason Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional rein- forcement learning. arXiv preprint 1rXiv:2107.0611
2021.
10

---

Under review as a conference paper at ICLR 2023
John Martin, Michal Lyskawinski, Xiaohu Li, and Brendan Englot. Stochastically dominant distribu- tional reinforcement learning. In International Conference on Machine Learning, pp. 6745-6754. PMLR. 2020.
Borislav Mavrin, Shangtong Zhang, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. International Conference on Machine Learning (ICML), 2019.
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference Machine Learning, pp- 6820-6829. PMLR, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 51 3(77540)522-533. 2015.
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Parametric return density estimation for reinforcement learning. arXiv preprint arXiv: 1203.3497 2012.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint NEK.vvI90060262.. 2019.
Thanh Tang Nguyen, Sunil Gupta, and Svetha Venkatesh. Distributional reinforcement learning with maximum mean discrepancy. Association for the Advancement of Artificial Intelligence (AAAI), 2020.
Martin Riedmiller. Neural fitted q iteration first experiences with data efficient neural reinforce- ment learning method. In European conference on machine learning, pp. 317-328. Springer, 2005.
Mark Rowland, Mare Bellemare, Will Dabney, Rémi Munos, and Yee Whye Teh. An analysis of categorical distributional reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 29-37. PMLR, 2018.
Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Marc G Bellemare, and Will Dab- ney. Statistics and samples in distributional reinforcement learning. International Conference on Machine Learning (ICML), 2019.
Ke Sun, Yi Liu, Yingnan Zhao, Hengshuai Yao, Shangling Jui, and Linglong Kong. Exploring the robustness of distributional reinforcement learning against noisy state observations. arXiv preprint ar arXiv:2109.087 2021.
Ke Sun, Yingnan Zhao, Liu, Bei Jiang, and Linglong Kong. Distributional reinforcement learning via sinkhorn iterations. arXiv preprint arXiv:2202.00777 2022.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An Introduction. MIT press, 2018.
Larry Wasserman. All of nonparametric statistics. Springer Science & Business Media, 2006.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-292, 1992.
Ronald J Williams and Jing Peng. Function optimization using rennn learning algorithms. Connection Science, 3(3):241-268, 1991.
Derek Yang, Li Zhao. Zichuan Lin, Tao Qin, Jiang Bian, and Tic- Yan Liu. Fully parameterized quan- tile function for distributional reinforcement learning. Advances in neural information processing systems, 32:6193 220222 2019.
Fan Zhou, Jianing Wang, and Xingdong Feng. Non-crossing quantile regression for distributional reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
11

---

Under review as a conference paper at ICLR 2023
A CONVERGENCE GUARANTEE OF CATEGORICAL DISTRIBUTIONAL RL
Categorical Distributional RL (Bellemare et al. 2017a) uses the heuristic projection operator IIc that was defined as
8
21 2i y S 2+41 y > 2
IL (6y)
82, #441
8244
(13)
Ri-1
OaK
and extended affinely finite mixtures of Dirac measures, so that for mixture of Diracs E p: we have Te (Zim1 = - P:lle ( The Cramér distance was recently studied as an alternative to the Wasserstein distances the context of generative models (Bellemare) al. 2017b). Recall the definition of Cramér distance.
Definition 1. (Definition 3 (Rowland al. (018)) The Cramér distance l between two distribu- tions V1, E (RR, with cumulative distribution functions Fuy Fuz respectively, is defined by:
1/2
82 (2), (F Fey
dr
Further, the supremun-Cramét metric &g defined between two distribution functions 1). 11 9P ( by
12(n.p)= sup (2,0)EXXA
le (1(101) (
Thus, the contraction of categorical distributional RL can be guaranteed under Cramér distance: Proposition 5. (Proposition 2 (Rowland ef al. 2018 The operator TcT" is a V5-contraction in lz.
An insight behind this conclusion is that Cramér distance endows particular subset with tion of orthogonal projection, and the orthogonal projection onto the subset is exactly the heuristic projection IT (Proposition in (Rowland et al. 2018))
B PROOF OF PROPOSITION
Proposition
I Denote ( E Δς) PE/A. Following the density function decomposition in Pf E A.)AA is valid probability density function < 2 Pg-
Eq.
Proof. Recap a valid probability density function requires non-negative and one-bounded probabil- ity in each bin and all probabilities should sum to 1.
Necessity. (1) When x E F. Eq. 3 can simplified as pF/ (1 <)/ A t A, where //t Āa( Ag). Thus, // PE
2 if 2 1 1PPE. Obviously, // PE 1 < 1-t
guaranteed by the validity of PE (2) When x & AE. we have pi/ = ep." AA i.e., When x & AE.
We immediately have p."
P. <
Pg < 1 when 2 - PE. Also, 1/," = E 22
Sufficiency, (1) When x E Ar, let / Pg - 2 0, we have 2 1 PE. 1td P3 I-c <
in nature. (2) When x & AE. D. P: 2 0 in nature. Let / = P < 1, we have p; - E. We need
to take the intersection set of (1) and (2), find that > Pe = 2 1 Pe Z P: that satisfies
condition in (2). Thus, the intersection set of (1) and (2) would be 2 PE-
In summary, 2 1 pr is both the necessary and sufficient condition, we have the conclusion that (ix) is a valid probability density function < 2 1 PF-
12

---

Under review as a conference paper at ICLR 2023
C PROOF OF THEOREM 1
Theorem 1 Suppose p* (a) is Lipschitz continuous and the support of X is partitioned by N bins with bin size A. Then
log N n
sup 1 (x) " (x)] (00)
(14)
Proof. Our proof mainly refers to Wasserman 2006 pai (x) can be written as
In particular, the difference of (*m(())
(^'0()) (x) (x) (F (x)) p(a)
(x) - (T)*
(15)
bias
stnchastic variation
(1) The first bias term. Without loss of generality, we consider æ E Ar. we have
(x)) P(X Ax)
tk - 1)A ply)dy
(16)
F(20 + (k: 1)4) F(zo (k- 1)4)
20 + kΛ (z0 + (k - 1)A)
->*a(x" (xx),
where the last equality is based on the mean value theorem. According the L-Lipschitz continuity property, we have
E (x)) (x)) (1"0((")) pr (x))
L|2x xx
(17)
$ LA
(2) The second stochastic variation term. we let x As, then fr.a Pk 1 E 1(x,
A we thus have
P
sup 1 (x) E (5- (x))))))>
())AA >E)
P
max
jm N
im1
P
max
Σ - > >AC d=1
(18)
j=1," .N
Σ
- P(X >
< N exp (-2mA2e2) (by Hoeffding's incquality),
where in the last inequality we know that the indicator function bounded in [0, 1J. We then let the last term be constant independent of N, n, A, thus,
(V
VLog log N
sup
i a (z) E (f>0())) =Op
nA²
(19)
In summary, as the above inequality holds for each x, we thus have the uniform convergence rate of histogram density estimator
sup 1
(x)) sup
E (T (x)) ( (x)) sup (x) E ( (x:))
log
(20)
= =
n
13

---

Under review as a conference paper at ICLR 2023
D PROOF OF PROPOSITION 2
Proposition 2 Suppose the target categorical distribution c= E=E== prò., and the target histogram function h(x) = E p; E A.)A,, updating the parameterized categorical distribution Gø
under KL divergence equivalent to updating the parameterized histogram function ha-
Proof. For the histogram density estimator he and the true target density function p(æ), simplify the KL divergence as follows.
N
Pi
DKL h2) :0)= =
log
dæ
1=1
(3
(1
pi(x)
h
log
de
log
dx
Z6-1
Z1-1
(21)
N
(a(a)
h
log
de
A
im
N
h
N
(1(x)
log
Σ
A
where hy is determined by i and and independent æ. For categorical distribution estimator co with the probability p: in for each atom 2i, we also have its target categorical distribution p(x) with each probability pt, we have:
N
Pr
(KI co)
Ps log
1=1
N
Ps log Pi
Σ 1=1
pa log cj
(22)
i=1
pj log cp
In categorical distributional RL we only discrete categorical distribution with probabilities centered on the fixed atoms { while the histogram density estimator in our analysis continuous function defined [20, EN] We reveal that minimizing the KL. divergence regarding the parameterized categorical distribution in Eq. 22 is equivalent to minimizing the cross entropy loss regarding the parameterized histogram function Eq.
E PROPERTIES OF KL DIVERGENCE IN DISTRIBUTIONAL RL
Proposition Given two probability measures μ and v, we define the supreme DRL. as a functional P(Z)SxA x P(X)SXAA + IR, i.e., D&C (u,v) sup Dxr (u(x, a), v(x, a)).
have;
(1) T is non-expansive distributional Bellman operator under D i.e., Dog Zı, I Zz) S (Z,, Z2). (2) DR (Z,, Z) > O implies the Wasserstein distance Wp (Zn, Z) + 0, (3) the ex- pectation of 7. still 7-contractive under D&g i.e., |ES Z1 ET" 72 < 7 |EZ1 B.Z
Proof. We firstly assume Zo is absolutely continuous and the supports of two distributions in KL divergence have negligible intersection (Arjovsky Bottou 2017), under which the KL divergence is well-defined.
14

---

Under review as a conference paper at ICLR 2023
(1) Please refer (Morimura al. 2012) for the proof. Therefore, have DiL (I Z1, 5 Z2) -
Rii (Z1, Z2), implying that ' Is non expansive operator under Dki.
(2) By the definition of Di. we have sups.a ( a), (s,, a)) 0 implies D (Zn, 2)
0. DKL (Zn, Z) + 0 implies the total variation distance ö(Zn, Z) + 0 according to straightfor-
ward application of Pinsker's s inequality
8 ( 7) S V DKI. (Z ?) - 0
(23)
8 (2, - V DKL (Z, Zn) -
Based on Theorem 2 in WGAN (Arjovsky et al., 2017). 8( 2) implies Wp 2) + 0. This is trivial by recalling the fact that ò and W give the strong weak topologies on the dual of (C(X), (lo)) when restricted to Prob(X().
(3) The conclusion holds because the I" degenerates T regardless of the metric dp (Bellemare ct al. 2017a). Specifically, due to the linearity of expectation, we obtain that
ET Z1 ET 2o ET T'EZal> EZ|
EZ2
(24)
This implies that the expectation of 2 under DxL. exponentially converges to the expectation of 2", ie., conttaaciion.
F PROOF OF PROPOSITION 3
Proposition 3 Denote do (x) the histogram density function of Zh (s, a) in Neural FZI. Based on the decomposition in Eq. 3 and KI. divergence as dp, Neural FZI in Eq. 2 is simplified as
Z argmin
log q0 si (Ai)) + aH(pfiazle)
.00 (
(25)
72
Proof. Firstly, given fixed p(z) we know that minimizing (pL 90) is equivalent to minimizing H(p, q) by following
p'(x)/A
(RL(p, qo) =
Σ
pi(a)/ A log
dr
/
N Σ
N
pi(x)/ A log / A dx
p (1(z)/Alo log//// dz) (26)
1=1
1=1
H((p 90) H((p) H(pp
Based on H(p, qo), we use (x))((((((x)) to denote the target probability density function of the random variable R(s;, a.) 7.J. (8/, (z())) Then, derive the objective function within
15

---

Under review as a conference paper at ICLR 2023
each Neural FZI as
H(p ( (x))
(1 (1-c)H(1(z EEE 40 "/.0// /2) + eH(l / 98 10 //.00/ /4))
N -11 E)
Σ
1(x E ( 22)
90 2..0 (Aj)/Adz Σ
(A))) log "0'0 (AA))AA
35-1
j=1 2/-1
log 2 . (A)) p log (A,))/ (1- c)(
(1 c) log 9Ø 1i (A))) + HH(T 90 )1.)a.))
n
10 "1 (Ai)) aH )g.a)) where a 1
log
n1
(27)
9g (1 where the cross entropy H(77(,z2(6))
) is based on the discrete distribution when i = =1, 1,... N.
A' represent the interval that E [2,, (T falls into, i.e., E [Z" (s/ #2 (x())) E A L
G PROOF OF PROPOSITION
Proposition 4 In Eq. of Neural FZI, if the function class {Zo E O}} is sufficiently large such that it contains {Y:) ) A + 0 (N +oo), we have
p(z (s. a) = T Q a)) =1
(28)
where T (s, a) is the target in Eq. of Neural FQI.
Proof. Firstly, we define the distributional Bellman optimality operator T as follows:
T ((s 2 a(s a²) S s(),s,a,, a" argmaxx [Z a')
(29)
If [Zo :
}} } is sufficiently large enough such that it contains TTp Zo-, then optimizing Neural FZI in Eq. 2 leads to Z TTp 20-
We apply the action-value density function decomposition on the target histogram function (xx) Consider the parameterized histogram density function he and denote 1/ / / as the bin height in the bin Ag. under the KL divergence between the first histogram function (æ Ag) with he (x), the objective function is simplified
hE log A
DRL(1(x e Ae)/A, x -
dr
JEAD
(30)
α log h
Since {Zo 0 E } is sufficiently large enough, the KL minimizer would be he = 1(x 1 E AE)AA in expectation. Then, arg min,,, lima-+0 DKL (1 (x E AE)/A, ha(z)) = )g.2uue(s.a))- where )).2))((s.a)) is a Dirac Delta function centered at E (s, a)] and can be viewed as a gener- alized probability density function. The limit behavior from histogram function ÿ to continuous one for Lurget is guaranteed by Theorem [ and this also applies from hg to Zg. In Neural FZI, we have uageet Zp.. According to the definition of Dirac function, as A - O, attain
P(Z)+1 (s, a) E Ih 2 (s, a)))
(31)
16

---

Under review as a conference paper at ICLR 2023
Due to the linearity of expectation analyzed in Lemma 4 of (Bellemare al. 2017a), have
E I (s, as
I [2g (s,a)]
(32)
=T (s,
Finally, we obtain:
(
as 0
(33)
H PROOF OF CONVERGENCE OF SOFT DISTRIBUTIONAL POLICY ITERATION IN THEOREM 2
H.1 PROOF OF SOFT DISTRIBUTIONAL POLICY EVALUATION IN LEMMA
Lemma Battrrbbttion-Ettoo----eeee
Policy Evaluation) Consider the distributiion-ettooyy- regularized Bellman operator T in Eq. 8 and the behavior of expectation of (" a), i.e., Q(s, a).
Assume H 96 g.))) < M for all (5t, ar) E S x A. where M is a constant. Define Q
THQK, then Q will converge to a corrected Q-value of as k + co with the new objective function defined
T
J'(T)
= Σ
E E(ara))pp, ir (St, ad) + 7H (prtir ,4 ()))))))
(34)
t=
Proof. Firstly, we plug in (88+1) into RHS of the iteration in Eq. 8 then we obtain
Tí Q (s), az)
= r az) + IV (st+1)]
(35)
= =r(s;, at) + +7 (1c 1 )))))) + (1)))))))))) IQ (8141,
A rx (st, + + IQ ($141, a141)],
where (st, 0x) 2 r (St, a²) (Hl (n 9o ()) is the entropy augmented reward we redefine. Applying the standard convergence results for policy evaluation (Sutton & Barto 2018), we can attain that this Bellman updating under T is convergent under the assumption of [ < co and bounded entropy augmented rewards T=
H.2 POLICY IMPROVEMENT WITH PROOF
Lemma 2. (Disriibution-Ettrooy-Reeuuu
Policy Improvement) Let ग II and new policy
Tnew be updated via the policy improvement step in Eq. 11 Then Queee (st, a,) 2 Q Fai (st, as) for all (st, a.) E S x A with [A| sc.
Proof. The policy improvement in Lemma 2
implies that Ear TT (St, az)
Earwaa (Q a,),, we consider the Bellman equation via the distribution reg.lay--rguuueieed Bellman operator Tea:
The (St, 0z) A r (St, at) "Est4imp [V (S141)]
r (St, az) 7H (1et100, ,00 Fe ()()e))) + 71E ( are drop 1Quau (8t4 ax+1)
r (St, at) 7H (1er 16 ((ar)) + 71 (a4 propteo [Q TMM (st+1, Gc+1)]
Fa (st, 0x) + 1
TM (8t+1, 4441)]
(36)
< TT (5441,0++1),
where we have repeated expanded Tau on the RHS by applying the beapobutionnettoo--rrnogoe distributional Bellman operator. Convergence TTe follows from Lemma
17

---

Under review as a conference paper at ICLR 2023
H.3 PROOF OF SOFT DISTRIBUTIONAL POLICY ITERATION IN THEOREM 2
Policy Iteration) Assume H( Lo "0 S M for all (st; (t) É S A, where M is constant. Repeatedly applying instrbbtiioneettooy--regg [ and the policy improvement in Eq. TT the policy converges to an optimal policy x¹ such that Q" (st, 0e) 2 Q" (8t, ae) for all x E IT.
Theorem D
policy evaluation Eq.
Proof. The proof similar soft policy iteration (Haarnoja et 2018). For the completeness, we provide the proof here. By Lemma 2 as the number of iteration increases, the sequence Q at i-th iteration is monotonically increasing. Since we assume the risk-sensitive entropy bounded by M, the Q" is thus bounded as the rewards are bounded. Hence, the sequence will converge some n² Further, we prove that x is in fact optimal. At the convergence point, for all π II, it must case that:
Ba IQ (st,at)] 2 Ear TQ (st, [s))
According to the proof in Lemma 2 say, the "corrected" value function of any other policy in II is lower than the converged policy, indicating that x is optimal.
we can attain Q" (st, ac) > (st, a.) for (st, at). That is
I IMPLEMENTATION DETAILS
Our implementation is directly adapted from the source code in (Ma et al. 2020).
For Distributional SAC with C51, 51 atoms similar to the C51 (Bellemare al. 2017a). For distributional SAC with quantile regression, instead of using fixed quantiles in QR-DQN we
Table 1: Hyper- pappee-prameterss Sheet.
Hyperparameter
Value
Shared
Policy network learning rate
3e-4
(Quantile) Value network learning rate
3e-4
Optimization
Adam
Discount factor
0.99
Target smoothing
5e-3
Batch size
256
Replay buffer size
1e6
Minimum steps before training
le4
DSAC with C5I
Number of Atoms (N)
51
DSAC with IQN
Number of quantile fractions (N)
32
Quantile fraction embedding size
64
Huber regression threshold
Hyperparameter
Temperature Parameter 9
Max episode lenght
Walker2d-v2
0.2
1000
Swimmer-v2
0.2
1000
Reacher-v2
0.2
1000
Ant-v2
0.2
1000
HalfCheetah-v2
0.2
1000
Humanoid-v2
0.05
1000
HumanoidStandup-v2
0.05
1000
BipedalWalkerHaattcneee22
0.002
2000
18

---

Under review as a conference paper at ICLR 2023
leverage the quantile fraction generation based on IQN (Dbnnyy et al. 2018a) that uniformly samples quantile fractions in order to approximate the full quantile function. In particular, we fix the number of quantile fractions N and keep them in ascending order. Besides, adapt the sampling as Tu O, e/ _
where 6; U|0 1], N,,,.. N.
I.1
AND NETWORK STRUCTURE.
We adopt the same hyper- syper-parameters, which is listed in Table 1 and network structure as in the original distributional SAC paper (Ma al. 2020).
DERAC ALGORITHM
Algorithm Distribution- Hatrrion---ntooo--eegaaiiieeee Actor Critic (DERAC) Algorithm
Initialize two value networks qo: 96-, and policy network TΦ.
2: for each iteration do
3: for each environment step do
4:
(ar(s)))
5: 8:+1 ~ p($141155 at).
6: D DU {(st, a,, (st, az) $(+1)}
7:
end for
8:
for each gradient step do
9:
0 - 0 >g (0)
o+ v (o). ++ T0 (1 -)
11:
end for
13: end for
K
EXPERIMENTS
Figure suggests that C51 with cross entropy loss behaves similarly to the vanilla C51 equipped with KL divergence.
Breakout
Seaquest
Hero
12000
Average Return
400
CS1
DQN
7000
C51
Average Return
CS1
5000
O51_CE
8900
OS1_CE
Average
6900
3000
6.0
0.2
04
0.5
5.8
00
0.4
0.8
0.0
0.2
0.4
0.5
Time Steps (1e7)
Time Steps (1c7)
Time Steps (1e7)
ant
swimmer
bipedalwalkerhartccor
Average Return
6003
DSAc
Dexe
SaE
Averago Return
120
100
AM
D6AC
TRAC
Average Return
1.5
80
50
2003
1007
40
-1003
20
0.7
B.C
0.6
1..5
1.0
6.4
0.5
1.0
0.2
1.0
Time Stepo (1e5)
Time Stops (1.5)
Tiate Stepo Cleft
Figure 5: (First row) Learning curves of C51 under cross entropy loss Atari games over seeds. (Second row) Learning curves of DSAC with C51 under cross entropy loss on MuJoCo environments over 5 seeds.
Figure 6 shows that DERAC with different À in Eq. 12 may behave differently on the different environment. Learning curves of DERAC with increasing À will tend to DSAC (C51), e.g.
19

---

Under review as a conference paper at ICLR 2023
Bipeddllalkkerharddone,, where DERAC with À = in the green line tends to DSAC (C51) in the blue line. However, DERAC with a small À is likely to outperform DSAC (C51) by only leverage the expectation effect of value distribution, e.g., Bipedalwalkkerarddoor, where DERAC with
0, 0.5 bypass DERAC with À = 1.0.
swimmer
bipedalwalKerhardcore
120
BAC
Average Return
-005
4000
Average Return
100
DERACU - 1.0.
DEACICE1)
2000
DERACI -
DERACI
Average
100
1000
DERAC 100.2-10)
DSACYCSS)
DBACIOS1)
DERACjE=100 -102203)
-1000
VERAGiE=2:0: 1:022-000)--
198
DERAG 1.0.
10
Time Steps (1e6)
Time Steps (1c6)
Time Steps (1e6)
Figure 6: Learning curves of DERAC algorithms across different À on three MuJoCo environments over seeds.
20


Reviews:
[{"review_id": "qWlH5k5X2ed", "reviewer": "Reviewer_wnYb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a new regularization perspective on analyzing why the distribution RL methods perform better than the expectation-based RL methods empirically. By leveraging the decomposition assumption of the action-value distributions, this paper proposes to express the objective function of NeuralFZI as the expected effect and the distributional regularization effect. Based on this decomposition, the author(s) proposed DESPI, which is a variant of the soft policy iteration for distributional entropy regularization. Accordingly, to substantiate DESPI in more practical RL tasks, this paper presents a learning algorithm called DERAC, which performs well in some complex continuous control tasks.", "summary_of_the_review": "Overall this paper tackles an important question in distributional RL from a novel perspective. My main concerns are the critical assumption of the value distribution decomposition mentioned above as well as that the experiments look quite inconclusive and require more explanation.", "strength_and_weaknesses": "**Strength**\n\n- A new perspective of interpreting distributional RL: This paper aims to answer an important question in the distributional RL literature: why do the distributional RL methods have an empirical advantage over the expectation-based RL methods? This paper proposes to decompose the objective function into an expectation term and an additional regularization term. As far as I know, this perspective of distributional RL is quite novel.\n- This paper is easy to understand for most parts of the paper. \n- The proposed method enjoys one additional flexibility (compared to the standard C51) – the weight of value distribution decomposition for controlling the risk sensitivity. This feature could be useful in practice.\n\n**Weaknesses**\n- There are some concerns regarding the decomposition assumption of $F^{s, a}$ in Eq. (4): Given an arbitrary CDF $F^{s,a}$, it is not always true that the resulting $F_{\\mu}^{s,a}$ is also a valid CDF. This is a critical issue since in the proof of Proposition 3, the cross entropy term in the 4th line of Eq. (19) would not always be well-defined. In other words, to establish Proposition 3, one would need to first show that $\\mu$ is indeed a valid PDF. More justification about this assumption is needed.\n- The novelty of the proposed algorithm: The proposed DERAC method can be viewed as a variant of C51 (for discrete actions) and DSAC (for continuous actions) with a slightly different regularization term. It would be good to clarify and highlight the differences between DSAC and C51/DERAC.\n- Lack of discussion on the experiment results: Several experimental results do not appear very conclusive. \nFor example, regarding the result in Figure 4, can the authors explain why the curve of AC+RE+VE performs well in the environments of humanoid and walker2d (despite the conjecture mentioned by the authors)? \nMoreover, in Section 4.2, it is mentioned that “Our empirical result in Figure 3 has provided strong evidence to verify our theoretical results.” However, this is not completely clear to me since (i) Theorem 1 suggests “convergence to a global optimum” while Figure 3 can only show convergence to some stationary point, and (ii) the empirical result of DERAC cannot be used to corroborate the theoretical results of DERPI given that DERAC is a learning algorithm while DERPI in Theorem 1 is essentially a planning algorithm.\nSince DERAC is a reinterpretation of C51 (which is a distributional method that also uses KL divergence), it is a bit surprising that the performance of DERAC and C51 are rather different in Figure 2.\n\nAnother thing to mention is about the generality of the analysis: It appears that the analysis (cf. Propositions 1-3) requires specifically choosing the KL-divergence in the objective function. It it not immediately clear whether the analysis indeed capture the essence of distributional RL and can address other popular distributional distance metrics (e.g., Wasserstein distance). While this might not be a true weakness, it would be helpful to have a discussion on this somewhere in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i6xYlLGci1", "reviewer": "Reviewer_UayF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper performs an analysis of distributional reinforcement learning using a regularization perspective. Specifically, under the assumption of a particular decomposition of the value distribution, they argue that distributional RL provides a risk-sensitive entropy regularization when approximated via the neural fitted Z-iteration framework (something also introduced in this paper).", "summary_of_the_review": "I think this is a nice paper that can be an important contribution to further understanding distributional reinforcement learning and can help advance research in this area. However, there is a fair bit of work to do on the clarity front to make this paper useful for others wishing to build on it.\n\nAs it is, I do not believe this paper is ready for publication yet. However, I do believe the authors have an important contribution to provide, so I would encourage them to work on the clarity of the writing and exposition.", "strength_and_weaknesses": "# Strengths\nThis paper presents an interesting analysis of distributional RL that can be useful for better understanding its differences with traditional (expectational) RL, and likely lead to better algorithms.\n\n# Weaknesses\n## Clarity\nThe main weakness of this paper is its clarity. There is a lot going on and lots of notation to keep track of (sometimes not presented consistently) which makes it hard to follow. Some points related to clarity below:\n1. What is $\\mu$ in equation (4)? It has not been introduced yet.\n1. In the discussion after equation (4) up to the end of page 3 it's not very clear what $F^{s,a}$ and $F^{s,a}_{\\mu}$ are, how they differ, and what the role of each is. A more elaborate discussion may help address point 1 above.\n1. At the bottom of page 3 it says \"Next, we use $p^{s,a}(x)$ to express the true target probability...\", but then it doesn't seem like $p^{s,a}$ is used again. Should this be $\\mu^{s,a}$?\n1. In the discussion below equation (6) it's interesting and somewhat counterintuitive that estimates at the current $s,a$ are evaluated against the next state-action $s',a'$. It would be nice to have some more discussion, and perhaps even a diagram/figure for better exposing this.\n1. In equation (7), where did the horizon $T$ come from? This is the first time it's been introduced and seems to be in conflict with the discounted and infinite-horizon problem setup that has been introduced thus far.\n1. In equation (8) should the expectation be sampling from $P(\\cdot | s_t, a_t)$ instead of $\\rho^{\\pi}$?\n1. Equation (9) seems to me to be somewhat in conflict with what was presented in equation (6), where estimates at the current $s,a$ are evaluated against the next state-action $s',a'$. The rest of the discussion seems to follow equation (9), but then I'm confused as to how equation (6) (and the discussion following it) fits in.\n1. In equation (9) (and sentence following it) it seems we're in a tabular setting, yet you have $q_{\\theta}$. What is the role of $q_{\\theta}$ in a tabular scenario? What is it approximating and how?\n1. In the paragraph following equation (9) it says \"can normally be obtained via bootstrap estimate $\\mu^{s_{t+1},\\pi_Z(s_{t+1})}$ similar in Eq. 6.\" Can you elaborate on this? It's not clear what is meant by this statement.\n1. In Lemma 1 and Theorem 1 there is an assumption of an upper-bound $M$ on the entropy term. Can this be any constant? From the proof it seems all that is necessary is boundedness, so it would be good to clarify the role of $M$ here. In particular, for finite states and actions the entropy term will always be bounded, so the assumption seems to be only necessary when one or the other is not finite.\n1. More importantly, shouldn't the upper-boundedness assumption be on $f(\\mathcal{H}(\\ldots))$? Otherwise, it seems one can pick $f$ adversarially to make the theorem false.\n1. In Lemma 1, what is the $sd$ subscript in one of the $\\mathcal{T}$s? Should it just be $\\mathcal{T}_d$?\n1. Why are you using a finite horizon $T$ in the statement of Lemma 1? It does not appear to be necessary and is related to the point made above.\n1. Figure 1 is nice, but it's not totally clear what the authors are trying to show there. Providing more details in the caption (or in the main text) would help illustrate how it connects to the paper's analyses.\n1. In the first line of page 7 is the $\\epsilon$ with $\\mathbb{E}[\\epsilon]=0$ the same $\\epsilon$ from equation (4)?\n1. In the line after equation (12) it says \"we consider a particular increasing function $f(\\mathcal{H})$, but there is no $f$ in equation (12).\n1. In the first paragraph of section 4 it says QR-DQN is also evaluated, but then later it seems the authors use IQN, not QR-DQN.\n1. In section 4.1 there is a sentence \"Firstly, it is a fact that the value distribution decomposition... owing to the leverage of target networks.\" It is not clear what is meant by this sentence.\n1. In section 4.1 it says \"demonstrate that C51 algorithm can still achieve similar results _under the cross entropy loss_\", which is a little confusing, since C51 was introduced with a cross entropy loss.\n1. In the first paragraph of section 4.1 there is some discussion using $\\varepsilon$ which is rather confusing:\n    1. Is it same as the $\\epsilon$ used earlier in the paragraph (for the continuous decomposition) or different?\n    1. The same $\\varepsilon$ is used to refer to \"the proportion probability\" _and_ \"the true $\\varepsilon$\", which makes things confusing.\n    1. It appears $\\varepsilon$ is referring to the same $\\epsilon$ object from equation (4), but they are different symbols, which makes things confusing.\n1. In general, I found the first paragraph of section 4.1 really hard to follow. Since it is quite important to the paper's contributions, it needs to be clarified.\n\n## Assumptions\nIn section 3.2 the authors assume that $F^{s,a}$ satisfies ... eqn (4). The rest of the paper's analysis seems to hinge on this assumption, but it is not clear to me how reasonable an assumption it is to make.\n\n## Comparison to related work\nAnother weakness is that it is missing discussion with a very related work:\n* [Clare Lyle, Pablo Samuel Castro, Marc G. Bellemare. A Comparative Analysis of Expected and Distributional Reinforcement Learning. AAAI 2019](https://arxiv.org/abs/1901.11084)\n\nLyle et al. prove that for tabular and linear function approximation, distributional and expectational RL are equivalent; with non-linear approximators distributional and expectational are _different_, but sometimes distributional can hurt performance. This seems to be in contrast with some of the wording in the paper under review, where the authors make statements like (emphasis is mine): \"we illuminate the **superiority** of distributional RL over expectation-based RL.\".\n\n# Minor suggestions\n* Second line of page 2 should read: \"result is based on two analy**tical** components, ... by leverag**ing a** variant of gross...\"\n* Top line of page 5 should say \"risk-neu**t**ral\".\n* In section 3.3 when you introduce _Distribution-Entropy-Regularized Policy Iteration_ you may as well introduce the acronym DERPI (instead of waiting until the next page).\n* Above equation (12) it should say \"**objective** function\", not \"objection function\".\n* Below equation (12) it says \"we use the target value distribution neural network $q_{\\theta^*}$\". Then why not just use $q_{\\theta^*}$ directly in equation (12) to make it clear?\n* In **Environments** in section 4 it says three Atari games were used: Breakout, Seaquest, and Asterix. But it seems Hero is being used instead of Asterix.\n* Please specify what the shaded areas represent in your figures.\n* Please restate lemmas/theorems in the appendix when you present the proofs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g8DIDgI0Ezk", "reviewer": "Reviewer_8Ekx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates an important problem of interpreting why distributional RL outperforms conventional RL. Specifically, the author separates the action value function into the expectation part and regularization part, and attributes the superiority to the regularization part. In addition, the author proposes a new algorithm called DERPI for both tabular and function approximation settings. The experimental results manifest the effectiveness. ", "summary_of_the_review": "Generally speaking, this paper aims to address an important problem. However, the paper does not solve the problem very well, since the result can only interpret KL-divengence based distributional RL. In addition, there are also not enough experimental scenarios to support the theoretical results.", "strength_and_weaknesses": "Pros\nThe paper investigates an important problem.\nThe paper provides solid theoretical analysis.\n\nCons\n1. Many assumptions are made to gain the theoretical results, but the author does not explain them clearly. e.g., (1)In Section 3.2, the author assumes the action value function satisfies the expectation decomposition of Eq.(4). However, the author needs to convince us why it is true. It seems proposition 1 tries to do that, but it is still not clear why action value function satisfies the decomposition.\n\n2. The paper is hard to follow. The theoretical derivation is not explained clearly. For example, in proposition 1, why the inf is taken over F_\\mu^{s,a}, also the corresponding prof in Appendix A, what does the “||” mean in the first inequality of Eq.(15)?\n\n3. As the title of the article shows, the main idea of this paper is to explain the superiority of distributional RL. But actually, the author only explains distributional RL that uses KL divergence, and the experiments are also only performed with the C51 that is based on KL divergence. So I think the topic of the paper should be further narrowed down , since the results can not represent all the distributional RL algorithms.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "58JUXEBOHA", "reviewer": "Reviewer_RieR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "his paper addresses the lack of theoretical analysis on why distributional RL works so well and approaches it from the framework of risk-sensitive entropy regularization. It introduces an entropy term in the distributional bellman update which is different from that of conventional MaxEnt frameworks, as it is an entropy of the value-distribution and is state-action-wise rather than just state-wise. \n", "summary_of_the_review": "I think this paper requires additional polish and refinement, in particular some empirical demonstrations of the usefulness of the theoretical analyses made. I would lean towards rejecting.", "strength_and_weaknesses": "Strengths:\n- This paper studies an important topic in distributional RL, by looking at where the superior performance of distributional RL algorithms comes from. It further presents a new framework by incorporating an entropy term of the value-distribution itself (as opposed to the policy entropy) into the Q-value estimation. Furthermore, it retrofits the analysis into the policy iteration and policy evaluation frameworks, which is a familiar theoretical framework. \n- The proposed entropy regularization term is compatible with existing distributional RL algorithms, in particular, the quantile based ones and for both discrete/continous control. \n\nWeaknesses:\n- The theoretical contributions, while interesting are performed on a KL divergence as a distance measure between distributional value-estimates. While the authors acknowledge this is done because Wasserstein distance is less manageable in this theoretical framework, the KL divergence is non-expanding while most state-of-the-art distributional RL papers have moved to contractive distance measures such as Wasserstein or MMD. This discrepancy would seem to lessen the contribution of this paper. \n- The experimental results are not convincing beyond to study the effects of regularizing the update using the proposed DERAC algorithm. The DSAC and C51 comparisons use \"vanilla entropy regularization\" and no regularization and outperfoms DERAC on a majority of tasks. I understand the authors state the empircal results are to corroborate the theoretical results and not necessarily to perform well against existing algorithms, but then it seems there should be additional/different experiments analyzing how exactly the it converges to the optimal risk-sensitive policy. \n- The experiments given in Figure 4. are confusing in terms of drawing conclusions, since they combine two different frameworks for RL. Notably, it is unclear how VE and RE tradeoff against one another.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject