A Distributional Perspective on Actor-Critic Framework
Authors: Daniel Wontae Nam
Younghoon Kim
Chan Youn Park
Venue:  (2021)
Keywords: Value distribution learning
reinforcement learning
deep learning
distributional reinforcement learning
distributional actor-critic

Abstract: Recent distributional reinforcement learning methods, despite their successes, still contain fundamental problems that can lead to inaccurate representations of value distributions, such as distributional instability, action type restriction, and conflation between samples and statistics. In this paper, we present a novel distributional actor-critic framework, GMAC, to address such problems. Adopting a stochastic policy removes the first two problems, and the conflation in the approximation is alleviated by minimizing the Crame ́r distance between the value distribution and its Bellman target distribution. In addition, GMAC improves data efficiency by generating the Bellman target distribution through the Sample-Replacement algorithm, denoted by SR(λ), which provides a distributional generalization of multi-step policy evaluation algorithms. We empirically show that our method captures the multimodality of value distributions and improves the performance of a conventional actor-critic method with low computational cost in both discrete and continuous action spaces, using Arcade Learning Environment (ALE) and PyBullet environment.

Under review as a conference paper at ICLR 2021
A DISTRIBUTIONAL PERSPECTIVE ON ACTOR-CRITIC FRAMEWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Recent distributional reinforcement learning methods, despite their successes, still contain fundamental problems that can lead to inaccurate representations of value distributions, such as distributional instability, action type restriction, and confla- tion between samples and statistics. In this paper, we present novel distributional actor-critic framework, GMAC, to address such problems. Adopting stochastic policy removes the first two problems, and the conflation in the approximation is alleviated by minimizing the Cramér distance hetween the value distribution and its Bellman target distribution. In addition, GMAC improves data efficiency by generating the Bellman target distribution through the Cample-Replacement algorithm, denoted by SR(A), which provides distributional generalization of multi-step policy evaluation algorithms. We empirically show that our method captures the multimodality of value distributions and improves the performance of a conventional actor-critic method with low computational cost in both discrete and continuous action spaces, using Arcade Learning Environment (ALE) and PyBullet environment.
INTRODUCTION
The ability learn complex representations via neural networks has enjoyed success in various ap-
plications of reinforcement learning (RL), such as pixel-based video gameplays (Mnih et al.) (0115), the game of Go (Silver et al.) (2166). robotics (Levine al., (016)). and high dimensional controls like humanoid robots (Lillicrap al.] 2016 [Schulman et al. 2015). Starting from the seminal work of Deep Q-Network (DQN) (Mnih al. (015), the advance value prediction network, in partieular, has been one of the main driving forces for the breakthrough.
Among the milestones of the advances in value function approximation, distributional reinforce- ment learning (DRL) further develops the scalar value function to distributional representation. The distributional perspective offers various benefits by providing more information on the charac- teristies and the behavior of the value. One such benefit is the preservation of multimodality in value distributions, which leads to more stable learning of the value function (Bellemare et al. 2017a).
Despite the development, several issues remain, hindering DRL from becoming robust framework. First, a theoretical instability exists in the control setting of value-based DRL, methods (Bellemare) al. 2017a). Second, previous DRL algorithms discrete (Bellemare et al. 2017aa] Dabney et al.) 2018b a) or continuous (Barth-Maron et al.) 2018 Singh et 2020). Third, a common choice of loss function is Huber quantile regression loss which is vulnerable to conflation between samples and statistics without an imputation strategy (Rowland al. 2019).
limited
single type of action space, either
The instability issuc is not present if trainable policy is introduced, i.c., the evaluation setting of the Bellman operator is used, as shown by the convergence of distributional Bellman operator (Bellemare et al.] 2017a). In addition, the general form of the stochastic policy gradient method does not assume a specific type of action space, e.g. discrete or continuous (Williams) 1988 1992 Sutton et al.) 1999). Because Wasserstein distance has biased sample gradients (Bellemare et al. 2017b), in practice, directly minimizing the Wasserstein distance is often not preferred as loss function of a neural network and thus some of the exemplary works (Dabney et al. 2018b a) of deep DRL minimizes the Huber quantile regression loss (Huber 1964) instead. To this end, we treat the methods which minimize the Huber quantile loss our baseline. However, proven in Rowland

---

Under review as a conference paper at ICLR 2021
ION Wasserstein
GMM Cramer
0.0
Polccy Improsvement
0:49
(a) State of interest
(b) The evaluated value distributions
Figure 1: Modality of value distribution during the learning process of Breakout-v4. (a) An arrow is added in the inset to indicate the ball's direction of travel. The episode reaches a terminal state if the paddle misses the ball. (b) Probability density functions of the value distributions learned by each actor-critic when {0.2. 0.4, 1.2)M frames are collected. As the policy improves, the probability of losing a turn (V 0) decreases, and the probability of earning scores (V >> increases. Note that the modality transition from V = O is clearly captured by the GMM + Cramér method.
et al. 2019), representing a distribution using the Huber quantiles instead of samples can lead to conflation between statistics and samples, and thus imputation strategy is required to learn more accurate representation of distribution. Despite theoretical soundness, the imputation strategy introduce computational overheads depending the statistics (e.g. quantiles, expectiles). To avoid this issue, we formulate the DRL problem through samples and parameters instead of statistics, and minimize the Cramér distance between the value distributions. Combining these solutions, arrive at distributional actor-critic with Cramér distance as the value distribution loss.
On the other hand, many actor-critic methods suffer from data inefficiency and often include multi- step algorithms like the A-return algorithm (Watkins 1989) and off-policy updates, e.g. importance sampling. Here address this problem by adapting multi-step off-policy updates to the distri- butional perspective, by defining generalized form of the multi-step distributional Bellman tar- gets. Furthermore, we introduce novel value-distribution learning method which call Sample- Replacement, denoted by SR(A). We show that the expectation of the target distribution from SR(A) equivalent to the scalar A-return. Additionally, we propose to parameterize the value distribution as a Gaussian mixture model (GMM). When combining GMM with the Cramér distance, we can derive an analytic solution and obtain unbiased sample gradients at much lower computational cost compared to the method using Huber quantile loss. Altogether, we call our framework GMAC (Gaussian mixture actor-critic).
We present experimental results to demonstrate that this framework successfully outperforms the bascline algorithm with scalar value function in discrete action space, and can be expanded to con- tinuous action spaces without any architectural algorithmic modification. Furthermore, we show that more accurate representation value distributions is learned, with a less computational cost.
2 RELATED WORKS
Bellemare et al. (217aa has shown that the distributional Bellman operator derived from the dis- tributional Bellman equation contractor in maximal form of the Wasserstein distance. Based on this point, Bellemare et al. (2017a) proposed a categorical distributional model, C51, which is later discussed to be minimizing the Cramér distance the projected distributional space (Row- land et al. 2018 Bellemare et al. 2019). Dabney et al. (2018b) proposed quantile regression-basse models, QR-DQN, which parameterizes the distribution with a uniform mixture of Diraes and uses sample-based Huber quantile loss (Huber) 1964). Dabney et al. (201 Sa) later expand it further so that a full continuous quantile function can be learned through the implicit quantile network (IQN). Yang et al. (2019) then further improved the approximation of the distribution by adjusting the set of quantiles. Rowland et al. (2019) proposed expectile regression in place of quantile regression for learning categorical distribution to address the error in the Bellman target approximation. Choi et al. (2019) suggested parameterizing the value distribution using Gaussian mixture and minimizing the Tsallis-Jenson divergence as the loss function on value-hased method. Outside of RI., Bellemare
2

---

Under review as a conference paper at ICLR 2021
et al. (017b) proposed to Cramér distance in place of Wasserstein distance used in WGAN due to its unbiasedness in sample gradients (Arjovsky et al. 2017).
There have been many applications of the distributional perspective, which exploit the additional information from value distribution. Dearden et al.] (1998) modeled parametric uncertainty and Morimura et al. (2010a b) designed risk-sensitive algorithm using distributional perspective, which can be seen as the earliest concept of distributional RL. Mavrin et al. (2019) utilized the idea of the uncertainty captured from the variance of value distribution. Nikolov et al. (2019) has also utilized the distributional representation of the value function by using information-direcn
sampling for better exploration of the value-based method. While multi-step Bellman target was considered (Hessel et al.] 2018), the samplc-efficiency was directly addressed by combining multi- step off- off-policy algorithms like Retrace(A) (Gruslys et al. 2017).
Just as C51 has been expanded deep RL to disitributional perspective, Barth-Maron et al. (2018 (018) studied distributional perspective on DDPG (Lillicrap et al. 2016). an actor-critic method, by pa- rameterizing distributional critic categorical distribution and Gaussian mixture model. [Singh]
eet al. (020)) has further expanded the work by using an implicit quantile network for the critic.
Several works (Duan et al. 2020, Kuznetsov et al. 2020: Ma et al., 2020) have proposed a distri- butional version of the cctt-accor-critic (SAC) framework to address the error from estimaattng the value. However, previous works concentrated on extending specific actor-critic framework to the distributional setting. Therefore, we aim to suggest methods which may be easily adopted in the process of expanding& scalar value methods to a distributional perspective, along with attempt to address the previously mentioned issues present in the value-based ditributional algorithms.
3 DISTRIBUTIONAL REINFORCEMENT LEARNING
We consider a conventional RL setting, where an agent's 's interaction with its environment is de- scribed by Markov Decision Process (MDP) (X', A, R, 7), where X' and A state and action spaces, R(x, a) is the stochastic reward function for a pair of state x and action a, P(x [tx a) is the transition probability of observing x' given the pair (x. a), and y E (0.1) is time discount factor. A policy (-11r) maps state E X' to a probability distribution over actions a E A.
The objective of RL is to maximize the expected return, B[G,] where G, Ei 7 R(,,,, at) the sum of discounted rewards from state 2, given a policy π at time t. Then for any state 2t, the value V and acteon value Q under the given policy π can be defined as
V(z,) E[G, X zz),
Qx),, = E[G, X 21, A a.)
(1)
recursive relationship in the value in terms of the reward and the random transition is described by the Bellman equation (Bellman 1957) given by
Qxx, E[ a)] + 7 [Q(a', (()=',a))=
(2)
where the first expectation is calculated over a given state-action pair (x, a) and the second expecta- tion is taken over the next possible states x' P(- a) and actions a' r--12:).
DRL extends the Bellman equation to an analogous recursive equation, termed the distributional
Bellman equation (Morimura et al.] 2010a by Bellemare et al. 2017a), using distribution of the
possible sum of discounted rewards Z(r, a):
Z(x, 2(zz,)) 7
(3)
where denotes having equal distributions and Q(x,a) = (1,(x,a)), Then Z is learned through distributional Bellman operator T defined as
T" a) R(z,a) + +
(4)
where pr : Z + Z is a state transition operator under policy т, p Z(z, a) D gously, the distributional Bellman optimality operator T can be defined
Z Z(x', a²). Analo-
(Z( _ R(z, a) 77(,', arg max Egwr [7 d')))-
(5)
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1: SR(A)
Input: Trajectory of states and values ((x1, 21).... (IT, Zr)} for given length T, discount factor y, weight parameter À
Output: Set of X-returns (Z Z
X Collect m samples {X1,...; ) from ZT
for t T - 1 to 1 do
X - Ti + x // Bellman operator
b[Ai t - òx, // empirical distribution using m Diracs
X' + Collect samples (Xi..... ) from 2,AA
for i to m do
X X with probability 1 1-A
end
end
The distributional Bellman operator has been proven to be y-contraction in a maximal form of Wasserstein distance (Bellemare et al. 2017a), which has practical definition given by
1/P
do(u, () (% FF (w) (F (W)
(6)
where U, V are random variables and FU, Ev are their cumulative distribution functions (cdf).
However, unlike the distributional Bellman operator, the distributional Bellman optimality operator is not contractor in any metric (Bellemare et al. 2017a). Thus, the distance d,(T2, Z1, T Z2) between some random variables Z1. Zg may not converge to unique solution. This issue has been discussed in Bellemare al. 2017a). with an example of oscillating value distribution caused by specific bie-breaker design of the argmax operator. One way to remove this issue from consideration is by learning the value distributions via expected Bellman operator with trainable stochastic policy and finding an optimal policy under principles of conservative policy iteration by Kakade & Langford 2002). See Apppendix A for more discussion.
AL.GORITHM
In this section, we incrementally develop the building blocks of our proposed method. First, present an efficient distributional version X-retum algorithm called Cemmlle-Replacement denoted by SR(A). Then, we show that minimizing the energy distance, which is equivalent to specific form of the Cramér distance, between Gaussian mixtures can be better solution than using quantile regressions when working with distributional actor-critic and SR(A).
4.1 SR(A): SAMPLE- SAMPLE-REPLAACEENN FOR X-RETURN DISTRIBUTION
The actor-critic method is temporal-differenn (TD) learning method in which the value function, the critic, is Jearned through the TD error defined by the difference between the TD target given by n-step return, ((()) = >i 1d-4 +I*V(r and the current value estimate V(x;). A special case of TD method, called TD(A) (Sutton 1988). generates weighted average of n-step returns for the TD target, also known as the X-retur,
1
G. =1 Σ x- G( X [0,
n=1
to mitigate the variance and bias trade-off between Monte Carlo and the TD(0) return enhance data efficiency. From an alternative perspective, equation can be thought of as finding TD target via taking the expectation of a random variable G whose sample space is the set of all n-step returns, (G ,..., ((()) J. Then the probability distribution of & is given by
Pr(m G.")) (1 --
(8)
4

---

Under review as a conference paper at ICLR 2021
Similar to G we define n-step approximation of the value distribution
(n)
R(z14i Gess) + +y" Br' ( a
(9)
A=0
where ][Z = G(a)) Then, a distributional analogy of equation 8 can written as
Pr[Z = - A)A-11
(10)
where Z is random variable whose sample space is of all n-step approximations, (2,0) . 2, 3. Unlike G, we cannot directly calculate an expectation over a set of random variables, Z. Instead, we redefine equation 10 in terms of cdfs:
Pr(β- |22=0]
A)A=-1
(11)
F z denotes the cdf of the n-step return z and F {F 2..0......, able over the set of 2.0.. Then, for any 2, we can successfully rewrite equation 11 as a linear
F2, } is random vari-
combination of F m
BIA (1 A) Σ A" 2
(12)
Let us define random variable 2,-) that has E[F] its edf. Then the expectation of Z7A) and the expectation of Z((*) have an analogous relationship equation 7 (see Appendix B), meaning that its behavior in expectation is equal to that of the A-return. Therefore, we treat the resulting random variable (7(²) distributional analogy of the A-return. We note that, in practice, collecting infinite
horizon trajectory is infeasible and thus we use a truncated sum (Cichosz 1995 van Seijen et al.,
2011:
F 22(20) -1
A)
F
((1) + A (M
(13)
#=1
Given a trajectory of length N, naively speaking, finding 2,²2 for each time step requires finding N different -!!), As result, need to find total of O(N2) different distributions find (x) for all states in the given trajectory. The number of distributions to find can be reduced to O(N), in
practice, by approximating the distribution of z!") with mixture of diracs, as described in Dabney et al. (2018b), because the Bellman operations can be applied to the same set of samples to find Z,
for different t's. Then, the approximation takes the form of
2 Zo(
80.(
(14)
m7
where 0 X + 1 is some parametric model. To obtain the target samples for Ze(x;), we start with m samples the last value distribution in the sampled trajectory. Traversing through the sampled trajectory in a reversed order, replace each sample with state with probability of A. Then the obtained set X,, is equivalent to a set of samples from the approximated distribution of the X-returns, ((A)) A more detailed description of the algorithm can be found in Algorithm
sample from the next
For sasple-based methods like the implicit quantile network, one can directly use this set X.. as the target samples. However, Rowland et al. (2019) has shown that the quantiles predicted from the Huber quantile loss cannot be interpreted as samples, thus an imputation strategy is required to generate a distribution from the statistics. We propose to minimize the Cramér distance instead in which the predicted parameters be samples or the parameters of the distribution itself.
5

---

Under review as a conference paper at ICLR 2021
4.2 CRAMÉR DISTANCE
Let P and Q be probability distributions over IK. If we define the cdf of P. as Fp, PQ respectively, the Lp family of divergence between P and Q
(P, ( [FP(x) Fo(x)) (o( 1/9
(15)
When = 2, it is termed the Cramér distance. The distributional Bellman operator in the evaluation setting is a /|l//-ccotraattion mapping in the Cramér metric space (Rowland et al. 2019). worked out proof can be found in Appendix
whose
notable characteristic of the Cramér distance is the unbiasedness of the sample gradient,
V Vol[(Pm. Qa) = Vol((P, (P, Qa) X~Q
E
(16)
where X.. :- (X1, ..., Xm3 are samples drawn from P. Pm. 1 >i= Óx; is the empirical distribution. and Qo is a parametric upproximation of a distribution. The unbiased sample gradient makes it suitable to use Cramér distance with stochastic gradient descent method and empirical distributions for updating the value distribution.
Szzkelly (2002) showed that, in the univariate case, the squared Cramér distance is equivalent to one half of energy distance (13( (P,PPOO Q) = (P, Q)) defined as
E(P, Q) = {U, V) = 2E
2 |IU - Vil2
IU U'l12 E E V'12,
(17)
where U, U' and V, V are random variables that follow P. Q, respectively.
4.3 ENERGY DISTANCE BETWEEN GAUSSIAN MIXTURE MODELS
So far, we have described the components required to formulate general setting of the suggested distributional extension to actor-critic methods. Here, we take one step further enhance the ap- proximation accuracy and computational efficiency by considering the parameterized model of the value distribution as Gaussian mixture model (Choi et al. 2019: Sarth-Maron et al., 2018). Fol- lowing the same assumption used for equation the approximation using Gaussian mixture is given using parametric models μl, a, w X R
Zo(xe) = Σ wi
where Σ
uy (xy(Fe) = 1.
(18)
If random variables X, Y follow the distributions P. parameterized as GMM, the energy distance has the following closed-form
E(X,Y) Y) -10(5,() - 5(X,X) 8(Y,Y',
where VV, = Σ WanWiry E. [[( Moj: ou + o
(19)
Here, Mai refers to the 7th component for random variable and same applies for a and w. With Gaussian mixtures, the closed-form solution of the energy distance defined in equation 19 has computational advantage over sample-based approximations like the Huber quantile loss. When using GMM, the analytic approximation of equation 14|can be derived
K
Σ (1 1) Wikk ((;; Prnks o
T Σ N(2; Pakk (h))
1
(20)
where Mon refers to th component of (zy()) for simplicity of notation, is sampled from Geo A) and the index & {1,.... K) is sampled proportional to the mixture weights {wi...., w. This is equivalent to having a mixture of m Gaussians, thus we can simply perform sample replacement on the parameters (μ, o²), instead of realizations of the random variables as in equation 14 Then, the loss function described in equation 19 can easily be applied.
When bringing all the components together, have distributional actor-critic framework with SR(A) that minimizes the Cramér distance between Gaussian mixture value distributions. We call this method GMAC. A brief sketch of the algorithm is shown in Appendix E

---

Under review as a conference paper at ICLR 2021
~ ((0.0.13)
PPO
240
To 14(-111)
2001
GMAC- Cramer
S1
101
2000 4000
E000
8000
10300
05
1.5
26
(a)
(b)
(c)
Figure 2: (a) An environment with two states and stochastic rewards with expected value of zero. (b) The probability density function (above) and the Cramér distance between the ground truth and the estimated distributions for IQ. IQE, and GM (below). (c) Learning curve of Montezuma's Revenge using modality information as intrinsic reward.
EXPERIMENTS
In this section, we present experimental results for three different distributional versions of Proximal Policy Optimization (PPO) with SR(A): IQAC (IQN + Huber quantile), IQAC-E (IQN + energy distance), and GMAC (GMM + energy distance), in the order of the progression of our suggested approach. The performance of the scalar version of PPO with value clipping (Schulman et al. 2016) is used as the baseline for comparison. Details about the loss function each method can found in [pppendixx D For fair comparison, we keep all common hyperparameters consistent across the algorithms except for the value heads and their respective hyperparameters (see Appendipii F).
The results demonstrate three contributions of our proposed DRL framework: 1) the ability to cor- rectly capture the multimodality of value distributions, 2) generalization to both discrete and contin- uous action spaces, and 3) significantly reduced computational cost.
Representing Multimodality As discussed throughout Seccioon 4 we expect minimizing the Cramér distance to produce more appropriate depiction of distribution compared minimiz- ing the Huber quantile loss. First, we demonstrate this with a simple value regression problem for an MDP of five sequential states, as shown in Figure 2] (a). The reward function r% of last two state S; is stochastic, with ra from a uniform discrete distribution and rs. from normal distribution. Then the value distribution of S should be bimodal with expectation of zero (Figure (b)). this example, minimizing the Huber quantile loss (labeled IQ-naive) of implicit quantile network underestimates the variance of S due to conflation and does not capture the locations the modes. Applying an imputation strategy suggested in Rowland et al. (2019), improvement the under- estimation of variance can be seen. On the other hand, minimizing the Cramér distance converges to correct mode locations using both implicit quantile network and Gaussian mixture model, labeled as IQE and GM respectively in the figure. More details about the experimental setup and further results can be found in Appendix G The comparison can be extended more complex tasks such as the Atari games, of which sample result is shown in Figure and additional visualizations of the value distribution during the learning process from different games can be found in Appendix G
So what can be achieved with correct modality? By capturing the correct modes of the distribution, an additional degree of freedom on top of the expected value can be accurately obtained, from which richer information can be derived to distinguish states by their value distributions. In particular, the extra information may be utilized as an intrinsic motivation in sparse-reward exploration tasks. To demonstrate, we compare using Cramér distance between value distributions as intrinsic reward to using TD error between scalar value estimates in sparse reward environment of Montezuma's Revenge in Figure 2 (c), which shows clear improvement in the performance.
Discrete and Continuous Action Spaces Experimental results from the ALE (Bellemare et al. 2013) and the PyBullet environments (Coumans & Bai 2016-2020) show that our algorithm can be
7

---

Under review as a conference paper at ICLR 2021
Nomallzed
alized
040
GAC
GAC
300
69776
423
0.25
3.75
1.00
352
0.25
0.5
8.79
3.00 Pommes
525
350
1.7
2.00
Rhrmes
ALE mean humaa--ormliize
(b) ALE median human-normlized score
Jesbul
Ik
3000
P10
2500
2500
124c
1866-T
2000
2500
1900
FPO
1000
KACC
500
ln7
167
(c) Ant
(d) HalfCheetah
(e) Humanood
Figure 3: Learning curves for Atari games from ALE and continuous control tasks from PyBullet.
applied to both discrete and continuous action spaces. While GMAC shows overall improvement in 61 Atari games (Appendix G), we present the full learning curves of 3 Atari games in Figure 3) in which previous distributional methods have shown improvements over the scalar version. One can observe that the results from GMAC show the most significant improvement from the baseline.
All experiments are run over random seeds with consistent hyperparameters. In Figure solid line represents an average of mean scores over 100 recent episodes and the shaded area represents the standard deviation over the seeds. For visual clarity, the plots are smoothed over 5M frames and 1.25M frames in Atari and PyBullet, respectively. Non-smoothed learning curves on more tasks and the final scores over the 61 ALE tasks can be found in Appendix
Computational Cost Table I shows the number parameters and the number floating-point operations (FLOPs) required for single inference and update step of each agent. We emphasize three points here. First, the implicit quantile network requires more parameters due to the intermedi-
ate embeddings of random quantiles. Second, the difference between the FLOPs for single update in IQAC and IQAC-E indicates that the proposed energy distance requires less computation than the Huber quantile regression. Last, the results for GMAC show that using Gaussian Mixtures greatly reduce the cost even to match the numbers of PPO while having improved performance.
Table FLOP measurement results for a single process in Breakout-v4
FLOPs (G)
Algorithm
Params (M)
Inference
Update
PPO
0.44
1.73
5.19
IQAC
0.52
2.98
12.98
IQAC-E
0.52
2.98
8.98
GMAC
0.44
1.73
5.27
8

---

Under review as a conference paper at ICLR 2021
6 CONCLUSION
In this paper, we have developed the distributional perspective of the actor-critic framework which integrates the SR(A) method, Cramér distance, and Gaussian mixture models for improved perfor- mance in both discrete and continuous action spaces at lower computational cost. Furthermore, show that our proposed method can capture the correct modality in the value distribution, while the extension of the conventional method with the stochastic policy fails to do so.
Capturing the correct modality of value distributions can improve the performance of various policy- based RL applications that exploit statistics from the value distribution. Such applications may include training risk-sensitive policies and learning control tasks with sparse rewards that require heavy exploration, where transient information from the value distribution can give benefit to the learning process, We leave further development of these ideas as future works.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Gabriel
Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In International Conference Learning Representations (ICLR), 2018.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017a.
Mare G. Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The cramer distance as solution to biased wasserstein gradi- ents, 2017b.
Marc G. Bellemare, Nicolas Le Roux, Pablo Samuel Castro, and Subhodeep Moitra. Distribu- tional reinforcement learning with linear function approximation. In Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 2203-2211, 2019. 2019.
Richard Bellman. Dynamic Programming. Dover Publications, 1957.
Dimitri P. Bertsekas and John N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1st edition, 1996.
Y. Choi, K. Lee, and S. Oh. Distributional deep reinforcement learning with mixture of gaussians. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9791-9797, 2019.
P. Cichosz. Truncating temporal differences: On the efficient implementation of TD(A) for rein- forcement learning. Journal on Artificial Intelligence, 2:287-318, 1995.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet.orrg 2016-2020.
Will Dabney, Georg Ostrovski. David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning (ICML), 2018a.
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In AAAL, pp. 2892-2901, 2018b.
Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian q-learning. Proceedings of the fifteenth national/tenth conference on Artificial inteligenceelnnovative applications of artificial intelligence, pp. 761 768, 1998.

---

Under review as a conference paper at ICLR 2021
Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, and Bo Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. arXiv preprint urXiv:2001.02811, 2020.
Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot, Mare Bellemare, and Remi Munos. The reactor: fast and sample-efficient actor-critic agent for reinforcement learn- ing. In International Conference on Learning Representation (ICLR), 2017.
Matteo Hessel, Joseph Modayil, H. V. Hasselt, Schaul, Georg Ostrovski, W. Dabney, Dan Horgan, B. Piot, Mohammad Gheshlaghi Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018
Peter J. Huber. Robust estimation of location parameter. The Annals of Mathematical Statistics, 35(1):73-101, 1964.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the 19th International Conference Machine Learning (ICML), 2002.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry P. Vetrov. Controlling overes- timation bias with truncated mixture of continuous distributional quantile crities. arXiv preprint arXiv:2005.0472 .0..2, 2020.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. Journal of Machine Learning Research, 17:39:1-39:40, 2016
Timothy Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proceedings the 33rd International Conference on Learning Representations (ICML), 2016.
Xiaoteng Ma. Qiyuan Zhang, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Distribu- tional soft actor critic for risk sensitive learning. arXiv preprint arXiv:2004. 14547, 2020.
Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional rein- forcement learning for efficient exploration. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):5295533 2015.
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Parametric return density estimation for reinforcement learning. In UAI, pp. 668-375. AUAT Press, 2010a.
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametric return distribution approximation for reinforcement learning. Proceedings of the 27th International Conference on Machine Learning (ICML), 2010b.
Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information- directed exploration for deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2019.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1st edition, 1994.
Mark Rowland, Mare G. Bellemare, Will Dabney, Rémi Munos, and Yee Whye Teh. An analysis of categorical distributional reinforcement learning. In Amos J. Storkey and Femnand Pérez-Crur (eds.), Artificial Intelligence and Statistics (AISTATS), volume 84 of Proceedings of Machine Learning Research, pp. 29-37. PMLR, 2018.
10

---

Under review as a conference paper at ICLR 2021
Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Mare G. Bellemare, and Will Dab- ney. Statistics and samples in distributional reinforcement learning. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning (ICML), 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In International Conference Learning Representations (ICLR), 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: NAKYv:1777.064444 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. Nature. 529(7587):484 4894 2016.
Rahul Singh, Keuntack Lee, and Yongxin Chen. Sample-based distributional policy gradient. arXiv preprint arXiv:2001.0260 2020.
Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3(1):9-44, aug 1988. ISSN 0885-6125. doi: (..1033/A:10223333111
Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Proceedings the 12th International Conference on Neural Information Processing Systems (NeurIPS), Cambridge, MA, USA, 1999.
Gabor Székely. E-statistics: The energy of statistical samples. 10 2002. doi: 10.13140/RG.2.1 5063.9761.
Harm van Seijen, Shimon Whiteson, Hado van Hasselt, and Marco Wiering. Exploiting best-match equations for efficient reinforcement learning. Journal of Machine Learning Research, 12:2045- 2094, 2011.
C. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Oxford, 1989.
R. J. Williams. Toward theory of reinforcementt connectionist systems. Technical Report NU-CCS-88-3, College of Comp. Sci., Northeastern University. Boston, MA, 1988.
R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229-2566 1992.
Derek Yang, Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Tie-Yan Liu. Fully parameterized quantile function for distributional reinforcement learning, In 33rd Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 6190-6199, 2019.

---

Under review as a conference paper at ICLR 2021
Appendices
A DISCUSSION ON THE CHOICE OF PROXIMAL POLICY OPTIMIZATION AS A BASELINE
A general learning process of RL be described using policy iteration, which consists of two iterative phases: policy evaluation and policy improvement (Sutton & Barto 1998). In policy it- eration, the value function assumed to be exact, meaning that given policy, the value function is learned until convergence for the entire state space, which results in a strong bound on the rate of convergence the optimal value and policy (Puterman 1994).
But the exact value method is often infeasible from resource limitation since it requires multiple sweeps over the entire state space. Therefore, in practice, the value function is approximated, i.c. it is not trained until convergence nor across the entire state space on each iteration. The approxi- mate version of the exact value function method, also known as asynchronous value iteration, still converges to the unique optimal solution of the Bellman optimality operator. However, the Bellman optimality only describes the limit convergence, and thus the best we can practically consider is to measure the improvement on each update step.
Bertsekas & Tsitsiklis (1996) have shown that, when we approximate the value function V2 of some policy т with V, the lower bound of a greedy policy n' is given by
27e
(x) 2 V=(2) 1-7
(21)
where = max [=() is the Low error of value approximation V. This means a greedy policy from an approximate value function guarantees that its exact value function will not degrade more than 2ne. However, there no guarantee on the improvement, i.e. Vx) > (x) Kakade
& Langford 2002).
As solution this issue, Kakade & Langlord (0022) have proposed policy updating scheme named conservative policy iteration,
Tnew (a|z) (1 a) Todd(aa)) + av (a(r),
(22)
which has an explicit lower bound on the improvement
2ey
(TTwww) Timasa (Tnew)
(1 7)2 a²
(23)
where max, [E,/ [A, a)ll. A= a) denotes the expected sum reward under the policy n,
Q(z,a) V Vzx) is the advantage function, n(TT)
Σ R ar)
(24)
and Lroia is the local approximation of with the state visitation frequency under the old policy.
From the definition distributional Bellman optimality operator in equation one can see that the lower bound in equation 23 also holds when n' is greedy with respect to the expectation of the value distribution, i.e., Exr= (a,, ; d')]. Thus the improvement of the distributional Bellman update is guaranteed in expectation under conservative policy iteration, and the value functions are guaranteed to converge in distribution to a fixed point by 7-contraction.
Schulman et (2015) takes this further, suggesting algorithm called trust region policy optimiza- tion (TRPO), which extends conservative policy iteration to general stochastic policy by replacing with Kullback-Leibles (KL) divergence between two policies,
D (xt, #) max DKL (m(-|2))|
(25)
Then, the newly formed objective is to maximize the following, which is form of constraint opti- mization with penalty:
n (a .xa) Å, (T(()e)) ((1o))) ((((()))) Thè [re(mm)) (aniiAA-BDDeeeeeee (3Dx)((((1))
E2,
(26)
((a xx)
12

---

Under review as a conference paper at ICLR 2021
where r(T) refers the ratio r(*))
However, in practice, choosing fixed penalty
a(a, [.)""
coefficient 3 difficult and thus Schulman et (2015) uses hard constraint instead of the penalty.
max E. [T((T))
(27)
s.t.
8
(28)
Schulman al. (2077) simplifies the loss function even further in proximal policy optimization (PPO) by replacing KL divergence with ratio clipping between the old and the new policy with the following:
=R, - min (ra(t)) ( +1))))))
(29)
Thus, by using PPO the baseline, we aim to optimize the value function via unique point con- vergence of distributional Bellman operator for a policy being approximately updated under the principle of conservative policy.
B EXPECTATION VALUE OF (A)
Continuing from equation
12 let us define a random variable that has a cumulative distribution
function of [[F22] as Z(A), Then, its cumulative distribution function is given by
F 222 = (1 - A)
F 2.
(30)
If we assume that the support of (A) is defined in the extended real line [-o0, 20],
dz 1- F
B|Z
(1
2(1))
Pzzrdzz
(31)
J
20
0
8
1
A) Σ x- F 2(())
de
(1- X)
Σ
1n
F
(32)
2
-00
= =1 (1- - A) Σ 1 1/,0 (1- 2(m)
0
dz
F
(33)
n=1
= (1 - A) Σ \" ((x))
=G(A)
(34)
Thus we can arrive at the desired expression of E 1 =
C DISTRIBUTIONAL BELLMAN OPERATOR AS CONTRACTION IN CRAMÉR METRIC SPACE
The Cramér distance possesses the following characteristies (detailed derivation of each can be found in (Bellemare et al. 2017b)):
X (((,)))
(35)
Using the above characteristics, the Bellman operator in I, divergence is
lp (T Z1, T Z2(x, a)) +((( R(I, 7 (2(
(36)
< 1/ sup (u) d'))
z,
13

---

Under review as a conference paper at ICLR 2021
Substituting the result into the definition of the maximal form of the Cramér distance yields
(T(T 21, T T2)
=sup (7 p
Z1 ((((1)))
(2(
3,0
$ $1> 1/p
sup (p (r', dd),
(37)
z'aa'
22)
Thus distributional Bellman operator is ht/////coo/// ccottraciin mapping the Cramér metric space, which was also proven in Rowland et al. 2019).
/
Similar characteristics as in equation 35 can derived for the energy distance
EAA+ E X Y) < EX,
E(cX,cY) = eX,
(38)
showing that the distributional Bellman operator a y-contractor in energy distance
E(T T 2 < 7E 72)
(39)
D LOss FUNCTIONS
As in other policy gradient methods, our value distribution approximator models the distribution of the value, V V(t)), not the state-action value (Iz, dz), and denote as Zo(x;) parametrized with 0, whose cumulative distribution function defined as
(z)(
Σ T(a, (t) acA
(40)
Below, we provide the complete loss function of value distribution approximation for each of the cases used in experiments (Section 5).
D.1 IMPLICIT QUANTILE + HUBER QUANTILE (IQAC)
For the value loss IQAC, we follow the general flow of Huber quantile loss described in Dabney ct al. (2018b). For two random samples T, T' U ([0, 1)).
z (xt, ac; 7') T)( 87
(41)
where A(Aa is generated via SR(A) and 7(x;7) (T) realization of Z(X) given æ and Then, the full loss function of value distribution is given by
N
N'
Lz,
6
(42)
N
where and N' are number of samples of T, 7', respectively, and p is the Huber quantile loss
(=((1)) = |- - -1 < <0}1 (drs))
with
(43)
if S otherwise.
(x(g)) (n(181g) 1k),
(44)
D.2 IMPLICIT QUANTILE ENERGY DISTANCE (IQAC-E)
Here, we replace the Huber quantile loss in equation 42 with sample-based approximation of energy distance defined in equation 19
N
L
2
ΣΣ
(45)
N
2
N72
14

---

Under review as a conference paper at ICLR 2021
D.3 GAUSSIAN MIXTURE ENERGY DISTANCE (GMAC)
Unlike the two previous losses, which use samples at generated by the implicit quantile network Zo (2417) here discuss case in which the distribution is k-component Gaussian mixture param- eterized with (142, oz wk))
Using the expectation of a folded normal distribution, we define å between two Gaussian distribu- tions as
2 Vo 03 + to
(- (mi - m2)² 2( + 07)
(Mz (4) 2Φ
v2
(46)
Let (z) and Z, (A) be Gaussian mixtures parameterized with (pos, og woi), (Maj: Bzg. way). re- spectively. Then, the loss function for the value head is given by
2
Lz
Σ
No 8 Maj. Maj. Rg)
NN
1
d o Hove oz20
(47)
N2
i=
P=1
N'
N'
1,, ozy))
2
j=1 =1
15

---

Under review as a conference paper at ICLR 2021
E PSEUDOCODE OF GMAC
Algorithm 2: GMAC
Input: Initial policy parameters 8g. initial value function parameters Φu. length of trajectory T,
number of environments E, clipping factor e, discount factor 7, weight parameter À
for k ===0,
for 1, ..... E do
Collect samples of discounted sum of rewards (Zi...., Zr} by running policy
Tk
T(0x) in the environment
Compute the parameters (/4: 0s w)) for each of the A-returns {2(A)
by
SR(A) Algorithm I
Compute advantage estimates Å, using GAE (Schulman et al. 2016), based on the
current value function VoL
end
Gather the data from environments
Update policy using the clipped surrogate loss:
0 arg max E min To((elbte) As, g(c, A.)
To2 (1)
0
via stochastic gradient ascent.
Update value function using the energy distance between Gaussian mixtures (Equation 19):
arg min & (Vo(s)), ]
Øk-
via stochastic gradient descent. end
The clipping function g(e, A) shown in the algorithm is defined follows:
c)
if
g(e, A) = (1
i
<
Note that expectation of each loss is taken over each the collection of trajectories and environments.
16

---

Under review as a conference paper at ICLR 2021
F IMPLEMENTATION DETAILS
For producing a categorical distribution, a softmax layer was added to the output of the network. For producing Gaussian mixture distribution, the mean of each Gaussian is simply the output of the network, the variance is kept positive by running the output through a softplus layer, and the weights of each Gaussian is produced through the softmax layer. Since our proposed method takes archi-
Table 2: Network architecture for GMAC on atari
Layer Type
Specifications
Filter size, stride
Input
84 84 x 4
Conv
x 20 x 32 9 9 64
8 8 32, 4 64, 3 3 x 32,
Conv2
Conv3
x 7 32
FC1
512
Heads
Policy
Value
(FC)
action dim
of modes (= 5)
tecture which only changes the value head of the original PPO network, we base our hyperparameter
settings from the original paper (Schulman al. 2017). We performed hyperparameter search on Pirimizers=[Adamyy RMSprop). learning r2te=- 1.0e-4). number of
subset of variables: epochs=(4, 10). batch size=/256, 512], and number of environments=(116 32, 64, 128) over
atari tasks of Breakout, Gravitar, and Seaguest, for which there of PPO.
degrade in the performance
Table 3: Parameter settings for training atari games
Parameter
PPO
IQAC
IQAC-E
GMAC
Learning rate Optimizer Total frames
2.5c-4
Adam
2c8
Rollout steps
128
Skip frame
Environments Minibatch size
64
512
Epoch
4
0.99
0.95
Dirac samples
64
Mixtures
5
Table 4: Parameter settings for training PyBullet games
Parameter
PPO
IQAC
IQAC-E
GMAC
Learning rate Optimizer
le-4
Adam
Total env steps
5e-7
Rollout steps
512
Skip fram Environments
1
64
Minibatch
2048
Epoch
10
0.99
0.95
Dirac samples
64
Mixtures
5
17

---

Under review as a conference paper at ICLR 2021
G MORE EXPERIMENTAL RESULTS
G.1 FIVE-STATE MDP
1.2
0.1
0.
2000
4000
6030
B00G
10000
0
2000
4003
6000
5000
10000
(a) Quantile Regression
(b) Expectile Regression
Figure 4: In addition to 2 quantile and expectile regressions also evaluated in the 5-state MDP with imputation using neural networks architectures of implicit quantile network against IQE and Gaussian mixture with their respective loss functions.
0.5
-2
B.0
01
0.50
0.4
0.2
2500
5000
7500
10000 12500 15000
7500
5000
7500
13000 12500 15000
7500 5000
7500 13000 12500 11000
(a) Huber Quantile Regression
(b) Quantile Regression
(c) Expectile Regression
0.50
5.0
0.0
2.5
00
10-
Day
0 2500 5000
7500
10000 12500 11000
7500 5000
7500
13000
17500 15000
7500 5000
7500
13000
1:000
Huber Quantile Regression
(c) Quantile Regression
Expectile Regression
Figure 5: Evaluation of the five-state MDP under tabular setting on symmetric reward distribution (top row) and on assymetric reward distribution (bottom row). Huber quantile(1: 1), quantile, and expectile regressions are compared to the energy distance minimization between samples and Gaussian mixtures.
Here we provide more details on the five-state MDP presented in Figure For each cases in the figure, 15 diracs are used for quantile based methods and 5 mixtures are used GMM balance the total number of parameters required to represent a distribution. For the cases with the label "naīve"" the network outputs (quantiles, expectiles, etc.) are used to create the plot. On the other hand, the cases with "imputation" labels apply appropriate imputation strategy the statistics to produce samples which are then used to plot the distribution. all cases, the network trained on full batches of all states for different empirical samples with Adam optimizer and learning rate of le-3, Sample based energy-distance was used to calculate the distance from the true distribution for all cases. The tabular setting of five-state MDP in Figure 5 Juses same exact experimental set-up as the one with neural network. The only difference is that the statistics are from evenly spaced quantilc/expectile targets e [0, 1].
18

---

Under review as a conference paper at ICLR 2021
G.2
VALUE DISTRIBUTION LEARNING IN ATARI GAMES
IQAC
IQAC-E
GMAC
000
Pollcy Improvament
0.8MM
1.0M
Vix)
aa) Breakout-v4
(b) Breakout-v4
IQAC
IQAC-E
GMAC
Polloy Improvement
+1.
1.0M
1.4M
1.0M
Vixi
(c) Seaquest-v4 v4
(d) Seaquest-v4
IQAC
IQAC-E
GMAC
20112 01
Policy Improvament
2.2M
3.2M
AYAA
4.2M
Vix)
(e) BeamRider-v4
(l) BeamRider-v4
IQAC
IQAC-E
GMAC
Pollay Improvement
3.4M
5.0M
G.GH
Vix)
Visi
(g) Gravitar-v4
(h) Gravitar-v4
Figure 6: More value distributions different tasks. All states are chosen such that the agent is in place of near-death near positive score. Thus, when the policy is not fully trained, such as in a very early stage, the value distribution should include a notion of death indicated by mode positioned at positioned at zero while the other two methods, IQN energy distance (IQAC-E) and GMM energy distance (GMAC) captures the mode in the early stage of policy improvement. Again, the visual representation is maxpool of the 4 frame stacks in the given state.
In all games, IQN Huber quantile (IQAC) fails correctly capture mode
19

---

Under review as a conference paper at ICLR 2021
G.3
LEARNING CURVES
Searriceeed
PRO
300
PPO
10AC
10AC
IOAC
RCAD
GPAC
CMAC
4000
1000
FPQ
2000
R3AC
500
RAAC
1.0
1.5
16B
1cB
led
(a) BeamRider
(b) Breakout
(c) Enduro
Hkr Worcoomooosgooo
PPO
3500
PR0
PPO
25000
I0AC
10MC
GMAL
2500
3300
IQAC
I0AC
LMAA
I0AC
10AC
20000
2300
15000
500
10000
1300
-300
1.0
155
1e0
1e0
lee
(d) Gopher
(c) Gravitar
(f) Montezuma's 's Revenge
Mafuronf
FPO
1500
1260
1240 8
GMMC
1500
1000
1000
300
111
500
I0AC
10AC-
GAAC
0.5
1.5
leB
Je0
(g) Seaquest
(h) Venture
Figure 7: Raw learning curves for 8 selected atari games.
20

---

Under review as a conference paper at ICLR 2021
P90
200
1500
1000
3.00
PrD
FTC
IQAC
500
Ioar-E
-1000
1060-E
(a) Hopper
(b) Walker2D
(c) HalfCheetah
3000
3000
PPO
124C
2500
K2AC-E
2000
2000
1300
P/0
1000
103C
105C-C
500
GHMC
(d) Ant
(c) Humanoid
Figure 8: Raw learning curves for selected PyBllet continuous control tasks.
21

---

Under review as a conference paper at ICLR 2021
Adventure
ArRaid
Allen
Annidar
Asseult
Asterix
Astercids
Eankireist
BattleZone
BeamiRider
Bowling
Boxing
Breakout
Cariivl
Centipede
ChopperCunmand
CrazyCimber
DemonArtack
DoubleDunk
Elevatoractian
Encluro
FichingDerty
Freeway
Prostb
Gopher
Gravitar
Hero
Icellockey
A AmM
Jimesshond
Jo' Fraa
Kangaaon
10 SingF MMsher
MaPacman
Namre MerneeeieGGrreee
Phodenix
Rifell
Porig
Pacyyan
PrivateEye
Obert
Riverraid
RosdRunner
Robotankk
Seeqqest
Skiing
Solaris
Speceirveders
Vennis
TimePilot
Tutankhem
UpNDown
Venbure
Videafinbell
WicerdoFWaa
YarsRevenge
Zzanoon
PPO
IQAC IQAC-E
GMAC
Figure 9: Full learning curves of 61 atari games from ALE
22

---

Under review as a conference paper at ICLR 2021
Table 5: Average score over last 100 episodes in 200M frame collected for training 61 atari games. The algorithms are trained using same single seed and hyperparameters. Random and Human scores taken from Wang al.
GAMES
RANDOM
HUMAN
PPO
IQAC-E
GMAC
Adventure
NA
NA
0,00
0.00
0.00
AirRaid
NA
NA
10,205.75
7,589.50
62,328.75
Alien
227.80
7,127.70
2,918.60
2,704,20
3,687.10
Amidar
5.80
1,719.50
1,244.12
932.11
1,363.72
Assault
222.40
742.00
7,508.03
8,589.55
10,281.73
Asterix
210.00
8,503.30
13,367.00
15,426.00 00
22,650.00
Asteroids
719.10
47,388.70
2,088.10 10
2,332.00
2,597.50
Atlantis
12,850.00
29,028.10
3.073,796.0
3,373,635.00
3,141,534.00
BankHeist
14.20
753.10
1,263.80
1,286.60
1,274.30
BattleZone
2,360.00
37,187.50
18,540.00
21,310.00
32,490.00
BeamRider
363.90
16,926.50
5,913.84
6,507.68
8.718.72
Berzerk
123.70
2,630.40
1,748.10
887.50
3,081.20
Bowling
23.10
160.70
33.54
30.00
19.39
Boxing
0.10
12.10
96.79
97.10
99.89
Breakout
1.70
30.50
384.29
445.64
462.68
Carnival
NA
NA
5,079.20
4,401.00
6,344.20
Centipede
2.090.90
12,017.00
5,205.25
4,864.69
4,303.10
ChopperConmand
811.00
7,387.90
872.00
1,314.00
1,795.00
CrazyClimber Climber
10,780.50
35,829.40
112,640.00
121,550.00
125,143.00
DemonAttack
152.10
1,971.00
50,590.65
236.839.85
411,118,85
DoubleDunk
-18.60
-16.40
-3.26
-8,28
-2.72
FlevatorAction
NA
NA
10,449.00
8,516.00
14,254.00
Enduro
0.00
860.50
1,588.68
1,612.17
2,092.65
FishingDerby
-91.70
-38.70
37.01
33.13
37.52
Freeway
0.00
29.60
32.53
33.68
32.84
Frosthite
62.50
4,334.70
3,571.50
307.10
3,392.40
Gopher
257.60
2,412.50
8,199.80
16,934.60
25,266.80
Gravitar
173.00
3,351.40
1,151.50
2,178.50
2,401.00
Hero
1,027.00
30,826.40
37,725.55
43,065.95
41,509.05
IceHockey
-11.20
0.90
-1.90
2.13
0.34
Jameshond
29.00
302.80
642.50
961.00
1,512.00
Jouane
NA
NA
-607.00
-840.00
-680.00
Kangaroo
52.00
3,035.00
1,742.00
12,208
12,909.00
Krull
1,598.00
2,665.50
9,605.51
9,514.03
9,127.62 63
KungFuMaster
258.50
22,736.50
26,846.00
33,378
31,025.00
Montezuma.Revenge
4,753.30
0.00
0.00
0.00
MsPaeman
307.30
6,951.60
3,674.20
4,699.00
3.884.40
NaimmeiGemme
2,292.30
8.049.00
13,229.10
13,454.00
14.031.30
Phoenix
761.40
7,242.60
37,263.70
25.154.00
42,664.00
Pitfall
-229.40
6,463.70
-18.86
-3.36
Pong
-20.70
14.60
20.87
20.88
20.97
Pooyan
4.018.95
3,674.70
4,178.65
PrivateEye
24.90
69,571.30
100.00
196.30
100.00
Qbert
163.90
13.455.00
25,519.25
21,599.50
23.176.25
Riverraid
1,338.50
17.118.00
15,983.00
18,073
19,761.30
ReadRunner
11.50
7,845.00
56,321.00
56,121.00
68,272.00
Robotank
11.90
23.45
36.69
45.82
Seaquest
42.054.70
1,832.00
1,814.60
1.838.40
Skiing
-17,098.10
-4,36.990
-7,958.81
-29.971.02
29,975.52
Solaris
1,236.30
12,326.70
2,452.80
2,204.80
2,579.20
SpaceInvaders
148.00
1,668.70
2,544.10
2,410,90
2,228.30 30
StarGunnner
664.00
10,250.00
74,848.00
97,450.00
184,188.00
Tennis
-23.80
-8.30
-8.16
-7.54
-5.90
TimePilot
3,568.00
5,229.20
12,157.00
11,704.00
13,227.00
Tutankham
11,40
167.60
206.32
208.72
209.82
UpNDown
533.40
11,693.00
158,629.50
161,328.40
129,243.70
Venture
0.00
1,187.50
0.00
1,339.00
1,181.00
VideoPinball
16,256.90
17,667.90
279,504.81
59,988 90
55,272.87 82
WizardOfWar
563,50
4,756.50
8,749.00
9,165.00
11,388.00
YarsRevenge
3,092.90
54,576.90
92,709.90 94
100,082.55
103,895.05
Zaxxon
32.50
9.173.00
13,336.00
14,882.00
18,436.00
23


Reviews:
[{"review_id": "wiUGjkPrdrR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a Gaussian-mixture Actor Critic (GMAC) framework to address the problems of distributional RL (distRL): distributional instability, action-type restrictions, and biased approximation (replacing Wasserstein distance with Cramer distance. The framework uses Gaussian mixtures to represent the distribution of value functions, and learn the Gaussian distribution parameters. \n\nUsing GMMs for distRL: it appears explored before in ref 1, but not cited. \n\nmy problem of GMMs: the model may be not Gaussian. The distribution learned by DistRL is prevoulsy shown to be asymmetirc (Fig 5 of Mavrin et. al. 2019). \n\none question: since the three problems also exist for distRL alg's like QRDQN, why don't you improve the original value function approximation distRL alg's, but instead on AC algorithms?\n\nDRL:using it for distributional rl is a bit unconventional. DRL: deep reinforcement learning. \n\n\nMorimura et al. (2010a;b) designed a risk-sensitive algorithm using a distributional perspective: this paper is perhaps the earliest concept of distRL. \n\n\nMavrin et al. (2019) utilized the idea of the uncertainty captured from the variance of value distribution with a decay factor to add an intrinsic reward to the objective of conventional greedy policy: \"utilized the idea of the uncertainty captured from the variance of value distribution\" is correct. DLTV uses the distribution/quantiles/variance to estimate the upper confidence bound and use it for action selection. It's not \"adding an intrinsic reward\". \n\nour work is the first to connect stochastic policy as a solution to\nthe problems in value-based DRL: this isn't very clear. \n\nWe believe that the findings from this paper can easily generalize\nto other actor-critic frameworks as well: are you sure the other alg's all have the same kind of the (three) issues?\n\nBelow eq 15:\ndo you mean Bellman optimality operator is a contraction mapping? It appears so from Appendix C. this is interesting because with Wasserstein distance the Bellman optimality operator is not a contraction in any norm (pls confirm this is correct).  \n\nExperiments were performed on two-state MDP (to test the Cramer loss func is less biased while Huber loss function used in QRDQN is biased), \n\nRepresenting Multimodality section:\nHere the experiment lacks a study of assymetric models where GMMs cannot represents. Previously experiments by Mavrin et. al. 2019 showed in Pong, e.g., the Q value function distribution is not symmetric. \n\nDiscrete and Continuous Action Spaces section:\nDo you use the same algorithm for both cases?\tWhy PPO, IQAC IQAC-E GMAC are selected? How these alg's compare to distRL alg's like DQN, C51, QRDQN?\n \n\nRef 1:\nDistributional Deep Reinforcement Learning\nwith a Mixture of Gaussians\nhttp://cpslab.snu.ac.kr/publications/papers/2019_icra_ddrl_mog.pdf\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "jX5Gw5D5ycB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a distributional Actor critic framework (GMAC) based on GMM, Actor critic and Cramer distance.\nAuthors introduce SR(λ) a distributional version of the λ-return algorithm and to minimize the Cramer distance - as opposed to minimizing the Wasserstein distance using Huber quantile regression- between the value distribution, this helps in obtaining unbiased sampled gradients, this is shown to be more effective in preserving modality that can provide extra information in sparse-reward exploration tasks as well as more stability during training. Finally, authors choose to parametrize value distributions as a GMM this provides a closed-form to the energy distance (eq19) which can reduce computational costs achieving very close numbers to PPO. \n\nClarity: The paper is easy to follow and well written. The motivations are quite clear from the beginning. The paper would have benefited from highlighting why GMAC specifically is suitable to handle discrete and continuous actions and the challenges behind each case (see q1).\n\nNovelty: This work introduces many novel aspects, importantly the use of the Cramer + GMM for getting an unbiased sample gradient plus computational efficiency.   \n\nExperiments and significance of the empirical results: \nAuthors evaluate GMAC using a two-state MDP, a set of discrete and continuous action space tasks. the majority of the results presented show convincing improvements of GMAC over IQAC and the PPO baselines. IQN + energy distance (IQAC-E) shows improvements in capturing the modality over IQAC confirming the intuition behind the proposed use of Cramer distance. Both GMAC and IQAC-E show improvements in the computational costs. However, I do have some concerns considering the selection of the displayed examples and table of results in the appendix including only the baseline PPO (Table 5) (see q2&q3). \n\n\nQuestions:\nq1: in Figure 7, the PyBullet learning curves, In 3/5 of the learning curves IQN + Huber quantile (IQAC) seems to be performing on par or better than GMAC. this makes me wonder what conceptually makes GMAC specifically suitable for both Discrete and continuous action spaces.\n\nq2: Since there are no space limitations in the appendix. In Figure 6, I am wondering the reasons behind displaying only the learning curves of 8 selected games?  \n\nq3: Could you In Table 5 display the Average scores of IQAC and IQAC-E. Some of the reported results of GMAC are inferior to those (QR-DQN & IQN) which are reported in (Dabney et al 2018a) If those results comparable, It would be better to put them side by side for comparison to confirm the claims wrt performance superiority. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "a3Clmt4sfxW", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to learn a Gaussian Mixture Model of the distribution of returns and use it as the critic in an actor-critic RL agent. From what I can tell the principal novel contribution of this work is the Sample-Replacement method, in particular the observation that when paired with a GMM the replacement can be done at the level of modes of the mixture instead of individual samples. Another potential contribution is showing that the GMM can be optimized using the Cramer metric, although obviously this metric has been fairly widely studied previously.\n\nHowever, this work has several problems that make it unpublishable at the moment. I'll begin with the least severe (lack of clarity and poor attribution) and then move to the much more problematic (misleading statements and factual errors).\n\n\nUnclear:\n\nSection 4.3, assumptions:\n\"1) the reward given a state-action pair R(x, a) follows a single distribution with finite variance\"\n\nWhat does this mean? Finite variance is clear, what do you mean \"a single distribution\".\n\n\"3) the policy follows a distribution which can be approximated by Dirac mixtures\"\n\nWhat does this mean? Approximated how well? Under what metric?\n\nEquation 18, second line, \\mu and \\sigma seem like they should both be functions of (x, a).\n\n\nPoor attribution:\n\nExisting work has used Gaussian Mixture Models for distributional RL, as well as for the actor-critic setting (D4PG, among others).\n\nExisting work has considered multi-step returns in distributional RL (Rainbow, Reactor, as well as almost all methods that use AC with Dist. RL). However, the Sample-Replacement method is an interesting contribution that is novel compared with this existing work.\n\n\"The distributional Bellman operator is a [...] contraction mapping in the Cramer metric space, whose proof can be found in Appendix C.\"\n\nThis has previously been proven in the Rowland et al. (2019) paper the authors cite, but do not attribute such a result to.\n\n\nMisleading statements:\n\n\"Third, the Wasserstein distance that is commonly used in DRL does not guarantee unbiasedness in sample gradients\"\n\nWhile this is true for direct minimization, the quantile regression work cited in this paper does guarantee unbiased sample gradients.\n\n\"The instability issue is not present under the stochastic policy... Combining these solutions, we arrive at a distributional actor-critic...\" (Much later) \"One way to overcome this issue is learning value distributions under the stochastic policy and finding an optimal policy under principles of conservative policy iteration...\"\n\nThe instability issue the authors reference here is that the distribution of returns, though not its mean, can be an expansion under any probability metric when applying the optimality operator. While this is an interesting topic, the authors do not actually address it or contribute towards its understanding or solution in any way. The evaluation operator was already known to be a contraction in Wasserstein (as well as for Cramer), which is the relevant operator when considering an actor-critic framework. Unlike the authors' claim that this is due to using a stochastic policy, it is in fact due to performing evaluation as opposed to optimality operators.\n\n\"Barth-Maron et al (2018) expanded DDPG by training a distributional critic through quantile regression.\"\n\nThis is completely incorrect, as they considered categorical distributions and Gaussian mixtures, but not quantile regression.\n\n\n\"The actor-critic method is a specific case of temporal-difference (TD) learning method in w hich the value function, the critic, is learned through the TD error defined by the difference...\"\n\nActor-critic uses TD to learn the critic, but it is not a specific case of TD.\n\n\"However, the Wasserstein distance minimized in the implicit quantile network cannot guarantee the unbiasedness of the sample gradient, meaning it may not be suitable for empirical distributions like equation 13.\"\n\nThis is 100% false and shows a lack of understanding of multiple papers being cited in this work.\n\nFigure 2 and \"Wasserstein distance (labeled as IQ) converges to a local minimum which does not correctly capture the locations of the modes\"\n\nThis seemed off to me so I went ahead and reimplemented this experiment myself. This has nothing to do with the Wasserstein distance and is exceedingly misleading to the reader. Suggestion to read the Rowland et al. (2019) paper that the authors cite for better understanding. Huber-quantiles are not quantiles. The authors learn Huber-quantiles and then treat them as quantiles and observe they look wrong. If you run IQ with the Huber parameter at 0 (corresponding to quantiles) then you get the correct (unbiased) distribution. If you instead learn Huber-quantiles and use the imputation in the Rowland you again get the right distribution.\n\nThe experimental results in the main text look promising for GMAC, but looking at the full set of results in the appendix paints a much more mixed picture.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RCVGbazoq7e", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes an actor-critic algorithm based on distributional RL. Distributions are approximated as mixtures of Gaussians, Bellman targets are computed using lambda-returns, and the training loss is computed using an energy distance.\n\nReview Summary\n\nThere is a solid algorithmic contribution in this paper, although at present the comparison against baselines considered feels a little incomplete. I would be happy to review my rating if further results are reported.\n\nSignificance\n\nWhile several aspects of the proposed algorithm have been explored before (multi-step returns in distributional RL, Cramer distance in training loss), these components are combined to yield a novel algorithm, and the treatment of multi-step returns differs from previous work. \n\nQuality & Clarity\n\nThe proposed method is presented reasonably clearly in Algorithm 1 and related discussion in Section 4. There is some looseness of mathematical notation in Section 4, such as the use of densities, and the discussion around Equation (18), but broadly the paper is clear. \n\nTechnical comments\n\nThe paper mentions three issues with DRL: instability in the control setting, specificity of algorithms to certain action spaces, and biasedness in sample gradients. I think the first two points stand, but am less sure about the third. Wasserstein distances have been used in the analysis of distributional RL algorithms, but as far as I know in general they are not used algorithmically. Many distributional algorithms use methods that do yield unbiased gradients, such as the quantile regression approach used by QR-DQN, IQN, and FQF, and the Cramer distance used by S51 in \"Distribution Reinforcement Learning with Linear Function Approximation\". This issue is mentioned again below Equation (14) in comparison with IQN, which uses a quantile regression loss, rather than an empirical Wasserstein distance loss. Can the authors comment on this?\n\nDiscussion around Equation (18): Is this a formal claim (that the density of Z(x, a) can be approximated by a Gaussian mixture distribution)? It seems as though without being quantitative, this argument could be made for any distribution with a density. As an additional note, what if Z(x, a) does not have a density?\n\nExperiment hyperparameters: How were the hyperparameters (e.g. learning rate, Adam epsilon, lambda, etc.) selected for the methods considered in the paper? Were any hyperparameter sweeps undertaken? What does \"Epoch\" in Tables 3 and 4 refer to?\n\nFigure 2 experiment. The text mentions that minimizing the Wasserstein distance (labelled as IQ) leads to a local minimum. Can the authors clarify whether they are minimizing the Wasserstein distance between approximate distributions, or minimizing a quantile regression loss?\n\nThe intrinsic motivation experiments are interesting. I couldn't find details in the appendix as to precisely how the intrinsic reward is used - is the raw TD/Cramer distance used as a reward/is there any scaling applied? Is the Cramer error typically of a different magnitude to the TD error? One might expect the raw TD error to be of a very low magnitude, which may mean that it requires scaling before use as an intrinsic reward (although of course scaling introduces an additional hyperparameter).\n\nHow were the Atari results displayed in Figures 3 & 6 selected? It looks as though in all games in Figure 3, GMAC comes out on top, although judging from Figure 6 this is not always the case. Can the authors include the results of IQAC/IQAC-E in Table 5? Can the authors report summary statistics for the performance of the algorithms across the suite of games considered, such as human-normalized mean/median performance? At present the comparison of GMAC against the baselines feels a little incomplete; as far as I can tell, full results for IQAC/IQAC-E are not reported in the paper.\n\nGMAC implementation details. I couldn't find precise details on the architecture used for GMAC, in particular the outputs of the network. Presumably each mixture component requires 3 heads (for mixture weight, mean, and variance/stddev). Is a softmax used to ensure the sum of the weights is 1? How is non-negativity of the variance parameters enforced?\n\nIQN-based agent implementation details. Is the embedding of tau as in Equation (4) of \"Implicit Quantile Networks for Distributional Reinforcement Learning\", but with n=32?\n\nMinor comments\n\n\"Note that the cdf of \\tilde{Z} has a different domain from \\tilde{F}_Z\". This wasn't clear to me, can the authors expand?\n\nBelow Equation (12), the authors discuss pdfs of Z^{(n)}_t etc., but presumably these random variables may not have pdfs?\n\nI didn't understand the comment that the cdf F_{Z^{(\\lambda)}_t} has a simple form, and that evaluating requires O(n^2) time and memory. Presumably if the CDFs that appear in the mixture in Equation (12) don't have a simple form, then neither will F_{Z^{(\\lambda)}_t}? Can the authors give more detail on where O(n^2) time and space complexity come from?\n\nSection 4.3: \"The reward given a state-action pair R(x, a) follows a single distribution with finite variance\". What is meant by \"single distribution\" here?\n\nEquation (19): I think there are missing factors of 1/2 in front of the \\delta(X, X') and \\delta(Y, Y') terms.\n\nMinor comments on formatting etc.\nConsider using \\eqref rather than \\ref when referring to equations.\nReferences should point to conference versions of papers (rather than arxiv) where appropriate (e.g. WGAN reference).\n\n\n\nPost-rebuttal update\n\nOverall, I am still borderline on this paper. I appreciate the effort the authors have put in during the rebuttal phase, and would say the paper is now clearer. \nThe inclusion of mean/median normalized scores for the Atari results in the main paper has improved the experimental section. However, the main drawbacks that remain are empirical; the smaller scale comparisons between methods need to be updated to give a like-for-like comparison in terms of numbers of parameters etc., as the authors acknowledge, and the paper still lacks baseline distributional agents in the large-scale experiments. This is important, as it means it is difficult to assess the impact that, for example, SR(lambda) may be having on the experimental results.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a distributional perspective on the value function and uses it to modify PPO for both discrete and continuous control reinforcement learning tasks. The referees had noticed a number of wrong/misleading statements in the initial version of the submission, and the AC had also pointed out several problematic statements in a revised version. While the authors had acknowledged these mistakes and made appropriate corrections, there are several places that still need clear improvement before the paper is ready for publication. The paper seems to introduce a novel actor-critic algorithm. However, the correctness of its key step, the  SR($\lambda)$ algorithm, has not been rigorously justified. For example, it is unclear how the geometric random variables would arise in that algorithm. For experiments, the AC seconds the comments provided by Reviewer 2 during the discussion: "The empirical comparisons are overall still lacking: for the smaller-scale experiments, whilst the authors have been actively engaged in improving these comparisons during the rebuttal, at present, they are still in need of updating to make a fair comparison, for example in terms of the number of parameters included. The authors have acknowledged this, although the rebuttal period ran out before they were able to post new plots. The large-scale empirical results are still lacking reasonable baselines against existing distributional RL agents."


Decision: Reject