Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a distributional Actor critic framework (GMAC) based on GMM, Actor critic and Cramer distance.
Authors introduce SR(λ) a distributional version of the λ-return algorithm and to minimize the Cramer distance - as opposed to minimizing the Wasserstein distance using Huber quantile regression- between the value distribution, this helps in obtaining unbiased sampled gradients, this is shown to be more effective in preserving modality that can provide extra information in sparse-reward exploration tasks as well as more stability during training. Finally, authors choose to parametrize value distributions as a GMM this provides a closed-form to the energy distance (eq19) which can reduce computational costs achieving very close numbers to PPO. 

Clarity: The paper is easy to follow and well written. The motivations are quite clear from the beginning. The paper would have benefited from highlighting why GMAC specifically is suitable to handle discrete and continuous actions and the challenges behind each case (see q1).

Novelty: This work introduces many novel aspects, importantly the use of the Cramer + GMM for getting an unbiased sample gradient plus computational efficiency.   

Experiments and significance of the empirical results: 
Authors evaluate GMAC using a two-state MDP, a set of discrete and continuous action space tasks. the majority of the results presented show convincing improvements of GMAC over IQAC and the PPO baselines. IQN + energy distance (IQAC-E) shows improvements in capturing the modality over IQAC confirming the intuition behind the proposed use of Cramer distance. Both GMAC and IQAC-E show improvements in the computational costs. However, I do have some concerns considering the selection of the displayed examples and table of results in the appendix including only the baseline PPO (Table 5) (see q2&q3). 


Questions:
q1: in Figure 7, the PyBullet learning curves, In 3/5 of the learning curves IQN + Huber quantile (IQAC) seems to be performing on par or better than GMAC. this makes me wonder what conceptually makes GMAC specifically suitable for both Discrete and continuous action spaces.

q2: Since there are no space limitations in the appendix. In Figure 6, I am wondering the reasons behind displaying only the learning curves of 8 selected games?  

q3: Could you In Table 5 display the Average scores of IQAC and IQAC-E. Some of the reported results of GMAC are inferior to those (QR-DQN & IQN) which are reported in (Dabney et al 2018a) If those results comparable, It would be better to put them side by side for comparison to confirm the claims wrt performance superiority. 

review_title: Interesting novel ideas but some concerns w.r.t evaluation.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes to learn a Gaussian Mixture Model of the distribution of returns and use it as the critic in an actor-critic RL agent. From what I can tell the principal novel contribution of this work is the Sample-Replacement method, in particular the observation that when paired with a GMM the replacement can be done at the level of modes of the mixture instead of individual samples. Another potential contribution is showing that the GMM can be optimized using the Cramer metric, although obviously this metric has been fairly widely studied previously.

However, this work has several problems that make it unpublishable at the moment. I'll begin with the least severe (lack of clarity and poor attribution) and then move to the much more problematic (misleading statements and factual errors).


Unclear:

Section 4.3, assumptions:
"1) the reward given a state-action pair R(x, a) follows a single distribution with finite variance"

What does this mean? Finite variance is clear, what do you mean "a single distribution".

"3) the policy follows a distribution which can be approximated by Dirac mixtures"

What does this mean? Approximated how well? Under what metric?

Equation 18, second line, \mu and \sigma seem like they should both be functions of (x, a).


Poor attribution:

Existing work has used Gaussian Mixture Models for distributional RL, as well as for the actor-critic setting (D4PG, among others).

Existing work has considered multi-step returns in distributional RL (Rainbow, Reactor, as well as almost all methods that use AC with Dist. RL). However, the Sample-Replacement method is an interesting contribution that is novel compared with this existing work.

"The distributional Bellman operator is a [...] contraction mapping in the Cramer metric space, whose proof can be found in Appendix C."

This has previously been proven in the Rowland et al. (2019) paper the authors cite, but do not attribute such a result to.


Misleading statements:

"Third, the Wasserstein distance that is commonly used in DRL does not guarantee unbiasedness in sample gradients"

While this is true for direct minimization, the quantile regression work cited in this paper does guarantee unbiased sample gradients.

"The instability issue is not present under the stochastic policy... Combining these solutions, we arrive at a distributional actor-critic..." (Much later) "One way to overcome this issue is learning value distributions under the stochastic policy and finding an optimal policy under principles of conservative policy iteration..."

The instability issue the authors reference here is that the distribution of returns, though not its mean, can be an expansion under any probability metric when applying the optimality operator. While this is an interesting topic, the authors do not actually address it or contribute towards its understanding or solution in any way. The evaluation operator was already known to be a contraction in Wasserstein (as well as for Cramer), which is the relevant operator when considering an actor-critic framework. Unlike the authors' claim that this is due to using a stochastic policy, it is in fact due to performing evaluation as opposed to optimality operators.

"Barth-Maron et al (2018) expanded DDPG by training a distributional critic through quantile regression."

This is completely incorrect, as they considered categorical distributions and Gaussian mixtures, but not quantile regression.


"The actor-critic method is a specific case of temporal-difference (TD) learning method in w hich the value function, the critic, is learned through the TD error defined by the difference..."

Actor-critic uses TD to learn the critic, but it is not a specific case of TD.

"However, the Wasserstein distance minimized in the implicit quantile network cannot guarantee the unbiasedness of the sample gradient, meaning it may not be suitable for empirical distributions like equation 13."

This is 100% false and shows a lack of understanding of multiple papers being cited in this work.

Figure 2 and "Wasserstein distance (labeled as IQ) converges to a local minimum which does not correctly capture the locations of the modes"

This seemed off to me so I went ahead and reimplemented this experiment myself. This has nothing to do with the Wasserstein distance and is exceedingly misleading to the reader. Suggestion to read the Rowland et al. (2019) paper that the authors cite for better understanding. Huber-quantiles are not quantiles. The authors learn Huber-quantiles and then treat them as quantiles and observe they look wrong. If you run IQ with the Huber parameter at 0 (corresponding to quantiles) then you get the correct (unbiased) distribution. If you instead learn Huber-quantiles and use the imputation in the Rowland you again get the right distribution.

The experimental results in the main text look promising for GMAC, but looking at the full set of results in the appendix paints a much more mixed picture.
review_title: Concerns with clarity, attribution and correctness

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a Gaussian-mixture Actor Critic (GMAC) framework to address the problems of distributional RL (distRL): distributional instability, action-type restrictions, and biased approximation (replacing Wasserstein distance with Cramer distance. The framework uses Gaussian mixtures to represent the distribution of value functions, and learn the Gaussian distribution parameters. 

Using GMMs for distRL: it appears explored before in ref 1, but not cited. 

my problem of GMMs: the model may be not Gaussian. The distribution learned by DistRL is prevoulsy shown to be asymmetirc (Fig 5 of Mavrin et. al. 2019). 

one question: since the three problems also exist for distRL alg's like QRDQN, why don't you improve the original value function approximation distRL alg's, but instead on AC algorithms?

DRL:using it for distributional rl is a bit unconventional. DRL: deep reinforcement learning. 


Morimura et al. (2010a;b) designed a risk-sensitive algorithm using a distributional perspective: this paper is perhaps the earliest concept of distRL. 


Mavrin et al. (2019) utilized the idea of the uncertainty captured from the variance of value distribution with a decay factor to add an intrinsic reward to the objective of conventional greedy policy: "utilized the idea of the uncertainty captured from the variance of value distribution" is correct. DLTV uses the distribution/quantiles/variance to estimate the upper confidence bound and use it for action selection. It's not "adding an intrinsic reward". 

our work is the first to connect stochastic policy as a solution to
the problems in value-based DRL: this isn't very clear. 

We believe that the findings from this paper can easily generalize
to other actor-critic frameworks as well: are you sure the other alg's all have the same kind of the (three) issues?

Below eq 15:
do you mean Bellman optimality operator is a contraction mapping? It appears so from Appendix C. this is interesting because with Wasserstein distance the Bellman optimality operator is not a contraction in any norm (pls confirm this is correct).  

Experiments were performed on two-state MDP (to test the Cramer loss func is less biased while Huber loss function used in QRDQN is biased), 

Representing Multimodality section:
Here the experiment lacks a study of assymetric models where GMMs cannot represents. Previously experiments by Mavrin et. al. 2019 showed in Pong, e.g., the Q value function distribution is not symmetric. 

Discrete and Continuous Action Spaces section:
Do you use the same algorithm for both cases?	Why PPO, IQAC IQAC-E GMAC are selected? How these alg's compare to distRL alg's like DQN, C51, QRDQN?
 

Ref 1:
Distributional Deep Reinforcement Learning
with a Mixture of Gaussians
http://cpslab.snu.ac.kr/publications/papers/2019_icra_ddrl_mog.pdf


review_title: An ambitious AC framework trying to solve 3 problems with Distributional RL

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary

This paper proposes an actor-critic algorithm based on distributional RL. Distributions are approximated as mixtures of Gaussians, Bellman targets are computed using lambda-returns, and the training loss is computed using an energy distance.

Review Summary

There is a solid algorithmic contribution in this paper, although at present the comparison against baselines considered feels a little incomplete. I would be happy to review my rating if further results are reported.

Significance

While several aspects of the proposed algorithm have been explored before (multi-step returns in distributional RL, Cramer distance in training loss), these components are combined to yield a novel algorithm, and the treatment of multi-step returns differs from previous work. 

Quality & Clarity

The proposed method is presented reasonably clearly in Algorithm 1 and related discussion in Section 4. There is some looseness of mathematical notation in Section 4, such as the use of densities, and the discussion around Equation (18), but broadly the paper is clear. 

Technical comments

The paper mentions three issues with DRL: instability in the control setting, specificity of algorithms to certain action spaces, and biasedness in sample gradients. I think the first two points stand, but am less sure about the third. Wasserstein distances have been used in the analysis of distributional RL algorithms, but as far as I know in general they are not used algorithmically. Many distributional algorithms use methods that do yield unbiased gradients, such as the quantile regression approach used by QR-DQN, IQN, and FQF, and the Cramer distance used by S51 in "Distribution Reinforcement Learning with Linear Function Approximation". This issue is mentioned again below Equation (14) in comparison with IQN, which uses a quantile regression loss, rather than an empirical Wasserstein distance loss. Can the authors comment on this?

Discussion around Equation (18): Is this a formal claim (that the density of Z(x, a) can be approximated by a Gaussian mixture distribution)? It seems as though without being quantitative, this argument could be made for any distribution with a density. As an additional note, what if Z(x, a) does not have a density?

Experiment hyperparameters: How were the hyperparameters (e.g. learning rate, Adam epsilon, lambda, etc.) selected for the methods considered in the paper? Were any hyperparameter sweeps undertaken? What does "Epoch" in Tables 3 and 4 refer to?

Figure 2 experiment. The text mentions that minimizing the Wasserstein distance (labelled as IQ) leads to a local minimum. Can the authors clarify whether they are minimizing the Wasserstein distance between approximate distributions, or minimizing a quantile regression loss?

The intrinsic motivation experiments are interesting. I couldn't find details in the appendix as to precisely how the intrinsic reward is used - is the raw TD/Cramer distance used as a reward/is there any scaling applied? Is the Cramer error typically of a different magnitude to the TD error? One might expect the raw TD error to be of a very low magnitude, which may mean that it requires scaling before use as an intrinsic reward (although of course scaling introduces an additional hyperparameter).

How were the Atari results displayed in Figures 3 & 6 selected? It looks as though in all games in Figure 3, GMAC comes out on top, although judging from Figure 6 this is not always the case. Can the authors include the results of IQAC/IQAC-E in Table 5? Can the authors report summary statistics for the performance of the algorithms across the suite of games considered, such as human-normalized mean/median performance? At present the comparison of GMAC against the baselines feels a little incomplete; as far as I can tell, full results for IQAC/IQAC-E are not reported in the paper.

GMAC implementation details. I couldn't find precise details on the architecture used for GMAC, in particular the outputs of the network. Presumably each mixture component requires 3 heads (for mixture weight, mean, and variance/stddev). Is a softmax used to ensure the sum of the weights is 1? How is non-negativity of the variance parameters enforced?

IQN-based agent implementation details. Is the embedding of tau as in Equation (4) of "Implicit Quantile Networks for Distributional Reinforcement Learning", but with n=32?

Minor comments

"Note that the cdf of \tilde{Z} has a different domain from \tilde{F}_Z". This wasn't clear to me, can the authors expand?

Below Equation (12), the authors discuss pdfs of Z^{(n)}_t etc., but presumably these random variables may not have pdfs?

I didn't understand the comment that the cdf F_{Z^{(\lambda)}_t} has a simple form, and that evaluating requires O(n^2) time and memory. Presumably if the CDFs that appear in the mixture in Equation (12) don't have a simple form, then neither will F_{Z^{(\lambda)}_t}? Can the authors give more detail on where O(n^2) time and space complexity come from?

Section 4.3: "The reward given a state-action pair R(x, a) follows a single distribution with finite variance". What is meant by "single distribution" here?

Equation (19): I think there are missing factors of 1/2 in front of the \delta(X, X') and \delta(Y, Y') terms.

Minor comments on formatting etc.
Consider using \eqref rather than \ref when referring to equations.
References should point to conference versions of papers (rather than arxiv) where appropriate (e.g. WGAN reference).



Post-rebuttal update

Overall, I am still borderline on this paper. I appreciate the effort the authors have put in during the rebuttal phase, and would say the paper is now clearer. 
The inclusion of mean/median normalized scores for the Atari results in the main paper has improved the experimental section. However, the main drawbacks that remain are empirical; the smaller scale comparisons between methods need to be updated to give a like-for-like comparison in terms of numbers of parameters etc., as the authors acknowledge, and the paper still lacks baseline distributional agents in the large-scale experiments. This is important, as it means it is difficult to assess the impact that, for example, SR(lambda) may be having on the experimental results.
review_title: Interesting application of distributional RL to actor-critic, although with scope for further empirical comparison

Meta-review: The paper proposes a distributional perspective on the value function and uses it to modify PPO for both discrete and continuous control reinforcement learning tasks. The referees had noticed a number of wrong/misleading statements in the initial version of the submission, and the AC had also pointed out several problematic statements in a revised version. While the authors had acknowledged these mistakes and made appropriate corrections, there are several places that still need clear improvement before the paper is ready for publication. The paper seems to introduce a novel actor-critic algorithm. However, the correctness of its key step, the  SR($\lambda)$ algorithm, has not been rigorously justified. For example, it is unclear how the geometric random variables would arise in that algorithm. For experiments, the AC seconds the comments provided by Reviewer 2 during the discussion: "The empirical comparisons are overall still lacking: for the smaller-scale experiments, whilst the authors have been actively engaged in improving these comparisons during the rebuttal, at present, they are still in need of updating to make a fair comparison, for example in terms of the number of parameters included. The authors have acknowledged this, although the rebuttal period ran out before they were able to post new plots. The large-scale empirical results are still lacking reasonable baselines against existing distributional RL agents."


Decision: Reject