Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper suggests exploiting the awareness and available identification of multiple reward sources to factorise a distributional value function and exploit this explicit structure to learn and exploit the correlation structure of distinct reward sources. The overall architecture predicts representative samples that describe the joint (total) value distribution. The paper presents experimental results that confirm efficacy of the approach. The paper appears to be correct overall, although there are some technical issues that mar the overall impression. 

main_review: First noticeable issue with the paper is Algorithm-1. Authors suggest that it is supposed to calculate the gradient of the MMD loss. However, it actually calculates an estimate of the MMD loss itself. This is in a sharp contrast to the preceding paragraph that speaks of the necessary simplifications. The appendix is of no help in this situation. Authors do not describe how the loss estimate will translate into it's gradient estimate. The text of the algorithm is also festered by repeated use of symbols at distinct semantic locations ("a prime" serves too many roles here, so is Z_{1:M}). It is difficult to imagine how this pseudo-code was translated into the actual code to function correctly. Notwithstanding the practical success of the code, this does not bode well for the alignment of the theory and practice of this paper. 

It is also not clear why borrowing a few factorisation tricks from the MARL literature, where the correlation of independent executions is implicitly learned, can not be of use here. QR-MIX and MMD-MIX would be among the more recent structures that can be successfully deployed here. Authors cannot avoid a justification of not using these architectures (at least as a baseline) in their domain. 

Authors also miss more important references that bear key similarities to their approach:
"Distributional Multivariate Policy Evaluation and Exploration with the Bellman GAN" by Freirich et al
"Sample-based distributional policy gradient" by Singh, Lee and Chen
It would be interesting if authors could compare their work to the above two (at least in the rebuttal).

Peeves and typos:

The paper is also overly reliant on the Appendix. Though the claims to appear to be correct, there are intuitions that may be gathered here, that do not appear in the main text. 

Line 169 "Crame_r" 
Line 170 "Operator" 
Line 277, what is the exact nature of an "iteration" here? 


limitations_and_societal_impact: Work is sufficiently distant from real world applications to avoid social impact.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The authors present MD3QN, which leverages the distributional approximation capacity of C51 and its likes as well as the reward multiplicity of some reinforcement learning tasks. The line or research centered around C51 has the advantage of learning an entire distribution of the reward (instead of just the expectation), which is often considered a better representation of the RL task at hand. Meanwhile, the line of research around HRA separately model the value function for each reward source to better the algorithm's sample efficiency. MD3QN combines the said advantages in a correlated multiple-reward setting where the agent learns the joint reward distribution for a better task representation and a better sample efficiency. The proposed algorithm is shown to outperform one previous work (HRA) in a number of Atari games, and it seems to learn the reward correlations from custom-designed mazes. 
main_review: Originality:
I think the presentation could do a much better job at highlighting the novelty of the algorithm design. The current paper does not pinpoint the challenges encountered in developing MD3QN. I struggled to identify the unique contribution of MD3QN in the transition from Section 2 to Section 3, where MD3QN is illustrated almost like a mere combination of C51 and HRA when that is certainly not the case according to the introduction. The strength of MD3QN must then lie in capturing the correlated randomness in source-specific returns, but the experiments and the figures do not point to that strength. Figure 2 is especially obscure in that I wasn't convinced by its effectiveness or its message. How better does MD3QN do in identifying the correlations than do previous works? What quantitative measures could you utilize to gauge the performance in capturing the correlations? Also, why have the maze-multireward rewards been split into blue-orange and red-green pairs?

On a separate note, the paper claims multiple times that it "corrects the misunderstanding" of existing works, but I would like to raise a few points on this claim. The first mention of this claim is in Section 1, where "We correct a misunderstanding in existing works on distributional RL, and argue that we can use other metrics than the Wasserstein metric over the joint distribution as training losses to approximate the Bellman operator." The second and third mentions of the claim both appear in Section 3: "... misunderstanding that we must use the same metric for proving convergence and as training loss." Now, those two understandings are very different from each other, and the rest of the paper does not clarify which "misunderstanding" that the authors are "correcting". Is it that only the Wasserstein metric must be used? Or is it that the same metric must be used in convergence and in training loss? Furthermore, the first mention of the "misunderstanding in existing works" requires references. The second and third mentions of the "misunderstanding" do contain references to existing works, but as far as my understanding of Bellemare2017 and Nguyen2020 goes, those two papers in no way claim  that "only the Wasserstein metric must be used" or "the same metric must be used in convergence and in training loss". In fact, the MD3QN authors only list some other works that employ different metrics for showing convergence and in training loss. Consequently, I am lost as to which misunderstanding in which previous work the authors are referring to. If the "corrects the misunderstanding" statement is not addressed properly, it will put the submitted paper in a very bad light. You would be claiming to have corrected a misunderstanding that never even existed to begin with. Far worse than that would be to wrongly attribute that non-existent misunderstanding to a well-established prior art. It is very much needed that the authors cite exactly which sources make the said misunderstanding, but even that would not be enough to credit the MD3QN authors with "correcting a common misunderstanding" because many studies that the authors themselves have cited already use different metrics for convergence and in training loss. I am left puzzled as to why such a statement is needed in this paper after all.


Quality:
The paper was easy to read. There is considerable effort in founding a mathematical establishment before delving into the experiments, notations are consistent, as far as I can tell.
Some minor issues:
Section 2.1 is are sampled from the replay memory --> are sampled from the replay memory
Section 3.2 Crame loss --> Cramer loss
Section 5.1 The move has no effect --> The attempted move does not succeed (because, it's not that there is a move and then it has no effect. It's that there is no move in the first place.)


Clarity:
The sectioning feels a bit awkward. For example, what was meant to be the distinction between Section 2.1 and Section 3.1? At first, it looked like Section 2.1 is generally applicable to related research while Section 3.1 is specific to MD3QN, but that does not seem to be the case after continuing on to Section 3.2, where a concept (Bellman operator) much broader than this specific paper is introduced. The scopes of Sections 2 and 3 did not help in clarifying the notations, background, or the settings.

Again, referring to the "common misunderstanding" claim that the authors are making: authors write that they "clarify that the metric for proving convergence and the metric for training loss play different roles, and are not necessarily the same metric". No, they have not clarified that statement. They have cited a number of works that do not adhere to the "common misunderstanding", which then calls to question whether the said misunderstanding is indeed "common". It remains unclear i) who said that convergence metric and the training metric have to be the same, as well as ii) what mathematical tools the MD3QN authors have wielded to disprove the misconception.

The term "exclusive" is used without definition in Section 5.1. Is Figure 2 (a) supposed to show negative correlation?

Significance:
The key element of the paper is the correlatedness of the multiple rewards, and the two theorems do not show how the correlation plays a pivotal role in the design philosophy of MD3QN. Theorem 1 being the contraction mapping theorem for two joint distributions and Theorem 2 being the unbiasedness of the joint distribution estimator, neither of the two are, strictly speaking, contributions that are attributable to MD3QN, even if they show the "interesting" part when it comes to the correlation of the multiple rewards.

The experiments do not particularly outshine other submitted papers. For instance, the maze setting looks a bit artificial, and it would have been much more effective if accompanied with a real-world scenario, despite the fact that only some simplified analogues are tested.

Also, reward multiplicity poses some intriguing opportunities for rewards defined "outside the game". Intrinsic motivation papers geared towards better exploration or more diversity are examples of reward structure tuning. It would be enlightening to observe how MD3QN performs under those circumstances and whether the distributional RL advantage and/or the sample efficiency stand out in an ablation study.

The effectiveness of Figure 2 is unclear. The figures do not quantify how well MD3QN does, and they do not compare MD3QN to previous works. More importantly, why have the Atari Games been limited to just four games? For example, Atari Seaquest game grants points for destroying the sharks as well as some points for rescuing the divers. Why has this game been excluded?
limitations_and_societal_impact: The authors acknowledged that, despite MD3QN being a general framework, its actual application to goal-conditioned RL and successor states are left to future works. I believe the same kind of limitation acknowledgement is in order for i) testing with only four multiple-reward Atari games, ii) not clarifying a "common misunderstanding" in existing works despite claiming to have done so, and iii) the rather factitious nature of the maze experiments.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new learning algorithm for modeling the distribution of multi-dimensional reward in the reinforcement learning (RL) setting. It proposes a bellman evaluation operator for the distribution of multi-dimensional reward and shows its optimality. Moreover, a MMD based metric is proposed to measure the difference between two distributions and this metric is fit for the typical RL learning settings. The new learning method is tested and performs the best on one designed maze environment and multiple Atari games.
main_review: Originality:
The idea of the learning distribution of multi-dimensional reward in RL is straightforward. But building a learning framework upon that is complex and significant. 

Quality:

The learning framework is designed based on value-based reinforcement learning framework. The proof of the bellman operator is sound. Moreover, it uses different losses for proving and training. The MMD based training loss shows good performance in the experimental section.

Clarity:

The paper is well-written. And I do like the paper's structure.  It frames each main theorems and proposition, then explains the relation between them. Moreover, the authors also use many intuitions among them. Though the proof may not be fancy, however, the structure does make the understanding much easier.

Significance:

The problem to model the distribution of multi-dimensional reward is vital to push the performance of RL algorithms.

Questions:

1. I am wondering can you explain more about the kernel selection? Is any kernel that can make MMD a metric is a good kernel? 
2. It would be better if you can compare MMD with some other distance metric and show the superiority of MMD.

Minor:
The $r_t$ of equation (2) should be bolded.
limitations_and_societal_impact: N/A.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a method for distributional reinforcement in environments with multiple channels of reward. The joint distributional Bellman operator is introduced and its properties analysed, and an implementation based on MMD distances is described, with experiments carried on a small-scale gridworld and several Atari games.
main_review: __Post-discussion__

I thank the authors for their detailed rebuttal. I like the fundamental idea of the paper, although I think that the updates required to the paper are substantial enough that further review may be beneficial, particularly around the unbiasedness claims for MMD estimation, "common misunderstanding" remarks, and experimental details. Based on this, I would not recommend acceptance at this time, but I think that with further updates along these lines, this will be a strong paper.

__Main points__

There is an interesting central contribution in this paper, that I think should be of interest broadly to the RL community. Both distributional reinforcement learning and multi-reward channel RL are existing ideas, but their combination is an interesting idea, and the perspective the authors take in this paper is novel as far as I am aware. The exposition is generally clear, and the experimental results show interesting results from the newly-proposed approach. I would therefore like to see this paper accepted.

However, there are several parts of the paper that I think require clarification before the paper is ready for acceptance:

1. I found the discussion in the paragraphs at Lines 107 & 160 very unclear, for several reasons:
      * First, it is known that metrics for analysis and for projection play different roles in distributional reinforcement learning; see for example the analysis of Dabney et al. (2018b), in which a W_1 projection is used, while the contraction of the projected Bellman operator is performed under W_infty.
      * Second, the paragraph at Line 160 is unclear when mentioning "convergence" as to what it is referring to.  Repeated application of the distributional Bellman operator (or joint distributional Bellman operator) can be shown to be convergent using a variety of metrics, but this is distinct from understanding convergence of approximate algorithms. The example of C51 mentioned in this paragraph is useful to consider: global minimization of the expected KL loss leads to the same result as the application of the projected Bellman operator, this does not imply that using the KL loss leads to a convergent algorithm. Analysis of the projected operator, for example, would be required to show this. On this basis, I don't think the sentence "We illustrate that, once the convergence of the Bellman operator is proved under a certain metric, we can use other metrics than the metric for proving convergence, as training loss to approximate the Bellman operator." is true.
2. I think currently there is not enough discussion of the fact that multiple independent samples from T mu (s, a; \theta_i) are required to unbiasedly estimate gradients of the MMD^2 loss. This is an important fact, since multiple independent samples from a single given state-action pair are typically not available in online/replay-based reinforcement learning (e.g. Atari), which makes the sentence "the temporal difference loss MMD^2 can be optimized without bias by transition samples" potentially misleading.
3. Experiments: Overall these experiments nicely demonstrate the methods introduced by the paper, but there are some missing details that should be added. I couldn't find values for some hyperparameters such as the number of particles used, and in other cases I couldn't find details of how hyperparameter sweeps (e.g. on kernel bandwidth, learning rates, etc.) were carried out, or on how the Atari environments were selected, or their multichannel reward functions were designed (i.e. were other alternative rewards/games tried, did these work as well, etc.)

__Minor points__

Proposition 1: The metric of convergence should be specified; presumably it is \overline{d}_p.

There is some previous work on multi-channel distributional reinforcement learning that it would be good to comment on in the related work section, such as "Distributional Reward Decomposition for Reinforcement Learning" (NeurIPS, 2019).

The paper would naturally be stronger with an evaluation carried out on the full set of Atari games. However, I don't think this should be necessary for acceptance. The experiments that are provided in the paper serve as a useful proof-of-concept that the approach can learn meaningful joint distributions over rewards, and that this approach can have benefits in large-scale RL agents.

__Notation/typos__

E_\Sigma vs \mathbb{E}_\Sigma
"Guassian" -> "Gaussian"
limitations_and_societal_impact: See main review for comments on limitations. I did not see any discussion around societal impact, although this work is concerned with a core reinforcement learning algorithm, rather than applications.
review_title: null

Meta-review: The reviewers generally like the fundamental ideas of this paper.  However, there was a lot of confusion regarding the notation and the supposedly common misunderstandings of the community. The reviewers also raised some concerns about the experiments.  While the rebuttal clarified many concerns, the revisions required for the paper to be accepted are significant and therefore another round of reviews will be needed.  Hence, the authors are encouraged to revise the paper and to resubmit it at another venue.
Decision: Accept (Poster)