Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present RMIX, which is a value-based MARL method incorporating a risk evaluator to prevent overly optimistic actions in teamwork among agents. In particular, the CVaR metric is chosen for its computational advantage and theoretically sound foundation. Experiments in the SMAC environment show better win rates than do recent baselines.
main_review: Originality: Contribution of the work is clear. There is a well-defined niche in the literature among MARL and safety RL. Authors appropriately position RMIX in the intersection of value-based MARL and risk-sensitive RL. The non-stationarity challenge addressed in the paper is unique to the multi-agent setting; a simple application of the single-agent CVaR to the multi-agent setting would not work. However, the said challenge are not specific to RMIX, and thus the (partial) solutions (utilizing recurrent models and CTDE) also borrow from existing literature. It would have been even better if the non-trivialities arising from the multi-agent nature of the risk-sensitive RL problem were highlighted.

Quality: The paper was easy to follow. Figure 2 delivers the overall structure clearly. Evaluation includes some of the more recent works and a thorough ablation study and discussion, without which the complexity of the SMAC tasks might have clouded the effectiveness of the risk sensitivity.

Clarity: The theory that supports the work feels rather fragmented. For example, the appendix states Proposition 2 as key to understanding and proving Theorem 1. What compelled the authors to include contraction mapping theorem, which in my opinion is the relatively "standard" one, and leave out Proposition 2 instead? Please consider unifying the theory overarching the work.

Significance: RMIX is being compared with the baselines that it should be compared with, plus some other family of MARL algorithms as well. It is shown to excel in most scenarios. A brief explanation on how QPLEX beat RMIX in the 8m_vs_9m task would be informative. In fact, most SMAC tasks are rather complex, so simpler didactic tasks would probably better illustrate the effectiveness of RMIX. For instance, if risk-unaware algorithms (such as QPLEX) outperformed better than RMIX, would it be correct to say that the 8m_vs_9m task carries little to no risk? There is some guesswork (on my part) involved when attempting to decide whether a situation carries much risk or not. Simpler, albeit toy-ish, tasks may be built to better compare two distinctively constructed scenarios of varying risk levels.
limitations_and_societal_impact: I would up my rating if the authors would kindly come up with some toy examples (one low-risk setting showing little to no performance difference between RMIX and risk-unaware algorithms and one high-risk setting showing larger performance difference between them). Something like the CliffWalk example illustrating the difference between Q-learning and SARSA would do. Or, any equivalently valid explanation about interpreting the intrinsic risk levels of a task and the corresponding effects in RMIX would be much appreciated.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This is an interesting article introducing risk-averse distributional RL ideas into the multi-agent setting in a new and it seems useful way. It builds on methods like Q-Mix and VDN but replaces normal returns with CVar_alpha, the expected return for the worst alpha-percentile outcomes. They also introduces a method for picking alpha different for different timesteps and different agents.
main_review: This is an interesting article introducing risk-averse distributional RL ideas into the multi-agent setting in a new and it seems useful way. It builds on methods like Q-Mix and VDN but replaces normal returns with CVar_alpha, the expected return for the worst alpha-percentile outcomes. They also introduces a method for picking alpha different for different timesteps and different agents, before mixing them into the total, which slightly confuses me and I would like to see more conceptual discussion on what that choice is really optimized for and why it is a good idea, though their ablation study shows that it does better than any fixed choice so something smart is probably done but I have not really understood the what and why it is good.

The empirical evaluation first shows superior performance in the SCII micromanagement tasks compared to start-of-the-art algorithms which is interesting, while reminiscent of what was seen with risk-averse distributional rl for the atari suite of tasks when adding this to DQN-based agents. There have been various hypotheses why this was the case including that it just usefully takes more information into account akin to auxiliary tasks. The author goes on to show the usefulness for the tasks with extra inserted noise, which is more directly testing the aimed for robustness, though Q-mix was doing too badly in this either (Fig 7).

It seems like a new useful iteration of MARL algorithms for the team reward setting, VDN -> Q-Mix -> R-Mix, where they also show that adding it straighter to VDN giving RDN is also clearly useful.

I would mainly like to better understand the “prediction” of alpha better the thinking behind how this is done. 

The contribution builds on things already developed for the single agent setting, but requires some amount of innovation for the marl setting.

limitations_and_societal_impact: ok
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a multi-agent RL algorithm based on CVaR. In the proposed algorithm, the return distribution is learned as in QR-DQN, and CVaR is estimated using the trained model. The centralized training for multi-agent settings is based on QMIX. In the proposed algorithm, Q-values in QMIX is replaced with CVaR.  The proposed method is evaluated with several scenarios in StarCraftII, and the proposed methods outperformed baseline methods.
main_review: Strong points:
-	The proposed method was rigorously evaluated and outperformed baseline methods.

Weak points:
-	The contribution seems incremental: Risk-aware RL algorithm based on QR-DQN and CVaR was proposed in [1], although it is not cited. The proposed algorithm RMIX is basically the combination of QMIX and Q2-Opt proposed in [1].

I appreciate the authors' efforts for performing comparison with baseline methods and ablation study. On the other hand, the use of the QR-DQN for computing CVaR is not novel.  In the context of robotic manipulation, Bodnar et al. proposed an algorithm called Q2-Opt, which is a risk-aware RL algorithm based on QR-DQN and CVaR[1]. Adding [1] to the reference is recommended. The proposed algorithm RMIX seems the combination of QMIX and Q2-Opt. Namely, the proposed algorithm can be obtained by replacing Q values in QMIX with the CVaR in Q2-Opt. 

However, the risk predictor is not proposed in Q2-Opt, and it is the contribution of this paper. The proposed algorithm is rigorously evaluated and outperformed baseline methods. If Q2-Opt paper [1] is added to the related work and the authors clearly state that a risk-aware RL algorithm based on QR-DQN and CVaR has been proposed in [1], I think the paper can meet the standard of NeurIPS.

[1] Cristian Bodnar, Adrian Li, Karol Hausman, Peter Pastor, Mrinal Kalakrishnan.  Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping. R:SS, 2020.

=== comments after the author response ===

I went through other reviewers' comments and the author response. I think that the authors addressed the concerns raised by reviewers.
I increase the score from 6 to 7.
limitations_and_societal_impact: Although the policies are trained in a risk-sensitive manner, the proposed algorithm does not guarantee the safety during the training phase. I recommend authors to clarify this point in the paper in order not to confuse/mislead audiences, who are not machine learning experts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work takes risk-sensitive learning to cooperative multiagent learning. It first, learns the conditional value at risk of the return distribution then, a second component is a risk-level predictor which dynamically adapts the risk. Lastly, the paper builds on QMIX to deal with the credit assignment problem since there' is a single global reward.

The problem the authors are tackling are very hard and I see the benefits of their approach. 
I think the experimental results are interesting and informative, however the paper needs some clarifications to be more understandable to a bigger audience.


main_review: Originality
Risk-sensitive approaches have been proposed mainly in single-agent RL. Within MARL, the area seems less explored.

Clarity
The paper has a lot of content in fact, I think removing some parts might help the readability, I have some small suggestions to improve.

- The theoretical result is interesting but I don't see it as the major contribution, actually I think that section can be shortened.
- You added the pseudocode of the algorithm to the main paper, however, I suggest to add accompanying text to it.
- I know about the lazy agent issue but I'm not sure I understand your comment: "we train RMIX in an end-to-end manner where each agent shares a single agent network and a risk predictor network to solve the lazy-agent issue".
- I'm not sureI agree with:  "the overall structure of QPLEX is more complex and benefits much from increased model capacities than RMIX."
- Figure 7 is quite small to see the differences
- Is Figure 4 really necessary to have?
- Extend the caption in Figure 3

Quality

I see the benefits, but I would like to know what are the disadvantages of modeling the return distribution with a mixture of Dirac delta functions? Is this the same approach as Distributional RL with implicit quantile networks ?

Why there is a need to compute the discrepancy wrt to the historical distribution? (Section 3.2)
- Why the historical information is useful?


Significance

- The experimental results look solid and the ablation results show the benefits of the proposed components.

Minor:

"discreet decision"
"It gains popularity in various engineering and finance applications. " ??
"However, few works have been done in MARL and it cannot be easily extended." ??

limitations_and_societal_impact: No discussion from the authors.
review_title: null

Meta-review: All reviewers felt the author rebuttals satisfactorily addressed their concerns and three reviewers increased their scores as a result.  Overall, there was a unanimous decision to accept.  The authors are strongly encouraged to incorporate review comments and rebuttal discussion into their final revision.
Decision: Accept (Poster)