Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces GRSP (Generalizable Risk-Sensitive MARL algorithm), a method for training agents in mixed-games to converge to cooperative nash-equilibrium. The method's impact is in its applicability to much more relaxed situations than other MARL algorithms (usually Centralised Training Decentralised Execution). The key insights for this method are two fold: 

1) cooperative equilibrium are usually “high-risk” so simple reward maximisation rarely converges to this.

2) instead of requiring full access to co-players during training, simply being provided their action is suffice for opponent modelling. This is closer to independent learning than existing methods.
 
The method is then applied in two evaluation protocols: Evaluation Of Returns during training (can they converge to the desired equilibrium) and Generalisation to Novel Co-players at test-time. In both situations compelling arguments are made to show GRSP efficacy. Finally ablations are applied to distinguish how important the “high-risk” vs “opponent-modelling” insights and associated components are.

strengths_and_weaknesses: 
1.  Clean presentation
2.  Good experiment protocol
3. Very good breath of environments
4. Solid theoretical justification
5. Original work.
6. Clearly explains intuitions for method.


Weaknesses;
1) Very little time is given to explain training models, i’m unsure how sample efficient this method is compared to others. 
2) Generalisation Study Results could be better displayed. The opponent shaping literature has much nicer payoff graphs for showing how much an agent has cooperated with another -> these could help explain your findings.

3) I have little understanding of the distortion methods discussed would be useful. 

questions: 
I’d like more understanding of the training dynamic:


1. How does using the distribution RL effect sample efficiency?
2. How does risk-sensitive approaches effect exploration? 


limitations: None to mention :)
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper considers the problem of coordination in the more realistic general-sum game settings, by applying techniques in distributional reinforcement learning which allows the agents to leverage risk measures to guide the exploration toward risky cooperative strategies. Besides, an auxiliary opponent modeling task is used to improve the generalizability in face of non-cooperative players.
The method is compared with previous works in four tasks, and the empirical results demonstrate that the proposed method could improve the coordination ability of agents even without access to other agents' rewards, and quickly adapt to non-cooperative agents by fine-tuning the feature extractor.
strengths_and_weaknesses: Strengths:

1. This paper takes a look into a valuable point. Coordination is often studied in the fully cooperative setting, but general-sum game is more realistic. This paper considers an important but less studied problem. Also, generalization is a major but often overlooked issue in current MARL methods.
2. The proposed method applies the risk-seeking bonus in distributional RL to solve the exploration problem in multi-agent tasks, which is well-motivated and has the potential to improve the coordination ability.
3. The method used in this article combines risk-sensitive learning and agent modeling, both of which are promising techniques to handle the non-stationarity caused by environmental and agent level changes in multi-agent settings.

Weaknesses:

1. The most important part method is too short compared with the preliminaries and experiments. It is unclear to me how the feature extractor $\mathcal{E}$ and the decision maker $D$ work, more details are needed to introduce how your framework functions as a whole.
2. 1. There have been some works considering risk-sensitive or distributional Q-functions in MARL, such as [DFAC](https://arxiv.org/abs/2102.07936) and [D4PG](https://ieeexplore.ieee.org/document/9311945). This part of the method in this paper seems to be an implementation of risk-sensitive Q-function upon independent Q-learning (IQL), thus having rather restricted innovations.
3. The risk-seeking bonus results in optimistic behavior during training, this is beneficial when training with cooperative agents, but on the other side, which can also be exploited by non-cooperative agents and thus fail to learn a generalizable cooperative policy.
4. The paper claims to have generalization ability across different opponents during execution, but the way of achieving this requires long-term fine tuning during test phase (even just a part of the network will be tuned). I think such level of generalization should be expressed as "policy transferring" for clarity in order not to confuse others to think of zero-shot coordination (ZSC) or ad-hoc teamwork (AHT).

Minors:
1. The equation 7 makes it hard to understand what is the output of the function $D$. Besides, the parameters to be optimized on the LHS are $\phi^i$ and $\psi^i_s$, but $\phi^i$ disappears on the RHS.
2. The loss term $\mathcal{J}$ in equation 8 is not defined. I know that you have introduced it in the preliminaries, but please define it again under your framework according to your parameterization.

questions: 1. Since long-term fine tuning is needed at test phase to achieve generalization, why keep $\psi^i_a$ fixed and not tuning anymore?
2. In line 285, you claim that the cooperative agents and non-cooperative agents are trained by GRSP and MADDPG, respectively. Can you explain why these two different algorithms can produce these two desired strategies?
3. The proposed method seems to work well on the 5 by 5 grid-world environments, which left me wondering how well it performs on other popular Markov game environments. For example, social dilemma extends the prisoner's dilemma into multi-step settings, and the results on it can be more convincing.
4. Can you explain why your method can converge to the highest global reward in IPD. This is very unintuitive because it is not an equilibrium, implying that self-interest agents would deviate from it and try to defect at last. Your method does not take the rewards of other agents into consideration and the learned policy is self-interest even though you have incorporated risk-seeking bonus.
limitations: 1. Experiments in this work consists of "Comparison of Returns" only. More comprehensive demonstration experiments can better explain why the method works.
2. Generalization of the method used in this paper requires long-term fine tuning, while a lot of works can tackle ad-hoc teamwork efficiently right now. The application value of this work is limited.
3. During test time, the adaptation needs access to the full action history of all other agents, and fine-tune the feature extractor on it. This can be a very restrictive assumption and limits its use in practice.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a generalizable and sample-efficient algorithm for multi-agent coordination in decentralized general-sum games without any access to other agents’ rewards or observations.  It first learns the distributions over the return of individuals and estimates a dynamic risk-seeking bonus to encourage agents to discover risky coordination strategies. Then it proposes an auxiliary opponent modeling task so that agents can infer their opponents’ type and dynamically alter corresponding strategies during execution. Empirical studies show that the trained agents outperform other baseline methods via achieving mutual coordination during training and avoiding being exploited by non-cooperative opponents during execution.
strengths_and_weaknesses: Strengths: Risk-sensitive learning in zero-sum games is important topic in multi-agent systems. This paper applies risk-sensitive learning in zero-sum games. It uses the well-received distributional RL to learn return distributions and uses distribution distortion. The opponent learning was used to avoid overfitting opponents’ coordination strategies during training.

Weaknesses: 
1. This paper combines many well-studies methods and applies them in zero-sum game scenario. The methodological contributions are limited.

2. The evaluation scenarios are mainly on 2-agent cases, which is simple for didactic studies. However, it is hard to see if the proposed method can also perform well on complex 3+ agent scenarios. For example, social dilemma and zero-sum games scenarios in melting pot [1].

3. The writing is not good. Sec. 4.2 is not clear. It is not easy to follow. A figure showing the architecture is needed.

4. The Monster-hunt and Escalation scenarios were modified in [2], which is not cited in this paper.

[1] Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot

[2] DISCOVERING DIVERSE MULTI-AGENT STRATEGIC BEHAVIOR VIA REWARD RANDOMIZATION


questions: After reading the paper, I have some questions:

1. Sec. 4.2 is not clear. A figure showing the architecture is needed.

2. Why left truncated variance is used here? Why did the index start from M/2? 

3. Why c_tj is used here?  Why combing Eqn. 4 and Eqn. 5 in Eqn. 6 is reasonable? It lacks the motivation

4. LIAM is actually a single-agent RL method, how did you conduct the experiments in MARL scenarios?

5. Did you compare your method with LOLA [3]?

[3] Learning with Opponent-Learning Awareness

limitations: Please see the above comments.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper suggests learning risk-seeking policies in general-sum games in order to promote cooperation. The authors seek the strategy using distributional RL and risk-seeking weighting. Empirical evaluation on a range of domains shows that this algorithm finds the cooperative outcomes where other multi-agent RL methods do not, even though some of them focus on opponent modeling and coordination.
strengths_and_weaknesses: *Positives*
 
I like the idea of using risk-seeking behavior for creating coordination and I like the use of distributional RL for implementing it. I also like the empirical evaluation, which uses a sufficiently wide range of domains and clearly shows the superiority of the proposed approach over the chosen baselines.

*Major concerns*

My main concerns are lack of theoretical grounding and depth of understanding conveyed by the experimental results.

Learning in games is a relatively old and well studied field with many solution concepts and algorithms to achieve them. This paper presents an intuitive solution to promote cooperation, but does not analyze at all what the proposed algorithm is supposed to converge to theoretically. Are there any guarantees on the behavior of the algorithm even in simple matrix games? Is there some relation between the parameters for risk seeking and the gap between cooperative and non-cooperative equilibria of the game? Does the algorithm converge to correlated equilibria as many other learning dynamics under some conditions or even that is not guaranteed?

I believe that theoretical understanding of algorithms is important, but not absolutely necessary for a good paper, if the empirical evaluation chooses the right baselines and provides an in-depth analysis of why and when the proposed algorithm is superior. Since there are many papers dealing with trying to learn cooperative outcomes in general sum games (such as [A,B]) I consider the choice of baselines insufficient. Furthermore, the experiments do not provide any insights into when and why the proposed method performs better. The ablation study is definitely a step in the right direction, but I still do not know when to expect the algorithm to work well and when not.

[A] Littman ML. Friend-or-foe Q-learning in general-sum games. InICML 2001 Jun 28 (Vol. 1, pp. 322-328).
[B] Crandall JW, Goodrich MA. Learning to compete, compromise, and cooperate in repeated general-sum games. InProceedings of the 22nd international conference on Machine learning 2005 Aug 7 (pp. 161-168).

To summarize, the overall idea of the paper is interesting to me, but the thoroughness of setting in the related work and the depth of the analysis is slightly below the bar I would expect from a NeurIPS publication.

*Minor suggestions*

Lines 31-32 claim that training a population of strategy profiles is infeasible in real world settings, which does not seem to be true with algorithms, such as AplhaStar.

It is not clear what exactly are the error areas in the graphs. Confidence intervals? Variance? STD?

Some formulations are hard to follow, such as L196-197, L236, 
questions: Why did you not compare to the existing algorithms for learning cooperative outcomes in general sum games?

What are the main reasons why the proposed algorithm finds the cooperation and the existing ones fain in some of the specific domains?
limitations: There is not obvious potential negative societal impact. The limitations are IMO not explored sufficiently well. It is not clear when the proposed algorithm starts failing.
review_title: null

Meta-review: metareview: The paper presents a novel approach for improving coordination in general-sum games by using risk-sensitive policies based on distributional RL. While the idea is promising, there are significant questions about the paper. 

For example, there is concern about the lack of theoretical guarantees and intuition about when the approach will work well. 

There should also be a more extensive discussion of related work. For example, distributional and risk-sensitive RL have been used in multi-agent RL but more should be said about how the proposed method differs and why they can't be included in the experiments (e.g., the decentralized training methods should have no problem running in the general sum case). 

More extensive experiments would also be helpful. Additional domains and baselines would more clearly show the benefits of the method (and when it could potentially fail). 
confidence: Less certain
Decision: Reject