Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Incentivizes agents to be diverse from each other in cooperative multi-agent tasks by getting them to maximize the mutual information between their trajectories and their ID. Breaks this down into action-diversity (maximizing the KL divergence between each agent’s action distribution & the group average), as well as observation-diversity (using a variational approximation of I(o_t;ID|tau,a_t)). Also propose a new architecture that allows sharing of information between agents, while retaining some agent-specific parameters. Thorough evaluation with 6 state-of-the-art multi-agent baselines show the method gives consistently much higher performance than prior techniques at both Google Research Football and Starcraft. Great analysis of the learned behaviors and why diversity helps in both environments, cool ablation studies.

main_review: **Quality:**
- The paper proposes a well-motivated and effective technique for improving cooperation and learning in multi-agent RL (MARL). It combines a (somewhat) novel diversity objective with a novel architecture that enables sharing information between agents while allowing for agent-specific policies. 
-  The experimental evaluation is extremely thorough, and includes 6 baseline methods spanning state-of-the-art multi-agent techniques. The performance gains over these methods are convincing, across 2 challenging MARL environments.
- The paper also provides experiments in a 3rd environment which provide insight into the method.
- Thorough ablation studies analyze exactly which components of the algorithm contribute to performance. Interestingly, this differs between the multi-agent environments under consideration. 

**Significance:**
- Given the magnitude of the performance between this technique and several SOTA baselines, this paper is definitely of interest to the MARL community
- The progress on GRF and the analysis of how diversity contributes to this is also compelling. 

**Originality/Novelty:**
- The biggest issue with the paper is novelty. Similar diversity objectives have been proposed in DIAYN (https://arxiv.org/abs/1802.06070), and in the multi-agent case with EOI (https://arxiv.org/pdf/2006.05842.pdf) and CMRL (https://arxiv.org/abs/1903.02710). However, the paper benchmarks against EOI and shows convincing performance improvements above this method. DIAYN is appropriately cited. 

**Clarity:**
- The paper is well motivated, and the introduction is clear and compelling. 
- Intro figure clearly sets up the issue with learning shared policies and why diversity is important. 
- The derivation of the method in section 3.1 is very clear and crisp, especially highlighting each of the three terms in equation 2. 
- However, it would improve clarity to briefly describe (even in 1-2 sentences) how p(a_t|\tau_t) and p(o_{t+1}|\tau_t,a_t) are estimated in the main text. 
- The L1 regularization could be more clearly motivated. Why L1 not L2? 
- Figure 3 would be much more compelling if it included a heatmap for one of the baseline methods (as in the EOI paper). This helps make the case for the importance of explicitly incentivizing diversity, rather than just optimizing shared reward. 
- Lines 255-260 talk about how during phase 2 of learning in GRF, the agents "lose their individuality" to focus on scoring. And yet even in this phase, the performance of CDS is significantly higher than the baselines (~25% vs~10%). If the agents have lost their individuality, what explains this performance gap?

**Detailed feedback/suggestions:**
- Lines 54-57 in the intro could be made more clear/crisp. 
- Although the results of the paper are convincing, calling your own results "extraordinary" (line 62) might be considered a bit grandiose.
- "agents need to behave differently to /highlight/ themselves from others" -> not proper grammar. "distinguish themselves from others" would be more appropriate 
- When referring to QPLEX in line 153 you should include the citation of the QPLEX paper.
- The idea of rewarding agents for mutual influence was first proposed in https://arxiv.org/abs/1810.08647. It would be appropriate to cite this work in lines 208-209 of the related work when introducing EITI and EDTI.
limitations_and_societal_impact: The introduction clearly identifies real-world problems for which this technique could provide social benefit, but the paper does not include a discussion of limitations and potential negative impact in the main text. It might be worth mentioning the limitation that because the method relies on centralized training, it is not applicable to multi-agent problems in which some of the other agents are humans, and thus are outside of the control of the learning algorithm. E.g. it is not applicable to autonomous driving.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a CDS method that adaptively trades off diversity and sharing in multi-agent reinforcement learning. The authors introduce an information-theoretical objective to encourage diversity and decompose individual Q-functions as the sum of shared and non-shared local Q-functions to boost shared knowledge usage whenever possible. Visualizations show that the balance between identity-aware diversity and homogeneity promotes sophisticated strategies.
main_review: Originality: It is novel and interesting to introduce diversity into the common-used parameter sharing technique. The method is new and differs from previous works in this area. The related works are adequately cited and reasonably compared.

Quality: I have not checked all details, but the paper appears to be technically sound. The experimental results illustrate the method’s superiority to some extent, but some visualizations may not support some of the ideas in the paper very well.

Clarity: This paper is clearly written and well organized. It provides the reader with some essential information to reproduce its results.

Significance: This submission introduces diversity into the parameter sharing technique, and to some extent, advances the field of multi-agent systems. Other researchers are likely to use the ideas in this paper.
limitations_and_societal_impact: The authors do not address the limitations of their work, but I have some questions and suggestions about this paper: 

1. The essence of exploration should be for the team as a whole to interact thoroughly with the environment, so why should the agents within a team have to behave differently? How does the algorithm perform in scenarios that require multiple agents to work together on a single task (for instance, a door that requires two agents to open together)? 

2. Figure 3 is attractive and illustrates some of the intrinsic mechanics and advancements of the method, but I did not understand the role of independent Q very well with this example. Is the passage of agents through different paths a reflection of different strategies? Passing diverse paths can also be summarized macroscopically as the same strategy. I suggest designing more complex scenarios, such as some agents passing through paths while others need to stay in the central room to accomplish the task together. These may be more like manifestations of different strategies. 

3. Why were the agents' identifications not included when calculating the local Q-functions in the experiments? 

4. Figure 5 shows why the CDS method works, illustrating the process of training in which the agents first personalize, then assimilate, and finally find a balance between diversity and homogeneity. This visualization, although interesting, does not seem to be a convincing illustration of this method. Because this process can also be explained by the fact that the agents first focus on self-exploration and then learn to cooperate, which is the same with the vast majority of MARL approaches. An ablation experiment is a more proper way to prove this. I suggest removing modules that encourage individualization and assimilation, respectively, and visualize how these cases perform at the same number of training steps. These experiments can show that the CDS is effective because of its balance between diversity and homogeneity.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper focuses on the centralized training with decentralized execution (CTDE) setting within cooperative multi agent reinforcement learning (MARL). The authors observe that in some cases training with a joint Q function can lead to myopic agents that do not effectively coordinate with one another. The authors introduce CDS, a new method that factorizes the Q function into shared and individual components, with an additional loss term maximizing the mutual information between the individual Q-function and the agent ID. They demonstrate the effectiveness of the approach on a toy problem and then outperform a set of strong baselines on two challenging benchmarks. Overall the work seems like a solid contribution to an active area of research. 

I vote to weak accept the paper, with a possible increase if my questions below are answered in a satisfactory manner. The only reason why this paper is not an accept is a general impression that the authors prioritized SoTA over transparency. There is no discussion of how hyperparameters were tuned, very limited discussion of limitations and no discussion of how environments were chosen. Overall it leaves the impression that maybe the method doesn't generalize to other environments, and is brittle to hyperparameters.
main_review: Strengths of the paper:

- It is well-written and motivated. The ideas fit well with the current literature.

- The idea of optimizing mutual information between agent ID and the trajectory is intuitive, and the breakdown into action and observation diversity is well done.

- The toy example is well presented and explained. In particular, the result that the agents follow their local Q-functions along the paths then use the shared one for the rooms is interesting and certainly demonstrates the method effectively.

- The experimental results are strong, outperforming recent (and relevant) baselines, and on both GRF and SMAC (vs. usually just SMAC for related works).

Weaknesses/Questions: 

1) How many seeds were used for the pacman task? What does the plot actually show? Mean/median? What are the error bars?

2) The paper introduces four new hyperparameters, all of which were tuned for different environments. There are no ablations showing the sensitivity of the performance to these hyperparameters. How were they chosen? How brittle is the algorithm to these changes? 

3) It seems for SMAC the parameters were individually tuned per task, was there not a single parameter configuration that could perform well on all three tasks? This is a separate question from above because it also highlights another issue - how does CDS perform on other tasks? The authors mention that it is not expensive to run their algorithm, yet only use four StarCraft maps, which is fewer than other works. Then for GRF, one of the three environments chosen was hand designed, presumably to favor this specific approach. 

4) For GRF, how were the number of timesteps chosen? It looks to be arbitrary, and makes it appear that the other methods may outperform later on. 

5) The language is, at times, a little overzealous. In particular, referring to the experimental results as "extraordinary" (l.62). I don't think this is needed, *the results are strong as they are without sensationalizing them*.

6) The limitations of the method are barely mentioned. There is one paragraph at the end of the appendix, and 50% of it is reiterating the strength of the approach. It would be useful to understand when this fails. I think this gives the general impression that the authors are trying to only show strengths, maybe through fear of reviewers criticising weaknesses. In my opinion it would actually be better for the paper to show when it does not work, so that future research can be done to improve upon it.

7)  The Appendix shows the objective is actually using the mean over IDs, which the authors say themselves should not be accurate, so it is possible they tuned hyperparameters to a noisy objective and picked the best result. 

Minor issues and typos (did not impact score):

- l.16 "provides a promise" → "provides promise"

- l.63 "achieve" → "achieves"

- l. 64 "talent" → "talented"

- l.105 "between individual" → "between an individual"

- l. 112 "about agent's" → "about an agent's"

- l.335 "into" → "in"
limitations_and_societal_impact: Limitations should be discussed more adequately in the main body of the paper. Societal impact is likely minimal.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper deals with the problem of imposing policy diversity along the different agents of a MARL problem. To do so, the paper proposes to introduce a novel objective to maximize the mutual information between the agents identities and trajectories. Additionally, the paper proposes a decoupled Q function architecture to represent shared and non-shared goals.
main_review: The paper is well written and presents both quantitative and qualitative metrics to show the performance of the proposed method. Additionally, the paper introduces an insightful ablation study to evaluate the effect of the proposed elements in the learning performance.

However, I think there are few points that should be clarified:

- In Line 120, the paper claims the intractability of the Term 3, given an integral term, but the Term 3 is not explicitly showing any integral term in Equation 2.

- Equation 5, presents the gradient over a Loss function, but this loss function has not been previously introduced properly.

- Even if the paper proposes a decoupled architecture between shared and non-shared Q functions, the learning is done jointly. The paper claims that L1 regularization over the independent Q function, allows the learning to be balanced between the shared and non-shared terms. Can the authors provide any theoretical guarantee of that claim or only experimental?
limitations_and_societal_impact: Not applicable.
review_title: null

Meta-review: The authors present a novel approach to cooperative multi-agent reinforcement learning that focuses on balancing diverse behaviors with shared information. The approach incorporates several new insights (mutual information losses to encourage action and observation diversity, shared and local architecture components and a regularization term) and conceptually and empirically demonstrate how each contributes to the algorithms performance. Empirical validation is extensive and includes challenging environments, often showing dramatic performance improvements compared to baselines. Detailed ablations and analysis provide further insights into how the proposed approach works.

Initial reviews were positive, highlighting the novel insights and strong empirical results achieved in the paper. The paper was assessed as well written and largely clear. Open questions included experimental details (e.g., the procedure for hyper-parameter tuning and impact of hyperparameters on performance), positioning of the paper relative to related work (e.g., DIAYN, and the general question of how this work relates to previous approaches that encourage diversity), and conceptual questions (e.g., potential conflicts between diverse behavior and the need for coordinated exploration, the role of L1 regularization over the independent Q functions).

The authors addressed reviewer questions and suggestions in detail, providing clarifications as well as additional empirical results. As a result, several reviewers increased their scores and all indicated that their concerns and questions had been addressed. The AC agrees with this consensus. 
Decision: Accept (Poster)