Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper connects policy gradient method in joint policy optimization of multi-agent RL to MI maximization framework and shows that joint policy optimization essentially leads to MI maximization. Thus joint policy optimization produces skills that are transferable across agents but it costs in diversity and performance of agents. To address this issue, the authors propose a novel minimax algorithm called M&M that is able to learn agents that specialize at different skill via conditioning on the stable generalist policies obtained through MI regularization. Experimental results demonstrate the similarity between the agents learned through joint policy optimization and how specialist agents learned by their algorithm shift away from Generalist’s behavior to specialize in their own specific behaviors. They also presented a qualitative comparison of their algorithm with prior works like NeuPL.
strength_and_weaknesses: - The paper is well motivated and seeks to learn agents with diversified behavior pattern in a multi-agent policy optimization framework.
    
- The experimental section demonstrated the advantage of their algorithm over prior works.
    
- However, the technical sections 2.3 and 3 lacks clarity and has some issues(see comments in next section for more details).
clarity,_quality,_novelty_and_reproducibility: - The technical section of the paper is a bit difficult to read particularly because the terms used are not well defined and some notations are ambiguous. Please see the list below :
    
    - Is $\{\Pi^g_{\theta^\*}\}_g$ a solution policy set obtained from $\epsilon$\-Nash joint policy optimization or is it the optimization problem itself, first line of section 3.2?
        
    - The non-Markov two-player game setting is not well specified - how is agents utility $U$ different from reward $R$ ? What does weighted edges represent and what constitutes an adversarial interactions discussed just above section 3.2?
        
    - $[0:1]$ is a python notation; better to use $\gamma \in [0,1)$.
        
    - Multiplying a policy $\Pi^g_{\theta^\*}$ which is a function of $\mathcal O \times \mathcal A$, to an expectation term as in equation 7 is ambiguous.
        
    
    - How does equation $(3)$/(8) reduces to the form in equation $(4)$/(9) respectively? how did the $Q$ and $V$ function vanish? The statement after eqn (3) “With T steps of optimization, the integration of gradients results to a long-term behavioral change of mutual information maximization” ([pdf](zotero://open-pdf/library/items/ER8E6IYM?page=3)) is also not very clear.
        
    - This statement is not clear “The formulation can be extended to any two-player exchange of (X,Y) of N distinct agents {i,ii,..N}. MI is expanded as Interaction Information”.
summary_of_the_review: The paper is well motivated and the proposed M&M algorithm appears quite novel. However, technical section 2.3 and section 3 lacks clarity and should be improved further.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper offers a new perspective to solve the behavior homogeneity issue in current joint policy optimization approaches. The paper points out the connections between policy gradient in multi-agent joint policy optimization and mutual information (MI) maximization followed by the drawbacks of MI maximization. The paper proposes a minimax formulation of MI, M&M, to improve agents’ diversity and specialization while maintaining the population stability.
strength_and_weaknesses: Strengths: The key idea is straightforward and easy to understand. The paper is backed by a succinct theoretical analysis when it talks about the underlying relationship between joint policy gradient and MI maximization. 

Weaknesses: Even though the method is well-motivated, the proposed method is not easy to follow, with terms and symbols left undefined. Moreover, the experiment section needs more work. I would like to see a more thorough evaluation and clearer explanation of the M&M results compared to NeuPL, such as the root cause of Agent A’s significant improvement. In addition, it would be good to show how M&M works in games with more cooperation among agents like MuJoCo Football mentioned in the NeuPL paper.

clarity,_quality,_novelty_and_reproducibility: Clarity and quality: 
The motivation and related work sections are well-written, but the writing of the proposed method is unclear. There are several typos in the paper.

Novelty:
In my perspective, the approach has a certain novelty in it.

Reproducibility:
The source code is not provided and the way the method is introduced makes it challenging to reproduce the results solely based on the paper.
summary_of_the_review: This paper purposes a minimax MI approach to mitigate the behavior homogeneity issue among agents in joint policy optimization. The idea proposed in the paper could be inspiring to the community, but I would expect a better presentation of the proposed method and a more comprehensive evaluation and discussion.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: As implied by the author, multi-agent systems can be studied in the sense of social behavior. Where each agent needs to be an expert in a particular task, they should share a common understanding of the whole space with other agents. To this end, a wide variety of methods are proposed to fulfill both of these criteria. One can think of that common understanding/knowledge as the ability to generalize from bottom to top. That means when an agent is an expert in one task, it can easily be extended to contribute to the solution of a larger task. Since an intelligent agent can be showcased by using a single policy function, many of the existing methods in this matter focus on relations between different agents(a.k.a policies). This paper generally centers around proposing an optimization framework in multi-agent systems by using Reinforcement Learning notions and classic game theory. Specifically, by using the Mutual Information concept, which conveys the level of shared understanding of the same random variable, a gap will rise between generalization and expertise. Having grounds of knowledge about almost everything will, obviously, be counterproductive in the case of getting narrow in a specific task. Therefore, the author tries to employ classic game theory to balance this dilemma.
strength_and_weaknesses: The most important merit of this paper is the contribution toward an equilibrium between generalization and expertise in multi-agent systems without having any assumptions about cooperation or competitive setup. Once the initialization happens, the proposed algorithm will basically follow the min-max strategy to reach an equilibrium satisfying both generalization and expertise. 

**Strengths**
* No assumption on the cooperation/competition.
* Practical on policy optimization that provides better performance compared to value-based methods.
* Intuitive representation of results using radial charts.

**Weaknesses**
* The contributions are not highlighted enough in the paper.
* Although it cannot be a direct weakness, we should try to avoid using KL in policy optimization due to complexity.
* Background and Results are limited to two player games.
* The results are not compared to any state-of-the-art method.
* The algorithm is not discussed in details.

clarity,_quality,_novelty_and_reproducibility: * There are some minor issues (see weaknesses), in terms of writing that needs to be further improved.
* The main idea is only marginally significant and not well-supported by experiments and also lack of a comparison with state-of-the-art models.
summary_of_the_review: * As stated before, the contribution is not perfectly annotated in the paper. The author pretty much employs the related techniques in the proposed algorithm and does not clearly claim the novelty.
* The proposed method closes an essential gap in multi-agent systems, but still the generalist seems to take so much time to converge
* The overall quality of paper sounds acceptable, but slight improvements on the presentation, specifically, on the contribution will make it more understandable
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper deal with the problem of mutii-agent specialization: which is the issue related to the developing of a single general learning algorithms for a relatively diverse population. Typically each agent in a multi-agent paradigm is trained using separate behavioral policies. But this has now led way to jointly optimized learning solutions, 

The authors here report that the joint policy is related to a simple information theoretic measure: mutual information (MI) among agents. This solution however leads to a bottleneck, which is that individual agents do not specialize. To circumvent this issue, the authors derive a minimax MI solution (M&M) and show that the agent trained using this method not are better specialisists but do better on competitive play metrics. 

strength_and_weaknesses: Pros:
1. This study is a valiant attempt to demystify multi-agent learning in current RL models
2. The proposed solution is intuitively easy to understand.

Cons:
1. The paper is really hard to follow and there are several typos.
2. It is hard to know if the experiments truly generalize. Besides the theoretical argument, the experiments were performed on a single game: Naruto Mobile. While the authors report considerable improvement there is considerable work to be done. 
3. There is an obvious link between stability (specilization) and plasticity (generalization to new tests). The authors have not discussed this theme. It is not always desirable for agents in a multi-agent game to specialize and I wonder how the authors think about this in context of their work. I worry that M&M may NOT in fact the most desirable in most case of AI based training methods. 
4. The conclusion and discussion sections were notably sparse on details. 

clarity,_quality,_novelty_and_reproducibility: The paper is not easy to follow (even for me who is fairly well integrated in this field). The paper has several typos which I have tried to overlook in this review, but urge the reviewers to check carefully.  I did not find any mention about the code. So it is unclear whether the solution is reproducible. 
summary_of_the_review: The paper makes an interesting observation regarding the training of multi agent models. The proposed solution however seems rather extreme and the context under which this would be relevant in the training of other multi-agent models remains to be seen. The current results on a single benchmark are well taken.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Given Joint policy optimization encourages knowledge sharing and social behaviors, this paper connects the dots between policy gradient formulation of joint policy optimization and MI maximization. Authors have proposed a minimax formulation of MI (M&M) that enables agents specialization with stable regularization. It’s pointed as a good sweet spot to balance the tradeoffs with MI maximization/minimization alone - M&M optimizes agent specialization while regularizing the learning instability with conditional MI minimization. Authors have included a theoretical and experimental analysis on the social behavioral change of agent population and demonstrate that M&M benefits in population social diversity and specialists competitive play.
strength_and_weaknesses: Authors have well drafted the related work/background section, clearly explaining past work in competitive & cooperative behavioral learning. Solid motivation pointing out the trade offs with MI maximization/minimization alone. Gains in empirical evaluation are significant - performance gain from agent specification is significant(few agents beating sota by >20 points interms of winrate%). Also, 30 % reduction in population performance deviation, as authors pointed out, shows that equity of a population can be improved when individual agents specialize.

Some sub-sections under experiment aren’t clear/seek more details. In section 4.5, Winrate % is not consistent across all agents, specifically seems to be more beneficial for weaker agents. For example, agent T, agent D see no/little performance gain with M&M specialization. Also, training efficiency and competitiveness is not clear and require more details - for ex, why are benchmarks limited/specific to 2 agents (K and M)? Lastly, what would the impact of number of unique agents on joint policy optimization benefits (in these experiments, 16 unique agents of Naruto Mobile characters were chosen).
clarity,_quality,_novelty_and_reproducibility: Clarity: Motivation and methodology part of this paper is very clear. I appreciate the authors efforts in well drafting the related work section. I wish there was a bit more discussion for qualitative analysis in experiments section. 

Quality/Novelty: minimax formulation of mutual information for agents specialization sound to be novel in multi-agent learning settings. I wish there was more space in the main paper to expand on experimental details for some of the sections and qualitative analysis, but the paper is quite content-full as is.

summary_of_the_review: The motivation of this paper is technically sound and empirical evaluations shown significant performance gains. This paper features minimax formulation of MI (M&M) that enables agents specialization with stable regularization. experimental results have shown significant benefits in population social diversity and specialists competitive play. However, some sub-sections of the experimental section aren't clear/seek more details. It would have been further helpful if authors included qualitative analysis. For more details, please refer to the strengths and weaknesses part of the review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers a multi-agent reinforcement learning setup and takes an information-theoretic approach to the joint policy gradient method. The first observation is that the joint policy gradient maximizes the mutual information (MI) between the agents -- an observation that follows by definition of MI. The authors extract an important insight from the observation, namely, that the joint policy gradient is in fact maximizing the stability of social behavior, and may compromise the individual performances of the agents. In order to address this challenge, they propose Minimax MI algorithm, which specializes for individual goals by maximizing individual performance conditioned on the socially stable behaviors. The authors provide experimental evaluations of the proposed method, showcasing its ability to specialize. 
strength_and_weaknesses: ### Strengths

1. The paper considers an interesting problem which is relevant to the MARL community. 
2. The key observation that the authors make provides an interesting insight into the intuition behind joint policy gradient, albeit it being a straightforward derivation.
3. The proposed algorithm seems to address the specialization issue of joint policy gradient. 
4. Experimental evaluation of the method is present, and seems to prove the effectiveness of the algorithm with respect to vanilla joint policy gradient. 

### Weaknesses

The most major weakness of the paper is that it is poorly written:
 1.  There are typos everywhere, from abstract to conclusion.

 2.  Definitions of mathematical objects are practically non-existent:
     - x and y, or $\Pi$, in the beginning of Section 2.3; 
     - $\psi^k_i$ and $\epsilon$-Nash in Section 3.2. 
     - $S: O \times O$ is said to be a function, but I do not understand what the range of the function is.

I understand that most things are left implicit, but this practice makes the paper prone to errors, since the derivations are not based on well-defined objects, and makes the paper very hard to read. 

 3. The paper could benefit from additional remarks or further elaborations:
    - There should be an explicit formulation of the graph solver from (Chen et al. 2022) and its relevance to the problem considered, not just a reference. 
    - What is the meaning of $Self-play$ function in the pseudocode?
    - What is the meaning of $D_{kl}(gradient_{step}(\pi_{\psi^k_\tau}))$?
    - What is the meaning of $Eval(\pi_{\psi^k_\tau}, \{ \Pi_g\}^N_{g=I})$, and why is the first index $I$ in these definitions?
clarity,_quality,_novelty_and_reproducibility: ### Clarity and Quality

As mentioned in the previous section, the paper suffers heavily from a low clarity. A thorough rewriting might be necessary in order to make the paper conference material. The considered problem and proposed approach seem interesting, and I think that the authors can do a much better job in presenting their work. 

### Novelty

The proposed approach seems to be novel, to the best of my knowledge. 

### Reproducibility

There is no provided code in the present manuscript. 
summary_of_the_review: In the current state, I would not recommend the paper for publication, due to the concerns I have raised in the Strengths and Weaknesses section. I believe that the observations made by the authors and their proposed algorithm are interesting and the topic is relevant to the community, but the paper needs a lot of improvement (see Strengths and Weaknesses section). With that being said, I am willing to reconsider my score, conditioned on the authors addressing the raised concerns.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a minimax formulation of MI (M&M) that enables agents specialization with stable regularization. In this paper, authors have derived two formulations of MI from multi-agent policy gradients. One brings a new perspective on the existing work of joint policy optimization, the other explores the potential of conditional MI minimization. Then authors empirically evaluated M&M against the prior SOTA MARL framework, and analyzed the social behavior change in performance, diversity, and the stability of their social graphs.

strength_and_weaknesses: Strength:

The paper provided a new perspective on the existing work of joint policy optimization, The experimental environment selected a novel Naruto game. Naruto Mobile is a real-time 2D mobile Fighting Game of 1 vs 1 gameplay. The game has a pool of over 300 unique characters, and each has different character attributes in attack speed, range and movement variations. Additionally, each character has skills of different cooldown, status effects and duration. The large population of characters and real time competitive interactions create a testbed environment for research on heterogeneous specializations, embedded agents and Game Theory of Nash Equilibria.

Weaknesses:

* The writing of this paper is confusing, and there are many basic mistakes. For example, PSRO is Policy Space Response Oracle, not Policy Search Response Oracle. 

* The objective function in Eq.(1) is confusing. As this is a competitive game, how is the advantage function $Adv(s,a^x,a^y)$ computed?

* In eq(2), how could the policy gradient of two competitive players be computed together?

* It’s not clear to me how mutual information helps in multi-agent specialization.

* The definition of $\epsilon$-nash is not given.

* This work would benefit from testing on more tasks.

clarity,_quality,_novelty_and_reproducibility: Clarity: Very unclear and confusing

Quality: The quality of the figures can be improved and the explanation of the results can be enhanced by making the key information clear.

Novelty: The considered problem is interesting and novel, but the methodology is confusing to me.

Reproducibility: The author did not provide the source code, making it hard to evaluate the reproducibility.
 

summary_of_the_review: The paper provided a new perspective on joint policy optimization to enable agent specialization. The considered problem is interesting. But the reviewer found the present methodology is hard to follow and the experiments can be enriched by testing on more scenarios.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies agent specialization for multi-agent reinforcement learning. The authors proposed a min-max formulation of mutual information. The authors claim that the proposed method enables agents specialization with stable regularization. The proposed approach is evaluated on the newly introduced ‘Naruto Mobile’ which  is a real-time 2D mobile game with two competitive players.
strength_and_weaknesses: ### Strength:
The paper studied social behavior change and agent specialization, which are important problems for multi-agent reinforcement learning. 

### Weakness:

The writing of the paper is casual and lacking the rigor needed by a research paper. Many terms and symbols are undefined. Many sentences and paragraphs are unclear. Please see the following for more details:
1. In Eq 1, symbols such as $\theta, a^x, a^y, s, \pi$ and the term ADV are all undefined. Without defining the symbol, the authors left the readers guessing what each symbol meant.   
2. In P3, Q, V are undefined.   
3. In P.3, what are Generalists and Specialists? Please provide a formal definition.  
4. In P.4, what is a graph solver F? Please provide definition and context.   
5. In P.5, what is a ‘recorded vector’? Please define and explain.   
6. The term social graph is used throughout the paper without definition.     

The above list is not exhaustive. There are still a lot of undefined terms.   

clarity,_quality,_novelty_and_reproducibility: ### Clarity:
The paper writing is casual and informal. Major revision is needed. 

### Quality/Novelty
The unclear writing makes it hard to judge the paper’s technical quality and novelty. 

### Reproducibility:
No code is provided and the presentation of the technical content is unclear. The reviewer thinks the results may be difficult to reproduce. 

summary_of_the_review: In summary, the reviewer found the paper not ready for publication due to the writing quality.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers agreed that the paper proposes a novel minimax formulation of mutual information for multi-agent policy optimization. However, the reviewers pointed out several weaknesses in the paper and shared common concerns. We want to thank the authors for their detailed responses. Based on the raised concerns and follow-up discussions, unfortunately, the final decision is a rejection. Nevertheless, the reviewers have provided detailed and constructive feedback. We hope the authors can incorporate this feedback when preparing future revisions of the paper.
justification_for_why_not_higher_score: The reviewers pointed out several weaknesses in the paper and reached a rejection decision.
Decision: Reject