Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes the coordinated proximal policy optimization (CoPPO) algorithm for cooperative multi-agent systems. CoPPO is a novel algorithm with centralized training and decentralized execution (CTDE) which achieves better coordination between agents via a coordinated adapting process of the agents' step size when updating policies. The empirical results in penalty games and SMAC show that CoPPO outperforms the SOTA algorithms such as MAPPO and DOP.
main_review: The paper is technically sound and well written. I think it is a good paper. However, I still have one concern which I think is quite important for making the final decision: Why did the author only run 6 maps in the SMAC experiments? Why not compare the algorithms in more SMAC maps such as MMM, MMM2, 10m_vs_11m?
limitations_and_societal_impact: See the above
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a multi-agent generalization of proximal policy optimization algorithm (PPO) (Schulman'17). The authors provide monotonic policy improvement guarantee and use double clipping technique (Ye'20) in order to mitigate the exploration issues caused by multiplication of importance weights in off-policy sampling. They also interpret this objective function as dynamic credit assignment among agents which can reduce the variance in update. They test the proposed approach on cooperative matrix game and Starcraft tasks to show its outperformance over other multi agent RL algorithms.
main_review: The flow of the paper is nice and readable. The idea of the paper on multi agent RL with PPO is interesting as well.  
In summary, I see this paper as combination PPO and double clipping technique(Ye'20) in multiagents. Because the derivations are pretty straightforward and similar to  TRPO and PPO papers. Overall, The results are nice but not surprising at all. 
Some questions:
Does double clipping affect the convergence or policy improvement lemma? Can clipping be applied separately to each importance weight to avoid the problem?
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a new CTDE algorithm CoPPO for cooperative MARL. CoPPO is based on the extended objective of TRPO in MARL settings. CoPPO uses the clip objective of PPO to implement the trust region constraint of TRPO and proposes a double clipping trick to adjust the step sizes of different agents.
main_review: The double clip trick is interesting, and it also has the efficacy of dynamic credit assignment. However, I have the following concerns.

1. The objective of CoPPO is actually a direct extension of TRPO in multi-agent settings. If we take a centralized view, suppose the policies of agents are mutually independent, following the conclusion of TRPO, we can direct derive the same result. So the main contribution of this paper may lie in the double clipping trick for optimizing the objective. 

2. The paper is unclear in many ways  at least from the writing. For example, is a joint Q(s,a)  learned to compute A^i? where does c^i come from? 

3. Morever, the empirical results on SMAC maps seem to have some problems. The performance of MAPPO in the Hard and Super Hard maps are much lower than the results in the Table 1 in the original paper of MAPPO. And the learning curves of MAPPO in these maps are different with the original paper. 

4. Moreover, the performance of CoPPO in these maps is lower than the results of MAPPO in the original paper.

I expect these concerns can be addressed during rebuttal. 
limitations_and_societal_impact: As stated above
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work presents an extension of the popular proximal policy optimization (PPO) algorithm for cooperative multi-agent tasks.
main_review: # Strengths
* The proposed method falls into a promising direction for cooperative MARL methods. Much of the focus in this field has been in value-based methods, but simple extensions of single-agent policy-optimization methods have recently been shown to be competitive with the SOTA value-based ones. As such, this work is timely.
* The exposition is clear and succinct. Theoretical contributions are well motivated and explained intuitively.
* The expected baselines are present in the experiments, and the proposed method reliably outperforms them.

# Weaknesses
I don't believe the paper has any serious shortcomings, though there are a few minor points that could be addressed.
* The colors should be consistent in Figures 1 and 2. In other words, whichever setting for the inner clipping threshold that is used in Figure 1 should be blue in figure 2. Perhaps the other settings should use colors or line patterns that aren't used in figure 1.
* It would be good to see how CoPPO performs in more of the Hard and (especially) Super Hard tasks from SMAC. The MAPPO paper performs experiments on the whole set of tasks.

# Overall Thoughts
This paper is a well thought out and sensible follow up to recent works which show the relatively untapped potential of policy-based methods for cooperative MARL.
limitations_and_societal_impact: On the subset of tasks from SMAC selected for experimentation, the proposed methods performs well. I imagine that on at least some tasks from SMAC, the proposed method does not outperform the baselines, so it would be good to know what these are and what the authors' hypotheses are for the lack of improvement in those settings , if they exist.
review_title: null

Meta-review: The reviewers and AC discussed the paper. There was agreement that the approach is interesting and the results are impressive. The author response was also helpful in addressing some of the concerns of the reviewers (e.g., additional experimental results). There are still some concerns about the novelty of the approach and the comparisons with MAPPO. These should be clarified. 
Decision: Accept (Poster)