Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper performs extensive experiments on MAPPO, i.e., shared parameter PPO with Centralized value function in cooperative multi-agent settings. The authors ran PPO-based MARL algorithms on the MPE, SMAC, Hanabi, and GRF environments compared to off-policy baselines. Demonstrate that PPO-based algorithms can achieve competitive performance to those state-of-the-art off-policy algorithms. Based on the results, the authors also give suggestions for implementation tricks for on-policy MARL algorithms.
strengths: 1. This paper performs extensive experiments on the MPE, SMAC, Hanabi, and GRF environments compared with different MARL algorithms.
1. The authors ran ablations on their MAPPO implementation and gives tricky suggestions based on the results.
weaknesses: 1. The results are not significant enough.
1. The paper is organized like a kind of blog or technical report rather than an academic paper. In this paper, the author simply lists the results and then gives the conclusion that "MAPPO outperforms ...". The experimental analysis is not sound enough.
1. Followed by the weakness above, the authors did not do any theoretical analysis of the results. The suggestions given in the paper may not effective in other settings where the paper was not tested.
1. In lines 33, 266, and 291, the authors use the term "we hypothesize that ...". The conclusions may need more evidence, e.g., theoretical analysis.
1. In this paper, the author only discussed MAPPO in cooperative environments with homogeneous agents. That is only a subset of cooperative multi-agent games. None of the heterogeneous settings are benchmarked.
1. Nearly all methods utilize parameter sharing, lack of results on non-sharing settings. Do the suggestions given by the author still hold without parameter sharing?
1. The authors discussed that PPO-based on-policy algorithms can achieve comparable performance to the off-policy methods with comparable sample efficiency. But for off-policy methods, the sampling and training stages can be executed in parallel. Are on-policy methods as time efficient as the off-policy methods?
correctness: The experiment design is almost appropriate. But almost all experiments are tested in homogenous agent settings. None of the cooperative games with heterogeneous settings.
clarity: Yes.
relation to prior work: Yes.
additional feedback: 1. In this paper's title: "The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games", but the author only studied the CTDE setting of MAPPO. The centralized methods are not discussed in this paper. It would be better to add the centralized PPO algorithm as an additional baseline.
2. The authors have tweaked the PDF page margin to fit the paper into 9 pages. Some tables and figures have gone too close to the page margin.
review_title: Empirical Study for Shared Parameter PPO with Centralized-V in Cooperative Multi-Agent Games

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors evaluate multi-agent versions of PPO (IPPO and MAPPO) in four multi-agent environments, and compare it against several baselines. The authors perform several ablation studies to evaluate how different implementation details affect PPO's achieved returns. 

strengths: * The paper does a good job in evaluating PPO and comparing it against several baselines.
* The paper is in general well-written (besides some cases that I mention below).
* Section 5 does a good job in showing how different implementation details affect PPO's  performance.
* The code to reproduce the experiments is open-source.
weaknesses: My biggest concern in the paper is the evaluation consistency. 
For example:
* line 104: "nearly all baseline methods utilize parameter sharing..." why not all of them?
* line 173: "The reported results for SAD and VDN are obtained from [15]." This makes the comparison inconsistent as different implementations lead to completely different achieved returns.
* Table 2: "Results are reported over at-least 3 seeds." Why not the same number of seeds?

Another concern is the suggestions of Section 5. In general, I liked these studies, BUT, the obtained conclusions are only applicable to the specific evaluation tasks. In general, I use PPO in my own MARL research, and I have seen many cases where the suggestions of the authors would result in lower performance. I would recommend to make it clear in the text that these suggestions will not always result in better performance, and that the findings are only applicable to the specific tasks.

* The definition of the Dec-POMDP in lines 91-97 is incorrect. The definition is missing the observation conditional probabilities function.
correctness: Please see my comment about the evaluation consistency. 
clarity: There are some language issues with phrasing. For example:
* line 10: "strong off-policy methods". what does strong mean in the context of the paper?
* line 11-12: rewards ---> returns (I have seen that several times in the paper) 

relation to prior work: Yes
additional feedback: * I recommend to the authors to fix the references as there are several inconsistencies. 

* I recommend to the authors to increase the size of Figure 1, as it is difficult to read.
-----------------------------------------------------------

In general, the paper is well-known already in the community, counting more than 100 citations. It does a good job in evaluating PPO for MARL, and I believe that it should appear in the conference.
review_title: Official Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper makes serval contributions:
1. Compare PPO’s  performance with off-policy baseline algorithm
2. Compare PPO’s sample efficiency with off-policy baseline algorithm
3. Provide tricks on how to setup training configs
strengths: The paper is easy to read and it contains multiple empirically useful suggestion to the PPO practitioner on how to setup their PPO training, each backed up with ablation study. The experiments contains fair comparisons, hyper parameter search and detailed ablation study on the tricks it suggests.
weaknesses: One of the limitation of the paper is the depth of analysis provided.
correctness: I believe the evaluation methods and experiment design are correct.
clarity: Yes.
relation to prior work: The PPO clipping can be performed in different ways in multi-agent cases, such as the [Coordinated Proximal Policy Optimization. zifan et al.].  Any benchmark experiments on clipping schemes?
additional feedback: How does the multi-headed actor-critic network structure affects the PPO’s performance?
review_title: An exhaustive evaluation of PPO in cooperative MARL tasks and provides empirical suggestion.

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper implements PPO on four benchmarks and finds that PPO is a strong baseline for multiagent reinforcement learning. 
strengths: The authors provide the details of how to implement PPO on four benchmarks. 
weaknesses: This work lacks scientific novelty. I do not think the performance of PPO in multiagent settings like SMAC is surprising. I have been using PPO in environments like SMAC long ago. 
The followup work of this work has already been published elsewhere: https://arxiv.org/pdf/2206.07505.pdf

correctness: Yes. 
clarity: this paper is well written. 
relation to prior work: yes. 
additional feedback: A blog track might be the proper venue for this kind of work.
review_title: I do not think the workload, or the finding, or the idea is worth a scienfic research paper at NeurIPS.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper aims to demonstrate the promising performance of PPO in multi-agent reinforcement learning (MARL) problems compared with several off-policy MARL algorithms. The experiments are conducted in four benchmark environments, including multi-agent particle world env, StarCraft, Hanabi, and Google Research Football. Many up to date off-policy algorithms are included in the experiments and the parameter settings are given. 
strengths: As described in the paper, there is little work studying on-policy algorithm in multi-agent RL literature. This work shows the performance of PPO in four multi-agent environments.
weaknesses: Since this is a benckmark paper, my main concern is the missing of some technique details and the experiment setup, see Correctness and Clarity.
correctness: + In Section 3, the reward is conditioned on joint actions, i.e. $R(s,A)$, while the reward in objective function $J(\theta)$ is $R(s^t,a^t)$, which seems not conditioned on joint actions.

+ The experiment results in SMAC are provided in Appendix D.1. MAPPO uses AS and FP inputs, while a variant of QMIX uses MG inputs, which is “concatenation of the default environment global state”. For RODE, QPLEX, CWQMix, AIQMix, the input is not described. If different methods use different inputs, I doubt whether the comparison is reasonable. Same problem raises in other experiments, e.g., in Section 4.3, it seems the features used by MAPPO and SAD are different.

  
clarity: 1. This paper proposes MAPPO and IPPO in multi-agent scenarios. However, the details of IPPO are missing in the main paper and the appendix. Its name “independent PPO” is misleading at first glance. Generally, independent learning in multi-agent RL means that each agent optimizes its own return regardless of the interaction with other agents. In cooperative Markov game where only a single team reward is available, reward decomposition or credit assignment is needed. In Section 4.4, the authors claim that when “local observations contain a full description of the environment state, there is no distinction between MAPPO and IPPO”. It seems IPPO is still a centralized training method instead of an independent learning method. Please clarify how IPPO is trained, i.e., when the environment global state is unavailable and only a team reward is provided. 

2. In Section 4.2, the experiments in SMAC, MAPPO uses Agent-Specific (AS) global state and Featured-Pruned (FP) Agent-Specific Global State. What input for the comparing off-policy methods is not discussed.

3. In Section 5.2, "the **Agent-Specific Global State (AS)** which creates a global state for agent $i$ by concatenating the EP state and $o_i$". It seems **AS** is created for each agent. If we have $n$ agents, we will get $n$ AS features. It is unclear to me how these $n$ features are used to train a centralized value function.
relation to prior work: The paper makes good connections to many prior works.
additional feedback: N/A
review_title: OK but not good enough

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Most previous works regarding extensive benchmarks of the PPO algorithm focused on the single-agent setting, hence this work is different in that all of the benchmarks performed in this work concern the multi-agent setting. 
First, the authors compare the performance of MAPPO against its variants and other MARL algorithms in four cooperative game environments. Then the authors proceed to identify five key implementation and hyperparameter factors that are influential in PPO's performance in multi-agent settings, with a particular focus on the SMAC environment. The authors provide their empirical analysis and suggestions for each of the factors accordingly. To my knowledge, this paper is one of the early works that systematically study the performance of MAPPO.
\
Compared to the version published on ArVix, the death mask section is moved to the Supplementary Material while a new section regarding PPO batch size is made to replace it.
strengths: **(+)** Extensive experiments to prove the authors' points and the experiments are carefully planned and laid out in a thoughtful way.
\
\
**(+)** Well-structured main text, well-presented figures and tables. Empirical analysis and suggestions are provided for each influential factor. Most of the important details regarding the experiment setup are there.
\
\
**(+)** I generally agree with the authors' finding of "critical point" batch size. Some literature I have read in the past has reported a similar conclusion.
weaknesses: Note: I use **(~)** to denote my doubt or question but may not necessarily be a weakness of the paper.

**(-)** I'm worried that this work becomes a technical guide on how to tune hypermeters of the MAPPO algorithm while training in the SMAC environment. Though I definitely appreciate most of the authors' suggestions drawn from their experimental analysis since SMAC has become one of the most standard benchmarks for the MARL algorithms. I was wondering if the authors believe that to what extent their conclusions may be applicable to other MARL environments, where data frequency, reward landscape, the difficulty of credit assignment, etc., will differ. For instance, have you thought about performing some benchmarks on temporal sequential social dilemma (SSD) environments (i.e. involve mixed-motive agents) to see if your key findings hold across the board? I'm raising this concern because Section 5.2 seems specifically applies to the information structure of SMAC.
\
\
**(~)** Perhaps blaming everything on the non-stationarity of MARL might be too generic? One example might be Line 266-268. I am hoping for more insightful reasoning behind the performance gap caused by a change in hyperparameter.
\
\
**(-)** I believe that many factors in the author's analysis, such as PPO batch size, number of mini-batches, training epochs, and rollout fragment length, are tightly entangled together. Therefore, I think the author should combine Section 5.3 and 5.5 and perhaps add the rollout fragment length since they would affect the GAE horizon. Maybe more extensive grid searches that test out different combinations of these hyperparameters I mentioned in the first sentence could better convince me.
\
\
**(-)** From what I can see in Tables 3 and 4 from the supplementary material, QMix and MADDPG employ $\epsilon$\_greedy exploration while it is not clear to me what the PPO has used. Stochastic sampling?
\
\
**(-)** I have mixed opinions regarding the use of the "Median evaluation win rate" metric for SMAC. It makes some of the comparisons inconclusive especially when 100% win rates are obtainable. What about showing the average return? 
correctness: The evaluation method looks correct and the results seem reliable. Most experiments that I am aware of were run on multiple seeds (3~6). Training curves were provided with confidence intervals. Error bars and one standard deviation were provided in the bar charts and the tables respectively. 

Line 152:
> "take the median of the final ten evaluations as the performance for each seed"

Are they taking the top-10 evaluations sorted by win-lose or are you taking the actual last 10 games from the sequence of evaluations? Why not take the median over all 32 games?

Some data in Table 1 indicate significantly large variances in win rates, for example, the results for MAPPO(AS), IPPO in 6hvs8z, and MAPPO(FP) in 3s5zvs3s6z.
clarity: The figures and tables are information and well-structured. The main text is well-written and easy to follow.
relation to prior work: The authors have cited most of the relevant influential works that I can think of (namely [9],[1],[24]) and differ from them by analyzing the performance of the PPO algorithms exclusively in multi-agent environments or have performed more experiments on SMAC.
additional feedback: * Reference [24], G. Papoudakis et. al 2020, was an accepted paper of this track last year, please update the citation accordingly.
* Some tables and figures have gone over the top margin limit for almost an inch. Clear signs of abusing reduce `\vspace` command and it shows when Line number 259 and 260 are almost squished together. The titles, the figure and the text are all crammed together and causing minor discomfort while I was reviewing this paper. 

This work experienced a few iterations of modifications over the past year, and I will admit that it is now in better shape compared to the previous versions. Therefore I am voting for marginal acceptance of this work.
review_title: A comprehensive study on the performance of the multi-agent PPO variant in the multi-agent settings

Meta-review: metareview: The paper presents extensive evidence showing PPO to be a (surprisingly) strong algorithm in multi-agent RL environments compared to more sophisticated and specialized approaches. The paper provides a helpful codebase along with elaboration on a variety of implementation details which significantly matter for PPO's good performance. 

The reviewers had conflicting views of the paper. I'm glad to see that the submission authors used the review period wisely to significantly improve the clarity and organization of the paper, which was one of the major concerns (thanks to the reviewers for raising these concerns!). Even so, while most reviewers appreciated the extensive and detailed experiments and ablations of the paper, concerns lingered regarding novelty and significance of the work. In response to these concerns, I have consulted with a few domain experts in the field of multi-agent RL. Through these consultations, I have determined that the submission is indeed impactful, with its findings already being extensively used and built upon. Thus, altogether I believe this submission meets the bar for acceptance. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept