Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The major contribution of this paper is benchmarking 5 MARL algorithms on 4 cooperative multi-agent environments. Also, this paper found that under constrained hyperparameter search budgets, the multi-agent PPO algorithm has more consistent performance over the other algorithms across different tested multi-agent environments. The code base is open-sourced for public use, which benefits the MARL community.

Researchers in MARL often find it difficult to find a useful benchmark for multiagent learning algorithms and multiagent environments. Thus this paper makes a good contribution to the community, but, on the other hand, since this paper is not really intended to present any major technical contributions, that could be considered a weak point for an ICLR submission.

To the best of the reviewer's knowledge, there has not been any previous work that tries to produce a benchmark for multi-agent deep reinforcement learning. As the first work that attempts to fill this gap, this paper presents a comprehensive implementation of popular MARL algorithms tested on a representative list of MA environments, which is the major reason for acceptance suggestion. But there are some limitations of this paper. For example, there is little discussion about the results and the authors did not attempt to do more technical investigation for understanding the their findings about the algorithms. In this sense, this paper is less than a qualified research paper.
In particular, it would be helpful if  the authors have any insight on why the MAPPO works more consistently than the other algorithms.

One major suggestion is to add standard deviation to the numeric values presented in Table 1, 2, 3 and 4.

If this paper get accepted to this conference, peer researchers could be incentivized to contribute similarly comprehensive benchmark to this field, which would benefit the advancements of this field in the long run.    

I have read over the rebuttal and discussion and will keep my evaluation score unchanged as I see value in benchmarking papers such as this for the community.
review_title: Comparison of MARL algorithms that could benefit the community

final_score: rating: 3: Clear rejection
review_content: The paper aims to benchmark a suite of Multi-Agent Deep Reinforcement Learning algorithms across different environments in the cooperative multi-agent setting. The paper compares standard algorithms alongside extensions of well-known policy gradient algorithms to the multi-agent setting, i.e. PPO (MAPPO), SAC (MASAC) and TD3 (MATD3). The paper investigates the ease of using an algorithm by doing a fair hyperparameter search for different algorithms. The paper concludes by saying that MAPPO is a promising choice for tackling a multi-agent problem. 

Such a benchmarking paper would absolutely be useful. In its current form, however, it is not yet ready for publication. There are three primary reasons. First, the scope is misplaced: the paper starts with the scope of benchmarking different algorithms but seems to shift focus primarily to MAPPOs benefits. Second, the experiments only include 3 runs, with large overlapping deviations, limiting the ability to make claims about any significant differences. This is particularly problematic for a paper where the main focus is on experiments to provide insight. Finally, it seems like Pop-art is only used with MAPPO (please clarify this if I am mistaken), calling into question if the main reason for improvements in MAPPO is this choice. Reward scaling, done by Pop-art, can significantly affect learning performance as pointed out in Henderson et al. (2017). Using Pop-art only for MAPPO makes the comparison unfair between different approaches and potentially invalidates the conclusions presented in the paper.

As two other more minor points:

1. Multi-agent methods are built around numerous agents learning separate policies. The choice of sharing the same weights (Page 4, Sec 3.3) among different policies doesn’t seem to me like a fair choice. This also makes the algorithms different from their original intended use. 

2. There are several inconsistencies or questionable choices in the paper: 

a. As per appendix, MADDP, MATD3, MASAC, QMix have gamma parameters, whereas there is no such parameter mentioned for MAPPO. Having a different gamma value for the methods essentially changes the problem.

b. There is a difference between similar hyperparameters for different methods, i.e. MAPPO uses a gradient clipping of 20, whereas other methods use a gradient clipping of 10. How was this chosen and why is it different?

c. Appendix (Hanabi), MAPPO uses a learning rate of 7e-4, whereas the learning rate is not mentioned as a value being tested in the search grid. 

d. MADDPG and MASAC are said to use a centralized critic, but according to Algorithm 1 and Algorithm 2, they seem to estimate separate critics. 

e. MASAC should have a tuple of three parameters (value functions, q function and policy), but Algorithm 2 talks about only two parameters (q function and policy). 


A few questions: 
1. In SMAC environments, why are MASAC, MATD3 and MADDPG not run for other settings (Table 2)? 

2. In MPE, I was not able to compare the results for MADDPG to the original paper. Can you point out the results (i.e. example the table number in Lowe et al. 2017)?  This also seems strange as Lowe et al. 2017 use a non-recurrent network with a different parameterization for each of the policies. In contrast, the current paper uses a recurrent network and shared policies, making the models quite different. 

Suggestions: 
1. Although the paper compares all the well-known policy gradient methods, comparing against a simple Vanilla Actor-Critic network would provide useful insight by removing some of the complexity.

2. It would be good to include hyperparameters of PPO like the \epsilon clipping.
 
3. The graphs in Figure 3 do not have a visible axis.

4. Including parameter sensitivity would provide much more insight into the properties of the algorithms,  and how they perform for general configuration of parameters. 

5. Section 3.1, P(s’|s,u) : \mathcal{S} \times U \times \mathcal{S} \rightarrow [0,1], should have a U^n instead of U.  

References
1. Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2019). Deep Reinforcement Learning that Matters. 
2. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., & Mordatch, I. (2020). Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. 

--- Update after reading the rebuttal and other reviews

Though some concerns have been addressed, a critical issue remains unresolved, which is that the experiments use only 3 runs. 

As an additional point, it is useful that you've identified MAPPO as a good multi-agent algorithm. However, for clarity, it is better to focus the paper around MAPPO rather than strictly calling it a benchmarking paper. Further, it would be good to clearly state in the paper that the results now include reward normalization. 
review_title: Official Review for Reviewer 1

final_score: rating: 4: Ok but not good enough - rejection
review_content: In order to compare different MARL algorithms more fairly, the author compared the differences in the performance of different algorithms(MADDPG, MATD3, MASAC, Qmix, MAPPO) in different environments(Particle-World, StarCraft Micromanagement, Hanabi, The Hide-and-Seek Domain).  The author's writing and organization are very good, so that I can clearly understand the content of the paper, and also has certain highlights, but I think it has not reached the ICLR criteria.

First of all, as an article about BENCHMARKING MULTI-AGENT DEEP REINFORCE- MENT LEARNING ALGORITHMS, I think the author should compare a series of algorithms to bring more insightful analysis and inspiration. Unfortunately, I found that the author simply I enumerate the performance differences of different algorithms in different environments, and there is also a lack of analysis of what methods are suitable for what tasks. I suggest that the author conduct further analysis and experiments, and also pay attention to the advantages and disadvantages of different algorithms. I hope that the author can adjust the content of the paper, and don't make people feel that the author has collected different official implementations, and then simply run them and compare directly.

In addition, I think the author may be missing some important MARL algorithms. For example, in terms of policy-base, it is recommended that the author consider adding algorithms such as COMA[1] and MAAC[2] (the authors of these algorithms have already announced their official implementations). At the same time, most of the author's algorithms have official implementations on the Internet. How to compare different algorithms fairly is of great concern, but the author's content is still relatively weak, and the author is recommended to improve this part of the discussion.

Overall, I vote for a rejection

[1] Counterfactual Multi-Agent Policy Gradients, Foerster et.al. AAAI 2018

[2] Actor-Attention-Critic for Multi-Agent Reinforcement Learning, Iqbal and Sha, ICML 2019
review_title: Ok but not good enough

final_score: rating: 4: Ok but not good enough - rejection
review_content: ## Summary
The paper provides a useful benchmark (a suite of cooperative multi-agent RL tasks) and a nice comparison of common as well as uncommon-but-successful algorithms on these tasks. A limited budget of hyperparameter tuning is applied to each algorithm in order to give each one a chance at good performance, and a clear winner emerges for consistent strong performance across tasks.

## Quality & Clarity
The paper is presented according to a high standard of quality. They document and explain their methods clearly, including hyperparameters. They have clean descriptions of each algorithm and easily readable text.

Nit: for table 2, it would be easier to read if top scores (or, top scores within stdev) were bolded.
Nit: for table 3 with 2 players and possibly for table 4, it might be better to report results using a graph as this will show the progress that each algorithm made and document where they were cut off. This could even just be in the appendix.
Nit: you use box-locking-easy and box-locking-simple interchangeably--best remove one.

## Originality
I’m not aware of other work which provides as clean a comparison of both different tasks and different methods, so there is novelty in the open source benchmark that this provides. The work contains some original tasks built on top of existing ones (i.e. in the Hide and Seek domain). There are no original algorithms in this paper as it is not the focus of the work.

## Significance
From the conclusion:
> “Given access to equally sized compute budgets for hyperparameter optimization, MAPPO is the most consistently successful algorithm across the studied environments, performing well at SMAC, as well as any algorithm in the MPEs and the best in hide-and-seek and Hanabi-small. Surprisingly, we also find that the centralized Q function variants DDPG, SAC, and TD3 almost entirely fail to solve any of the studied tasks, particularly in slightly harder environments.”

The main conclusion of the paper is “MAPPO is a good starting point for tackling new MARL problems”. This is a useful practical tip for researchers in the field. However, the work provides little insight into why that is the case. The next lines of the conclusion confirm that:

> “Of course, these results are likely dependent on the choice of hyperparameters and architectural choices. Thus it may be the case that the initial point around which we perform our grid search for MA-DDPG/TD3/SAC does not contain any good solutions and that performance of a set of hyperparameters on the MPEs is not predictive of performance in other domains. Finding a better set of hyper-parameters over which to search for the off policy algorithms is an important direction for future work.”

The authors admit that they were only able to explore a limited space of hyperparameters and their work may not hold in other circumstances. There is plenty of room for continued exploration of hyperparameters, certainly, but even more valuable would be any indicators into why different techniques might have different performance on the different tasks. I think in order for the publication to get accepted to a conference like ICLR it likely needs a bit more investigation, though researchers may certainly find the benchmark useful in research.
review_title: Useful benchmark but limited significance (so far)

Meta-review: The paper presents a thorough comparison of different algorithms for multi-agent Deep RL methods. The conclusions of the paper is that across a variety of envionment and hyperparameter tuning, multi-agent PPO seems to peform well relatively to the competitors.

The reviewers agreed that the paper fills a gap in the literature regarding a fair and thorough comparison of algorithm, and that the paper clearly presents the results. As it stands, the code to reproduce the experiments and the results are a useful contribution to the community. However, the reviewers felt the technical contribution to be below the bar for ICLR, as the paper does not help in understanding the differences between algorithms, or develop insights as to how to further improve algorithms. The large standard deviations of the various algorithms also makes the main experimental insight (MAPPO works consistently well) relatively weak.


Decision: Reject