Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper's main contribution is the proposed standardized performance evaluation protocol for cooperative MARL. Specifically, this paper performs meta-analyses and identifies the following potential issues in related works:
1. Lack of experimental details, open-sourced code, and ablation studies (Section 2.1)
2. Lack of standardized statistical tools and uncertainty estimation (Section 2.2)
3. Lack of standardized environment sets for preventing domain overfitting (Section 2.3)

Based on these observations, this paper provides the standardized evaluation protocol for cooperative settings in Section 3, including benchmark/baseline, evaluation parameters, statistical quantification, and reporting.
strengths_and_weaknesses: **Strengths:**
1. The performed meta-analyses provide interesting perspectives on recent trends in cooperative MARL, such as types of MARL algorithms used in papers, ablation studies, and types of performed uncertainty quantification.
2. In general, I agree with the main points of this paper that MARL may lack standardization and thus establishing the standard protocol is important.
3. The paper is written well and conveys the main points clearly.

**Weakness:**
While I agree with the main points, my main concern is the novelty of this paper compared to related works. In particular, the importance of reporting experimental details (Section 2.1) and computing uncertainty estimation (Section 2.2) is already discussed in prior works (Henderson, 2018; Agarwal, 2021), so these aspects have been considered in recent MARL papers. Compared to these works, the domain overfitting issue discussed in Section 2.3 adds a new perspective to the MARL community, but I am unsure whether this perspective alone has sufficient novelty for acceptance.

**References:**
Henderson, et al., Deep Reinforcement Learning that Matters, AAAI 2018
Agarwal, et al., Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS 2021
questions: 1. (Related to Weakness) I understand that extending to general-sum/competitive settings is a future work, but I hope to ask for a clarification on why this paper performed meta-analyses limited to cooperative settings. In particular, the main differences between RL and MARL result from the fact that there are multiple learning agents with non-stationary policies. Hence, establishing the standard protocol for general-sum and/or competitive games is an open question because it is often not clear when the learning should end under non-stationarity compared to cooperative settings. As noted in the paper, there are close similarities between RL and cooperative MARL evaluation, and the related works of Henderson (2018) and Agarwal (2021) focus on single-agent settings. As such, if this paper could provide a general protocol that includes general-sum and/or competitive settings, the contribution could have been more significant.
2. In Figure 3(f), there might be a typo in the x-axis label: "Year" to "Number of independent runs"
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper is a interesting meta-research of cooperative MARL methods. The authors claim at least three major problems in current MARL research: report of implementation and evaluation details, report of uncertainty with proper statistical tool, the choice of benchmark environments. The paper tries to appeal for a standard protocol for MARL experiments, which should be encouraged for benefiting the community. 
strengths_and_weaknesses: This paper conduct heavy statistical analysis of previous MARL algorithms, which are reported in the diagrams. The results are clear and thorough. The efforts of the authors should be appreciated. The description of the problems are clear. More importantly, recommendations for solving the problems are provided after each section.

One major weak side of the work is that it seems all three aspects of lessons are inherited from single-agent RL, with no additional components specifically caused by cooperative MARL settings. The problems of reporting implementation details, statistical uncertainty estimation and choice of environments are existing problems in single-agent RL, which makes this work looks like a replicate of commonly known problems in single-agent RL to MARL domain. It could be more interesting to see at least one or two more lessons specifically considered in MARL setting.
questions: For ablation study analysis in Sec. 2.1, it seems only the number of papers using ablation study are discussed. Is there any deeper investigation of the number and types of ablation study within each paper, and what should be a standard for ablation study?

The results in Fig.4 (e) show that some methods even have a decreasing performance along the time. Does that mean the later work does not report the performance as good as the original paper claims?
limitations: As mentioned in the weakness, more insights of problems specific for MARL settings are interesting to know, apart from the commonly known single-agent RL problems.

Although the authors propose a performance evaluation protocol for cooperative MARL in Sec. 3, some other problems raised by authors are not concretely solved, like how to choose a standard set of environments for different MARL settings, etc. I understand this problem may be hard to give an answer and the authors may want to receive more feedback from the community. I think if this paper is the one trying to establish the protocol for MARL experiments, these problems are better solved with concrete solutions even if in the follow-up works.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper reveals the trends and worrying issues of evaluation via an extensive study of recent MARL works. Several concerns are raised regarding inconsistent results, poor statistical aggregation, and environmental misuse.  Based on the analysis, the authors propose several recommendations and a standardized protocol, specifying the methodologies of how evaluation parameters selection, uncertainty estimation, and reporting should be performed.
strengths_and_weaknesses: ## Strengths

+ The logic of this paper is easy to follow, and the meta-analysis is comprehensive.

+ The summary and proposal of a standardized evaluation protocol is valuable.

As a practitioner, I quite understand that reproducibility is a huge problem in recent MARL works. It is worthwhile (however difficult) to summarize the evaluation methods applied in these works. I really appreciate the efforts of the authors and hope the standardized protocol could be utilized by researchers in the field.

## Weaknesses

I don't find any significant weaknesses in this paper.
questions: + In L280, by using the term "timesteps", do the authors mean environment frames or training steps? What is the batch size and the frequency of updating parameters?
+ L191: "SMAC and it's" -> "SMAC and its"
+ L289: please use \citet
limitations: I don't know much about the works investigating the evaluation protocol in single-agent RL. It seems that the authors just translate the designed protocols into the domain of cooperative MARL. However, as the authors mentioned in the introduction section, the contribution of this paper is not an innovation of evaluation protocol. This limitation can be minor.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
Towards a Standardized Performance Evaluation Protocol for Cooperative MARL
In this paper, the authors review existing papers on multi-agent reinforcement learning, and show three lessons from these papers: 1) we need to study what exactly makes the improvement and don’t hide details; 2) we need to have standardized results reporting and uncertainty analysis; 3) we need to avoid exploiting the environments and overfitting it.
And authors also propose a standard evaluation protocol for future MARL research.

strengths_and_weaknesses: 
Originality: The paper discusses a very important topic in multi-agent RL that draws more and more attention recently. Some of the findings, for example the inconsistent reported performance, hidden performance tricks under the sleeves, improper handling of results reporting and missing uncertainty evaluations, have been already discovered as problems in general RL research. That being said, a lot of the findings have already been discovered in Agarwal et al. (2022), Whiteson et al. (2011), Henderson et al. (2018).

On the bright side, the paper gives a very good systematic review on multi-agent reinforcement learning algorithms. Figures such as Figure 2, Figure 4 are quite inspiring and the discussion about QMIX, VDN, COMA and IQL give quite valuable information about the actual performance of these algorithms.
The paper also provides a concluding protocol suggestion, which will benefit the community.


Quality and clarity: The paper is well written, supported with informative figures and plots.
Each section has a very clear topic and the authors provide detailed explanations for every proposal in the paper.

Significance: A lot of the problems have been discussed by other related papers, which potentially will make the impact of this paper less. However the proposed protocol is beneficial to the multi-agent RL community.

questions: 1) What does the author believe are the problems in multiagent RL that are different or unique as opposed to the ones also commonly seen in general RL research?
2) What do the authors propose to help all the researchers to follow the correct protocol and practices in RL? Is there a code-base or website MARL researchers could use for MA benchmarks?
3) There’s a very related paper here [4] (Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning). This is a paper that was available very recently, but the author might want to cite it as a contemporary work in the final version.



limitations: 
I don’t find anything that would potentially have a negative societal impact.

[1] Agarwal, Rishabh, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc Bellemare. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.
[2] Henderson, Peter, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. "Deep reinforcement learning that matters." In Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1. 2018.
[3] Whiteson, Shimon, Brian Tanner, Matthew E. Taylor, and Peter Stone. "Protecting against evaluation overfitting in empirical reinforcement learning." In 2011 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pp. 120-127. IEEE, 2011.
[4] Fu, Wei, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. "Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning." arXiv preprint arXiv:2206.07505 (2022).
review_title: null

Meta-review: metareview: This paper performs a meta-research on cooperative MARL research, identifies three main issues, and proposes a standard evaluation protocol. All the reviewers agree that the paper is well written, provides interesting perspectives, and recommends reasonable solutions. A common concern that was shared by all reviewers is that the discussed issues were already highlighted in prior works on single-agent RL. The "Reply to all reviewers" in the rebuttal clearly addressed the above concern. After the discussion, all reviewers agree that publishing this study, including both the meta-analyses and the proposed evaluation protocol, would be beneficial for the MARL community. Thus, I recommend accepting this paper.
confidence: Certain
Decision: Accept