Towards a Standardised Performance Evaluation Protocol for Cooperative MARL
Authors: Rihab Gorsane
Omayma Mahjoub
Ruan John de Kock
Roland Dubb
Siddarth Singh
Arnu Pretorius
Venue: NeurIPS 2022 Accept (2022)
Keywords: “multi-agent reinforcement learning”
“MARL”
“Evaluation”
“Cooperative MARL”
“Standardised evaluation protocol”

Abstract: Multi-agent reinforcement learning (MARL) has emerged as a useful approach to solving decentralised decision-making problems at scale. Research in the field has been growing steadily with many breakthrough algorithms proposed in recent years. In this work, we take a closer look at this rapid development with a focus on evaluation methodologies employed across a large body of research in cooperative MARL. By conducting a detailed meta-analysis of prior work, spanning 75 papers accepted for publication from 2016 to 2022, we bring to light worrying trends that put into question the true rate of progress. We further consider these trends in a wider context and take inspiration from single-agent RL literature on similar issues with recommendations that remain applicable to MARL. Combining these recommendations, with novel insights from our analysis, we propose a standardised performance evaluation protocol for cooperative MARL. We argue that such a standard protocol, if widely adopted, would greatly improve the validity and credibility of future research, make replication and reproducibility easier, as well as improve the ability of the field to accurately gauge the rate of progress over time by being able to make sound comparisons across different works. Finally, we release our meta-analysis data publicly on our project website for future research on evaluation accompanied by our open-source evaluation tools repository.

Towards a Standardised Performance Evaluation Protocol for Cooperative MARL
Rihab Gorsane
Omayma Mahjoub 12-1
Ruan de Kock !+
Roland Dubb 13f
Siddarth Singh'
Arnu Pretorius*
1 InstaDeep
National School of Computer Science. Tumisia University of Cape Town, South Africa
Abstract
Multi-agent reinforcement learning (MARL) has emerged as a useful approach to solving decentralised decision-making problems at scale. Research in the field has been growing steadily with many breakthrough algorithms proposed in recent years. In this work, we take closer look at this rapid development with focus evaluation methodologies employed across a large body of research in cooperative MARL. By conducting detailed meta-analysis of prior work, spanning 75 papers accepted for publication from 2016 to 2022, we bring to light worrying trends that put into question the true rate of progress. We further consider these trends in a wider context and take inspiration from single-agent RL literature on similar issues with recommendations that remain applicable to MARL. Combining these recommendations, with novel insights from our analysis, we propose a standardised performance evaluation protocol for cooperative MARL. We argue that such a standard protocol, if widely adopted, would greatly improve the validity and credibility of future research, make replication and reproducibility easier, as well as improve the ability of the field to accurately gauge the rate of progress over time by being able make sound comparisons across different works. Finally, we release our meta-ans meta-analysis data publicly on our project website for future research on evaluation accompanied by our
evaluation tools repository
Introduction
Empirical evaluation methods in single-agent reinforcement learning (RL) have been closely scruti- nised in recent years (Islam et al., 2017; Machado et al., 2017; Henderson, 2018; Zhang et al., 2018; Chan et al., 2020; Jordan al., 2020; Engstrom et al., 2020; Agarwal et al., 2022). In this context, the impact of lack of rigour and methodological standards has already been observed. Fortunately, just as these issues have arisen and been identified, they have also been accompanied by suggested solutions and recommendations from these works.
Henderson al., 2018; Colas et al., 201 2018a, 2019;
"Equal contribution. Corresponding author: r.gorsane @ instadeep.com Work done during an internship at InstaDeep.
'htis:///sites. http://sttes..oggel....e wowgle.com/view//nee comm/viw//aarl-saadnat sanad--tanddrdproooeeee
"http:://ittuub.c httpps///gituub.oo//ssaaata https///oatateeeaaaaaa.....ao
'In this paper, study, which MARL is subfield.
the "RI." exclusively refer single-agent RI., as opposed RI. field of
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Multi-agen Mulli--agent reinforcement learning (MARL) extends RL with capabilities to solve large-scale de- centralised decision-maken tasks where many agents are expected to coordinate achieve shared objective (Foerster, 2018; 8; Oroojlooyjadid and Hajinezhad, 2019; Yang and Wang, 2020). In this cooperative setting, where there common goal and rewards are shared between agents, sensi- ble evaluation methodology from the single-agent case often directly translates to the multi-agent case. However, despite MARL being far less developed and entrenched than RL, arguably making adoption of principled evaluation methods easier, the field remains affected by many of the same issues as found RL. Implementation variance, inconsistent baselines, and insufficient statistical rigour still affect the quality of reported results. Although work specifically addressing these issues in MARL have been rare, there have been recent publications that implicitly observe some of the aforementioned issues and attempt to address their symptoms but arguably not their root cause (Yu et al., 2021; Papoudakis et al., 2021; Hu et al., 2022). These works usually attempt to perform new summaries of performance or look into the code-level optimisations used in the literature provide insight into the current state of research.
In this paper, we argue that to facilitate long-term progress in future research, MARL could benefit from the literature found in RL on evaluation. The highlighted issues and recommendations from RL may serve as guide towards diagnosing similar issues in MARL, well as providing scaffolding around which a standardised protocol for evaluation could be developed. In this spirit, we survey key works in RL over the previous decade, each relating to particular aspect of evaluation, and use these insights to inform potential standards for evaluation in MARL.
For each aspect of evaluation considered, we provide detailed assessment of the corresponding situation in MARL through a meta-analysis of prior work. more detail, our meta-analysis involved manually annotating MARL evaluation methodologies found in research papers published between 2016 to 2022 from various conferences including NeurIPS, ICML, AAMAS and ICLR, with a focus on deep cooperative MARL (see Figure 1). In total, we collected data from 75 cooperative MARL papers accepted for publication. Although we do not claim our dataset to comprise the entire field of modern deep MARL, to the best of our knowledge, our data includes all popular and recent deep MARL algorithms and methodologies from seminal papers. We believe this dataset is the first of its kind and we have made it publicly available for further analysis.
By mining the data on MARL evaluation from prior work, we highlight how certain trends, wor- rying inconsistencies, poor reporting, lack of uncertainty estimation, and general absence of proper standards for evaluation, is plaguing the current state of MARL research, making it dif- ficult to draw sound conclusions from compara- tive studies. We combine these findings with the earlier issues and recommendations highlighted in the literature RL evaluation, propose a standardised performance evaluation protocol for cooperative MARL research.
Number of papers publishedd
16 20
15
13-
5-
0-
2016
2017
2018
2019
2020
2021
2022
Year
Figure 1: Recorded papers by year in the meta- meta-analysis on evaluation methodologies in cooperative MARL.
In addition standardised protocol for evalu- ation, we expose trends in the use of benchmark environments and suggest useful standards for environment designers that could further improve the state of evaluation in MARL. We touch upon aspects of environment bias, manipulation, level/map cherry picking as well as scalability and compu- tational considerations. Our recommendations pertain to designer specified standards concerning the use of a particular environment, its agreed upon settings, scenarios and version, as well as improved reporting by authors and preferring the of environments designed test generalisation.
There are clear parallels between RL and MARL evaluation, especially in the cooperative setting. Therefore, we want to emphasise that our contribution in this work is less focused on innovations in protocol design (of which much can be ported from RL) and more focused on data-driven insights on the current state of MARL research and a proposal of standards for MARL evaluation.

---

(a)
100
80
Number paper
80
8.
2011 d0n
SAL n. 308
Tel
Contron
120
2 w
OMT MADDPG
VOR
MAPPO
CENTRAL
inC
1dit:: vt. 10
Algoritmm
(c)
SG.46
Wenien
(d)
Dartalrs udleaan saudy
SC24
The Tevustas (Mllan))
2.0
4.0
56.2
2010
2017
2010
65/
2000
202'
2032
Figure 2: Inconsistencies in performance reports and a lack of ablation studies. (a) MARI. algorithms ranked by popularity. (b) Historical performance of QMIX on different SMAC maps across papers. (c) Performance of QMIX the MMM2 SMAC map reported in different papers. (d) A count of papers the years containing any type ablation study as part of their evaluation of newly proposed algorithm.
From RL to MARL evaluation: lessons, trends and recommendations
In this section, provide list of key lessons from RL that applicable MARL evaluation. We highlight important issues identified from the literature on RI. evaluation, and for each issue, provide assessment of the corresponding situation and trends in MARL. Finally, conclude each lesson with recommendations stemming from our analysis and the literature.
2.1 Lesson 1: Know the true source of improvement and report everything
Issue in RL - Confounding code-level optimisations and poor reporting: It has been shown empiri- cally that across some RL papers there considerable variance the reported results for the same algorithms evaluated on the same environments ( (Henderson et al., 2018; 8; Jordan et al., 2020). This variance impedes the development mance comparisons and making direct comparisons of results across papers difficult. Implementation and code-level optimisation differences can have significant impact on algorithm performance and may act as confounders during performance evaluation (Engstrom et al., 2020; Andrychowicz et al., 2020). It is rare that these implementation and code-level details are reported, or that appropriate ablations are conducted to pinpoint the true source of performance improvement.
novel algorithmic developments by creating misleading perfor-
The situation in MARL Similar inconsistencies and poor reporting but promising rise in ablation studiex: There already exist some work in MARL showing the effects of specific code- level optimisations evaluation performance (Hu al., 2021). Employing different optimisers, exploration schedules, or simply setting the number of rollout processes to be different, can have significant effect on algorithm performance. To better understand the variance performance reporting across works in MARL, we focused on QMIX (Rashid et al., 2018), the most popular algorithm our dataset (as shown in Figure (a)). In Figure (b), we plot the performance of QMIX tested on different maps from the StarCraft multi-agent challenge (SMAC) environment (Samvelyan al., 2019), a popular benchmark in MARL. On several maps, we find wildly different reported performances with large discrepancies across papers. Although it is difficult to pin down the exact source of these differences in reports, we zoom with our analysis to only consider a single environment, in this case MMM2. We find that some of the variance is explained by differences in the environment version as well as the length of training time, as shown Figure 2 (c). However, when both of these aspects controlled for, well any implementation evaluation

---

details mentioned in each paper, differences in performance are still observed (as seen by comparing orange and blue circles, respectively). This provides evidence that unreported implementation details, or differences in evaluation methodology account for some of the observed variance and act as confounders when comparing performance across papers (similar inconsistencies in other maps are shown in the Appendix). We finally consider studying the explicit attempts in published works at understanding the sources of algorithmic improvement through the use of ablation studies. We find that very few of these studies were performed in the earlier years of MARL (see Figure 2 (d)). However, even though roughly 40% of papers in 2021 still lacked any form of ablation study, we find promising trend showing that ablation studies have become significantly more prevalent in recent years.
Recommendations - Report all experimental details, release code and include ablation studies: Henderson et al. (2018) emphasise that for results to be reproducible, it important that papers report all experimental details. This includes hyperparameters, code-level optimisations, tuning procedures, as well as precise description of how the evaluation was performed on both the baseline and novel work. It is also important that code be made available to easily replicate findings and stress test claims of general performance improvements. Furthermore, Engstrom et al. (2020) propose that algorithm designers be more rigorous in their analysis of the effects of individual components and how these impact performance through the use of detailed ablation studies. It is important that researchers practice diligence in attributing how the overall performance of algorithms and their underlying algorithmic behavior are affected by different proposed innovations. implementation details and code-level optimisations. In the light of our above analysis, argue that the situation is no different in MARI, and therefore suggest the field adopt more rigorous reporting and conduct detailed ablation studies of proposed innovations.
2.2 Lesson 2: Use standardised statistical tooling for estimating and reporting uncertainty
in papers do not take into account uncertainly: We have discussed how different implementations of the same algorithm with the same set of hyperparameters can lead to drastically different results. However, even under such high degree of variability, typical methodologies often ignore the uncertainty in their reporting (Colas et al., 2018a, 2019; Jordan et al., 2020; Agarwal et al., 2022). Furthermore, most published results in RL make use of point estimates like the mean or median performance and do not take into account the statistical uncertainty arising from only using a finite number of testing runs. For instance, Agarwal et al. (2022) found that the current norm of using only few runs to evaluate the performance of an RL algorithm is insufficient and does not account for the variability of the point estimate used. Furthermore, Agarwal et al. (2022) also revealed that the manner in which point estimates are chosen varies between authors. This inconsistency invalidates direct comparison between results across papers.
Issue in
The situation in MARL- - A lack of shared standards for uncertainty estimation and concerning omissions in reporting: In Figure 3 (a)-(c), we investigate the use of statistical aggregation and uncertainty quantification methods in MARL. We find considerable variability in the methods used, with little indication of standardisation. Perhaps more concerning is a complete absence of proper uncertainty quantification from one third of published papers. On a more positive note, we observe an upward trend the use of standard deviation as an uncertainty measure recent years, particularly in 2021. Furthermore, it has become fairly standard in MARL to evaluate algorithms at regular intervals during the training phase by conducting certain number of independent evaluation runs in the environment using the current set of learned parameters. This procedure is then followed for each independent training run and results are aggregated to assess final performance. In our analysis, we find that key aspects of this procedure are regularly omitted during reporting, as shown in Figure 3 (d)-(c). Specifically, in (d), we find many papers omit details on the exact evaluation interval used, and in (c), similar trend in omission regarding the exact number of independent evaluation runs used. Finally, in Figure 3 (f), we plot the number of independent runs used during training, showing no clear standard. Given the often high computational requirements of MARL research, it is not surprising that most works opt for a low number of independent training runs, however this remains of concern when making statistical claims.
Recommendations Standardised statistical tooling and uncertainty estimation including detailed reporting: As mentioned before, the computational requirements in MARL research often make it prohibitively difficult to run many independent experiments to properly quantify uncertainty. One approach to make sound statistical analysis more tractable, is to pool results across different tasks

---

(c)
33.8%
Scandial besidd
655 Corfiiin interid
Kode
of
Fie Hamge
20.5%
20
05.0%
13.0%
5-
3,8%
2016
2017
332
80%
lunden
(d)
(e)
2055
Figure 3: Trends performance aggregation, uncertainty quantification and omissions in reporting key axpects of evaluation methodology. (a) Distribution of performance aggregation metrics. None means that the aggregation metric was not specified in the paper. (b) Distribution of uncertainty methods: a% confidence interval (CI), standard deviation (STD) and interquartile range (IQR). (c) Trends in uncertainty quantification over time. (d) Reporting of the evaluation interval used. (e) Reporting on the number of evaluation rums used. (f)
Number of independent runs used in published work.
using the bootstrap (Efron, 1992). In particular, for RL, Agarwal et al. (2022) recommend computing stratified bootstrap confidence intervals, where instead of only using the original set of data to calculate confidence intervals, the data is resampled with replacement from M tasks, each having N runs. This process is repeated as many times as needed approximate the sampling distribution of the statistic being calculated. Furthermore, when making summary of overall performance across tasks it has been shown that the mean and median are insufficient, the former being dominated by outliers and the latter having higher variance. Instead, Agarwal et al. (2022) propose the use of the interquartile mean (IQM) which more robust to outlier scores than the mean and more statistically efficient than the median. Finally, Agarwal et al. (2022) propose the use of probability of improvement scores, sample efficiency curves and performance profiles, which are commonly used to compare the performance of optimization algorithms (Dolan and Moré, 2001). These performance profiles are inherently robust to outliers on both ends of the distribution tails and allow for the comparison of relative performance
glance. In the shared reward setting, where it is only required track single return value, we argue that these tools fit the exact needs cooperative MARL, as they do in RL. Furthermore, in light of our analysis, strongly recommend universal standard in the use, and reporting of, evaluation parameters such as the number of independent runs, evaluation frequency, performance metrics and statistical tooling, make comparisons across different works easier and more fair.
2.3 Lesson 3: Guard against environment misuse and overfitting
Issue in RL Dover-tunin algorithmss to perform well on a specific environment: As early as 2011 issues with evaluation in RL came to the foreground in the form of environment overfitting. Whiteson al. (2011) raised the concern that in the context of RL, researchers might over-tune algorithms to perform well on a specific benchmark at the cost of all other potential use cases. More specifically, Whiteson al. (201 1) define environment overfitting terms of a desired target distribution. When an algorithm performs well in specific environment but lacks performance over target distribution of environments, it can be decmed to have overfit that particular environment.
The situation in MARL One environment to rule them all on the use and misuse of SMAC: The StarCraft multi-agent challenge (SMAC) (Samvelyan et al., 2019) has quickly risen to prominence as key benchmark for MARL research since it's release in 2019, rivaled only in use by the multi-agent particle environment (MPE) introduced by Lowe al. (2017) (see Figure (a)). SMAC and its accompanied MARL framework PyMARL (introduced in the same paper), fulfills several desirable properties for benchmarking: offering multiple maps of various difficulty that test key aspects of MARL algorithms and providing unified API for running baselines as well as state-of- of-the-art algorithms on SMAC. Unfortunately, the wide-spread adoption of SMAC has also caused several issues, relating to environment overfitting and cherry picking of results, putting into question the
5

---

Maznfrsse
40-
14-
MPE
Custom
Number of papoors
12-
Sta coaft
Number
10-
Mall's Games
SMAC
LBF
20
MAMLJOCO
6-
10-
2016
2017
2018
2019
2020
2021
2002
Number ervironments (n.g. SMACIMPE)
(c)
(d)
200
100-
Scenario:
of diparss
-00-
Top win rate (96)
Colectdd e8t
600-
60-
200
-0-
100
20-
1002 Jets
Namber 16 De.. Extoprnnitoo (ma))
scatarbommape
2017
2018
20:0
2020
2021
Year
SIAMG map dfficdlly
Figure Environment popularity, usage trends in papers and potential evidence of overfitting on SMAC. (a) En- vironment adoption over time. (b) Number environments used papers. (c) Number scenarios/tasks/mapp used in papers. (d) Distribution of task difficulty of SMAC maps used in papers. (e) Performance trends on popular SMAC maps: Aggregation is the aggregate function used for the different reported values.
credibility of claims made while using it as benchmark.
To illustrate the above point, we start by highlighting that many MARL papers use only a single environment (e.g. SMAC or MPE) for evaluation, as shown in Figure 4 (b). This is often deemed acceptable since both SMAC and MPE provide many different tasks, maps. For instance, in SMAC, there are 23 different maps providing a wide variety in terms of the number of agents, agent types and game dynamies. However, there no standard, or agreed upon set of maps use for benchmarking novel work, which makes it easy for authors to selectively subsample maps post-experiment based on the outcomes of their proposed algorithm. shown in Figure (c), although environments like SMAC and MPE offer many different testing scenarios, it is typical for papers to only use small number of these in their reported experiments.
To concretely expose the potential danger in this author map selection bias, we redo the original analysis performed by Samvelyan et al. (2019), using the authors' exact experimental data that was made publicly available containing five independent runs for IQL (Tampuu et al., 2015), COMA (Foerster et al., 2018) (). VDN (Sunehag et al., 2017) and QMIX (Rashid al., 2018) on 14 SMAC maps.? The top row of Figure 5 shows the results of this analysis performed using the statistical tools recommended by Agarwal et al. (2022), including the probability of improvement between algorithms. performance profiles and sample efficiency curves. The results support the original claims made by Samvelyan et (2019), namely that QMIX is superior algorithm that of VDN, COMA and IQL. both in terms of performance and sample efficiency. However, by simply sampling smaller set of two easy, medium and hard maps (a common spread in the literature, see Figure 4 (d)), from the original 14, giving 6 maps in total (a reasonable number according to prior work, see Figure 4 (c)), we are able to change the outcome of the analysis in support of no difference in performance between VDN and QMIX, well as finding VDN to be more sample efficient. This is shown in the bottom row of Figure and highlights the danger of a lack of standards regarding which fixed set of maps should be used for benchmarking.
We end our investigation into SMAC (and refer the reader to the Appendix for additional discrepancies uncovered), by looking at historical performance trends. In Figure 4 (e), we show the top win rate achieved by an algorithm in a specific year for 14 of the most popular maps used prior work. We find that by 2021, most of these maps have converged to a win rate close or equal to 100%, while only a few maps are still situated around 80-90%. Given that many of these maps repeatedly feature
We applaud the authors for making their raw evaluation data publicly available. This data can be found here: https://gittbb.cmmmooaiiii/iiii com//owhhi//mmma..cc.........
'The maps chosen were: lc3s5z, 2c 3s_ (42g, bane_ Daan__s__tt sese_ss__bbeee ba_e, MMM2, 10m_vs. I 11m, 27m_ vs_ 30m.
5m, vs. 6m, 2s3z, 2s vs. 1sc, s52, 3s 52, 6h vs. 8z, 3s52, vs. 3s6z and corridor

---

Algorithh
Algorithm
61
0.6
VDN
QMX
00375
ONX
QMOX
COKA
QMIX
COMA
VON
0.50
IQL
QMIX
0.23
0.00
PIX
Tes: biltl
10
0.0
0.5
10
1.00
Algorithm X
Algorithm
COMA
2.5
VDN
QMIX
VIN
VOn
GMIX
0.4
COMA
0.50
IQU
QMIX
0.2
IDL
VON
0.25
10L
COMA
0.0
0.2
0.4
PVX
Y1
0.6
0.5
02
10
Timesteos In millonss
Figure 5: Reanalysis of original SMAC experiments conducted by Samvelyan et al. (2019) including probubility of improvement, performance profiles and sample efficiency curves. Top row: All 14 SMAC maps used in the original analysis. Bottom row: Subset of maps, including casy, 2 medium and 2 hard: $__vs_sc 3s_vs_5z, banc_vs_bant 5m_vs_6m, 6h_vs_8z and corridor
across papers, and will likely be used in future work, begs the question what extent the MARL community has already overfit to SMAC as an evaluation benchmark.
Recommendations Standardised environment sets and testing for generalisation: To solve the issue of enviromment overfitting, Whiteson et al. (2011) propose the use of a generalised evaluation methodology. In this approach, environments (for tuning algorithms) are freely sampled from some generalised environment set. Separately, algorithm evaluation is performed on a second set of sampled environments from the same generalised environment set, acting test set analogous to that used in supervised learning. Recent work in this direction include benchmarks such as Proegen (Cobbe al., 2020), which use procedural generation to implicitly construct distribution from which to sample test tasks. In SMAC and other environments, it is common practice to only evaluate on the exact map the algorithm was trained on, under that exact same conditions, and to not specifically test for generalisation across unseen tasks, However, many MARL algorithms are still highly sensitive to small changes in the environment and often fail to generalise to new unseen tasks (Carion et al., 2019; Zhang et al., 2020; Mahajan et al., 2022). This calls for more work on MARL generalisation and we recommend stronger focus on benchmarks designed specifically to test generaliation. However, has been noted that procedurally generated benchmarks may reduce the precision of research (Kirk et al., 2021), 1), making it more difficult to track progress. Furthermore, MARL exibits several unique and challenging difficulties when it comes to building algorithms able to generalise, likely requiring many years of future work to surmount (Mahajan et al., 2022). Therefore, in certain cases, it might make more sense for researchers to take smaller and more precise steps towards key innovations in algorithm design by still relying on traditional environment sets. In this setting, we strongly advocate using fixed environment sets, where ideally these are selected by the designers of each environment and are accompanied by exact instruction for their configuration, so as to be consistent across papers.
3 Towards standardised evaluation protocol for MARL
In this section, we pool together the observations and recommendations from the previous section to provide standardised performance evaluation protocol for cooperative MARL. We are realistic in our efforts, knowing that a single protocol is unlikely to be applicable all MARL research. However, echoing recent work on evaluation (Ulmer et al., 2022), we stress that many of the issues highlighted previously stem from a lack of standardisation. Therefore, we believe default "off-the-shelff protocol that is able capture most settings, could provide great value to the community. If widely adopted, such a standardised protocol would make it easier and more accurate to compare across different works and remove some of the noise in the signal regarding the true rate of progress in MARL research. A summarised version of our protocol is given in the blue box at the end of this section and concrete demonstration of its usage can be found in the Appendix.
7

---

Benchmarks and Baselines. Before giving details on the proposed protocol, we first briefly comment on benchmarks and baselines used experiments. These choices often depend on the research question of interest, the novel work being proposed and key algorithmic capabilities to be tested. However, as alluded to in our analysis, we recommend that environment designers take full ownership regarding how their environments are to be used for evaluation. For example, if an environment has several available static tasks, the designers should specify a fixed compulsory minimum set for experiments to avoid biased subsampling by authors. It could also be helpful if designers keep track of state-of-the-art (SOTA) performances on tasks from published works and allow authors submit these for vetting. We also strongly recommend using more than single environment (e.g. SMAC) and preferring environments that test generalisation. Regarding baselines, recommend that at minimum the published SOTA contender to novel work should be included. For example, if the novel proposal is a value-based off-policy algorithm for discrete environments, at minimum, it must be compared to the current SOTA value-based off-policy algorithm for discrete environments. Finally, all baselines must be tuned fairly with the same compute budget.
Evaluation parameters. In Figure 6, we show the evaluation parameters for the num- ber of evaluation runs, the evaluation interval (top) and the training time (bottom left) used in papers. We find it is most common to use 32 evaluation episodes at every 10000 timesteps (defined as the steps taken by agents when act- ing in the environment) and to train for million timesteps in total. We note that these numbers are skewed towards earlier years of SMAC evalu- ation and that recent works have since explored far longer training times. However, we argue that these longer training times are not always justified (e.g. see the bottom right of Figure 6). Furthermore, SMAC is one of the most ex- pensive MARL environments (Appendix A.7 Papoudakis et al. (2021)) and for the future accessibility of research in terms of scale and for fair comparisons across different works, we recommend the above commonly used values as reasonable starting defaults and support the view put forward by Dodge et al. (2019) that results should be interpreted as function of the compute budget used. We of course recognise that these evaluation parameters can be very specific to the environment, or task, and again urge environment designers to play a role in helping the community develop and adopt sensible standards.
Eacy 40000100100 1o 32
Dach 10830 for
Elee scceds for
Each 10
100050
Each
20000 Tar 24
Exch = ta 5000
Namberol papert
Figure 6: Top: Evaluation methodologics in papers: how frequent and for how many runs to do evaluation. "E" refers episode and "T" to timestep. Bottom: Number of total training timesteps used in papers (left) and win rates per training timesteps on 2s3z (right).
Performance and uncertainty quantification. To aggregate performance across evaluation intervals, we recommend using the absolute performance metric proposed by Colas et al. (2018b), computed using the best average (over training runs) joint policy found during training. Typically, practitioners checkpoint the best performing policy parameters to use as the final model, therefore it makes sense to do evaluation in a similar way. However, to account for not averaging across different evaluation intervals, Colas et al. (2018b) recommend increasing the number of independent evaluation runs using the best policy by factor of 10 compared to what was used at other intervals. To quantify uncertainty, we recommend using the mean with 95% confidence intervals (CIs) at each evaluation interval (computed over independent evaluation runs), and when aggregating across tasks within an environment, we recommend using the tools proposed by Agarwal et al. (2022), in particular, the inter-quartile mean (IQM) with 95% stratified Bootstrap Cls.
Reporting. We strongly recommend reporting all relevant experimental details including: hyper- parameters, code-level optimisations, computational requirements and frameworks used. Taking inspiration from model cards (Mitchell et al., 2019), we provide templates for reporting in the Ap- pendix. Furthermore, we recommend providing experimental results in multiple formats, including plots and tables per task and environment as well as making raw experimental data and code publicly available for future analysis and easy comparison. Finally, we encourage authors to include detailed ablation studies in their work, so as to be able to accurately attribute sources of improvement.
8

---

Standardised Performance Evaluation Protocol for Cooperative MARI.
Input: Environments with tasks from set 7 Algorithms A. including baselines and novel work.
Evaluation parameters defaults
Number of training timesteps, 7 2 million.
Number independent training runs, R = (from Agarwal al. (2022))
Number of independent evaluation episodes per interval, F 32.
Evaluation intervals, E Z, at every 10000 timestops.
Performance and uncertainty quantification
Performance metric: Always use returns G (applicable all environments), and the environ-
ment specific metric (c.g. Win rate).
Per task evaluation: Compute the mean G7 over E episodes at each evaluation interval where Cg the return of algorithm a on task t, with 95% CI, for a.
3. Per environment evaluation:
Compute the normalised absolute return (Colas et al., 201 as the mean return of 10 x E = 320 evaluation episodes using the best joint policy found during training and normalising the return to be in the range [0, 1] using (G; smin(Ge) (maxx (G.)
task L. (ain(G))((max((t)
min(G.)), where G, is the return for all algorithms
For each algorithm form evaluation matrix with shape (R, [T]) where each entry the normalised absolute return for specific training
specific task.
Compute the IQM and optimality gap with 95% stratified Bootstrap Cls, probability improvement scores and performance profiles, to compare algorithms, using the tools proposed by Agarwal et al. (2022). Sample efficiency curves can computed by using normalised returns at each evaluation interval.
Reporting
Experiments: All hyperparameters, code-level optimisations, computational requirements and framework details.
Plots: task environment evaluations as well as ablation study results.
Tables: Normalised absolute performance per task with 95% CI for all tasks, IQM with 95% stratified Bootstrap Cls per environment for all environments.
Public repository: Raw evaluation data and code implementations.
These can be found rliable library: https:/gillub..oo/gggggereeereee reeerriiiieeeeeerrtttiiiilllee
4 Conclusion and future work
In this work, we argue for the power of standardisation. In a fast-growing field such as MARL, it becomes ever more important be able to dispel illusions of rapid progress, potentially misleading the field and resulting in wasted time and effort. We hope to break the spell by proposing sensible standardised performance evaluation protocol, motivated part by the literature on evaluation in RI., as well as by meta-analysis of prior work in MARI.. If widely adopted, such protocol could make comparisons across different works much faster, easier and more accurate. However, certain aspects of evaluation are better left standardised outside of the control influence of authors, such as protocols pertaining to the use of benchmark environments. We believe this is an overlooked issue and important area for future work by the community, and specifically environment designers, to jointly establish better standards and protocols for evaluation and environment use. Finally, we encourage the community to move beyond the use of only or two environments with static task sets (e.g. MPE and SMAC) and focus more on building algorithms, environments and tools for improving generalisation in MARL.
clear limitation of our work is our focus on the cooperative setting. Interesting works have developed protocols and environments for evaluation in both the competitive and mixed settings (Omidshafici al., 2019; Rowland et al., 2019; Leibo et al., 2021). We find this encouraging and argue for similar efforts in the adoption of proposed standards for evaluation.

---

Acknowledgments and Disclosure of Funding
The authors would like to kindly thank the following people for useful discussions and feedback on this work: Jonathan Shock, Matthew Morris, Claude Formanek, Asad Jeewa, Kale-ab Tessera, Sinda Ben Salem, Khalil Gorsan Mestiri, Chaima Wichka and Sasha Abramowitz.
References
R. Islam, P. Henderson, M. Gomrokchi, and D. Precup, "Reproducibility of benchmarked deep reinforcement learning tasks for continuous control," 2017. [Online]. Available: https//arxiv.or///b///88.... 770.00433
M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, and M. Bowling, "Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents," 2017. [Online]. Available: https//aaxivorr//bb///00... 709.0//0
Henderson, "Reproducibility and reusability in deep reinforcement learning," Master's thesis, McGill University, 2018.
A. Zhang, N. Ballas, and J. Pineau, "A dissection of overfitting and generalization in continuous reinforcement learning," 2018. [Online]. Available: https://axxivorg/abb// 806.07937
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, "Deep Reinforcement Learning that Matters," " 2018.
C. Colas, O. Sigaud, and P.-Y. Oudeyer, "How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments," 2018.
"A Hitchhiker's Guide Statistical Comparisons of Reinforcement Learning Algorithms," 2019.
S. C. Y. Chan, S. Fishman, J. Canny, A. Korattikara, and S. Guadarrama, "Measuring the Reliability of Reinforcement Learning Algorithms," 2020.
M. Jordan, Y. Chandak, D. Cohen, M. Zhang, and P. S. Thomas, "Evaluating the Performance of Reinforcement Learning Algorithms," 2020.
L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry, "Implementation matters in deep policy gradients: A case study on ppo and trpo." 2020.
R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare, "Deep Reinforcement Learning at the Edge of the Statistical Precipice," 2022.
J. N. Foerster, "Deep multi-agent reinforcement learning," Ph.D. dissertation, University of Oxford, 2018.
A. Oroojlooyjadid and D. Hajinezhad, "A review of cooperative multi-agent deep reinforcement learning," CoRR, vol. abs/1908.03963, 2019. [Online]. Available: http://arxiv.org//bs//// http://wwww.rrr/a///0000000666
Y. Yang and J. Wang, "An Overview of Multi-Agent Reinforcement Learning from Game Theoretical Perspective," CoRR, vol. abs/2011.00583, 2020. [Online]. Available: https:/arxxiv.or//bb//// 1.00583
C. Yu, A. Velu, E. Vinitsky, Y. Wang, A. M. Bayen, and Y. Wu, "The surprising effectiveness of MAPPO in cooperative, multi-agent games." CORR, vol. abs/2103.01955, 2021. [Online]. Available: https//anxivoor/bb///0030.01111
G. Papoudakis, F. Christianos, I. Schäfer, and S. V. Albrecht, "Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks," in Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2021. [Online]. Available: http://axiv.org/abb///o0/ 006696
Hu, S. Jiang, S. A. Harding, H. Wu, and wei Liao, "Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning." 2022.
10

---

M. Andrychowicz, A. Raichuk, P. Stariczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski et al., "What matters in on-policy reinforcement learning? a large-scale empirical study," arXiv preprint arXiv:2006.05990, 2020.
Hu, S. Jiang, S. A. Harding, H. Wu, and wei Liao, "Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning," 2021.
Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson, "Qmix: Mono- tonic value function factorisation for deep multi-agent reinforcement learning," in International Conference on Machine Learning. PMLR, 2018, pp. 4295-43044
M. Samvelyan, T. Rashid, C. S. de Witt, G. Farguhar, N. Nardelli, T. G. Rudner, C.-M. Hung, P. H. S. Torr, Foerster, and S. Whiteson, "The StarCraft Multi-Agent Challenge," " CoRR, vol. 0oss/192..4044 2019.
B. Efron, Bootstrap Methods: Another Look at the Jackknife. New York, NY: Springer New York, 1992, pp. 569-593. [Online]. Available: https://doi.org/100 10000/977---4111-45000001
E. D. Dolan and J. J. Moré, "Benchmarking optimization software with performance profiles," 2001. [Online]. Available: https://axiv..og//abscc/00000000
S. Whiteson, B. Tanner, M. E. Taylor, and P. Stone, "Protecting against evaluation overfitting in empirical reinforcement learning." in 2011 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL). IEEE, 2011, pp. 120-127.
R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and Mordatch, "Multi-agent actor-critic for mixed cooperative-compttttte environments," Advances in neural information processing systems, vol. 30, 2017.
A. Tampuu, T. Matüisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente, "Multiagent cooperation and competition with deep reinforcement learning," 2015.
J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, "Counterfactual multi-agent policy gradients," Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018.
P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Son- nerat, J. Z. Leibo, K. Tuyls et al., "Value-decoomoostion networks for cooperative multi-agent 2017.
learning," arXiv preprint
K. Cobbe, C. Hesse, J. Hilton, and J. Schulman, "Leveraging procedural generation to benchmark reinforcement learning," in International conference machine learning. PMLR, 2020, pp. 0048-2056..
N. Carion, N. Usunier, G. Synnaeve, and A. Lazaric, "A structured prediction approach for gen- eralization in cooperative multi-agent reinforcement learning." Advances in neural information processing systems, vol. 32, 2019.
T. Zhang, H. Xu, X. Wang, Y. Wu, K. Keutzer, J. E. Gonzalez, and Y. Tian, "Multi-agent collaboration via reward attribution decomposition." arXiv preprint arXiv:2010008531,, 2020.
A. Mahajan, M. Samvelyan, T. Gupta, B. Ellis, M. Sun, T. Rocktäschel, and S. Whiteson, "General- ization cooperative multi-agent systems," arXiv preprint arXhv:2202.00104, 2022.
R. Kirk, A. Zhang, E. Grefenstette, and T. Rocktäschel, "A survey of generalisation in deep reinforce- ment learning." arXiv preprint arXiv:2111.0979. 2021.
D. Ulmer, E. Bassignana, M.
D. Varab, M. Zhang, C. Hardmeier, and B. Plank, "Experimental standards for deep learning research: A natural language processing perspective." arXiv preprint 2:Xiv:2204.0625). 2022.
Dodge, Gunurangan, D. Card, R. Schwartz, and N. Smith, "Show your work: Improved reporting of experimental results." arXiv preprint 1rXiv:1909.03000 2019.
11

---

C. Colas, O. Sigaud, and P.-Y. Oudeyer, "Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms." International conference machine learning. PMLR. 2018, pp. 1039-1048.
M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and Gebru, "Model cards for model reporting," " in Proceedings of the conference on fairness, accountability, and transparency, 2019, pp. 220-229.
S. Omidshafiei, C. Papadimitriou, G. Piliouras, K. Tuyls, M. Rowland, J.-B. Lespiau, W. M. Czamecki, M. Lanctot, J. Perolat, and R. Munos, "a-rank: Multi-agent evaluation by evolution," Scientific reports, vol. 9, no. pp. 1-29, 2019.
M. Rowland, S. Omidshafiei, K. Tuyls, Perolat, M. Valko, G. Piliouras, and R. Munos, "Multiagent evaluation under incomplete information." 2019. [Online]. Available: hitps: /arxiv.org/abs/1900 wttps//////99999999999
Z. Leibo, Duéñez-Guzmin A. Vezhnevets, J. P. Agapiou, P. Sunehag, R. Koster, Matyas, C. Beattie, I. Mordatch, and T. Graepel, "Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot," 2021.
12


Reviews:
[{"review_id": "t36P4KgVgYz", "reviewer": "Reviewer_F87n", "summary": "\nTowards a Standardized Performance Evaluation Protocol for Cooperative MARL\nIn this paper, the authors review existing papers on multi-agent reinforcement learning, and show three lessons from these papers: 1) we need to study what exactly makes the improvement and don’t hide details; 2) we need to have standardized results reporting and uncertainty analysis; 3) we need to avoid exploiting the environments and overfitting it.\nAnd authors also propose a standard evaluation protocol for future MARL research.\n", "questions": "1) What does the author believe are the problems in multiagent RL that are different or unique as opposed to the ones also commonly seen in general RL research?\n2) What do the authors propose to help all the researchers to follow the correct protocol and practices in RL? Is there a code-base or website MARL researchers could use for MA benchmarks?\n3) There’s a very related paper here [4] (Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning). This is a paper that was available very recently, but the author might want to cite it as a contemporary work in the final version.\n\n\n", "limitations": "\nI don’t find anything that would potentially have a negative societal impact.\n\n[1] Agarwal, Rishabh, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc Bellemare. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021): 29304-29320.\n[2] Henderson, Peter, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. \"Deep reinforcement learning that matters.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1. 2018.\n[3] Whiteson, Shimon, Brian Tanner, Matthew E. Taylor, and Peter Stone. \"Protecting against evaluation overfitting in empirical reinforcement learning.\" In 2011 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pp. 120-127. IEEE, 2011.\n[4] Fu, Wei, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. \"Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning.\" arXiv preprint arXiv:2206.07505 (2022).\n\n", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nOriginality: The paper discusses a very important topic in multi-agent RL that draws more and more attention recently. Some of the findings, for example the inconsistent reported performance, hidden performance tricks under the sleeves, improper handling of results reporting and missing uncertainty evaluations, have been already discovered as problems in general RL research. That being said, a lot of the findings have already been discovered in Agarwal et al. (2022), Whiteson et al. (2011), Henderson et al. (2018).\n\nOn the bright side, the paper gives a very good systematic review on multi-agent reinforcement learning algorithms. Figures such as Figure 2, Figure 4 are quite inspiring and the discussion about QMIX, VDN, COMA and IQL give quite valuable information about the actual performance of these algorithms.\nThe paper also provides a concluding protocol suggestion, which will benefit the community.\n\n\nQuality and clarity: The paper is well written, supported with informative figures and plots.\nEach section has a very clear topic and the authors provide detailed explanations for every proposal in the paper.\n\nSignificance: A lot of the problems have been discussed by other related papers, which potentially will make the impact of this paper less. However the proposed protocol is beneficial to the multi-agent RL community.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YwaNFgPdX0c", "reviewer": "Reviewer_hmEq", "summary": "This paper's main contribution is the proposed standardized performance evaluation protocol for cooperative MARL. Specifically, this paper performs meta-analyses and identifies the following potential issues in related works:\n1. Lack of experimental details, open-sourced code, and ablation studies (Section 2.1)\n2. Lack of standardized statistical tools and uncertainty estimation (Section 2.2)\n3. Lack of standardized environment sets for preventing domain overfitting (Section 2.3)\n\nBased on these observations, this paper provides the standardized evaluation protocol for cooperative settings in Section 3, including benchmark/baseline, evaluation parameters, statistical quantification, and reporting.", "questions": "1. (Related to Weakness) I understand that extending to general-sum/competitive settings is a future work, but I hope to ask for a clarification on why this paper performed meta-analyses limited to cooperative settings. In particular, the main differences between RL and MARL result from the fact that there are multiple learning agents with non-stationary policies. Hence, establishing the standard protocol for general-sum and/or competitive games is an open question because it is often not clear when the learning should end under non-stationarity compared to cooperative settings. As noted in the paper, there are close similarities between RL and cooperative MARL evaluation, and the related works of Henderson (2018) and Agarwal (2021) focus on single-agent settings. As such, if this paper could provide a general protocol that includes general-sum and/or competitive settings, the contribution could have been more significant.\n2. In Figure 3(f), there might be a typo in the x-axis label: \"Year\" to \"Number of independent runs\"", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The performed meta-analyses provide interesting perspectives on recent trends in cooperative MARL, such as types of MARL algorithms used in papers, ablation studies, and types of performed uncertainty quantification.\n2. In general, I agree with the main points of this paper that MARL may lack standardization and thus establishing the standard protocol is important.\n3. The paper is written well and conveys the main points clearly.\n\n**Weakness:**\nWhile I agree with the main points, my main concern is the novelty of this paper compared to related works. In particular, the importance of reporting experimental details (Section 2.1) and computing uncertainty estimation (Section 2.2) is already discussed in prior works (Henderson, 2018; Agarwal, 2021), so these aspects have been considered in recent MARL papers. Compared to these works, the domain overfitting issue discussed in Section 2.3 adds a new perspective to the MARL community, but I am unsure whether this perspective alone has sufficient novelty for acceptance.\n\n**References:**\nHenderson, et al., Deep Reinforcement Learning that Matters, AAAI 2018\nAgarwal, et al., Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CtSoCqWAua", "reviewer": "Reviewer_qdGs", "summary": "This paper is a interesting meta-research of cooperative MARL methods. The authors claim at least three major problems in current MARL research: report of implementation and evaluation details, report of uncertainty with proper statistical tool, the choice of benchmark environments. The paper tries to appeal for a standard protocol for MARL experiments, which should be encouraged for benefiting the community. ", "questions": "For ablation study analysis in Sec. 2.1, it seems only the number of papers using ablation study are discussed. Is there any deeper investigation of the number and types of ablation study within each paper, and what should be a standard for ablation study?\n\nThe results in Fig.4 (e) show that some methods even have a decreasing performance along the time. Does that mean the later work does not report the performance as good as the original paper claims?", "limitations": "As mentioned in the weakness, more insights of problems specific for MARL settings are interesting to know, apart from the commonly known single-agent RL problems.\n\nAlthough the authors propose a performance evaluation protocol for cooperative MARL in Sec. 3, some other problems raised by authors are not concretely solved, like how to choose a standard set of environments for different MARL settings, etc. I understand this problem may be hard to give an answer and the authors may want to receive more feedback from the community. I think if this paper is the one trying to establish the protocol for MARL experiments, these problems are better solved with concrete solutions even if in the follow-up works. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper conduct heavy statistical analysis of previous MARL algorithms, which are reported in the diagrams. The results are clear and thorough. The efforts of the authors should be appreciated. The description of the problems are clear. More importantly, recommendations for solving the problems are provided after each section.\n\nOne major weak side of the work is that it seems all three aspects of lessons are inherited from single-agent RL, with no additional components specifically caused by cooperative MARL settings. The problems of reporting implementation details, statistical uncertainty estimation and choice of environments are existing problems in single-agent RL, which makes this work looks like a replicate of commonly known problems in single-agent RL to MARL domain. It could be more interesting to see at least one or two more lessons specifically considered in MARL setting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5Se4RXhiJHs", "reviewer": "Reviewer_8mDm", "summary": "This paper reveals the trends and worrying issues of evaluation via an extensive study of recent MARL works. Several concerns are raised regarding inconsistent results, poor statistical aggregation, and environmental misuse.  Based on the analysis, the authors propose several recommendations and a standardized protocol, specifying the methodologies of how evaluation parameters selection, uncertainty estimation, and reporting should be performed.", "questions": "+ In L280, by using the term \"timesteps\", do the authors mean environment frames or training steps? What is the batch size and the frequency of updating parameters?\n+ L191: \"SMAC and it's\" -> \"SMAC and its\"\n+ L289: please use \\citet", "limitations": "I don't know much about the works investigating the evaluation protocol in single-agent RL. It seems that the authors just translate the designed protocols into the domain of cooperative MARL. However, as the authors mentioned in the introduction section, the contribution of this paper is not an innovation of evaluation protocol. This limitation can be minor.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n+ The logic of this paper is easy to follow, and the meta-analysis is comprehensive.\n\n+ The summary and proposal of a standardized evaluation protocol is valuable.\n\nAs a practitioner, I quite understand that reproducibility is a huge problem in recent MARL works. It is worthwhile (however difficult) to summarize the evaluation methods applied in these works. I really appreciate the efforts of the authors and hope the standardized protocol could be utilized by researchers in the field.\n\n## Weaknesses\n\nI don't find any significant weaknesses in this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept