Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work does a benchmark evaluation of multi-agent RL methods across a number of tasks.
strengths: The evaluations are thorough and it's nice that a confidence interval is computed for the evaluations.
weaknesses: None that stand out.
correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: N/A
review_title: An exhaustive evaluation of Multi-Agent RL Methods

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper makes several contributions.
Firstly, it includes an extension of PyMARL to handle more algorithms and allow tuning more details of those algorithms.
Secondly, it includes two new MARL environments.
Thirdly, it includes experimental results comparing most of the commonly used algorithms in this field, as well as analysis of those algorithms' performances. 
The resulting experiments show that the new environments provide unique challenges compared to previously available benchmarks.
A hyper-parameter search was used to tune the algorithms for each benchmark, and statistical analyses are used to indicate which found results were significant.
strengths: The first major strength of this paper is that it contains multiple useful tools, which are all reasonably licensed and packaged.
The experiments are also strong, since they used an appropriate hyper-parameter search, statistical analysis, and explanations for differences found between algorithms.
weaknesses: The only notable limitations of this work is the depth of analysis provided for any particular algorithm. However, given the scope of the paper, this is completely understandable.
correctness: The claims in this submission appear to be correct. The evaluation and experiment design are appropriate.
clarity: The paper is clear and well written.
relation to prior work: The clearly indicate what work is new work of their own. However, I'm not sufficiently familiar with MARL to comment on its coverage of prior work.
additional feedback: I have no additional feedback or respond to provide.
review_title: The new environments look useful, the libraries are well packaged, and the experiments appear useful and trustworthy.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper proposes a suite of environments (three preexisting, two novel) for cooperative MARL. It then evaluates a set of popular MARL algorithms across the suite, and draws tentative conclusions regarding the value of key algorithm design axes such as centralised vs. independent training, VDN vs. QMIX, etc.
strengths: [S1] Extensive comparison. The number of compared algorithms and tasks is around the limit of what it is possible to compare in one paper with academic-level compute. The selection of algorithms and tasks also covers several important axes of variation for MARL, such as PG vs. TD, centralised vs. decentralised training, full vs. partial observability, etc.

[S2] The benchmark does in fact provide some generalisable insights in Section 6. None of these felt extremely surprising (e.g. indep. RL often works surprisingly well; centralised VFs help under partial observability), but there's value in providing empirically rigorous evidence for them nonetheless.

[S3] Writing is clear, hyperparametrs and experiment details were fairly thorough.
weaknesses: [W1] The proposed benchmark suite is mostly a collection of existing environments and re-implementations of existing algorithms (modulo the newly-released LBF benchmark, and the completely new RWARE benchmark). It therefore has limited novelty—I assume most of the value of the "benchmark" is in having a predefined suite for testing on in future?

[W2] The paper has an unusual mix of contributions, and it's not clear which are meant to be the primary ones. Is it meant to be an experimental paper that provides standalone insights, or a paper that proposes a new benchmark to beat? If the former, the findings seem somewhat underwhelming on their own. If the latter, how easy is it to run all the proposed algorithms on all the proposed environments? What about adding new algorithms or environments to the bake-offf? It seems like the environment code is split across a few different repositories, which might make this harder.
correctness: [CO1] In Table 1, scores on the matrix games seem to be maxed out almost uniformly across all algorithms. This suggests that they are not challenging enough. It may be worth decreasing the number of time steps to make the benchmark more informative.
clarity: [CL1] What are the "cooperation settings" referred to on L154?

[CL2] How are returns being normalised in these figures?
relation to prior work: This is mostly a compilation of prior work into a new suite. Existing discussion therefore seems adequate.
additional feedback: Currently leaning accept on this. Results are not earth-shatting and the contributions are somewhat incremental in nature (a new environment here, an extension of an existing codebase there). However, the benchmark was well-executed, and I think this kind of work is valuable for creating common knowledge in the field about what approaches do or don't work.

------

(review last checked/updated 2021-07-19)
review_title: Review of cooperative MARL benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper develops a set of evaluations and comparisons of multi-agent RL algorithms. They also extend a previous codebase with a few additional algorithms and a redesigned api (for example allowing parameter sharing), and new environments. 

The paper considers three algorithms for independent learning, centralized training decentralized execution, and value decomposition. There are 25 different learning tasks that are cooperative in the sense of shared reward signals; there is a taxonomy of difficulties in environments. Two matrix games follow previous work, and two new environments. 

===
Thanks to the authors for the clarifications; I've raised my score. 
strengths: The paper considers an empirical evaluation of multi agent deep RL algorithms of different types and compares their performance on different environments; two of which are new to this paper or seem to have appeared previously in the literature but are newly integrated in a codebase with others. Given the complexity of the task, I find the first half of the paper nicely clear as a summary of differences between the algorithms and environments. 

The paper's strengths in comprehensiveness lead naturally to its weaknesses in drawing structured conclusions from the empirical evaluation and emphasizing fundamental algorithmic or environment differences related to the math, rather than the phenomenology of specific simulations or environments. 
weaknesses: Evaluation protocol 4.1: a bit more detail could be helpful here. 
Section 4.3, performance metrics: 
what about AUC; learning curves, or other summaries besides maximum and average requirements? 
Fig. 1 is impossibly difficult to read. At least the legend could be made larger so it is legible. 
Similarly, table 3 is also extremely difficult to read. Can the information be summarized further for the main text? 
Statistical tests of fig. 5: multiple testing adjustments? 
Section 5.3 parameter sharing: 
Can you include more details on how the parameters were shared in the main text, how the update rules were coordinated and so on, concurrency and so on? 

Overall: It could have been helpful to have more detailed comparisons of algorithms from the point of view of aspects of cooperation of the different environments. Some of this is gestured towards, e.g. lines 324-325 but this kind of structured comparison is most helpful. 
Similarly section 332-348 would benefit from not just "phenomenological descriptions" of what happens in the experiments, e.g. " so that agents are not stuck in the narrow passages" but maybe with more specific descriptions of algorithm internals or how different algorithms succeed at these types of things. 

Overall I appreciate the comprehensiveness of the evaluation but it is difficult for an outsider to glean conclusions from the extensive evaluation; more structured comparison, or organization in conclusions could be helpful here to draw on the extensive work done to provide crisper recommendations. 
correctness: To the best of my knowledge. There could be improvements with regards to statistical testing and more information on empirical performance evaluation. 
clarity: First half is more clear than the second half. 
relation to prior work: This paper seems to be a synthesis paper, e.g. building on prior work; while I appreciate the clarity of the summary, it may be clearer to precisely delineate what differs. 
additional feedback: The paper could be improved by more structured comparison of conclusions (I am not an expert on multi-agent RL so perhaps this would be more self-evident to someone working in the field).
review_title: review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: - Efforts towards benchmarking Multi Agent Deep Reinforcement Learning (MARL)
- Compare and contrasts 3 different classes of MARL on proposed benchmarks 
- Open source codebase with new environments
strengths: - Comprehensive overview of the problem space and existing algorithms 
- Open source codebase 
- Well timed paper consolidating progress in a rapidly evolving field to facilitate fair comparisons
- A good collection of task suits from different environments
- Authors enlist their computational foot print

weaknesses: - My primary concern with the paper its positioning as a benchmarks paper. The paper to a large extent is a literature review summarizing ideas from the field into a single space. Both task (except two) and algorithms are rehash of previously published works. I'm fairly convinced about the contributions of the paper but I'm conflicted if this should instead be restructured as a review paper. Reason being -- review paper is underlined as a summary of other's contributions, instead of authors' contributions.
- Authors do propose the RWARE environments but Im not convinced if that is contribution enough to warrant a publication. Additionally  there is less evidence / experiments to support if these new environments are well structured, calibrated, and indeed useful to the community at large.
- Section 4.1: Under the premise of a fast simulator, authors propose adjustment for on-policy and off-policy methods. The goal of algorithmic developments are to make progress in real world where cost of sample acquisition is significant. (even for sim2real paradigm, compute cost is significant for non-grid world problems). I'm not convinced that this adjustment is adequate and justified.
- Authors largely rehash existing ideas of the field but miss on the opportunity to cater to the future needs (see Additional Feedback section for details)
correctness: Authors present data and analysis to substantiate their claims 
clarity: Paper is easy to follow.
relation to prior work: Authors are candid about relation to prior works. They appropriately point relevance and contrast their contributions to existing literature.
additional feedback: While the broader field of Robotics and Deep RL have seen significant success in the complexity of tasks that can be solved, MARL seems to be suck in grid world and simple settings. Real world is neither of those. Real world problems are complex with various nuances that are being ignored. Authors are very positioned to design a benchmark for the future needs of the MARL field. A few suggestions below

- Move beyond grid world and matrix games settings
- Device problems using real world data -- e.g. financial-markets 
- Introduce heterogenous players -- players in a team rarely have same responsibilities and rewards
- Introduce co-operative and competitive settings
- Introduce continuous control problem -- Mujoco problems in openAIGym, dm_control [Emergent Complexity via Multi-Agent Competition. Trapit et al.]
- Team sports are great settings - e.g. football [From Motor Control to Team Play in Simulated Humanoid Football. siqi et al.]
review_title: A review paper or a benchmarking paper?

Meta-review: This paper has multiple contributions. Firstly it extends the existing PyMARL, which is called as EPyMARL (Extended
PyMARL) to implement more algorithms and allow for flexible configuration of different implementation details. Second, the paper implemented and open-sourced two new multi-agent environments: Level-Based Foraging (LBF) and Multi-Robot Warehouse (RWARE). Lastly it provides an extensive study of different MARL algorithms on various MARL environments.

The paper focuses on addressing important issue in MARL: the lack of clear benchmark and evaluation of the MARL agents across a diverse set of MARL environments. The reviewers in general found the contributions of the paper meaningful. The authors did a good job in the rebuttal. Thus, this paper deserves to be presented as a poster at NeurIPS 2021 Datasets and Benchmarks Track.
Decision: Accept