Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors propose a comprehensive MARL algorithm library (MARLlib) for solving multi-agent problems. MARLlib manages to unify tens of algorithms, including different types of independent learning, centralized critic, and value decomposition methods. And MARLlib goes beyond current work by integrating diverse environment interfaces and providing flexible parameter sharing strategies
strength_and_weaknesses: Strengths:
1.	An integrated library suite for MARL algorithms, Multi-Agent RLlib (MARLlib), is introduced.
2.	MARLlib unifies diverse environment interfaces with a newly proposed Gym-style interface.
3.	Open source codebase
Weaknesses:
1.	For example, in the 3s_vs_5z map of SMAC, why the performance conducted in MARLib has an obvious different from EPyMARL?  
2.	For other environments (MPE, GRF, MAMuJoCo), why the performances of MARLlib are only included?
3.	Author may clearly illustrate the benefits obtained from agent-level distributed dataflow, a unified agent-environment interface, and effective policy mapping in terms of implementations or experiments.
4.	Some statements, like “value iteration used by VDN and QMIX prefers a dense reward function”, lack solid explanations. 
clarity,_quality,_novelty_and_reproducibility: Original work.
summary_of_the_review: In this paper, authors propose a comprehensive MARL algorithm library (MARLlib) for solving multi-agent problems. MARLlib manages to unify tens of algorithms, including different types of independent learning, centralized critic, and value decomposition methods. And MARLlib goes beyond current work by integrating diverse environment interfaces and providing flexible parameter sharing strategies
Strengths:
1.	An integrated library suite for MARL algorithms, Multi-Agent RLlib (MARLlib), is introduced.
2.	MARLlib unifies diverse environment interfaces with a newly proposed Gym-style interface.
3.	Open source codebase
Weaknesses:
1.	For example, in the 3s_vs_5z map of SMAC, why the performance conducted in MARLib has an obvious different from EPyMARL?  
2.	For other environments (MPE, GRF, MAMuJoCo), why the performances of MARLlib are only included?
3.	Author may clearly illustrate the benefits obtained from agent-level distributed dataflow, a unified agent-environment interface, and effective policy mapping in terms of implementations or experiments.
4.	Some statements, like “value iteration used by VDN and QMIX prefers a dense reward function”, lack solid explanations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a unified coding framework for MARL based on RLlib and Ray. The frame support a relatively large number of different algorithms (of different types), and a relatively large number of MARL environments.
strength_and_weaknesses: **Strength**
- This work implements a relatively large number of MARL algorithms on multiple environments. This is useful for researchers to compare new algorithms to existing baselines.
- Extensive evaluation is conducted for the proposed framework
- The proposed framework integrates a large number of environments

**Weaknesses and Questions**
- As a unified framework to support the future development of MARL, the correctness of algorithms is important. For example, in Table 2, for SMAC 3s_vs_5z and IPPO algorithm, it seems the performance difference between your framework and EPyMARL is relatively big. Why is there such a difference?
- As a unified frame, how does your framework compare to other frameworks in terms of performance? It would be good to have some evaluation results (e.g. training clock time, memory usage, etc) both in the single machine case and distributed case.
- The original RLlib also has some support for MARL, how does your implementation differ from it? Providing more discussion in terms of your contributions and difference can help the reader better understand the framework.
- Extensibility is also important for a unified framework, how does your framework support future development of new MARL algorithms? Do you provide API, methods, or instructions that can make future development MARL algorithm easy?


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is easy to follow.
Novelty: The paper is somewhat novel, but the novelty should be further emphasized 
Reproducibility: the well-implemented code is provided, so it should be reproducible.
summary_of_the_review: Overall, this work can be useful for the MARL community, but the contribution should be clarified and concerns mentioned above should be considered.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduced a new development of a multi-agent reinforcement learning platform based on RLlib to support cutting-edge research on multi-agent reinforcement learning technologies. This paper introduced several key platform design ideas that help to build a flexible and general platform for developing and experimenting multi-agent reinforcement learning algorithms.
strength_and_weaknesses: Strength:
With the increasing popularity of multi-agent reinforcement learning, it is important to develop a reliable and high-quality multi-agent reinforcement learning platform to foster fruitful future research in this booming field. This paper introduced such a platform with some interesting system design ideas, such as agent-level distributed dataflow that unifies diverse learning paradigms. Experiment results show that this platform provides high-quality implementation of many existing multi-agent reinforcement learning algorithms and can be very useful for researchers who are interested in developing and using multi-agent reinforcement learning technologies.

Weakness:
Many libraries have been developed to support single-agent reinforcement learning. It is not clear why the authors chose to use RLlib as the basis for developing their new multi-agent reinforcement learning platform. Is it possible for the multi-agent platform to support multiple existing reinforcement learning libraries? This might be helpful to satisfy researchers' diverse preferences over reinforcement learning libraries.

To determine the overall usefulness of the newly developed multi-agent platform, it might be important to understand which computing platform and infrastructure it supports. Furthermore, it is also interesting and important to know whether the new platform supports distributed deployment for many practical applications. Can the platform support scalable learning through large-scale parallel processing? Is the platform compatible with (or support) any publicly accessible parallel computing services and facilities?

For any platform to be successful, it needs to meet certain efficiency requirements. While the authors evaluated the effectiveness of several multi-agent reinforcement learning algorithms that are supported by the new platform, in terms of computation time and computation resources required, it is not clear whether the new platform is more competitive than any existing platforms.

Another major concern is the user experience. I understand that the new platform supports some ease-of-use features. For example, users can configure several aspects of their multi-agent reinforcement learning pipeline. In line with this feature, I was wondering whether the platform also allows users to configure certain details regarding experiment deployment, experiment progress monitoring, result collection and reporting. Perhaps a case study can also be carried out to demonstrate how the new platform can help researchers develop and evaluate their new multi-agent reinforcement learning algorithms. Additionally, the extensibility of the new platform may need to be examined in more depth.

In line with the above, the robustness of the new platform implementation may also need to be verified and reported in the paper. For example, it might be interesting to find out how likely the new platform may run into any errors (for example errors due to instability of numerical calculation) while conducting a large-scale experiment.

According to my understanding, the new platform does not support some algorithm-specific tricks. While this is understandable to a certain extent, it remains unclear about (1) whether users can easily implement and support any tricks while conducting experiments using the new platform and (2) due to the absence of algorithm-specific tricks, how big the performance difference could be between the results published by the respective algorithm inventors and the results obtained from using the new platform. Moreover, the paper only compared the performance of several algorithms by using two alternative multi-agent reinforcement learning platforms. As only four random seeds are used and the performance differences cannot be verified statistically, it remains unclear whether the new platform indeed provides high-quality implementations of all the multi-agent reinforcement learning algorithms it supports.

Perhaps the authors can comment more about documentation support for the new platform. The usefulness of any software platform depends on its documentation support. The authors may want to discuss how their documentation can help researchers quickly learn their platform. What kind of background knowledge researchers must have in order to understand the documentation? This question can be clarified in the paper too.
clarity,_quality,_novelty_and_reproducibility: The paper is in general well-written and easy to follow.

My biggest concern is regarding the reproducibility. In fact, I spent sometime to try to understand the new library based on the provided link to the source code and online documentation. The library does not seem to work well. I experienced a lot of errors while trying to test some multi-agent reinforcement learning algorithms supported by the library.

Below I list some errors I found during my trial runs:

Change pip install gym==1.21.0 to pip install gym==0.21.0 in the readme file.

The icecream library is not included in the pip list but is used in centralized_critic_hetero.py and happo.py

The command on running an example on SMAC --algo_config=MAPPO should be --algo_config=mappo

When we do python add_patch.py -y -p, Pommerman must be installed since it is referenced somewhere in the library.

The given example in the readme is not working. I tried other examples, none of them seem to work properly.

Additionally, I found that many algorithms claimed to be supported by the new library were already implemented/supported by RLlib and Ray. For the newly implemented algorithms in the library, they do not seem work without errors.
summary_of_the_review: The paper considered an important problem of building a flexible and extensible platform for multi-agent reinforcement learning. While building such a platform clearly has significant research values, I have major concerns regarding the reproducibility, the quality of the code and the correctness/clarity of the online documentation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a library - MARLlib - to run multi-agent reinforcement learning experiments with different environments, task settings (cooperative, competitive, or mixed), and algorithms. The core contribution is defining how observations, predictions, and other data are shared among agents that are otherwise distributed during training. The structure defined is flexible enough to support existing MARL algorithms that vary in using centralized critics or mixing functions for value decomposition and supports environments that use synchronous or asynchronous updates. In practice the interface modifies the standard gym definition to accommodate per-agent data and uses RLlib to provide parameter sharing functionality. Experiments include a comparison with EPyMARL algorithm implementations in SMAC and include results on several other MARL benchmarks (MPE, GRF, MAMuJoCo).
strength_and_weaknesses: ## Strengths

The main strengths of the paper:
1. Algorithm versatility. The parameter sharing implementation and data flow structure supports a meaningful subset of MARL implementations.
2. Environment breadth. Including cooperative, competitive, and mixed scenarios facilitates a wide variety of research. This may also encourage further development of algorithms that perform well across all contexts and deeper analysis of how task vs environment structure influence algorithm performance.


## Weaknesses

1. No evidence on runtime performance. How fast and scalable is MARLlib compared to alternatives? Can researchers quickly run experiments given typical hardware? How does the throughput vary with number of agents? How close are the algorithm implementations to their competitors in terms of runtime efficiency? Clarifying runtime characteristics would strengthen the case that MARLlib is a useful addition to the tools available for running MARL experiments.
2. Limited evidence compared to existing algorithms. The only comparison made is to EPyMARL, with no reference baselines provided for other environment algorithms.
3. Lack of baseline comparisons. The configurations used remove many "tricks" applied in other algorithms. It is hard to gauge if the implementations are correct due to the changes needed to fit the config files. It would help to include reference to previously reported performance numbers for comparison.


## Feedback & Questions

Table 2 would benefit from reporting baseline performance results from other implementations for comparison.

Runtime performance merits a discussion. Specifically addressing scaling in terms of memory and runtime, when varying number of agents, and so on. This would benefit from comparisons to existing methods.

Some questions:
- Can MARLlib support agents that have different lifespans (not around entire task)? That is, tasks where agents "die" during the task.
- What are the "virtual" vs "physical" policies mentioned in section 3.3? Does physical mean the data structures used?
- How can developers/users add "tricks" to their algorithms that are not already provided by the configuration?
clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is readable. Not exceptionally clear but the text was not confusing or hard to follow.
- Quality: Effectively address the problem of a unified algorithm and environment interface for MARL (at least this definition of unified).
- Originality: Incrementally extends efforts at general MARL interfaces by explicitly supporting diverse task settings.
- Reproducibility: Looks very good. Code repository is included and all results are hosted there as well.
summary_of_the_review: The contributions of this paper come from unifying existing approaches and the novelty of the formulation of that unification. From a technical standpoint the novelty is a distributed dataflow formulation to enable distributed agent learning through decoupling agents from each other during optimization. This is a useful insight, but it is not clear it unifies all the existing MARL approaches nor does the paper explore what new algorithms may be facilitated. The main contribution is the experiment platform provided, more than the insights facilitated by its conceptual foundation.

The empirical results lack baselines in most cases. Thus, it is hard to tell if the implementations are "correct" relative to previously published results. The results do not claim any novelty and are intended to demonstrate comparability instead. Without proper baselines it is only possible to assess that EPyMARL and MARLlib offer roughly similar agent training convergence in many, but not all, cases (ex: IQL and MAA2C in 3s_vs_5z do substantially worse). No empirical results offer insight into the runtime performance of the platform, which would be important if researchers are to adopt the tool in lieu of other more targeted environments.

Taken together, the paper offers a potentially useful platform for others to consider. The lack of baselines makes it difficult to gauge whether the implementations being provided are close to other options. Without runtime or other performance data it is hard to tell if this framework is useful for others to adopt instead of existing options.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes MARLlib, a multi-agent counterpart of the reinforcement learning framework RLlib. The framework supports a wide range of MARL algorithms as well as environments, potentially a very useful platform for replicating baseline method experiments and developing new algorithms. 

Reviewers and AC had further discussions on the merit of publication as an ICLR paper at the current stage of development. The challenges with frameworks like this is that they must show some evidence that they will endure the test of time. I think this is the main reason why reviewers were hesitant to give enthusiastic comments. I also think that it’s not too late for the authors to resubmit the paper once MARLlib has gained some degree of popularity among researchers. It’s just too early to have a peer-reviewed archival paper about the software at the current stage.


justification_for_why_not_higher_score: software framework too early in the development stage
Decision: Reject