MARLlib: Extending RLlib for Multi-agent Reinforcement Learning
Authors: Siyi Hu
Yifan Zhong
Minquan Gao
Weixun Wang
Hao Dong
Zhihui Li
Xiaodan Liang
Yaodong Yang
Xiaojun Chang
Venue: Submitted to ICLR 2023 (2023)
Keywords: MARL

Abstract: Despite the fast development of multi-agent reinforcement learning (MARL) methods, there is a lack of commonly-acknowledged baseline implementation and evaluation platforms. As a result, an urgent need for MARL researchers is to develop an integrated library suite, similar to the role of RLlib in single-agent RL, that delivers reliable MARL implementation and replicable evaluation in various bechmarks. To fill such a research gap, in this paper, we propose Multi-Agent RLlib (MARLlib), a comprehensive MARL algorithm library that facilitates RLlib for solving multi-agent problems. With a novel design of agent-level distributed dataflow, MARLlib manages to unify tens of algorithms, including different types of independent learning, centralized critic, and value decomposition methods; this leads to a highly composable integration of MARL algorithms that are not possible to unify before. Furthermore, MARLlib goes beyond current work by integrating diverse environment interfaces and providing flexible parameter sharing strategies; this allows to create versatile solutions to  cooperative, competitive, and mixed tasks with minimal code modifications for end users. A plethora of experiments  are conducted to substantiate the correctness of our implementation, based on which we further derive new insights on the relationship between the  performance and the design of algorithmic components. With MARLlib, we expect researchers to be able to tackle broader real-world multi-agent problems with trustworthy solutions. Our code\footnote{\url{https://github.com/ICLR2023Paper4242/MARLlib}} and documentation\footnote{\url{https://iclr2023marllib.readthedocs.io/}} are released for reference.

Under review as a conference paper at ICLR 2023
MARLLIB: EXTENDING RLLIB FOR MULTI-AGENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the fast development of multi-agen reinforcement learning (MARL) meth- ods, there is lack of conmonly-acknowledged baseline implementation and evaluation platforms. As result, an urgent need for MARL researchers is to develop an integrated library suite, similar to the role of RLlib in single-agent RL, that delivers reliable MARL implementation and replicable evaluation in various bechmarks. To fill such research gap, in this paper. we propose Multi-Agent RLlib (MARLlib), a comprehensive MARL algorithm library that facilitates RLlib for solving multi-agent problems. With a novel design of agent-level distributed dataflow, MARLlib manages to unify tens of algorithms, including different types independent learning, centralized critic, and value decomposition methods; this leads highly composable integration of MARL algorithms that are not possible to unify before. Furthermore, MARLlib goes beyond current work by integrating diverse environment interfaces and providing flexible parameter sharing strategies; this allows to create versatile solutions to cooperative, competitive, and mixed tasks with minimal code modifications for end users. plethora of experiments are conducted to substantiate the correctness of our implementation, based on which we further derive new insights on the relationship between the performance and the design of algorithmic components. With MARLlib, we expect researchers to be able tackle broader real-world multi-agent problems with trustworthy solutions. Our code and documentatior are released for reference.
INTRODUCTION
Multi-Agent Reinforcement Learning (MARL) is a prosperous research field that has many real- world applications and holds revolutionary potential for advanced collective intelligence 6 38 36 Existing work 2 33 [ has shown that agents are able to learn strategies that could outperform human experts and help guide human's 's decision-makinn outcomes are, the algorithm implementations are always task-specific, making it hard to compare algorithm performances, observe algorithm robustness across tasks, use them off the shelf. Thus, developing a commonly-ackun www-aackkowwdeeeee baseline implementation and a unified tool suite for MARL research is in urgent demand.
process in reverse. Significant as these
While single-agent RL has witnessed successful unification for both algorithms (e.g. SpinningUp [..
Tianshou 35 RLlib [19]. Dopamine (7 and Stable-Basell Basbll-ns series [10 12 25 and environments
(e.g. Gym 4 ), multi-agent RI. has unique challenges in building a comprehensive and high-quality library. Firstly, there exist diverse MARL algorithm pipelines. MARL algorithms diverge in learning targets such as working as group and learning to cooperate, or competing with other agents and finding a strategy that can maximize individual reward while minimizing others. Algorithms also have different restrictions on agent parameters sharing strategies, with HATRPO agents forced to not share parameters and MAPPO capitalizing on sharing. Different styles of central information utilization such as mixing value functions (e.g. VDN 30 or centralizing value function (e.g. MADDPG (00) introduce extra challenge on algorithm learning style unification. Existing libraries such as EPyMARL (23) attempt to unify MARL algorithms under framework by introducing independent learning, centralized critic, and value decomposition categorization but still lack the effort to address

---

Under review as a conference paper at ICLR 2023
IDMPG TERPO
HATRPO
MAger,
Independentt Leeriin
Cetrriized Calie
Value Desempostion
Podproomsing (dea dusing)
Gaming é Simslation M4 Mnjel'n Pormertronn Hrrel, Shrrnaf, dr:
Mapniag stactog)
Agent Lavel Distributed Datallo
Tacb/Scerart
Alpoctbra Ryperperucseee
Agurt Configuations
Beal World Problee: MotaDiles, Google Bewarch Fouibell ax:
MARLLb Exsiromment Interface
RLIi Algosttimss
Rey RRIlib
Figure I: An overview of Multi-Agent RLlib (MARLlib). MARLlib unifies environment interfaces decouple environments and algorithms. Beyond, it unifies independent learning, centralized critic, and value decomposition algorithms with an agent-level distributed dataflow, and allows flexible parameter sharing by means of policy mapping. The whole pipeline can be fully determined by configuration files. To our best knowledge, with the widest coverage of algorithms and environments, MARLlib is one of the most comprehensive MARL research platform.
all the problems above. The diversity of MARL algorithms is still a huge challenge for unification. Secondly, various multi-agent environment interfaces mutually inconsistent, they originally designed to fit the task nature (e.g. asynchronous interaction used in Hanabi, action masks are provided additional information in SMAC (28). local observation and global state mixed in MAgent (39).. The inconsistency hinders directly unified agent-environmen interaction processing and results in the issue of coupling between algorithm implementation and task environment; an algorithm implementation for one environment can not be directly applied to another due to interface changes. While PettingZoo 32 builds a collection of diverse multi-agent tasks, it inconvenient for CTDE- algorithm implementation as important information such as global state and action mask is not explicitly provided. Towards the inconsistency problem, other work, such as MAPPO benchmark (7], provides each environment with unique runner script. Nevertheless, this solution creates hurdles for long-term maintenance as well as uneasiness for new task extensions.
To address the above challenges in one work, we build new library called MARLlib based on Ray (2 and RLlib, By inheriting core advantages from RLlib and providing the following four novel features, MARLlib serves as comprehensive platform for MARL research community.
Unified algorithm pipeline with a newly proposed agent-level distributed dataflow: To unify algorithms under diverse MARL topics and enable them to share the same learning pipeline while preserving their unique optimization logics, we construct MARLlib under the guidance of a key observation: all multi-agent learning paradigms
be equivalently transformed the combination of single-agent learning processes; thus each agent maintains its own dataflow and optimizes the policy regardless of other agents. With this philosophy, algorithms are implemented in unified pipeline to tackle various types of tasks, including cooperative (team-reward-on), cooperation). collaborative
cooperation), competitive (individual competition), and mixed (teamwork based competition) tasks. We further categorize algorithms based on how they utilize central information, thereby enabling module sharing and extensibility. As shown in Figure] [[ MARLlib manages to unify tens of algorithms with the proposed agent-level distributed dataflow, validating effectiveness.
2. Unified multi-agent environment interface: In order to fully decouple algorithms from envi- ronments, we propose a new interface following Gym standard, with data structure design that compatible with most of the existing multi-agent environments, supports asynchronous agent- environment interaction, and provides necessary information to algorithms. To show the advantage of interface design, MARLlib supports ten environments (SMAC (28) MAMuloCo (24), GRF [16),
MPE 20) LBF (23,, RWARE [23]. MAgent 39), Pommerman 27]. MetaDrive [18], and Hanabi
3) picked from the zoo of multi-agent tasks because of their inter-diversity, covering various task
2

---

Under review as a conference paper at ICLR 2023
Table comparison between current MARL libraries and our MARLIib. (x) stands for the number of available algorithms. denotes that the benchmark has a unique framework of its own.
Library
Supported Eay
Algorithm
Parameter
Sharing
Sampling
Asyue
Framework
Independeni Learning (1)
PyMARL
1
cooperative
Centralized Critic (1)
full full-sharing
Decompositonn
Independent Learning (1)
PyMARL2
eceperative
Centralized Critic
full-sharing
PyMARL
Value Decompostionn (9)
CTDE (6)
Communication (1)
MARL-Algorithns
couperative
Graph (1)
full-sharing
Multi-task
EPyMARI 22
cooperative
Independent Learning (3)
Contralized Critic (4)
full shuring
PyMARL
Value Decomposition (2)
sharing
2++
full-shoring
self-play
PettinZZoo OpenSpiel
Population-
group-sharing
17]
non-shar sharing.
MАPPO benchmark
full-sharing
pytoreh-n2e- ppo-acktr-gill [S
couperative
Multi-agent
non non-sbarinn sharing
couperative collaborative competitive mixed
Independeat Learning (6)
full-sharing
MARLiib
10 PettinZooo
Centralized Critic (7)
group shuring
Ray 22 RLLbb
Value Decomposition (5)
non non-sbaring
settings in MARL, including differences of task mode, observation dimension, action space property, agent-environment interaction style, etc.
Effective policy mapping: Flexible parameter sharing the key to enabling one algorithm to tackle different task modes. To reduce the manual effort regulating policies assignment, MARLlib provides three parameter sharing strategies, namely full-sharing, non-sharing, and group-sharing, by implementing the policy mapping API of RLlib. By simply changing the configuration file, users can switch among these strategies regardless of algorithms or scenarios. Therefore, although policy mapping controls the correspondence between policies and agents throughout the pipeline, it is fully decoupled from algorithms and environments, enabling further customization of sharing strategies.
Exhaustive performance evaluation: We all suitable algorithms on 23 selected scenarios of five most common and diverse environment suites under four random seeds on average, which sums up over one thousand experiments in total. The empirical results not only substantiate the correctness of MARLLib, but they also serve as useful and trustworthy reference for the MARL research community as the comprehensiveness and fairness of comparison are guaranteed by the unified implementation approach. In addition, parram-prrreeer tables are provided to ensure reproducibility. Based on these results, we derive key observations and analyze them in depth in Section 5
With the above characteristics for new MARL benchmark, MARLlib becomes one of the most general platforms for building, training, and evaluating MARL algorithms.
RELATED WORK
Building a unified platform for MARL research meaningful yet challenging. MARL research has witnessed the development of algorithm library starting from single task with limited number of algorithms to more enriched tools and APIs, covering diverse tasks and advanced algorithms.
PyMARL [ is the first and most well-known MARL library. All algorithms in PyMARL are built for SMAC (28) where agents learn to cooperate for a higher team reward. However, PyMARL has not been updated for long time and can not catch up with the recent progress. To address this, the extension versions of PyMARL are presented including PyMARL2 [3] and EPyMARL 23
PyMARL2 [3] focuses credit assignment mechanism and provide finetuned QMIX (26 with state-of-ad art tee--art--effff performance on SMAC. The number of available algorithms increases to ten, with more code-level tricks incorporated.
3

---

Under review as a conference paper at ICLR 2023
EPyMARL 233 is another extension for PyMARL that aims to present comprehensive view on how to unify cooperative MARL algorithms. It first proposed independent learning, value decomposition, and centralized critic categorization but is restricted to cooperative algorithms. Nine algorithms implemented in EPYMARI. Three more cooperative environments LBF (8) RWARE g. and MPE 20 are incorporated to evaluate the generalization of the algorithms.
All PyMARL-based libraries follow the centralized training decentralized execution setting as PyMARL. There also exist other MARL libraries that are built in different styles and serve their unique purposes.
MARI-Algorithmn 2 is library that covers broader topics compared to PyMARL including learning better credit assignment, communication based learning, graph- based learning, and multi- task curriculum learning. Each topic has least one algorithm, with nine implemented algorithms in total. The testing bed is limited to SMAC.
MAPPO benchmark 37 is the official code base of MAPPO 37
focuses on cooperative MARL and covers four environments. It aims at building strong baseline and only contains MAPPO.
MAlib 40
is a recent library for population-based MARL which combines game-theory and MARL algorithm to solve multi-agent tasks in the scope of meta-game.
Existing libraries and benchmarks provide good platforms for developing and comparing MARL algorithms in different environments. However, there are essential limitations to the current work. Firstly, these work limited in task coverage. As shown Table [ most existing work only supports one task mode. Moreover, the number of supported environments is insufficient for researching general MARL. Secondly, existing work pays little attention to how algorithms are organized but only focuses implementing existing work. This results in poor extensibility and bloated code structure. Our MARLLib, a comprehensive and unified library based on Ray and RLlib, effectively solves the dilemma by providing better algorithm unification and categorization, implementing more algorithms both in quantity and in diversity, covering four task modes, supporting ten environment suites, allowing flexible parameter sharing, and being friendly to different training demands.
MARLLIB ARCHITECTURE
In this section, mainly explain how MARLlib addresses the two major challenges. namely the diversity of MARL algorithms and inconsistency of environment interfaces, with a newly proposed agent-level distributed dataflow, unified agent-environmeen interface, and effective policy mapping.
3.1 AGENT-LEVEL DISTRIBUTED DATAFLOW FOR ALGORITHM UNIFICATION
common framework to solve multi-agent problems is Centralized Training Decentralized Execution (CTDE), where agents maintain their own policies for independent execution and optimization, and centralized information can be utilized to coordinate agents' update directions during the training phase. Under this framework, existing libraries split the whole learning pipeline into two stages: data sampling and model optimization. In the model optimization stage, all data sampled in the data sampling stage are available make the training centralized. However, in this way, choosing proper data and using these data to optimize the model are coupled in the same stage. As a result, extending an algorithm fit other task modes (e.g. both cooperative and competitive) becomes more challenging and requires redesigning the whole learning pipeline.
MARLlib addresses this issue by equivalently decomposing the original grouped dataflow into agent- level distributed dataflow. Essentially, it takes every agent in multi-agent training as an independent unit during sample collection and optimization, but shares centralized information among agents during the notetpoocsiinn phase (postproces) in is a RLlib API for processing sampled data before model optimization; we enrich it to accommodate diverse algorithms) to ensure the equiv- alence. In postprocessiio wingosssssnn..................rs agents share observed data (data sampled from the environment) and predicted data (actions taken by their policies or Q values) with others. All agents maintain individual data buffers, which store their experiences and necessary information shared by other agents. After entering the learning stage, no information sharing is needed among agents and they can optimize themselves independently. In this way, we distribute originally combined dataflow agents and fully
4

---

Under review as a conference paper at ICLR 2023
mu-palky
otf-ally
obwerve
podgrecesing
shore
Larget agonts
sumpiing
cillectpreiic
MARL lib interface
Multi Multi-agent Tasks
Environmen Engine
cillc
Biser
Agens- Optimization
Agent-Leve Level Optiizztiion
A Agnnt-Lore Lovel Optiiization
(a) Independent Leaming
(b) Centralized Critic
(c) Value Decomposition
Figure 2: Agent-level distributed dataflow of MARLIib, Observed data refer to data sampled from environments, such as rewards or global states. Predicted data refer data generated by agents, such as Q values or chosen actions. postprocess maintains its own learning pipeline where collected data are used to optimize the policy of agent itself. Therefore, the dataflow is agent-level distributed. There are three types of dataflow, namely independent learning, centralized critic, and value decomposition [23], which are separated according to their central information utilization style. Independent learning algorithms (e.g. IQL (31) are inherently distributed dataflow from the single-agent perspective and the information- shariingnssnngggggii process skipped, as shown in (a). For centralized critic algorithms (e.g. MAPPO (37)), central information, including both observed data and predicted data, collected and shared the
refers to the data sharing process. Each agent
s ing function before entering the training stage ensure the distributed dataflow, shown in (b). In value decomposition category (e.g. FACMAC (24)), predicted data from all agents must be shared, whereas the observed data is optional, depending the algorithm's 's mixing function. The corresponding dataflow is shown (c).
decouple data sharing and optimization, thereby allowing the same implementation to solve multiple modes of tasks.
Moreover, while all STDE-based algorithms share similar agent-level dataflow in general, they still have unique data processing logic. Inspired by EPyMARI 23 we further classify algorithms into independent learning, centralized critic, and value decomposition categories to enable module sharing and extensibility. Independent learning algorithms agents Iearn independently; centralized critic algorithms optimize the critic with shared information, which then guides the optimization of decentralized actors; value decomposition algorithms learn a joint value function as well as its decomposition into individual value functions, which agents then employ to select actions during execution. According to their algorithmic properties, we implement suitable data sharing strategies in postprocessin phase, illustrated in Figure
Therefore, preserving the unique properties of all algorithms, the agent-level distributed dataflow unifies diverse learning paradigms. Our implementation approach shows its value in unifying algorithms following CTDE in single pipeline, which is capable of handling all task modes and achieving similar performance the original implementation.
3.2 A UNIVERSAL INTERFACE FOR AGENT-ENVIRONMER INTERACTION
In RL, Gym interface, obs, reward, done, info, is universally used. However, this stan- dardized interface cannot be trivially extended to MARL. In MARL, multiple agents coexist, each having its own experience data. Extra information is sometimes available, such as action mask and global state. Depending the task mode, the reward may be scalar dictionary. Moreover, the
5

---

Under review as a conference paper at ICLR 2023
action
ogerto
Dotfon
Sp
Hoctloe
agent
ogpnnl
agent
EIV
agent
MARI EIV
MARL Erv
Trasilion Data reward
spack
Transllin Bata
ogert
Transtion
morro into
Peward
dome
um
ogport
done
observation actirumart giobol
(a) Gym
(b) PyMARL
(c) MARLlib (ours)
Figure 3: Agent-environment interaction style comparison between Gym (standard RL), mostly used multi-agent library PyMARL, and MARLlib. MARLlib unifies diverse environment interfaces with a newly proposed Gym-style interface obs, reward, done, info and supports both synchronous and asynchronous sampling. done a shared signal about termination, and the others are dictionaries containing agents-specific data indexed by agent id. In this figure, we organize the data by agent for better illustration.
agents may not interact with the environments synchronously, posing another challenge. Towards these issues, MARLlib unifies the multi-agent interface from two aspects.
Firstly, MARLlib unifies the interface data structure. MARLlib proposes a new interface following Gym standard, obs, reward, done, info, that compatible with diverse multi-agent task settings in practice, covering both data structure and task modes. As illustrated in Figure 3c] in MAR- Llib, the observation returned from the environment This design satisfies most circumstances and is compatible with RLlib's data processing logic. Other Doservation-relatee information is included in info. reward is dictionary with agent id as the key. To accommodate cooperative tasks, sealar team reward transformed into dictionary structure by copying it for agent number times. done is a dictionary containing single key " "_all_" ", which true only when all agents are terminated.
dictionary with three keys: observate on,
action_maail
llobal__saate
Secondly, MARLlib supports both synchronous and asynchronous agent-environment interaction. Ex- isting MARI libraries before MARÍlib, such PyMARI (Figure do not support asynchronous sampling and mainly focus on synchronous cases. However, asynchronous agent-environment in- teraction is common in multi-agent tasks like Go and Hanabi. MARLlib supports synchronous and asynchronous agent-environe geentennvironment interaction, thanks to Ray/RLlib's 's flexible data collection mechanism: the data are collected and stored with agent id. Only when we receive the terminal signal done will all data be returned for subsequent usage. This sampling process is illustrated in Figure 3c
3.3 EFFECTIVE POLICY MAPPING
In multi-agent scenarios, a proper parameter sharing strategy can improve the algorithm's 's performance. Important it is, most existing work supports insufficient sharing modes and the implementation is repetitious - MAPPO benchmark rewrites everything for shared and separated settings, while EPyMARL repeats model structures to support both. In MARLlib, we support full-sharing (all agents share parameters), non-sharing (no agents share parameters), and group-sharing (agents within the same group share parameters) of parameters by implementing the policy mapping API of RLlib. Intuitively, it maps the virtual policies of agents to physical policies that are actually maintained, used, and optimized. Agents mapped to the same physical policy share parameters. As policy mapping is transparent to agents, they actually sample data and do optimization with the physical policies. Therefore, different types of parameter sharing can take place without affecting algorithm implementation. In practice, every enviromment with all the relevant information to support multiple sharing modes. A more customized parameter sharing strategy can be realized by revising the policy mapping API to suit the needs.
only need to maintain policy mapping dictionary for
4 MARLLIB CONFIGURATION AND USAGE
MARLlib allows users to regulate the whole training pipeline by customizing configuration files, which is clean for usage and convenient for the experimental report. For a complete MARL pipeline, configurations of four different aspects are supported, including task configuration, algorithm con-

---

Under review as a conference paper at ICLR 2023
mmarl
lenvs/base enw.confiiiiggggee
lalgos/hyperparams
Imodels/configs
ray.yami
env_args:
algo argo:
model_ arch_argsi
run workers: 0
nep_name: "3m"
gae: True
fc_layer: out_din_fe_ 128 cone_anch: "gru"
run_apus:
difficuity: "7"
lambda: 1.8
run_ wtt_cpus__ee__worcee
raward_scale_enee
k1 coeff: 8.2
pun_epus__er__ooker
Scenario
A Algorithm
Model
Ray/RLlib
RLI
i b
python ma -algo config=$algo --nnv config=$env with env_ aug.a.agg.ppp _ameassaaaammmmppp
Figure 4: Functionalities and locations of configuration files in MARLlib. With these configuration files, MARLlib allows customization of environments, algorithms, models, and basic RLlib settings. When launching training, only high-level choices of algorithms and tasks need to be specified, thereby avoiding long training commands and enabling easy experimental report.
figuration, agent model configuration, and basic training settings for Ray and RLlib. The contents and locations of them are shown in Figure 4 Task configuration maintains the key parameters of the task scenario when generating from the environment engine. For instance, the level of the enemy in SMAC can be decided by users. Algorithm configuration is in charge of the algorithm's learning procedure, In addition, tricks can be turned on or off by simply changing the corresponding key values. As MARLlib aims at building a general benchmark, tricks that are only applicable for one task but not for others are not incorporated, which guarantees that MARLlib's s al- gorithm configuration is valid for diverse multi-agen tasks. Agent model configuration is responsible for constructing the agent learning unit: a neural network. Basic training settings for Ray and RLlib control the computation resource allocation. By configuring the four aspects, the whole pipeline is determined, and the training can be launched by simply running the entrance script ma inniin. py. After the training begins, all configurations will be recorded. Referring the record file can help reproduce the same performance and conduct a fair experiment.
parameters of the
5 BENCHMARKING RESULTS AND ANALYSIS
In this section, we evaluate 17 algorithms on 23 tasks of five MARL testing beds including SMAC (28). MPE [20]. GRF [16), MAMuJoCo 244, and MAgent [39], which are chosen for their popularity MARL research and their diversity in task modes, observation shape, additional information, action space, sparse or dense reward, and homogeneous or heterogeneous agent types. We report the mean reward of experiments under four random seeds, which sums up to over one thousand experiments in total. Experiments results are shown in Table @ Figure Based on these results, we substantiate the quality of implementation and provide insightful analysis.
5.1 QUALITY OF IMPLEMENTATION
To show the correctness of MARLIib, we compare the performances of MARLIib implementation on SMAC to those reported by EPyMARL with the important hyper-parameterr kept the same. Results of EPyMARL consume 40 million steps on on-policy algorithms and four million on off-policy algorithms. MARLlib only consumes half of them respectively, as we find it enough for training to converge. Even with fewer training steps, we match most of the performances reported by EPyMARL, as shown in Table For all performance pairs available to compare, MARLlib attains similar results on 63% of them (the reward difference is less than 1.0), achieves superior results on 25% of them, and appears inferior the rest 12%. Since every algorithm exhibits expected performances and for generality and stability, we do not rely on specifce tricks. These experimental results substantiate the correctness of implementation. In this table, also report for the first time the performances
7

---

Under review as a conference paper at ICLR 2023
Table 2: Algorithm performances (in reward) for cooperative tasks. Both discrete control tasks (MPE, SMAC, GRF) and continuous control tasks (MAMuloCo) are covered. Among four environment suites, SMAC has two rows for each scenario. The first row is the performances reported by EPyMARL, and the second row is performances with MARLlib. For other environments, only MARLIib performances are included. represents no data reported. Dark cells indicate the top two performances in each scenario.
tm
Sennarto
IQL
IPG
Indeperdere
11.3C
Leaming
ITRPO
IP90
MAJAC
COMA
Canimilon Crins
MATRPO
MAPPO
VDN
Vilie Decompatiion
QMIX
VDDAC
VDPPO
16.72
20.24
20.24
20.20
11.04
20.25
18.04
19.01
16.09
20.07
20.07
20.16
20.18
20.09
10.32
20:23
20.21
16.3
17:25
15.61
20.24
345cc
16.44
16.33
10.78
15:55
13.49
10.04
13.36
14.3
19.95
1521
18.90
9.78
12.1
19.91
19.53
19.38
19:57
19.30
19.66
8.58
13,15
SMAC
MMM2
12.08
13,59
9.21
10.17
10.70
8.04
11:37
10.37
10:37
16.08
6.95
6.7
7.62
17.78
16.65
18.49
19:31
18.40
19:34
2.2
9.31
21.15
4.42
19.36
6.68
3.25
18.17
15.03
15.04
16.08
56
10.79
3.19
7.05
12.14
4.79
19.30
17:24
18.55
19:50
96
14.61
MPP
simple simpk-ppakkr listenar
-99.61
-4,00
-251.n5
-5383
22.05
153.15
-78.16
-50.17
-65.74
-58.29
-27.76
-3.37
-71.64
467.6
-44.01
-77.63
-34.4
46 26
-35.66
-90.5
-89.22
25.65
190.66
-54.37
-213.0
44.61
simple, reference
72.30
-36.3
-35.0
-0052
35.05
-37.9
31.53
-6.35
-3582
GRF
and shooe
and shout
-0115
0.07
0.6
-0.07
-0.05
-000
0.5
40.05
-0.09
-000
4000
0124
-0.34
-0.11
-0.24
0.05
40.09
0.01
40.15
3. with
0.02
0.35
0.01
0:37
0.05
0.5
0.13
0:45
-0.88
-0.06
142C
1DDFG
ITRPO
IP90
MAAIC
MADDPG
MAPPO
HAPPO
FACMAC
VDPPO
MAMBIOO
AAgeiAm
143.22
-258.2
44.60
153.46
730.5
133.06
457.11
197.85
493.3
333.95
644.53
3Agen
05.76
12549
S5L.75
-971.28
750.06
137.43
1005.30
151.5
457.68
-164.72
-302.99
-196.6
1400.34
653.78
257.75
442.48
-151.05
-588.66
.544.29
of five algorithms SMAC and MPE, twelve GRF, and ten MAMuJoCo for community reference. All the experiments are reproducible, as we provide learning curves and complete training configurations in our code repository
5.2 PERFORMANCE INHERITANCE OF AGENNT RL
Empirically, we find that developing MARL algorithms based on a strong RL algorithm is a wise choice. For instance, PPO is primarily used in single-agent RL because of its better empirical performance than vanilla PG and A2C. This superiority affects the performance of their multi-agent counterparts MAPPO and VDPPO surpass MAA2C and VDA2C in most scenarios. Another piece of evidence that corroborates this conclusion is the robustness of value iteration methods. Value iteration- based algorithms are less syperparameteer-ensitiiin than policy-gradies methods and more sample efficient. The multi-agent version of Q learning like IQL, VDN, and QMIX also inherits this advantage and shows robust performance most scenarios like SMAC and MPE.
5.3 THE EFFECTIVENESS OF MARL ALGORITHMS
From T we find algorithms of different categorization show superiority on specific tasks that share similar task patterns.
Independent learning is effective when the central information is not necessary. While coordination among agents is essential for MARL algorithms and independent learning is theoretically subop- timal, existing work [9] has pointed out that independent learning can surpass other algorithms. In Table 2 we find that independent learning algorithms are better than their centralized critic counterparts in scenarios like sample_spreale and pected to behave similarly and central information is not necessary for policy optimization. By the same logic, without global view, independent learning fails to solve coordination tasks such as a simplesspakee
_sooot, where agents are ex-
stener and s simple_refereee
Centralized critic is better at learning diverse yet coordinated behaviors. In multi-agent task, agents can take diffferent roles, and their behaviors are expected to be role-specific [14] 34). Centralized critic suits these tasks since local observations and global information are both well utilized. A good example is MAPPO, a representative algorithm of centralized critic that performs well on most cooperative tasks in SMAC, MPE, and GRF. HAPPO is the heterogenous version of MAPPO that achieve robust performance MAMuJoCo. MAPPO and HAPPO are strong baselines for cooperative tasks where agents have diverse behaviors.
8

---

Under review as a conference paper at ICLR 2023
0.0
-4
-10
-4
-0.26
a) battlefield
b) battle
C) adversarial pursuit
d) tiger deer
150
L00
-20
-20
40
e) simple crypto
f) simple tag
simple adversary
simple push
Figure 5: Reward curves of eight mixed scenarios (agents compete in group) in MAgent (a-d) and MPE (e-h). Different styles of curves stand for different agent groups. The reward curves of competing groups show a dynamic balance during the learning procedure, while the balance point depends on both algorithms and tasks.
Value decomposition dominates the popular cooperative benchmarks, except for two cases. The first case is continuous control. Well-known value decomposition algorithms like VDN and QMIX are unsuitable for continuous control tasks, and VDA2C and VDPPO are inferior compared to other algorithms. The second case a long-term planning problem with sparse reward function like in GRF Empirically, the performances of value decomposition methods are significantly worse than algorithms of other categories, such as ITRPO and MAPPO. We identify two primary reasons: I) value iteration used by VDN and QMIX prefers dense reward function; 2) the mixer can hardly decompose Q function close to zero. Except for these two cases, value decomposition algorithms achieve robust performance with the best sample efficiency.
5.4 ALGGORITHM EVALUATION MIXED SCENARIOS
Benchmarking algorithms in mixed tasks is challenging. Agents in mixed tasks behave both coopera- tively (with teammates) and competitively (to their opponents). hard to justify which algorithm is better based on the reward gained as the policies are always in dynamic balance: when one policy better optimized, the performances of its opponents' policies are degraded.
Under mixed task mode, algorithms can be evaluated by the summed reward of all different policies. One policy optimization forces competitive policies to get a higher reward. Therefore, the higher the summed reward, the better the algorithms (Figure 5 a-d]). However, there are exceptions (Figure e- h)). The summed reward is a constant value or around a constant value and policies quickly reach equilibrium with mirrored learning curves between competitive policies significant pattern. A fair and general criterion to evaluate algorithms on constant-sum tasks is still an active research direction.
CONCLUSION
In this paper, we introduced MARLlib, an integrated library that covers broad algorithms and tasks in the MARL research. MARLlib unifies diverse learning paradigms and multi-agent environment interfaces with newly proposed agent-level distributed dataflow, interface unification, and flexible parameter sharing strategies. Thousands of experiments are conducted validate the correctness of our implementation. To summarize, MARLlib serves as a comprehensive and solid library suite for MARL algorithm training, evaluation, and comparison. We believe MARLlib can benefit large- scale multi-agent applications in the long term. Moreover, it can also benefit the MARL research community by serving as an educational portal for new researchers. At last, we strongly recommend readers to read our code and documentation where more details are served.

---

Under review as a conference paper at ICLR 2023
REFERENCES
[1] Joshua Achiam. Spinning Up in Deep Reinforcement Learning, 2018.
[2] Bowen Baker, Ingmar Kanitscheider, Todor M. Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In ICLR, 2020.
[3] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch. Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 2020.
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[5] Noam Brown and Tuomas Sandholm. Superhuman ai multiplayer poker. Science, 365 (6456):885-899 2019.
[6] Lucian Busoniu, Robert Babuska, and Bart De Schutter. comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, 38(2):156-177 2008.
[7] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G. Belle- mare. Dopamine: Research Framework for Deep Reinforcement Learning. 2018.
[8] Filippos Christianos, Lukas Schäfer, and Stefano Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. In NIPS, 2020.
[9] Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Tom, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.0953: 2020.
[10] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines, 2017.
[11] Python MARL Framework. URL
[12] Ashley Hill, Antonin Raffin, Maximilian Emestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines, 2018.
[13] Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih wei Liao. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning. 2021.
[14] Siyi Hu, Chuanlong Xie, Xiaodan Liang, and Xiaojun Chang, Policy diagnosis via measuring role diversity in cooperative multi-agent rl. In /CML, 2022.
[15] Ilya
Kostrikovk.
URL
sttps://githu..com/ikeko
p-torch-a22-ppo-ccck-----
[16] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly. Google research football: A novel reinforcement learning environment. In AAAI, 2020.
[17] Mare Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et al. Openspiel: A framework for reinforcement learning games. arXiv preprint arXiv: 1908. 09453, 2019.
[18] Quanyi Li, Zhenghao Peng, Lan Feng. Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
10

---

Under review as a conference paper at ICLR 2023
[19] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In ICML, 2018
[20]
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitve environments. In NIPS, 2017.
[21]
MARL-Algorithss.
URL
MARL Aall-Algoorittt
[22] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang. William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging {AI} applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), 2018.
[23] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Bench- marking multi-agent deep reinforcement learning algorithms in cooperative tasks. In NIPS Track on Datasets and Benchmarks, 2021.
[24] Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pleare-Aeexadre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. NIPS, 2021.
[25] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. JMLR, 2021.
[26] Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML, 2018
[27] Cinjon Resnick, Wes Eldridge, David Ha, Denny Britz, Jakob N. Foerster, Julian Togelius, Kyunghyun Cho, and Joan Bruna. Pommerman: A multi-agent playground. In Jichen Zhu (ed.), Joint Proceedings of the ALIDE 2018 Workshops co-located with 14th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE 2018), 2018.
[28] Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung. Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In AAMAS, 2019.
[29] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In ICLR, 2016.
[30] Jianyu Su, Stephen Adams, and Peter A Beling. Value-decompositionn multi-agent actor-critics. In AAAI, 2021.
[31] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In ICML, 1993.
[32] Justin K. Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S. Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Verez-Vicente Niall L. Williams, Yashas Lokesh, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learning. In Marc Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), NIPS, 2021.
[33] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung. David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782)3350-3355 2019.
[34] Tonghan Wang, Tarun Gupta, Anuj Mahajan. Bei Peng. Shimon Whiteson, and Chongjic Zhang. RODE: learning roles to decompose multi-agent tasks. In ICLR, 2021.
[35] Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Hang Su, and Jun Zhu. Tianshou: highly modularized deep reinforcement learning library. CoRR, 2021.
11

---

Under review as a conference paper at ICLR 2023
[36] Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. CoRR, abs/2011.0055. 2020.
[37] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu, The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint 2021.
[38] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pp. 321-384 2021.
[39] Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. In AAAI, 2018.
[40] Ming Zhou, Ziyu Wan, Hanjing Wang, Muning Wen, Runzhe Wu, Ying Wen, Yaodong Yang, Weinan Zhang, and Jun Wang. Malib: A parallel framework for population-base -based multi-agent reinforcement learning, arXiv preprint arXiv:2106.07553 2021.
A EFFICIENCY
We conducted experiments to show the efficiency regarding the training clock time and the memory usage of MARLlib compared to EPyMARL and On-policy bascline (official MAPPO (37). This experiment is conducted on a local server with NVIDIA RTX A6000 GPU and AMD Ryzen Thread-
ripper PRO 5945WX 12-Cores CPU. The testing scenario is MMM2 from SMAC (28) and the testing
algorithm is MAPPO. The total consumed timesteps are 10 The result can be found in Table.
Table 3: Efficency and hardware usage comparison between EPyMARL, MAPPO, and MARLLib
Framework/Benchhaakk (ihh)
Training Clocktime (min:sec)
Memory Usage (GB) GPU Memory Usage (MB)
3:29/3:14/2:2
8.4/12.3/15.8
2245/2309/2329
MAPPPO(hhcad::5/00115) 3ARL
5:12/3: 13/2:42
8.9/12.3/16.3
2157/2277/2389
3:29/2:16/1:244
11.2/15.4/20.4
5025/5327/5351
As result, MARLlib significantly more efficient than other frameworks in clock time. However, the largely increased training speed comes with relatively higher memory usage and GPU memory usage. This is partly due to Ray/RLlib's scheduling mechanism, where data is cached on GPU as long as there is still available memory. And since each agent is asked to maintain its own data buffer in MARLlib (which a premise of tackling task modes such as competitive and mixed), the memory usage higher than the other two.
B EXTENSIBILITY
The extensibility of MARLlib is guaranteed by the highly abstracted architecture, which consists of five major parts: configuration, training script, algorithm, model, and environments. Each part has corresponding API, methods, instructions explained in our documentation. Here we provide a brief description MARLlib's extensibility gained from each part on different aspects MARL research:
To control the experiment deployment, such as scale up to multi-GPU parallel training, one needs to focus on the configutation part as illustrated Sec and documentation' s handbook part. For instance, ray.yaml enables flexible resource allocation, such as CPU cores and numbers of available GPUs, to each worker and learner.
To extend one algorithm to tackle more task modes, such as cooperative only to mixed scenarios, one needs to focus on the script part provided in under marl allo/ ser secripte ipts directory. For instance, enabling an algorithm tackle additional task modes requires reorganizing the policy mapping function in the script part.
To enable one algorithm to tackle complex task data structures or partial observable settings, needs to focus marl/mo 1s directory. For instance, the well-known large-scale multi- task Neural-MMi returns an observation containing both 3D observation and ID state.
12

---

Under review as a conference paper at ICLR 2023
To develop algorithm, one needs to focus the algorithm part build new functions based on existing basic algorithm implementation under marl/algos /cooooscccceeeeoeo /core directory and marl/utils directory. For instance, building communication-base MARL al- gorithm requires agents to share their decision in the postprocessing function defined in marl/algos/ is wwe.eettaaalll wwi_eeiddiiicpppp
To quickly build baseline for newly proposed multi-agent tasks that are not already included, one needs to focus the envs directory. Following the gym-like interface introduced See 3.2 a new task can be easily incorporated, and all algorithms available on MARLLib can also be evaluated this new task.
C TRICKS
In MARLlib, task-specific tricks are removed. Common tricks are still preserved and available to choose from. RLlib supports rich tricks standard RL, such as gradient clip, parameter clip, value function clip, GAE(A) [29], etc. All these tricks are still available in MARLlib by modifying the hyperparameter in the configuration file. As for tricks specific to MARL, currently, there are three most commonly seen tricks in MARL.
First is choosing a better input representation for the value function. This trick is first mentioned in MАPPO (37] where the authors find that combining local and global obser- vation improves model performance. In MARLlib, we implement these tricks by adding additional hyperparameters in environment configuration files. For instance, to remove global observation of the critic function, simply change the looballstateee False and cut the global_steen state off in the environment return.
to
Second is a suitable model size. Increasing the neuron number of model can enlarge the model's capacity. However, larger model size can slow down the learning speed. In MARLLib, model size can be easily controlled by changing the model configuration file. For instance, increase the number of encoder layers.
Third is applying algorithm-specifie tricks like increasing minibatch sgd-iteration number in MAPPO's training stage. These tricks are all included in MARLlib and can be adjusted by changing the parameter in the algorithms configuration.
We conducted an ablation study on adding//emovinggadjussinng some tricks used in MARLlib although there is no comparable result reported before. We test trick MAPPO and two QMIX [26]. The testing bed is SMAC scenario MMM2. Different groups use different tricks combination, indicated by V.
Table Input representation trick on MAPPO
local observation
global state
opponent action
reward
Group 1 Group Group 3
15.74
16.13
16.86
Table 5: Reward normalization trick on QMIX with different model size
layer
dimension
reward norm
reward
Group 1
128
5.68
Group 2
1
128
18.34
Group 3
2
128164
4.71
Group 4
2
128164
16.23
Group 5
3
2561128164
3.76
Group 6
3
2866128164
15.98
As result, find the input representation has minor effect the final performance of the centralized critic-based method MAPPO. At the same time, tricks like reward normalization have a significant impact the value crration-based method QMIX.
13

---

Under review as a conference paper at ICLR 2023
D RUNNING OPTIONS
MARLlib provides three options building and running MARLlib, including standard, docker-based, and website-based.
Standard way is best for learning the workflow of MARLIib and developing new algorithms on a local machine.
Docker- based way is proposed for fast deployment and testing the framework's utility.
Website- based way is most suitable for the visualization of training results and easy configu- ration manipulation.
E DOCUMENTATION
MARLlib is proposed with documentation affiliated. The documentation consists of four major parts.
The first part is the MARLlib handbook, which contains the installation of MARLlib and the usage of incorporated environments"]
The second part is guidance for new MARL researchers navigate from RL MARL, with a collective survey of MARI
The third part is the algorithm's documentation, with five self-contained algorithm families, discussing the internal relationship within different algorithms and how they organized in MARL
The fourth part the resources of MARI., including an awesome list of algorithm papers and existing MARI benchmarke
This documentation provides MARLLib users with comprehensive understanding of MARL's basic idea and representative algorithms. We recommend MARLlib users have basic ideas DL and RL. Still, when reading the documentation, users can easily find the background knowledge and related information with linked URL or brief introduction.
F MAPPO BENCHMARK, EPYMARL, AND MARLLIB
To further ensure the MARLIib algorithm implementation' 's trustworthiness and correctness, we conduct extra experiments that compare the performance (reward gain) of MARLlib and the official MAPPO implementation [37] on widely-used SMAC benchmark (28) as the testing bed. Here we also include EPyMARL's 20 million timesteps performance for reference.
Framewwork/Bechhmakk 2s vs lsc
3s5z
MMM2
VS
EPyMARL
20.20
18.71
16.67
17.62
MAPPO
20.21
18.34
18.20
13.78
MARLlib
20.21
19.52
16.86
17.24
As shown in the table above, the model performance among all three benchmarks is similar, which proves that the implementation in MARLlib is trustworthy.
aaii aamily.hmm.
html
14


Reviews:
[{"review_id": "iZCyf3qKwpE", "reviewer": "Reviewer_EKiW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a unified coding framework for MARL based on RLlib and Ray. The frame support a relatively large number of different algorithms (of different types), and a relatively large number of MARL environments.", "summary_of_the_review": "Overall, this work can be useful for the MARL community, but the contribution should be clarified and concerns mentioned above should be considered.", "strength_and_weaknesses": "**Strength**\n- This work implements a relatively large number of MARL algorithms on multiple environments. This is useful for researchers to compare new algorithms to existing baselines.\n- Extensive evaluation is conducted for the proposed framework\n- The proposed framework integrates a large number of environments\n\n**Weaknesses and Questions**\n- As a unified framework to support the future development of MARL, the correctness of algorithms is important. For example, in Table 2, for SMAC 3s_vs_5z and IPPO algorithm, it seems the performance difference between your framework and EPyMARL is relatively big. Why is there such a difference?\n- As a unified frame, how does your framework compare to other frameworks in terms of performance? It would be good to have some evaluation results (e.g. training clock time, memory usage, etc) both in the single machine case and distributed case.\n- The original RLlib also has some support for MARL, how does your implementation differ from it? Providing more discussion in terms of your contributions and difference can help the reader better understand the framework.\n- Extensibility is also important for a unified framework, how does your framework support future development of new MARL algorithms? Do you provide API, methods, or instructions that can make future development MARL algorithm easy?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ki4EyDiArC", "reviewer": "Reviewer_o2ZC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a library - MARLlib - to run multi-agent reinforcement learning experiments with different environments, task settings (cooperative, competitive, or mixed), and algorithms. The core contribution is defining how observations, predictions, and other data are shared among agents that are otherwise distributed during training. The structure defined is flexible enough to support existing MARL algorithms that vary in using centralized critics or mixing functions for value decomposition and supports environments that use synchronous or asynchronous updates. In practice the interface modifies the standard gym definition to accommodate per-agent data and uses RLlib to provide parameter sharing functionality. Experiments include a comparison with EPyMARL algorithm implementations in SMAC and include results on several other MARL benchmarks (MPE, GRF, MAMuJoCo).", "summary_of_the_review": "The contributions of this paper come from unifying existing approaches and the novelty of the formulation of that unification. From a technical standpoint the novelty is a distributed dataflow formulation to enable distributed agent learning through decoupling agents from each other during optimization. This is a useful insight, but it is not clear it unifies all the existing MARL approaches nor does the paper explore what new algorithms may be facilitated. The main contribution is the experiment platform provided, more than the insights facilitated by its conceptual foundation.\n\nThe empirical results lack baselines in most cases. Thus, it is hard to tell if the implementations are \"correct\" relative to previously published results. The results do not claim any novelty and are intended to demonstrate comparability instead. Without proper baselines it is only possible to assess that EPyMARL and MARLlib offer roughly similar agent training convergence in many, but not all, cases (ex: IQL and MAA2C in 3s_vs_5z do substantially worse). No empirical results offer insight into the runtime performance of the platform, which would be important if researchers are to adopt the tool in lieu of other more targeted environments.\n\nTaken together, the paper offers a potentially useful platform for others to consider. The lack of baselines makes it difficult to gauge whether the implementations being provided are close to other options. Without runtime or other performance data it is hard to tell if this framework is useful for others to adopt instead of existing options.", "strength_and_weaknesses": "## Strengths\n\nThe main strengths of the paper:\n1. Algorithm versatility. The parameter sharing implementation and data flow structure supports a meaningful subset of MARL implementations.\n2. Environment breadth. Including cooperative, competitive, and mixed scenarios facilitates a wide variety of research. This may also encourage further development of algorithms that perform well across all contexts and deeper analysis of how task vs environment structure influence algorithm performance.\n\n\n## Weaknesses\n\n1. No evidence on runtime performance. How fast and scalable is MARLlib compared to alternatives? Can researchers quickly run experiments given typical hardware? How does the throughput vary with number of agents? How close are the algorithm implementations to their competitors in terms of runtime efficiency? Clarifying runtime characteristics would strengthen the case that MARLlib is a useful addition to the tools available for running MARL experiments.\n2. Limited evidence compared to existing algorithms. The only comparison made is to EPyMARL, with no reference baselines provided for other environment algorithms.\n3. Lack of baseline comparisons. The configurations used remove many \"tricks\" applied in other algorithms. It is hard to gauge if the implementations are correct due to the changes needed to fit the config files. It would help to include reference to previously reported performance numbers for comparison.\n\n\n## Feedback & Questions\n\nTable 2 would benefit from reporting baseline performance results from other implementations for comparison.\n\nRuntime performance merits a discussion. Specifically addressing scaling in terms of memory and runtime, when varying number of agents, and so on. This would benefit from comparisons to existing methods.\n\nSome questions:\n- Can MARLlib support agents that have different lifespans (not around entire task)? That is, tasks where agents \"die\" during the task.\n- What are the \"virtual\" vs \"physical\" policies mentioned in section 3.3? Does physical mean the data structures used?\n- How can developers/users add \"tricks\" to their algorithms that are not already provided by the configuration?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "65JcDlZyZz", "reviewer": "Reviewer_Fhyv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors propose a comprehensive MARL algorithm library (MARLlib) for solving multi-agent problems. MARLlib manages to unify tens of algorithms, including different types of independent learning, centralized critic, and value decomposition methods. And MARLlib goes beyond current work by integrating diverse environment interfaces and providing flexible parameter sharing strategies", "summary_of_the_review": "In this paper, authors propose a comprehensive MARL algorithm library (MARLlib) for solving multi-agent problems. MARLlib manages to unify tens of algorithms, including different types of independent learning, centralized critic, and value decomposition methods. And MARLlib goes beyond current work by integrating diverse environment interfaces and providing flexible parameter sharing strategies\nStrengths:\n1.\tAn integrated library suite for MARL algorithms, Multi-Agent RLlib (MARLlib), is introduced.\n2.\tMARLlib unifies diverse environment interfaces with a newly proposed Gym-style interface.\n3.\tOpen source codebase\nWeaknesses:\n1.\tFor example, in the 3s_vs_5z map of SMAC, why the performance conducted in MARLib has an obvious different from EPyMARL?  \n2.\tFor other environments (MPE, GRF, MAMuJoCo), why the performances of MARLlib are only included?\n3.\tAuthor may clearly illustrate the benefits obtained from agent-level distributed dataflow, a unified agent-environment interface, and effective policy mapping in terms of implementations or experiments.\n4.\tSome statements, like “value iteration used by VDN and QMIX prefers a dense reward function”, lack solid explanations. ", "strength_and_weaknesses": "Strengths:\n1.\tAn integrated library suite for MARL algorithms, Multi-Agent RLlib (MARLlib), is introduced.\n2.\tMARLlib unifies diverse environment interfaces with a newly proposed Gym-style interface.\n3.\tOpen source codebase\nWeaknesses:\n1.\tFor example, in the 3s_vs_5z map of SMAC, why the performance conducted in MARLib has an obvious different from EPyMARL?  \n2.\tFor other environments (MPE, GRF, MAMuJoCo), why the performances of MARLlib are only included?\n3.\tAuthor may clearly illustrate the benefits obtained from agent-level distributed dataflow, a unified agent-environment interface, and effective policy mapping in terms of implementations or experiments.\n4.\tSome statements, like “value iteration used by VDN and QMIX prefers a dense reward function”, lack solid explanations. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1Wwsq7Hw5qH", "reviewer": "Reviewer_RpXo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduced a new development of a multi-agent reinforcement learning platform based on RLlib to support cutting-edge research on multi-agent reinforcement learning technologies. This paper introduced several key platform design ideas that help to build a flexible and general platform for developing and experimenting multi-agent reinforcement learning algorithms.", "summary_of_the_review": "The paper considered an important problem of building a flexible and extensible platform for multi-agent reinforcement learning. While building such a platform clearly has significant research values, I have major concerns regarding the reproducibility, the quality of the code and the correctness/clarity of the online documentation.", "strength_and_weaknesses": "Strength:\nWith the increasing popularity of multi-agent reinforcement learning, it is important to develop a reliable and high-quality multi-agent reinforcement learning platform to foster fruitful future research in this booming field. This paper introduced such a platform with some interesting system design ideas, such as agent-level distributed dataflow that unifies diverse learning paradigms. Experiment results show that this platform provides high-quality implementation of many existing multi-agent reinforcement learning algorithms and can be very useful for researchers who are interested in developing and using multi-agent reinforcement learning technologies.\n\nWeakness:\nMany libraries have been developed to support single-agent reinforcement learning. It is not clear why the authors chose to use RLlib as the basis for developing their new multi-agent reinforcement learning platform. Is it possible for the multi-agent platform to support multiple existing reinforcement learning libraries? This might be helpful to satisfy researchers' diverse preferences over reinforcement learning libraries.\n\nTo determine the overall usefulness of the newly developed multi-agent platform, it might be important to understand which computing platform and infrastructure it supports. Furthermore, it is also interesting and important to know whether the new platform supports distributed deployment for many practical applications. Can the platform support scalable learning through large-scale parallel processing? Is the platform compatible with (or support) any publicly accessible parallel computing services and facilities?\n\nFor any platform to be successful, it needs to meet certain efficiency requirements. While the authors evaluated the effectiveness of several multi-agent reinforcement learning algorithms that are supported by the new platform, in terms of computation time and computation resources required, it is not clear whether the new platform is more competitive than any existing platforms.\n\nAnother major concern is the user experience. I understand that the new platform supports some ease-of-use features. For example, users can configure several aspects of their multi-agent reinforcement learning pipeline. In line with this feature, I was wondering whether the platform also allows users to configure certain details regarding experiment deployment, experiment progress monitoring, result collection and reporting. Perhaps a case study can also be carried out to demonstrate how the new platform can help researchers develop and evaluate their new multi-agent reinforcement learning algorithms. Additionally, the extensibility of the new platform may need to be examined in more depth.\n\nIn line with the above, the robustness of the new platform implementation may also need to be verified and reported in the paper. For example, it might be interesting to find out how likely the new platform may run into any errors (for example errors due to instability of numerical calculation) while conducting a large-scale experiment.\n\nAccording to my understanding, the new platform does not support some algorithm-specific tricks. While this is understandable to a certain extent, it remains unclear about (1) whether users can easily implement and support any tricks while conducting experiments using the new platform and (2) due to the absence of algorithm-specific tricks, how big the performance difference could be between the results published by the respective algorithm inventors and the results obtained from using the new platform. Moreover, the paper only compared the performance of several algorithms by using two alternative multi-agent reinforcement learning platforms. As only four random seeds are used and the performance differences cannot be verified statistically, it remains unclear whether the new platform indeed provides high-quality implementations of all the multi-agent reinforcement learning algorithms it supports.\n\nPerhaps the authors can comment more about documentation support for the new platform. The usefulness of any software platform depends on its documentation support. The authors may want to discuss how their documentation can help researchers quickly learn their platform. What kind of background knowledge researchers must have in order to understand the documentation? This question can be clarified in the paper too.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject