Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms
Authors: Chao Yu
Akash Velu
Eugene Vinitsky
Yu Wang
Alexandre Bayen
Yi Wu
Venue:  (2021)
Keywords: Multi-agent Reinforcement Learning
Benchmarking

Abstract: We benchmark commonly used multi-agent deep reinforcement learning (MARL) algorithms on a variety of cooperative multi-agent games. While there has been significant innovation in MARL algorithms, algorithms tend to be tested and tuned on a single domain and their average performance across multiple domains is less characterized. Furthermore, since the hyperparameters of the algorithms are carefully tuned to the task of interest, it is unclear whether hyperparameters can easily be found that allow the algorithm to be repurposed for other cooperative tasks with different reward structure and environment dynamics. To investigate the consistency of the performance of MARL algorithms, we build an open-source library of multi-agent algorithms including DDPG/TD3/SAC with centralized Q functions, PPO with centralized value functions, and QMix and test them across a range of tasks that vary in coordination difficulty and agent number. The domains include the particle-world environments, starcraft micromanagement challenges, the Hanabi challenge, and the hide-and-seek environments. Finally, we investigate the ease of hyper-parameter tuning for each of the algorithms by tuning hyper-parameters in one environment per domain and re-using them in the other environments within the domain.

Under review as a conference paper at ICLR 2021
BENCHMARKING MULTI-AGENT DEEP REINFORCE- MENT LEARNING ALGORITHMS
Anonymous authors Paper under double-blind review
ABSTRACT
We benchmark commonly used ageet deep reinforcement learning (MARL) algorithms variety of cooperative multi-agent games. While there has been significant innovation MARL algorithms, algorithms tend to be tested and tuned single domain and their average performance across multiple domains is less characterized. Furthermore, since the hyperparameters of the algorithms are carefully tuned to the task of interest, it is unclear whether hyperparameters easily be found that allow the algorithm to be repurposed for other cooperative tasks with different reward structure and environment dynamics. To investigate the consistency of the performance of MARL algorithms, we build an open-source library of multi-agent algorithms including DDPG/TD3/SAC with centralized Q functions, PPO with centralized value functions, as well as QMix, and test them across a range of tasks that vary in coordination difficulty and agent number. The domains include the Multi-agent Particle World environments, StarCraftll micromanagement challenges, the Hanabi challenges, and the Hide-And-Seet environments. Finally, we investigate the ease of hyperparameters tuning for each of the algorithms by tuning hyperparameters in one environment per domain and re-using them in the other environments within the domain. The open-source code and more details can be found in our website: https::/ /tsstteessoooll........... google comm wiew//ee ma Ibenchharks
INTRODUCTION
Widespread availability of high-speed computing, neural network architectures, and advances in reinforcement learning (RI.) algorithms have led to continuing series of interesting results in building cooperative artificial agents: agents collectively playing Hanabi to expert level (Hu & Foerster 2019), designing cooperative StarCraftII bots (Rashid et al. 2018) that outperform hand-designed heuristics, and constructing emergent languages between agents (Mordatch & Abbeel 2017). Each of these aforementioned results have often come with the introduction a new algorithm, leading to proliferation of new algorithms that is rapidly advancing the field.
However, these algorithms are often designed and tuned to get optimal performance a particular algorithm come with new
deployment environment. In particular, it is not unusual for each proposed benchmark on which it is evaluated. Consequently it is not obvious that these algorithms can easily be re-purposed for new tasks; subtle interactions between the algorithm, the architecture and the environment may lead to high asymptotic performance on one task and total f'ilure when applied new task. Without examining algorithm across range of tasks, it is difficult assess how general purpose it is.
Furthermore, the high asymptotic rewards that are often presented may hide complexities in using the algorithms in practice. The amount of time that researchers spent in finding optimal hyperparameters often obscured, making it unclear how extensive of a hyperparameter search was needed to find the good hyperparameters. That is, researchers will often report a grid search of hyperparameters but not the prior work that was done to pick out a hyperparameter grid that actually contained good hy hyprpaarameters. Furthermore, the amount of computation provided to tune the studied algorithm may not be provided to the baseline algorithms that it will be compared against. This can lead to an inflated performance of the proposed algorithm relative to the benchmarks. All these problems can arise without any ill intent on the part of the authors, but they make the problem of assessing algorithms quite challenging.

---

Under review as a conference paper at ICLR 2021
The downstream consequence of this proliferation of algorithms coupled with an absence of standard benchmarks is a lack of clarity on the part of practitioners as to which algorithm will give consis- tent, high performance with minimal tuning. Researchers are often operating under computational constraints that limit how extensive of a hyper-parameter sweep they can perform; the ease with which good hyperparameters can be found is consequently useful metric. When tackling new multi-agent problem, researchers have no clear answer to the questions: 1) which MARL algorithm should use to maximize performance and 2) given my computational resources, which algorithm is likeliest to work under my constraints?
We present an attempt evaluate the performance, robustness and the relative ease of using these algorithms by benchmarking them across a wide variety of environments that vary in both agent number, exploration difficulty, and coordination complexity. By exploring large range of possible environments, we identify algorithms that perform well on average and serve as a strong starting point for a variety of problems. We tackle the question of relative difficulty in finding hyperparameters by looking at how hyperparameters transfer across settings: tuning hyperparameters on one set of environments and applying them without re-tuning on the remaining environments. Using this procedure, can provide effective recommendations on algorithm choice for researchers attempting to deploy deep multi-agent reinforcement learning while operating under constrained hyper- pppam-err budgets. We test Proximal Policy Optimization (Schulman et al.) (017)) with centralized value func- tions (MAPPO), Multi-Agent DDPG (MADDPG) (Lowe et al. 2017). Multi-Agent TD3 (Fujimoto) al. 2018a) (MATD3), Multi-Agent variant of Soft Actor Critic (Haarnoja et al. 2018) (MASAC), and QMiX (Rashid et al. 2018). We focus specifically on the performance of these algorithms on fully cooperative tasks, this avoids game theoretic issues around computing the distance Nash equilibria, and allows us to solely characterize performance in terms of asymptotic reward.
The contributions of this paper are the following
Benchmarking multi- multi-agent variants of single-agent algorithms across wide range of
possible tasks including StarCraftII micromanagement (Rashid et al. (219)), Multi-agent Particle World (Mordatch & Abbeel, 2017), Hanabi (Bard et al. 2020 and the Hide-And- Seek domain (Baker et al., 2019).
Establishing that under constrained hyperparameter searching budgets, the multi-agent variant of PPO appears to be the most consistent algorithm across different domains.
The design and release of new multi-agent library of various on/off-policy learning algorithms with recurrent policy support.
2 RELATED WORK
MARL algorithms have a long history but have, until recently, primarily been applied in tabular settings (Littman 1994 Busoniu et al. 2008). Notions of using a Q-function that operated on the actions of all agents, known as Joint-Action Learners (Claus & Boutilier 1998) have existed in the literature since its inception with algorithms like Hyper-Q [Tesauro] (2044) using inferred estimates of other agent strategies in the Q-function. Recent MARL algorithms have built upon these ideas by incorporating neural networks (Tampuu et al., (2017), policy-gradient methods (Foerster et al. 2017]. and finding ways to combine local and centralized Q-functions enable centralized learning with decentralized execution (Lowe et al. 2017 Sunchaa et al. 2018).
Alongside the proliferation of algorithms has come wide variety of new, cooperative MARL benchmarks. Unlike single-agent RL, where MuJoCo (Todorov et al. 2012) and Atari (Mnih al. 2013) have become standard benchmarks, there sigmificantly less consensus on appropriate benchmarks. In this work, we consider 4 popular multi-agent environments, which we believe are the most representative in the community. Besides those we considered in this work, other interesting cooperative environments may include MAgent (Zheng et al. (017), a platform that can efficiently support hundreds of particle agents for cooperative tasks, multi-agent MuJoCo, in which each joint is an independent agent (Schroeder de Witt al. 2020), and CityFlow (Zhang al. 2019), which studies large-scale decentralized traffic light control.
There also has been variety of attempts to benchmark MARL algorithms that differ in scope from our paper. Gupta et al. (2017) benchmarks a similar set of algorithms to ours on a wide variety of environments. However, they do not consider algorithms that train in centralized fashion
2

---

Under review as a conference paper at ICLR 2021
while acting decentralized and instead perform comparison between fully centralized training and execution and full decentralized algorithms. They establish that parameter sharing is an essential component of getting quick convergence in MARL algorithms. [Schroeder de Witt et al. (2020) benchmark algorithms with centralized Q and value functions on a decentralized variant of the MuloCo environments; however, they primarily study variants of QMix and MADDPG and do not compare with on-policy algorithms.
3
MARL ALGORITHMS
3.1 PRELIMINARIES
We study decentralized partially observed Markov decision processes (DEC-POMDP) (Olichock) ct al. 2016) with global rewards. A DEC-POMDP is defined by an eight tuple (S, (S, U. Z, O 7) is state space describing sufficient set of state variables make the system dynamics Markovian. For simplicity we assume the agents share action space U and each agent a E (1..... picks action ut U which are concatenated to form join action E U We denote the joint action without the action of agent as u-a- Joint actions u and state are fed to state transition function P(s'ls, u): S x U + [o, 1]. These are cooperative tasks so all agents share a bounded reward function r(s, u) S x U - R and have shared discount factor E [0, 1]
Each agent has an observation function (s) S Z which defines how the global state is mapped onto a local observation E Z. Each agent maintains an action-observation history 7 E T' e (Z U)" which it conditions its policy (u"((r") TxU + [0, 1] on. Finally, given the joint policy T (u) II, x (u we can define a joint value function V (se) Esi Di= [ss) and joint Q function Q" (st; ue) Es Ei >"rt+//+ u....
We assume that the learning algorithm has access to both true states S, as well as the trajectories of all agents 7" The agents however, only have access to 7 for computing their policy. The goal of the agents is to jointly optimize the quantity J = E D
- 7'r(s,, u]].
3.2 BASELINE ALGORITTMS
We introduce all the baseline algorithms we consider, including MADDPG, MATD3, MASAC, QMix and MAPPO. For all problems considered, the action space is discrete. More algorithmic details and the complete pscudo-code can be found in the appendix.
MADDPG: The MADDPG algorithm is perhaps the most popular general-purpose purpose off-policy MARL algorithm. The algorithm was proposed by Lowe al. (2017), based the DDPG algorithm (Lil- lierap et al. 2015), and uses a centralized Q-function taking observations and actions from all the agents to alleviate the non-stationarity issue and stabilize multi-agent training. Note that although
DDPG was originally designed for continuous actions, MADDPG adopts the gumbel-softuax (Jang
al. 2016 trick handle discrete actions.
MATD3: The TD3 algorithm (Fujimoto et al. 2018b) is a popular enhanced version of the standard DDPG algorithm (Lillicrap al. 2016). We similarly apply the same centralized critic technique introduced in MADDPG to TD3 to derive a multi-agent variant of TD3, i.c., MATD3. The only difference between MATD3 and MADDPG is the use of twin delayed crities and the addition of small amounts of noise to the actions sampled from the buffer.
The Soft Actor-Critic (SAC) algorithm (Haarnoja al. 2018) is an extremely popular off-policy algorithm and has been considered as state-of-the-an art baseline for diverse range of RL problems with continuous actions. Similar to MADDPG, we introduce a centralized critic in SAC to achieve another general- purpose MARL algorithm, MASAC.
MASAC:
QMix (Rashid et 2018) is a Q-learning algorithm designed for multi-agent cooperative tasks with global reward. The core idea of QMix is value decomposition, which formulates the global Q function, Qtot as the output of a "mixer" neural network whose inputs are the individual agent Q functions, Qa: The weights of this "mixer" network constrained to be positive in order to insure that aQcat 0, V agents a. This ensures that by acting greedily to maximize their local 8Q2
QMix:
Q functions, agents will also be maximizing the global Q function. QMix first introduced in the StarCraftII micromanagement and has been a popular benchmark algorithm for this challenge. However, it is rarely tested in other domains.
3

---

Under review as a conference paper at ICLR 2021
MAPPO: In addition to the off-policy algorithms above, we also consider an on-policy MARL algorithm, ie., a multi-agent variant of PPO (MAPPO). We enhance the standard decentralized PPO algorithm by learning centralized critic that takes in the global state or the concatenation of each agent's local observation (when global state is not available). This idea of centralized value functions was originally introduced in the COMA algorithm (Foerster et al. 2018). Our implementation follows the details in Baker et al. (2019), including using two separate networks for policy and value function, GAE, advantage normalization and a Pop-Art value normalizer (Hessel et al., 2019), which uses running statistics of the values to normalizes the regression target of value network to zero mean and a standard deviation of one.
3.3 KEY IMPLEMENTATION DETAILS
We highlight some important techniques here. More details can be found in Appendix Sec. B
To increase the likelihood that differences in performance are primarily coming from the algorithm, we ensure all the policies use the same network architecture for both actor and critic. We use parameter sharing for all agents, i.e., all the agents share the same weights.
To handle partial observability, recurrent networks are used in all the algorithms. For MAPPO, we follow Bakker et al. (2019), which cuts full trajectory into small chunks (i.c., 10 timesteps per chunk) and maintains the initial hidden state of each chunk. Then the RNN policy trained over a batch of data chunks. A similar chunking technique has also been introduced for off-policy learning by Kapturowski et al. (2018), which, however, does not work well in our experiments. Instead, we follow the procedure in Rashid et al. (2020) and Hausknecht & Stone (215)), which trains the recurrent policy over a batch of complete trajectories. More specifically, for each timestep in a training trajectory, start with a zero-vector the initial hidden state and re-feed all timesteps up to to make sure the hidden state of time t is up-to-date.
Lastly, different from standard multi-agent games where all the agents act at the same time, the Hanabi challenge a turn-based game, i.e., all the agents take actions one by one. Hence, in an N-player Hanabi game, we decompose trajectory into N separate sub-trajectories for each agent and each agent's 's policy is only trained on its own state-action sequence. We make sure that for each agent's 's training trajectories, all the other agents' actions are properly observed in the state.
ENVIRONMENTS
We briefly outline the settings used test the algorithms well as key details on the modifications we made to their original implementations. We will refer to each of these settings as domains and each of the individual problems in the domains as environments. From each domain we only include the fully cooperative tasks.
Multi-agent Particle World environments were introduced in Lowe et al. 2017). These environ- ments consist of various multi-agent games in a 2D world with small particles navigating within a square box. We will refer to these environments as the MPEs, i.e., multi- particle environments. We consider the 3 fully cooperative tasks from the original set shown in Fig. I(a] simple-spread, speaker-listener, and simple-reference. Note that since the two agents in speaker-listener have different observation and action spaces, this is the only setting in this paper where we do not share parameters but train separate policies for each agent.
As will be discussed in Sec. 5 almost all algorithms solve all of the MPE tasks, we primarily use it as a basic check that our algorithms are likely correctly implemented.
StarCraftII Micromanagement (SMAC) D was introduced in Samvelyan et al. 2019). In these tasks, decentralized agents must cooperate to defeat adversarial bots in various scenarios with a wide range of agent numbers (from 2 27). We use the global game state to train our centralized critics or Q-functions. Note that an agent may "die" in an episode. We masked out the inputs of dead agents by zero and make sure they do not contribute to the learning objective. Fig. 2(a) example StarCraftl environments.
(left) shows two
Hanabi a turn- based card game, introduced MARL challenge in Bard al 2020). where each agent observes other players' cards except their own cards. A visualization of the game is shown in Fig. [(b] The goal of the game is to send information tokens to others and cooperatively take actions
Version 4.10 is used in this paper.
4

---

Under review as a conference paper at ICLR 2021
Agent
B
Listener
Agent
333
Agent O
Speaker
0.0 "Blas"
Staddss
Desic
Diccmdde
Agent 1
'Blae'
B2
Agent 2
simple spread
speaker listener
simple reference
(a) MPE tasks
(b) Hanabi-small with 4 players
Task visualizations. (a) the Particle- World domain. simple-spread (left): agents need to cover all the landmarks and do not have a color preference for the landmark they navigate to; speaker- listener (middle): the listener needs to navigate to specific landmarks following the instruction from the speaker; simple-reference (right): both agents only know the other's goal landmark and needs to communicate to ensure both agents move to the desired target. (b) The simplified Hanabi domain (ianabi-smalll) We use just two colors instead of the five used in the original Hanabi game.
Figure 1:
A
(a) Two representative StarCraftll micromanagement scenarios (2c_vs_64zg and corridor).
(b) Tasks in the hide-and-seck domain: construction, box-locking and box-locking-ene
Figure 2: Visualizations of the StarCraftIl domain and the hide-and-seek domain.
to stack as many cards as possible ascending order to collect points. Since the original full Hanabi game may require up over 100 million samples for a ceneral-purposs algorithm to converge (Foerster et al. 2019), we consider a simplified variant, Staanaiissall from [Brdd et al. (2020), which only has colors, hand size life token, information tokens and maximum score of 10.
The Hide-And-Seek Domain, introduced by Baker et al. (2019), a MuloCo-based physical world with agents, walls and manipulable objects. We modify the original environment to generate three cooperative games adapted from the transfer task set in [akkr et al. (2019), including box-locking, where agents need lock all the boxes as fast as possible, and construction, where agents need to manipulate boxes to desired locations. We also consider a simplified version of box-locking, named box-locd hox-locking-e easy, with much smaller maze size for faster training and hyperparameter tuning. Note that in box-locking, all the boxes need to be locked for success while in construction, all the boxes have to be placed on the target sites for success. visualizations of the environments in Fig. 2(b)
5
EXPERIMENT RESULTS
5.1 EXPERIMENTAL DETAILS
To ensure a fair comparison over different algorithms, we (1) make sure that all policy architectures and weight initialization consistent across algorithms (2) similarly sized grid searches for all algorithms with slightly larger grid searches for the worst performing algorithms. For the algorithms we tune the following hyperparameters and report the values and sweeping procedure Appendix Sec. B.I:
MADDPG, MATD3: learning rate and the polyak update rate for target-network updates. MASAC: learning rate, polyak update rate for target networks, target entropy coefficient.
QMix: learning rate, target network update frequency.
MAPPO: learning rate, epoch.
These algorithms, particularly the on-policy MAPPO algorithm, are highly sensitive to batch size. In domains other than the MPEs, where batch size has nominal effect, we use the largest possible batch-size possible that can be fit into GPU memory. Every single environment is conducted in single desktop machine using single GPU card for training. To enable easy comparison of wall-clock time between algorithms, we run all the algorithms without any parallelism in the environment. For each experiment, keep training until convergence at most 3 days. All the results are averaged
5

---

Under review as a conference paper at ICLR 2021
scenarios
МАРРО
QMix
MASAC
MATD3
MADDPG
simple-spread
-122.131
-127.95
-129.497
-142.907
-156.161
speaker-listener
-12.3095
-13.3582
-13.098
-3499122 -44.4698
-31.0721
simple-reference
-9.84231
-13.9181
-13.1636
-49.9884
Table 1: Average episode rewards in MPE.
-055
Miled
MUSC
MAPOO
HAPOO
(a) Learning curves different algorithms w.r.t environment steps.
39
A311
-30 WeiTinab
(b) Learning curves of different algorithms w.r.t. running time.
Figure 3: Evolution cumulative episode rewards according step-number (top) and wall-clock time (bottom) for different algorithms in the particle-world domain.
over 5 random seeds in the StarCraftII domain, Hanabi and Hide-And-Seek domain, and over 10 seeds in the particle-worf domain.
For hyperparameter optimization we perform a sequential search per hyperparameter, optimizing them at time and then proceeding onto the next hyper-paramete using the best value found previously. For example, we might tune learning rate, find the best learning rate, and then move on to optimizing target update frequency while keeping learning rate fixed to the best value. We perform this procedure on one representative environment in each domain and re-use the hyperparameters for the other environments.
The only exception to our outlined hyperparameter tuning is QMix in the StarCraftII domain, where simply inherit the hyperparameters from the original QMi paper as these parameters already tuned for good performance. For similar reason, in these environments we re-use the QMix batch size of 32 episodes for all the off-policy algorithms in the StarCraftlI domain.
5.2 PARTICLE-WORLD
Hyperparameter Selection: The hyperparameters tuned the simple-spread scenario for each algorithm and reused across the other two scenarios.
The final average scores convergence shown in Table. I We find that all of the algorithms perform somewhat similarly, with some degradation in performance for MATD3 and MADDPG in the speaker-listener and simple-reference scenarios. Note that the results for MADDPG we obtained are similar to the numbers reported in the original MADDPG paper.
Asymptotic reward:
Wall-clock time: Fig. shows the evolution of the rewards both step number and wall-clock time. The run-time is dominated by gradient steps rather than environment collection and consequently we observe that MAPPO, which takes many fewer gradient steps, converges to good solution much quicker than the other algorithms. Note that we have turned off the parallel sample collection in MAPPO. Hence, with enough computational resources, the wall-clock time of MAPPO could be even lower.

---

Under review as a conference paper at ICLR 2021
Maps
МАРРО
QMix
MASAC
MATD3
MADDPG
2m_vs.1z
100
100
90.88
89.84
89.06
3m
99.875
94.71
55.54
60.125
48.63
2s. s..vs.ls. Isc
99.56
95.25
20.06
46.63
2s3z.
98.5
94.07
23.52
3.125
14.59
3s_vs_3z
99.5625
97.03
38_vs_4z
99.27
95.98
0
0
0
2c.vs. vs. vs.647g
99.375
96.08
many baneling
99.68
93.18
62,43
45.28
56.15
8m
99.48
95.07
43.65
MMM
98.59
96.95
16.83
Ic3s5z
100
96.09
3s5z.vs_ss6r
80
79.03
0
0
bane vs bane
94,4792
91.97
0
25m
96.88
86.15
0
5m_vs_6m
75.01
71,46
$m_vs_90
92.97
93.44
0
10m_vs_i imn_vsllmm
84.27
94.85
3s5z
90.94
82.15
MMM2
0
86.43
3s vs 5z
23.35
96.58
6h vs 8z.
8.59
79.32
corridor
75.625
77m_vs_30m
10.625
64.06
0
0
Table 2: Eval Win rate in SMAC maps over 32 trials.
5.3 STARCRAFTII MICOMANAGEMENT
Hyperparameter Selection: We use the easier maps 3m and 2s3z to do the hyperparameter tuning where easier is characterized both by how quickly QMix finds solution well as how many agents are involved. For MAPPO we found that all hyperparameters worked for 3m so we used 2s3z for the tuning whereas for MASAC/MATDAMMADDDDA the easiest map, so we stuck to tuning parameters on that map. As mentioned before, for QMix we simply re-use the hyperparameters from the original QMix paper (Rashid al. 2020). We each of the maps for 10 million steps. Note that this longer than in the QMix paper (Rashid et al. 2020) and may lead to better results than reported there.
we could not maximize performance on 3m,
We evaluate the algorithms on 32 testing episodes and consider the test winning rate as our perfor- mance metric. Table. 2 represents the performance metric for all of our algorithms. The results demonstrate that in easy and medium environments, MAPPO performs as well, if not better, than QMix. However, in the hardest maps, MAPPO underperforms in relation QMix. A deeper investi- gation of this trend revealed that with MAPPO's 's performance in these difficult maps significantly improves with lower number of epochs specifically. 5 epochs instead 15. Appendix A.1 shows the results for MAPPO with 5 epochs. This may be caused by the fact that a higher epoch number results in larger amount of data reuses, potentially rendering the hidden states of the recurrent network stale. We also note that the off-policy actor critic algorithms, while showing promise in the easiest maps, quickly drop in performance on the medium and difficult maps, which they consistently fail.
5.4 HANABI
For Hanabi, tune hyperparameters the 2-player game and re-use the chosen hyperparameters on the remaining games. We trained all algorithms for either 100 million steps or 3 days, whichever came first.
We report the averaged training rewards of the best hyperparameters Table B No hyperparameters for the off- ooolyc algorithms recieved score above despite the same hyperparameter grid working for other domains. Since the task only gets harder as we increase agent number, we only report the results for increasing agent number for MAPPO.
7

---

Under review as a conference paper at ICLR 2021
МАРРО
QMix
MASAC
MATD3
MADDPG
2 players
6.8777
0.2907
1.094
0.1026 X
0.4211
players
5.1788
X
X
X
4 players 3,9557
X
X
X
X
5 players
3.5
X
X
X
X
Table 3: Average score in the Hanabi-Smal. with different number of players. X's represent trials that due underperformance.
5.5 THE HIDE-AND-SEEL SEEK DOMAIN
We tune hyperparameters the box-locking-easy environment and the chosen hyperparameters re-used on the other environments. We report the final success rate for each of the tasks in Table. 4 We find that MAPPO performs well across the environments, while of the off-policy algorithms, only MASAC has high performance on the easy task. Even after days of training, MASAC was not able solve the the two harder tasks. However, MAPPO is able to achieve its results within half a day of training time despite operating without parallelism.
Tasks
MAPPO
QMix
MASAC
MATD3
MADDPG
box-locking-eas,
97.0
10.6
93.0
56.5
73.23
box-locking
96
0
2.3
0.0
construction 48.0
0
9
0,2
Table 4: Success rate each of the algorithms in the Hide-And-Sede domain.
5.6 SUMMARY
Across environments we find that MAPPO has strong average performance on all domains while being occasionally outperformed by other algorithms on a few environments. More problematically. we find that we cannot easily tune MADDPG/MATD//MAAAC outside of the particle environments and that the same issue applies QMix outside of the SMAC environments. These issues could probably alleviated with a more extensive grid search for the off-policy algorithms but suggest either that 1) MAPPO is more stable algorithm across environments 2) that the default PPO hyperparameters around which we search are well configured for MARL tasks.
CONCLUSION
The results in Sec. Sec.3.sse.....p... suggest that MAPPO is good starting point for tackling new MARL problems. Although QMix occasionally outperforms PPO on certain maps in the StarCraftIl domain, this is achieved using tuned hyperparameters from the original implementation and MAPPO might achieve similar performance given equal tuning budget. Given access to equally sized compute budgets for hyperparameter optimization, MAPPO is the most consistently successful algorithm across the studied environments, performing well at SMAC, as well as any algorithm in the MPEs and the best in ande-and-seck and Hanabi-smin small. Surprisingly, we also find that the centralized Q function variants DDPG, SAC, and TD3 almost entirely fail to solve any of the studied tasks, particularly in slightly harder environments.
Of course, these results are likely dependent on the choice of hyperparameters and architectural choices. Thus, it may be the that the initial point around which perform grid search for MADDPG/MATID//MAA
does not contain any good solutions and that performance of a set of predictive of performance in other domains. Finding better
hyperparameters on the MPEs of byperparameters over which to search for the off policy algorithms is an important direction for future work. Similarly, seeing additionally be valuable. It is possible that our conclusions are function of the restricted length of time we ran the algorithms for; it would be interesting to investigate the asymptotic performance of these algorithms over much longer training times. Finally, it would be worthwhile to investigate the failures of these off-policy algorithms these settings to understand more deeply the performance discrepancy.
conclusions hold over larger set of problem domains would
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffre E. Hinton. Layer normalization, 2016.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv: 1909.07588 2019.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprchensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), (8(2))156-177) 2008.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. NIPS 2014 Workshop on Deep Learning. December 2014, 2014.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. AAAI/IAAI, 1998(746-752):, 1998.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. arXiv preprint 0rXiv:7705.08926. 2017.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI 2018: Proceedings of the Thirty-Secone AAAI Conference on Artificial Intelligence, February 2018.
Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, lain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi-agent reinforcement learning. volume 97 of Proceedings of Machine Learning Research, pp. 1942-1951, Long Beach, California, USA, 09-15 Jun 2019.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error actor- critic methods. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmässsan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 201 8a.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.0080 2018b.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pp. 66-83. Springer, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.012900 2018.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. arXiv preprint arXiv: 1507.00577 2015.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, W. Czarnecki. Simon Schmitt, and H. V. Hasselt. Multi-task deep reinforcement learning with popart. In AAAI, 2019.
Hengyuan Hu and Jakob N Foerster. Simplified action decoder for deep multi-agent reinforcement learning. arXiv preprint 1rXiv: 2019.
Wei Hu, Lechao Xiao, and Jeffrey Pennington. Provable benefit of orthogonal initialization in optimizing deep linear networks. arXiv preprint N005922001.05992,.0 2020.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: 1.01 144, 2016.

---

Under review as a conference paper at ICLR 2021
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations. 2018.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 159,009,,, 2015.
Timothy Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, In ICLR (Poster), 2016.
02911
Michael Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Picter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compettiiee environments. Neural Information Processing Systems (NIPS), 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv: 770300498,, 2017.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. ICML' 10, pp. 807-814. Omnipress, 2010. ISBN 9781 605589077.
Frans A Olichock, Christopher Amato, al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. volume 80 of Proceedings of Machine Learning Research, pp. 4295 4304. PMLR, 10-15 Jul 2018.
Tabish Rashid, Philip HS Torr, Gregory Farquhar, Chia-Man Hung, Tim GJ Rudner, Nantas Nardelli, Shimon Whiteson, Christian Schroeder de Witt, Jakob Foerster, and Mikayel Samvelyan. The star- craft multi-agent challenge. volume 4, pp. 2186-2188. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv:2003.08839/ 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 1902.04043 2019.
Christian Schroeder de Witt, Bei Peng, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv, pp. arXiv-2003, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06344 2017.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki. Vinícius Flores Zam- baldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning based on team reward. In AAMAS, pp. 2085-2087, 2018.
10

---

Under review as a conference paper at ICLR 2021
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4))o0172395, 2017.
Gerald Tesauro. Extending q-learning general adaptive multi-agent systems. In Advances in neural information processing systems, pp. 871-878, 2004.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. IEEE, 2012.
Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario. In The World Wide Web Conference, pp. 3620-3624, 2019.
Lianmin Zheng, Jiacheng Yang, Han Cai, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. arXiv preprint arXiv:1712.00600,, 2017.

---

Under review as a conference paper at ICLR 2021
common hyperparameters
value
ppo epoch
15
recurrent data chunk length use clipped value. loss max clipped value loss
10
True
0.2
entropy coef gradient clip norm gae lamda gamma
0.01
10.0
0.95
0.99
loss huber delta
huber loss
10.0
batchsize
parallel envs x episode length x num_agents
mini biii-
batch-size num-mini.bbtch
Table 5: common hyperparameters used in MAPPO across all domains.
A TRAINING DETAILS
A.1 ALGORITHM HYPER- PARAMETERS
Below, we list the details of the hyperparameters used for all experiments, as well as the hyperparam- eter sweeping procecure for hyperparameters that were tuned.
For MAPPO, certain hyperparameters were kept constant across all environments; these are listed in Tab. [5] and are mostly inherited from the baseline implementation of PPO. Note that since we use parameter sharing and mix all agents' data, the actual batch-size will be larger with more agents. Regarding the hini-bbtch-size we use the largest possible
under the constraint of GPU memory limit. Note that while we list "parallel envs" for MAPPO, we actually run it without any parallelism to create fair comparisons of wallclock speed between algorithms. For MAPPO, "parallel envs" is only used control the batch size.
Network architecture hyperparameters: All neural networks are implemented with the following overall structure: m fully connected layers, followed by Gated Recurrent Unit (GRU) (Chung et al. 2014) layers, and lastly by fully connected layers. In the following hyperparameter tables, the "num fc before" hyperparameter corresponds to m, "num GRU layers" corresponds k, and "num fc after" corresponds to n. The nonlinearity function used is the rectified linear unit (ReLU) (Nair & Hinton 2010), with gain value equal the "ReLU gain" hyperparameter. Lastly, The "use feature normalization" hyperparameter describes if layer normalization (Ba et al., 2016) is applied to the inputs.
Tuned byperparameters: For MADDPG, MATD3, and MASAC, the "tau" hyperparameter corre- sponds to the rate of the polyak average technique used to update the target networks. For MASAC, the "target entropy "oef" hyperparameter determines the target entropy; specifically, the target entropy given by: H = B * (-log( )). where |A] is the dimension of the discrete action space, and is equal to the "target entropy coef" hyperparameter. For QMix, the "hard target update interval" hyperparameter specifies the number of gradient updates which must elapse before the target network parameters are updated to equal the live network parameters. Note that the tables specifying the sweeping procedure for each domain list the hyperparameters in the order which they were tuned.
For each domain, we describe the network architectures, the hyperparameters kept constant across that domain for each algorithm. as well the hyperparameter sweeping procedure for tunable hyperparameters for each algorithm.
12

---

Under review as a conference paper at ICLR 2021
Multi-agent Particle World (MPE):
QMix
Ir le-4, 5e-4, 7c-4, 1e-3]; hard update interval 2200, 500, 700, 900, 1500]
MADDPG
Ir [e-4, 5c-4, 7e-4, le-3, 1e-2]; tau [0,01, 0.005, 0,001]
MATD3
Ir Ie-4, 5e-4, 7c-4, le-3, 1e-2]; tau [0.01, 0.005, 0.001]
MASAC
Ir 11e4,, Se-4, 7e-4, 1e-3, 1e-2]; tau [0.01, 0.005, 0.001;; J; target entropy coef [0.1, 0.3, 0.5, 0.7]
MАPPO
Ir [le-3, 7e-4, 5e-4, le-4]; ppo epoch [5, 10, 15, 20, 25]
Table 6: sweeping procedure in MPE, tau denotes the target network update rate, the bold font indicates the value adopted.
network hyperparameters
value
initialization method num GRU layers RNN hidden state dim fe layer dim num fo before num fe after use feature normalization optimizer optimizer eps weight decay ReLU ReLU gain
orthogonal
64
64
1
True
Adam
le-5
0
True
sqr((2)
Table 7: network hyperparameters used in the MPE domain by all algorithms
hyperparameters
value
episode length last action layer gain
25
0.01
gamma
0.99
buffer size
5000
batch size
32
epsilon
from 1.0 to 0.05
epsilon anneal time
50000
Q function loss
MSE loss
Table 8: Hyperparameters used in the MPE domain by MADDPG, MATD3, MASAC, and QMix. Buffer size and batch size calculated by episodes.
hyperparameters
value
parallel envs
128
last action layer gain
0.01
num mini batch
Table 9: Hyperparameters used in the MPE domain by MAPPO
13

---

Under review as a conference paper at ICLR 2021
StarCraftII Micromanagement (SMAC): All algorithms are trained until convergence, 10M timesteps is reached, or for a maximum of 3 days. Since the maximum length an episode varies per map, change the maximum episode length based on the maximum amount allowed by the map.
QMix
use open-source hyperparameters
MADDPG
Ir [e-3, 5e-4, le-4] tau [0.01, 0.005, 0.001]
MATD3
Ir [1e-3, 5e-4, le-4] tau [0.01, 0.005, 0.001]
MASAC
Ir [le-3, 5e-4, le-4] tau [0,01, 0.005, 0.001] target entropy coef [0.1. 0.3, 0.5, 0.7]
MАPPO
[le-3, 7c-4, 5e-4, lc-4], ppo epoch [5, 10, 15, 20, 25]
Table 10: Sweeping procedure the SMAC domain, the bold font indicates the value adopted.
network hyperparameters
value
initialization method num GRU layers
orthogonal
RNN hidden state dim fe layer dim num fo before after use feature normalization
64
64
2
True
optimizer optimizer eps weight decay ReLU ReLU gain
Adam
le-5
True
sqrt(2)
Table 11: network hyperparameters used in the StarCraft domain by all algorithms
hyperparameters
value depends on maps
episode length
last action layer gain
gamma
0.99
buffer size
5000
batch size
32
epsilon
from 1.0 0.05
epsilon anneal time Q-function loss MSE loss
50000
Table 12: Hyperparameters used in SMAC by MADDPG, MATD3, MASAC, and QMix. buffer size and batch size is calculated by episodes.
hyperparameters
value
parallel envs
episode length 400
last action layer gain
Table 13: Hyperparameters used in the SMAC domain by MAPPO
14

---

Under review as a conference paper at ICLR 2021
Maps
МАРРО
QMix
MASAC
MATD3
MADDPG
epoch=15
2m_vs_lz
100
100
90.88
89.84
89.06
3m
99.875
94.71
55.54
60.125
48.63
2s vs Isc
99.56
95.25
20.06
0
46.63
2s3z
98.5
94.07
23.52
3.125
14.59
38.v8_3
99.5625
97.03
0
0
0
3z
99.27
95.98
0
2c_vs_64zg
99.375
96.08
0
0
0
so_many_bamt baneling
99.68
93.18
62.43
45.28
56.15
8m
99.48
95.07
43.65
0
0
MMM
98.59
96.95
16.83
0
lc3s5z
100
96.09
24
0
0
3s57_vs_3s67
80
79.03
0
0
8m_vs_9m
92.97
93.44
0
bane_y vs. bane
94,4792
91.97
0
0
25m
96.88
86.15
0
5m_vs_0l
64.58
71.46
0
0
6h. vsv8._e 8z
8.59
79.32
corridor
0
75.625
0
0
0
epoch-5
3s_vs_5z
96.88
96.58
0
3s5z
96.84
82.15
0
0
MMM2
87.5
86.43
0
10m_vs_1lm
87.5
94.85
0
0
0
1r=7e-4epoch=5
27m_vs_30m
61.25
64.06
0
0
Table 14: Eval win rate in SMAC domain over 32 trials.
Hanabi
QMix
1e[1e-3, 5e-4, le-4, 2.5e-5]; hard update interval [200, 500, 900, 1500]
MADDPG
In[ 5c-4, Ic-4, 2.5c-5]; tau [0.001, 0.005, 0.01]
MATD3
Ir[le-3, 5e-4, le-4, 2.5e-5]; tau [0.001, 0.005, 0.01]
MASAC
1. 5e-4, le-4, 2.5c-5]: tau [0.001, 0.005, 0.01]; target entropy coef (.11, 0.3, 0.5, 0.7]
MAPPO
Ir[le-3, 7e-4, 5c-4, Ic-4]; ppo epoch [5, 10. 15, 20, 25]
Table 15: Sweeping procedure in the Hanabi domain, the bold font indicates the value adopted.
network hyperparameters
value orthogonal
initialization method num GRU layers
RNN hidden state dim
512
fo layer dim fc before num fe after use feature normalization optimizer optimizer eps weight decay ReLU ReLU gain
64
False
Adam
le-5
0
True
sqri(2)
Table 16: Network hyperparameters used in the Hanabi domain by all algorithms
15

---

Under review as a conference paper at ICLR 2021
hyperparameters episode length
value
80
last action layer gain
0.01
gamma
0.99
buffer size
5000
batch size
64
epsilon epsilon anneal time Q function loss
from 1.0 to 0.05
80000
MSE loss
Table 17: Hyperparameters used in the Hanabi domain by MADDPG, MATD3, MASAC, and QMix. Buffer size and batch size calculated by episodes.
hyperparameters
value
parallel envs
1000
episode length
80
last action layer gain
0.01
Table 18: Hyperparameters used in the Hanabi domain by MAPPO
The Hide-And-Seek (HNS) domain
QMix Ir[le-3, 5e-4, Te-4]; hard update interval [200, 500, 700, 900]
MADDPG
Ir[le-3, 5e-4, le-4]; tau [0.01, 0.005 0.001]
MATD3
Ir[1e-3, 5e-4, lc-4]; tau [0.01, 0.005 0.001]
MASAC
Ir[le-3, Se-4, le-4]; tau [0.01, 0.005, 0.001]
МАPPO
Ir| 1r]3e-3, 7e-, ppo epoch [5, 10,15, 20, 25]
Table 19: Sweeping procedure in the HNS domain
network hyperparameters
value
initialization method num GRU layers RNN hidden state dim fe layer dim num fc before num after feature normalization optimizer optimizer eps weight decay ReLU ReLU gain
orthogonal
64
64
2
True
Adam
le-5
0
True
sqrt(2)
Table 20: Network hyperparameters used in the HNS domain by all algorithms
16

---

Under review as a conference paper at ICLR 2021
hyperparameters
value
boxlocking: 120
episode length
blueprint construction: 200
last action layer gain
0.01
gamma
0.99
buffer size
5000
batch size
256
epsilon
from 1.0 0.05
epsilon anneal time Q function loss
50000
MSE loss
Table 21: Hyperparameters used in the HNS domain by MADDPG, MATD3, MASAC, and QMix.
Buffer size and batch size is calculated by episodes.
hyperparameters
value
boxlocking: 250
parallel envs
blueprint construction: 400
2 x task horizons
episode length
boxlocking: 240 construction:400
last action layer gain
0.01
Table 22: hyperparameters used in the HNS domain by MAPPO
B ALGORITHM DETAILS
We policies to refer to each unique actor and critic network pair and agents to refer to the actors in the environment. We will use m to denote the number of policies being trained, and n to be the number of active agents in the environment. We will each agent to the policy which controls it, and A, to be the number of agents under the control of policy In setting where all agents share the same parameters, policy, and would map all agents to this single policy. In a setting in which all agents have their own critic and actor, map each agent to its unique policy containing the actor and critic networks.
g(o) refer to the function which maps
1, there is only single
n,, the number of policies equals the number of agents, and g would
As notational simplification, in all of the outlined algorithms we will assume that the environment and all agents have a fixed horizon length T; modifying the algorithms to work with varying time horizons simple. We will also assume that the agents have discrete actions. For the MADDPG, MATD3, and MASAC algorithms, in order for actions to be differentiable in discrete setting, we use the gumbel softmax trick [aang al. (2016). which approximately samples an action from categorical distribution parameterized by the output logits of the actor network. Finally, we will always assume that the number of active agents in the environment is fixed across the horizon; this lets ensure that there is constant length input to the centralized critic.
The psuedocode of MADDPG, MASAC, QMix and MAPPO with the support of recurrent policies are shown in Algo. Algo.
Algo. 3 and Algo. 4 respectively
17

---

Under review as a conference paper at ICLR 2021
Algorithm 1: MADDPG
Initialize .1.-.. mm the parameters of agent critic networks, and 01.... Om the parameters of
the agent actor networks, using Orthogonal initialization (Hu et al. 2020);
Set the function mapping agent a index of the policy which controls the agent:
Set the learning rate batch size B and replay buffer D ():
0r 02 9,- Φ; for i 1...m;
step=0, episodes=0; while step < slepmar do
so = initial state;
initialize kn ... (m) actor RNN states;
{} empty trajectory:
for timestep t=l...7 do
for each agent do
i - g() Get the index the policy controlling agent a;
epsilon-schccull(ssep)
1
(a)
- :o)) using the Gumbel Softmax Trick randint(1,
pi(o:
(0)
with probability 1 with probability
end
Get reward rt, next state 8141, done d,, observations o!.....o,:: == TU {(s,, un Tri :]]]
step step + I;
end D DU:
episodes = episodes +
episodes 2 and iainninteerval training steps have passed then for each policy in I...m do
b + random batch B episodes from D for policy i; for each timestep in each episode in batch b do
Q: = Qi(se, ui ...., ut
(n)
..Q,1-1:0.))
0,);
Compute M441 (o h (/ ( for each agent a;
(a)
Q++1 (2(4111,1111)) M141,
(1) Mo-1
(n) (())
Qi = = + (1 - (e
Update RNN states 11 1/
11 (())
n.t jj [1,m] using transitions from batch
end
[(0,) 1 Ex
(Q 27)
0, - o,,,
Reset RNN states n1 11 i
for each timestep each episode in batch b do
for each agent controlled by policy i do
Q(a) Q( ui ,..., (o(o, (a) ((),())())
(n) (.) u(rh),)))
-i 0z);
Update RNN states h h ntt using transitions from batch b
end
end
(())
111 E.
A, BT BT
à: 6; -V,o C(ot):
0 = (1 -A)0, +A0,
6f (1 -A)Ø; + Adi
end
end
end
18

---

Under review as a conference paper at ICLR 2021
Algorithm 2: MASAC
Initialize .1.-.. fm the parameters of agent critic networks, and p1.... .... om the parameters of
the agent actor networks, using Orthogonal initialization (Hu et al. 2020);
Set the learning rate 6, polyak update rate and replay buffer U:
the initial entropy temperature, a, and the target entropy HI;
0, = fg oF- = Φ; for 1...m;
step=0, episodes=0;
while step S stepmar do
s0 initial state, ( empty trajectory;
initialize h10 ... hg
(m) actor RNN states;
for timestep t= do
for each agent do
i t g(a);
(a) (welo (a) (
ui
Update RNN state h to (a), end
Get reward re, next state $1415
т - ((),, Ut, fty 81+1)] step = step + 1;
end
D D UT:
episodes = episodes +
if episodes 2 and train-interval training steps have passed then for each policy in I...m do
b random batch of B episodes from D for policy i; for each timestep in each episode in batch b do
Q: (st, ui ,..,, u(()) ..Q 0:); ....,
(1)
(n)
Compute 4/41 ~ (g(a)(a++) 10 1 (7(a) for each agent a;
'(a),
Q++1 Q2(8141,, 0+1 ... 0++ 11 (())
(n)
Q, 0,);
Vi+1 Q++1 - (oo((T (ar+1 0
(a) (a))
())))
Oi = = (1-4))
Update RNN states 11 12 1
18)
x.... j [1, n] using transitions from batch
end
[(0,) = B7 E. 2
0, - -vo, ((,)):
Reset RNN states n 1
for each timestep t each episode in batch b do
for each agent controlled by policy do
Te (a) ~ x(ūe of (a) 1i ) Sample u. (a) using Gumbel Softmax Trick; Q. Q(st, ui
(1) u.. (a) (..,,,. un (8) 12)+
0z);
Update RNN states (a) (()) using transitions from batch b
end
end
((),)
111 E Σ, E, log(T(T))))))) ((0)
(0) (o)-1:)) Qui
A, BT
à di cV L(r));
0, = (1 A)0,- +A0;
6 (1 -AjØ, + +xo;;
((a) = A. BT
111 Et E. -a (ellog(<(7//))) - + A]:
(a) (a)
a= cV.L(a)
end
end
end
19

---

Under review as a conference paper at ICLR 2021
Algorithm 3: QMix
Initialize in. the hypernetwork parameters, and o, the parameters of the agent Q networks for
policy i, for {1 m). using Orthogonal initialization (Hu et al. 2020);
Set A = =0h, mm J. the collection of all parameters.;
Set 8 = 8;
Set the learning rate and replay buffer D ():
step = 0, episodes = 0:
while step slepmaz do
80 = initial state, T = ( empty trajectory;
initialize k1) hg actor RNN states;
for timestep t= ==I.. do
for each agent a do
i + g(a);
c = epsilon-sccheculllsste::
arg max ((w) (:(0, u( ,1()): randint(L,(U1)
with probability with probability
end
Get reward ra, next state 8141i == T {(s, u. Ti, st+1);; step step + 1;
end
D D UT:
episodes episodes + 1; episodes 2 B and
training steps have passed then b random batch of B episodes from D;
for each timestep I in each episode in batch b do
Q,
Hixer-net(Qgas(00 (1) 1 (g()), (o(n) M(( Q++1
dg()) hyperne((sr; 0n)):
Mixer-net(Qocy)(0ff)) (n) hi
(1)) o T0()) (o(m)
(n) ((n) : ((tn)) hypernet(844: 0 ));
= (1 de d++1i
Update RNN states
E [1, n] using transitions from batch
end
(()) B -,00- Q.)::
0 0= 8 aVoo(
end
if update-intervat training steps have passed then
end
end
20

---

Under review as a conference paper at ICLR 2021
Algorithm 4: MAPPO
Initialize 8,, the parameter for policy and Oi the parameter for critic i, for i {1 -m, using Orthogonal initialization (Hu et al. 2020
the learning rate o;; while step S slepmax do set data buffer D = { for batchsize b do
5 =
empty trajectory list;
initialize h/) ())... (1)))) actor RNN states;
for timestep := do
for each agent a do i + g(a);
u; (a) 1 = n (a) (8t, (0) :.):
o y(a) (st, i+)
end
Get reward Tt, next state 8t+1; T T [[(s,, un h, Vt, T2s 81+1)]}:
end
compute advantage A and value target V for 7 by applying GAB on // process a trajectory into chunks of length 10;
for timestep = 11,21, .I,...,,T
do
D D Du 10), Alt + 10), V' : + 10));
end
end
normalize all the advantages in to zero mean and one standard deviation; for mini-batch k= do for each policy i=l... do b 4- random mini-batch from D for policy i; for each data chunk e in the mini-batch do
update the RNN hidden states for each timestep in from the first hidden state;
end
Adam update for 0g with PPO objective and data
Adam update for di with value regression objective and data 6;
end
end
end
21


Reviews:
[{"review_id": "ryUkD0veg7u", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims to benchmark a suite of Multi-Agent Deep Reinforcement Learning algorithms across different environments in the cooperative multi-agent setting. The paper compares standard algorithms alongside extensions of well-known policy gradient algorithms to the multi-agent setting, i.e. PPO (MAPPO), SAC (MASAC) and TD3 (MATD3). The paper investigates the ease of using an algorithm by doing a fair hyperparameter search for different algorithms. The paper concludes by saying that MAPPO is a promising choice for tackling a multi-agent problem. \n\nSuch a benchmarking paper would absolutely be useful. In its current form, however, it is not yet ready for publication. There are three primary reasons. First, the scope is misplaced: the paper starts with the scope of benchmarking different algorithms but seems to shift focus primarily to MAPPOs benefits. Second, the experiments only include 3 runs, with large overlapping deviations, limiting the ability to make claims about any significant differences. This is particularly problematic for a paper where the main focus is on experiments to provide insight. Finally, it seems like Pop-art is only used with MAPPO (please clarify this if I am mistaken), calling into question if the main reason for improvements in MAPPO is this choice. Reward scaling, done by Pop-art, can significantly affect learning performance as pointed out in Henderson et al. (2017). Using Pop-art only for MAPPO makes the comparison unfair between different approaches and potentially invalidates the conclusions presented in the paper.\n\nAs two other more minor points:\n\n1. Multi-agent methods are built around numerous agents learning separate policies. The choice of sharing the same weights (Page 4, Sec 3.3) among different policies doesn’t seem to me like a fair choice. This also makes the algorithms different from their original intended use. \n\n2. There are several inconsistencies or questionable choices in the paper: \n\na. As per appendix, MADDP, MATD3, MASAC, QMix have gamma parameters, whereas there is no such parameter mentioned for MAPPO. Having a different gamma value for the methods essentially changes the problem.\n\nb. There is a difference between similar hyperparameters for different methods, i.e. MAPPO uses a gradient clipping of 20, whereas other methods use a gradient clipping of 10. How was this chosen and why is it different?\n\nc. Appendix (Hanabi), MAPPO uses a learning rate of 7e-4, whereas the learning rate is not mentioned as a value being tested in the search grid. \n\nd. MADDPG and MASAC are said to use a centralized critic, but according to Algorithm 1 and Algorithm 2, they seem to estimate separate critics. \n\ne. MASAC should have a tuple of three parameters (value functions, q function and policy), but Algorithm 2 talks about only two parameters (q function and policy). \n\n\nA few questions: \n1. In SMAC environments, why are MASAC, MATD3 and MADDPG not run for other settings (Table 2)? \n\n2. In MPE, I was not able to compare the results for MADDPG to the original paper. Can you point out the results (i.e. example the table number in Lowe et al. 2017)?  This also seems strange as Lowe et al. 2017 use a non-recurrent network with a different parameterization for each of the policies. In contrast, the current paper uses a recurrent network and shared policies, making the models quite different. \n\nSuggestions: \n1. Although the paper compares all the well-known policy gradient methods, comparing against a simple Vanilla Actor-Critic network would provide useful insight by removing some of the complexity.\n\n2. It would be good to include hyperparameters of PPO like the \\epsilon clipping.\n \n3. The graphs in Figure 3 do not have a visible axis.\n\n4. Including parameter sensitivity would provide much more insight into the properties of the algorithms,  and how they perform for general configuration of parameters. \n\n5. Section 3.1, P(s’|s,u) : \\mathcal{S} \\times U \\times \\mathcal{S} \\rightarrow [0,1], should have a U^n instead of U.  \n\nReferences\n1. Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2019). Deep Reinforcement Learning that Matters. \n2. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., & Mordatch, I. (2020). Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. \n\n--- Update after reading the rebuttal and other reviews\n\nThough some concerns have been addressed, a critical issue remains unresolved, which is that the experiments use only 3 runs. \n\nAs an additional point, it is useful that you've identified MAPPO as a good multi-agent algorithm. However, for clarity, it is better to focus the paper around MAPPO rather than strictly calling it a benchmarking paper. Further, it would be good to clearly state in the paper that the results now include reward normalization. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "LwVVAxtCY67", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The major contribution of this paper is benchmarking 5 MARL algorithms on 4 cooperative multi-agent environments. Also, this paper found that under constrained hyperparameter search budgets, the multi-agent PPO algorithm has more consistent performance over the other algorithms across different tested multi-agent environments. The code base is open-sourced for public use, which benefits the MARL community.\n\nResearchers in MARL often find it difficult to find a useful benchmark for multiagent learning algorithms and multiagent environments. Thus this paper makes a good contribution to the community, but, on the other hand, since this paper is not really intended to present any major technical contributions, that could be considered a weak point for an ICLR submission.\n\nTo the best of the reviewer's knowledge, there has not been any previous work that tries to produce a benchmark for multi-agent deep reinforcement learning. As the first work that attempts to fill this gap, this paper presents a comprehensive implementation of popular MARL algorithms tested on a representative list of MA environments, which is the major reason for acceptance suggestion. But there are some limitations of this paper. For example, there is little discussion about the results and the authors did not attempt to do more technical investigation for understanding the their findings about the algorithms. In this sense, this paper is less than a qualified research paper.\nIn particular, it would be helpful if  the authors have any insight on why the MAPPO works more consistently than the other algorithms.\n\nOne major suggestion is to add standard deviation to the numeric values presented in Table 1, 2, 3 and 4.\n\nIf this paper get accepted to this conference, peer researchers could be incentivized to contribute similarly comprehensive benchmark to this field, which would benefit the advancements of this field in the long run.    \n\nI have read over the rebuttal and discussion and will keep my evaluation score unchanged as I see value in benchmarking papers such as this for the community.", "extra_scores": {}, "extra_text": {}}, {"review_id": "JhfGV9EKQn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe paper provides a useful benchmark (a suite of cooperative multi-agent RL tasks) and a nice comparison of common as well as uncommon-but-successful algorithms on these tasks. A limited budget of hyperparameter tuning is applied to each algorithm in order to give each one a chance at good performance, and a clear winner emerges for consistent strong performance across tasks.\n\n## Quality & Clarity\nThe paper is presented according to a high standard of quality. They document and explain their methods clearly, including hyperparameters. They have clean descriptions of each algorithm and easily readable text.\n\nNit: for table 2, it would be easier to read if top scores (or, top scores within stdev) were bolded.\nNit: for table 3 with 2 players and possibly for table 4, it might be better to report results using a graph as this will show the progress that each algorithm made and document where they were cut off. This could even just be in the appendix.\nNit: you use box-locking-easy and box-locking-simple interchangeably--best remove one.\n\n## Originality\nI’m not aware of other work which provides as clean a comparison of both different tasks and different methods, so there is novelty in the open source benchmark that this provides. The work contains some original tasks built on top of existing ones (i.e. in the Hide and Seek domain). There are no original algorithms in this paper as it is not the focus of the work.\n\n## Significance\nFrom the conclusion:\n> “Given access to equally sized compute budgets for hyperparameter optimization, MAPPO is the most consistently successful algorithm across the studied environments, performing well at SMAC, as well as any algorithm in the MPEs and the best in hide-and-seek and Hanabi-small. Surprisingly, we also find that the centralized Q function variants DDPG, SAC, and TD3 almost entirely fail to solve any of the studied tasks, particularly in slightly harder environments.”\n\nThe main conclusion of the paper is “MAPPO is a good starting point for tackling new MARL problems”. This is a useful practical tip for researchers in the field. However, the work provides little insight into why that is the case. The next lines of the conclusion confirm that:\n\n> “Of course, these results are likely dependent on the choice of hyperparameters and architectural choices. Thus it may be the case that the initial point around which we perform our grid search for MA-DDPG/TD3/SAC does not contain any good solutions and that performance of a set of hyperparameters on the MPEs is not predictive of performance in other domains. Finding a better set of hyper-parameters over which to search for the off policy algorithms is an important direction for future work.”\n\nThe authors admit that they were only able to explore a limited space of hyperparameters and their work may not hold in other circumstances. There is plenty of room for continued exploration of hyperparameters, certainly, but even more valuable would be any indicators into why different techniques might have different performance on the different tasks. I think in order for the publication to get accepted to a conference like ICLR it likely needs a bit more investigation, though researchers may certainly find the benchmark useful in research.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bs8rH5w0G7C", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In order to compare different MARL algorithms more fairly, the author compared the differences in the performance of different algorithms(MADDPG, MATD3, MASAC, Qmix, MAPPO) in different environments(Particle-World, StarCraft Micromanagement, Hanabi, The Hide-and-Seek Domain).  The author's writing and organization are very good, so that I can clearly understand the content of the paper, and also has certain highlights, but I think it has not reached the ICLR criteria.\n\nFirst of all, as an article about BENCHMARKING MULTI-AGENT DEEP REINFORCE- MENT LEARNING ALGORITHMS, I think the author should compare a series of algorithms to bring more insightful analysis and inspiration. Unfortunately, I found that the author simply I enumerate the performance differences of different algorithms in different environments, and there is also a lack of analysis of what methods are suitable for what tasks. I suggest that the author conduct further analysis and experiments, and also pay attention to the advantages and disadvantages of different algorithms. I hope that the author can adjust the content of the paper, and don't make people feel that the author has collected different official implementations, and then simply run them and compare directly.\n\nIn addition, I think the author may be missing some important MARL algorithms. For example, in terms of policy-base, it is recommended that the author consider adding algorithms such as COMA[1] and MAAC[2] (the authors of these algorithms have already announced their official implementations). At the same time, most of the author's algorithms have official implementations on the Internet. How to compare different algorithms fairly is of great concern, but the author's content is still relatively weak, and the author is recommended to improve this part of the discussion.\n\nOverall, I vote for a rejection\n\n[1] Counterfactual Multi-Agent Policy Gradients, Foerster et.al. AAAI 2018\n\n[2] Actor-Attention-Critic for Multi-Agent Reinforcement Learning, Iqbal and Sha, ICML 2019", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a thorough comparison of different algorithms for multi-agent Deep RL methods. The conclusions of the paper is that across a variety of envionment and hyperparameter tuning, multi-agent PPO seems to peform well relatively to the competitors.

The reviewers agreed that the paper fills a gap in the literature regarding a fair and thorough comparison of algorithm, and that the paper clearly presents the results. As it stands, the code to reproduce the experiments and the results are a useful contribution to the community. However, the reviewers felt the technical contribution to be below the bar for ICLR, as the paper does not help in understanding the differences between algorithms, or develop insights as to how to further improve algorithms. The large standard deviations of the various algorithms also makes the main experimental insight (MAPPO works consistently well) relatively weak.


Decision: Reject