Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms

We benchmark commonly used multi-agent deep reinforcement learning ( MARL ) algorithms on a variety of cooperative multi-agent games . While there has been significant innovation in MARL algorithms , algorithms tend to be tested and tuned on a single domain and their average performance across multiple domains is less characterized . Furthermore , since the hyperparameters of the algorithms are carefully tuned to the task of interest , it is unclear whether hyperparameters can easily be found that allow the algorithm to be repurposed for other cooperative tasks with different reward structure and environment dynamics . To investigate the consistency of the performance of MARL algorithms , we build an open-source library of multi-agent algorithms including DDPG/TD3/SAC with centralized Q functions , PPO with centralized value functions , as well as QMix , and test them across a range of tasks that vary in coordination difficulty and agent number . The domains include the Multi-agent Particle World environments , StarCraftII micromanagement challenges , the Hanabi challenges , and the Hide-And-Seek environments . Finally , we investigate the ease of hyperparameters tuning for each of the algorithms by tuning hyperparameters in one environment per domain and re-using them in the other environments within the domain . The open-source code and more details can be found in our website : https : //sites.google . com/view/marlbenchmarks . 1 INTRODUCTION . Widespread availability of high-speed computing , neural network architectures , and advances in reinforcement learning ( RL ) algorithms have led to a continuing series of interesting results in building cooperative artificial agents : agents collectively playing Hanabi to an expert level ( Hu & Foerster , 2019 ) , designing cooperative StarCraftII bots ( Rashid et al. , 2018 ) that outperform hand-designed heuristics , and constructing emergent languages between agents ( Mordatch & Abbeel , 2017 ) . Each of these aforementioned results have often come with the introduction of a new algorithm , leading to a proliferation of new algorithms that is rapidly advancing the field . However , these algorithms are often designed and tuned to get optimal performance in a particular deployment environment . In particular , it is not unusual for each new algorithm to come with a new proposed benchmark on which it is evaluated . Consequently it is not obvious that these algorithms can easily be re-purposed for new tasks ; subtle interactions between the algorithm , the architecture and the environment may lead to high asymptotic performance on one task and total failure when applied to a new task . Without examining an algorithm across a range of tasks , it is difficult to assess how general purpose it is . Furthermore , the high asymptotic rewards that are often presented may hide complexities in using the algorithms in practice . The amount of time that researchers spent in finding optimal hyperparameters is often obscured , making it unclear how extensive of a hyperparameter search was needed to find the good hyperparameters . That is , researchers will often report a grid search of hyperparameters but not the prior work that was done to pick out a hyperparameter grid that actually contained good hyperparameters . Furthermore , the amount of computation provided to tune the studied algorithm may not be provided to the baseline algorithms that it will be compared against . This can lead to an inflated performance of the proposed algorithm relative to the benchmarks . All these problems can arise without any ill intent on the part of the authors , but they make the problem of assessing algorithms quite challenging . The downstream consequence of this proliferation of algorithms coupled with an absence of standard benchmarks is a lack of clarity on the part of practitioners as to which algorithm will give consistent , high performance with minimal tuning . Researchers are often operating under computational constraints that limit how extensive of a hyper-parameter sweep they can perform ; the ease with which good hyperparameters can be found is consequently a useful metric . When tackling a new multi-agent problem , researchers have no clear answer to the questions : 1 ) which MARL algorithm should I use to maximize performance and 2 ) given my computational resources , which algorithm is likeliest to work under my constraints ? We present an attempt to evaluate the performance , robustness and the relative ease of using these algorithms by benchmarking them across a wide variety of environments that vary in both agent number , exploration difficulty , and coordination complexity . By exploring a large range of possible environments , we identify algorithms that perform well on average and serve as a strong starting point for a variety of problems . We tackle the question of relative difficulty in finding hyperparameters by looking at how hyperparameters transfer across settings : tuning hyperparameters on one set of environments and applying them without re-tuning on the remaining environments . Using this procedure , we can provide effective recommendations on algorithm choice for researchers attempting to deploy deep multi-agent reinforcement learning while operating under constrained hyper-parameter budgets . We test Proximal Policy Optimization ( Schulman et al. , 2017 ) with centralized value functions ( MAPPO ) , Multi-Agent DDPG ( MADDPG ) ( Lowe et al. , 2017 ) , Multi-Agent TD3 ( Fujimoto et al. , 2018a ) ( MATD3 ) , a Multi-Agent variant of Soft Actor Critic ( Haarnoja et al. , 2018 ) ( MASAC ) , and QMix ( Rashid et al. , 2018 ) . We focus specifically on the performance of these algorithms on fully cooperative tasks , as this avoids game theoretic issues around computing the distance to Nash equilibria , and allows us to solely characterize performance in terms of asymptotic reward . The contributions of this paper are the following • Benchmarking multi-agent variants of single-agent algorithms across a wide range of possible tasks including StarCraftII micromanagement ( Rashid et al. , 2019 ) , Multi-agent Particle World ( Mordatch & Abbeel , 2017 ) , Hanabi ( Bard et al. , 2020 ) , and the Hide-AndSeek domain ( Baker et al. , 2019 ) . • Establishing that under constrained hyperparameter searching budgets , the multi-agent variant of PPO appears to be the most consistent algorithm across different domains . • The design and release of a new multi-agent library of various on/off-policy learning algorithms with recurrent policy support . 2 RELATED WORK . MARL algorithms have a long history but have , until recently , primarily been applied in tabular settings ( Littman , 1994 ; Busoniu et al. , 2008 ) . Notions of using a Q-function that operated on the actions of all agents , known as Joint-Action Learners ( Claus & Boutilier , 1998 ) have existed in the literature since its inception with algorithms like Hyper-Q ( Tesauro , 2004 ) using inferred estimates of other agent strategies in the Q-function . Recent MARL algorithms have built upon these ideas by incorporating neural networks ( Tampuu et al. , 2017 ) , policy-gradient methods ( Foerster et al. , 2017 ) , and finding ways to combine local and centralized Q-functions to enable centralized learning with decentralized execution ( Lowe et al. , 2017 ; Sunehag et al. , 2018 ) . Alongside the proliferation of algorithms has come a wide variety of new , cooperative MARL benchmarks . Unlike single-agent RL , where MuJoCo ( Todorov et al. , 2012 ) and Atari ( Mnih et al. , 2013 ) have become standard benchmarks , there is significantly less consensus on appropriate benchmarks . In this work , we consider 4 popular multi-agent environments , which we believe are the most representative in the community . Besides those we considered in this work , other interesting cooperative environments may include MAgent ( Zheng et al. , 2017 ) , a platform that can efficiently support hundreds of particle agents for cooperative tasks , multi-agent MuJoCo , in which each joint is an independent agent ( Schroeder de Witt et al. , 2020 ) , and CityFlow ( Zhang et al. , 2019 ) , which studies large-scale decentralized traffic light control . There also has been a variety of attempts to benchmark MARL algorithms that differ in scope from our paper . Gupta et al . ( 2017 ) benchmarks a similar set of algorithms to ours on a wide variety of environments . However , they do not consider algorithms that train in a centralized fashion while acting decentralized and instead perform a comparison between fully centralized training and execution and full decentralized algorithms . They establish that parameter sharing is an essential component of getting quick convergence in MARL algorithms . Schroeder de Witt et al . ( 2020 ) benchmark algorithms with centralized Q and value functions on a decentralized variant of the MuJoCo environments ; however , they primarily study variants of QMix and MADDPG and do not compare with on-policy algorithms . 3 MARL ALGORITHMS . 3.1 PRELIMINARIES . We study decentralized partially observed Markov decision processes ( DEC-POMDP ) ( Oliehoek et al. , 2016 ) with global rewards . A DEC-POMDP is defined by an eight tuple 〈S , U , P , r , Z , O , n , γ〉 . s ∈ S is a state space describing a sufficient set of state variables to make the system dynamics Markovian . For simplicity we assume the agents share action space U and each agent a ∈ { 1 , . . . , n } picks an action ua ∈ U which are concatenated to form join action u ∈ Un . We denote the joint action without the action of agent a as u−a . Joint actions u and state are fed to state transition function P ( s′|s , u ) : S × U × S → [ 0 , 1 ] . These are cooperative tasks so all agents share a bounded reward function r ( s , u ) : S × Un → R and have shared discount factor γ ∈ [ 0 , 1 ] Each agent i has an observation function Oa ( s ) : S → Z which defines how the global state is mapped onto a local observation z ∈ Z . Each agent maintains an action-observation history τa ∈ T ∈ ( Z ×U ) ∗ which it conditions its policy πa ( ua|τa ) : T×U → [ 0 , 1 ] on . Finally , given the joint policy π ( u ) = ∏ i π a ( ua|τa ) we can define a joint value function V π ( st ) = Est+1 : ∞ [ ∑∞ i=0 γ irt+i|st ] and joint Q function Qπ ( st , ut ) = Est+1 : ∞ [ ∑∞ i=0 γ irt+i|st , ut ] . We assume that the learning algorithm has access to both true states S , as well as the trajectories of all agents τa . The agents however , only have access to τa for computing their policy . The goal of the agents is to jointly optimize the quantity Jπ = E [ ∑∞ i=0 γ ir ( st , u ) ] . 3.2 BASELINE ALGORITHMS . We introduce all the baseline algorithms we consider , including MADDPG , MATD3 , MASAC , QMix and MAPPO . For all problems considered , the action space is discrete . More algorithmic details and the complete pseudo-code can be found in the appendix . MADDPG : The MADDPG algorithm is perhaps the most popular general-purpose off-policy MARL algorithm . The algorithm was proposed by Lowe et al . ( 2017 ) , based on the DDPG algorithm ( Lillicrap et al. , 2015 ) , and uses a centralized Q-function taking observations and actions from all the agents to alleviate the non-stationarity issue and stabilize multi-agent training . Note that although DDPG was originally designed for continuous actions , MADDPG adopts the gumbel-softmax ( Jang et al. , 2016 ) trick to handle discrete actions . MATD3 : The TD3 algorithm ( Fujimoto et al. , 2018b ) is a popular enhanced version of the standard DDPG algorithm ( Lillicrap et al. , 2016 ) . We similarly apply the same centralized critic technique introduced in MADDPG to TD3 to derive a multi-agent variant of TD3 , i.e. , MATD3 . The only difference between MATD3 and MADDPG is the use of twin delayed critics and the addition of small amounts of noise to the actions sampled from the buffer . MASAC : The Soft Actor-Critic ( SAC ) algorithm ( Haarnoja et al. , 2018 ) is an extremely popular off-policy algorithm and has been considered as a state-of-the-art baseline for a diverse range of RL problems with continuous actions . Similar to MADDPG , we introduce a centralized critic in SAC to achieve another general-purpose MARL algorithm , MASAC . QMix : QMix ( Rashid et al. , 2018 ) is a Q-learning algorithm designed for multi-agent cooperative tasks with a global reward . The core idea of QMix is value decomposition , which formulates the global Q function , Qtot as the output of a ” mixer ” neural network whose inputs are the individual agent Q functions , Qa ; The weights of this ” mixer ” network are constrained to be positive in order to insure that ∂Qtot∂Qa ≥ 0 , ∀ agents a . This ensures that by acting greedily to maximize their local Q functions , agents will also be maximizing the global Q function . QMix was first introduced in the StarCraftII micromanagement and has been a popular benchmark algorithm for this challenge . However , it is rarely tested in other domains . MAPPO : In addition to the off-policy algorithms above , we also consider an on-policy MARL algorithm , i.e. , a multi-agent variant of PPO ( MAPPO ) . We enhance the standard decentralized PPO algorithm by learning a centralized critic that takes in the global state or the concatenation of each agent ’ s local observation ( when global state is not available ) . This idea of centralized value functions was originally introduced in the COMA algorithm ( Foerster et al. , 2018 ) . Our implementation follows the details in Baker et al . ( 2019 ) , including using two separate networks for policy and value function , GAE , advantage normalization and a Pop-Art value normalizer ( Hessel et al. , 2019 ) , which uses running statistics of the values to normalizes the regression target of value network to zero mean and a standard deviation of one .

Summary: The major contribution of this paper is benchmarking 5 MARL algorithms on 4 cooperative multi-agent environments. Also, this paper found that under constrained hyperparameter search budgets, the multi-agent PPO algorithm has more consistent performance over the other algorithms across different tested multi-agent environments. The code base is open-sourced for public use, which benefits the MARL community.