Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning

1 INTRODUCTION . Most real-world problems involve interactions between multiple agents and the problem becomes significantly harder when there exist complex cooperation and competition among agents . Inspired by the tremendous success of deep reinforcement learning ( RL ) in single-agent applications , such as Atari games ( Mnih et al. , 2013 ) , robotics manipulation ( Levine et al. , 2016 ) , and navigation ( Zhu et al. , 2017 ; Wu et al. , 2018 ; Yang et al. , 2019 ) , it has become a popular trend to apply deep RL techniques into multi-agent applications , including communication ( Foerster et al. , 2016 ; Sukhbaatar et al. , 2016 ; Mordatch & Abbeel , 2018 ) , traffic light control ( Wu et al. , 2017 ) , physical combats ( Bansal et al. , 2018 ) , and video games ( Liu et al. , 2019 ; OpenAI , 2018 ) . A fundamental challenge for multi-agent reinforcement learning ( MARL ) is that , as the number of agents increases , the problem becomes significantly more complex and the variance of policy gradients can grow exponentially ( Lowe et al. , 2017 ) . Despite the advances on tackling this challenge via actor-critic methods ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) , which utilize decentralized actors and centralized critics to stabilize training , recent works still scale poorly and are mostly restricted to less than a dozen agents . However , many real-world applications involve a moderately large population of agents , such as algorithmic trading ( Wellman et al. , 2005 ) , sport team competition ( Hausknecht & Stone , 2015 ) , and humanitarian assistance and disaster response ( Meier , 2015 ) , where one agent should collaborate and/or compete with all other agents . When directly applying the existing MARL algorithms to complex games with a large number of agents , as we will show in Sec . 5.3 , the agents may fail to learn good strategies and end up with little interaction with other agents even when collaboration is significantly beneficial . Yang et al . ( 2018 ) proposed a provably-converged meanfield formulation to scale up the actor-critic framework by feeding the state information and the average value of nearby agents ’ actions to the critic . However , this formulation strongly relies on the assumption that the value function for each agent can be well approximated by the mean of local ∗Equal contribution †Equal advising pairwise interactions . This assumption often does not hold when the interactions between agents become complex , leading to a significant drop in the performance . In this paper , we propose a general learning paradigm called Evolutionary Population Curriculum ( EPC ) , which allows us to scale up the number of agents exponentially . The core idea of EPC is to progressively increase the population of agents throughout the training process . Particularly , we divide the learning procedure into multiple stages with increasing number of agents in the environment . The agents first learn to play in simpler scenarios with less agents and then leverage these experiences to gradually adapt to later stages with more agents and ultimately our desired population . There are two key components in our curriculum learning paradigm . To process the varying number of agents during the curriculum procedure , the policy/critic needs to be population-invariant . So , we choose a self-attention ( Vaswani et al. , 2017 ) based architecture which can generalize to an arbitrary number of agents with a fixed number of parameters . More importantly , we introduce an evolutionary selection process , which helps address the misalignment of learning goals across stages and improves the agents ’ performance in the target environment . Intuitively , our within-stage MARL training objective only incentivizes agents to overfit a particular population in the current stage . When moving towards a new stage with a larger population , the successfully trained agents may not adapt well to the scaled environment . To mitigate this issue , we maintain multiple sets of agents in each stage , evolve them through cross-set mix-and-match and parallel MARL fine-tuning in the scaled environment , and select those with better adaptability to the next stage . EPC is RL-algorithm agnostic and can be potentially integrated with most existing MARL algorithms . In this paper , we illustrate the empirical benefits of EPC by implementing it on a popular MARL algorithm , MADDPG ( Lowe et al. , 2017 ) , and experimenting on three challenging environments , including a predator-prey-style individual survival game , a mixed cooperative-and-competitive battle game , and a fully cooperative food collection game . We show that EPC outperforms baseline approaches by a large margin on all these environments as the number of agents grows even exponentially . We also demonstrate that our method can improve the stability of the training procedure . 2 RELATED WORK . Multi-Agent Reinforcement Learning : It has been a long history in applying RL to multi-agent games ( Littman , 1994 ; Shoham et al. , 2003 ; Panait & Luke , 2005 ; Wright et al. , 2019 ) . Recently , deep RL techniques have been applied into the multi-agent scenarios to solve complex Markov games and great algorithmic advances have been achieved . Foerster et al . ( 2016 ) and He et al . ( 2016 ) explored a multi-agent variant of deep Q-learning ; Peng et al . ( 2017 ) studied a fully centralized actor-critic variant ; Foerster et al . ( 2018 ) developed a decentralized multi-agent policy gradient algorithm with a centralized baseline ; Lowe et al . ( 2017 ) proposes the MADDPG algorithm which extended DDPG to the multi-agent setting with decentralized policies and centralized Q functions . Our population curriculum approach is a general framework for scaling MARL which can be potentially combined with any of these algorithms . Particularly , we implement our method on top of the MADDPG algorithm in this paper and take different MADDPG variants as baselines in experiments . There are also other works studying large-scale MARL recently ( Lin et al. , 2018 ; Jiang & Lu , 2018 ; Yang et al. , 2018 ; Suarez et al. , 2019 ) , which typically simplify the problem by weight sharing and taking only local observations . We consider a much more general setting with global observations and unsharedweight agents . Additionally , our approach is a general learning paradigm which is complementary to the specific techniques proposed in these works . Attention-Based Policy Architecture : Attention mechanism is widely used in RL policy representation to capture object level information ( Duan et al. , 2017 ; Wang et al. , 2018 ) , represent relations ( Zambaldi et al. , 2018 ; Malysheva et al. , 2018 ; Yang et al. , 2019 ) and extract communication channels ( Jiang & Lu , 2018 ) . Iqbal & Sha ( 2019 ) use an attention-based critic . In our work , we utilize an attention module in both policy and critic , inspired by the transformer architecture ( Vaswani et al. , 2017 ) , for the purpose of generalization to an arbitrary number of input entities . Curriculum Learning : Curriculum learning can be tracked back to Elman ( 1993 ) , and its core idea is to “ start small ” : learn the easier aspects of the task first and then gradually increase the task difficulty . It has been extended to deep neural networks on both vision and language tasks ( Bengio et al. , 2009 ) and much beyond : Karras et al . ( 2017 ) propose to progressively increase the network capacity for synthesizing high quality images ; Murali et al . ( 2018 ) apply a curriculum over the control space for robotic manipulation tasks ; several works ( Wu & Tian , 2016 ; Florensa et al. , 2017 ; Sukhbaatar et al. , 2017 ; Wang et al. , 2019 ) have proposed to first train RL agents on easier goals and switch to harder ones later . Baker et al . ( 2019 ) show that multi-agent self-play can also lead to autocurricula in open-ended environments . In our paper , we propose to progressively increase the number of the agents as a curriculum for better scaling multi-agent reinforcement learning . Evolutionary Learning : Evolutionary algorithms , originally inspired by Darwin ’ s natural selection , has a long history ( Bäck & Schwefel , 1993 ) , which trains a population of agents in parallel , and let them evolve via crossover , mutation and selection processes . Recently , evolutionary algorithms have been applied to learn deep RL policies with various aims , such as to enhance training scalability ( Salimans et al. , 2017 ) , to tune hyper-parameters ( Jaderberg et al. , 2017 ) , to evolve intrinsic dense rewards ( Jaderberg et al. , 2018 ) , to learn a neural loss for better generalization ( Houthooft et al. , 2018 ) , to obtain diverse samples for faster off-policy learning ( Khadka & Tumer , 2018 ) , and to encourage exploration ( Conti et al. , 2018 ) . Leveraging this insight , we apply evolutionary learning to better scale MARL : we train several groups of agents in each curriculum stage and keep evolving them to larger populations for the purpose of better adaptation towards the desired population scale and improved training stability . Czarnecki et al . ( 2018 ) proposed a similar evolutionary mix-and-match training paradigm to progressively increase agent capacity , i.e. , larger action spaces and more parameters . Their work considers a fixed environment with an increasingly more complex agent and utilizes the traditional parameter crossover and mutation during evolution . By contrast , we focus on scaling MARL , namely an increasingly more complex environment with a growing number of agents . More importantly , we utilize MARL fine-tuning as an implicit mutation operator rather than the classical way of mutating parameters , which is more efficient , guided and applicable to even a very small number of evolution individuals . A similar idea of using learning for mutation is also considered by Gangwani & Peng ( 2018 ) in the single-agent setting . 3 BACKGROUND . Markov Games : We consider a multi-agent Markov decision processes ( MDPs ) ( Littman , 1994 ) . Such an N -agent Markov game is defined by state space S of the game , action spacesA1 , ... , AN and observation spaces O1 , ... , ON for each agent . Each agent i receives a private observation correlated with the state oi : S 7→ Oi and produces an action by a stochastic policy πθi : Oi × Ai 7→ [ 0 , 1 ] parameterized by θi . Then the next states are produced according to the transition function T : S ×A1 × ... ×AN 7→ S . The initial state is determined by a distribution ρ : S 7→ [ 0 , 1 ] . Each agent i obtains rewards as a function of the state and its action ri : S ×Ai 7→ R , and aims to maximize its own expected return Ri = ∑T t=0 γ trti ( s t , ati ) , where γ is a discount factor and T is the time horizon . To minimize notation , we omit subscript of policy when there is no ambiguity . Multi-Agent Deep Deterministic Policy Gradient ( MADDPG ) : MADDPG ( Lowe et al. , 2017 ) is a multi-agent variant of the deterministic policy gradient algorithm ( Silver et al. , 2014 ) . It learns a centralized Q function for each agent which conditions on global state information to resolve the non-stationary issue . Consider N agents with deterministic policies µ = { µ1 , ... , µN } where µi : Oi 7→ Ai is parameterized by θi . The policy gradient for agent i is : ∇θiJ ( θi ) = Ex , a∼D [ ∇θiµi ( oi ) ∇aiQ µ i ( x , a1 , ... , aN ) |ai=µi ( oi ) ] , ( 1 ) Here D denotes the replay buffer while Qµi ( x , a1 , ... , aN ) is a centralized action-value function for agent i that takes the actions of all agents , a1 , . . . , aN and the state information x ( i.e. , x = ( o1 , ... , oN ) or simply x = s if s is available ) . Let x′ denote the next state from the environment transition . The replay bufferD contains experiences in the form of tuples ( x , x′ , a1 , . . . , aN , r1 , . . . , rN ) . Suppose the centralized critic Qµi is parameterized by φi . Then it is updated via : L ( φi ) = Ex , a , r , x′ [ ( Qµi ( x , a1 , . . . , aN ) − y ) 2 ] , y = ri + γ Q µ′ i ( x ′ , a′1 , . . . , a ′ N ) ∣∣ a′j=µ ′ j ( oj ) , ( 2 ) where µ′ = { µθ′1 , ... , µθ′N } is the set of target policies with delayed parameters θ ′ i . Note that the centralized critic is only used during training . At execution time , each policy µθi remains decentralized and only takes local observation oi .

Summary: This paper proposes a new method of scaling multi-agent reinforcement learning to a larger number of agents using evolution. Specifically, the procedures (EPC) involves starting with a small number of agents, training multiple sets in parallel, and doing crossover to find the set of agents that generalize best to a larger number of agents. This is motivated by the intuition that agents that perform best in small groups may not be the ones that perform best in larger groups. These claims are empirically verified in three games based on the particle world set of environments.