Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination

1 INTRODUCTION . Cooperative multiagent reinforcement learning ( MARL ) studies how multiple agents can learn to coordinate as a team toward maximizing a global objective . Cooperative MARL has been applied to many real world applications such as air traffic control ( Tumer and Agogino , 2007 ) , multi-robot coordination ( Sheng et al. , 2006 ; Yliniemi et al. , 2014 ) , communication and language ( Lazaridou et al. , 2016 ; Mordatch and Abbeel , 2018 ) , and autonomous driving ( Shalev-Shwartz et al. , 2016 ) . Many such environments endow agents with a team reward that reflects the team ’ s coordination objective , as well as an agent-specific local reward that rewards basic skills . For instance , in soccer , dense local rewards could capture agent-specific skills such as passing , dribbling and running . The agents must then coordinate when and where to use these skills in order to optimize the team objective , which is winning the game . Usually , the agent-specific reward is dense and easy to learn from , while the team reward is sparse and requires the cooperation of all or most agents . Having each agent directly optimize the team reward and ignore the agent-specific reward usually fails or is sample-inefficient for complex tasks due to the sparsity of the team reward . Conversely , having each agent directly optimize the agent-specific reward also fails because it does not capture the team ’ s objective , even with state of the art multiagent RL algorithms such as MADDPG ( Lowe et al. , 2017 ) . One solution to this problem is to use reward shaping , where extensive domain knowledge about the task is used to create a proxy reward function ( Rahmattalabi et al. , 2016 ) . Constructing this proxy reward function is difficult in complex environments , and is domain-dependent . Apart from requiring domain knowledge and manual tuning , this approach also poses risks of changing the underlying problem itself ( Ng et al. , 1999 ) . Simple approaches to creating a proxy reward via linear combinations of the two objectives also fail to solve or generalize to complex coordination tasks ( Devlin et al. , 2011 ; Williamson et al. , 2009 ) . In this paper , we introduce Multiagent Evolutionary Reinforcement Learning ( MERL ) , a state-ofthe-art algorithm for cooperative MARL that does not require reward shaping . MERL is a split-level training platform that combines gradient-based and gradient-free optimization . The gradient-free optimizer is an evolutionary algorithm that maximizes the team objective through neuroevolution . The gradient-based optimizer is a policy gradient algorithm that maximizes each agent ’ s dense , local rewards . These gradient-based policies are periodically copied into the evolutionary population . The two processes operate concurrently and share information through a shared replay buffer . A key strength of MERL is that it is a general method which does not require domain-specific reward shaping . This is because MERL optimizes the team objective directly while simultaneously leveraging agent-specific rewards to learn basic skills . We test MERL in a number of multiagent coordination benchmarks . Results demonstrate that MERL significantly outperforms state-of-the-art methods such as MADDPG , while using the same observations and reward functions . We also demonstrate that MERL scales gracefully to increasing complexity of coordination objectives where MADDPG and its variants fail to learn entirely . 2 BACKGROUND AND RELATED WORK . Markov Games : A standard reinforcement learning ( RL ) setting is often formalized as a Markov Decision Process ( MDP ) and consists of an agent interacting with an environment over a finite number of discrete time steps . This formulation can be extended to multiagent systems in the form of partially observable Markov games ( Littman , 1994 ; Lowe et al. , 2017 ) . An N -agent Markov game is defined by a global state of the world , S , and a set of N observations { Oi } and N actions { Ai } corresponding to theN agents . At each time step t , each agent observes its corresponding observation Oti and maps it to an action A t i using its policy πi . Each agent receives a scalar reward rti based on the global state St and joint action of the team . The world then transitions to the next state St+1 which produces a new set of observations { Oi } . The process continues until a terminal state is reached . Ri = ∑T t=0 γ trti is the total return for agent i with discount factor γ ∈ ( 0 , 1 ] . Each agent aims to maximize its expected return . TD3 : Policy gradient ( PG ) methods frame the goal of maximizing the expected return as the minimization of a loss function . A widely used PG method for continuous , high-dimensional action spaces is DDPG ( Lillicrap et al. , 2015 ) . Recently , ( Fujimoto et al. , 2018 ) extended DDPG to Twin Delayed DDPG ( TD3 ) , addressing its well-known overestimation problem . TD3 is the state-of-the-art , off-policy algorithm for model-free DRL in continuous action spaces . TD3 uses an actor-critic architecture ( Sutton and Barto , 1998 ) maintaining a deterministic policy ( actor ) π : S → A , and two distinct criticsQ : S ×A → Ri . Each critic independently approximates the actor ’ s action-value function Qπ . A separate copy of the actor and critics are kept as target networks for stability and are updated periodically . A noisy version of the actor is used to explore the environment during training . The actor is trained using a noisy version of the sampled policy gradient computed by backpropagation through the combined actor-critic networks . This mitigates overfitting of the deterministic policy by smoothing the policy gradient updates . Evolutionary Reinforcement Learning ( ERL ) is a hybrid algorithm that combines Evolutionary Algorithms ( EAs ) ( Floreano et al. , 2008 ; Lüders et al. , 2017 ; Fogel , 2006 ; Spears et al. , 1993 ) , with policy gradient methods ( Khadka and Tumer , 2018 ) . Instead of discarding the data generated during a standard EA rollout , ERL stores this data in a central replay buffer shared with the policy gradient ’ s own rollouts - thereby increasing the diversity of the data available for the policy gradient learners . Since the EA directly optimizes for episode-wide return , it biases exploration towards states with higher long-term returns . The policy gradient algorithm which learns using this state distribution inherits this implicit bias towards long-term optimization . Concurrently , the actor trained by the policy gradient algorithm is inserted into the evolutionary population allowing the EA to benefit from the fast gradient-based learning . Related Work : Lowe et al . ( 2017 ) introduced MADDPG which tackled the inherent non-stationarity of a multiagent learning environment by leveraging a critic which had full access to the joint state and action during training . Foerster et al . ( 2018b ) utilized a similar setup with a centralized critic across agents to tackle StarCraft micromanagement tasks . An algorithm that could explicitly model other agents ’ learning was investigated in Foerster et al . ( 2018a ) . However , all these approaches rely on a dense agent reward that properly captures the team objective . Methods to solve for these types of agent-specific reward functions were investigated in Li et al . ( 2012 ) but were limited to tasks with strong simulators where tree-based planning could be used . A closely related work to MERL is ( Liu et al. , 2019 ) where Population-Based Training ( PBT ) ( Jaderberg et al. , 2017 ) is used to optimize the relative importance between a collection of dense , shaped rewards automatically during training . This can be interpreted as a singular central reward function constructed by scalarizing a collection of reward signals where the scalarization coefficients are adaptively learned during training . In contrast , MERL optimizes its reward functions independently with information transfer across them facilitated through shared replay buffers and policy migration directly . This form of information transfer through a shared replay buffer has been explored extensively in recent literature ( Colas et al. , 2018 ; Khadka et al. , 2019 ) . 3 MULTIAGENT EVOLUTIONARY REINFORCEMENT LEARNING . MERL leverages both agent-specific and team objectives through a hybrid algorithm that combines gradient-free and gradient-based optimization . The gradient-free optimizer is an evolutionary algorithm that maximizes the team objective through neuroevolution . The gradient-based optimizer trains policies to maximize agent-specific rewards . These gradient-based policies are periodically added to the evolutionary population and participate in evolution . This enables the evolutionary algorithm to use agent-specific skills learned by training on the agent-specific rewards toward optimizing the team objective without needing to resort to reward shaping . Algorithm 1 Multiagent Evolutionary Reinforcement Learning 1 : Initialize a population of k multi-head teams popπ , each with weights θπ initialized randomly 2 : Initialize a shared critic Q with weights θQ 3 : Initialize an ensemble of N empty cyclic replay buffersRk , one for each agent 4 : Define a white Gaussian noise generatorWg random number generator r ( ) ∈ [ 0 , 1 ) 5 : for generation = 1 , ∞ do 6 : for team π ∈ popπ do 7 : g , R = Rollout ( π , R , noise=None , ξ ) 8 : _ , R = Rollout ( π , R , noise=Wg , ξ = 1 ) 9 : Assign g as π ’ s fitness 10 : end for 11 : Rank the population popπ based on fitness scores 12 : Select the first e teams π ∈ popπ as elites 13 : Select the remaining ( k − e ) teams π from popπ , to form Set S using tournament selection 14 : while |S| < ( k − e ) do 15 : Single-point crossover between a randomly sampled π ∈ e and π ∈ S and append to S 16 : end while 17 : for Agent k=1 , N do 18 : Randomly sample a minibatch of T transitions ( oi , ai , li , oi+1 ) from Rk 19 : Compute yi = li + γ min j=1,2 Q′j ( oi+1 , a∼|θQ ′ j ) 20 : where a∼ = π′pg ( k , oi+1|θπ ′ pg ) [ action sampled from the kth head of π′pg ] + 21 : Update Q by minimizing the loss : L = 1T ∑ i ( yi −Q ( oi , ai|θQ ) 2 22 : Update πkpg using the sampled policy gradient ∇θπpgJ ∼ 1 T ∑ ∇aQ ( o , a|θQ ) |o=oi , a=ai∇θπpgπ k pg ( s|θπpg ) |o=oi 23 : Soft update target networks : θπ ′ ⇐ τθπ + ( 1− τ ) θπ′ and θQ′ ⇐ τθQ + ( 1− τ ) θQ′ 24 : end for 25 : Migrate the policy gradient team popj : for weakest π ∈ popjπ : θπ ⇐ θπpg 26 : end for Policy Topology : We represent our multiagent ( team ) policies using a multi-headed neural network π as illustrated in Figure 1 . The head πk represents the k-th agent in the team . Given an incoming observation for agent k , only the output of πk is considered as agent k ’ s response . In essence , all agents act independently based on their own observations while sharing weights ( and by extension , the features ) in the lower layers ( trunk ) . This is commonly used to improve learning speed ( Silver et al. , 2017 ) . Further , each agent k also has its own replay buffer ( Rk ) which stores its experience defined by the tuple ( state , action , next state , local reward ) for each interaction with the environment ( rollout ) involving that agent . Team Reward Optimization : Figure 2 illustrates the MERL algorithm . A population of multi-headed teams , each with the same topology , is initialized with random weights . The replay bufferRk is shared by the k-th agent across all teams . The population is then evaluated for each rollout . The team reward for each team is disbursed at the end of the episode and is considered as its fitness score . A selection operator selects a portion of the population for survival with probability proportionate to their fitness scores . The weights of the teams in the population are probabilistically perturbed through mutation and crossover operators to create the next generation of teams . A portion of the teams with the highest relative fitness are preserved as elites . At any given time , the team with the highest fitness , or the champion , represents the best solution for the task . Policy Gradient : The procedure described so far resembles a standard EA except that each agent k stores each of its experiences in its associated replay buffer ( Rk ) instead of just discarding it . However , unlike EA , which only learns based on the low-fidelity global reward , MERL also learns from the experiences within episodes of a rollout using policy gradients . To enable this kind of `` local learning '' , MERL initializes one multi-headed policy network πpg and one critic Q . A noisy version of πpg is then used to conduct its own set of rollouts in the environment , storing each agent k ’ s experiences in its corresponding buffer ( Rk ) similar to the evolutionary rollouts . Agent-Specific Reward Optimization : Crucially , each agent ’ s replay buffer is kept separate from that of every other agent to ensure diversity amongst the agents . The shared critic samples a random mini-batch uniformly from each replay buffer and uses it to update its parameters using gradient descent . Each agent πkpg then draws a mini-batch of experiences from its corresponding buffer ( Rk ) and uses it to sample a policy gradient from the shared critic . Unlike the teams in the evolutionary population which directly seek to optimize the team reward , πpg seeks to maximize the agent-specific local reward while exploiting the experiences collected via evolution . Skill Migration : Periodically , the πpg network is copied into the evolving population of teams and can propagate its features by participating in evolution . This is the core mechanism that combines policies learned via agent-specific and team rewards . Regardless of whether the two rewards are aligned , evolution ensures that only the performant derivatives of the migrated network are retained . This mechanism guarantees protection against destructive interference commonly seen when a direct scalarization between two reward functions is attempted . Further , the level of information exchange is automatically adjusted during the process of learning , in contrast to being manually tuned by an expert designer . Algorithm 1 provides a detailed pseudo-code of the MERL algorithm . The choice of hyperparameters is explained in the Appendix . Additionally , our source code 1 is available online . 1https : //tinyurl.com/y6erclts

Summary: This paper proposes an algorithm to learn coordination strategies for multi-agent reinforcement learning. It combines gradient-based optimization (Actor-critic) with Neuroevolution (genetic algorithms style). Specifically, Actor-critic is used to train an ensemble of agents (referred to as “team”) using a manually designed agent-specific reward. Coordination within a team is then learned with Neuroevolution. The overall design accommodates sharing of data between Actor-critic and Neuroevolution, and migration of policies. Evaluation is done using the multi-particle environments (Lowe et. al. 2017) and a Rover domain task.