Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination

1 INTRODUCTION . Cooperative multiagent reinforcement learning ( MARL ) studies how multiple agents can learn to coordinate as a team toward maximizing a global objective . Cooperative MARL has been applied to many real world applications such as air traffic control ( Tumer and Agogino , 2007 ) , multi-robot coordination ( Sheng et al. , 2006 ; Yliniemi et al. , 2014 ) , communication and language ( Lazaridou et al. , 2016 ; Mordatch and Abbeel , 2018 ) , and autonomous driving ( Shalev-Shwartz et al. , 2016 ) . Many such environments endow agents with a team reward that reflects the team ’ s coordination objective , as well as an agent-specific local reward that rewards basic skills . For instance , in soccer , dense local rewards could capture agent-specific skills such as passing , dribbling and running . The agents must then coordinate when and where to use these skills in order to optimize the team objective , which is winning the game . Usually , the agent-specific reward is dense and easy to learn from , while the team reward is sparse and requires the cooperation of all or most agents . Having each agent directly optimize the team reward and ignore the agent-specific reward usually fails or is sample-inefficient for complex tasks due to the sparsity of the team reward . Conversely , having each agent directly optimize the agent-specific reward also fails because it does not capture the team ’ s objective , even with state of the art multiagent RL algorithms such as MADDPG ( Lowe et al. , 2017 ) . One solution to this problem is to use reward shaping , where extensive domain knowledge about the task is used to create a proxy reward function ( Rahmattalabi et al. , 2016 ) . Constructing this proxy reward function is difficult in complex environments , and is domain-dependent . Apart from requiring domain knowledge and manual tuning , this approach also poses risks of changing the underlying problem itself ( Ng et al. , 1999 ) . Simple approaches to creating a proxy reward via linear combinations of the two objectives also fail to solve or generalize to complex coordination tasks ( Devlin et al. , 2011 ; Williamson et al. , 2009 ) . In this paper , we introduce Multiagent Evolutionary Reinforcement Learning ( MERL ) , a state-ofthe-art algorithm for cooperative MARL that does not require reward shaping . MERL is a split-level training platform that combines gradient-based and gradient-free optimization . The gradient-free optimizer is an evolutionary algorithm that maximizes the team objective through neuroevolution . The gradient-based optimizer is a policy gradient algorithm that maximizes each agent ’ s dense , local rewards . These gradient-based policies are periodically copied into the evolutionary population . The two processes operate concurrently and share information through a shared replay buffer . A key strength of MERL is that it is a general method which does not require domain-specific reward shaping . This is because MERL optimizes the team objective directly while simultaneously leveraging agent-specific rewards to learn basic skills . We test MERL in a number of multiagent coordination benchmarks . Results demonstrate that MERL significantly outperforms state-of-the-art methods such as MADDPG , while using the same observations and reward functions . We also demonstrate that MERL scales gracefully to increasing complexity of coordination objectives where MADDPG and its variants fail to learn entirely . 2 BACKGROUND AND RELATED WORK . Markov Games : A standard reinforcement learning ( RL ) setting is often formalized as a Markov Decision Process ( MDP ) and consists of an agent interacting with an environment over a finite number of discrete time steps . This formulation can be extended to multiagent systems in the form of partially observable Markov games ( Littman , 1994 ; Lowe et al. , 2017 ) . An N -agent Markov game is defined by a global state of the world , S , and a set of N observations { Oi } and N actions { Ai } corresponding to theN agents . At each time step t , each agent observes its corresponding observation Oti and maps it to an action A t i using its policy πi . Each agent receives a scalar reward rti based on the global state St and joint action of the team . The world then transitions to the next state St+1 which produces a new set of observations { Oi } . The process continues until a terminal state is reached . Ri = ∑T t=0 γ trti is the total return for agent i with discount factor γ ∈ ( 0 , 1 ] . Each agent aims to maximize its expected return . TD3 : Policy gradient ( PG ) methods frame the goal of maximizing the expected return as the minimization of a loss function . A widely used PG method for continuous , high-dimensional action spaces is DDPG ( Lillicrap et al. , 2015 ) . Recently , ( Fujimoto et al. , 2018 ) extended DDPG to Twin Delayed DDPG ( TD3 ) , addressing its well-known overestimation problem . TD3 is the state-of-the-art , off-policy algorithm for model-free DRL in continuous action spaces . TD3 uses an actor-critic architecture ( Sutton and Barto , 1998 ) maintaining a deterministic policy ( actor ) π : S → A , and two distinct criticsQ : S ×A → Ri . Each critic independently approximates the actor ’ s action-value function Qπ . A separate copy of the actor and critics are kept as target networks for stability and are updated periodically . A noisy version of the actor is used to explore the environment during training . The actor is trained using a noisy version of the sampled policy gradient computed by backpropagation through the combined actor-critic networks . This mitigates overfitting of the deterministic policy by smoothing the policy gradient updates . Evolutionary Reinforcement Learning ( ERL ) is a hybrid algorithm that combines Evolutionary Algorithms ( EAs ) ( Floreano et al. , 2008 ; Lüders et al. , 2017 ; Fogel , 2006 ; Spears et al. , 1993 ) , with policy gradient methods ( Khadka and Tumer , 2018 ) . Instead of discarding the data generated during a standard EA rollout , ERL stores this data in a central replay buffer shared with the policy gradient ’ s own rollouts - thereby increasing the diversity of the data available for the policy gradient learners . Since the EA directly optimizes for episode-wide return , it biases exploration towards states with higher long-term returns . The policy gradient algorithm which learns using this state distribution inherits this implicit bias towards long-term optimization . Concurrently , the actor trained by the policy gradient algorithm is inserted into the evolutionary population allowing the EA to benefit from the fast gradient-based learning . Related Work : Lowe et al . ( 2017 ) introduced MADDPG which tackled the inherent non-stationarity of a multiagent learning environment by leveraging a critic which had full access to the joint state and action during training . Foerster et al . ( 2018b ) utilized a similar setup with a centralized critic across agents to tackle StarCraft micromanagement tasks . An algorithm that could explicitly model other agents ’ learning was investigated in Foerster et al . ( 2018a ) . However , all these approaches rely on a dense agent reward that properly captures the team objective . Methods to solve for these types of agent-specific reward functions were investigated in Li et al . ( 2012 ) but were limited to tasks with strong simulators where tree-based planning could be used . A closely related work to MERL is ( Liu et al. , 2019 ) where Population-Based Training ( PBT ) ( Jaderberg et al. , 2017 ) is used to optimize the relative importance between a collection of dense , shaped rewards automatically during training . This can be interpreted as a singular central reward function constructed by scalarizing a collection of reward signals where the scalarization coefficients are adaptively learned during training . In contrast , MERL optimizes its reward functions independently with information transfer across them facilitated through shared replay buffers and policy migration directly . This form of information transfer through a shared replay buffer has been explored extensively in recent literature ( Colas et al. , 2018 ; Khadka et al. , 2019 ) . 3 MULTIAGENT EVOLUTIONARY REINFORCEMENT LEARNING . MERL leverages both agent-specific and team objectives through a hybrid algorithm that combines gradient-free and gradient-based optimization . The gradient-free optimizer is an evolutionary algorithm that maximizes the team objective through neuroevolution . The gradient-based optimizer trains policies to maximize agent-specific rewards . These gradient-based policies are periodically added to the evolutionary population and participate in evolution . This enables the evolutionary algorithm to use agent-specific skills learned by training on the agent-specific rewards toward optimizing the team objective without needing to resort to reward shaping . Algorithm 1 Multiagent Evolutionary Reinforcement Learning 1 : Initialize a population of k multi-head teams popπ , each with weights θπ initialized randomly 2 : Initialize a shared critic Q with weights θQ 3 : Initialize an ensemble of N empty cyclic replay buffersRk , one for each agent 4 : Define a white Gaussian noise generatorWg random number generator r ( ) ∈ [ 0 , 1 ) 5 : for generation = 1 , ∞ do 6 : for team π ∈ popπ do 7 : g , R = Rollout ( π , R , noise=None , ξ ) 8 : _ , R = Rollout ( π , R , noise=Wg , ξ = 1 ) 9 : Assign g as π ’ s fitness 10 : end for 11 : Rank the population popπ based on fitness scores 12 : Select the first e teams π ∈ popπ as elites 13 : Select the remaining ( k − e ) teams π from popπ , to form Set S using tournament selection 14 : while |S| < ( k − e ) do 15 : Single-point crossover between a randomly sampled π ∈ e and π ∈ S and append to S 16 : end while 17 : for Agent k=1 , N do 18 : Randomly sample a minibatch of T transitions ( oi , ai , li , oi+1 ) from Rk 19 : Compute yi = li + γ min j=1,2 Q′j ( oi+1 , a∼|θQ ′ j ) 20 : where a∼ = π′pg ( k , oi+1|θπ ′ pg ) [ action sampled from the kth head of π′pg ] + 21 : Update Q by minimizing the loss : L = 1T ∑ i ( yi −Q ( oi , ai|θQ ) 2 22 : Update πkpg using the sampled policy gradient ∇θπpgJ ∼ 1 T ∑ ∇aQ ( o , a|θQ ) |o=oi , a=ai∇θπpgπ k pg ( s|θπpg ) |o=oi 23 : Soft update target networks : θπ ′ ⇐ τθπ + ( 1− τ ) θπ′ and θQ′ ⇐ τθQ + ( 1− τ ) θQ′ 24 : end for 25 : Migrate the policy gradient team popj : for weakest π ∈ popjπ : θπ ⇐ θπpg 26 : end for Policy Topology : We represent our multiagent ( team ) policies using a multi-headed neural network π as illustrated in Figure 1 . The head πk represents the k-th agent in the team . Given an incoming observation for agent k , only the output of πk is considered as agent k ’ s response . In essence , all agents act independently based on their own observations while sharing weights ( and by extension , the features ) in the lower layers ( trunk ) . This is commonly used to improve learning speed ( Silver et al. , 2017 ) . Further , each agent k also has its own replay buffer ( Rk ) which stores its experience defined by the tuple ( state , action , next state , local reward ) for each interaction with the environment ( rollout ) involving that agent . Team Reward Optimization : Figure 2 illustrates the MERL algorithm . A population of multi-headed teams , each with the same topology , is initialized with random weights . The replay bufferRk is shared by the k-th agent across all teams . The population is then evaluated for each rollout . The team reward for each team is disbursed at the end of the episode and is considered as its fitness score . A selection operator selects a portion of the population for survival with probability proportionate to their fitness scores . The weights of the teams in the population are probabilistically perturbed through mutation and crossover operators to create the next generation of teams . A portion of the teams with the highest relative fitness are preserved as elites . At any given time , the team with the highest fitness , or the champion , represents the best solution for the task . Policy Gradient : The procedure described so far resembles a standard EA except that each agent k stores each of its experiences in its associated replay buffer ( Rk ) instead of just discarding it . However , unlike EA , which only learns based on the low-fidelity global reward , MERL also learns from the experiences within episodes of a rollout using policy gradients . To enable this kind of `` local learning '' , MERL initializes one multi-headed policy network πpg and one critic Q . A noisy version of πpg is then used to conduct its own set of rollouts in the environment , storing each agent k ’ s experiences in its corresponding buffer ( Rk ) similar to the evolutionary rollouts . Agent-Specific Reward Optimization : Crucially , each agent ’ s replay buffer is kept separate from that of every other agent to ensure diversity amongst the agents . The shared critic samples a random mini-batch uniformly from each replay buffer and uses it to update its parameters using gradient descent . Each agent πkpg then draws a mini-batch of experiences from its corresponding buffer ( Rk ) and uses it to sample a policy gradient from the shared critic . Unlike the teams in the evolutionary population which directly seek to optimize the team reward , πpg seeks to maximize the agent-specific local reward while exploiting the experiences collected via evolution . Skill Migration : Periodically , the πpg network is copied into the evolving population of teams and can propagate its features by participating in evolution . This is the core mechanism that combines policies learned via agent-specific and team rewards . Regardless of whether the two rewards are aligned , evolution ensures that only the performant derivatives of the migrated network are retained . This mechanism guarantees protection against destructive interference commonly seen when a direct scalarization between two reward functions is attempted . Further , the level of information exchange is automatically adjusted during the process of learning , in contrast to being manually tuned by an expert designer . Algorithm 1 provides a detailed pseudo-code of the MERL algorithm . The choice of hyperparameters is explained in the Appendix . Additionally , our source code 1 is available online . 1https : //tinyurl.com/y6erclts

Summary: This paper proposes to use a two-level optimization process to solve the challenge of optimizing the team reward and the agent's reward simultaneously, which are often not aligned. It applies the evolutionary algorithm to optimize the sparse team reward, while using RL (TD3) to optimize the agent's dense reward. In this way, there is no need to combine these two rewards into a scalar that often requires extensive manual tuning.