Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning

1 INTRODUCTION . Multi-Agent Reinforcement Learning ( MARL ) refers to the task of training an agent to maximize its expected return by interacting with an environment that contains other learning agents . It represents a challenging branch of Reinforcement Learning ( RL ) with interesting developments in recent years ( Hernandez-Leal et al. , 2018 ) . A popular framework for MARL is the use of a Centralized Training and a Decentralized Execution ( CTDE ) procedure ( Lowe et al. , 2017 ; Foerster et al. , 2018 ; Iqbal & Sha , 2019 ; Foerster et al. , 2019 ; Rashid et al. , 2018 ) . It is typically implemented by training critics that approximate the value of the joint observations and actions , which are used to train actors restricted to the observation of a single agent . Such critics , if exposed to coordinated joint actions leading to high returns , can steer the agents ’ policies toward these highly rewarding behaviors . However , these approaches depend on the agents luckily stumbling on these actions in order to grasp their benefit . Thus , it might fail in scenarios where coordination is unlikely to occur by chance . We hypothesize that in such scenarios , coordination-promoting inductive biases on the policy search could help discover coordinated behaviors more efficiently and supersede task-specific reward shaping and curriculum learning . In this work , we explore two different priors for successful coordination and use these to regularize the learned policies . The first avenue , TeamReg , assumes that an agent must be able to predict the behavior of its teammates in order to coordinate with them . The second , CoachReg , supposes that coordinating agents individually recognize different situations and synchronously use different subpolicies to react to them . In the following sections we show how to derive practical regularization terms from these premises and meticulously evaluate them1 . Our contributions are twofold . First , we propose two novel approaches that aim at promoting coordination in multi-agent systems . Our methods augment CTDE MARL algorithms with additional multi-agent objectives that act as regularizers and are optimized jointly with the main return-maximization objective . Second , we design two new sparse-reward cooperative tasks in the 1Source code for the algorithms and environments will be made public upon publication of this work . multi-agent particle environment ( Mordatch & Abbeel , 2018 ) . We use them along with two standard multi-agent tasks to present a detailed evaluation of our approaches against three different baselines . Finally , we validate our methods ’ key components by performing an ablation study . Our experiments suggest that our TeamReg objective provides a dense learning signal that helps to guide the policy towards coordination in the absence of external reward , eventually leading it to the discovery of high performing team strategies in a number of cooperative tasks . Similarly , by enforcing synchronous sub-policy selections , CoachReg enables to fine-tune a sub-behavior for each recognized situation yielding significant improvements on the overall performance . 2 BACKGROUND . 2.1 MARKOV GAMES . In this work we consider the framework of Markov Games ( Littman , 1994 ) , a multi-agent extension of Markov Decision Processes ( MDPs ) with N independent agents . A Markov Game is defined by the tuple 〈S , T , P , { Oi , Ai , Ri } Ni=1〉 . S , T , and P respectively are the set of all possible states , the transition function and the initial state distribution . While these are global properties of the environment , Oi , Ai and Ri are individually defined for each agent i . They are respectively the observation functions , the sets of all possible actions and the reward functions . At each time-step t , the global state of the environment is given by st ∈ S and every agent ’ s individual action vector is denoted by ait ∈ Ai . To select their action , each agent i has only access to its own observation vector oit which is extracted by its observation function Oi from the global state st . The initial global state s0 is sampled from the initial state distribution P : S → [ 0 , 1 ] and the next states of the environment st+1 are sampled from the probability distribution over the possible next states given by the transition function T : S×S×A1× ... ×AN → [ 0 , 1 ] . Finally , at each time-step , each agent receives an individual scalar reward rit from its reward functionRi : S × S ×A1 × ... ×AN → R. Agents aim at maximizing their expected discounted return E [ ∑T t=0 γ trit ] over the time horizon T , where γ ∈ [ 0 , 1 ] is a discount factor . 2.2 MULTI-AGENT DEEP DETERMINISTIC POLICY GRADIENT ( MADDPG ) . MADDPG ( Lowe et al. , 2017 ) is an adaptation of the Deep Deterministic Policy Gradient algorithm ( DDPG ) ( Lillicrap et al. , 2015 ) to the multi-agent setting . It allows the training of cooperating and competing decentralized policies through the use of a centralized training procedure . In this framework , each agent i possesses its own deterministic policy µi for action selection and critic Qi for state-action value estimation , which are respectively parametrized by θi and φi . All parametric models are trained off-policy from previous transitions ζt : = ( ot , at , rt , ot+1 ) uniformly sampled from a replay buffer D. Note that ot : = [ o1t , ... , oNt ] is the joint observation vector and at : = [ a 1 t , ... , a N t ] is the joint action vector , obtained by concatenating the individual observation vectors oit and action vectors a i t of all N agents . Each centralized critic is trained to estimate the expected return for a particular agent i using the Deep Q-Network ( DQN ) ( Mnih et al. , 2015 ) loss : Li ( φi ) = Eζt∼D [ 1 2 ( Qi ( ot , at ; φ i ) − ( rit + γQi ( ot+1 , at+1 ; φ̄i ) ) ) 2∣∣∣∣ ajt+1=µ j ( ojt+1 ; θ̄ j ) ∀j ] ( 1 ) For a given set of weights w , we define its target counterpart w̄ , updated from w̄ ← τw + ( 1− τ ) w̄ where τ is a hyper-parameter . Each policy is updated to maximize the expected discounted return of the corresponding agent i : J iPG ( θ i ) = Eot∼D [ Qi ( ot , at ) ∣∣ ait=µ i ( oit ; θ i ) , ajt=µ j ( ojt ; θ̄ j ) ∀j 6=i ] ∇θiJ iPG ( θi ) = Eot∼D [ ∇θiµi ( oit ; θi ) ∇ait Q i ( ot , at ) ∣∣ ait=µ i ( oit ; θ i ) , ajt=µ j ( ojt ; θ̄ j ) ∀j 6=i ] ( 2 ) By taking into account all agents ’ observation-action pairs when guiding an agent ’ s policy , the valuefunctions are trained in a centralized , stationary environment , despite taking place in a multi-agent setting . In addition , this mechanism can allow to implicitly learn coordinated strategies that can then be deployed in a decentralized way . However , this procedure does not encourage the discovery of coordinated strategies since high-reward behaviors have to be randomly experienced through unguided exploration . This work aims at alleviating this limitation . 3 RELATED WORK . Many works in MARL consider explicit communication channels between the agents and distinguish between communicative actions ( e.g . broadcasting a given message ) and physical actions ( e.g . moving in a given direction ) ( Foerster et al. , 2016 ; Mordatch & Abbeel , 2018 ; Lazaridou et al. , 2016 ) . Consequently , they often focus on the emergence of language , considering tasks where the agents must discover a common communication protocol in order to succeed . Deriving a successful communication protocol can already be seen as coordination in the communicative action space and can enable , to some extent , successful coordination in the physical action space ( Ahilan & Dayan , 2019 ) . Yet , explicit communication is not a necessary condition for coordination as agents can rely on physical communication ( Mordatch & Abbeel , 2018 ; Gupta et al. , 2017 ) . Approaches to shape RL agents ’ behaviors with respect to other agents have also been explored . Strouse et al . ( 2018 ) use the mutual information between the agent ’ s policy and a goal-independent policy to shape the agent ’ s behavior towards hiding or spelling out its current goal . However , this approach is only applicable for tasks with an explicit goal representation and is not specifically intended for coordination . Jaques et al . ( 2019 ) approximate the direct causal effect between agent ’ s actions and use it as an intrinsic reward to encourage social empowerment . This approximation relies on each agent learning a model of other agents ’ policies to predict its effect on them . In general , this type of behavior prediction can be referred to as agent modelling ( or opponent modelling ) and has been used in previous work to enrich representations ( Hernandez-Leal et al. , 2019 ) , to stabilise the learning dynamics ( He et al. , 2016 ) or to classify the opponent ’ s play style ( Schadd et al. , 2007 ) . In our work , agent modelling is extended to derive a novel incentive toward team-predictable behaviors . Finally , Barton et al . ( 2018 ) propose convergent cross mapping ( CCM ) to measure the degree of effective coordination between two agents . Although this may represent an interesting avenue for behavior analysis , it fails to provide a tool for effectively enforcing coordination as CCM must be computed over long time series which makes it an impractical learning signal for singlestep temporal difference methods . In this work , we design two coordination-driven multi-agent approaches that do not rely on the existence of explicit communication channels and allow to carry the learned coordinated behaviors at test time , when all agents act in a decentralized fashion . 4 COORDINATION AND POLICY . REGULARIZATION Intuitively , coordination can be defined as an agent ’ s behavior being informed by the one of another agent , i.e . structure in the agents ’ interactions . Namely , a team where agents act independently of one another would not be coordinated . To promote such structure , our proposed methods rely on team-objectives as regularizers of the common policy gradient update . In this regard , our approach is closely related to General Value Functions and Auxiliary tasks ( Sutton & Barto , 2018 ) used in Deep RL to learn efficient representations ( Jaderberg et al. , 2019 ) . However , this work ’ s novelty lies in the explicit bias of agents ’ policy towards either predictability for their teammates or synchronous sub-policy selection . Pseudocodes of our implementations are provided in Appendix C ( see Algorithms 1 and 2 ) . 4.1 TEAM REGULARIZATION . The structure of coordinated interactions can be leveraged to attain a certain degree of predictability of one agent ’ s behavior with respect to its teammate ( s ) . We hypothesize that the reciprocal also holds i.e . that promoting agents ’ predictability could foster such team structure and lead to more coordinated behaviors . This assumption is cast into the decentralized framework by training agents to predict their teammates ’ actions given only their own observation . For continuous control , the loss is defined as the Mean Squared Error ( MSE ) between the predicted and true actions of the teammates , yielding a teammate-modelling secondary objective . While the previous work of Hernandez-Leal et al . ( 2019 ) focus on stationary , non-learning teammates and exclusively use this approach to learn richer internal representations , we propose to extend this objective to drive the teammates ’ behaviors closer to the prediction by leveraging a differentiable action selection mechanism . We call team-spirit this novel objective J i , jTS between agents i and j : J i , jTS ( θ i , θj ) = Eot∼D [ −MSE ( âi , jt , a j t ) ∣∣∣ajt=µj ( ojt ) , âi , jt =µ̂i , j ( oit ) ] ( 3 ) = −Eot∼D 1 2 |Aj |∑ k=1 ( âi , jt , k − a j t , k ) 2 ∣∣∣ajt=µj ( ojt ) , âi , jt =µ̂i , j ( oit ) ] ( 4 ) where µ̂i , j is the policy head of agent i trying to predict the action of agent j . The total gradient for a given agent i becomes : ∇θiJ itotal ( θi ) = ∇θiJ iPG ( θi ) + λ1 ∑ j ∇θiJ i , jTS ( θ i , θj ) + λ2 ∑ j ∇θiJj , iTS ( θ j , θi ) ( 5 ) where λ1 and λ2 are hyper-parameters that respectively weight how well an agent should predict its teammates ’ actions , and how predictable an agent should be for its teammates . We call TeamReg this dual regularization from team-spirit objectives . Figure 1 summarizes these interactions . 4.2 COACH REGULARIZATION In order to foster structured agents interactions , this method aims at teaching the agents to recognize different situations and synchronously select corresponding sub-behaviors .

Summary: This paper proposed two approaches to encourage cooperation among multi-agents under the *centralized training decentralized execution* framework. The main contribution of this paper is that they propose to allow agents to predict the behavior of others and introduce this prediction loss into the RL learning objective. One approach is teammate-regularization where each agent predicts the behavior of all others (and therefore makes the total model complexity increasing quadratically with the number of agents), while the other is a centralized coach-regularization. The performance of these two approaches are compared with MADDPG and its two variants on 4 games. Experiments show that their approaches surpass a few baselines in certain game settings. However, the novelty of this algorithm is not strong, given that the similar idea of 'predicting the behavior of other agents' can be found in related work with discrete action spaces. Experimental results also show unstable advantages of their approaches.