Intention Propagation for  Multi-agent Reinforcement Learning

1 INTRODUCTION . Collaborative multi-agent reinforcement learning is an important sub-field of the multi-agent reinforcement learning ( MARL ) , where the agents learn to coordinate to achieve joint success . It has wide applications in traffic control ( Kuyer et al. , 2008 ) , autonomous driving ( Shalev-Shwartz et al. , 2016 ) and smart grid ( Yang et al. , 2018 ) . To learn a coordination , the interactions between agents are indispensable . For instance , humans can reason about other ’ s behaviors or know other peoples ’ intentions through communication and then determine an effective coordination plan . However , how to design a mechanism of such interaction in a principled way and at the same time solve the large scale real-world applications is still a challenging problem . Recently , there is a surge of interest in solving the collaborative MARL problem ( Foerster et al. , 2018 ; Qu et al. , 2019 ; Lowe et al. , 2017 ) . Among them , joint policy approaches have demonstrated their superiority ( Rashid et al. , 2018 ; Sunehag et al. , 2018 ; Oliehoek et al. , 2016 ) . A straightforward approach is to replace the action in the single-agent reinforcement learning by the joint action a = ( a1 , a2 , ... , aN ) , while it obviously suffers from the issue of the exponentially large action space . Thus several approaches have been proposed to factorize the joint action space to mitigate such issue , which can be roughly grouped into two categories : • Factorization on policy . This approach explicitly assumes that π ( a|s ) : = ∏N i=1 πi ( ai|s ) , i.e. , policies are independent ( Foerster et al. , 2018 ; Zhang et al. , 2018 ) . To mitigate for the instability issue caused by the independent learner , it generally needs a centralized critic . • Factorization on value function . This approach has a similar spirit but factorizes the joint value function into several utility functions , each just involving the actions of one agent ( Rashid et al. , 2018 ; Sunehag et al. , 2018 ) . However , these two approaches lack of the interactions between agents , since in their algorithms agent i does not care about the plan of agent j . Indeed , they may suffer from a phenomenon called relative over-generalization in game theory observed by Wei & Luke ( 2016 ) ; Castellini et al . ( 2019 ) ; Palmer et al . ( 2018 ) . Approaches based on the coordinate graph would effectively prevent such cases , where the value function is factorized as a summation of utility function on pairwise or local joint action ( Guestrin et al. , 2002 ; Böhmer et al. , 2020 ) . However , they only can be applied in discrete action , small scale game . Furthermore , despite the empirical success of the aforementioned work in certain scenarios , it still lacks theoretical insight . In this work , we only make a simple yet realistic assumption : the reward function ri of each agent i just depends on its individual action and the actions of its neighbors ( and state ) , i.e. , ri ( s , a ) = ri ( s , ai , aNi ) , ( 1 ) where we use Ni to denote the neighbors of agent i , s to denote the global state . It says the goal or decision of agent is explicitly influenced by a small subset Ni of other agents . Note that such an assumption is reasonable in lots of real scenarios . For instance , • The traffic light at an intersection makes the decision on the phase changing mainly relying on the traffic flow around it and the policies of its neighboring traffic light . • The main goal of a defender in a soccer game is to tackle the opponent ’ s attacker , while he rarely needs to pay attention to opponent goalkeeper ’ s strategy . Based on the assumption in equation 1 , we propose a principled multi-agent reinforcement learning algorithm in the framework of probabilistic inference , where the objective is to maximize the long term reward of the group , i.e. , ∑∞ t=0 ∑N i=1 γ trti ( see details in section 4 ) . Note since each agent ’ s reward depends on its neighbor , we still need a joint policy to maximize the global reward through interactions . In this paper , we derive an iterative procedure for such interaction to learn the joint policy in collaborative MARL and name it intention propagation . Particularly , • In the first round , each agent i makes an independent decision and spreads out his plan µ̃i ( we name it intention ) to neighbors . • In the second round , agents i changes its initial intention properly based on its neighbors ’ intention µ̃j , j ∈ Ni and propagates its intention µ̃i again . • In the third round , it changes the decision in the second round with a similar argument . • As this procedure goes on , we show the final output of agents ’ policy converges to the mean field approximation ( the variational inference method from the probabilistic graphical model ( Bishop , 2006 ) ) of the joint policy . In addition , this joint policy has the form of Markov Random Field induced by the locality of the reward function ( proposition 1 ) . Therefore , such a procedure is computationally efficient when the underlying graph is sparse , since in each round , each agent just needs to care about what its neighbors intend to do . Remark : ( 1 ) Our work is not related to the mean-field game ( MFG ) ( Yang et al. , 2018 ) . The goal of the MFG is to find the Nash equilibrium , while our work aims to the optimal joint policy in the collaborative game . Furthermore , MFG generally assumes agents are identical and interchangeable . When the number of agents goes to infinity , MFG can view the state of other agents as a population state distribution . In our problem , we do not have such assumptions . ( 2 ) our analysis is not limited to the mean-field approximation . When we change the message passing structure of intention propagation , we can show that it converges to other approximation of the joint policy , e.g. , loopy belief propagation in variational inference ( Yedidia et al. , 2001 ) ( see Appendix B.2 ) . Contributions : ( 1 ) We propose a principled method named intention propagation to solve the joint policy collaborative MARL problem ; ( 2 ) Our method is computationally efficient , which can scale up to one thousand agents and thus meets the requirement of real applications ; ( 3 ) Empirically , it outperforms state-of-the-art baselines with a wide margin when the number of agents is large ; ( 4 ) Our work builds a bridge between MARL and neural embedded probabilistic inference , which would lead to new algorithms beyond intention propagation . Notation : sti and ati represent the state and action of agent i at time step t. The neighbors of agent i are represented asNi . We denote X as a random variable with domain X and refer to instantiations of X by the lower case character x . We denote a density on X by p ( x ) and denote the space of all such densities as by P . 2 RELATED WORK . We first discuss the work of the factorized approaches on the joint policy . COMA designs a MARL algorithm based on the actor-critic framework with independent actors πi ( ai|s ) , where the joint policy is factorized as π ( a|s ) = ∏N i=1 πi ( ai|s ) ( Foerster et al. , 2018 ) . MADDPG considers a MARL with the cooperative or competitive setting , where it creates a critic for each agent ( Lowe et al. , 2017 ) . Other similar works may include ( de Witt et al. , 2019 ; Wei et al. , 2018 ) . Another way is to factorize the value functions into several utility functions . Sunehag et al . ( 2018 ) assumes that the overall Q function can be factorized as Q ( s , a1 , a2 , .. , aN ) = ∑N i=1Qi ( si , ai ) . QMIX extends this work to include a richer class of function , where it assumes the overall Q function is a monotonic function w.r.t . each Qi ( si , ai ) ( Rashid et al. , 2018 ) . Similarly , Son et al . ( 2019 ) further relax the structure constraint on the joint value function . However these factorized methods suffer from the relative overgeneralization issue ( Castellini et al. , 2019 ; Palmer et al. , 2018 ) . Generally speaking , it pushes the agents to underestimate a certain action because of the low rewards they receive , while they could get a higher one by perfectly coordinating . A middle ground between the ( fully ) joint policy and the factorized policy is the coordination graph ( Guestrin et al. , 2002 ) , where the value function is factorized as a summation of the utility function on the pairwise action . Böhmer et al . ( 2020 ) ; Castellini et al . ( 2019 ) combine deep learning techniques with the coordination graph . It addresses the issue of relative overgeneralization , but still has two limitations especially in the large scale MARL problem . ( 1 ) The max-sum algorithm can just be implemented in the discrete action space since it needs a max-sum operation on the action of Q function . ( 2 ) Even in the discrete action case , each step of the Q learning has to do several loops of max-sum operation over the whole graph if there is a cycle in graph . Our algorithm can handle both discrete and continuous action space cases and alleviate the scalability issue by designing an intention propagation network . Another category of MARL is to consider the communication among agents . The attention mechanism is used to decide when and who to communicate with ( Das et al. , 2018 ) . Foerster et al . ( 2016 ) propose an end-to-end method to learn communication protocol . In ( Liu et al. , 2019 ; Chu et al. , 2020 ) , each agent sends the action information to it neighbors . In addition , Chu et al . ( 2020 ) require a strong assumption that the MDP has the spatial-temporal Markov property . However , they utilizes neighbor ’ s action information in a heuristic way and thus it is unclear what the agents are learning ( e.g. , do they learn the optimal joint policy to maximize the group reward ? ) . Jiang et al . ( 2020 ) propose DGN which uses GNN to spread the state embedding information to neighbors . However each agent still uses an independent Q learning to learn the policy and neglects other agents ’ plans . In contrast , we propose a principled algorithm , where each agent makes decision considering other agents ’ plan . Such procedure can be parameterized by GNN and other neural networks ( see section 4.1 and appendix B.2 ) . We prove its convergence to the solution of variational inference methods . 3 BACKGROUNDS . Probabilistic Reinforcement Learning : Probabilistic reinforcement learning ( PRL ) ( Levine , 2018 ) is our building block . PRL defines the trajectory τ up to time step T as τ = [ s0 , a0 , s1 , a1 , ... , sT , aT , sT+1 ] . The probability distribution of the trajectory τ induced by the optimal policy is defined as p ( τ ) = [ p ( s0 ) ∏T t=0 p ( s t+1|st , at ) ] exp ( ∑T t=0 r ( s t , at ) ) . While the probability of the trajectory τ under the policy π ( a|s ) is defined as p̂ ( τ ) = p ( s0 ) ∏T t=0 p ( s t+1|st , at ) π ( at|st ) . The objective is to minimize the KL divergence between p̂ ( τ ) and p ( τ ) . It is equivalent to the maximum entropy reinforcement learning max π J ( π ) = T∑ t=0 E [ r ( st , at ) +H ( π ( at|st ) ) ] , where it omits the discount factor γ and regularizer factor α of the entropy term , since it is easy to incorporate them into the transition and reward respectively . Notice in this framework the max operator in the Bellman optimality equation would be replaced by the softmax operator and thus its optimal policy is a softmax function related to the Q function ( Haarnoja et al. , 2017 ) . Such framework subsumes state-of-the-art algorithms such as soft-actor-critic ( SAC ) ( Haarnoja et al. , 2018 ) . In each iteration , SAC optimizes the following loss function of Q , π , V , and respectively . E ( st , at ) ∼D [ Q ( st , at ) − r ( st , at ) − γEst+1∼p [ V ( st+1 ) ] ] 2 , Est∼DEat∼π [ log π ( at|st ) −Q ( st , at ) ] Est∼D [ V ( st ) − Eat∼πθ [ Q ( st , at ) − log π ( at|st ) ] ] 2 , where D is the replay buffer . Function Space Embedding of Distribution : In our work , we use the tool of embedding in Reproducing Kernel Hilbert Space ( RKHS ) to design an intention propagation procedure ( Smola et al. , 2007 ) . We let φ ( X ) be an implicit feature mapping and X be a random variable with distribution p ( x ) . Embeddings of p ( x ) is given by µX : = EX [ φ ( X ) ] = ∫ φ ( x ) p ( x ) dx where the distribution is mapped to its expected feature map . By assuming that there exists a feature space such that the embeddings are injective , we can treat the embedding µX of the density p ( x ) as a sufficient statistic of the density , i.e. , any information we need from the density is preserved in µX ( Smola et al. , 2007 ) . Such injective assumption generally holds under mild condition ( Sriperumbudur et al. , 2008 ) . This property is important since we can reformulate a functional f : P → R of p ( · ) using the embedding only , i.e. , f ( p ( x ) ) = f̃ ( µX ) . It also can be generalized to the operator case . In particular , applying an operator T : P → Rd to a density can be equivalently carried out using its embedding T ◦ p ( x ) = T̃ ◦ µX , where T̃ : F → Rd is the alternative operator working on the embedding . In practice , µX , f̃ and T̃ have complicated dependence on φ . As such , we approximate them by neural networks , which is known as the neural embedding approach of distribution ( Dai et al. , 2016 ) .

Summary: The paper considers the cooperative multiagent MARL setting where each agent’s reward depends on the state and the actions of itself and its neighbors The paper has a theoretical claim that, for such reward structure, the optimal maximum entropy joint policy in the form that can be factored into potential functions, one for each agent. In particular, if the sum of all agents’ rewards is a function on pairwise actions, those potential functions are one for each agent and one for each pair of actions (i.e. the equation after Proposition 1).