Communication-Efficient Actor-Critic Methods for Homogeneous Markov Games

Recent success in cooperative multi-agent reinforcement learning ( MARL ) relies on centralized training and policy sharing . Centralized training eliminates the issue of non-stationarity MARL yet induces large communication costs , and policy sharing is empirically crucial to efficient learning in certain tasks yet lacks theoretical justification . In this paper , we formally characterize a subclass of cooperative Markov games where agents exhibit a certain form of homogeneity such that policy sharing provably incurs no suboptimality . This enables us to develop the first consensus-based decentralized actor-critic method where the consensus update is applied to both the actors and the critics while ensuring convergence . We also develop practical algorithms based on our decentralized actor-critic method to reduce the communication cost during training , while still yielding policies comparable with centralized training . 1 INTRODUCTION . Cooperative multi-agent reinforcement learning ( MARL ) is the problem where multiple agents learn to make sequential decisions in a common environment to optimize a shared reward signal , which finds a wide range of real-world applications such as traffic control ( Chu et al. , 2019 ) , power grid management ( Callaway & Hiskens , 2010 ) , and coordination of multi-robot systems ( Corke et al. , 2005 ) . Efficient learning for large and complex cooperative MARL tasks is challenging . Naively reducing cooperative MARL to single-agent RL with a joint observation-action space imposes significant scalability issues , since the joint space grows exponentially with the number of agents . Approaches that treat each agent as an independent RL learner , such as Independent Q-Learning ( Tan , 1993 ) , overcome the scalability issue yet fail to succeed in complicated tasks due to the nonstationarity caused by other learning agents ’ evolving policies . To address these challenges , the paradigm of Centralized Training and Decentralized Execution ( CTDE ) is then proposed , where a centralized trainer is assumed to access to information of all agents during training to approximate the global ( action- ) value function , whereas each agent only needs local information for its action selection during decentralized policy execution ( Lowe et al. , 2017 ; Foerster et al. , 2017 ) . The centralized critic eliminates non-stationarity during training , while the policy decentralization ensures scalability during execution . Besides , existing CTDE methods almost always enable policy parameter sharing to further improve learning scalability and efficiency , where agents also share the parameters of their decentralized policies . However , in many real-world scenarios , there is not a readily available centralizer that conveniently gathers the global information from all agents , and therefore agents need to rely on all-to-all communication for centralized training , incurring enormous communication overheads for large numbers of agents . This motivates us to think about whether it is possible to train agents in a decentralized and communication-efficient manner , while still keeping the benefits of the centralized training of CTDE . Moreover , despite its wide adoption , little theoretical understanding has been provided to justify policy parameter sharing . Agents should at least exhibit a certain level of homogeneity before it is feasible to share their policies . For example , if the observation and/or action spaces vary across agents , then their decentralized policies can not even have the same architecture . Even if it is feasible , it is unclear whether restricting the agents to share their policy parameters will introduce any suboptimality . In this paper , we address these aforementioned issues centered around the CTDE framework . We begin by formally characterizing a subclass of Markov games where the cooperative agents exhibit a certain form of homogeneity such that it is not only feasible but also incurs no suboptimality to share their decentralized policies , thus providing a first theoretical justification for policy parameter sharing . We then develop a decentralized actor-critic algorithm for homogeneous MGs where agents share their critic and actor parameters with consensus-based updates , for which we prove an asymptotic convergence guarantee with linear critics , full observability , and other standard assumptions . To our knowledge , this is the first decentralized actor-critic algorithm that enjoys provable convergence guarantees with policy ( i.e. , actor ) consensus . To account for communication efficiency , we develop a simple yet effective bandit-based process that wisely selects when and with whom to perform the parameter census update based on the feedback of policy improvement during training . To further account for partial observability , we develop an end-to-end learnable gating mechanism for the agents to selectively share their observations and actions for learning the decentralized critics . This series of innovations are capable of transforming any CTDE algorithm into its decentralized and communication-efficient counterpart , with policy consensus in homogeneous MGs for improved efficiency . Our empirical results demonstrate the effectiveness of these innovations when instantiated with a state-of-the-art CTDE algorithm , achieving competitive policy performance with only a fraction of communication during training . Our contribution is therefore summarized as three-fold : ( 1 ) the characterization of a subclass of cooperative Markov games , i.e . homogeneous Markov games ( MGs ) , where policy sharing provably incurs no loss of optimality ; ( 2 ) a decentralized MARL algorithm for homogeneous MGs that enjoys asymptotic convergence guarantee with policy consensus ; and ( 3 ) practical techniques that transform CTDE algorithms to their decentralized and communication-efficient counterparts . 2 RELATED WORK . Communication in cooperative MARL . Communication is key to solving the issue of nonstationarity in cooperative MARL . The CTDE paradigm ( Lowe et al. , 2017 ; Foerster et al. , 2017 ) assumes a centralized unit during training to learn a joint value function . Other methods , such as CommNet ( Sukhbaatar et al. , 2016 ) and BiCNet ( Peng et al. , 2017 ) , do not assume a centralized unit and instead allow agents to share information by all-to-all broadcasting , effectively relying on centralized communication . These methods require centralized/all-to-all communication that impedes their application to large numbers of agents . Although follow-up work such as IC3Net ( Singh et al. , 2018 ) and VBC ( Zhang et al. , 2019 ) proposes algorithms to learn when to communicate , agents there still perform all-to-all communication before others decide whether to receive . We instead entirely abandon centralized/all-to-all communication , letting each agent decide whom to communicate to purely based on its local observation . There is another line of work , Networked MARL ( NMARL ) ( Zhang et al. , 2018 ) , that where agents lie on a predefined network such that neighboring agents can freely communicate . Our approach instead learns sparse communication that is dynamically adjusted during decentralized training , even if the predefined network topology can be dense . Policy parameter sharing and consensus . Policy parameter sharing is widely adopted in MARL where agents share the same action space , yet it has not been theoretically justified except under the mean-field approximation where the transition dynamics depends on the collective statistics of all agents and not on the identities and ordering of individual agents ( Nguyen et al. , 2017a ; b ; Yang et al. , 2018 ) . A recent result from Kuba et al . ( 2021 ) states that enforcing policy parameter sharing in a general cooperative MG can lead to a suboptimal outcome that is exponentially-worse with the number of agents . We are the first to 1 ) formally characterize the subclass of homogeneous MGs without the notion of mean-field approximation , where enforcing policy parameter incurs no suboptimality and 2 ) develop an algorithm that performs policy parameter sharing in homogeneous MGs in a soft manner with decentralized consensus-based policy update with convergence guarantees . Zhang & Zavlanos ( 2019 ) also develop a policy consensus algorithm for decentralized MARL , yet they do not assume homogeneity and thus need each agent to represent the joint policy for consensus . Communication-efficient MARL . There have been several recent works that also aim to achieve communication-efficiency in decentralized MARL . Chen et al . ( 2021b ) use pre-specified communication topology and reduce communication frequency for actor-critic via mini-batch updates ; in contrast , our work adaptively learn sparse communication topology during the decentralized training process . Chen et al . ( 2021a ) generalize their method of communication-efficient gradient descent from distributed supervised learning ( Chen et al. , 2018 ) to distributed reinforcement learning with policy gradient methods , where they assume the existence of a centralized controller that gather the policy gradients from decentralized agents which only communicate when the change in gradient exceeds a predefined threshold ; in contrast , our method does not rely on a centralized controller , and we empirically demonstrate the benefit of our adaptive communication learning over a rulebased baseline inspired by Chen et al . ( 2021a ) . Gupta et al . ( 2020 ) learn discrete messages among agents with a fixed communication topology , where the communicated messages are used to form the policy for action selection rather than for decentralized training . 3 HOMOGENEOUS MARKOV GAME . We consider a cooperative Markov game ( MG ) 〈N , S , A , P , R〉 with N agents indexed by i ∈ N = { 1 , ... , N } , state space S , action spaceA = A1×· · ·×AN , transition functionP : S×A×S → [ 0 , 1 ] , and reward functionsR = { Ri } i∈N withRi : S×A → R for each i ∈ N . In Section 3 , we assume full observability for simplicity , i.e. , each agent observes the state s ∈ S. Under full observability , we consider joint policies , π : S × A → [ 0 , 1 ] , that can be factored as the product of local policies πi : S × Ai → [ 0 , 1 ] , π ( a|s ) = ∏ i∈N π i ( ai|s ) . Let r ( s , a ) : = 1N ∑ i∈N R i ( s , a ) denote the joint reward function , and let γ ∈ [ 0 , 1 ] denote the discount factor . Define the discounted return from time step t as Gt = ∑∞ l=0 γ lrt+l , where rt : = r ( st , at ) is the reward at time step t. The agents ’ joint policy π = ( π1 , ... , πN ) induce a value function , which is defined as V π ( st ) = Est+1 : ∞ , at : ∞ [ Gt|st ] , and action-value function Qπ ( st , at ) = Est+1 : ∞ , at+1 : ∞ [ Gt|st , at ] . The agents are cooperative in the sense that they aim to optimize their policies with respect to the joint reward function , i.e. , maxπ J ( π ) = Es0 : ∞ , a0 : ∞ [ G0 ] . 3.1 HOMOGENEOUS MG : DEFINITION , PROPERTIES , AND EXAMPLES . As along as the action spaces { Ai } i∈N are homogeneous , policy sharing among { πi } i∈N is feasible . However , such policy sharing can incur suboptimal joint policies for general MGs , as we will see in an example introduced by Kuba et al . ( 2021 ) and revisited in this subsection . Here , we characterize a subclass of Markov games in Definition 1 requiring conditions stronger than homogeneous action spaces , where policy sharing provably incurs no suboptimality . Definition 1 ( Homogeneous Markov game ) . Markov game 〈N , S , A , P , R〉 is homogeneous if : ( i ) The local action spaces are homogeneous , i.e. , Ai = Aj ∀i , j ∈ N . Further , the state is decomposed into local states with homogeneous local state spaces , i.e. , s = ( s1 , ... , sN ) ∈ S = S1 × · · · × SN with Si = Sj ∀i , j ∈ N . ( ii ) The transition function and the joint reward function are permutation invariant . Formally , for any st = ( s1t , ... , s N t ) , st+1 = ( s 1 t+1 , ... , s N t+1 ) ∈ S and at = ( a1t , ... , aNt ) ∈ A , we have P ( Mst+1|Mst , Mat ) =P ( M ′st+1|M ′st , M ′at ) , R ( Mst , Mat ) = R ( M ′st , M ′at ) for any M , M ′ ∈ M , where R ( s , a ) : = ( R1 ( s , a ) , ... , RN ( s , a ) ) , Mx denotes a permutation M of ordered list x = ( x1 , ... , xN ) , andM is the set of all possible permutations . ( iii ) Each agent i ∈ N has access to a bijective function oi : S → O ( i.e. , each agent has full observability ) that maps states to a common observation space O . These observation functions { oi } i∈N are permutation preserving with respect to the state , i.e. , for any s ∈ S and any M ∈M , ( o1 ( Ms ) , ... , oN ( Ms ) ) = M ( o1 ( s ) , ... , oN ( s ) ) . By Definition 1 , besides requiring homogeneous action spaces , our characterization of homogeneous MGs further requires that the global state can be factored into homogeneous local states ( condition ( i ) ) such that the transition and reward functions are permutation invariant ( condition ( ii ) ) . Moreover , condition ( iii ) requires each agent to have an observation function to form its local representation of the global state . The main property of the homogeneous MG is that , after representing the global state with the observation functions , policy sharing incurs no suboptimality . This is formally stated in Theorem 1 . Proofs of all the theorems are included in the appendix . Theorem 1 . Let Π be the set of state-based joint policies , i.e. , Π = { π = ( π1 , ... , πN ) : πi : S × Ai → [ 0 , 1 ] } , and let Πo be the set of observation-based joint policies , i.e. , Πo = { πo = ( π1o , ... , π N o ) : π i o : O ×Ai → [ 0 , 1 ] } . In homogeneous MGs , we have max π= ( π1 , ... , πN ) ∈Π J ( π ) = max πo= ( π1o , ... , π N o ) ∈Πo J ( πo ) = max πo= ( π1o , ... , π N o ) ∈Πo : π1o= ... =πNo J ( πo ) . To provide more intuition for homogeneous MGs , we here give an example from Multi-Agent Particle Environment ( MPE ) ( Lowe et al. , 2017 ) and a non-example from Kuba et al . ( 2021 ) . Appendix D provides more examples and non-examples show the generality of our homogeneous MG subclass . Example : Cooperative Navigation . In a Cooperative Navigation task in MPE , N agents move as a team to cover N landmarks in a 2D space . The landmarks are randomly initialized at the beginning of an episode , and fixed throughout the episode . Under full observably where each agent can observe the information ( locations and/or velocities ) of all agents and landmarks , we can cast a Cooperative Navigation task as a homogeneous MG by verifying the three conditions in Definition 1 : ( i ) The local state of agent i consists of its absolute location li = ( lix , l i y ) ∈ R2 and its absolute velocity vi = ( vix , v i y ) ∈ R2 with respect to the common origin , as well as the absolute locations of all N landmarks , { ck = ( ckx , cky ) } Nk=1 . Therefore , the location state spaces are homogeneous , and the concatenation of all the location states preserves the global state of the task . Since local action is the change in velocity , the local action spaces are also homogeneous . ( ii ) The transition function determines the next global state by the current state and all agents ’ actions according to physics , and thus it is permutation invariant . The reward function Ri determines the reward for agent i according to the distances between all the agents and the landmarks to encourage coverage , as well as penalties to discourage collisions if any , resulting in a permutation invariant joint reward function . ( iii ) In MPE , agents ’ observations are based on relative , instead of absolute , locations and/or velocities of other agents and/or landmarks . For Cooperative Navigation , such observations happen to define observation functions that are bijective and permutation preserving . Specifically , function oi yields the observation for agent i that consists of its absolute location li and velocity vi , the relative location lij : = l j − li and velocity vij : = vj − vi of other agents j ∈ N \ { i } , and the relative location cik : = c k − li of all the landmarks k = 1 , .. , N . Non-Example : a stateless MG. Kuba et al . ( 2021 ) recently shows that enforcing policy parameter sharing is exponentially-worse than the optimality without such a restriction in the following stateless MG : Consider a cooperative MG with an even number of N agents , a state s fixed as the initial state , and the joint action space { 0 , 1 } N , where 1 ) the MG deterministically transits from state s to a terminal state after the first time step , and 2 ) the reward in state s is given by R ( s , 01 : N/2 , 11+N/2 : N ) = 1 and R ( s , a1 : N ) = 0 for all other joint actions . It is obvious that the optimal value of this MG ( in state s ) is 1 , while Kuba et al . ( 2021 ) prove that the optimal value under policy parameter sharing is 1/2N−1 . This MG is not a homogeneous MG : the agents are relying on the raw state to represent their policies , and therefore their observation functions are identity mappings oi ( s ) = s , which is not permutation preserving and violates Definition 1 ( iii ) .

Summary: This paper proposes a consensus-based actor-critic algorithm for multi-agent RL with limited communication during the training phase. This is in contrast to standard centralized training approaches such as MADDPG or COMA that assume access to the complete data form all agents for training critics. The paper motivates the proposed algorithm by an analysis of homogeneous Markov games (MGs), a subclass of MGs where parameter sharing preserves policy optimality. Inspired by the networked MARL consesus method of Zhang et al. (2018), the paper proposes a policy consensus algorithm that learns with which neighbours to share actions and observations or to perform consesus-based updates on policy and critic parameters using a bi-level bandit method. The effectiveness of the proposed algorithm in reducing the required communication during training is empirically verified in a suite of experiments in the multi-agent particle environment (MPE) domains.