Communication-Efficient Actor-Critic Methods for Homogeneous Markov Games

Recent success in cooperative multi-agent reinforcement learning ( MARL ) relies on centralized training and policy sharing . Centralized training eliminates the issue of non-stationarity MARL yet induces large communication costs , and policy sharing is empirically crucial to efficient learning in certain tasks yet lacks theoretical justification . In this paper , we formally characterize a subclass of cooperative Markov games where agents exhibit a certain form of homogeneity such that policy sharing provably incurs no suboptimality . This enables us to develop the first consensus-based decentralized actor-critic method where the consensus update is applied to both the actors and the critics while ensuring convergence . We also develop practical algorithms based on our decentralized actor-critic method to reduce the communication cost during training , while still yielding policies comparable with centralized training . 1 INTRODUCTION . Cooperative multi-agent reinforcement learning ( MARL ) is the problem where multiple agents learn to make sequential decisions in a common environment to optimize a shared reward signal , which finds a wide range of real-world applications such as traffic control ( Chu et al. , 2019 ) , power grid management ( Callaway & Hiskens , 2010 ) , and coordination of multi-robot systems ( Corke et al. , 2005 ) . Efficient learning for large and complex cooperative MARL tasks is challenging . Naively reducing cooperative MARL to single-agent RL with a joint observation-action space imposes significant scalability issues , since the joint space grows exponentially with the number of agents . Approaches that treat each agent as an independent RL learner , such as Independent Q-Learning ( Tan , 1993 ) , overcome the scalability issue yet fail to succeed in complicated tasks due to the nonstationarity caused by other learning agents ’ evolving policies . To address these challenges , the paradigm of Centralized Training and Decentralized Execution ( CTDE ) is then proposed , where a centralized trainer is assumed to access to information of all agents during training to approximate the global ( action- ) value function , whereas each agent only needs local information for its action selection during decentralized policy execution ( Lowe et al. , 2017 ; Foerster et al. , 2017 ) . The centralized critic eliminates non-stationarity during training , while the policy decentralization ensures scalability during execution . Besides , existing CTDE methods almost always enable policy parameter sharing to further improve learning scalability and efficiency , where agents also share the parameters of their decentralized policies . However , in many real-world scenarios , there is not a readily available centralizer that conveniently gathers the global information from all agents , and therefore agents need to rely on all-to-all communication for centralized training , incurring enormous communication overheads for large numbers of agents . This motivates us to think about whether it is possible to train agents in a decentralized and communication-efficient manner , while still keeping the benefits of the centralized training of CTDE . Moreover , despite its wide adoption , little theoretical understanding has been provided to justify policy parameter sharing . Agents should at least exhibit a certain level of homogeneity before it is feasible to share their policies . For example , if the observation and/or action spaces vary across agents , then their decentralized policies can not even have the same architecture . Even if it is feasible , it is unclear whether restricting the agents to share their policy parameters will introduce any suboptimality . In this paper , we address these aforementioned issues centered around the CTDE framework . We begin by formally characterizing a subclass of Markov games where the cooperative agents exhibit a certain form of homogeneity such that it is not only feasible but also incurs no suboptimality to share their decentralized policies , thus providing a first theoretical justification for policy parameter sharing . We then develop a decentralized actor-critic algorithm for homogeneous MGs where agents share their critic and actor parameters with consensus-based updates , for which we prove an asymptotic convergence guarantee with linear critics , full observability , and other standard assumptions . To our knowledge , this is the first decentralized actor-critic algorithm that enjoys provable convergence guarantees with policy ( i.e. , actor ) consensus . To account for communication efficiency , we develop a simple yet effective bandit-based process that wisely selects when and with whom to perform the parameter census update based on the feedback of policy improvement during training . To further account for partial observability , we develop an end-to-end learnable gating mechanism for the agents to selectively share their observations and actions for learning the decentralized critics . This series of innovations are capable of transforming any CTDE algorithm into its decentralized and communication-efficient counterpart , with policy consensus in homogeneous MGs for improved efficiency . Our empirical results demonstrate the effectiveness of these innovations when instantiated with a state-of-the-art CTDE algorithm , achieving competitive policy performance with only a fraction of communication during training . Our contribution is therefore summarized as three-fold : ( 1 ) the characterization of a subclass of cooperative Markov games , i.e . homogeneous Markov games ( MGs ) , where policy sharing provably incurs no loss of optimality ; ( 2 ) a decentralized MARL algorithm for homogeneous MGs that enjoys asymptotic convergence guarantee with policy consensus ; and ( 3 ) practical techniques that transform CTDE algorithms to their decentralized and communication-efficient counterparts . 2 RELATED WORK . Communication in cooperative MARL . Communication is key to solving the issue of nonstationarity in cooperative MARL . The CTDE paradigm ( Lowe et al. , 2017 ; Foerster et al. , 2017 ) assumes a centralized unit during training to learn a joint value function . Other methods , such as CommNet ( Sukhbaatar et al. , 2016 ) and BiCNet ( Peng et al. , 2017 ) , do not assume a centralized unit and instead allow agents to share information by all-to-all broadcasting , effectively relying on centralized communication . These methods require centralized/all-to-all communication that impedes their application to large numbers of agents . Although follow-up work such as IC3Net ( Singh et al. , 2018 ) and VBC ( Zhang et al. , 2019 ) proposes algorithms to learn when to communicate , agents there still perform all-to-all communication before others decide whether to receive . We instead entirely abandon centralized/all-to-all communication , letting each agent decide whom to communicate to purely based on its local observation . There is another line of work , Networked MARL ( NMARL ) ( Zhang et al. , 2018 ) , that where agents lie on a predefined network such that neighboring agents can freely communicate . Our approach instead learns sparse communication that is dynamically adjusted during decentralized training , even if the predefined network topology can be dense . Policy parameter sharing and consensus . Policy parameter sharing is widely adopted in MARL where agents share the same action space , yet it has not been theoretically justified except under the mean-field approximation where the transition dynamics depends on the collective statistics of all agents and not on the identities and ordering of individual agents ( Nguyen et al. , 2017a ; b ; Yang et al. , 2018 ) . A recent result from Kuba et al . ( 2021 ) states that enforcing policy parameter sharing in a general cooperative MG can lead to a suboptimal outcome that is exponentially-worse with the number of agents . We are the first to 1 ) formally characterize the subclass of homogeneous MGs without the notion of mean-field approximation , where enforcing policy parameter incurs no suboptimality and 2 ) develop an algorithm that performs policy parameter sharing in homogeneous MGs in a soft manner with decentralized consensus-based policy update with convergence guarantees . Zhang & Zavlanos ( 2019 ) also develop a policy consensus algorithm for decentralized MARL , yet they do not assume homogeneity and thus need each agent to represent the joint policy for consensus . Communication-efficient MARL . There have been several recent works that also aim to achieve communication-efficiency in decentralized MARL . Chen et al . ( 2021b ) use pre-specified communication topology and reduce communication frequency for actor-critic via mini-batch updates ; in contrast , our work adaptively learn sparse communication topology during the decentralized training process . Chen et al . ( 2021a ) generalize their method of communication-efficient gradient descent from distributed supervised learning ( Chen et al. , 2018 ) to distributed reinforcement learning with policy gradient methods , where they assume the existence of a centralized controller that gather the policy gradients from decentralized agents which only communicate when the change in gradient exceeds a predefined threshold ; in contrast , our method does not rely on a centralized controller , and we empirically demonstrate the benefit of our adaptive communication learning over a rulebased baseline inspired by Chen et al . ( 2021a ) . Gupta et al . ( 2020 ) learn discrete messages among agents with a fixed communication topology , where the communicated messages are used to form the policy for action selection rather than for decentralized training . 3 HOMOGENEOUS MARKOV GAME . We consider a cooperative Markov game ( MG ) 〈N , S , A , P , R〉 with N agents indexed by i ∈ N = { 1 , ... , N } , state space S , action spaceA = A1×· · ·×AN , transition functionP : S×A×S → [ 0 , 1 ] , and reward functionsR = { Ri } i∈N withRi : S×A → R for each i ∈ N . In Section 3 , we assume full observability for simplicity , i.e. , each agent observes the state s ∈ S. Under full observability , we consider joint policies , π : S × A → [ 0 , 1 ] , that can be factored as the product of local policies πi : S × Ai → [ 0 , 1 ] , π ( a|s ) = ∏ i∈N π i ( ai|s ) . Let r ( s , a ) : = 1N ∑ i∈N R i ( s , a ) denote the joint reward function , and let γ ∈ [ 0 , 1 ] denote the discount factor . Define the discounted return from time step t as Gt = ∑∞ l=0 γ lrt+l , where rt : = r ( st , at ) is the reward at time step t. The agents ’ joint policy π = ( π1 , ... , πN ) induce a value function , which is defined as V π ( st ) = Est+1 : ∞ , at : ∞ [ Gt|st ] , and action-value function Qπ ( st , at ) = Est+1 : ∞ , at+1 : ∞ [ Gt|st , at ] . The agents are cooperative in the sense that they aim to optimize their policies with respect to the joint reward function , i.e. , maxπ J ( π ) = Es0 : ∞ , a0 : ∞ [ G0 ] . 3.1 HOMOGENEOUS MG : DEFINITION , PROPERTIES , AND EXAMPLES . As along as the action spaces { Ai } i∈N are homogeneous , policy sharing among { πi } i∈N is feasible . However , such policy sharing can incur suboptimal joint policies for general MGs , as we will see in an example introduced by Kuba et al . ( 2021 ) and revisited in this subsection . Here , we characterize a subclass of Markov games in Definition 1 requiring conditions stronger than homogeneous action spaces , where policy sharing provably incurs no suboptimality . Definition 1 ( Homogeneous Markov game ) . Markov game 〈N , S , A , P , R〉 is homogeneous if : ( i ) The local action spaces are homogeneous , i.e. , Ai = Aj ∀i , j ∈ N . Further , the state is decomposed into local states with homogeneous local state spaces , i.e. , s = ( s1 , ... , sN ) ∈ S = S1 × · · · × SN with Si = Sj ∀i , j ∈ N . ( ii ) The transition function and the joint reward function are permutation invariant . Formally , for any st = ( s1t , ... , s N t ) , st+1 = ( s 1 t+1 , ... , s N t+1 ) ∈ S and at = ( a1t , ... , aNt ) ∈ A , we have P ( Mst+1|Mst , Mat ) =P ( M ′st+1|M ′st , M ′at ) , R ( Mst , Mat ) = R ( M ′st , M ′at ) for any M , M ′ ∈ M , where R ( s , a ) : = ( R1 ( s , a ) , ... , RN ( s , a ) ) , Mx denotes a permutation M of ordered list x = ( x1 , ... , xN ) , andM is the set of all possible permutations . ( iii ) Each agent i ∈ N has access to a bijective function oi : S → O ( i.e. , each agent has full observability ) that maps states to a common observation space O . These observation functions { oi } i∈N are permutation preserving with respect to the state , i.e. , for any s ∈ S and any M ∈M , ( o1 ( Ms ) , ... , oN ( Ms ) ) = M ( o1 ( s ) , ... , oN ( s ) ) . By Definition 1 , besides requiring homogeneous action spaces , our characterization of homogeneous MGs further requires that the global state can be factored into homogeneous local states ( condition ( i ) ) such that the transition and reward functions are permutation invariant ( condition ( ii ) ) . Moreover , condition ( iii ) requires each agent to have an observation function to form its local representation of the global state . The main property of the homogeneous MG is that , after representing the global state with the observation functions , policy sharing incurs no suboptimality . This is formally stated in Theorem 1 . Proofs of all the theorems are included in the appendix . Theorem 1 . Let Π be the set of state-based joint policies , i.e. , Π = { π = ( π1 , ... , πN ) : πi : S × Ai → [ 0 , 1 ] } , and let Πo be the set of observation-based joint policies , i.e. , Πo = { πo = ( π1o , ... , π N o ) : π i o : O ×Ai → [ 0 , 1 ] } . In homogeneous MGs , we have max π= ( π1 , ... , πN ) ∈Π J ( π ) = max πo= ( π1o , ... , π N o ) ∈Πo J ( πo ) = max πo= ( π1o , ... , π N o ) ∈Πo : π1o= ... =πNo J ( πo ) . To provide more intuition for homogeneous MGs , we here give an example from Multi-Agent Particle Environment ( MPE ) ( Lowe et al. , 2017 ) and a non-example from Kuba et al . ( 2021 ) . Appendix D provides more examples and non-examples show the generality of our homogeneous MG subclass . Example : Cooperative Navigation . In a Cooperative Navigation task in MPE , N agents move as a team to cover N landmarks in a 2D space . The landmarks are randomly initialized at the beginning of an episode , and fixed throughout the episode . Under full observably where each agent can observe the information ( locations and/or velocities ) of all agents and landmarks , we can cast a Cooperative Navigation task as a homogeneous MG by verifying the three conditions in Definition 1 : ( i ) The local state of agent i consists of its absolute location li = ( lix , l i y ) ∈ R2 and its absolute velocity vi = ( vix , v i y ) ∈ R2 with respect to the common origin , as well as the absolute locations of all N landmarks , { ck = ( ckx , cky ) } Nk=1 . Therefore , the location state spaces are homogeneous , and the concatenation of all the location states preserves the global state of the task . Since local action is the change in velocity , the local action spaces are also homogeneous . ( ii ) The transition function determines the next global state by the current state and all agents ’ actions according to physics , and thus it is permutation invariant . The reward function Ri determines the reward for agent i according to the distances between all the agents and the landmarks to encourage coverage , as well as penalties to discourage collisions if any , resulting in a permutation invariant joint reward function . ( iii ) In MPE , agents ’ observations are based on relative , instead of absolute , locations and/or velocities of other agents and/or landmarks . For Cooperative Navigation , such observations happen to define observation functions that are bijective and permutation preserving . Specifically , function oi yields the observation for agent i that consists of its absolute location li and velocity vi , the relative location lij : = l j − li and velocity vij : = vj − vi of other agents j ∈ N \ { i } , and the relative location cik : = c k − li of all the landmarks k = 1 , .. , N . Non-Example : a stateless MG. Kuba et al . ( 2021 ) recently shows that enforcing policy parameter sharing is exponentially-worse than the optimality without such a restriction in the following stateless MG : Consider a cooperative MG with an even number of N agents , a state s fixed as the initial state , and the joint action space { 0 , 1 } N , where 1 ) the MG deterministically transits from state s to a terminal state after the first time step , and 2 ) the reward in state s is given by R ( s , 01 : N/2 , 11+N/2 : N ) = 1 and R ( s , a1 : N ) = 0 for all other joint actions . It is obvious that the optimal value of this MG ( in state s ) is 1 , while Kuba et al . ( 2021 ) prove that the optimal value under policy parameter sharing is 1/2N−1 . This MG is not a homogeneous MG : the agents are relying on the raw state to represent their policies , and therefore their observation functions are identity mappings oi ( s ) = s , which is not permutation preserving and violates Definition 1 ( iii ) .

Summary: This paper considers a subclass of cooperative Markov games where agents exhibit a certain form of homogeneity such that policy sharing provably incurs no sub-optimality. Based on this property, the paper develops a consensus-based decentralized actor-critic method where the consensus update is applied to both the actors and the critics while ensuring convergence. Furthermore, heuristic algorithms have been developed and incorporated into the consensus-based decentralized actor-critic method to reduce the communication cost during training.