Learning to Schedule Communication in Multi-agent Reinforcement Learning

1 INTRODUCTION . Reinforcement Learning ( RL ) has garnered renewed interest in recent years . Playing the game of Go ( Mnih et al. , 2015 ) , robotics control ( Gu et al. , 2017 ; Lillicrap et al. , 2015 ) , and adaptive video streaming ( Mao et al. , 2017 ) constitute just a few of the vast range of RL applications . Combined with developments in deep learning , deep reinforcement learning ( Deep RL ) has emerged as an accelerator in related fields . From the well-known success in single-agent deep reinforcement learning , such as Mnih et al . ( 2015 ) , we now witness growing interest in its multi-agent extension , the multi-agent reinforcement learning ( MARL ) , exemplified in Gupta et al . ( 2017 ) ; Lowe et al . ( 2017 ) ; Foerster et al . ( 2017a ) ; Omidshafiei et al . ( 2017 ) ; Foerster et al . ( 2016 ) ; Sukhbaatar et al . ( 2016 ) ; Mordatch & Abbeel ( 2017 ) ; Havrylov & Titov ( 2017 ) ; Palmer et al . ( 2017 ) ; Peng et al . ( 2017 ) ; Foerster et al . ( 2017c ) ; Tampuu et al . ( 2017 ) ; Leibo et al . ( 2017 ) ; Foerster et al . ( 2017b ) . In the MARL problem commonly addressed in these works , multiple agents interact in a single environment repeatedly and improve their policy iteratively by learning from observations to achieve a common goal . Of particular interest is the distinction between two lines of research : one fostering the direct communication among agents themselves , as in Foerster et al . ( 2016 ) ; Sukhbaatar et al . ( 2016 ) and the other coordinating their cooperative behavior without direct communication , as in Foerster et al . ( 2017b ) ; Palmer et al . ( 2017 ) ; Leibo et al . ( 2017 ) . In this work , we concern ourselves with the former . We consider MARL scenarios wherein the task at hand is of a cooperative nature and agents are situated in a partially observable environment , but each endowed with different observation power . We formulate this scenario into a multi-agent sequential decision-making problem , such that all agents share the goal of maximizing the same discounted sum of rewards . For the agents to directly communicate with each other and behave as a coordinated group rather than merely coexisting individuals , they must carefully determine the information they exchange under a practical bandwidth-limited environment and/or in the case of high-communication cost . To coordinate this exchange of messages , we adopt the centralized training and distributed execution paradigm popularized in recent works , e.g. , Foerster et al . ( 2017a ) ; Lowe et al . ( 2017 ) ; Sunehag et al . ( 2018 ) ; Rashid et al . ( 2018 ) ; Gupta et al . ( 2017 ) . In addition to bandwidth-related constraints , we take the issues of sharing the communication medium into consideration , especially when agents communicate over wireless channels . The stateof-the-art standards on wireless communication such as Wi-Fi and LTE specify the way of scheduling users as one of the basic functions . However , as elaborated in Related work , MARL problems involving scheduling of only a restricted set of agents have not yet been extensively studied . The key challenges in this problem are : ( i ) that limited bandwidth implies that agents must exchange succinct information : something concise and yet meaningful and ( ii ) that the shared medium means that potential contenders must be appropriately arbitrated for proper collision avoidance , necessitating a certain form of communication scheduling , popularly referred to as MAC ( Medium Access Control ) in the area of wireless communication . While stressing the coupled nature of the encoding/decoding and the scheduling issue , we zero in on the said communication channel-based concerns and construct our neural network accordingly . Contributions In this paper , we propose a new deep multi-agent reinforcement learning architecture , called SchedNet , with the rationale of centralized training and distributed execution in order to achieve a common goal better via decentralized cooperation . During distributed execution , agents are allowed to communicate over wireless channels where messages are broadcast to all agents in each agent ’ s communication range . This broadcasting feature of wireless communication necessitates a Medium Access Control ( MAC ) protocol to arbitrate contending communicators in a shared medium . CSMA ( Collision Sense Multiple Access ) in Wi-Fi is one such MAC protocol . While prior work on MARL to date considers only the limited bandwidth constraint , we additionally address the shared medium contention issue in what we believe is the first work of its kind : which nodes are granted access to the shared medium . Intuitively , nodes with more important observations should be chosen , for which we adopt a simple yet powerful mechanism called weight-based scheduler ( WSA ) , designed to reconcile simplicity in training with integrity of reflecting real-world MAC protocols in use ( e.g. , 802.11 Wi-Fi ) . We evaluate SchedNet for two applications : cooperative communication and navigation and predator/prey and demonstrate that SchedNet outperforms other baseline mechanisms such as the one without any communication or with a simple scheduling mechanism such as round robin . We comment that SchedNet is not intended for competing with other algorithms for cooperative multi-agent tasks without considering scheduling , but a complementary one . We believe that adding our idea of agent scheduling makes those algorithms much more practical and valuable . Related work We now discuss the body of relevant literature . Busoniu et al . ( 2008 ) and Tan ( 1993 ) have studied MARL with decentralized execution extensively . However , these are based on tabular methods so that they are restricted to simple environments . Combined with developments in deep learning , deep MARL algorithms have emerged ( Tampuu et al. , 2017 ; Foerster et al. , 2017a ; Lowe et al. , 2017 ) . Tampuu et al . ( 2017 ) uses a combination of DQN with independent Q-learning . This independent learning does not perform well because each agent considers the others as a part of environment and ignores them . Foerster et al . ( 2017a ) ; Lowe et al . ( 2017 ) ; Gupta et al . ( 2017 ) ; Sunehag et al . ( 2018 ) , and Foerster et al . ( 2017b ) adopt the framework of centralized training with decentralized execution , empowering the agent to learn cooperative behavior considering other agents ’ policies without any communication in distributed execution . It is widely accepted that communication can further enhance the collective intelligence of learning agents in their attempt to complete cooperative tasks . To this end , a number of papers have previously studied the learning of communication protocols and languages to use among multiple agents in reinforcement learning . We explore those bearing the closest resemblance to our research . Foerster et al . ( 2016 ) ; Sukhbaatar et al . ( 2016 ) ; Peng et al . ( 2017 ) ; Guestrin et al . ( 2002 ) , and Zhang & Lesser ( 2013 ) train multiple agents to learn a communication protocol , and have shown that communicating agents achieve better rewards at various tasks . Mordatch & Abbeel ( 2017 ) and Havrylov & Titov ( 2017 ) investigate the possibility of the artificial emergence of language . Coordinated RL by Guestrin et al . ( 2002 ) is an earlier work demonstrating the feasibility of structured communication and the agents ’ selection of jointly optimal action . Only DIAL ( Foerster et al. , 2016 ) and Zhang & Lesser ( 2013 ) explicitly address bandwidth-related concerns . In DIAL , the communication channel of the training environment has a limited bandwidth , such that the agents being trained are urged to establish more resource-efficient communication protocols . The environment in Zhang & Lesser ( 2013 ) also has a limited-bandwidth channel in effect , due to the large amount of exchanged information in running a distributed constraint optimization algorithm . Recently , Jiang & Lu ( 2018 ) proposes an attentional communication model that allows some agents who request additional information from others to gather observation from neighboring agents . However , they do not explicitly consider the constraints imposed by limited communication bandwidth and/or scheduling due to communication over a shared medium . To the best of our knowledge , there is no prior work that incorporates an intelligent scheduling entity in order to facilitate inter-agent communication in both a limited-bandwidth and shared medium access scenarios . As outlined in the introduction , intelligent scheduling among learning agents is pivotal in the orchestration of their communication to better utilize the limited available bandwidth as well as in the arbitration of agents contending for shared medium access . 2 BACKGROUND . Reinforcement Learning We consider a standard RL formulation based on Markov Decision Process ( MDP ) . An MDP is a tuple < S , A , r , P , γ > where S and A are the sets of states and actions , respectively , and γ ∈ [ 0 , 1 ] is the discount factor . A transition probability function P : S × A → S maps states and actions to a probability distribution over next states , and r : S × A → R denotes the reward . The goal of RL is to learn a policy π : S → A that solves the MDP by maximizing the expected discounted return Rt = E [ ∑∞ k=0 γ krt+k|π ] . The policy induces a value function V π ( s ) = Eπ [ Rt|st = s ] , and an action value function Qπ ( s , a ) = Eπ [ Rt|st = s , at = a ] . Actor-critic Method The main idea of the policy gradient method is to optimize the policy , parametrized by θπ , in order to maximize the objective J ( θ ) = Es∼pπ , a∼πθ [ R ] by directly adjusting the parameters in the direction of the gradient . By the policy gradient theorem Sutton et al . ( 2000 ) , the gradient of the objective is : ∇θJ ( πθ ) = Es∼ρπ , a∼πθ [ ∇θ log πθ ( a|s ) Qπ ( s , a ) ] , ( 1 ) where ρπ is the state distribution . Our baseline algorithmic framework is the actor-critic approach Konda & Tsitsiklis ( 2003 ) . In this approach , an actor adjusts the parameters θ of the policy πθ ( s ) by gradient ascent . Instead of the unknown true action-value function Qπ ( s , a ) , its approximated version Qw ( s , a ) is used with parameter w. A critic estimates the action-value function Qw ( s , a ) using an appropriate policy evaluation algorithm such as temporal-difference learning Tesauro ( 1995 ) . To reduce the variance of the gradient updates , some baseline function b ( s ) is often subtracted from the action value , thereby resulting in Qπ ( s , a ) − b ( s ) Sutton & Barto ( 1998 ) . A popular choice for this baseline function is the state value V ( s ) , which indicates the inherent “ goodness ” of the state . This difference between the action value and the state value is often dubbed as the advantage A ( s , a ) whose TD-error-based substitute δt = rt+γV ( st+1 ) −V ( st ) is an unbiased estimate of the advantage as in Mnih et al . ( 2016 ) . The actor-critic algorithm can also be applied to a deterministic policy µθ : S → A . By the deterministic policy gradient theorem Silver et al . ( 2014 ) , we update the parameters as follows : ∇θJ ( µθ ) = Es∼ρµ [ ∇θµθ ( s ) ∇aQµ ( s , a ) |a=µθ ( s ) ] . ( 2 ) MARL : Centralized Critic and Distributed Actor ( CCDA ) We formalize MARL using DECPOMDP ( Oliehoek et al. , 2016 ) , which is a generalization of MDP to allow a distributed control by multiple agents who may be incapable of observing the global state . A DEC-POMDP is described by a tuple < S , A , r , P , Ω , O , γ > . We use bold-face fonts in some notations to highlight the context of multi-agents . Each agent i ∈ N chooses an action ai ∈ A , forming a joint action vector a = [ ai ] ∈ An and has partial observations oi ∈ Ω according to some observation function O ( s , i ) : S × N 7→ Ω. P ( s′|s , a ) : S × An 7→ [ 0 , 1 ] is the transition probability function . All agents share the same reward r ( s , u ) : S × An 7→ R. Each agent i takes action ai based on its own policy πi ( ai|oi ) . As mentioned in Section 1 , our particular focus is on the centralized training and distributed execution paradigm , where the actor-critic approach is a good fit to such a paradigm . Since the agents should execute in a distributed setting , each agent , say i , maintains its own actor that selects i ’ s action based only on what is partially observed by i . The critic is naturally responsible for centralized training , and thus works in a centralized manner . Thus , the critic is allowed to have the global state s as its input , which includes all agents ’ observations and extra information from the environment . The role of the critic is to “ criticize ” individual agent ’ s actions . This centralized nature of the critic helps in providing more accurate feedback to the individual actors with limited observation horizon . In this case , each agent ’ s policy , πi , is updated by a variant of ( 1 ) as : ∇θJ ( πiθ ) = Es∼ρπ , a∼πθ [ ∇θ log πiθ ( ai|oi ) ( r + γV ( st+1 ) − V ( st ) ) ] . ( 3 )

Summary: The authors present a study on scheduling multi-agent communication. Specifically, the authors look into cases where agents share the same reward and they are in a partially observable environment, each of them with different observations. The main contribution of this work is that authors provide a model for communication scheduling for dealing with cases where only a certain number of agents is allowed to communicate.