Learning Predictive Communication by Imagination in Networked System Control

1 INTRODUCTION . Networked system control ( NSC ) is extensively studied and widely applied , including connected vehicle control ( Jin & Orosz , 2014 ) , traffic signal control ( Chu et al. , 2020b ) , distributed sensing ( Xu et al. , 2016 ) , networked storage operation ( Qin et al. , 2015 ) etc . In NSC , agents are connected via a communication network for a cooperative control objective . For example , in an adaptive traffic signal control system , each traffic light performs decentralized control based on its local observations and messages from connected neighbors . Although deep reinforcement learning has been successfully applied to some complex problems , such as Go ( Silver et al. , 2016 ) , and Starcraft II ( Vinyals et al. , 2019 ) , it is still not scalable in many real-world networked control problems . Multiagent reinforcement learning ( MARL ) addresses the issue of scalability by performing decentralized control . Recent decentralized MARL performs decentralized control based on the assumptions of global observations and local or global rewards ( Zhang et al. , 2018 ; 2019a ; Qu et al. , 2019 ; 2020b ; a ) , which are reasonable in multi-agent gaming but not suitable in NSC . A practical solution is to allow each agent to perform decentralized control based on its local observations and messages from the connected neighbors . Various communication-based methods are proposed to stabilize training and improve observability , and communication is studied to enable agents to behave as a group , rather than a collection of individuals ( Sukhbaatar & Fergus , 2016 ; Chu et al. , 2020a ) . Despite recent advances in neural communication ( Sukhbaatar & Fergus , 2016 ; Foerster et al. , 2016 ; Chu et al. , 2020a ) , delayed global information sharing remains an open problem that widely exists in many NSC applications . Communication protocol not only reflects the situation at hand but also guides the policy optimization . Recent deep neural models ( Sukhbaatar & Fergus , 2016 ; Foerster et al. , 2016 ; Hoshen , 2017 ) implement differentiable communication based on available connections . However , in NSC , such as traffic signal control , each agent only connects to its neighbors , leading to a delay in receiving messages from the distant agents in the system , and the non-stationarity mainly comes from these partial observation ( Chu et al. , 2020a ) . Communication with delayed global information limits the learnability of RL because RL agents can only use the delayed information and not leverage potential future information . Moreover , it is not efficient in situations where an environment is sensitive when the behaviours of agents change . It is therefore of great practical relevance to develop algorithms which can learn beyond the communication with the delayed information sharing . In this paper we introduce ImagComm that learns communication by imagination for multi-agent reinforcement learning in NSC . We leverage the model of the agent ’ s world to provide an estimate of farsighted information in latent space for communication . At each time step , the agent is allowed to imagine its future states in an abstract space and convey this information to its neighbors . Therefore unlike previous works , our communication protocol conveys not only the current sharing information but also the imagined sharing information . It is applicable whenever communication changes frequently , e.g . at every time step agents may receive new communication information . We summarize our main contributions as follows : ( 1 ) We first introduce the imagination module that can be used to learn latent dynamics for communication in networked multi-agent systems control . ( 2 ) We predict the future state of each local agent and allow each agent to convey the latent state to neighbors as messages , which reduce the delay of global information . ( 3 ) We demonstrate that leveraging the predictive communication by imagination in latent space succeeds in networked system control . We explore this model on a range of NSC tasks . Our results demonstrate that our method consistently outperform baselines on these tasks . 2 RELATED WORK . Networked system control ( NSC ) considers the problem where agents are connected via a communication network for a cooperative control objective , such as autonomous vehicle control ( Jin & Orosz , 2014 ) , adaptive traffic signal control ( Chu et al. , 2020b ) , and distributed sensing ( Xu et al. , 2016 ) , etc . Recently reinforcement learning has become popular for NSC through decentralized control and communications by networked agents . Communication is an important part for multi-agent RL to compensate for the information loss in partial observations . Heuristic communication allows the agents to share some certain forms of information , such as policy fingerprints from other agents ( Foerster et al. , 2017 ) and averaged neighbor ’ s policies ( Yang et al. , 2018 ) . Recently end-to-end differentiable communications have become popular ( Foerster et al. , 2016 ; Sukhbaatar & Fergus , 2016 ; Chu et al. , 2020a ) since the communication channel is learned to optimize the performance . Attention-based communication ( Hoshen , 2017 ; Das et al. , 2019 ; Singh et al. , 2019 ) selectively send messages to the agents chosen , however , these are not suitable for NSC since the communication is allowed only between connected neighbors . Our method adopts differentiable communication with end-to-end training . Compared to existing works , we introduce a new predictive communication module through learning latent dynamics . Learning latent dynamics has been studied to solve single agent tasks , such as E2C ( Watter et al. , 2015 ) , RCE ( Banijamali et al. , 2018 ) , PlaNet ( Hafner et al. , 2019 ) , SOLAR ( Zhang et al. , 2019b ) and so on . Lee et al . ( 2019 ) and Gregor et al . ( 2019 ) learn belief representations to accelerate modelfree agents . World Models ( Ha & Schmidhuber , 2018 ) learn latent dynamics in a two-stage process to evolve linear controllers in imagination . I2A ( Racanière et al. , 2017 ) hands imagined trajectories to a model-free policy based on a rollout encoder . In contrast to these works , our work considers multi-agent tasks and learns predictive communication by imagination in latent space . 3 PRELIMINARIES . In networked system control problem , we work with a networked system , which is described by a graph G ( V , E ) , where i ∈ V denotes the ith agent and ij ∈ E denotes the communication link between agents i and j . The corresponding networked ( cooperative ) multi-agent MDP is defined by a tuple ( G , { Si , Ai } i∈V , { Mij } ij∈E , p , { ri } i∈V ) . Si and Ai are the local state space and action space of agent i . Let S : = ∪i∈VSi and A : = ∪i∈VAi , the MDP transitions follow a stationary probability distribution p : S×A×S → [ 0 , 1 ] . The global reward is denoted by r : S×A → R and defined as r = 1|V| ∑ i∈V ri indicating that all local rewards are shared globally . The communication is limited to neighborhoods . M denotes the message space for the communication model . That is each agent i observes s̃i , t : = si , t ∪mNii , t , where si , t ∈ Si denotes local state space of agent i and mNii , t : = { mji , t } j∈Ni andNi : = { j ∈ V|ji ∈ E } . Message mji , t ∈Mji denotes all the available information at an agent ’ s neighbor . In NSC , the system is decentralized and the communication is limited to neighborhoods . Each agent i follows a decentralized policy πi : S̃i × Ai → [ 0 , 1 ] to choose its own action ai , t ∼ πi ( ·|s̃i , t ) at time t. The objective is to maximize Eπ [ R0 ] , where Rt = ∑∞ l=0 γ lrt+l and γ is a discount factor . 4 METHODOLOGY . Our goal is to learn predictive communication on a particular observation or environment state . We start by introducing the networked MDP with neighborhood communications and delayed information issue in communication . Then , we describe ImagComm that utilizes predictive communication , which we learn the agent ’ s world model to provide an additional context for communication . 4.1 DELAYED COMMUNICATION IN NETWORKED SYSTEM CONTROL . Following the setting of NSC in ( Chu et al. , 2020a ) , we assume that all messages sent from agent i are identical and we denote mij = mi , ∀j ∈ Ni . The message explicitly includes state s and policy π and agent belief h , i.e. , mi , t = si , t ∪ πi , t−1 ∪ hi , t−1 in communication . Note that πi , t−1 is the probability distribution over discrete actions . Thus for each agent in NSC , s̃i , t : = sVi , t ∪ πNi , t−1 ∪ hNi , t−1 . Note the communication phase is prior-decision , so only hi , t−1 and πi , t−1 are available . This protocol can be easily extended for multi-pass communication . We assume that any information that agent j knows at time t can be included in mji , t and mji , t = sj , t ∪ { mkj , t−1 } k∈Nj . Then s̃i , t : = si , t ∪ { sj , t+1−dij } j∈V/ { i } , which includes the delayed global observations . dij indicates the distance between i and j , i.e . the hops between two agents on the graph of the networked system . We illustrate the delayed information in Figure 1 . A more rigorous analysis of this conclusion can be found in the Appendix A . 4.2 PREDICTIVE COMMUNICATION . To reduce the delay of global information , we consider a forward model for predicting future states of each agent j , then sj , t+1 can be encoded as a message for communication , and agent i can benefit from this information . Let ŝi , t be the abstract state of ith agent , Wi ∈ Wi be a world model of the transition dynamics from ŝi , t to the abstract state ŝi , t+1 , and let bi , t : = ∪kτ=1ŝi , t+τ denote the predictive message . We aim to build a policy based on delayed global observations and predictive messages . The value of policy πi can be defined as V π , W i ( s ) based on the model Wi : V π , Wi ( s , aNi ) = Eai , t∼πi ( ·|s̃i , t , bi , t ) [ Rπi , t | s̃t = s , aNi , t = aNi ] . ( 1 ) Learning based on ( 1 ) has the benefit of reduced delay in global information compared to that without bi , t ; this is formally presented in Proposition 1 . Proofs are provided in Appendix A . Proposition 1 . ImagComm can reduce the delay of global information by incorporating a predictive model in the communication protocol . We are now interested in constructing an abstract model Ŵi ( · ; ϕ ) to approximateWi , which operates on an abstract state . Let ŝi , t+1 be the new abstract state sampled by ŝi , t+1 ∼ Ŵi ( ŝi , t ) . We want to minimize‖ŝi , t+1 − gi ( si , t+1 ; ψ ) ‖ , where gi ( · ; ψ ) is an embedding of raw states . Let V π , Ŵi be the value function of the policy on the estimated model Ŵi . Towards optimizing V π , W∗ i ( s , aNi ) , we build a lower bound as follows and maximize it iteratively : V π , W ∗ i ( s , aNi ) ≥ V π , Ŵi ( s , aNi ) −D ( Ŵ , π ) , ( 2 ) where D ( Ŵ , π ) ∈ R bounds the discrepancy between V π , W ? i and V π , Ŵ i . In practice , D ( Ŵi , πi ) is defined as Dπrefi ( Ŵi , πi ) = α · Es0 , ... , st , ∼πrefi [ ‖Ŵi ( ŝi , t ) − gi ( si , t+1 ) ‖ ] , ( 3 ) where α is a hyperparameter , πrefi is the policy used for sampling . For each agent , we solve the following problem : πk+1 , W k+1 = argmax π∈Π , W∈W V π , Wi −Dπki , δ ( W , π ) . ( 4 ) With the predictive imagination module , each agent utilizes the estimate of predictive state information to learn its belief and optimize the control performance of all other agents . Follow the analysis in ( Luo et al. , 2018 ) , we can show that ImagComm can lead to monotonic improvement in policy iteration . Proofs are defered to Appendix A . Proposition 2 . Suppose that W ∗i ∈ Wi is the optimal model and the optimization problem in equation ( 4 ) is solvable at each iteration . Solving ( 4 ) produces a sequence of policies π0i , . . . , π T i with monotonically increasing values : V π 0 , W∗ i ≤ V π 1 , W∗ i ≤ · · · ≤ V π T , W∗ i . A conclusion following directly from Proposition 2 is that solving ( 4 ) will converge to a local maximum . ImagComm considers build a world model and predict the farsighted state by a imagination module to eliminate the delay in global information and henceforth reduce the negative influence of the partial observability . Because the future information after time t compensate for some of the delayed information at time t. Next we will present the differentiable neural communication with imagination .

Summary: The paper proposes to communicate predicted local states between neighboring agents to address the problem of delayed information in networked multi-agent reinforcement learning. To enable agents to predict future states, a world model is learned at each agent. It is empirically demonstrated that the proposed method has good performance in traffic signal control and cooperative adaptive cruise control.