Learning Predictive Communication by Imagination in Networked System Control

1 INTRODUCTION . Networked system control ( NSC ) is extensively studied and widely applied , including connected vehicle control ( Jin & Orosz , 2014 ) , traffic signal control ( Chu et al. , 2020b ) , distributed sensing ( Xu et al. , 2016 ) , networked storage operation ( Qin et al. , 2015 ) etc . In NSC , agents are connected via a communication network for a cooperative control objective . For example , in an adaptive traffic signal control system , each traffic light performs decentralized control based on its local observations and messages from connected neighbors . Although deep reinforcement learning has been successfully applied to some complex problems , such as Go ( Silver et al. , 2016 ) , and Starcraft II ( Vinyals et al. , 2019 ) , it is still not scalable in many real-world networked control problems . Multiagent reinforcement learning ( MARL ) addresses the issue of scalability by performing decentralized control . Recent decentralized MARL performs decentralized control based on the assumptions of global observations and local or global rewards ( Zhang et al. , 2018 ; 2019a ; Qu et al. , 2019 ; 2020b ; a ) , which are reasonable in multi-agent gaming but not suitable in NSC . A practical solution is to allow each agent to perform decentralized control based on its local observations and messages from the connected neighbors . Various communication-based methods are proposed to stabilize training and improve observability , and communication is studied to enable agents to behave as a group , rather than a collection of individuals ( Sukhbaatar & Fergus , 2016 ; Chu et al. , 2020a ) . Despite recent advances in neural communication ( Sukhbaatar & Fergus , 2016 ; Foerster et al. , 2016 ; Chu et al. , 2020a ) , delayed global information sharing remains an open problem that widely exists in many NSC applications . Communication protocol not only reflects the situation at hand but also guides the policy optimization . Recent deep neural models ( Sukhbaatar & Fergus , 2016 ; Foerster et al. , 2016 ; Hoshen , 2017 ) implement differentiable communication based on available connections . However , in NSC , such as traffic signal control , each agent only connects to its neighbors , leading to a delay in receiving messages from the distant agents in the system , and the non-stationarity mainly comes from these partial observation ( Chu et al. , 2020a ) . Communication with delayed global information limits the learnability of RL because RL agents can only use the delayed information and not leverage potential future information . Moreover , it is not efficient in situations where an environment is sensitive when the behaviours of agents change . It is therefore of great practical relevance to develop algorithms which can learn beyond the communication with the delayed information sharing . In this paper we introduce ImagComm that learns communication by imagination for multi-agent reinforcement learning in NSC . We leverage the model of the agent ’ s world to provide an estimate of farsighted information in latent space for communication . At each time step , the agent is allowed to imagine its future states in an abstract space and convey this information to its neighbors . Therefore unlike previous works , our communication protocol conveys not only the current sharing information but also the imagined sharing information . It is applicable whenever communication changes frequently , e.g . at every time step agents may receive new communication information . We summarize our main contributions as follows : ( 1 ) We first introduce the imagination module that can be used to learn latent dynamics for communication in networked multi-agent systems control . ( 2 ) We predict the future state of each local agent and allow each agent to convey the latent state to neighbors as messages , which reduce the delay of global information . ( 3 ) We demonstrate that leveraging the predictive communication by imagination in latent space succeeds in networked system control . We explore this model on a range of NSC tasks . Our results demonstrate that our method consistently outperform baselines on these tasks . 2 RELATED WORK . Networked system control ( NSC ) considers the problem where agents are connected via a communication network for a cooperative control objective , such as autonomous vehicle control ( Jin & Orosz , 2014 ) , adaptive traffic signal control ( Chu et al. , 2020b ) , and distributed sensing ( Xu et al. , 2016 ) , etc . Recently reinforcement learning has become popular for NSC through decentralized control and communications by networked agents . Communication is an important part for multi-agent RL to compensate for the information loss in partial observations . Heuristic communication allows the agents to share some certain forms of information , such as policy fingerprints from other agents ( Foerster et al. , 2017 ) and averaged neighbor ’ s policies ( Yang et al. , 2018 ) . Recently end-to-end differentiable communications have become popular ( Foerster et al. , 2016 ; Sukhbaatar & Fergus , 2016 ; Chu et al. , 2020a ) since the communication channel is learned to optimize the performance . Attention-based communication ( Hoshen , 2017 ; Das et al. , 2019 ; Singh et al. , 2019 ) selectively send messages to the agents chosen , however , these are not suitable for NSC since the communication is allowed only between connected neighbors . Our method adopts differentiable communication with end-to-end training . Compared to existing works , we introduce a new predictive communication module through learning latent dynamics . Learning latent dynamics has been studied to solve single agent tasks , such as E2C ( Watter et al. , 2015 ) , RCE ( Banijamali et al. , 2018 ) , PlaNet ( Hafner et al. , 2019 ) , SOLAR ( Zhang et al. , 2019b ) and so on . Lee et al . ( 2019 ) and Gregor et al . ( 2019 ) learn belief representations to accelerate modelfree agents . World Models ( Ha & Schmidhuber , 2018 ) learn latent dynamics in a two-stage process to evolve linear controllers in imagination . I2A ( Racanière et al. , 2017 ) hands imagined trajectories to a model-free policy based on a rollout encoder . In contrast to these works , our work considers multi-agent tasks and learns predictive communication by imagination in latent space . 3 PRELIMINARIES . In networked system control problem , we work with a networked system , which is described by a graph G ( V , E ) , where i ∈ V denotes the ith agent and ij ∈ E denotes the communication link between agents i and j . The corresponding networked ( cooperative ) multi-agent MDP is defined by a tuple ( G , { Si , Ai } i∈V , { Mij } ij∈E , p , { ri } i∈V ) . Si and Ai are the local state space and action space of agent i . Let S : = ∪i∈VSi and A : = ∪i∈VAi , the MDP transitions follow a stationary probability distribution p : S×A×S → [ 0 , 1 ] . The global reward is denoted by r : S×A → R and defined as r = 1|V| ∑ i∈V ri indicating that all local rewards are shared globally . The communication is limited to neighborhoods . M denotes the message space for the communication model . That is each agent i observes s̃i , t : = si , t ∪mNii , t , where si , t ∈ Si denotes local state space of agent i and mNii , t : = { mji , t } j∈Ni andNi : = { j ∈ V|ji ∈ E } . Message mji , t ∈Mji denotes all the available information at an agent ’ s neighbor . In NSC , the system is decentralized and the communication is limited to neighborhoods . Each agent i follows a decentralized policy πi : S̃i × Ai → [ 0 , 1 ] to choose its own action ai , t ∼ πi ( ·|s̃i , t ) at time t. The objective is to maximize Eπ [ R0 ] , where Rt = ∑∞ l=0 γ lrt+l and γ is a discount factor . 4 METHODOLOGY . Our goal is to learn predictive communication on a particular observation or environment state . We start by introducing the networked MDP with neighborhood communications and delayed information issue in communication . Then , we describe ImagComm that utilizes predictive communication , which we learn the agent ’ s world model to provide an additional context for communication . 4.1 DELAYED COMMUNICATION IN NETWORKED SYSTEM CONTROL . Following the setting of NSC in ( Chu et al. , 2020a ) , we assume that all messages sent from agent i are identical and we denote mij = mi , ∀j ∈ Ni . The message explicitly includes state s and policy π and agent belief h , i.e. , mi , t = si , t ∪ πi , t−1 ∪ hi , t−1 in communication . Note that πi , t−1 is the probability distribution over discrete actions . Thus for each agent in NSC , s̃i , t : = sVi , t ∪ πNi , t−1 ∪ hNi , t−1 . Note the communication phase is prior-decision , so only hi , t−1 and πi , t−1 are available . This protocol can be easily extended for multi-pass communication . We assume that any information that agent j knows at time t can be included in mji , t and mji , t = sj , t ∪ { mkj , t−1 } k∈Nj . Then s̃i , t : = si , t ∪ { sj , t+1−dij } j∈V/ { i } , which includes the delayed global observations . dij indicates the distance between i and j , i.e . the hops between two agents on the graph of the networked system . We illustrate the delayed information in Figure 1 . A more rigorous analysis of this conclusion can be found in the Appendix A . 4.2 PREDICTIVE COMMUNICATION . To reduce the delay of global information , we consider a forward model for predicting future states of each agent j , then sj , t+1 can be encoded as a message for communication , and agent i can benefit from this information . Let ŝi , t be the abstract state of ith agent , Wi ∈ Wi be a world model of the transition dynamics from ŝi , t to the abstract state ŝi , t+1 , and let bi , t : = ∪kτ=1ŝi , t+τ denote the predictive message . We aim to build a policy based on delayed global observations and predictive messages . The value of policy πi can be defined as V π , W i ( s ) based on the model Wi : V π , Wi ( s , aNi ) = Eai , t∼πi ( ·|s̃i , t , bi , t ) [ Rπi , t | s̃t = s , aNi , t = aNi ] . ( 1 ) Learning based on ( 1 ) has the benefit of reduced delay in global information compared to that without bi , t ; this is formally presented in Proposition 1 . Proofs are provided in Appendix A . Proposition 1 . ImagComm can reduce the delay of global information by incorporating a predictive model in the communication protocol . We are now interested in constructing an abstract model Ŵi ( · ; ϕ ) to approximateWi , which operates on an abstract state . Let ŝi , t+1 be the new abstract state sampled by ŝi , t+1 ∼ Ŵi ( ŝi , t ) . We want to minimize‖ŝi , t+1 − gi ( si , t+1 ; ψ ) ‖ , where gi ( · ; ψ ) is an embedding of raw states . Let V π , Ŵi be the value function of the policy on the estimated model Ŵi . Towards optimizing V π , W∗ i ( s , aNi ) , we build a lower bound as follows and maximize it iteratively : V π , W ∗ i ( s , aNi ) ≥ V π , Ŵi ( s , aNi ) −D ( Ŵ , π ) , ( 2 ) where D ( Ŵ , π ) ∈ R bounds the discrepancy between V π , W ? i and V π , Ŵ i . In practice , D ( Ŵi , πi ) is defined as Dπrefi ( Ŵi , πi ) = α · Es0 , ... , st , ∼πrefi [ ‖Ŵi ( ŝi , t ) − gi ( si , t+1 ) ‖ ] , ( 3 ) where α is a hyperparameter , πrefi is the policy used for sampling . For each agent , we solve the following problem : πk+1 , W k+1 = argmax π∈Π , W∈W V π , Wi −Dπki , δ ( W , π ) . ( 4 ) With the predictive imagination module , each agent utilizes the estimate of predictive state information to learn its belief and optimize the control performance of all other agents . Follow the analysis in ( Luo et al. , 2018 ) , we can show that ImagComm can lead to monotonic improvement in policy iteration . Proofs are defered to Appendix A . Proposition 2 . Suppose that W ∗i ∈ Wi is the optimal model and the optimization problem in equation ( 4 ) is solvable at each iteration . Solving ( 4 ) produces a sequence of policies π0i , . . . , π T i with monotonically increasing values : V π 0 , W∗ i ≤ V π 1 , W∗ i ≤ · · · ≤ V π T , W∗ i . A conclusion following directly from Proposition 2 is that solving ( 4 ) will converge to a local maximum . ImagComm considers build a world model and predict the farsighted state by a imagination module to eliminate the delay in global information and henceforth reduce the negative influence of the partial observability . Because the future information after time t compensate for some of the delayed information at time t. Next we will present the differentiable neural communication with imagination .

Summary: The paper provides an interesting way to add structure to MARL problems that have delay in the communication of state information. By explicitly building a predictive module for the future latent state of the agent and including that predicted state in the passed messages, it is possible that the agent will appropriately pass information that removes the effect of the delay in message passing across the network. They then apply this model to some interesting traffic light and cooperative vehicle control tasks.