Year: 2020

Abstract:
Dealing with multi-agent control in networked systems is one of the biggest challenges in Reinforcement Learning (RL) and limited success has been presented compared to recent deep reinforcement learning in single-agent domain. However, obstacles remain in addressing the delayed global information where each agent learns a decentralized control policy based on local observations and messages from connected neighbors. This paper first considers delayed global information sharing by combining the delayed global information and latent imagination of farsighted states in differentiable communication. Our model allows an agent to imagine its future states and communicate that with its neighbors. The predictive message sent to the connected neighbors reduces the delay in global information. On the tasks of networked multi-agent traffic control, experimental results show that our model helps stabilize the training of each local agent and outperforms existing algorithms for networked system control.

heading: 1 INTRODUCTION
text: Networked system control (NSC) is extensively studied and widely applied, including connected vehicle control (Jin & Orosz, 2014), traffic signal control (Chu et al., 2020b), distributed sensing (Xu et al., 2016), networked storage operation (Qin et al., 2015) etc. In NSC, agents are connected via a communication network for a cooperative control objective. For example, in an adaptive traffic signal control system, each traffic light performs decentralized control based on its local observations and messages from connected neighbors. Although deep reinforcement learning has been successfully applied to some complex problems, such as Go (Silver et al., 2016), and Starcraft II (Vinyals et al., 2019), it is still not scalable in many real-world networked control problems. Multiagent reinforcement learning (MARL) addresses the issue of scalability by performing decentralized control. Recent decentralized MARL performs decentralized control based on the assumptions of global observations and local or global rewards (Zhang et al., 2018; 2019a; Qu et al., 2019; 2020b;a), which are reasonable in multi-agent gaming but not suitable in NSC. A practical solution is to allow each agent to perform decentralized control based on its local observations and messages from the connected neighbors. Various communication-based methods are proposed to stabilize training and improve observability, and communication is studied to enable agents to behave as a group, rather than a collection of individuals (Sukhbaatar & Fergus, 2016; Chu et al., 2020a).
Despite recent advances in neural communication (Sukhbaatar & Fergus, 2016; Foerster et al., 2016; Chu et al., 2020a), delayed global information sharing remains an open problem that widely exists in many NSC applications. Communication protocol not only reflects the situation at hand but also guides the policy optimization. Recent deep neural models (Sukhbaatar & Fergus, 2016; Foerster et al., 2016; Hoshen, 2017) implement differentiable communication based on available connections. However, in NSC, such as traffic signal control, each agent only connects to its neighbors, leading to a delay in receiving messages from the distant agents in the system, and the non-stationarity mainly comes from these partial observation (Chu et al., 2020a). Communication with delayed global information limits the learnability of RL because RL agents can only use the delayed information and not leverage potential future information. Moreover, it is not efficient in situations where an environment is sensitive when the behaviours of agents change. It is therefore of great practical relevance to develop algorithms which can learn beyond the communication with the delayed information sharing.
In this paper we introduce ImagComm that learns communication by imagination for multi-agent reinforcement learning in NSC. We leverage the model of the agent’s world to provide an estimate of farsighted information in latent space for communication. At each time step, the agent is allowed to imagine its future states in an abstract space and convey this information to its neighbors. Therefore unlike previous works, our communication protocol conveys not only the current sharing information but also the imagined sharing information. It is applicable whenever communication changes frequently, e.g. at every time step agents may receive new communication information.
We summarize our main contributions as follows: (1) We first introduce the imagination module that can be used to learn latent dynamics for communication in networked multi-agent systems control. (2) We predict the future state of each local agent and allow each agent to convey the latent state to neighbors as messages, which reduce the delay of global information. (3) We demonstrate that leveraging the predictive communication by imagination in latent space succeeds in networked system control. We explore this model on a range of NSC tasks. Our results demonstrate that our method consistently outperform baselines on these tasks.

heading: 2 RELATED WORK
text: Networked system control (NSC) considers the problem where agents are connected via a communication network for a cooperative control objective, such as autonomous vehicle control (Jin & Orosz, 2014), adaptive traffic signal control (Chu et al., 2020b), and distributed sensing (Xu et al., 2016), etc. Recently reinforcement learning has become popular for NSC through decentralized control and communications by networked agents.
Communication is an important part for multi-agent RL to compensate for the information loss in partial observations. Heuristic communication allows the agents to share some certain forms of information, such as policy fingerprints from other agents (Foerster et al., 2017) and averaged neighbor’s policies (Yang et al., 2018). Recently end-to-end differentiable communications have become popular (Foerster et al., 2016; Sukhbaatar & Fergus, 2016; Chu et al., 2020a) since the communication channel is learned to optimize the performance. Attention-based communication (Hoshen, 2017; Das et al., 2019; Singh et al., 2019) selectively send messages to the agents chosen, however, these are not suitable for NSC since the communication is allowed only between connected neighbors. Our method adopts differentiable communication with end-to-end training. Compared to existing works, we introduce a new predictive communication module through learning latent dynamics.
Learning latent dynamics has been studied to solve single agent tasks, such as E2C (Watter et al., 2015), RCE (Banijamali et al., 2018), PlaNet (Hafner et al., 2019), SOLAR (Zhang et al., 2019b) and so on. Lee et al. (2019) and Gregor et al. (2019) learn belief representations to accelerate modelfree agents. World Models (Ha & Schmidhuber, 2018) learn latent dynamics in a two-stage process to evolve linear controllers in imagination. I2A (Racanière et al., 2017) hands imagined trajectories to a model-free policy based on a rollout encoder. In contrast to these works, our work considers multi-agent tasks and learns predictive communication by imagination in latent space.

heading: 3 PRELIMINARIES
text: In networked system control problem, we work with a networked system, which is described by a graph G(V, E), where i ∈ V denotes the ith agent and ij ∈ E denotes the communication link between agents i and j. The corresponding networked (cooperative) multi-agent MDP is defined by a tuple (G, {Si,Ai}i∈V , {Mij}ij∈E , p, {ri}i∈V). Si and Ai are the local state space and action space of agent i. Let S := ∪i∈VSi and A := ∪i∈VAi, the MDP transitions follow a stationary probability distribution p : S×A×S → [0, 1]. The global reward is denoted by r : S×A → R and defined as r = 1|V| ∑ i∈V ri indicating that all local rewards are shared globally. The communication is limited to neighborhoods. M denotes the message space for the communication model. That is each agent i observes s̃i,t := si,t ∪mNii,t, where si,t ∈ Si denotes local state space of agent i and mNii,t := {mji,t}j∈Ni andNi := {j ∈ V|ji ∈ E}. Message mji,t ∈Mji denotes all the available information at an agent’s neighbor. In NSC, the system is decentralized and the communication is limited to neighborhoods. Each agent i follows a decentralized policy πi : S̃i × Ai → [0, 1]
to choose its own action ai,t ∼ πi(·|s̃i,t) at time t. The objective is to maximize Eπ[R0], where Rt = ∑∞ l=0 γ lrt+l and γ is a discount factor.

heading: 4 METHODOLOGY
text: Our goal is to learn predictive communication on a particular observation or environment state. We start by introducing the networked MDP with neighborhood communications and delayed information issue in communication. Then, we describe ImagComm that utilizes predictive communication, which we learn the agent’s world model to provide an additional context for communication.

heading: 4.1 DELAYED COMMUNICATION IN NETWORKED SYSTEM CONTROL
text: Following the setting of NSC in (Chu et al., 2020a), we assume that all messages sent from agent i are identical and we denote mij = mi,∀j ∈ Ni. The message explicitly includes state s and policy π and agent belief h, i.e., mi,t = si,t ∪ πi,t−1 ∪ hi,t−1 in communication. Note that πi,t−1 is the probability distribution over discrete actions. Thus for each agent in NSC, s̃i,t := sVi,t ∪ πNi,t−1 ∪ hNi,t−1.
Note the communication phase is prior-decision, so only hi,t−1 and πi,t−1 are available. This protocol can be easily extended for multi-pass communication. We assume that any information that agent j knows at time t can be included in mji,t and mji,t = sj,t ∪ {mkj,t−1}k∈Nj . Then s̃i,t := si,t ∪{sj,t+1−dij}j∈V/{i}, which includes the delayed global observations. dij indicates the distance between i and j, i.e. the hops between two agents on the graph of the networked system. We illustrate the delayed information in Figure 1. A more rigorous analysis of this conclusion can be found in the Appendix A.

heading: 4.2 PREDICTIVE COMMUNICATION
text: To reduce the delay of global information, we consider a forward model for predicting future states of each agent j, then sj,t+1 can be encoded as a message for communication, and agent i can benefit from this information. Let ŝi,t be the abstract state of ith agent, Wi ∈ Wi be a world model of the transition dynamics from ŝi,t to the abstract state ŝi,t+1, and let bi,t := ∪kτ=1ŝi,t+τ denote the predictive message. We aim to build a policy based on delayed global observations and predictive messages. The value of policy πi can be defined as V π,W i (s) based on the model Wi:
V π,Wi (s, aNi) = Eai,t∼πi(·|s̃i,t,bi,t) [ Rπi,t | s̃t = s, aNi,t = aNi ] . (1)
Learning based on (1) has the benefit of reduced delay in global information compared to that without bi,t; this is formally presented in Proposition 1. Proofs are provided in Appendix A.
Proposition 1. ImagComm can reduce the delay of global information by incorporating a predictive model in the communication protocol.
We are now interested in constructing an abstract model Ŵi(·;ϕ) to approximateWi, which operates on an abstract state. Let ŝi,t+1 be the new abstract state sampled by ŝi,t+1 ∼ Ŵi(ŝi,t). We want to minimize‖ŝi,t+1 − gi(si,t+1;ψ)‖, where gi(·;ψ) is an embedding of raw states. Let V π,Ŵi be the value function of the policy on the estimated model Ŵi. Towards optimizing V π,W∗
i (s, aNi), we build a lower bound as follows and maximize it iteratively:
V π,W ∗
i (s, aNi) ≥ V π,Ŵi (s, aNi)−D(Ŵ , π), (2)
where D(Ŵ , π) ∈ R bounds the discrepancy between V π,W ? i and V π,Ŵ i . In practice, D(Ŵi, πi) is defined as Dπrefi (Ŵi, πi) = α · Es0,...,st,∼πrefi [‖Ŵi(ŝi,t)− gi(si,t+1)‖], (3)
where α is a hyperparameter, πrefi is the policy used for sampling. For each agent, we solve the following problem:
πk+1,W k+1 = argmax π∈Π,W∈W V π,Wi −Dπki ,δ(W,π). (4)
With the predictive imagination module, each agent utilizes the estimate of predictive state information to learn its belief and optimize the control performance of all other agents. Follow the analysis in (Luo et al., 2018), we can show that ImagComm can lead to monotonic improvement in policy iteration. Proofs are defered to Appendix A.
Proposition 2. Suppose that W ∗i ∈ Wi is the optimal model and the optimization problem in equation (4) is solvable at each iteration. Solving (4) produces a sequence of policies π0i , . . . , π T i with monotonically increasing values: V π 0,W∗ i ≤ V π 1,W∗ i ≤ · · · ≤ V π T ,W∗ i .
A conclusion following directly from Proposition 2 is that solving (4) will converge to a local maximum. ImagComm considers build a world model and predict the farsighted state by a imagination module to eliminate the delay in global information and henceforth reduce the negative influence of the partial observability. Because the future information after time t compensate for some of the delayed information at time t. Next we will present the differentiable neural communication with imagination.

heading: 4.3 DIFFERENTIABLE NEURAL COMMUNICATION
text: In our approach, an agent performs the following operations throughout the agent’s life time: learning the latent dynamics model from the dataset of past experience to predict future states of itself, encoding the imagined features into message and learning differentiable communication models together with the predictive information. Specifically, the predictive model lets us predict the states ahead in the latent space without having to observe.
Different with previous works that use hi,t = gVi(hi,t−1, si,t,mNi,t), we propose to learn communication with imagination (ImagComm), as shown in Figure 1, to add the imagined delayed information to communication:
hi,t = gVi(hi,t−1, si,t,mNi,t, bNi,t) (5)
where bNi,t indicates the module of the predictive message, mNi,t represents the standard communication module, which is the same as previous communication work. gVi is a differentiable function to extract informaton for the agent’s beliefs. For example, gVi can be LSTM (Hochreiter & Schmidhuber, 1997).
Compared to (Chu et al., 2020a), ImagComm uses imagination module to provide an augmented observation of agents and reduces delays in global information. Compared to model-based approaches (Luo et al., 2018; Janner et al., 2019), the differences are two-fold: i) instead of learning a model for a single-agent MDP, each agent learns a decentralized predictive model locally; ii) instead of aiming to improve the sampling efficiency, we aim to augment the message for communication, thus reducing the delay in global information from each agent’s view; this can be viewed as a combination of model-based and model-free aspects.

heading: 4.4 PRACTICAL IMPLEMENTATIONS
text: Simply we assume each agent is based on A2C (Advantage Actor-Critic) models. Let {πθi}i∈V and{Vωi}i∈V be the decentralized actor-critics, and {(si,τ ,mNii,τ , ai,τ , ri,τ , bNii,τ )}i∈V,τ∈B be the onpolicy minibatch from networked MDPs under stationary policies {πθi}i∈V . For each agent with the belief hi,t, the actor and critic become πθi(hi) and Vωi(hi, aNi) for fitting the optimal policy π ∗ i and value function V πi . ImagComm has three components based on (5). mNi,t represents the message that will be passed to hi,t with mNi,t = sNi,t ∪ πNi,t−1 ∪ hNi,t−1. For the predictive message bi,t, ŝi,t = fi(sVi,t, hi,t−1;φi), and ŝi,t+1 = Ŵi(ŝi,t). Let Φi = {φi, ψi, ϕi} denote the parameters for abstract state embedding, raw state embedding, and the imagination module for agent i. Then each actor and critic are updated by losses
L (θi) = 1 |B| ∑ τ∈B (− log πθi (ai,τ | hi,τ ))A π i,τ + βH[πθi (ai | hi,τ )], (6a)
L (ωi) = 1 |B| ∑ τ∈B ( Rπi,τ − Vωi (hi,τ , aNi,τ ) )2 , (6b)
L(Φi) = α |B| ∑ τ∈B [ ‖Ŵi(fi(sVi,τ , hi,τ−1))− gi(si,τ+1)‖ ] , (6c)
where R̂πi,τ = ∑τB−1 τ ′=τ γ
τ ′−τri,τ ′ + γτB−τvi,τB is the target action-value, vi,τ = Vωi− (s̃i,τ , aNi,τ ) is the state-value as baseline, Aπi,τ = R π i,τ − vi,τ is the advantage function as critic, and β is the hyperparameter of the entropy loss. In implementation of Eq. (6c), we adopt the root mean square loss (RMSE). The overall algorithm is provided in Algorithm 1.
Algorithm 1 Imagination-based policy optimization
1: Initialize policy {π0i }i∈V , predictive model {Ŵ 0i }i∈V , empty mini-batch DB. 2: for k = 0, 1,..., T do 3: Collect a batch of data with {πki }i∈V in real environment: DB = {(si,τ ,mNii,τ , ai,τ , ri,τ , bNii,τ )}i∈V,τ∈B . 4: Update policy {πk+1i }i∈V under imagination by solving (6a). 5: Update critic {V k+1i }i∈V under imagination by solving (6b). 6: Update fk+1i , g k+1 i and Ŵ k+1 i on dataset DB by solving (6c). 7: end for

heading: 5 NUMERICAL EXPERIMENTS
text: We evaluate ImagComm on several challenging environments in networked system control and compare it to current state-of-the-art algorithms for communication.

heading: 5.1 ENVIRONMENTS
text: We use four existing simulation environments: ATSC Grid, ATSC Monaco, Cooperative Adaptive Cruise Control (CACC) Catch-up and CACC Slow-down (Chu et al., 2020b). The ATSC environments are developed based on SUMO (Krajzewicz et al., 2012). In ATSC, ATSC Grid simulates a 5 × 5 synthetic traffic grid , as shown in Figure 2a. ATSC Monaco simulates a real-world 28- intersection traffic network from Monaco city, as shown in Figure 2b. In homogeneous scenario, i.e. ATSC Grid, all agents have the same action space consisting of five pre-defined signal phases. While in heterogeneous scenario, i.e. ATSC Monaco, agents have a variety of action spaces. For both scenarios, The objective of ATSC is to adaptively control traffic lights at the intersections to minimize traffic congestion based on real-time road-traffic measurements. Local state is defined as st,i = {waitt[l],wavet[l]}ji∈E,l∈Lji , where l is each incoming lane of intersection i. Wait[·] measures the cumulative delay of the first vehicle and wave[·] measures the total number of approaching vehicles along each incoming lane within 50m to the intersection. Rewards for each agent are defined as ri,t = − ∑ ji∈E,l∈Lji ( queuet+∆t[l] ) , where queue[·] denotes the number of queuing vehicles on an approaching lane, which is measured by induction-loop detectors (ILD). In
Under review as a conference paper at ICLR 2021 Under review as a conference paper at ICLR 2020
200m
x1 x2 x3
x4
x5
x6
x7x8x9
x10
x11
x12
f1
f2
F1F2
(a) Synthetic traffic grid.
0 500 1000 1500 2000 2500 3000 3500 Simulation time (sec)
0
200
400
600
800
1000
Fl ow
ra te
(v eh
/h r)
f1 F1 f2 F2
(b) Traffic flows within the grid.
F1
F2
F3
F4
(c) Monaco traffic network.
Figure 2: ATSC scenarios for NMARL. (a) Synthetic traffic grid, with major and minor traffic flows shown in solid and dotted arrows. (b) Simulated time-variant traffic flows within the traffic grid. (c) Monaco traffic network, with traffic flow collections shown in colored arrows.
5.1.2 COOPERATIVE ADAPTIVE CRUISE CONTROL
The objective of CACC is to adaptively coordinate a platoon of vehicles to minimize the car-following headway and speed perturbations based on real-time vehicle-to-vehicle communication. Here we implement two CACC scenarios: “Catch-up” and “Slow-down”, with physical vehicle dynamics. To save space, the setup details are deferred to C.4.1.
5.2 ALGORITHM SETUP
For fair comparison, all MARL approaches are applied to A2C agents with learning methods in Eq. (3)(4), and only neighborhood observation and communication are allowed. IA2C performs independent learning, which is an A2C implementation of MADDPG (Lowe et al., 2017) as the critic takes neighboring actions (see Eq. (4)). ConseNet (Zhang et al., 2018) has the additional consensus update to overwrite parameters of each critic as the mean of those of all critics inside the closed neighborhood. FPrint (Foerster et al., 2017) includes neighbor policies. DIAL (Foerster et al., 2016) and CommNet (Sukhbaatar et al., 2016) are described in Section 4. IA2C, ConseNet, and FPrint are non-communicative policies since they utilize only neighborhood information. In contrast, DIAL, CommNet, and NeurComm are communicative policies. The implementation details are in C.1.
All algorithms use the same DNN hidden layers: one fully-connected layer for message encoding eλ, and one LSTM layer for message extracting gν . All hidden layers have 64 units. The encoding layer implicitly learns normalization across different input signal types. We train each model over 1M steps, with γ = 0.99, actor learning rate 5× 10−4, and critic learning rate 2.5× 10−4. Also, each training episode has a different seed for generalization purposes. In ATSC, β = 0.01, |B| = 120, while in CACC, β = 0.05, |B| = 60, to encourage the exploration of collision-free policies.
5.3 ABLATION STUDY
We perform ablation study in proposed scenarios, which are sorted as ATSC Monaco > ATSC Grid > CACC Slow-down > CACC Catch-up by task difficulty. ATSC is more challenging than CACC due to larger scale (>=25 vs 8), more complex dynamics (stochastic traffic flow vs deterministic vehicle dynamics), and longer control interval (5s vs 0.1s). ATSC Monaco > ATSC Grid due to more heterogenous network, while CACC Slow-down > CACC Catch-up due to more frequently changing leading vehicle profile. To visualize the learning performance, we plot the learning curve, that is, average episode return (R̄ = 1T ∑T−1 t=0 ∑ i∈V ri,t) vs training step. For better visualization, all learning curves are smoothened using moving average with a window size of 100 episodes.
First, we investigate the impact of spatial discount factor, by comparing the learning curves among α ∈ {0.8, 0.9, 1} for IA2C and CommNet. Fig. 3 reveals a few interesting facts. First, α∗CommNet is always higher than α∗IA2C. Indeed, α ∗ CommNet = 1 in almost all scenarios (except for ATSC Monaco). This is because communicative policies perform delayed global information sharing, whereas noncommunicative policies utilize neighborhood information only, causing difficulty to fit the global return. Second, learning performance becomes much more sensitive to α when the task is more
(a) ATSC Grid.
Under review as a conference paper at ICLR 2020
200m
x1 x2 x3
x4
x5
x6
x7x8x9
x10
x11
x12
f1
f2
F1F2
(a) Synthetic traffic grid.
0 500 1000 1500 2000 2500 3000 3500 Simulation time (sec)
0
200
400
600
800
1000
Fl ow
ra te
(v eh
/h r)
f1 F1 f2 F2
(b) Traffic flows within the grid.
F1
F2
F3
F4
(c) Monaco traffic network.
Figure 2: ATSC scenarios for NMARL. (a) Synthetic traffic grid, with major and minor traffic flows shown in solid and dotted arrows. (b) Simulated time-variant traffic flows within the traffic grid. (c) Monaco traffic network, with traffic fl w collections shown in colored arrows.
5.1.2 COOPERATIVE ADAPTIVE CRUISE CONTROL
The objective of CACC is to adaptively coordinate a platoon of vehicles to minimize the car-following headway and speed perturbations based on real-time vehicle-to-vehicle communication. Here we implement two CACC scenari s: “Catch-up” and “Slow-down”, with physical vehicle dynamics. To save space, the setup details are deferred to C.4.1.
5.2 ALGORITHM SETUP
For fair comparison, all MARL approaches are applied to A2C agents with learning methods in Eq. (3)(4), and only neighborhood observation and communication are allowed. IA2C performs independent learning, which is an A2C implementation of MADDPG (Lowe et al., 2017) as the critic takes neighboring actions (see Eq. (4)). ConseNet (Zhang et al., 2018) has the additional consensus update to overwrite parameters of each critic as the mean of those of all critics inside the closed neighborhood. FPrint (Foerster et al., 2017) includes neighbor policies. DIAL (Foerster et al., 2016) and CommNet (Sukhbaatar et al., 2016) are described in Sectio 4. IA2C, ConseNet, and FPrint are non-communicative policies sinc they ut lize only eighborhood information. In contrast, DIAL, CommNet, and NeurComm are communicative policies. The implementation details are in C.1.
All algorithms use the sam DNN hidden layers: one fully-conn cted layer for messag e coding eλ, and one LSTM layer for message extracting gν . All hidden layers have 64 units. The encoding layer implicitly learns normalization across different input signal types. We train each model over 1M steps, with γ = 0.99, actor learning rate 5× 10−4, and critic learning rate 2.5× 10−4. Also, each training episode has a different seed for generalization purposes. In ATSC, β = 0.01, |B| = 120, while in CACC, β = 0.05, |B| = 60, to encourage the exploration of collision-free policies.
5.3 ABLATION STUDY
We perform ablation study in proposed scenarios, which are sorted as ATSC Monaco > ATSC Grid > CACC Slow-down > CACC Catch-up by task difficulty. ATSC is more challenging than CACC due to larger scale (>=25 vs 8), more complex dynamics (stochastic traffic flow vs deterministic vehicle dynamics), and longer control interval (5s vs 0.1s). ATSC Monaco > ATSC Grid due to more heterogenous network, while CACC Slow-down > CACC Catch-up due to more frequently changing leading vehicle profile. To visualize the learning performance, we plot the learning curve, that is, average episode return (R̄ = 1T ∑T−1 t=0 ∑ i∈V ri,t) vs training step. For better visualization, all learning curves are smoothened using moving average with a window size of 100 episodes.
First, we investigate the impact of spatial discount factor, by comparing the learning curves among α ∈ {0.8, 0.9, 1} for IA2C and CommNet. Fig. 3 reveals a few interesting facts. First, α∗CommNet is always higher than α∗IA2C. Indeed, α ∗ CommNet = 1 in alm st all scenari s (except for ATSC Monaco). This is because communicative policies perform d layed global information sharing, whereas noncommunicative policies utilize neighborhood information only, causing difficulty to fit the global return. Second, learning performance becomes much more sensitive to α when the task is more
6
(b) ATSC Monaco.
V2V V2V V2VV2V
Speed, acceleration, headway Speed, acceleration, headway
(c) CACC (Catch-up and Slow-down).
Figure 2: Environments on adaptive traffic signal control (ATSC) and cooperative adaptive cruise control (CACC) systems.
CACC, CAC Catch-up scenario simulates a string of 8 vehicles for 60s with a 0.1s control interval, where target speed v∗t = 15m/s and initial headway h1,0 > hi,0,∀i 6= 1. CACC Slow-down also simulates a string of 8 vehicles for 60s with a 0.1s control interval, where initial headway hi,0 = h∗ and target speed v∗t linearly decreases to 15m/s during the first 30s and then stays at constant. In both scenarios, the objective is to adaptively coordinate a platoon of vehicles to minimize the carfollowing headway and speed perturbations based on real-time vehicle-to-vehicle communication. Each vehicle observes and shares its headway h, velocity v, and acceleration a to neighbors within one step. Models are trained to recommend appropriate hyper-parameters (α◦, β◦) for each OVM controller(Bando et al., 1995), selected from four levels {(0,0),(0.5,0),(0,0.5),(0.5,0.5)}. Rewards are designed as a cost function. Assuming the target headway and velocity profile are h∗ = 20m and v∗t , respectively, the cost of each agent is (hi,t− h∗)2 + (vi,t− v∗t )2 + 0.1u2i,t. Whenever a collision happens ( i,t < 1m), a large penalty of 1000 is a signed t each agent and the state becomes absorbing. An additional cost 5 (2hst − hi,t)2+ is provided in training for potential collisions.

heading: 5.2 BASELINES AND SETUP
text: All the baselines are implemented based on the A2C agent (Mnih et al., 2016) following the methods in Eq. (6a)(6b). The baselines include one non-communicative policy IA2C (Mnih et al., 2016), and three communicative policies NeurComm (Chu et al., 2020a), CommNet (Sukhbaatar & Fergus, 2016), DIAL (Foerster et al., 2016). The details will be provided in Appendix B.
For ImagComm, we found that two-hop information already lead to compelling performance and we use one-step imagination. We use hi,t = LSTM(hi,t−1, concat(relu(sVi,t), relu(πNi,t−1), relu(hNi,t−1), relu(bNi,t))), and ŝi,t+1 = tanh(concat(relu(sVi,t), hi,t−1)). Then hi,t is fed into two fully-connected neural networks producing policy and value separately. Note that for state encoding and gi(·), we use one fullyconnected layer with tanh activation. For message extracting gVi , we use a LSTM layer. All layers are set up with 64 hidden units. For the imagination module, we stack two fully-connected lay rs. Other settings include actor learni g rat 5× 10−4, criti learning rate 2.5× 10−4, entropy coefficient β = 0.01, batch size |B| = 120. Each method is trained over 1M steps under an actor-critic fra ework. In ATSC, entropy c efficient β = 0.01, batch size |B| = 120 and in CACC, β = 0.05, batch size |B| = 60. We use a different random seed to nitialize the v ronme t witho t loss of generality. As different initial seeds ma c use fluc uation, we smooth the learning curve using moving aver ges with a wi dow siz of 100 episode f llowing NeurC mm(Chu et al., 2020a).

heading: 5.3 TRAINING RESULTS
text: Figure 3 compares the learning curves of different methods on four environments.The results show that ImagComm overall performs better than others for these environments. In both ATSC environments, our model learns quickly with good stability and performs well in the end. In CAC , the standard deviation of episode returns is high due to the large penalty of collisions. In ATSC Grid, CommNet and DIAL gain little performance improvement even after 0.4M training steps, while ImagComm and NeurComm learn with a faster speed and end with lower deviation. Imag-
6
Comm learns slower at the initial state of training owing to the learning of imagination module, but it quickly outperforms NeurComm after 0.5M steps, showing the benefits of incorporating the predictive communication. In ATSC Monaco, DIAL and NeurComm learn fast before 0.3M steps, but NeurComm does not improve much after that, and ImagComm outperforms all the baselines after 0.4M steps. In the CACC environments, ImagComm works better and more stable than others, with a better performance in the end. It shows that adding predictive communication is useful for these tasks. Though IA2C works well sometimes, it is very unstable, which proves the effectiveness of communication. NeurComm and CommNet also work stable in both environments, but suffering from delayed communication in such real-time scenarios, they perform worse than ImagComm.
Hyper parameter study We investigate the impact of coefficient α of D(Ŵ , π) in our ImagComm by comparing the learning curves among {0.01, 0.1, 1, 10, 100} on the ATSC and CACC environments in Figure 4. The results show that different α values have different results. In ATSC environments, α = 1, 0.1 work best for ATSC Grid and Monaco respectively. In CACC Catch-up scenario, different α values yield similar results. But in CACC Slow-down scenario, α = 1 performs best.

heading: 5.4 EXECUTION PERFORMANCE
text: Additionally, we investigate the execution results of different methods. We use two more metrics, average queue length and average intersection delay, to get a deeper understanding of controllers’ execution impact. Average intersection delay is the mean of waiting time of all vehicles, which is another congestion metric.
Figure 5a and Figure 5b compare average queue length of different trained models in one episode. As shown in the figures, ImagComm achieves the best performance in both scenarios. In ATSC Grid, IA2C and CommNet both fail to reduce traffic pressure and the congestion level remains high in the end. In contrast, ImagComm helps agents communicate with each other effectively, so the queue length increases slower and remains low in the end.
Figure 5c and Figure 5d compare average intersection delay of different trained models in one episode. In Figure 5c, all four communicative policies outperform IA2C dramatically, which proves the effectiveness of communication. However, when it comes to ATSC Monaco, results turn out differently. In Figure 5d, IA2C achieves the lowest average intersection delay while communicative policies perform worse. This can be explained by the emphasis on the queue length as we only consider queue length in reward functions. Intersection delay is not explicitly included in rewards. Thus communicative models may tend to block vehicles in short queues. Although this inclination helps
reduce the queue length, average intersection delay becomes longer. This problem can be alleviated through adjusting the reward functions.
We freeze and evaluate our model for another 50 episodes using different seeds and present the average rewards in Table 1. The results show that in ATSC scenarios, ImagComm outperforms other models by a large margin. We further investigate queue length and vehicle speed in ATSC enviroments and present the results in Table 2. In ATSC Grid, ImagComm achieves the lowest queue length and highest average vehicle speed, which means vehicles flow smoothly and ImagComm greatly reduces congestion level. In ATSC Monaco, ImagComm achieves the lowest queue length, but CommNet attains a higher average vehicle speed. This means ImagComm makes vehicles move steadily rather than getting blocked on the road for a long time, but with a sacrifice of moving slowly.

heading: 6 CONCLUSIONS
text: In this work, we study the delayed communication problem for decentralized MARL in networked system control. We have introduced an imagination module to predict farsighted information for predictive communication. ImagComm combines the delayed global information and predictive state information and performs end-to-end training of the neural communication and imagination module to optimize the control performance in NSC. Extensive empirical studies demonstrate that by leveraging world models for learning the latent delayed information for communication, our method achieves the compelling performance gains in the challenging traffic signal control and adaptive cruise control tasks. We hope that our work will provide inspiration for the research in model-based learning for (networked) multi-agent systems.

heading: A PROOFS
text: Chu et al. (2020a) has shown that the communication protocol allows the local agent to utinize the delayed global information. We cite the proof here to stay self-contained. Lemma 1. (Chu et al., 2020a) By communicating through hi,t = gVi(hi,t−1, si,t,mNi,t), the delayed global information is utilized to estimate each hidden state, that is
hi,t ⊃ si,0:t ∪ { sj,0:t+1−dij , πj,0:t−dij } j∈V\{i}
where x ⊃ y if information y is utilized to estimate x, and x0:t := {x0, x1, . . . , xt}.
Proof. Based on the definition of communication protocol, mi,t ⊃ hi,t−1, and hi,t ⊃ hi,t−1 ∪ sVi,t ∪ πNi,t−1 ∪mNi,t. Hence, hi,t ⊃ si,t ∪ {sj,t, πj,t−1}j∈Ni ∪ {hj,t−1}j∈Vi ⊃ si,t ∪ {sj,t, πj,t−1}j∈Ni ∪ { sj,t−1 ∪ {sk,t−1, πk,t−2}k∈Nj ∪ {hk,t−2}k∈Vj } j∈Vi
= si,t−1:t ∪ {sj,t−1:t, πj,t−2:t−1}j∈Ni ∪ {sj,t−1, πj,t−2}j∈{V|dij=2} ∪ {hj,t−2}j∈{V|dij≤2} ⊃ · · · ⊃ si,0:t ∪ {sj,0:t, πj,t−2:t−1}j∈Ni ∪ {sj,0:t−1, πj,0:t−2}j∈{V |dij=2}
∪ . . . ∪ {sj,0:t+1−dmax , πj,0:t−dmax}j∈{V|dij=dmax} ,
(7)
which concludes the proof.
A.1 PROPOSITION 1
Proof of Proposition 1. Based on (5), we have that
hi,t ⊃ {mj,t}j∈Ni ∪ {bj,t}j∈Ni ⊃ {hj,t−1}j∈Ni ⊃ {mj,t−1}j∈{V|dij=2} ∪ {bj,t−1}j∈{V|dij=2} ⊃ · · · ⊃ {mj,t+1−d}j∈{V|dij=d} ∪ {bj,t+1−d}j∈{V|dij=d} ⊃ · · ·
(8)
Since mj,t = sj,t ∪ πj,t−1 ∪ hj,t−1 and bj,t = ∪kτ=1ŝj,t+τ with k the step of forward imagination, hi,t ⊃ {sj,t+1−d}j∈{V|dij=d} ∪ {ŝj,t+k+1−d}j∈{V|dij=d} . (9)
Hence, ŝi,t=τ is included in the observation of agent j at time τ + dij − k − 1, si,t=τ , ahead of si,t=τ at time τ + dij − 1 by k steps.
A.2 PROPOSITION 2
Proof of Proposition 2. We follow the sketch in (Luo et al., 2018). From (2), we have that
V πk+1,W
∗
i ≥ V πk+1,Ŵ i −Dπk(Ŵ , πk+1). By solving (4), V πk+1,Ŵ i −Dπk(Ŵ , πk+1) ≥ V πk,W
∗
i − Dπk(W ∗, πk) with Dπk(W ∗, πk) = 0. Thus we have V πk+1,W ∗ i ≥ V πk,W ∗
i which completes the proof.

heading: B EXPERIMENT DETAILS
text: We describe the baselines used in our experiments as follows:
• NeurComm (Chu et al., 2020a): it is one of the state-of-the-art methods for NSC. It formulates the neighborhood communication under a spatiotemporal MDP and performs independent learning for actors and critics. The messages include the state of current step, policy fingerprints and hidden state of last time step.
• CommNet (Sukhbaatar & Fergus, 2016): it allows agents to communicate through broadcasting a communication vector, which is the average of neighbors’ hidden states. The messages include the current state and last step hidden state.
• DIAL (Foerster et al., 2016): each agent encodes the received messages instead of averaging them, but still sums all encoded inputs. It uses the observations of neighbors as the message.
• IA2C (Mnih et al., 2016): it is an advantage actor-critic method that trains decentralized policies and critics for each agent. Each agent does not communicate with nearby agents. It is implemented similar to MADDPG Lowe et al. (2017) as the critic takes neighboring actions.
• ImagComm: our method adopts an imagination module to predict the delayed message and mitigate nonstationarity coming from partial observations in training. Fig. 6 illustrates the diagram of policy representation.
IA2C is a non-communicative policy while the rest four approaches are communicative policies requiring messages from the neighbors.
In terms of baseline, the communication is based on hi,t = gVi(hi,t−1, si,t,mNi,t), and the algorithm implementation details are listed below: IA2C: hi,t = LSTM(hi,t−1, relu(si,t)). NeurComm: hi,t = LSTM(hi,t−1, concat(relu(sVi,t), relu(πNi,t−1), relu(hNi,t−1))). DIAL: hi,t = LSTM(hi,t−1, relu(sVt,t) + relu(relu((hi,t−1))) + onehot(ai,t−1). CommNet: hi,t = LSTM(hi,t−1, tanh(sVi,t) + linear(mean(hNi,t−1))). In terms of ImagComm, the communication is based on (5) with hi,t = LSTM(hi,t−1, concat(relu(sVi,t), relu(πNi,t−1), relu(hNi,t−1), relu(bNi,t))). For the imagination module, ŝi,t+1 = tanh(concat(relu(sVi,t), hi,t−1)).

heading: C ADDITIONAL RESULTS
text: Figure 7 shows the convergence of the learning curve for the imagination module. Different from the other three tasks, the ATSC Monaco is tasked with heterogeneous agents in which each agent’s state has different dimensions. Thus in Figure 7b we plot the loss that is the summation of the square error of N agents without average normalization by state dimensions. In Figure 7a, 7c and 7d, the y-axis is averaged RMSE loss for N agents.
Figure 8 compares the performance of ImagComm and a modified version of NeurComm-2Hop on CACC task. ImagComm performs one-step prediction to implicitly include the information of two-hop neighbors. NeurComm-2Hop directly uses two-hop information. It is interesting to see that ImagComm outperforms NeurComm-2hop, which proves the effectiveness of the imagination module. Due to the one-step imagination before communication, the receiver obtains the information of the neighbors that are two-hop away and the next-step information of neighbors one-hop away.

Review summaries:
The paper proposes to communicate predicted local states between neighboring agents to address the problem of delayed information in networked multi-agent reinforcement learning. To enable agents to predict future states, a world model is learned at each agent. It is empirically demonstrated that the proposed method has good performance in traffic signal control and cooperative adaptive cruise control. 

The paper provides an interesting way to add structure to MARL problems that have delay in the communication of state information. By explicitly building a predictive module for the future latent state of the agent and including that predicted state in the passed messages, it is possible that the agent will appropriately pass information that removes the effect of the delay in message passing across the network. They then apply this model to some interesting traffic light and cooperative vehicle control tasks.