Learning Predictive Communication by Imagination in Networked System Control
Authors: Yali Du
Yifan Zhao
Meng Fang
Jun Wang
Gangyan Xu
Haifeng Zhang
Venue:  (2021)
Keywords: Reinforcement Learning
Multi-agent Reinforcement Learning
Networked System Control

Abstract: Dealing with multi-agent control in networked systems is one of the biggest challenges in Reinforcement Learning (RL) and limited success has been presented compared to recent deep reinforcement learning in single-agent domain. However, obstacles remain in addressing the delayed global information where each agent learns a decentralized control policy based on local observations and messages from connected neighbors. This paper first considers delayed global information sharing by combining the delayed global information and latent imagination of farsighted states in differentiable communication. Our model allows an agent to imagine its future states and communicate that with its neighbors. The predictive message sent to the connected neighbors reduces the delay in global information.  On the tasks of networked multi-agent traffic control, experimental results show that our model helps stabilize the training of each local agent and outperforms existing algorithms for networked system control.

Under review as a conference paper at ICLR 2021
LEARNING PREDICTIVE COMMUNICATION BY IMAGI- NATION IN NETWORKED SYSTEM CONTROL
Anonymous authors Paper under double-blind review
ABSTRACT
Dealing with multi-agent control in networked systems is one of the biggest chal- lenges in Reinforcement Learning (RL) and limited success has been presented compared to recent deep reinforcement learning in single-agent domain. However, obstacles remain in addressing the delayed global information where each agent learns decentralized control policy based on local observations and messages from connected neighbors. This paper first considers delayed global information sharing by combining the delayed global information and latent imagination farsighted states in differentiable communication. Our model allows agent imagine its future states and communicate that with its neighbors. The predictive message sent to the connected neighbors reduces the delay in global information. On the tasks of networked multi-agent traffic control, experimental results show that our model helps stabilize the training of each local agent and outperforms existing algorithms for networked system control.
INTRODUCTION
Networked system control (NSC) is extensively studied and widely applied, including connected vehicle control (Jin & Orosz 2014). traffic signal control (Chu al.] (020b). distributed sensing (Xul al., 2016), networked storage operation (Qin al.) 2015) etc. In NSC, agents are connected via a communication network for a cooperative control objective. For example, in an adaptive traffic signal control system, each traffic light performs decentralized control based on its local observations and messages from connected neighbors. Although deep reinforcement learning has been successfully applied to some complex problems, such Go (Silver et al. 2016). and Starcraft II (Vinyals et al. (219). it is still not scalable in many real-world networked control problems. Multi- agent reinforcement learning (MARL) addresses the issue of scalability by performing decentralized control. Recent decentralized MARL performs decentralized control based on the assumptions of global observations and local or global rewards (Zhang et al., 2018 2019a Qu et al. 2019; 2020b a) which are reasonable in multi-agent gaming but not suitable NSC. A practical solution is allow each agent to perform decentralized control based on its local observations and messages from the connected neighbors. Various communication-base methods
proposed stabilize training and improve observability, and communication is studied to enable agents to behave as a group, rather than collection of individuals (Sukhbaatar & Fergus 2016 Chu al. 2020a).
Despite recent advances in neural communication (Sukhbaatar & Fergus 2016 Foerster et al. 2016 Chu et al.) 2020a)), delayed global information sharing remains open problem that widely exists in many NSC applications. Communication protocol not only reflects the situation at hand but also guides the policy optimization. Recent deep neural models (Sukhbaatar & Fergus 2016 Foerster Hoshen
et al., Hoshenn 2017) implement differentiable communication based on available connections. However, NSC, such as traffic signal control, each agent only connects to its neighbors, leading to a delay in receiving messages from the distant agents in the system, and the non-stationarity mainly comes from these partial observation (Chu et al, 2020a)). Communication with delayed global infor- mation limits the learnability of RL because RL agents can only use the delayed information and not leverage potential future information. Moreover, it is not efficient in situations where an environ- ment is sensitive when the behaviours of agents change. It is therefore of great practical relevance to develop algorithms which can learn beyond the communication with the delayed information sharing.

---

Under review as a conference paper at ICLR 2021
In this paper we introduce ImagComm that learns communication by imagination for multi-agent reinforcement learning in NSC. We leverage the model of the agent's 's world to provide an estimate of farsighted information in latent space for communication. At each time step, the agent is allowed to imagine its future states in an abstract space and convey this information to its neighbors. Therefore unlike previous works, our communication protocol conveys not only the current sharing informa- tion but also the imagined sharing information. It is applicable whenever communication changes frequently, e.g. at every time step agents may receive new communication information.
We summarize our main contributions as follows: (1) We first introduce the imagination module that can be used to learn latent dynamies for communication in networked multi-agent systems control. (2) We predict the future state of each local agent and allow each agent to convey the latent state to neighbors as messages, which reduce the delay of global information. (3) We demonstrate that leveraging the predictive communication by imagination in latent space succeeds in networked system control. We explore this model on a range of NSC tasks. Our results demonstrate that our method consistently outperform baselines on these tasks.
2 RELATED WORK
Networked system control (NSC) considers the problem where agents are connected via a communi- cation network for a cooperative control objective, such as autonomous vehicle control (Jin & Orosz, 2014), adaptive traffic signal control (Chu et al. 2020b), and distributed sensing (Xu et al. 2016). etc. Recently reinforcement learning has become popular for NSC through decentralized control and communications by networked agents.
Communication is an important part for multi-agent RL to compensate for the information loss in partial observations. Heuristic communication allows the agents to share some certain forms of information, such as policy fingerprints from other agents (Foerster et al. 2017] and averaged neighbor's 's policies (Yang et al. 2018). Recently end-to-end differentiable communications have become popular (Foerster et al.,
2016 Sukhbaatar & Fergus 2016 Chu et al.) 2020a) since the communication channel is learned to optimize the performance. Attention-based communication (Hoshen) 2017 Das et al. 2019 Singh al.) 2019) selectively send messages to the agents chosen, however, these are not suitable for NSC since the communication is allowed only between connected neighbors. Our method adopts differentiable communication with end-to-end training. Compared to existing works, we introduce a new predictive communication module through learning latent dynamics.
Learning latent dynamics has been studied to solve single agent tasks, such as E2C (Watter et al. 2015), RCE (Banijamali al. 2018). PlaNet (Hafner et 2019), SOLAR Zhang et al. (019b) and so on. Lee et al. (2019) and Gregor et al. (2019) learn belief representations to accelerate model- free agents. World Models (Ha Schmidhuber 2018) learn latent dynamics in a two-stage process to evolve linear controllers in imagination. 12A (Racanière et al. (017)) hands imagined trajectories to a model- free policy based on a rollout encoder. In contrast to these works, our work considers multi-agent tasks and learns predictive communication by imagination in latent space.
3 PRELIMINARIES
In networked system control problem, we work with a networked system, which is described by a graph G(V,E), where E V denotes the ith agent and ij E & denotes the communication link between agents i and j. The corresponding networked (cooperative) multi-agent MDP is defined by a tuple (G, {St, A, hiv, [Mighije P, (riliev). S; and A, are the local state space and action space of agent Let S := U and A UievA, the MDP transitions follow a stationary probability distribution
S x Ax + [0, 1]. The global reward is denoted by r S S A - R and defined as r 1 Σiev r; indicating that all local rewards are shared globally. The communication M is limited to neighborhoods. M denotes the message space for the communication model. That is each agent observes & == 81,1 U mAGLA, where 8;, E S₁ denotes local state space of agent and mAGi, := {m hee, and N == {j E Viji E E). Message mjj. E MM denotes all the available information at an agent's neighbor. In NSC, the system is decentralized and the communication is limited to neighborhoods. Each agent follows decentralized policy n; S, x A: - [0,
2

---

Under review as a conference paper at ICLR 2021
world model
mN.5
mNne
$i
(i-3)
(1-2)
(1-1)
()
(Agent)
(1-3)
(1-2)
(1-1)
(i)
t-2
t-1
Time
t-2
t-1
Delayed information sharing
Communication with imagination
Flow diagram of predictive communication. Left: An illustration networked system example is adaptive cruise control (ACC) system with siticleeto-vehicle (V2V) communication. For the information sharing, agent knows information at time / including &i. U - L U 2 U 84-3,1-2- which are the delayed global observations. For the communication, based world models b,, is learned from its neighbor's future states, which compensate for delayed information. Right: policy representation that includes messages of neighbors generated by the imagined module.
Figure I:
to choose its own action a, (.)5) at time t. The objective is to maximize Ex, [Ro], where R₂ Ei=o rhr and y is discount factor.
4 METHODOLOGY
Our goal is to learn predictive communication particular observation environment state. We start by introducing the networked MDP with neighborhood communications and delayed informa- tion issue communication. Then, describe ImagComm that utilizes predictive communication, which we learn the agent's 's world model to provide an additional context for communication.
4.1 DELAYED COMMUNICATION NETWORKED SYSTEM CONTROL
Following the setting of NSC in CChu al. 2020a), assume that all messages sent from agent are identical and we denote may = m, Vj Nr, The message explicitly includes state and policy and agent belief i.c., mi,t st, TT,1-1 U h in communication. Note that πT, the probability distribution over discrete actions. Thus for each agent in NSC, & sV, U TTA--1 U IN..-11-
Note the communication phase is prior-decision, so only h and Ti,-1 are available. This protocol can be easily extended for multi-pass communication. We assume that any information that agent j knows time & can be included in Mÿi, and Mjj.t 85., (IK, Then 8i, == $, U (*5,,411-4; bey J/e//1). which includes the delayed global observations. dy indicates the distance between and J, i.e. the hops between two agents on the graph of the networked system. We illustrate the delayed information in Figure A more rigorous analysis of this conclusion can be found in the Appendix A
4.2 PREDICTIVE COMMUNICATION
To reduce the delay of global information, we consider a forward model for predicting future states of each agent j, then 8j.++1 can be encoded message for communication, and agent benefit from this information. Let & be the abstract state of ith agent, W; W; be a world model of the transition dynamics from $1,; to the abstract state $1,4+1, and let by,e := 1 denote the predictive message. We aim build a policy based on delayed global observations and predictive messages. The value of policy n, can be defined as V..W based on the model W;:
V.n (s, aN,)
R
5 = s, ON = ax.].
(1)
Learning based on ① has the benefit of reduced delay in global information compared to that with- Proofs are provided Appendix A
out byg this is formally presented in Proposition] [
Proposition 1. Iomm can reduce the delay of global information by incorporating a predictive model in the communication protocol.
3

---

Under review as a conference paper at ICLR 2021
We are now interested in constructing abstract model ( (p) to approximate W,, which operates on an abstract state. Let $1,4+1 be the new abstract state sampled by 81,411 (&c,))). We want to minimize| |4,,11 - g (ii,4+1 Ill where g:(:;)) +)) is an embedding of raw states. Let V. be the value function of the policy on the estimated model W, Towards optimizing V. (8, (8,0))) we build lower bound as follows and maximize it iteratively:
V.n (8, aN.) V. (s, aN;) - D(W m)
(2)
where D(W, æ² E bounds the discrepancy between V.a. and V. In practice D(W;, m;) is defined as
D Dapor ( pe (B..)) g.
where is hyperparameter, गे is the policy used for sampling. For each agent, we solve the following problem:
Wk+1=
V.TWW D (W,
argmaa WWILWEE
With the predictive imagination module, each agent utilizes the estimate of predictive state informa- tion to learn its belief and optimize the control performance of all other agents. Follow the analysis in (Luo et al. 2018), we can show that ImagComm can lead to monotonic improvement in policy iteration. Proofs are defered Appendix
Suppose that W E Wi is the optimal model and the optimization problem in equation 4 is solvable each iteration. Solving (4) produces sequence of policies T. ......, n.
Proposition 2.
with monotonically increasing values: V. yaa. ,W"
W S ... SV, .
A conclusion following directly from Proposition 2 is that solving (4) will converge to a local max- imum. ImagComm considers build world model and predict the farsighted state by imagination module to eliminate the delay in global information and henceforth reduce the negative influence of the partial observability. Because the future information after time compensate for some of the delayed information at time f. Next we will present the differentiable neural communication with imagination.
4.3 DITERENTIABLE NEURAL COMMUNICATION
In our approach, an agent performs the following operations throughout the agent's 's life time: learn- ing the latent dynamics model from the dataset of past experience predict future states of itself, encoding the imagined features into message and learning differentiable communication models to- gether with the predictive information. Specifically, the predictive model lets us predict the states ahead in the latent space without having to observe.
Different with previous works that use hig = 9v, (h,,,1,,, 81,t: my.r), we propose to learn com- munication with imagination (ImagComm), as shown in Figure [ to add the imagined delayed information to communication:
l = gv, (ha 81,4, mAG.s bwrr)
where byi.. indicates the module of the predictive message, mvg, represents the standard com- munication module, which is the same as previous communication work. gv, is differentiable function to extract informaton for the agent's 's beliefs. For example, sv, can be LSTM (Hochreiter & Schmidhuber 1997).
Compared to (Chu et al., 2020a), ImagComm uses imagination module to provide an augmented ob- servation of agents and reduces delays global information. Compared to model-based approaches (Luo al. 2018 Janner al.) (019), the differences are two-fold: i) instead of learning model for a single-agent MDP, each agent learns a decentralized predictive model locally; ii) instead of aiming to improve the sampling efficiency, aim to augment the message for communication. thus reduc- ing the delay in global information from each agent's view; this can viewed as a combination of model-based and model-free aspects.
4

---

Under review as a conference paper at ICLR 2021
4.4 PRACTICAL IMPLEMENTATIONS
Simply we assume each agent is based on A2C (Advantage Actor-Critic) models. Let {To, hev and (Ww be the decentralized actor-critics, and {si,7) MMNi,r Qi,r; Ti,,, (Nss.)) (ir.:........ be the on- policy minibatch from networked MDPs under stationary policies {Te, J.vv For each agent with the belief hs, the actor and critic become TT, (h4) and Vu, (h,, dxx, for fitting the optimal policy nं and value function V ImagComm has three components based (5). m.N.t represents the message that will be passed to hig with myga = 8NG,a U TV.,t-1 1N,,.-1- For the predictive message bi,t,
Bi. = fi,,,, hhi.:,,. o.), and 81,+11 (. Let 2. = {dr, tha soi denote the parameters for abstract state embedding, raw state embedding, and the imagination module for agent i. Then each actor and critic are updated by losses
L (0:)
Σ TEB
(-log To, (as,7 (),-)) Al. + [ (as hs,;)],
(6a)
B
C (w:)
Σ TEB
(R Vas (IIG, aNG,))
(6b)
B
(().)= =
(. gi (++++++)]]]
(6c)
B
5B where R Σ r' /p - - + + 0i,s is the target action-value, Ui,r = Vis,- (5i., aN..))
the state-value as baseline, A,, R U/rr the advantage function critic, and B is the hyperparameter of the entropy loss. In implementation of Eq. (6c). adopt the root mean square loss (RMSE). The overall algorithm provided in Algorithm
Algorithm Imagination- policy optimization
1: Initialize policy )iV), predictive model (W;"ee empty mini-bater DB.
2: for k ==00 0, I,..., do
3: Collect a batch of data with (#f) in real environment:
DDs = [(si,r, mAGi,; Fi.r, bx..
4: Update policy {mh-1 jiev under imagination by solving (6a
5: Update critic (V*++1 JiEy under imagination by solving (6b)
7: end for " k+1 and Wh+ on dataset Dg by solving 66c
6: Update flet1
5 NUMERICAL EXPERIMENTS
We evaluate ImagComm on several challenging environments in networked system control and com- pare current state-of-the-ee algorithms for communication.
5.1 ENVIRONMENTS
We use four existing simulation environments: ATSC Grid, ATSC Monaco, Cooperative Adaptive Cruise Control (CACC) Catch-up and CACC Slow-down Chu et al. 2020b). The ATSC environ- ments are developed based SUMO (Krajzewicz et al. 2012). In ATSC, ATSC Grid simulates a 5 x 5 synthetic traffic grid. as shown in Figure 2a ATSC Monaco simulates a real-world 28- intersection traffic network from Monaco city, shown in Figure 2b In homogeneous scenario, ie. ATSC Grid, all agents have the same action space consisting of five pre-defined signal phases. While heterogeneous scenario, i.e. ATSC Monaco, agents have a variety of action spaces. For both scenarios, The objective of ATSC is to adaptively control traffic lights at the intersections to minimize traffic congestion based on real-time road-traffic measurements, Local state defined as St,i [wait,|1) [2], [ave Jee where / is each incoming lane of intersection i. Wait[-] measures the cumulative delay of the first vehicle and wave[-] measures the total number of ap- proaching vehicles along each incoming lane within 50m to the intersection. Rewards for each agent are defined as Ti. = - -,,eee,,,,, queuing vehicles approaching lane, which is measured by induction-loog detectors (ILD). In
sell)) where queue(-] denotes the number of
5

---

Under review as a conference paper at ICLR 2021
x12
x11
(c) CACC (Catch-up and Slow-down).
(a) ATSC Grid.
(b) ATSC Monaco.
Figure 2: Environments adaptive traffic signal control (ATSC) and cooperative adaptive cruise control (CACC) systems.
CACC, CACC Catch-up scenario simulates a string of 8 vehicles for 60s with a 0.1s control interval, where target speed oz 15m/s and initial headway h1, h4,0; Vi o 1. CACC Slow-down also simulates a string of 8 vehicles for 60s with a 0.1s control interval, where initial headway h,.0 = h" and target speed ot linearly decreases 15m/s during the first 30s and then stays constant. In both scenarios, the objective is to adaptively coordinate platoon of vehicles to minimize the car- following headway and speed perturbations based on real-time cchicle-to-vehiie Each vehicle observes and shares its headway h. velocity and acceleration to neighbors within one step. Models are trained to recommend appropriate hyper-parameters (a . for each OVM Rewards
communication.
controller(Bande et al. 1995). selected from four levels are designed as a cost function. Assuming the target headway and velocity profile are = 20m and of respectively, the cost of each agent is (fi,, h² + (v, of )² + 0.1u,, Whenever col- lision happens (hi,a < 1m), a large penalty of 1000 is assigned to each agent and the state becomes absorbing. An additional cost 5 (2hat ha,t) is provided in training for potential collisions.
5.2 BASELINES AND SETUP
All the baselines are implemented based on the A2C agent (Mnih et al. 2016) following the methods
in Eq. (6a) 6b). The baselines include one non-communicative policy IA2C (Mnih et 2016) and
three communicative policies NeurComm (Chu et al., 2020a). CommNet (Sukhbaatar Fergus
2016), DIAL (Foerster et al.] (016). The details will be provided in Appendix B
For ImagComm, we found that two-hop information already lead to com- pelling performance and we use one-step imagination. I.STM((,,,11 concat re):(TNN;--1); relu(hx,..-1), relu(by,.))), and & canhcconca(rrelu(y... h4,)-)). Then h4, is fed into two fully-connected neural networks producing policy and value separately. Note that for state encoding and gc(-), we use one fully- connected layer with tanh activation. For message extracting Iv., layers are set up with 64 hidden units. For the imagination module, we stack two fully-connected layers. Other settings include actor learning rate x 10-4, critie learning rate 2.5 10-4 entropy coefficient 3 = 0.01, batch size [8] = 120. Each method is trained over 1M steps under an actor-critic framework. In ATSC, entropy coefficient 3 0.01, batch size |B] 120 and in CACC, 3 0.05, batch size [B3 = 60. We use different random seed to initialize the environment without loss of generality. As different initial seeds may cause fluctuation, we smooth the learning curve using moving averages with a window size of 100 episodes following NeurCommíChu et al. 2020a).
We use hig
LSTM layer. All
5.3 TRAINING RESULTS
Figure 3 compares the learning curves of different methods on four environments, The results show that ImagComm overall performs better than others for these environments. In both ATSC envi- ronments, our model learns quickly with good stability and performs well the end. CACC, the standard deviation of episode returns is high due to the large penalty of collisions. In ATSC Grid, CommNet and DIAL gain little performance improvement even after 0.4M training steps, while ImagComm and NeurComm learn with faster speed and end with lower deviation. Imag-

---

Under review as a conference paper at ICLR 2021
10 dap
100 Uap
TeaKiig
(a) ATSC Grid.
(b) ATSC Monaco.
(c) CACC Catch-up.
(d) CACC Slow-down.
Figure 3: Learning curves for our method and different baselines on four environments
Uie came
100g sctee
(a) ATSC Grid.
(b) ATSC Monaco.
(c) CACC Catch-up.
(d) CACC Slow-down.
Figure 4: Learning curves for ImagComm with different a.
Comm learns slower at the initial state of training owing the learning of imagination module, but it quickly outperforms NeurComm after 0.5M steps, showing the benefits of incorporating the predictive communication. In ATSC Monaco, DIAL and NeurComm learn fast before 0.3M steps, but NeurComm does not improve much after that, and ImagComm outperforms all the baselines after 0.4M steps. In the CACC environments, ImagComm works better and more stable than others, with a better performance in the end. It shows that adding predictive communication is useful for these tasks. Though IA2C works well sometimes, it very unstable, which proves the effectiveness of communication. NeurComm and CommNet also work stable in both environments, but suffering from delayed communication such real-time scenarios, they perform worse than ImagComm.
We investigate the impact of coefficient a of D(W, π.) in our Imag- Comm by comparing the learning curves among (0.01, 0.1, 10, 100) on the ATSC and CACC environments in Figure 4 The results show that different values have different results. In ATSC environments, a = 1, 0.T work best for ATSC Grid and Monaco respectively. CACC Catch-up scenario, different values yield similar results. But in CACC Slow-down scenario, performs best.
Hyper parameter study
5.4 EXECUTION PERFORMANCE
Additionally, investigate the execution results of different methods. We more metrics, average queue length and average intersection delay, to get deeper understanding of controllers' exccution impact. Average intersection delay the mean of waiting time of all vehicles, which is another congestion metric.
Figure Sa and Figure compare average queue length of different trained models in one episode. As shown in the figures, ImagComm achieves the best performance in both scenarios. In ATSC Grid, IA2C and CommNet both fail reduce traffic pressure and the congestion level remains high in the end. In contrast, ImagComm helps agents communicate with each other effectively, so the queue length increases slower and remains low in the end.
Figure 5c| and Figure 5d compare average intersection delay of different trained models in one episode. In Figure 5c all four communicative policies outperform IA2C dramatically, which proves the effectiveness of communication. However, when it comes to ATSC Monaco, results turn out differently. In Figure 5d IA2C achieves the lowest average intersection delay while communicative policies perform worse. This be explained by the emphasis sider queue length in reward functions. Intersection delay is not explicitly included in rewards. Thus communicative models may tend to block vehicles in short queues. Although this inclination helps
the queue length
only

---

Under review as a conference paper at ICLR 2021
ATSE
Nomarn
ATISC Monaco
Carar Pe
DML
MIL
1333
(a) Averaged queue length (b) Averaged queue length (c) Averaged intersection (d) Averaged intersection in ATSC Grid.
in ATSC Monaco.
delay in ATSC Grid. delay in ATSC Monaco.
Figure 5: Performance on average queue length (a, b) and average intersection delay (c, d) in the ATSC setting.
Table 1: Average reward comparison over trained models.
Scenario Name
ImagComm
NeurComm
CommNet
DIAL
IA2C
ATSC Grid
-240.91
-302.68
-339.16
-354.52
-615.85
ATSC Monaco
-168.18
-318.90
-509.26
-202991
-385.71
CACC Catch-up
-196.60
-170.27
-364.16
-346.85
-449.27
CACC Slow-down
-1771.99
-2109.40
-1727.24
-2483.48
-1732.46
Table 2: Performance of MARL controllers in ATSC environments: synthetic traffic grid (top) and
Monaco traffic network (bottom). Best values are in bold.
Temporal Average Metrics
ImagComm
NeurComm
CommNet
DIAL
IA2C
avg queue length [veh]
0.88
1.17
1.47
1.64
3.84
avg vehicle speed [m/s]
3.65
3.43
3.15
2.89
0.73
trip delay [s]
194
215
225
262
950
avg queue length [veh] avg vehicle speed [m/s]
0.98
1.75
2.65
1.18
2.10
0.49
0.46
0.75
0.86
1.15
trip delay [s]
594
1445
719
566
479
reduce the queue length, average intersection delay becomes longer. This problem can be alleviated through adjusting the reward functions.
We freeze and evaluate our model for another 50 episodes using different seeds and present the average rewards in Table [ The results show that in ATSC scenarios, ImagComm outperforms other models by large margin. We further investigate queue length and vehicle speed in ATSC enviroments and present the results in Table In ATSC Grid, ImagComm achieves the lowest queue length and highest average vehicle speed, which means vehicles flow smoothly and ImagComm greatly reduces congestion level. In ATSC Monaco, ImagComm achieves the lowest queue length, but CommNet attains a higher average vehicle speed. This means ImagComm makes vehicles move steadily rather than getting blocked the road for long time, but with sacrifice of moving slowly.
CONCLUSIONS
In this work, we study the delayed communication problem for decentralized MARL in networked system control. We have introduced an imagination module to predict farsighted information for predictive communication. ImagComm combines the delayed global information and predictive state information and performs end-to-end training of the neural communication and imagination module to optimize the control performance in NSC. Extensive empirical studies demonstrate that by leveraging world models for learning the latent delayed information for communication, our method achieves the compelling performance gains in the challenging traffic signal control and adaptive cruise control tasks. We hope that our work will provide inspiration for the research in model-based learning for (networked) multi-agent systems.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Masako Bando, Katsuya Hasebe, Akihiro Nakayama, Akihiro Shibata, and Yuki Sugiyama. Dy- namical model of traffic congestion and numerical simulation. Physical Review E, 51(2):1035, 1995.
Ershad Banijamali, Rui Shu, Hung Bui, Ali Ghodsi, et al. Robust locally-linear controllable embed- ding. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 1751- 1759. PMLR. 2018.
Tianshu Chu, Sandeep Chinchali, and Sachin Katti. Multi-agent reinforcement learning for net- worked system control. In International Conference on Learning Representations (ICIR), 2020a. URL httph:ttp///t......tt pttttps://openreevieeee
m 1d=
Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li. Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions on Intelligent Transportation Systems, 21: 1086-1095, 2020b.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference Machine Learning (ICML), pp. 1538-1546, 2019.
Jakob Foerster, loannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2137-2145, 2016.
Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In In International Conference on Machine Learning (ICML), pp. 1146-1155, 2017.
Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aaron den Oord. Shaping belief states with generative environment models for rl. In Advances in Neural Information Processing Systems (NeurlPS), pp. 13475 13487, 2019.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint 1803. 103.2 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning (ICMI.), pp. 2555-2566 PMLR, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997.
Yedid Hoshen. Vain: Attentional multi-agent predictive modeling. Advances in Neural Informa- tion Processing Systems (NeurIPS), pp. 2701-2711, 2017.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), pp- 12519-12530, 2019.
I Ge Jin and Gábor Orosz. Dynamics of connected vehicle systems with delayed acceleration feed-
back. Transportation Research Part C: Emerging Technologies, 46:46 64, 2014.
Daniel Krajzewicz, Jakob Erdmann, Michael Behrisch, and Laura Bieker. Recent development and applications of sumo-simulation of urban mobility. International Journal Advances in Systems and Measurements, 5(3&4), 2012.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. arXiv preprint arXiv: 1ate::99000055550 2019.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-comper environments. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6379-6390, 2017.

---

Under review as a conference paper at ICLR 2021
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In Interna- tional Conference on Learning Representations (ICLR), 2018.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference Machine Learning (ICML), pp. 1928-1937, 2016.
Junjie Qin, Yinlam Chow, Jiyan Yang, and Ram Rajagopal. Distributed online modified greedy algorithm for networked storage operation under uncertainty. IEEE Transactions on Smart Grid, 7(2):1 1000:100-1110 106-1188, 2015.
Chao Qu, Shie Mannor, Huan Xu, Yuan Qi, Le Song, and Junwu Xiong. Value propagation for decentralized networked deep multi-agent reinforcement learning. In Advances in Neural Infor- mation Processing Systems (NeurIPS), pp. 1182-1191, 2019.
Guannan Qu, Yiheng Lin, Adam Wierman, and Na Li. Scalable multi-agent reinforcement learning for networked systems with average reward. arXiv preprint arXiv:2006. 06626, 2020a.
Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning of localized policies for multi-agent networked systems. Learning for Dynamics and Control, pp. 256-266. PMLR, 2020b.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. imagination-augmmnned agents for deep reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5690-5701, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, and Marc Lanctot. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484, 2016.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when communicate at scale in multiagent cooperative and competitive tasks. In International Conference on Learning Repre- sentations (ICLR), 2019.
Sainbayar Sukhbaatar and Rob Fergus. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2244-2252, 2016.
Oriol Vinyals, Igor Babuschkin, Wojeiech Czamnekki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3500-354 2019.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2746-2754, 2015.
Yong Xu, Renquan Lu, Peng Shi, Hongyi Li, and Shengli Xie. Finite-tinn distributed state estima- tion over sensor networks with round-robin protocol and fading channels. JEEE Transactions on Cybernetics, (511)3336-355 2016.
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. In International Conference on Machine Learning (ICML), pp- 5571-5580, 2018.
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi- agent reinforcement learning with networked agents. International Conference on Machine Learning (ICML), pp. 5872-5881, 2018
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Decentralized multi-agent reinforcement learning with networked agents: Recent advances. arXiv preprint arXiv:1912.0m0 038211, 2019a.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured representations for model- based reinforcement learning. In International Conference Machine Learning (ICLR), pp. 7453. PMLR, 2019b.
10

---

Under review as a conference paper at ICLR 2021
A PROOFS
Chu et al. 2020a) has shown that the communication protocol allows the local agent to utinize the delayed g global information. We cite the proof here stay self-contained.
Lemma 1. (Chu et al. 2020a) By communicating through h4,: gv, (,,, 84,1, mNG,), the de- layed global information uttlized to estimate each hidden state, that is
his C 81,0:t U {81,0211-ds "j,.ttdss) jEV\(1
where 5 y information y is utilized to estimate 2, and To:t { #1,..,,..
Proof. Based on definition communication protocol, ma,t C hig-t, and hist hig-l SV U TN,,1-1 mAi,t- Hence,
his 8i. (88a, T- Uhga-1))ev.
S.s (s U (3a-1 U Usuu- U Uh,, Sev
811-aa (5 -- seN, (85 - [vidy=a} (kga-2))
...
8s,0 T3 HeN, U 18 T3,01 )E (Vidg==)
T5a-
U...U U (5,00+11 dmnn J Amne je =dma)
which concludes the proof.
A.1 PROPOSITION
Proof of Proposition Based on (5). we have that
higt (mjak (bgie thze-13EEN, (ms,a-1)
(bb0-1) (e D...
(8)
(m., jelvld, =d) U de 3
Since mj,t = 83,0 1 h and رyt = 3 with k the step of forward imagination,
has 3 1 (VV U (8) je
Hence, 8,,er is included in the observation of agent at time dis - k 1, 8,,1 ahead of 81,=r at time T + diy 1 by steps.
A.2 PROPOSITION
Proof of Proposition We follow the sketch in (Luo al. 2018).
From 2 have that
V, V. 2 V. - -D (W, Tt+1)) By solving 4 V, V.
Dm (W Tk+1) V.TaWW'
Day (W*, TK) with Da, " = 0. Thus we have V. v proof.
2 a which completes the
B EXPERIMENT DETAILS
We describe the baselines used in experiments follows:
NeurComm (Chu et al. 2020a): it is one of the state-of-the-art methods for NSC. It for- mulates the neighborhood communication under spatiotemporal MDP and performs in- dependent learning for actors and critics. The messages include the state of current step, policy fingerprints and hidden state of last time step.
CommNet (Sukhbaatar & Fergus 2016); it allows agents to communicate through broad- casting a communication vector, which is the average of neighbors' hidden states. The messages include the current state and last step hidden state.
11

---

Under review as a conference paper at ICLR 2021
Imagination module
Plan on W
bx
Policy x,(a,, k:y))
Figure 6: Policy representation that includes messages of neighbors generated by the imagined module.
DIAL (Foerster et al. 2016): each agent encodes the received messages instead of aver- aging them, but still sums all encoded inputs. uses the observations of neighbors as the message.
IA2C (Mnih et al. 2016): it is an advantage actor-critic method that trains decentralized policies and critics for each agent. Each agent does not communicate with nearby agents. It is implemented similar to MADDPG Lowe et al. (217)) the critic takes neighboring actions.
ImagComm: our method adopts an imagination module to predict the delayed message and mitigate nonstationarity coming from partial observations in training. Fig. 6 illustrates the diagram of policy representation.
IA2C is non-communicative policy while the rest four approaches communicative policies requiring messages from the neighbors.
In terms of baseline, the communication is based hig gv, (hig 82,5 mAG,), and the algorithm implementation details are listed below: IA2C: hit = LSTM( relu(s..)))
NeurComm: hi,t = LSTM(H concat (elu((NN,._1) relu
DIAL: his
LSTM relu(sv,.e) + relu(relu( ((s,, + onehot(a LSTM( tanh(sy..)))
CommNet: h4.
linear (mea((hx..- In terms of ImagComm, the communication is based on (5) with h4 LSTM((,,-1+ concat
rel(hy,,. relu(bx..))). For the imagi- nation module, 81,4+1 lanh (relusy),)) hia-1)).
C ADDITIONAL RESULTS
Figure 7 shows the convergence of the learning curve for the imagination module. Different from the other three tasks, the ATSC Monaco tasked with heterogeneous agents which each agent's state has different dimensions. Thus in Figure 7b we plot the loss that is the summation of the square
error of N agents without average normalization by state dimensions. Figure 7al 7c y-axis is averaged RMSE loss for N agents.
and 7 the
Figure compares the performance of ImagComm and a modified version NeurComm-2HHo on CACC task. ImagComm performs one-step prediction to implicitly include the information of two-hop neighbors. NeurComm 2Hop directly uses two-hop information. is interesting to see that ImagComm outperforms NeurComm-22 module. Due to the one-step imagination before communication, the receiver obtains the information of the neighbors that are two-hop away and the next-step information of neighbors one-hop away.
which proves the effectiveness of the imagination
12

---

Under review as a conference paper at ICLR 2021
Traring stop
101 Traiiing Step
Tea stop
Traning 999
(a) ATSC Grid.
(b) ATSC Monaco.
(c) CACC Catch-up.
(d) CACC Slow-down.
Figure 7: Training curve the imagination module on ATSC (a, b) and CACC (c, d) setting.
1000
-1500
2000
-400
2500
6600
3300
-3000
-800
-0000
-1000
45500
ImegGComm
NeurEomm-27Hpp
ImagCoom
Neu-Comm-21ipp
-1200
0.1M 1M.0.2 0.3M 0.4M 0.5M 0.7M 0.7M 0.5M 1.0M
5500
1M 0.2M 0.3M
0-4M O. 6.6M 0.7M D.5M 1.0M Training step
Training step
(a) Averaged Reward of CACC Catch-up.
(b) Averaged Reward of CACC Slow-down.
Figure 8: Comparisons between ImagComm and NeurComm-2Hop ImagComm performs one-step prediction implicitly include the information of two-hop neighbors. NeurComm-2HHo directly uses two-hop information.
13


Reviews:
[{"review_id": "tZW8nNarvv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to communicate predicted local states between neighboring agents to address the problem of delayed information in networked multi-agent reinforcement learning. To enable agents to predict future states, a world model is learned at each agent. It is empirically demonstrated that the proposed method has good performance in traffic signal control and cooperative adaptive cruise control. \n+ The idea of communicating predicted local states is interesting and could mitigate the problem of delayed information.\n+ The proposed method is empirically verified in two scenarios and outperforms other communication methods in MARL.\n+ The tuning of hyperparameter (alpha) is also verified by experiments.\n\nConcerns:\n- The main concern about this paper is lack of novelty. It seems just applying the world model for single agent directly to MARL without any adaptation. Equations 2, 3, 4 are taken directly from (Luo at el, 2018) without any consideration of the characteristics of MARL. For example, why do these still hold in multi-agent case? Without rigorous mathematical analysis, it is hard to say it is valid. The proofs given in Appendix are not informative at all. \n- As in MARL, the state transition depends on the joint action of agents, why does equation 3 hold in practice?  It is just not clear. Any assumption is made for that? I am wondering how well the world model is learned in the experiments. \n- In the experiments, only one-step state prediction is used. It is claimed that two-hop information already leads to good performance. However, it is not supported by experiments. What if each agent directly uses two-hop information without prediction (it is easy to implement in simulation). This can serve as a baseline so as to illustrate how the learned model affects the performance empirically. \n-  The communication baselines, i.e., CommNet and DIAL, are pretty old. Why not use more recent methods such as ATOC, IC3Net, TarMAC, which perform much better than CommNet and DIAL.\n\n---\n**After rebuttal**\n\nThe authors' responses do not address my major concerns (the first two). I do not think the responses directly answer my questions.\nSo, I keep my score unchanged.   \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZI_xXkZa4W", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To reduce the delay of global information, the authors propose an imagination module to predict future information for communication. The agent predicts its future state and shares that with its neighbors. The information delay is an important problem in real-world applications. The proposed ImagComm outperforms other baselines on a range of NSC tasks.\n\nHowever, I have some concerns:\n\nWhat are the difference between predicting the future information at the sender end and the receiver end? Since the sender makes a prediction based on its history information, and the history information has been communicated to the receiver, the receiver could make the same prediction using its collected information, which could reduce the communication cost.\n\n$f_i$, $g_i$, and $W_i$ are trained by Eq 6c. The Eq 6c would make the outputs of $W_i$ and $g_i$ close to each other. Would it cause mode collapse, which means that the outputs of $W_i$ and $g_i$ become a constant representation?\n\nIn experiments, the ablation study that removes the optimization objective Eq 6c in Algorithm 1 could help to verify that the performance gain is caused by the predictive information rather than the additional information.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "D1NtKLi2tYd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n \nThe paper provides an interesting way to add structure to MARL problems that have delay in the communication of state information. By explicitly building a predictive module for the future latent state of the agent and including that predicted state in the passed messages, it is possible that the agent will appropriately pass information that removes the effect of the delay in message passing across the network. They then apply this model to some interesting traffic light and cooperative vehicle control tasks.\n##########################################################################\n\nReasons for score: \n \nOverall, I vote for rejecting. The idea is a nice contribution but the paper is quite confusingly written and the experiments do not adequately show the impact of the approach. In particular, the deviation between the method and the baseline does not appear to be particularly significant and the authors do not outline their hyperparameter search grid for both method and baselines which makes it quite difficult to tell whether a fair comparison was made. I really enjoy the applications and the idea but feel more careful experimentation is needed to demonstrate that the approach is valuable.\n ##########################################################################\n\nPros: \n \n1. The paper tackles a practical issue, namely that traffic control architectures are often necessarily decentralized and delayed. This is a good problem to study and a useful add on to existing work studying communication architectures in MARL settings.\n\n2. The idea of adding a predictive forwards model whose output is passed to other agents is a clever way of adding structure to the problem to get around delays. Technically, this does not add anything as an LSTM could internally learn to do this type of prediction as it constructs its messages but structure can often be useful.\n3. This partial model based approach may make the controller more robust to changing dynamics conditions or evolving agents.\n\n \n##########################################################################\n\nCons: \n\n1. While you tune the value of alpha for your method, it is not clear that you have also given equal amounts of hyperparameter optimization efforts to the baselines. This makes it unclear if your method is actually better or whether you have simply found better hyperparameters. \n2. It would be worth expanding on the details of your baselines in the appendix more. It’s not necessary and doesn’t affect my review, but it would make for a better paper! \n3. The authors suggest that the world model may “reduce the impacts of nonstationarity” but do not run any experiments that demonstrate this. Additionally, since the world model is implicitly a function of the other agents in the system, the opposite may be true and experiments are not run to investigate this possibility.\n\n \nA few ablations / changes that might lead to a more insightful paper but are not strictly necessary:\n(1) Many of the environments you describe are many-hop and many time-step delayed. It would be interesting to see how your results evolve with longer imagination horizons. \n(2) It would be useful for the reader to see the evolution of the world model accuracy over time, to see if it is in fact stationary or convergent. Without this graph, it’s unclear if you are even able to learn a good future predictive model. One way of demonstrating this (though by no means the only way) would be to learn a decoder on the hidden state and see if you can reconstruct the relevant world state variables.\n(3) Please add a hyperparameter grid to the appendix so we can see if fair comparison was made to the baselines. Since there do not appear to be existing benchmarks for the baseline methods on these tasks, it is probably necessary to give them equal amounts of tuning effort.\n\n#########################################################################\n\nThings that would improve readability:\n\n- What does it mean to share the policy with the agents around you? What is the representation that is shared?\n- The description of environments is a wall of text that is hard to read. Maybe describe the environments at a higher level and move detailed descriptions to the appendix?\n- Grammar: grammar throughout the paper is pretty inconsistent and there are too many grammar issues to point them out individually. It may be helpful to pass the paper through an automatic grammar checker and do several more proofreads.\n- There is a \\varphi in section 4.4 whose purpose I don’t understand; it does not seem to be used anywhere.\n- It would be useful to motivate the lower bound that appears in equation (2) \n- Including the architecture of the module for a 2-agent case directly in the body of the paper rather than in the appendix would probably make the notation a lot easier to read. As it was, I am still not 100% certain that I understood the notation correctly.\n- Figures are small and low-res; I can’t see them very well even by zooming in.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a technique of communicating predicted local states between agents in multi-agent reinforcement learning to deal with the delay in communication.  While the paper addresses an important practical problem, the reviewers have concerns about the insufficiency of novelty and experimental validation.
Decision: Reject