Multi-agent Reinforcement Learning for Networked System Control
Authors: Tianshu Chu
Sandeep Chinchali
Sachin Katti
Venue:  (2020)
Keywords: deep reinforcement learning
multi-agent reinforcement learning
decision and control

Abstract: This paper considers multi-agent reinforcement learning (MARL) in networked system control. Specifically, each agent learns a decentralized control policy based on local observations and messages from connected neighbors. We formulate such a networked MARL (NMARL) problem as a spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent. Further, we propose a new differentiable communication protocol, called NeurComm, to reduce information loss and non-stationarity in NMARL. Based on experiments in realistic NMARL scenarios of adaptive traffic signal control and cooperative adaptive cruise control, an appropriate spatial discount factor effectively enhances the learning curves of non-communicative MARL algorithms, while NeurComm outperforms existing communication protocols in both learning efficiency and control performance.

Published as a conference paper at ICLR 2020
MULTI-AGENT REINFORCEMENT LEARNING FOR NETWORKED SYSTEM CONTROL
Tianshu Chu
Sandeep Chinchali & Sachin Katti Stanford University Stanford, CA 94305, USA
Uhana Inc.
Palo Alto, CA 94304, USA cts maa com
(csandeep, s skatti j @stanford.eduu
ABSTRACT
This paper considers multi-agent reinforcement learning (MARL) in networked system control. Specifically, each agent learns a decentralized control policy based on local observations and messages from connected neighbors. We formulate such networked MARL (NMARL) problem as spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent. Further, we propose a new differentiable communication protocol, called NeurComm, to reduce information loss and non-stationarity in NMARL. Based on experiments realistic NMARL scenarios of adaptive traffic signal control and cooperative adaptive cruise control, appropriate spatial discount factor effectively enhances the learning curves of non-communicative MARL algorithms, while NeurComm outperforms existing communication protocols in both learning efficiency and control performance.
INTRODUCTION
Reinforcement learning (RL), formulated Markov decision process (MDP), is promising data-driven approach for learning adaptive control policies (Sutton & Barto 1998). Recent advances in deep neural networks (DNNs) further enhance its learning capacity on complex tasks. Successful algorithms include deep Q-network (DQN) (Mnih et al. 2015), deep deterministic policy gradient (DDPG) (Lillicrap al. 2015), and advantage actor critic (A2C) (Mnih et al.) 2016). However, RI. not scalable in many real-world control problems. This scalability issue is addressed in multi-agent RL (MARL), where each agent learns its individual policy from only local observations. However, MARL introduces new challenges in model training and execution, due non-stationarity and partial observability in a decentralized MDP from the viewpoint of each agent. To address these challenges, various learning methods and communication protocols are proposed to stabilize training and improve observability.
This paper considers networked MARL (NMARL) in the context of networked system control (NSC), where agents are connected via a communication network for a cooperative control objective. Each agent performs decentralized control based on its local observations and messages from connected neighbors. NSC is extensively studied and widely applied. Examples include connected vehicle control (Jin & Orosz 2014), traffic signal control (Chu al. 2019), distributed sensing (Xu et al. 2018), and networked storage operation (Qin et al. 2016). We expect an increasing trend of NMARL based controllers in the near future, after the development of advanced communication technologies such as 5G and Internetof--Thine
Recent works studied decentralized NMARL under assumptions of global observations and local rewards (Zhang et al.) 2018) Qu et al., 2019), which are reasonable in multi-agent gaming but not suitable in NSC. First, the control infrastructures are distributed in a wide region, so collecting global observations in execution increases communication delay and failure rate, and hurts the robustness. Second, online learning is not common due to safety and efficiency concerns. Rather, each model is trained offline and tested extensively before field deployment. In online execution, the model only runs forward propagation, and its performance is constantly monitored for triggering re- re-training. To reflect these practical constraints in NSC, we assume 1) each agent is connected limited number

---

Published as a conference paper at ICLR 2020
of neighbors and communication is restricted to its neighborhood, and 2) training is offline and global information available in rollout training minibatches, despite a decentralized training process.
The contributions of this paper are three-fold. First, we formulate NMARL under the aforementioned NSC assumptions as a decentralized spatiotemporal MDP, and introduce a spatial discount factor to stabilize training, especially for non-communicative algorithms. Second, we propose new neural communication protocol, called NeurComm, to adaptively share information on both system states and agent behaviors. Third, we design and simulate realistic NMARL environments evaluate and compare our approaches against recent MARL baselines.
RELATED WORK
MARL works can be classified into four groups based on their communication methods. The first group is non-communicative and focuses on stabilizing training with advanced value estimation methods. In MADDPG, each action-value is estimated by centralized critic based on global observations and actions (or inferred actions) (Lowe et al.) (017). COMA extends the same idea to A2C and estimates each advantage using centralized critic and counterfactual baseline (Foerster et al. 2018). In Dec-HDRQQ (Omidshafiei et al., 2017) and PS-TRPO (Gupta et al.) 2017). ) the centralized critic takes local observations, but the parameters shared globally. In the NMARL work of Zhang et al. (2018), the critic is fully decentralized but each takes global observations and performs consensus updates. In this paper, we empirically confirm that spatial discount factor helps stabilize the training of non-communicative algorithms under neighborhood observation.
The second group considers heuristic communication protocols or direct information sharing. Foerster et al. (2117) shows performance gains with directly-shared low dimensional policy fingerprints from other agents. Similarly, mean field MARI takes the average of neighbor policies for informed action-value estimation (Yang et al.] 2018). The major disadvantage of this group is that, unlike NeurComm, the communication not explicitly designed for performance optimization, which may cause inefficient and redundant communications in execution.
The third group proposes learnable communication protocols. In DIAL, the message is generated together with action-value estimation by each DQN agent, then it is encoded and summed with other input signals at the receiver side (Foerster et al. 2016). CommNet is a more general communication protocol, but it calculates the mean of all messages instead of encoding them (Sukhbaatar et al.) 2016). Both works, especially CommNet, incur an information loss due to aggregation on input signals. Another collection of works focuses on communications in strategy games. In BiCNet (Peng et al.. 2017), bi-directional RNN is used to enable flat communication among agents, while in Master- Slave (Kong et al.) 2017). two-way message passing is utilized in a hierarchical RNN architecture of master and slave agents. In contrast to existing protocols, NeurComm 1) encodes and concatenates signals, instead of aggregating them, minimize information loss, and 2) includes policy fingerprints in communication to reduce son-stationarity.
The fourth group focuses on communication attentions to selectively send messages. ATOC (Jang Lu 2018) learns soft attention which allocates a communication probability to each other agent, while IC3Net (Singh et al., 2018) learns a hard binary attention which decides communicating or not. These works especially useful when each agent has to prioritize the communication targets. NMARL is less likely the case since the communication range is restricted to small neighborhoods.
SPATIOTEMPORAL RL
This section formulates the NMARL problem as a decentralized spatiotemporal MDP, and introduces the spatial discount factor to reduce its learning difficulty. To simplify the notation, we assume the true system state is observable, and use "state" and "observation" interchangeably. This does not affect the validity of proposed methods in practice. To save space, all proofs are deferred A
CCode link: hitm::/pppr//tttoobo.ooo////////
2

---

Published as a conference paper at ICLR 2020
3.1 NETWORKED MARL
The networked system is represented by a graph G(V, E) where i E V is each agent and ij E & is each communication link. The corresponding MDP is characterized as (G, Ai)ee p, where S, and A, are the local state space and action space of agent i. Let S == XigvS; and A = x xiv,A be the global state space and action space, MDP transitions follow a stationary probability distribution p: S x A x & + [0, 1], and global step rewards be denoted by r S x A + R. In multi-agent MDP, each agent follows a decentralized policy T; S; x A, - [o. 1] to chose its own action ai, ~ Tú( (- at time I. The MDP objective is to maximize E[ where R& = =-= 3-= is the long-term global return with discount factor 7- Here the expectation is taken over the global policy A - [0, 1]. the initial distribution 9, p. and the transition 87+1 step reward r, r(s,, a, ). VT T. and the terminal reward rr rr (ST) | The same system can be formulated as a centralized MDP. Defining (=(s) E 8, s] as the state-value function and (*(saa) = E R |s 8, a = al as the action-value function, we have E Rg] = E.Es ((s) p(s V (*())= - EneA s( (s, a). and the advantage function A (s a) (= (T()).
pl- [r. a, regarding the
MARL provides a scalable solution for controlling networked systems, but it introduces partial observability and con-stationarity in decentralized MDP of each agent, leading to inefficient and unstable learning performance. To see this, note $i,, E S; C S does not provide sufficient information for π.. Even assuming Si, St, the transition p. ( St,,, as,t) Ea_.edd (==(o_i,(ss)
p(8++1 18t, di.t: (.) is non-stationary if the behavior policies of other agents - := (mz Dev evolving over time. In this paper, enforce practical constraints and only allow local observations and neighborhood communications, which makes MARL even more challenging.
Definition 3.1 (Networked Multi-agent MDP with Neighborhood Communication). cooperative multi-agent MDP (G, {St, A Aiviv, (Mij Jijee, P, {ri hev) with the message space M, Liev Ti. All local rewards are shared globally, whereas the
networked
the global reward is defined communication is limited to neighborhoods, that is, each agent observes S = 81, U mAçi, Here N₁- = [j E Viji E E). myA. := emns,e)ENN and each message mjit E MM is derived from all the available information that neighbor.
3.2 SPATIOTEMPORAL RL
Definition 3.2 (Spatiotemporal MDP). We assume local transitions are independent of other agents given the neighboring agents, that is,
Pa 1 ai.))
Σ
II
(5,(s,1)5)))) p(8..t1 118.t118yy,, 0i,t; (NG,));
ON,, JEN.
NG {i} is the closed neighborhood, and p is abused to denote any stationary transition. Then from the viewpoint of each agent Definition 3.J is equivalent to a decentralized spatiotemporal
where :
MDP, characterized as (S,, A., [MyleeN. Pt, (rihiv)) by optimizing the discounted return
T
R
Σ Aarz
(2)
JEV
where 0 S a < 1 is the spatial discount factor, and di distance between agents i and j.
The major assumption in Definition 3. is that the Markovian property holds both temporally and spatially, so that the next local state depends on the neighborhood states and policies only. This assumption is valid in most networked control systems such as traffic and wireless networks, as well as the power grid, where the impact of each agent is spread over the entire system via controlled flows, or chained local transitions. Note in NSC, each agent is connected to limited number of neighbors (the degree of G is low). So spatiotemporal MDP is decentralized during model execution, and it naturally extends properties of MDP. To reduce the learning difficulty of spatiotemporal MDP, a spatiotemporally discounted return is introduced in Eq, (2) to scale down reward signals further away (which are more difficult to fit using local information). When +
In infinite MDP, (s(s) = -B DES
3

---

Published as a conference paper at ICLR 2020
each agent performs local greedy control; when - 1, each agent performs global coordination and R = R.. Vi E V. Further, we have Q7 (s, a) = Q (s, ay,) = , = 8, ay,, = dy.]. 8;) (s, av.).. since the immediate local reward of
and (8, = V ax.) = Ea Ti(a; each agent is only affected by controls within its closed neighborhood.
Now we assume each agent is A2C, with parametric models Te, (5:) and V2, (3;. aNs) for fitting the optimal policy गे and value function V*, Note if 8, is able to provide global information through cascaded neighborhood communications, both To, and Va, are able to fit return Re,,-- Also, global and future information, such Re, and (N..r, are always available from each rollout minibatch in offline training. In contrast, only local information št, is allowed in online execution of policy T
Proposition 3.1 (Spatiotemporal RL with A2C). Let hev and (Ve be the decentralized actor-critics, and {(s.,, m,, Tt,r.Ti,r) (ue,,, FE be the on on-policy minibatch from spatiotemporal MDPs under stationary policies { Nev. Then each actor and critic are updated by losses
((0.)
1
Σ
log To, (ai, (a,,, +
Σ o, o.E.A
no, (a:(5,)))))) log no, (a: (ar1k(.))
(3)
[
FEB
(Wwi) =
1
feg - Vu, aN..))
(4)
B
TEB
where Ar, R. Usr is the estimated advantage, R Σt' x Exevatury
y TTBTTUi is the sampled action-valus Uj.r = Va, ($i,r: ONG,7) is the estimated state-value, and is the coefficient of the entropy loss.
Usrra
4 SPATIOTEMPORAL RL WITH NEURAL COMMUNICATION
For efficient and adaptive information sharing, we propose new communication protocol called NeurComm. To simplify the notation, we assume all messages sent from agent i are identical, i.e., Maj m,, Vj E NN Then
hi = 9v. (hr, (SV,,.)) exp (TN,, Exp (hw).--1)),
(5)
where h is the hidden state (or the belief)) of each agent and ex, and Iv, are differentiable message encoding and extracting functions To avoid dilution of state and policy information (the former is for improving observability while the later is for reducing non-stationarity) state and policy are explicitly included in the message besides agent belief, i.e., mi,t = 8i, U T U h4,t-1, or 5r, := sy, "y,,U Uhy, 1-1 as in Eq. (5). Note the communication phase is prior-decision, so only hirr- and TT,--1 are available. This protocol can be easily extended for multi-pass communication: exp (N) ((=-1))),
)). where n hs,t-1, and denotes each of the communication passes. The communication attentions can be integrated either at the sender as ui,, (m,,,), or at the receiver as fe,,e (mA,,)). Replacing the input (5,,) of Eq. 4 with the belief (h,,)). .) the actor and critic become ne, (-124,0) and Vw; (ha,e, aN,,)). and the frozen estimations are Ti, and ui,,, respectively.
hhe g, (hir (n((-1) exy (sv,.)), exp (MN,..-)),
Proposition 4.1 (Neighborhood Neural Communication). In spatiotemporal RL with neighborhood NeurComm, each agent utilizes the delayed global information to learn its belief, and it learns the message optimize the control performance of all other agents.
NeurComm enabled MARL can be represented using single meta-DNN since all agents are connected by differentiable communication links, and 5; are the intermediate outputs after com- munication layers. Fig. la illustrates the forward propagations inside each individual agent and Fig. [b]ssows the broader multi-step spatiotemporal propagations. Note the gradient propagation of this meta-DNN is decentralized based on each local loss signal. As time advances, the involved parameters in each propagation expand spatially in the meta-DNN, due to the cascaded neigh- borhood communication. To see this mathematically, Ta,, (- Tæ,,, (-1sv,, TAG,--1), with
o., (Ai, Ug 0:3: while To, (-114441) -Tr (-18,,e Tattt (SNGLe Tag.e- JiEN)), with
AAdiiiional cell state needs to he maintained if I.M. is used.
4

---

Published as a conference paper at ICLR 2020
0.,+11 UIJ,,, U (A,, 0z). In other words, (4, will be updated for improving actors TT, Vij E V. as soon they are included 0,: meanwhile, r; will be included in R5. In contrast, the policy is fully decentralized in execution, as 9r,, depends on 8; only.
Ai 14
14-1.
9k-14
H4-2,6-2
m
R2
FLL
V2
AEe-1
m
Rit-1.t
Ak--
Encodinn Layers
Peliet
Actoe
Crtte
Batel
Eneading
Encoding
Ligers
Layera
Layers
(a) Intra-step propagations.
(b) Inter-step propagations.
Figure 1: Forward propagations of NeurComm enabled MARL, illustrated queueing system. (a) Single-step forward propagations inside agent Different colored boxes and arrows show different outputs and functions, respectively. Solid and dashed arrows indicate actor and critic propagations, respectively. (b) Multi-step forward propagations for updating the belief of agent i.
NeurComm is general enough and has connections to other communication protocols. CommNet performs more lossy aggregation since the received messages are averaged before encoding, and all encoded inputs are summed up (Sukhbaatar et al. (016)) In DIAL, each DQN agent encodes the received messages instead of averaging them, but still it sums all encoded inputs (Foerster et 2016). Also, both CommNet and DIAL do not have policy fingerprints included in messages.
5 NUMERICAL EXPERIMENTS
5.1 ENVIRONMENT SETUP
There are several benchmark MARL environments such as cooperative navigation and predator- but few of them represent NSC. Here design two NSC environments: adaptive traffic signal control (ATSC) and cooperative adaptive cruise control (CACC). Both ATSC and CACC are extensively studied in intelligent transportation systems, and they hold assumptions of spatiotemporal MDP.
5.1.1 ADAPTIVE TRAFFIC SIGNAL CONTROL
The objective of ATSC is to adaptively adjust signal phases minimize traffic congestion based real-time coad-traffic measurements. Here we implement two ATSC scenarios: 5 x 5 synthetic traffic grid and real-world 28-intersection traffic network from Monaco city, using standard microscopic traffic simulator SUMO (Krajzewicz et al. 2012).
General settings. For both scenarios, each episode simulates the peak-hour traffic, and a 5s control interval is applied to prevent traffic light from too frequent switches, based on RL control latency and driver response delay. Thus, one MDP step corresponds to simulation and the horizon 720 steps. Further, a 2s yellow time is inserted before switching to red light for safety purposes. In ATSC, the real-time traffic flow, that is, the total number of approaching vehicles along each incoming lane, measured by near-intersectton induction- detectors (ILDs) (shown the blue areas of example intersections in Fig. 2). The cost of each agent is the sum of queue lengths along all incoming lanes.
Scenario settings. Fig. 2a illustrates the traffic grid formed by two-lane arterial streets with speed limit 20m/s and one-lane avenues with speed limit m/s. We simulate the peak- peak-hour hour traffic dynamics through four collections of time-variant traffic flows, with both loading and recovering phases. At beginning, three major flows F are generated with origin- destination (O-D) pairs Tin 2 425 and 212-Ug, meanwhile three minor flows fi generated with O-D pairs 21 -377, #2- and 13-27g

---

Published as a conference paper at ICLR 2020
After 15 minutes, F1 and f1 start to decay. while their opposite flows F2 and fz start dominate, as shown in Fig. 2b] Note the flows define the high-level demand only, the particular route of each vehicle is randomly generated. The grid is homogeneous and all agents have the same action space, which is a set of five pre-defined signal phases. Fig. 2c illustrates the Monaco traffic network, with controlled intersections in blue. NMARL in this scenario is more challenging since the network is heterogeneous with a variety of observation and action spaces. Four traffic flow collections are generated to simulate the peak-hour traffic, and each flow multiple of "unit" flow of 325veh/hr, with randomly sampled O-D pairs inside rectangle areas in Fig. [c F1 and F2 are simulated during the first 40min, as [1, 2, 4, 4, 2, unit flows with 5min intervals; F3 and F1 are generated in the same way, but with delay of 15min. See code for more details.
200m
800
x11
400
x10
2006
3930
(a) Synthetic traffic grid. (b) Traffic flows within the grid.
(c) Monaco traffic network.
Figure 2: ATSC scenarios for NMARL. (a) Synthetic traffic grid, with major and minor traffic flows shown in solid and dotted arrows. (b) Simulated time-variant traffic flows within the traffic grid. (c) Monaco traffic network, with traffic flow collections shown in colored arrows.
5.1.2 COOPERATIVE ADAPTIVE CRUISE CONTROL
The objective of CACC is to adaptively coordinate a platoon of vehicles to minimize the car-following headway and speed perturbations based on real-time vehicle-to-vehice communication. Here we implement CACC scenarios: "Catch-up" and "Slow-down", with physical vehicle dynamics.
General settings. For both CACC tasks, we simulate a string of 8 vehicles for 60s, with a 0.1s control interval. Each vehicle observes and shares its headway h, velocity v, and acceleration a to neighbors within two steps. The safety constraints are: h 2 1m, v S 30m/s, |a| S 2.5m/s". Safe RL relevant here, but itself is big topic and out of the scope of this paper. So we adopt simple heuristic optimal velocity model (OVM) (Bando et al. 1995) to perform longitudinal vehicle control under above constraints, whose behavior is affected by hyper-panee paperr-arameters: headway gain aº, relative velocity gain 3 stop headway ha = 5m and full-speed headway hgo = 35m. Usually (a 889) represent the human driver behavior, here we train NMARL recommend appropriate (a ,88 for each OVM controller, selected from four levels (0. 0), (0.5, 0), (0, 0.5), (0.5, 0.5)). Assuming the target headway and velocity profile are h' = 20m and vi, respectively, the cost of each agent is (hr, h²)² + (Vi,z v²)² + 0.1u,, Whenever collision happens (24, 1m), a large penalty of 1000 assigned to each agent and the state becomes absorbing. An additional cost 5(2hg h,. is provided in training for potential collisions.
Scenario settings. Since exploring a collision-free CACC strategy itself is challenging for on- policy RL, we consider simple scenarios. In Catch-up scenario, V..0 vi =15m/s and h,o0 h", vi * whereas h1,0 = h², with E U[3, 4). Slow-down scenario, Vi,a = v b- 15ms. b E [1.5,, 2.5], and h4p0 = h", Vi, whereas vē linearly decreases to 15m/s during the first 30s and then stays at constant.
5.2 ALGORITHM SETUP
For fair comparison, all MARL approaches are applied to A2C agents with learning methods in Eq. (3)4), and only neighborhood observation and communication independent learning. which is an A2C implementation of MADDPG (Lowe et al., 2017) as the critic takes neighhoring actions (see Eq. (4)) ConseNet (Zhang et al. 2018) has the additional consensus
allowed. IA2C performs

---

Published as a conference paper at ICLR 2020
update overwrite parameters of each critic the mean of those of all critics inside the closed neighborhood. FPrint (Foerster et al. 2017)) includes neighbor policies. DIAL (Foerster et al. 2016) and CommNet (Sukhbaatar et al. 2016) are described in Section 4 IA2C, ConseNet, and FPrint non-communicative policies since they utilize only neighborhood information. In contrast, DIAL, CommNet, and NeurComm
communicative policies. Note communicative policies require more messages to be transferred and so higher communication bandwidth. In particular, the local message sizes are ([s: + | + [b;l) for DIAL and NeurComm, O[ss; |b,]) for CommNet, O[1s: + (T:)))
for FPrint, and ([ss)) for IA2C and ConseNet. The implementation details are in C.1
All algorithms use the same DNN hidden layers: one fully-connected layer for message encoding ex- and one LSTM layer for message extracting 9v- All hidden layers have 64 units. The encoding layer implicitly learns normalization across different input signal types. We train each model over 1M steps, with 7 0.99, actor learning rate 5 x 10-1 and critic learning rate 2.5 x 10-4. Also, each training episode has different seed for generalization purposes. In ATSC, 3 0.01, [B] 120, while in CACC, & 0.05, [B] 60, to encourage the exploration of collision- ffree policies. Each training takes about 30 hours on a 32GB memory, Intel Xeon CPU machine.
5.3 ABLATION STUDY
We perform ablation study in proposed scenarios, which are sorted as ATSC Monaco > ATSC Grid > CACC Slow-down > CACC Catch-up by task difficulty. ATSC is more challenging than CACC due larger scale (>=25 vs 8), more complex dynamics (stochastic traffic flow vs deterministic vehicle dynamics), and longer control interval (5s vs 0.)s). ATSC Monaco > ATSC Grid due to more heterogenous network, while CACC Slow-down > CACC Catch-up due to more frequently changing leading vehicle profile. To visualize the learning performance,
plot the learning curve, that is, average episode return (R + Ei- Eiey Fi,t) vs training step. For better visualization, all learning curves are smoothened using moving average with window size of 100 episodes.
First, we investigate the impact of spatial discount factor, by comparing the learning curves among {0.8, 0.9, }} for IA2C and CommNet. Fig. 3 reveals few interesting facts. First, Coommeee is always higher than a. Indeed, Commmer This is because communicative policies perform delayed global information sharing, whereas non- communicative policies utilize neighborhood information only, causing difficulty fit the global return. Second, learning performance becomes much more sensitive to a when the task is more difficult. Specifically, all a values lead to similar learning curves in CACC Catch-up, whereas appropriate values help IA2C converge to much better policies more steadily in other scenarios. Third, a* is high enough: a'Ac = 0.9 except for CACC Slow-down where aj = 0.8. This is because the discounted problem must be similar enough to the original problem in execution.
almost all scenarios (except for ATSC Monaco).
Next, we investigate the impact of NeurComm under a = 1. We start with a bascline which is similar existing differentiable protocols, i.e., his LSTM(h;, relusyy. + rel relu(mx,..) ...).. We then evaluate two intermediate protocols "Concat Only" and "FPrint Only", in which encoded inputs concatenated and neighbor policies are included, respectively. Finally evaluate their combination NeurComm. As shown in Fig. 3 all protocols have similar learning curves in easy CACC Catch-up scenario. Otherwise, both "Concat" and "FPrint" are able enhance the baseline learning curves in certain scenarios and their affects are additive in NeurComm.
5.4 TRAINING RESULTS
Fig. compares the learning curves of all MARL algorithms, after tuned a E {0.6, 0.9, 0.95, 1}. As expected, a" for nonccommunicative policies are lower than those for communicative policies.
Table I: Best spatial discount factors a across NMARL scenarios.
Scenario Name
NeurComm
CommNet
DIAL
IA2C
FPrint
ConseNet
ATSC Grid
1.0
1.0
1.0
0.9
0.95
0.9
ATSC Monaco
1.0
0.9
0.9
0.9
0.9
0.9
CACC Catch-up
1.0
1.0
1.0
1.0
1.0
1.0
CACC Slow-down
1.0
1.0
1.0
0.8
0.9
0.8
7

---

Published as a conference paper at ICLR 2020
-20
200
-1500
-00
-1000
-00
350
-3000
Teal 6ip
Tearng
1100000
Tuanin
-20
122
-600
-1100
300
-330
-50
-20
-304
0.30
0.24
8.3
334
8.30
0.34
0,48
0.5M
3.34
0.04
2.2m
1.50
3.54
0.EM
6.44
2:20
2.40
0.50
0.24
(a) ATSC Grid.
(b) ATSC Monaco.
(c) CACC Catch-up.
(d) CACC Slow-down.
Figure 3: Sensitivity and ablation study of spatial discount factor (top) and NeurComm (bottom). The
best learning curves are in bold.
-200
200
-100
055
-300
-100
300
-100
-30
-000
Tucliig king
Th.
(a) ATSC Grid.
(b) ATSC Monaco.
(c) CACC Catch-up.
(d) CACC Slow-
Figure 4: Training performance comparison after tuned spatial discount factors.
Tab. summarizes a" of controllers across different NMARL scenarios. For challenging scenarios like ATSC Monaco, lower preferred by almost all policies (except NeurComm). This demonstrates that is an effective way to enhance MARL performance in general, especially for challenging tasks like ATSC Monaco. From another view point, a serves as an informative indicator on problem difficulty and algorithm coordination level. Based Fig. 4 NeurComm is at least competitive in CACC scenarios, and clearly outperforms other policies on both sample efficiency and learning stability in more challenging ATSC scenarios. Note in CACC big penalty is assigned whenever collision happens, so the standard deviation of episode returns is high.
5.5 EXECUTION RESULTS
We freeze and evaluate trained MARL policies another 50 episodes, and summarize the results in Tab. 2 In CACC scenarios, a enhanced FPrint policy achieves the best execution performance. Note NeurComm still outperforms other communicative algorithms, so this result implies that delayed information sharing may be helpful in easy but real-time and safety-critical CACC tasks. In contrast, NeurComm achieves the best execution performance for ATSC tasks. We also evaluate the execution performance of ATSC and CACC using domain-specinn metrics in Tab. 3 and Tab. respectively. The results are consistent with the reward-defined ones in Tab. 2
Further, we investigate the performance top policies in ATSC scenarios. For each ATSC scenario, we select the top two non-communicative and communicative policies and visualize their impact on network traffic by plotting the time series of network averaged queue length and intersection delay in Fig. 5 Note the line and shade show the mean and standard deviation of each metric across execution runs, respectively. Based on Fig. 5a NeurComm achieves the most sustainable traffic control in ATSC Grid, that the congested grid starts recovering immediately after the loading phase ends at 3000s. During the same unloading phase, CommNet prevents the queues from further increasing while non-communicative policies are failed do so. Also, FPrint is less robust than IA2C as
8

---

ConseNet Published as a conference paper at ICLR 2020
Table Execution performance comparison over trained MARL policies. Best values are in bold.
Scenario Name
NeurComm
CommNet
DIAL
IA2C
FPrint
ATSC Grid
-136.1
-165.1
-21..4
-160.2
-155,9
-187.5
ATSC Monaco
-226.3
-263.0
-339.4
-369.7
-359.4
-528.9
CACC Catch-up
-94.6
-95.6
-246.4
-419.7 -261.7
-57.8
CACC Slow-down
-934.7
-950.8
-1112
-2209
-697.9
-1038
330
100
100
Siraaton
(a) Average queue length (b) Average queue length ATSC Grid.
(c) Average intersection delay ATSC Grid.
(d) Average intersection
ATSC Monaco.
delay ATSC Monaco.
Figure 5: Execution performance comparison among top policies in ATSC scenarios, measured as average queue length and average intersection delay over time.
introduces sudden congestion jump at 1000s. Similarly, NeurComm achieves the lowest saturation rate in ATSC Monaco (Fig. 5b).
Intersection delay is another key metric in ATSC. Based Fig. 5c communicative policies are able to reduce intersection delay as well in ATSC Grid, though it is not explicitly included in the objective and so is not optimized by communimmnee policies. In contrast, communicative policies have fast increase on intersection delay in ATSC Monaco. This implies that communicative algorithms are able to capture the spatiotemporal traffic pattern in homogeneous networks whereas they still have the risk of overfitting on queue reduction in realistic and heterogenous networks. For example, they block the short source edges on purpose to reduce on-road vehicles by paying a small cost of queue length.
Finally, investigate the robustness (string stability) of top policies in CACC scenarios. In particular, we plot the time series headway and velocity for the first and the last vehicles in the platoon. The profile of the first vehicle indicates how adaptively the controller pursues h" and v", while that of the last vehicle indicates how stable the controlled platoon is. Based on Tab. [ and Tab. the top communicative and non-communicative controllers NeurComm and FPrint.
Fig. 6 shows the corresponding headway and velocity profiles for the selected controllers. Interest- ingly, MARL controllers are able to achieve steady state v and h* for the first vehicle of platoon, whereas they still have difficulty to eliminate the perturbation through the platoon. This may be because of the heuristic low-level controller as well the delayed information sharing.
MURCCATY AIRE
thine uPhi
then
(a) Headway profiles in CACC Catch-up
(b) Velocity profiles in CACC Catch-up
(c) Headway profiles in CACC Slow-down
(d) Velocity profiles
in CACC Slow-down
Figure 6: Headway and velocity profiles the first and last vehicles of the platoon, controlled by top
communicative and non-communicative policies in execution.

---

Published as a conference paper at ICLR 2020
6 CONCLUSIONS
We have formulated the spatiotemporal MDP for decentralized NSC under neighborhood commu- nication. Further, we have introduced the spatial discount factor enhance non-communicative MARL algorithms, and proposed neural communication protocol NeurComm to design adaptive and efficient communicative MARL algorithms. We hope this paper provides rethink on developing scalable and robust MARL controllers for NSC, by following practical engineering assumptions and combining appropriate learning and communication methods rather than reusing existing MARL algorithms. One future direction is improving the recurrett units to naturally control spatiotemporal information flows within the meta-DNY in a decentralized way.
ACKNOWLEDGMENTS
We would like to thank Marco Pavone and Alexander Anemogiannis for valuable discussions and insightful comments.
REFERENCES
Masako Bando, Katsuya Hasebe, Akihiro Nakayama, Akihiro Shibata, and Yuki Sugiyama. Dy- namical model of traffic congestion and numerical simulation. Physical review E, 51(2):1035, 1995.
Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li. Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions on Intellient Transportation Systems, 2019.
Jakob Foerster, loannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-2145, 2016.
Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Philip Torr, Pushmeet Kohli, Shimon Whiteson, et al. Stabilising experience replay for deep multi-agent reinforcement learning. arXiv preprint arXiv:1702.08880 2017.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pp. 66 13. Springer, 2017.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. In Advances in Neural Information Processing Systems, pp. 7254-7264, 2018.
I Ge Jin and Gábor Orosz. Dynamics of connected vehicle systems with delayed acceleration feedback. Transportation Research Part C: Emerging Technologies, 46:46-64, 2014.
Xiangyu Kong, Bo Xin, Fangehen Liu, and Yizhou Wang. Revisiting the master-slave architecture in multi-agent deep reinforcement learning. arXiv preprint arXiv:1712.07305. 2017.
Daniel Krajzewicz, Jakob Erdmann, Michael Behrisch, and Laura Bieker. Recent development and applications of SUMO Simulation of Urban MObility. International Journal On Advances in Systems and Measurements, 5(3&4): 12:4)::221111333 138, December 2012.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint /wtp:/099900..... 2015.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed enperaatvv-competttiy environments. Advances in Neural Information Processing Systems, pp. 6379-6699 6699,,3900000 2017.
10

---

Published as a conference paper at ICLR 2020
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 5 518(7540)5529 5335533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference Machine Learning, pp. 1928 1928-1937, 2016.
Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P How, and John Vian. Deep decentralized multi-task multi-agent under partial observability. arXiv preprint arXiv: XXiv:/703.0618. 2017.
Peng Peng. Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent wwa.eetiiinnllyoooonniiii nets: Emergence human-level coordination learning to play stareraft combat games. arXiv preprint
10069, 2017.
Junjie Qin, Yinlam Chow, Jiyan Yang, and Ram Rajagopal. Distributed online modified greedy algorithm for networked storage operation under uncertainty. IEEE Transactions on Smart Grid, (2):1 10-1106-118,, 2016.
Chao Qu, Shie Mannor, Huan Xu, Yuan Qi, Le Song, and Junwu Xiong. Value propagation for de- centralized networked deep multi-agent reinforcement learning. arXiv preprint 1901:1901.093266, 2019.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. arXiv preprint arXiv: 1rXiv:1812.09755, 2018.
Sainbayar Sukhbaatar, Rob Fergus, et Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems, pp. 2244-2252, 2016.
Richard S Sutton and Andrew G Barto. Reinforcement learning: an introduction. Neural Networks,
IEEE Transactions on, 155:10544, 1998.
Yong Xu, Renquan Lu, Peng Shi, Hongyi Li, and Shengli Xie. Finite-time distributed state estimation over sensor networks with round-robin protocol and fading channels. IEEE transactions cybernetics, 48(1) 483333336334 2018.
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. arXiv preprint arXiv: arXiv: /802.05800 2018.
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Başar. Fully decentralized multi- agent reinforcement learning with networked agents, arXiv preprint arXiv: 1002 2018.
APPENDIX
A PROOFS
A.1 PROOF OF PROPOSITION 3.1
Proof. The proof follows the learning method in A2C MnCh et al. (2016). which shows that
[())= 1
log To (a (5)) + B Σ
no (a(s)) log (al(s))))
B
aEA
((w)
R
Va((+))
B
where - R Uz, R
7 r+, + + 78 and U, = Vu- (s7), based on on-policy
minibatch from MDP ((s,, ar, TrEB-
Now we consider spatiotemporal MDP, which has transition in Eq. I optimizes return in Eq. 2 and
collects experience (si,e; MAjtt 0i,, Fe.), where fr.t Ejey ryg.. y In Theorem 3.1 of Zhang
11

---

Published as a conference paper at ICLR 2020
et al. 2018). the decentralized actor and critic are *e, (s) and (s. for fitting m7 (-|s) and E.A,A E.A, [a; (e (s, a) under global observations, respectively. Now assuming the observations restricted to each neighborhood as in Definition 3.1 then the actor and critic and communications
become To, (8:) No, (s) and Va, (&s, aN;) Va, (s,a-i), with the best observability.
Hence, replacing Th Falaas (a)s) s)s Vu (s), r by Te, (a,]&;), Va, (8s, awr), and i; respectively, we establish Eq. 3 from Eq. 66 which concludes the proof.
Note partial observability and non-stationarity are present in ne. (c))3)) and Va, (3,, ay,) Fortunately, communication improves the observability, Based on Definition 3.1 any information that agent knows at time t can be included in mji.-- We assume $j. U (maja-1 TRe, C myj.t- Then
f C Si, {33 U (migao-1RENN bev.
c (sueien (s30-1 (mNn
c (subien,
U f0jr-2 U (muNit-)RReN,eeeee
C Si,e (8 ()
Thus, 3t, includes the delayed global observations. On the other hand, Eq. 1 non-stationarity. To see this mathematically,
mitigate the
Excplirce ar] =Em (r.. + a En.,p Iraelos, ay,
Σ
JEN.
Amaa
Eyp.risssy,,,.a,ppl av,.el
Ep bp,leeelsy,
(E(V)d
where the further away reward signals are discounted more. Note communication is allowed, each agent will have delayed global observations, and the non-stationarity mainly comes from limited information of future actions.
A.2 PROOF OF PROPOSITION
This proposition contains two statements regarding neural communication based global information sharing in forward and backward propagations. We establish each of them separately. Lemma A.1 (Spatial Information Propagation). In NeurComm, the delayed global information is utilized to estimate each hidden state, that is,
hir 3 81,0st U {8,0 Tt,0tt-4 ) jEV(()'
(8)
where x D y if information is utilized to estimate and Fo: := {wo, T1. IIt
Proof. Based on the definition of NeurComm protocol (Eq, (5 mi, h4 and h4 5 hi,e- U
8v,, U U U...I- Hence,
bit si. ( thje
C sha U
(s (sh.c- U 2he.e-2kee, Jev. S U U (,
U8
2
5
$4.0t - U U1 T - (acooe2))jeeill 2e(1dd
U... U (51,0:41-dmmss Tf00tt densse Je (V =dmx))
which concludes the proof.
Lemma A.2 (Spatial Gradient Propagation). In NeurComm, each message is learned to optimize the performance other agents, that is, {vr, A,) receive almost all gradients from L(O,), L(w,). dj (Vlj * i).
12

---

Published as a conference paper at ICLR 2020
Proof. If we rewrite the required information for given hidden state ha using intermediate messages instead of inputs, the result Lemma A.1 becomes
h4g
(mgL)JENs D {hy,e-1)EN,
3 {m 5
Hence, mi,r is included in the meta-DNN of agent time + dij 1. In other words, {vr, A receive gradients from [(0,). ). L(w,). Vj [VIj * i}, except for the first dej 1 experience samples. Assuming dmax < [B], {v, ) receive almost all gradients from loss signals of all other agents, which concludes the proof.
B ALGORITHMS
Algo. presents the algorithm of model training synchronous way, following descriptions in Section and Four iterations performed each step: the first iteration (lines 3-5) updates and sends messages; the second iteration (lines 6-10) updates hidden state, policy, and action; the third iteration (lines 11-14) updates value estimation and executes action; the fourth iteration (lines 22-26) performs gradient updates on actor, critic, and neural communication. On the other hand, Algo, P presents the algorithm of decentralized model execution in asynchronous way. It runs job that repeatedly measures traffic, sends message, receives messages, and performs control.
Algorithm 1: Multi agent A2C with NeurComm (Training)
Parameter tax 8, T, [B), Yuu lla.
Result: {A; Ms; ws, ws, V.).
initialize So. T- h-1, 0, + 0, B B+
2 repeat
for i V do send mi, = fx, (hi,--1);
end
for i V do
observe ši,t 8V;, TT UmNa.
update his & 9v. (hi,1-1, Tie + no, (-1h.,); update Og,t Ta,
end 11
for i E V do
12
update Vi Vas (Iii,,,a,,.)) execute 0:.4
13
14
end
simulate ri,t)
16
update B + B U (8t,t, Ti,,-1 a,,-,,,,,,, The heee
update + t+ 1, k 1;
18
if - then
19
initialize 8o. 1- h x 0:
20
end
21
if A:
then for i V do
22
update Re Ans Vr B, based on Proposition 3.1
24
update (Ag Uy Vjjvv U (w:]. based on 7h V L[(w);; update Vijev U (0:). based on ne v L(0.);
25
end initialize B 8. 1 00
27
28
end
until Stop condition is reached:
13

---

Published as a conference paper at ICLR 2020
Algorithm 2: Multii-geent A2C with NeurComm (Execution)
Parameter: (,,, 0 Nev, pommm. Accontoo--
for i V do
initialize h; + 0, ni + 0, {85, Ty, my m5Jsen + 0: repeat
observe si:
update my 6 f2, (hi):
send 8t, Tq, m,;; for j E N do
receive and update 8j, Ty. my within Ataommi end
10
update 8, < sv; U UTN UmN:
11
update h; + gv, (his 3i). Ti f (-1ha)
12
execute Tī;
13
sleep Accorrlli
14
until Stop condition is reached;
end
C EXPERIMENT DETAILS
C.I ALGORITHM SETUP
Detailed algorithm implementations are listed below, in term of Eq. 5.. IA2C: hit
relussy.)) ConseNet: same IA2C but with consensus critic up- NeurComm:
date. FPrint: h4, = LSTM(h,,-1, (roce relu(,,,.)) DIAL: hat
relu(w,,,-)) CommNet: h1 = + relu(relu(hh,-i) + onehot LSTM(h;,-1. tan.) + 1-iiearr [inearrmeannhhv:: (h...)))) For ConseNet, only do consensus up- date on the LSTM layer, since the input and output layer sizes may not be fixed across agents. Also, the actor and critic ,,. so coftmaat Ftmaa(aaaa)) (ha,,)), and U224 linea riceacaccnaatt (hi,,, (aM)))(aN;:)))
hat LSTM(h,,-1,
C.2 EXPERIMENTS IN ATSC ENVIRONMENT
C.2.1 ACTION SPACE
Fig. 7 illustrates the action space of five phases for each intersection in the ATSC Grid scenario. The ATSC Monaco scenario has complex and heterogeneous action spaces, please see the code for more details. To summarize, there 11 two-phase intersections, 3 three-phase intersections, 10 four-phase intersections, five-phase intersection, and 3 six six-phase intersections.
C.2.2 SUMMARY EXECUTION PERFORMANCE
Table 3 summarizes the key metrics in ATSC. The spatial average is taken at each second, and then the temporal average is calculated for all metrics (except for trip delay, which is directly aggregated all trips). NeurComm outperforms all baselines delay. Interestingly, even though IA2C is good at optimizing the given objective of queue length, performs poorly on optimizing intersection and trip delays.
minimizing queue length and intersection
C.2.3
VISUALIZATION OF EXECUTION PERFORMANCE
Fig. 8 and Fig. show screenshots of traffic distributions the grid at different simulation steps for execution episode with random seed 2000. Clearly, communicative MARL controllers have better performance on reducing the intersection delay. NeurComm and CommNet have the best overall performance.
each MARL controller. The visualization is based
14

---

Published as a conference paper at ICLR 2020
Phase 1
Phase 2
Phase 3
Right of Way
Yield
Stop
Phase
Phase 5
Figure 7: Possible signal phases at each intersection.
Table 3: Performance of MARL controllers in ATSC environments: synthetic traffic grid (top) and
Monaco traffic network (bottom). Best values are in bold.
Temporal Average Metrics
NeurComm
CommNet
DIAL
IA2C
FPrint
ConseNet
avg queue length [veh]
1.16
1.44
2.36
1.63
1.62
2.04
avg intersection delay [s/veh]
68
111
145
376
415
366
avg vehicle speed [m/s]
2.28
1.82
1.78
0.26
0.23
0.29
trip delay [s]
293
455
1949
2067
1949
321
avg queue length [veh]
1.27
1.56
1.88
1.93
1.87
2.74
avg intersection delay [s/veh]
221.1
236.5
231.3
147.4
174.8
187.3
avg vehicle speed [m/s]
0.55
0.61
0.94
2.36
1.26
1.03
trip delay [s]
569
847
506
295
428
540
C.3 EXPERIMENTS IN CACC ENVIRONMENTS
C.3.1 SUMMARY OF EXECUTION PERFORMANCE
Table summarizes the key metrics in CACC. The best headway and velocity averages are closest ones 20m, and v* 15m/s. Note the averages are only computed from safe execution episodes, and we use another metric "collision number" to count the number of episodes where an collision happens within the horizon. Ideally, "collision-free" is the top priority. However, safe not the focus of this paper so trained MARL. controllers cannot achieve this goal in the experiments of CACC.
Table 4: Performance of MARL controllers in CACC environments: catch-up (above) and slow-down (below). Best values are bold.
Temporal Average Metrics
NeurComm
CommNet
DIAL
IA2C
FPrint
ConseNet
avg vehicle headway [m]
20.45
20.47
21.99
22.02
20.44
21.45
vehicle headway [m]
1.20
1.18
0.20
0.19
1.03
0
avg vehicle velocity [m/s]
15.33
15.33
15.07
15.07
15.33
15.00
std vehicle velocity [m/s]
0.90
0.87
0.16
0.18
0.75
0
collision number
0
0
0
0
0
avg vehicle headway [m]
15.84
16.24
14.42
18.21
11.60
std vehicle headway [m]
2.10
2.16
1.70
2.40
0.49
avg vehicle velocity [m/s]
13.43
13.82
12.28
15,47
8.59
std vehicle velocity [m/s]
2.77
2.88
2.49
3.37
1.19
collision number
13
12
16
50
8
23
15

---

Published as a conference paper at ICLR 2020
t 1500 see
t
2500 sec
3500 sec
IA2C
ConseNet
FPrint
Figure 8: Screenshots of traffic distribution in the grid. Each row is non-communicative MARL controller and each column is simulation step. The traffic condition along each lane visualized as line segment, with the color indicating the queue length or congestion level (grey: 0% traffic, green: 25% traffic, yellow: 50% traffic, orange: 70% traffic, red: 90% traffic, intermediate traffic condition shown the interpolated color), while the thickness indicating the intersection delay (the thicker the longer waiting time at intersection).
16

---

Published as a conference paper at ICLR 2020
1500 sec 1
1 == 2500 sec
3500 sec
NeurComm
CommNet
DIAL
Figure 9: Screenshots of traffic distribution in the grid. Each row is communicative MARL controller and each column a simulation step.
17


Reviews:
[{"review_id": "SkloMOu3FB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is concerned with network multi-agent RL (N-MARL), where agents need to update their policy based on messages obtained only from neighboring nodes. This is done under sensible restrictions on the state transition distribution, which can be claimed to hold true in realistic networked settings. The authors argue that introducing a spatial discount factor (along a temporal one), where neighboring nodes have a small distance, stabilizes learning. Also, they provide a way of learning a networked communication protocol. Experiments are done on somewhat realistic simulations of traffic.\n\nI am not an expert in this type of distributed MARL, and what the SotA there is. What is proposed here makes sense to me, even though it also does not look very surprising to me. Spatial discounting seems to make sense. The authors do not comment on the relationship between the d_{ij} (or how to choose them) with the neighborhood graph, I suppose that neighbors have short distances, while non-neighbors have not. The NeurComm protocol is a bit vague to me, and I cannot say how it relates to other proposals. I find it a bit strange (unless I misunderstood something) that complete state and policy information is transmitted to every neighbor, and on top some latent vector. This sounds expensive to me, surely these communication channels are limited? It'd be more interesting to consider bandwidth limitations here. Without knowing related work on other protocols in detail, I suspect they probably work less well than NeurComm, simply because they allow for smaller messages only (f.ex., whereas they talk about \"policy fingerprints\", they seem to submit the complete policy parameters to neighbors -- that is not just a fingerprint). Maybe I am wrong, but if so, the paper does not explain things properly (there is some encoding and decoding going on, but just to propagate the hidden states at each node).\n\nTo me as an outsider, this looks like interesting work with well-simulated experiments. Of course, these are simulated and allow for no real-world conclusions, but then much of RL work does neither. It is just I'd be hard pressed to say what is really surprising in this paper, and what could be learned from it.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Skg3YJBCFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary\n\nThe authors use decentralized MARL for networked system control. Each agent might control a traffic light (exp 1) or a car in traffic (exp 2). Some features of their approach are a spatial Markov assumption (only neighborhood matters), a spatial discount factor, and NeurComm: a general message passing scheme between agent policies. The authors compare their method with CommNet (averages messages before broadcast), DIAL (small-scale direct communication), etc.\n\n2. Decision (accept or reject) with one or two key reasons for this choice.\n\nWeak accept.\n\n3. Supporting arguments\n\nThe experiments and analysis of the more general communication scheme are nice and the assumptions used make sense for the environments considered (spatial interactions and dynamics).\nThe 2 environments used are nice, but it would be nice to see non-traffic applications as well.\nThe methodological contributions make sense in the spatial domain, but it would be interesting to see how neighborhood-based assumptions can be used in non-spatial / non-Euclidean domains.\n\n4. Additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.\n\nIt would be nice to explain why / what communication strategy enables FPrint to do better than NeurComm.\n\n5. Questions", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlJ342CYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a multiagent learning algorithm for networked system control. The main contributions are 1) a spatial discount factor that can stablize the learning process, 2) a differentiable communication protocol NeurComm. The paper was evaluated thoroughly by comparing against recent MARL baselines on two realistic environments.\n\nI voted for \"Weak Accept\" because this paper can have important real-world applications such as traffic control, autonomous driving and power grid. I really like its evaluations on the realistic environments. In addition, the paper is clearly written, the algorithm seems reasonable and the evaluations are comprehensive. \n\nI did not give it a higher rating because the proposed algorithm only outperforms the baselines in two out of four test cases. It is not a very strong result. In this situation, I would hope that see a more detailed discussion why in CACC experiments, NeurComm does not perform as well. And I would also suggest the paper adding more test cases to show that the proposed algorithm indeed can dominate in most of the cases.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on multi-agent reinforcement learning applications in network systems control settings. A key consideration is the spatial layout of such systems, and the authors propose a problem formulation designed to leverage structural assumptions (e.g., locality). The authors derive a novel approach / communication protocol for these settings, and demonstrate strong performance and novel insights in realistic applications. Reviewers particularly commended the realistic applications explored here. Clarifying questions about the setting, experiments, and results were addressed in the rebuttal, and the resulting paper is judged to provide valuable novel insights.
Decision: Accept (Poster)