Delayed Propagation Transformer: A Universal Computation Engine towards Practical Control in Cyber-Physical Systems
Authors: Wenqing Zheng
Qiangqiang Guo
Hao Frank Yang
Peihao Wang
Zhangyang Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transformers
Control
Reinforcement Learning
Cyber-Physical System

Abstract: Multi-agent control is a central theme in the Cyber-Physical Systems (CPS). However, current control methods either receive non-Markovian states due to insufficient sensing and decentralized design, or suffer from poor convergence. This paper presents the Delayed Propagation Transformer (DePT), a new transformer-based model that specializes in the global modeling of CPS while taking into account the immutable constraints from the physical world. DePT induces a cone-shaped spatial-temporal attention prior, which injects the information propagation and aggregation principles and enables a global view. With physical constraint inductive bias baked into its design, our DePT is ready to plug and play for a broad class of multi-agent systems. The experimental results on one of the most challenging CPS -- network-scale traffic signal control system in the open world -- show that our model outperformed the state-of-the-art expert methods on synthetic and real-world datasets. Our codes are released at: https://github.com/VITA-Group/DePT.

Delayed Propagation Transformer: A Universal Computation Engine towards Practical Control in Cyber-Physical Systems
Wenging Zheng The University Texas at Austin
Qiangqiang Guo University of Washington guoqq1 Derggg1@@wweer.
Hao Yang University of Washington haoyaluw edu
w. zhengeutexas.ed.
Peihao Wang
Zhangyang Wang The University of Texas at Austin at laswang@ut exat edu
The University of Texas Austin peihaowang[@ut exas edu
Abstract
Multi-agent control is central theme in the Cyber-Physical Systems (CPS). How- ever, current control methods either receive non-Markovian states due insuf- ficient sensing and decentralized design, or suffer from poor convergence. This paper presents the Delayed Propagation Transformer (DePT), new transformer- based model that specializes in the global modeling of CPS while taking into account the immutable constraints from the physical world. DePT induces a cone-shaped spatial-temporal attention prior, which injects the information prop- agation and aggregation principles and enables global view. With physical constraint inductive bias baked into its design, our DePT is ready to plug and play for a broad class of multi-agent systems. The experimental results on of the most challenging CPS network-scale traffic signal control system in the open world - show that our model outperformed the state-of-the-aa ex- pert methods synthetic and real-world datasets. Our codes are released http:::///ithuute www/vTT...//c /tt.ta/bccc/oooooooor//////eT
Introduction
The Saber-Physical System (CPS) is ubiquitous in our modern society, examples include intelligent transportation systems. power grids, autonomous automobile systems, industrial control systems, and robotic swarms. A CPS consists of multiple physical agents that interact and cooperate from time time; as well as a cyberspace that is responsible for monitoring the status of the physical system, predicting future states, and/or assigning control actions to the physical agents 2 How to effectively execute multi-agent control over CPS is an open problem of vital importance.
In general, there are two types of methods to solve the CPS multi-agent control problem: decentralized and centralized ones. The decentralized methods usually design individual controller for each agent based on local states. Due to the agents' continuous interaction, the features and the influence of actions propagate between agents, hence the state transition for each agent relies on not only the locally observed states and local actions but also states and actions of other agents. Therefore, the locally observed states alone cannot make up a Markov Decision Process (MDP) 3 which requires the controller to balance the global as well as the local features [5, 7. Existing methods tackle this issue by either complementing local information with information from the node's s immediate neighbors or self history, or using Graph Convolutional Networks (GCNs) instead to recursively model information propagation and aggregation. However, no aforementioned method is free of the
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Space (x, v.
Event Propagation Cone Shape
Event Propagation from ( to ++
Time (t)
Space (v)
wvent y Al, start propagete
event (x. .)
Space
Space (X)
Illustration of the typical impact propagation pattern of an event in a (CPS). Analogous to the "light cone" concept in the special theory of relativity, the correlation pattern in the space-time spans a cone-shaped distribution, which could be set as our prior for DePT.
Figure 1:
System
limited localized horizon issue: an n-layer GCNs can view at most no-hop neighbors away; meanwhile, deeper GCNs usually suffer from the "bottleneck phenomenon" [8] and "over-smoothness" issues [9], which make GNNs practically hard to train and perform well. Hence, such methods are inherently restricted to making a complete MDP state. addition, the edges a CPS graph often have physical directions to propagating information. That caused "feature mismatching" [10] [1] when modelling by normal GCNs, and several methods [12] [4] have studied directed graph modeling as remedies.
In contrast to decentralized methods, centralized methods partially avoided the non- Markovvnn barrier introduced by mutual interactions. However, due to the incomplete coverage/deploymenn of sensors and the sensing limitations, the ideal MDP assumption cannot always be met by real-world CPS observations [15] [6]. Moreover, it difficult for the traditional centralized methods to converge due to the huge state and action spaces [17. If one considers N agents with ds dimensional state space and a actions per agent, then the state space size is ds * N, and the total number of actions (i.e., number of logits in the output layer of traditional centralized model) will become (d
Fortunately, the booming of transformer models shed new light on addressing the aforementioned difficulties [1] 19 20 21 [2 [33 24 25 26 The Transformer architectures belong to the centralized methods due their global view in the attention mechanism. Their global views free them from the locality inductive bias of GCNs, and make them promising candidates for globally controlling multi-agent CPS problems. Also thanks to the tokenization and decentralized processing of each tokens, transformers are also free from the huge state/action space issues: given dg dimensional state space and A actions for each agent, transformer's s input/output are still ds/dA, regardless of the agent quantity N. However, the traditional transformer is not immediately ready to be plugged in tackling CPS control problems due to the following two arising challenges:
#1. Lack of physics in the attention modeling, The classical transformers benefit from their fully flexible self-attention mechanisms to capture the complex interactions within data. This free-form is desirable for vision and NLP tasks [27 28), but no longer valuable when it comes to real CPS due to its unawareness of many physical constraints. One example is the directional propagation and firection-featur coupling issue previously mentioned. For another prominent example, the real passage of information flow between nodes and the propagation of the effect of past actions are subject to the physical propagation speed. Previous studies about graph learning using transformers mainly focus on non-physical systems and encourage between-node communication without any notion of physical latency [29] 30) In CPS, however, features take time to propagate through the physical world. Without that important physical delay constraint in mind, the learned self-attention might mislead the attention to non-relevant temporal events or physically disjoint node pairs.
#2. The difficulty of training and convergence under noisy CPS data. Transformers are strong universal representers free of inductive bias, such as those in convolutions or recurrence. But the blessing of unprecedented flexibility and larger capacity can turn into the curse training: transformers take much longer time to converge. They are resource- heavier and much more data-hungry compared to classical convolutional, or recurrent networks 24 26). In CPS control problems, the training data will suffer from even higher variance due to the unavoidable randomness in state transitions. Such issue further amplified in the multi-agent setting. How to properly train transformers to stable convergence in this scenario still present a daunting challenge.

---

In view of the above, we propose to model the delayed propagation effect in CPS by enforcing the physical constraints into the transformer, via cone-shaped camporal-spatial prior that the features propagation in physical world process naturally forms the cone shape in the time-space B1 32 We customized multi-level heterogeneous attention mechanism that has the cone shaped prior baked into its design. With such design, the attention between input tokens now have physical interpretation, hence guiding the controller to learn more effectively in exploring the collaborative strategy across agents. Our main technical innovations are summarized below:
We model the CPS control problem under a transformer- based framework. This a naturally motivated step: transformers are free from the locality restrictions, and learn inter- inter-agent correlations across the whole system with constant state/action space size.
We propose the Delayed Propagation Transformer (DePT), new type of transformer with apatio-temporal priors baked into the attention design, to better characterize the inductive bias of physical information propagation latency, accompanied with better interpretability.
We build CPS controller based on DePT, and take the well-received CPS benchmark transportation signal control system as study case. The proposed controller achieves the state-of-the-art performance in the challenging urban scale traffic signal control task.
2
Related Works
CPS Control and Learning. With the rise of Reinforcement Learning (RL), learning- based control systems are adapted into CPS systems, including traffic network [33 344. smart gird system [35] 36), and autonomous vehicle (37]. Using traffic network control as an example. IntelliLight [38] developed deep RL model with policy interpretations. PressLight (39] considered a multi-intersection problem, where an RL agent was trained for every individual intersection. [40] introduced a fully localized RL agent to control traffic signals at every intersection. To efficiently model the influence of the neighbor intersections, graph attentional network was further introduced in CoLight [4]. 42 reviewed
conventional and RL-based methods for TSCP in detail.
To boost the optimization process and enhance the model transferability, 43 proposed to speed up the learning process by leveraging the knowledge learned from existing scenarios. To investigate the cooperation mechanism among various types of nodes and enhance the model utility, AttendLight [44] designed multi-level attention mechanism handle various numbers of roads-lanes, thus enabling decision-making with different phases in an intersection. However, the previous control scheme in CPS, especially for traffic network control, either assumed idealistic sensing condition, or tended to downplay or even ignore inter-agent cooperation and global signal propagation.
Transformers. Transformers have witnessed great success in the application of natural language processing 24 18] and recently in computer vision too [2 [5] [6). In general, researchers use transformers process the node embedding in two orthogonal directions: first, through the node-wise residual feature transformation, an arbitrary type of intra-node transformation is enabled [88 47 48] :
second, through the attention mechanism, features from different nodes are dynamically aggregated and the inter-nodes relationships are captured 48] Previous efforts have shown the potential of transformers in multi-agent system [49], by flattening connections features across time and agents. [50] used transformers to tackle the sparse communication in multi-agent settings, and achieved oftte-of-the-rre (SOTA) performance. 30) introduced transformer for the sepphtoo-sqqenne learning in translation, which also inspired us to use explicit relation encoding to allow direct cooperation between two distant nodes in agent features sharing. Yet our best knowledge, existing transformers have not explicitly tackled the spatial-temporal features together with the connections encoding, and no effort has been devoted to their learning under CPS physical constraints.
Problem Settings
A general CPS can be modeled by graph G = (V,E) with node set 1/ and an edge set E. One key property distinguishes the CPS graph from many other graph problems: CPS, each individual node i E > possesses physical-world location E RR2/RR Such physical location exists as an immutable attribute, and exerts a special constraint to CPS. Unlike pure cyber world graphs such user- user-product graphs where the information can be instantancously propagated from one node to
3

---

Tranffomer
Maii
01c
Artention
Fezuures
Featuee chservation
Inpuc: Nodes
Physucal Construints
Spaliosenpuual infoemation
40-00 Hikee trehekking
NX
Spui
Cow Staped Aneies
A
Fundienari
Aui
Figure 2: The DePT illustration of self-Attention layers (above) and the internal structure (below).
another, the information can can only propagate under a limited speed in the physical world, and therefore the interactions between nodes in CPS are governed by their physical distances.
We are interested in the control problem in CPS, where each node is an agent that constantly interacts with the environment as well as with each other. At every timestamp 1, the agents observe the state g(t) from state space O, and take actions (f) from the action space A. The system then get a reward R(++),, The target of optimizing the CPS control to come up with a collective optimal policy Ty. under which the accumulated future reward is maximized:
k R(+*++1)
where 7 E [o, 1] is the discount factor. the discrete action space settings, the agents can be trained with Q-learning types of algorithms:
min (()) = E R((+1) +y max 8
((o((-1) (0+11); Q()(1) a aa0)))))
(2)
where 0 is the parameters of the the learned Q-function Q(s, a( 0 +
DePT: Delayed Propagation Transformer
4.1 Preliminary: Transformers for Cyber-Physical Systems
We begin by introducing the transformer as the centralized agent to handle control problems in CPS. The transformer takes series of inputs, collected from all nodes spatially, and across the most recent 1 T'ma timestamps temporally, making up the collected inputs as: X = { i V, T E (0, ...... T.mas Every 4,,-, in X is referred to as token, which can be uniquely indexed via the ID of its corresponding physical node and time difference relative the current timestamp /.
For each token $) in addition to feeding the transformer with the observed features (denoted as fi,--), also make our transformer aware of the policy information via the policy embedding: initialize trainable embedding matrix P E RR [AI, and for each action ai,--r R taken by node at time т, we index out the corresponding embedding P P(: ai..a,,,.), and concatenate p with the features fi and use [fit-r/] Ipii, the input.
For each pair of tokens Φ and Φ where i.j E V and S T, S Tinas the traditional design of transformer will compute the pre-softmax attention via:

---

a (i (0.--r:0)1--))) = - We W
(3)
where W and We are weight matrices to compute the key and query components, respectively. For every token o (1) i.t- at the I-th layer, its output from attention layer
E e (5 Wr
(4)
jEV,PE(Tmaz)
where e Pj,t-p) softmax;ev.ne(TT... (a (0..-r: 1,t- /T), and is the tempera- ture factor, the weight matrix Wy maps the input to the value component, and e(---) denotes the pairwise soft-softmax attention. In multi-head attention, each attention head will have output (d(k), h 1, N, where N is the number of attention heads. The computed attention heads are concatenated together and remapped to the output dimension via a linear block.
Following the attention layer, we append layer normalization module with a skip connection. Subsequently, the output will be fed into Feed-Forward Network (FFN) followed by skip- connected layer with normalization. Multi-head attention, normalization, and FFN constitute single encoding layer. We stack such layers building L-layer transformer. At the output layer, we only read out the feature embedding of the nodes at the current time f, i.c., Pi Vi E V. Afterward, we employ another fully connected layer to map the feature embedding to the pre-action q q-values. The architecture is illustrated in Fig. 2
4.2 Enforcing Spatial-temporal Constraints
As illustrated in Fig the spatial-temporal token pairs that have strong connections are more likely to be located around the cone-shaped area. We argue that in order to capture the most relevant spatial-temporal attention, the transformer agent operating in such settings should have their attention to follow the distribution of cone-shaped prior.
To enforce such prior attention distribution, we train a new function a ( Φ') (D for DePT), in addition to the Traditional transformer attention (7") in Eq 3 These two functions are added up to obtain the pre-softmax attention value. Further, (P)) (sb, o' consists three components: the learned pair-relation Look-Up-Table Up-Table embedding (attm)) j] (LUT), the cone shaped correlation decay function ((ii,.i.))) (ConeDecay), the temporal correlation decay function a(T p) (TimeDecay).
Look-Up- Up-Table embedding (LUT), To capture the Docatio--assoiiate. patterns, we enable our DePT to learn scalar attention value X (attn) [i for every pair of nodes (i,j) (tokens associated to the same nodes but with different timestamp share the same embedding). Such LUT is learned independently for every attention head in the original transformer.
ConeDecay. In the physical part of CPS, both the effect of nodes' actions and the passage of between
features can be modeled as information flow that are constantly scchaggng/proppagatio nodes. Think of information flow with speed v that originates from node at time t p. and heads for node at time t т. This flow will induce a causal connection in the physical world between two associated tokens -,,-- and -
It is straightforward to define the causal deviation follows:
E),r,. (0j 21.-7)) pp |u Ug
Under this definition, Shoii < O and Ej,pú.r > O corresponds to the expected "past" and "future" for the incident "Ø),.pp D - is exactly reachable by flow with speed v². We use the function 7 Wx) parameterized by W,y penalize those "unreachable" token pairs, hence 7(-) is expected to obtain its global maximum at (j,pi.r = O, and is a bi-directional decreasing function. It's d=((),mitr
gradient dej,mi.res
depicts how fast the attention should drop and how "wide" the attention should
spread, for the token j.tpp that deviates from the cone of Φ For the estimation the speed, is the average of three learnable functions: the originatiin//destinatio speed, and a speed LUT (different from the above A (attn)); â 1 (1)))()) to W.w)) (0', Worr) + (i,)),
5

---

TimeDecay. Information from more distant past should be payed with less attention. To fit such time dissipation effect, given two tokens Pi,--r: Pjs--p= we our DePT learn an additive function aT p: Wa) parameterized by Wa for the pairwise attention.
Overall Attention Form. The ConeDecay, TemaDecay. and LUT are all learned separately for every transformer encoder block and every attention head. To further guide the capturing of desired attention as well as accelerate the training/inference in two-fold speed, we mask out the attention between token pairs (0).- Pjttp)) whenever p > т, that is, not letting node n: pay attention to node ny when nj is in the future timestamp of n, and have no chance to pass information to n. Distilling the essence up to this point, the architectures of the proposed DePT encoder blocks is shown in Fig. 2 and the pre-softmax attention between token pair (Pi,t-r' Pj,e-p) is computed via:
( ((ipptt)) (T + ( tPi WT
if if >>> T
[1,]]
(6)
Otherwise
4.3
Training DePT Faster: Prior Pre-fitting and Imitation Learning
Same as all other centralized control methods, transformers are not immune to slow convergence issues during RL. To leverage the favorable attention prior of DePT, We propose to pre-train the ConeDecay. TimeDecay, LUT components of DePT before interacting with the environments. In addition to pre-training, we also adopt imitation learning the early stage of reinforcement learning.
Pre-fitting the Priors. There are six learnable components in every group of priors: the ConaDecay
function 7(-), three speed estimators to compute (,,ri, for the ConeDecay: v("(() p(() (-), [( [+.]]
the Deme function (-),, and the attention LUT [(att) -]. Above them, the attention LUT and the speed LUT are matrices and can be initialized as Mo and M, respectively, where M2 = 0, 8)
stands for random matrix with all entries i.i.d. and have mean value and V is the average flow speed according to the statistics of the system. 7(-) and (-)) are univariate functions, and be pre-trained to fit certain desired analytical functions. In practice,
y/ kxx² to pre-fit fit (c) and o (.), where l is the normalization factor depending on the variance of input features. The two last functions p(a) (-), (-), though multi-variate, can still be casily pre-fitted to labels sampled from (V, 0.1), and also lead to good initializations. All pre- pre-fitings/inittaizzattt
involve no costly tensor computation, and can therefore be accomplished within minutes by CPU. Such rreefitiing/iintialiizaiioss can largely boost the convergence of DePT, to be shown in section 6.1
Warm-up with Imitation Learning. Since the decentralized controller is typically easier to con- verge than the centralized method, we adopt the imitation learning(II.) [5] warm-up RI. Before RI., a decentralized baseline controller is first trained and acts as the teacher model. During the II. stage, the actions are imitated by DePT and evaluated by the teacher model. During the subsequent training, DePT takes actions and evaluates the Q-values by its own interaction.
5 Experiments and Discussions
We use the network-wide urban traffic signal control (TSC) problem under connected and autonomous vehicles (CAVs) environment as CPS control example to illustrate the performance of DePT. CAVs have great promises to help improve the performance of traffic signal control system, since they can provide traffic information (i.e., vehicle position, speed, acceleration, etc.) that are erucial to determining the signal phases and timings. The network-wide TSC under CAV is multi-agent control problem where distributed control methods will encounter the non-MDP property for single agent. In contrast, centralized control methods usually suffer from inefficient learning and slow convergence. In this section, verify the ability of DePT under this challenging setting. We first. define the TSC problem in Section and Then, show the simulation results and discuss them in Section 5.3
and introduce the dataset and simulation tools in Section 5.2
5.1 widwork-wide Urban TSC problem

---

In the TSC setting, the graph is the urban traffic network, where the nodes/agents represent intersections and edges indicate the connect- ing roads between intersections. The viable action at each node is the traffic light signal to control the traffic flow. For a specific node, we take the surrounded traffic information provided by CAVs as the state, and use the signal phases as actions. Specifically, assume there is k; incoming lanes for intersection we define the state of this intersection as:
curTiime IR: current time in simulation;
numIn E Nk: the number of CAVs on each of the k lanes;
numQue E N the number of stopped CAVs (i.c., those CAVs with speeds < 0.1m/8) on each of the h; lanes;
New York roadmap considered in the experiments.
Figure 3:
For intersection i, we define the action space as 21, where n, is the total number of phases for intersection As an example, consider an intersection that has two phases (i.e., n; 2): EWstt~East, East and {South North North-+South); we define a 1 as West--Eas/East--Wer green and the other conflict phase red, 2 as Nouth--Noth/Noort--Suut green and the other conflict phase red.
5.2 Dataset
We use Cityflow 52 as the simulation platform due to the high computation efficiency. To evaluate DePT, we run experiments both synthetic and real urban traffic networks built by [41]. The data is for scientific research usages, which is open-source and does not contain offensive content.
For the synthetic traffic networks, we use X 6 synthetic grid network with 36 agents taking actions every 10 seconds. There are 12 movements associated with every intersec- West~>outhh East->West, East Nasth:North,
tion (i.e.. West-+East, South-NNoth,, Sesth-West, Nouth->East North North->East, North-Westt). We de- sign four phases for each intersection: Fast,, West-+South, Fast-WWsst, North) Wostt-North, Fasth+South), [South [South->West, North-+East) There are 12 incoming lanes for each intersection, and the length of each lane is 300m. Similar to [4]], we test two flow settings for this network: bi-directional (noted as Grid-Bi) and uni-directional (noted as Grid-Uni). In Grid-Bi, we generate 300 vehicles/lanc/hour for West-East and East West movements and 90 vehieles/lane/hour for South-NNoth and North movements. In Grid-Uni, we generate the same flows, but only for West-+East and North movements.
South->ast,, North->South North and
For the real-world scenarios, agents as shown in Figure 3 The road network imported from OpenStreetMap. and the flow is the average travel time (noted Avg in
the upper east side of Manhattan, New York, with 28 7 active
extracted by taxi data. For the evaluation metrics, seconds) of all vehicles, and the average queue length (noted as AvgQue, in vehicles).
5.3 Experimental Settings and Results
The number of different timestamps to be fed into the DePT (Tmax) ) is set to be 10 in our experiments. The DePT trained for 200 rounds for each dataset, while the first 100 rounds are imitation learning and the second 100 rounds are independent training. During each round, we run simulations for 4 hours and record state/action pairs, then we train DePT for 100 epochs with Double-DQN using the newly generated data. DePT is compared with three typical network-wide urban TSC methods:
Fixed-time 53 A classical signal planning method that uses daily vehicle volumes to calculate pre-timed signal plans based on operation manuals. The signal plans will not change once they are generated.
Max-pressure 54 A network-wide signal control method that dynamically generates greedy signal plans by maximizing the "pressure" (calculated from queue length) of upstream and downstream roads.
7

---

Grid-Bi
Grid-Uni
YorYork
Dataset
AvgQue
Tvg
AvgQue
AvgTT
AvgQue
AvgTT
Fixed-time 53
0.11
225.62
0.06
225.62
1.72
1957.35
Max-pressuree [4]
0.09
208.13
0.04
198.93
1.75
1628.64
(29
0.06
204.34
0.05
192.44
1.73
1688.30
Co-light[41
0.05
191.66
0.04
188.23
1.69
1605,56
DePT
0.05
184.22
0.04
180.71
1.52
1555.47
Table 1: Evaluation results of DePT against others.
Stages Rounds
Imitation Learning
Independent Training
30
60
90
100
130
160
190
DePT
1143.1
244.3
211.6
202.4
199.6
188.2
185.0 184.9
n0-pre-fit
997.2
782.1
243.8
218.9
216.2
192.4
190.1 189.2
no-ConeDecay
1042.6
616.4
317.3
227.7
225.2
240.3
285.4
269.3
TTE
1215.4
1418.9
955.8
326.7
299.5
265.9
297.1 313.7
Table 2: Ablation study results: AvgTT on Grid-Bi. TTE refer to the traditional transformer encoder.
Co-light 41: A state-of-art multi-agent RL-based network-wide signal control method
that uses graph attentional networks to incorporate the temporal and spatial impacts among different intersections.
The teacher model we use during the II. stage is the Co-light model. The results for all models are shown in [ As can be observed, DePT achieves the best performance among the four models, and it especially outperforms its teacher model Co-light.
Ablation Studies and Model Interpretations
6.1 Ablation Studies
We report two ablation studies of DePT: the effect of the prior pre-fitting, and the existence of newly proposed priors. Since the convergence rate is prohibitively slow if no imitation learning, all experiments are still executed with imitation learning. The AvgTT results on Grid-Bi dataset are shown in Table In the table, the "DePT" corresponds the proposed DePT that has its priors pre- fitted, where the "no-pre-fit" corresponds to the DePT without pre-fitting prior, the "ho-aa is the DePT without ConeDecay component but still pre-fitted other prior, and the TTE corresponds to the traditional transformer encoder without any prior. As can be observed, the "DePT" achieves the fastest convergence speed as well as the best performance, where the "no-pre-fit" model can also slowly converge to satisfactory performance. However, when the ConeDecay component is removed, both the Dno-ConeeDeccy" and the TTE model fail to converge properly.
6.2 Model Visualizations
The attention patterns are visualized figure 4 Subfigure (a) is a visualization of the urban grid roadmap, while subfigures (b)-(g) are sampled attention matrices at the 30-th timestamp, coming from the 1st attention head at the last transformer encoder block.
all subfigures in (b)-(g), we display the attention as 4 parts: O the ConeDecay prior 7()). 2 the TimeDecay prior af-) plus the attention LUT [.ttm))....] @ the traditional attention a (-)
residual component, and O the overall attention, which is the summation of the previous three components.
Denote the size of the grid as h x w, then the token number is Ntoken = Tmaa x h x w. The subfigures (f)()) are the Ntoken Ntoken attention matrix itself, coming from the Grid-Bi dataset. The attention matrix in (b)(c) come from the New York dataset, and are sampled in the following way: we pick the certer node nig of the x grid New York, then select all nodes that are in the same vertical line as ng. Then attention ni paid to these 28 nodes in the past Tmaz timestamps, forming a sub-matrix of of shape Tmaz x h (reshaped from the union of multiple discontinuous segments in certain rows of A).
extract the attention from A that correspond to the

---

Tex
Ove alAleee
(a) Scenario illustration
(b) Attention for New York
(c) Attention for New York
(d) ConeDecay y
(c) + attn
(f) Residual a
(g) Overall Attention
Figure 4: Visualized Attention Distributions
As can be seen from the figures, the overall learned attention pattern displays the decay effect both along the cone-device deviatin direction and along the time axis, which is the same as it is designed for. Observing the cone attention in subfigure (b) (the above one), the learned cause decay effect shows asymmetry: the outwards direction (relative future) decays faster than the inwards direction (relative past), which is in line with the underlying Poisson distribution for the arrival of traffic flows. Note that such attention patterns rely on the instant dynamic features, and hence can vary across time, center tokens considered, and across different encoder blocks. Such observations demonstrate the interpretability of our DePT.
Conclusions
In this paper, we targeted to address several major challenges multi-agent CPS control: the large state/action space for centralized methods, and the tension between localized observations and global control targets for decentralized methods. We proposed new transformer-based model for feature learning and control in CPS the Delayed Propagation Transformer (DePT). DePT induces physically-inspreed attention priors into the transformer, which help boost the transformer's 's ability to capture the desired attention patterns. We choose one of the most complicated open-word system traffic network to test the potential of performance of DePT on both synthetic and real open-source datasets, and our result shows that DePT not only converges faster and better than the traditional transformer but also outperforms other expert neural networks for traffic network control.
We several technical and practical impacts of DePT. As successful global-view controller, DePT offers centralized control scheme that has the potential to solve the control problem under limited sensing. On the other hand, DePT is designed for general CPS under the constraint of limited physical propagation speed, hence is ready to plug-in for many other CPS problems, such as electrical power grids, industrial control systems, and the e-commerce delivery systems.
Acknowledgement
Z.W. supported by US Army Research Office Young Investigator Award (W911NF2010240), and a P. Morgan Faculty Research Award.
References
[1] Stefano Zanero. Cyber-physical systems. Computer, (0(4):14-16 2017. 2017.

---

[2] Yang Liu, Yu Peng, Bailing Wang, Sirui Yao, and Zihe Liu. Review on cyber- bobber- systems. IEEE/CAA Journal of Automatica Sinica, 4(1):27-40, 2017.
[3] Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pages 66-83. Springer, 2017.
[4] Thomas Morstyn, Branislav Hredzak, and Vassilios G Agelidis. Cooperative multi-agen control of heterogencous storage devices distributed a de microgrid. IEEE Transactions on Power Systems, 3144):977-2298 2015.
[5] Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li. Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions on Intelligent Trunsporiation Systems, 21 118)):008-11095 2019.
[6] Teodora Sanislav, Sherali Zeadally, George Mois, and Hacène Fouchal. Multi-agent architec- ture for reliable syber-physieet systems (cps). In 2017 IEEE Symposium on Computers and Communications (ISCC), pages 170-175. IEEE, 2017.
[7] Tianshu Chu, Sandeep Chinchali, and Sachin Katti. Multi-agent reinforcement learning for networked system control. arXiv preprint arXiv:2004.01339, 2020.
[8] Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. arXiv preprint arXiv:2006.05222 2020.
[9] Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. International Conference on Learning Representations, 2020.
[10] Frank L Lewis, Hongwei Zhang, Kristian Hengster-Movric, and Abhijit Das. Cooperative control of multi-agent systems: optimal and adaptive design approaches. Springer Science & Business Media, 2013.
[11] Hashim A Hashim, Sami El-Ferik, and Frank L Lewis. aauro-adaptive cooperative tracking control with prescribed performance of unknown higher-order nonlinear multi-agent systems. International Journal of Control, 92(2):445 460, 2019.
[12] Zekun Tong, Yuxuan Liang. Changsheng Sun, David S Rosenblum, and Andrew Lim. Directed graph convolutional network. arXiv preprint arXiv:2004. 13970, 2020.
[13] Xitong Zhang, Nathan Brugnone, Michael Perlmutter, and Matthew Hirn Magnet: A magnetic neural network for directed graphs. arXiv preprint arXiv:2102.1139) 2021.
[14] Veronika Thost and Jie Chen. Directed acyclic graph neural networks. arXiv preprint arXiv:2101.07965, 2021.
[15) Matthijs 'J Spaan. Partially observable markov decision processes. In Reinforcement Learning, pages 387-414. Springer, 2012.
[16] Khac Duc Do. Bounded controllers for formation stabilization of mobile agents with limited sensing ranges. IEEE Transactions on Automatic Control, 52(3):569-576 2007.
[17] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv: 1708.04782 2017.
[18] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint (tto::170003666 2017.
[19] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. arXiv preprint arXiv:2012.07436, 2020.
10

---

[20] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 1055-4064. PMLR, 2018.
[21] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey visual transformer. arXiv preprint arXiv:2012. / 2556, 2020.
[22] Igor V Tetko, Pavel Karpov, Ruud Van Deursen, and Guillaume Godin. State-of thite-of-thhaart aug- mented nlp transformer models for direct and single-step retrosynthesis. Nature communications, 11(1):1-11, 2020.
[23] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint 191iv:111..1000 2019.
[24] Jacob Devlin, Ming-Wiirr Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[25] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagonuyko. End-to-end object detection with transformers. In European Conference Computer Vision, pages 13-2299. Springer, 2020.
[26] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[27] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey, arXiv preprint arXiv:2101.01 169, 2021.
[28] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprini arXiv:2009.06732. 2020.
[29] Seongjun Yun, Minbyul Jeong, Rachyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks. Advances in Neural Information Processing Systems, 32:1 1983-11993, 2019.
[30] Deng Cai and Wai Lam. Graph transformer for graph-to-sequence learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7464-7471, 2020.
[31] N Christ, B Hasslacher, and Alfred H Mueller. Light-cone behavior of perturbation theory. Physical Review D, 6(12):3543, 1972.
[32] Mare Cheneau, Peter Barmettler, Dario Poletti, Manuel Endres, Peter Schauß, Takeshi Fukuhara, Christian Gross, Immanuel Bloch, Corinna Kollath, and Stefan Kuhr. Light-cone like spreading of correlations in a quantum many-body system. Nature, 48 (7382):4844487 2012.
[33] Xiaoyuan Liang. Xunsheng Du, Guiling Wang, and Zhu Han. deep reinforcement learning network for traffic light cycle control. IEEE Transactions on Vehicular Technology, 68(2):1243 1253, 2019.
[34] Wade Genders and Saiedeh Razavi. Using deep reinforcement learning agent for traffic signal control. arXiv preprint 161 142, 2016.
[35] Jakob Stoustrup, Anuradha Annaswamy, Aranya Chakrabortty, and Zhihua Qu. Smart grid control: overview and research opportunities. 2018
[36] M Ourahou, W Ayrir, B EL Hassouni, and A Haddi. Review smart grid control and reliability in presence of renewable energies: Challenges and prospects. Mathematics and Computers in Simulation, 167:19-31, 2020.

---

[37] Sampo Kuutti, Richard Bowden. Yaochu Jin, Phil Barber, and Saber Fallah. A survey of deep learning applications to autonomous vehicle control. IEEE Transactions on Intelligent Transportation Systems, 2020.
[38] Hua Wei, Guanjie Zheng. Huaxiu Yao, and Zhenhui Li. Intellilight: A reinforcement learn- ing approach for intelligent traffic light control. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2405, 2018.
[39] Hua Wei, Chacha Chen, Guanjie Zheng, Kan Wu, Vikash Gayah, Kai Xu, and Zhenhui Li. Presslight: Learning max pressure control to coordinate traffic signals in arterial network. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & 2019.
Data Mining, pages
[40] Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang, Yuanhao Xiong, Kai Xu, and Zhenhui Li. Toward a thousand lights: Decentralized deep reinforcement learning for large- scale traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3414-3421, 2020.
[41] Hua Wei, Nan Xu, Huichu Zhang, Guanjie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. Colight: Learning network-level cooperation for traffic signal control. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 193-1122, 2019.
[42] Hua Wei, Guanjie Zheng, Vikash Gayah, and Zhenhui Li. A survey on traffic signal control methods. arXiv preprint arXiv: 00000400871.... 17, 2019.
[43] Xinshi Zang, Huaxiu Yao, Guanjie Zheng. Nan Xu, Kai Xu, and Zhenhui Li. Metalight: Value-based meta-reiiffrreemet learning for traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1153-1166 1160, 2020.
[44] Afshin Oroojlooy, Mohammadrezz Nazari, Davood Hajinezhad, and Jorge Silva. Attendlight: Universal attention-based reinforcement learning model for traffic signal control. arXiv preprint arXiv:20/0.05772, 2020.
[45] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.001 2021.
[46] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two transformers can make one strong gan. arXiv preprint arXiv:2102.07074, 2021.
[47] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Fukasz Kaiser. Uni- versal transformers. arXiv preprint 180..318,,. 2018.
[48] Chen-Hsuan Lin, Ersin Yumer, Oliver Wang. Eli Shechtman, and Simon Lucey, St-gan: Spatial transformer generative adversarial networks for image compositing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9455-9464, 2018.
[49] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris Kitani. Agentformer: Agent- transformers for socio-temporal multi-agent forecasting. arXiv preprint arXiv:2103.14002. 2021.
[50] Jeevana Priya Inala, Yichen Yang, James Paulos, Yewen Pu, Osbert Bastani, Vijay Kumar, Martin Rinard, and Armando Solar-Lezama. Neurosymbolic transformers for multi-agent communication. arXiv preprini arXiv:2101.03:: 2021.
[51] Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. arXiv preprint 1xXiv:1703.08841 2017.
[52] Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario. In The World Wide Web Conference, pages 8620-3624, 2019.
[53] Peter Koonce and Lee Rodegerdts. Traffic signal timing manual. Technical report, United States. Federal Highway Administration, 2008.
12

---

[54] Pravin Varaiya. The max- maex-preseeee controller for arbitrary networks of signalized intersections. In Advances in Dynamic Network Modeling in Complex Transportation Systems, pages 27-66. Springer, 2013.
13


Reviews:
[{"review_id": "xt2utK-kxfH", "reviewer": "Reviewer_BFpw", "summary": "The authors used transformer to model the spatial-temporal dynamics in a real-world system with information propagation delay, which is claimed to be specialized at global modeling. The transformer design has taken the physical constraints into consideration. It shows promise in learning to control multi-agent systems.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main novelty of this paper is to bake in physics prior in transformers to model CPS. Previous works utilized transformer to model networks or graphs, but usually from non-physical systems and without a notion of information propagation latency. By taking the important physical delay constraint into account, the new transformer has essentially restricted a physically realizable subspace of solutions, which the authors argue to be the key for modeling CPS.  \n\nTo formulate the constraint, the authors designed a cone-shaped temporal-spatial prior, as the feature propagation in physical world process naturally forms the cone shape in the time-space. That prior guide the design of a multi-level heterogeneous attention mechanism. \n\nThe proposed controller achieved the state-of-the-art performance in the traffic signal control task, in a large-scale network with heterogeneous node and edge types. Specifically, the new transformer approach outperformed the state-of-the-art Co-light method, that uses graph attentional networks to model the temporal-spatial space.\n\nOverall, this is a solid paper tackling a worthy problem. The novelty is clear. The writing is also good quality. \n\nQuestions & Suggestions:\n\n(1)\tTransformer belongs to the centralized methods and can be hard to converge due to the huge state and action spaces. Although the authors presented two empirical tweaks that can alleviate training difficulty, I double the transformer approach may still have difficulty to scaling up to large amounts of agents in a real city (currently two experiments had only 36 and 196 agents). On the other hand, graph NNs may scale up better due to its locality-based update and aggregations. \n\n(2)\tGraph NNs also can apply to directional graphs too, e.g., “Directed Graph Convolutional Network”. Therefore, line 43-49 claims are inaccurate.\n\n(3)\tI think “universal” is too high in tone since the authors only demonstrated one application of traffic signal control. Similarly, the authors should avoid many exaggerative yet imprecise/unnecessary buzzwords, such as ““light cone/ Special Relativity”, or “Causality”.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No significant social risk", "review_text": "", "extra_scores": {"ethical_concerns": "Not found", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "esIFeegtB0W", "reviewer": "Reviewer_qqzv", "summary": "The paper presents a transformer-based centralized method to approach the multi-agent control problem in cyber-physical systems (CPS). They formulate the graph-based CPS control as a reinforcement learning problem, and adapt transformer to represent the Q-network by introducing spatial-temporal constraints in computing self-attention. The constraints include a cone-shaped prior for neighboring distribution, a location-associated embedding and time decay effect. To accelerate training of such centralized method, they also propose to retrain three constraints and adopt imitation learning for warm-up. They experiment on one of the CPS scenarios, traffic signal control, and the results show the effectiveness of the proposed method compared to the baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**== Originality ==**  \nThis is the first work to approach CPS control problem via transformer framework and the idea of incorporating the cone-shaped spatial-temporal prior into the self-attention is interesting and new to me. In particular, the authors do not directly transformer on the CPS scenario, but make some improvements of the model in consideration of direction-feature coupling issue in directed graphs and time decay in message propagation. For the training convergence speedup, the pertaining RL agent for warmup is a common technique for RL-based application besides CPS and hence this part is not new to me.\n\n**== Quality ==**  \nIt is appropriate to adopt transformer for the CPS control problem and the modification in terms of the issues in centralized CPS is reasonable. However, I have some concerns regarding the DePT training speedup and the experiment results are not convincing to me.\n\nFor the training speedup, it seems that the slow convergence is partially caused by the proposed spatial-temporal prior constraints introduced in calculating transformer attention, and that's why additional pre-fitting priors are required in Section 4.3. In other words, the proposed acceleration method is only specific to the model in this work, but not generalized to other centralized methods for CPS control. Therefore, the common difficulty of training centralized methods under huge state and action space still cannot be addressed, as stated in the introduction section.\n\nFor the experiment part, there are three major issues. \n1. The model is only evaluated on one scenario of CPS (i.e., traffic signal control) and hence the generalizability of the method under other CPS domains is unknown. Even for the selected traffic signal control task, only one real-world dataset (NYC city) is adopted for evaluation. The results would be more convincing if more real-world datasets (e.g., other cities used in CoLight [36]) are experimented.\n2. The selected baselines are relatively weak. For example, the recent work Attendlight [39] can be considered as a strong baseline to demonstrate the effectiveness of the proposed model. Besides, no decentralized method such as [35][36] is selected for comparison. Since the authors claim that the proposed model can better capture the global-view of CPS system compared to the decentralized methods, it would be better to show it in the experiment.\n3. The experiment analysis on the proposed model is not thorough. Since the major technical contribution of this work is on the modification of transformer, evaluation of other transformers variants on the CPS control is also encouraged. For example, the GraphTransformer [a] may be suitable in the graph-based CPS problem. In addition, the ablation study in Table 2 also fails to consider other centralized methods in evaluating the training convergence. Since the imitation learning is a common technique for any RL-based models, so what if it is applied to CoLight to speed up training? \n\n[a] Yun, Seongjun, et al. \"Graph transformer networks.\" Advances in Neural Information Processing Systems 32 (2019): 11983-11993.\n\n**== Clarify ==**  \nIn general, the paper is well-structured and easy to follow in most of the parts. However, some details in problem formulation and model description should be clarified.\n1. The definition of MDP in CPS is unclear. I believe the detail of traffic signal control problem is not familiarized in the NeurIPS community and hence it would be better to clearly define the MDP formulation, e.g., what is  the reward function.\n2.  In section 4.1, what is the policy embedding P and what is notation E? Since Q learning is adopted to approach the RL problem, why policy is required in this case? Is the observed features f correspond to section 5.1?\n3.  Although the source codes are provided, it is still encouraged to describe implementation detail of the proposed method in the paper.\n4.  There are many typos and grammatical errors in the paper, which should be fixed in the final version.\n  - line 32, \"the\" -> \"The\"\n  - line 78, \"have\" -> \"has\"\n  - line 80, \"learned\" -> \"learn\"\n  - line 113, \"model\" -> \"modeling\"\n  - line 142, \"interact\" -> \"interacts\"\n  - line 157, \"feed\" -> \"feeding\"\n  - line 185, \"consists\" -> \"consists of\"\n  - line 221, \"pre-train\" -> \"per-training\"\n  - line 294, \"feed\" -> \"feeding\"\n\n**== Significance ==**  \nThe paper targets a specific task of traffic signal control rather than a general CPS problem. Although some technical contribution of adapting transformers in the task is novel to me, the evaluation is somehow problematic and the results are not convincing. The work may be only interesting to a small group of audience in the community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address two specific issues regarding how to adopt transformers in approaching the CPS control problem. However, the general limitations brought by centralized methods are still remained to be solved, e.g., as stated in the introduction section, \"it is hard for the centralized methods to converge due to the huge state and action spaces\". It is unclear how the proposed method deals with huge state and action spaces.\n\nIn addition, the motivation of using transformer is kind of vague. It would be better to clearly state the benefits of using transformer in solving this problem. The citations of transformers in line 51 are all about NLP and CV domains. Why transformers are good at removing locality inductive bias and capturing global view in CPS, compared to existing GNN based approaches. Also the direction-feature coupling is not a big issue for GNN, as directed graph neural networks [b][c] have been proposed.\n\n[b] Zhang, Xitong, et al. \"MagNet: A Magnetic Neural Network for Directed Graphs.\" arXiv preprint arXiv:2102.11391 (2021).  \n[c] Thost, Veronika, and Jie Chen. \"Directed acyclic graph neural networks.\" arXiv preprint arXiv:2101.07965 (2021).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Y3c0b8w34lG", "reviewer": "Reviewer_pDRN", "summary": "This paper proposed a new transformer for feature learning and control in CPS, called Delayed Propagation Transformer (DePT). DePT induces cone-shaped spatial-temporal attention priors into the transformer, which help boost the transformer’s ability to capture the desired attention pattern as ruled by immutable physical constraints. The authors demonstrated their new model on a traffic signal control application. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n-\tThis is the first paper leveraging transformer to multi-agent system control as far as I know. \n-\tTo model the information propagation in physical systems, the authors pointed out that the correlation pattern in space-time follows a neighboring distribution of the cone-shaped distribution, which is set as an important prior for the transformer attention.\n-\tThe model seems “universally” applicable to model CPS dynamics, due to being free of inductive bias while being physics aware.\n-\tThe authors conduct thorough experiments on both synthetic and real urban traffic networks and observe new state-of-the-art results. The overall learned attention pattern also displays the desired decay effect.\n-\tThe paper is clear and easy to read. The releases of codes and data are promised.\n\nWeakness\n-\tI am not convinced that transformer free of locality-bias is indeed the best option. In fact, due to limited speed of information propagation, the neighborhood agents should naturally have more impacts on each other, compared to far away nodes. I hope the authors to explain more why transformer’s no-locality won’t make a concern here.\n-\tDue to the above, I feel graph networks seem to capture this better than the too-free transformer, and their lack of global context/ the “over-squashing” might be mitigated by adding non-local blocks (e.g., check “Non-Local Graph Neural Networks” or several other works proposing “global attention” for GNNs). \n-\tThe authors also claimed “traditional GNNs” cannot handle direction-feature coupling: that is not true. See a latest work “MagNet: A Neural Network for Directed Graphs” and I am sure there were more prior arts. Authors are asked to consider whether those directional GNNs can possibly suit their task well too.\n-\tTransform is introduced as a centralized agent. Its computational overhead can become formidable when the network gets larger. Authors shall discuss how they prepare to address the scalability bottleneck. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No significant risk or concern", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "BZQKO0xWfjT", "reviewer": "Reviewer_2aDi", "summary": "Cyber-Physical Systems (CPS), which is used to control and monitor the health of multiple agents and systems, and predict their future health, has a major challenge in effectively implementing multi-agent control. In CPS, an ideal MDP cannot be established due to incomplete coverage/placement of the sensor and limitations of current sensing technology. The author’s purpose is a new transformer-based Delayed Propagation Transformer (DePT) model that can specialize in global modeling of CPS while considering the constraints of the physical world. The transformer-based model gets rid of the agent’s inherent locality restrictions, and learns global views of correlations across the whole system. The author customized a multi-level heterogeneous attention mechanism that has the causality cone prior backed into its design. With such design, the attention between input tokens now has a physical interpretation, hence guiding the controller to learn more effectively in exploring the collaborative strategy across agents. Taking advantage of these strengths, we plan to build a DePT-based CPS controller and use it in a traffic signal control system.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the techniques of CPS and the problem of multi-agent control are well known. Edges in the  CPS  system graph have physical orientations, and these orientations are inherently coupled with node/edge functions and can be translated into completely different meanings. For this reason, the representative problem of CPS control is the difficulty of training due to the physically limited space and noisy CPS data. The authors of this paper have a good understanding of the problems of the conventional transformer model and tried to solve the problem using a unique model called the Delayed Propagation Transformer (DePT). The authors have a good understanding of the problems with existing technologies and have technically described the proposed techniques to solve them in a technically friendly manner. However, it would have been better if the core difference between the existing Transformer model and the DePT model was clearly revealed. With a clear theoretical/experimental approach, the authors leveraged existing data to achieve better results than before. In addition, everything in the paper is well laid out and provides the reader with relevant information.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The simulator is a real product used in the real world or a fake product that has a similar feel to it. Either way, since they are ‘real things’ that exist under the physical environment, physical limitations and operation and maintenance will inevitably require considerable time and cost. The simulators that overcome these physical limitations do not exist yet, so there is always an error between the real world and the virtual world. In order to overcome this limitation, this paper needs to be proven by operating it in a real environment.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "1SvfpiOZZd", "reviewer": "Reviewer_SpfT", "summary": "The paper proposes a novel centralized multi-agent control technique. This technique relies on a novel transformer architecture that has been designed to account for the various domain-specific constraints in the dynamic multi-agent control setting, namely the presence of time delays in signal propagation across the network of agents. The paper details the specifics of this novel architecture and performs experiments on the CityFlow traffic control simulation platform. Various ablation studies analyze the contributions of various parts of the architecture to the overall system performance.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The main contribution of the paper is in the design of the transformer architecture. This is certainly well motivated by the application area in which the model is to be deployed. However, the contribution seems incremental, and in some ways it is similar to common masking schemes that are widely used to prevent \"cheating\" of standard transformer architectures during training. A more thorough literature review of masking techniques and priors that are used in transformer architectures would be helpful context. \n\nQuality: Overall the claims are generally supported, although some further detail could do a great deal to substantiate the novelty and importance of the proposed architecture. For example, some further ablation studies could supplement the existing analysis. Namely, imitation learning is done for all experiments. It is therefore unclear how much marginal benefit is provided by the novel architecture components on top of the baseline of imitation learning. Furthermore, experimental comparison with other priors proposed in the literature could be useful as well. In this sense, the paper appears to be a strong start but is still a work in progress.\n\nClarity: Aside from a few scattered grammar issues and typos, the paper is written clearly and is easy to follow. The organization makes sense for the proposed contribution (i.e. the lack of theoretical guarantees or analysis is expected for this line work).\n\nSignificance: As mentioned above, it is unclear how novel the contribution is. I think the significance of the work will be in in its ability to outperform other methods in benchmark tasks. Currently, the experiments are on a bespoke dataset/simulation platform so it is hard to compare against other widespread techniques. Further experimentation and comparison with existing techniques will take this work further into showcasing the significance of its novel transformer architecture. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Although I do not think it is necessary, the authors may want to more explicitly mention current limitations of the architecture and plans for future work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work has ben extensively discussed by reviewers, and the AC (myself) has also stepped in to review the paper. 

In general this work studies an application of model-free RL, together with a newly proposed transformer architecture with a cone-shaped attention to handle information flow in the temporal-spatial sense. This paper is more of an applied work, with not much algorithmic/theoretical contribution. Still I believe the problem studied is refreshing to the RL community, and the results which are decently good (though not super impressive) when compared with SOTA, demonstrates initial success of applying this ML technique to a more realistic problem. Most importantly, this also shows how transformers can be applied to problems beyond language and small-scale control problems in MuJoCo with novel features in the architectural design, which is new on both transformer design research and the CPS application. The paper is generally well-written, with the newly proposed transformer architecture clearly stated and discussed. On the overall, it's easy to follow and intuitive. 

Several reviewers express concerns on the significance of this work, issues in evaluation, and the value to the broader community. I agree these points are valid, but given the new ideas proposed for transformer-based architecture and the applications to multi-agent systems with these techniques are quite interesting, I think the benefits outweighs the drawback. On the overall, I'd recommend acceptance for this paper.
Decision: Accept (Poster)