Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a novel centralized multi-agent control technique. This technique relies on a novel transformer architecture that has been designed to account for the various domain-specific constraints in the dynamic multi-agent control setting, namely the presence of time delays in signal propagation across the network of agents. The paper details the specifics of this novel architecture and performs experiments on the CityFlow traffic control simulation platform. Various ablation studies analyze the contributions of various parts of the architecture to the overall system performance.
main_review: Originality: The main contribution of the paper is in the design of the transformer architecture. This is certainly well motivated by the application area in which the model is to be deployed. However, the contribution seems incremental, and in some ways it is similar to common masking schemes that are widely used to prevent "cheating" of standard transformer architectures during training. A more thorough literature review of masking techniques and priors that are used in transformer architectures would be helpful context. 

Quality: Overall the claims are generally supported, although some further detail could do a great deal to substantiate the novelty and importance of the proposed architecture. For example, some further ablation studies could supplement the existing analysis. Namely, imitation learning is done for all experiments. It is therefore unclear how much marginal benefit is provided by the novel architecture components on top of the baseline of imitation learning. Furthermore, experimental comparison with other priors proposed in the literature could be useful as well. In this sense, the paper appears to be a strong start but is still a work in progress.

Clarity: Aside from a few scattered grammar issues and typos, the paper is written clearly and is easy to follow. The organization makes sense for the proposed contribution (i.e. the lack of theoretical guarantees or analysis is expected for this line work).

Significance: As mentioned above, it is unclear how novel the contribution is. I think the significance of the work will be in in its ability to outperform other methods in benchmark tasks. Currently, the experiments are on a bespoke dataset/simulation platform so it is hard to compare against other widespread techniques. Further experimentation and comparison with existing techniques will take this work further into showcasing the significance of its novel transformer architecture. 

limitations_and_societal_impact: Although I do not think it is necessary, the authors may want to more explicitly mention current limitations of the architecture and plans for future work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a transformer-based centralized method to approach the multi-agent control problem in cyber-physical systems (CPS). They formulate the graph-based CPS control as a reinforcement learning problem, and adapt transformer to represent the Q-network by introducing spatial-temporal constraints in computing self-attention. The constraints include a cone-shaped prior for neighboring distribution, a location-associated embedding and time decay effect. To accelerate training of such centralized method, they also propose to retrain three constraints and adopt imitation learning for warm-up. They experiment on one of the CPS scenarios, traffic signal control, and the results show the effectiveness of the proposed method compared to the baselines.
main_review: **== Originality ==**  
This is the first work to approach CPS control problem via transformer framework and the idea of incorporating the cone-shaped spatial-temporal prior into the self-attention is interesting and new to me. In particular, the authors do not directly transformer on the CPS scenario, but make some improvements of the model in consideration of direction-feature coupling issue in directed graphs and time decay in message propagation. For the training convergence speedup, the pertaining RL agent for warmup is a common technique for RL-based application besides CPS and hence this part is not new to me.

**== Quality ==**  
It is appropriate to adopt transformer for the CPS control problem and the modification in terms of the issues in centralized CPS is reasonable. However, I have some concerns regarding the DePT training speedup and the experiment results are not convincing to me.

For the training speedup, it seems that the slow convergence is partially caused by the proposed spatial-temporal prior constraints introduced in calculating transformer attention, and that's why additional pre-fitting priors are required in Section 4.3. In other words, the proposed acceleration method is only specific to the model in this work, but not generalized to other centralized methods for CPS control. Therefore, the common difficulty of training centralized methods under huge state and action space still cannot be addressed, as stated in the introduction section.

For the experiment part, there are three major issues. 
1. The model is only evaluated on one scenario of CPS (i.e., traffic signal control) and hence the generalizability of the method under other CPS domains is unknown. Even for the selected traffic signal control task, only one real-world dataset (NYC city) is adopted for evaluation. The results would be more convincing if more real-world datasets (e.g., other cities used in CoLight [36]) are experimented.
2. The selected baselines are relatively weak. For example, the recent work Attendlight [39] can be considered as a strong baseline to demonstrate the effectiveness of the proposed model. Besides, no decentralized method such as [35][36] is selected for comparison. Since the authors claim that the proposed model can better capture the global-view of CPS system compared to the decentralized methods, it would be better to show it in the experiment.
3. The experiment analysis on the proposed model is not thorough. Since the major technical contribution of this work is on the modification of transformer, evaluation of other transformers variants on the CPS control is also encouraged. For example, the GraphTransformer [a] may be suitable in the graph-based CPS problem. In addition, the ablation study in Table 2 also fails to consider other centralized methods in evaluating the training convergence. Since the imitation learning is a common technique for any RL-based models, so what if it is applied to CoLight to speed up training? 

[a] Yun, Seongjun, et al. "Graph transformer networks." Advances in Neural Information Processing Systems 32 (2019): 11983-11993.

**== Clarify ==**  
In general, the paper is well-structured and easy to follow in most of the parts. However, some details in problem formulation and model description should be clarified.
1. The definition of MDP in CPS is unclear. I believe the detail of traffic signal control problem is not familiarized in the NeurIPS community and hence it would be better to clearly define the MDP formulation, e.g., what is  the reward function.
2.  In section 4.1, what is the policy embedding P and what is notation E? Since Q learning is adopted to approach the RL problem, why policy is required in this case? Is the observed features f correspond to section 5.1?
3.  Although the source codes are provided, it is still encouraged to describe implementation detail of the proposed method in the paper.
4.  There are many typos and grammatical errors in the paper, which should be fixed in the final version.
  - line 32, "the" -> "The"
  - line 78, "have" -> "has"
  - line 80, "learned" -> "learn"
  - line 113, "model" -> "modeling"
  - line 142, "interact" -> "interacts"
  - line 157, "feed" -> "feeding"
  - line 185, "consists" -> "consists of"
  - line 221, "pre-train" -> "per-training"
  - line 294, "feed" -> "feeding"

**== Significance ==**  
The paper targets a specific task of traffic signal control rather than a general CPS problem. Although some technical contribution of adapting transformers in the task is novel to me, the evaluation is somehow problematic and the results are not convincing. The work may be only interesting to a small group of audience in the community.
limitations_and_societal_impact: The authors address two specific issues regarding how to adopt transformers in approaching the CPS control problem. However, the general limitations brought by centralized methods are still remained to be solved, e.g., as stated in the introduction section, "it is hard for the centralized methods to converge due to the huge state and action spaces". It is unclear how the proposed method deals with huge state and action spaces.

In addition, the motivation of using transformer is kind of vague. It would be better to clearly state the benefits of using transformer in solving this problem. The citations of transformers in line 51 are all about NLP and CV domains. Why transformers are good at removing locality inductive bias and capturing global view in CPS, compared to existing GNN based approaches. Also the direction-feature coupling is not a big issue for GNN, as directed graph neural networks [b][c] have been proposed.

[b] Zhang, Xitong, et al. "MagNet: A Magnetic Neural Network for Directed Graphs." arXiv preprint arXiv:2102.11391 (2021).  
[c] Thost, Veronika, and Jie Chen. "Directed acyclic graph neural networks." arXiv preprint arXiv:2101.07965 (2021).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Cyber-Physical Systems (CPS), which is used to control and monitor the health of multiple agents and systems, and predict their future health, has a major challenge in effectively implementing multi-agent control. In CPS, an ideal MDP cannot be established due to incomplete coverage/placement of the sensor and limitations of current sensing technology. The author’s purpose is a new transformer-based Delayed Propagation Transformer (DePT) model that can specialize in global modeling of CPS while considering the constraints of the physical world. The transformer-based model gets rid of the agent’s inherent locality restrictions, and learns global views of correlations across the whole system. The author customized a multi-level heterogeneous attention mechanism that has the causality cone prior backed into its design. With such design, the attention between input tokens now has a physical interpretation, hence guiding the controller to learn more effectively in exploring the collaborative strategy across agents. Taking advantage of these strengths, we plan to build a DePT-based CPS controller and use it in a traffic signal control system.
main_review: In general, the techniques of CPS and the problem of multi-agent control are well known. Edges in the  CPS  system graph have physical orientations, and these orientations are inherently coupled with node/edge functions and can be translated into completely different meanings. For this reason, the representative problem of CPS control is the difficulty of training due to the physically limited space and noisy CPS data. The authors of this paper have a good understanding of the problems of the conventional transformer model and tried to solve the problem using a unique model called the Delayed Propagation Transformer (DePT). The authors have a good understanding of the problems with existing technologies and have technically described the proposed techniques to solve them in a technically friendly manner. However, it would have been better if the core difference between the existing Transformer model and the DePT model was clearly revealed. With a clear theoretical/experimental approach, the authors leveraged existing data to achieve better results than before. In addition, everything in the paper is well laid out and provides the reader with relevant information.
limitations_and_societal_impact: The simulator is a real product used in the real world or a fake product that has a similar feel to it. Either way, since they are ‘real things’ that exist under the physical environment, physical limitations and operation and maintenance will inevitably require considerable time and cost. The simulators that overcome these physical limitations do not exist yet, so there is always an error between the real world and the virtual world. In order to overcome this limitation, this paper needs to be proven by operating it in a real environment.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors used transformer to model the spatial-temporal dynamics in a real-world system with information propagation delay, which is claimed to be specialized at global modeling. The transformer design has taken the physical constraints into consideration. It shows promise in learning to control multi-agent systems.
main_review: The main novelty of this paper is to bake in physics prior in transformers to model CPS. Previous works utilized transformer to model networks or graphs, but usually from non-physical systems and without a notion of information propagation latency. By taking the important physical delay constraint into account, the new transformer has essentially restricted a physically realizable subspace of solutions, which the authors argue to be the key for modeling CPS.  

To formulate the constraint, the authors designed a cone-shaped temporal-spatial prior, as the feature propagation in physical world process naturally forms the cone shape in the time-space. That prior guide the design of a multi-level heterogeneous attention mechanism. 

The proposed controller achieved the state-of-the-art performance in the traffic signal control task, in a large-scale network with heterogeneous node and edge types. Specifically, the new transformer approach outperformed the state-of-the-art Co-light method, that uses graph attentional networks to model the temporal-spatial space.

Overall, this is a solid paper tackling a worthy problem. The novelty is clear. The writing is also good quality. 

Questions & Suggestions:

(1)	Transformer belongs to the centralized methods and can be hard to converge due to the huge state and action spaces. Although the authors presented two empirical tweaks that can alleviate training difficulty, I double the transformer approach may still have difficulty to scaling up to large amounts of agents in a real city (currently two experiments had only 36 and 196 agents). On the other hand, graph NNs may scale up better due to its locality-based update and aggregations. 

(2)	Graph NNs also can apply to directional graphs too, e.g., “Directed Graph Convolutional Network”. Therefore, line 43-49 claims are inaccurate.

(3)	I think “universal” is too high in tone since the authors only demonstrated one application of traffic signal control. Similarly, the authors should avoid many exaggerative yet imprecise/unnecessary buzzwords, such as ““light cone/ Special Relativity”, or “Causality”.
limitations_and_societal_impact: No significant social risk
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposed a new transformer for feature learning and control in CPS, called Delayed Propagation Transformer (DePT). DePT induces cone-shaped spatial-temporal attention priors into the transformer, which help boost the transformer’s ability to capture the desired attention pattern as ruled by immutable physical constraints. The authors demonstrated their new model on a traffic signal control application. 
main_review: Strength
-	This is the first paper leveraging transformer to multi-agent system control as far as I know. 
-	To model the information propagation in physical systems, the authors pointed out that the correlation pattern in space-time follows a neighboring distribution of the cone-shaped distribution, which is set as an important prior for the transformer attention.
-	The model seems “universally” applicable to model CPS dynamics, due to being free of inductive bias while being physics aware.
-	The authors conduct thorough experiments on both synthetic and real urban traffic networks and observe new state-of-the-art results. The overall learned attention pattern also displays the desired decay effect.
-	The paper is clear and easy to read. The releases of codes and data are promised.

Weakness
-	I am not convinced that transformer free of locality-bias is indeed the best option. In fact, due to limited speed of information propagation, the neighborhood agents should naturally have more impacts on each other, compared to far away nodes. I hope the authors to explain more why transformer’s no-locality won’t make a concern here.
-	Due to the above, I feel graph networks seem to capture this better than the too-free transformer, and their lack of global context/ the “over-squashing” might be mitigated by adding non-local blocks (e.g., check “Non-Local Graph Neural Networks” or several other works proposing “global attention” for GNNs). 
-	The authors also claimed “traditional GNNs” cannot handle direction-feature coupling: that is not true. See a latest work “MagNet: A Neural Network for Directed Graphs” and I am sure there were more prior arts. Authors are asked to consider whether those directional GNNs can possibly suit their task well too.
-	Transform is introduced as a centralized agent. Its computational overhead can become formidable when the network gets larger. Authors shall discuss how they prepare to address the scalability bottleneck. 

limitations_and_societal_impact: No significant risk or concern
review_title: null

Meta-review: This work has ben extensively discussed by reviewers, and the AC (myself) has also stepped in to review the paper. 

In general this work studies an application of model-free RL, together with a newly proposed transformer architecture with a cone-shaped attention to handle information flow in the temporal-spatial sense. This paper is more of an applied work, with not much algorithmic/theoretical contribution. Still I believe the problem studied is refreshing to the RL community, and the results which are decently good (though not super impressive) when compared with SOTA, demonstrates initial success of applying this ML technique to a more realistic problem. Most importantly, this also shows how transformers can be applied to problems beyond language and small-scale control problems in MuJoCo with novel features in the architectural design, which is new on both transformer design research and the CPS application. The paper is generally well-written, with the newly proposed transformer architecture clearly stated and discussed. On the overall, it's easy to follow and intuitive. 

Several reviewers express concerns on the significance of this work, issues in evaluation, and the value to the broader community. I agree these points are valid, but given the new ideas proposed for transformer-based architecture and the applications to multi-agent systems with these techniques are quite interesting, I think the benefits outweighs the drawback. On the overall, I'd recommend acceptance for this paper.
Decision: Accept (Poster)