T-Graphormer: Using Transformers for Spatiotemporal Forecasting

Authors: Hao Yuan Bai, Xue Liu
Primary area: learning on time series and dynamical systems
Keywords: Deep Learning, Spatiotemporal Forecasting, Transformer, Graph Neural Network

Abstract:
Time series data is ubiquitous and appears in all fields of study. In multivariate time series, observations are interconnected both temporally and across components. For instance, in traffic flow analysis, traffic speeds at different intersections exhibit complex spatiotemporal correlations. Modelling this dual structure poses significant challenges. Most existing forecasting methods tackle these challenges by separately learning spatial and temporal dependencies. In this work, we introduce T-Graphormer, a Transformer-based approach designed to model spatiotemporal correlations directly. Extending the Graphormer architecture to incorporate temporal dynamics, our method updates each node representation by selectively attending to all other nodes within a graph sequence. This design enables the model to capture rich spatiotemporal patterns with minimal reliance on predefined spacetime inductive biases. We validate the effectiveness of T-Graphormer on real-world traffic prediction benchmark datasets, achieving up to 10% reductions in both root mean squared error (RMSE) and mean absolute percentage error (MAPE) compared to state-of-the-art methods.

Decision: reject
Reviewer scores: 1, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1729964448310

thread_type:
official_review

<thread_object>
creator: reviewer_vhlu

<content>
summary:
The paper presents the Temporal-Graphormer (T-Graphormer) framework, which encodes spatial-temporal data in the temporal dimension and employs the attention mechanism to effectively capture complex spatial and temporal relationships. Comprehensive experiments also demonstrate its effectiveness.

strengths:
1. The time series forecasting problem is a valuable and impactful area of research, with significant applications across various fields.
2. This paper introduces a centrality encoding and positional encoding combination approach, with the attention mechanism, to capture complex spatial-temporal correlations. This approach can be easily adapted to various domains.
3. This paper provides a clear presentation and thorough description of the methodology and experimental setup.

weaknesses:
1. This paper claims that T-Graphormer performs well on spatial-temporal data; however, only traffic datasets were used in the experiments, so testing on additional datasets is encouraged. Additionally, the experiments focus primarily on main results and ablation studies, and further experiments, such as hyperparameter analysis, are recommended.
2. This paper would benefit from incorporating more recent citations on state-of-the-art Transformer research, particularly in the baseline comparisons. Adding newer relevant works, such as DSformer [Yu, 2023] and LDT [Feng, 2024], is recommended.
3. Even though detailed information about the implementation are provided, no code provided would still make it hard to evaluate the results.

Reference:
[1] Feng, S., Miao, C., Zhang, Z., & Zhao, P. (2024). Latent Diffusion Transformer for Probabilistic Time Series Forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 38(11), 11979-11987. https://doi.org/10.1609/aaai.v38i11.29085.
[2] Yu, Chengqing, et al. "Dsformer: A double sampling transformer for multivariate time series long-term prediction." Proceedings of the 32nd ACM international conference on information and knowledge management. 2023.

questions:
1. The encoding strategy in the methodology seems primarily focused on a single data dimension. However, spatial-temporal data inherently involves complex interactions between spatial and temporal dimensions. Given that the embedding preprocessing is focused solely on the temporal dimension, how could the current method effectively  capture dependencies across both dimensions?
2. The attention mechanism is indeed effective at capturing correlations, but have you considered testing the encoding strategy with alternative mechanisms, such as replacing attention with a Graph Neural Network (GNN), in ablation studies? This would help address the lack of sufficient Transformer or attention-based baselines for comparison in the experiments.
3. Could you provide a brief summary of how the graph was generated and the adjacency matrix constructed? While I understand the implementation follows another paper, it would be helpful if these details were mentioned directly in your paper for clarity.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730155649608

thread_type:
official_review

<thread_object>
creator: reviewer_nhks

<content>
summary:
In this paper, the authors develop a spatial temporal forecasting/traffic prediction task with transformer-based structure. Unlike other methods, T-Graphformer can capture spatial-temporal correlations directly and it has a better performance than some of the baseline methods.

strengths:
1. Well written related work.
2. Nice figure 2. It clearly explains the model framework.
3. The ablation study is somehow strong and it shows that different model components are useful.

weaknesses:
1. The baseline methods are outdated. Authors need to compared this model with more advanced and latest baselines, instead of SVM and FC-LSTM. Additionally, more datasets should be included. For example, you might include STID, MegaCRN and PDFormer. 
References:
Zezhi Shao, Zhao Zhang, Fei Wang, Wei Wei, and Yongjun Xu. 2022. Spatialtemporal identity: A simple yet effective baseline for multivariate time series forecasting. In Proceedings of the 31st ACM International Conference on Information
& Knowledge Management. 4454–4458.
Renhe Jiang, Zhaonan Wang, Jiawei Yong, Puneet Jeph, Quanjun Chen, Yasumasa Kobayashi, Xuan Song, Shintaro Fukushima, and Toyotaro Suzumura. 2023. Spatio-temporal meta-graph learning for traffic forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 8078–8086.
Jiawei Jiang, Chengkai Han, Wayne Xin Zhao, and Jingyuan Wang. 2023. PDFormer: Propagation Delay-aware Dynamic Long-range Transformer for Traffic Flow Prediction. In AAAI. AAAI Press.
For datasets, more traffic datasets, such as PEMS03, PEMS04, PEMS07 and PEMS08 can be included. It is not sufficient to only use two datasets to demonstrate the effectiveness of your model. 
2. The writing should be improved. The authors use 'spatial temporal forecasting', 'multivariate time series forecasting', 'traffic prediction' in the introduction and section. Although these terms are correlated, they are different and they might be better to focus on one specific task while writing this paper. Since your data is traffic, it is better to frame it as a traffic prediction problem and emphasize it in your introduction. 
3. The methodology itself is not innovative. For example, authors do not make modifications to the traditional transformer structure to make improvement. Many papers adjust the attention mechanism to better capture temporal/spatial correlations. Why you only use the original transformer?

questions:
1. See weaknesses.
2. Why you choose STEP as the baseline? It seems that your model does not have a higher performance.
3. Since your title is spatial temporal forecasting, do you agree that more spatial temporal dataset should be concluded, such as weather dataset and crime dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730608275377

thread_type:
official_review

<thread_object>
creator: reviewer_jisb

<content>
summary:
In this paper, the authors propose an approach called Temporal Graphormer(T-Graphormer). It can models spatiotemporal correlations directly by capturing spatiotemporal correlations and adding special tokens.

strengths:
S1. The author has effectively enhanced Graphormer for spatiotemporal prediction, incorporating improvements in both embeddings and tokens.

S2. T-Graphormer demonstrates superior performance, particularly when forecasting the next 12 time steps (1-hour window).

weaknesses:
Weaknesses:

W1.Key points lack details. While the existing content and formulas in Graphormer are well-articulated, The description of the novel contributions is not sufficiently detailed. In Section 4.2, the paper only presents the modified transformer method for T-Graphormer only in brief with the expression l=T'*N. In Section 4.3, the paper doesn't clearly convey the formulas for special tokens. Could you provide detailed mathematical formulas and subsequent explanations for these original contributions?

W2. Lack of recent related work. The related work mentioned in Chapter 2, Related Works, basically references studies prior to 2020. There's also a lack of references to newer traffic prediction models. Including recent works such as  STAEformer [1]  would strengthen the baselines.

[1]Liu H, Dong Z, Jiang R, et al. Spatio-temporal adaptive embedding makes vanilla transformer sota for traffic forecasting[C]//Proceedings of the 32nd ACM international conference on information and knowledge management. 2023: 4125-4129.

W3. Lack of visualization. While the main claim of the paper is that directly modeling spatiotemporal correlations is better than modeling them separately, there is no accompanying visualization or case study to substantiate this claim, reducing the argument's credibility.

Could you provide a visualization of an attention example for a particular sensor to illustrate that the T-Graphormer indeed learns spatiotemporal patterns effectively compared to previous methods?

Minor remarks:

* In the last paragraph of 4.2, the symbol upright phi appears without prior definition.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687456966

thread_type:
official_review

<thread_object>
creator: reviewer_sdza

<content>
summary:
The manuscript focuses on spatiotemporal forecasting. To simultaneously learn both spatial and temporal correlations, the authors propose T-Graphormer based on the transformer architecture and design temporal and spatial encoding methods for this purpose.

strengths:
Two novel spatial feature extraction methods are proposed in this paper, one for encoding the in-degree and out-degree of the graph structure and the other for encoding the shortest travel distance between graph nodes.

weaknesses:
1. There have been several methods [1][2] that apply transformer architecture to spatio-temporal forecasting. The idea of using transformers to simultaneously capture temporal and spatial correlations is not sufficiently novel.
2. Section 4.3 does not adequately explain the setting of additional tokens. These tokens are typically used in the NLP domain, but the authors fail to clarify their significance in spatio-temporal forecasting or provide details on their implementation.
3. The experimental design has shortcomings. First, the dataset selection is insufficient. The study focuses on spatio-temporal forecasting but only uses two traffic speed datasets to validate the model, which is inadequate to demonstrate the method’s superiority. Second, the experimental baseline lacks comparisons with advanced transformer-based methods [1][2]. Finally, the hyperparameter experiments do not sufficiently explore hyperparameter values.
4. The manuscript's structure is also not well-balanced. Chapter 4, which describes the main method, occupies only one and a half pages, while other sections take up considerably more space.
5. The language presentation is rough. The adjacency matrix dimension in Section 3.1 is incorrect. Section 4.1 consists of only two sentences and lacks a period. Additionally, the summary in Section 6 is confusing and poorly presented.

Reference:
[1] Liu H, Dong Z, Jiang R, et al. Spatio-temporal adaptive embedding makes vanilla transformer sota for traffic forecasting[C]//Proceedings of the 32nd ACM international conference on information and knowledge management. 2023: 4125-4129.
[2] Jiang J, Han C, Zhao W X, et al. Pdformer: Propagation delay-aware dynamic long-range transformer for traffic flow prediction[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(4): 4365-4373.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732278286552

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Phase 1 Comment to Address Some Common Questions from All Reviewers

comment:
We sincerely thank all reviewers for their valuable feedback and thoughtful comments, which have significantly helped us improve our work. Below, we address the common concerns raised across the reviews, including baseline comparisons, dataset selection, and the novelty of our method.

**Baseline Comparisons**

Despite STEP (Shao et al., 2022) having a Transformer-based architecture, we acknowledge the need for more comprehensive comparisons with more recent Transformer-based baselines. In our revised manuscript, we have incorporated advanced models such as PDFormer (Jiang et al., 2023) and STAEformer (Liu et al., 2023) into our experiments. These additions, along with detailed results in Tables 2 and 3, demonstrate that T-Graphormer still consistently achieves state-of-the-art performance on traffic datasets such as METR-LA and PEMS-BAY. These updates strengthen our claims about the effectiveness of T-Graphormer and its ability to model spatiotemporal data holistically.

**Dataset Selection**


We understand the importance of evaluating T-Graphormer across a diverse set of datasets. In our initial submission, we focused on METR-LA and PEMS-BAY due to their prevalence in traffic prediction studies and their challenging properties. However, in response to reviewers' feedback, we are actively expanding our experiments to include additional traffic datasets such as PEMS03, PEMS04, PEMS07, and PEMS08. Furthermore, we are exploring the potential of applying T-Graphormer to other spatiotemporal domains, such as weather and crime datasets, and will include these results in future work.

**Novelty**

We clarify that T-Graphormer introduces significant innovations in spatiotemporal modelling:

* *Unified Spatiotemporal Learning*: Unlike existing Transformer-based methods (e.g., STAEformer, PDFormer) that separate spatial and temporal learning into distinct modules, T-Graphormer treats spatiotemporal data as a flattened sequence. This approach leverages the global attention mechanism of Transformers to simultaneously model interdependencies across both dimensions.
* *Extended Structural Encoding*: While we adopt the centrality and shortest-path-distance encodings from Graphormer, our novel contribution lies in extending these encodings to the temporal dimension as well as adding positional encoding. These are critical enhancements for handling dynamic spatiotemporal data.

These contributions represent fundamental shifts in how spatiotemporal data can be modelled using Transformer architectures, which we believe constitutes a meaningful advance in the field.

**Conclusion**

We have reflected these answers in our revised manuscript. In response to reviewers' requests for visualizing spatiotemporal dependencies, we have also added Figures 3 and 4, showcasing model predictions across time steps for individual sensors. This provides a clearer illustration of T-Graphormer’s ability to capture spatiotemporal patterns. We hope these changes address the concerns raised and demonstrate the novelty, effectiveness, and broad applicability of T-Graphormer.

We look forward to further correspondence and are happy to provide any additional clarifications.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732690771476

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Phase 2 Comment, Updates to Manuscript

comment:
Dear Reviewers and Area Chairs,

Thank you again for your efforts during the review process. Your feedback has been very helpful in shaping our work towards better clarity and rigour. 

On that note, here are some of the changes we made to improve our manuscript:
## Main Text:

* **Related Works and Baselines Comparisons**:

    We included two additional baselines and added their prediction results on the PEMS-BAY and METR-LA datasets in **Table 2**. The table has been reorganized, ranking models based on their performance on the horizon-12 prediction. Additionally, the second-best-performing model for each metric is now underlined for better readability.
* **Overall Writing**:

    Corrected typos. Improved phrasing at necessary parts (e.g. Conclusion) to ensure smoother flow and coherence.


## Appendix:
* **Additional Results**:

    Summarized the T-Graphormer prediction results. Notably, as our training loss is the mean squared error, the most significant improvement of our method over the baselines is observed in the RMSE test metric. Furthermore, the prediction improvements are more pronounced in larger datasets.

* **Figures**:

    * Added **Figures 3, 4** to visualize T-Graphormer's predictions on PEMS-BAY and METR-LA datasets. 
    * Added **Figure 8** to support pre-training potentials mentioned in the conclusion.



Thank you again for your time and feedback. 

We look forward to further active discussions. We are more than happy to provide clarifications or additional details.

Best,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a transformer model that aims to model spatiotemporal correlations directly for forecasting applications. In particular, it extends the Graphormer architecture to incorporate temporal dynamics. The proposed model is evaluated on some traffic prediction benchmark datasets.

Major strengths:
- This work explores a tighter integration of the spatial and temporal dimensions by operating on spatiotemporal data directly.
- The structural encoding techniques in T-Graphormer are extended to the temporal domain for spatio-temporal forecasting.

Major weaknesses:
- The technical contributions of this work are not very novel and significant.
- The original submission only included very few benchmark datasets and baseline methods for comparison, but they were slightly expanded during the rebuttal period.
- A more comprehensive literature review should be conducted, especially to include the very recent works.
- The linguistic quality and presentation of this paper have room for improvement.

We thank the authors for making an effort to address some issues raised in the reviews, which includes conducting more experiments. Nevertheless, we still feel that even the revised manuscript is below the acceptance standard of ICLR for the major weaknesses as summarized above. The authors are encouraged to improve their paper for future submission by considering the comments and suggestions of the reviewers.