GPT-ST: Generative Pre-Training of Spatio-Temporal Graph Neural Networks
Authors: Zhonghang Li
Lianghao Xia
Yong Xu
Chao Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: Spatial Temporal Prediction
Deep Neural Networks
Pre-training Model
TL;DR: A generative pre-training framework for improving the spatio-temporal prediction performance of downstream models.

Abstract: In recent years, there has been a rapid development of spatio-temporal prediction techniques in response to the increasing demands of traffic management and travel planning. While advanced end-to-end models have achieved notable success in improving predictive performance, their integration and expansion pose significant challenges. This work aims to address these challenges by introducing a spatio-temporal pre-training framework that seamlessly integrates with downstream baselines and enhances their performance. The framework is built upon two key designs: (i) We propose a spatio-temporal mask autoencoder as a pre-training model for learning spatio-temporal dependencies. The model incorporates customized parameter learners and hierarchical spatial pattern encoding networks. These modules are specifically designed to capture spatio-temporal customized representations and intra- and inter-cluster region semantic relationships, which have often been neglected in existing approaches. (ii) We introduce an adaptive mask strategy as part of the pre-training mechanism. This strategy guides the mask autoencoder in learning robust spatio-temporal representations and facilitates the modeling of different relationships, ranging from intra-cluster to inter-cluster, in an easy-to-hard training manner. Extensive experiments conducted on representative benchmarks demonstrate the effectiveness of our proposed method. We have made our model implementation publicly available at https://github.com/HKUDS/GPT-ST.

GPT-ST: Generative Pre-Training of Spatio-Temporal Graph Neural Networks
Zhonghang Li
Lianghao Xia²
Yong Xu1.3,4
Chao Huang2*
1 South China University of Technology. University of Hong Kong, PAZHOU LAB, Guangdong Key Lab of Communication and Computer Network {bjdwh. zzh, chachuang7550@gaiilccoo
aka_xia@formail. com
yxudscut edu,
Abstract
In recent years, there has been a rapid development of spatio-temporal prediction techniques in response to the increasing demands of traffic management and travel planning. While advanced end-to-end models have achieved notable success improving predictive performance, their integration and expansion pose signif- icant challenges. This work aims to address these challenges by introducing a spatio-temporal pre-training framework that seamlessly integrates with downstream baselines and enhances their performance. The framework is built upon two key de- signs: (i) We propose a spatio-temporal mask autoencoder for learning spatio-tempone dependencies. The model incorporates customized pa- rameter learners and hierarchical spatial pattern encoding networks. These modules are specifically designed to capture spatio-temporal customized representations and intra- and inter-cluster region semantic relationships, which have often been neglected in existing approaches. (ii) We introduce an adaptive mask strategy as part of the pre-training mechanism. This strategy guides the mask autoencoder in learning robust spatio-temporal representations and facilitates the modeling of different relationships, ranging from intra-cluster inter-cluster, in an easy-to-hard training manner. Extensive experiments conducted on representative benchmarks demonstrate the effectiveness our proposed method. We have made our model implementation publicly available at filhtps:: //t::///ttttt com/ wwt.C/CDs//HHCCC.. HKUDDS/GDTPST
pre-tratee pre-training model
Introduction
Intelligent Transportation Systems (ITS) is a rapidly growing research field driven by advancements in sensors and communications technology [1 12 Spatio-Temporal (ST) prediction, including traffic flow and ride demand, aims to forecast future trends, facilitating decision-making traffic management and risk response [43. and enhancing people's lives with future travel plans [6] Deep learning methods, such as RNNs and CNNs, have been widely adopted to model temporal and spatial correlations [5] 15 50 45). Recently, GNNs have gained attention for spatial and temporal prediction tasks, achieving remarkable results. Initially, researchers built adjacency matrices based on region attributes, such as distance [48 54 Subsequent studies introduced learnable graph structures to
capture spatial correlations [40 39]. while others focused on dynamic relationships between regions, proposing methods for constructing dynamic graphs [4 44). These advancements have significantly contributed to the development of spatio-temporal prediction techniques.
Despite the remarkable results achieved by existing methods in spatio-temporal prediction, there are several issues that remain inadequately addressed. i) Lack customized representations of specific temtoall patterns. Customization can be categorized into two key aspects: time- dynamic and node-specific properties in both the time and spatial domains. Time-dynamic patterns exhibit variations across different time periods, such as contrasting patterns between weekends and
Cooresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

weekdays within the same region. Moreover, correlations between regions dynamically evolve over time, surpassing the capabilittess of static graph representations. Node-specific patterns highlight the distinct time series observed in different regions, rather than shared pattern. Additionally, it important to ensure that different regions retain their individual node characteristics even after message aggregation, to prevent interference from prominent nodes in the spatial domain. We believe that encoding all of these customized characteristics is essential to guarantee the model robustness. However, existing works often fall short in providing comprehensive consideration of these factors. ii) Insufficient consideration of spatial dependencies at different levels. Most approaches primarily focus pairwise associations between regions when modeling spatial dependency, but they overlook the semantic relevance at different spatial levels. In the real world, regions with similar functions tend to exhibit similar apatio-temporal patterns. By performing clustering analysis on different regions, the model can explore common characteristies among similar regions, thereby facilitating improved spatial representation learning. Moreover, there is a lack of adequate modeling of high-level regional relationships across time in current studies. Spatio-temporal patterns between different types of high-level regions may exhibit dynamic transfer relationships. For instance, during work hours, there is a notable movement of people from residential areas to work areas, as illustrated in the right part of Figure [ In this context, changes in people flow within residential areas can provide valuable auxiliary signals for predicting people flow in work areas. It highlights the importance of incorporating both fine- mne-grained and coarse- -raineed correlations among different levels of regions in order to enhance the predictive capabilities of -patooltemporal prediction models.
An intuitive approach to address the aforementioned challenges is to develop an end-to-end model. However, the cur- rent models face difficulties in benefiting from such an approach, as each module in the state-of-the-ar (SOTA) model is intricately refined, and any disassembly and integration may lead to a degradation in prediction performance. So, is there strategy that can allow existing spatio- temporal methods to leverage the advan- tages of an end-to-end model? The an- swer is affirmative. Recent pre-training frameworks, such as ChatGPT [3] (2) in
Weel
Figure 1: The motivations behind GPT-ST. The upper left figure demonstrates that traffic patterns in the same region exhibit variations across different time periods. Similarly, the lower left figure depicts how this variation can occur between different regions during the same time period. In contrast, existing works typically construct static graphs, failing to capture the dynamic nature of region relation- ships over time, shown in the middle figures (lower), where region relationships change dynamically. Lastly, the right figure illustrates the commuting phenomenon between regions belonging to different categories.
the field of natural language processing (NLP) and MAE [5 in computer vision (CV), have been proposed by pioneers and widely studied in their respective do- mains. These pre-training architectures construct unsupervised training tasks, in- volving masking or other techniques, to learn better representations and improve downstream task performance. However, the application of such scalable pre-training models in the field of spatio-temporal prediction has been limited thus far. To address the aforementioned challenges, we propose a novel framework called Generative Pre-Training for Spatio-Temport pre- diction (GPT-ST). This framework is designed seamlessly integrate into existing spatio-temport prediction models, enhancing their performance. Our contributions are summarized as follows:
We present GPT-ST, a novel pre-training framework designed for spatio- temporal prediction. This framework seamlessly integrates into existing spatio-temporal neural networks, resulting in performance enhancements. Our approach combines a model parameter customization scheme with self-supervised masking autoencoding, enabling effective spatio-temporal pre-training.
GPT-ST ingeniously utilizes a hierarchical hypergraph structure to capture spatial dependencies at different levels from global perspective. Through collaboration with thoughtfully designed adaptive mask strategy, the model acquires the capability to model both intra-cluster and inter- cluster spatial relations among regions, thereby generating robust ppatio-temporal representations.
We conduct extensive experiments on real-world datasets, and the demonstrated improvement in the performance of diverse downstream basclines showcases the superior performance of GPT-ST.

---

2 Related Work
Deep Spatio-temporal Prediction Techniques. Numerous studies have focused on designing deep neural networks for spatio-temporal data prediction. To capture temporal dependencies, D- LSTM 49 and GWN [40] have developed predictive frameworks based on Recurrent Neural Networks (RNNs) and Temporal Convolutional Networks (TCNs), respectively. GMAN 55 and
STGNN (6) have utilized temporal self-attention networks to facilitate long-range temporal learning. In spatial relation learning, DMVST-Net
44 adopted convolutional network to model region [23] exploited Graph Neural Networks (GNNs) for message passing between regions. STAN (26] consider the interrelationships between regions based on Graph Attention Networks (GAT). Temporal graph networks serve as a similar research baseline aimed at reasoning about dynamic graph structures, such as CAWs [37 and PINT [33] More recently, some works have explored the combination of ST models with advanced methods, such as Neural-ODE- based [10 [8] and self-supervised learning-bas based (24), for better spatial and temporal dependencies modelling. Compared to these works, our proposed GPT-ST framework aims at empower diverse deep spati-temporal models with the pre-training stage.
associations. STGCN 48) and DCRNN
Pre-trainning methods. Pre- Pre-training models have made significant advancements in recent years. with notable examples including BERT [7]28), vision transformers [9 g] masked autoencoders [15] 20 and language models (3 27 34 The success of the autoencoder- based approach, exemplified by MAE, has demonstrated the powerful expressive capability of the mask-reconstruction pre-training framework. Building upon this, STEP [3T] proposed a pre-training model specifically for time series prediction. In other domains, CMSF 41 introduced a master-slave framework for identifying urban villages. The master model pre-trains region representations, while the slave model fine-tunes specific regions for accurate identification. Researchers have also recognized the need for more efficient mask mechanisms than random masking. AttMask [19] and SemMAE [21] leverage attention mechanisms and semantic information, respectively, to guide the mask operation. AdaMAE (2] proposes adaptively selecting visible signals with greater reconstruction loss, as they contain more information, and using higher proportion of masking strategy accordingly. Despite the significant progress of pre- pre-training models in the fields of natural language processing (NLP) and computer vision (CV), their application spatio-temporal prediction remains relatively limited. Additionally, computational efficiency has been a major bottleneck for transformer architectures in general.
supervied learning methods for GNNs. In recent years, self-supervised learning methods for graph data have gained significant attention [ 38 Contrastive learning-basee GNNs generate different views of the original graph through data augmentation techniques [42 29 A loss function is then employed to maximize the consistency of positive sample pairs while minimizing the consistency of negative sample pairs across views. For example, GraphCL 47 generates two views of the graph by applying node dropping and edge shuffling, and performs contrastive learning between them. Another research direction focuses generative graph neural networks, where the graph data itself serves as a natural supervisory signal for representation learning through reconstruction. GPT- GNN (17 conducts pre-training by reconstructing graph features and edges, while GraphMAE [16) utilizes node feature masking in both the graph encoder and decoder to reconstruct features and learn graph representations. However, spatio-temporal forecasting tasks require simultaneous consideration of complex temporal evolution patterns and spatial correlation mechanisms. Pre-training paradigms specifically designed for such tasks are still an area of exploration and research.
3 Preliminaries
Opatio-Temporee Data X. To capture the ST information, we represent it as three-way tensor X E R RxTXF where R, T. and F correspond the numbers of regions, time slots, and feature dimensions, respectively. Each entry Xy,t,/ is the f-th feature of the r-th region in the t-th time slot.
Spatio-Temporal Hypergraphs. We employ ST hypergraphs for ST modeling. A hypergraph H = (V. &, H) is composed of three parts: i) Vertices V = (vr, r E R, E T), each of which represents region in specific time slot f. iii H hyperedges & = {ei, ez each of which connects multiple vertices to reflect the multipartite region-wise relations (e.g.. all residential regions can be connected by one hyperedge). iii) The vertex-hyperedg connections II E NxH where N denotes the number of vertices. To fully excavate the potential of hypergraphs in region-wise relation learning, we adopt a learnable hypergraph scheme where H is derived from trainable parameters.
3

---

Pre-training Paradigm. Our GPT-ST framework aims at developing a pre-trained ST represen- tation method that facilitates the accuracy on downstream ST prediction tasks such as traffic volume forecasting. As depicted in Figure 2 the workflow of GPT-ST can be divided into the pre-training stage and the downstream task stage. We formulate the two stages follows:
Pre-training Stage
Downstream Tasks Stage
Eutoenooder
comporlltemporll Neural Networks
MMM
Mask Autolnooder
The pre-training stage of our GPT-ST framework adopts the masked autoencoding (MAE) task as the train- Figure 2: Overall workflow of GPT-ST. ing objective. The goal of this MAE task is to reconstruct masked information of the spatio-temporal data according to the unmasked ones by learning a ST representation function f. For the training ST
data during the period of [1x 1 we aim to minimize the following objective:
C ((1 M) X titttt W W.f f(M @ 0 XIK-
where L denotes the measure of prediction deviation, M {0, 1} represents the mask tensor, and O denotes the element-wise multiplication operation. W denotes a predictive linear layer.
ii) After the pre-training stage, the results of GPT-ST are then utilized in the downstream task stage. The pre-trained model yields high-quality ST representations to facilitate the downstream prediction tasks such traffic forecasting. Specifically, the downstream stage formulated as:
Ç
Xix- = s(S, Xtx -x411))
where { refers to the representation generated by f according to the historical spatio-temporal data from the previous L time slots before the K-th time slot. The output is the predictions for the next P time slots. Various existing spatio-temporal neural networks can serve the prediction function g.
Methodology
This section introduces the technical details of the proposed ST pre-training framework GPT-ST. develops customized temporal encoder and hierarchical spatial encoder to obtain large modeling capacity for the pre-trained model. An adaptive mask strategy proposed to facilitate effective MAE training. Figure 3 demonstrates the model design of our pre-trained spatio-temporal encoding model.
4.1 Customized Temporal Pattern Encoding
Initial Embedding Layer. We begin by constructing an embedding layer to initialize the represen- tation of the spatio-temporal data X. The original data undergoes normalization using the Z-Score function 1 [39] and is then masked using the mask operation. Subsequently, a linear transformation is applied to augment the representation as E,,- = M,, @ X Eg. where E,, E Rd, M,, E RR", and X,, E I represent the representation, mask operation, and normalized spatio-temporal data for the r-th region in the (-th time slot. The variable d denotes the number of hidden units. Additionally, Eg E RR Fxd represents the learnable embedding vectors for the feature dimensions.
Temporal Hypergraph Neural Network. To facilitate global relation learning, we employ the hypergraph neural network for the temporal pattern encoding, specifically as follows:
= =F, W. + b. + E,); E, HyperPropagate(F,, (F,) oH, a (HH,,..,
(3)
where rr, F,, E, E Rxd denote the result, intermediate, and initial region embeddings for the 1-th time slot, respectively. W, Rdxd b, Rd represent the 1-th time slot-specific parameters. (-) denotes the LeakyReLU activation. The intermediate embeddings E E IR lxxxxd are calculated by the hypergraph information propagation. It employs the region-
hypergraph H, E H to propagate the temporal embeddings E, RT'd for the r-th region along the connections between time slots and H hyperedges, so as to capture the multipartite relations among time periods.
Customized Parameter Learner. To characterize the diversity of temporal patterns, our temporal encoder conducts model parameter customization for both different regions and different time periods. Specifically, the aforementioned time-specific parameters W b, and the region-speciio hypergraph

---

Parameter Learner
Hypergraph Capsule Clustering Network
Cross-dass Relation Learning
Time-dynamtn parameters
Region-speciic parameten
Hjst
M
Sperio
Customized Temporal Pattern Encoding
E. egil eit, K.))
Figure 3: The detailed framework of the proposed GPT-ST.
parameters H, are generated through a learnable process, rather than directly utilizing independent parameters. Specifically, the customized parameters are learned as follows:
- W. = d. W: b d, b; d = ML ( e1 + (w)) e2)
where H E Idd Hdd' XT W E Rdd xdxd b E Rd are independent parameters for the three generated parameters, respectively. C, d, E Rd' denote representations for the r-th region and the i-th time slot, respectively. c, for r E R is a free-form parameter, while de for t E T is calculated from the normalized time-of-day features z!d) and day-of-week features _!"²) e1 E Rdd are their corresponding learnable embeddings. This parameter learner realizes spatial and temporal
customization by generating parameters according to the features of specific time slots and regions.
4.2
Hierarchical Spatial Pattern Encoding
4.2.1 Hypergraph Capsule Clustering Network
Current spatial encoders primarily focus on capturing local adjacent region- -iise relations while disregarding the widespread similarities that exist between distant regions. For example, business districts that are geographically apart can still exhibit similar ST patterns. In light of this, our GPT-ST introduces hypergraph capsule clustering network to capture global region-wise similarities. This network explicitly learns multiple cluster centers as hyperedges, characterizing the global region-wise similarities. To further enhance the hypergraph structure learning, we incorporate the dynamic routing mechanism of capsule network. This mechanism iteratively updates hyperedge representations and region-hyperedee connections based on semantic similarities. As result, it improves the clustering capability of hyperedges and facilitates the global modeling of dependencies among regions.
Specifically, we firstly obtain a normalized region embedding Vr. E Rd for each region in time slot t, using the previous embedding rr., and the squash function (30). This embedding is then used to calculate the transferred information Valr.e _d from each region each cluster center (hyperedge) i, within the t-th time slot. Formally, these two variables are calculated by:
|
2
x
Vr, = squash( (VT, + c); Valrst = squash(H Ha Vr.e) @ Vr.ei squash(x)
=
1
X 2
||]|
(5)
where V E ddd and e E Id are free-form learnable parameters. The hypergraph connectivity matrix H E I * records the relations between R regions and Hs hyperedges as the cluster centroids. It is tailored to the t-th time slot using the aforementioned customized parameter learner, specifically by H. softmax(d, (d,"T H'). Here, d' and H' are temporal features and hypergraph embeddings.
Iterative Hypergraph Structure Learning. With the initialized region embeddings v., and hyper- graph connection embeddings Vilra, we follow the dynamic routing mechanism of capsule network to enhance the clustering effect for the hyperedges. The j-th iteration is described as follows:
S DR= C. Vilrai
exp(())..)) Cun. = E. Exp((, bl, 62, bl,n. + vT, squash
(6)
where Sq, E IRd represents the iterative hyperedge embedding. It is calculated utilizing the iterative hyperedge-region weights Gi,,, E The weight Ctr.t turn calculated from the hyperedge em- beddings S.,, from the last iteration. With this iterative process, the relation scores and the hyperedge representations are mutually adjusted by each other, to better reflect the semantic similarities between regions and the spatial cluster centroids represented by hyperedges.
5

---

After the iterations of the dynamic routing algorithm, to make use of both b and H. for better region-hyperedd relation learning, GPT-ST combines the two sets of weights and generates the final embeddings S E Rd. We begin by replacing bi,rt with (bi,r + H. to obtain new weight vector Ct E R. and then utilize Cr,,, E TR to calculate the final embeddings 8,,-
4.2.2 Cross-Cluster Relation Learning
With the clustered embeddings Si,,, we propose to model the inter-cluster relations via a high-level hypergraph neural network. Specifically, the refined cluster embeddings S E R x Txd are calculated by message passing between the Hs cluster centroids and Ни high-level hyperedges as follows:
S HyperPropagate(S) = squash( o (H" " 5)) S)
where S E HsTd denotes the reshaped embedding matrix obtained from Si, for i E Hs and t E T. H" E R HHTT denotes the high-level hypergraph structure, which is obtained through the aforementioned personalized parameter leamnr. This parameter customization aggregates the temporal features 2t 7 (w) for all E 7 the input, and generates the parameters as Eq
After refining the cluster representations S,, Rd for i Hs, E T, we propagate the clustered embeddings back to the regional embeddings with the low-level hypergraph structure, as follows:
r.t = o EE Cs W + b" rra))
where rr, IRd denotes the new region embedding for the r-th region in the f-th time slot. Cr,,t E R denotes the weight for the low-level hypergraph capsule network. W RRddd and b" E IR denotes the region-specific transformation and bias parameters generated by the customized parameter learner.
4.3 Cluster-aware Masking Mechanism
Inspired by semantic-guided MAE (21), we design cluster-aware masking mechanism to enhance the intra- and inter-cluster relation learning for our GPT-ST. The adaptive masking strategy incorporates the foregoing learned clustering information Cr.r, to develop easy- sasy-to-hard masking process. In particular, at the beginning of the training, we randomly mask portion of regions for each cluster, in which case the masked values be easily predicted by referring to in-cluster regions that share similar ST patterns. Subsequently, we gradually increase the masking proportion for certain categories, to increase the prediction difficulty for these clusters by reducing their relevant information. Finally, we completely mask the signals of some clusters, facilitating the ability of cross-cluster knowledge transfer for the pre-trained model. This adaptive masking process is depicted in Figure
However, directly utilizing the learned cluster information Gj.a to generate the cluster-aware mask M is not feasible. This is because the cluster information calculated by deeper layers of our GPT- ST network, while the generated mask is required as input to the network (M @ X). To address this, we employ a two-layer MLP network with customized parameters to predict the learned results of G.r,t- In particular, the transformations and bias vectors in this MI.P
Cluster-aware Masking Strategy
Caieguyy
Uniaasked
Masked
From easy-to-hrad hard
network replaced with lime-dynamie and node-specific parameters generated by the customized parameter learner (Eq[4). 4) Subsequently, a linear layer and softmax(-) function are used to obtain the predictions Dirr, E IR for i E Hs,r E R2 E T. To optimize the distribution of Birr, we utilize the KL divergence loss function Lus with the ground truth C. It is important to note that the backpropagation of ēt,,. prevented in this step. The category is the maximum probability according to a. is taken as the classification result. See Algorithm
Figure 4: Illustration for the adaptive masking strategy.
in A.2 for more details.
5 Evaluation
In this section, we verify the effectiveness of GPT-ST by answering the following five questions:
RQ1: How does our proposed pre-training framework GPT-ST benefit different downstream baseline methods, in terms of prediction accuracy diverse spatio-temporal prediction tasks?

---

RQ2: How effective are the designed technical modules of our GPT-ST framework?
RQ3: How the interpretation abilitty of the learned global region clusters and the inter-cluster transition patterns in the hierarchical spatial pattern encoding of our GPT-ST framework.
RQ4: How efficient is GPT-ST in both the pre-training stage and the downstream task stage?
RQ5: How does varying the mask ratio setting affect the performance of GPT-ST?
5.1 Experimental Setting
To evaluate the effectiveness of our proposed method, we conduct experiments on four real-world ST datasets including: PEMS08 32), METR-LA [33, NYC Taxi and NYC Citi Bike [46], which records the traffic flow, traffic speed, taxi order records, and bike order records, respectively.
We choose Mean Absolute Error (MAE), Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE) as the evaluation metrics, which are widely used in ST prediction tasks (52). The lower value of the metrics, the better performance of the model. Following previous works 13 32 [2]] the number of time slots L is set to 12. We divide METR-LA dataset into training set, validation set and test set in a ratio of 7:1:2 and 6:2:2 for others datasets. The final model parameters are selected according to the optimal effect (minimum MAE value) of the validation set.
For optimization, in the pre-trainning phase, the absolute enror loss function L, is used to optimize the parameters and a hyperparameter À is adopted to balance the weight of C, and Lu-- In the downstream task stage, we firstly fuse the embedding C and the raw signal X as the input to the downstream model, and the optimization strategies are distinct for different downstream models. Detailed descriptions of datasets and evaluation presented in the supplementary material
5.2 Main Results (RQ1)
In this section, we mainly investigate whether GPT-ST improves the forecasting for downstream tasks. To achieve this, we evaluate both the original performance and enhanced performance (w/ GPT-ST) of different baselines on four datasets, and the results shown in Table[] Note that since DMVSTGGN, STMGCN and CCRNN are specially designed for demand forecasting, we climinate them in traffic forecasting tasks. The results indicate that the proposed model significantly improves the prediction performance of different downstream baselines on all datasets, which confirms the effectiveness of the framework. We analyze the promotion effect of GPT-ST from three dimensions:
Consistent improvements by GPT-ST. We observe that GPT-ST improves different types of methods (e.g., GNN-based, or attention-based models), and such positive effect does not partialize to certain category baselines, which verifies the generalization ability of GPT-ST. We attribute this improvement to the MAE pre-training with the awareness of intra- and inter-cluster ST relations.
Difference among baseline models. When compared to recently proposed methods like MSDR, our GPT-ST exhibits more substantial improvements when applied to classical baselines such as STGCN. Specifically, in traffic flow prediction using the PEMS08 dataset, MSDR shows improvements of 0.55 in MAE, 0.60 in RMSE, and 0.2 % in MPAE, while STGCN demonstrates improvements of 1.61 in MAE, 2.71 in RMSE, and 0.63% MPAE. One possible explanation for these findings is that advanced baselines like MSDR are already well-designed and comprehensive in modeling various factors. Consequently, they are capable of encoding abundant knowledge independently, which may diminish the utility of the additional signals provided by the pre-training model. Conversely, classical baselines like STGCN may derive greater benefits from the insights offered by our GPT-ST due to their simpler design and potentially limited capacity capture complex relationships.
Comparison with pre-training methods. In addition to evaluating our pre-training method against seatio-temporal prediction methods without pre-training, we also compare it with the competitive pre- are-training baseline STEP. This method utilizes long-term time series as input for pre-training to enhance the performance of the downstream model (GWN). Despite our GPT-ST using only short-term data during the pre-training stage, it outperforms STEP on several metrics. Furthermore, our GPT-ST demonstrates an even larger performance advantage in scenarios where long-term data is insufficient, e.g., the taxi and bike demand prediction tasks. This highlights the broader applicability of our proposed GPT-ST framework, emphasizing its ability to excel in various settings.
7

---

Table 1: Overall performance comparison different datasets in terms of MAE, RMSE and MAPE.
Dataset
PEMSO8
METR-LA
Nacia RMSE
NYC
Model
Metries
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
MAPE
MAE
RMSE
MAPE
STGCK
17.85
28.64
11.08%
3.55
7.39
10.02%
5.68
11.55
38.10%
2.04
51.10%
W/ ST
16.24
25.93
10.45%
3.29
6,85
9.14%
5.09
10.05
35.21%
1.82
2.82
49.15%
DMVSINET
14.25
69.86%
2.27
61.39%
w/ SP
5.27
9.18
40.56%
1.89
2.86
55,01 %
TGCN
21.44
31.82
15.67%
3.70
1.28
10.63%
19.39
83.30%
2.23
62.90%
w/ ST
17.34
26.53
12.26%
3.16
6.41
8,88%
6.88
13.80
51.84%
2.00
3.22
58.84% %
ASTOCN
17.55
26.97
11.78%
3.57
7.13
6.54
52.17%
2.06
55.18%
w/ ST
15.89
24.87
10.28%
3.10
6.26
8.74%
5.90
10.38
45.78%
1.90
2.93
53.60 %
GWN
15.29
24.39
9.98%
3.15
6:44
8.67%
42.19%
1,50
2.988
53.00%
w/ SP
14.72
24.12
9.42%
3.02
6.18
8,29%
5.12
8.89
37.49%
1,80
2.70
$1.69 %
STMGUN
12.20
49.38%
1,96
55.99%
w/ ST
5.18
9.000
38.38 %
1.81
2.73
52.95% %
MIGGN
15.28
24.70
9.72%
3.12
6.44
8.60%
5.15
36.02%
1.84
51.55%
w/ ST
14.93
24.46
9.67%
2.99
6.19
8.21%
4.90
8.55
34.97%
1.78
2.67
51.16%
STSCON
17.82
27.88
11.49%
3.58
1.36
9.95%
6.02
10.98
46.87%
2.04
56. 56.73%%
w/ SP
16.29
26.14
10.59%
3.16
6.51
8.74%
5.00
8.77
36.28% %
1.88
2.79
55.23 %
STEGNN
17.13
27.32
11.10%
3:29
6.69
9.14%
5.91
10.64
46.32%
2.03
56.31%
w/ ST
15.89
25.78
10.53%
3.13
6.47
8.84%
4.97
8.69
36.37%
1,86
2.74
54.84% 9%
STGODE
17.73
27.53
11.39%
3:47
7:02
9.91%
12.14
55.03%
2.08
36.47%
W/ ST
15.68
24.84
10.46%
3.19
6.44
9.30%
5.61
9.91
43.09%
1.90
2.89
52.86 %
CCRNN
5.41
38.7%%
1,92
52.44%
W/ ST
5.08
9.47
37.04%
1.82
2.77
51.33%
MSDR
16.47
25.63
10.34%
3.19
6.42
8.83%
10.45
42.33%
1,93
53.44%
w/ SP
15.92
25.03
10.33%
3.07
6.27
8.50%
5.27
9.33
38.90%
1.82
2.74
52.78 %
STWA
16.16
25.80
10.35%
3.19
6.36
8.68%
38.08%
50007
W/ ST
15.29
24.50
9.81%
3.09
6.32
8.59%
5.06
8.82
36.63% %
1,83
2,80
49.91 %
GWN
13.97
22.94
9.28%
2.95
5.99
8.01%
5.18
9.08
35.75%
2.02
3.12
55. 16%
w/ STEP
14.08
23.33
9.07%
2.92
5.91
7.98%
5.33
9.24
36, 36%
2.02
3.12
53.86%
w/ GPT-ST
13.96
22.83
16%
2.92
5.97
7.93%
5.07
8.82
34.97%
1,98
3,07
51.44%
5.3 Model Ablation Study (RQ2)
This section investigates the impact of major components in our GPT-ST. To evaluate the impact of different components, we re-conduct pre-training for multiple ablated variants, and evaluate the performance of the downstream method using the newly pre-trained ablated models. GWN is utilized as the downstream approach, and two datasets METR-LA and NYC Taxi are employed for evaluation. The results are shown in Figure 5 From the results, we make the following observations:
Impact of basic components. i) -P. We remove the customized parameter learner in GPT-ST. ii) -C. We disable the hypergraph capsule clustering network to cancel the clustering operation. In this case, the spatial hypergraph NNs works on fine-grained regions directly. iii) -T. We leave the cross- cluster relation learning unexplored by removing the high-level spatial hypergraph. The noticeable performance decay demonstrates the positive benefits brought by all three components, indicating that generating personalized ST parameters and modeling the intra- and inter-cluster ST dependencies can effectively capture complex ST correlations to benefit the predictions. Among the three components, the removal of the hypergraph capsule clustering network causes the most significant performance drop. This is because the clustering results also play an important role in many other components, including the cross-cluster dependencies and the cluster-aware adaptive masking.
Impact of mask mechanism. i) Ran0.22 and Ran0.75. We replace our adaptive mask strategy with random masking with mask ratio of 0.25 (mask ratio same as ours) and 0.75 (mask ratio used in MAE [I5] and STEP BII,, respectively. The results consistently demonstrate that our proposed mask strategy outperforms the random mask strategies. This can be attributed to the fact that our mask strategy effectively promotes the learning of intra- and inter-cluster relationships by GPT-ST, resulting in the generation of high-quality representations. Similar results are obtained in additional experiments conducted on the other two datasets, as shown in Table 2 ii) GMAE and AdaMAE. We compare our approach to two variants using the mask strategies proposed by GraphMAE [16] and AdaMAE 22). Both variants perform worse than our methods, highlighting the importance of considering spatial and temporal patterns in the masking strategy. This further confirms the superiority of our adaptive masking approach that leverages clustering information.
Impact of pre-training strategy. To further investigate the effectiveness of the masssreccnsttuuction pre-training approach in spatio-temporal pre-training, we compare it to other pre-training methods: local-global infomax and contrastive pre-training. We used DGI (35] and GraphCL 47 as baselines, which are widely recognized pre-training strategies for GNNs. By replacing GPT-ST's pre-training strategy with these methods (denoted as "r/"), we observe improvements in performance compared to
8

---

models without pre-training. This indicates the adaptability and benefits of infomax and contrastive pre-traiming for representation learning in our model. However, our approach, which utilizes the mask- reconstruction task, achieved the most significant performance enhancement. This can be attributed to the higher correlation between the mask rronsttccntt task and the downstream regression task, leading to more effective learning of spatio-temporal representations. Additionally, our adaptive mask strategy plays a crucial role in facilitating the model to learn robust spatio-temporal representations by increasing the difficulty of the pre-training task.
Ran0 25
Pan0.75
GMAE
AGOMAE
DG
GCL
Ours
Table Comparison of Ran & Ada mask.
Dataset
PEMSINMMAF RMSE)
model
GWN
MSDR
STWA
Ran mask
14.30
16.50 25.63
15 15.397 24.85
mask
14.72/
25.03
24.50
Dulaset
NYC Ciü BIKGCMAE/ RMSE)
model
DMVSTNET
STMGCN
CORNN
mask
192/289 189 2,86
18222744
L 2,86
Figure 5: Ablation study of GPT-ST.
Ada mask
L8U 2,73
1882/ 2.77
5.4 Investigation on Clustering Effect (RQ3)
To comprehensively demonstrate the effectiveness of the clustering process, we assess the inter- pretability of our GPT-ST framework by analyzing the embeddings generated by the Hypergraph Capsule Clustering Network (HCCN), as illustrated Figure We employ the T-SNE algorithm to visualize the high-dimensional embeddings produced by HCCN, mapping them to 2-dimensional vectors. Each category represented by a distinct color, with 10 categories defined for hyperparam- eter Hs. The region clustering is determined based on the probabilities of belonging to different categories. Upon examining the visualized embeddings, we observe that regions belonging to the same class exhibit tight clustering within confined space, providing empirical evidence of the robust clustering capability of our hypergraph capsule clustering network.
In another case study conducted on the METR-LA dataset, we investigate the in-cluster region rela- tions derived from the hypergraph capsule clustering network, as well as the cross- cross-class dependencies obtained from the cross-clusted hypergraph network. The results, illustrated in Figure 7 demonstrate
that the top-3 regions within the same category, as depicted in Figure ((a) and [b] exhibit similar traffic patterns, indicating shared functionalities. For instance, regions near commercial areas (7()) experience an evening peak, while those near residential areas (7b)) remain relatively stable. These observations align with real-world scenarios. Furthermore, focusing on Figure [cc] we analyze the top-2 regions from two categories that have undergone shifts in traffic patterns over a specific time period, while sharing similar hyperedge weights in the cross-class transition. The results re- veal that regions undergoing pattern shifts exhibit distinct traffic patterns while maintaining close interconnections within short driving distance. These findings provide further evidence of the cross-cluster transition learning's 's ability to capture semantic-leve relationships between regions, reflecting real-world traffic scenarios. These advantages contribute to the generation of high-qu nighquuaity representations in our GPT-ST framework, leading to improved performance in downstream tasks.
5.5 Model Efficiency Study (RQ4)
In this section, we assess the efficiency of our model. Specifically, we measure the per-epoch training time of all methods the PEMS08 dataset, and
Table 3: Computational time cost investigation.
Model timelepach(6)
Model
timelepach(3)
STGCN
1.72
GWN
9,87
the results are summarized in Table 3 To ensure
w/ WPG-TST
12.52
w/ GPT-ST
13.46
fairness, all experiments are conducted on system equipped with a GTX 3090 GPU and an Intel Core 19-129 CPU. The batch size for all methods, ex-
ASTGCN
12.47
TGCN
7.63
w/ W/PPSTT
18.20
w/ GPEEST
14.04
MIGNN
9.40
STSOCK
18.71
WCPTST
13.02
w/ GPFEST
21.43
STFGNN
16.03
STGODE
22.93
cept those tagged with 'b8', is set to 64 (where 'b8'
GPTSST
19.54
w/ GPT-ST
19.45
denotes batch size of 8). By combining the results from [a we can conclude that while most of
MSDR
29.15
STWA
46.72
GPT-ST
33.23
w/ GPT-ST
49.90
Downsiream Magge
training stage
the baselines experience minor decrease in train- ing efficiency when enhanced by GPT-ST, there is significant improvement in their performance. For instance, some previous methods (e.g., STGCN) equipped with GPT-ST achieve comparable perfor-
GWN 68
210.98
STEPMN
327.78
w/ STEP w/ GPT-ST b8
425.12
GPT-ST b8
40.18
220.29
GPT-ST
12,50

---

Figure 6: Visualized embbeddings of the hypergraph capsule clustering network.
(a) Case A for learned clusters
(b) Case B learned clusters
(c) Case for cross-class transitions
Figure 7: Case study for the spatial encoder. (a) and (b) present the real traffic conditions and their geographic locations belonging to two specific categories learned by the hypergraph capsule network;
(c) illustrates the learned traffic transition relation between two categories of regions.
mance to advanced methods (e.g.. MSDR), while maintaining superior training efficiency. In the comparison between GPT-ST and STEP, we observe that STEP incurs a substantial time cost, both in the pre-training stage and the downstream task stage, whereas GPT-ST enhances existing baselines in a lightweight manner. These cases demonstrate that our framework achieves a win-win situation in terms of performance and training efficiency, making it well-suited for practical applications.
5.6 Model Hyperparameter Experiment (RQ5)
To examine the impact of different hyperparameters on GPT-ST, we conduct parameter experiments on the NYC Taxi dataset using GWN the downstream model. The results of these experiments are depicted in Figure 8 where the evaluation metrics are plotted relative values.
MAE
RMSE
In our investigation, we specifically focus on the total mask ratio r; and adaptive mask ratio Fa as the main parameters of interest. Based on the findings presented in Figure X we observe that the optimal effect is achieved when the total mask ratio is set to 0.25. Increasing the mask ratio beyond this value does not improve the model's ability to learn better representations; instead, it leads to a decline in prediction performance. Furthermore, we explore the impact of the adaptive mask ratio r,, which determines the number of clusters that are entirely masked, thereby influencing the entire training process. The best performance is obtained when the adaptive mask ratio is set to 1. This suggests that this specific configuration enhances the model's ability to learn regional relations, both within individual clusters and across different clusters.
(a) Total mask ratio
RMSE
0.55
0.50
0.75
1.30
(b) Adaptive mask ratio ma
Figure 8: Hyperparame- ter study of my and m.-
6 Conclusion
In this work, we introduce a scalable and effective pre-training framework tailored for spatio-temporal prediction tasks. Our framework starts with foundational pre-training model that focuses on cap- turing apatio-tempora dependencies. It leverages customized parameter learner and hierarchical hypergraph networks to extract customized spatio-temporal features and region- wise semantic associ- ations, respectively. To further improve the model's performance, we propose an adaptive masking strategy. This strategy guides the model to learn inferential reasoning capabilities by considering both intra-class and inter-class relations during the pre-training stage. We conduct extensive experi- ments on four real-world datasets, demonstrating the efficacy our proposed GPT-ST framework in enhancing the performance of downstream baselines across different spatio-temporal tasks.
10

---

References
[1] Lei Bai, Lina Yao, Can Li, Xianzhi Wang. and Can Wang. Adaptive graph convolutional recurrent network traflic forecasting. NeurlPS, pages 17804-17815, 2020.
Wele Gedara Chaminda Bandara, Naman Patel, Ali Gholami, Mehdi Nikkhah, Motilal Agrawal, and Vishal M. Patel. Adamae: Adaptive masking for efficient spatiotemporal learning with masked autoencoders. In NeurlPS, 2022.
[3] Tom B. Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared Kaplan, Prafulla Dhariwal, et al. Language models are few-shot learners. In NeurlPS, 2020.
[4] Razvan-Gabried Cirstea, Bin Yang. Chenjuan Guo, Tung Kieu, and Shirui Pan. Towards spatio- temporal aware traffic time series forecasting. In ICDE, pages 2900-2913, 2022.
[5] Zhiyong Cui, Kristian Henrickson, Ruimin Ke, and Yinhai Wang. Traffic graph convolutional recurrent neural network: A deep learning framework for network-scale traffic learning and forecasting. TITS, pages 4883 4894, 2020.
[6] Rui Dai, Shenkun Xu, Qian Gu, Chenguang Ji, and Kaikui Liu. Hybrid spatio-temporal graph convolutional network improving traffic prediction with navigation data. In KDD, pages 3074-3082, 2020.
[7] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre rre-training of deep bidirectional transformers for language understanding. In NAACI, pages 4171-4186, 2019.
[8] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: general vision transformer backbone with cross-shaped windows. In CVPR, pages 22124-12144, 2022.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR.
2021.
[10] Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. In KDD, pages 364-373, 2021.
[11] Xu Geng, Yaguang Li, Leye Wang, Lingyu Zhang, Qiang Yang. Jieping Ye, and Yan Liu. Spatiotemporal multi-graph convolution network for ridc-hailing demand forecasting. In AAAI, pages 3656-3663, 2019.
[12] Kan Guo, Yongli Hu, Yanfeng Sun, Sean Qian, Junbin Gao, and Baocai Yin. Hicrarchical graph convolution network for traffic forecasting. In AAAI, pages 151 159, 2021.
[13] Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial-temporal graph convolutional networks for traffic flow forecasting. In AAAI, pages 922-929, 2019.
[14] Liangzhe Han, Bowen Du, Leilei Sun, Yanjie Fu, Yisheng Lv, and Hui Xiong. Dynamic and spatio-temporal deep learning for traffic speed forecasting. In KDD, pages 547-555, 2021.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders 2022.
are scalable vision learners. In CVPR, pages
[16] Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong. Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self supersupe masked graph autoencoders. In KDD, pages 594-604, 2022.
[17] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre- pre-training of graph neural networks. In KDD, pages 186777 2020.
[18] Ming Jin, Yu Zheng, Yuan-Fang Li, Siheng Chen, Bin Yang, and Shirui Pan. Multivariate time series forecasting with dynamic graph neural odes. TKDE, pages 1-14, 2022.
[19] Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. What to hide from your students: Attention-guided masked im- age modeling. In ECCV, pages 318, 2022.
[20] Chaoliu Li, Lianghao Xia, Xubin Ren, Yaowen Ye, Yong Xu, and Chao Huang. Graph transformer for recommendation. arXiv preprint 4rXiv:2306.02330. 2023.
[21] Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. SemMAE: Semantic-guidee masking for learning masked autoencoders. In NewrIPS, 2022.
11

---

[22] Mengzhang Li and Zhanxing Zhu. Spatial-temporal fusion graph neural networks for traffic flow forecasting. In AAAI, pages 4189-4196, 2021.
[23] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: data-driven traffic forecasting. In ICLR, 2018.
[24] Zhonghang Li, Chao Huang, Lianghao Xia. Yong Xu, and Jian Pei. Spatial-tempora hypergraph self- supervised learning for crime prediction. In ICDE. pages 2984-2998 2022.
[25] Dachuan Liu, Jin Wang, Shuo Shang, and Peng Han. Msdr: Multi-step dependency relation networks for spatial temporal forecasting. In KDD, pages 1042-1050, 2022.
[26] Yingtao Luo, Qiang Liu, and Zhaocheng Liu. Stan: recommendation. In WWW, pages 2177-2181 2021.
attention network for next location
[27] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, et al. Training language models to follow instructions with human feedback. arXiv preprint urXiv:2203.02/555 2022.
[28] Zhaopeng Qiu, Xian Wu, Jingyue Gao, and Wei Fan. U-bert: Pre- tre-training user user representatioon for improved recommendation. In AAAI, number 5, pages 4320-433 4327, 2021.
[29] Xubin Ren, Lianghao Xia. Jiashu Zhao, Dawei Yin, and Chao Huang. Disentangled contrastive collabora- tive filtering. arXiv preprint arXivv2305.02759, 2023.
[30] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In NeurlPS, 2017.
[31] Zezhi Shao, Zhao Zhang, Fei Wang, and Yongjun Xu. Pre-training enhanced apatial-tempore graph neural network for multivariate time series forecasting. In KDD, pages 1567-1577. 2022.
[32] Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. Spatial-tempont synchronous graph convolu- tional networks: A new framework for spatial-temporal network data forecasting. In AAAI, pages 914-921, 2020.
[33] Amauri Souza. Diego Mesquita, Samuel Kaski, and Vikas Garg. Provably expressive temporal graph networks. In NeurlPS, pages 32257-32269, 2022.
[34] Jiabin Tang, Yuhao Yang, Wei Wei, Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. arXiv preprint arXiv:2310. 13023, 2023.
[35] Petar Veličkovié, William Fedus, William L. Hamilton, Pictro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In International Conference on Learning Representations, 2019.
[36] Xiaoyang Wang, Yao Ma, Yiqi Wang. Wei Jin, Xin Wang, Jiliang Tang, Caiyan Jia, and Jian Yu. Traffic flow prediction via spatial temporal graph neural network. WWW, pages 1082-1092, 2020.
[37] Yanbang Wang, Yen-Ye Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representation learning in temporal networks via causal anonymous walks. In ICLR, 2021.
[38] Wei Wei, Chao Huang, Lianghao Xia, and Chuxu Zhang. Multi molti-modal self self- supervised learning for recommendation. In WWW, pages 790- 800, 2023.
[39] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In KDD, pages 753-763, 2020.
[40] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In IJCAI, 2019.
[41] Congxi Xiao. Jingbo Zhou, Jizhou Huang, Hengshu Zhu, Tong Xu, Dejing Dou, and Hui Xiong. A contextual mlaver-slave framework urban region graph for urban village detection. In ICDE, pages 736-748, 2023.
[42] Dongkuan Xu, Wei Cheng, Dongsheng Luo, Haifeng Chen, and Xiang Zhang. Infogel: Information-aware graph contrastive learning. NewrIPS, pages 30414-30425, 2021.
[43] Huaxiu Yao, Xianfeng Tang, Hua Wei, Guanjie Zheng, and Zhenhui Li. Revisiting spatial-temport similarity: A deep learning framework for traffic prediction. In AAAI, pages 5668-5675, 2019.
12

---

[44] Huaxiu Yao, Fei Wu, Jintao Ke, Xianfeng Tang. Yitian Jia, Siyu Lu, Pinghua Gong, Jieping Ye, Didi Chuxing, and Zhenhui Li. Deep multi-view spatial-temporal network for taxi demand prediction. In AAAI, pages 2588-2595,, 2018.
[45] Junchen Ye, Leilei Sun, Bowen Du, Yanjie Fu, Xinran Tong. and Hui Xiong. Co-prediction of multiple transportation demands based on deep
neural network. In KDD, pages 305-313, 2019.
[46] Junchen Ye, Leilei Sun, Bowen Du, Yanjie Fu, and Hui Xiong. Coupled layer-wise graph convolution for transportation demand prediction. In AAAI, pages 4617 4625, 2021.
[47] Yuning You. Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurIPS, pages 3812-5823, 2020.
[48] Bing Yu, Haoteng Yin, al. Spatio-temporal graph convolutional networks: deep leamning framework traflic forecasting. IJCAI, 2018
[49] Rose Yu, Yaguang Li, Cyrus Shahabi, Ugur Demiryurek, and Yan Liu. Deep learning: A generic approach for extreme condition traffic forecasting. In SDM, pages 777-785, 2017.
[50] Jinlei Zhang, Feng Chen, Zhiyong Cui, Yinan Guo, and Yadi Zhu. Deep leamnngg architecture for short-term passenger flow forecasting urban rail transit. TITS. pages 7004-7014, 2021.
[51] Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. AAAI, 2017.
[52] Qianru Zhang, Chao Huang, Lianghao Xia, Zheng Wang, Zhonghang Li, and Siuming Yiu. Automated spatio-temporal graph contrastive learning. In WWW. pages 295-305, 2023.
[53] Zaixi Zhang, Qi Liu, Hao Wang, Chenggiang Lu, and Chee-Kong Lee. Motif-hased graph self-supervised learning for molecular property prediction. NeurIPS, pages 15870-15882, 2021.
[54] Ling Zhao, Yujiao Song, Chao Zhang, Yu Liu, Pu Wang, Tao Lin, Min Deng, and Haifeng Li. T-gen: temporal graph convolutional network traffic prediction. T7TS, pages 3848-3858, 2020.
[55] Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. Gman: A graph multi-attention network for traffic prediction. In AAAI, pages 1234-1241, 2020.
13

---

A Supplementary Material
In the supplementary material, we provide additional information and details A.1 This section covers the introduction of data, key parameter settings, comparisons with baselines, optimization methods, and the algorithm process of our method. Furthermore, A.2 presents supplementary experiments for our model, including visualization experiments and replication studies. Additionally, we discuss the reasons behind utilizing hypergraphs as the temporal encoder in A.3 Finally, the limitations and broader impacts of our work are discussed in A.4
A.1 Data and Implementation Details
Data. The statistical information of the aforementioned four real-world datasets is presented in These datasets primarily consist of daily spatio-temporal statistics in the United States. Specifically, the PeMS08 and METR-LA datasets were collected from roads in the San Bernardino area and the road network of Los Angeles County, respectively. On the other hand, the NYC Taxi and NYC Citi Bike datasets were obtained from New York City.
Table 4
Parameters. The latent representation dimensionality (d) and the customized parameter (d") are both 64 and 16, respectively. The number of hyperedges (Hr, Hs. HM) is 8, 10, and 16, respectively. We perform 2 dynamic routing iterations. Additionally, the balance ratio (A) for C, and Ckt is 0.1, and the total mask ratio (r+) is 0.25.
Table 4: Statistical Information of Experimental Datasets.
Dataset PEMS08
Data Record
Node
Time Steps
Sample Rate
Sample Date
traffic flow
170
17856
5min
I/Jul/2016 0/////006-1//ug/2100 31/Aug/2016
METR-LA
traffic speed
207
34272
5min
1/Mar/2012 30/Jun/2012
NYC Taxi
taxicab records
266
4368
30min
1/Apr/2016 33//Jn/20166
NYC Citi Bike
bike orders
250
4368
30min
1/Apr/2016 30/Jun/2016
Baselines. We selected 13 methods as baselines and categorized them into distinct groups:
Hybrid spatio-temporal prediction method:
DMVSTNET [44]: This framework for demand forecasting utilizes RNNs, convolutional networks, and fully connected layers. RNNs capture temporal correlation, convolutional networks handle spatial correlation, and fully connected layers address regional semantic correlation.
Spatio-temporal prediction method based on GNNs:
STGCN 48
It employs convolution operations model both temporal and spatial dependencies.
GWN 40
This method utilizes learnable graph structure capture spatial dependencies and incorporates the diffuse convolution technique to capture temporal dependencies.
GWN* To evaluate the effectiveness GWN, we utilize long-term time series data as input. Our evaluation focuses predicting the data for the next hour using the data from the preceding two weeks. We employ a simple linear layer to process the long-term features in this prediction.
TGCN (54] In this method, both RNNs and GNNs are employed to model temporal dependence and capture spatial correlation, respectively.
This method utilizes learnable graph structure to model associations between multiple variables. incorporates dilated convolution and skip connections to jointly capture spatio-temporal dependencies.
MTGNN 39
This model proposes a variant of RNNs to make full use of historical time step information, and combines GNNs to model long-range spatio-temporal dependence.
MSDR 25
STMGCN [IT]: This method is temtooal framework designed for demand forecasting. It incorporates region association from various aspects and combines the power of RNNs and GNNs to effectively model both spatial and temporal relations.
This is approach specifically designed for traffic demand prediction. It leverages multiple layers of GNNs, each assigned with different adjacency matrices, to capture hierarchical spatial associations. RNNs employed to establish temporal dependencies within the network.
CCRNN 46

---

STSGCN [32]
This work captures pptio-tempore correlations by constructing a local spatio- temporal graph, enabling the synchronous modeling these correlations.
P2 This work proposes data-driven approach that utilizes gated convolution method to generate spatio-temporee graphs. By learning spatial and temporal dependencies, the approach effectively captures the correlations within the data.
STEGNN
Attention-based spatio-temporal prediction method:
ASTGCN 13 : It utilizes attention and GNNs to capture spatio-temporal periodic dependencies.
STWA 4 integrates location -speccifc and time-varying parameters into the attention network to effectively capture dynamic spatio-tempord correlations.
Spatio-temporal prediction via differential equation:
STGODE [TO]
This method captures spatial dependencies by enhancing GNNs through the use of ordinary differential equations. Additionally, temporal dependencies are modeled using a dilated temporal convolutional network.
A.1.1 Optimization Method
In the GPT-STfrramwwrkk multiple temporal encoders (Section 4.1 and spatial encoders (Section 4.2) are stacked together to generate the final embeddings. The architecture includes two temporal encoders followed by a spatial encoder, forming a spatio-temporal (ST) encoding block. The final embeddings are generated by passing the data through two spatio-temporal encoding blocks.
During the pre-training phase, the predictions Y R RxTXF are computed by applying linear layer to the hidden dimension. To optimize the parameters, the absolute error loss function is utilized, following the approach employed in prior works such [55
P C, RIFT: ΣΣΣ
R
T
- 1=1
YY..1)
(9)
Let X represent the input, consisting of F features across R regions in the previous T time slots. The two layers of the MLP network discussed in Sec 4.3 can be formalized as follows:
= ( W W? + by))
(10)
In the given formulation, W ddd and by E R2 represent the region-specific parameters, while W E gdxd and bt Id are the time-dynamic parameters created as in Equation 4 The predictions E RR xRXT are obtained from Q through a linear layer followed by a softmax function. The KL
divergence loss function Lal can be formulated as follows:
R
T
H
Cu
Ciraa (log Ci,,, - log (..r.))
(11)
In this case, E E H. considered as the ground truth classification result. To balance the contribution of the two loss functions, we introduce parameter À to adjust their ratio, as follows:
L - L, + AL#
(12)
In the downstream task stage, utilize the pre-trained embeddings Ç E Rxxxxx along with the raw data representation E' X eo (without any mask operation) as the input for the downstream model. To fuse these two inputs, we employ simple gated fusion layer proposed by (55) which can be formalized as follows:
H = -1-(+(11)) E; == Ö(C + EW bb)
(13)
In the given formulation, W1.1, W,.2 E Bddd and ba E Rd are learnable parameters. The variable represents the gate operation, and 5(-) denotes the sigmoid activation function. It is important to note that we prevent the backpropagation of Ç at this stage. By fusing Ç and E' using the gated fusion layer, downstream models can leverage the knowledge gained during the pre-training stage for improved predictions. The specific optimization methods employed may vary depending on the
downstream models, such as mean absolute error [| (25 Huber loss [22 32).

---

A.1.2 Algorithm Process
Algorithm represents the process of the adaptive mask strategy described in Section 4.3
On the
other hand, Algorithm illustrates the algorithmic process during the pre-training stage.
Algorithm 1: Adaptive Mask Strategy
Input: ST data X RRXIXxF maximum epoch number E, total mask ratio rt, number of X elements J Output: mask matrix M E [RxxTxx
for 1 to do
Calculate the classification results from X according to Sec 4.3
Calculate adaptive mask ratio by
E)²
Calculate the number of masked elements by me Jr,, and then obtain the adaptive masked number me and random masked number my by m2 = Jra: my my ma
Randomly select n categories from the classification list until the total numher of elements of these categories is greater than Mask all the elements of the first n -1 categories, and randomly mask the elements of the n-th category with the residual masked number in m.
Randomly mask the remaining elements with random masked number mer
end
Algorithm 2: Learning Process of GPT-ST Framework
Input: Spatio-temporal data X E RRxXXF, mask matrix M E RRXXXF, dynamic routing iterations number R, maximum epoch number E, learning rate
Output: Trained parameters +
Initialize all parameters +
for 1 to E do
Calculate approximate classification results according to Eq 10 and then generate the mask matrix M according [
Mask the elements with then calculate initial representation E of masked traffic data for each region in each time slot
Calculate the raw temporal features d, according to Eg 4 and initialize the free-form region embedding matrices
Encode the temporal traffic pattern with T integrating customized parameters and e, into the temporal hypergraph neural network according to Eq
Generate the normalized region embedding v and calculate the transferred information Vijna E IRd from each region to each cluster center (hyperedge) i according to Eq S
for r,, R do
Perform the dynamic routing algorithm characterize semantic similarities between the regions
(low-level capsules) and the spatial cluster centroids (high-level capsules) according to Eq
10 end
Generate the final cluster embedding for cross-class relationships learning
Generate the personalized high-level hypergraph structure and conduct the reshaped embedding š to generate š model inter-olasses relationships according to Eq
13
Conduct the customized low-level hypergraph structure to propagate the clustered embeddings s back to the regional embeddings Y according to Eq
14
Make predictions and calculate the absolute error loss L, according to Eq
15
Calculate the KL divergence loss Lxt based on Eg 11
16
Calculate the final loss L. according Eq
for E @ do
18
0 7 aL/00
19
end
end
21
return parameters
A.2 Additional experiments
A.2.1 Visualization Study
Figure TigurePpresent presents the reconstruction results of the model during the pre-training stage. the figure, gray, red, and blue lines respectively represent the masked signals, predicted values, and visible

---

signals. The results demonstrate that GPT-ST can accurately predict the masked signals based on the visible signals, regardless of whether random mask or an adaptive mask is used. Furthermore, can be observed that the masked signals generated by the adaptive strategy exhibit greater temporal consistency compared to those generated by the random strategy. This because the category attribute of region is less likely to change within short period of time. The continuous mask and cluster mask are effective in increasing the difficulty of the reconstruction task, enabling GPT-ST to learn robust spatio-temporal representations even with low masking ratios.
Figure 10 summarizes the enhancement effect your model on downstream baselines. The figure showcases the predicted performance of four baselines on the PEMS08 dataset, where the blue, green, and red lines respectively represent the ground truth, the original performance, and the enhanced performance of the baselines. With the assistance of GPT-ST. the prediction performance of the baseline models experiences significant improvements on certain subsets, as highlighted by the red box. This confirms that the pre-trained model, equipped with customized parameter learners and mechanisms for encoding intra- and inter-class spatial patterns, can provide a discriminative and semantic representation for downstream tasks. As result, it effectively compensates for the limitations different baselines.
(a) Case A for Ran mask
(b) Case A for Ada mask (c) Case B for Ran mask
(d) Case B for Ada mask
Figure 9: Visualization experiments of reconstruction performance in the pre-training stage.
(a) STGCN (b) TGCN
(c) GWN
(d) MSDR
Figure 10: Visualization experiments of different baselines in the downstream stage.
A.2.2 Replication Study
In this section, the deviation of GPT-ST's performance over random parameter initialization is investigated. The statistical results are presented in Table [S] For each performance evaluation on the PEMS08 dataset, we randomly selected 5 seeds. The results indicate that GPT-ST exhibits strong adaptability to different parameter initialization settings, consistently providing stable and high-quality spatio-temporal representations for downstream models.
Table 5: Replication study on PEMS08 dataset.
Model
MAE
RMSE
MAPE
Model
MAE
RMSE
MAPE
STGCN
17.99 14
28.5640.18
11.22=0.24
GWN
15.3740.28
24.61
9,7420.15
GPT-ST
16.4020.16
26.27:0.29
10.62=0.20
w/ GPT-ST
14.7740.05
24.19:0.08
9.52+0.07
ASTGCN
18.05+0.30
27.76:0047
11.40:0.26
TGCN
21.47:0.04
31.82+0.03
16.06s0.25
w/ GPT-ST
16.83+0.50
26.40=0.71
10.78=0.45
w/ GPT-ST
17.33+0.08
26.50:0.11
12.3940.11
MTGNN
15.34+0.04
24.55+0009
9.7240.04
STSGCN
17.9640.
28. 19+0.02
11.65+0.17
w/ GPT-ST
14.9620.05
24.47=0.10
9.6620.04
w/ GPT-ST
16.2820.07
26.05+0.09
10.5120.08
STFGNN 17.20+0.07
27.55+0.18
11.09=0.10
STGODE
17.8140.06
27.6480.08
11.3340.07
w/ GPT-ST
15.9020.03
25.8620.06
10.42=0.16
w/ GPT-ST
15.89+0.16
25.05+0.16
10.59#0.32
MSDR
16.5240.16
25,70 19
10.69
STWA
15.80&0.2 22
25. 1640.6
10. 15.000
GPT-ST
15.96:0.06
25.09:0.08
10.33+0.08
w/ GPT-ST
15.30+0.05
24.6420.12
9.9920.16

---

A.3 Analysis of Hypergraph as Temporal Encoder
Existing time encoders in spatio-temporal prediction schemes, such as recurrent neural networks (RNNs), temporal convolutional networks (TCNs), and attention mechanisms, have certain limitations. RNNs are prone losing long-term information due to the vanishing gradient problem. TCNs can only capture temporal information within a limited neighborhood defined by the size of the convolution kernel. Although attention mechanisms consider the interrelationship between time steps, their computational efficiency decreases significantly when the length of the time series (T) is large, resulting time complexity of O(T²). These limitations hinder the ability of existing approaches to effectively capture and model the temporal dynamics in spatio-temporal data.
To address the aforementioned limitations, propose to utilize the hypergraph neural network as a temporal encoder for time-dependent modeling. A hypergraph is composed of multiple sets of hyperedges. where each hyperedge serves as learnable information hub connecting time-step information with different weights. By treating different hyperedges as a means of integrating temporal relationships across different dimensions, the hypergraph neural network offers a flexible control over complexity by adjusting the number of hyperedges. This approach allows for modeling long-term temporal dependencies while maintaining low computational costs. The hypergraph neural network combines the advantages of capturing temporal dynamics effectively and efficiently, making a promising solution for spatio-temporal prediction tasks.
A.4 Limitations and Broader Impacts
The proposed GPT-ST has demonstrated its effectiveness in improving the prediction performance of downstream models. However, it also has two main limitations: i) Task-specific pre-training GPT-ST requires pre-training for each specific downstream task. This is because different prediction tasks often have distinct data formats and distributions, necessitating task-specific pre-training. For instance, GPT-ST praitrdined on task A cannot be directly applied to task B. ii) Increased time cost: Both the pre-training process and the enhancement process the downstream task stage add to the prediction time cost. These additional computations can impact real-time applications or scenarios with stringent time constraints. future research, we aim to explore more generalized and versatile ppatio-temporal pre- pre-training frameworks, along with lightweight algorithms, to address the task-specific pre- pre-training requirement and further reduce the computational overhead.
5


Reviews:
[{"review_id": "VvhGsZcTvK", "reviewer": "Reviewer_H5wT", "summary": "This paper introduces a spatio-temporal pre-training framework called GPST (Generative Pre-training framework for Spatial Temporal prediction) to enhance the performance of existing models in traffic management and travel planning. The goal is to address the challenges of integrating and expanding refined models, while achieving better predictive performance.\n\nThe framework consists of two main components. (1) a spatio-temporal knowledge extractor that utilizes personalized parameter learners and hierarchical hypergraph networks. These modules are designed to model personalized representations and semantic relationships between regions, which have been overlooked in previous work. (2) an adaptive mask strategy is proposed to guide the knowledge extractor in learning robust spatio-temporal representations.\n\nThe paper conducts extensive experiments on representative benchmarks and demonstrates the effectiveness of the proposed GPST method.", "questions": "Thanks for clarifying the above mentioned weaknesses.", "limitations": "NA", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The two research questions are interesting. Question (1) in personalized information extraction is well studied in STGNN. Question (2) is relatively novel since region functions could make a difference but not considered too much in the literature.\n2. The experiments results look promising. ", "weaknesses": "1. Related works should include SSL works from both Graph and Multivariate Time Series. I think current version has some missing references in esp. Graph SSL, which could be related to the strategy developed in this paper (needs further justification)\n2. Figure 3 is really small to understand clearly. Visual illustration is better decoupled into text description.\n3. Section 4 has too many details that could be moved to appendix since these detailed designs are not quite novel, though important. I'd suggest this section to be more compact and highlighted.\n4. I have some concerns about the improvement by using GPST. It is clear in Table 1 that some methods benefit a lot and some not that much. I expect a more detailed analysis on this. Is some design of GPST redundant in GWN, MTGNN, etc? What is the core part of GPST that makes performance better? Or, is GPST dependent of certain architecture designs? How are these two factors (architecture and ssl strategy) decoupled? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Response to Reviewer H5wT***  \n**Comment 1:** The inclusion of the suggested related work.  \n**Response:** Thank you for your suggestion. The self-supervised learning methods for graph networks and multivariate time series forecasting will be incorporated into the related work in our revised version. Due to character limitations, we have included the discussion of all related works mentioned by the reviewers in the global response. Please refer to the author  rebuttal to access it.  \n  \n**Comment 2:** Explanation of figure size and corresponding text description.  \n**Response:** We sincerely apologize for reducing the size of Figure 3 due to page limit. We will adjust it to its regular size in our revision. In addition, all textual descriptions for the figures will be added and updated. In instance, for Figure 1:  Upper left figure shows that the traffic patterns of same region in different time period behaves difference, while this situation also occurs in different regions during the same period (lower left); the middle figures show the relationship between regions changes dynamically over time (lower), while most of the existing works only construct static graphs (upper).The right figure shows the migration relationship between regions of different categories.  \n\n**Comment 3:** Further discussion of the organization in Section 4.  \n**Response:** Thank you for your valuable feedback. We appreciate your suggestions and will make the necessary adjustments to the structure and description of Section 4 to highlight the key points of our model. Specifically, to emphasize the key elements in each section, we will simplify the personalized parameter operations mentioned in Sections 4.2-4.4 by referencing their standard forms and moving their specific implementations to the supplementary material. Additionally, the specific details of the embedding layer mentioned in Section 4.2, which involve conventional operations, will also be included in the supplementary material. In Section 4.3, we will highlight the concept of Intra- & Inter-class Spatial Patterns Encoding and focus on describing the key module for modeling semantic correlations among multiple regions. The fundamental operations related to capsule networks will also be streamlined (e.g., Equation (6)).  \n  \n**Comment 4:** Further analysis of the performance improvement over the baseline GPST.  \n**Response:** (1) Explanation of the performance improvement of GPST on different baselines. For the mentioned GWN and MTGNN, GPST shares some similar insights with these baselines, as all approaches adopt learnable modules to model spatial correlations. In GWN and MTGNN, learnable node embeddings are utilized to generate graph structures and learn node associations, while GPST employs a learnable hypergraph structure combined with capsule networks to capture region associations. Particularly, GPST considers the spatial-temporal semantic correlations in urban scenes, providing effective compensatory signals for GWN and MTGNN. Furthermore, experimental results demonstrate that with the inclusion of GPST, GWN and MTGNN exhibit faster loss reduction during training, as shown in the table below.  \n  \n| model/epoch | 5 |  10 |  20  |\n|:-----:|:-----:|:-----:|:-----:|\n| GWN | 19.79 | 18.67 | 17.14 |\n| w/ GPST |19.77 | 17.61 | 15.94 |\n| MTGNN | 17.84 | 16.75 | 16.21 |\n| w/ GPST |16.94 | 15.93 | 15.05 |\n  \n(2) Does GPST rely solely on a specific module? From ablation study (Section 5.3), we observed that the personalized parameter learner, Intra- & Inter-class spatial patterns encoding and the adaptive mask strategy are all crucial modules in GPST, and they collectively contribute to its performance improvement. The injection of discriminative parameters can help the model to better capture the spatio-temporal relationship in different periods and different regions. Intra- & Inter-class spatial patterns encoding empowers the model with the ability to aggregate the information from regions with similar functionalities in the global perspective and discover inter-class correlation patterns. The adaptive mask further promotes Intra- & Inter-class association learning. These three components complement each other, contributing to the optimal performance achieved by GPST.  \n  \n(3) GPST exhibits decoupling and correlation in model architecture and training strategy. On one hand, model architecture and training strategy are distinct components with different responsibilities. The training strategy serves as a guide for the model architecture by employing masked signals to direct the learning process. The model architecture, acting as a feature generator, encodes complex spatio-temporal features driven by the training task. On the other hand, their common objective is to generate effective spatio-temporal representations. The proposed adaptive masking strategy is designed based on the model architecture, guiding the model to learn intra-class and inter-class associations in an easy-to-hard way, thereby generating higher-quality representations."}, {"review_id": "UHKckpsQN2", "reviewer": "Reviewer_fFpq", "summary": "The authors introduce a spatio-temporal pre-training framework that can be easily integrated into downstream baselines. The framework comprises personalized parameter learners and hierarchical hypergraph networks. The former enables the acquisition of spatio-temporal personalized representation and the latter capture the intra-class and inter-class correlations among regions. Besides, the authors design an adaptive mask strategy to guide the model in learning diverse relationships across regions. Experiments are conducted on representative benchmarks on four real-world datasets to evaluate the proposed framework.", "questions": "1. What’s the “another initial” in Line 206 of Section 4.3.1 mean?\n\n2. Is the aforementioned c in Section 4.4 from Section 4.3.1?\n", "limitations": "The authors have not adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors propose a novel pre-training framework for Spatio-Temporal prediction which can be easily applied to existing advanced ST neural networks.\n\n2. The authors devise an adaptive mask strategy that guides the model in a progressive manner, facilitating the acquisition of useful knowledge from other categories to recover unknown categories. The comparisons with other mask strategies demonstrate the superiority of the designed mechanism.\n\n3. The authors evaluate both the original performance and enhanced performance with the proposed framework of different baselines on four datasets. The results indicate that the proposed model significantly improves the downstream baseline prediction performance and confirm the effectiveness of the framework.\n", "weaknesses": "1. The presentation of this paper should be improved:\n\n    (1) The explanations of some concepts are not clear. For instance, the meaning of the symbol $M_{r,t}$ in Section 4.2 is unclear. Does it represent the mask operation? Furthermore, the first-level hyperedges mentioned in Section 4.3.2 lack clear explanation, as it is the first occurrence of the term \"first level\" in the paper.\n\n    (2) Equation (4) is referenced multiple times in the paper to illustrate the process of generating embeddings. However, it is necessary to provide a more detailed explanation of how the equation is actually implemented. For instance, in Line 173 of Section 4.2, it would be helpful to provide specific equations for the computations involved. The inclusion of concrete examples would greatly enhance the understanding of Equation (4) and its functionality.\n\n    (3) In Section 4.2, there are multiple subscripts used, including (r,t), r, and t. It would be better if provide a more detailed explanation to clarify the differences between these descriptions. \n\n    (4) In Section 5.1.1, the paper mentions the use of the absolute error loss function to optimize the parameters. However, there is a lack of detailed description regarding how this loss function is precisely employed. It would be beneficial to provide a more thorough explanation of how the optimization process utilizes the absolute error loss function, including any specific computations.\n\n2. The authors adopt the hierarchical hypergraph structure to capture the semantic-level association of multiple regions from a global perspective. However, the construction of the hypergraph requires a more detailed description. Specifically, the definition of hyperedges and capsules in Section 4.3 needs to be clarified. The paper mentions that hyperedges are treated as high-level capsules, but what exactly does a capsule consist of? Furthermore, it would be helpful to explain the type of hypergraph connection that a hyperedge represents.\n\n3. Some related work with similar ideas is not discussed. For example, the paper [1] also builds a pre-training model on urban data and builds a personalized model in a specific area. Though the idea is not the same, the author should discuss the difference in their related work at least.\n\n4. It would be beneficial to include a brief discussion on future work in the paper.\n\n[1] A Contextual Master-Slave Framework on Urban Region Graph for Urban Village Detection, ICDE 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Response to Reviewer fFpq***  \n**Comment 1:** Clarifications of notations and formulas.  \n**Response:** We sincerely apologize for any confusion that may have arisen regarding the notations and equations, and greatly appreciate your feedback. In order to enhance the clarity of our work, we will provide detailed explanations for each of these issues and will supplement and update them in subsequent versions.  \n(1) $\\textbf{M}_{r,t}$ represents the mask operation for the $r$-th region in the $t$-th time slot. The first-level hyperedges denotes the final embeddings $\\bar{\\textbf{s}}\\in\\mathbb{R}^{H_S\\times T\\times d}$ obtained from Section 4.3.1. In detail, the designed hierarchical hypergraph neural architecture contains the hypergraph capsule clustering network (Section 4.3.1) and the classes aware hypergraph network (Section 4.3.2). We consider the former as the first-level and the latter as the second-level (also high-level).  \n  \n(2) For the time-dynamic encoding, GPST generates time-dynamic feature extraction parameters $\\textbf{W}_t\\in\\mathbb{R}^{d\\times d}, \\textbf{b}_t\\in\\mathbb{R}^d$ using the temporal features $\\textbf{D}_t\\in\\mathbb{R}^{d'}$. Formally, the customization is conducted by $\\textbf{W}_t = \\textbf{D}_t^\\top \\cdot \\bar{\\textbf{W}},~\\textbf{b}_t = \\textbf{D}_t^\\top \\cdot \\bar{\\textbf{b}}$, where $\\bar{\\textbf{W}}\\in\\mathbb{R}^{d'\\times d\\times d}, \\bar{\\textbf{b}}\\in\\mathbb{R}^{d'\\times d}$ are learnable parameters for transformations and bias vectors.  \n  \n(3) $r$ and $t$ are the indexes of $R$ regions and $T$ time slots. For example, $\\textbf{E}_{r,t}\\in\\mathbb{R}^d$ is the representation of the ST-data for the $r$-th region in the $t$-th time slot; $\\textbf{E}_t\\in\\mathbb{R}^{R\\times d}$ denotes the embedding matrix for all the $R$ regions in the $t$-th time slot; $\\textbf{E}_r\\in\\mathbb{R}^{T\\times d}$ denotes the embedding matrix for all the $T$ time slots in the $r$-th region.  \n  \n(4) The details of the loss function are presented in the supplementary material (Section A.1.3) for reference. Due to space constraints, we refrain from presenting it here again.  \n  \n(5) Another initial temporal embeddings $\\textbf{D}_t^{'}$ is new temporal embeddings that  generated based on Equation (3). The aforementioned $\\bar{c}$ in Section 4.4 is from Section 4.3.1.  \n  \n**Comment 2:** Further explanation of hypergraphs and capsule networks.  \n**Response:** (1) We provide an explanation of the definition and construction of hypergraphs in Section 3 (Starting from line 114), as below:  \n A hypergraph $\\mathcal{H} = \\{\\mathcal{V}, \\mathcal{E}, \\textbf{H}\\}$ is composed of three parts: i) Vertices $\\mathcal{V}=\\{v_{r,t}: r\\in R, t\\in T\\}$, each of which represents a region $r$ in a specific time slot $t$. ii) $H$ hyperedges $\\mathcal{E}=\\{e_1,...,e_H\\}$, each of which connects multiple vertices to reflect the multipartite region-wise relations. iii)} The vertex-hyperedge connections $\\textbf{H}\\in\\mathbb{R}^{N\\times H}$, where $N$ denotes the number of vertices. To fully excavate the potential of hypergraphs in region-wise relation learning, we adopt a learnable hypergraph scheme where $\\textbf{H}$ is derived from trainable parameters.  \n   \n(2) According to [1], a capsule is a group of neurons whose activity vector represents the instantiation parameters of a specific type of entity such as an object or an object part. We use the length of the activity vector to represent the probability that the entity exists and its orientation to represent the instantiation parameters. This characteristic empowers GPST with the ability to express region features, such as the functionality of a region. In contrast to low-level capsules, high-level capsules can represent the characteristics of more advanced entities, such as common features of a certain category of regions. By computing the agreement between low-level capsules and high-level capsules using a dynamic routing algorithm, the model can better capture the dynamic semantic information of regions.  \n  \n(3) According to the definition of hypergraphs, the hyperedges are not predefined but derived from trainable parameters. Therefore, each hyperedge in the hypergraph structure can represent a certain underlying semantic correlation. For example, in section 4.3.1, a hyperedge (also high-leval capsule) represents the probability of a region belonging to a certain category.  \n  \n[1] Dynamic Routing Between Capsule, NIPS 2017.  \n  \n**Comment 3:** The inclusion of the suggested work CMSF.  \n**Response:** Thank you very much for your suggestion. We will incorporate it into the discussion of related work. Due to character limitations, we have included the discussion of all related works mentioned by the reviewers in the global response. Please refer to the author  rebuttal to access it.  \n  \n**Comment 4:** Further discussion of our future work for further investigation.  \n**Response:** Thank you for your suggestions. We briefly mention future work in the supplementary material (Section A.4), and we will expand on it based on the latest developments.  \nFuture works：In future work, we will continue to explore more generalizable and versatile spatio-temporal pre-training frameworks. For example, one interesting research direction is addressing the issue of disparate distributions between training and testing data to enhance the practicality of pre-training models. Additionally, lightweight algorithms are also worth investigating, considering the high demand for computational efficiency in real-world urban scenarios."}, {"review_id": "yHrgoOFDnz", "reviewer": "Reviewer_8n4n", "summary": "This paper introduce pre-training task for spatio-temporal. The algorithm design includes a spatial-temporal knowledge extractor and a pre-training mechanism.\n\nThe design of the pre-training methods is to resolve the two limitations of existing spatial-temporal learning: (1) lack of comprehensive personalization (2) insufficient consideration of modelling semantic.\n\n", "questions": "1. Could you please explicitly summarize what is the SSL pre-training method different from the method on static graph? My impression is the main contribution is due to spatial-temporal vs static graph setting. The mask and predict\n\n2. The hyper-edge definition is not very clear to me. At line 116 \"each of which connects multiple vertices\", could you please explain how an edges can connect to more than 2 nodes?\n\n3. How intra-class pattern and inter-class relation are defined (line 138). Is this coming from dataset, or the authors define them based on human knowledge per dataset.\n\n4. Is the pre-training task a link prediction task? Do you explicitly consider the potential information leakage issue during pre-training. For example, the positive edges for pre-training might be sampled as a neighbor when computing the node representation.\n\n5. Will GNN size affect the pre-training efficiency? \n", "limitations": "No", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This paper study an important problem: GNN-pretraining.", "weaknesses": "1. The presentation wasn't very clear, please refer to questions.\n\n2. Missing many recent baselines, e.g., TGAT, JODIE, TGN. The authors can compare with those works using https://github.com/amazon-science/tgl. Besides, there are most more recent methods [1-3] need to be included in discussion.\n\n[1] Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks\n[2] Provably expressive temporal graph networks\n[3] Do We Really Need Complicated Model Architectures For Temporal Networks?\n\n3. Experiment dataset size are all relatively small. I am not sure this method can scale to very large graph from experiments.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Response to Reviewer 8n4n***  \n**Comment 1:** The difference between our pre-training method and the static graph approach.  \n**Response:** Thank you for your suggestion. Compared to static graphs, spatio-temporal (ST) graphs focus more on uncovering the temporal evolution patterns of node features and the dynamic correlations between nodes. The differences between these two approaches primarily lie in the design of the models and the pre-training strategies.  \n(1) Model design. Static graph models typically rely on time-agnostic GNNs for capturing fixed graph structures without time information. However, in ST graph models, it becomes crucial to capture node-specific temporal information and time-evolving spatial dependencies. To tackle these challenges, our GPST employs a temporal encoder and personalized parameter learners to capture the dynamics of node features. Additionally, GPST utilizes a hierarchical hypergraph network to capture the evolving patterns of region-wise spatial relations. As a result, GPST can effectively encode complex ST relations that are not adequately addressed by static graph models.  \n  \n(2) Pre-training strategy: Pre-training in static graph methods typically involves masked autoencoding for features and structures. However, in ST modeling, handcrafted structure information is often noisy and incomplete. This limits the effectiveness of structure reconstruction for pretraining in ST graphs. To address this, we propose an adaptive mask strategy based on ST data for feature autoencoding. This strategy enables the reconstruction of valuable knowledge in an easy-to-hard manner, facilitating high-quality ST pre-training.  \n  \n**Comment 2:** Further explanation of hyperedge.  \n**Response:** Thank you for your feedback. In contrast to conventional edges that transfer information between two nodes, hyperedges expand the notion of edges to act as intermediate information hubs among multiple nodes [1-2]. Specifically, the hyperedge connections is denoted $\\textbf{H}\\in\\mathbb{R}^{N\\times H}$, where $N$ and $H$ are the number of nodes and the number of hyperedges. Each hyperedge captures the connectivity among nodes and hyperedges, with messages initially propagated from all nodes to each hyperedge, followed by transmitting aggregated messages back to each node.  \n  \n[1] Hypergraph Neural Networks  \n[2] Dynamic Hypergraph Neural Networks\n  \n**Comment 3:** Further explanation of intra-class pattern and inter-class relation.  \n**Response:** Thank you for your comment. Relevant concepts are mentioned in the introduction (starting from line 44). In urban settings, similar regions exhibit comparable ST patterns (e.g., residential areas sharing similar traffic patterns), while there are also some associations between different categories of regions (e.g., people moving from residential areas to office areas). These patterns are neither directly obtained from the dataset nor artificially defined. It is a potential knowledge that we aim to extract during the training process, achieved by the iterative clustering effect of capsule networks [3] integrated into the hypergraph network.  \n  \n[3] Dynamic routing between capsules  \n  \n**Comment 4:** Further clarification of the pre-training task.  \n**Response:** Thank you for your feedback. The pre-training task is to mask and reconstruct node features, which is meticulously defined and explained in Preliminaries (Section 3 starting from line 121).  Upon thorough verification, we have confirmed the absence of information leakage in this work. Additionally, since our pre-training task does not involve link prediction, the mentioned sampling-induced information leakage scenario does not occur.  \n\n**Comment 5:** Further discussion on the impact of GNN Size on the proposed model.  \n**Response:** Thank you for your suggestion. The complexity of our Intra- & Inter-class spatial patterns encoding module (spatial encoder) is proportional to the number of time slots $T$ and the number of regions $R$, similar to a vanilla GCN. Hence, our model exhibits scalability to large datasets, akin to other graph architectures.  \n  \n**Comment 6:** Clarification of recent methods for temporal graph networks.  \n**Response:** Thank you for your suggestion. We observe that the mentioned methods predominantly involve works related to temporal graph networks (TGNs), rather than the field of ST prediction that we are concerned about, and we will elaborate from the following two perspectives.  \n(1) Why we did not consider the mentioned methods as baselines and include them in the discussion of this paper?  \ni) There is a gap between the tasks addressed by TGNs and ST prediction. ST prediction primarily deals with forecasting future values based on historical data in a spatial and temporal context. While temporal graph networks focus on dynamic reasoning over graph structures (e.g. link prediction). Different tasks require tailored model architectures to meet their specific requirements.  \nii) We followed the experimental setup used in existing research on ST prediction [4-5], and the baseline designed specifically for ST prediction tasks may be more representative.\n  \n[4] Spatial-temporal fusion graph neural networks for traffic flow forecasting  \n[5] MSDR: Multi-step dependency relation networks for spatial temporal forecasting  \n  \n(2) We adapted the mentioned baselines to suit the ST prediction task and discussed the works related to TGNs.  \ni) Additional comparative experiment of JODIE and TGN. Results are presented in the following table (in terms of MAE).  \n  \n| model/dataset | PEMS08 | METR-LA | NYC TAXI | NYC Citi Bike |\n|:--:|:--:|:--:|:--:|:--:|\n| JODIE | 18.53 | 3.45 | 6.92 | 2.09 |\n| w/ GPST | 16.59 | 3.11 | 6.19 | 1.90 |\n| TGN | 19.33 | 3.61 | 7.58 | 2.09 |\n| w/ GPST | 16.11 | 3.16 | 5.94 | 1.89 |\n  \nii) Related works. Thank you for your suggestion. Due to character limitations, we include the discussion of related works in the global response."}, {"review_id": "ez9p2PEtVR", "reviewer": "Reviewer_83m7", "summary": "This paper proposes a novel pre-training framework (GPST) for ST prediction, it mainly integrates the ST parameter personalization scheme and the region-wise semantic association mechanism into the pre-training model, and conduct the training in an unsupervised manner. Besides, the proposed model GPST adopts the hierarchical hypergraph structure to capture the semantic-level association of multiple regions from a global perspective. Extensive experiments are conducted.", "questions": "See above.", "limitations": "Noone", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-presented and well-organized.\n2. This paper introduces a spatio-temporal pre-training framework that can be easily integrated into downstream baselines and improve their performance.\n3. Detailed experimental results are provided to validate the model.\n", "weaknesses": "1. The model's performance seems to heavily rely on the pre-training stage. If the pre-training is not done correctly or if the pre-training data is not representative of the test data, the model's performance could suffer, how can this case be handled?\n2. Why not include POIs, which should benefit the prediction performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Response to Reviewer 83m7***  \n**Comment 1:** Further discussion on the design of the pre-training stage and data distribution.  \n**Response:** Thank you for your feedback. This work primarily introduces a spatio-temporal pre-training framework to enhance downstream baselines, where the effectiveness of the enhancement is highly dependent on the designs of the pre-training stage. If the pre-training model or mechanisms are not appropriately designed, the embeddings generated by the model may lack superior representational capacity, consequently compromising the performance of downstream baselines. To address this issue, we address the limitations of prior research in spatio-temporal modeling and comprehensively consider these concerns in the construction of the pre-training model. In terms of the pre-training mechanism,  an adaptive mask strategy is proposed to guide the model to learn robust spatio-temporal representations.  \nThe performance of most models suffers significantly if the training data fails to represent the test data, which remains a major challenge in the field of machine learning. Following prior works (such as GWN[1] and STEP[2] ), we assume that the training (or pre-training) and test data distributions are approximately similar, and this work does not address the scenario where there exists a significant distribution discrepancy between the training and test data. There exists a research line dedicated to exploring this issue (such as domain adaptation, domain generalization, and out-of-distribution (OOD)), which is  an intriguing research direction. In future work, we will continue to investigate the related issues concerning the generalization ability of spatiotemporal models (such as different distributions between test and training data, data distribution drift, and so on).  \n  \n[1] Graph wavenet for deep spatial-temporal graph modeling. IJCAI 2019.  \n[2] Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting, KDD 2022.\n  \n**Comment 2:** Explanation for the exclusion of POIs.  \n**Response:** Thank you for your constructive comment. Intuitively, incorporating POIs information can enhance the predictive performance of the model. However, we did not utilize such information primarily based on the following two considerations. i) Difficulties in obtaining POIs data. Due to the absence of POIs in the utilized dataset, their inclusion would require manual collection. However, collecting and determining POIs data in real-world urban scenarios can be extremely challenging. This is primarily attributed to the dataset's extensive coverage of numerous urban regions with significant geographical spans, demanding substantial human resources and time commitment to accomplish this task. ii) Fairness of experiments. Previous works did not incorporate POIs when evaluating on the utilized datasets. To present a fair comparison and showcase the improvement achieved by our model, we refrained from introducing additional artificially created data during the pre-training stage."}]

Author rebuttal:
***Responses Regarding Literature Review***  
We extend our heartfelt gratitude to the reviewers for their valuable time and effort dedicated to evaluating our work. Due to constraints on character count, we will address the discussion of all relevant works mentioned by the reviewers in this section.  
  
***Response to Reviewer H5wT***  
**Comment 1:** The inclusion of the suggested related work.  
**Response:** Thank you for your suggestion. The self-supervised learning methods for graph networks and multivariate time series forecasting will be incorporated into the related work in our revised version.  
SSL methods for Graph and MTS: In recent years, self-supervised learning methods for graph have received significant attention in recent years. GNNs based on contrastive learning generate different views through data augmentation.  Following that, a loss function is employed to maximize the consistency of positive sample pairs while minimizing the consistency of negative sample pairs across views. For instance, GraphCL [1] generates two views of the original graph by applying node dropping and edge shuffling, and performs contrastive learning between them. Furthermore, JOAO [2] proposes an approach that automatically selects graph augmentations to facilitate contrastive learning. Another research direction involves generative graph neural networks, where these methods leverage the graph data itself as a natural supervisory signal and achieve representation learning through reconstruction. GPT-GNN [3] pretrains by reconstructing graph features and edges, while GraphMAE [4] employs node feature masking in both the graph encoder and decoder to reconstruct features and learn graph representations. In multivariate time series prediction, STEP [5] integrates long-term temporal features to generate temporal representations using a MAE-based pretraining method. COST [6] argues that unraveling temporal features is beneficial for time series prediction and proposes a contrastive learning approach applied to decouple trend and seasonal representations. However, spatio-temporal forecasting requires considering complex temporal evolution patterns and spatial correlation mechanisms simultaneously, and the pre-training paradigm for such tasks is still an area of exploration.
  
[1] Graph Contrastive Learning with Augmentations, NIPS 2020.  
[2] JOAO: Graph Contrastive Learning Automated, ICML 2021.  
[3] GPT-GNN: Generative Pre-Training of Graph Neural Networks, KDD 2020.  
[4] GraphMAE: Self-Supervised Masked Graph Autoencoders, KDD 2022.  
[5] Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting, KDD 2022.  
[6] CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting, ICLR 2022.  
  
***Response to Reviewer 8n4n***  
**Comment 6:** Discussion of recent methods for temporal graph networks.  
Thank you for your suggestion. In response to this valuable feedback, we will incorporate a dedicated paragraph in our revised version to provide a detailed analysis and discussion of the mentioned methods and their relevance to the field of spatio-temporal prediction scenarios.  
Temporal graph networks serve as a similar research baseline aimed at reasoning about dynamic graph structures. CAWs [7] proposed a causal anonymous walks approach to inductively represent dynamic graph networks, specifically addressing interaction prediction for newly appearing nodes. PINT [8] introduced injective temporal message passing and relative positional features to achieve more expressive temporal graph networks. Additionally, GraphMixer [9] argued that complex model structures may not necessarily suitable for temporal networks and presented a conceptually and technically simple architecture based on MLP for link prediction.  

[7] Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks, ICLR 2021.  
[8] Provably expressive temporal graph networks, NIPS 2022.  
[9] Do We Really Need Complicated Model Architectures For Temporal Networks, ICLR 2023.  
  
***Response to Reviewer fFpq***  
**Comment 3:** The inclusion of the suggested work CMSF.  
**Response:** Thank you very much for your suggestion. We will incorporate it into the discussion of related work.   
CMSF [10] proposes a master-slave framework for identifying urban villages. The master model generates region representations in a pre-trained manner, while the slave model fine-tunes specific regions for accurate identification.  
The proposed model can be distinguished from CMSF in the following three aspects.  
(1) Task. CMSF focuses on identifying urban villages in the city, which falls under the classification task. In contrast, our model aim to predict the future spatio-temporal conditions given the historical records, thus belonging to a regression task.  
(2) Technical approach. While CMSF employs techniques like GNNs to establish semantic correlations in static scenes, our task involves temporal dynamics. To address this, we have designed a hierarchical hypergraph framework with personalized parameter learner to model time-aware semantic correlations in regions. In addition, an adaptive masking strategy is proposed to further enhance the model's performance.  
(3) Generalizability: After pre-training, CMSF requires specific configurations for the slave model to adapt to the master model's predictions, limiting its applicability to enhancing other similar models. Recognizing that different downstream baselines have distinct advantages and expressive capabilities when dealing with different data distributions, we have designed a universal pre-training framework that can enhance existing baselines to improve their performance.  
  
[10] A Contextual Master-Slave Framework on Urban Region Graph for Urban Village Detection, ICDE 2023.

Meta-review: The paper introduces GPST, a spatio-temporal pre-training framework designed to improve downstream prediction models. GPST incorporates personalization and region-wise semantic association in an unsupervised manner and utilizes a hierarchical hypergraph structure. Across four reviews, there is a consensus on the paper's innovative approach to spatio-temporal learning. The paper has been praised for its well-organized presentation, its ease of integration into existing models, and detailed experimental validation.

However, there are concerns across reviews about presentation clarity, scalability, and comparison with recent baselines. One reviewer questions the model's heavy reliance on the pre-training stage, suggesting that if the pre-training is not done adequately or the data used isn't representative, the model could underperform. Another reviewer suggests the inclusion of Points of Interest (POIs) for enhanced prediction performance.

In summary, Reviewers' ratings vary from borderline reject to accept, but leaning toward accept.
Decision: Accept (poster)