Title: GPT-ST: Generative Pre-Training of Spatio-Temporal Graph Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Spatial Temporal Prediction

Deep Neural Networks

Pre-training Model
Abstract: In recent years, there has been a rapid development of spatio-temporal prediction techniques in response to the increasing demands of traffic management and travel planning. While advanced end-to-end models have achieved notable success in improving predictive performance, their integration and expansion pose significant challenges. This work aims to address these challenges by introducing a spatio-temporal pre-training framework that seamlessly integrates with downstream baselines and enhances their performance. The framework is built upon two key designs: (i) We propose a spatio-temporal mask autoencoder as a pre-training model for learning spatio-temporal dependencies. The model incorporates customized parameter learners and hierarchical spatial pattern encoding networks. These modules are specifically designed to capture spatio-temporal customized representations and intra- and inter-cluster region semantic relationships, which have often been neglected in existing approaches. (ii) We introduce an adaptive mask strategy as part of the pre-training mechanism. This strategy guides the mask autoencoder in learning robust spatio-temporal representations and facilitates the modeling of different relationships, ranging from intra-cluster to inter-cluster, in an easy-to-hard training manner. Extensive experiments conducted on representative benchmarks demonstrate the effectiveness of our proposed method. We have made our model implementation publicly available at https://github.com/HKUDS/GPT-ST.

GPT-ST: Generative Pre-Training of Spatio-Temporal Graph Neural Networks

Abstract

In recent years, there has been a rapid development of spatio-temporal prediction techniques in response to the increasing demands of traffic management and travel planning. While advanced end-to-end models have achieved notable success in improving predictive performance, their integration and expansion pose significant challenges. This work aims to address these challenges by introducing a spatio-temporal pre-training framework that seamlessly integrates with downstream baselines and enhances their performance. The framework is built upon two key designs: (i) We propose a spatio-temporal mask autoencoder as a pre-training model for learning spatio-temporal dependencies. The model incorporates customized parameter learners and hierarchical spatial pattern encoding networks. These modules are specifically designed to capture spatio-temporal customized representations and intra- and inter-cluster region semantic relationships, which have often been neglected in existing approaches. (ii) We introduce an adaptive mask strategy as part of the pre-training mechanism. This strategy guides the mask autoencoder in learning robust spatio-temporal representations and facilitates the modeling of different relationships, ranging from intra-cluster to inter-cluster, in an easy-to-hard training manner. Extensive experiments conducted on representative benchmarks demonstrate the effectiveness of our proposed method. We have made our model implementation publicly available at https://github.com/HKUDS/GPT-ST .

1 Introduction

Intelligent Transportation Systems (ITS) is a rapidly growing research field driven by advancements in sensors and communications technology [1, 12]. Spatio-Temporal (ST) prediction, including traffic flow and ride demand, aims to forecast future trends, facilitating decision-making in traffic management and risk response [43], and enhancing people's lives with future travel plans [6]. Deep learning methods, such as RNNs and CNNs, have been widely adopted to model temporal and spatial correlations [5, 51, 50, 45]. Recently, GNNs have gained attention for spatial and temporal prediction tasks, achieving remarkable results. Initially, researchers built adjacency matrices based on region attributes, such as distance [48, 54]. Subsequent studies introduced learnable graph structures to capture spatial correlations [40, 39], while others focused on dynamic relationships between regions, proposing methods for constructing dynamic graphs [14, 40]. These advancements have significantly contributed to the development of spatio-temporal prediction techniques.

Despite the remarkable results achieved by existing methods in spatio-temporal prediction, there are several issues that remain inadequately addressed. i) Lack of customized representations of specific spatio-temporal patterns. Customization can be categorized into two key aspects: timedynamic and node-specific properties in both the time and spatial domains. Time-dynamic patterns exhibit variations across different time periods, such as contrasting patterns between weekends and

weekdays within the same region. Moreover, correlations between regions dynamically evolve over time, surpassing the capabilities of static graph representations. Node-specific patterns highlight the distinct time series observed in different regions, rather than a shared pattern. Additionally, it is important to ensure that different regions retain their individual node characteristics even after message aggregation, to prevent interference from prominent nodes in the spatial domain. We believe that encoding all of these customized characteristics is essential to guarantee the model robustness. However, existing works often fall short in providing a comprehensive consideration of these factors. ii) Insufficient consideration of spatial dependencies at different levels. Most approaches primarily focus on pairwise associations between regions when modeling spatial dependency, but they overlook the semantic relevance at different spatial levels. In the real world, regions with similar functions tend to exhibit similar spatio-temporal patterns. By performing clustering analysis on different regions, the model can explore common characteristics among similar regions, thereby facilitating improved spatial representation learning. Moreover, there is a lack of adequate modeling of high-level regional relationships across time in current studies. Spatio-temporal patterns between different types of high-level regions may exhibit dynamic transfer relationships. For instance, during work hours, there is a notable movement of people from residential areas to work areas, as illustrated in the right part of Figure 1. In this context, changes in people flow within residential areas can provide valuable auxiliary signals for predicting people flow in work areas. It highlights the importance of incorporating both fine-grained and coarse-grained correlations among different levels of regions in order to enhance the predictive capabilities of spatio-temporal prediction models.

An intuitive approach to address the aforementioned challenges is to develop an end-to-end model. However, the current models face difficulties in benefiting from such an approach, as each module in the state-of-the-art (SOTA) model is intricately refined, and any disassembly and integration may lead to a degradation in prediction performance. So, is there a strategy that can allow existing spatiotemporal methods to leverage the advantages of an end-to-end model? The answer is affirmative. Recent pre-training frameworks, such as ChatGPT [3, 27] in the field of natural language processing (NLP) and MAE [15] in computer vision (CV), have been proposed by pioneers and widely studied in their respective domains. These pre-training architectures construct unsupervised training tasks, involving masking or other techniques, to learn better representations and improve

A

B

C

D

C

D

0.8

0.1

0.9

0.2

0.8

0.7

0.1

6-8pm

7-9am

Work day

9-12am

Week day

downstream task performance. However, the application of such scalable pre-training models in the field of spatio-temporal prediction has been limited thus far. To address the aforementioned challenges, we propose a novel framework called Generative Pre-Training for Spatio-Temporal prediction (GPT-ST). This framework is designed to seamlessly integrate into existing spatio-temporal prediction models, enhancing their performance. Our contributions are summarized as follows:

We present GPT-ST, a novel pre-training framework designed for spatio-temporal prediction. This framework seamlessly integrates into existing spatio-temporal neural networks, resulting in performance enhancements. Our approach combines a model parameter customization scheme with a self-supervised masking autoencoding, enabling effective spatio-temporal pre-training.

GPT-ST ingeniously utilizes a hierarchical hypergraph structure to capture spatial dependencies at different levels from a global perspective. Through collaboration with a thoughtfully designed adaptive mask strategy, the model acquires the capability to model both intra-cluster and intercluster spatial relations among regions, thereby generating robust spatio-temporal representations.

We conduct extensive experiments on real-world datasets, and the demonstrated improvement in the performance of diverse downstream baselines showcases the superior performance of GPT-ST.

2 Related Work

Deep Spatio-temporal Prediction Techniques. Numerous studies have focused on designing deep neural networks for spatio-temporal data prediction. To capture temporal dependencies, DLSTM [49] and GWN [40] have developed predictive frameworks based on Recurrent Neural Networks (RNNs) and Temporal Convolutional Networks (TCNs), respectively. GMAN [55] and STGNN [36] have utilized temporal self-attention networks to facilitate long-range temporal learning. In spatial relation learning, DMVST-Net [44] adopted convolutional network to model region associations. STGCN [48] and DCRNN [23] exploited Graph Neural Networks (GNNs) for message passing between regions. STAN [26] consider the interrelationships between regions based on Graph Attention Networks (GAT). Temporal graph networks serve as a similar research baseline aimed at reasoning about dynamic graph structures, such as CAWs [37] and PINT [33]. More recently, some works have explored the combination of ST models with advanced methods, such as Neural-ODEbased [10, 18] and self-supervised learning-based [24], for better spatial and temporal dependencies modelling. Compared to these works, our proposed GPT-ST framework aims at empower diverse deep spati-temporal models with the pre-training stage.

Pre-trainning methods. Pre-training models have made significant advancements in recent years, with notable examples including BERT [7, 28], vision transformers [9, 8], masked autoencoders [15, 20], and language models [3, 27, 34]. The success of the autoencoder-based approach, exemplified by MAE, has demonstrated the powerful expressive capability of the mask-reconstruction pre-training framework. Building upon this, STEP [31] proposed a pre-training model specifically for time series prediction. In other domains, CMSF [41] introduced a master-slave framework for identifying urban villages. The master model pre-trains region representations, while the slave model fine-tunes specific regions for accurate identification. Researchers have also recognized the need for more efficient mask mechanisms than random masking. AttMask [19] and SemMAE [21] leverage attention mechanisms and semantic information, respectively, to guide the mask operation. AdaMAE [2] proposes adaptively selecting visible signals with greater reconstruction loss, as they contain more information, and using a higher proportion of masking strategy accordingly. Despite the significant progress of pre-training models in the fields of natural language processing (NLP) and computer vision (CV), their application to spatio-temporal prediction remains relatively limited. Additionally, computational efficiency has been a major bottleneck for transformer architectures in general.

Self-supervised learning methods for GNNs. In recent years, self-supervised learning methods for graph data have gained significant attention [53, 38]. Contrastive learning-based GNNs generate different views of the original graph through data augmentation techniques [42, 29]. A loss function is then employed to maximize the consistency of positive sample pairs while minimizing the consistency of negative sample pairs across views. For example, GraphCL [47] generates two views of the graph by applying node dropping and edge shuffling, and performs contrastive learning between them. Another research direction focuses on generative graph neural networks, where the graph data itself serves as a natural supervisory signal for representation learning through reconstruction. GPTGNN [17] conducts pre-training by reconstructing graph features and edges, while GraphMAE [16] utilizes node feature masking in both the graph encoder and decoder to reconstruct features and learn graph representations. However, spatio-temporal forecasting tasks require simultaneous consideration of complex temporal evolution patterns and spatial correlation mechanisms. Pre-training paradigms specifically designed for such tasks are still an area of exploration and research.

3 Preliminaries

Spatio-Temporal Data X . To capture the ST information, we represent it as a three-way tensor X ∈ R R × T × F , where R , T , and F correspond to the numbers of regions, time slots, and feature dimensions, respectively. Each entry X r,t,f is the f -th feature of the r -th region in the t -th time slot.

Spatio-Temporal Hypergraphs . We employ ST hypergraphs for ST modeling. A hypergraph H = {V , E , H } is composed of three parts: i) Vertices V = { v r,t : r ∈ R,t ∈ T } , each of which represents a region r in a specific time slot t . ii) H hyperedges E = { e 1 , ..., e H } , each of which connects multiple vertices to reflect the multipartite region-wise relations ( e . g ., all residential regions can be connected by one hyperedge). iii) The vertex-hyperedge connections H ∈ R N × H , where N denotes the number of vertices. To fully excavate the potential of hypergraphs in region-wise relation learning, we adopt a learnable hypergraph scheme where H is derived from trainable parameters.

Spatio-Temporal Pre-training Paradigm . Our GPT-ST framework aims at developing a pre-trained ST representation method that facilitates the accuracy on downstream ST prediction tasks such as traffic volume forecasting. As depicted in Figure 2, the workflow of GPT-ST can be divided into the pre-training stage and the downstream task stage . We formulate the two stages as follows:

The pre-training stage of our GPT-ST framework adopts the masked autoencoding (MAE) task as the train-

Mask AutoEncoder

Mask Operation

Pre-training Stage

Downstream Tasks Stage

Spatial-temporal Neural

Networks

෡

Y

Mask AutoEncoder

ing objective. The goal of this MAE task is to reconstruct masked information of the spatio-temporal data according to the unmasked ones by learning a ST representation function f . For the training ST data during the period of [ t K -L +1 , t K ] , we aim to minimize the following objective:

where L denotes the measure of prediction deviation, M ∈ { 0 , 1 } represents the mask tensor, and ⊙ denotes the element-wise multiplication operation. W denotes a predictive linear layer.

ii) After the pre-training stage, the results of our GPT-ST are then utilized in the downstream task stage . The pre-trained model yields high-quality ST representations to facilitate the downstream prediction tasks such as traffic forecasting. Specifically, the downstream stage is formulated as:

where ζ refers to the representation generated by f according to the historical spatio-temporal data from the previous L time slots before the K -th time slot. The output is the predictions for the next P time slots. Various existing spatio-temporal neural networks can serve as the prediction function g .

4 Methodology

This section introduces the technical details of the proposed ST pre-training framework GPT-ST. It develops a customized temporal encoder and a hierarchical spatial encoder to obtain large modeling capacity for the pre-trained model. An adaptive mask strategy is proposed to facilitate effective MAE training. Figure 3 demonstrates the model design of our pre-trained spatio-temporal encoding model.

4.1 Customized Temporal Pattern Encoding

Initial Embedding Layer . We begin by constructing an embedding layer to initialize the representation of the spatio-temporal data X . The original data undergoes normalization using the Z-Score function [1, 39] and is then masked using the mask operation. Subsequently, a linear transformation is applied to augment the representation as E r,t = M r,t ⊙ ¯ X r,t · E 0 , where E r,t ∈ R d , M r,t ∈ R F , and ¯ X r,t ∈ R F represent the representation, mask operation, and normalized spatio-temporal data for the r -th region in the t -th time slot. The variable d denotes the number of hidden units. Additionally, E 0 ∈ R F × d represents the learnable embedding vectors for the F feature dimensions.

Temporal Hypergraph Neural Network . To facilitate global relation learning, we employ the hypergraph neural network for the temporal pattern encoding, specifically as follows:

where Γ t , ¯ E t , E t ∈ R R × d denote the result, intermediate, and initial region embeddings for the t -th time slot, respectively. W t ∈ R d × d , b t ∈ R d represent the t -th time slot-specific parameters. σ ( · ) denotes the LeakyReLU activation. The intermediate embeddings ¯ E ∈ R R × T × d are calculated by the hypergraph information propagation. It employs the region-specific hypergraph H r ∈ R H T × T to propagate the temporal embeddings E r ∈ R T × d for the r -th region along the connections between time slots and H T hyperedges, so as to capture the multipartite relations among time periods.

Customized Parameter Learner . To characterize the diversity of temporal patterns, our temporal encoder conducts model parameter customization for both different regions and different time periods. Specifically, the aforementioned time-specific parameters W t , b t and the region-specific hypergraph

Temporal

…

…

…

𝐯𝑟,𝑡

= squash(𝐕𝚪 𝑟,𝑡 +𝐜)

Primary capsules

Dynamic routing

𝐬𝑖,𝑡 = ෍

𝑅

𝑐 𝑖,𝑟,𝑡

⋅ ത

𝐯𝑖|𝑟,𝑡

Clustering

Hypergraph Capsule Clustering Network

Back to

regions

High-level capsules

𝑐 𝑖,𝑟,𝑡 =

exp(𝑏𝑖,𝑟,𝑡

)

σ𝑟 exp(𝑏𝑖,𝑟,𝑡

)

𝑏𝑖,𝑟,𝑡

Regions clustering

Patterns  migration

High-level

Low-level

Generate

ത

𝐯𝑖|𝑟,𝑡

= 𝑠𝑞𝑢𝑎𝑠ℎ(𝑢) ⊺

⋅ 𝐯

𝑟,𝑡

𝑢 = 𝐇 𝑖,𝑟,𝑡

′

⋅ 𝐯

𝑟,𝑡

Time-dynamic parameters

𝐃𝑡

Inject

t1

t2

t1

t2

Rr

R2

R1

…

…

ത

𝐄𝑟

= 𝜎 𝐇𝑟

⊺

⋅ 𝜎(𝐇𝑟

⋅ 𝐄

𝑟

)

Time-HyperPropagate

Customized Temporal Pattern Encoding

Cross-class Relation Learning

Customized Parameter Learner

Region-specific parameters

Inject

Specific

𝐂𝑟

t1

…

tn

…

Specific

ഥ 𝐇

Rr

R2

R1

…

parameters H r are generated through a learnable process, rather than directly utilizing independent parameters. Specifically, the customized parameters are learned as follows:

where ¯ H ∈ R d ′ × H T × T , ¯ W ∈ R d ′ × d × d , ¯ b ∈ R d ′ × d are independent parameters for the three generated parameters, respectively. c r , d t ∈ R d ′ denote representations for the r -th region and the t -th time slot, respectively. c r for r ∈ R is a free-form parameter, while d t for t ∈ T is calculated from the normalized time-of-day features ¯ z ( d ) t and day-of-week features ¯ z ( w ) t . e 1 , e 2 ∈ R d ′ are their corresponding learnable embeddings. This parameter learner realizes spatial and temporal customization by generating parameters according to the features of specific time slots and regions.

4.2 Hierarchical Spatial Pattern Encoding

4.2.1 Hypergraph Capsule Clustering Network

Current spatial encoders primarily focus on capturing local adjacent region-wise relations while disregarding the widespread similarities that exist between distant regions. For example, business districts that are geographically apart can still exhibit similar ST patterns. In light of this, our GPT-ST introduces a hypergraph capsule clustering network to capture global region-wise similarities. This network explicitly learns multiple cluster centers as hyperedges, characterizing the global region-wise similarities. To further enhance the hypergraph structure learning, we incorporate the dynamic routing mechanism of capsule network. This mechanism iteratively updates hyperedge representations and region-hyperedge connections based on semantic similarities. As a result, it improves the clustering capability of hyperedges and facilitates the global modeling of dependencies among regions.

Specifically, we firstly obtain a normalized region embedding v r,t ∈ R d for each region r in time slot t , using the previous embedding Γ r,t and the squash function [30]. This embedding is then used to calculate the transferred information ¯ v i | r,t ∈ R d from each region r to each cluster center (hyperedge) i , within the t -th time slot. Formally, these two variables are calculated by:

where V ∈ R d × d and c ∈ R d are free-form learnable parameters. The hypergraph connectivity matrix H ′ t ∈ R H S × R records the relations between R regions and H S hyperedges as the cluster centroids. It is tailored to the t -th time slot using the aforementioned customized parameter learner, specifically by H ′ t = softmax ( d ′ ⊤ t ¯ H ′ ) . Here, d ′ t and ¯ H ′ are temporal features and hypergraph embeddings.

Iterative Hypergraph Structure Learning . With the initialized region embeddings v r,t and hypergraph connection embeddings ¯ v i | r,t , we follow the dynamic routing mechanism of capsule network to enhance the clustering effect for the hyperedges. The j -th iteration is described as follows:

where s i,t ∈ R d represents the iterative hyperedge embedding. It is calculated utilizing the iterative hyperedge-region weights c i,r,t ∈ R . The weight c i,r,t is in turn calculated from the hyperedge embeddings s i,t from the last iteration. With this iterative process, the relation scores and the hyperedge representations are mutually adjusted by each other, to better reflect the semantic similarities between regions and the spatial cluster centroids represented by hyperedges.

After the iterations of the dynamic routing algorithm, to make use of both b i,r,t and H ′ i,r,t for better region-hyperedge relation learning, GPT-ST combines the two sets of weights and generates the final embeddings ¯ s i,t ∈ R d . We begin by replacing b i,r,t with ( b i,r,t + H ′ i,r,t ) to obtain a new weight vector ¯ c i,r,t ∈ R , and then utilize ¯ c i,r,t ∈ R to calculate the final embeddings ¯ s i,t .

4.2.2 Cross-Cluster Relation Learning

With the clustered embeddings ¯ s i,t , we propose to model the inter-cluster relations via a high-level hypergraph neural network. Specifically, the refined cluster embeddings ˆ S ∈ R H S × T × d are calculated by message passing between the H S cluster centroids and H M high-level hyperedges as follows:

where ˜ S ∈ R H S T × d denotes the reshaped embedding matrix obtained from ¯ s i,t for i ∈ H S and t ∈ T . H ′′ ∈ R H M × H S T denotes the high-level hypergraph structure, which is obtained through the aforementioned personalized parameter learner. This parameter customization aggregates the temporal features ¯ z ( d ) t , ¯ z ( w ) t for all t ∈ T as the input, and generates the parameters as in Eq 4.

After refining the cluster representations ˆ s i,t ∈ R d for i ∈ H S , t ∈ T , we propagate the clustered embeddings back to the regional embeddings with the low-level hypergraph structure, as follows:

where Ψ r,t ∈ R d denotes the new region embedding for the r -th region in the t -th time slot. c i,r,t ∈ R denotes the weight for the low-level hypergraph capsule network. W ′′ r ∈ R d × d and b ′′ r ∈ R d denotes the region-specific transformation and bias parameters generated by the customized parameter learner.

4.3 Cluster-aware Masking Mechanism

Inspired by semantic-guided MAE [21], we design a cluster-aware masking mechanism to enhance the intra- and inter-cluster relation learning for our GPT-ST. The adaptive masking strategy incorporates the foregoing learned clustering information ¯ c i,r,t to develop an easy-to-hard masking process. In particular, at the beginning of the training, we randomly mask a portion of regions for each cluster, in which case the masked values can be easily predicted by referring to in-cluster regions that share similar ST patterns. Subsequently, we gradually increase the masking proportion for certain categories, to increase the prediction difficulty for these clusters by reducing their relevant information. Finally, we completely mask the signals of some clusters, facilitating the ability of cross-cluster knowledge transfer for the pre-trained model. This adaptive masking process is depicted in Figure 4.

However, directly utilizing the learned cluster information ¯ c i,r,t to generate the cluster-aware mask M is not feasible. This is because the cluster information is calculated by deeper layers of our GPTST network, while the generated mask is required as input to the network ( M ⊙ X ). To address this, we employ a two-layer MLP network with customized parameters to predict the learned results of ¯ c i,r,t . In particular, the transformations and bias vectors in this MLP network are replaced with time-dynamic and node-specific parameters generated by the customized parameter learner (Eq 4). Subsequently, a linear layer and softmax ( · ) function are used to obtain the predictions q i,r,t ∈ R for i ∈ H S , r ∈ R,t ∈ T . To optimize the distribution of q i,r,t , we utilize the KL divergence loss function L kl with the ground truth ¯ c i,r,t . It is important to note that the backpropagation of ¯ c i,r,t is prevented in this step. The category is the maximum probability

Cluster-aware Masking Strategy

From-easy-to-hard

Unmasked

Masked

Category

according to q i,r,t is taken as the classification result. See Algorithm 1 in A.2 for more details.

5 Evaluation

In this section, we verify the effectiveness of GPT-ST by answering the following five questions:

RQ1 : How does our proposed pre-training framework GPT-ST benefit different downstream baseline methods, in terms of prediction accuracy on diverse spatio-temporal prediction tasks?

RQ2 : How effective are the designed technical modules of our GPT-ST framework?

RQ3 : How are the interpretation ability of the learned global region clusters and the inter-cluster transition patterns in the hierarchical spatial pattern encoding of our GPT-ST framework.

RQ4 : How efficient is GPT-ST in both the pre-training stage and the downstream task stage?

RQ5 : How does varying the mask ratio setting affect the performance of GPT-ST?

5.1 Experimental Setting

To evaluate the effectiveness of our proposed method, we conduct experiments on four real-world ST datasets including: PEMS08 [32], METR-LA [23], NYC Taxi and NYC Citi Bike [46], which records the traffic flow, traffic speed, taxi order records, and bike order records, respectively.

We choose Mean Absolute Error (MAE), Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE) as the evaluation metrics, which are widely used in ST prediction tasks [52]. The lower value of the metrics, the better performance of the model. Following previous works [13, 32, 22], the number of time slots L is set to 12. We divide METR-LA dataset into training set, validation set and test set in a ratio of 7:1:2 and 6:2:2 for others datasets. The final model parameters are selected according to the optimal effect (minimum MAE value) of the validation set.

For optimization, in the pre-trainning phase, the absolute error loss function L r is used to optimize the parameters and a hyperparameter λ is adopted to balance the weight of L r and L kl . In the downstream task stage, we firstly fuse the embedding ζ and the raw signal X as the input to the downstream model, and the optimization strategies are distinct for different downstream models. Detailed descriptions of datasets and evaluation are presented in the supplementary material A.1.

5.2 Main Results (RQ1)

In this section, we mainly investigate whether GPT-ST improves the forecasting for downstream tasks. To achieve this, we evaluate both the original performance and enhanced performance (w/ GPT-ST) of different baselines on four datasets, and the results are shown in Table 1. Note that since DMVSTGCN, STMGCN and CCRNN are specially designed for demand forecasting, we eliminate them in traffic forecasting tasks. The results indicate that the proposed model significantly improves the prediction performance of different downstream baselines on all datasets, which confirms the effectiveness of the our framework. We analyze the promotion effect of GPT-ST from three dimensions:

Consistent improvements by GPT-ST . We observe that GPT-ST improves different types of methods ( e . g ., GNN-based, or attention-based models), and such positive effect does not partialize to a certain category of baselines, which verifies the generalization ability of GPT-ST. We attribute this improvement to the MAE pre-training with the awareness of intra- and inter-cluster ST relations.

Difference among baseline models. When compared to recently proposed methods like MSDR, our GPT-ST exhibits more substantial improvements when applied to classical baselines such as STGCN. Specifically, in traffic flow prediction using the PEMS08 dataset, MSDR shows improvements of 0.55 in MAE, 0.60 in RMSE, and 0.21% in MPAE, while STGCN demonstrates improvements of 1.61 in MAE, 2.71 in RMSE, and 0.63% in MPAE. One possible explanation for these findings is that advanced baselines like MSDR are already well-designed and comprehensive in modeling various factors. Consequently, they are capable of encoding abundant knowledge independently, which may diminish the utility of the additional signals provided by the pre-training model. Conversely, classical baselines like STGCN may derive greater benefits from the insights offered by our GPT-ST due to their simpler design and potentially limited capacity to capture complex relationships.

Comparison with pre-training methods. In addition to evaluating our pre-training method against spatio-temporal prediction methods without pre-training, we also compare it with the competitive pre-training baseline STEP. This method utilizes long-term time series as input for pre-training to enhance the performance of the downstream model (GWN). Despite our GPT-ST using only short-term data during the pre-training stage, it outperforms STEP on several metrics. Furthermore, our GPT-ST demonstrates an even larger performance advantage in scenarios where long-term data is insufficient, e . g ., the taxi and bike demand prediction tasks. This highlights the broader applicability of our proposed GPT-ST framework, emphasizing its ability to excel in various settings.

5.3 Model Ablation Study (RQ2)

This section investigates the impact of major components in our GPT-ST. To evaluate the impact of different components, we re-conduct pre-training for multiple ablated variants, and evaluate the performance of the downstream method using the newly pre-trained ablated models. GWN is utilized as the downstream approach, and two datasets METR-LA and NYC Taxi are employed for evaluation. The results are shown in Figure 5. From the results, we make the following observations:

Impact of basic components. i) -P. We remove the customized parameter learner in GPT-ST. ii) -C. We disable the hypergraph capsule clustering network to cancel the clustering operation. In this case, the spatial hypergraph NNs works on fine-grained regions directly. iii) -T. We leave the crosscluster relation learning unexplored by removing the high-level spatial hypergraph. The noticeable performance decay demonstrates the positive benefits brought by all three components, indicating that generating personalized ST parameters and modeling the intra- and inter-cluster ST dependencies can effectively capture complex ST correlations to benefit the predictions. Among the three components, the removal of the hypergraph capsule clustering network causes the most significant performance drop. This is because the clustering results also play an important role in many other components, including the cross-cluster dependencies and the cluster-aware adaptive masking.

Impact of mask mechanism. i) Ran0.25 and Ran0.75. We replace our adaptive mask strategy with random masking with a mask ratio of 0.25 (mask ratio same as ours) and 0.75 (mask ratio used in MAE [15] and STEP [31]), respectively. The results consistently demonstrate that our proposed mask strategy outperforms the random mask strategies. This can be attributed to the fact that our mask strategy effectively promotes the learning of intra- and inter-cluster relationships by GPT-ST, resulting in the generation of high-quality representations. Similar results are obtained in additional experiments conducted on the other two datasets, as shown in Table 2. ii) GMAE and AdaMAE. We compare our approach to two variants using the mask strategies proposed by GraphMAE [16] and AdaMAE [2]. Both variants perform worse than our methods, highlighting the importance of considering spatial and temporal patterns in the masking strategy. This further confirms the superiority of our adaptive masking approach that leverages clustering information.

Impact of pre-training strategy. To further investigate the effectiveness of the mask-reconstruction pre-training approach in spatio-temporal pre-training, we compare it to other pre-training methods: local-global infomax and contrastive pre-training. We used DGI [35] and GraphCL [47] as baselines, which are widely recognized pre-training strategies for GNNs. By replacing GPT-ST's pre-training strategy with these methods (denoted as "r/"), we observe improvements in performance compared to

models without pre-training. This indicates the adaptability and benefits of infomax and contrastive pre-training for representation learning in our model. However, our approach, which utilizes the maskreconstruction task, achieved the most significant performance enhancement. This can be attributed to the higher correlation between the mask-reconstruction task and the downstream regression task, leading to more effective learning of spatio-temporal representations. Additionally, our adaptive mask strategy plays a crucial role in facilitating the model to learn robust spatio-temporal representations by increasing the difficulty of the pre-training task.

-P

-C

-T

Ours

Ran0.25

Ran0.75

GMAE

AdaMAE

r/ GCL

METR-LA

2.9

3.0

3.1

MAE

METR-LA

6.1

6.2

6.3

6.4

RMSE

NYC_TAXI

5.0

5.1

5.2

5.3

5.4

MAE

NYC_TAXI

8.6

8.8

9.0

9.2

RMSE

Dataset

PeMS08(MAE/ RMSE)

model

GWN

MSDR

STWA

Ran mask

14.80/ 24.34

16.50/ 25.63

15.39/ 24.85

Ada mask

14.72

/

24.12

15.92

/

25.03

15.29

/

24.50

Dataset

NYC Citi Bike(MAE/ RMSE)

model

DMVSTNET

STMGCN

CCRNN

Ran mask

1.92/ 2.89

1.82/ 2.74

1.85/ 2.86

Ada mask

1.89

/

2.86

1.81

/

2.73

1.82

/

2.77

5.4 Investigation on Clustering Effect (RQ3)

To comprehensively demonstrate the effectiveness of the clustering process, we assess the interpretability of our GPT-ST framework by analyzing the embeddings generated by the Hypergraph Capsule Clustering Network (HCCN), as illustrated in Figure 6. We employ the T-SNE algorithm to visualize the high-dimensional embeddings produced by HCCN, mapping them to 2-dimensional vectors. Each category is represented by a distinct color, with 10 categories defined for hyperparameter H S . The region clustering is determined based on the probabilities of belonging to different categories. Upon examining the visualized embeddings, we observe that regions belonging to the same class exhibit tight clustering within a confined space, providing empirical evidence of the robust clustering capability of our hypergraph capsule clustering network.

In another case study conducted on the METR-LA dataset, we investigate the in-cluster region relations derived from the hypergraph capsule clustering network, as well as the cross-class dependencies obtained from the cross-cluster hypergraph network. The results, illustrated in Figure 7, demonstrate that the top-3 regions within the same category, as depicted in Figure 7(a) and 7(b), exhibit similar traffic patterns, indicating shared functionalities. For instance, regions near commercial areas (7(a)) experience an evening peak, while those near residential areas (7(b)) remain relatively stable. These observations align with real-world scenarios. Furthermore, focusing on Figure 7(c), we analyze the top-2 regions from two categories that have undergone shifts in traffic patterns over a specific time period, while sharing similar hyperedge weights in the cross-class transition. The results reveal that regions undergoing pattern shifts exhibit distinct traffic patterns while maintaining close interconnections within a short driving distance. These findings provide further evidence of the cross-cluster transition learning's ability to capture semantic-level relationships between regions, reflecting real-world traffic scenarios. These advantages contribute to the generation of high-quality representations in our GPT-ST framework, leading to improved performance in downstream tasks.

5.5 Model Efficiency Study (RQ4)

In this section, we assess the efficiency of our model. Specifically, we measure the per-epoch training time of all methods on the PEMS08 dataset, and the results are summarized in Table 3. To ensure fairness, all experiments are conducted on a system equipped with a GTX 3090 GPU and an Intel Core i9-12900K CPU. The batch size for all methods, except those tagged with 'b8', is set to 64 (where 'b8' denotes a batch size of 8). By combining the results from Table 1, we can conclude that while most of the baselines experience a minor decrease in training efficiency when enhanced by GPT-ST, there is a significant improvement in their performance. For

instance, some previous methods ( e . g ., STGCN) equipped with GPT-ST achieve comparable perfor-

1152 [97, 119, 74] 66.888885 6

1152 [27, 7, 177] 66.888885 9

Commercial district

(a) Case A for learned clusters

Residential district

(b) Case B for learned clusters

Commercial district

Residential district

Traffic speed

(c) Case for cross-class transitions

70

60

60

70

-T141201

50

50

paads se

40

Traffic speed

40

30

30

69

79

20

20

10

Region_98

Region_120

10

Region_28

Region_8

0

00:00

Region_75

04:00

00:80

Time of Day

12:00

16:00

20:00

24:00

0

00:00

Region_178

04:00

00:80

Time of Day

12:00

16:00

20:00

24:00

29

9

00:00

Regioe_192

04.00

,142

0980

12:00

Tine of Dey

16:0020:00

24:00

S.O74EE.

2880 [52, 141] [15, 191] 68.0 6

mance to advanced methods ( e . g ., MSDR), while maintaining superior training efficiency. In the comparison between GPT-ST and STEP, we observe that STEP incurs a substantial time cost, both in the pre-training stage and the downstream task stage, whereas GPT-ST enhances existing baselines in a lightweight manner. These cases demonstrate that our framework achieves a win-win situation in terms of performance and training efficiency, making it well-suited for practical applications.

5.6 Model Hyperparameter Experiment (RQ5)

To examine the impact of different hyperparameters on GPT-ST, we conduct parameter experiments on the NYC Taxi dataset using GWN as the downstream model. The results of these experiments are depicted in Figure 8, where the evaluation metrics are plotted as relative values.

In our investigation, we specifically focus on the total mask ratio r t and adaptive mask ratio r a as the main parameters of interest. Based on the findings presented in Figure 8, we observe that the optimal effect is achieved when the total mask ratio r t is set to 0.25. Increasing the mask ratio beyond this value does not improve the model's ability to learn better representations; instead, it leads to a decline in prediction performance. Furthermore, we explore the impact of the adaptive mask ratio r r , which determines the number of clusters that are entirely masked, thereby influencing the entire training process. The best performance is obtained when the adaptive mask ratio is set to 1. This suggests that this specific configuration enhances the model's ability to learn regional relations, both within individual clusters and across different clusters.

6 Conclusion

In this work, we introduce a scalable and effective pre-training framework tailored for spatio-temporal prediction tasks. Our framework starts with a foundational pre-training model that focuses on capturing spatio-temporal dependencies. It leverages a customized parameter learner and hierarchical hypergraph networks to extract customized spatio-temporal features and region-wise semantic associations, respectively. To further improve the model's performance, we propose an adaptive masking strategy. This strategy guides the model to learn inferential reasoning capabilities by considering both intra-class and inter-class relations during the pre-training stage. We conduct extensive experiments on four real-world datasets, demonstrating the efficacy of our proposed GPT-ST framework in enhancing the performance of downstream baselines across different spatio-temporal tasks.

0.0

0.2

0.4

0.6

0.8

1.0

.00

.02

.04

.06

Relative values

MAE

RMSE

(a) Total mask ratio

m

t

0.25

0.50

0.75

1.00

.00

.01

.02

Relative values

MAE

RMSE

(b) Adaptive mask ratio m a

References

Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. In NeurIPS , pages 17804-17815, 2020.

Wele Gedara Chaminda Bandara, Naman Patel, Ali Gholami, Mehdi Nikkhah, Motilal Agrawal, and Vishal M. Patel. Adamae: Adaptive masking for efficient spatiotemporal learning with masked autoencoders. In NeurIPS , 2022.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, et al. Language models are few-shot learners. In NeurIPS , 2020.

Razvan-Gabriel Cirstea, Bin Yang, Chenjuan Guo, Tung Kieu, and Shirui Pan. Towards spatio- temporal aware traffic time series forecasting. In ICDE , pages 2900-2913, 2022.

Zhiyong Cui, Kristian Henrickson, Ruimin Ke, and Yinhai Wang. Traffic graph convolutional recurrent neural network: A deep learning framework for network-scale traffic learning and forecasting. TITS , pages 4883-4894, 2020.

Rui Dai, Shenkun Xu, Qian Gu, Chenguang Ji, and Kaikui Liu. Hybrid spatio-temporal graph convolutional network improving traffic prediction with navigation data. In KDD , pages 3074-3082, 2020.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL , pages 4171-4186, 2019.

Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR , pages 12124-12134, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. In KDD , pages 364-373, 2021.

Xu Geng, Yaguang Li, Leye Wang, Lingyu Zhang, Qiang Yang, Jieping Ye, and Yan Liu. Spatiotemporal multi-graph convolution network for ride-hailing demand forecasting. In AAAI , pages 3656-3663, 2019.

Kan Guo, Yongli Hu, Yanfeng Sun, Sean Qian, Junbin Gao, and Baocai Yin. Hierarchical graph convolution network for traffic forecasting. In AAAI , pages 151-159, 2021.

Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial-temporal graph convolutional networks for traffic flow forecasting. In AAAI , pages 922-929, 2019.

Liangzhe Han, Bowen Du, Leilei Sun, Yanjie Fu, Yisheng Lv, and Hui Xiong. Dynamic and multi-faceted spatio-temporal deep learning for traffic speed forecasting. In KDD , pages 547-555, 2021.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR , pages 15979-15988, 2022.

Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In KDD , pages 594-604, 2022.

Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In KDD , pages 1857-1867, 2020.

Ming Jin, Yu Zheng, Yuan-Fang Li, Siheng Chen, Bin Yang, and Shirui Pan. Multivariate time series forecasting with dynamic graph neural odes. TKDE , pages 1-14, 2022.

Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. What to hide from your students: Attention-guided masked image modeling. In ECCV , pages 300-318, 2022.

Chaoliu Li, Lianghao Xia, Xubin Ren, Yaowen Ye, Yong Xu, and Chao Huang. Graph transformer for recommendation. arXiv preprint arXiv:2306.02330 , 2023.

Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. SemMAE: Semantic-guided masking for learning masked autoencoders. In NeurIPS , 2022.

Mengzhang Li and Zhanxing Zhu. Spatial-temporal fusion graph neural networks for traffic flow forecasting. In AAAI , pages 4189-4196, 2021.

Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: data-driven traffic forecasting. In ICLR , 2018.

Zhonghang Li, Chao Huang, Lianghao Xia, Yong Xu, and Jian Pei. Spatial-temporal hypergraph selfsupervised learning for crime prediction. In ICDE , pages 2984-2996, 2022.

Dachuan Liu, Jin Wang, Shuo Shang, and Peng Han. Msdr: Multi-step dependency relation networks for spatial temporal forecasting. In KDD , pages 1042-1050, 2022.

Yingtao Luo, Qiang Liu, and Zhaocheng Liu. Stan: Spatio-temporal attention network for next location recommendation. In WWW , pages 2177-2185, 2021.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.021555 , 2022.

Zhaopeng Qiu, Xian Wu, Jingyue Gao, and Wei Fan. U-bert: Pre-training user representations for improved recommendation. In AAAI , number 5, pages 4320-4327, 2021.

Xubin Ren, Lianghao Xia, Jiashu Zhao, Dawei Yin, and Chao Huang. Disentangled contrastive collaborative filtering. arXiv preprint arXiv:2305.02759 , 2023.

Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In NeurIPS , 2017.

Zezhi Shao, Zhao Zhang, Fei Wang, and Yongjun Xu. Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting. In KDD , pages 1567-1577, 2022.

Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. In AAAI , pages 914-921, 2020.

Amauri Souza, Diego Mesquita, Samuel Kaski, and Vikas Garg. Provably expressive temporal graph networks. In NeurIPS , pages 32257-32269, 2022.

Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. arXiv preprint arXiv:2310.13023 , 2023.

Petar Veliˇ ckovi´ c, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In International Conference on Learning Representations , 2019.

Xiaoyang Wang, Yao Ma, Yiqi Wang, Wei Jin, Xin Wang, Jiliang Tang, Caiyan Jia, and Jian Yu. Traffic flow prediction via spatial temporal graph neural network. In WWW , pages 1082-1092, 2020.

Yanbang Wang, Yen-Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representation learning in temporal networks via causal anonymous walks. In ICLR , 2021.

Wei Wei, Chao Huang, Lianghao Xia, and Chuxu Zhang. Multi-modal self-supervised learning for recommendation. In WWW , pages 790-800, 2023.

Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In KDD , pages 753-763, 2020.

Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In IJCAI , 2019.

Congxi Xiao, Jingbo Zhou, Jizhou Huang, Hengshu Zhu, Tong Xu, Dejing Dou, and Hui Xiong. A contextual master-slave framework on urban region graph for urban village detection. In ICDE , pages 736-748, 2023.

Dongkuan Xu, Wei Cheng, Dongsheng Luo, Haifeng Chen, and Xiang Zhang. Infogcl: Information-aware graph contrastive learning. NeurIPS , pages 30414-30425, 2021.

Huaxiu Yao, Xianfeng Tang, Hua Wei, Guanjie Zheng, and Zhenhui Li. Revisiting spatial-temporal similarity: A deep learning framework for traffic prediction. In AAAI , pages 5668-5675, 2019.

Huaxiu Yao, Fei Wu, Jintao Ke, Xianfeng Tang, Yitian Jia, Siyu Lu, Pinghua Gong, Jieping Ye, Didi Chuxing, and Zhenhui Li. Deep multi-view spatial-temporal network for taxi demand prediction. In AAAI , pages 2588-2595, 2018.

Junchen Ye, Leilei Sun, Bowen Du, Yanjie Fu, Xinran Tong, and Hui Xiong. Co-prediction of multiple transportation demands based on deep spatio-temporal neural network. In KDD , pages 305-313, 2019.

Junchen Ye, Leilei Sun, Bowen Du, Yanjie Fu, and Hui Xiong. Coupled layer-wise graph convolution for transportation demand prediction. In AAAI , pages 4617-4625, 2021.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurIPS , pages 5812-5823, 2020.

Bing Yu, Haoteng Yin, et al. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In IJCAI , 2018.

Rose Yu, Yaguang Li, Cyrus Shahabi, Ugur Demiryurek, and Yan Liu. Deep learning: A generic approach for extreme condition traffic forecasting. In SDM , pages 777-785, 2017.

Jinlei Zhang, Feng Chen, Zhiyong Cui, Yinan Guo, and Yadi Zhu. Deep learning architecture for short-term passenger flow forecasting in urban rail transit. TITS , pages 7004-7014, 2021.

Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. In AAAI , 2017.

Qianru Zhang, Chao Huang, Lianghao Xia, Zheng Wang, Zhonghang Li, and Siuming Yiu. Automated spatio-temporal graph contrastive learning. In WWW , pages 295-305, 2023.

Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self-supervised learning for molecular property prediction. NeurIPS , pages 15870-15882, 2021.

Ling Zhao, Yujiao Song, Chao Zhang, Yu Liu, Pu Wang, Tao Lin, Min Deng, and Haifeng Li. T-gcn: A temporal graph convolutional network for traffic prediction. TITS , pages 3848-3858, 2020.

Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. Gman: A graph multi-attention network for traffic prediction. In AAAI , pages 1234-1241, 2020.

A Supplementary Material

In the supplementary material, we provide additional information and details in A.1. This section covers the introduction of data, key parameter settings, comparisons with baselines, optimization methods, and the algorithm process of our method. Furthermore, A.2 presents supplementary experiments for our model, including visualization experiments and replication studies. Additionally, we discuss the reasons behind utilizing hypergraphs as the temporal encoder in A.3. Finally, the limitations and broader impacts of our work are discussed in A.4.

A.1 Data and Implementation Details

Data . The statistical information of the aforementioned four real-world datasets is presented in Table 4. These datasets primarily consist of daily spatio-temporal statistics in the United States. Specifically, the PeMS08 and METR-LA datasets were collected from 7 roads in the San Bernardino area and the road network of Los Angeles County, respectively. On the other hand, the NYC Taxi and NYC Citi Bike datasets were obtained from New York City.

Parameters . The latent representation dimensionality ( d ) and the customized parameter ( d ′ ) are both set to 64 and 16, respectively. The number of hyperedges ( H T , H S , H M ) is set to 8, 10, and 16, respectively. We perform 2 dynamic routing iterations. Additionally, the balance ratio ( λ ) for L r and L kl is set to 0.1, and the total mask ratio ( r t ) is set to 0.25.

Baselines . We selected 13 methods as baselines and categorized them into distinct groups:

Hybrid spatio-temporal prediction method:

DMVSTNET [44]: This framework for demand forecasting utilizes RNNs, convolutional networks, and fully connected layers. RNNs capture temporal correlation, convolutional networks handle spatial correlation, and fully connected layers address regional semantic correlation.

Spatio-temporal prediction method based on GNNs:

STGCN [48]: It employs convolution operations to model both temporal and spatial dependencies.

GWN [40]: This method utilizes a learnable graph structure to capture spatial dependencies and incorporates the diffuse convolution technique to capture temporal dependencies.

GWN* [40]: To evaluate the effectiveness of GWN, we utilize long-term time series data as input. Our evaluation focuses on predicting the data for the next hour using the data from the preceding two weeks. We employ a simple linear layer to process the long-term features in this prediction.

TGCN [54]: In this method, both RNNs and GNNs are employed to model temporal dependence and capture spatial correlation, respectively.

MTGNN [39]: This method utilizes a learnable graph structure to model associations between multiple variables. It incorporates dilated convolution and skip connections to jointly capture spatio-temporal dependencies.

MSDR [25]: This model proposes a variant of RNNs to make full use of historical time step information, and combines GNNs to model long-range spatio-temporal dependence.

STMGCN [11]: This method is a spatio-temporal framework designed for demand forecasting. It incorporates region association from various aspects and combines the power of RNNs and GNNs to effectively model both spatial and temporal relations.

CCRNN [46]: This is an approach specifically designed for traffic demand prediction. It leverages multiple layers of GNNs, each assigned with different adjacency matrices, to capture hierarchical spatial associations. RNNs are employed to establish temporal dependencies within the network.

STSGCN [32]: This work captures spatio-temporal correlations by constructing a local spatiotemporal graph, enabling the synchronous modeling of these correlations.

STFGNN [22]: This work proposes a data-driven approach that utilizes a gated convolution method to generate spatio-temporal graphs. By learning spatial and temporal dependencies, the approach effectively captures the correlations within the data.

Attention-based spatio-temporal prediction method:

ASTGCN [13]: It utilizes attention and GNNs to capture spatio-temporal periodic dependencies.

STWA [4]: It integrates location-specific and time-varying parameters into the attention network to effectively capture dynamic spatio-temporal correlations.

Spatio-temporal prediction via differential equation:

STGODE [10]: This method captures spatial dependencies by enhancing GNNs through the use of ordinary differential equations. Additionally, temporal dependencies are modeled using a dilated temporal convolutional network.

A.1.1 Optimization Method

In the GPT-STframework, multiple temporal encoders (Section 4.1) and spatial encoders (Section 4.2) are stacked together to generate the final embeddings. The architecture includes two temporal encoders followed by a spatial encoder, forming a spatio-temporal (ST) encoding block. The final embeddings are generated by passing the data through two spatio-temporal encoding blocks.

During the pre-training phase, the predictions ˆ Y ∈ R R × T × F are computed by applying a linear layer to the hidden dimension. To optimize the parameters, the absolute error loss function is utilized, following the approach employed in prior works such as [55, 1].

Let X represent the input, consisting of F features across R regions in the previous T time slots. The two layers of the MLP network discussed in Sec 4.3 can be formalized as follows:

In the given formulation, W p r ∈ R d × d and b p r ∈ R d represent the region-specific parameters, while W p t ∈ R d × d and b t p ∈ R d are the time-dynamic parameters created as in Equation 4. The predictions q ∈ R H s × R × T are obtained from Q through a linear layer followed by a softmax function. The KL divergence loss function L kl can be formulated as follows:

In this case, ¯ c ∈ R H s × R × T is considered as the ground truth classification result. To balance the contribution of the two loss functions, we introduce a parameter λ to adjust their ratio, as follows:

In the downstream task stage, we utilize the pre-trained embeddings ζ ∈ R R × T × d along with the raw data representation E ′ = ¯ X · e 0 (without any mask operation) as the input for the downstream model. To fuse these two inputs, we employ a simple gated fusion layer proposed by [55], which can be formalized as follows:

In the given formulation, W h, 1 , W h, 2 ∈ R d × d and b h ∈ R d are learnable parameters. The variable z represents the gate operation, and δ ( · ) denotes the sigmoid activation function. It is important to note that we prevent the backpropagation of ζ at this stage. By fusing ζ and E ′ using the gated fusion layer, downstream models can leverage the knowledge gained during the pre-training stage for improved predictions. The specific optimization methods employed may vary depending on the downstream models, such as mean absolute error [1, 25] or Huber loss [22, 32].

A.1.2 Algorithm Process

Algorithm 1 represents the process of the adaptive mask strategy described in Section 4.3. On the other hand, Algorithm 2 illustrates the algorithmic process during the pre-training stage.

Algorithm 2: Learning Process of GPT-ST Framework

A.2 Additional experiments

A.2.1 Visualization Study

Figure 9 presents the reconstruction results of the model during the pre-training stage. In the figure, gray, red, and blue lines respectively represent the masked signals, predicted values, and visible

signals. The results demonstrate that GPT-ST can accurately predict the masked signals based on the visible signals, regardless of whether a random mask or an adaptive mask is used. Furthermore, it can be observed that the masked signals generated by the adaptive strategy exhibit greater temporal consistency compared to those generated by the random strategy. This is because the category attribute of a region is less likely to change within a short period of time. The continuous mask and cluster mask are effective in increasing the difficulty of the reconstruction task, enabling GPT-ST to learn robust spatio-temporal representations even with low masking ratios.

Figure 10 summarizes the enhancement effect of your model on downstream baselines. The figure showcases the predicted performance of four baselines on the PEMS08 dataset, where the blue, green, and red lines respectively represent the ground truth, the original performance, and the enhanced performance of the baselines. With the assistance of GPT-ST, the prediction performance of the baseline models experiences significant improvements on certain subsets, as highlighted by the red box. This confirms that the pre-trained model, equipped with customized parameter learners and mechanisms for encoding intra- and inter-class spatial patterns, can provide a discriminative and semantic representation for downstream tasks. As a result, it effectively compensates for the limitations of different baselines.

0

100

200

300

400

500

600

Time of Day

100

200

300

400

500

600

Traffic flow

y_mask

y_pred

y_unmask

(a) Case A for Ran mask

0

100

200

300

400

500

600

Time of Day

100

200

300

400

500

600

Traffic flow

y_mask

y_pred

y_unmask

(b) Case A for Ada mask

2000

2100

2200

2300

2400

2500

2600

Time of Day

50

100

150

200

250

300

350

400

Traffic flow

y_mask

y_pred

y_unmask

(c) Case B for Ran mask

2000

2100

2200

2300

2400

2500

2600

Time of Day

50

100

150

200

250

300

350

400

Traffic flow

y_mask

y_pred

y_unmask

(d) Case B for Ada mask

GPST

(a) STGCN

GPST

(b) TGCN

GPST

(c) GWN

GPST

(d) MSDR

A.2.2 Replication Study

In this section, the deviation of GPT-ST's performance over random parameter initialization is investigated. The statistical results are presented in Table 5. For each performance evaluation on the PEMS08 dataset, we randomly selected 5 seeds. The results indicate that GPT-ST exhibits strong adaptability to different parameter initialization settings, consistently providing stable and high-quality spatio-temporal representations for downstream models.

A.3 Analysis of Hypergraph as Temporal Encoder

Existing time encoders in spatio-temporal prediction schemes, such as recurrent neural networks (RNNs), temporal convolutional networks (TCNs), and attention mechanisms, have certain limitations. RNNs are prone to losing long-term information due to the vanishing gradient problem. TCNs can only capture temporal information within a limited neighborhood defined by the size of the convolution kernel. Although attention mechanisms consider the interrelationship between time steps, their computational efficiency decreases significantly when the length of the time series ( T ) is large, resulting in a time complexity of O ( T 2 ) . These limitations hinder the ability of existing approaches to effectively capture and model the temporal dynamics in spatio-temporal data.

To address the aforementioned limitations, we propose to utilize the hypergraph neural network as a temporal encoder for time-dependent modeling. A hypergraph is composed of multiple sets of hyperedges, where each hyperedge serves as a learnable information hub connecting time-step information with different weights. By treating different hyperedges as a means of integrating temporal relationships across different dimensions, the hypergraph neural network offers a flexible control over complexity by adjusting the number of hyperedges. This approach allows for modeling long-term temporal dependencies while maintaining low computational costs. The hypergraph neural network combines the advantages of capturing temporal dynamics effectively and efficiently, making it a promising solution for spatio-temporal prediction tasks.

A.4 Limitations and Broader Impacts

The proposed GPT-ST has demonstrated its effectiveness in improving the prediction performance of downstream models. However, it also has two main limitations: i) Task-specific pre-training: GPT-ST requires pre-training for each specific downstream task. This is because different prediction tasks often have distinct data formats and distributions, necessitating task-specific pre-training. For instance, a GPT-ST pre-trained on task A cannot be directly applied to task B. ii) Increased time cost: Both the pre-training process and the enhancement process in the downstream task stage add to the prediction time cost. These additional computations can impact real-time applications or scenarios with stringent time constraints. In future research, we aim to explore more generalized and versatile spatio-temporal pre-training frameworks, along with lightweight algorithms, to address the task-specific pre-training requirement and further reduce the computational overhead.

Peer reviews:
comments: SUMMARY:
This paper proposes a novel pre-training framework (GPST) for ST prediction, it mainly integrates the ST parameter personalization scheme and the region-wise semantic association mechanism into the pre-training model, and conduct the training in an unsupervised manner. Besides, the proposed model GPST adopts the hierarchical hypergraph structure to capture the semantic-level association of multiple regions from a global perspective. Extensive experiments are conducted.

STRENGTHS:
1. This paper is well-presented and well-organized.
2. This paper introduces a spatio-temporal pre-training framework that can be easily integrated into downstream baselines and improve their performance.
3. Detailed experimental results are provided to validate the model.

WEAKNESSES:
1. The model's performance seems to heavily rely on the pre-training stage. If the pre-training is not done correctly or if the pre-training data is not representative of the test data, the model's performance could suffer, how can this case be handled?
2. Why not include POIs, which should benefit the prediction performance.

QUESTIONS:
See above.

LIMITATIONS:
Noone
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduce pre-training task for spatio-temporal. The algorithm design includes a spatial-temporal knowledge extractor and a pre-training mechanism.\n\nThe design of the pre-training methods is to resolve the two limitations of existing spatial-temporal learning: (1) lack of comprehensive personalization (2) insufficient consideration of modelling semantic.\n\nSTRENGTHS:\nThis paper study an important problem: GNN-pretraining.\n\nWEAKNESSES:\n1. The presentation wasn't very clear, please refer to questions.\n\n2. Missing many recent baselines, e.g., TGAT, JODIE, TGN. The authors can compare with those works using https://github.com/amazon-science/tgl. Besides, there are most more recent methods [1-3] need to be included in discussion.\n\n[1] Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks\n[2] Provably expressive temporal graph networks\n[3] Do We Really Need Complicated Model Architectures For Temporal Networks?\n\n3. Experiment dataset size are all relatively small. I am not sure this method can scale to very large graph from experiments.\n\nQUESTIONS:\n1. Could you please explicitly summarize what is the SSL pre-training method different from the method on static graph? My impression is the main contribution is due to spatial-temporal vs static graph setting. The mask and predict\n\n2. The hyper-edge definition is not very clear to me. At line 116 \"each of which connects multiple vertices\", could you please explain how an edges can connect to more than 2 nodes?\n\n3. How intra-class pattern and inter-class relation are defined (line 138). Is this coming from dataset, or the authors define them based on human knowledge per dataset.\n\n4. Is the pre-training task a link prediction task? Do you explicitly consider the potential information leakage issue during pre-training. For example, the positive edges for pre-training might be sampled as a neighbor when computing the node representation.\n\n5. Will GNN size affect the pre-training efficiency?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a spatio-temporal pre-training framework that can be easily integrated into downstream baselines. The framework comprises personalized parameter learners and hierarchical hypergraph networks. The former enables the acquisition of spatio-temporal personalized representation and the latter capture the intra-class and inter-class correlations among regions. Besides, the authors design an adaptive mask strategy to guide the model in learning diverse relationships across regions. Experiments are conducted on representative benchmarks on four real-world datasets to evaluate the proposed framework.\n\nSTRENGTHS:\n1. The authors propose a novel pre-training framework for Spatio-Temporal prediction which can be easily applied to existing advanced ST neural networks.\n\n2. The authors devise an adaptive mask strategy that guides the model in a progressive manner, facilitating the acquisition of useful knowledge from other categories to recover unknown categories. The comparisons with other mask strategies demonstrate the superiority of the designed mechanism.\n\n3. The authors evaluate both the original performance and enhanced performance with the proposed framework of different baselines on four datasets. The results indicate that the proposed model significantly improves the downstream baseline prediction performance and confirm the effectiveness of the framework.\n\nWEAKNESSES:\n1. The presentation of this paper should be improved:\n\n    (1) The explanations of some concepts are not clear. For instance, the meaning of the symbol $M_{r,t}$ in Section 4.2 is unclear. Does it represent the mask operation? Furthermore, the first-level hyperedges mentioned in Section 4.3.2 lack clear explanation, as it is the first occurrence of the term \"first level\" in the paper.\n\n    (2) Equation (4) is referenced multiple times in the paper to illustrate the process of generating embeddings. However, it is necessary to provide a more detailed explanation of how the equation is actually implemented. For instance, in Line 173 of Section 4.2, it would be helpful to provide specific equations for the computations involved. The inclusion of concrete examples would greatly enhance the understanding of Equation (4) and its functionality.\n\n    (3) In Section 4.2, there are multiple subscripts used, including (r,t), r, and t. It would be better if provide a more detailed explanation to clarify the differences between these descriptions. \n\n    (4) In Section 5.1.1, the paper mentions the use of the absolute error loss function to optimize the parameters. However, there is a lack of detailed description regarding how this loss function is precisely employed. It would be beneficial to provide a more thorough explanation of how the optimization process utilizes the absolute error loss function, including any specific computations.\n\n2. The authors adopt the hierarchical hypergraph structure to capture the semantic-level association of multiple regions from a global perspective. However, the construction of the hypergraph requires a more detailed description. Specifically, the definition of hyperedges and capsules in Section 4.3 needs to be clarified. The paper mentions that hyperedges are treated as high-level capsules, but what exactly does a capsule consist of? Furthermore, it would be helpful to explain the type of hypergraph connection that a hyperedge represents.\n\n3. Some related work with similar ideas is not discussed. For example, the paper [1] also builds a pre-training model on urban data and builds a personalized model in a specific area. Though the idea is not the same, the author should discuss the difference in their related work at least.\n\n4. It would be beneficial to include a brief discussion on future work in the paper.\n\n[1] A Contextual Master-Slave Framework on Urban Region Graph for Urban Village Detection, ICDE 2023.\n\nQUESTIONS:\n1. What’s the “another initial” in Line 206 of Section 4.3.1 mean?\n\n2. Is the aforementioned c in Section 4.4 from Section 4.3.1?\n\nLIMITATIONS:\nThe authors have not adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a spatio-temporal pre-training framework called GPST (Generative Pre-training framework for Spatial Temporal prediction) to enhance the performance of existing models in traffic management and travel planning. The goal is to address the challenges of integrating and expanding refined models, while achieving better predictive performance.\n\nThe framework consists of two main components. (1) a spatio-temporal knowledge extractor that utilizes personalized parameter learners and hierarchical hypergraph networks. These modules are designed to model personalized representations and semantic relationships between regions, which have been overlooked in previous work. (2) an adaptive mask strategy is proposed to guide the knowledge extractor in learning robust spatio-temporal representations.\n\nThe paper conducts extensive experiments on representative benchmarks and demonstrates the effectiveness of the proposed GPST method.\n\nSTRENGTHS:\n1. The two research questions are interesting. Question (1) in personalized information extraction is well studied in STGNN. Question (2) is relatively novel since region functions could make a difference but not considered too much in the literature.\n2. The experiments results look promising.\n\nWEAKNESSES:\n1. Related works should include SSL works from both Graph and Multivariate Time Series. I think current version has some missing references in esp. Graph SSL, which could be related to the strategy developed in this paper (needs further justification)\n2. Figure 3 is really small to understand clearly. Visual illustration is better decoupled into text description.\n3. Section 4 has too many details that could be moved to appendix since these detailed designs are not quite novel, though important. I'd suggest this section to be more compact and highlighted.\n4. I have some concerns about the improvement by using GPST. It is clear in Table 1 that some methods benefit a lot and some not that much. I expect a more detailed analysis on this. Is some design of GPST redundant in GWN, MTGNN, etc? What is the core part of GPST that makes performance better? Or, is GPST dependent of certain architecture designs? How are these two factors (architecture and ssl strategy) decoupled?\n\nQUESTIONS:\nThanks for clarifying the above mentioned weaknesses.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}