SlowGuess/NeurIPS_2023_OCR huggingface dataset:
After the iterations of the dynamic routing algorithm, to make use of both  $b_{i,r,t}$  and  $\mathbf{H}_{i,r,t}'$  for better region-hyperedge relation learning, GPT-ST combines the two sets of weights and generates the final embeddings  $\bar{\mathbf{s}}_{i,t} \in \mathbb{R}^d$ . We begin by replacing  $b_{i,r,t}$  with  $(b_{i,r,t} + \mathbf{H}_{i,r,t}')$  to obtain a new weight vector  $\bar{c}_{i,r,t} \in \mathbb{R}$ , and then utilize  $\bar{c}_{i,r,t} \in \mathbb{R}$  to calculate the final embeddings  $\bar{\mathbf{s}}_{i,t}$ .
4.2.2 Cross-Cluster Relation Learning
With the clustered embeddings  $\bar{\mathbf{s}}_{i,t}$ , we propose to model the inter-cluster relations via a high-level hypergraph neural network. Specifically, the refined cluster embeddings  $\hat{\mathbf{S}}\in \mathbb{R}^{H_S\times T\times d}$  are calculated by message passing between the  $H_{S}$  cluster centroids and  $H_{M}$  high-level hyperedges as follows:
$$
\hat {\mathbf {S}} = \text {H y p e r P r o p a g a t e} (\tilde {\mathbf {S}}) = \operatorname {s q u a s h} \left(\sigma \left(\mathbf {H} ^ {\prime \prime \top} \cdot \sigma \left(\mathbf {H} ^ {\prime \prime} \cdot \tilde {\mathbf {S}}\right)\right) + \tilde {\mathbf {S}}\right) \tag {7}
$$
where  $\tilde{\mathbf{S}}\in \mathbb{R}^{H_S T\times d}$  denotes the reshaped embedding matrix obtained from  $\bar{\mathbf{s}}_{i,t}$  for  $i\in H_S$  and  $t\in T$ .  $\mathbf{H}''\in \mathbb{R}^{H_M\times H_S T}$  denotes the high-level hypergraph structure, which is obtained through the aforementioned personalized parameter learner. This parameter customization aggregates the temporal features  $\bar{\mathbf{z}}_t^{(d)},\bar{\mathbf{z}}_t^{(w)}$  for all  $t\in T$  as the input, and generates the parameters as in Eq 4.
After refining the cluster representations  $\hat{\mathbf{s}}_{i,t} \in \mathbb{R}^d$  for  $i \in H_S, t \in T$ , we propagate the clustered embeddings back to the regional embeddings with the low-level hypergraph structure, as follows:
$$
\Psi_ {r, t} = \sigma \left(\sum_ {i = 1} ^ {H _ {S}} c _ {i, r, t} \cdot \hat {\mathbf {s}} _ {i, t} \mathbf {W} _ {r} ^ {\prime \prime} + \mathbf {b} _ {r} ^ {\prime \prime} + \boldsymbol {\Gamma} _ {r, t}\right) \tag {8}
$$
where  $\Psi_{r,t} \in \mathbb{R}^d$  denotes the new region embedding for the  $r$ -th region in the  $t$ -th time slot.  $c_{i,r,t} \in \mathbb{R}$  denotes the weight for the low-level hypergraph capsule network.  $\mathbf{W}_r'' \in \mathbb{R}^{d\times d}$  and  $\mathbf{b}_r'' \in \mathbb{R}^d$  denotes the region-specific transformation and bias parameters generated by the customized parameter learner.
4.3 Cluster-aware Masking Mechanism
Inspired by semantic-guided MAE [21], we design a cluster-aware masking mechanism to enhance the intra- and inter-cluster relation learning for our GPT-ST. The adaptive masking strategy incorporates the foregoing learned clustering information  $\overline{c}_{i,r,t}$  to develop an easy-to-hard masking process. In particular, at the beginning of the training, we randomly mask a portion of regions for each cluster, in which case the masked values can be easily predicted by referring to in-cluster regions that share similar ST patterns. Subsequently, we gradually increase the masking proportion for certain categories, to increase the prediction difficulty for these clusters by reducing their relevant information. Finally, we completely mask the signals of some clusters, facilitating the ability of cross-cluster knowledge transfer for the pre-trained model. This adaptive masking process is depicted in Figure 4.
However, directly utilizing the learned cluster information  $\bar{c}_{i,r,t}$  to generate the cluster-aware mask  $\mathbf{M}$  is not feasible. This is because the cluster information is calculated by deeper layers of our GPT-ST network, while the generated mask is required as input to the network  $(\mathbf{M} \odot \mathbf{X})$ . To address this, we employ a two-layer MLP network with customized parameters to predict the learned results of  $\bar{c}_{i,r,t}$ . In particular, the transformations and bias vectors in this MLP network are replaced with time-dynamic and node-specific parameters generated by the customized parameter learner (Eq 4). Subsequently, a linear layer and softmax(\cdot) function are used to obtain the predictions  $q_{i,r,t} \in \mathbb{R}$  for  $i \in H_S, r \in R, t \in T$ . To optimize the distribution of  $q_{i,r,t}$ , we utilize the KL divergence loss function  $\mathcal{L}_{kl}$  with the ground truth  $\bar{c}_{i,r,t}$ . It is important to note that the backpropagation of  $\bar{c}_{i,r,t}$  is prevented in this step. The category is the maximum probability according to  $q_{i,r,t}$  is taken as the classification result. See Algorithm 1
Figure 4: Illustration for the adaptive masking strategy.
In this section, we verify the effectiveness of GPT-ST by answering the following five questions:
- RQ1: How does our proposed pre-training framework GPT-ST benefit different downstream baseline methods, in terms of prediction accuracy on diverse spatio-temporal prediction tasks?
- RQ2: How effective are the designed technical modules of our GPT-ST framework?
- RQ3: How are the interpretation ability of the learned global region clusters and the inter-cluster transition patterns in the hierarchical spatial pattern encoding of our GPT-ST framework.
- RQ4: How efficient is GPT-ST in both the pre-training stage and the downstream task stage?
- RQ5: How does varying the mask ratio setting affect the performance of GPT-ST?
To evaluate the effectiveness of our proposed method, we conduct experiments on four real-world ST datasets including: PEMS08 [32], METR-LA [23], NYC Taxi and NYC Citi Bike [46], which records the traffic flow, traffic speed, taxi order records, and bike order records, respectively.
We choose Mean Absolute Error (MAE), Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE) as the evaluation metrics, which are widely used in ST prediction tasks [52]. The lower value of the metrics, the better performance of the model. Following previous works [13, 32, 22], the number of time slots  $L$  is set to 12. We divide METR-LA dataset into training set, validation set and test set in a ratio of 7:1:2 and 6:2:2 for others datasets. The final model parameters are selected according to the optimal effect (minimum MAE value) of the validation set.
For optimization, in the pre-training phase, the absolute error loss function  $\mathcal{L}_r$  is used to optimize the parameters and a hyperparameter  $\lambda$  is adopted to balance the weight of  $\mathcal{L}_r$  and  $\mathcal{L}_{kl}$ . In the downstream task stage, we firstly fuse the embedding  $\zeta$  and the raw signal  $\mathbf{X}$  as the input to the downstream model, and the optimization strategies are distinct for different downstream models. Detailed descriptions of datasets and evaluation are presented in the supplementary material A.1.
5.2 Main Results (RQ1)
In this section, we mainly investigate whether GPT-ST improves the forecasting for downstream tasks. To achieve this, we evaluate both the original performance and enhanced performance (w/ GPT-ST) of different baselines on four datasets, and the results are shown in Table 1. Note that since DMVSTGCN, STMGCN and CCRNN are specially designed for demand forecasting, we eliminate them in traffic forecasting tasks. The results indicate that the proposed model significantly improves the prediction performance of different downstream baselines on all datasets, which confirms the effectiveness of the our framework. We analyze the promotion effect of GPT-ST from three dimensions:
Consistent improvements by GPT-ST. We observe that GPT-ST improves different types of methods (e.g., GNN-based, or attention-based models), and such positive effect does not partially to a certain category of baselines, which verifies the generalization ability of GPT-ST. We attribute this improvement to the MAE pre-training with the awareness of intra- and inter-cluster ST relations.
Difference among baseline models. When compared to recently proposed methods like MSDR, our GPT-ST exhibits more substantial improvements when applied to classical baselines such as STGCN. Specifically, in traffic flow prediction using the PEMS08 dataset, MSDR shows improvements of 0.55 in MAE, 0.60 in RMSE, and  $0.21\%$  in MPAE, while STGCN demonstrates improvements of 1.61 in MAE, 2.71 in RMSE, and  $0.63\%$  in MPAE. One possible explanation for these findings is that advanced baselines like MSDR are already well-designed and comprehensive in modeling various factors. Consequently, they are capable of encoding abundant knowledge independently, which may diminish the utility of the additional signals provided by the pre-training model. Conversely, classical baselines like STGCN may derive greater benefits from the insights offered by our GPT-ST due to their simpler design and potentially limited capacity to capture complex relationships.
Comparison with pre-training methods. In addition to evaluating our pre-training method against spatio-temporal prediction methods without pre-training, we also compare it with the competitive pre-training baseline STEP. This method utilizes long-term time series as input for pre-training to enhance the performance of the downstream model (GWN). Despite our GPT-ST using only short-term data during the pre-training stage, it outperforms STEP on several metrics. Furthermore, our GPT-ST demonstrates an even larger performance advantage in scenarios where long-term data is insufficient, e.g., the taxi and bike demand prediction tasks. This highlights the broader applicability of our proposed GPT-ST framework, emphasizing its ability to excel in various settings.
Table 1: Overall performance comparison on different datasets in terms of MAE, RMSE and MAPE.
Model	Dataset Metrics	PEMS08	METR-LA	NYC Taxi	NYC Citi Bike
MAE	RMSE	MAPE	MAE	RMSE	MAPE	MAE	RMSE	MAPE	MAE	RMSE	MAPE
STGCN	17.85	28.64	11.08%	3.55	7.39	10.02%	5.68	11.55	38.10%	2.04	3.34	51.10%
w/ GPT-ST	16.24	25.93	10.45%	3.29	6.85	9.14%	5.09	10.05	35.21%	1.82	2.82	49.15%
DMVSTNET	-	-	-	-	-	-	7.73	14.26	69.86%	2.27	3.65	61.89%
w/ GPT-ST	-	-	-	-	-	-	5.27	9.18	40.56%	1.89	2.86	55.01%
TGCN	21.44	31.82	15.67%	3.70	7.28	10.65%	9.73	19.39	83.50%	2.23	3.72	62.90%
w/ GPT-ST	17.34	26.53	12.26%	3.16	6.41	8.88%	6.88	13.80	51.84%	2.00	3.22	58.84%
ASTGCN	17.55	26.97	11.78%	3.57	7.13	10.19%	6.54	11.81	52.17%	2.06	3.35	55.18%
w/ GPT-ST	15.89	24.87	10.28%	3.10	6.26	8.74%	5.90	10.38	45.78%	1.90	2.93	53.60%
GWN	15.29	24.59	9.98%	3.15	6.44	8.67%	5.64	9.92	42.19%	1.90	2.98	53.00%
w/ GPT-ST	14.72	24.12	9.42%	3.02	6.18	8.29%	5.12	8.89	37.49%	1.80	2.70	51.69%
STMGCN	-	-	-	-	-	-	6.53	12.20	49.38%	1.96	3.12	55.99%
w/ GPT-ST	-	-	-	-	-	-	5.18	9.00	38.38%	1.81	2.73	52.95%
MTGNN	15.28	24.70	9.72%	3.12	6.44	8.60%	5.15	9.19	36.02%	1.84	2.82	51.45%
w/ GPT-ST	14.93	24.46	9.67%	2.99	6.19	8.21%	4.90	8.55	34.97%	1.78	2.67	51.16%
STSGCN	17.82	27.88	11.49%	3.58	7.36	9.95%	6.02	10.98	46.87%	2.04	3.19	56.73%
w/ GPT-ST	16.29	26.14	10.59%	3.16	6.51	8.74%	5.00	8.77	36.28%	1.88	2.79	55.23%
STFGNN	17.13	27.32	11.10%	3.29	6.69	9.14%	5.91	10.64	46.32%	2.03	3.17	56.51%
w/ GPT-ST	15.89	25.78	10.53%	3.13	6.47	8.84%	4.97	8.69	36.37%	1.86	2.74	54.84%
STGODE	17.73	27.53	11.39%	3.47	7.02	9.93%	6.62	12.14	55.03%	2.08	3.27	56.47%
w/ GPT-ST	15.68	24.84	10.46%	3.19	6.44	9.30%	5.61	9.91	43.09%	1.90	2.89	52.86%
CCRNN	-	-	-	-	-	-	5.41	10.51	38.47%	1.92	3.01	52.44%
w/ GPT-ST	-	-	-	-	-	-	5.08	9.47	37.04%	1.82	2.77	51.33%
MSDR	16.47	25.63	10.54%	3.19	6.42	8.83%	5.73	10.45	42.33%	1.93	2.98	53.44%
w/ GPT-ST	15.92	25.03	10.33%	3.07	6.27	8.50%	5.27	9.33	38.90%	1.82	2.74	52.78%
STWA	16.16	25.80	10.55%	3.19	6.56	8.68%	5.25	9.34	38.08%	1.89	2.96	50.07%
w/ GPT-ST	15.29	24.50	9.81%	3.09	6.32	8.59%	5.06	8.82	36.63%	1.83	2.80	49.91%
GWN*	13.97	22.94	9.28%	2.95	5.99	8.01%	5.18	9.08	35.75%	2.02	3.12	55.16%
w/ STEP	14.08	23.33	9.07%	2.92	5.91	7.98%	5.33	9.24	36.17%	2.02	3.12	53.86%
w/ GPT-ST	13.96	22.83	9.16%	2.92	5.97	7.93%	5.07	8.82	34.97%	1.98	3.07	51.44%
5.3 Model Ablation Study (RQ2)
This section investigates the impact of major components in our GPT-ST. To evaluate the impact of different components, we re-conduct pre-training for multiple ablated variants, and evaluate the performance of the downstream method using the newly pre-trained ablated models. GWN is utilized as the downstream approach, and two datasets METR-LA and NYC Taxi are employed for evaluation. The results are shown in Figure 5. From the results, we make the following observations:
Impact of basic components. i) -P. We remove the customized parameter learner in GPT-ST. ii) -C. We disable the hypergraph capsule clustering network to cancel the clustering operation. In this case, the spatial hypergraph NNs works on fine-grained regions directly. iii) -T. We leave the cross-cluster relation learning unexplored by removing the high-level spatial hypergraph. The noticeable performance decay demonstrates the positive benefits brought by all three components, indicating that generating personalized ST parameters and modeling the intra- and inter-cluster ST dependencies can effectively capture complex ST correlations to benefit the predictions. Among the three components, the removal of the hypergraph capsule clustering network causes the most significant performance drop. This is because the clustering results also play an important role in many other components, including the cross-cluster dependencies and the cluster-aware adaptive masking.
Impact of mask mechanism. i) Ran0.25 and Ran0.75. We replace our adaptive mask strategy with random masking with a mask ratio of 0.25 (mask ratio same as ours) and 0.75 (mask ratio used in MAE [15] and STEP [31]), respectively. The results consistently demonstrate that our proposed mask strategy outperforms the random mask strategies. This can be attributed to the fact that our mask strategy effectively promotes the learning of intra- and inter-cluster relationships by GPT-ST, resulting in the generation of high-quality representations. Similar results are obtained in additional experiments conducted on the other two datasets, as shown in Table 2. ii) GMAE and AdaMAE. We compare our approach to two variants using the mask strategies proposed by GraphMAE [16] and AdaMAE [2]. Both variants perform worse than our methods, highlighting the importance of considering spatial and temporal patterns in the masking strategy. This further confirms the superiority of our adaptive masking approach that leverages clustering information.
Impact of pre-training strategy. To further investigate the effectiveness of the mask-reconstruction pre-training approach in spatio-temporal pre-training, we compare it to other pre-training methods: local-global infomax and contrastive pre-training. We used DGI [35] and GraphCL [47] as baselines, which are widely recognized pre-training strategies for GNNs. By replacing GPT-ST's pre-training strategy with these methods (denoted as "r/"), we observe improvements in performance compared to
models without pre-training. This indicates the adaptability and benefits of infomax and contrastive pre-training for representation learning in our model. However, our approach, which utilizes the mask-reconstruction task, achieved the most significant performance enhancement. This can be attributed to the higher correlation between the mask-reconstruction task and the downstream regression task, leading to more effective learning of spatio-temporal representations. Additionally, our adaptive mask strategy plays a crucial role in facilitating the model to learn robust spatio-temporal representations by increasing the difficulty of the pre-training task.
-P C -T Ran0.25 Ran0.75 GMAE AdaMAE r/DGI r/GCL Ours Table 2: Comparison of Ran & Ada mask.
Figure 5: Ablation study of GPT-ST.
Dataset	PeMS08(MAE/RMSE)
model	GWN	MSDR	STWA
Ran mask	14.80/24.34	16.50/25.63	15.39/24.85
Ada mask	14.72/24.12	15.92/25.03	15.29/24.50
Dataset	NYC Citi Bike(MAE/RMSE)
model	DMVSTNET	STMGCN	CCRRN
Ran mask	1.92/2.89	1.82/2.74	1.85/2.86
Ada mask	1.89/2.86	1.81/2.73	1.82/2.77
5.4 Investigation on Clustering Effect (RQ3)
To comprehensively demonstrate the effectiveness of the clustering process, we assess the interpretability of our GPT-ST framework by analyzing the embeddings generated by the Hypergraph Capsule Clustering Network (HCCN), as illustrated in Figure 6. We employ the T-SNE algorithm to visualize the high-dimensional embeddings produced by HCCN, mapping them to 2-dimensional vectors. Each category is represented by a distinct color, with 10 categories defined for hyperparameter  $H_{S}$ . The region clustering is determined based on the probabilities of belonging to different categories. Upon examining the visualized embeddings, we observe that regions belonging to the same class exhibit tight clustering within a confined space, providing empirical evidence of the robust clustering capability of our hypergraph capsule clustering network.
In another case study conducted on the METR-LA dataset, we investigate the in-cluster region relations derived from the hypergraph capsule clustering network, as well as the cross-class dependencies obtained from the cross-cluster hypergraph network. The results, illustrated in Figure 7, demonstrate that the top-3 regions within the same category, as depicted in Figure 7(a) and 7(b), exhibit similar traffic patterns, indicating shared functionalities. For instance, regions near commercial areas (7(a)) experience an evening peak, while those near residential areas (7(b)) remain relatively stable. These observations align with real-world scenarios. Furthermore, focusing on Figure 7(c), we analyze the top-2 regions from two categories that have undergone shifts in traffic patterns over a specific time period, while sharing similar hyperedge weights in the cross-class transition. The results reveal that regions undergoing pattern shifts exhibit distinct traffic patterns while maintaining close interconnections within a short driving distance. These findings provide further evidence of the cross-cluster transition learning's ability to capture semantic-level relationships between regions, reflecting real-world traffic scenarios. These advantages contribute to the generation of high-quality representations in our GPT-ST framework, leading to improved performance in downstream tasks.
5.5 Model Efficiency Study (RQ4)
In this section, we assess the efficiency of our model. Specifically, we measure the per-epoch training time of all methods on the PEMS08 dataset, and the results are summarized in Table 3. To ensure fairness, all experiments are conducted on a system equipped with a GTX 3090 GPU and an Intel Core i9-12900K CPU. The batch size for all methods, except those tagged with 'b8', is set to 64 (where 'b8' denotes a batch size of 8). By combining the results from Table 1, we can conclude that while most of the baselines experience a minor decrease in training efficiency when enhanced by GPT-ST, there is a significant improvement in their performance. For
Table 3: Computational time cost investigation.
Model	time/epoch(s)	Model	time/epoch(s)
STGCN	7.72	GWN	9.87
w/ GPT-ST	12.52	w/ GPT-ST	13.46
ASTGCN	12.47	TGCN	7.63
w/ GPT-ST	18.20	w/ GPT-ST	14.04
MTGNN	9.40	STSGCN	18.71
w/ GPT-ST	13.02	w/ GPT-ST	21.43
STFGNN	16.03	STGODE	22.93
w/ GPT-ST	19.54	w/ GPT-ST	19.45
MSDR	29.15	STWA	46.72
w/ GPT-ST	33.23	w/ GPT-ST	49.90
Downstream tasks stage	Pre-training stage
GWN* b8	210.98	STEP b8	327.78
w/ STEP b8	425.12	GPT-ST b8	40.18
w/ GPT-ST b8	220.29	GPT-ST	12.50
instance, some previous methods (e.g., STGCN) equipped with GPT-ST achieve comparable perfor
Figure 6: Visualized embeddings of the hypergraph capsule clustering network.
(a) Case A for learned clusters
(b) Case B for learned clusters
Figure 7: Case study for the spatial encoder. (a) and (b) present the real traffic conditions and their geographic locations belonging to two specific categories learned by the hypergraph capsule network; (c) illustrates the learned traffic transition relation between two categories of regions.
(c) Case for cross-class transitions
mance to advanced methods (e.g., MSDR), while maintaining superior training efficiency. In the comparison between GPT-ST and STEP, we observe that STEP incurs a substantial time cost, both in the pre-training stage and the downstream task stage, whereas GPT-ST enhances existing baselines in a lightweight manner. These cases demonstrate that our framework achieves a win-win situation in terms of performance and training efficiency, making it well-suited for practical applications.
5.6 Model Hyperparameter Experiment (RQ5)
To examine the impact of different hyperparameters on GPT-ST, we conduct parameter experiments on the NYC Taxi dataset using GWN as the downstream model. The results of these experiments are depicted in Figure 8, where the evaluation metrics are plotted as relative values.
In our investigation, we specifically focus on the total mask ratio  $r_t$  and adaptive mask ratio  $r_a$  as the main parameters of interest. Based on the findings presented in Figure 8, we observe that the optimal effect is achieved when the total mask ratio  $r_t$  is set to 0.25. Increasing the mask ratio beyond this value does not improve the model's ability to learn better representations; instead, it leads to a decline in prediction performance. Furthermore, we explore the impact of the adaptive mask ratio  $r_r$ , which determines the number of clusters that are entirely masked, thereby influencing the entire training process. The best performance is obtained when the adaptive mask ratio is set to 1. This suggests that this specific configuration enhances the model's ability to learn regional relations, both within individual clusters and across different clusters.
(a) Total mask ratio  $m_t$
(b) Adaptive mask ratio  $m_{a}$
Figure 8: Hyperparameter study of  $m_t$  and  $m_a$ .
In this work, we introduce a scalable and effective pre-training framework tailored for spatio-temporal prediction tasks. Our framework starts with a foundational pre-training model that focuses on capturing spatio-temporal dependencies. It leverages a customized parameter learner and hierarchical hypergraph networks to extract customized spatio-temporal features and region-wise semantic associations, respectively. To further improve the model's performance, we propose an adaptive masking strategy. This strategy guides the model to learn inferential reasoning capabilities by considering both intra-class and inter-class relations during the pre-training stage. We conduct extensive experiments on four real-world datasets, demonstrating the efficacy of our proposed GPT-ST framework in enhancing the performance of downstream baselines across different spatio-temporal tasks.