Expand and Compress: Exploring Tuning Principles for Continual Spatio-Temporal Graph Forecasting
Authors: Wei Chen
Yuxuan Liang
Venue: ICLR 2025 Poster (2025)
Keywords: Spatio-temporal Graph
Continual Forecasting
Tuning Principle
Primary area: learning on time series and dynamical systems
TL;DR: We introduce EAC, which follows the two fundamental tuning principle and learns prompt parameters  pool only through expand and compress, simply, effectively and efficiently solving the continual spatio-temporal graph forecasting problem.

Abstract: The widespread deployment of sensing devices leads to a surge in data for spatio-temporal forecasting applications such as traffic flow, air quality, and wind energy. Although spatio-temporal graph neural networks (STGNNs) have achieved success in modeling various static spatio-temporal forecasting scenarios, real-world spatio-temporal data are typically received in a streaming manner, and the network continuously expands with the installation of new sensors. Thus, spatio-temporal forecasting in streaming scenarios faces dual challenges: the inefficiency of retraining models over newly-arrived data and the detrimental effects of catastrophic forgetting over long-term history. To address these challenges, we propose a novel prompt tuning-based continuous forecasting method, **_EAC_**, following two fundamental tuning principles guided by empirical and theoretical analysis: _**e**xpand **a**nd **c**ompress_, which effectively resolve the aforementioned problems with lightweight tuning parameters. Specifically, we integrate the base STGNN with a continuous prompt pool, utilizing stored prompts (\ie, few learnable parameters) in memory, and jointly optimize them with the base STGNN. This method ensures that the model sequentially learns from the spatio-temporal data stream to accomplish tasks for corresponding periods. Extensive experimental results on multiple real-world datasets demonstrate the multi-faceted superiority of **_EAC_** over the state-of-the-art baselines, including effectiveness, efficiency, universality, etc.

Published as a conference paper at ICLR 2025
EXPAND AND COMPRESS: EXPLORING TUNING PRIN- CIPLES FOR CONTINUAL SPATIO- -EMPORL GRAPH FORECASTING
Wei Chen, Yuxuan Liang
The Hong Kong University of Science and Technology (Guangzhou) onedeanxxxx@gaiil com, yuxliange out look com
ABSTRACT
The widespread deployment of sensing devices leads to a surge in data for spatio- temporal forecasting applications such as traffic flow, air quality, and wind energy. Although spatio-temporal graph neural networks (STGNNs) have achieved suc- cess modeling various static spatio-temporal forecasting scenarios, real-world spatio-temporal data are typically received in streaming manner, and the net- work continuously expands with the installation of new sensors. Thus, spatio- temporal forecasting in streaming scenarios faces dual challenges: the inefficiency of retraining models over newly-arrived data and the detrimental effects of catas- trophic forgetring over long-term history. To address these challenges, pose novel prompt tuning-based continuous forecasting method, EAC, follow- ing two fundamental tuning principles guided by empirical and theoretical analy- sis: expand and compress, which effectively resolve the aforementioned problems with lightweight tuning parameters. Specifically, we integrate the base STGNN with a continuous prompt pool, utilizing stored prompts (i.e., few learnable param- eters) in memory, and jointly optimize them with the base STGNN. This method ensures that the model sequentially learns from the apatio-temporal data stream to accomplish tasks for corresponding periods. Extensive experimental results on multiple real-world datasets demonstrate the multi- faceted superiority of EAC over the state-of ofie--of-hhe thet-art baselines, including effectiveness, efficiency, universality, etc. Our code repository is available at hetps://gitthe www/Onedean//nnnecceeccmee EAC
pro-
INTRODUCTION
Spatio-temporal data is ubiquitous in various applications, such as traffic management (Avila & Mezic 2020), air quality monitoring (Liang et al. (023), and wind energy deployment (Yang al.) 2024). Spatio-temporal graph neural networks (STGNNs) (Jin et al., 2023; 2024) have become predominant paradigm for modeling such data, primarily due to their powerful spatio-temports representation learning capabilities, which consider the both spatial and temporal dimensions of data by learning temporal representations of graph structures. However, most existing works (Li al., 2017 Wu et al. 2019 Bai et al., 2020): Cini et al., 2023 Han et al., 2024) assume a static setup, where STGNN models are trained on the entire dataset over a limited time period and maintain fixed parameters after training is completed. In contrast, real-worle spatio-temporal data (Liu et al.) 2024 Yin al., 2024) typically exists in streaming format, with the underlying network structure expanding through the installation of new sensors in surrounding areas, resulting in a constantly network. Due to computational and storage costs, it is often impractical to store all data and retrain the entire STGNN model from scratch for each time period.
evolving
To address this problem, several straightforward solutions are available, as illustrated in Figure I The simplest approach involves pre-training an STGNN (using node-count-free graph convolution operators) for testing across subsequent periods. However, due to distribution shifts (Wang et al.) 2024a), this method often fails to adapt to new period data. Another approach involves model retraining and prediction on different data windows due to graph expansion. Unfortunately. this neglects the informational gains from historical data, leading to limited performance improvements. To simultaneously resolve the challenges posed by these two issues, more effective solution is to adopt continual learning paradigm (Wang al. 2024d), which is research area focused how

---

Published as a conference paper at ICLR 2025
Period #1 Dota
Period 82 Dota
Period MN Doto
Pre-training
R2
#N
Re-training
RN
Online traininng
Ours
HV
Period
Model
Test
Prompt Pool Tuning
Frazen
Re-load
Figure 1: Comparison of classic schemes and EAC for continual spatio-temporal forecasting.
systems learn sequentially from continuous related data streams. Specifically, the core idea is to load the previously trained model for new period data and conduct online training the current period. Nevertheless, the notorious problem of catastrophic forgetting (van de Ven et al. 2024) in neural networks often hinders the improvement of online learning performance.
Current methods for continual spatio-temporal forecasting typically follow various types of contin- ual learning approaches for improvement. For example, TrafficStream (Chen al. 202 com- prehensively integrates regularization and replay-based methods to learn and adapt to ongoing data streams while retaining past knowledge enhance performance. PECPM (Wang
2023b) and STKEC [Wang et al. 2023a) further refine replay strategies to detect stable and changing node data for better adaptation. TFMoE (Lee & Park 2024) method considers training a sets of mix- ture of experts models for adapting to new nodes, thereby improving efficiency. Though promising, the aforementioned methods still involve optimizing the entire STGNN model, resulting in complex tuning costs and failing to mitigate the problem of catastrophic forgetling in a principle way.
To this end, we propose EAC, a novel continual spatio-temporal graph forecasting framework based on continuous prompt parameter pool for modeling streaming ppatio-temporal data. Specifically, we freeze the base STGNN model during the continual learning process to prevent knowledge for- getting, adapting solely through dynamically adjustable prompt parameter pool to accommodate the continuously emerging expanded node data while further storing the knowledge acquired from streaming ppatio-tempont data. Notably, we explore two fundamental tuning principles, expand and compress, through empirical and theoretical analyses to balance model effectiveness and effi- ciency. EAC has five distinctive characteristics: (i) Simplicity: it accomplishes complex continual learning tasks solely by tuning the prompt parameter pool; (ii) Effectiveness: it demonstrates consis- tent performance across multiple real-world datasets; (ii)) Universality it demonstrates consistent performance across different STGNN architectures; (iv) Efficiency: it accelerates model training speed by effectively freezing the backbone model; and (v) Lightweight: it requires adjustment of only limited number of parameters in the prompt pool. In summary, our main contributions are:
We propose prompt- based continual spatio-temporal forecasting framework EAC that is simple, effective, and efficient with lightweight tunable parameters.
Through empirical observations and theoretical analysis, explore two tuning principles for tinual apatio-tempont forecasting: the heterogeneity property for expansion and low-rank property for compression our continuous prompt parameter pool.
Based on the two proposed tuning principles, we introduce two implementation schemes: contin- uous prompt pool growth and continuous prompt pool reduction.
Experimental results different scenarios of world datasets from different domains demon- strate the effectiveness and universal superiority of EAC
2 RELATED WORK
Spatio-temporal Forecasting, Spatio-temporal forecasting originates from time series analysis and can be viewed as a temporal data modeling problem within an underlying network. Traditional statistical models, such as ARIMA (Box & Pierce 1970) and VAR (Biller & Nelson 2003), well
2

---

Published as a conference paper at ICLR 2025
as advanced time series deep learning models (Nie al. 2023 Wu et al. 2022), can simplify this into a single time series forecasting task. Even though, these methods fail to capture correlations between different locations, leading to suboptimal performance. STGNNs, due to their inherent ability to aggregate local spatio-temporal information, are considered powerful tools for modeling this data. STGNNs consist of two key components: a graph operator module for spatial relationship modeling, typically categorized into spectral GNNs (Yu et al. (017)) spatial GNNs (Li al. 2017), or hybrids, and a sequence operator module for temporal relationship modeling, which can be recurrent-based (Pan et al., 2019). cnvvolution-based (Wu et al. 2019), attention -basd (Guo al. 2019), or a combination of these networks. However, most models focus static settings with limited-periali forecasting scenarios.
forecastin
Continual Learning. Continual learning technique for sequentially training models as data from related tasks arrives in streaming manner. Common approaches include regularization- based (Kirkpatrick et al. 2017), replay-based (Rolnick et al., 2019), and prototype-based (De Langel & Tuytelaars] 2021) methods, all aimed at learning knowledge from new tasks while retaining knowledge from previously tasks. However, these methods primarily focus on vision and text do- mains (Wang al. 2024d), assuming that samples are i.i.d. TrafficStream (Chen et al. 2021b) first integrates the ideas of regularization and replay into continual apatio-temport forecasting scenarios. PECPM (Wang al. 2023b) and STKEC (Wang et al.) 2023a) further incorporate prototype-base ideas for enhancement. TFMoE (Lee & Park] (024) advances replay data into a generative recon- struction approach, equipped with mixture of experts model. Additionally, some methods consider diverse perspectives such as few-shot scenarios (Wang et al., 2024b), large-scale contexts (Wang al., 2024c), and combinations with reinforcement learning (Xiao et al. 2022) and data augmen- tation (Miao et al. 2024). Nonetheless, most methods in dynamic scenarios still require tuning all STGNN parameters, resulting in the dual challenges of catastrophic forgetting and inefficiency.
Prompt Learning, Prompt learning suggest simply tuning frozen language or vision models to perform downstream tasks by learning prompt parameters attached to the input to guide model pre-
dictions. Some studies (Yuan et al. 2024 Li et al. 2024) attempt to integrate it into spatio-tempores forecasting, but they still focus on static scenarios. Other methods have applied it to continual learn- ing contexts; however, they only concentrate
vision (Wang al. 2022) and text (Razdaibicdinal ee al., 2023) domains. In contrast to the various prompt-based approaches in the existing literature, a naive application of prompt learning in context is to append learnable parameters P (referred to as prompts) to the original spatio-temporal data X. resulting in a fused embedding X' = [X || P), which is then fed into the base STGNN model fo(X') for spatio-temporal forecasting. Notably, we design a novel prompt pool learning mechanism, guided by two luning principles derived from empirical and theoretical analysis, to model continual spatio-temports forecasting.
PRELIMINARIES
Definition (Dynamic Streaming Spatio-temporal Graph). We consider dynamic streaming spatio-temporal graph G = (G,, "hh,,,,.,,9 GT), ) for every time interval т, the network dynamically grows, i.e., 9, = 9--1 1 + Ag Specifically, the network in the 7-th time interval is modeled by the
graph G2 = (V,,& E,, ), where V, is the set of nodes corresponding to the |V+| = n sensors in the network, and & signifies the edges connecting the node set, which can be further represented by the adjacency matrix A, E IR".xpr, The node features are represented by firee-dimensionl tensor X, E Raxexc denoting the e features of the records of all nodes observed on the graph G, in the past / time steps. Following (Chen et al. 2021b), here is usually only numerical value.
Problem (Continual Spatio-temporal Graph Forecasting). The continual spatio-temporal graph forecasting can be viewed as learning the optimal prediction model for the current stage from dynamic streaming spatio-temporal graph data. Specifically, given the training data D = (D, Y=))?= P from sequence of streaming data, our goal is to incrementally learn the optimal model parameters fg- from the sequential training set. For the current 7-th time interval, the model is optimized to minimize:
før->- argmin argminEp,,pte) ED, (.... ((r)
where fai-). represents the optimal model that achieves the minimum loss when trained on the data from the current period r. The loss function L(.) measures the discrepancy between the predicted signals P, - soc-> X+) E Baxixe for next time steps and the ground-truth Y,.
3

---

Published as a conference paper at ICLR 2025
4 METHODOLOGY
In this section, we propose two tuning principles, expand and compress, through detailed empirical observations and theoretical analysis. Based on these principles, we apply them to a prompt pa- rameter pool to develop the continual spatio-temporal graph forecasting framework EAC (as shown in Figure 2 Specifically, we design a node-level prompt parameter pool corresponding to the put spatio-temporal data of different nodes, jointly optimized within the STGNN backbone. O For the expand process, empirical studies reveal that the prompt parameter pool adapts to dynamic heterogeneity, which further analyze theoretically. Building on this, show that expanding prompt parameters for newly introduced nodes effectively accommodates heterogeneity in contin- uous spatio-temporal scenarios. @ For the compress process, empirical results indicate that the prompt parameter pool exhibits a low-rank property, which we formalize through detailed analy- sis. Based this, show that nigh-dimensional prompt parameters low-dimensional components, mitigating parameter inflation caused by expansion in continuous spatio-temporal scenarios. We also summarize the workflow of EAC in Algorithm 1 detailed explanation of the continual learning process in Appendix B
be compressed into two
and provide
Time
Period #1
Period #2
Period MN
Stable Node
Node
Expand
Expand
Tuning Frozen Prompi
Continual
x g²
X²,Ga
X G
Training
Prompt Expand
Prompt Compress
Prompl Expand
Prunpt Compress
STGNN Backbone
STGNN Backbone
STGNN Backbone
Prompt Parameter Pool
Relond Weights
Reload Wrights
Reload Weights
Tuning Principle
Tuning Principle
Figure 2: The overall architecture of our proposed EAC.
4.1 EXPAND: HETEROOEENETY-GUUDEE CONTINUOUS PROMPT POOL GROWTH
Insight. As mentioned above, fine-tuning existing STGNN model with new data streams often leads to catastrophic forgetting (van de Ven et al. 2024). While previous methods have proposed some mitigative strategies (Chen et al. [222202110 lb; Wang et al. ())..lb). these solutions are not entirely avoidable. A straightforward solution is to isolate parameters, freeze the old model, and dynamically adjust the network structure incorporate adaptable learning parameters. Recently, there has been an increasingly common consensus in spatio-temporal forecasting to introduce node-specific trainable parameters spatial identifiers achieve higher performance (Shao et al. [2022 Liu al., 2023 Dong et al. 2024 Ych et al., 2024). Although some empirical evidence (Shao et al.) 2023 Cini et al. 2024) supports their predictive performance in static scenarios, there has been root analysis to explain why they are useful, when they are applicable, and in what contexts they are most suitable. However, we find this closely aligns with our motivation and extend it to the continual spatio-temporal forecasting setting by providing reasonable explanation from the perspective of heterogeneity to address these questions. Specifically, spatio-temporal data generally exhibit two characteristics: correlation and heterogeneity (Geetha et al.] 2008) Wang et al. (020). The former is naturally captured by various STGNNs, as they automatically aggregate local spatial and temporal information. However, given the message- paassagg-pasingg mechanism of STGNNs. the latter is clearly not captured. Therefore, we argue that the introduction of node prompt parameter pool likely enhances the model's ability capture heterogeneity by expanding the expressiveness of the feature space.
Empirical Observation. To quantitatively analyze heterogeneity, we consider the dispersion of node feature vectors in the feature space (Fan et al. 2024). We first define the Average Node Devia- tion (D(-) metric as: ((X) pxn E Σn E (Xu X,x)², where X Raxd represent the feature matrix composed of n node vectors, each with d dimensions. This metric quantifies the
degree of dispersion between pairs of node vectors within the feature matrix, reflecting the ability to express heterogeneity. We use this indicator to plot the dispersion degree of the feature matrix
4

---

Published as a conference paper at ICLR 2025
PEMS-Stream
for the stmemm dataset across different pe- riods as node prompt parameters are injected during the learning process. As shown in Fig-
Average Node Deviation
Year
40
2011
ure 3 two phenomena are clearly observed: ① Within the same period, the dispersion of the node feature space continuously expands throughout the learning process, reflecting the increasing ability of prompt parameters to repre- Across different periods, the dispersion of the feature space in the current period expands further compared the previous period, showing the continuous capture ability of the prompt parameter for heterogeneity.
2012
2016
30
2017
2014
20
10
sent heterogeneity.
8
10
12
14
16
Steps saving models
Figure 3: Heterogeneity measurement.
Theoretical Analysis. Below, we provide a theoretical analysis of the above empirical results.
Proposition 1. For an original node input feature matrix X [æ,,... [Fp] E Tgaxd we introduce a node prompt parameter matrix [pi.... .Po) E Ruxd Through a spatio-temporal learning function So with invariance, a new feature matrix X f(; X, P) is obtained, satisfying:
D(X") - D(X) - 1P |1 2 0,
(2)
i=1
where p0 = [p..... ... E Raxd represents the optimized prompt parameter matrix, and pl =
PP. E i= 1 is the mean vector of the parameter matrix.
Proof. For more details, refer to the supplementary materials in appendix A.1
Tuning Principle 1: Prompt Parameter Pool Can Continuously Adapt to Heterogeneity Property.
Implementation Details. Based on the above analysis, we present the implementation details for expand process in continuous spatio-temporal forecasting scenarios. Specifically, we continuously maintain a prompt parameter pool P. For the initial static stage, we provide each node with a learnable parameter vector, and the matrix p(!) of all such vectors is added to the parameter pool P = [PO)] Follow the Occam's razor, we adopt a simple yet effective fusion method, where the prompt pool and the corresponding input node features are added element-wise. The prompt parameter pool P is then trained together with the base STGNN model. For subsequent period т, we only provide prompt parameter vectors for newly added nodes, and the resulting matrix p(r) is added the prompt pool P [P(1) p(2) ..... p. As analyzed, we weffeeze the STGNN backbone and only tuming the prompt pool P, effectively reducing computational costs and accelerate training.
4.2 COMPRESS: LOW-RANK-GUIDER CONTINUOUS PROMPT POOL REDUCTION
Insight. While node-customized prompt parameter pools are highly effective, an unavoidable chal- lenge arises in our scenario of continuous spatio-temporal forecasting: the number of prompt pa- rameters continuously increases with the addition of new nodes across consecutive periods, leading to parameter inflation. Despite the existence of numerous well-established studies that enhance the efficiency of spatio-temporal prediction (Bahadori et al. 2014 Yu et al. 2015 Chen et al. 2023 Ruan et al. 2024) and imputation (Chen et al. 2020; 202 Nie et al. 2024) tasks using techniques such compressed sensing and matrix tensor decomposition, these study typically focus solely on the original spatio-temporal data. An intuitive solution is to similarly apply low-rank matrix approximations the prompt learning parameter pool, thereby reducing the number of learnable parameters while maintaiining performance. However, for the prompt learning parameter pool, it remains to be validated whether it exhibits redundancy characteristics akin to spatio-temporal data and whether these properties hold in the continuous spatio-temporal forecasting setting.
Empirical Observation. To explore redundancy, we conduct a spectral analysis of the prompt parameter pool P.. Specifically, for the models optimized annually on the PEMS- Stream dataset, we first apply singular value decomposition to the extended prompt parameter pool introduced in the
5

---

Published as a conference paper at ICLR 2025
PEMS-Siream
previous section and plot the normalized cu- mulative singular values for different years, as shown in Figure left. It can be observed that all years exhibit a clear long-tail spectral dis- tribution, indicating that most information from the parameter matrix P can be recovered from the first few largest singular values. In Fig- frihtt we also present a heatmap of the nor- malized cumulative singular values the sixth largest singular value for different years at dif- ferent steps, revealing that, despite some vari- ations across years, the overall processes for all years maintain a high concentration informa- tion (> 0.75), suggesting low-rank property for P.
Normalized Cumulaive Singnilar Value
2.01
2017 2016 2015 2015 2012 2011
0.66
0.5
0.84
Yoar
0.5
2012
Years
0.82
0.50
0.4
2014
2016
0.2
2017
0.76
Index the largest cigenvalue
Steps of saving models
Figure 4: Low-rank measurement.
Theoretical Analysis. Below, we provide a theoretical analysis the above empirical results.
Proposition Given the node prompt parameter matrix E Baxd there will always be two matrices A E Bg** and B E Rkxd such that P can be approximated AB when the nodes grow large, and satisfy the following probability inequality:
Pr P - AB|l S <l]P)] 2 o(1) and h: (log(min(n, d)))
where o(1) represents a term that becomes negligible even as n grows large.
Proof. For more details, refer to the supplementary materials in appendix A.2
Tuning Principle II: Prompt Parameter Pool Can Continuously Satisfy the Low-rank Property.
Implementation Details. Formally, we present the implementation details for compress process in continuous spatio-temporal forecasting scenarios based on the aforementioned analysis. Specifi- cally, for the initial static stage, we approximate the original prompt parameter p(1) using the prod- uct of the subspace parameter matrix (1) and the adjustment parameter matrix B. For subsequent periods т, we provide only the subspace parameter matrix A for the newly added node vectors, approximating the prompt parameter p(r) through the product with the adjustment parameter matrix B. As analyzed, the dimensionality of the subspace parameter matrix A is significantly smaller than that of the prompt parameter P. while the number of parameters the adjustment matrix remains constant; thus, we effectively mitigate the inflation issue.
EXPERIMENTS
In this section, we conduct extensive experiments to investigate the following research questions:
RQ1: Can EAC outperform previous methods in accuracy across various tasks? (Effectiveness)
RQ2: Can EAC have consistent improvement on various types of STGNNs? (Universality)
RQ3: How efficient is EAC compared different methods during the training phase? (Efficieeccy)
RQ4: How many parameters does EAC require tuning compared to baselines? (Lightweight)
RQS: How does EAC compare to other common prompt-adaptive learning method? (Simplicity)
5.1
EXPERIMENTAL SETUP
We use real-world spatio-temport graph datasets from three do- transportation, weather, and energy, encompassing common streaming apatio-tempora fore- casting scenarios. The transportation dataset, SEream is derived from benchmark datasets in previous research (Chen et al. 2021b), covering dynamic traffic flow in Northern California from 2011 1 to 2017 across seven periods. The weather dataset, Air-Stream, originates from the real-time urban air quality platform of the Chinese Environmental Monitoring Center capturing dynamic air quality indicators at monitoring stations across various regions of China from 2016 to 2019 over
Dataset and Evaluation Protocol.
mains:
hups://air.cmeme.com 1n:.:me........1.1:.: 1807//

---

Published as a conference paper at ICLR 2025
four periods. The energy dataset, Energy-Stream, comes from spatial dynamic wind power fore- casting dataset provided by a power company during the KDD Cup competition (Zhou et al.) 2022). containing monitoring metries for wind farms over a span of 245 days across four periods. During the experiments, each dataset's 's temporal dimension is split into training, validation, and test sets in 6:2:2 ratio for each period, employing an early stopping mechanism. According the established protocol (Chen et al. 2021b), we use the past 12 steps to predict the next 3, 6. 12 steps, and the mean value. Evaluation metries include Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE) averaged over all periods. Further details regarding the datasets and evaluation metrics are available in Appendix C.I
Baseline and Parameter Setting. Following the default settings (Chen et al. 2021b Wang et al.,
2023a Lee & Park, 2024), we employ the same STGNN as the backbone network and consider the
following baseline methods for comparison:
Pretrain-ST: For each dataset, we train the STGNN backbone using only the spatio-tempore graph data from the first period and directly use this network to predict results on the test sets in subsequent periods.
Retrain-ST:: For each dataset, we train a new backbone network for the spatio-temporal graph data of each period and use the corresponding network predict results on the test set of the current.
Online-ST: For each dataset, we iteratively train a backbone network in an online manner, where the model weights from the previous period serve as initialization for the current period.
* Online-ST-ANN Train on the complete node data of the current period's spatio-temporal graph, tuning the entire model with the model trained from the previous year initialization.
* Online-ST-NN: Train on the newly added node data of the current period's spatio-tempont graph. tuning the entire model with the model trained from the previous year as initialization. TFMoE (Lee& & Park 2024) improved this using a mixture of expert models technique.
$ Online-ST-Nae Train on the mixed node data (new nodes some old nodes) of the current period's ' spatio-temporal graph, tuning the entire model with the model trained from the pre- vious year as initialization. Existing methods typically foeus this aspect, including Traffic- Stream (Chen et al. 20215). PECPM (Wang et al. 20235). and STKEC (Wang et al. 2023a).
For both the baseline method and our method, ommendations in previous paper (Chen et al.) 202 to ensure fair comparison. Our only hyper- parameter & set to 6. We repeated each experiment five times and report the mean and standard de- viation (indicated by gray +) of all methods. More details about baseline settings, see Appendix C.2
parameters uniformly according to the rec-
5.2 EFFECTIVENESS STUDY (RQ1)
Overall Performance. We report a comparison between EAC and typical schemes (including rep- resentative improved methods P in Table I where the best results are highlighted in bold pink and the second-best results in underlined blue. A indicates the reduction of MAE compared to the second- best result, or the increase of other results relative to the second best result. Moreover, due to the unavailability of official source code for PECMP and TFMoE, along with the more complex backbone network used by TFMoE, the comparisons may be unfair. Nonetheless, we also include comparable reported values, as shown in Table 2 Based on the results, we observe the following:
Pretrain-ST methods generally yield the poorest results, especially on smaller datasets (i.e,, Engery-Stram), aligning with the intuition that they directly use a pre-trained model for zero-shot forecasting in subsequent periods. Even with better pre-training on larger dataset (i.e., Air-Stream), performance remains mediocre. rely on limited data to train specific phase models without effectively utilizing historical informa- tion gained from the pretrained model. @ Online-ST-NN methods perform poorly, as they fine- tune the pretrained model using only new node data differing from the old pattern. Despite TF- MoE's improvements through complex design, severe catastrophic forgetting remains an issue. t Online-ST-MN methods strike a balance between performance and efficiency, showing some im- provements, particularly on small datasets (e.g., STKEC in Energy-Stream), due limited node pat- tern memory. @ Online-ST-AN methods typically achieve suboptimal results, as they fine-tune the pretrained model on the full data across different periods, approximating the performance boundary
Retrain-ST methods also exhibit unsatisfactory results, they
Notably, the core code of STKEC is not available, so we carefully reproduce and report average results.
7

---

Published as a conference paper at ICLR 2025
Table I: Comparison of the overall performance of the classical scheme and EAC.
Datasets
Air Stream (1087 1262)
PEMS Stream (655
871)
Energy Strray-Strrom (203
134)
Method
Metric
Avg.
12
Ang.
Aug.
MAG
18.50 38
23
24 25
21.33
12.96
14.06:000
16.36
14.24
5,50anu
5.40
5.45a00m
5.48
RatrainSST
RMSE
20:20
39.61
42
20.88-
22.96-0
26.
19
23.20-0
5.78
5. 6.70
5.80
1.72
MAPE (8)
23.72m 27.67a
42.50m
27.54
18.51s2
19,98:042
23.31.524
20.30:0444
$4.35a211
54.61200
55.00.1.5
54.74:2.06
-.588
-1.03%
0.525
+0.99%
1.255%
1.00%
1.17%
11.33%
4.70%
2.24%
0.733%
2.23%
MAE
29.08a000 22.72a03
25.00
44.4
14
15.17.006
17.00u520
10.05
10.06.002
17.10a5-22
17.05
Pretrin-ST
RMSE
$1.46
36.
41.00 41
11 36, 15
21.77
23.79
27.78
27.78
24.04.
10.88
10.92.011
11.02.01
10.9
MAPE
24.06a112
33.4841.13
24.00u3.4
21.201.3.11
888.88
172.77.425
174.07.488
172.71:411
6.99%
6.61%
+ 5.26%
6.25%
10.39%
+ 8.90%
7.22%%
8.87%
100.5668
99.62288
216.00%
218.00%
12
12.80
188.92000
16.17
6.47a00s
6.46a
6.47a
6.47.000
Onlise-ST-AN
RMSF
26 24.54
33
53 40
25-0
20.00-000
no 22.7 23-73
20.04-015
22.00-000
6.62
5. 6.66-nos
6.70-mmm
6.67-005
MAPE
27.4
19.37.0m
19.78.077
52. 70.70.1.54
58 58.2641.51
64.30-171
- -..01%
-..44%
+ +1.00%
2.00%
MAB
19.38a1.77
24 25.30a155
22.05s1::
14.08s2.0
16.57
16.95.1.0
5.51a005
5.50an0s
5.40
5.50
Ouline-ST-AV
RMSE
29.57
$4 17
39.62a1.0
53 97 71
24.50.2.00
26,2
36,
20.05-3
5.65en04
5. 5.68
6,70s0
5.70=004
MAPE à
28.02:2.9
27.96s2
58
20.09:340
24.89usv
21.15
54.981157
55. 10
55.17
-5.90%
-4.36%
+ +3.23%
+4440%
14.68%
19.00%
+ 277644
-20.38%
-3.76%
2.99%
1.47%
2.61%
MAE
18.00.1:11
21.30aau9
24.90um::
21.33
12.89a2.0
14.036001
16.39u00s
14.22.0.:
5.58
5.57
5.88
6.67.00s
TrafficStream
RMSE
20.0641:1
34
30.54.1.0
00 33.05-
20.78+013
22,90.00
26.984055
23.16-0.0
6.734005
G. 6.76
6.86.007
6.77-00
MAPE
24.2341.30 28.1261.30
32.9061333
28.0031133
17.806a041
19.50.000
19.95a31
58.87a199
54.06a124
64.M2
54.16.121
-1966
-1311
-0.00%
1..27%
0.700%
0.799%
-1.56%
+ +0.995
- -.008%%
4,300%
4 3.144
3.91%
MAE
19.42.1.27
22:2441:17
25.44+1.05
22.06+1
12.85
13.98.00
16.25/004
44.14
6.31
6.844125
G.41atts
STNEC
RMSE
30.2841.5
35.00+1
40.1144
34.61+1
20.73-3.0
22.81-00
26.
26.79-0m0
23.04-am
8,50m0
6. 5:
6.724000
6.50-017
MAPE (8)
25.20
28
23.83
53.30a1.64
28.71x223
17.8758.1
10.255417
22.00
10.53
50, 10.10
50 10
52.40+1100
-6.1
-4.36%
+2.99%%
+.4.5%%
+.0399%
0.44%
+-0.49%
0.445%
MAE RMSE
18:11:02
20.87.017
24.15
20.75un3
12,65urea
13.45u0.s
14.92a1.11
13.83uice|
5.08
5.09a0.00
5.15c9.10
5.10:a10
27.78:047
32.88:042
38.22:0:31
32.35
20.2460
21,86:0
24. 17
21.77
5.26:9.00
5.3
5.46:8
5.33e3.0
MAPE
2312:420
26.91.007
31.79
26.89
17,80uxes
18, 18.79
20 82
18.98.0.0
47.58:2.71
48, 20
50.55:2.0
48.56G2
1.0388
2.00%
-2.26%
1.75%
-1.77%
3.53%
7.73%%
3.90%
-4.33%
4..8%
-4.00%
4.85%
of continual-based methods while still suf-
Table 2: Performance comparison of the improved method with EAC on PEMS-Stream benchmark.
fering some knowledge forgetting. @ Our
EAC consistently improves all metrics across
all types of datasets. We attribute this to its
Model
Avg. @ MAE
Avg. RMSE
Simplicity
Lightweight
TrafficStram STKEC
14.22-0:3
16
ability to continuously adapt the complex information and knowledge forgetting chal- lenges inherent in continual spatio-temporal learning scenarios by tuning the prompt pool with weterooeenetty-cattrii parameters.
14.14.054
23.04umm
PECMP
14,85
26.62
x
TFMoB
14,18
23.54
x
EAC
13,53a10
21.77au
Few-Shot Performance. While the current experiments encompass spatio-temporal datasets of various scales, we aim to examine more general and complex few-shot scenarios. Specifically, for continuous periodic spatio-temporal data, only a limited amount of training data is typically available for each period. We train or fine-tune the model using only 20% of the data for each period. As shown in Figure 5) we present the 12 step and average RMSE performance metrics across different years of the -MMreen dataset. Our observations are as follows:
Avg. RMSE
12 RMSE
32
32
Retrain-ST
TraffieStream
30-
Pretrain-ST
STKEE
30
Online-STAN
EAC
RMSE Value
28
Online-ST-NN
28
RMSE
26
26
24
24
Retrain-ST
TraffiSttream
Pretrain-ST
STAFC
22-
22
Online-ST.A
EAC
Online ST-NN
20
20
2011
2012
2013
2014 Year
2015
2016
2017
2011
2012
2013
2014 2015
2016
2017
Figure 5: Few-Shot Scenario Forecasting in PEMS-Stream benchmark.
(Robustness) All methods exhibit a decline in performance compared to the complete data sce- nario, with Online-ST-NN demonstrating significant sensitivity due to its inherent catastrophic for- getting issue However, our EAC method exhibits controllable robustness across all years, outper- forming all other methods. @ (Adaptabilitt) The performance of all methods consistently declines as the periods extend, yet our EAC method demonstrates a relatively mild decline, particularly evident in the 12-step metrics, where significantly outperforms all other approaches.
8

---

Published as a conference paper at ICLR 2025
5.3 UNIVERSALITY STUDY (RQ2)
Setting. We further aim to demonstrate the universality of our EAC in enhancing the performance of various STGNN backbones, an aspect largely overlooked in previous studies. Specifically, STGNN can be categorized into spectral-based and spatial-based graph convolution operators, as well as recurrent-basee convolution-basee and attention- basedd sequence modeling operators. We select representative operator from each category to form six distinct models, where the core architecture consists of two interleaved graph convolution modules and one sequence module. A detailed de- scription of the different operators can be found in Appendix E Additionally, adapt different models to the prompt parameter pool proposed by EAC to compare the performance impact.
Table 3: Effect of EAC the average performance of different STGNN component the PEMS- Stream. C-based: Convolution-base R-based: Recurrent-basee A-based: Attention- based..
Spatial-ussed
Spectral-bused
Methods
C-hased
R-based
A-hased
C-based
R-based
A-based
Metric
MAB
RMSE
MAE
RMSE
MAB
RMSE
MAB
RMSE
MAE
RMSE
MAE
RMSE
w/o
14.07-006
22 22.03-0
13.23-9
21.36
14.09-342
23.335 59
14.01-000
13.73
21 21.87s1 34
14.64
23.12
RAC
13.72-0 0 22.14.0 m
15
14.64-355
22.700055
13.62-4 1s
21.80
13.00+0
2008000
13
21.05-a
A
2.48%
3.44%
33.0%
33.00%
0.34%
2.31%
2.78%
-4.21%
-4.66%
4.89%
6.48%
6.35%
Result Analysis. Due to page limitations, we present MAE and RMSE metrics at average time steps using the PEMS- Stre-S dataset. As shown in Table 3
observe that:
Our EAC consistently demonstrates performance improvements across different combinations, highlighting its universality for various architectures. @ Compared to spatial domain-based graph convolution operators, our EAC shows a more pronounced enhancement for spectral domain- based methods. 2 The advantages of recurrent-buse dent, achieving the best results, while attention- based methods perform the worst. This aligns with intuition, directly introducing vanilla multi-head attention may lead excessive parameters and over-fitting; however, our approach still provides certain gains in these cases.
sequence modeling operators particularly evi-
5.4 EFFICIENCY & LIGHTWEIGHT STUDY (RQ3 & RQ4)
Overall Analysis. We first conduct a comprehensive comparison of the EAC with other baselines terms of performance, training speed, and memory usage. All models are configured with the same batch size ensure faimess. As illustrated in Figure 6 we visualize the performance, average tuning parameters, and average training time (per period) of different methods on both the smallest follows:
dataset (Energy-Stream) and the largest dataset (Air-Stream). Our observations
Energy-Stream
Air-Stream
5.8
Rewain-ST
3
3383,72
Strune
1907, 111
SIKEE
23.0
STKEC
54t
Odline STNN
143
3 17A
ST
22.0
5.4
BAD 100
Onliie--S STAN
2383 9%
3385 1155
21.0
Tra
5.0
20.5
EmlineST--NN
4.3
BAC T.IEK
1383, 1846
Menncr Foeqriar
4.6
40
150
120
140
19.5
160
150
250
250
Average Twining Time period)
Average Training Time period)
Figure 6: Efficiency & Lightwight & Performance Study.
1 On datasets with smaller number of nodes, our EAC consistently outperforms the others, achiev- ing superior performance with only half the number of tuning parameters. Furthermore, the average training time per period accelerates by factor of 1.26 to 3.02. In contrast, other methods such as TrafficStream and STKEC, despite employing second-order subgraph sampling techniques, do not benefit from efficiency improvements due to the limited number of nodes, which typically ne- cessitates covering the entire graph. @ On datasets with a larger number of nodes, although the EAC exhibits slightly higher number of tuning parameters, the freezing of the backbone model

---

Published as a conference paper at ICLR 2025
results in faster training speeds while still achieving superior performance. According the tun- ing principle of compression, we set A = 2 to replace 6, resulting in the EAC -Efficient version, which maintains relative performance superiority on larger datasets using only ~ 63% parameters compared to others. This demonstrates the superiority of the compression principle we propose.
Hyper-parameter Analysis. We further examine our sole hyper-parameter k, proposed through the compress principle, and its mutual influence on performance and tuning parameters. As shown in Figure 7 the horizontal axis denotes the values of k, while the vertical axes depict the performance on the PEMS-Stream. The color of the bars indicates the averaged percentage of tuning parameters relative to the total number of parameters across all periods. Our observations are as follows:
10
12
2
10
12
Avg. Turing Parm. Percentage (%K
1000
13.3
21.8
59% Parm
80
MAE
13.6
50% Parm.
74% Parm
RMSE
22.1
60
13.9
Patn
40
14.2
(Second Best)
(Second Best)
20
14.5
Figure 7: ppper-parameter study in PEMS- -Steam benchmark.
From right left, as the value of I decreases, indicating reduction in tuning parameters, the overall performance of the model deteriorates significantly, accompanied by increased volatility (i.e., higher standard deviation). This aligns with our findings in Figure the effective representational information of the approximate prompt parameter pool is constrained with decreasing k, leading to default. Although performance continues
marked decline in performance. @ We set & to improve with increasing k. the gains are minimal. Moreover, excessively high values of k may result in negative effects due to redundant parameters, leading to over-fitting. Consequently, our method achieves satisfactory performance using only approximately 59% of the tuning parameters, effectively balancing performance and parameter efficiency.
5.5 SIMPLICITY STUDY (RQ5)
Simplicity Analysis. Lastly, we aim explore the simplicity of the node parameter prompt pool, well the effectiveness of the expan- sion and compression principle. We selected common low-rank adaptation (LoRA) (Hu al. 2021 Ruan et al., 2024) technique, which has recently been widely used large language models. Following the default architecture, we added low-rank adaptation layers to the sequence operators, setting the rank to 6, and fine-tuned the back- bone model during each period. As shown in Table we observe that simply applying LoRA layers without considering the specific spatio-temporal context of streaming parameters may not be highly effective. Moreover, our method enjoy shorter training times compared to LoRA-basel approaches, further validating the superiority of our proposed expansion and compression tuning principle.
Table 4: Performance comparison of LoRA-Based method with EAC on PEMS-Stream benchmark.
Model
[12 @ MAE 12 @ RMSE 12 @ MAPE| Avg. Time
LoRe-Based
16.22-013
26.81s023
337.31-230 12.78-108
14.92-0.1
24.17s0.1
20.82.0:16
224.33-2035
6 CONCLUSION
In this paper, we derive two fundamental tuning principle: expand and compress for continual spatio- temporal forecasting scenarios through empirical observation and theoretical analysis. Adhering to these principle, we propose novel prompt-based continual forecasting method, EAC, which ef- fectively adapts to the complexities of dynamic continual spatio-temporal forecasting problems. Experimental results across various datasets from different domains in the real world demonstrate that EAC possesses desirable characteristics such as simplicity, effectiveness, efficiency, and univer- sity. In the future, we will further explore large-scale scale pre- traiirann methods for apatio-tempora data, leveraging the tuning principle proposed in this paper, which could broadly benefit a wide range of downstream continual forecasting tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like to thank the anonymous reviewers for their valuable comments. This work is mainly supported by the National Natural Science Foundation of China (No. 62402414), This work Joint Funding Program (No. 2024A03J0620),
also supported by the Muungghou-HHKUSTI Guangzhou Municipal Science and Technology Project (No. 2023A03J001 1), the Guangzhou In- dustrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628), and a grant from State Key Laboratory of Resources and Environmental Information System, and Guangdong Provin- cial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things (No. 2023B 12 12010007.
REFERENCES
Allan M Avila and Igor Mezić. Data-driven analysis and forecasting of highway traffic dynamics. Nature communications, 11(1):2090, 2020.
Mohammad Taha Bahadori, Qi Rose Yu, and Yan Liu. Fast multivariate spatio-temporal analysis via low rank tensor learning. Advances in neural information processing systems, 27, 2014.
Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. Advances in neural information processing systems, 33:17804- 17815, 2020.
Bahar Biller and Barry 1. Nelson. Modeling and generating multivariate time-series input pro- cesses using vector autoregressive technique. ACM Transactions on Modeling and Computer Simulation (TOMACS), 13(3):211-2377 2003.
George EP Box and David Pierce. Distribution of residual autocorrelations autoregressive- integrated moving average time series models. Journal of the American statistical Association, 65 1539):1509-1526, 1970.
Xinyu Chen, Jinming Yang, and Lijun Sun. A nonconvex low-rank tensor completion model for spatiotemporal traffic data imputation. Transportation Research Part Emerging Technologies, 117:102673, 2020.
Xinyu Chen, Mengying Lei, Nicolas Saunier, and Lijun Sun. Low-rank autoregressive tensor com- pletion for spatiotemporal traffic data imputation. IEEE Transactions on Intelligent Transportation Systems, 23(8): 2021a.
Xinyu Chen, Chengyuan Zhang, Xiaoxu Chen, Nicolas Saunier, and Lijun Sun. Discovering dynamic patterns from spatiotemporal data with time-varying low-rank autoregression. IEFF Transactions on Knowledge and Data Engincering, 2023.
Xu Chen, Junshan Wang, and Kunging Xie. Trafficstream: A streaming traffic flow forecasting framework based on graph neural networks and continual learning. In UCAL pp. 3620-3626. Montreal, Canada, 202
Andrea Cini, Ivan Marisca, Filippo Maria Bianchi, and Cesare Alippi. Scalable spatiotemporal graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp- 7218-7226, 2023.
Andrea Cini, Ivan Marisca, Daniele Zambon, and Cesare Alippi. Taming local effects in graph-based spatiotemporal forecasting. Advances in Neural Information Processing Systems, 36, 2024.
Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non- stationary data streams. In Proceedings of the IEEE/CVF international conference on computer vision. pp. 9250-8259 2021.
Zheng Dong, Renhe Jiang, Haotian Gao, Hangchen Liu, Jinliang Deng, Qingsong Wen, and Xuan Song. Heterogeneity-infon informeed meta- veta-parameter learning for spatiotemporal time series forecast- ing. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 631 1311,41, 2024.

---

Published as a conference paper at ICLR 2025
Chao Fan, Yang Yang, and Ali Mostafavi. Neural embeddings of urban big data reveal spatial structures in cities. Nature, Humanities and Social Sciences Communications, 11(1):1-15, 2024.
R Geetha, N Sumathi, S Sathiyabama, and Tiruchengode Tiruchengode Tiruchengode. A survey of spatial, temporal and spatio-temporal data mining. journal of computer applications, 1(4):31-33, 2008.
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 922-929, 2019.
Jindong Han, Weijia Zhang, Hao Liu, Tao Tao, Naiqiang Tan, and Hui Xiong. Bigst: Linear com- graph neural network for traffic forecasting on large-scale road networks. Proceedings of the VLDB Endowment, 17(5):1081-100 2024.
plexity
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
Guangyin Jin, Yuxuan Liang, Yuchen Fang, Zezhi Shao, Jincai Huang, Junbo Zhang, and Yu Zheng. Spatio-temporal graph neural networks for predictive learning in urban computing: A survey. IEEE Transactions on Knowledge and Data Engincering, 2023.
Ming Jin, Huan Yee Koh, Qingsong Wen, Daniele Zambon, Cesare Alippi, Geoffrey I Webb, Irwin King, and Shirui Pan. A survey on graph neural networks for time series: Forecasting, classifi- cation, imputation, and anomaly detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska Barwiiskk, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):352133222 2017.
Sanghyun Lee and Chanyoung Park. Continual traffic forecasting via mixture of experts. arXiv preprint arXiv:2406.0314p 2024.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusson convolutional recurrent neural net- work: Data-driven traffic forecasting, arXiv preprint arXiv:1707.01926, 2017.
Zhonghang Li, Lianghao Xia, Yong Xu, and Chao Huang. Flashst: A simple and universal prompt- tuning framework for traffic prediction. International conference on machine learning, pp. 28978 28988. PMLR, 2024.
Yuxuan Liang, Yutong Xia, Songyu Ke, Yiwei Wang, Qingsong Wen, Junbo Zhang, Yu Zheng, and Roger Zimmermann. Airformer: Predicting nationwide air quality in china with transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 14337, 2023.
Hangchen Liu, Zheng Dong, Renhe Jiang, Jiewen Deng, Jinliang Deng, Quanjun Chen, and Xuan Song. Spatio-temporal adaptive embedding makes vanilla transformer sota for traffic forecast- ing. In Proceedings the 32nd ACM international conference on information and knowledge management, pp. 4129-4129, 2023.
Xu Liu, Yutong Xia, Yuxuan Liang. Junfeng Hu, Yiwei Wang, Lei Bai, Chao Huang, Zhenguang Liu, Bryan Hooi, and Roger Zimmermann. Largest: A benchmark dataset for large-scale traffic forecasting. Advances Neural Information Processing Systems, 36, 2024.
Hao Miao, Yan Zhao, Chenjuan Guo, Bin Yang. Kai Zheng, Feiteng Huang, Jiandong Xie, and Christian Jensen. A unified replay-based continuous learning framework for prediction streaming data. arXiv preprint arXiv:2404.1 14999, 2024.
12

---

Published as a conference paper at ICLR 2025
Tong Nie, Guoyang Qin, Wei Ma, Yuewen Mei, and Jian Sun. Imputeformer: Low rankness- induced transformers for generalizable spatiotemporal imputation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.
2024.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023.
Zheyi Pan, Yuxuan Liang, Weifeng Wang, Yong Yu, Yu Zheng, and Junbo Zhang. Urban traffic prediction from spatio-temporal data using deep meta learning. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, pp. 1720-1736 2019.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. arXiv preprint arXiv:2301.12314, 2023.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Weilin Ruan, Wei Chen, Xilin Dang, Jianxiang Zhou, Weichuang Li, Xu Liu, and Yuxuan Liang. Low-rank adaptation for spatio-temporal forecasting. arXiv preprint arXiv:2404007919, 2024.
Zezhi Shao, Zhao Zhang, Fei Wang. Wei Wei, and Yongjun Xu. Spatial-temporal identity: A simple yet effective baseline for multivariate time series forecasting. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 4454-4458, 2022.
Zezhi Shao, Fei Wang, Yongjun Xu, Wei Wei, Chengqing Yu, Zhao Zhang, Di Yao, Guangyin Jin, Xin Cao, Gao Cong, et al. Exploring progress in multivariate time series forecasting: Compre- hensive benchmarking and heterogeneity analysis. arXiv preprint arXiv:2310.061 19, 2023.
Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics, 12:389 434, 2012.
Gido M van de Ven, Nicholas Soures, and Dhireesha Kudithipudi. Continual learning and catas- trophic forgetting. arXiv preprint 240::240000517555 2024.
Binwu Wang, Yudong Zhang, Jiahao Shi, Pengkun Wang. Xu Wang. Lei Bai, and Yang Wang- Knowledge expansion and consolidation for continual traffic prediction with expanding graphs. IEEE Transactions on Intelligent Transportation Systems, 24(7):7190-7201, 2023a.
Binwu Wang, Yudong Zhang. Xu Wang, Pengkun Wang, Zhengyang Zhou, Lei Bai, and Yang Wang. Pattern expansion and consolidation on evolving graphs for continual traffic prediction. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 2223-2232, 2023b.
Binwu Wang, Jiaming Ma, Pengkun Wang, Xu Wang, Yudong Zhang, Zhengyang Zhou, and Yang Wang. Stone: A spatio-temporal ood learning framework kills both spatial and temporal shifts. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2948 29499,299,2225 2024a.
Binwu Wang, Pengkun Wang, Yudong Zhang, Xu Wang, Zhengyang Zhou, Lei Bai, and Yang Wang. Towards dynamic spatial-temporal graph learning: A decoupled perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 9089-9097, 2024b.
Binwu Wang, Pengkun Wang, Zhengyang Zhou, Zhe Zhao, Wei Xu, and Yang Wang. Make bricks with a little straw: Large-scale spatio-temporal graph learning with restricted gpu-memory capac- ity. In IJCAI, pp. 2388-2396. Jeju, South Korea, 2024c.
Liyuan Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024d.
Senzhang Wang, Jiannong Cao, and Yu Philip. Deep learning for spatio-temporal data mining: A survey. IEEE transactions knowledge and data engineering, (099)3681-37000 2020.
13

---

Published as a conference paper at ICLR 2025
Zifeng Wang, Zizhao Zhang, Chen-Yu Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 139-
149, 2022.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations. arXivpreprint, 2022.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. arXiv preprint arXiv: Ntv.199060012111 2019.
Yanan Xiao, Minyu Liu, Zichen Zhang, Lu Jiang. Minghao Yin, and Jianan Wang. Streaming traffic flow prediction based on continuous reinforcement learning. In 2022 IEEE International
Conference on Data Mining Workshops (ICDMW), pp. 16-22. IEEE, 2022.
Xiaosong Yang. Thomas L Delworth, Liwei Jia, Nathamiel C Johnson, Feiyu Lu, and Colleen McHugh. Skillful seasonal prediction of wind energy resources in the contiguous united states. Communications Earth & Environment, 5(1):313, 2024.
Chin-Chia Michael Yeh, Yujie Fan, Xin Dai, Uday Singh Saini, Vivian Lai, Prince Osei Aboagye, Junpeng Wang, Huiyuan Chen, Yan Zheng, Zhongfang Zhuang, et al. Rpmixer: Shaking up time series forecasting with random projections for large spatial-temporal data. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3919-3930, 2024.
Zhongchao Yi, Zhengyang Zhou, Qihe Huang, Yanjiang Chen, Liheng Yu, Xu Wang, and Yang Wang. Get rid of task isolation: A continuous multi-task spatio-temporal learning framework. arxiv preprint arXiv:2410.10524, 2024.
Du Yin, Hao Xue, Arian Prabowo, Shuang Ao, and Flora Salim. XxItraffic: Expanding
and extremely long traffic dataset for ultra-dynamic forecasting challenges. arXiv preprint arXiv:2406. 122.92236,.,,,,, 2024.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. arXiv preprint urXiv:1709.04877 2017.
Rose Yu, Dehua Cheng, and Yan Liu. Accelerated online low rank tensor learning for multivariate spatiotemporal streams. In International conference on machine learning, pp. 238 238-247. PMLR, 2015.
Yuan Yuan, Jingtao Ding, Jie Feng, Depeng Jin, and Yong Li. Unist: a prompt-empowered uni- versal model for urban -etiiottmmporl prediction. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4095-4106, 2024.
Jingbo Zhou, Xinjiang Lu, Yixiong Xiao, Jiantao Su, Junfu Lyu, Yanjun Ma, and Dejing Dou. Sdwpf: dataset for spatial dynamic wind power forecasting challenge at kdd cup 2022. arXiv preprint arXiv:2208.04 2022.
14

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY MATERIAL
EXPAND AND COMPRESS: EXPLORING TUNING PRINCIPLES FOR CONTINUAL TETTO-TEMPORAL GRAPH FORECASTING
TABLE OF CONTENTS
A Theoretical Proof
16
A.1 Prompt Parameter Pool Can Continuously Adapt to Heterogeneity Property
16
A.2 Prompt Parameter Pool Can Continuously Satisfy the Low-rank Property
17
B Method Detail
18
B.1 Method Algorithm
18
B.2 Method Process
19
Experimental Details
19
C.I Datasets Details
19
C.2 Baseline Details
20
D More Results
21
STGNNs Component Details
22
E.1
Spectral-based Graph Convolution Formula
22
E.2 Spatial-based Graph Convolution Formula
23
E.3
Recurrent-based Sequence Modeling Formula
................................
23
E,4
Convolution- based Sequence Modeling Formula
23
E.5
Attention-basal Sequence Modeling Formula
24
E.6
Summary
24
F
More Discussion
24
E1
Discussion...
24
F2
Limitation
25
E3 Future work
26
15

---

Published as a conference paper at ICLR 2025
A THEORETICAL PROOF
A.1 PROMPT PARAMETER POOl CAN CONTINCOUSLY ADAPT TO HETEROGENEITY PROPERTY
Proof. First, based on the definition of the average node vector dispersion, the original feature matrix can be further rewritten as:
( =
I. - 2312
(3)
72
For the spatio-temporal learning function fa with invariance (i.e., frozen STGNN backbone net- work), have X f(;; X p if the network converges, f can be obtained through nonlinear fitting. Therefore, for vectors in the matrix, we denote:
x² xx
(4)
Similarly, substituting this into the new feature matrix, it can be rewritten as:
D(X))
20 + p. Iz - 1311
(5)
n
1=1
Based on the following equation property:
Σ
|0 - 22 = 2n
1 1
Σ 2
(6)
J
1=1
Thus, the difference between Equation 5 and Equation 3 can be rewritten as:
D(X - ((X)
I + p
2j -9112
12
122
ビ= 1=
(=1
j=1
2
n
1
2n
c + 1 22
Σ (x; )
2n
T2
2
2
2
|2:
111
Σ (x; + p:)
| 1
+
2
n
n²
1=1
The mean vectors of the original feature matrix and the prompt parameter matrix defined as follows:
1
Tin
P
P
(8)
Therefore, we have:
20
n141
Σ p
np
(9)
16

---

Published as a conference paper at ICLR 2025
Substituting this into Equation 7 and further simplifying yields:
D(X") - -DXX)
2
2 n
||: + +111== n2
mp + npppl
e1
Σ
||0:| 1|
+
||a1
n
n
i=1
2
n Σ
|| + 11
2 + 1111
|0:1| + 2||u11
n
12
2
Σ
a
| 2 + Σ
a
11 + Σ
n
2 (11=112 + 2/x T mi + 11
n 2
2 n
2 + |0112
34 P1
n
N
2
4
11?
+
Σ
T
Ap T 12p 2112g11
1=1
2
l| 11?
+
Σ T1
n
n
D! )
4p ug -211x211
11
1
f1 i=1
i=1
2
4 (1u)
112
+ n
T pg
4u T pg 21 n2!1
n
2(
|p -1
(10)
According to the corollary of the Cauchy- SehwarSS inequality (Horn & Johnson 2012):
n
Σ
xx,|| 2
Σ
x4
(11)
72
Therefore, for any set of vectors {p?} J. we have:
2
n|
ml112112.
(12)
That is:
Σ
lp 2 - 114211 2
(13)
n 1=1
Thus,
n Σ
(1x))- =2(
1pr2112 11 2
(14)
7
This completes the proof.
A.2 PROMPT PARAMETER POOL CAN CONTINCOUSLY SATISFY THE LOW-RANK PROPERTY
Proof. We first construct Random Matrix & E R Let /b be random matrix with entries 6ij drawn independently from the standard normal distribution scaled by 1/V/:
1 0iz N 0, ke
Next, we can define matrices A= = T Rank and B = &P E kxd Thus, we have
AB = क T ($P) = ( T P.
Consider the approximation error:
PP- AB ||(T, - $ bT>) < In Φ TT 011 ||]
17

---

Published as a conference paper at ICLR 2025
Thus, goal is transform to bound In, + 2112- According concentration inequality for tail
bounds of spectral norms of sums of random matrices (Tropp 2012), for any (0, 1):
Pr (1 Lnll2 26) S 2n
We further show that, when At
4 log min((n,))
obtain:
Pr (1 T + Inll2 2 - 2n
log min(nd) 2n min (n, d) -2
Since min(n, d) S n, it follows that:
2 + n1
2n min(n, d) 2 S 2n 11 -
as 11 - oc.
We have:
Pr p - <) 2 - (1).
Therefore, due to the symmetry of the matrix norm, with probability at least 1 - o(1), we have:
|P-ABBle I F - T 1 plle P|l.
Thus, there exist matrices E Bonk and B E R with & O (log(min(n,,)) d))) such that:
(r - AB I P| ž o(1),
where o1) represents term that becomes negligible as grows large.
This completes the proof.
B METHOD DETAIL
B. METHOD ALGORITHM
Algorithm The workflow of EAC for continual graph forecasting
Input:
Dynamic streaming
graph G "a,,",,...
Observation data X = (X1, X2, XT).
Output:
prompt parameter pool P (in memory).
Pipeline:
I: while Stream Graph G remains do
2: if = then
3:
Construct an initial prompt parameter pool: P A
>> Tunnng Principle II: Compress Fusion of observed data X and prompt parameter pool P: X, = X, P. Jointly optimize the hase STGNN fo and P: fo- argmin, fo (x+,
4:
clse
7:
Reload the prompt pool P and model for
8:
Detect nodes and construct a prompt parameter matrix A(*)
9:
Add new node prompts to the prompt parameter pool: P Paoppend ((T) B)
Tuning Principle I: Expand
10: Fusion of observed data X and prompt parameter pool P: X, = X, P.
11: Jointly optimize the frozen STGNN fa- and P. with data X..
end if
13:
Use the STGNN and prompt parameter pool P for current period prediction.
14:
end while
15:
return Prompt Parameter Pool P
Here, present the workflow of EAC for continual spatio-temporal forecasting in Algorithm
18

---

Published as a conference paper at ICLR 2025
B.2
METHOD PROCESS
Our input consists of a series of continual spatio-temporal graphs and corresponding observational data. Our objective is maintain continuous dynamic prompt parameter pool. Specifically, we first construct an initial prompt parameter pool P that includes the initial node prompt parameters p()), which we approximate using subspace parameters A and fixed parameters B (line 3). We then perform element-wise addition of the observational data X1 with the prompt parameter pool at the node level (line 4), and feed this into the base STGNN model fo for joint training (line 5). In the subsequent 7-th period, we detect new nodes compared to the
phase and construct prompt parameter matrix A ( for all newly added nodes (line 8), which is then incorporated into the prompt parameter pool d (line 9). We again perform element-wiss addition of the observational data D- with the prompt parameter pool at the node level (line 10) and input this into the frozen base STGNN model fo- for training (line I1).
After completing each period of training, we use the current prompt parameter pool and the base STGNN fo to predict the results for the current period (line 13).
C EXPERIMENTAL DETAILS
C.I DATASETS DETAILS
Our experiments are carried on three real-world datasets from diffrent domain. The statistics of these stream spatio-temporal graph datasets are shown in Table 5
Table 5: Summary of datasets used for continual spatio-tempone datasets.
Dataset
Domain
Time Range
Period
Node Evolute
Frequency
Frames
1087 + 1154
Air-Stream
Weather
01/01/2016 12/31/2019
4
hour 34,065
+ 1193 + 1202
655 > 715 786
PEMS-Stream
Traffic
07/10/2011 09/08/2017
7
822 834 850
min 61,992
871
103 - 113
Energy-Stream
Energy
Unknown (245 days)
4
10 min
34,560
> 122
134
Specifically, we first utilize the PEMS-Stream dataset, which serves benchmark dataset, for analysis. The objective is to predict future traffic flow based on historical traffic flow observations from directed sensor graph. Additionally, we construct Air-Stream dataset for the meteorolog- ical domain focused on air quality, aiming to predict future air quality index (AQI) flow based on observations from various environmental monitoring stations located in China. We segment the data into four periods, corresponding to four years. We also construct a Energy-Stream dataset for wind power in the energy domain, where the goal is to predict future indicators based on the generation metrics of a wind farm operated by a specific company (using the temperature inside the turbine na- celle as substitute for active power flow observations). We also divided the data into four periods, corresponding to four years, based on the appropriate sub-period dataset size.
We follow conventional practices (Li et al., 2017 Bai et al., 2020) to define the graph topology for the air quality and wind power datasets. Specifically, we construct the adjacency matrix A, for 7-th year using a threshold Gaussian kernel, defined as follows:
1-2
if exp (- 2 and í otherwise
#
exp
A+(i)]
where dy represents the distance between sensors i and j. a is the standard deviation of all distances, and is the threshold. Empirically, select values of 0.5 and 0.99 for the air quality and wind power datasets, respectively.
There are some differences in the datasets of these three fields. Specifically, the pri- mary difference between the three spatio-temporal datasets lies in the underlying ppatiottemporal dynamics, spatial dependencies: 0 (Differences in Underlying Spatio-temporal Dynamics) In
Discussion:
19

---

Published as a conference paper at ICLR 2025
order, Snergy-Stream involves small wind farm with closely spaced turbines that share similar spa- tial patterns. PEMS Stream represents the entire transportation system in Southern California, with moderate spatial dependencies. Air-Stream goes further, encompassing air quality records from air monitoring stations across all of China, leading to much more complex spatial dependencies. These differences are reflected in the MAE and RMSE metrics, with the performance progressively de- grading as the complexity increases. @ (Special Characteristics of Energy Data) Regarding the MAPE metric, we must note that in the Anergy--tream dataset, turbines occasionally face overheat- ing protection or shutdown for inspection, which can cause some devices to produce very small values during certain time periods. This causes large pereentage errors when using MAPE.
C.2 BASELINE DETAILS
We have provided a detailed overview of various types of methods for continual spatio-temporal forecasting in the main body of the paper. The current core improvements primarily focus on online- based methods. Below is a brief introduction to these advance methods:
TrafficStream (Chen et al., 2021b): The paper first introduces novel forecasting framework leveraging spatio-temporal graph neural networks and continual learning to predict traffic flow expanding and evolving networks. addresses challenges in long-term traffic flow prediction by integrating emerging traffic patterns and consolidating historical knowledge through strate- gies like historical data replay and parameter smoothing. https://github. cwm/
Trafficst
ree main in
STKEC (Wang et 2023ah: The paper also presents continual learning framework, which is designed for traffic flow prediction on expanding traffic networks without the need for historical graph data. The framework introduces a pattern bank for storing representative network patterns and employs a pattern expansion mechanism to incorporate new patterns from evolving networks. https://gg
com/ wangb nwol 31 61 /TKC
PECPM (Wang et al., 2023b): The paper also discusses a method for predicting traffic flow on graphs that change time. To address the challenges presented by these dynamic graph, the paper proposes a framework that includes two main components: Knowledge Expansion and Knowledge Consolidation. The former is responsible for detecting and incorporating new pat- terns and structures as the graph expands. The latter aims to prevent the loss of knowledge about traffic patterns learned from historical data as the model updates itself with new information.
TFMoE (Lee & Park 2024): The paper introduces Traffic Forecasting Mixture of Experts for continual traffic forecasting on evolving networks. The key innovation is segmenting traffic flow into multiple groups and assigning dedicated expert model to each group. This addresses the challenge of catastrophic forgetting and allows each expert to adapt to specific traffic patterns without interference.
Horizon
3
12
Metric
MAE
RMSE
MAPE(%)
MAE
RMSE
MAPE(%)
ST-CRE
18.41
24.63
22.64
24.45
35.11
29.40
BAC
12.6540.03
20.2410.06
17.8020.08
14.92+0.11
24.1740.17
20.8240.16
Table 6: Comparison of ST-CRL and our method on the PEMS-Stream benchmark.
We also highlight some related works and explain why they are not included as baselines:
ST-CRL (Xiao al.) 2022); This work integrates reinforcement learning into continual spatio- temporal graph learning. We exclude it because its code is unavailable and its methods are not reproducible. Additionally, its poor performance further justifies this decision. Notably, according to the original paper, the comparison between ST-CRL and our method on the PEMS-Strean benchmark, shown in Table 6 provides evidence for this.
URCL (Miao al., 2024): This work incorporates data augmentation into continual spatio- temporal graph learning. However, it essentially treats spatio-temporal graphs static, with only observed instances evolving over time. Thus, this approach is not directly comparable to ours.
UniST (Yuan et al,, 2024): This work can be viewed as a parallel effort in two different directions. It focuses on large-scale pre-training in the initial stage. While it also uses some empirical prompt
20

---

Published as a conference paper at ICLR 2025
learning methods for fine-tuning, it is not suitable for continuous incremental scenarios. Another key point is that is limited to spatio-temporal grid data.
FlashST (Li et al. ):24): This work essentially limited to static spatio-temporal graphs, adjust- ing data distributions via prompt embedding regularization to achieve efficient model adaptation across different spatio-temporal prediction tasks. Therefore, it cannot be reasonably applied to continuous spatio-temporal graph learning.
CMuST (Yi et al. 2024); This recent work addresses spatio-temporal learning in continuous multitask scenarios, enhancing individual tasks by jointy modeling learning tasks in the same spatio-temporal domain. However, this framework mainly focuses on task-level continual learn- ing, transitioning from task to another. It does not address the continuous apatio-tempori graph learning characteristics encountered in real spatio-temporal prediction scenarios. Hence, this approach is also not suitable for direct comparison our single-task dynamic scenarios.
Metrics Detail, We use different metrics such as MAE, RMSE, and MAPE. Formally, these metrics formulated following:
MAE
Dsl. RMSE
1
y: Σ 1=1
6.)3. MAPE
100%
Σ
1:
y/s
Y2
y/
i=1
where n represents the indices of all observed samples, y; denotes the i-th actual sample, and y; the corresponding prediction.
Table 7: Hyperparameters setting.
Graph operator hidden dimension
64
Sequence operator (TCN) kernel size
Graph layer
Sequence layer
Training epochs
100
Batch size
128
Adam
1e-8
Adam |3
(0.9,0.999)
Learning rate
0.03 / 0.01
Loss Function
MSE
Dropout
0.1 0.0
Parameter Detail. Detailed hyperparameters settings are shown in Table We use the same parameter configurations for our EAC along with the other baseline methods according to the recommendation of previous studies Chen et al. (2021b); Wang et al. (003a)). Our only hyper- parameter & is set to by default. All experiments are conducted on a Linux server equipped with a x AMD EPYC 7763 128-Core Processor CPU (256GB memory) and 2 NVIDIA RTX A6000 (48GB memory) GPUs. To carry out benchmark testing experiments, all baselines are set to run for duration of 100 epochs by default, with specific timings contingent upon the method with early stop mechanism. The source code, data, experimental results, logs, and model weights can be accessed in the anonymous repository at https: :/pt::///ii thub. com haam//Ondean//// /EAC
D MORE RESULTS
To evaluate the performance differences between EAC and models trained individually for each period, we compare the average forecasting performance over time steps for each period on the PEMS-Stream dataset. This comparison includes our EAC and the individually trained Retrain-ST method, with results averaged over five random runs and shown Table 8
SR Underperformance) Retrain-ST exhibits unsatisfactory perfor- mance because it relies solely on limited data to train models for specific periods, failing to effec- tively leverage historical information from pre-trained models. In continuous spatiotemporal graph
Result Analysis: !
21

---

Published as a conference paper at ICLR 2025
Methods
Metric
2011
2012
2013
2014
2015
2016
2017
Average
MAE.
14.26
13.09 40.75
13.88.0.1
14.76
14.14
a3
15.26.0.5
14.24
Retrain-ST
RMSE
21.97
-0.00
22.50-0.1
23 23.82-011 19
23. 35
24
24.98-041
23.20
MAPE(%)
b9.33
20.19
22.06x2.0
20.33e1.57
19.48al
21.82:3.9
20.30
MAF
13.46i0
13.00-00e
13.07-n05
14.00
13.01
14.57-mm
13.53
SAC
RMSE
20.49w025
20.19
20.90uauo
22.27
21990ua30
23.08
23.54...1
21.77
()
18.12
18.98k0m
Table Comparison of the average performance of Retrain-ST and EAC across different periods in PEMS-Stream benchmark.
scenarios, underlying spatiotemporal dependencies typically shared across periods. Leverag- ing historical training data helps models perform better, a benefit clearly observed with our EAC approach. @ (Performance Variability Across Different Periods): Performance differences across periods are evident even with the full dataset. These differences primarily stem from factors such as data noise and varying levels of learning difficulty in different periods.
Further Analysis: We visualize the noise levels across periods in the PEMS-Stream dataset, as shown in Figure [ Specifically, Fourier transform is used to convert time series from the time do- main to the frequency domain, where noise is typically represented by high-frequency components. higher proportion high-frequency energy indicates greater noise. Analyzing the frequency- domain characteristics of the spatio-temporal data reveals that for 2017 (red line), low-frequency components are minimal (Ratio > 0.42), while in 2014 (purple line), high-frequency components more concentrated (0.46 ~ 0.48). This indicates that data from these periods are noisier and harder to learn, a trend consistent with the results in Table
2011
2012
2013
2014
2015
2016
2017
Procunno
High-Freguency Energy Ratio
Figure 8: High-freug feghfrrey energy ratios across different periods in TEMS-Stream benchmark.
E STGNNS COMPONENT DETAILS
In the design of ppatio-temporal graph neural network (STGNN), spectral based and spatial- -asee graph convolutions have different operation modes, and sequence convolution operators also be divided into recurrent-based. convolution-basedd and attention-based. Below are the formulas and explanations of each type.
E.1 OPECTRAI--BSED GRAPH CONVOLUTION FORMULA
Spectral graph convolution relies on the eigenvalue decomposition of the graph Laplacian matrix. The specific formula is as follows:
Graph Convolution:
Z = Uga (A)U
U is the matrix of eigenvectors of the graph Laplacian L =D= A.
22

---

Published as a conference paper at ICLR 2025
A is the diagonal matrix of eigenvalues of the Laplacian.
is the node feature matrix.
(A) is the convolution kernel function the eigenvalues, typically parameterized (e.g., using Chebyshev polynomial approximations).
Approximation of Convolution Kernel:
Using Chebyshev polynomial approximations, one can express the formula as:
K
Z TT)x
L =22/> 2L/Amax - I is the normalized Laplacian matrix.
TA (L) is the Chebyshev polynomial, and 0₂ are the learnable parameters.
E.2 SPATIAL-BASED GRAPH CONVOLUTION FORMULA
Spatial graph convolution aggregates information directly using the adjacency matrix. The classic formula for spatial graph convolution is follows:
= (AxW)
Á D D-1//AD-1/ 1/2 is the normalized form of the adjacency matrix.
is the node feature matrix.
W is the learnable weight matrix.
(-) is a nonlinear activation function (e.g., ReLU).
E.3 RECURRENT-BASER SEQUENCE MODELING FORMULA
Recurrent-baad based sequence convolution typically uses recurrent neural networks (RNNs), such as LSTM or GRU. we choose LSTM, the formulas are as follows:
LSTM Equations:
Forget Gate: f; = a (W [h, + by)
Input Gate: i, = a (W; [h,-1, x1| b,)
Candidate Memory: Č = tanh (W [ x,] + bc)
Output Gate: = α (W, (h x,] + b.)
Cell State Update: C, f, @ C,-1 + i, @ C
Hidden State Update: h, = O, O tanh(C.)
E.4 CONVOLUTION- BASED SEQUENCE MODELING FORMULA
Convolution- baased sequence convolution employs one-dimensional (ID) convolution to process time series data. The formula is as follows:
Z = (Conv ID(X, W, b))
X B is the input sequence, where B is the batch size, Cj the number of input channels, and T is the time steps.
W E C is the convolution kernel, with K being the kernel size.
b is the bias vector.
(-) the activation function.
23

---

Published as a conference paper at ICLR 2025
E.5 ATTENTION-BASE BASED SEQUENCE MODELING FORMULA
The attention-based model commonly uses the formula for the multi-head self-attention mechanism (e.g., in Transformers):
Attention Weights:
QK
Attention(Q. K, V)
softmax
vdk
Q X WW is the query matrix.
K XWx K is the key matrix.
V XWy is the value matrix.
dę the dimension of the key vectors.
Multi-head Attention:
MultiHead K) Concat (head), ....., heads) Wo
Each head is defined as head, Attention (Qriionn(QQ,4 Ki, Vi).
E.6 SUMMARY
Spectral-based Graph Convolution Operator: Achieves graph convolution through the eigen- value decomposition of the graph Laplacian.
Spatial-based Graph Convolution Operator: Directly aggregates node features using the adja- cency matrix.
Recurrent-based Sequence Operator: Uses LSTM or GRU to capture temporal dependencies.
Convolution-based Sequence Operator: Employs 1D convolution handle time series data with fixed window sizes.
Attention-based Sequence Operator: Uses multi-head self-attention model global dependen- cies in long sequences.
These formulas form the of spatio-temporal graph neural networks, allowing the combination of different operations to form the basis architecture of most current models.
F MORE DISCUSSION
F.1 DISCUSSION
Parameter Inflation: In addition to the analysis mentioned in the main text, we further provide a more detailed analysis of the concerns about parameter inflation. Specifically, this includes the following points:
(Adjustable Parameter Count): As we introduced in main text, there is no free lunch in this re- gard. While introducing the prompt parameter pool significantly improves performance, there is an inevitable risk of parameter inflation as the dataset size grows with more nodes. Therefore, dedicated entire section empirical observations and theoretical analysis explain how to reasonably eliminate redundant parameters, offering compress principle. Consequently, the adjustable parameter count can be dynamically reduced with changes in k. In the hyperparameter analysis section, we further demonstrate that even with limited number of tuning parameters, our approach still achieves SOTA performance. Thus, in large-scale scenarios, we can appropriately select the hyperparameter & to balance performance and efficiency.
(Freezing Backbone Model); One of the main advantages of EAC is that, compared to existing methods, freezing the backbone model directly leads to significant efficiency improvements, even in lcrge-scale datasets. For example, air-stream dataset, and this can further accelerated by adjusting parameter Therefore. our approach is clearly the best choice compared to others.
maintained the fastest average training speed the
24

---

Published as a conference paper at ICLR 2025
(Advantages of In-Memory Storage): Another point worth noting is that we need to point out that our prompt parameter pool is separate from the backbone model. Therefore, we can naturally store it in memory and only load it when needed. Therefore, for practical applications, overloading the prompt parameter pool is unnecessary worry.
(Easy-to-manage solution): As for even larger global datasets, the current backbone model size would clearly be insufficient and would need to be scaled up. In contrast, the growth the prompt parameter pool can be considered a more manageable solution.
Stacking all historical data together for training might seem reasonable at first glance, but it is actually impractical and reflects a misunderstanding of continuous spatio- temporal graph learning. As we pointed out in the last sentence of the first paragraph of the intro- duction: Due to computational and storage costs, it is often impractical to store all data and retrain the entire STGNN model from scratch for each time period. Thus, the primary motivation behind continuous spatio-temporal graph modeling methods is:
Use All Historical Data:
(Training and Storage Costs): Storing all historical data and retraining is associated with unac- ceptable training and storage costs. Training costs easy understand, but storage costs significant because the model is usually only a fraction of the size of the data (e.g. in the PEMS- Stream benchmark, the model size per year is 36KB compared the dataset size of 1.3GB, ap- proximately (.5).
In addition this fundamental motivation, we would like share further insights:
(Privacy Risks): In common continuous modeling tasks such as vision and text, a key improve- ment direction to avoid accessing historical data, as this poses privacy risks beyond storage costs (Wang et al. (2024d). Accessing models that store knowledge from historical data is clearly safer. Common improvements, such as regularization-ben bbasd and prototype-bast methods, are moving in this direction.
(Existing Approximation Methodsj: The existing Online-ST methods can be seen as an approxi- mation solution to training with all historical data. However, this often suffers from catastrophic forgetting and the need for full parameter adjustments, issues that our EAC effectively addresses.
F.2 LIMITATION
In this paper, we thoroughly investigate methods for continual spatio-temporal forecasing. Based on empirical observations and theoretical analysis, we propose two fundamental tuning principle for sustained training. In practice, we consider this kind of continual learning to fall under the paradigm of continual fine-tuning. However, given the superiority and generality of our approach, we believe this provides an avenue for future exploration of continual pre-training. While have made a small step in this direction, several limitations still warrant attention.
All current baselines and datasets primarily focus scenarios involving the continuous expansion of spatio-temporal graphs, with little consideration given to their reduction. This focus is reason- able, as spatio-temporal data is typically collected from observation stations, which generally do not disappear once established. Consequently, existing streaming spatio-temporal datasets are predom- inantly aspansion-oriented Nonetheless, there are exceptional cireumstances, such as monitoring anomalies at stations or the occurrence of natural disasters leading to station closures, resulting in node disappearance. But, we want to emphasize that our method can effectively handle such situ- ations, as our prompt parameter pool design is node-level, allowing for flexible selection of target node sets for parameter integration.
All current baselines and datasets span maximum of seven years. We believe this is reasonable constraint, as a longer time frame might lead to drastic changes in spatio-temporal patterns. A more practical approach would be to retrain the model directly in the current year. However, research extending beyond this time span remains worthy of exploration, which we leave for future work.
Due to the node-level design of our prompt parameter pool, the issue of parameter inflation is inevitable. Although we have proposed effective compression principles in this paper, other avenues for compression, such parameter sparsification and pruning, also merit investigation. Given that we have innovatively provided guiding principles for compression in this study, we reserve further improvement efforts for future research.
25

---

Published as a conference paper at ICLR 2025
F.3 FUTURE WORK
One lesson learned from our experiments is that the initially pre-trained spatio-temporal graph model crucial for the subsequent continuous fine-tuning process. This is highly analogous to today's large language models, which compress rich intrinsic knowledge from vast corpora, allowing them to per- form well with minimal fine-tuning in specific domains. Therefore, we consider a significant future research direction to be the training of sufficiently large foundation spatio-temporal model from scratch, utilizing data from diverse fields and scenarios. While some discussions and studies have emerged recently, we believe that truly foundation spatio-tempons model remains considerable distance away. Thus, we view this as long-term goal for future work.
26


Reviews:
[{"review_id": "2vJRm7E1B0", "reviewer": "Reviewer_S8oR", "summary": "This paper proposes EAC, a continuous spatio-temporal graph forecasting framework based on a continuous prompt parameter pool, aiming to address prediction challenges in dynamic streaming spatio-temporal data. EAC’s core idea is to freeze the base STGNN model and dynamically adjust the prompt parameter pool to adapt to new node data, achieving efficient knowledge transfer and mitigating catastrophic forgetting. The two tuning principles proposed in the paper, “expansion” and “compression,” along with their corresponding implementation schemes, demonstrate innovation and practical value.", "questions": "Please address the questions above.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a prompt-based continuous spatio-temporal forecasting framework, EAC, introducing the “expansion” and “compression” principles and offering a new perspective on solving dynamic streaming spatio-temporal data prediction problems.\n2. EAC can be combined with different STGNN architectures and performs well on various spatio-temporal data types.\n3. By freezing the base STGNN model and adjusting a limited number of parameters in the prompt parameter pool, EAC can improve speed and reduce the number of parameters to be adjusted, demonstrating its efficiency.", "weaknesses": "Overall, my concerns are mainly about experiments.\n(1) How does the performance of the schema adopt all historical spatio-temporal data for training, which is not mentioned in Fig. 1? It would be better if the performance of such schema were also discussed and included in the performance comparison.\n(2) Section 5.2 provides a detailed comparison between different methods, and a further discussion on the difference in results across different domains (weather, traffic, and energy) should also be provided.\n(3) The efficiency of EAC is observed to be largely influenced by the scale of the dataset in Section 5.4. Thus, a more in-depth analysis of the impact of the dataset scale on the model performance should be provided. This makes the real-world application questionable.\n(4) Many details of the baselines and datasets are missing.\n(5) More baselines published in 2023 and 2024 should be considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MuMge390KL", "reviewer": "Reviewer_7vMi", "summary": "This paper introduces a prompt-tuning approach for continual spatio-temporal graph forecasting, specifically addressing the challenges of dynamic data streams. The authors propose the EAC framework, guided by two tuning principles, \"Expand\" and \"Compress,\" to handle continual learning in STGNNs. By utilizing a continual prompt pool, EAC allows the base STGNN to accommodate new data while minimizing catastrophic forgetting. The authors demonstrate the approach’s effectiveness across various datasets, showcasing improvements in efficiency and adaptability compared to other methods.", "questions": "See the above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "S1.  EAC’s application of prompt tuning principles in continual spatio-temporal forecasting is novel, integrating dynamic prompt pool adjustments to effectively handle incoming data.\nS2. The methodology is backed by both empirical and theoretical analysis, and the explanations are clear.\nS3. The experimental results are impressive.", "weaknesses": "W1: While EAC is compared with several traditional and just-in-time tuning baselines, it is not included in comparison with other recent continuous learning techniques, such as combinations with reinforcement learning (Xiao et al., 2022) and data augmentation (Miao et al., 2024) mentioned in RELATED WORK. The reasons for the missing baselines are required. \nW2: The Prompt Parameter Pool in EAC may introduce an issue of parameter bloat, which needs to be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oKfZkYiwM7", "reviewer": "Reviewer_vM4w", "summary": "This paper introduces a novel framework, EAC, for continual spatio-temporal graph forecasting. The authors address the challenges of retraining inefficiency and catastrophic forgetting in streaming spatio-temporal data scenarios by proposing a prompt tuning-based approach. They present two tuning principles—expand and compress—that guide both empirical and theoretical analysis. The expand principle addresses the dynamic heterogeneity of the data, while the compress principle tackles parameter inflation. Results demonstrate that EAC is effective, efficient, universal, and lightweight in tuning, with extensive experiments on real-world datasets supporting these claims.", "questions": "- The authors note that the choice of pre-training backbone model is crucial. Does this imply that their method is more effective with larger-scale STGNN backbones?\n\n- How would the EAC model adapt if the graph were to shrink, for instance, due to the removal of sensors or monitoring stations? Why was this scenario not included in the comparisons?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem addressed is significant, as spatio-temporal graph forecasting has applications in areas such as traffic flow and air quality monitoring. The proposed solution offers potential improvements in efficiency and model effectiveness in dynamic, real-world environments compared to previous methods.\n\n- The paper presents a novel approach to continual learning within the context of spatio-temporal graph forecasting. The exploration of prompt tuning principles is innovative, and the authors offer a detailed and well-supported discussion of existing paradigms along with an extensive experimental analysis.\n\n- The methodology is well-developed, with clear explanations of the theoretical foundations and empirical insights leading to the expand and compress tuning principles. While node-level parameters and low-rank decomposition are common in the field, the authors’ thorough analysis and discussion bring valuable new perspectives.\n\n- The paper is well-organized and clearly written, making complex concepts accessible. The figures and tables are clear and complement the textual explanations effectively.", "weaknesses": "- While the prompt-based tuning paradigm for continual spatio-temporal forecasting is novel, similar recent methods [1,2,3] are only briefly mentioned in related work. A more detailed discussion of these approaches and their connection to the present work would be beneficial.\n\n[1] Yuan, Yuan, et al. \"Unist: a prompt-empowered universal model for urban spatio-temporal prediction.\" SIGKDD, 2024.\n\n[2] Li, Zhonghang, et al. \"FlashST: A Simple and Universal Prompt-Tuning Framework for Traffic Prediction.\" ICML, 2024.\n\n[3] Yi, Zhongchao, et al. \"Get Rid of Task Isolation: A Continuous Multi-task Spatio-Temporal Learning Framework.\" NIPS, 2024.\n\n- The approach still has several limitations, such as performance over long time spans and parameter inflation. However, the authors appropriately address these limitations in detail in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P3NVstRiuv", "reviewer": "Reviewer_YV6D", "summary": "The paper addresses the challenges of continual learning in spatio-temporal forecasting, particularly for data streams that evolve due to the deployment of new sensors. Traditional spatio-temporal graph neural networks struggle with retraining inefficiencies and catastrophic forgetting when applied to such streaming data scenarios. To overcome these issues, the authors propose a novel method called EAC (Expand and Compress). The proposed approach enhances the model’s capacity to manage evolving data without the need for full retraining, ensuring efficient and effective handling of dynamic spatio-temporal data streams.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a highly practical problem in spatio-temporal graph forecasting. On the one hand, new sensors are deployed over time, on the other hand, the patterns of spatio-temporal dynamics evolve.\n\n2. One of the strengths of the paper lies in its strong motivation, backed by both empirical and theoretical analysis.  The authors provide a clear rationale for addressing catastrophic forgetting and the challenges of handling dynamic, continuously evolving spatio-temporal data.\n\n3. The proposed method is reasonable.  I appreciate the idea of fixing the backbone of the spatio-temporal graph model and updating the prompt pool.  This approach strikes a balance: on the one hand, it preserves knowledge from previously trained samples, and on the other hand, it adapts to new incoming data effectively.", "weaknesses": "1. The design of the prompt pool is not clearly explained. Specifically, it is unclear what the prompt pool contains and how exactly these prompts are utilized within the model. Additionally, there is a lack of clarity on how the system handles the incorporation of new sensors in dynamic environments, which is a crucial aspect of the proposed approach.\n\n2. The evaluation lacks a comparison with models trained separately for each period. While the proposed continual learning method shows promising results, it is essential to establish a performance upper bound by comparing it to a scenario where separate models are trained for different periods. \n\n3. While the method outperforms baselines, I am concerned about its long-term effectiveness. In Figure 5, the model’s performance shows significant degradation over time, with the RMSE increasing from 24 to 28—indicating a more than 10% reduction in performance. Although this is in the context of few-shot learning, I suspect a similar trend would be observed in non-few-shot scenarios as well.\nWhile separate training for each period may be more time-consuming, it could potentially achieve better performance, and it is storage-efficient since only the latest model needs to be saved. Therefore, it is crucial to assess whether the trade-off between reduced performance and computational efficiency is truly justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)