Spatio-Temporal Few-Shot Learning via Diffusive Neural Network Generation
Authors: Yuan Yuan
Chenyang Shao
Jingtao Ding
Depeng Jin
Yong Li
Venue: ICLR 2024 poster (2024)
Keywords: Spatio-Temporal Few-Shot Learning
Diffusion models
Neural Network Generation
Primary area: generative models
TL;DR: This paper leverages diffusion models for neural network generation to achieve effective spatio-temporal few-shot learning.

Abstract: Spatio-temporal modeling is foundational for smart city applications, yet it is often hindered by data scarcity in many cities and regions. To bridge this gap, we propose a novel generative pre-training framework, GPD, for spatio-temporal few-shot learning with urban knowledge transfer. Unlike conventional approaches that heavily rely on common feature extraction or intricate few-shot learning designs, our solution takes a novel approach by performing generative pre-training on a collection of neural network parameters optimized with data from source cities.  We recast spatio-temporal few-shot learning as pre-training a generative diffusion model, which generates tailored neural networks guided by prompts, allowing for adaptability to diverse data distributions and city-specific characteristics. GPD employs a Transformer-based denoising diffusion model, which is model-agnostic to integrate with powerful spatio-temporal neural networks.  By addressing challenges arising from data gaps and the complexity of generalizing knowledge across cities, our framework consistently outperforms state-of-the-art baselines on multiple real-world datasets for tasks such as traffic speed prediction and crowd flow prediction. The implementation of our approach is available: https://github.com/tsinghua-fib-lab/GPD.

Published as a conference paper at ICLR 2024
SPATIO-TEMPORAL FEW-SHOT LEARNING VIA DIFFU- SIVE NEURAL NETWORK GENERATION
Yuan Yuan", Chenyang Shao', Jingtao Ding! Depeng Jin, Yong Li
Department of Electronic Engineering, BNRist, Tsinghua University, Beijing, China y y-yuan2 0$mails tsinghua. edu. en, {dingjingtao, 1iyong07]etsingeeeee. edu. en
ABSTRACT
Spatio-temporal modeling is foundational for smart city applications, yet it is often hindered by data scarcity in many cities and regions. To bridge this gap, we propose novel generative pre-training framework, GPD, for spatio-temporal few-shot learning with urban knowledge transfer. Unlike conventional approaches that heavily rely common feature extraction intricate few-shot learning de- signs, our solution takes a novel approach by performing generative pre-training collection of neural network parameters optimized with data from source cities. We recast spatio-temporal few-shot learning as pre-training a generative diffusion model, which generates tailored neural networks guided by prompts, allowing for adaptability to diverse data distributions and city-specific character- isties. GPD employs Transformer-basee denoising diffusion model, which is -adel-agnostic to integrate with powerful teaporle neural networks. By addressing challenges arising from data gaps and the complexity of generalizing knowledge across cities, our framework consistently outperforms state-of-the-art baselines on multiple real-world datasets for tasks such as traffic speed predic- tion and crowd flow prediction. The implementation of our approach is available: / wtt.:uu....mm...iitt.e..e.. fib lab/GPD
INTRODUCTION
Spatio-temporal prediction is fundamental problem in various smart city applications (Xia et al., 2024| [Zhou al., 2024) Wang et al., [203a bb). Many deep learning models are proposed solve this problem, whose successes however rely on large-scale spatio-temppral data. Due imbalanced development levels and different data collection policies, urban spatio-temporal data, such as traffic and crowd flow data, are usually limited in many cities and regions. Under these circumstances, the model's 's transferability under data-scarce scenarios of pressing importance.
To address this issue, various transfer learning approaches have emerged for spatio-temporal mod- eling. Their primary goal is to leverage knowledge and insights gained from one or multiple source cities and apply them effectively to target city. These approaches can be broadly classified into two main categories.
(1) Coarse- Coarse-grained grained methods consider each city as unified entity and transfer the learned knowledge the city level (Pang al., 20200; He et al. 2020 Ding al. 2019 Tang al.) 2022). (2) Fine-graing -re-ggrined methods dissect cities into smaller regions to enable a more refined explo- ration of region-level knowledge (Wang al. 2019 Guo al.) 2018 Yao al., 2019 Liu al., 2021 Jin et al., 2022: Lu et al. (2022). which have shown better performance due to the inherent disparities between source and target cities. However, existing fine-grained methods largely rely on elaborated matching designs, such as utilizing auxiliary data for similarity calculation (Wang et al.) 2019) or incorporating multi-task learning to obtain implicit representations (Lu et al. 2022). How to enable more general knowledge transfer to automated retrieving similar characteristics across source and target cities still remains unsolved.
"Equal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2024
Recently, pre-trained models have yielded significant breakthroughs in the fields of Natural Lan- guage Processing (NLP) (Brown et al. 2020 Vaswani et al. (017). Prompting techniques are also introduced to reduce the gap between fine-tume tuning and pre-training (Brown et al. 2020). At its core, the adoption of pre-trained models embodies the fundamental principles of transfer learning, allow- ing the model to acquire broad understanding of various patterns and subsequently adapt to ad- dress specific tasks. Nowadays, what's 's particularly noteworthy is that advanced pre-trained models no longer require laborious fine-tuning, but leverage effective prompting techniques for fast adap- tation (Brown et al., 2020; Rombach et al., 2022). Such capability serves as crucial remedy for the current limitations in spatio temporal few-shot learning. which offers the potential to enhance fine-grained transfer by general matching techniques.
However, despite these remarkable advancements of pre-trained models, there remains a notable gap in developing pre-trained models tailored for spatio-temporal scenarios. This disparity can be attributed to several challenges. Firstly, NLP benefits from a shared vocabulary that can be applied across various scenarios or tasks, while urban areas across different cities are geographically disjoint, lacking common elements that enable straightforward knowledge transfer. Secondly, substantial divergence often exists in data distributions between source and target cities, leading the potential noise or even counterproductive information in the knowledge acquired from source cities (Jin et al.) 2022]. Additionally, pattern divergence exists even within one city due regions with different functions. These divergences pose challenges to the effective transfer of knowledge to a target city. In other words, the knowledge obtained from the data often contains noise and bias, making it hard to train a universal model that perfectly fits patterns with high variations in different cities. Achieving effective transfer in this context requires addressing a more complex setting of generalization across cities. The key challenge, therefore, lies in determining what transferable knowledge, analogous to the shared semantic structure in NLP, can be established for urban settings.
In this work, we present a generative pre- training framework for Instead of fitting the spatio-temporal data with unified model, we propose a novel pre-training strat- egy that captures universal patterns from optimized neural network parameters. In simpler terms, we recast spatio-temporal few-shot learning as pre-training a generative hypernetwork. This hyper- network is designed to adaptively generate unique parameters for spatio-temporal prediction models guided by prompts.
few-shot learning.
Essentially, pre- pre-training approach empowers the capability to adaptively generate distinctive neural networks in response to diverse data distributions, which addresses the challenges arising from data gaps across cities regions. To elaborate, timized for spatio-temporal predictions. Then we design a Transformer- diffusion model to generate network parameters from Gaussian noise conditioned the prompt. We also design class of conditioning strategies for the prompt to guide the denoising process. Consequently, when presented with target prompt encoding spatio-temporal characteristics of the target scenario, the diffusion model generates corresponding neural networks for accurate predictions. Our framework -onel-agnostic, ensuring compatibility with state-of-the-art apatio-temporal prediction models. We summarize our contributions as follows:
begin with set of neural networks op-
We propose to leverage pre-training paradigm to achieve effective fine- fine-grained knowledge transfer across different cities, which stands as a pioneering practice in handling urban data-scarce scenarios with pretrained models.
We propose novel Generative Pre-training framework based on Diffusion models, called GPD. It leverages a Transformer-bree diffusion model and city-specific prompts to generate neural networks, opening new possibilities for improving spatio-temporal modeling.
Extensive experiments on multiple real-world scenarios demonstrate that GPD achieves superior performance towards data-scarce scenarios with an average improvement of 7.87% over the best baseline on four datasets.
2 RELATED WORKS
Spatio-Tene PeattioTemmpral Few-Shot Learning. Addressing data scarcity is pervasive challenge in ma- chine learning-drived driven urban computing applications (Wei et al.] [016:] He et al.) 2020; Zhou et al.] 2023ab. This issue is particularly pronounced in cities lacking advanced digital infrastructure in
2

---

Published as a conference paper at ICLR 2024
the initial stages of deploying sensors, as they struggle to accumulate adequate data. To solve these issues, spatio-temporal few-shot learning (Zhuang et al. 2020) has emerged as a promising solu- tion. Existing solutions be broadly categorized into coarse-grained and fine- fineggriiedd methods.
Coarse-grained methods (Fang et al. 2022 Liu et al. 2018 Pang et al. 2020; He et al.] 2020 Ding et al. 2019: Tang al. [022] Yao al.] 2019) treat each city as whole and apply transfer learning lechniques, such as adversarial learning (Fang et al. (022) and meta-learning (Yao et al. 2211)) to
leverage knowledge across urban contexts. Differently, fine-grained methods (Wang et 2019
Guo et al. 2018 Liu et al. 2021] Jin et al. (022) divide cities into smaller regions and identify similar region pairs for knowledge transfer. They use various techniques, such as Similarity- based matching (Wang et al. 2019) or re-weighting (Jin et al.) (022). to facilitate the transfer of insights between these regions. Fine-grained methods have demonstrated better performance due to the in- herent disparities between source and target cities (Jin et al. 2022). (Lu et al. 2022) proposed to learn meta- kett-knooweeeee to facilitate node-level knowledge transfer. Instead, we propose a generative pre-training framework made up of a diffusion-basee hypernetwork and adaptive conditioning strat- egy. Our framework not only exhibits powerful parameter generation capabilities but also offers remarkable flexibility by allowing the use of various forms of prompts.
Diffusion Models. Diffusion probabilistic models (Ho et al., 2020: Song et al., 2020 Nichol & Dhariwal (02T) have emerged powerful alternative to generation tasks, which not only out- perform Generative Adversarial Networks (Goodfellow et al. (020) on image generation tasks with higher fidelity (Dhariwal & Nichol, 2021 Rombach et al. 2022). but also enable effective cross-
modal conditioning (Bao et al. 2023 Nichol et al.) (022). In addition to image generation, diffusion models have also been utilized for other tasks, such as video generation (Ho al. 2022 Luo al.)
2023 text generation (Li et al., 2022 Gong et al. 2022). implicit neural fields generation (Erkog jt al. 2023), network learning (Peebles et al.) 2022] ) graph generation (Vignac et al., 2023). and spatio-temporal prediction (Wen et al. 2023 Yuan al. 2023). In contrast, our approach leverages diffusion models to generate neural network parameters of spatio-temporal prediction models.
Hypernetworks. Hypernetworks represent a class of generative models tasked with producing pa- rameters for other neural networks (Ha et al., (017). These hypernetworks serve two primary pur- poses: (I) the generation of task-specific parameters (Ha et al. 2017, Alaluf et al. 2022). where hypernetworks are usually jointly trained with specific task objectives, and (2) the exploration of neural network characteristies (Schürholt al. [022 2021). which focus on learning representa- tions of neural network weights to gain insights of network properties. In our work, we align with the first purpose, focusing on the generation of parameters tailored the target city for spatio- temporal prediction. There are also hypernetworks for spatio-temporal prediction (Pan et al.) 2019)] Bai et al. 2020 Pan al., 2020 2018 ent nodes. Our key differences lie in three aspects. Firstly, our framework ensures compatibility with state-of-the-aat models, while those works propose specific model designs. Secondly, different from the pre-training solution (Li et al. (2023) with mask autoencoder, we focus on the pre-training of model parameters. Thirdly, our approach is spatio-temporal few-shot learning framework, capable of acquiring knowledge from multiple cities and effectively transferring it to few-shot scenarios. In contrast, those works have certain limitations data-scarce urban environments.
Li et al. 2023), which use non-shared parameters for differ-
PROPOSED METHOD
3.1 PRELIMINARY
We introduce some preliminaries related to our research problem.
Definition 1 (Spatio-Temporal Graph). A spatio-temporal graph (STG) as GST = -V, where V is the node set, & is the edge set, A is the adjacency matrix, and X denotes the node feature. The node features can encompass various aspects such as crowd flows and traffic speed.
For example, for the crowd flow dataset, nodes correspond to segmented regions within the city, with edges indicating their geographic adjacency. For the traffic speed dataset, nodes represent the sensors deployed along the road network, and edges signify the connections between these sensors. The construction of the spatio-temporal graph is adaptable to the specific requirements of each task.
Problem Temporal Prediction). For a apatio-temport graph GST, suppose we have I. historical signals [xt-++1, X!-L+2 ,...,, Xt] for each node and predict the future steps
3

---

Published as a conference paper at ICLR 2024
(a) Preparation of Neural Networks
(b) Pre-Train the Diffusion Model
Target
Dataset of Source Cities
y
Noise
W.
Gy
Prediction Model
Source Prompt
Reconstruction
Figure 1: An overview of the proposed framework. (a) A collection of optimized spatio-temporal prediction models based on the dataset of source cities; each model's parameters are transformed into a vector-based format. (b) Pre-training the diffusion model to generate neural network parameters from the noise given the prompt. network parameters for the target city based on the target prompt.
(c) Utilizing the pre-trained diffusion model to generate neural
.X......,,tt The prediction task is formulated as learning a O-parameterized model given "0--++,...,X".
spatio-temporal graph GST: [X++] , Xt+x) - 1
Problem Temppoll Few-Shot Learning). Spatio-temporal few-shot learning formulated as learning knowledge K. from source cities Ct. {Crource, ..... ,...,Crrreee
Csouree) }. and then transfer the
learned knowledge to the target city with few-shot structured data to facilitate tentooaa predictions.
Problem 3. We formulate the apatio-temporde few-shot learning as pre-training diffusion model conditionally generates neural network parameters, and then utilizing generate parameters for the target city's prediction model. Suppose have collective of learned prediction models = {Fer, Feg. Fox h from a set of data-rich source cities (:P =[Crouree ,..., C...,ep }, we aim pre-train diffusion model generate F with source-city prompts. The optimized diffusion model serves as the learned knowledge K. which can be transferred to the target city.
3.2 OVERALL FRAMEWORK
Figure prooviees an illustrative overview of our proposed framework, which contains three phases. The left part shows the preparation model parameters, from which we can obtain a collection of optimized neural network parameters. The middle part illustrates the pre-training phase within source cities, where the diffusion model G, is trained to generate meaningful parameters from Gaus- sian noise given source prompts. The right part demonstrates the how transfer the learned diffu- sion model to facilitate spatio-temporal predictions in the target city, In the following subsections, elaborate the details of these phases.
3.3 SPATIO-TERT TEMPORAL PREDICTION MODEL
Our framework is model-agnostic. ensuring compatibility with saate-of-the-att spatio-temporal pre- diction models. Here utilize well-established STG models, specifically STGCN (Yu et al., 2017) and GWN (Wu et af.) 2019), and MI.P-based model STID (Shao et al., 2022a) as prediction mod- ell! In this work, train the prediction model (denoted Fs.) separately for each region within a given city.
3.4 GENERATIVE PRE-TR TRE-TRAINING PARAMETER SPACE
Our approach is conditional generative framework designed to directly leam from model parame- ters source cities. This pre-training process enables the generation new model parameters for target cities. Our training paradigm encompasses three-phase approach as follows.
Appendix A.4 provides details of the three models.
4

---

Published as a conference paper at ICLR 2024
Preparation of Neural Networks. As the initial step, optimize spatio-temporal prediction mod- els for each region within source cities and save their optimized network parameters. It's important to note that, while parameters is uniquely optimized for its respective region. other words, there is no parameter sharing, and each model is meticulously trained achieve peak performance specific its region. This optimization process can be formulated as follows:
the same network architecture for different regions, each set of model
0, arg min Σ [X
L+1 X! L+2
X/: (X ..... x
T {01, 02........................
00, AM M-
Σ ns
where i denotes the den region within source city, and 0g represents parameters of its dedicated prediction model. N denotes the total number source cities, and n, indicates the number of regions in source city s. T(0) encompasses the optimized parameters of prediction models.
To optimize these parameters, we employ the standard Adam optimizer. These meticulously op- timized parameters serve as the ground truth for the subsequent generative pre-training process. Algorithm T in Appendix A.3 illustrates the training process. Then we transform the parameters of each model into vector-based format.
Generative Pre-Training of the Diffusion Model. Using the dataset comprising pre-trained pa- rameters of prediction models and region prompts, we employ a generative model denoted G,2 to learn the process of generating model parameters in single pass. Specifically, G, predicts the distribution of parameters PG ps)) where p; corresponds to the prompt of region i. We adopt diffusion models (Ho et al. 2020 our generative model due to its efficacy in various generation tasks (Ho et al. 2022 Li al. 2022 Vignac et al. 2023). Moreover, it has shown superior perfor-
mance multi-modal conditional generation (Bao et al. 2023 Nichol et al. 2022 Saharia et al. 2022).
We train the diffusion model to sample parameters by gradually denoising the vector from the Gaus- sian noise. This process intuitively reasonable as intriguingly mirrors the optimization journey from random initialization which is well-established practice in existing optimizers like Adam. Our model takes two parts as the input: a woise-corrupted parameter vector 8 and a region prompt p. with & representing the step in the forward diffusion process. The training objective as follows:
L = Ego (- |||e (, (,(O k))))
where e denotes the noise to obtain 0 from 0º, Algorithm 2 illustrates the pre-training procedure.
Sampling. After pre-training, can generate parameters 0, by querying G.y using region prompt p for the target cities. When the region prompt in the target city closely resembles region in multiple source cities, domain. In essence, we harness the power of the prompt to facilitate efficient knowledge transfer and precise parameter matching, which leverages the inherent similarities between regions across cities. The generation is an iterative sampling process from step k = K to A = O, which denoises the Gaussian noise into meaningful parameters. The generation process is formulated follows:
seamlessly approximate the model parameters specific to the target
By
e, (0* k),))) + Vo,z,
(2)
ark
VI
ak
where z ~ NOO, I) for k > and z 0 for lo = 1. Algorithm 3 Billustrates the procedure.
3.5 ARCHITECTURE
The generative model is a transformer-bast diffusion architecture that learns the relationships across different layers of the prediction model via effective self-ane attention. It has been shown that the Transformer can effectively capture relationships of each token in long sequences. We find it to be good choice learn inter-layer and intra-layer interactions.
5

---

Published as a conference paper at ICLR 2024
Predicted noise
Unear and Reshape
Pointwise Feedforward
Feedforward
Prompt
Layer norm
Temporal Prompt
Spatial Prompt
Layer norm
Layer
Quenry
Attention
Mutti-haad
Self-
Adaptive Emb
Layer
ksttop
Embed
Prompt
Chunking
toklens
Stop Emb
Step
Timestop
Prompt
[a] Transformer based Denoising Network
(b) Pre Adoptive Conditioning Strategy
(c] Pre Conditioning Strategy
Figure 2: The network architecture of the denoising network. (b) and (c) illustrate two conditioning strategies, and we provide other used conditioning strategists in Appendix A.3
Parameter Tokenizers. Spatio-temporal prediction models always exhibit complicated architec- tures consisting of various neural network layers, such as temporal layers, spatial layers, and spatio- temporal interaction layers. Usually, these layers are characterized by different shapes. These het- erogeneous layer structures pose a challenge we want to leverage the Transformer model to capture intricate relationships between them. To provide clearer conceptual understanding, Table and and Ta- ble 5 Appendix [A.4] illustrate the layer structures of an spatio-temporal model. It is evident that different layers possess parameter tensors of diverse shapes. Therefore, it is necessary decom- pose these parameters into vector-base tokens of uniform dimensions while preserving the inherent connectivity relationships within the original prediction model.
We achieve this by determining the greatest common divisor (denoted as g) of the parameters amount across all layers, then we perform layer segmentation by reshaping each layer into several tokens as: = Nn. g. where n.m is the number of tokens for the layer m and Nm is the number of parameters of the layer m. After that, the tokens are connected in a sequential manner, ensuring that layers that are adjacent in the network structure also maintain adjacency within the resulting sequence. This systematic approach facilitates the effective utilization of Transformer-based architectures in handling spatio-temporal prediction models with heterogeneous layer structures.
Region Prompt. The selection of region prompts offers flexibility, as long as they can capture the distinctive characteristics of a specific region. Various static features, such as population, region area, function, and distribution of points of interest (POI), can be leveraged for this purpose. this work, we utilize region prompts from two aspects: spatial and temporal domains. For the spatial prompt, we utilize node embeddings extracted from a pre-trained knowledge graph. It only utilizes relations like region adjacency and functional similarity, which are easily obtained in all cities. Simultaneously, we employ self-supervised learning techniques (Shao et al. 2022b) on the very limited time series data (three days) to derive the embedding for the temporal prompt. Appendix A.2 provides more details of the prompt design.
Denoising Network. Figure 2 demonstrates the architecture of the denoising network, which adopts prompt-based transformer diffusion model. After the layer segmentation, the parameters are re- structured into token sequence. In the denoising process, the transformer layers operate on the noised token sequence. In addition to the noised sequence, our transformer diffusion model also takes in the timestep i and region prompt p. We anticipate that the model capable of generating outputs conditioned on the region prompt. To this end, we explore several conditioning approaches, which introduce small but important modifications to the standard transformer layer design. The conditioning strategies are shown in Figure 2 and We also explore more conditioning strate- gies, such as "Post-adaptive conditioning" and "Adaptive norm conditioning" (see Appendix A.3)
Pre-conditioning. "Pre" denotes that the prompt is integrated into the token sequence before being fed into aelf-attention layers. In this conditioning strategy, we simply add the spatial prompt and temporal prompt into a spatio-temporal prompt. Then, we uniformly add to each token. This design allows us leverage standard transformer layers without requiring any modifications.

---

Published as a conference paper at ICLR 2024
Pre-conditioning with inductive bias. When adding the vector embeddings of k and p to the token embeddings, we introduce inductive bias: the timestep vector is added uniformly to each token, while the spatial prompt and temporal prompt incorporated into spatial-related and temporal-related parameters, respectively. Inductive hias can be introduced flexibly based on the used spatio-temporal model architectures.
Pre-adaptive conditioning. In this variant, the operation on the timestep embedding remains the same. However, concerning the prompt. we treat the embeddings of p as two-element sequence. An attention layer is introduced to realize the "adaptive" mechanism, dynamically determining what extent the prompt should be added to specific token embeddings. This approach aims to empower the model to learn how to adaptively utilize the prompts, enhancing its conditioning capabilities.
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Datasets. We conduct experiments on two types of spatio-temporal prediction tasks: crowd flow prediction and traffic speed prediction. As for crowd flow prediction, we conducted experiments three real-world datasets, including New York City, Washington, D.C., and Baltimore. Each dataset contains hourly urban flow of all regions, and use half of the day (12 hours) training sample. As for traffic speed prediction, we conduct experiments on four real-world datasets, including Meta- LA, PEMS-BAy, Didi-Chengdu, and Didi-Shenzhen. Their time intervals and 10min. We use 12 points as training sample. For both tasks. we categorized the datasets into source cities and one target city. For example, if one specific city is set as the target dataset, we assume access to only limited quantity of data, such as three-day data (existing models usually require several months of data train the model). The diffusion model is trained using the other source cities with rich data. This same division and training strategy is consistently applied when targeting different cities. Appendix B.I provides more details of the used datasets.
Smin, Smin, 10min,
Baselines and Metrics. To evaluate the performance of our proposed framework, we compare it against classic models and state-of fatte-of-thh-art the-art urban transfer approaches, including History Average (HA), ARIMA, RegionTrans (Wang et al., 2211)). DASTNet (Tang et al. (2022), AdaRNN (Du et al., 202T). MAML (Finn et al. 2017). TPB (Liu et al. 2023), and ST-GFSL (Lu et al. 2022). We use two widely-used regression metrics: Mean Absolute Error (MAE) and Root Mean Squared Error
(RMSE). Appendix B.2 and B.3 provide more details of baselines and implementation.
4.2 RESULTS
Performance Comparison. We compare the performance of our proposed GPD with baselines on two tasks: crowd flow prediction and traffic speed prediction. In both tasks, our goal is to transfer knowledge from the source cities the target city. Due to space limits, Table only reports the comparison results for the mean error of prediction over the subsequent 6 steps for some datasets. Comprehensive evaluations, including step-wise performance on all datasets, can be found in Ap- pendix C
Based on these results, we have these noteworthy observations:
Consistent Superiority. GPD consistently achieves the best performance compared with baseline approaches. When compared against the best- pest-performing baselines (indicated by underlining in Table T). GPD achieves average error reduction of 4.31%, 17.1%. 2.1% and 8. 8.17% in terms of MAE for Washington D.C., Baltimore, LA, and Chengdu. These improvements indicate that GPD enables effective knowledge transfer regarding parameter generation.
Competitive Long- Term Prediction. GPD exhibits its competitiveness particularly in long-term prediction scenarios, as shown in Tables in Appendix C For example, when considering Baltimore as the target city (Appendix Table IT). compared with the state-of-the-art baseline STGFSL, GPD shows an improvement in MAE performance of up to 22.1% the 69h step, compared a 5.9% improvement at the ast step. This notable trend can be attributed to the inherent design of our framework, which facilitates the transfer of long-term temporal knowledge to the target city.
Performance across Cities. To investigate the influence of different source cities on our experi- mental outcomes, conducted series of experiments using various source cities for the target
7

---

Published as a conference paper at ICLR 2024
Washington D.C.
Baltimore
LA
Didi-Chengdu
Model
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
HA
21.520
47.122
15.082
26.768
3.257
6.547
3.142
4.535
ARIMA
19.164
42,480
12.902
23.758
7.184
10.91
5.381
7.087
RegionTrans
13.853
31.951
7.375
13.715
3.349
5.728
2.828
4.130
DASTNet
14.808
32.219
7.532
13.975
3.236
5.603
2.810
3,953
MAML
13.930
31.009
7.978
14.901
3.257
5,558
2.874
4.041
TPB 13.552
31.024
7,421
14.011
3.198
5.574
2.654
3,709
STGFSL
12.224
28.625
7.365
13.746
3.184
5.577
2.570
3.832
GPD (ours)
11.697
27.308
6.110
11.669
3.119
5.450
2.366
3.393
Reduction%
-4.31%
-4.62%
17.1%
-14.9%
-2.1%
-3.2%
-8.17%
-8.62%
crowd flow datasets (Washington D.C. and Baltimore) and two traffic speed datasets (LA and Didi-Chengdu) terms of MAE and RMSE, We the average prediction errors six steps the result. Bold denotes the best results and underline denotes the second-best results.
Table 1: Performance comparison of few-shot scenarios
MAE
Target city
Source city
Step
Step 2
Step
Step
Step 5
Step 6
B
29.039
28.226
29.358
28.146
30.026
31.268
Washington D.C.
N
27.569
28.631
27.285
28.229
27.858
29.667
B+N
26.679
26.624
26.27
26.262
26.639
26.112
W
15.492
15.676
15.575
17.213
17.915
17.338
Baltimore
N
17.234
17.737
17.395
17.325
18.734
18.179
W+N
15.259
14.219
14.309
15.334
14.68
14.876
W
51.344
54,478
50.078
49.87
48.612
52.143
NYC
B 53.145
51.088
48.292
54.658
62.494
60.525
W+B
42.337
42.392
42.417
42.304
42.312
42.28
Table Performace comparison with different source cities for crowd flow predictions. "W", "B", and "N" represent Washington D.C., Baltimore, and NYC, respectively.
city. For instance, when considering Washington D.C. as the target city, we analyzed three distinct source city scenarios: (1) using New York City (N) the source city, (2) utilizing Baltimore (B) as the source city, and (3) incorporating both NYC and Baltimore (N+B) as source cities. The results summarized in Table indicaee benefits for both short-term and long-term predictions across all three target cities. This suggests that our framework effectively learned useful and transferable knowledge from these source cities. In one-step predictions, the inclusion of two source cities, as opposed to just one, results in perfor- mance improvements: 8.1 and 3.2% for Washington, 1.5% and 11.4% for Baltimore, and 17.5% and 20.3% for NYC. When extending our analysis to six-step predictions, the advantages of utiliz- ing two source cities over one are even more pronounced, with performance improvements of 16.4% and 12.0% for Washington, 14.2% and 18. 18.1% for Baltimore, and 18 18.9% and 30.1% for NYC. These results emphasize the significance of leveraging data from multiple source cities for the pre-training.
indicate that incorporating data from multiple source cities offers substantial
4.3 FLEXIBILITY OF GPD
Our framework is designed to be model-agnostic, providing support for cutting-edge spatio-temporal models. To validate the versatility of our framework, we also integrated another two advanced spatio-temporal prediction models, Graph WaveNet (GWN) (Wu et al. (009)) and Spatial-Temponee Identity (STID) (Shao al. 2022a), into our framework. Specifically, STGCN and GWN are graph- based architectures, while STID does not leverage graph neural networks.
Figure in Appendix presents a comprehensive performance comparison across the three spatio- temporal models-STGCN GWN, and STID across four datasets encompassing both crowd flow prediction and traffic speed prediction. The results reveal that STGCN and GWN exhibit nearly com- parable performance, with GWN slightly outperforming. STID, given its relatively simpler design
8

---

GPD introduces a pioneering generative pre-training frame- Published as a conference paper at ICLR 2024
Ind-Bias
Pre-Adap
Pre-Cond
Adap-Norm
Post-Adap
35
80
60
100
70
40
AMSE
MAE
30-
80
25
50
30
60
20
40 10
MAE
IMSE
20
40
RMSE
Washington
Balámore
Figure 3: Performance comparison across different conditioning strategies.
and potential limitations in modeling complex spatio-temport relationships, lags slightly behind STGCN and GWN. Therefore, we implement GWN as the base model for baselines to examine the effectiveness of our framework. Appendix C. provides the detailed comparison results. This comparison underscores our framework's ' adaptability to integrate existing spatio-tempordl models and its inherent potential to enhance prediction performance, particularly as more powerful models become available.
4.4 IN-DEPTH STUDY OF GPD
Conditioning Strategy. We investigate how the conditioning strategies affect the model's perfor- mance. Figure3 the comparison results. As we can observe, inducing inductive bias to the conditioning strategy consistently leads to lower prediction errors compared to other conditioning approaches across various datasets. The second-best performing strategy is the "Adaptive Sum of Prompts", which employs an attention mechanism to flexibly aggregate prompts related to different aspects for each token. This approach exhibits promise in addressing diverse apatio-temporal models in general and flexible manner. Conversely, the conditioning strategy without self-attention opera- tion demonstrates the poorest performance. This outcome indicates the importance of incorporating the prompt within the Transformer's 's self-attention layers for the generative pre-training process.
Prompts. We conduct experiments to investigate the influ- ence of prompt selection on the final performance. Figure illustrates the comparison results of three prompt methods: the spatio-temporal prompt (ours), the spatial prompt (w/o temporal), and the temporal prompt (w/o spatial). As can observe, eliminating either the spatial or temporal prompt leads reduction in the model's s performance, while em- ploying both the spatio-temporal prompt yields the most fa- vorable performance. These findings underscore the signifi- cance of harnessing the characteristics of the target city from both spatial and temporal perspectives in the context of spatio- temporal prediction.
OUFS
Fooa spatoal
Mro tempora)
40
90
80
700
MAE
60
25
50
20
40
MAE
RMSE
Wachington D.C.
Figure 4: Performance comparison of different prompts with Washing- ton D.C. as the target city.
CONCLUSION
work for spatio-temporal few-shot learning, addressing the challenges posed by data scarcity and heterogeneity in smart city applications. By conducting pre-training in the parameter space, GPD addresses the inherent disparities present in the data space across different cities and enables new but effective knowledge transfer. Its model-agnostic nature ensures compatibility with existing ur- ban computing models, making it valuable tool for researchers and practitioners the field. Our framework represents significant advancement in urban transfer learning, which has the potential to revolutionize smart city applications in data-scarce environments and contribute to more sustainable and efficient urban development. As for future work, researchers explore more sophisticated methods for prompt selection, such as leveraging large language models to capture the unique char- acteristics of cities.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under U23B2030, U22B2057 and U20B2060, and the National Key Research and Development Program of China under grant 2020YFA0711400 This work is also supported by a grant from the Guoqiang Institute, Tsinghua University under 2021GQG1005.
REFERENCES
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inver- sion with hypernetworks for real image editing. In Proceedings of the IEEE/CVE conference on computer Vision and pattern recognition, pp. 18511-18521, 2022.
Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. Advances in neural information processing systems, 33:17804- 17815, 2020.
Ivana Balažević, Carl Allen, and Timothy M Hospedales. Tucker: Tensor factorization for knowl- edge graph completion. arXiv preprint 0et:::9911099900 2019.
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33: 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-877 8794,, 2021.
Jingtao Ding, Guanghui Yu, Yong Li, Depeng Jin, and Hui Gao. Learning from hometown and current city: Cross-city poi recommendation via interest drift and transfer learning. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(4):1-28, 2019.
Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. Adamnn: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM inter- national conference on information & knowledge management, pp. 402-411, 2021.
Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generat- ing implicit neural fields with weight-space diffusion. arXiv preprint urXiv:2303. /7015, 2023.
Ziquan Fang, Dongen Wu, Lu Pan, et al. When transfer learning meets cross-city urban flow predic- tion: spatio-temporal adaptation matters. IJCAI'22, pp. 2030-2034 2022.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint
000077, 2020.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11)) (:(11))1139-) 1444 2020.
Bin Guo, Jing Li, Vincent W Zheng, Zhu Wang, and Zhiwen Yu. Citytransfer: Transferring inter- and intra-city knowledge for chain store site recommendation based multi-source urban data. Proceedings of the ACM on Interactive. Mobile, Wearable and Ubiquitous Technologies, 1(4); 1-23, 2018
10

---

Published as a conference paper at ICLR 2024
David Ha, Andrew M Dai, and Quoe V Le. "hypemetworks," in 5th international conference learning representations, iclr 2017. In Conference Track Proceedings, OpenReview. net, pp. 24- 26, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVT conference on computer vision and paltern recognition, pp. 16000-16009, 2022.
Tianfu He, Jie Bao, Ruiyuan Li, Sijie Ruan, Yanhua Li, Li Song, Hui He, and Yu Zheng. What is the human mobility in a new city: Transfer mobility knowledge across cities. In Proceedings of The Web Conference 2020. pp.
2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-66 6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300: 2022.
Yilun Jin, Kai Chen, and Qiang Yang. Selective cross-city transfer learning for traffic prediction via source city region re- re-weighting. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 731-741, 2022.
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion- Im improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328-434 2022.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusson convolutional recurrent neural net- work: Data-driven traffic forecasting. In International Conference on Learning Representations, 2018.
Zhonghang Li, Lianghao Xia, Yong Xu, and Chao Huang. Gpt-st: Generative pre-training of spatio- temporal graph neural networks. arXiv preprint arXiv:231 1.04245, 2023.
Yan Liu, Bin Guo, Daqing Zhang, Djamal Zeghlache, Jingmin Chen, Ke Hu, Sizhe Zhang, Dan Zhou, and Zhiwen Yu. Knowledge transfer with weighted adversarial network for cold-start store site recommendation. ACM Transactions on Knowledge Discovery from Data (TKDD), 15(3): 1-27, 2021.
Zhanyu Liu, Guanjie Zheng, and Yanwei Yu. Cross-city few-shot traffic forecasting via traffic pattern bank. arXiv preprint arXiv:2308.0m 2977777700 2023.
Zhaoyang Liu, Yanyan Shen, and Yanmin Zhu. Inferring dockless shared bike distribution in new cities. In Proceedings of the eleventh ACM international conference on web search and data mining, pp. 578-386,7 2018
Bin Lu, Xiaoying Gan, Weinan Zhang, Huaxiu Yao, Luoyi Fu, and Xinbing Wang. Spatio-temporal graph few-shot learning with cross-city knowledge transfer. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1162-1172. 2022.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nilion, pp. 10209-10218, 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, pp. 16784-16804. PMLR, 2022.
Zheyi Pan, Yuxuan Liang, Junbo Zhang, Xiuwen Yi, Yong Yu, and Yu Zheng. Hyperst-net: Hyper- networks for spatio-temporal forecasting. arXiv preprint arxiv: /809. 10889, 2018.
11

---

Published as a conference paper at ICLR 2024
Zheyi Pan, Yuxuan Liang. Weifeng Wang, Yong Yu, Yu Zheng, and Junbo Zhang. Urban traffic prediction from spatio-temporal data using deep meta learning. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 7720-1730 2019.
Zheyi Pan, Wentao Zhang, Yuxuan Liang, Weinan Zhang, Yong Yu, Junbo Zhang, and Yu Zheng. Spatio-temporal meta learning for urban traffic prediction. IEEE Transactions on Knowledge and 2020.
Data Engineering, 34(3):
Yanbo Pang. Kota Tsubouchi, Takahiro Yabe, and Yoshihide Sekimoto. Intercity simulation of human mobility at rare events via reinforcement learning. In Proceedings of 28th International Conference on Advances in Geographic Information Systems, pp. 293-302, 2020.
William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint arXiv:2209. 12892, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-36494 2022.
Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learn- ing on neural network weights for model characteristic prediction. Advances in Neural Informa- tion Processing Systems, 34: (14411111194,,,, 2021.
Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper- representations as generative models: Sampling unseen neural network weights. Advances in Neural Information Processing Systems, 35:27906-27999 2022.
Zezhi Shao, Zhao Zhang, Fei Wang, Wei Wei, and Yongjun Xu. Spatial-temporal identity: A simple yet effective baseline for multivariate time series forecasting. In Proceedings of the 31st ACM International Conference Information & Knowledge Management, pp. 4454 4458, 2022a.
Zezhi Shao, Zhao Zhang, Fei Wang, and Yongjun Xu. Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting. In KDD, pp. 1567-1577, 2022b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- lional Conference on Learning Representations, 2020.
Yihong Tang, Ao Qu, Andy HF Chow, William HK Lam, SC Wong, and Wei Ma. Domain adver- sarial spatial-temporal network: a transferable framework for short-term traffic forecasting across cities. In Proceedings of the 31 ACM International Conference Information & Knowledge Management, pp. 1905-1915, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Clément Vignac, Igor Krawezuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. In Proceedings of the 11th International Conference on Learning Representations, 2023.
Binwu Wang, Yudong Zhang, Xu Wang, Pengkun Wang, Zhengyang Zhou, Lei Bai, and Yang Wang. Pattern expansion and consolidation evolving graphs for continual traffic prediction. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2222,,232,22 2023a.
Leye Wang, Xu Geng, Xiaojuan Ma, Feng Liu, and Qiang Yang. Cross-city transfer learning for deep spatio-tene tempoall prediction. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 1893-1899, 2019.
12

---

Published as a conference paper at ICLR 2024
Xu Wang, Lianliang Chen, Hongbo Zhang, Pengkun Wang, Zhengyang Zhou, and Yang Wang. A multi-graph fusion based spatiotemporal dynamic learning framework. In Proceedings of the Sixteenth ACM International Conference Web Search and Data Mining, pp. 294-302, 2023b.
Xu Wang, Pengfei Gu, Pengkun Wang, Binwu Wang, Zhengyang Zhou, Lei Bai, and Yang Wang. Graph-free learning in graph-structured data: A more efficient and accurate spatiotemporal learn- ing perspective. arXiv preprint arXiv:2301.11742 2023c.
Ying Wei, Yu Zheng, and Qiang Yang. Transfer knowledge between cities. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp- 1905-1914, 2016.
Haomin Wen, Youfang Lin, Yutong Xia, Huaiyu Wan, Qingsong Wen, Roger Zimmermann, and Yuxuan Liang. Diffstg: Probabilistic spatio-temporal graph forecasting with denoising diffusion models. In Proceedings of the s1 ACM International Conference on Advances in Geographic Information Systems, pp. 1-12, 2023.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling, arXiv preprint arXiv:/906.00/21, 2019.
Yutong Xia, Yuxuan Liang, Haomin Wen, Xu Liu, Kun Wang. Zhengyang Zhou, and Roger Zim- mermann. Deciphering spatio-temporal graph forecasting: A causal lens and treatment. Advances in Neural Information Processing Systems, 36, 2024.
Huaxiu Yao, Yiding Liu, Ying Wei, Xianfeng Tang, and Zhenhui Li. Learning from multiple cities: A meta-learning approach for spatial-tempora prediction. In wwW, pp. 2181-2191, 2019.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. arXiv preprint arXiv: (709004770 2017.
Yuan Yuan, Jingtao Ding, Chenyang Shao, Depeng Jin, and Yong Li. Spatio-temporal diffusion point processes. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 3173-3184, 2023.
Zhengyang Zhou, Qihe Huang, Kuo Yang, Kun Wang, Xu Wang, and Yudong Zhang. Maintaining the status quo: Capturing invariant relations for ood spatiotemporal learning. 2023a.
Zhengyang Zhou, Jiahao Shi, Hongbo Zhang. Qiongyu Chen, Xu Wang. Hongyang Chen, and Yang Wang. Crest: A credible spatiotemporal learning framework for uncertainty-awant traffic fore- casting. In The 17th ACM International Conference on Web Search and Data Mining, pp. 1-10, 2024.
Zhilun Zhou, Jingtao Ding, Yu Liu, Depeng Jin, and Yong Li. Towards generative modeling of urban flow through knowledge-enhanee denoising diffusion. arXiv preprint arXiv:2309. 10547, 2023b.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1): 43-76, 2020.
13

---

Published as a conference paper at ICLR 2024
(a) Data-space Knowledge
(b) Parameter-space Knowledge
Source city
Target city
Source city
Target city
hypernetwork
Hypernetwork
learned parameter
Transfer the
Pre-train
Generate
Figure 5: Comparison of data-space and parameter-space knowledge across cities. Colorful points represent divided regions in the city.
A METHODOLOGY DETAILS
A.1 DIFFERENT KNOWLEDGE FOR TRANSFER
Figure SegureScooppnnt compares the difference between our framework and other urban transfer learning methods, which focus parameter-space knowledge and data-space knowledge, respectively.
A.2 REGION PROMPTS
We design region prompts to leverage auxiliary data that captures city characteristics. This prompt- ing technique facilitates to utilize external information more flexibly. In experiments, adopt a granular approach by crafting distinct prompts for each region within the city. These prompts are meticulously designed to encapsulate the unique characteristics of each region. To facilitate aceu- rate spatio-temporal predictions, we create two specific types of prompts: spatial prompt and temporal prompt. The spatial prompt is tailored to provide in-depth representation of the spa- tial attributes, encompassing geographical features, environmental conditions, and interconnections with neighboring regions. Complementing the spatial prompt, we also introduce temporal prompt. This prompt captures the temporal dynamics of each region.
Spatial Prompt. The spatial prompt is obtained by pre-training an urban knowledge graph (UKG) (Zhou et al. (003b)), which is meticulously designed to encapsulate the extensive environ- mental information within a city. Specifically, we represent urban regions distinct entities within the UKG framework. We use relations "BorderBy" and "NearBy" to capture the spatial adjacency among regions. By leveraging this adjacency representation, we aim to capture the influence that proximate regions exert functional similarity between these urban regions. This insight is quantified by computing the cosine similarity of the distribution of Points of Interest (POI) categories between region pairs. We establish a "SimilarFune" relation to establish connections between regions that exhibit functional similarity, which emphasizes the critical role played by shared functions in shaping the urban landscape.
another. Furthermore, UKG incorporates an understanding of the
Relation
Head & Tail Entity Types
Semantic Information
BorderBy
(Region, Region)
Regions share part of the boundary Regions within certain distance
NearBy
(Region, Region) (Region, Region)
SimilarFunc
Regions have similar functions
Table 3: The details of relations in the urban knowledge graph.
To extract the spatial prompts from the constructed Urban Knowledge Graph (UKG), we employ state-of-the-ar KG embedding model, TuckER (Balažević et al. 2019), to learn embedding representation for each region. TuckER evaluates the plausibility of triplets as follows:
@h,r, W X1 Ch X2 er X3 Ct.
(3)
14

---

Published as a conference paper at ICLR 2024
where W Reke is learnable tensor, X, denotes the tensor product along the nu dimension, and Cbe Cr, C₁ E KigG are the embeddings of head entity h, tail entity t and relation respectively. The primary objective of the KG embedding model is maximize the scoring function for triplets that exist the UKG, thereby preserving the knowledge contained within the UKG.
In summary, UKG leverages the 'BorderBy' and 'NearBy' relationships articulate spatial connections and the 'SimilarFunc' relationship to underscore functional parallels between urban regions. The benefits of UKG twofold. First, it integrates various relationships within the city, allowing the learned embeddings of regions to provide descriptive information about their respective urban environments. Secondly, in contrast time series data collected by sensors or GPS devices, the features utilized in the Urban Knowledge Graph (UKG) are readily available in all urban ar- eas. This accessibility makes the UKG scalable and adaptable to cities, even those with limited development levels. The details of relations in UKG are shown in Table
The temporal prompt is derived through a strategic application of an unsuper- vised pre-training model designed for time series, as introduced by Shao et al (Shao al. 2022b). This approach shares similarities with the concept of a Masked AutoEncoder (MAE) (He et al. 2022) for sequence data. Initially, the time series data for each region is subjected to masking procedure, where random patches within the time series are concealed. Subsequently, an encoder- decoder model is trained on this modified data to reconstruct the original time series. This training process is centered around the objective of reconstructing the complete time series based solely on the partially observable series patches. Given that time series data often exhibits lower information density, a relatively high masking ratio (75%) is employed. This higher masking ratio is crucial for creating self-supervised learning challenge that encourages the model to capture meaningful temporal patterns from incomplete observations. Upon successful completion of the self-supervised leamning phase for time series data, the output of the encoder yields the temporal embeddings. In essence, this method capitalizes on self-supervised learning to extract valuable temporal features from time series data, which can subsequently be used as temporal prompts.
Temporal Prompt.
A.3 CONDITIONING STRATEGIES
Pre-conditioning. "Pre" denotes that the prompt is integrated into the token sequence before being fed into self-attention layers. In this method, we simply add the region prompt p the token embeddings within the input sequence.
Pre-conditioning with inductive bias. In this variant, adopt different approach add the gion prompt to the token embeddings within the input sequence. The spatial prompt is incorporated into spatial-related parameters uniformly and the temporal prompt into temporal-related parameters uniformly as follows:
11,1, ".,1,'1 [.mm] = [x,,1 #x,1 [mmm + [p. Psi Ps
(4)
(01,1, [.,n] = [1 (01,1, [r +
Pe: Pc:
Pe
where and represent the number of tokens for spatial parameters and temporal parameters, P. denotes the spatial prompt, and Pt denotes the temporal prompt.
Pre-adaptive conditioning. In this variant, we introduce attention mechanism, which determines to what extent the prompt should be added to specific token embeddings. We denote the prompt as p 2 where E is the embedding size of spatial and temporal prompts. This approach aims to empower the model to learn how to adaptively utilize the prompts, enhancing its conditioning capabilities. The utilization of the prompt can be formulated follows:
15

---

Published as a conference paper at ICLR 2024
Prompt
Prompt
Pointwise Feedforward
Temporal Prompt
Spatial Prompt
Pointwise Feedforward
Temporal Prompt
Spatial
Prompt
Layer norm
Query
Attention
Layer norm
Attention
Adaptive Emb
Adaptive Emb
Multi-head Self-Attention
Multi-head Self-Attention
Re-Scale
Re-Scale
Layer norm
Layer norm
Noised tokens
Step Emb
Noised tokens
Step Emb
Post-Adaptive Conditioning Strategy
(b) Adaptive Norm Conditioning Strategy
Figure 6: Illustration of two conditioning strategies: (a) "Post-Adaptive Conditioning" and (b) "Adaptive Norm Conditioning".
U = lanh(WwP) + bw),j E {0,1}
(5)
exp(u? xz)
0g En exp(u xz)
P. Σ atyPy:J {0,1}
where Po and Pı represent the spatial prompt and temporal prompt, respectively, P; denotes the aggregated prompt from two aspects for the deh. token.
Post-adaptive Conditioning. Figure (a) illustrates this conditioning strategy. The aggregated prompt based on the attention mechanism is added after the multi-head self-attention in each trans- former layer. Specifically, the query used for spatio-temporal attentive aggregation is the output of the multi-head self-attention layer.
Adaptive norm conditioning. Figure 6 (b) illustrates this conditioning strategy. The aggregated
prompt based on the attention mechanism is used for re-scaling the output in each layer norm.
A.4 NETWORK LAYERS OF SPATIO-TEMPORAL PREDICTION MODELS
We provide details of parameter tokenizers introduced in Section 3.5 In our experiments, we implement framework three spatio-temporal prediction models, STGCN (Yu et al. (017). GWN (Wu et al. 2019), and STID (Shao et al.) 2022a). We present how to transform their network layers into a vector-based token sequence. According Table 4 and Table the transformation from parameter layers to token sequence can be formulated as follows:
ged = GC (CDDmmllllss) numellss)).......... numel((m))
L C; reumel(s:)
(8)
L = Σ Li
where GCD denotes the calculation of the Greatest Common Divisor, m denotes the number of different layer types, 8; denotes the shape, numel denotes the total number of elements in the tensor
16

---

Published as a conference paper at ICLR 2024
Layer name
Parameter shape
Count
Boocklkkkillookk
[32,2,1,4]
3
Block1
Block1 Brrckkl-rrpCConyy Thetal
[32,8]
Hlockl-TimeBllock--eo
[32,8.1,4]
3
TimeBlock
last TimeBlock-conv
[32,32,1.4]
3
Linear
Fully Connected Layer
[6,96]
Table 4: Parameter structure of STGCN Model
Layer name
Parameter shape
Count
filter.conv..x
32, 1
8
convs.X
33, 1,2
8
residual cosvv.al.conv..x
[32, 1,
8
skip_convs.x
32,32, 1,
bn.x
[32] + [32]
8
gconv.x.mlp.mlp
[60, 1,1]
8
start conv
[32, 1,
conv1
[32, 02,32,14 1,
1
end_conv2
66,32, 1,1
1
Table 5: Parameter structure of a GWN Model
Layer name
Parameter shape
Count
Time_series_enil
conv
[32,24,1,1|+[3]
LayerI-fe
[32,32,1,1]
2
Encoderl.ayen Layers
Layer2-fe
(32,32,1,11+{322
2
Layer3-fe
[32.32.1,1]+(32]
2
Regression_layer
conv
[6,32,1,1]
1
Table 6: Parameter structure of a STID Model
denotes the count of this type of layer. In this way, we obtain a token sequence with length as L and embedding size as ged.
STGCN (Yu et al. 2017). Spatio-temporal graph convolution network. Different from regular convolutional and recurrent units, this model build convolutional structures on graphs. We use 3-layer STGCN block, and utilize a I-layer MLP as the output predictor. shows the detailed network layers of STGCN.
GWN (Wu et al. 2019). Graph WaveNet. This model developed a novel adaptive dependency matrix and learned through node embeddings to capture the spatial dependency. It also combines with stacked dilated causal convolution component. We use 2-layer 4-block GWN model. Table shows the detailed network layers of a GWN.
STID (Shao et al., 2022a).
Identity. This model is simple Multi-Layer Percep- trons (Maes)- approach. By identifying the indistinguishability samples in both spatial and temporal dimensions, it is simple yet effective for spatio-temporal prediction.
A.5 ALGORITHMS
We present the training algorithm for spatio-temporal graph prediction in Algorithm Besides, present the training algorithm and sampling algorithm for the diffusion model Algorithm 2 and Algorithm respectively.
B EXPERIMENT DETAILS
B.1 DATASETS
In this section, we introduce the details of the used real-world datasets.
17

---

Published as a conference paper at ICLR 2024
Algorithm Model Parameter Preparation
1: Input: Dataset D {D1, D2...., DM, neural networks F (So, fez ................................ so } of the
spatio-temporal prediction model, loss function L, parameter storage S.
2: Output: Parameter storage S.
3: Initialize: Learnable parameters 6m for fms parameter storage = {
4: for m .1, M.) do
5: for epoch E (1.2... Niter) do
6: Sample mini-batch of inputs and labels from the dataset Dm (2. Dm
7:
Compute the predictions ÿ + fem (x)
8:
Compute the loss L L(ÿ.
9: Update the model's parameters 6m t update C; 6m) end for
11: Save the optimized model S S {0m)
12: end for
Algorithm Generative Pre-training of the Diffusion Model
1: Input: Parameter storage S, diffusion model G.
2: Initialize: Learnable parameters 7 for G
3: for epoch {1, 2, NNter do
4: Sample a parameter sample 0g ~ q(0b) and ~~ NO0
5: Sample diffusion step Uniform(1, (nif,
6: Take gradient descent step on
V en (Vooio + v1 -.xe,, k)
7: end for
Algorithm Parameter Sampling
1: Input: Gaussian noise 0x ~ (O, I), prompts P = {p1, ... M}
2: Output: Parameters 0, = [0,,, 02,0 00.
3: Initialize: Learnable parameters for G. 0. = (..
4: for m {1,2, ......... Mc} do
5: for k K + do
6:
K > then (((0T)
7:
8: else
9:
==0 end if
10:
1
Bk (mm. + BBzz
(8m,k
Vap
V1 a
12:
end for
13:
0, = 0, U 0m.
14: end for
18

---

Published as a conference paper at ICLR 2024
Datasets #Nodes
New York City
Washington, D.C.
Baltimore
195
194
267
#Edges
555
504
644
Interval
hour
hour
hour
2016.01.01.
2019.01.01-
2019.01.00 1010
Time span
2016.06.30
2019.05.31
2019.05.31
Mean
70.066
30.871
18.763
Std 71.852
58.953
28.727
Table 7: The basic information and statistics of four real-world datasets for crowd flow prediction.
Datasets
METR-LA
PEMS-BAY
Didi-Chengdu
Didi-Shenzhen
#Nodes
207
325
524
627
#Edges
1722
2694
1120
4845
Interval
5 min
5 min
10 min
10 min
2012.05.01-
2017.01.1-
2018.1.1-
2018.1.1-
Time span
2012.06.30
2017.06.30
2018.4.30
2018.4.30
Mean
61.776 58.274
31.001 29.023 9.662
Std 13.128
9.285
10.969
Table 8: The basic information and statistics of four real-world datasets for traffic speed prediction.
B.1.1 CROWD FLOW PREDICTION.
NYC Dataset. this dataset, we define regions as census tracts and aggregate taxi trips from NYC Open Data to derive hourly inflow and outflow information. The division of train and test regions based on community districts. Specifically, the Manhattan borough comprises munity districts, and we designate 9 of them train regions, reserving the remaining 3 for testing. Region-specific features encompass the count of Points Interest (POIs), area, and population.
com-
Washington, D.C. Dataset. Regions in this dataset are defined as census tracts, and inflow data is calculated based on Point of Interest (POI)-level hourly visits. The partitioning of train and test regions is done by counties. Specifically. regions within the District of Columbia are selected as train regions, and regions Arlington County are designated as test regions. This dataset comprises rich region features, including demographies and socioeconomic indicators.
Baltimore Dataset. Similar to the D.C. dataset, regions, and inflow data in the Baltimore dataset are obtained in the same manner. Train regions consist of regions in Baltimore City, while test regions encompass Baltimore County. This dataset includes the same set of features as the D.C. dataset.
B. 1.2 TRAFFIC SPEED PREDICTION.
We conducted performance evaluation using four real-world traffic speed datasets, following the data preprocessing procedures established in prior literature (Li et al. 2018 Lu et al.) 2022). To construct the spatio-temporal graph, we treated each traffic sensor or road segment as an indi- vidual vertex within the graph. We then computed pairwise road network distances between these sensors. Finally, we construct the adjacency matrix of the nodes using road network distances and thresholded Gaussian kemel.
METR-LA (Li et al., 2018; Lu et al. 2022). The traffic data in our study were obtained from observation sensors situated along the highways of Los Angeles County. We utilized total of 207 sensors, and the dataset covered span of four months, ranging from March 1, 2012, to June 30, 2012. To facilitate analysis, the sensor readings were aggregated into 5 5-minute intervals.
PEMS-BAY Li et al. 2018: Lu et al., 2022). The PEMS BAY dataset comprises traffic data collected over period of months, from January 1st. 2017. to June 30th, 2017, within the Bay Area. The dataset is composed of records from 325 traffic sensors strategically positioned throughout the region.
19

---

Published as a conference paper at ICLR 2024
Didi-Chengdu (Lu et al. 2022). We utilized the Traffic Index dataset for Chengdu, China, which was generously provided by the Didi Chuxing GAIA Initiative. Our dataset selection encompassed the period from January April 2018 and covered 524 roads situated within the central urban area of Chengdu. The data was collected at 10-minute intervals to facilitate our analysis.
Didi-Shenzhen al. 2022). We utilized the Traffic Index dataset for Shenzhen, China, which was generously provided by the Didi Chuxing GAIA Initiative. Our dataset selection included data from January April 2018 and encompassed 627 roads located in the downtown area of Shenzhen. The data collection was conducted at 10-minute intervals to facilitate our analysis.
B.2 BASELINES
HA. Historical average approach models time series as a seasonal process and leverages the av- erage of previous seasons for predictions. In this method, we utilize a limited set of target city data to compute the daily average value for each node. This historical average then serves as the baseline for predicting future values.
ARIMA. Auto-regressive Integrated Moving Average model a widely recognized method for comprehending and forecasting future values within time series.
Region Trans (Wagg et al. 2019). Region Trans assesses the similarity between source and target nodes, employing as a means to regulate the fine-tuning of the target. We use STGCN and GWN as its base model.
Network, which undergoes pre-training data from multiple source networks and then proceeds to fine-tune using the data specific to the target network's 's traffic.
DASTNet (Tang et al. 2022). Domain Adversarial
AdaRNN (Du et al. 202T). This cutting-edge transfer learning framework is designed for stationary time series data. The primary objective of this model is to mitigate the distribution disparity within time series data, enabling the training of an adaptive model based on recurrent neural networks (RNNs).
MAML Finn et al. 2017). Model-Agnostic Meta Learning is an advanced meta-learning tech- nique designed to train model's parameters in way that minimal number of gradient updates result rapid learning on a novel task. MAML achieves this by acquiring an improved initial- ization model through the utilization of multiple tasks guide the learning process of the target task.
TPB (Liu al. 2023). Traffic Pattern Bank-based approach. TPB employs a pre-trained traffic patch encoder to transform raw traffic data from cities with rich data into a high-dimensional space. In this space, traffic pattern bank is established through clustering. Subsequently, the traffic data originating from cities with limited data availability can access and interact with the traffic pattern bank to establish explicit relationships between them.
ST-GFSL (Lu et al., 2022). ST-GFSL generates node-specific parameters based node-level meta- wan--nnowledgge drawn from the graph-based traffic data. This approach ensures that parameters are tailored individual nodes, promoting parameter similarity among nodes that exhibit similar- ity in the traffic data.
B.3 IMPLEMENTATION DETAILS
In the experiments, we the number of diffusion steps N=500. The learning rate is 8e-5 and the number of training epochs ranges from 3000 to 12000. The dimensions of KG embedding and time embedding are both 128. Regarding the spatio-temporal prediction, we use 12 historical time steps to predict 6 future time steps. Our framework can be effectively trained within 3 hours and all experiments were completed on one NVIDIA GeForce RTX 4090.
C ADDITIONAL RESULTS
C.I FLEXIBILITY OF GPD
To demonstrate the flexibility of our framework, we compare the performance with GWN as the base model. Figure to Figure illustrate the comparison results. As we observe, frame-
20

---

Published as a conference paper at ICLR 2024
work still achieves the best performance all datasets. The advantage of our framework is more significant for long-step predictions.
20
STGCN
GWN
15
STID
MAE
10
Baltimore
NYC
Chengdu
LA
Figure 7: Performance comparison of different spatio-temporal prediction models on crowd flow prediction task (Baltimore and NYC) and traffic speed prediction task (Chengdu and LA).
Washington D.C.
18
Ours
DASTNet
16
STGFSL
1:11::
RegionTrans
TPB
14-
MAE
10
Prediction Step
Figure 8: Performance comparison with GWN the base model Washington D.C. dataset.
C.2 TIME CONSUMPTION ANALYSIS
Model Computational cost
RrgionTrans
DASTNet 1h
MAML
TPB
STGESL
GPD
~ 1.5mmn
2h
1h
2h
3h
Table 9: Training time consumption of our framework and baseline solutions.
Table present a detailed comparison of the computational cost of our proposed model against baselines. Our model demonstrates efficient training, completing within 3 hours on single GPU (0TX-2080Ti), which we believe is an acceptable time consumption. It is important to note that the slightly longer time consumption in our model is attributed to the step-by-step denoising process implemented in DDPM approach (Ho et al.) 2020).
We choose DDPM because it is classic, simple but effective diffusion framework. Notably, our framework is designed to be compatible with more efficient diffusion models, such as DDIM (Song al. 2020). Employing these models has the potential to significantly reduce the current computa- tional cost. We want to underscore that our framework has the ability to achieve remarkably better performance with relatively lower overhead.
21

---

Published as a conference paper at ICLR 2024
Baltimore
9
Ours
M000
DASTNet
8
STGFSL
RegionTrans
TPB
7
MAE
6
5
4
3
Prediction Step
Figure 9: Performance comparison with GWN the base model Baltimore dataset.
NYC
Ours
DASTNet
STGFSL
RegionTrans
25
TPB
MAE
20
10
5
3
6
Prediction Step
Figure 10: Performance comparison with GWN as the base model on NYC dataset.
C.3 FINE-GRAINED PERFORMANCE ANALYSIS
conducted fine- fineggraieed evaluations to discern the types of regions in target cities that benefit more from knowledge transfer in our pretraining framework. For this analysis, selected Balti- more as the target city, while Washington D.C. and NYC served as source cities. To initiate our investigation, analyzed the performance differences across various regions within Baltimore. The distribution of performance across these regions is visualized in Figure 11 This observation of varied performance across different regions prompted performance variance, aiming to gain insights into more effective knowledge transfer strategies.
explore the factors contributing this
In our analysis, employed clustering the time series data from the source cities. Figure 12 pro- vides a visual representation the time series data through Principal Component Analysis (PCA). Simultaneously, we conducted an in-depth analysis focusing on regions within the target city. Specif- ically, we selected two regions with relatively superior performance (MAE 2.53 and 2.33) and two regions with comparatively lower performance (MAE 30.2, 31.8). As illustrated Figure 12 regions with better performance (depicted in red as the "good case") seamlessly align with one of the clusters identified in the source cities. contrast, regions with lower performance (depicted in blue as the "bad case") are situated farther from the clustering centers, struggling to align with any specific clusters. This insightful observation aligns with the intuitive notion that regions sharing similar patterns more receptive to knowledge transfer within pretraining framework. Con- sequently, leveraging more diverse cities as source cities for pretraining the generative framework promises improve overall performance. This insight is consistent with the principles observed
22

---

Published as a conference paper at ICLR 2024
in Large Language Models, which often exhibit enhanced performance when trained on highly diverse dataset (Gao al. 2020).
Baltimore
0.15
Density
0.10
0.05
0.00
0
10
20 MAE
30
40
Figure 11: Performance distribution of different regions in Baltimore.
Good case
200
Bad case
150
100
50
-50
-100
100
200
Figure 12: Clustering visualization of the source city data and the case study of the target city.
C.4 SYNTHETIC DATA EXPERIMENTS
The effectiveness of our approach is grounded in two key hypotheses:
Assumption 1. Regions with distinct spatio-temporal patterns correspond to different optimized model parameters. This relationship be mathematically expressed as follows:
Pf P if D(X,, X(),,
where P represents the optimized model parameters for region X, denotes the apatio-temporal pattern of the region i, D signifies the dissimilarity metric between the patterns of two regions.
Assumption 2. There exists mapping relation between the conditions from euch region C2 and the optimized model parameters P.
P. = foC..) where
= argmin Σ Distance(P: fo(x))
(10)
23

---

Published as a conference paper at ICLR 2024
2.0
450
1.5
400
1.0
0.5
350
0.0
-0.5
250
-1.0
200
-15
100
200
300
400
-2.0
Timestep
(a) Parameters of Region A
(b) Time Series of Region A
2.0
1.5
1.0
400
0.5
vale
0.0
300
-05
250
-1.0
-15
0
100
200
300
400
2.0
Timestep
(c) Parameters of Region B
(d) Time Series of Region
2.0
1.5
1800
1.0
0.5
Value
0.0
1400
-0.5
1200
11.0
-1.5
1000
100
200
300
400
-2.0
Timestep
(e) Parameters of Region C
(f) Time Series of Region C
Figure 13: Visualization of parameters and time series for three regions in the source city.
where P; is the generated model parameters.
To empirically validate these hypotheses, we conducted experiments on two synthetic datasets, al- lowing us to manipulate pattern similarities between nodes. This design enables an investigation into the framework's ability to generate effective parameters. The experimental setup involves two cities: designated as the source city, and the other the target city, each comprising 200 regions. We first construct a graph by randomly adding edges between regions. Then, for each region, we generated time series following specific patterns while incorporating random noise. This approach ensured that regions have distinct npatio-temporal patterns.
In Figure 13 present the parameter maps of well-trained models for three regions in the source city. Notably, regions (Figure [3(b) and B (Figure [l] exhibit highly similar time series patterns, whereas region C (Figure 13) displays distinctly different patterns. In the meantime. the
24

---

Published as a conference paper at ICLR 2024
2.0
2.0
1.5
1.5
1.0
1.0
0.5
0.5
0.0
0.0
-0.5
-0.5
520m
-1.
-1.0
-1.5
-2.
2.0
(a) Ground-truth Parameters of Region M
(b) Generated Parameters of Region M
2.0
2.0
1.5
1.5
1.0
1.0
0.5
0.5
0.0
0.5
-05
-1.0
-1.
wnw.aaatiiininn auttoua wam Vi
-2.0
-2.0
(d) Ground-truth Parameters of Region N (d) Generated Parameters of Region
Figure 14: Visualization of ground-truth parameters and generated parameters for two regions in the target city.
symmetrical positions of nodes A and B in the graph (depicted in Figure 15) suggest that regions A and B have very similar spatial patterns. Therefore, we can assume that regions and B have very similar spatio-temporal patterns. Consequently, the parameter maps for regions A and B showcase similar distributions, diverging from the parameter map of region C. This observation underscores the necessity of employing non-shared parameters for regions to effectively adapt diverse spatio- temporal patterns.
Moving on to the parameter generation results for the target city, selected two regions labeled as and N, characterized by similar time series patterns and spatial connection locations (see Fig- ure T6p. Figure [4] illustrates the comparison results. Specifically, Figures 14 a) and [4] c) depict the ground truth parameter maps for these two regions, revealing similar distribution patterns. Fig- ures b)) and [4[d) illustrate the generated parameter maps for the generated parameter maps closely align with their corresponding ground-truth distributions. This comprehensive analysis of parameter generation, presented in Figures [3] Figures [4] Figure [15] and Figure [6] collectively validates the capability of our framework in effectively generating pa- rameters that capture diverse spatio-temporal patterns across different regions.
regions. Notably, the
C.5 PREDICTION RESULTS OF THE REMAINING CITIES
Table 10 to Table show the prediction results of other datasets.
25

---

Published as a conference paper at ICLR 2024
A
B
Figure 15: The sub-graph of the source city, which contains nodes A, B, and C. The graph structure denotes spatial connection relationships.
M
N
Figure 16: The sub-graph of the target, which contains nodes M and N.
26

---

Published as a conference paper at ICLR 2024
MAE
RMSE
Model
Step 1
Step
Step
Step 1
Step 3
Step
HA
21.520
21.520
21.520
47.122
47.122
47.122
ARIMA
19.703
15,063
27.083
37.809
35.093
61.675
RegionTrans
12.116
13.538
15.501
27.622
30.999
36.095
DASTNet
11.501
14.255
17.676
25.551
31.466
38.004
MAML
10.831
13.634
16.192
24.455
30.740
36,271
TPB
9.153
11,870
15.236
23,420
28.756
33,424
STGFSL
9.636
12.178
14.116
22.362
28.287
33.547
GPD
10.339
11.454
13.144
23.348
26.798
30.959
Table 10: Performance comparison of few-shot scenarios Washington D.C. dataset different prediction steps in terms of MAE and RMSE. Bold denotes the best results and underline denotes the second-best results.
MAE
RMSE
Model
Step 1
Step
Step 6
Step 1
Step 3
Step
HA
15.082
15.082
15.082
26.768
26.768
26.768
ARIMA
11.150
12.344
18.557
19.627
21.665
35.520
RegionTrans
6.782
7.371
7.895
12.454
13.778
14.648
DASTNet
6.454
7.461
8.424
12.304
13.960
15.225
MAML
6.765
8.170
8.834
13.227
14.470
16.953
TPB
6.014
7.322
8.571
9.832
14.512
16.308
STGESL
5.925
7.244
8.356
11.157
13.450
15.444
GPD
5.570
5.971
6.582
10.165
11.344
13.003
Table 11: Performance comparison of few-shot scenarios on Baltimore dataset at different prediction steps in terms MAE and RMSE. Bold denotes the best results and underline denotes the second- best results.
MAE
RMSE
Model
Step
Step
Step
Step 1
Step
Step
HA
34.705
34.705
34.705
52.461
52.461
52.461
ARIMA
27.865
27.695
33.771
40.643
45.003
59.359
Region Trans
16.138
19.318
22.103
26.248
32.489
37.654
DASTNet
16.480
22.464
27.147
25.788
36.717
43.834
MAMI.
14.083
19.753
24.493
23.473
33.079
40.407
TPB
16.616
21.835
28.005
20.50
28.386
37.701
STGESL
13.479
18.654
23.054
21.918
31.106
37.818
GPD
13.580
16.076
18.923
22.081
27.329
32.260
Table 12: Performance comparison few-shot scenarios on NYC dataset at different prediction steps in terms of MAE and RMSE. Bold denotes the best results and underline denotes the second- best results.
27

---

Published as a conference paper at ICLR 2024
MAE
RMSE
Model
Step
Step 3
Step
Step 1
Step 3
Step
HA
3.257
3.257
3.257
6.547
6.547
6.547
ARIMA
7.176
7.262
7.114
10.84
11.01
10.89
Region Trans
2.846
3.278
3.925
4.417
5.729
7.039
DASTNet
2.695
3,205
3.809
4.267
5,516
7.028
MAML
2.756
3.121
3.896
4.182
5.584
6,909
TPB
2.485
3.129
3.680
4.094
5,513
6,816
STGFSL
2.679
3.187
3.686
4.147
5.599
6.987
GPD
2.587
3.098
3.674
4.135
5.502
6.715
Table 13: Performance comparison of few-shot scenarios on METR-LA dataset at different pre- diction steps in terms MAE and RMSE. Bold denotes the best results and underline denotes the second-best results.
MAE
RMSE
Model
Step
Step 3
Step
Step 1
Step 3
Step
HA
3.142
3.142
3.142
4.535
4.535
4.535
ARIMA
5.179
5.456
5.413
6.829
7.147
7.144
Region' Trans
2.388
2.845
3.071
3.464
4.166
4.488
DASTNet
2.278
2.818
3.164
3.256
3,909
4.427
MAML
2.396
2.885
3.181
3.327
4.050
4.499
TPB
2.156
2.593
3.116
3.082
3.637
4.382
STGESL
2.133
2.565
2.901
3.183
3.815
4.291
GPD
2.031
2.381
2.550
2.948
3.422
3.630
Table 14: Performance comparison of few-shot scenarios on Didi-Chengdu dataset at different pre- diction steps in terms MAE and RMSE, Bold denotes the best results and underline denotes the second-ber beotd results.
28


Reviews:
[{"review_id": "UXw1ZwnIqs", "reviewer": "Reviewer_zXEa", "summary": "The proposed GPDiff framework in this paper aims to tackle the challenging transfer learning problem in spatio-temporal graph (STG) predictions. At first, it basically trains a transformer-based diffusion model on collected data in source cities, to capture the mapping function from a set of region-specific features to a set of model parameters. Then GPDiff can directly generate corresponding model parameters for unseen regions in new cities, based on the spatio-temporal features of these regions. The first stage is described as the pretraining and the second stage is the prompt-based finetuning. Experiments on several datasets illustrate that GPDiff empirically performs well on different prediction tasks, compared with a series of state-of-the-art methods.", "questions": "please check the weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tGenerally, the proposed method is sound and reasonable.\n2.\tThe proposed framework seems flexible, as it can adapt to different model structures and support training on data collected from any number of source cities.\n3.\tThe results show a large improvement. Ablation studies and case studies seem sufficient.\n4.\tI have checked the released code and found that it is well organized. The reproducibility should be good.", "weaknesses": "1. While the proposed method is sound, it is not clear what is the theoretical foundation behind GPDiff.\n2. How do we guarantee that model parameters of regions among different cities are within the same parameter space?\n3. Why choose diffusion for parameter generation? How much time does it consume in the denoising process?\n4. The experimental setup is not clear, which confused me why only DiDi-Chengdu is used for speed experiments. Besides, the caption of table 1 is not correct if didi-cheng is for speed evaluation. \n5. It seems that the method uses a newly proposed network. It is unclear whether the performance gain comes from the Diffusion or the network. Could the framework generalized to other networks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bAKYvGzpjI", "reviewer": "Reviewer_yfzH", "summary": "This paper provides a generative pretraining framework for spatio-temporal graph (STG) transfer learning tasks in smart city applications. Unlike traditional transfer learning approaches in this field, authors leverage the power of pretraining and design a novel approach that first pretrains a diffusion-based hypernetwork and then, based on the spatio-temporal prompt, directly generates model parameters for each region in a target city. Experiments on 7 real-world datasets covering two typical STG prediction tasks, i.e., crowd flow prediction and traffic speed prediction, demonstrate the superiority of the proposed GPDiff framework over SOTAs, with an improvement of 9.8%.", "questions": "Please refer to the weaknesses part.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1.\tThis paper provides a promising solution for an important research problem. Specifically, it is a pioneering practice in handling urban data-scarce scenarios that explores the paradigm of pretraining and prompt-based finetuning.\nS2.\tThe novelty of the proposed GPDiff framework is sound. Unlike existing works, it tackles the STG transfer learning problem from a new angle, i.e., pretraining a generative hypernetwork that captures the region-conditional distribution of optimized model parameters. This design overcomes the difficulties in applying pretraining in STG learning.\nS3.\tBuilding spatial prompts with easy-to-accessible data like POIs and region attributes is reasonable and practical. Building temporal prompts with a self-supervised representation learning technique is also technically sound.\nS4.\tBy leveraging the generation power of transformer-based diffusion models, GPDiff achieves sota performance on two typical STG prediction tasks, covering 7 real-world datasets.", "weaknesses": "W1.\tThe presentation of experimental results can be improved. For example, in Figure 4, why only show the temporal variations? What about spatial information? Visualization of model parameters needs to be more apparent.\nW2.\tI find that the presented results on the traffic speed prediction task seem much less than those of the crowd flow prediction task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lkenTlWk5N", "reviewer": "Reviewer_fmNn", "summary": "This paper addresses the problem of transfer learning on spatio-temporal graphs between cities. Inspired by recent advances in NLP, the paper proposes a new pre-training approach to generate different models for various regions based on a DDPM. The major insight here is to fit different data distributions across regions and cities. The proposed framework has been evaluated on two real-world spatio-temporal applications.", "questions": "Please reply to W1 to W3.\n\nAdditional questions:\n- Should region prompts also include information about the entire city? Merely focusing on regional features might not provide comprehensive information.\n- Some alternative methods may consider transferring at different levels or scales, whereas this model appears to only consider regional features. Is there a way to adaptively incorporate large-scale features as well?\n- The authors mentioned the issue of negative transfer between cities as the second limitation in the introduction. However, it is not clear how the proposed model explicitly addresses this problem. Can the authors provide more discussion and experimental evidence?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses an interesting real-world problem.\n2. The paper is easy to follow.\n3. I appreciate the design of prompts in this paper, i.e., from both spatial and temporal domains.\n4. The source code has been provided.", "weaknesses": "Although this paper has the above merits, I would like to point out the following concerns.\n\nW1. Insufficient related work and baselines:\n\nThe examination of related works in the paper appears to be extremely insufficient, with a noticeable lack of comparison to existing studies. From my understanding, numerous studies [1,2,3,4, 5] have delved into the application of meta-learning or hypernetworks to tailor prediction models for each node or region, especially [1] and [2] with high impacts in the field of traffic forecasting. It seems that implementing these directly could address the issue of varying data distributions across regions or cities. Therefore, I find it necessary for this paper to provide a compelling justification for the utilization of the framework proposed in this manuscript. What's more, they should be considered as baselines for empirical comparison.\n\nW2. Limited technical novelty:\n\nCompared to existing meta-learning-based methods, this paper has no significant novelty and technical contributions. The primary difference lies in the utilization of a DDPM (along with some prompts) to generate model parameters for each region, without theoretical contributions from the model side. This approach doesn't seem to surpass the acceptance threshold set by ICLR. The originality and technical value of this study could benefit from further enhancement.\n\nW3. Weak evaluation:\n\nFirstly, while the paper asserts that the proposed framework is model-agnostic, this claim is substantiated solely with evaluations on two outdated baselines, published before 2019, while recent methodologies have been overlooked, e.g., [6, 7, 8], see more in [9]. Secondly, the affirmations made in Section 4.4 lack persuasiveness. Figure 4 doesn't offer any insightful revelations, except reporting their differences. For instance, is there any proximity amongst various regions based on their respective POI information? These elements need to be elaborated on for a comprehensive understanding. Thirdly, the paper lacks a discussion on the model's efficiency. Customizing models for each region always brings considerable computational overhead. By the way, the efficiency of the proposed method should be benchmarked against the existing approaches as mentioned in W1. \n\nW4. Some SOTA baselines are missing, such as CrossTReS, MetaST, etc.\n\n\nReference:\n\n[1] Urban traffic prediction from spatio-temporal data using deep meta learning, KDD 2019.\n\n[2] Adaptive graph convolutional recurrent network for traffic forecasting, NeurIPS 2020.\n\n[3] Spatio-temporal meta learning for urban traffic prediction, TKDE 2020.\n\n[4] Region Profile Enhanced Urban Spatio-Temporal Prediction via Adaptive Meta-Learning, CIKM 2023.\n\n[5] Hyperst-net: Hypernetworks for spatio-temporal forecasting, arXiv.\n\n[6] Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting, KDD 2022.\n\n[7] Graph Neural Controlled Differential Equations for Traffic Forecasting, AAAI 2022.\n\n[8] Spatial-temporal identity: A simple yet effective baseline for multivariate time series forecasting, CIKM 2022\n\n[9] Spatio-temporal graph neural networks for predictive learning in urban computing: A survey, arXiv.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3XqpOCkcZo", "reviewer": "Reviewer_cSYa", "summary": "The paper proposes a generative pre-training framework based on diffusion models, GPDiff, to generatively pre-train a set of model parameters and generate tailored model parameters using data from the source city guided by prompts. The proposed solution addresses the challenges of inter-city transfer caused by data gaps across cities.", "questions": "- Case Study only wrote for different regions to generate a different distribution of parameters, indicating that the parameters can be generated for different data, but can explain the parameters are effective?\n\n- ABLATION STUDY content is too coupled, it can be split into CONTRAST STUDY, or directly changed to Study of GPDiff.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors design a novel generative pretraining framework for STG transfer problems, especially the unique angle of generating fine-grained model parameters that can serve as a good initial point for model optimizations on small-scale datasets.\n- Utilizing diffusion modeling for generative pretraining is technically sound. The designed model structure based on transformers is also appropriate and flexible for supporting the model-agnostic requirement.\n- The conducted experiments are sufficient to demonstrate the superiority of the proposed GPDiff framework in terms of not only prediction performance but also compatibility with different STG prediction models.\n- The paper presentation and writing are good.", "weaknesses": "- The time consumption of this framework needs to be provided.\n- Some details require further clarification. For example, how exactly is the improvement value of 9.8% calculated? As in the last part of the Introduction part, i.e., “an improvement of 9.8%”. Similar cases are in the description text of Table 2.\n- I think the rationality behind the excellent performance of GPDiff is easy to understand. But I still expect some further explanations by designing some more pertinent experiments. For example, which kind of regions in target cities would be much easier to benefit from knowledge transfer in this pretraining framework?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)