Taming Local Effects in Graph-based Spatiotemporal Forecasting
Authors: Andrea Cini
Ivan Marisca
Daniele Zambon
Cesare Alippi
Venue: NeurIPS 2023 poster (2023)
Keywords: time series forecasting
spatiotemporal forecasting
graph-based spatiotemporal forecasting
graph neural networks
TL;DR: The paper studies the interplay between globality and locality in graph-based spatiotemporal forecasting, proposing a framework to rationalize the practice of including trainable node embeddings in such architectures.

Abstract: Spatiotemporal graph neural networks have shown to be effective in time series forecasting applications, achieving better performance than standard univariate predictors in several settings. These architectures take advantage of a graph structure and relational inductive biases to learn a single (global) inductive model to predict any number of the input time series, each associated with a graph node. Despite the gain achieved in computational and data efficiency w.r.t. fitting a set of local models, relying on a single global model can be a limitation whenever some of the time series are generated by a different spatiotemporal stochastic process. The main objective of this paper is to understand the interplay between globality and locality in graph-based spatiotemporal forecasting, while contextually proposing a methodological framework to rationalize the practice of including trainable node embeddings in such architectures. We ascribe to trainable node embeddings the role of amortizing the learning of specialized components. Moreover, embeddings allow for 1) effectively combining the advantages of shared message-passing layers with node-specific parameters and 2) efficiently transferring the learned model to new node sets. Supported by strong empirical evidence, we provide insights and guidelines for specializing graph-based models to the dynamics of each time series and show how this aspect plays a crucial role in obtaining accurate predictions.

Taming Local Effects in Graph-based Spatiotemporal Forecasting
Andrea Cini +1, Ivan Marisca "1 Daniele Zambon ', Cesare Alippi 12
TTe Swiss Lab IDSIA USI-SUPSI, Università della Svizzera italiana, Politecnico di Milano (andrea. cini, ivan.
ivan.marisca, marisca, daniele. zambon, cesare. alippi)
Abstract
Spatiotemporal graph neural networks have shown to be effective in time series forecasting applications, achieving better performance than standard univariate pre- dictors in several settings. These architectures take advantage of a graph structure and relational inductive biases to learn single (global) inductive model to predict any number of the input time series, each associated with a graph node. Despite the gain achieved in computational and data efficiency w.r.t. fitting set of local models, relying on single global model can be a limitation whenever some the time series are generated by a different spatiotemporal stochastic process. The main objective this paper is to understand the interplay between globality and locality in graph based spatiotemporal forecasting, while contextually proposing methodological framework to rationalize the practice of including trainable node embeddings in such architectures. We ascribe trainable node embeddings the role of amortizing the learning of specialized components. Moreover, embeddings allow for effectively combining the advantages of shared cesssag-ppsssing layers with node-specific parameters and 2) efficiently transferring the learned model to new node sets. Supported by strong empirical evidence, we provide insights and guidelines for specializing graph-base models to the dynamics of each time series and show how this aspect plays a crucial role in obtaining accurate predictions.
Introduction
Neural forecasting methods I take advantage of large databases of related time series learn models for each individual process. If the time series in the database are not independent, functional dependencies among them can be exploited obtain more accurate predictions, e.g., when the considered time series are observations collected from a network of sensors. In this setting, we use the term spatiotemporal time series to indicate the existence of relationships among subsets of time series (sensors) that span additional axes other than the temporal one, denoted here as spatial in a broad sense. Graph representations effectively model such dependencies and graph neural networks (GNNs) 24 can be included as modules in the forecasting architecture to propagate information along the spatial dimension. The resulting neural architectures are known in the literature as spatiotemporal graph neural networks (STGNNs) (5 6 and have found widespread adoption in relevant applications ranging from traffic forecasting embed inductive biases typical of graph deep learning and graph signal processing [10] and, thus, have several advantages over standard multivariate models; in fact, a shared set of learnable weights used to obtain predictions for each time series by conditioning nodes. Nonetheless, has become more and more common to see node-specific trainable parameters being introduced means extract node(sensor)-level features then used as spatial identifiers within the processing [| [2]. By doing so, the designer accepts a compromise transferability that often
to energy analytics [ 9. These models
observations at the neighboring
Equual contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

empirically leads to higher forecasting accuracy on the task at hand. We argue that the community has yet to find proper explanatory framework for this phenomenon and, notably, has yet to design proper methodologies to deal with the root causes of observed empirical results, in practical applications.
In the broader context of time series forecasting, single models learned from set of time series are categorized as global and are opposed to local models, which instead are specialized for any particular time series in the set [13 [4]. Global models are usually more robust than local ones, as they require a smaller total number of parameters and are fitted on more samples. Standard STGNNs, then, fall within the class of global models and, as result, often have an advantage over local multivariate approaches; however, explicitly accounting for the behavior of individual time series might be problematic and require a large memory and model capacity. As an example, consider the problem of electric load forecasting: consumption patterns of single customers are influenced by shared factors, e.g., weather conditions and holidays, but are also determined by the daily routine of the individual users related by varying degrees of affinity. We refer to the dynamics proper to individual nodes as local effects.
Local effects can be accounted for by combining a global model with local components, e.g., by using encoding and/or decoding layers specialized for each input time series paired with a core global processing block. While such an approach to building hybrid global-local models can be effective, the added model complexity and specialization can negate the benefits of using a global component. In this paper, we propose to consider and interpret learned node embeddings as mechanism to amortize the learning of local components; in fact, instead of learning a separate processing layer for each time series, node embeddings allow for learning single global module conditioned on the learned (local) node features. Furthermore, we show that - within proper methodological framework node embeddings can be fitted to different node sets, thus enabling an effective and efficient transfer of the core processing modules.
Contributions In this paper, we analyze the effect of node-specific components in spatiotemporal time series models and assess how incorporate them in the forecasting architecture, while pro- viding an understanding of each design choice within a proper context. The major findings can be summarized in the following statements.
S1 Local components can be crucial to obtain accurate predictions in spatiotemporal forecasting.
S2 Node embeddings can amortize the learning of local components.
S3 Hybrid global-local STGNNs can capture local effects with contained model complexity and smaller input windows w.r.t. fully global approaches.
S4 Node embeddings for time series outside the training dataset be obtained by fitting relatively small number of observations and yield more effective transferability than fine-tuning global models.
S5 Giving structure to the embedding space provides an effective regularization, allowing for similarities among time series to emerge and shedding light on the role of local embeddings within a global architecture
Throughout the paper, we reference the findings with their pointers. Against this backdrop, our main novel contributions reside in:
A sound conceptual and methodological framework for dealing with local effects and designing node-specific components in STGNN architectures.
An assessment of the role of learnable node embeddings in STGNNs and methods to obtain them.
comprehensive empirical analysis of the aforementioned phenomena representative architectures across synthetic and real-world datasets.
Methods to structure the embedding space, thus allowing for effective and efficient reuse of the global component in transfer learning scenario.
We believe that our study constitutes an essential advancement toward the understanding of the interplay of different inductive biases in graph-based predictors, and argue that the methodologies and conceptual developments proposed in this work will constitute a foundational piece of koww-how for the practitioner.

---

2 Preliminaries and problem statement
Consider set of N time series with graph-side information where the i-th time series is composed by a sequence of dz dimensional vectors z E IR. observed at time step i; each time series {athe might be generated by a different stochastic process. Matrix X, E xd, encompasses the N observations at time & and, similarly, X indicates the sequence of observations within the time interval [4, ( + T). Relational information is encoded by a weighted adjacency matrix A E W that accounts for (soft) functional dependencies existing among the different time series. We use interchangeably the terms node and sensor to indicate the entities generating the time series and refer to the node set together with the relational information (graph) sensor network. Eventual exogenous variables associated with each node are indicated by U, E R- the tuple G. = (X,, U,, A) indicates all the available information associated with time step
We address the multistep-ahead time-series forecasting problem, i.e., we are interested in predicting. for every time step & and some H, W 2 1, the expected value of the next H observations X и given a window :t of W past measurements.
case data from multiple sensor networks are available, the problem can be formalized as learning from M disjoint collections of spatiotemporal time series
D [9 [9, ( t2 tm +T }. potentially without overlapping time frames. In the latter case, we assume sensors to be homogeneous both within single network and among different sets. Furthermore, we assume edges to indicate the same type of relational dependencies, e.g..
g(
physical proximity.
3 Forecasting with Spatiotemporal Graph Neural Networks
This section provides taxonomy of the different components that constitute STGNN. Based on the resulting archetypes, reference operators for this study are identified. The last paragraph of the section broadens the analysis to fit STGNNs within more general time series forecasting frameworks.
Spatiotemporal message-passing message-passing (STMP) layers s.t.
We consider STGNNs obtained by stacking spatiotemporal
H = STMP (H)
where H Rg indicates the stack of node representations h time step at the I-th layer. The shorthand I indicates the sequence of all representations corresponding to the time steps up to t (included). Each STMP' (-) layer is structured as follows
h 11:1+1 = p² (hi/ AGGR { (h2t hel az)) }
}).
(2)
JEN(()
where p' and 7 are respectively the update and message functions, e.g.. implemented by multilayer perceptrons (MLPs) or recurrent neural networks (RNNs). AGGR( indicates generic permutation invariant aggregation function, while (i)) refers to the set of neighbors of node i, each associated with edge with weight ajt. Models of this type are fully inductive, in the sense that they can be used to make predictions for networks and time windows different from those they have been trained on, provided a certain level of similarity (e.g., homogenous sensors) between source and target node sets [5]].
Among the different implementations of this general framework, can distinguish between time- then-space (TTS) and time-and-space (T&S) models by following the terminology of previous works 16 Specifically, in TTS models the sequence of representations h is encoded by sequence model, c.g., an RNN, before propagating information along the spatial dimension through message passing (MP) [6]. Conversely, in T&S models time and space are processed in a more integrated fashion, c.g., by a recurrent GNN [5] or by spatiotemporal convolutional operators [7]. In the remainder of the paper, take for TTS model STGNN composed by Gated Recurrent Unit (GRU) 18] followed by standard MP layers [19]
H = GRU (HQ.),
H = MP' (H), A)
where I 1..... L. 1 and GRU( processes sequences node-wise. Similarly, we consider as reference T&S model a GRU with an MP network at its gates 20), that process input data as
H H
GRU' (IMP' (H), A))
3

---

Moreover, STGNN models can be further categorized w.r.t. the implementation of message function; in particular, by loosely following Dwivedi al. [2T], we call isotropic those GNNs where the message function o only depends on the features of the sender node h conversely, we call anistropic GNNs where 7 takes both h and h as input. In the following, the case-study isotropic operator is
R.J+1 = WW + MEAN h
(Wsh?")) JEN(S)
where Wi and W32 are matrices of learnable parameters and E( a generic activation function. Conversely, the operator of choice for the anisotropic case corresponds
m W/ W||
af a Womm,
(6)
h h {( WWihh
{att
mi m))
SUM JEN(G)
where matrices W E IR 12_xdm WT. W3 and W3 are learnable parameters, a( (() is the sigmoid activation function and || the concatenation operator applied along the feature dimension (see Appendix A.2 for a detailed description).
Global and local forecasting models Formally, a time series forecasting model is called global if its parameters are fitted to a group of time series (either univariate or multivariate), while local models specific to single (possibly multivariate) time series. The advantages of global models have been discussed at length in the time series forecasting literature [22] [4] [3] [ and are mainly ascribable to the availability of large amounts of data that enable generalization and the use of models with higher capacity w.r.t. the single local models. As presented, and further detailed in the following section, STGNNs are global, yet have peculiar position in this context, as they exploit spatial dependencies localizing predictions w.r.t. each node's neighborhood. Furthermore, the transferability of GNNs makes these models distinctively different from local multivariate approaches enabling their use in cold-start scenarios
23 and making them inductive both temporally and spatially.
Locality and globality in Spatiotemporal Graph Neural Networks
We now focus the impact of local effects in forecasting architectures based STGNNs. The section starts by introducing a template to combine different processing layers within a global model, then continues by discussing how these can be turned into local. Contextually, we start to empirically probe the reference architectures.
4.1 A global processing template
STGNNs localize predictions in space, i.e., with respect to single node, by exploiting an MP operator that contextualizes predictions by constraining the information flow within each node's neighborhood. STGNNs are global forecasting models s.t.
X F' (G--wai Φ)
(8)
where ó are the learnable parameters shared among the time series and Xt++H indicate the H-step ahead predictions of the imput time series given the window of (structured) observations X In particular, we consider forecasting architectures consisting of an encoding step followed by STMP layers and final readout mapping representations to predictions; the corresponding sequence of operations composing FG can be summarized as
h 1.0
ENCODER (0i-1,"1, 11
(9)
H 1-1 STMP A). 1= 1=
(10)
E DECODER (n:
(11)
ENCODER(> and DECODER( indicate generic encoder and readout layers that could be imple- mented in several ways. In the following, the encoder is assumed to be standard fully connected
linear layer, while the decoder is implemented by an MI.P with a single hidden layer followed by an output (linear) layer for each forecasting step.

---

4.2 Local effects
Differently from the global approach, local models are fitted to a single time series (e.g., see the standard Box-Jenkins approach) and, in our problem settings, can be indicated as
B (ak- 0')
(12)
where / (-:0+) is a sensor-specific model, e.g., an RNN with its dedicated parameters 0', fitted on the i-th time series. While the advantages of global models have already been discussed, local effects, i.e., the dynamics observed at the level of the single sensor, are potentially more easily captured by a local model. In fact, if local effects are present, global models might require an impractically large model capacity to account for all node-specific dynamics [13], thus losing some of the advantages of using a global approach (S3) In the STGNN case, then, increasing the imput window for each node would result in large computational overhead. Conversely, purely local approaches fail to exploit relational information among the time series and cannot reuse available knowledge efficiently in an inductive setting.
Combining global graph-based components with local node-level components has the potential for achieving a two-fold objective: 1) exploiting relational dependencies together with side information to learn flexible and efficient graph deep learning models and 2) making at the same time specialized and accurate predictions for each time series. In particular, we indicate global-local STGNNs as
a Fet+ F w:: 0,6")
(13)
where function F and parameter vector & are shared across all nodes, whereas parameter vector 0 is time-series dependent. Such a function F( could be implemented, for example, as a sum between a global model (Eq. ( and a local one (Eq. 12h:
(1) 144H F6 (G.-wai o),
tu++ 1 = -i (xi
..(2)
(14)
a alatn = b++H + (.)
(1))
(15)
or with more integrated approach - by using different weights for each time series at the encoding and/or decoding steps. The latter approach results in using different encoder and/or decoder for each i-th node in the template STGNN (Eq, I to extract representations and, eventually, project them back into input space:
h ENCODER, (z/-114(11:00
(16)
Ee++ H1 DECODER, (h:-;; (8
(17)
MP layers could in principle be specialized as well, e.g., by using a different local update function 7x( ) for each node. However, this would be impractical unless subsets of nodes are allowed to share parameters to some extent (e.g., by clustering them).
To support our arguments, Tab. shows empir- ical results for the reference TTS models with isotropic message passing (TTS-IMP) on 2 popular traffic forecasting benchmarks (METR- LA and PEMS-BAY (6). In particular, we compare the global approach with 3 hybrid global-local variants where local weights are used in the encoder, in the decoder, or in both of them (see Eq. 17 and the light brown block Tab. D Notably, while fitting a separate RNN each individual time series fails (LocaIRNNs), exploiting a local encoder and/or decoder significantly improves perfor-
Table 1: Perfomance (MAE) of TTS-IMP variants and number of associated trainable parameters in PEMS-BAY (5-run average).
METR-LA
PEMS-BAY (# weights)
Global TTS
3.35 1001
1.72 10000
4.71x10*
Global-locss -ocal ITS
Weights
Encoder
3.15 40.1
1.66 4000
2075x10"
Decoder
3.09 #001
1.58 #0000
3.00x
Enc. Dec.
3.16 +0.01
1.70 #0.01
1.28x10*
Embed.
Encoder
3,08 10.0
1.58 10.00
5.96 10
Decoder
3.13 +0,00
1.60 +0000
1.9
Enc. + Dec.
3.07 10.0
1.58 10.00
6.16x 10
mance w.r.t. the fully global model (ST). Note
that the price of specialization is paid in terms of the number of learnalle parameters which an order of magnitude higher global-local variants. The table reports as reference also results for FC-RNN, multivariate RNN taking as input the concatenation of all time series. Indeed, having both encoder and decoder implemented as local layers leads to large number of parameters and has marginal impact on forecasting accuracy. The
FC-RNN
3.56 + n.3
2,32 = 001
3.04 10
LocaIRNNs
3.69 10.00
1.91 10000
1.10x100
5

---

light gray block in Tab. anticiiaaes the effect of replacing the local layers with the use of learnable node embeddings, an approach discussed in depth the next section. Additional results including TTS models with anisotropic message passing (TTS-AMP) are provided in Appendix B
5 Node embeddings
This section introduces node embeddings as mechanism to amortize the learning of local components and discusses the supporting empirical results. We then propose possible regularization techniques and discuss the advantages of embeddings in transfer learning scenarios.
Amortized specialization Static node features offer the opportunity to design and obtain node identification mechanisms across different time windows tailor predictions to a specific node, However, in most settings, node features are either unavailable or insufficient to characterize the node dynamics. A possible solution consists of resorting to learnable node embeddings, i.e., table of learnable parameters θ = V Nxd. Rather than interpret these learned representations as positional encodings, our proposal is consider them as a way of amortizing the learning of node-level specialized models. More specifically, instead of learning a local model for each time series, embeddings fed into modules of global STGNN and learned end-to-end with the forecasting architecture allow for specializing predictions by simply relying on gradient descent to find a suitable encoding.
The most straightforward option for feeding embeddings into the processing is to update the template model by changing the encoder and decoder as
0. h
ENCODER ve--1:4-111
(18)
d b DECODER
(ni o²)
(19)
which can be seen as amortized versions of the encoder and decoder in Eq. 16 17
The encoding scheme of Eq. 18 also facilitates the propagation of relevant information by identifying nodes, an aspect that can be particularly significant as message-passing operators in particular isotropic ones
can act low-pass filters that smooth out node-level features 24 25
Tab. [ (light gray block) reports empirical results that show the effectiveness of embeddings in amortizing the learning of local components (S2), with negligible increase in the number of trainable parameters w.r.t. the base global model. In particular, feeding embeddings to the encoder, instead of conditioning the decoding step only, results in markedly better performance, hinting the impact of providing node identification ahead of MP (additional empirical results provided in Sec.
5.1 Structuring the embedding space
The latent space in which embeddings are learned can be structured and regularized to gather benefits in terms of interpretability, transferability, and generality (S5). In fact, accommodating new embeddings can be problematic, as they must fit in a region of the embedding space where the trained model can operate, and, at the same time, capture the local effects at the new nodes. In this setting, proper regularization can provide guidance and positive inductive biases to transfer the learned model to different node sets. As an example, if domain knowledge suggests that neighboring nodes have similar dynamics, Laplacian regularization 266 27 can added to the loss. In the following, we propose two general strategies based respectively to impose soft constraints on the latent space. As shown in Sec. | the resulting structured space additionally allows us to gather insights into the features encoded in the embeddings.
variational inference and on a clustering loss
Variational regularization As a probabilistic approach to structuring the latent space, we propose to consider learned embeddings as parameters of an approximate posterior distribution - given the training data on the vector used to condition the predictions. In practice, we model each node embedding as sample from multivariate Gaussian v' q² (v' (D) = N (n, diag(a?) where (m, o)) are the learnable (local) parameters. Each node-level distribution is fitted on the training data by considering a standard Gaussian prior and exploiting the reparametrization trick 28 to minimize under the sampling of
27
6, = Evar
X - X
+ 3DDK((Q)))

---

where P (O.,1) is the prior, DKL the Kulback- Leibler divergence, and & controls the regular- ization strength. This regularization scheme results smooth latent space where it is easier to interpolate between representations, thus providing a principled way for accommodating different node embeddings.
Clustering regularization A different (and potentially complementary) approach to structuring the latent space to incentivize node embeddings to form clusters and, consequently, self-organize into different groups. We do so by introducing regularization loss inspired by deep K K-means algorithms [29]. In particular, besides the embedding table V E RNx,, we equip the embedding module with matrix C' E IRK xd, of K < N learnable centroids and cluster assignment matrix S E IR x encoding scores associated to each node-cluster pair. We consider scores as logits of a categorical (Boltzmann) distribution and learn them by minimizing the regularization term
eSa/T
Ereg EM V V M P(M, =)
ESuu/T
where is hyperparameter. We minimize Lreg which corresponds to the embedding-to--ennnen distance jointly with the forecasting loss by relying on the Gumbel softmax trick [30]. Similarly to the variational inference approach, the clustering regularization gives structure to embedding space and allows for inspecting patterns in the learned local components (see Sec.
Transferability of graph-based predictors Global models based GNNs make predictions for seeer--een-bbfooo node sets, and handle graphs of different sizes and variable topology. In practice, this means that the graph-based predictors can easily handle new sensors being added the network over time and be used for zero-shot transfer. Clearly, including in the forecasting architecture node-specific local components compromises these properties. Luckily, if local components replaced by node embedding, adapting the specialized components is relatively cheap since the number of parameters to fit w.r.t. the new context usually contained, and eventually both the graph topology and the structure of the embedding latent space can be exploited (S4). Experiments in Sec. provide an in-depth empirical analysis of transferability within our framework and show that the discussed regularizations can be useful in this regard.
Related works
GNNs have been remarkably successful in modeling structured dynamical systems 31 temporal networks [34 36 and sequences of graphs [37] 38). For what concerns time series processing,
recurrent GNNs (5 6 were among the first STGNNs being developed, followed by fully convolutional models 7 39 and attention-based solutions 440 Among the methods that focus on modeling
node-specific dynamics, Bai al. (TT] use factorization of the weight matrices in recurrent STGNN to adapt the extracted representation to each node. Conversely, Chen et al. 44 use a model inspired by Wang al. [4] consisting of a global GNN paired with a local model conditioned on the neighborhood of each node. Node embeddings have been mainly used in structure-learnigg modules to amortize the cost of learning the full adjacency matrix
45 12 and in attention- based 39
approaches as positional encodings 46
47 41 Shao et al. 48 observed how adding spatiotemporal identification mechanisms to the forecasting architecture can outperform several ffate-of-the-an
STGNNs. Conversely, Yin et al. [49 used a cluster-based regularization to fine-tune an AGCRN- like model on different datasets. However, none of the previous works systematically addressed directly the problem of globality and locality in STGNNs, nor provided a comprehensive framework accounting for learnable node embeddings within different settings and architectures. Finally, besides STGNNs, there several examples of hybrid global and local time series forecasting models. Wang et al. [44) propose an architecture where K global models extract dynamic global factors that are then weighted and integrated with probabilistic local models. Sen et al. [50] instead use matrix factorization scheme paired with a temporal convolutional network [1] to learn a multivariate model then used to condition second local predictor.
7 Experiments
This section reports salient results of an extensive empirical analysis of global and local models and combinations thereof in spatiotemporal forecasting benchmarks and different problem settings;
7

---

complete results of this systematic analysis can be found in Appendix B Besides the reference architectures, we consider the following baselines and popular ofte-of-thhearr architectures.
RNN: global univariate RNN sharing the same parameters across the time series.
FC-RNN: a multivariate RNN taking as input the time series as if they were a multivariate one.
LocaIRNNs: local univariate RNNs with different sets of parameters for each time series.
DCRNN 66] recurrent T&S model with the Diffussin Convolutional operator.
AGCRN Mi: the T&S global-local Adaptive Graph Convolutional Recurrent Network.
Graph WaveNet: the deep T&S spatiotemporal convolutional network by Wu et al. 339
We also consider global-local RNN, in which we specialize the model by using node embeddings in the encoder. Note that among the methods selected from the literature only DCRNN can considered fully global (see Sec. (). Performance is measured in terms of mean absolute error (MAE).
Synthetic data We start by assessing the performance of hybrid global-local spatiotemporal models in con- trolled environment, considering a variation of GP- VAR (52). a synthetic dataset based on a polynomial graph filter [53], that we modify to include local effects. In partieular, data are generated from the spatiotemporal process
Table 2: One-step-ahead forecasting er- ror (MAE) of the different models in GP- VAR datasets (5 runs).
MODELS
GPVAR
GPVAR-L
FC-RNN
5978 .323
LocalRNNs
.4077 000
.6100= 03m
ΣΣ H.
, X1-
RNN
.3999
.5440= 0000
Global
1=1
TTS-IMP
.32324 0000
.40.9 00012
X++1 = 2 tanh (H)) + b & tanh (X,-1) ne (20)
TTS-AMP
.3193
.3587 0009
RNN
.3991 1: (001
.4612 000
where @ ROXL a E RN
RN and
Emhh.
T ~ (N II).. We refer to this dataset as GPVAR- L: note that a and are node-specific parameters that inject local effects into the spatiotemporal process. We indicate simply as GPVAR the process obtained by fixing b = 0.5, i.e., by removing local effects. We use as the adjacency matrix the community graph used in prior works, increasing the number of nodes to 120 (see Appendix [A.I].
TIS-IMP
31955.0000
.3200= 0002
TTS-AMP
.194
=3199=0000
Optimal model
.3192
.3192
Tab.7 shows forecasting accuracy for reference architectures with G-steps window on data generated from the processes. In the setting with no local effects, all STGNNs achieve performance close to the theoretical optimum, outperforming global and local univariate models but also the multivariate FC-RNN that without any inductive bias struggles to properly the data. In GPVAR-L, global and univariate models fail to match the performance of STGNNs that include local components (ST): interestingly, the global model with anisotropic MP outperforms the isotropic alternative, suggesting that the more advanced MP schemes can lead to more effective state identification.
Table 3: TTS-IMP atee-eep- MAE on GPVAR-L with varying window length W and capacity d, (5 runs).
Finally, Tab. 3 shows that to match the performance of models with lo- cal components, in fully global mod- els both window size and capacity should be increased (53). This result is in line with what we should expect by considering the theory of local and global models and suggests that similar trade-offs might also happen in practical applications where both computational and sample complex- ity are concern.
Global
Embeddings
W
dp
16
d, = 32
d, = 64
dy = 16
da 32 d, 64
2
537 #.001
.6679.0006
.3198 nom .319=.00 33003 00)
6
4059+
0002
3351
.3000 .006
.201 0.00
12
,33622..0n 36722
2012
.3200
3200
omm
24
3485 .3286
.3200 :001
3200
0000 33111:000
We then compare the performance of reference architectures and baselines with and without node embeddings at the encoding and decoding steps. Note that, while reference architectures and DCRNN are purely global models, Graph WaveNet and AGCRN use node embeddings to obtain an adjacency matrix for MP. AGCRN, furthermore, uses embeddings to make the convolutional filters
Benchmarks
8

---

Table Forecasting enror (MAE) on benchmark datasets (5 runs). The best result between each model and its variant with embeddings is in bold. N/A indicates runs exceeding resource capacity.
MODELS
METR-LA
PEMS-BAY
CER-E
AQI
METR-LA PEMS-BAY
CER-E
AQI
Reference arch.
Global models
Global-Hocal models (with embeddings)
RNN
3,54.000 00
1.77.100 00
156.
14.02.104
3.15
1.59..00
421.50
13.73..04
T&S-IMP
3.
1.70..01
10.98
12.87
3.101.01
1,59..00
417.71.1.28
12,48 1.00
TTS-IMP
3.34:
1.72:t 00
43
12.74
3.08::01
1.58::.00
412.44
12.33=.02
T&S-AMP
3.22+
1.65
N/A
N/A
3.07++02
+.59
N/A
N/A
TTS-AMP
3.241.01
1.66
431.33108 10:8
12.30..0z
3.061.01
1
412.95 1.28
12.15
Baseline arch.
Original
Embeddings at Encoder and Decoder
DCRNN
3.22+
1.64:
428.36:1.23
12.96
3.07::02
1.60
412.87
12.53=.0z
Graph WaveNet
3.05
1.56
397.17
12.08
2.99
+
401.15#1.w
11.81=:04
AGCRN
3.16+ 01
1.61
44..80+1.25
13.33#
3.14::00
1.62# 00
436.84=2.6
13.28=.13
Average claster lead (KWh)
Claster
Chaser n.
Cluser
Cluser n.
Cluser
20
Carara Cletre
0.5
12:0
10:00
600.3
12:00
20.34
$2,00
40040
12:00
1240
16:00
12
Men
Wed
The
Sar
(a) Average cluster load by day of the week.
(b) Node embeddings.
Figure I: Time series clusters in CER-E, obtained by regularizing the embedding space. (a) Average load for each clusters. (b) t-SNE plot of the corresponding node embeddings.
adaptive w.r.t. the node being processed. We evaluate all models real-world datasets from three different domains: traffic networks, energy analytics, and air quality monitoring. Besides the already introduced traffic forecasting benchmarks (METR-LA and PEMS-BAY),
experiments
smart metering data from the CER-E dataset [54] and air quality measurements from AQI (55) by using the same pre-processing steps and data splits of previous works 22 47). The full experimental setup is reported in appendix A Tab. reports forecasting mean absolute error (MAE) averaged over the forecasting horizon. Global-local reference models outperform the fully global variants every considered scenario (S1). A similar observation can be made for the state-of of-trtt architectures, where the impact of node embeddings (at encoding and decoding) large for the fully global DCRNN and more contained in models already equipped with local components. Note that hyperparameters were not tuned to account for the change architecture. Surprisingly, the simple TTS-IMP model equipped with node embeddings achieves results comparable to that of state-of-the-art STGNNs with significantly lower number parameters and a streamlined architecture. Interestingly, while both global and local RNNs models fail, the hybrid global-local RNN ranks among the bert-performing models, outperforming graph-based models without node embeddings in most settings.
Structured embeddings To test the hypothesis that structure in embedding space provides insights on the local effects at play (S5). we consider the clustering regularization method (Sec. 5.1) and the reference TTS-IMP model trained on the CER-E dataset. We set the number of learned centroids K 5 and train the cluster assignment mechanism end-to-end with the forecasting architecture. Then, we inspect the clustering assignment by looking at intra-cluster statistics. In particular, for each load profile, we compute the weekly average load curve, and, for each hour, we look at quantiles of the energy consumption within each cluster. Fig. La shows the results of the analysis by reporting the median load profile for each cluster; shaded areas correspond to quantiles with 10% increments. Results show that users in the different clusters have distinctly different consumption patterns. Fig. lb

---

Table Forecasting error (MAE) in the transfer learning setting (5 runs average). Results refer to 1 1-week fine-tuning set size on all PEMS datasets.
TTS IMP
PEMS03
PEMS04
PEMS07
PEMS08
Fine tuning
Global
15.30 = D:3
21.59 = 0.11
23.82 + 0.03
15.90 + 0.07
Embeddings
14.64 = 0.05
20.27 = 0.11
22.23 = 0.08
15.45 +: 0.06
Variational
14.56 = IMLS
20.19 0.85
22.43 + 0.02
15,41 + 0.06
Clustering
14.60 1M12
19.91 = 0.11 22.16
15.41 + 0J06
Zero-shot
18.20 = 0.09
23.88 + 0.08
32.76 + 0.69
20.41 + 0.07
Table 6: Forecasting error (MAE) on PEMS04 in the transfer learning setting by varying fine-tuning set size (5 runs average).
Model
Training set size
TTS IMP
2 weeks
week
3 days
day
Global
20.86 + am
21.59 = 0.11 21.84 = 0.6
22.26 0.10
Embeddings
19.96 + 0.08
20.27 0.11
21.03 i 0.14
21.99 0.13
Variational
19.94 = 0.08
20.19
20.71 0.12
21.20 0.15
Clustering
19.69 + 0.06 19.91
20.48 + 0.09 21.91
shows a 2D t-SNE visualization of the learned node embeddings, providing a view of the latent space and the effects of the cluster-based regularization.
Transfer In this experiment, we consider the scenario in which an STGNN for traffic forecasting is trained by using data from multiple traffic networks and then used to make predictions for a disjoint set of sensors sampled from the same region. We use the PEMS03, PEMS04, PEMS07, and PEMS08 datasets [56). which contain measurements from 4 different districts California. We train models 3 of the datasets, fine-tune on 1 week of data from the target left-out dataset, validate on the following week, and test on the week thereafter. We compare variants of TTS-IMP with and without embeddings fed into encoder and decoder. Together with the unconstrained embeddings, we also consider the variational and clustering regularization approaches introduced in Sec. [.. At the fine-tuning stage, the global model updates all of its parameters, while in the hybrid global-local approaches only the embeddings are fitted to the new data. Tab. rebports results for the described scenario. The fully global approach is outperformed by the hybrid architectures in all target datasets (S4). Besides the significant improvement in performance, adjusting only node embeddings retains performance on the source datasets. Furthermore, results show the positive effects of regularizing the embedding space in the transfer setting (S5). This is further confirmed by results in Tab. 6 which report, for PEMS04, how forecasting error changes in relation the length of the Ane-tuning window. We refer to Appendix B for an in-depth analysis of several additional transfer learning scenarios.
8 Conclusions
We investigate the impact of locality and globality in graph-based spatiotemporal forecasting architec- tures. We propose a framework to explain empirical results associated with the use of trainable node embeddings and discuss different architectures and regularization techniques to account for local effects. The proposed methodologies are thoroughly empirically validated and, although not inductive, prove to be effective in transfer learning context. We argue that work provides necessary and key methodologies for the understanding and design of effective graph-based spatiotemporal forecasting architectures. Future works can build on the results presented here and study alternative, and even more transferable, methods to account for local effects.
Acknowledgements
This research was partly funded by the Swiss National Science Foundation under grant 204061: High-Order Relations and Dynamics in Graph Neural Networks.
10

---

References
[1] Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang Wang. Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael Sohlke--ccheiine David Salinas, Lorenzo Stella, François-Xavea Aubet, Laurent Callot, and Tim Januschowski. Deep learning for time series forecasting: Tutorial and literature survey. ACM Comput. Surv., 55(6), dec 2022. ISSN 0360-0300, doi: 10. 10./1333333382 URL littps://doo org/10. 1145/3533382
[2] Davide Bacciu, Federico Errica, Alessio Micheli, and Mareo Podda. gentle introduction to deep learning for graphs. Neural Networks, 129:203-221 2212 2020.
[3] Ljubiša Stanković, Danilo Mandic, Miloš Daković, Miloš Brajović, Bruno Scalzo, Shengxi Li, Anthony Constantinides, et al. Data analytics on graphs part ii:: machine learning on graphs. from graph topology to applications. Foundations and Trends@ in Machine Learning, 13(4): 332-530, 2020.
[4] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint vrXivv2104. 13478, 2021.
[5] Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks. In International Conference on Neural Information Processing, pages 362-373. Springer, 2018.
[6] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. In International Conference on Leurning Representations, 2018.
[7] Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: deep learning framework for traffic forecasting. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 3634-3640, 2018.
[8] Simone Eandi, Andrea Cini, Slobodan Lukovic, and Cesare Alippi. Spatio-temporal graph neural networks for aggregate load forecasting. In 2022 International Joint Conference on Neural Networks (ICCN)), pages 1-8. IEEE, 2022.
[9] Jonas Berg Hansen, Stian Normann Anfinsen, and Filippo Maria Bianchi. Power flow balancing with decentralized graph neural networks. IEEE Transactions on Power Systems, 2022.
[10] Antonio Ortega, Pascal Frossard, Jelena Kovačević, José MF Moura, and Pierre Vandergheynst. Graph signal processing: Overview, challenges, and applications. Proceedings of the IEEE, 106 (5)8808 828, 2018
[11] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. Advances in Neural Information Processing Systems, 33:17804-17815, 2020.
[12] Ailin Deng and Bryan Hooi. Graph neural network-based anomaly detection multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4027-4035, 2021.
[13] Pablo Montero-Manso and Rob Hyndman. Principles and algorithms for forecasting groups of time series: Locality and globality. International Journal of Forecasting, 37(4):6332-1553, 2021.
[14] Tim Januschowski, Jan Gasthaus, Yuyang Wang, David Salinas, Valentin Flunkert, Michael Schnneder,, and Laurent Callot. Criteria for classifying forecasting methods. Interna- lional Journal of Forecasting, 36(1):167-177. 2020.
[15] Luana Ruiz, Luiz Chamon, and Alejandro Ribeiro. Graphon neural networks and the trans- ferability graph neural networks. Advances in Neural Information Processing Systems, 33: 1702-1713 1712, 2020.
[16] Jianfei Gao and Bruno Ribeiro. On the equivalence between temporal and static equivariant graph representations. In International Conference on Machine Learning, pages 7052-7076. PMLR, 2022.
11

---

[17] Andrea Cini, Daniele Zambon, and Cesare Alippi. Sparse graph learning from spatiotemporal time series. Journal of Machine Learning Research, 24(242):1-36, 2023.
[18] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint arXiv: 1412.3351 2014.
[19] Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning. pages 1263-1272. PMLR, 2017.
[20] Andrea Cini, Ivan Marisca, and Cesare Alippi. Filling the g ap_s Multivariate time series imputation by graph neural networks. In International Conference on Learning Representations, 2022.
[21] Vijay Prakash Dwivedi, Chaitanya K. Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research, 24(43):1-48, 2023, URL http://jmlr org/ ptppppaapp//a////// 256677 .tmml
[22] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020.
[23] Kenneth B Kahn and Charles W Chase. The state of new-product forecasting. Foresight: The International Journal of Applied Forecasting, (51), 2018.
[24] Hoang Nt and Takanori Maehara. Revisiting graph neural networks: All we have is low-pass filters. arXiv preprint AKKiv/995509955 2019.
[25] Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In International Conference Learning Representations, 2020.
[26] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in neural information processing systems, 16, 2003.
[27] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
[28] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of International Conference on Learning Representations, 2013.
[29] Bo Yang, Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. Towards k-means m-mans--riendly spaces: Simultaneous deep learning and clustering. In international conference on machine learning, pages 3861-3870. PMLR, 2017.
[30] C Maddison, A Mnih, and Y Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations, 2017.
[31] Alvaro Sonzaez- Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International conference on machine learning, pages 8459-8468. PMLR, 2020.
[32] Zijie Huang, Yizhou Sun, and Wei Wang. Learning continuous system dynamics from ireeuulally-smpple partial observations. Advances in Neural Information Processing Sys- tems, 33:161777-61177, 2020.
[33] Song Wen, Hao Wang, and Dimitris Metaxas. Social ode: Multi-agent trajectory forecasting with neural ordinary differential equations. In European Conference on Computer Vision, pages 217-233, Springer, 2022.
[34] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Inductive repre- sentation learning on temporal graphs. International Conference on Learning Representations, 2020.
12

---

[35] Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth. and Pascal Poupart. Representation learning for dynamic graphs: A survey. The Journal of Machine Learning Research, 21(1):2648-277 2720, 2020.
[36] Antonio Longa, Veronica Lachi, Gabriele Santin, Monica Bianchini, Bruno Lepri, Pietro Lio, franco scarselli, and Andrea Passerini. Graph neural networks for temporal graphs: State of the art, open challenges, and opportunities. Transactions on Machine Learning Research, 2023.
ISSN 2835-8856. URL (tIphttts:// /ptp::///eppeeeeieeeeee wwt./oorummiddpeeeioieee
[37] Daniele Grattarola, Daniele Zambon, Lorenzo Livi, and Cesare Alippi. Change detection in graph streams by learning graph embeddings on constant-curvature manifolds. IEEE Transac- lions on neural networks and learning systems, 31 (1))))856-1866)) 1869,, 2019.
[38] Benjamin Paassen, Daniele Grattarola, Daniele Zambon, Cesare Alippi, and Barbara Eva Hammer. Graph edit networks. In International Conference on Learning Representations, 2020.
[39] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In Proceedings of the 28th International Joint Conference Artificial Intelligence, pages 1907-1913, 2019.
[40] Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, Irwin King, and Dit Yan Yeung. Gaan: Gated attention networks for learning on large and spatiotemporal graphs. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, 2018.
[41] Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. Gman: A graph multi- attention network for traffic prediction. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 1234-1241, 2020.
[42] Zonghan Wu, Da Zheng, Shirui Pan, Quan Gan, Guodong Long, and George Karypis. Tra- versenet: Unifying space and time in message passing for traffic forecasting. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[43] Hongjic Chen, Ryan A. Rossi, Kanak Mahadik, Sungchul Kim, and Hoda Eldardiry. Graph deep factors for forecasting with applications to cloud resource allocation. Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD '21, page 106-17 116, New York. NY, USA, 2021. Association for Computing Machinery. ISBN 978 1503833325 doi: 0441144/344454834467733 URL https::///oii.. org 10. 04445//445488.. 3467357
[44] Yuyang Wang, Alex Smola, Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski. Deep factors for forecasting. In International conference on machine learning, pages 6607-6617. PMLR, 2019.
[45] Chao Shang and Jie Chen. Discrete graph structure learning for forecasting multiple time series. In Proceedings of International Conference on Learning Representations, 2021.
[46] Victor Garcia Satorras, Syama Sundar Rangapuram, and Tim Januschowski. Multivariate time series forecasting with latent graph inference. arXiv preprint arXivv2203.0344 03222333333,, 2022.
[47] Ivan Marisca, Andrea Cini, and Cesare Alippi. Learning to reconstruct missing data from spatiotemporal graphs with sparse observations. Advances in Neural Information Processing Systems, 04:32069-32083 2022.
[48] Zezhi Shao, Zhao Zhang, Fei Wang, Wei Wei, and Yongjun Xu. Spatial-temporal identity: A simple yet effective baseline for multivariate time series forecasting. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, page 4454 4458, 2022.
[49] Xueyan Yin, Feifan Li, Yanming Shen, Heng Qi, and Baocai Yin. Nodetrans: A graph transfer learning approach for traffic prediction. arXiv preprint 2:X1v 2022.
[50] Rajat Sen, Hsiang-Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. Advances neural information processing systems, 32, 2019.
13

---

[51] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint orXiv:1803.0127/ 2018.
[52] Daniele Zambon and Cesare Alippi. AZ-whiteness test: test for uncorrelated noise on spatio-temporal graphs. Advances in Neural Information Processing Systems, 35: 1975-11966, 2022.
[53] Elvin Isufi, Andreas Loukas, Nathanacl Perraudin, and Geert Leus. Forecasting time series with VARMA recursions on graphs. IEEE' Transactions on Signal Processing, 67(18):4870-4888 2019.
[54] Commission for Energy Regulation. CER Smart Metering Project - Electricity Customer Behaviour Trial, 2009-2010 [dataset]. Irish Social Science Data Archive. SN: 0012-00, 2016. URL https::////ww ucd. ie/ wwwss/ddaaaaaaaaa commi wtt.:/oooooncccoooooooooooooocct air.aaaiiiicoorrrrrrrrrrraaaaaaa
[55] Yu Zheng, Xiuwen Yi, Ming Li, Ruiyuan Li, Zhangqing Shan, Eric Chang, and Tianrui Li. Forecasting fine -aaingr air quality based on big data. In Proceedings the the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 2267-2276, 2015.
[56] Shengnan Guo, Youfang Lin, Huaiyu Wan, Xiucheng Li, and Gao Cong. Learning dynamics and heterogeneity of spatial-temporal graph data for traffic forecasting. IEEE Transactions on Knowledge and Data Engineering,
2021.
[57] Guido Van Rossum and Fred L. Drake. Python 3 Reference Manual. CreateSpace, Scotts Valley,
CA, 2009. ISBN 1441412699
[58] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[59] William Falcon and The PyTorch Lightning team. PyTorc Lightning, 3 2019. URL https //github. cowm/PPyTorccliighnninng. http:/yyotocchhhieee -ighhhingggtiiiggngg
[60] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint 1903. 2019.
[61] Andrea Cini and Ivan Marisca. Torch Spatiotemporal, 3 2022. URL httth::/tttthhihhe com/ Torrhhpattiotemporal//taa
[62] Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array programming with numpy. Nature, 585(7825):357-36. 2020.
[63] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[64] Chao Chen, Karl Petty, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia. Freeway performance measurement system: mining loop detector data. Transportation Research Record, 1748(1):96-102, 2001.
[65] Andrea Cini, Ivan Marisca, Filippo Maria Bianchi, and Cesare Alippi. Scalable spatiotemporal graph neural networks. In NewrlPS 2022 Temporal Graph Learning Workshop, 2022.
[66] Xiuwen Yi, Yu Zheng, Junbo Zhang, and Tianrui Li. St-mvl: filling missing values in geo- sensory time series data. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, 2016.
[67] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. 3nd International Conference Learning Representations, ICLR 2015, 2015.
14

---

Appendix
The following sections provide additional details on the experimental setting and computational platform used gather the results presented the paper. Additionally, we also include supplementary empirical results and analysis.
A Experimental setup
Experimental setup and baselines have been developed with Python 57 by relying on the following open-source libraries:
PyTorch [58]
Py'oorch Lightning [59]
PyTorch Geometric 660
Torch Spatiotemporal
numpy 62
scikit-learn 63
Experiments were run on a workstation equipped with AMD EPYC processors and four NVIDIA
RTX A5000 GPUs. The code needed to reproduce the reported results is available online"
A.1
Datasets
Table 7: Statistics of datasets used in the experiments.
DATASETS
Type
Time steps
Nodes
Edges
Rate
GPVAR
Undirected
30,000
120
199
N/A
GPVAR-L
Undirected
30,000
120
199
N/A
METR-LA
Directed
34,272
207
1515
5 minutes
PEMS-BAY
Directed
52,128
325
2369
5 minutes
CER-E
Directed
25,728
485
4365
30 minutes
AQI
Undirected
8,760
437
2699
hour
PEMS03
Directed
26,208
358
546
5 minutes
PEMS04
Directed
16,992
307
340
5 minutes
PEMS07
Directed
28,224
883
866
5 minutes
PFMS08
Directed
17,856
170
277
5 minutes
In this section, we provide additional information regarding each one of the considered datasets.
Tab. reports summary of statistics.
GPVAR(-L) For the GPVAR datasets we follow the procedure described in Sec. 7 to generate data and then partition the resulting time series in 11%%/100/22 /20% splits for training, validation and testing, respectively. For GPVAR-L the parameters of the spatiotemporal process are set as
2.5 -2.0 -0.51
U (-2,2)
3.0
0.0.
~ NO diag(o²)),
0.4.
Fig. showos the topology of the graph used as support to the process. In particular, we considered a network with 120 nodes with 20 communities and added self-loops the graph adjacency matrix.
https//gghhhooccmmorgghhhhhhhhee Learning Sroup//aming-ooallfffffee.
15

---

Figure 2: GPVAR community graph. We used a graph with 20 communities resulting in network with 120 nodes.
Spatiotemporal forecasting benchmarks We considered several datasets coming from relevant application domains and different problem settings corresponding to real-world application scenarios.
Traffic forecasting We consider two popular traffic forecasting datasets, namely METR-LA and PEMS-BAY [6), containing measurements from loop detectors in the Los Angeles County Highway and San Francisco Bay Area, respectively. For the experiment on transfer learning, we use the PEMS03, PEMS04, PEMS07, and PEMS08 datasets from Guo al. [56] each collecting traffic flow readings, aggregated into 5-minutes intervals, from different areas in California provided by Caltrans Performance Measurement System (PeMS) [64].
Electric load forecasting We selected the CER-E dataset [34]. collection of energy consumption
readings, aggregated into 30-minutes intervals, from 485 smart meters monitoring small and medium-sized enterprises.
Air quality monitoring The AQI (55 dataset collects hourly measurements of pollutant PM2.5 from 437 air quality monitoring stations in China. Note that all of these datasets have been previously used for spatiotemporal forecasting and imputation
65).
For each dataset, we obtain the corresponding adjacency matrix by following previous works 20 6. 56). We use exogenous variables sinusoidal functions encoding the time of the day and the one-hot encoding of the day of the week. For datasets with an excessive number of missing values, namely METR-LA and AQI, we add corresponding value has been imputed.
exogenous variable also binary mask indicating if the
We split datasets into windows of W time steps, and train the models to predict the next H observa- tions. For the experiment Tab we set W 12, H for the traffic datasets, W 48, H for CER-E, and W 24, H 3 for AQL Then, for all datasets except for AQI, we divide the obtained windows sequentially into 70% 10% splits for training, validation, and testing, respec- tively. For AQI instead, we use as the test set the months of March, June, September, and December, following Yi et al. [66].
A.2 Architectures
In this section, we provide a detailed description of the reference architectures used in the study.
Reference architectures We consider 4 simple STGNN architectures that follow the template given (Eq. all the models, we use as ENCODER (Eq. 9 simple linear layer s.t.
h 4,0 Wene x I|u |
(21)
and as DECODER (Eq. 11 a 2 layer module structured as
WI. (Wacch"") 1=1....
(22)
where Wene. Wase and WI...., WH are learnable parameters. In hybrid models with local EN- CODER and/or DECODER, we use difffernnt parameter matrices for each node, i.c..
4.0 W [ai_1] [[]]]]]]]]]
(23)
cet Wis (aach! I=I.....
(24)
When instead node embeddings V E Nxd are used to specialize the model, they are concatenated the modules' input as
1,0 h
Wene Wen[arf-1 || 1 || [|]
(25)
EL+ WiE (Wace [h ||or)) h=
(26)
16

---

For the TTS architectures, we define the STMP module (Eq.
10 as a node-wise GRU
18
ri [ [ II (1))
(27)
0 - a [wi ||
(28)
c tanh tanh(W; [
u || @ m=))),
(29)
h of o h +1 of) oc.
(30)
The GRU is then followed by L MP layers
H MP (H), A) 1= --1.
(31)
We consider two variants for this architecture: TTS-IMP, featuring the isotropic MP operator defined
in Eq. 3 and TTS-AMP, using the anisotropic MP operator defined in Eq. 7 Note that all the parameters in the STMP blocks are shared among the nodes.
In T&S models, instead, we use a GRU where gates are implemented using MP layers:
MP
h
1 )A) 4.1
(32)
1
MMP).
([ [|
|- .4)).
(33)
d. tanh MMP] [A ||r o ( .4)). C
(34)
h of h + (1 o. o
(35)
Similarly to the TTS case, we indicate T&S-IMP the reference architecture in which MP operators are isotropic and T&S-AMP one featuring anisotropic MP.
Baselines For the RNN baselines we follow the same template of the reference architectures (Eq. III but use a single GRU cell as core processing module instead of the STMP block. The global-local RNN, then, uses node embeddings at encoding shown in Eq, 25 whereas NocalRNNs have different sets parameters for each time series. The FC-RNN architecture, instead, takes as input the concatenation all the time series as if they
single multivariate one.
Hyperparameters For the reference TTS architectures we use a GRU with single cell followed by 2 seassage-psssing layers. In T&S case number of neurons in each layer is set to 64 and the embedding size to 32 for all the reference architectures in all the benchmark datasets. Analogous hyperparameters were used for the RNN baselines. For GPVAR instead, we use 16 and as hidden and embedding sizes, respectively. For the baselines from the literature, use the hyperparameters used in the original papers whenever possible.
single graph recurrent convolutional cell. The
For GPVAR experiments we use batch size of 128 and train with early stopping for maximum of 200 epochs with the Adam optimizer (67 and a learning rate of 0.01 halved every epochs.
For experiments in Tab. | and 4 we instead trained the models with batch size 64 for maximum of 300 epochs each consisting of maximum 300 batches. The initial learning rate was set 0.003 and reduced every 50 epochs.
Transfer learning experiment In the transfer learning experiments we simulate the case where pre-trained spatiotemporal model for traffic forecasting is tested on new road network. We fine-tune on each of the PEMS03-08 datasets models previously trained on the left-out datasets. Following previous works [56), we split the datasets into 60% /0% 200 for training, validation, and testing, respectively. We obtain the mini-batches during training by uniformly sampling 64 windows from the 3 training sets. To train the models we use a similar experimental setting of experiments in Tab, 4 decreasing the number of epochs to 150 and increasing the batches per epoch 500. We then assume that 2 weeks of readings from the target dataset are available for fine-tuning, and use the first week for training and the second one as the validation set. Then, we test fine-tuned models on the immediately following week (Tab. 5). For the global model, we either tune all the parameters or none of them (zero-shot setting). For fine-tuning, we increase the maximum number of epochs to 2000 without limiting the batches processed per epoch and fixing the learning rate to 0.001. At the end of every
17

---

Table 8: Perfomance (MAE) of TMS-IMP and TTS-AMP variants and number of associated trainable
parameters in SEMS-BAY (5-rum average).
METR-LA
PEMS-BAY (# weights)
METR-LA
PEMS-BAY (# weights)
TTS-IMP
TTS-AMP
Global TTS
3.35 20.01
1.72 1000
4.71 10*
3.27 1001
1.68 10.00
6.41 x 10
Global local TTS
Weights
Encoder
3.15 +0.01
1.66 +001
2.7
3.12 +001
1.64 +0,00
2.92x 10
Decoder
3,09 10.01
1.58 1.000
3.00x
3.09 10.01
1,58
1,,00
3.17x102
Enc. + Dec.
3.16 20.01
1.70 2001
5.28x
3.14 10.01
1.69
=0.01
5,45 10
Embed.
Encoder
3.08 2:0.01
1.58 1:000
5.96x
3.05 +0.02
1,58
00001
7.66x10*
Decoder
3.13 10,00
1.60
1.000
5,96x 10
3.12 10.01
1.60
10000
7.66x10'
Enc. Dec.
3.07 +0.01
+:000
6.16x10*
+0.01
1.59
=0,01
7.86x10*
FC-RNN
3.56 1: 0.00
2.32
3.04x 10
LocaIRNNs
3.69 +0:00
1.91 +0000
1.10x10"
training epoch, we compute the MAE on the validation set and stop training if it has not decreased in the last 100 epochs, restoring the model weights corresponding to the best-performing model. For the global-local models with variational regularization on the embedding space, during training, we set B = 0.05 and initialize the distribution parameters as
H4 A U (-0.01,0.0) 0.01),
O, = 0.2.
For fine-tuning instead, we initialize the new embedding table V E R N'x, xd, as
V ~
where A Vd, and remove the regularization loss. For the clustering regularization, instead, use K 10 clusters and regularization trade-off weight À = 0.5. We initialize embedding, centroid, and cluster assignment matrices
V ) C~U(-A,A) S S~u (0.1)
respectively. For fine-tuning, we fix the centroid table C and initialize the new embedding table V' E R xd, and cluster assignment matrix S' E Nx following an analogous procedure. Finally, we increase the regularization weight À to 10.
B Additional experimental results
Local components Table 8, an extended version of Table shows the performance of reference architecture with and without local components. Here consider TTS-IMP and TTS-AMP, together with FC-RNN (a multivariate RNN) and LocaIRNNs (local univariate RNNs with a different set of parameters for each time series). For the STGNNs. we consider a global variant (without any local component) and global-local alternatives, where we insert node-specific components within the architecture by means of (1) different of weights for each time series (light brown block) or (2) node embeddings (light gray block). More precisely. we show how performance varies when the local components are added in the encoding and decoding steps together, or uniquely in one of the two steps. Results for TTS-AMP are consistent with the observations made for TTS-IMP, showing that the use of local components enables improvements up to 7% w.r.t. the fully global variant.
Transfer learning Tables to 12 show additional results for the transfer learning experiments in all the target datasets. In particular, each table shows results for the reference architectures w.r.t. different training set sizes (from 1 day to 2 weeks) and considers the settings where embeddings are fed to both encoder and decoder or decoder only. We report results on the test data corresponding to the week after the validation set but also on the original test split used in the literature. In the last columns of the table, we also show the performance that one would have obtained 100 epochs after the minimum in the validation enror curve; the purpose of showing these results is to hint at the performance that one would have obtained without holding out week of data for validation. The results indeed suggest that fine-tuning the full global model is more prone overfitting.
18

---

Table Forecasting error (MAE) on PEMS03 in the transfer learning setting (5 runs average).
Model
Testing subsequent week
Testing the standard split
Testing 100 epachs alter validation min.
TTS-IMP
2 wookss
1 week
days
day
2 weeks
week
days
day
2 weekss
wook
3 days
1 day
Global
14
15,30ame
16.26a00s
16.65
16.1sans
16.36grs
16,95
17.39am
16. 30300mm
16 16.5e
17.62a32
18.33
Exe Dac.
Brabeddingg
14
14.64gmm
15.87300
16.78
16.03aes
16.12
17.18
17.825415
16.99
16.66
17.28g318
17.94gmn
Varistioaal
14.50ga
14.56gmm
15:40gos
15
15,02a v
15.70gatt
16.33gn
16.5220n
15.70gm
15,70ge
o
16.35gt
16.54gm
Clustering
14,88
14.60.00
15,67
16.53
15.681049
15.21.200
16.76
17,36
15.70
15.74e00
16.80
12178
Embedkings
a4.79.am
14.8kgm
15449u00
16
16.06
16.12amm
16.74zm
17,25
16.08
16,13
16.75
17.29
DRc.
Variational
15:33-am
15.83
16.15sm22 16.00
16.20
16.60
16.75
16.15
16.20
16.60ssa-
16.76:we
Clustering
14.96-ax
saee
15.88
16.25 16.29
16.72
16.87
16.28
16.91 16.191
16.93
Table 10: Forecasting error (MAE) on PEMS04 in the transfer learning setting (5 runs average).
Model
Texting subsequent week.
Testing the standard split
Testing
100 epuchs after valiidation min.
TTS ITS-IMP
2 weeks
week
days
day
2 weeks
week
days
day
2 weeks
wook
3 days
day
Global
20.86.am
21.59amm
21.84aus 22 22.26
20.79aaaz
21. 21.68asto
22.10
22.59am
20.89
21.97aaas
22.88
23 23.85
Eve Deoc
Feshedkinngs
1 x. 20.27g01
21,03a 21 21.99
19.87a007
20.27ga0r
21,20gars
22,38g
19.88gmm
20.29
21.78g11
22.46
Varistional
19.9
20.19gm0
20.71ame
21 15
19.92
20.23amm
20.82a00
21 46200
19.92gmm
20.23ae15
20.82at
21.47
Clustering
19.69.a>
19.91
20/48100
21a9
19.70amms
19.96am1
20.62zme
22.28
19.72
19.97200
20.65.aa
22.29
Finheskings
20.10_a 20.27gn0
20.87200
21 24
18g88
20.39geas
21.01g00 goo
21.70ga0e
20. 19gu9
20401ges
21.03ga
21.74gaa
DRc-
Variational
20.79 a.e 20.94uue
21.23ssur
21.51
20.94aa2s
21.10aaas
21.40maue
21.7610u
20.94
21.101eas
21.40
21.77
Clustering
20.19 20.45
20.603000
21.08
20.27a629
20,56a310
20.81
21.281005
20.75
20.78aess
20.89
21.35.005
Table 11: Forecasting error (MAE) on PEMS07 in the transfer learning setting (5 runs average).
Model
Testing suisequentt week
Testig the standard split
Testing 100 epuchs aller validation min.
TTS-IMP
2 wooks
I weck
days
day
2 weeks
week
days
day
2 weekss
wook
3 days
day
Global
22
23.82gm0
24.52300
25
22.64
23.58gam
24.20g00x
25.04gm0
22.72gm0
23.72a00
24.45gmm
25.48gmm
Eve Dac.
krabedkings
21.68
22.23
23.5410
26.11
22 2010n
22.77
24.17
26.79100
22.11
22.784e25
24.21
26,82
Varistional
22.05
22.48g00
28.25g008
24 24.40
22 22.18
22.59.a0
23.44gm1
24.62300
22.18
22.59g604
23.44g110
24.62g00
Clustering
21.75
22.16suun
23.361031
26.
22.03
22.52a110
23.85
27.12
22.03
22.55xea1
23.85
27
Fmbbedkings
22,50_a14
22.82gm
23.50gn
24 80
22.68a0
23.13gr00
24.04gm
25.4
22.69gm0
23.1428
34.04
25.43gmm
DBc.
Varistional
24.32aate
24.6020te
25.12amm
25
24.25an34
24.60a115
25.16ann
25.562m11
24.25am0
24.60ze11
16,16s
25.57am11
Clustering
23.02:am
23.53
24/42sass
24.87
23.18an29
23,77ax
24.66:a11
25.24
23,91
24.10aeas
24.73a1m
25.27:xum
Table 12: Forecasting error (MAE) on PEMS08 in the transfer learning setting (5 runs average).
Model
Testing
anbsequent week
Testing on the standard spli
Testing
100 epochs after validation min.
TTS-IMP
2 wookss
weok
days
day
2 weeks
weok
days
day
2 weoks
wook
3 days
day
Global
15
15.90gmn
16.872m0
17.59
15336a9s
15.71am
16.71
1741200
15,47200
15.87ass
17.58
18.46
Eive Doc
Emabeddings
15.45
15.45.00e
16.34300
17.15302
15.34
15.32gs0
16,27gmm
17.11g00
15.32200
15.30geca
16.27a5a5
17.13
Variational
15.34
18.41g00
15.83g0
16.32
15:21a0es 15:27
15.70gm
16.19
15.21
15:272005
15 15.69
16, 1920
Clustering
15.4
15:41gmm
15.06200
16 19
15.77
15:27amm
15.90gme
16.092ms
15.30gno
15:28
15900
17.02
kmbeedingg
15.72-436
15,74g000
16.41300
16.97442
15.6130ss
15.61am0
16.33s00
16,00.00
15.61 so0e
15,61144s
16.35
16.92.016
DBc.
Variational
16.31at0
16.33eate
16.53ssas
16.74am22
16.13an29
16.14as14
16.34aa
16.55
16.14
16.35
16.56axm
Clustering
15.81-au
15.92
16.11100
16.5510.0
15.70
15.77auu
15.97
16.43imn
15,98
15:90
16.01
16.45
19


Reviews:
[{"review_id": "q7mW8gGcTm", "reviewer": "Reviewer_K1NT", "summary": "In this paper, the authors explore the influence of locality and globality in graph-based spatiotemporal forecasting architectures. Existing spatiotemporal models are global trained on multiple multivariate timeseires, which can capture the strong dependency among individual nodes in a network. Standard local models such as RNNs learn each timeseries independently which lost the interaction information with other nodes, but are fitted solely on each individual trajectory resulting in good short-term prediction performance. \n\nDirectly combine the predictions from global and local models would result in a large number of model parameters (introduced by individual models). The authors instead propose to use a learnable embedding vector to represent the locality for each node and incorporate it in the GNN message passing procedure. The guide the learning process for such node embeddings, the authors further proposed two regularization terms to make the model more generalizable, with the assumption that the underlying dynamics of nodes within the same network topology would not differ too much.\n\nExperiment result over several benchmark datasets show the proposed method is able to make better prediction results than compared baselines.\n\n", "questions": "1. Can the authors provide running time comparison during the testing stage, as the proposed method would need to retrain the model on unseen(new) nodes.\n2. Can the authors visualize some of the learned local node embeddings and show some case study to interpret their semantic meanings?", "limitations": "The authors have not discuss the limitations of their model.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "1. The writing for this paper is very easy to follow\n2. The idea to inject local information into existing global spatiotemporal models is interesting.\n3. The experiments are comprehensive, though some baselines are missing.", "weaknesses": "1. My major concern is the contribution/novelty of this paper. The authors propose to learn a node embedding to mimic the role of local models such as RNN trained on each individual timeseries. First of all, the node embeddings are static, whereas the output of RNN models are dynamic. Those local features for each individual node can changes over time which can be well-captured by any local models. Secondly, the learnable node embeddings seem to me are similar to those exogenous factors specific to each node, how to guarantee the learned embeddings would not serve as the same role as those exogenous factors? Finally, learning these embeddings would make the whole model not able to perform inductive tasks. When a new node/timeseries comes in, one needs to retrain the model instead of directly use the model to do the inference, opposed to existing spatiotemporal GNNs.\n\n2. Also there are some missing baselines in terms of spatiotemporal GNNs, such as continuous graphODE approaches [1][2] and other discrete methods [3].\n\n[1] Huang, Zijie, Yizhou Sun, and Wei Wang. \"Learning continuous system dynamics from irregularly-sampled partial observations.\" Advances in Neural Information Processing Systems 33 (2020): 16177-16187.\n\n[2] Song Wen, Hao Wang, and Dimitris Metaxas. 2022. Social ODE: Multi-agent Trajectory Forecasting with Neural Ordinary Differential Equations. In Computer Vision–ECCV 2022: 17th European Conference.\n\n[3]Sanchez-Gonzalez, Alvaro, et al. \"Learning to simulate complex physics with graph networks.\" International conference on machine learning. PMLR, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Before providing point-by-point answers, we’d like to remark that the main contribution of our paper is not in introducing an architecture but rather in studying a crucial aspect of graph-based forecasting, i.e., the interplay of local and global aspects of time series forecasting in such architectures. We would appreciate it if the reviewer could reconsider their evaluation of the novelty/contributions of our paper in light of this.\n\n> The authors propose to learn a node embedding to mimic the role of local models such as RNN trained on each individual timeseries. First of all, the node embeddings are static, whereas the output of RNN models are dynamic.\n\nThere might be some misunderstandings here. Node embeddings are learnable parameters and, as such, static once trained; the same holds true for the learnable parameters of an RNN, which are static as well; what is dynamic, instead, is the output of the models. Having node embeddings, rather than a fully local model, would only imply that most of the parameters involved in the dynamic processing of the data will be shared among time series. In other words, assuming the encoder of a global-local model is an RNN, the embeddings are passed as an additional input to provide localization, while the RNN parameters remain shared for all time series.\n\n> The learnable node embeddings seem to me are similar to those exogenous factors specific to each node, how to guarantee the learned embeddings would not serve as the same role as those exogenous factors?\n\nEmbeddings are indeed used similarly to exogenous features and indeed exogenous features can be used to localize predictions. However, as specified in the paper, such features are often not available in practice, and node embeddings are far more flexible as the encoding is learned end-to-end, thus becoming part of the model’s parameters. Furthermore, structuring the embedding space allows for regularizing the local components of the model (as shown in Section 5.1 and the transfer learning experiments).\n\n> Learning these embeddings would make the whole model not able to perform inductive tasks. When a new node/timeseries comes in, one needs to retrain the model instead of directly use the model to do the inference, opposed to existing spatiotemporal GNNs. \n\nYes, that is correct, adding learnable node embeddings makes the model not inductive. In this respect, a significant paper contribution is in showing that – with the proper regularizations – hybrid global-local models based on node embeddings can be adapted to new nodes using only a few observations, without training from scratch the full model. Finally, note that most of the state-of-the-art STGNN architectures are actually not inductive as they rely on some form of node identification (see, e.g., Graph Wavenet, AGCRN, etc.). We show that with our simple approach, we can get similar performance and that the resulting model can be easily transferred by fine-tuning only a very small number of parameters, drastically reducing sample complexity.\n\n> Also there are some missing baselines in terms of spatiotemporal GNNs, such as continuous graphODE approaches [1][2] and other discrete methods [3].\n\nIn the empirical section, we focused on SOTA architectures for the benchmarks and types of problems considered in the study. However, even if the baselines suggested by the reviewer have been developed in a different context, we think it is indeed worth discussing them in the related works section. We will do so in the revision of the paper, thanks for the suggestion.\n\n> Q1 Can the authors provide running time comparison during the testing stage, as the proposed method would need to retrain the model on unseen(new) nodes?\n\nIn the transductive setting, the models have the same computational costs as embedding results only in a small increase in the number of features and, as such, their impact w.r.t. time complexity is negligible.\nIn the transfer learning setting, the computational cost is again exactly the same at inference time. The only overhead is the cost of fine-tuning the model (no full re-training is needed), yet such cost does not depend at all on the methodology we propose, but on the complexity of the model being fine-tuned and on the number of observations available. Furthermore, fine-tuning the entire model – rather than the embeddings alone – can be more computationally expensive. Finally, note that fine-tuning needs to be performed only once and that the performance improvement w.r.t. the zero-shot model is very large, even for inductive models.\n\n> Q2 Can the authors visualize some of the learned local node embeddings and show some case study to interpret their semantic meanings?\n\nFig. 1 in the paper provides a visualization of the time series associated with different clusters of embeddings and, commenting on the figure, in Sec. 7 we discuss how the emerging clusters elucidate the role of embeddings as local components in the forecasting architectures. In addition to that, we include a t-SNE visualization of the learned embeddings with the different regularization mechanisms and different settings in the pdf attached to the rebuttal. The results confirm how regularization allows for structure to emerge in embedding space.\n\n> The authors have not discuss the limitations of their model.\n\nLimitations are discussed throughout the paper. In particular, we highlight several times the limits of the hybrid global-local model in the inductive settings and provide an in-depth discussion on the issue in Section 5.1. We will improve the discussion on the limitations of our study and include a comment on possible future works in the conclusions of the paper. \n"}, {"review_id": "hUvmojRB5A", "reviewer": "Reviewer_Zmtu", "summary": "This paper examines the interaction between global and local effects in graph-based spatiotemporal forecasting. It addresses the limitations of a single global model by introducing a framework that incorporates trainable node embeddings into graph-based architectures. This framework enables the learning of specialized components and combines the benefits of shared message-passing layers with node-specific parameters. Additionally, the framework facilitates model transfer to new node sets. The paper offers empirical evidence and provides guidelines for adapting graph-based models to the dynamics of each time series to improve prediction accuracy.", "questions": "I have to admit that I do not fully understand the paper, though I tried to read through the paper multiple times, so it may be helpful for me to understand the key contributions of the paper if the authors can provide some information of the following:\n\n1. **What is the guideline and main takeaway messages for the audience to design models for spatial-temporal graph learning?** I feel a little lost in a vast amount of information and empirical observations.\n2. **How do we know which design is the best for an application? Isn't it application specific?** If we want to know whether the \"local\" or \"global\" components in a STGNN are more important for an application, we may want to try it out or are there any ways to know it beforehand? To the best of my understanding, it could be very different from applications to applications, since global information is more important for some of them while for others local information is more important.\n3. **Could T&S-AMP design be a generic to go choice?** If we do not know the importance of global or local effect in an application beforehand, can T&S and AMP be an to-go option? Many well-established GNNs for dynamic graphs fall in this category, for example, use attention in MP (AMP) and use alternating message-passing layers and recurrent layers (T&S), e.g., Graph Recurrent Attention Networks (GRAN). ", "limitations": "No limitations and negative societal impacts are left unaddressed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "It is nice to see a paper that investigates the attribution of \"local\" and \"global\" learning in modeling spatial-temporal graphs. The evaluation is very comprehensive and the paper is very informative. It may have great impact that can benefit the broad community that researches on spatial-temporal graphs.", "weaknesses": "The paper tries to answer a set of very big questions (\"local\" vs \"global\"), which I feel could be too hard to find a concrete answer in a 9 pages conference paper.\n\nSimilar questions can be asked for GNNs as well: Is message-passing more important or the node feature encoding more important? Should I go fully inductive like GCN, GraphSAGE? Or I just stick to non-inductive GNNs? Is the isotropic message-passing enough like vanilla GCNs, or do I need anisotropic message-passing like graph attention networks (GAT)? Do I interleave the MP layers with node encoding layers like most GCNs do? Or I should stack multiple node encoding layers before doing message-passing?\n\nI feel it is a little too overwhelming to answer all these questions at once. I really appreciate the authors' efforts to investigate these questions, but it feel less convincing when it fits into a 9-pages conference paper, that each claim will be supported by less empirical evidences. Sometimes I will question that, how does this claim hold for other applications when the nature of a problem changes, would the conclusions change?\n\nWhat's more challenging is that, these questions seem to not having a general answer that hold for all the applications, making it especially hard to draw conclusions by merely relying on empirical studies (or you will need a lot of experiments across much more domains). \n\nIn general, it is overall a technical solid paper and an ambitious one as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and useful comments. We are happy that you found our paper interesting, please find our point-by-point answers below.\n\n> I feel it is a little too overwhelming to answer all these questions at once. [...] Sometimes I will question that, how does this claim hold for other applications when the nature of a problem changes, would the conclusions change?What's more challenging is that, these questions seem to not having a general answer.\n\nThe paper tackles complex problems from different perspectives. In particular, we understand that the presentation of many different template architectures for STGNNs without providing guidelines on which architecture should be preferred in general might generate some confusion. However, we believe that introducing the many possible design choices is instrumental in showing that the issues related to global and local aspects (which are the focus of the paper) appear across the full spectrum of architectures used in practice. In such regard, we believe that the paper succeeds (through experiments on both synthetic and real-world datasets) in showing that locality and globality are crucial aspects in graph-based forecasting and in showing how local components can be effectively and efficiently introduced in otherwise global architectures.\n\n> Q1 What is the guideline and main takeaway messages for the audience to design models for spatial-temporal graph learning?\n\nThe main takeaway messages are summarized in the 5 points in lines 64-75 in the paper.  In particular, 1) local components can be crucial to obtain accurate predictions in spatiotemporal forecasting; 2) node embeddings can amortize the learning of such components in otherwise global architectures; 3) hybrid local-global STGNNs with node embeddings can capture local effects with contained model capacity and reasonably long input window; 4) node embeddings make adapting models to different scenarios more efficient; 5) structuring the embedding space allow for regularizing the forecasting model.\n\n> Q2 How do we know which design is the best for an application? Isn't it application specific? If we want to know whether the \"local\" or \"global\" components in a STGNN are more important for an application, we may want to try it out or are there any ways to know it beforehand?\n\nIndeed it is application-specific. While fully global models are more flexible, hybrid architectures often perform better in practice. As there is no definitive answer, we suggest trying both architectures to decide whether including the local components is worth the compromise in flexibility. That being said, from our experience in real-world applications, adding local components consistently leads to better performance.\n\n> Q3 If we do not know the importance of global or local effect in an application beforehand, can T&S and AMP be an to-go option? \n\nA global-local T&S-AMP model is indeed a solid choice if the final performance at task is the only concern. However, T&S-AMP models are more computationally demanding than TTS-ISO models, which can nonetheless provide good performance. The practitioner should decide how to balance performance at task and computational costs and should be aware of the impact on the final performance of components that take local effects into account. The latter aspect is, as already mentioned, one of the main takeaways of the paper.\n"}, {"review_id": "wfRTCk2g3l", "reviewer": "Reviewer_jVAx", "summary": "This paper presents a methodological framework aimed at rationalizing the inclusion of trainable node embeddings in STGNNs for spatiotemporal forecasting applications. The authors examine the interplay between globality and locality in graph-based spatiotemporal forecasting and provide insights and guidelines for specification design. The paper demonstrates how incorporating trainable node embeddings in STGNNs can effectively combine the advantages of shared message-passing layers with node-specific parameters, while efficiently transferring the learned model to new node sets. The proposed framework is supported by empirical evidence and offers a principled approach for accommodating various node embeddings.", "questions": "The instructions in line 43 and lines 179-180 may appear contradictory. Could you provide a more detailed explanation?", "limitations": "see weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The authors investigate the interplay between globality and locality in graph-based spatiotemporal forecasting, resulting in five major findings.\n2. The paper illustrates how including trainable node embeddings in STGNNs can effectively combine the benefits of shared message-passing layers with node-specific parameters and efficiently transfer the learned model to new node sets.\n", "weaknesses": "1. The paper is not well-organized, making it difficult to understand the main points and arguments presented.\n2. The proposed framework adopts the TTS model as an STGNN, but some important TTS methods are not discussed in the related work, such as [1] and [2]. \n[1] Jianfei Gao and Bruno Ribeiro. On the equivalence between temporal and static equivariant graph representations. In International Conference on Machine Learning, pages 7052–7076. PMLR, 2022. \n[2] Da Xu, etc. Inductive representation learning on temporal graphs. In ICLR 2020.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review. Please find our answers below.\n\n> The paper is not well-organized, making it difficult to understand the main points and arguments presented.\n\nWe did our best to make the structure of the paper easy to follow, an aspect that was appreciated by the other reviewers. Currently, the presentation is structured as follows: 1) template global architectures; 2) issues with fully global models; 3) introduction of hybrid global-local architectures (with empirical evidence of the benefits); 4) node embeddings (i.e., more efficient hybrid architectures); 5) overcoming the limitations of hybrid models in transfer learning settings; 6) complete empirical results. There’s always room for improvement and we are open to making adjustments based on more specific feedback that the reviewer wishes to provide.\n\n> The proposed framework adopts the TTS model as an STGNN, but some important TTS methods are not discussed in the related work, such as [1] and [2]. \n\nWe indicated that the terminology was indeed adapted from [1], we will add a reference to [2] in the related works. However, it should be noted that both [1,2] focus on temporal graphs rather than on time series. Finally, note that although we use TTS models as one of the reference architectures, the focus of the paper is not on introducing a new architecture but rather to study the impact of local effects on existing ones.\n\n> The instructions in line 43 and lines 179-180 may appear contradictory. Could you provide a more detailed explanation?\n\nThe contradiction is only apparent. The point is that global models are indeed (in general) more efficient than local approaches [line 43] as the total number of parameters to be learned is smaller. However, if local effects are present in the data-generating process, this advantage might be compromised [lines 179-180], as more parameters might be needed to properly model them. This trade-off is what motivates the introduction of the hybrid models in the paper. We will make this apparent contradiction explicit in the paper. Thank you for the comment.\n"}, {"review_id": "SV9f4rkQ7S", "reviewer": "Reviewer_Lyxe", "summary": "This paper proposes a method to leverage local effects in graph-based spatio-temporal forecasting. The authors claim that existing spatio-temporal graph neural networks are global models, i.e. all nodes share the same set of parameters, and thus may fail to capture some node-specific patterns. On the other hand, local models, in which some layers within the models are node-specifically parameterized, have better performances compared to global ones, but at the cost of many additional parameters. The authors find a method, random node embeddings, to strike a balance between local and global methods. The authors also propose regularizations to improve transferability of the node embeddings and the resulting models. Experiments over real-world data are given where the proposed method achieves consistent improvements over a variety of models and datasets.  ", "questions": "I really like this paper and since it is in general clearly written, I do not have questions at this time.  ", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The studied problem is interesting. It challenges the assumption that a shared global STGNN is used for all nodes, which is standard in previous works. \n2. The proposed technique, i.e. trainable node embeddings, is simple and sound. The regularization terms designed also make sense. \n3. The proposed technique with trainable node embeddings are effective over various models (DCRNN, AGCRN, GWNet) and real-world datasets, which shows the generality of the proposed technique. \n4. The experimental results showing that node embeddings and regularizations are effective in terms of knowledge transfer are a plus. Intuitively, people may think that models with node specific parameters will not perform well in unseen nodes, but the results show the opposite. ", "weaknesses": "1. The proposed method with node specific embeddings is effective, but not new. Specifically, STID [40] proposes exactly the same technique in terms of trainable node embeddings. I am slightly concerned about whether the technical contribution meets the standard of NeurIPS with this existing work. \n2. The fine-grained categorization about spatio-temporal graph neural networks do not seem necessary, e.g. T&S, TTS and anistropic VS istropic. I fail to see how introducing these concepts help better understand the paper, and thus I would suggest these parts be removed. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and your positive opinion about our work. Please find our point-by-point answers below.\n\n> 1. The proposed method with node specific embeddings is effective, but not new. Specifically, STID [40] proposes exactly the same technique in terms of trainable node embeddings. I am slightly concerned about whether the technical contribution meets the standard of NeurIPS with this existing work.\n\nSimilar node embeddings have been used in some architectures. In this paper, we rationalize the practice of introducing such trainable components by providing an explanatory framework for the observed empirical results. We believe our contribution fully meets NeurIPS standards as it sheds light on extremely relevant challenges of very popular architectures. In fact, we provide a comprehensive methodology accounting for local components in several settings and across different (global) models. Our framework allows the practitioner to take full advantage of such hybrid models in both transductive and transfer learning settings.\n\n> 2. The fine-grained categorization about spatio-temporal graph neural networks do not seem necessary, e.g. T&S, TTS and anistropic VS istropic. I fail to see how introducing these concepts help better understand the paper, and thus I would suggest these parts be removed.\n\nThe introduction of the different model architectures and design choices is necessary to show that the issues related to global and local aspects are present across a variety of architectures. In other words, the introduced categorization of existing architectures was necessary to carry out a proper and comprehensive empirical evaluation of the phenomena studied in the paper.\n"}]

Author rebuttal:
We thank the reviewer for their insightful comments. 

We provide point-by-point answers to each reviewer and attach as supplementary results a visualization of the embedding space for different regularization strategies for load and traffic forecasting datasets (see the attached pdf for more details).

We hope that the rebuttal clarifies all the raised issues.


Meta-review: The paper proposes a novel graph-based spatiotemporal forecasting model that accounts for local effects. The reviewers think the notion of locality and embedding vector is novel and the experimental results are comprehensive. Please include the feedback and references provided by the reviewers in the final verison.
Decision: Accept (poster)