SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Andrea Cini  $^{*1}$ , Ivan Marisca  $^{*1}$ , Daniele Zambon  $^{1}$ , Cesare Alippi  $^{12}$
<sup>1</sup> The Swiss AI Lab IDSIA USI-SUPSI, Università della Svizzera italiana, <sup>2</sup> Politecnico di Milano {andrea.cini, ivan.marisca, daniele.zambon, cesare.alippi}@usi.ch
Spatiotemporal graph neural networks have shown to be effective in time series forecasting applications, achieving better performance than standard univariate predictors in several settings. These architectures take advantage of a graph structure and relational inductive biases to learn a single (global) inductive model to predict any number of the input time series, each associated with a graph node. Despite the gain achieved in computational and data efficiency w.r.t. fitting a set of local models, relying on a single global model can be a limitation whenever some of the time series are generated by a different spatiotemporal stochastic process. The main objective of this paper is to understand the interplay between globality and locality in graph-based spatiotemporal forecasting, while contextually proposing a methodological framework to rationalize the practice of including trainable node embeddings in such architectures. We ascribe to trainable node embeddings the role of amortizing the learning of specialized components. Moreover, embeddings allow for 1) effectively combining the advantages of shared message-passing layers with node-specific parameters and 2) efficiently transferring the learned model to new node sets. Supported by strong empirical evidence, we provide insights and guidelines for specializing graph-based models to the dynamics of each time series and show how this aspect plays a crucial role in obtaining accurate predictions.
Neural forecasting methods [1] take advantage of large databases of related time series to learn models for each individual process. If the time series in the database are not independent, functional dependencies among them can be exploited to obtain more accurate predictions, e.g., when the considered time series are observations collected from a network of sensors. In this setting, we use the term spatiotemporal time series to indicate the existence of relationships among subsets of time series (sensors) that span additional axes other than the temporal one, denoted here as spatial in a broad sense. Graph representations effectively model such dependencies and graph neural networks (GNNs) [2-4] can be included as modules in the forecasting architecture to propagate information along the spatial dimension. The resulting neural architectures are known in the literature as spatiotemporal graph neural networks (STGNNs) [5, 6] and have found widespread adoption in relevant applications ranging from traffic forecasting [6, 7] to energy analytics [8, 9]. These models embed inductive biases typical of graph deep learning and graph signal processing [10] and, thus, have several advantages over standard multivariate models; in fact, a shared set of learnable weights is used to obtain predictions for each time series by conditioning on observations at the neighboring nodes. Nonetheless, it has become more and more common to see node-specific trainable parameters being introduced as means to extract node(sensor)-level features then used as spatial identifiers within the processing [11, 12]. By doing so, the designer accepts a compromise in transferability that often
empirically leads to higher forecasting accuracy on the task at hand. We argue that the community has yet to find a proper explanatory framework for this phenomenon and, notably, has yet to design proper methodologies to deal with the root causes of observed empirical results, in practical applications.
In the broader context of time series forecasting, single models learned from a set of time series are categorized as global and are opposed to local models, which instead are specialized for any particular time series in the set [13, 14]. Global models are usually more robust than local ones, as they require a smaller total number of parameters and are fitted on more samples. Standard STGNNs, then, fall within the class of global models and, as a result, often have an advantage over local multivariate approaches; however, explicitly accounting for the behavior of individual time series might be problematic and require a large memory and model capacity. As an example, consider the problem of electric load forecasting: consumption patterns of single customers are influenced by shared factors, e.g., weather conditions and holidays, but are also determined by the daily routine of the individual users related by varying degrees of affinity. We refer to the dynamics proper to individual nodes as local effects.
Local effects can be accounted for by combining a global model with local components, e.g., by using encoding and/or decoding layers specialized for each input time series paired with a core global processing block. While such an approach to building hybrid global-local models can be effective, the added model complexity and specialization can negate the benefits of using a global component. In this paper, we propose to consider and interpret learned node embeddings as a mechanism to amortize the learning of local components; in fact, instead of learning a separate processing layer for each time series, node embeddings allow for learning a single global module conditioned on the learned (local) node features. Furthermore, we show that – within a proper methodological framework – node embeddings can be fitted to different node sets, thus enabling an effective and efficient transfer of the core processing modules.
Contributions In this paper, we analyze the effect of node-specific components in spatiotemporal time series models and assess how to incorporate them in the forecasting architecture, while providing an understanding of each design choice within a proper context. The major findings can be summarized in the following statements.
S1 Local components can be crucial to obtain accurate predictions in spatiotemporal forecasting.
S2 Node embeddings can amortize the learning of local components.
S3 Hybrid global-local STGNNs can capture local effects with contained model complexity and smaller input windows w.r.t. fully global approaches.
S4 Node embeddings for time series outside the training dataset can be obtained by fitting a relatively small number of observations and yield more effective transferability than fine-tuning global models.
S5 Giving structure to the embedding space provides an effective regularization, allowing for similarities among time series to emerge and shedding light on the role of local embeddings within a global architecture.
Throughout the paper, we reference the findings with their pointers. Against this backdrop, our main novel contributions reside in:
- A sound conceptual and methodological framework for dealing with local effects and designing node-specific components in STGNN architectures.
- An assessment of the role of learnable node embeddings in STGNNs and methods to obtain them.
- A comprehensive empirical analysis of the aforementioned phenomena in representative architectures across synthetic and real-world datasets.
- Methods to structure the embedding space, thus allowing for effective and efficient reuse of the global component in a transfer learning scenario.
We believe that our study constitutes an essential advancement toward the understanding of the interplay of different inductive biases in graph-based predictors, and argue that the methodologies and conceptual developments proposed in this work will constitute a foundational piece of know-how for the practitioner.
2 Preliminaries and problem statement
Consider a set of  $N$  time series with graph-side information where the  $i$ -th time series is composed by a sequence of  $d_x$  dimensional vectors  $\pmb{x}_t^i \in \mathbb{R}^{d_x}$  observed at time step  $t$ ; each time series  $\{\pmb{x}_t^i\}_t$  might be generated by a different stochastic process. Matrix  $\pmb{X}_t \in \mathbb{R}^{N \times d_x}$  encompasses the  $N$  observations at time  $t$  and, similarly,  $\pmb{X}_{t:t + T}$  indicates the sequence of observations within the time interval  $[t, t + T)$ . Relational information is encoded by a weighted adjacency matrix  $\pmb{A} \in \mathbb{R}^{N \times N}$  that accounts for (soft) functional dependencies existing among the different time series. We use interchangeably the terms node and sensor to indicate the entities generating the time series and refer to the node set together with the relational information (graph) as sensor network. Eventual exogenous variables associated with each node are indicated by  $\pmb{U}_t \in \mathbb{R}^{N \times d_u}$ ; the tuple  $\mathcal{G}_t = \langle \pmb{X}_t, \pmb{U}_t, \pmb{A} \rangle$  indicates all the available information associated with time step  $t$ .
We address the multistep-ahead time-series forecasting problem, i.e., we are interested in predicting, for every time step  $t$  and some  $H$ ,  $W \geq 1$ , the expected value of the next  $H$  observations  $X_{t:t + H}$  given a window  $\mathcal{G}_{t - W:t}$  of  $W$  past measurements. In case data from multiple sensor networks are available, the problem can be formalized as learning from  $M$  disjoint collections of spatiotemporal time series  $\mathcal{D} = \{\mathcal{G}_{t_1:t_1 + T_1}^{(1)},\mathcal{G}_{t_2:t_2 + T_2}^{(2)},\dots,\mathcal{G}_{t_m:t_m + T_m}^{(M)}\}$ , potentially without overlapping time frames. In the latter case, we assume sensors to be homogeneous both within a single network and among different sets. Furthermore, we assume edges to indicate the same type of relational dependencies, e.g., physical proximity.
3 Forecasting with Spatiotemporal Graph Neural Networks
This section provides a taxonomy of the different components that constitute an STGNN. Based on the resulting archetypes, reference operators for this study are identified. The last paragraph of the section broadens the analysis to fit STGNNs within more general time series forecasting frameworks.
Spatiotemporal message-passing We consider STGNNs obtained by stacking spatiotemporal message-passing (STMP) layers s.t.
$$
\boldsymbol {H} _ {t} ^ {l + 1} = \operatorname {S T M P} ^ {l} \left(\boldsymbol {H} _ {\leq t} ^ {l}, \boldsymbol {A}\right), \tag {1}
$$
where  $H_{t}^{l}\in \mathbb{R}^{N\times d_{h}}$  indicates the stack of node representations  $h_t^{i,l}$  at time step  $t$  at the  $l$ -th layer. The shorthand  $\leq t$  indicates the sequence of all representations corresponding to the time steps up to  $t$  (included). Each  $\mathrm{STMP}^l (\cdot)$  layer is structured as follows
$$
\boldsymbol {h} _ {t} ^ {i, l + 1} = \rho^ {l} \left(\boldsymbol {h} _ {\leq t} ^ {i, l}, \underset {j \in \mathcal {N} (i)} {\operatorname {A G G R}} \left\{\gamma^ {l} \left(\boldsymbol {h} _ {\leq t} ^ {i, l}, \boldsymbol {h} _ {\leq t} ^ {j, l}, a _ {j i}\right) \right\}\right), \tag {2}
$$
where  $\rho^l$  and  $\gamma^l$  are respectively the update and message functions, e.g., implemented by multilayer perceptrons (MLPs) or recurrent neural networks (RNNs). AGGR{·} indicates a generic permutation invariant aggregation function, while  $\mathcal{N}(i)$  refers to the set of neighbors of node  $i$ , each associated with an edge with weight  $a_{ji}$ . Models of this type are fully inductive, in the sense that they can be used to make predictions for networks and time windows different from those they have been trained on, provided a certain level of similarity (e.g., homogenous sensors) between source and target node sets [15].
Among the different implementations of this general framework, we can distinguish between time-then-space (TTS) and time-and-space (T&S) models by following the terminology of previous works [16, 17]. Specifically, in TTS models the sequence of representations  $h_{\leq t}^{i,0}$  is encoded by a sequence model, e.g., an RNN, before propagating information along the spatial dimension through message passing (MP) [16]. Conversely, in T&S models time and space are processed in a more integrated fashion, e.g., by a recurrent GNN [5] or by spatiotemporal convolutional operators [7]. In the remainder of the paper, we take for TTS model an STGNN composed by a Gated Recurrent Unit (GRU) [18] followed by standard MP layers [19]:
$$
\boldsymbol {H} _ {t} ^ {1} = \operatorname {G R U} \left(\boldsymbol {H} _ {\leq t} ^ {0}\right), \quad \boldsymbol {H} _ {t} ^ {l + 1} = \mathrm {M P} ^ {l} \left(\boldsymbol {H} _ {t} ^ {l}, \boldsymbol {A}\right), \tag {3}
$$
where  $l = 1, \dots, L - 1$  and  $\mathrm{GRU}(\cdot)$  processes sequences node-wise. Similarly, we consider as reference T&S model a GRU with an MP network at its gates [5, 20], that process input data as
$$
\boldsymbol {H} _ {t} ^ {l + 1} = \operatorname {G R U} ^ {l} \left(\left\{\mathrm {M P} ^ {l} \left(\boldsymbol {H} _ {t} ^ {l}, \boldsymbol {A}\right) \right\} _ {\leq t}\right). \tag {4}
$$
Moreover, STGNN models can be further categorized w.r.t. the implementation of message function; in particular, by loosely following Dwivedi et al. [21], we call isotropic those GNNs where the message function  $\gamma^l$  only depends on the features of the sender node  $h_{\leq t}^{j,l}$ ; conversely, we call anistropic GNNs where  $\gamma^l$  takes both  $h_{\leq t}^{i,l}$  and  $h_{\leq t}^{j,l}$  as input. In the following, the case-study isotropic operator is
$$
\boldsymbol {h} _ {t} ^ {i, l + 1} = \xi \left(\boldsymbol {W} _ {1} ^ {l} \boldsymbol {h} _ {t} ^ {i, l} + \underset {j \in \mathcal {N} (i)} {\operatorname {M E A N}} \left\{\boldsymbol {W} _ {2} ^ {l} \boldsymbol {h} _ {t} ^ {j, l} \right\}\right), \tag {5}
$$
where  $W_1^l$  and  $W_2^l$  are matrices of learnable parameters and  $\xi(\cdot)$  a generic activation function. Conversely, the operator of choice for the anisotropic case corresponds to
$$
\boldsymbol {m} _ {t} ^ {j \rightarrow i, l} = \boldsymbol {W} _ {2} ^ {l} \xi \left(\boldsymbol {W} _ {1} ^ {l} \left[ \boldsymbol {h} _ {t} ^ {i, l} | | \boldsymbol {h} _ {t} ^ {j, l} | | a _ {j i} \right]\right), \quad \alpha_ {t} ^ {j \rightarrow i, l} = \sigma \left(\boldsymbol {W} _ {0} ^ {l} \boldsymbol {m} _ {t} ^ {j \rightarrow i, l}\right), \tag {6}
$$
$$
\boldsymbol {h} _ {t} ^ {i, l + 1} = \xi \left(\boldsymbol {W} _ {3} ^ {l} \boldsymbol {h} _ {t} ^ {i, l} + \underset {j \in \mathcal {N} (i)} {\operatorname {S U M}} \left\{\alpha_ {t} ^ {j \rightarrow i, l} \boldsymbol {m} _ {t} ^ {j \rightarrow i, l} \right\}\right), \tag {7}
$$
where matrices  $\mathbf{W}_0^l \in \mathbb{R}^{1 \times d_m}$ ,  $\mathbf{W}_1^l$ ,  $\mathbf{W}_2^l$  and  $\mathbf{W}_3^l$  are learnable parameters,  $\sigma(\cdot)$  is the sigmoid activation function and  $|| \cdot |$  the concatenation operator applied along the feature dimension (see Appendix A.2 for a detailed description).
Global and local forecasting models Formally, a time series forecasting model is called global if its parameters are fitted to a group of time series (either univariate or multivariate), while local models are specific to a single (possibly multivariate) time series. The advantages of global models have been discussed at length in the time series forecasting literature [22, 14, 13, 1] and are mainly ascribable to the availability of large amounts of data that enable generalization and the use of models with higher capacity w.r.t. the single local models. As presented, and further detailed in the following section, STGNNs are global, yet have a peculiar position in this context, as they exploit spatial dependencies localizing predictions w.r.t. each node's neighborhood. Furthermore, the transferability of GNNs makes these models distinctively different from local multivariate approaches enabling their use in cold-start scenarios [23] and making them inductive both temporally and spatially.
4 Locality and globality in Spatiotemporal Graph Neural Networks
We now focus on the impact of local effects in forecasting architectures based on STGNNs. The section starts by introducing a template to combine different processing layers within a global model, then continues by discussing how these can be turned into local. Contextually, we start to empirically probe the reference architectures.
4.1 A global processing template
STGNNs localize predictions in space, i.e., with respect to a single node, by exploiting an MP operator that contextualizes predictions by constraining the information flow within each node's neighborhood. STGNNs are global forecasting models s.t.
$$
\widehat {\boldsymbol {X}} _ {t: t + H} = F _ {\mathrm {G}} \left(\mathcal {G} _ {t - W: t}; \phi\right) \tag {8}
$$
where  $\phi$  are the learnable parameters shared among the time series and  $\widehat{X}_{t:t + H}$  indicate the  $H$ -step ahead predictions of the input time series given the window of (structured) observations  $X_{t - W:t}$ . In particular, we consider forecasting architectures consisting of an encoding step followed by STMP layers and a final readout mapping representations to predictions; the corresponding sequence of operations composing  $F_{\mathrm{G}}$  can be summarized as
$$
\boldsymbol {h} _ {t} ^ {i, 0} = \operatorname {E N C O D E R} \left(\boldsymbol {x} _ {t - 1} ^ {i}, \boldsymbol {u} _ {t - 1} ^ {i}\right), \tag {9}
$$
$$
\boldsymbol {H} _ {t} ^ {l + 1} = \operatorname {S T M P} ^ {l} \left(\boldsymbol {H} _ {\leq t} ^ {l}, \boldsymbol {A}\right), \quad l = 0, \dots , L - 1 \tag {10}
$$
$$
\hat {\boldsymbol {x}} _ {t: t + H} ^ {i} = \operatorname {D E C O D E R} \left(\boldsymbol {h} _ {t} ^ {i, L}\right). \tag {11}
$$
ENCODER(  $\cdot$  ) and  $DECODER(\cdot)$  indicate generic encoder and readout layers that could be implemented in several ways. In the following, the encoder is assumed to be a standard fully connected linear layer, while the decoder is implemented by an MLP with a single hidden layer followed by an output (linear) layer for each forecasting step.
4.2 Local effects
Differently from the global approach, local models are fitted to a single time series (e.g., see the standard Box-Jenkins approach) and, in our problem settings, can be indicated as
$$
\hat {\boldsymbol {x}} _ {t: t + H} ^ {i} = f _ {i} \left(\boldsymbol {x} _ {t - W: t} ^ {i}; \theta^ {i}\right), \tag {12}
$$
where  $f_{i}\left(\cdot ;\theta^{i}\right)$  is a sensor-specific model, e.g., an RNN with its dedicated parameters  $\theta^i$ , fitted on the  $i$ -th time series. While the advantages of global models have already been discussed, local effects, i.e., the dynamics observed at the level of the single sensor, are potentially more easily captured by a local model. In fact, if local effects are present, global models might require an impractically large model capacity to account for all node-specific dynamics [13], thus losing some of the advantages of using a global approach (S3). In the STGNN case, then, increasing the input window for each node would result in a large computational overhead. Conversely, purely local approaches fail to exploit relational information among the time series and cannot reuse available knowledge efficiently in an inductive setting.
Combining global graph-based components with local node-level components has the potential for achieving a two-fold objective: 1) exploiting relational dependencies together with side information to learn flexible and efficient graph deep learning models and 2) making at the same time specialized and accurate predictions for each time series. In particular, we indicate global-local STGNNs as
$$
\hat {\boldsymbol {x}} _ {t: t + H} ^ {i} = F \left(\mathcal {G} _ {t - W: t}; \phi , \theta^ {i}\right) \tag {13}
$$
where function  $F$  and parameter vector  $\phi$  are shared across all nodes, whereas parameter vector  $\theta^i$  is time-series dependent. Such a function  $F(\cdot)$  could be implemented, for example, as a sum between a global model (Eq. 8) and a local one (Eq. 12):
$$
\widehat {\boldsymbol {X}} _ {t: t + H} ^ {(1)} = F _ {\mathrm {G}} \left(\mathcal {G} _ {t - W: t}; \phi\right), \quad \hat {\boldsymbol {x}} _ {t: t + H} ^ {i, (2)} = f _ {i} \left(\boldsymbol {x} _ {t - W: t} ^ {i}; \theta^ {i}\right), \tag {14}
$$
$$
\hat {\boldsymbol {x}} _ {t: t + H} ^ {i} = \hat {\boldsymbol {x}} _ {t: t + H} ^ {i, (1)} + \hat {\boldsymbol {x}} _ {t: t + H} ^ {i, (2)}, \tag {15}
$$
or - with a more integrated approach - by using different weights for each time series at the encoding and/or decoding steps. The latter approach results in using a different encoder and/or decoder for each  $i$ -th node in the template STGNN (Eq. 9-11) to extract representations and, eventually, project them back into input space:
$$
\boldsymbol {h} _ {t} ^ {i, 0} = \operatorname {E N C O D E R} _ {i} \left(\boldsymbol {x} _ {t - 1} ^ {i}, \boldsymbol {u} _ {t - 1} ^ {i}; \theta_ {\text {e n c}} ^ {i}\right), \quad (1 6) \quad \hat {\boldsymbol {x}} _ {t: t + H} ^ {i} = \operatorname {D E C O D E R} _ {i} \left(\boldsymbol {h} _ {t} ^ {i, L}; \theta_ {\text {d e c}} ^ {i}\right). \tag {17}
$$
MP layers could in principle be specialized as well, e.g., by using a different local update function  $\gamma_{i}(\cdot)$  for each node. However, this would be impractical unless subsets of nodes are allowed to share parameters to some extent (e.g., by clustering them).
To support our arguments, Tab. 1 shows empirical results for the reference TTS models with isotropic message passing (TTS-IMP) on 2 popular traffic forecasting benchmarks (METR-LA and PEMS-BAY [6]). In particular, we compare the global approach with 3 hybrid global-local variants where local weights are used in the encoder, in the decoder, or in both of them (see Eq. 16-17 and the light brown block in Tab. 1). Notably, while fitting a separate RNN to each individual time series fails (LocalRNNs), exploiting a local encoder and/or decoder significantly improves performance w.r.t. the fully global model (S1). Note that the price of specialization is paid in terms of the number of learnable parameters which is an order of magnitude higher in global-local
Table 1: Performance (MAE) of TTS-IMP variants and number of associated trainable parameters in PEMS-BAY (5-run average).
METR-LA	PEMS-BAY (# weights)
Global TTS	3.35 ±0.01	1.72 ±0.00	4.71×10^4
Global-local TTS	Weights	Encoder	3.15 ±0.01	1.66 ±0.01	2.75×10^5
Decoder	3.09 ±0.01	1.58 ±0.00	3.00×10^5
Enc. + Dec.	3.16 ±0.01	1.70 ±0.01	5.28×10^5
Embed.	Encoder	3.08 ±0.01	1.58 ±0.00	5.96×10^4
Decoder	3.13 ±0.00	1.60 ±0.00	5.96×10^4
Enc. + Dec.	3.07 ±0.01	1.58 ±0.00	6.16×10^4
FC-RNN	3.56 ±0.03	2.32 ±0.01	3.04×10^5
LocalRNNs	3.69 ±0.00	1.91 ±0.00	1.10×10^7
variants. The table reports as a reference also results for FC-RNN, a multivariate RNN taking as input the concatenation of all time series. Indeed, having both encoder and decoder implemented as local layers leads to a large number of parameters and has a marginal impact on forecasting accuracy. The
light gray block in Tab. 1 anticipates the effect of replacing the local layers with the use of learnable node embeddings, an approach discussed in depth in the next section. Additional results including TTS models with anisotropic message passing (TTS-AMP) are provided in Appendix B.
5 Node embeddings
This section introduces node embeddings as a mechanism to amortize the learning of local components and discusses the supporting empirical results. We then propose possible regularization techniques and discuss the advantages of embeddings in transfer learning scenarios.