SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For each dataset, we obtain the corresponding adjacency matrix by following previous works [20, 6, 56]. We use as exogenous variables sinusoidal functions encoding the time of the day and the one-hot encoding of the day of the week. For datasets with an excessive number of missing values, namely METR-LA and AQI, we add as an exogenous variable also a binary mask indicating if the corresponding value has been imputed.
We split datasets into windows of  $W$  time steps, and train the models to predict the next  $H$  observations. For the experiment in Tab.4, we set  $W = 12$ ,  $H = 12$  for the traffic datasets,  $W = 48$ ,  $H = 6$  for CER-E, and  $W = 24$ ,  $H = 3$  for AQI. Then, for all datasets except for AQI, we divide the obtained windows sequentially into  $70\% / 10\% / 20\%$  splits for training, validation, and testing, respectively. For AQI instead, we use as the test set the months of March, June, September, and December, following Yi et al. [66].
A.2 Architectures
In this section, we provide a detailed description of the reference architectures used in the study.
Reference architectures We consider 4 simple STGNN architectures that follow the template given in (Eq. 9-11). In all the models, we use as ENCODER (Eq. 9) a simple linear layer s.t.
$$
\boldsymbol {h} _ {t} ^ {i, 0} = \boldsymbol {W} _ {\text {e n c}} \left[ \boldsymbol {x} _ {t - 1} ^ {i} \| \boldsymbol {u} _ {t - 1} ^ {i} \right] \tag {21}
$$
and as DecODER (Eq. 11) a 2 layer module structured as
$$
\hat {\boldsymbol {x}} _ {t: t + H} ^ {i} = \left\{\boldsymbol {W} _ {h} \xi \left(\boldsymbol {W} _ {\mathrm {d e c}} \boldsymbol {h} _ {t} ^ {i, L}\right) \right\} _ {h = 1, \dots , H} \tag {22}
$$
where  $W_{\mathrm{enc}}$ ,  $W_{\mathrm{dec}}$ , and  $W_{1}, \ldots, W_{H}$  are learnable parameters. In hybrid models with local ENCODER and/or DECODER, we use different parameter matrices for each node, i.e.,
$$
\boldsymbol {h} _ {t} ^ {i, 0} = \boldsymbol {W} _ {\text {e n c}} ^ {i} \left[ \boldsymbol {x} _ {t - 1} ^ {i} \right| | \boldsymbol {u} _ {t - 1} ^ {i} \tag {23}
$$
$$
\hat {\boldsymbol {x}} _ {t: t + H} ^ {i} = \left\{\boldsymbol {W} _ {h} ^ {i} \xi \left(\boldsymbol {W} _ {\mathrm {d e c}} \boldsymbol {h} _ {t} ^ {i, L}\right) \right\} _ {h = 1, \dots , H}. \tag {24}
$$
When instead node embeddings  $\mathbf{V} \in \mathbb{R}^{N \times d_v}$  are used to specialize the model, they are concatenated to the modules' input as
$$
\boldsymbol {h} _ {t} ^ {i, 0} = \boldsymbol {W} _ {\text {e n c}} \left[ \boldsymbol {x} _ {t - 1} ^ {i} \mid \mid \boldsymbol {u} _ {t - 1} ^ {i} \mid \mid \boldsymbol {v} ^ {i} \right] \tag {25}
$$
$$
\hat {\boldsymbol {x}} _ {t: t + H} ^ {i} = \left\{\boldsymbol {W} _ {h} \xi \left(\boldsymbol {W} _ {\mathrm {d e c}} \left[ \boldsymbol {h} _ {t} ^ {i, L} \mid \mid \boldsymbol {v} ^ {i} \right]\right) \right\} _ {h = 1, \dots , H}. \tag {26}
$$
For the TTS architectures, we define the STMP module (Eq. 10) as a node-wise GRU [18]:
$$
\boldsymbol {r} _ {t} ^ {i} = \sigma \left( \right.\boldsymbol {W} _ {1} ^ {l} \left[ \right. \boldsymbol {h} _ {t} ^ {i, 0} \left. \right| \mid \boldsymbol {h} _ {t - 1} ^ {i, 1} \left. \right]\left. \right), \tag {27}
$$
$$
\boldsymbol {o} _ {t} ^ {i} = \sigma \left(\boldsymbol {W} _ {2} ^ {l} \left[ \boldsymbol {h} _ {t} ^ {i, 0} \mid \mid \boldsymbol {h} _ {t - 1} ^ {i, 1} \right]\right), \tag {28}
$$
$$
\boldsymbol {c} _ {t} ^ {i} = \tanh  \left(\boldsymbol {W} _ {3} ^ {l} \left[ \boldsymbol {h} _ {t} ^ {i, 0} \mid \mid \boldsymbol {r} _ {t} ^ {i} \odot \boldsymbol {h} _ {t - 1} ^ {i, 1} \right]\right), \tag {29}
$$
$$
\boldsymbol {h} _ {t} ^ {i, 1} = \boldsymbol {o} _ {t} ^ {i} \odot \boldsymbol {h} _ {t - 1} ^ {i, 1} + (1 - \boldsymbol {o} _ {t} ^ {i}) \odot \boldsymbol {c} _ {t} ^ {i}. \tag {30}
$$
The GRU is then followed by  $L$  MP layers
$$
\boldsymbol {H} _ {t} ^ {l + 1} = \operatorname {M P} ^ {l} \left(\boldsymbol {H} _ {t} ^ {l}, \boldsymbol {A}\right), \quad l = 1, \dots , L - 1. \tag {31}
$$
We consider two variants for this architecture: TTS-IMP, featuring the isotropic MP operator defined in Eq. 5, and TTS-AMP, using the anisotropic MP operator defined in Eq. 6-7. Note that all the parameters in the STMP blocks are shared among the nodes.
In T&S models, instead, we use a GRU where gates are implemented using MP layers:
$$
\boldsymbol {r} _ {t} ^ {i, l} = \sigma \left(\mathrm {M P} _ {r} ^ {l} \left(\left[ \boldsymbol {h} _ {t} ^ {i, l - 1} \right| \left| \boldsymbol {h} _ {t - 1} ^ {i, l} \right], \boldsymbol {A}\right)\right), \tag {32}
$$
$$
\boldsymbol {o} _ {t} ^ {i, l} = \sigma \left(\mathrm {M P} _ {o} ^ {l} \left(\left[ \boldsymbol {h} _ {t} ^ {i, l - 1} \right| \left| \boldsymbol {h} _ {t - 1} ^ {i, l} \right], \boldsymbol {A}\right)\right), \tag {33}
$$
$$
\boldsymbol {c} _ {t} ^ {i, l} = \tanh  \left(\mathbf {M P} _ {c} ^ {l} \left(\left[ \boldsymbol {h} _ {t} ^ {i, l - 1} \right| \left| \boldsymbol {r} _ {t} ^ {i, l} \odot \boldsymbol {h} _ {t - 1} ^ {i, l} \right], \boldsymbol {A}\right)\right), \tag {34}
$$
$$
\boldsymbol {h} _ {t} ^ {i, l} = \boldsymbol {o} _ {t} ^ {i, l} \odot \boldsymbol {h} _ {t - 1} ^ {i, l} + \left(1 - \boldsymbol {o} _ {t} ^ {i, l}\right) \odot \boldsymbol {c} _ {t} ^ {i, l}. \tag {35}
$$
Similarly to the TTS case, we indicate as T&S-IMP the reference architecture in which MP operators are isotropic and T&S-AMP the one featuring anisotropic MP.
Baselines For the RNN baselines we follow the same template of the reference architectures (Eq. 9-11) but use a single GRU cell as core processing module instead of the STMP block. The global-local RNN, then, uses node embeddings at encoding as shown in Eq. 25, whereas LocalRNNs have different sets of parameters for each time series. The FC-RNN architecture, instead, takes as input the concatenation all the time series as if they were a single multivariate one.
Hyperparameters For the reference TTS architectures we use a GRU with a single cell followed by 2 message-passing layers. In T&S case we use a single graph recurrent convolutional cell. The number of neurons in each layer is set to 64 and the embedding size to 32 for all the reference architectures in all the benchmark datasets. Analogous hyperparameters were used for the RNN baselines. For GPVAR instead, we use 16 and 8 as hidden and embedding sizes, respectively. For the baselines from the literature, we use the hyperparameters used in the original papers whenever possible.
For GPVAR experiments we use a batch size of 128 and train with early stopping for a maximum of 200 epochs with the Adam optimizer [67] and a learning rate of 0.01 halved every 50 epochs.
For experiments in Tab. 1 and 4, we instead trained the models with batch size 64 for a maximum of 300 epochs each consisting of maximum 300 batches. The initial learning rate was set to 0.003 and reduced every 50 epochs.
Transfer learning experiment In the transfer learning experiments we simulate the case where a pre-trained spatiotemporal model for traffic forecasting is tested on a new road network. We fine-tune on each of the PEMS03-08 datasets models previously trained on the left-out 3 datasets. Following previous works [56], we split the datasets into  $60\% /20\% /20\%$  for training, validation, and testing, respectively. We obtain the mini-batches during training by uniformly sampling 64 windows from the 3 training sets. To train the models we use a similar experimental setting of experiments in Tab. 4, decreasing the number of epochs to 150 and increasing the batches per epoch to 500. We then assume that 2 weeks of readings from the target dataset are available for fine-tuning, and use the first week for training and the second one as the validation set. Then, we test fine-tuned models on the immediately following week (Tab. 5). For the global model, we either tune all the parameters or none of them (zero-shot setting). For fine-tuning, we increase the maximum number of epochs to 2000 without limiting the batches processed per epoch and fixing the learning rate to 0.001. At the end of every
Table 8: Performance (MAE) of TTS-IMP and TTS-AMP variants and number of associated trainable parameters in PEMS-BAY (5-run average).
METR-LA	PEMS-BAY (# weights)	METR-LA	PEMS-BAY (# weights)
TTS-IMP	TTS-AMP
Global TTS	3.35 ±0.01	1.72 ±0.00	4.71×104	3.27 ±0.01	1.68 ±0.00	6.41×104
Weights	Encoder	3.15 ±0.01	1.66 ±0.01	2.75×105	3.12 ±0.01	1.64 ±0.00	2.92×105
Decoder	3.09 ±0.01	1.58 ±0.00	3.00×105	3.09 ±0.01	1.58 ±0.00	3.17×105
Enc. + Dec.	3.16 ±0.01	1.70 ±0.01	5.28×105	3.14 ±0.01	1.69 ±0.01	5.45×105
Embed.	Encoder	3.08 ±0.01	1.58 ±0.00	5.96×104	3.05 ±0.02	1.58 ±0.01	7.66×104
Decoder	3.13 ±0.00	1.60 ±0.00	5.96×104	3.12 ±0.01	1.60 ±0.00	7.66×104
Enc. + Dec.	3.07 ±0.01	1.58 ±0.00	6.16×104	3.04 ±0.01	1.59 ±0.01	7.86×104
FC-RNN	3.56 ±0.03	2.32 ±0.01	3.04×105
LocalRNNs	3.69 ±0.00	1.91 ±0.00	1.10×107
training epoch, we compute the MAE on the validation set and stop training if it has not decreased in the last 100 epochs, restoring the model weights corresponding to the best-performing model. For the global-local models with variational regularization on the embedding space, during training, we set  $\beta = 0.05$  and initialize the distribution parameters as
$$
\boldsymbol {\mu} _ {i} \sim \mathcal {U} (- 0. 0 1, 0. 0 1), \quad \sigma_ {i} = \mathbf {0}. 2.
$$
For fine-tuning instead, we initialize the new embedding table  $V^{\prime}\in \mathbb{R}^{N^{\prime}\times d_{v}}$  as
$$
\boldsymbol {V} ^ {\prime} \sim \mathcal {U} (- \Delta , \Delta),
$$
where  $\Delta = \frac{1}{\sqrt{d_v}}$  and remove the regularization loss. For the clustering regularization, instead, we use  $K = 10$  clusters and regularization trade-off weight  $\lambda = 0.5$ . We initialize embedding, centroid, and cluster assignment matrices as
$$
\boldsymbol {V} \sim \mathcal {U} (- \Delta , \Delta) \quad \boldsymbol {C} \sim \mathcal {U} (- \Delta , \Delta) \quad \boldsymbol {S} \sim \mathcal {U} (0, 1)
$$
respectively. For fine-tuning, we fix the centroid table  $\mathbf{C}$  and initialize the new embedding table  $\mathbf{V}' \in \mathbb{R}^{N' \times d_v}$  and cluster assignment matrix  $\mathbf{S}' \in \mathbb{R}^{N' \times K}$  following an analogous procedure. Finally, we increase the regularization weight  $\lambda$  to 10.
Local components Table 8, an extended version of Table 1, shows the performance of reference architecture with and without local components. Here we consider TTS-IMP and TTS-AMP, together with FC-RNN (a multivariate RNN) and LocalRNNs (local univariate RNNs with a different set of parameters for each time series). For the STGNNs, we consider a global variant (without any local component) and global-local alternatives, where we insert node-specific components within the architecture by means of (1) different sets of weights for each time series (light brown block) or (2) node embeddings (light gray block). More precisely, we show how performance varies when the local components are added in the encoding and decoding steps together, or uniquely in one of the two steps. Results for TTS-AMP are consistent with the observations made for TTS-IMP, showing that the use of local components enables improvements up to  $7\%$  w.r.t. the fully global variant.
Transfer learning Tables 9 to 12 show additional results for the transfer learning experiments in all the target datasets. In particular, each table shows results for the reference architectures w.r.t. different training set sizes (from 1 day to 2 weeks) and considers the settings where embeddings are fed to both encoder and decoder or decoder only. We report results on the test data corresponding to the week after the validation set but also on the original test split used in the literature. In the last columns of the table, we also show the performance that one would have obtained 100 epochs after the minimum in the validation error curve; the purpose of showing these results is to hint at the performance that one would have obtained without holding out 1 week of data for validation. The results indeed suggest that fine-tuning the full global model is more prone to overfitting.
Table 9: Forecasting error (MAE) on PEMS03 in the transfer learning setting (5 runs average).
Model	Testing on 1 subsequent week	Testing on the standard split	Testing 100 epochs after validation min.
TTS-IMP	2 weeks	1 week	3 days	1 day	2 weeks	1 week	3 days	1 day	2 weeks	1 week	3 days	1 day
Global	14.86±0.02	15.30±0.03	16.26±0.08	16.65±0.07	16.11±0.05	16.36±0.05	16.95±0.04	17.39±0.12	16.30±0.10	16.58±0.07	17.62±0.23	18.33±0.32
Embeddings	- Variational	14.53±0.02	14.64±0.05	15.87±0.08	16.78±0.12	16.03±0.05	16.12±0.05	17.18±0.16	17.82±0.15	16.09±0.06	16.16±0.05	17.28±0.18	17.94±0.17
- Clustering	14.50±0.04	14.56±0.03	15.40±0.06	15.65±0.11	15.69±0.10	15.70±0.12	16.33±0.06	16.52±0.10	15.70±0.10	15.70±0.12	16.35±0.07	16.54±0.10
14.58±0.02	14.60±0.02	15.67±0.08	16.53±0.13	15.65±0.07	15.71±0.08	16.76±0.15	17.76±0.15	15.70±0.05	15.74±0.07	16.80±0.14	17.78±0.14
DEC.	Embeddings	14.79±0.02	14.84±0.03	15.49±0.03	16.01±0.08	16.06±0.05	16.12±0.07	16.74±0.04	17.25±0.07	16.08±0.05	16.13±0.07	16.75±0.04	17.29±0.06
- Variational	15.33±0.03	15.39±0.02	15.83±0.04	16.03±0.04	16.15±0.02	16.20±0.02	16.60±0.04	16.75±0.06	16.15±0.02	16.20±0.02	16.60±0.04	16.76±0.06
- Clustering	14.96±0.06	15.09±0.06	15.88±0.07	15.81±0.03	16.25±0.08	16.29±0.06	16.72±0.08	16.87±0.07	16.28±0.07	16.19±0.05	16.91±0.10	16.93±0.07
Table 10: Forecasting error (MAE) on PEMS04 in the transfer learning setting (5 runs average).
Model	Testing on 1 subsequent week.	Testing on the standard split	Testing 100 epochs after validation min.
TTS-IMP	2 weeks	1 week	3 days	1 day	2 weeks	1 week	3 days	1 day	2 weeks	1 week	3 days	1 day
Global	20.86±0.03	21.59±0.11	21.84±0.06	22.26±0.10	20.79±0.02	21.68±0.10	22.10±0.10	22.59±0.11	20.89±0.05	21.97±0.13	22.88±0.17	23.85±0.22
Embeddings	- Variational	19.96±0.08	20.27±0.11	21.03±0.14	21.99±0.13	19.87±0.07	20.27±0.07	21.20±0.15	22.38±0.14	19.88±0.07	20.29±0.07	21.28±0.14	22.46±0.14
- Clustering	19.94±0.08	20.19±0.05	20.71±0.12	21.20±0.15	19.92±0.06	20.23±0.05	20.82±0.08	21.46±0.13	19.92±0.06	20.23±0.05	20.82±0.08	21.47±0.12
19.69±0.06	19.91±0.11	20.48±0.09	21.91±0.21	19.70±0.06	19.96±0.11	20.62±0.09	22.28±0.21	19.72±0.07	19.97±0.10	20.65±0.08	22.29±0.21
DEC.	Embeddings	20.10±0.06	20.27±0.04	20.87±0.08	21.44±0.09	20.18±0.07	20.39±0.05	21.01±0.09	21.70±0.08	20.19±0.07	20.40±0.05	21.03±0.08	21.74±0.08
- Variational	20.79±0.06	20.94±0.05	21.23±0.07	21.51±0.06	20.94±0.06	21.10±0.05	21.40±0.08	21.76±0.05	20.94±0.06	21.10±0.05	21.40±0.08	21.77±0.05
- Clustering	20.19±0.09	20.45±0.10	20.63±0.06	21.03±0.05	20.27±0.09	20.56±0.10	20.81±0.06	21.28±0.06	20.75±0.05	20.78±0.05	20.89±0.05	21.35±0.05
Table 11: Forecasting error (MAE) on PEMS07 in the transfer learning setting (5 runs average).
Model	Testing on 1 subsequent week	Testing on the standard split	Testing 100 epochs after validation min.
TTS-IMP	2 weeks	1 week	3 days	1 day	2 weeks	1 week	3 days	1 day	2 weeks	1 week	3 days	1 day
Global	22.87±0.05	23.82±0.03	24.52±0.06	25.40±0.06	22.64±0.04	23.58±0.02	24.20±0.06	25.04±0.06	22.72±0.05	23.72±0.02	24.45±0.11	25.48±0.05
Embeddings	- Variational	21.68±0.07	22.23±0.08	23.54±0.19	26.11±0.61	22.10±0.09	22.77±0.05	24.17±0.24	26.79±0.63	22.11±0.09	22.78±0.05	24.21±0.22	26.82±0.63
- Clustering	22.05±0.05	22.43±0.02	23.23±0.08	24.40±0.13	22.18±0.04	22.59±0.04	23.44±0.11	26.62±0.13	22.18±0.04	22.59±0.04	23.44±0.10	24.62±0.13
21.75±0.05	22.16±0.07	23.36±0.20	26.44±0.26	22.03±0.08	22.52±0.10	23.85±0.24	27.12±0.27	22.03±0.09	22.55±0.11	23.85±0.24	27.13±0.28
DEC.	Embeddings	22.50±0.14	22.83±0.13	23.59±0.12	24.89±0.19	22.68±0.12	23.13±0.10	24.04±0.09	25.41±0.20	22.69±0.12	23.14±0.10	24.04±0.09	25.43±0.20
- Variational	24.32±0.16	24.60±0.16	25.12±0.17	25.50±0.15	24.25±0.14	24.60±0.13	25.16±0.13	25.56±0.12	24.25±0.14	24.60±0.13	25.16±0.13	25.57±0.12
- Clustering	23.02±0.09	23.53±0.09	24.42±0.15	24.87±0.13	23.18±0.09	23.77±0.08	24.66±0.12	25.24±0.09	23.91±0.16	24.10±0.15	24.73±0.10	25.27±0.09
Table 12: Forecasting error (MAE) on PEMS08 in the transfer learning setting (5 runs average).
Model	Testing on 1 subsequent week	Testing on the standard split	Testing 100 epochs after validation min.
TTS-IMP	2 weeks	1 week	3 days	1 day	2 weeks	1 week	3 days	1 day	2 weeks	1 week	3 days	1 day
Global	15.51±0.03	15.90±0.07	16.87±0.05	17.59±0.04	15.36±0.03	15.71±0.06	16.71±0.06	17.41±0.03	15.47±0.06	15.87±0.04	17.58±0.16	18.46±0.12
ENC-DEC	Embeddings	15.45±0.08	15.45±0.06	16.34±0.07	17.15±0.08	15.34±0.07	15.32±0.04	16.27±0.07	17.11±0.08	15.32±0.09	15.30±0.03	16.27±0.05	17.13±0.08
-Variational	15.34±0.04	15.41±0.06	15.83±0.07	16.32±0.11	15.21±0.03	15.27±0.06	15.70±0.06	16.19±0.12	15.21±0.03	15.27±0.05	15.69±0.06	16.19±0.12
-Clustering	15.41±0.06	15.41±0.06	15.96±0.04	16.99±0.07	15.27±0.07	15.27±0.07	15.90±0.05	16.99±0.08	15.30±0.08	15.28±0.07	15.90±0.05	17.02±0.10
DEC	Embeddings	15.72±0.06	15.74±0.06	16.41±0.07	16.97±0.08	15.61±0.06	15.61±0.06	16.33±0.08	16.90±0.09	15.61±0.06	15.61±0.06	16.35±0.08	16.92±0.10
-Variational	16.31±0.10	16.33±0.15	16.53±0.15	16.74±0.12	16.13±0.09	16.14±0.14	16.34±0.13	16.55±0.11	16.13±0.09	16.14±0.13	16.35±0.14	16.56±0.11
-Clustering	15.81±0.11	15.92±0.14	16.11±0.08	16.55±0.10	15.70±0.10	15.77±0.11	15.97±0.08	16.43±0.10	15.98±0.06	15.90±0.06	16.01±0.08	16.45±0.11