On the Regularization of Learnable Embeddings for Time Series Forecasting
Authors: Luca Butera
Giovanni De Felice
Andrea Cini
Cesare Alippi
Venue: Accepted by TMLR ()

Abstract: In forecasting multiple time series, accounting for the individual features of each sequence can be challenging. To address this, modern deep learning methods for time series analysis combine a shared (global) model with local layers, specific to each time series, often implemented as learnable embeddings. Ideally, these local embeddings should encode meaningful representations of the unique dynamics of each sequence. However, when these are learned end-to-end as parameters of a forecasting model, they may end up acting as mere sequence identifiers. Shared processing blocks may then become reliant on such identifiers, limiting their transferability to new contexts. In this paper, we address this issue by investigating methods to regularize the learning of local learnable embeddings for time series processing. Specifically, we perform the first extensive empirical study on the subject and show how such regularizations consistently improve performance in widely adopted architectures. Furthermore, we show that methods attempting to prevent the co-adaptation of local and global parameters by means of embeddings perturbation are particularly effective in this context. In this regard, we include in the comparison several perturbation-based regularization methods, going as far as periodically resetting the embeddings during training. The obtained results provide an important contribution to understanding the interplay between learnable local parameters and shared processing layers: a key challenge in modern time series processing models and a step toward developing effective foundation models for time series.

Published in Transactions on Machine Learning Research (02/2025)
On the Regularization of Learnable Embeddings for Time Series Forecasting
Luca Butera
Università della Suizzero Italiana, IDSIA
Giovanni De Felice
g. gedeffelice
University of Liverpool
Andrea Cini
andrea.ccniie www.ciiiiiiiccmmmm ch
Universitd della Svizzera Italiana, IDSIA
Cesare Alippi Università della Svizzero Italiana, IDSIA Politecnico di Milano
cesore. aesare.alippi@usii.c
Reviewed OpenReview:
https: openreview. net/ forum? id= F5ALCh33GG
Abstract
In forecasting multiple time series, accounting for the individual features of each sequence can be challenging. To address this, modern deep learning methods for time series analysis combine a shared (global) model with local layers, specific to each time series, often imple- mented as learnable embeddings. Ideally, these local embeddings should encode meaningful representations of the unique dynamics of each sequence. However, when these are learned end-to-end as parameters identifiers. Shared processing blocks may then become reliant on such identifiers, limiting their transferability to contexts. In this paper, we address this issue by investigating methods to regularize the learning of local learnable embeddings for time series processing, Specifically, perform the first extensive empirical study on the subject and show how such regularizations consistently improve performance in widely adopted architectures. Further- morc, we show that methods attempting to prevent the co-adaptation of local and global parameters by means of embeddings perturbation are particularly effective in this context. In this regard, we include in the comparison several perturbation-basel regularization methods, going far as periodically resetting the embeddings during training. The obtained results provide important contribution to understanding the interplay between learnable local parameters and shared processing layers: key challenge in modern time series processing models and step toward developing effective foundation models for time series.
forecasting model, they may end up acting as mere sequence
1
Introduction
Collections of related time series characterize many applications of learning systems in the real world, such as
traffic monitoring Li et al.
2018
Yu et al. 2018 energy analytics (Dimoulkas et al. 2019 Gasparin et al.
2022
climate modeling Ma al. 2023 Chen al. 2023 and biomedical data processing (Jarrott al.
2021
Zhang al.
2022
The success of deep learning the associated tasks, e.g., forecasting (Shih al.
2019
Benidis et al.
2022
imputation (Cao et. al.
2018
Cini et al. 2022) and virtual sensing (Wu et al. 2021

---

Published in Transactions on Machine Learning Research (02/2025)
relies on effectively modeling shared patterns across time series while also accounting for their individual characteristics (Benidis et al. 2022) In this context, models must, rely on some attributes or positional encodings to tailor the processing to the target time series, with the risk of requiring long observation windows
and high model capacity when those are not available (Salinas ct al.) 2020: Montero-Manso & Hyndman 2021). Indeed, prior and positional information is often unavailable or insufficient to provide effective special ization. Notably, the study of methods to tune a shared time scries model on a specific task is a major concern for the development of foundation models, i.e., backbone models trained on large collections of time series and then applied to specific target applications (Garza & Alergenthaler-Canenn
2023 Liang et al. 2024). This problem is also particularly relevant in spataotemporal forecasting when dealing with multiple time series coming from a sensor network (Bai et al. 2020 Cini et al. 2023b).
Among different methods, research has addressed the problem by looking into hybrid global-local architectures, i.c., architectures that combine global models with local learnable components (e.g.. layers) specific to a target time scries (Wang et al. 2019 Smyl, 2020). As a prominent example, [Smy] (2020) won the M4 competition (Makridakis el al. 2020) by combining a shared recurrent neural network (RNN) with local exponential smoothing models fitted on each time series. In the context of spatiotemporal data, research has
shifted towards the adoption of learnable embeddings (Bai et al. 2020 Shao et al. 2022). i.c., vectors of learnable parameters, to reduce the cost of learning (more complex) local processing blocks (Cini et al. 2023b). Each embedding, associated with a specific time series, is fed into the shared modeling architecture and trained end-to-end alongside it. These representations can go beyond simply encoding coordinates, as they can account for the dynamics of each time series with respect to the other sequences in the collection (Cini et al. 2023b). Parameters of this kind are analogous to the word embeddings used in natural language processing (NLP) (Mikolov ct al. 2013; Peng et al. 2015). We will refer to them local embeddings.
While local embeddings offer significant advantages, their adoption introduces potential drawbacks. In the first place, differently from NLP applications that operate on a fixed-size dictionary, the target time series might change over time, and new sequences might be added to the collection. Secondly, since embeddings are Iearned jointly with the entire architecture, co-adaplation (Srivastava el al. 2014)) with the shared layers might result. in embeddings being used as simple sequence identifiers (Geirhos et al., 2020). This interdependence can hinder the flexibility of the shared procossing blocks and limit their transferability. Existing works (Yin et al., 2022 Cini et al. 2023b Prabowo et al.) 2024 show evidence that constraining the structure of the embedding space can lead to transferability improvements. However, no prior work has systematically addressed and evaluated regularization methods for local embeddings in forecasting architectures.
To fill this void, we investigate the impact of regularizing the learning of local embeddings for related time series within a selection of commonly used deep learning forecasting architectures. In particular, we perform an extensive empirical study comparing a variety of regularization strategies, ranging from standard approaches, such weight penalties and dropout, to more advanced methods, e.g., clustering and variational regularization. Our analysis considers range of scenarios, including transductive settings, transfer learning, and sonsitivity analyses. Empirical results show that methods that attempt to prevent the co-adaptation of the local and global blocks by perturbing the embeddings at training time are consistently among the best-performing approaches. To further validate this observation, we include in the analysis a forget-and- relearn strategy (Zhou et al., 2021), named forgetting regularization, that periodically resets embeddings during training. The results obtained by considering this approach further show that perturbation of local parameters offers good design principle for regularization strategies in this context. Our main findings can be summarized as follows.
(F1) In both transductive and transfer learning settings, regularizing the local embeddings can provide consistent performance improvements across several forccasting architectures.
(F2) Regularization strategies based on local parameters' perturbations aimed at preventing co-adaptation consistently lead to larger performance gains compared to other approaches.
F3) Finding F2) can be used as a design principle for designing new regularization strategics to mitigate overfitting and improve transferability, as highlighted by the competitive performance of the considered forgetting regularization.
2

---

Published in Transactions on Machine Learning Research (02/2025)
The regularization of local learnable embeddings emerges often neglected but central aspect, which, with negligible computational overhead, can lead to performance improvements across all the considered benchmarks. This makes our study an important missing piece for guiding practitioners and researchers in building and designing modern neural architectures for time series processing.
2 Preliminaries
This section introduces the notation and formalizes the problem of time series lorecasting, with a focus on the class of deep learning architectures we consider in this paper.
2.1 Problem settings
Consider a collection D of N time series, where each i-th time series consists of a sequence of ' observations {æj E dd 1 Specifically, we indicate as related time series set of homogenous time series coming from the same domain but generated by different sources (e.g,, different sensors). Examples include sales figures for different products, or energy consumption of various users. Time series might be acquired both asynchronously and/or synchronously; the latter case, we denote the stacked N observations at time step t by the matrix X, E Rxx,, We the shorthands X indicate the sequence of observations within the time interval it, (T)++T and X to indicate those up to time (included). Eventual (exogenous) covariates associated with each time series denoted by ut E Idd (U, (g.)...). We focus multistep-ahead time series forecasting, i... the problem of predicting the next H future values for each i-th time series æ H given exogenous variables and a window of W past observations. We focus on point forecasts, while probabilistic predictors might also be considered.
2.2 Hybrid global-local architectures for time series
We consider a broad class of models similar those in Benidis al. 2022 and Cini et al. 2023a In particular, consider predictors such
E++H F(x)_ ut W. 0), i 1,
(1)
where 0 are the learnable parameters of the model, #fa+ indicates the predicted values of a and F): is family of parametric forecasting models. Models in Eq. | do not take into account (spatial) dependencies that might exist among time series. In scenarios such as spatiotemporal forecasting, where such dependencies might be relevant to achieve accurate predictions, models that can operate on sets of (synchronous) correlated time scries should be preferred. In particular, we consider a family of models
X XL+H F(X UL 0),
VS C
where F(:; O) operates on subsets S of the collection of time scries D, and X E RlS)xd is the corresponding stack of observations at time step t. Models this kind can be implemented by architectures operating on sets (Zaheer et al., 2017), e.g., attention-based architectures (Vaswani et al. 2017 Grigsby et al. 2021] or spatiotemporal graph neural networks (STGNNs) (Jin al.) [2023] Cini al. 2023a) based on message passing GGimer et al.] 2017p. These models can eventually account for priors existing dependencies among time series, which could be encoded. e.g., by graph adjacency matrix A E NxN (Cini et al. 2023a Note that models in both families (Eqs. and 2 are global, i.c., they share parameters among all the time series being processed, with clear advantages in terms of sample efficiency (Montero-Manso & Hyndman 2021] Most of the recent successes in applying deep learning to time series forecasting are based on the idea of implementing such global models with neural network (Benidis et al.) (022). In the following, we will can be seen as special case where | 1. In particular, we consider the whole collection at (i.e., S N) and drop the superscript. S to simplify the notation.
consider models belonging to the family in Eq. 2 as Eq.
3

---

Published in Transactions on Machine Learning Research (02/2025)
History X Wa
Embeddings
Embeddings
Predictions
X-W
E
E
X X
Encoder
PropLayers
Decoder
time
shared (global) parameters
local parameters
concatenation
Figure Overview of the hybrid global-local time series forecasting framework.
Template architecture We use template architecture analogous that of Cini 2023b). It consists of the following three processing steps: encoder, one propagation layers, and decoder. Predictions at each time step and lime series are obtained
h 1.0 EXCODER (zi u/-1)
(3)
H ++ PROPLAYER (H' 1= 1= 0,0,1,..,44
(4)
8 * = DECODER
(1i (
(5)
where the additional index refers to the layer depth, hid e Rd is the representation associated with the i-th time series at the I-th layer, and H E R|S]xd, the stack of such representations. The ENCODER (+) and DECODER (+) blocks can be implemented in different ways, e.g. linear layer or a multilayer perceptron (MLP), and operate on single time steps and time series. Conversely, the PROPLAYER (+) blocks are the only components of the architecture propagating representations across the temporal dimension and across time series in the collection. Each PROPLAYER (-) can be implemented by any existing sequence modcling archi- lecture, e.g., RNNs (Hochreiter & Schmidhuber 1997 Cho el. al. 20144] or temporal convolutional networks (TCNs) (LeCun & Bengio 1998 Bai et al. 2018) and/ /or spatiotemporal operators, e.g., spatiotemporal attention (STAtt) models (Grigsby et al. 2021 Deihim et al.) (003).. In particular, STGNNs (Soo al. 2018 Yu et al.] 2018), i.e., models that combine sequence modeling architectures with message passing operators, are among the most popular architectures for implementing propagation layers when processing correlated (Jin et al.) 2023).
time series
Hybrid global-local architectures and local embeddings As anticipated in the introduction, despite their advantages, global models might struggle to account for the specific dynamics of each time series. Hybrid global-local architectures (Benidis et al., 2022 address this issue by including parameters specific to each target time series. If indicate these parameters as [ provide forecasts for the time series in the collection
...., N3. the resulting model family would
= (x.-
U. Wi-Wi-Hi 0, ()
(6)
Although many implementations of such models exist (Benidis et al. 2022 Cini al. 2023b), we consider models where the local components such that ① E and d' = e'. In particular, each local embedding e' is associated with the corresponding i-th time series and can be learned end-to-end jointly with the shared network weights. Embeddings are incorporated into the template architecture at both the encoder and decoder level by concatenating them to the input as
implemented embeddings E Nxd. of learnable parameters
h ENCODER (x/_1|) u(,u(_)) & = DECODER (n. (N uustH)
Fig. 1 provides an overview of the resulting reference architecture. The addition of these parameters comes at cost in terms of flexibility, as processing time series that were not observed at training time requires fitting parameters (Januschowski et al. 2020 Cini et. al. 2023b
4

---

Published in Transactions on Machine Learning Research (02/2025)
3
Related works
Learnable embeddings are key components of statc-of-the-art time series processing architectures such as STGNNs [Bai et al. 2020 Cini et al. 2023a and atlention-based models (Marisca et al. 2022 Liu al.] 2023
Xiao et al. 2024). In particular, Cini et al. (2023b) systematically addresses the role of such embeddings
in bybrid global-ocal STGNNs. Aside from modeling local dynamics, embeddings have been used extensively
to amortize the cost of learning the full adjacency matrix in graph-based models (Wu et al.) 2019 Shang
let al.
2021 Satorras et al. 2022
De Felice et al.) 2024), or as spatial positional encodings (Marisca et al.
2022
Shao et al. 2022 Liu et al.
2023 They are also routinely used in NLP as word embeddings (Mikolov
et al.
2013) and often as spatial positional encodings (Devlin et al., 2019 Yang al. 2019).
Methods to regularize learning architectures are obviously central to deep and machine learning in gen- Traditional examples include L1 (lasso) (Tibshirani 1996) and
eral (Ying] 2019: Tian & Zhang 2022).
1.2 (ridge, weight decay) (Krogh & Hertz
1991 | regularizations. Established approaches in deep learning
encompass dropout (Srivastava et al. 2014). layer normalization (Ba et al. (016) and weight normalization (Sal
imans & Kingma, 2016). Notably, several regularizations have been tailored to specific architectures (Zaremba
el al.] 2014 Gal & Ghahramani 2016 Dieng el al.
2018
Wang & Niepert 2019: Santos & Papal 2022) , and
designed improve transferability (Wang et al.
2018
Takada & Fujisawa [020 Abuddwwiili et al. 2021] p.
presents a comparative study of different regularizations for word embeddings in NLP. Regarding regularizations for learnable embeddings for time series, Yin et al 2022
Similarly in spirit to our work, Peng et al. (2015)
uses learned embedding clusters to regularize fine-tuning on target data, facilitating transfer in graph-basel forecasting. Similarly, Cini et al. (2023b) uses a cluster-based regularization, as well as a variational-basse approach. However, these works only address model transferability, without considering the impact of local embedding regularization in a broader context, e.g., transductive settings.
4 Regularization strategies for local embeddings
In this section, we discuss some possible shortcomings of hybrid global-local forecasting architectures and present the regularization strategies employed in our experimental analysis.
Global models are inherently less likely to overspecialize to individual sequences since all parameters are shared among time series. The introduction of local embeddings reduces the regularization effect brought by this inductive bias and jointly learning local and global parameters end-to-end on downstream task may result in model that is more likely to overfit to individual time series, with poten- tial negative impact on both performance and transfcrability (Cini et al.) 2023b). In the following, we use the term co-adaplation to refer to these overfitting phenomena coming from training the local embeddings end-to-end with the global model.
emb.
Dropout (whole model)
Nu regul. Dropout (emb. only)
2.80
2.75
MAE
2.70
Model regularization is a common approach to deal with overfitting and to control model and sample complexity. For instance, one might apply well-known techniques, such as weight decay (Krogh & Hertz 1991 or dropout (Srivastava et al. 2014), across the entire network. However, in many scenarios, the global processing blocks are not particularly overparametrized, as weights are shared, and regularization might not be necessary. In global-local models, one could consider regularizing the local parameters only, i.e., in settings, constraining how embeddings are Iearned. Fig. 2 empirically supports this inluition with an illustrative example: given a global time-series forecasting model with fixed hyperparameters (green), adding local embeddings (orange) improves performance, but makes it more likely to overfit (as model complexity increases). Regularizing the entire global-local model (purple), however, hinders performance
2.65
0
50
100
150
Epochs
Figure 2: Validation curves when reg- ularizing whole model or just local parameters in time series forecast- ing task (STGNN model, METR-LA datasct, 50 runs, 11std). Models and datasets are discussed in Sec. 5
5

---

Published in Transactions on Machine Learning Research (02/2025)
in this scenario. Conversely, applying regularization (dropout in this case) to the embeddings only (red), regularizes the model effectively. Note that regularization of the global parameters can obviously still be beneficial in the case of an over-parametrized global model.
4.1 Regularization methods
In the following, present the regularization methods that will later be part of our experimental analysis in Sec. 5 They range from adaptations of standard approaches to recent contributions from the literature. We also consider approach based on parameter resetting.
L1 and L2 regularization L2 regularization (Koghh & Hertz 1991p, also known as weight decny, consists in adding to the loss a penalty term proportional to the magnitude of the model's weights. When applied to the embeddings, the penalty term is C12 (E) = Ar2 |E2. Similarly, L1 regularization (Tibshirani 1996), also known as lasso, consists in applying a penalty term of L11 (E) = Au |E||1. The positive scalar values, \12 and An, control the regularization strength.
Dropout Dropout (Srivastava et al. 2014) is another widely recognized regularization. This consists randomly masking out individual neurons during training, with probability p, while scaling the others by I-p' This technique heuristically forces the network to learn robust representations, often preventing overfitting. In our context, we apply it to the embedding vectors by randomly masking out some parameters for each embedding.
The clustering regularization introduced in (Cini et al. 2023b) is based on learning set of cluster centroids and a cluster assignment matrix, alongside the embeddings. A regularization term, with weight Actat, is then added to the loss to minimize the distance between embeddings and the assigned centroid. It was originally designed to structure the embedding space and improve model transferability.
Clustering
Variational regularization Varialional regularization Cini et al. 2023b consists in modeling embeddings as samples from Gaussian posterior learned from data e' N (Hwar diag (otur)): where phar and ofar are the learnable local parameters. A penalty term based on the KL-divergence between the learned distribution and standard Gaussian prior, with weight Avar, is added to the loss. Similarly to the clustering regularization, it was introduced to enhance model transferability.
Forgetting We also consider a novel sorgelling regularization, based on the Jorgel-and-relearr paradigm (Zhou et al.) 202I). This refers to strategies where the parameters of some neural network
layers are occasionally reset during training. By doing this, it is possible to reduce memorization of training samples (Baldock et al.
2021), avoiding shortcuts and learning representations that generalize better (Geirhos
jet al. 2020 Zhhou et al., (021). ). We adopt this paradigm in the context of hybrid global-local time series forecasting architectures and propose the following procedure. Every K training epochs, we periodically reset local embeddings E to a sample from shared initialization distribution Pe. Concurrently, we similarly reset the ENCODER's and DECODER's weights (Eq. 7 that directly multiply the embeddings (see Appendix C.1 The value of K can be casily selected empirically (sce Appendix C.2 Additionally, resetting is halted after a certain amount of epochs to allow for convergence to a final configuration; this can be easily tuned manually, by considering the model's convergence speed or triggered automatically, by monitoring the validation loss. Note that resetting local parameters to regularize the training of related time series forecasting model has never been explored in the litcrature.
4.2 Preventing co-adaptation
As mentioned in Sec. 4 learning local and global parameters jointly on a downstream task can result in overfitting. We use the term co-adaptation to indicate this phenomenon, by referring to global and local parameters co-adapting during training. Specifically, local embeddings might simply act as sequence identifiers and result in models that rely entirely on this identification mechanism. Besides the negative effects on sample efficiency, the resulting architecture would likely lose flexibility in terms of the transferability of the
6

---

Published in Transactions on Machine Learning Research (02/2025)
learned representations and processing blocks. Furthermore, this behavior is in contrast with the intended one for local embeddings, iee., encoding meaningful characteristics of the target time series. One way in which regularization methods can aim to prevent such co-adaptation is to actively perturb the local parameters during training so that subsequent layers cannot rely on specific feature values (Srivastava et al.) 2014). In our case, the global processing block would be less likely to rely on specific values of the embeddings when processing the target time series. In the following, we will use the term perturbation in a broad sense, referring to strategies that modify the values of local parameters training time, e.g., resampling, zeroing, or adding noise.
Among the considered regularization methods, dropout perturbs the embeddings by randomly zeroing out parameters, as similarly done for the inputs/outpute of subsequent layers to avoid the co-adaptation of the associated weights (Srivastava et al.) 2014). The variational regularization, given a common prior, learns each embedding's sampling distribution end-to-end while resampling their actual values before providing them as inputs to the downstream at each forward pass. Finally, forgetting regularization introduces perturbation in the early stages of training by periodically resetting the local parameters. In particular, by considering this strategy for the first time in this context, aim to test whether actively perturbing local parameters offers an effective principle for the design of new regularizations for hybrid time scries forccasting models.
Regarding the other regularization methods considered in the analysis, L. and L.2 regularizations simply penalize the embeddings' magnitude, while clustering similarly penalizes the distance of the embeddings from the learned centroids, forcing them to occupy specific regions within the embedding space. While these regularizations can provide structure to the embedding space, they do not actively perturb embeddings" values. The empirical results presented in Sec. 5 altogether suggest that strategies based on embedding perturbations are more effective at preventing the kind of overfitting we previously discussed, i.e., co-adaptation.
5 Experiments
We evaluate the effectiveness of different regularization strategies for local embeddings under three different scenarios: time series lorecasting benchmarks (Sec. 5.1 transfer learning (Sec. 5.3) and a sensitivily analysis
through embedding perturbations (Sec. 5.4]. We consider six real-world datasets of time series collections,
spanning four different application domains: METR-LA and PEMS-BAY (Li et al., 2018) as two established
2015) from the air quality domain, CER-E (CER 2016) from the energy consumption domain, CLM-D De Felice et al. 2024) and EngRAD (Marisca et al, 2024) as two multivariate climatic datasets. Details on the datasets, data splits and forecasting settings can be found in Appendix A
benchmarks for traffic forecasting, AQI (Zheng et al.
Regarding the investigated models, we consider three different hybrid global-local architectures (Sec. 2.2 distinguished by three different implementations of the propagation layer (Eq. 4 In particular: 1) a RNN with gated recurrent units (GRUs) cells (Cho et al. 2014b) (RNN), 2) STGNN stacking GRU and anisotropic message- pariggg-ppsnngggggg layers (Bresson & Laurent 2017 (STGNN), and 3) GRU followed by multi-head attention across sequences (Vaswani et al. 2017) (STAtt). Such hybrid architectures are representative of the current state of the art in the considered benchmarks and of the dominant deep learning
frameworks for processing sets of related time series (Benidis et al. 2022 Cini et al. 2023a). Implementation
and architectural details are provided in Appendix while details on the experimental settings can be found in Appendix D
5.1 Time series forecasting benchmarks
In our first experiment, we consider the problem of time series forecasting in transductive setting, i.e., the set of time series to be forecast is the same set observed at training time. The forecasting horizon is dataset-dependent and reported in Tab. of Appendix A. We use models without local parameters and with un-regularized local embeddings as reference baselines. Then, we evaluate performance with regularized local embeddings, adopting the different strategies detailed in Sec. [.1] For model selection, a hyperparameter search on the hidden size (i.c., the number of units in cach model's hidden layer) and learning rate has been carried out independently for each model variant and dataset. All regularization hyperparameters have been set as detailed in Appendix D
7

---

Published in Transactions on Machine Learning Research (02/2025)
Table Forecasting test error under optimal model size and learning rate (5 runs, #1std). Methods equal to or better than the corresponding standard architectures (RNN, STGNN, STAtt) with embeddings (+ EMB.) are in bold. The best-performing method within each dataset and model is in red. '4 REGULARIZATION denotes the addition of that specific regularization only, top of ++ EMB.'
DATASET
METR-LA
BAMS-BAYY
CER-E
AQT
CLM-D
ENGRAD
MODEL
MAE
MAE
MAE +
MAE +
MMRE +
MMRR. &
RNN
3.5564.00
1.774 .002
0.4453 .0100
13.279
19.664.01
31.044.04
+ EMB.
3.148+.011
1.593+.000
0.4146 0025
13.247
19.41+.01
31.01+.10
+ L1
3.149+.007
1.590+.000
0.4079 +.029900023 .0233
13.147 +.047
19.47+.01
31.02+.11
+ L2
3.146+.009
1.586+.005
0.4058#.000
13.181
19.42+.01
30.97
+ CLUST.
3.138
1.588
0.4115 .0255 .0025
13.231
19.41
31.05+.07
+ DROP.
3.147+.012
1.580+.007
0.4104 .005
13.114+.038
19.43+.01
30.99
+ VARI.
3.132+.006
1.589
0.4050 +
13.113
19.39
30.98+.05
+ FORG.
3.149+.007
1.590+.007
0.4049 0007
13.185+.022
19.42+.02
30.92+.02
STGNN
3.239
1.660
0.4275 0006
11.814
19.19
28.04
+ EMB.
3.027+.009
1.593
0.4144 0032
11.881+.053
18.89+.04
27.52+.09
+ LI
3.040+.016
1.5874.000
0.4039+.00m
11.7894.043
18.92+.03
27.45
+ L2
3.023+.009
1.582+.00
0.4016+.001
11.795 11.795+.022
18.87
27.44 27.44+.14
+ CLUST.
3.025
1.580 .005
0.4075 .0220 .000
11.876 1.633
18.854.04
27.604.14
+ DROP.
3.036+.011
1.575 0006
0.4042 .000 .0088
11.712+.016
18.93+.04
27.41
+ VARI.
3.0134.005
1.5664.000
0.3989
100009..000 0012
11.7684.o0
18.854.02
27.534.10
+ FORG.
3.0501.017
1.578 4.07
0.4026 .006
11.793
18.834.02
27.47 1.12
STATT
3.538+.004
1.776+.002
0.1179 0014
13.311+.330
19.71+.02
29.25+.09
+ EMB.
3.074+.020
1.616+.012
0.4143 .0022
12.973+.402
19.10+.03
28.32+.08
+ L1
3.0614.021
1.590+.005
0.4072
12.799
28. 28.15 19.27
+ L2
3.058
1.593 0.00
0.4061 .0004
4.004
13.542
19.17
28. 78.124.10
+ CLUST.
3.061
1.590+.008
+.4084+.0017
12.535+.240
27.96 19.06+.01
+ DROP.
3.0584.005
1.565 0004
0.40684.00m
12.4434.482
19.30
28. 124.28
+ VARI.
3.041+.009
1.571 4.571+008 005
0.4030+.0014
12.616
19.10+.06
28. 01 +.13
+ FORG.
3.0584.011
1.579 +.00
0.4062#.0012
12.455+.416
27.9 27.94 19.164.02
Tab. shows the obtained results, while Tab. provides a concise summary of the performance of each regularization across experimental settings. Regard- less of the strategy, regularizing the learning of local embeddings provides consistent performance improvements over non-regularized models in most datasets F1 Considering the negligible computational overhead and ease of implementation. these results support the adoption of such techniques as standard practice. While there no clear winner among the different regularization modhods, the variational regularization appears be the most effective, average; followed by forgetting and dropout. This suggests that methods perturbing the embedding values can be more effective in preventing co-adaptation F2) F3). We can speculate that the effectiveness of the variational regularization stems from it jointly perturbing embeddings values while also providing structure to the embedding space. Additional results on the impact of regularizing embeddings when Eq. is implemented by a simple MLP are reported in Appendix E while results regarding the combination of multiple regularization techniques be found in Appendix F
Table 2: Summary statistics for Tab. 1 Columns indi- cate the average improvement (%) over the reference un- regularized model and mean relative rank. Best in red, second in bold.
REG.
ZIMPR.
RANK
L,1 L2
0.58
4.6
0.45
3.4
CLUST.
0.67
4.0
DROP.
0.94
3.2
VARI.
1.21
2.0
FORG.
0.95
3.0
8

---

Published in Transactions on Machine Learning Research (02/2025)
STÄtt. (8) METR-LA]
STAtt. (32) METR-LA]
STAtt. (128) [METR-LA
2.85
2.90
2.90
2.85
2.80
2.85
2.80
2.75
MAE
2.80
2.75
2.70
2.75
2.65
2.70
2.65
2.65
2.60
100
100
200
100
Epochs
Epochs
Epochs
No regularization
L2
Dropout Variational
Forgetting
L1
Clustering
Figure 3: Validation curves for different training scenarios (5 runs, #lstd). Plot names follow the convention model (embedding size) [dalasel].
5.2 Sensitivity analysis and learning curves
To provide additional insight, investigate how different regularizations affect the learning curve across different embedding sizes. In doing rate to Ir = 0.00075. Specifically, Fig. 3 shows the validation mean absolute error (MAE) across training epochs for some example scenarios; different colors correspond to different regularization strategies. For completeness, additional plots in complementary settings are reported Appendix G Comparing the different regularizations, we can see that dropout (red) and forgetting (blue) significantly affect the learning curves, even with large embeddings (Fig. 3 right). However, dropout, one could expect, might be problematic when applied to embeddings of limited size (Fig. left). The forgetting regularization is robust across the different configurations. The variational regularization appears less disruptive of the learning dynamics. The remaining regularizations (i.c., L1, L2, and clustcring) have little impact on the learning curve. Overall, regularization strategies that perturb the local embeddings seem more likely to positively alfect the learning curves, which might be a reason for the superior performance shown Tab. 2
fix the shared model hidden size to da = 64 and learning
5.3 Transfer learning
Our second main experiment consists of time scries forecasting under transfer learning setting, adapted from previous works (Cini et al. 2023b). This benchmark aims to verify the impact of regularizing the local embeddings on the transferability of the global (shared) processing blocks. We consider the 4 PEMS benchmark datasets from Guo et al. 2021 (i.e., PEMS03, PEMS04, PEMS07, PEMS08) and a reference STGNN model with local embeddings. For each subset of three datasets, we first train the entire model, then we reset the embeddings their initial values and finc-tune them exclusively on the held-out datasot. During fine-tuning, shared parameters are kept frozen. The different regularizations are applied only during the initial training procedure and disabled during fine-tuning. Tab. 3 reports the results for the un-regularized and regularized models the amount of data for fine-tuning varies (zero-shot refers to fine-tuning at all). It is evident how dropout, variational rogularization, and forgetting are the most reliable and effective strategies (F2) Notably, dropout excels in the zero-shot setting, while forgetting is consistent across fine- [F3) This suggests dropout can be particularly useful when data from the transfer domain scarce or absent. Conversely, regularizations based on structuring the embedding space, i.c., L1, L2, and the clustering regularization, show mixed results compared to the unconstrained model.
tuning lengths and excels when more data are available

---

Published in Transactions on Machine Learning Research (02/2025)
Table 3: Forecasting test error (MAE) in transfer learning (5 runs, #lstd, STGNN). Within each target dataset, different rows pertain to different fine-tuning data lengths. Bold denotes methods equal or better than the baseline (+ EMB.), best method is in red. '+ REGULARIZATION" denotes the addition of that specific regularization only, top of ++ EMB.'
TR. SIZE
+ EMB.
+ L1
+ L2
+ CLUST.
DROPOUT
VARIA.
FORGET.
ZERO-SHOT
26.19+.9 .88
26.06
=66
25.18%. +.40
21.45=.22
22.71 #.13
25.08+.40
PEMS03
DAY
18.92+.0s 00
18.89
18.87 =.08
18.77 +.05
17.99=.02
18.17 #.01
18.554.09
DAYS
18.4
18.41 1.07
18.35 2.05
18.53
17.86..05
17.91 8.00
18.19 19...0
1 WEEK
17.534.02
17.524.00
17.47
17.59
17.31=.04
17.264.02
17.344.00
2 WEEKS
17.34+.03
17.34
17.28 17.28=.01
17.37+.01
17.20=.02
17.16+.02
17. 17.20
HERO-SHOT
29.234.4 43
29.09 26
29.26
30.03
26.13
27.564.31
28.444.34
PEMS04
DAY
23.93+.16
24.00
23.96
23.71
23.04
23.33 #.12
23.544.13
3 DAYS
23.181.12
23.22
23.27
22.971.00
22.67
22.87 1.06
22.861.08
1 WEEK
22.47+.07
22.464.us
22.45_.07
22.494.07
22.29
22.38
22.264.08
WEEKS
21.92+.04
21.93
21.93
22.00
21.96+.03
22.04
21.79+.03
ZERO-SHOT
57.40
55.30
53.97 23773..5
56.9241.62
34.19
42.9942.84
42.67
PEMS07
DAY
29.61
29.63
29.37
29.00
27.25=.12
28.30 #.41
28.38
3 DAYS
27.654.14
27.744.14
27.56_.14
27.56
26.37
26.92
26.921.08
WEEK
26.60+.04
26.66+.00 .99
26.60=.07
26.90
25.87=.13
26.20
26.03
WEEKS
25.84 .04
25.90
25.76 =.07
26.31
25.53 :..5
25.66 1.
25.41
ZERO-SHOT
25.41
25.26+1.00
25.90=.47
25.60+.41
21.04-.17
22.60
23.90
PEMSO8
DAY
18.91
18.88 +
18.85
18.64
17.96#.06
18.23 1.
18.40
DAYS
18.114.10
18.164.13
18.15_.08
18.03
17.552.06
17.77
17.83
WEEK
17.33+.08
17.28
17. 17.33=.05
17.40#.01
17.22=.06
17.31
17.11
WEEKS
17.134.06
17.18
17.15
17.25
17.15
17.20
16.99 4.09
25
6
24
No regul.
A MAB
23
L1
MAE
5
L2
22
Clust.
Dropout
Vari.
Forget.
3
0
21
22
Rearranged Mean Sampled Perturbation
Noise O
Figure 4: Test performance degradation embeddings perturbation (5 runs, #1std, STGNN, METR-LA). Left: Adding zero-mean Gaussian noise. Right: (Left) random shuffling, (Middle) replaced with their mean, and (Right) replaced by draw from their sample normal.
10

---

Published in Transactions on Machine Learning Research (02/2025)
5.4 Robustness to local parameter perturbation
Finally, we investigate how a model's forccasting accuracy is affected by the perturbation of the learned local embeddings. This provides insights into the robustness of the shared learned parameters and can serve a praxy for the effectiveness of different regularizations in preventing co-adaptation. To avoid penalizing regularizations that are sensitive to the weights' magnitude (i.e., L2, L1), we consider perturbations that do not impact the scale of the learned representations. In particular, we experiment with four such strategies: adding zero-mean Gaussian noise to the embeddings (Noise a), randomly shuffling embeddings across sequences (Rearranged); replacing each embedding with their sample mean (Mean), resampling each embedding from their sample normal distribution (Sampled) (see Appendix D.1 for formal definition). Fig. 4 shows the results of the analysis after training an STGNN on the METR-LA dataset (complementary settings are illustrated in Appendix G As one might expect, regularization methods that actively perturb the embeddings while learning (i.e., dropout, forgetting, variational) consistently result in more robust models under all the considered perturbations
(F2) F3) Conversely, other regularizations have marginal, or even negative, impact. We observe some consistency between the results observed in Fig. f and performance transfer learning, shown in Tab.
5.5 Discussion
Considering the observed empirical results, we can summarize set of practical recommendations that can provide practitioners with a starting point for model search. In transductive settings, variational regularization performs reliably across benchmarks and it is safe choice. Morcover, suggest favoring techniques that perturb the embedding values during training, e.g., varialional regularization, forgelling or dropoul. In the context of transfer learning, our results suggest the adoption of dropout, when fine-tuning the model on a low amount of data, while forgetting or variational regularization can give better results as more data become available. The recommendation of prioritizing techniques that perturb the embeddings at training time remains valid in the transfer learning settings.
6 Conclusions
This paper highlights the importance of regularizing the learning of local embeddings in modern deep-learning architectures for time series forecasting. Our empirical study, across diverse datascts and scenarios, provides clear evidence that this practice is beneficial for a variety of reference architectures, representative of the state of the art. Notably, even simple techniques, when applied to local embeddings, can yield consistent performance gains. Furthermore, we observed that methods that actively perturb the embeddings at training time, such dropout, variational regularization, and forgetting, consistently rank among the top performers, both in transductive and transfer learning settings. The use of this property as guideline in designing new regularization methods is further supported by the effectiveness of the unprecedentedly considered forgetting regularization. Overall, our findings offer strong empirical argument for the adoption of embedding regularization as standard practice when designing hybrid global-local architectures for time series. We believe this an important building block towards developing more robust and transferable models for spatiotemporal data and designing foundation models for related time series processing.
Limitations and future works While it is clear that these regularizations provide consistent advantages, it is difficult to strongly identify the best- pest-performing method in the different scenarios. Hence, future works might focus on finding strategics that combine the best qualitics of the different methods. Furthermore, future studies could try to provide methods to quantitatively and analytically characterize the co-adaptation of global and local parameters, for which we can only identify indirect signs. In particular, they might design experiments to detect the degencration of local components into identifiers. Morcover, future rescarch could explore how the efficacy of different regularization methods varies with the number of input time series and different ream tasks.
11

---

Published in Transactions on Machine Learning Research (02/2025)
Acknowledgments
Partly supported by International Partnership Program of the Chinese Academy of Sciences under Grant 104GJJH72022013GC
References
Abulikemu Abuduweili, Xingjian Li, Humphrey Shi, Cheng-Zhong Xu, and Dejing Dou. Adaptive consistency regularization for semi-supervised transfer learning. In Proceedings of the IEEE/CVE conference on compuler vision and pallern recognition, pp. 6923 6932, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arxiv: Naoo1607.064550 2016.
Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. Advances neural information processing systems, 33:17804-17815, 2020.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neurol Information Processing Systems, 11-10876-18889) 2021.
Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang Wang, Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael Bohlke-Schneider, David Salinas, Torenzo Stella, et al. Deep learning for time series forecasting: Tutorial and literature survey. ACM Computing Surveys, 55(6):1 36, 2022.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL //wp:////wwwwww/ htttps://wwwwwadd..co www.b.comnddccm Software available from wandb.com.
Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv preprint arXix:1711.0755 2017.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. Advances in neural information processing systems, 31, 2018
Commission for Energy Regulation. CER. CER Smart Metering Project Electricity Customer Behaviour Trial, 2009-2010 [dataset]. Irish Social Science Data Archive, SN: 0012-00, 2016. URL https://www.u ucd
1e/ ae/issda/dan data/ cwwmissionnorreeeeeeooooooooocee commisssoorfforeerrrgggggaaaaooo
Shengchao Chen, Guodong Long, Jing Jiang, Dikai Liu, and Chengqi Zhang, Foundation models for weather and climate data understanding: comprehensive survey. arXiv preprint urXie:2312. 030114, 2023.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. Proceedings of SSST-8, Eighth Workshop on Synlar, Semantics and Structure in Statistical Translation, 2014a.
Kyunghyun Cho, Bart van Merriënboer, Çağlar Gulgehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, 2014b.
Andrea Cini and Ivan Marisca. TorchsSpttiotemppraa//aa]]]
Torch Spatiotemporal, 3 2022.
URL https://github.com
Andrea Cini, Ivan Marisca, and Cosare Alippi. Filling the _p__ss multivariate time scries imputation by graph neural networks. In International Conference on Learning Representations, 2022.
Andrea Cini, Ivan Marisca, Danicle Zambon, and Cesare Alippi. Graph deep learning for time series forecasting. Xix preprint ar orx::2310015999 15978, 2023a.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Andrea Cini, Ivan Marisca, Daniele Zambon, and Cesare Alippi. Taming local effects in graph-based spatiotemporal forecasting. Advances in Newral Information Processing Systems, 36, 2023b.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (clus). In Yoshua Bengio and Yann LeCun (ods.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerlo Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://aaxavvoorra/s//11........
Giovanni De Felice, Andrea Cini, Daniele Zambon, Vladimir V Gusev, and Cesare Alippi. Graph-based virtual sensing from sparse and partial multivariate observations. In International Conference on Learning Representations, 2024.
Azad Deihim, Eduardo Alonso, and Dimitra Apostolopoulou. Surre: A spalio-temporal transformer with relative embeddings for multivariate time series forecasting. Newral Networks, 168:549-559) -594, 2023.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre-training deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North Americon Chapler of the Associalion for Compulational Linguislies: Human Language Technologies, Volume (Long
and Short Papers), 2019.
Adji Bousso Dieng, Rajesh Ranganath, Jaan Altosaar, and David Blei. Noisin: Unbiased regularization for recurrent neural networks. In International Conference on Machine Learning, pp. 1252 1261. PMLR, 2018.
Taas Dimoulkas, Peyman Mazidi, and Lars Herre. Neural networks for gefcom2017 probabilistic load forecasting. International Journal of Forecnsling, 35(4):1409-1422) 2019.
William Falcon and The PyTorrh Lightning team. PyTorch Lightning, March 2019. URL https://github com/Lightning-Aln Aaa.i//gggA//iggiinnggg
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. Advances in neural information processing systems, 29, 2016.
Azul Garza and Max Mergenthaler-Canseen. TimeGPT-1. Xrxiv preprint arXii:2210003558 2023.
Alberto Gasparin, Slobodan Lukovic, and Cesare Alippi. Deep learning for time series forecasting: The electric load case. CAAI Transactions on Intelligence Technology, 7(1):1 25, 2022.
Robert Geirhos, Jörn- Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11): 665-673, 2020.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pp. 1263 1272. PMLR, 2017.
Jake Grigsby, Zhe Wang, and Yanjun Qi. Long-Range Transformers for Dynamic Spatiotemporal Forecasting, 2021.
Shengnan Guo, Youfang Lin, Huaiyu Wan, Xiucheng Li, and Gao Cong. Learning dynamics and hetero- geneity of spatial-temporal graph data for traffic forecasting. IEEE Transactions Knowledge and Data Engineering, 34(11):5415 5:411555422 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 988):1735-177 1780, 1997.
Tim Januschowski, Jan Gasthaus, Yuyang Wang, David Salinas, Valentin Flunkert, Michael Bohlko-Schneider, and Laurent Callot. Criteria for classifying forecasting methods. International Journal of Forecasting, 36 (1):167-177, 2020.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Daniel Jarrett, Jinsung Yoon, Ioana Bica, Zhaozhi Qian, Ari Ercole, and Mihacla van der Schaar. Clairvoyance: A pipeline toolkit for medical time series. In International Conference on. Learning Regresentations, 2021.
URL https://openrevierr
Ming Jin, Huan Yee Koh, Qingsong Wen, Daniele Zambon, Cesare Alippi, Geoffrey Webb, Irwin King, and Shirui Pan. A survey on graph neural networks for time series: Forecasting, classification, imputation, and anomaly detection. arXiv preprint arXiv:2307.03755 2023.
Diederík Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural informalion processing systems, 4, 1991.
Yann LeCun and Yoshua Bengio. Convolutional Networks for Images, Speech, and Time Series, pp. 255-258. MIT Press, Cambridge, MA, USA, 1998. ISBN 0262511029.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network:
Data-driven traflic forecasting. In International Conference on Learning Representations, 2018.
Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongiin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. arXiv preprint or 2rx:::2003.142 1473555 2024.
Hangchen Liu, Zheng Dong, Renhe Jiang, Jiewen Deng, Jinliang Deng, Quanjun Chen, and Xuan Song. Spatio-temporal adaptive embedding makes vanilla transformer sota for trallic forecasting. In Proceedings of the 32nd ACM international conference on information and knowledge management, pp. 4125 4129, 2023.
Minbo Ma, Peng Xie, Fei Teng, Bin Wang, Shenggong Ji, Junbo Zhang, and Tianrui Li. Histgun: Hierarchical spatio-temporal graph neural network for weather forecasting. Information Sciences, 648:119580) 2023.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1) 36(1):54-744 ):5-74, 2020.
Ivan Marisca, Andrea Cini, and Cesare Alippi. Learning to reconstruct missing data from spatiotemporal graphs with sparse observations. Advances in Neural Information Processing Systems, 01322699-20822 2022.
Ivan Marisca, Cesare Alippi, and Filippo Maria Bianchi. Graph-based forecasting with missing data through spatiotemporal downsampling. arXie preprint or arx:::2402. 10634, 2024.
Tomas Mikolov, llya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
Pablo Montero-Manso and Rob J Hyndman. Principles and algorithms for forecasting groups of time series: Locality and globality. International Journal of Forecasting, 37(4):1632-166) 1653, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perfoppefornance deep learning library. Advances in newral information processing systems, 32, 2019.
Hao Peng, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, and Zhi Jin. A comparative study on regularization strategies for enbedding-based neural networks. Conference on Empirical Methods in Natural Language Processing, 2015.
Arian Prabowo, Hao Xue, Wei Shao, Piotr Koniusz, and Flora D Salim. Traffic forecasting on new roads using spatial contrastive pre-training (scpt). Dala Mining and Knowledge Discovery, 38(3):913 937, 2024.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Jamuschowski. Deepar: Probabilistic forecasting with autoregrossive recurrent networks. International journal of forecasting, 36(3):1181-1191, 2020.
Claudio Filipi Gonçalves Dos Santos and João Paulo Papa. Avoiding overfitting: A survey on regularization methods for convolutional neural networks. ACM Compuling Surveys (CSUR), 54(10s):1 -51, 2022.
Victor Garcia Satorras, Syama Sundar Rangapuram, and Tim Januschowski. Multivariate time series forecasting with latent graph inference. arXiv preprint arXa::2033.055: 220:0220303225 2022.
Youngjoo Seo, Michaöl Defferrard, Pierre Vandorgheynst, and Xavier Bresson. Structured sequence modcling with graph convolutional recurrent norwsrks. In International Conference Neural Information Processing, pp. Springer, 2018.
Chao Shang, Jie Chen, and Jinbo Bi. Discrete graph structure learning for forecasting multiple time series. International Conference on Learning Representations, 2021.
Zezhi Shao, Zhao Zhang, Fei Wang, Wei Wei, and Yongjun Xu. Spatial-temporal identity: A simple yet effective baseline for multivariate time series forecasting, In Proceedings of the 31st ACM International Conference Information & Knowledge Management, pp. 4454-4458, 2022.
Shun-Yao Shih, Fan-Keng Sun, and Hung-yi Lec. Temporal pattern attention for multivariate time series forecasting, Machine Learning, 108:1421 1441, 2019.
Slawek Smyl. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecnsting, 36(1):75-80) 85, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Πya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
Masaaki Takada and Hironori Fujisawa. Transfer learning via l regularization. Advances in Newral Information Processing Systems, 33:14266-14277, 2020.
Yingjie Tian and Yuqi Zhang. A comprehensive survey regularization strategies in machine learning. Information Fusion, 80:146-166, 2022.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-28 288, 1996.
Guido Van Rossum and Fred L. Drake. Python 3 Reference Marual. CreateSpace, Scotts Valley, CA, 2009. ISBN 1441412697.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Cheng Wang and Mathias Niepert. State-regularized recurrent neural networks. In International Conference on Machine Learning, pp. 6596-6606. PMLR, 2019.
Yijun Wang, Yingee Xia, Li Zhao, Jiang Bian, Tao Qin, Guiquan Liu, and Tie-Yan Liu. Dual transfer learning for neural machine translation with marginal distribution regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Yuyang Wang, Alex Smola, Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Januschovski. Deep factors for forecasting. In International conference on machine learning, pp. 6607-6617. PMLR, 2019.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Yuankai Wu, Dingyi Zhuang, Aurelie Labbe, and Lijun Sun. Inductive graph neural networks for spatiotem- poral kriging. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 4478 4485, 2021.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial- temporal graph modeling. Proceedings of the Twenty-Eightn International Joint Conference Artificial Intelligence, 2019.
Jingyun Xiao, Ran Liu, and Eva L Dyer. Gaformer: Enhancing timescries transformers through group-aware embeddings. In International Conference on Learning Representations, 2024.
Omry Yadan. Hydra - a framework for elegantly configuring complex applications. Github, 2019. URL https://githur: com haa///fcoooorreeeee///ytttr
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. XLNet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Xucyan Yin, Feifan Li, Yanming Shen, Heng Qi, and Baocai Yin. Nodetrans: A graph transfer learning approach for traflic prediction. arXiv preprint orXic:2207.0333 01301, 2022.
Xue Ying. An overview of overfitting and its solutions. In Journal of physics: Conference series, volume 1168, pp. 022022. IOP Publishing, 2019.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, 2018
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in Neural Information Processing Systems, 30, 2017.
Wojeiech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arx preprint arXiv:1409.2388 2014.
Xiang Zhang, Marko Zeman, Theodoros Tsiligkarddis, and Marinka Zitnik. Graph-Guided Network For Irregularly Sampled Multivariate Time Series. In International Conference on Learning Representations, ICI.R, 2022.
Yu Zheng, Xiuwen Yi, Ming Li, Ruiyuan Li, Zhangqing Shan, Eric Chang, and Tianrui Li. Forecasting fine-grained air quality based on big data. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 2267-2276, 2013.
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in comnectionist networks. In International Conference on Learning Representations, 2021.
Patrick Zippenfenig. Open-meteo.com weather api. https://open-mmtto.ccmm 2023.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Appendix
A Datasets
Table 4: Dataset details.
DATASETS
TIME SERIES
TIME
CHANNELS
CONNECTIVITY
EDGES
SAMPLING RATE
TIME WINDOW
HORIZON
METR-LA
34,272
DIRECTED
1515
12
12
PEMS-BAY
325
52,128
DIRECTED
2369
MINUTES
12
12
CER-E
185
25,728
DIRECTED
4365
30 MINUTES
48
6
AQI
437
8,760
UNDIRECTED
2699
HOUR
24
3
CLM-D
235
10,958
10
NA
2699
1 DAY
14
ENGRAD
487
26, 26,304
NA
2699
HOUR
24
6
PEMS03
358
26,208
DIRECTED
546
MINUTES
12
12
PEMS04
16,992
DIRECTED
340
MINUTES
12
12
PEMSO7
28,224
DIRECTED
860
12
PEMS08
17,856
DIRECTED
277
12
12
METR-LA traffic data from Li et al. (2018). Traffic readings are from different loop detectors on highways in the Los Angeles County. Licensed under Attribution 4.0 International (CC BY 4.0).
PEMS-BAY traffic data from Li et al. 2018 Traffic readings are from different loop detectors on highways in the Los Angeles County. Licensed under Attribution 4.0 International (CC BY 4.0).
CER-E Electric load data from CER 2016). Data encompass energy consumption readings from smart meters in small and medium enterprises, collected in the context of the Commission for Energy Regulation (CER) Smart Metering Project. Data access can be requested through https: ued. ie/ ate/sssda/data/ coomissiioofrrreeeeerrrruaaaaatt
AQI Air quality data. from Zheng al. 2015 The dataset collects measurements of the PM2.5 pollutant. from air quality stations in 43 Chinese cities and is available at
www.oooo.....o.ooooommmmommo.oo. com/ her///n-u/// based on-
CLM-D satellite daily climatic dataset from (De Felice et al. 2024). Data were obtained from the POWER Project's Daily 2.3.5 version on 2023/02/26 and sampled in the correspondence of the 235 world capitals. Further information, together with data and APT, available at the project website (https //power.larc.nee las....cc...agaaaaoc...a..a... gov/). For daily data, select the following 10 variables: mean temperature ("C", lemperature range ("C), mazimum lemperalure (?C), wind speed (m/s), relative humidill (%), precipilation (mm/day), dew/frost point (?C', cloud amount (7), allsky surface shortwave irradiance (W/m²) and all-sky surface longwave irradiance (W/m²). Data extend for 30 years (1991 to 2022).
EngRAD hourly climatic dataset from Marisca et al. 2024 The measurements are provided by https
//oeen Jopen-meteo. meteo. com (Zippenfenig 2023) and licensed under Attribution 4.0 International (CC BY 4.0). Data are collected on grid in correspondence with cities in England. The variables correspond to air lemperature at meters above ground (%): summation of total precipitation
meters above ground (?C); relative humidity (rain, showers, snow) during the preceding hour (mm); total cloud cover (%); global horizontal irradiation (W/m?). Data extend for years (2018 to 2020).
PEMS03, PEMS04, PEMS07, and PEMS08 datasets from Guo et al. 2021 collect traffic detector data from 4 districts in California provided by Caltrans Performance Measurement System (PeMS). Data are aggregated into 5-minutes intervals.
All datascts were split 10%/10%/200 into train, validation and test along the temporal axis. Datascts with high number of missing values, i.c., METR-LA, PEMS-BAY and AQL, have a binary mask concatenated the input, indicating if the corresponding value has been artificially imputed. Note that, unless differently
17

---

Published in Transactions on Machine Learning Research (02/2025)
specified, used datasets are public domain. Where possible, pointed to the original source of the data to meaningful reference.
B
Models
In this section describe the models used in study. Note that all models used fixed hidden size dA for all layers. For all the architoctures, the ENCODER is parametrized by lincar layer, while the DECODER is l-layer MLP followed by H parallel linear layers, each decoding different step in the forecasting horizon. The RNN model is implemented by means a 1-layer GRU (Cho et al. 201 4b) shared among all sequences. On top of the same GRU architecture, for the STGNN model, we employ layers of message passing defined as
m W2 (W1 (h',),,a)))
=n
(8)
h = Wah's +
ta o pefkomm-y m
(9)
JEN(()
where Wo E Rdh W E Rd. 12d W2 E xd and Wa E Rdaxds are learnable parameters, [;]] is the concatenation operator along the feature dimension, @ is the Hadamard product, & denotes the edu (Clevert et al. 2016) activation function and yf the sigmoid activation function. Furthermore, (i) denotes the neighbours of the i-th sequence, induced by the adjacency matrix A, aj denotes the weight associated to odge j - and h' and he denote the hidden features associated with the i-th and j-th sequences respectively. Regarding the STAtt model, we use layers of multi-head attention (Vaswani et al., (017)) taking input tokens the hidden representations extracted by temporal encoder which shares the architecture the aforementioned RNN. Finally, for consistency with the original experiment (Cini et al, 2023b), we employ an STGNN consisting of 1-layer GRU and 2 message passing layer implementing
Σ (Wsh') = Wah + (1)( JEN())
(10)
where W4 Rdxxdd and Ws E Rdaxds
xds are learnable parameters, while || || is the cardinality operator.
C Additional details on forgetting regularization
For the sake of completeness, in the following, we list some additional details regarding the local forgetting regularization.
C.1 rebedding-eelated Encoder/Decodee parameters reset
When resetting the local parameters, we also reset the encoder's (Eq. 3 and decoder's (Eq. 5) parameters (i.e., coefficients of linear layer) that directly interact with the embeddings' features. As an illustrative example, consider ENCODER parametrized by MLP, with input linear layer
h |X.-1 UU. W b,
(11)
In this case, the encoder's parameters to be reset would correspond the last de columns of the weight matrix W. An equivalent behavior implemented for the decoder's input layer.
C.2 Forgetting period sensitivity
The introduced forgetting regularization has two hyperparameters: the reset period ie and the halting epoch. While the latter can be determined automatically, by monitoring the validation error before each reset, the former should be set empirically. To provide insight on the degree to which the selection of k can impact
18

---

Published in Transactions on Machine Learning Research (02/2025)
Learning Rate
0.0001
3.061
3.100
3.100
3.097
3.098
3.094
3.100
3.094
3.079
3.091
3.10
0.0005
3.027
3.049
3.039
3.038
3.033
3.038
3.035
3.033
3.038
3.027
MMA
0.001
3.045
3.052
3.040
3.032
3.037
3.045
3.045
3.042
3.045
3.051
0.005
3.067
3.064
3,064
3.070
3.071
3.079
3.073
3.069
3.072
3.063
-3.05
0.01
3.105
3.113
3.115
3.113
3.111
3.115
3.114
3.120
3.119
3.106
0
3
5
10
15
20
25
30
40
50
Period (epochs)
(a) METR-LA
Learning Rate
0.0001
1.617
1.633
1.632
1.638
1.635
1.636
1.635
1.635
1.632
1.632
0.0005
1.590
1.580
1.578
1.579
1.580
1.579
1.579
1.581
1.578
1.579
1.62
MAE
0.001
1.592
1.583
1.577
1.579
1.584
1.579
1.584
1.584
1.582
1.583
1.60
0.005
1.599
1.582
1.582
1.585
1.591
1.589
1.591
1.591
1.594
1.595
0.01
1.615
1.609
1.609
1.615
1.613
1.618
1.614
1.614
1.617
1.616
1.58
0
3
5
10
15
20
25
30
40
50
Period (epochs)
(b) PEMS-BAY
Learning Rate
0.0001
12.077
11.975
11.981
11.987
12.008
12.021
12.021
12.035
12.038
12.045
0.0005
11.916
11.731
11.738
11.755
11.801
11.796
11.808
11.823
11.829
11.823
MAE
12.0
0.001
11 .963
11.801
11.767
11.774
11.796
11.790
11.818
11.790
11.844
11.863
0.005
12.164
11.874
11.851
11.899
11 883
11 L29
12. 00.001
11.95
11.954
11.976
11.960
0.01
12.236
11.941
12.014
12.083
12.
12.158
12.080
12.145
12.149
12.093
12.176
11.8
0
3
5
10
15
20
25
30
40
50
Period (epochs)
(c) AQI
Figure 5: Sensitivity to forgetting period & across different learning rates and datasets (3 runs, STGNN). A period of 0 indicates regularization.
regularization performance, we perform sensitivity study on 3 datasets: METR-LA, PEMS-BAY and AQL The results are shown in Fig. 5 where we report the mean absolute error (MAF) with varying learning rates and values of k. We can see that performance is relatively stable in regard to changes in k; suggesting that hyperparameter search can be limited to few values (c.g., 3 for a short period and 30 for a long onc). A Another possibility, to avoid searches, is to select the value by hand, after inspecting the learning curves of the unregularized model, in order select reset period that allows performance recovery.
C.3 Forgetting warm-up
To avoid instabilities at the beginning of training, particularly when using short reset periods, the forgetting routine can be initiated after a warm-up period, which can be set empirically to a few epochs, as commonly done with learning rate warm-up procedures or with other regularizations (Cini et al. 2023b)
D Detailed experimental setting
Reproducibility Python code to reproduce the experiments is available onling Datasets will be automat- ically downloaded or access can be requested to the authors, aside from CER-E and EngRAD. Such datasets can be obtained by contacting the original papers' authors (EngRAD (Marisca et al. 2024h) or by requesting
19

---

Published in Transactions on Machine Learning Research (02/2025)
data access to the appropriate authority (CER-E CER 2016)) To specifically reproduce the experiments in Tab 3 regarding the columns for clustering and variational regularizations, access to the code can be requested to the original authors of Cini et al. (2023b). Appendix A contains pointers to request such access. Note that randomized operations in the code are controlled by fixed seeds for reproducibility purposes.
Shared settings All the models were trained with the Adam optimizer (Kingma & Ba
2015
with a batch
size of 64 and up to 300 batches per epoch. We used the Python Van Rossum & Drake 2009 programming language, leveraging Torch Spaliotemporal (Cini & Marisca 2022), Pytorch (Paszke et al. 2019) and Pytorch
Lightning (Falcon & The PyTorch Lightning team| 2019) implement all the experiments. Experiments were scheduled and logged by leveraging Hydra (Yadan) 2019) and Weights and Biases (Biewald 2020). The mean absolute error (MAE) was used as loss function in all experiments. For variational and clustering regularizations additional hyperparameters, aside from the regularization strength, i.c., Avar, Actst, were taken from the original paper. Unless specified, we employed and embedding size de = 32 for all models. Moreover, all datasets used temporal encodings as additional covariates, in particular, a one-hot encoding of the weekday and sinus and cosinus encodings with daily period.
Experiment specific hyperparameters For the experiments in Tab. [| optimal hyperparameters, i.c., learning rate Ir and hidden size d,, for cach model and dataset, were found via a grid-search over Ir E [0.00025, 0.00075, 0.0015, 0.003] and dn E [32, 64, 128, 256), with constraints caused by GPU memory capacity in some settings. Before the grid search for regularized models, regularization's hyperparameters Au, Am2 Avar and Adst, were set according to the following procedure. For each model and dataset (we here consider three datasets: METR-LA, PEMS-BAY and AQI), the validation error is computed for regularization strength values in [0.01,0.001, 0.001, 0.0001, 0.00001], while keeping the Ir and da fixed to the optimal values for the respective un-regularized model. For each regularization hyperparamcter, the corresponding validation results are ranked and the average rank is computed across models and datasets. The parameter with the top rank is selected and kept fixed. This resulted in: weight of L2 Ar2 = 0.0001, weight of L1 'v1 = 0.00001, weight of variational regularization Xear = 0.00005 and weight of clustering regularization Adst 0.0005. Dropout's probability of dropping connection was set to p 0.5, which is middle ground between soft and aggressive dropout. Moreover, we set the resetting period of sorgelling to A = 20 epochs with 30 epochs of warm-up. The analysis in Sec. C.2 shows how the method is not particularly sensitive in this regard. Forgetting was halted after 150 epochs. Each model was trained for 200 epochs and the best validation parameters were used for lesting.
For the experiments in Tab. 3 we used the STGNN in Eq. 10 and set dn 64, Aver = 0.05 and Actat = 0.5 as in Cini et al. (2023b), while for the other regularizations we used the same hyperparameters as in Tab. [] Furthermore, we used a learning rate Ir = 0.005 during training and Ir 0.001 during fine-tuning. The training lasted up to 150 epochs with 50 epochs of early stopping patience, while fine-tuning lasted up to 1000 epochs with 100 epochs of patience. No temporal encodings were used in this setting.
For experiments in Sec. 5.2 and Sec. 5.4 we used set d, = 64 and i = 0.00075. Regularization's hyperparam- eters were set to the values found for Tab. [ Training lasted up to 300 epochs, early stopping patience was set to 100, while forgetting was halted after 100 epochs. Note that the foryetting regularization ignores early stopping for the whole duration before being halted; this is to avoid triggering early stopping as a byproduct of the parameter resetting itself.
Metrics For univariate datasets, consider the mean absolute error (MAE), this is standard choice across the related literature. For multivariate datasets, i.c., CLM-D and EngRAD, following De Felice et al.] (2024). we compute the mean relative error (MRE) independently for each channel, then, take the whan average to obtain a unique final score, which we termed multivariate mean relative error (MMRE) in Tab.
Computing resources Experiments were run on A100 and A5000 NVIDIA GPUs. The vast majority of the experiments conducted in our work can be easily run on moderate GPU hardware, with at least 8 GBs of VRAM. However, not that some experiments, may require more. For instance, the transfer experiments require at least 20 GBs of VRAM on some of the benchmark dataset. Moreover, high hidden size configurations of STGNN and STAtt models, required up to 40 GBs of VRAM. In general, a single run (i.e., training one
20

---

Published in Transactions on Machine Learning Research (02/2025)
model dataset), given hardware that fulfills the memory requirements, takes from 30 minutes to 3 hours, depending on the specific configuration.
D.1 Embeddings perturbation
Here provide more formal definition of the perturbations adopted in Sec. 5.4 In particular, considering N embedding vectors:
Noise a consists in adding zero-mean gaussian noise, from an isotropic multivariate normal distribution, to the embeddings. Formally, this refers to substituting each embedding e with e' + where e ~ N (0, diag (o²)).
Rearrange refers to randomly reassigning the embedding vectors to different time-serics in the collection. Formally, this means substituting each embedding e' with an embedding e', where j ~ Multinomial ((O,..,,N N - 1}) is a sample from a multinomial distribution over the embedding indices, sampled without repetition.
Mean refers to setting each embedding to the mean values across embeddings themselves. Formally, replacing each embedding ct with N Ey-oc'. c².
Sampled consists in estimating the sample gaussian distribution of the embeddings, and then replacing each embedding with a draw from such distribution. Formally, this entails estimating the gaussian's parameters as He N -N-- e' and ae = N-T Em' (el -n)). Then we sample values for each embedding as e¹ ~ N NNe, (Me:
Note that, to avoid penalizing regularizations that are sensible to the weights' magnitude (i.c., L1, L2), consider perturbations that do not change the scale of the learned representations. As this does not hold true for the Noise perturbation, since it can potentially affect the embeddings magnitude, depending on the standard deviation, we selected most values to be within reason relative to the embeddings magnitude itself. Nonethcless, we also employed a few more extreme values, in order to observe their effects.
E Effect on simple global models
The purpose of model regularization is, usually, to limit model capacity by means of additional constraints and, in turn, obtain models that generalize better. This implies that excessive regularization might result a degradation of performance. In principle, regularization of local parameters should not have a negative impact underparametrized global models, as their parameters not constrained directly. To evaluate this, we employ simple 2-layer ML. our global model (Fq. 4 and train it with the same settings as in Sec. 5.
Tab. reports the obtained results in case tuned learning rate and hidden size of 64 and 128 units. Compared to models in Tab. [ the performance of the ML.P worse or comparable best, however, the regularizations relative offectiveness similar, considering each datasct, at hidden size 128. In the extreme case of the MLP with hidden size 64, we can observe additional scenarios in which most regularizations are ineffective (i.e., CER-E). Nonetheless, we do not observe catastrophic effects performance. Notably, with some datasots, i.c., AQT and EngRAD, adding local parameters to the global MLP model hurts performance, while regularization allows an effective exploitation of the embeddings.
F Effect of combining different regularizations
To further investigate whether different regularization techniques can be combined for improved performance, we applied L2 and dropout on top of the approaches specifically designed for the regularization of local embeddings, i.c., clustering, variational and forgetting regularizations. This covers the most reasonable combinations, as L2 and dropout are commonly used in combination with other regularization approaches. In particular, consider the same transductive setting as in Sec. 5.1 and adopt the optimal hyperparameter
21

---

Published in Transactions on Machine Learning Research (02/2025)
Table Forecasting error in transductive setting for MLP model (hidden size 64-128) (5 runs, #lstd), Equal to to/better than unregularized (+ EMB) in bold. Best in red. '+ REGULARIZATION denotes the addition of
that specific regularization only, on top of ++ EMB.
DATASET
METRRLLA
PEMS-BAY
CER-E
AQI
CLM-D ENGRAD MMRE + MMRE +
MODEL
MAE +
MAE +
MAE +
MAE +
MLP (64)
3.580
1.808
0.4658
13.398 -0.044
19.8640.01
31.3840..1
+ EMB.
3.15310.012
1.619
0.4233
19.5540.01 13.521
31.40
+ L1
3.150+0.000 40.005
1. 1.61440.006
0.4236
23.392-0.006
19.5640.01
31.3211004
+ L2
3.147 0000
1.613
0.4212+0.0010
13.389 =0.022
19.57+0.02
31.21+0.09
+ CLUST.
3.152+0.014
1.625-00.099
0.4236+0.0012
13.478=0.070
19.5440.01
31.25+00.4
+ DROP.
3.193
1.630
0.4317
13.328
19.6210.01
31.2040.07
+ VARI.
3.139+0.01
1.610+0.001
0.4216+0.0011
23.337=0.00
19.54+0.01
31.32+0.03
+ FORG.
3..533
1.617+0.000
00.0022
13.472
19.56+0.02
31 31.3440.09
MLP (128)
30.012
1.798
0.4057
13.390
00.044
19.7840.01
31.44
+ EMB.
3..5740.014
1.615+0.006
0.4141+0.0013
13.534 0.002
19.47+0.01
31.38+0.10
+ L1
3.002
+..00
0.4128
00.0007
13.417 00078
19.49+0.02
31.38+0.07
+ L,2
3.150
1.600
0.4117
10.0009
13.450
19.48
31.65.0.17
+ CLUST.
3.143
1.603
0.4134
13.612:0.119
19.47
31.4210.06
+ DROP.
3.195+0.008
1.624=0.003
0.4101 +0.0003
13.320=0.044
19.53+0.02
31.35+0.13
+ VARI.
3.139
1..03
0.4109 00.0112
13.346 00.038
19.4740.01
31.33
+ FORG.
3.15010.014
1.616=100110
0.418410.0008
13.427 00058
19.53
31.3610.09
Table 6: Forecasting test error under optimal model size and learning rate from Tab. (STGNN, 5 runs, #lstd) when adding additional regularizations. Combinations equal to or better than the corresponding base regularization are in bold. The best- perffrerrorming method within each dataset and base regularization is in rod. Regularization methods are always considered applied on lop of the '+ EMB.' model.
DATASET
METR-LA
PEMS-BAY
AQI MAE +
MODEL
MAE
MAE 4
+ CLUST.
3.025
1.580
11.876+.0555
+ CLUST. + L2
3.019
1.581 10.008
11.767 +0.023
+ CLUST. + DROP.
3.052
1.572
11.71240.037
+ VARI.
3.0131.000
1.566
11.768
+ VARI. + L2
3.037 +0.099
1.581+0.007
11.789+0.022
+ VARI. + DROP.
3.047+0.008
1.576+0.004
11.67540.023
+ FORG.
3.050
1.578+.006
11.793
+ FORG. + L2
3.045:0.012
1.576=0.007
11.764+0.00
+ FORG. + DROP.
3.087+0.005
1.570+0.003
11.75540.028
configuration (see Appendix D for details) previously found for clustering, variational, and forgetting, respectively.
Tab. 0 shows the obtained results. We can see that combining different regularization techniques can be beneficial. Notiocably, combining variational regularization and dropout yielded a new overall best score on the AQT dataset. However, improvements appear mostly case-dependent, without emerging clear patterns.
22

---

Published in Transactions on Machine Learning Research (02/2025)
RNN (32) [EER-LLA]
STGNN (32) METR-LA LA]
STAtt. (32) [METR-LA]
2.80
3.00
3.0
2.95
2.75
2.9
2.90
MAE
2.85
2.70
2.8
2.80
MAw
2.7
2.65
2.75
100
100
150
200
Epochs
Epochs
Epochs
No embeddings
No regularization
Dropout (whole model)
Dropout (embeddings)
Figure Validation curves for different model families runs, #1std). Curves show the application of dropout regularization at the embedding level over the entire architecture. Plot names follow the convention model (embedding size) [dataset].
G Additional experiments
For completeness, in this section, we provide experimental results in addition to what has been shown in the main paper.
Fig. shows an example of how regularizing local parameters only can have a different impact than regularizing all parameters on the training of an STGNN model. Fig. 0 complements what has been shown such an example by providing results for the other two models, i.e., RNN and STAtt. A similar pattern emerges for all the considered models, though with different degrees.
Similarly, Fig. 7 complements the results shown in Fig. 3 with the models that were not shown. We can notice a similar pattern in which dropout (red) can be problematic at smaller embedding sizes (lop row). Nonetheless, this seems to be less significant for the STGNN in this specific scenario. In general, the additional plots confirm that dropout (red) and forgetting (blue) lead to different learning curves compared to the un-regularized model (orange), while other regularizations have little impact on this aspect.
the PEMS-BAY dataset. We can see that, in this case, the validation curves barely plateau. In this scenario, dropout shows potentially problematic effects, similar to those observed for small embedding sizes.
Fig. X shows results obtained by training models with the same setting Sec. 5.2
Finally, Fig. and Fig. 10 provide complementary results for Fig. We can notice how the different. regularizations rank similarly, in terms of global model robustness, across different architectures.
23

---

Published in Transactions on Machine Learning Research (02/2025)
RNN (8) [METR-LA]
STGNN (8) [METR-LA]
STAtt. (8) [METR-LA]
3.00
3.0
3.0
2.95
2.90
MAE
2.85
2.8
2.8
2.80
2.7
2.7
2.75
50
100 Epochs
150
200
50
100
150
200
100 Epochs
200
Epochs
No embeddings
L1
Clustering
Variational
regularization
L2
Dropout
Forgetting
RNN (32) [METR-LA]
STGNN (32) METR-LA]
ST (32) [ETR-LA]
2.85
3.0
2.95
2.80
2.90
2.9
2.75
MAP
2.85
2.70
2.7
2.65
2.75
0
50
100
150
50
100 Epochs
150
100 Epochs
200
Epochs
No embeddings
L1
Clustering
Variational
No regularization
L2
Dropout
Forgetting
RNN (128) METR-LLA
STGNN (128) METR-LA
STAtt. (128)
METR--LA NETR-LAA]
3.00
3.0
2.95
2.9
2.9
2.90
MAR
2.8
2.8
2.85
2.7
2.7
2.75
2.6
2.6
2.70
50
100
150
100
Epochs
Epochs
Epochs
No embeddings
L1
Clustering Dropout
Variational Forgetting
No regularization
L2
Figure 7: Validation curves for different model families (5 runs, #1sld). Plot names follow the convention model (embedding size) [dataset].
24

---

Published in Transactions on Machine Learning Research (02/2025)
RNN (32) [PEMS-BAY]
STGNN (32) [PEMS-BAY]
STAtt. (32) [PEMS-BAL
1.9
1.9
1.9
1.8
1.8
MAE
1.7
1.7
1.7
1.6
1.6
1.6
100
200
0
100
200
100
200
Epochs
Epochs
Epochs
No embeddings
L1
Clustering Dropout
Variational
No regularization
L2
Forgetting
Figure 8: Validation curves for different model familics (5 runs, #1std). Plot names follow the convention model (embedding size) [dataselj.
RNN
STGNN
STAtt.
25
25
25
24
24
24
MAE
23
23
23
22
22
22
2
21
21
0
21
22
21
22
21
22
Noise 0
Noise O
Noise Forgetting
No regularization
L2
Dropout
L1
Clustering
Variational
Figure 9: Test performance degradation when adding pero-mean Gaussian noise to the embeddings, with increasing variance (5 runs, +lsld, METR-LA). Fixed hidden size and learning rate, 64 and 0.00075 respectively.
25

---

Published in Transactions on Machine Learning Research (02/2025)
RNN
STGNN
STAUL.
20.0
17.5
15.0
MAE
12.5
5
10.0
7.5
5.0
Rearranged Mean Sampled
Rearranged
Mean
Sampled
Rearranged
Mean
Sampled
Perturbation
Perturbation
Perturbation
No regularization
L2
Dropout
Forgetting
Clustering
Variational
Figure 10: Test performance degradation on embeddings perturbation (5 runs, 1lstd, METR-LA) random shuffling, (Middle) replaced with their mean, and (Right) replaced by a draw from their sample normal. Fixed hidden size and learning rate, 64 and 0.00075 respectively.
26


Reviews:
[{"review_id": "wRDYD44zjR", "reviewer": "Reviewer_fGiC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n+ The paper is well written and easy to follow\n+ Having a clear empirical evidence of what kinds of regularization are important in which scenarios is a very useful thing for the community\n+ The results clearly show, that the regularizations that prevent \"co-adaption\" are indeed more helpful\n\n## Weaknesses\n- The term co-adaption is only explained very shallowly and the grouping of the regularization methods by the binary question wether they are preventing this is also not very formal\n- What is not observed is, to which extent for adding different regularization methods the enhancements stack up, see my first request for changes for more details\n- A clear recommendation how to use the findings is missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors discuss how to regularize time-series specific embeddings in hybrid models which process multiple time-series at once. The authors study to which extent L1,L2-regularization, dropout, clustering-regularization, variational-regularization and random forgetting of the local embeddings help to improve performance in time-series analysis. Their main finding is, that in hybrid models, especially regularization techniques that prevent co-adaption are useful.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- How do I read for example Table 1? Does \"+ Clust\" means, that I have also all regularization  above and now add \"Clust\", meaning I have Embeddings + L1+ L2+Clust? Or do I just have Embeddings and Clust? If it is the first, then results for having only this specific regularization are missing. If it is the latter, the results for the question \"Which regularization stack up regarding performance enhancement?\" are missing\n\n- To make this work important for the community, It would be super helpful to have a section \"recommendations\" where it is clearly stated what kind of regularization I now should use when I am doing time-series forecasting with an own dataset\n\n- I was not able to fully understand the co-adaption phenomenon. I get the rough idea, but it would be crucial for me to formalize it: What is a useful embedding and what is a meaningless identifier? And how can I exactly connect specific regularization methods to it?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GxDkcKLSgU", "reviewer": "Reviewer_hmzB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The manuscript is well-written and easy to follow. The background preliminaries and problem settings are clearly defined, making it easier for readers to understand the objectives of this work.\n- Regularization techniques for time-series forecasting are systematically studied, and the experimental analysis is thorough, including diverse model architectures and datasets.\n\n# Weaknesses\nThe contributions of this work are limited, as the authors do not develop any novel regularization techniques based on their findings. Additionally, in my opinion, Findings (1) and (2) are already common knowledge and widely accepted within the deep learning community, i.e., regularization and the disentangling of global and local parameters can improve the generalization performance and robustness of deep learning systems. Furthermore, Finding (3) should not be considered a separate or novel contribution, as it essentially duplicates Finding (2).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study empirically evaluates the significance of regularizing local embeddings across various regularization techniques, model architectures, and time-series forecasting datasets. In particular, regularization methods that mitigate co-adaptation between global and local parameter learning result in greater performance improvements.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "All methods and experiments are conducted within the context of time-series forecasting. Therefore, it should be emphasized more clearly in the title and Introduction section that this study focuses on forecasting, rather than on \"Time Series Processing\", as stated in the title.\n\nIn general, when considering the \"Evaluation Criteria\" of TMLR, the claims made in the submission are supported by accurate, convincing, and clear evidence. However, I have concerns regarding the limited contribution of this work, as the findings may already be widely recognized within the deep learning community. This could mean that the audience of TMLR may not find the findings of this paper particularly interesting.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0TnfjrE5fn", "reviewer": "Reviewer_LgE4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe paper shows, that regularizing the local components yields better performance in both settings.\nIn particular, the paper shows that Dropout is most effective in the first setting, if less information is available.\n\n## Weaknesses\nThe writing could use some work.\n\nEspecially in the second setting, the claimed \"clear performance improvements\" seem to be very small for most models. The conclusion of the authors seem to imply much stronger results.\n\nEven after looking at the appendix, it is not entirely clear, which parts of which models were tuned and which parts were taken from related work, making it difficult to fully contextualize the presented results.\n\nI am not certain about the novelty of the proposed \"forgetting regularization\". After a quick look into the cited papers, i fail to see significant differences to prior work and the paper does not provide a satisfying comparison.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Given a collection of time series, this paper considers forecasting in two settings: \n(1) Forecasting in an unknown time series with little to no information on the new time series during training time. \n(2) Forecasting in different realizations of the known time series.\nWhen processing sets of time series, one can learn a single (global) model processing all time series equally, an individual model for each time series, or a combination of both. This paper considers a general architecture combining a local encoder and a local decoder with a global model in between. The main contribution of the paper is an experimental comparison of multiple regularization techniques applied to the local parts of the combined model in both settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The language in the summary of the findings seems too strong. \"clear improvements\", \"larger performance gains\", and \"significant performance improvements\" seem to imply much stronger results than the experiments seem to show. A more conservative wording might be more appropriate. \n\nThe paper spends a lot of time on the definition of the models in varying generality. That space might be better used to expand the discussion of the results, the conclusion, and the discussion of the regularization techniques, in particular the proposed method.\n\nIt is not clear, how the \"simple setting\" in Figure 2 relates to the benchmarks. Is there a strong reason, why global regularization should not also be considered in all evaluations as a baseline?\n\nThe paper would greatly benefit from some general improvements on the writing.\nHere are some issues i noticed:\n- P2§2 \"[...] that operate on fixed size dictionary [...]\" -> \"[...] that operate on a fixed size dictionary [...]\"\n- \"w.r.t.\" is usually more appropriate for theoretical papers.\n- Equation (3) seems to be missing W\n- Section 3 §1 \"[...] and, often, as spatial positional encodings [...]\" -> \"[...] and, often as spatial positional encodings [...]\"\n- Section 4.1 §2 \"L1 and L2 regulariation\" -> \"L1 and L2 regularization\n- \"embedding\" is usually used for representations of data.\n- $\\sigma$ is used in multiple different contexts.\n- Caption of Table 3 \"[...]  methods equal or better [...]\" -> \"[...]  methods equal to or better [...]\"\n- Caption of Table 5 \"trasductive\" -> \"transductive\"\n- Using $\\\\Vert$ for concatenation seems a bit confusing at first, especially when used in close proximity to the norm, as e.g. in (8) and (10).\n- There seems to be a problem with the definition of $W_0, \\\\ldots, W_3$ or equation (8-9). $W_1$ and $W_3$ don't seem to come from the same space.\n- Sec 2.2 §1 \"With full generality [...]\" seems to be contradicting the following statement, where only a special case is considered.\n- Sec 2.2 §3 The indices of the parameters should probably also start at 1.\n- $<t$ is usually used to refer to values less than $t$. Using $\\leq t$ might make the equations more intuitive.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers expressed the opinion that the contributions of the work are limited, and two of the reviewers provided a recommendation of “leaning to reject”. 

Having said this, two of the reviewers did acknowledge that the submission did meet the acceptance criteria of TMLR, namely that the claims of the submission are supported by clear evidence, and the results will be of interest to some individuals in TMLR’s audience. 

The third reviewer is of the opinion that claims are not sufficiently supported because the paper lacks a comprehensive comparison to global regularization methods and the number of evaluated models is too small. Unfortunately, this reviewer did not express in the original review that the number of evaluated models is too small. While the results would be more convincing with more models, the experiments do include state-of-the-art methods. 

Concerning the comparison to global regularization methods, the authors provided a response to this, explaining that the benefits of global regularization are well known and for this reason the paper focuses on the less-studied issue of regularization of the local parameters. The reviewer did not respond to or acknowledge the authors’ response to the criticism. The official recommendation does not provide a clear argument about why this response is unsatisfactory. 

In my view, the paper’s experiments do support the claims that are put forward in the paper, provided they are read in the context of “for the models that we experiment with”, which is implied in the introduction.

Concerning audience interest, the reviewers expressed concerns that the contribution is limited, and that the results may be either common knowledge or would emerge through standard hyperparameter tuning. In general, however, there is a consensus that the results for such an investigation have not been published for local parameters of time-series models, and that some researchers working on time-series analysis and forecasting are likely to be interested in the findings.
Decision: Accept (poster)