DeepRRTime: Robust Time-series Forecasting with a Regularized INR Basis
Authors: Chandramouli Shama Sastry
Mahdi Gilany
Kry Yik-Chau Lui
Martin Magill
Alexander Pashevich
Venue: Accepted by TMLR ()

Abstract: This work presents a simple, inexpensive, theoretically motivated regularization term to enhance the robustness of deep time-index models for time-series forecasting. Recently, DeepTime demonstrated that this class of models can rival state-of-the-art deep historical-value models on the long time-series forecasting (LTSF) benchmarks. The DeepTime framework comprises two key components: (1) a time-indexed basis parameterized as an implicit neural representation (INR), and (2) a meta-learning formulation that fits observed data to this basis via ridge regression, then extrapolates the result to generate forecasts. Our regularization term encourages the time-indexed basis elements to be more unit standardized and less mutually correlated, intended to enable more robust ridge regression. The regularized variant matches or outperforms DeepTime on all LTSF benchmarks. Moreover, it is significantly more resilient to missing values in the lookback window at test time, enhances forecast accuracy when applied to higher-frequency data than it was trained on, and boosts performance when trained on smaller datasets. Overall, we conclude that our regularized approach sets a new state-of-the-art for deep time-index models.

Published in Transactions on Machine Learning Research (01/2025)
DeepRRTime: Robust Time-series Forecasting with a Regularized INR Basis
Chandramouli Sastry!
chendrammoui.ssstt@@@aaa.oo
Dalhousie University
Vector Institute
RBC Borealis
Mahdi Gilany?
maldii. Baalldigilaneyyuueeesevve
Queen's University
RBC Borealis
Kry Yik-Chau Lui
jikchhou...kf@bboreelii..coo
RBC Borealis
Martin Magill
marttn.magglleoorruiiss.cc
RBC Borealis
Alexander Pashevich
deeanderpocheiicceebrrrlaaa.....
RBC Borealis
Reviewed on OpenReview: https: // openreview. net/ formm? id= #DR#ORP177
Abstract
This work presents a simple, inexpensive, theoretically motivated regularization term to enhance the robustness deep time-index models for time-series forccasting. Recently, Deep- Time demonstrated that this class of models can rival state-of-the-arl deep historical-value models on the long time-series forecasting (LTSF) benchmarks. The Deep'Time framework compriscs two key components: (1) a time-indexed basis parameterized representation (INR), and (2) meta-learning formulation that fits observed data to this basis via ridge regression, then extrapolates the result to generate forecasts. Our regularization term encourages the time-indexed basis elements to be more unit standardized and less mutually correlated, intended to enable more robust ridge regression. The regularized variant significantly
implicit neural
matches or outperforms DeepTime on all LTSF benchmarks. Moreover, more resilient. to missing values in the lookback window at test time, enhances forecast accu- racy when applied to nigher-frequency data than was trained on, and boosts performance when trained on smaller datasets. Overall, conclude that our regularized approach sets new state-of-the-art for deep time-index models.
1 Introduction
Time-series forecasting is a rich topic with applications spanning science (e.g., in weather or climate science, astronomy, biology (Ghaderpour et al., 2021; Rackauckas al., 2020)), engincering (c.g., control enginoering, communications engineering (Krishna al., 2023; Hua al., 2017)), and business (e.g., in econometrics, mathematical finance, demand forecasting, capacity planning and management (Carbonneau et al., 2008; Laptev et al., 2017)). The rise of deep learning models and the increasing availability of large time-series datasets are stimulating a shift towards highly expressive time-series models calibrated using big data and big compute, rather than the classical approach of simpler models calibrated using strong assumptions and human expertise. However, leveraging large and expressive models based on deep learning for time-series forecasting has proven challenging. For example, Zeng et (2022) showed that many state-of-the-at
Work was while the author was intern RBC Borealis

---

Published in Transactions on Machine Learning Research (01/2025)
Transformer-based time-series models could be outperformed by "embarrassingly simple" linear models on common long time-series forecasting (LTSF) benchmarks.
Recently, the Deee model (Woo et al., 2023) demonstrated the promise of deep time-inder models, as opposed to the more popular historical-value models. Historical-value models (e.g., ARIMA (Box et al., 2015) and popular Transformer-basee models (Vaswani et al., 2017)) typically produce forecasts at pre-defined msment(s) in the future based on a pre-defined sequence of recent values, e.g., h++ O(ye, yt-1,.... Vt-L), L), where is the current time, ar is the model output for time t At, Y/ denotes ground truth observations at time t, and ó is the function learned by the model. In contrast, time-index models are continuous functions of time, e.g., 1t+ar= = o(t + At). In this case, some parameters of the model are typically refit at inference so that N+tAr & At+ for / + A values in the past, and predictions are made by extrapolating Φ(1 + At). Classical examples of such approaches include Prophet (Taylor & Letham, 2018) and Gaussian processes (Rasmussen, 2004). Time-index models have some appealing properties. For instance, the continuous dependence of Φ on time is arguably a useful inductive bias for reprosenting typical time-series data (Woo et al., 2023). Morcover, historical-vale value models often struggle with inconsistent input sequences or larget horizons, while time-index models are well-suited for handling irregular sampling rates, missing values, and forecasts over continuous horizon.
The core idea of Teee is to parameterize a time-indexed basis of features ((A)) by an implicit neural representation (INR) network. However, simply fitting a generic neural network to past values of at inference time slow and leads to poor extrapolation into the future. DeepTime overcomes this issue by framing forecasting as meta-learning problem: globally, learn a basis of time-indexed features f(At) such that, locally, forecasting can be done by simple ridge regression. The INR training objective is the forecasting error, with the goal of finding basis that can simultaneossly (a) fit lookback values in reliable manner and (b) extrapolate that fit to the forecast period accurately. This formulation achieves results competitive with other deep forecasting models on common LTSF benchmarks (Woo et al., 2023).
In this work, we improve upon DeepTTime via a regularization term designed to better condition the ridge regression stage as shown in Figure 1. Fssentially, we refine the meta-learning objective to be: globally, learn a basis of normalized and uncorvelated time-indexed features such that, locally, forecasting can be done robustly by simple ridge regression. As widely understood by practitioners, and provably true under cerlain assumptions (Krikheli & Leshem, 2021), linear regression performs better when the explanatory variables are more uniformly normalized and less mutually correlated. Our regularizer is simple, efficient, and encourages these properties in the learned INR basis. Our model code is available at https://githun cww.com//oaaoe.cce hamm/BorealssAA//m BorealisAt/DeepaRrime We call our robust, regularized DeepTime method DeepRRTime and summarize our contributions as follows:
1. We introduce a novel regularization technique for improving the forecasting accuracy of Deep- Time models. Our rogularization tochnique is thcoretically motivated, casy to implement, and computationally inexpensive.
2. We demonstrate empirically that the proposed regularizer facilitates the learning of more unit- normalized and less mutually correlated basis representations, improving performance on several benchmarks at little cost and setting new state-of-the-art for time-index models.
3. We show that the proposed regularizer helps to improve forecast accuracy in more challenging settings: (a) forecasting with missing values, (b) training on smaller datasets, (c) forecasting at a higher-frequennc at test-time compared to the training data.
2 Related work
2.1 Models for time-series forecasting
Classically, timo-scries forecasting has typically been conducted using simple models and strong assumptions about the underlying data generation process, relying on human expertise. Famous examples of classical historical-value models include exponential smoothing models like ETS (Holt, 2004; Hyndman, 2018) and the ARIMA family of models (Box et al., 2015). Classical time-series models in the class of timo-index models (as defined by Woo et al. (2023)) include the Prophet algorithm (Taylor & Letham, 2018) and Gaussian processes (Rasmussen, 2003). DeepTime (Woo et al., 2023) and TimeFlow (Naour et al., 2024) are recent.
2

---

Published in Transactions on Machine Learning Research (01/2025)
Forecast (k.x3)
Forecast 6. Ag Coy
Forecast
DeepTime
DeepRRTine (our method]
Forecest
Ridge Regression
Ridge Regress on
busis
busis
busis
busis
basis
basis
basis
basis
basis
basis
bases similarity
bases similarity
Time-Indek INR
Time-Inded INR
Index
Time Index
(b) Similarity matrices for Timp (top) and DeepRRTime (bottom).
(a) Comparison of our approach with Deep'Iime.
Figure 1: (a): Our proposed approach is similar to DeepTTmme, which takes time indices input, maps them into basis, and applies ridge regression to predict future time series. However, DeepRRTTime includes an additional regularization term added to the forecast loss, promoting more standardized and less correlated Lime-indexed basis elements. This regularization leads to more distinct basis elements, resulting in better conditioning for ridge regression and more robust forecasting. (b): Cosine similarity matrices for 100 of the 256 basis clements (truncated for visibility), demonstrating that the basis learned by DeepRRTime is much closer to orthonormal. The top matrix shows the similarity between the bases learned by DeepTTime, while the bottom matrix illustrates the similarity between the bases learned by DeepRRTime the same dataset. (ETTm2, forecast horizon 96).
models that utilize time-index models for time-series forecasting, and Deep'Time in particular was the first demonstration deep Lime-index model with performance competitive to historical-value models.
The time-series forecasting research community has shown significant interest in historical-vahue models based deep lcarning, with the hope that their increased exprossivity will capture more complex relationships in data. Oreshkin et al. (2019) proposed N-BEATS, deep learning model based on trend and seasonality decomposition, while Challu et al. (2023) introduced N-HiTS, which employs hierarchical interpolation and multi-rate sampling. LogTrans (Li ct al., 2019), FEDformer (Zhou et al., 2022), and ETSFormer (Woo et al., 2022) models attempt to leverage attention-basee based archilectures (Vaswani et al., 2017) to model dependencies across time steps. Adaptive RNNs (Du et al., 2021), RevIN (Kim al., 2021), and Non-stationary Transformers (Liu et al., 2022) introduce various methods aimed handling distribution shifts.
Zeng et al. (2022) recently published simple linear models that outperform all the Transformer baased models listed above on all the standard LTSF benchmarks, and argued against applying attention layers to individual time steps. PatchTST (Nie et al., 2023) effectively addressed this concern, segmenting time-series data into patches to address the potential lack of semantic meaning in individual time steps. PatchTST achieved state-of-the-art performance on almost all the LTSF benchmark problems, matching or outperforming the lincar basclines of (Zeng et al., 2022). However, one notable exception is the Exchange datasot, which was omitted from the TST experiments. The authors highlighted it as singularly challenging among all the benchmarks, and noted that even simple martingale model is formidable baseline to overcome for this dataset.
2.2 Meta-learning
Meta-learning, or the "learning to learn" paradigm, entails training across multiple tasks to learn to generalize to new, unseen tasks. The widely recognized MAML (Finn et al., 2017) algorithm consists of an outer-step optimization spanning multiple tasks and an inner-step optimization on each task. The goal of the inner stop is to perform well on query set by fitting a neural network to a corresponding support set. The outer step aims to learn neural work initialization such that, for all tasks, a small amount of fine-tuning on the support.
3

---

Published in Transactions on Machine Learning Research (01/2025)
set is sufficient to perform well on the query set. Reptile (Nichol et al., 2018) and ANIL (Raghu et al., 2020) demonstrated that MAMT's 's expensive inner-loop fine-tuning can often be replaced by cheaper approximate or partial fine-tuning without hindering performance. Bertinetto et al. (2019a) eschewed the expensive iterative inner step entircly by employing a closed-form ridge regression solver for the classification head in a meta-learning model, significantly enhancing their efficiency. In DeepTime's meta-learning framework, each forecast is considered distinct task. The support sets correspond to the rocently observed data in the lookback window, and the query sets are the future values in the horizon window. In this framework, learning the INR. basis is the outer loop, while ridge regression is the inner loop. As in Bertinetto et al. (2019a), Deee employs closed-form solution to the inner-loop optimization, improving efficiency.
While we are not aware of previous work applying covariance regularization to meta-learning frameworks, as proposed in this paper, we note that covariance regularization has been considered for supervised and self-supervised tasks in computer vision (Cogswell et al., 2016; Zbontar et al., 2021; Bardes et al., 2022).
2.3 Implicit Neural Representation
Implicit neural representations (INR) are deep neural networks trained to approximate varions types of signal including views of scene (Mildenhall et al., 2021); images (Chen et al., 2021; Dupont et al., 2021), and 3D shapes (Chen & Zhang, 2019). For example, an INR trained to reconstruct an image might map a two-dimensional vector representing coordinates of an image pixel to three-dimensional vector representing RGB colors at this pixel. Many INR architectures are relatively simple, consisting of simple variations of multilayer perceptrons (MLPs). SIRENs, proposed by (Sitzmann et al., 2020b), are MLPs with sinusoidal activation functions, chosen to facilitate the representation of high-frequency features. Deep uses a variant of the INR design proposed in (Tancik et al., 2020). Similar to SIRENs, the architecture of (Tancik el al., 2020) is also designed lo capture figh-frreqency features, but uses a single Fourier embedding layer at the beginning of the network rather than periodic activations after each layer. Besides forecasting, INRs have been explored for other time-series applications: Jeong & Shin (2022) utilize INR representation errors as a metric for anomaly detection while Fons et al. (2022) and Szatkowski et al. (2023) explore applications to time-series generative and missing value imputation respectively. Meta-learning on INRs, as done in DeepTime, has been interpreted as learning a prior over the space of signals (Sitzmann et al., 2020a; Tancik et al., 2021; Dupont et al., 2022) and shown to have analogies to dictionary learning (Yüce et al., 2022).
3 Background
In this section, we review DeepTime and then present our method DeepRRTimee, We first describe inference, which is the same for both approaches, and then contrast their training procedures.
3.1 Notation
Consider an INR network fo T + 2, where T E [0, 1] is called the time-index and 2, E DR represents the values of the D-dimensional basis at time т. The time index T is divided between the lookback and forecast regions as follows: given a lookback length and a forecast horizon H, the range S T -+H-I indexes the lookback window while I+#-I S SS1 1 indexes the forecast period. Specifically,
1-1
I+# is the moment at which a forecast is to be made, in keeping with the convention of Woo et al. (2023).
L
We denote observed values of the target variable y in the lookback window by Y = (Ur, Vra..... Vr with T. < Tj when < j, Ti 2 0, and To S I+ For regularly sampled data, we specialize this to Ti = 0, I I##-I. We will refer to the model's forecast by O(T) for I## ST S 1. We will generally consider a discrete set of forecasts Y, = (0 Dre.g... .. For regularly sampled data, we specialize this to Te+1 L+H-I Tet2
T2- I and so on to
L+1 -+ and to Teth = 1.
3.2 Inference of Time and DeepRRTime
The inference step of our method DeepRRTime is essentially identical to that of the original Deep'Time method, which we repeat here. Inference begins with the evaluation of the INR basis at the time index values for which lookback data are available: Z:- = f 2ro.... 2r, } = (T)( fo(2)), ...... fo(r))). We then "cxplain the past" by solving the following system of equations for W and b. Ye = Z,W + b. Specifically, we solve for the ridge-regression optimal parameters:
W b² arg aigmin |Y Z.W b1l2 + A1 (1 1 + [b112)
(1)
W
4

---

Published in Transactions on Machine Learning Research (01/2025)
where X1 is an L2 penalty coefficient.
The parameters (W*,5% 6*) which solve the ridge regression problem over the lookback window can be obtained T Ye where Z = [Ze; 1] is obtained by concatenating with the glosed-form solution (W) ) = (Z, \1d) - z
a vector of ones. They are then used to extrapolate O(T) = 2, W + b* into the forecast region and generate the set of predictions:
Y W +b",
(2)
where Zi, (Zre41 Pre421... Zr4tn
3.3
Training of Deme
We begin by reviewing the training procedure for Deep'Iime, and later explain how our method DeepRRTime differs. The core idea is that the meta-lcarner, fe, is exposed during training to the way in which the base learner, ridge regression, solves for the optimal parameters (W*, b*). The meta-learning routine can therefore learn an INR basis that allows the base learner to forecast the future reliably.
Deep optimizes the INR fo to learn basis {2+} } that, through the above inference procodure, minimizes the forecasting error. Specifically, given Y,, Ze and Zn, we first compute (W*,"% and then use Equation 2 to compute Yn. We then optimize the INR parameters, 0, to minimize the forecast mean squared error (MSE):
L Forecast (y. 9. 8) = arg min ||Y, (,, W (*). 6" (0). (0))
(3)
where Y, represents the ground-truth observations at the same time indices as Yn. Note that the inner optimization step itself depends on 0, as implied by the notation (0), (0)). The closed form solution to ridge regression enables Dime to propagate efficient, high-quality gradients through the entire inner optimization step. The meta-learning step is key to DeepTimme's adaptation for time-series forecasting: among all possible representations, the INR. seeks to learn one that allows the ridge regression solver to explain the past values, Ye, in such a way as to reliably forecast the future values, Yn.
4 DeepRRTime: regularized basis for improved forecasting
In this section, we introduce a regularization technique to improve the forecasting accuracy of the DeepTime model by refining the basis functions learned by the INR. Informally, we consider the question of how to learn temporal patterns in the basis elements that will improve the forecasting accuracy. Since "good" temporal patterns are highly dependent on the specific forecasting task, it is difficult to define them in a domsin-agnostic way. Therefore, we approach this problem through the lens of the ridge regression employed by Deep'Time, asking instead how to learn basis elements that are better suited for ridge regression. In the following, we identify key propertics of such a basis and propose a regularization objective to encourage these properties during the training process.
We first note the sharp contrast between our method and traditional regularization techniques to improve the conditioning of linear regression correlated variates, c.g., (Adnan & Hura Ahmad, 2006; Paul, 2006; Herawati et al., 2018; Chan et al., 2022). As rule, such methods assume that the correlated variates are specified externally to the problem, and then seek to process them one way or another to minimize the impact of the collinearity on the regression process. In contrast, the INR representation in Time is entirely learned; there are no externally specified collinear variates to disentangle. Our method exploits the unique freedom in this setting to construct these variates specifically such that they lead to a well-conditioned regression problem.
Linear regression problems have been the subject of numerous theoretical analyses owing to their simplicity and efficiency. In the following, use the theoretical results from Krikheli & Leshem (2021) to develop an understanding of how the basis influences the prediction errors. Theorem III.1 in (Krikheli & Leshem, 2021) suggests that the smallest and the largest eigenvalues (Amin and Xmax) of the sample covariance matrix influence the sample-effciincyy for a required error threshold. Therefore, for a given number of train samples, a basis whose covariance matrix has larger Amin and smaller Amax would result in lower prediction error. Motivated by this observation, we propose to improve the ridge regression optimization by controlling the eigenvalues of the covariance matrix through regularization term.
5

---

Published in Transactions on Machine Learning Research (01/2025)
In standard applications, linear regression is applied directly to observed data with fixed data-determined covariance matrix. In contrast, the basis 2- in our case is learned, enabling direct control of the covariance matrix properties. While it is possible in principle to directly minimize the largest eigenvalue and maximize the smallest cigenvalue of the sample covariance matrix, the key parameters in the theorems of Krikheli & Leshem (2021), we have found it more practical to make two variations as discussed below.
4.1 Centered covariance matrix regularizer
Motivated by the theory of Krikheli & Leshem (2021) on robust regression, we aim to develop a regularizer that encourages the INR to learn basis with better-condiitoned covariance matrix. Instead of regularizing the uncentered covariance matrix, which arises directly in the theory, we instead regularize the centered covariance matrix,
1
Go L ZZ pu
(4)
where Go(is) indicates the covariance between the i-th and j-th basis elements, Z [Ze; Zy.] is the con- catenation of the lookback and forecast bases and μ I+Z Σ, 2, RD is the mean along the temporal axis for each of the D basis dimensions. Compared to regularizing the uncentered covariance matrix, this approach leaves the absolute means of the basis elements unconstrained; preliminary experiments suggested this flexibility was advantageous to the learning process.
From the Weyl inequalities, we show below that controlling the eigenvalues of the centered covariance matrix will also control most of the eigenvalues of the uncentered covariance matrix. The exception is the largest eigenvalue, which can be poorly controlled if the basis elements are large.
Theorem 1. (Weyl, 1912) For Hermätian matrices A, B E C", the eigenvalues of A+ B related to the eigenealues of the individual matrices as follows:
Aitj- (A + B) S (x(A) +B,( n+1
(5)
(x) - (B) S Aits-n
B), i+j2n+1
(6)
where i,j = 1,...., n and An I An- S ... S A1-
Specifically, if A = Gø, B = pp and A + B = L+H = ZZ= we can find upper- and lower-bounds for the cigenvalues of the uncentered covariance matrix'. Considering i = n and j = n in Equation (6), we conclude that Ao (Ga) S Am (It Zzz That is, the smallest eigenvalue of the uncentered covariance matrix (which must not be too small, according the theory of Krikheli & Leshem (2021)) is lower-bounded by the smallest. eigenvalue of the centered covariance Gg. Thus, it is sufficient to control the smallest eigenvalue via Gø- Considering j = 1 and = =1 in Equation (5), we see that X1 ( ( zzT) S Ai(GG) + a μ.. That is, the largest eigenvalue of the uncentered covariance matrix exceed that of Go by JTT for basis means μ.
In practice, we have found that the INR generally does not appear to learn basis elements with pathologically large mcans. The INR initialization and training protocols do not seem conducive to representing such functions a priori, and we would expect the forecast error term to be sufficient to prevent them from emerging during training. Conversely, as noted above, our early experiments empirically showed that the extra flexibility of unconstrained basis means appears to be very advantageous to the training process. Thus, we find that regularizing Go instead of the uncentered covariance matrix is the more practical approach.
4.2 Indirect regularization of eigenvalues
In our preliminary experiments, directly regularizing the largest and smallest eigenvalues of Go led to instabilities in the optimization process. Instead, we found it more tractable and effective to regularize Go towards the identity matrix, thereby regularizing all of its eigenvalues towards 1, using a term of the form:
LCW (0)
1 D
Σ
2olis) + Σ (Go(i) - 1)2 D2
1 Goo 1|?: -11;:
1S
13I5D
where I|- stands for Frobenius norm. Intuitively, the first sum (C)r penalizes non-zero covariances between elements in the centered basis, while the second sum encourages the variances of each element in the
NNote that since (pp )) = p(n (1) (n p)p, and since up T is rank one, the only non-zero eigenvalue of pq is p TT
6

---

Published in Transactions on Machine Learning Research (01/2025)
centered basis to be close to 1. Therefore, when (f) is small the centered basis is closer to orthonormal, and it is orthonormal if and only if Go equals the identity matrix. The fact that all the eigenvalues of Go lie within an interval near 1, with the size of the interval upper- boundee in proportion to (C), directly follows from Theorem and we refer the reader to Appendix A for further details.
Theorem 2. (Gershgorin, 1981); or, e.g., (Johnson & Horn, 1985) for an English presentation. Let A be complex n x n matriz with entries agj. For i n. let R3 be the sum of the absolute values of the non-diagonal entries in the i-th row:
R₁ = Elagl-
571
Let D(ag; Ra) C C be a closed disc centered at au with radius Rg Such disc is called Gershgorin disc.
Then every eigenvalue of A lies wilhin at least one of the Gershgorin discs D(asi Rz).
Although the theory of Krikheli & Leshem (2021) does not directly require all eigenvalues to be near 1 for regression to be robust, this is certainly a sufficient condition for their robustness results to apply. Specifically, this theory's regression error bounds diverge as the smallest eigenvalue of the uncentered covariance matrix approaches zero, which is an outcome that (0) directly discourages.
4.3 Summary of DeepRRTime In summary, our proposed method, DeepRRTime,, differs from the original framework of the primary objective for the outer loop of training to the following:
Deep'Time by altering
Forecast (y. g. 0) + AgCcor
(9)
where A2 is the covariance regularizer coefficient.
Constraining Ccov(0) provably improves the conditioning of the INR basis. The smallest eigenvalue of the uncentered covariance matrix is discouraged from being too small, with an upper bound of its distance below controlled in proportion to (0). The largest eigenvalue of the centered covariance matrix is also bounded near 1, with the largest eigenvalue of the uncentered covariance matrix growing only insofar as the basis means are large (which does not appear to occur in practice). As we show in Figure 6 in the Appendix, minimizing Equation (9) does reduce Lcov of the DeepRRTime model substantially compared to DeepTTime In fact, Lcov grows with training epochs for DoepTime, suggesting that the basis is becoming increasingly ill-conditioned over time. In the following section, we demonstrate empirical evidence to support these views: Deep Time indeed exhibits the instabilities that one would expect from a correlated basis, whereas DeepRRTime exhibits the robustness we anticipated from a basis better suited for regression.
5 Results
In this section, we evaluate the effectiveness of our regularization method, DeepRRTTime, on real-world datasets and compare it with relevant baselines. In addition to the regular timescries forecasting problem, we also test its nobustness in three more challenging settings to illuminate advantages of the proposed regularization technique: forecasting with missing values in the lookback window, training with reduced datasot size, and forecasting on a finer time grid at test time than the one used during training.
5.1 Experimental setup
For evaluation, we use real-world benchmarks for LTSF which include Electricity Transformer Temperature (ETTm2), Electricity Consumption Load (ECL), Exchange, Traffic, Weather, and Influenza-like IlIness (ILI), detailed in Appendix B. For each dataset, we evaluate our model on tasks with four distinct forecast horizons, as is standard practice in the related literature. We also use the standard train, validation, and test sets splits: 60/20/20 for ETTm2 and 70/10/20 for the remaining 5 datasets (Woo et al., 2023). We preprocess each dataset by standardization based train set statistics. To compare our method with other approaches, we employ two metrics: mean squared error (MSE) and mcan absolute error (MAE). For cach experiment, we report error statisties across 10 random network initializations: only average errors are reported in the main text, with standard deviations included in the Appendix. Hyperparameter selection, when applied, is performed for the lookback length multiplier y which defines the length of the lookback window relative to the prediction horizon H as L pH. We search through the values μ E (1,3, 5, 7, 9), and select the best value based on the validation loss. Throughout this work, we use identical values for all common
7

---

Published in Transactions on Machine Learning Research (01/2025)
hyperparameters of DeepTimm and DeepRRTime besides the lookback multiplier μ.. We recompute the results of Teme using the original open source code implementation, and report errors over 10 random network initializations. We emphasize that our implementation is largely based on the original code of Deep'Time and the proposed regularization requires minimal code changes. Please refer to Appendix C for a complete list of hyper parameters and further implementation details.
5.2 Multivariate forecasting
In Table 1, we consider the regular multivariate Lime-series forecasting problem and compare our proposed approach, DeepRRTime, with other time-index models for multivariate time-series forecasting including TimeFlow (Naour et al., 2024) and Gaussian Processes (GP) (Rasmussen, 2004). We emphasize the comparison of DeepRRTTime with Deep'Time in particular, to understand the relative benefits of our proposed regularization. We also compare DeepRRTime to historical-value models including recent Transformer-baser architectures such as PatchTST (Nie et al., 2023) as well as NLinear, a linear timeseries forecasting model, shown to be competitive with many of the far more computationally expensive Transformer models (Zeng et al., 2022). Lastly, we include the performance of a simple martingale model that outputs the last observed value as a forecast and was shown to be competitive some LTSF benchmarks. Additionally, Tables 5a and 5b in the Appendix extends the comparison of DeepRRTime to a broader set of historical-value models including NS Transformer (Liu et al., 2022), N-HiTS (Challu et al., 2023), ETSFormer (Woo et al., 2022), FEDformer (Zhou et al., 2022), CrossFormer (Wang et al., 2022), TimesNet (Wu et al., 2023) and i'Trnsformer (Liu et al., 2023). All experiments in this section are conducted with the regularization coefficient fixed to A2 = 1, as jointly optimizing X2 with the lookback multiplier fa for the full set of benchmarks was prohibitively expensive.
Time-index models
Historical-valee models
DecpRRTine
DeepTime
GP
TimeFlow
NLincar
Martingale
PatchTST
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTm2
0.166
0.258
0.166
0.258
0.442
0.422
0.269
0.322
0.167
0.255
0.266
0.328
0.166
0.256
192
0.224
0.300
0.223
0.299
0.605
0.505
0.394
0.399
0.221
0.203
0.340
0.371
0,223
0.296
336
0.276
0.338
0.278
0.339
0.731
0.569
0.523
0,471
0.274
0.327
0.412
0.410
0.274
0.329
720
0.368
0.397
0.383
0.411
0.959
0.609
0.663
0.557
0.368
0.384
0.521
0.463
0,362
0.385
96
0.137
0.238
0.137
0.238
0.503
0.538
0.141
0,240
0,141
0.237
1.588
0.946
0.129
0.222
ECL
192
0.152
0.251
0.152
0.252
0.505
0.543
0,155
0.251
0.154
0.248
1.595
0.950
0,147
0.240
336
0.165
0.267
0.166
0.268
0.612
0.614
0.170
0.268
0.171
0.265
1.617
0.961
0.163
0.259
720
0.202
0.303
0.202
0.302
0.652
0.635
0.203
0.300
0.210
0.297
1.647
0.975
0.197
0.290
Exchaang
0.078
0.197
0.079
0.199
0.136
0.267
0.307
0.395
0.080
0.208
0.081
0.196
0.088*
0.207
192
0.153
0.284
0.152
0.285
0.229
0.348
1,450
0.658
0.180
0.300
0.167
0.289
0.191
0.312
336
0.257
0.375
0.324
0.424
0.372
0.447
3.691
1.063
0.331
0.415
0.305
0.396
0.358
0.436*
720
0.541
0.540
0.675
0.592
1.135
0.810
8.184
1.626
1.033
0.780
0.823
0.681
0.932*
0.728
96
0.390
0.274
0.390
0.274
1.112
0.665
2.623
0.287
0.410
0.279
2.723
1.079
0.360
0.249
Traffic
192
0.402
0.278
0.402
0.278
1.133
0.671
5.621
0.305
0.423
0.284
2.756
1.087
0.379
0.256
336
0.416
0.285
0.416
0.289
1.274
0.723
23.648
0.331
0.435
0.290
2.791
1.095
0.392
0.264
720
0.450
0.307
0.450
0.308
1,280
0.719
15.013
0.357
0.464
0.307
2,811
1.097
0.432
0.286
Weather
96
0.166
0.222
0.167
0.223
0.395
0.356
0,186
0.242
0.182
0.232
0.259
0.254
0.149
0.198
192
0.207
0.260
0.207
0.260
0.450
0.398
0.252
0.299
0.225
0.269
0.309
0.292
0.194
0.241
336
0.251
0.298
0.252
0.300
0.508
0.440
0.318
0.343
0.271
0.301
0.377
0.338
0.245
0.282
720
0.312
0.348
0.313
0.350
0.498
0.450
0.393
0.394
0,338
0.348
0.465
0.394
0.314
0,334
24
2.317
1.044
2.558
1.115
2.331
1.036
3.199
1.228
1.683
0.858
6.587
1.701
1.319
0.754
36
2.253
1.022
2.264
1.042
2.167
1.002
3.166
1.212
1.703
0.859
7.130
1.884
1.579
0.870
48
2.292
1.033
2.302
1.027
2.961
1.180
3.128
1,150
1.719
0.884
6.575
1.798
1.553
0.815
2.301
1.035
2.292
1.030
3.108
1.214
3.563
1.277
1.819
0.917
5.893
1.677
1.470
0.788
Table 1: Comparison of DeepRRTime with time-scries forecasting. For reference, we also provide the corresponding results for representative historical- value models. Best results for time-index models are bolded, and best results overall are colored in blue. *The results of PStchTST on the Exchange dataset were not reported by the authors and are therefore computed by while the remaining results are obtained from the respective papers. We summarize our findings from these experiments as follows:
models on multivariate benchmarks for long sequence
(a) DeepRRTime matches or exceeds the performance of other time-index models overall. DeepRRTimee achieves the best performance out of the time-index models in 38 out of 48 settings in
8

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1. Moreover, considering the standard deviations and the fourth digit of precision reported in Table 6 in the Appendix, we find that the cases where DeepTTmme outperforms DeepRRTime are generally not statistically significant: only the difference in MSE on ECL/720 comes close, with a difference slightly exceoding two standard errois. Similarly, the underperformance relative to GP on ILI/24 is not stalistically significant, although the underperformance relative to TimeFlow in of metrics on ECI./720 and relative to GP on ILI/36 are significant. Altogether, accounting for measurement uncertainty, DeepRRTime matches or exceeds the other time-index models in 45 out of 48 results, and matches or exceeds DeepTTmme in all settings. Conversely, 17 of the cases where DeepRRTime outperforms Deep are statistically significant as determined by one-sided Welch's t-test with a significance level of 0.05. Most notably, the covariance regularization introduces significant improvements on the Exchange dataset, especially on the longest. forecast horizons (336, 720) where we achieve between 10% and 20% improvement in MAE and MSE over Deep'Time. Altogether, these results suggest that the DoepRRTime regularizer a clear improvement to DeepTime's already statc-of-tho-art performance among lime-index models.
(b) DeepRRTime achieves state-of-the-art performance on the challenging Exchange dataset and performs comparably with historical-value models on other datasets. Despite the improvements of DeepRiTTime over Deee'Time, PatchTST outperforms both almost uniformly on 5 out of of the LTSF benchmarks. We find this suggestive of strong inductive bias in the PatchTST model that is very beneficial to modelling many types of time-series data. However, we see that both DeepTime and DeepRRTimm outperform PatchTST by a large margin on the Exchange dataset, suggesting this same inductive bias can be counterproductive for some data types. The Exchange dataset is considered to be a very challenging LTSF benchmark due to its low signal-to toiisse ratio and its low degree of stationarity (sce Table 8 in the Appendix reporting the ADF test statistic (Liu et al., 2022), a measure of non- stationarity). Indeed, the authors of PatchTST (Nie et al., 2023) acknowledged omitting Exchange from their experiments for this reason. Besides PatchTST, Table 5a in the Appendix shows that DeepRRTime outperforms the other Transformer-base based historical-value models uniformly on ECL, Exchange, and Traffic, and on nearly all the ETTm2 metrics, although results are more mixed on Weather and ILI datasets.
The results of this section demonstrate that DeepRRTime a statc-of-the-art time-index model, and that time-index models can be competitive with state-of-the-art historical-value models in at least some cases. Although PatchTST still exhibits strong advantage in predictive performance many LTSF datasets, it is also worth noting that time-index models have other practical advantages. For one, time-index models are generally less computationally expensive, especially at inference time (in particular, DecpRRTime is much more efficient than PStchTST as shown in Table 11 in Appendix). The next several sections explore use cases highlighting other putative advantages of time-index models over historical-value models, and demonstrate that the regularization added to DecpRRTime is key to capitalizing on these advantages.
Evalustion
No missing values
50% missing lookback values
Average improvement
Highest improvernent
Average improvement
Highest improvement
Train portion
Data
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTm2
1.05%
0.84%
3.92%
3.41%
4.13%
3.21%
8.66%
5.77%
ECL
0.15%
0.11%
0.60%
0.40%
0.50%
0.56%
0.94%
0.93%
Exchange
10.28%
5.42%
20.68%
11.56%
24.46%
11.58%
53.66%
23.52%
Full
Traffic
0.00%
0.48%
0.00%
1.38%
0.445
0.88%
0.71%
1.60%
Weather
0.33%
0.42%
0.60%
0.67%
-0.11%
0.12%
0.50%
0.72%
ILI 2.49%
1880%
9.42%
6.37%
1.85%
0.99%
6.84%
3.97%
ETTm2 22.65%
16.39%
41.91%
28%
23.75%
16.92%
42.33%
29.20%
ECL 14.64%
13.07%
20.40%
16.96%
10.13%
8.52%
16.59%
12.12%
10% of data
Traffic 0.43%
1.21%
0.65%
1.941%
0.5256
0.96%
0.68%
0.99%
Weather
40.53%
31.11%
60.36%
46.70%
46.92%
37.62%
60.95%
47 80%
Table 2: Summary of improvements introduced with the proposed regularizer (i.e., DeepTime DeepRRTime) for all combinations of training (e.g., full-data 10% of training data) and evaluation (e.g., 50% missing lookback values vs no missing values). We observe that the regularization technique introduces higher improvements for more challenging settings of train and/or evaluation. For each datasel, we average results over forecast horizons and report average and highest relative improvements in terms of MSE and MAE,
9

---

Published in Transactions on Machine Learning Research (01/2025)
H 96
H 192
336
H= 720
2 10"
10"
10a
10
MSE
MSE
MSE
6x 10-
MSE
10"
x 10-
10
3 10-1
6x 10
25
50
75
99
25
50
75
99
25
50
75
99
25
50
75
99
Missing lookback valtoes, %
Missing lookback valuss, %
Missing lookback values, %
Missinn lookback values, %
DeepTTwe
DeepRRRiime(A> 1)
Figure 2: Plots of mcan squared error (MSE) of DeepRRTime and DeepTime function of missing lookback values percentage for different forecast horizons on Exchange dataset. The shadow areas report standard deviations over 10 network initializations. While the performance of DeepTime deteriorates significantly for higher missing rates, our model exhibits a remarkable level of robustness to the missing values, even when 90% of the values are missing.
50% missing lookback values
No missing values
50% missing lookback values
No missing values DeepRieTime
Methods
DeepTTime
DecpRleTime
DeepRRTime
Methods
Deep
DeepRRTime
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
ETTTm2
96
0.200
0,301
0.183
0.284
0.165
0.258
Traffic
96
0.417
0.295
0.417
0.293
0.389
0.273
192
0.233
0.315
0.230
0.312
0.224
0.300
192
0.424
0.297
0.422
0.295
0.401
0.278
336
0.282
0.347
0.278
0.343
0.276
0.337
336
0.438
0.305
0.435
0.300
0.416
0.285
720
0.383
0.414
0.363
0.394
0.368
0.397
720
0.475
0.325
0.473
0.322
0.450
0.307
96
0.155
0.263
0.154
0.262
0.136
0.237
Weather
96
0.176
0.242
0.177
0.243
0.166
0.222
ECL
192
0.166
0.274
0.165
0.272
0.151
0.250
192
0.216
0.275
0.218
0.277
0.207
0.260
336
0.180
0.289
0.179
0.287
0.165
0.266
336
0.262
0.314
0.260
0.312
0.251
0.298
720
0.215
0.319
0.214
0.319
0.201
0.302
720
0.318
0.359
0.316
0.357
0.312
0.348
Exchange
0.175
0.268
0.081
0.205
0.077
0.197
24
2.571
1.120
2.395
1.076
2.317
1.043
192
0.166
0.303
0.158
0.292
0.152
0,284
36
2.309
1.046
2.291
1.033
2.252
1.022
336
0.311
0.416
0.259
0.379
0.256
0.375
48
2.352
1.032
2.344
1.039
2.291
1.033
720
0.665
0.593
0.516
0.532
0.541
0.539
2.328
1.033
2.341
1.039
2.301
1.034
Table 3: Our covariance regularization introduces further improvements upon Deep'Time when forecasting with 50% of lookback values missing. For reference, we also include the results of DeepRRTime when the whole lookback window is available. See Table for the standard-deviaaioos..
5.3 Missing values in the lookback window
We now explore a more challenging setting to showcase additional benefits of the covariance regularization: forecasting with missing observations in the lookback window. In contrast to historical-value models, which typically assume fixed lookback window, time-index models can, in principle, handle missing values in lookback windows without any extra architectural modifications even when trained with regularly-sampled training data (i.e., no missing values). While other model families, such as Neural ODEs (Rubanova et al., 2019) and Graph Networks (Yalavarthi et al., 2024), have been designed to handle irregular time-series, we do not compare to them in this work and focus instead on time-index models. We will show in this section, however, that although DeepTime is technically capable of making forccasts in spitc of missing values, it does not perform very well in this setting. In contrast, we find DeepRRTine is extremely robust to missing values in the lookback window, consistent with the expected advantages of learning better-conditioned basis.
For this experiment, we do not change the training of DecpTime and DeepRRTime, but randomly mask out 50% of the samples in the lookback window at test lime when computing the regression coelficients W' and b* (Equation 1). Table 3 presents the performance of both models in this setting, compared to the performance of DecpRRTime without missing lookback values. We notice that DecpRRTime is more resilient to this test than Teee on all datasets except Weather; on the shortest two horizons of Weather, DeepRRTime underperforms DeepTime by a statistically significant but nonetheless small amount. Importantly, the missing-
10

---

Published in Transactions on Machine Learning Research (01/2025)
(a) No missing lookback values
(b) 50% missing lookback values
10% of data
Full data
10% of data
Full data
Methods
Deep'T'me Time
DeepRRTimee
Deep
Methors
Deep'Tine Time
DeepRRTime
DeepRRTTim
Metries
MSE
MAE
MSE
MAE
MSE
MAE
Metries
MSE
MAE
MSE
MAE
MSE
MAE
ETTm2
96
0.210
0.306
0.181
0.276
0.165
0.258
ECL ETTm2
96
0.214
0.309
0.180
0.277
0.183
0.284
192
0.285
0.357
0.241
0.313
0.224
0.300
192
0.287
0.359
0.239
0.312
0.230
0.312
336
0.374
0.412
0.301
0.351
0.276
0.337
336
0.374
0.412
0.299
0.350
0.278
0.343
720
0.653
0.566
0.379
0.403
0.368
0.397
720
0.652
0.565
0.376
0.400
0.363
0.394
96
0,200
0.313
0.159
0.260
0.136
0.237
96
0.217
0.330
0.181
0.290
0.154
0.262
ECL
192
0.209
0.323
0.175
0.276
0.151
0.250
192
0.217
0.331
0.192
0.299
0.165
0.272
336
0,231
0.343
0.193
0.292
0.165
0.266
336
0.240
0.351
0.220
0.325
0.179
0.287
720
0.264
0.365
0.249
0.342
0.201
0.302
720
0.270
0.370
0.259
0.352
0.214
0.319
Weather Traficc
96
0.414
0.288
0.411
0.283
0.389
0.273
Weather Traflic
0.438
0.303
0.435
0.300
0.417
0.293
192
0.428
0.302
0.425
0.299
0.401
0.278
192
0.448
0.315
0.445
0.312
0.422
0.295
336
0.446
0.298
0.446
0.295
0.416
0.285
336
0.473
0.316
0.472
0.313
0.435
0.300
96
0.273
0.346
0.188
0.252
0.166
0.222
96
0.313
0.389
0.185
0.249
0.177
0.243
192
0.394
0.435
0.256
0.317
0.207
0.260
192
0.425
0.464
0.237
0.293
0.218
0.277
336
0.535
0.509
0.344
0.388
0.251
0.298
336
0.543
0.518
0.317
0.364
0.260
0.312
720
0.900
0.713
0.357
0.380
0.312
0.348
720
0.886
0.710
0.346
0.370
0.316
0.357
Table 4: The proposed regularizer further improves forecast upon DeepTime when models are trained using 10% of the most recent data. For reference, we also provide the results when using the full training dataset.
value experiment helps to reveal differences between DeepTime and DeepRRTime that were not evident in the regular forecasting setup - in Table 2, aggrogating the improvements of DeepRRTime over Tome in different scenarios, we generally observe that the difference between Deep'Time and DeepRRTiime widens in the missing-value setting as compared to the default, setting: examples where average MSE improves are ECL Based
1..15%00.0005 Traffic 10.00%%0.44%% ETTm2 (1.05%-44.55)) and Exchange (10.28%-224.4%%) on the Welch's ono-sided t-test, we find that DeepRRTime achieves statistically significant improvements in 30 out of 40 metrics on the missing-value test. Interestingly, the state-of-the-art performance of DeepRRTime the longer forecasting horizons of Exchange barely degrades at all, compared to its performance without masked inputs, illustrating the robustness conferred by our regularization.
We further extend our analysis and plot MSE of DeepRRTime and DeepTTmme with missing rates of 25%, 50% 75,, 90%, and 99% on the Exchange dataset in Figure 2. The plots show that while the MSE of DeepTime grows up to an order of magnitude with increasing masking rate, DeepRRTime exhibits little degradation even when 90% of the lookback values are missing. Also see Figure in Appendix for results extended to other datasets.
In summary, the regularization we have added to DeepRRTime enables robust forecasting in the presence of missing lookback values. This is a putative advantage of iine-index models over historical-value models, but our experiments reveal that the original Deep'Time model does not actually perform stably in this setting, We also reiterate that most historical-value models, including PatchTST, cannot natively operate with lookback window samples that differ from the exact sequence on which they were trained ( [or e.g., see Table 12 for an evaluation of Patch-TST with missing-values using innear-nterpolltionn// tatterpplatioo//errrsutsttttt /eerrs--obsiitttiinnnnnn to handle missing values): the forward evaluation of these models is simply undefined for sequences of different length. We expect that the robustness of DeepRRTime to missing samples in the lookback window should be particularly useful in certain real-world scenarios where only irregularly sampled values are available at test time.
5.4 Smaller training dataset size
A common method to evaluate new regularization technique is to test its ability to improve generalization when training with smaller datascts (Srivastava et al., 2014). To create smaller training datascts, we select the last 10% of the training set, resulting in 10x fewer observations. As this leads to reduced updates per epoch as compared to the default training scenario, we increase the number of epochs, early-stopping patience and warmup epochs by 10x. We normalize the data using mcan and standard deviation statistics estimated using the reduced training dataset. To obtain error metries comparable to the other settings, we report MSE and MAF obtained by renormalizing model outputs using statistics estimated using the full training dataset.
11

---

Published in Transactions on Machine Learning Research (01/2025)
ETTm2
ECL
Traffic
Weather
0.6
0.250
0.48
0.8
0.225
MSB
MND
MSB
0.44
MSD
0.6
0.200
0.4
0.175
0.42
0.2
0.2
10
10
10
10
10
Regularizer strength, A
Regularizer strength, Ag
Regularizer strength,
Regularizer strength, Ag
DeepRRTiime
DerpTime
11-96
11-192
II-330 I1-220
Figure Plots showing influence of the regularizer strength A2 on MSE when models are trained on 10% of the most recent data. The results are shown for different datasets and forecast horizons (color-coded according to the legend). The baseline performance of Deep'Time is shown with dashed lines, while performance of DeepRRTime is shown with solid lines. The shadow areas report standard deviations over 10 network initializations. We observe that the covariance regularization introduces significant improvements with increasing regularization strength. Note that X2 is shown logarithmic-scale.
We exclude the Exchange and ILI datasets well Traffic/720 from this experiment, since reducing these datasets 10x leaves too few training samples to form a single training batch. In this section, evaluate DeepRRTTime by selecting 22 from {1, 10, 25, 50, 75, 100} based on the validation loss.
Table presents a comparison between DeepRRTime and Deep'Time when trained on 10% of data for both the settings without. missing lookback values and with 50% of lookback values missing. As observed Table 4a, Deep'Time experiences a significant performance drop when trained on reduced datasets. In contrast, our regularization considerably narrows this performance gap, with DeepRRTime outperforming Deep by approximately 20%, and the average MSE of DeepRRTime increasing by 11.8% with reduced dataset sizes compared to a 30% increase in the case of DeepTime. A similar trend is observed in Table 4b when forecasting with missing lookback values. Figure 3 shows the performance of DeepRRTime on every dataset for different values of 22 varying from X2 = 0 (i.e., no regularization corresponding to DeepTTime) to X2 = 100. These results suggest that, in most cases, tuning A2 for a specific problem improves performance, and that the results in Scction 5.2 might be further improved if sufficient resources were allocated to training with different values of A2. Based these experiments, suggest that our regularization objective is partieularly likely to add value over the unregularized Deep'Time when training on relatively small training datasets.
5.5 Time interpolation
Lockback Window
FortonstHrrton
Figure 4: An illustration of the test-time interpolation setting where the time-index model is used to generate forecasts at 2x the frequency it was trained at. At train time, the grey points on the time grid are used to estimate the lincar-rogression parameters while the green points on the time grid are used to compute the train loss. At Lest time, we exploit the flexibility of time-index models to interpolate between the training time grid (i.e., green points) to forecast at a higher frequency (i.e. both green and red points). Note that the rod points on the time grid denote time indexes that are only seen at test time.
In this section, we consider the problem of forecasting at a greater frequency at test-time (e.g., every 30 minutes) using a model trained to forecast at a smaller frequency (c.g., every hour). Recall that the DeepTime model first solves the least-squares optimization problem defined in Equation (1) to compute the optimal parameters W*, by based on lookback observations; next, the optimal parameters applied the basis 2, so(r) to
12

---

Published in Transactions on Machine Learning Research (01/2025)
96
== 192
336
H 720
1.0
1.0
1.0
0.8
0.5
MSE
0.35
MSE
0.6
0.6
Farosurt leequency multtppies,
Forecast lrqquency multipleer
Foreour lrequenc maltipless
Pocecast Irequency muhtiplorr
DocpTTmme
DespltTTime
Figure Comparison of our regularized method with the baseline on the test-time interpolation setting the ETTm2 dataset. The shadow areas report standard deviations over 10 network initializations. We observe that DeepRRTTime achieves significant improvements over DeepTTime when forecasting at a frequency that is y times higher at test time than the frequency observed during training.
generate the forecast for the following values of r: { (I-Ā-I. L-H-T,"... where T. is the lookback window
L+1
and H is the forecast horizon. If the model is trained over hourly observations, then I++ corresponds to temporal difference of one hour. In order to use this model to generate forecast at frequency of 30 minutes,
L+1 -+1_... T.......... } see we could use the following sequence of time indexes instead: T.+ T+ T++H-T T-
1-055
L+D.5
Figure for illustration. More generally, to forecast at integer frequency higher at test time compared to train time, apply the parameters W*, b* over the basis representation obtained for the following time indexes: 1h-1 P(L+H-Ij - seen by the network at train time and evaluates the network in terms of its ability to generalize to novel time indexes seen only at test time.
"L
wL+1 - ...--.....11 }} Note that this setting interpolates between time indexes
To evaluate the model's ability to forecast at higher frequency, subsample the training data at different frequencies, using the ETTm2 dataset with observations spaced 15 minutes apart. For instance, when v = 4, the training data is subsampled to simulate observations at hourly intervals. At test time, we generate forecasts at the original higher frequency (e.g., every 15 minutes). We evaluate models on different combinations of y and forecast horizons H, choosing j E (1, 5, 7,9} for both DeepTime and DeepRRTime and X2 E {1, 10, 50} for DeepRRTime based on the validation loss. The results presented in Figure 5 show that covariance regularization significantly improves performance across all (v, H) pairs compared to DeepTime.
6 Conclusion
Our simple, inexpensive covariance regularizer leads to improvements on several LTSF benchmarks, including very significant improvements on the Exchange dataset. The enhanced robustness particularly evident in the more challenging test settings: where lookback window values missing at test time, forccasting is done at higher frequency than training, or training is restricted to datasets with reduced sizes. These results suggest that regularization can greatly enhance deep time-index models. While we have only conducted preliminary hyperparameter tuming in this work, we expect that further tuning could potentially yield additional performance euhancement in practice.
7
Future Work
We believe that the results shown in our work open new opportunities for further research of deep time-index models. Moreover, we believe that the impact of our regularization method is not limited to the Deep'ime model or time-series forecasting and consider the following directions for future work. Below, we list several directions for future work that we consider promising and important:
Efficient hyperparameter tuning. Future work may consder exploring learning the optimal X2 and j through gradient- based optimization or, more broadly speaking, employing techniques for automated hyperparameter tuning methods. Furthermore, developing techniques to enhance the computational efficiency of hyperparameter search presents a valuable future research direction in the field of time-index models as well as time-series forecasting, in general.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Direct control of covariance matrix eigenvalues. Given the demonstrated efficacy of the simple covarianco-regilariiatio technique presented in our work, we believe that directly controlling the eigenvalues of the covariance matrix may be fruitful direction for developing more powerful regularization techniques.
Non-linear generalization. The benefits of the covariance regularization presented in our paper can motivate further exploration of time-index models employing non-linear base-learners (e.g., an MLP operating on the meta-learner's feature space) and regularization techniques that go beyond linear correlations.
Extensions to other applications. We believe that our proposed regularization is not specific to time-series applications and can benefit other applications that use linear networks atop meta-learner's representations (e.g., (Bertinetto et al., 2019b]) which should be considered as future work.
Extending the evaluation to more benchmarks. Given that deep time-index models is an emerging rescarch arca, it is important studied historical-value models by expanding the evaluation to diverse datasets: such datasets with high frequency seasonal patterns, highly non-stationary behavior, data from specific application domains.
understand their competitive advantages over better
References
Norliza Adnan and Robiah Adnan Hura Ahmad. A comparative study on some methods for handling multicollinearity problems. MATEMATIKA, 22(2):109 119, 2006.
Oliver D. Anderson and M. G. Kendall. Tine-series. 2nd edn. The Statistician, 25:308, 1976. URL
https://api..
htt//pppissnmnnicccchollllrr wot..iaurrrrr/roo.SDD..... org//orrpssD::340011755 134001785.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https:
baaxxiv.or//bba.. 064500.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varianceiimariance-cooraan regularization for self-supervised learning. In International Conference on Learning Representations, 2022. URL https:
// //openreview. wwoperreview.net/frrruee wtt.:///orrmmddd///062222111 21dxxx6YD62D1UU.
L Bertinetto, J Henriques, P Torr, and A Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations (ICLR), 2019. International Conference on Learning Representations, 2019a.
Luca Bertinetto, Joao F. Henriques, Philip Tort, and Andrea Vedaldi. Meta-learning with diffcrentiable closed-form solvers. In International Conference on Learning Represenlations, 2019b. URL https:
// //openreview net/forum? .ett:/torumiiiilyyzzohthttt
G.F.P. Box, G.M. Jenkins, G.C. Reinsel, and G.M. Ljung. Time Series Analysis: Forecusting and Control. Wiley Series in Probability and Statistics. Wiley, 2015. ISBN 9781118674925. URL https://books.
google.ca/boole borookks?ddrNNttggMQB...
Real Carbonneau, Kevin Laframboise, and Rustam Vahidov. Application of machine learning techniques for supply chain demand forecasting. Europeun journal of operational research, 184(3):1140 -1544, 2008.
Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 6989 6997, 2023.
Jireh Yi-Le Chan, Steven Mun Hong Leow, Khean Thye Bea, Wai Khuen Cheng, Seuk Wai Phoong, Zeng- Wei Hong, and Yen-Lin Chen. Mitigating the multicollinearity problem and its machine learning approach: a review. Mathematics, 1088): 2022.
Yinbo Chen, Sifci Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8628 8638, 2021.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVE Conference on Computer Vision and Patlern Recognilion, pp. 5939-55990 1199-5948,40 2019.
Michael Cogswell, Faruk Ahmed, Ross B. Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrclating representations. Yoshua Bengio and Yann LeCun (ods.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv, org/ 0t////111...011
Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. Adarnn: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM international conference on information & knowledge management, pp. 402-411, 2021.
Emilien Dupont, Adam Goliński, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet, Coin: Compression with implicit neural representations. arXiv preprint arXiv:2103.081 2021.
Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa:
Your data point is a function and you can treat it like one. arxiv preprint or arxiv:2201. 12204, 2022.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Elizabeth Fons, Alejandro Sztrajman, Yousef El-Laham, Alexandros Iosifidis, and Svitlana Vyetrenko. Hypertime: Implicit neural representations for time series. In NewrIPS 2022 Workshop on Synthelic Dala for Empowering ML Research, 2022. URL httpe://opppeeerreeee http://peneeviinn.ntffotrtttttte htt//forurrccco/2Faaoli
S Gershgorin. Uber die abgrenzung der eigenwerte einer matrix. lav. Akad. Nauk. USSR. Otd. Fiz-Met. Nauk, 7:749-754, 1931.
Ebrahim Ghaderpour, Spiros D Pagiatakis, and Quazi K Hassan. A survey on change detection and time series analysis with applications. Applied Sciences, (1(33):614,, 2021.
Netti Herawati, Khoirin Nisa, Eri Setiawan, Nusyirwan Nusyirwan, and Tiryono Tiryono. Regularized multiple regression methods to deal with severe multicollinearity. International Journal of Slotistics and Applications, 8(4):167-11) 172, 2018.
Charles C Holt. Forecasting seasonals and trends by exponentially weighted moving averages. International journal of forecasting, 20(1):5-10, 2004.
Jia-Chen Hua, Farzad Noorian, Duncan Moss, Philip HW Leong, and Gemunu H Gunaratne. High-dimensional time series prediction using kernel-basee koopman mode regression. Nonlinear Dynamics, 90:1785-1800 2017.
RJ Hyndman. Forecasting: principles and practice. OTexts, 2018.
Kyeoug-Joong Jeong and Mong-Min Shin. Time-series anomaly detection with implicit neural representation. arXiv preprint rrXtx:2201.11900 2022.
Charles R Johnson and Roger A Horn. Matriz analysis. Cambridge university press Cambridge, 1985.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate timo-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Michael Krikheli and Amir Leshem. Finite sample performance of linear least squares estimation. Journal of the Franklin Institute, 358(15,,7955-7991, 2021. ISSN 0016-0032. doi: httpss//totoor///.00//////aaaa 2021.07.04 URL http://wwww.ccccceeeeeeeee.e http://wwwaaaaeeeococcocoococtta
15

---

Published in Transactions on Machine Learning Research (01/2025)
Somanchi Hari Krishna, Abhiruchi Passi, Vinitha Kanaka, Ishwarya Kothandaraman, and Thirumala Reddy Vijaya Lakshmi. An enhanced time series analysis to improve the performance of 5g communication systems. Engineering Proceedings, (9(1):103, 2023.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp- 35-104, 2018.
Nikolay Laptev, Jason Yosinski, Li Erran Li, and Slawek Smyl. Time-series extreme event forecasting with neural networks at uber. International conference on machine learning, volume 34, pp. 1-5. sn, 2017.
SHIYANG Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. ArXie, abs/ 2os//197.0000 00035, 2019. URL https://apis wwwanticscholalttt... Pttg/CorppII:::97666887 195766887.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting. Conference on Neural Information Processing Systems, 2022.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long, itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint (XXw:2310006662) 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 99 10-106,20 2021.
Etienne Le Naour, Louis Serrano, Léon Migus, Yuan Yin, Ghislain Agoua, Nicolas Baskiotis, patrick gallinari, and Vincent Guigue. Time series continuous modcling for imputation and forecasting with implicit neural represenlations. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreviite nett:forummid0PP2220
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. ar preprinl arXiv:1800 800.00999, 2018.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forccasting with transformers. In International Conference on Learning Representations, 2023.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Represenlations, 2019.
Ranjit Kumar Paul. Multicollincarit: Causes, effects and remedies. TASRI, New Delhi, 1(1):58 05888-65211 2006.
Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning, 2020. URL https://doi. org/ 21203/rr.3..... rs- rs-5512//1..
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of maml. International Conference on Learning Representations, 2020.
Carl Edward Rasmussen, Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
Carl Edward Rasmussen. Gaussian Processes in Machine Learning. Springer Berlin Heidelberg, 2004.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time scries. Advances in neural énformation processing systems, 32, 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Jamuschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181-1199) 2020.
16

---

Published in Transactions on Machine Learning Research (01/2025)
Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. Advances in Neural Information Processing Systems, 53:10136-10147 2020a.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in neural information processing systems, 33: 7462-7473, 2020b.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Lya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 2014.
Filip Szatkowski, Karol Piczak, Przemyslaw Spurek, Jacek Tabor, and Tomasz Trzciúski. Hypernetworks build implicit neural representations of sounds. Joint European Conference on Machine Learning and Knowledge Discovery in Dalabases, pp. 661 -676. Springer, 2023.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kei, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537 7547, 2020.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2846 2855, 2021.
Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 72(1):37 45, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Crossformer: A versatile vision transformer hinging on cross-scale attention. In International Conference on Learning Representations, ICLR, 2022. URL https://openreviiet net/forumiid= Pat.iiiddd__PLIIIxI.....I
Hermann Weyl. Das asymptotische vertellungsgesetz der eigenwerte linearer partieller differentialgeeichnnen (mit einer anwendung auf die theorie der hohlraumstrahlung). Mathemalische Annalen, 71 71(4):441-47 1912. doi: AL.00//FF01556004 URL attps:///oo..o
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. ArXiv, abs/ abs//2202.0131,, 2022. URL https://api. semanticscholar. org/ tagg/CrrpusID /CorppusIDD2244487722 246485702.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Learning deep time-index models for time series forccasting. In International Conference Machine Learning, pp. 37217-37237. PMLR, 2023.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d- variation modeling for general time series analysis. In International Conference on Leorning Representations, 2023.
Vijaya Krishna Yalavarthi, Kiran Madhusudhanan, Randolf Scholz, Nourhan Ahmed, Johannes Burchert, Shayan Jawed, Stefan Born, and Lars Schmidl-Thienn Grafiti: Graphs for forecasting irregularly sampled time series. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 16255-16263, 2024.
Gizem Yüce, Guillermo Ortiz-Jiméncz, Beril Besbinar, and Pascal Frossard. structured dictionary perspective on implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 19238, 2022.
17

---

Published in Transactions on Machine Learning Research (01/2025)
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning 2021. URL
via redundancy reduction. CoRR,
03230.
Ailing Zeng, Mu-Hwa Chen, L. Zhang, and Qiang Xu. Are transformers effective for time series forecasting?
AAAT Conference Artificial Intelligence, 2022. URL http:/ppp...mmanticcccooool. hatrmaaniiiccoolooooooo///ooollp 249097444.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference arlificial intelligence, volume 35, pp. 11106-11111 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning, pp. 27268-27222 27286. PMLR, 2022.
18

---

Published in Transactions on Machine Learning Research (01/2025)
|0o Ale
20
Train Epoch
DoepTime
DeepRRTTime
Figure 6: [Go] ||Ga F plotted across training epochs on the ETTm2 dataset with forecast horizon 96 and lookback multiplicr μ 1. The shadow arcas report standard deviations 10 network initializations. While ||Gg /|E for DeepTTmm grows with epochs, our regularizer effectively decreases this value which results in less mutually correlated basis. As predicted by theory discussed in Section 4, this leads to empirical improvements discussed throughout this section.
A Connection between the DeepRRTime regularizer and eigenvalues of Go
By the Gershgorin circle thcorem (Theorem 2), we see more prociscly that when Lcor (0) is small all the eigenvalues of Go will indeed be close to 1.
In the context of regularizing Ga by (O), note first that the radii of the Gershgorin discs of Go are constrained to be small when (f) is small because
G
Σ Ge((iii]
(DD-1)
Σ ISGASD
Golit)² < (D Cco Ccov
(10)
ISFiSD
Moreover, the center of each i-th disc close to according to (Go(ti) - 1)² D2 (Cvv
(11)
For sufficiently small (0) (for any fixed choice of D) all eigenvalues therefore lie within Gershgorin discs having small radii and centers close to 1.
Since Go Hermitian, its eigenvalues are real, and for small Ccor the cigenvalues must therefore lic small intervals near In particular, the smallest eigenvalue bounded below by
An(Go) 2 (V(D 10 (0) + Lcr
(0) = (VD- D (co,
(12)
when Loov (0) 1/ - 1) D we have \n (Go) > 0 and the basis is non-degenerate. Similarly, the largest eigenvalue of Gg also cannot be very large when Coov(() is small.
B Datasets
ETTm2 (Zhou et al., 2021) Electricity Transformer Temperature dataset provides measurements from an electricity transformer such as load and oil temperature at a 15 minutes frequency.
ECL The Electricity Consumption Load dataset comprises electricity usage data for 321 households, gathered between 2012 and 2014. Originally recorded every 15 minutes, the data is compiled into hourly aggregates.
Exchange (Lai et al., 2018) Dataset provides exchange rates of USD with currencies of eight countries (Australia, United Kingdom, Canada, Switzerland, China, Japan, New Zealand, and Singapore) from 1990 to 2016 at a daily frequency.
htttpe///githu. hwwp://///ooooooo//////ttttaeee
'attps://archive.. -ics uci www.:u.c...uullll/.co httpp///ttttt..ccccaaaaaaaaaaaat
19

---

Published in Transactions on Machine Learning Research (01/2025)
Methoos
Deep
PascITTT
NS Transloruer
N-HITS
ETStomeer
FEDioreer
NLinear
DLinear
Martingale
Metries
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
RCL BTTa2
96
0.166
0.258
0.166
0.256
0.192
0.274
0.176
0.255
0.189
0.250
0.203
0.287
0.167
0.255
0.167
0.260
0.266
0.328
192
0.224
0.300
0.223
0.256
0.280
0.339
0.245
0.305
0.253
0.319
0.260
0.328
0.221
0.293
0.224
0.340
0.371
336
0.278
0.339
0.274
0.929
0.334
0.361
0.295
0.346
0.314
0.357
0.325
0.366
0.274
0.327
0.281
0.342
0.412
0.410
720
0.368
0.307
0.362
0.385
0.417
0.401
0.425
0.414
0.421
0.415
0.384
0.397
0.521
0.465
96
0.197
0.238
0.129
0.222
0.160
0.273
0.147
0.240
0.187
0.304
0.183
0.281
0.141
0.297
0.140
0.237
1.589
0.946
192
0.152
0.251
0.147
0.240
0.182
0.286
0.167
0,289
0.199
0.315
0.195
0.308
0.154
0.245
0.153
0.249
1,505
0.950
336
0.165
0.267
0.163
0.279
0.200
0.304
0.186
0.290
0.212
0.329
0.232
0.313
0.171
0.265
0.169
0.267
1.617
0.961
720
0.202
0.303
0.197
0.290
0.222
0.321
0.243
0.340
0.233
0.345
0.231
0.343
0.210
0.297
0.203
0.301
1.647
0.975
Exchaniee
96
0.078
0.197
0.088
0.207
0.111
0.237
0.092
0.211
0.085
0.204
0.1.9
0:277
0.089
0.208
0.081
0.203
0.081
0.196
192
0.153
0,284
0.151
0.3122
0.219
0.208
0.182
0,300
0,235
0.369
0.180
0.300
0.157
0,253
0.167
0.289
336
0.257
0.375
0.398
0.436"
0.421
0.476
0.371
0.443
0.348
0.125
0.425
0.466
0.333
0.415
0.305
0.414
0.305
0.396
720
0.541
0.540
0.532"
0.728
1.002
0.760
0.888
1.025
0.774
1.000
0.500
1.033
0.780
0.643
0.823
0.681
96
0.390
0.274
0.360
0.249
0.612
0.338
0.402
0.282
0.807
0.302
0.562
0.349
0.410
0.278
0.410
0.282
2.723
1.079
Tralte
192
0.402
0.278
0.379
0.256
0.513
0.420
0.621
0,250
0.316
0,422
0.423
2.756
1,087
336
0.416
0. 0.285
0.392
0.264
0.618
0.328
0.448
0.313
0.622
0.396
0.570
0.323
0.435
0.290
0.436
0.206
2.791
1.095
720
0:400
0.307
0.482
0.286
0.653
0.355
0.539
0.632
0.396
0.595
0.308
0.466
1.097
Weatber
96
0.106
0.222
0.1449
0.158
0.1775
0.223
0.158
0.195
0.197
0.281
0.217
0:2546
0.182
0.232
0.176
0.235
0.250
0.254
192
0.207
0.260
0.191
0.241
0.215
0.285
0.211
0.237
0.336
0.269
0.230
0.309
0.292
-396
0:251
0.208
00.244
0.2822
0.3221
0.338
0.274
0.300
0. 0.298
0.353
0.333
0.380
0.277
0.301
0.265
0.331
0.377
0.338
720
0.312
0.318
0.314
0.334
0.414
0.110
0.351
0.352
0.355
0.428
0.323
0.465
0.351
24
2.317
1.041
1.319
0.754
2.291
0.945
1.862
0.869
2, 2.527
1,020
2.203
01.653
1,683
0.858
2,215
1.081
6.587
1.701
2.255
1.022
1.579
0.870
1.825
0.848
2.071
2.615
2.2722
0:976
1.708
7:130
48
2.292
1.033
1.553
0.815
2.010
0,900
2,366
1.082
2 359
0.972
2.209
0.981
1.719
0.884
2,130
1.024
6.575
1.798
1.035
1.470
0.788
2.178
2.500
2.487
1.061
1.096
0.801
(a) Best results are highlighted in bold, and second best results are underlined. *Thh results of PatchTST on the Exchange dataset were not reported by the authors and are therefore computed by us while the remaining results are obtained from the respective papers.
Methoola
CrossFaeuus
TimssNet
PuichTST
Metiiios
XISE
MAR
MSS
MAA
MAR
MSS
MAE
MAE
56
0.166
0.258
0.316
0:267
0.38
0.100
0.250
192
0.224
0.5
0.434
0.402
0.309
0.25
0.238
0.206
336
0.2275
0.338
0.507
0.549
0.321
0:351
0.311
0.318
0.274
0.329
720
0.365
0.397
1.70
1.042
0.408
0.403
0.412
0.407
0.362
0.385
96
0.133
0.238
0.219
0.311
0.108
0.272
0.148
0.26
0.129
0.222
192
0.152
0:251
0.231
0.322
0.184
0.289
0.162
0.253
0.147
0.240
386
0.165
0.267
0.216
0.357
0.195
0.3
0.178
0.360
0.168
0.250
720
0.202
0.303
0.363
0.225
Exclange
96
0.078
0.197
0.256
0.367
0.294
0.085
0.058
192
0.153
0.505
0.177
0.191
336
0.257
0.375
1,258
0,883
0.367
0.468
0.321
0.358
0.136
720
0.541
0.54
L7707
1.00s
0.961
0.735
0.547
0.201
0,002
0.728
56
0.39
0.274
0,522
0.29
0.283
0.321
0.305
0.268
0.360
0.249
192
0.8072
0.275
0.53
0.213
0.517
0.335
0.417
0.2776
0.377
0.256
306
0.416
11,255
0.558
0.265
0.629
0.336
0.433
0.253
0.392
0.264
720
0.45
0,307
0,550
0.325
0.467
0,286
Weatbee
96
0.106
0.222
0.25
0.22
0.374
0.140
0.198
392
0.307
0.277
0:221
0.194
38G
0.251
0.20%
0.385
0.28
0.305
0.278
0.206
0.245
0.262
770
0.312
0.348
0.388
0.118
0.365
0.359
0.358
0.344
0.314
0.333
(b) Best results amongst DeepRITTime, CrossFormer, TimesNet and i'Iransformer are highlighted in bold, and second best results are underlined. The best results overall are highlighted in blue. The CrossFormer, TimesNet and iTransformer results in this table are copied directly from Liu et al. (2023).
Table 5: Comparison of DeepRRTime with historical-value models on multivariate forecasting benchmarks for long sequence time-series forecasting.
T 1 Dataset from the California Department of Transportation provides road occupancy rates from 862 sensors located on the freeways of the San Francisco Bay area at a hourly frequency.
Weather Dataset provides measurements of 21 meteorological indicators such air temperature and humid- ity throughout 2020 at 10 minute frequency from the Weather Station of the Max Planck Biogeochemistry Institute.
Inteps://pams.aot. dot. gov/ h/tp:::///wxxbbe-jeee mpg. de/vetter/
20

---

Published in Transactions on Machine Learning Research (01/2025)
Methods
Deep'Time
DeepRRTime
Metrics
MSE
MAE
MSE
MAE
ETIm2
0.1658=00.0099
0.258110.0019
0.1656=0.0000
0.258510.000
192
0.2227++000119
0.2994+0.0015
0.2211+000002
0.3001+0.0022
336
0.2778=0.004 0.09
0.3386=0.00339
0.2764
0.3378:10.000
720
0,3830=0.0062
0.4114=0.0056
0.3681=0.0037
0.3970:10.004
0.1373=0.0002
0.2381=000001
0.1369=000000
0.2375+000003
ECL
192
0.1523=0.0000
0.2517=0.0005
111517=0.000:
0.2507
336
0.1656=0.000M
0.2677=000009
0.1653=0.0008
0.2669++0.001
720
0.2015++0.000
0.3023=000000
0.2018++0.0001
0.3025++0.0005
Exchange
96
0.0786=0.0018
0.1993=0.0083
0.0775=0.00005
0.19741:.00066
192
0.1519+0.0015
0.2851=0.0019
0.1528++0000322
0.28-10+0.0025
336
0.3245=0.0287
0.4991=0.0190
0.2568=0.0087
0.3752110.004
720
0.6751=0.2377
0.5918+0.1004
0.5411=0.0600 0.0664
0.5396+10.0001
96
039002=000003
0.2741=000005
0.3899=000000
0.2738
Traliie
192
0.4016:t0.00m 0.2784=0.0005
0.4018+0.000
0.2784+10.0005
336
0.4160++0.002 0.2085=000019
0.4162++0.0008
0.2854++00006
720
0.4505=
0.3078=000007
0.4502=0.000s
0.3073+00.013
Weather
96
0.1667=0.0012
0.2233:10.0015
0.1661=0.0008
0. 0.2223
192
0.2070+10.0010
0.2603:0.0011
0.2073+10.000
0.2603+000007
336
0.2522=0.0010
0.3001=0.0000
0.2510=0.0011
0.2983:10.0011
720
08131==00000
0.3501=0.0011
0.3121=0.0007
0.3481+t0.0016
24
2.5578=001427
1.1151=000357
2.3171=0.1312
1.0437+00.016
2.2642==0.12
10447=0.044J
1.02244:0.0204 2.2527=0.0597
48
2.3019
1.0270=0.0315
2.2919
1. 1.0330+:.0019
2.292110.1186
1.0296=0.0369
2.301440.1531
1.0349+0.0457
Table 6: Comparison of DeepRRTTime with Deep'Time multivariate benchmarks for long sequence time- scries forecasting. Best results are highlighted in bold. The table reports mcan and standard deviation over random worw initializations.
Hyperparameter
Value
Epochs
50
Learning rate
le-3
A learning rate
1.0
Warm up epochs
5
Batch size
256
Early stopping patience
7
Parameters inherited from Deep
Max gradient norm Layer size
10.0
256
Aj initialization
0.0
Scales
[0.01, 5,0, 50,100000 100
Fourier features size
4096
INR dropout rate
0.1
Lookback length multiplier, μ μ E (1,3,5,7,9)
Our parameters
A2
1.0
Table 7: Hyperparameters used in our experiments. parameters inherited from DeepTime.
We emphasize that we do not change any of the
21

---

Published in Transactions on Machine Learning Research (01/2025)
Dataset
Number of variables
Frequency
Number of samples
ADF test statistic
Exchange
8
1 Day
7,588
-1.889
ILI
7
1 Week
966
-5.406
ETTm2
7
15 Minutes
69,680
-6.225
ECL.
321
1 Hour
26,304
-8.483
Traffic
862
1 Hour
17,544
-15.046
Weather
21
Minutes
52,695
-26.661
Table 8: Summary of LTSF datasets with their ADF test statisties (Liu et al., 2022) where smaller ADF means more stationary dataset.
ILI Infuueza--lkke Illness dataset provides ratio of patients seen with ILI and the total number of patients, collected by the Centers for Disease Control and Prevention of the United States between 2002 and 2021 at a weekly frequency.
Table 8 presents further characteristics of the datasets including number of variables and samples, frequeney, and ADF test statistic.
C Implementation details
We use the code provided by the authors of Teep paper while only making minimal changes related to the proposed regularizer and the conditioning network. We do not change any of the hyperparameters inherited from DeepTTme and provide their values Table 7. Below we repeal the description given by the authors in their paper (Woo et al., 2023).
DeepTime model hyperparameters. DeepTime is trained using Adam optimizer (Kingma & Ba, 2014) with a learning rate scheduler following a linear warm-up and cosine annealing scheme. We use gradient clipping by norm. The ridge regressor regularization coefficient, Å1, is trained at a higher learning rate compared to the rest of meta parameters. The model is trained with early stopping based on validation loss, with a fixed patience parameter defined as the number of epochs for which the loss can increase before the training is stopped. We learn the ridge regression regularization coefficient parameter and constrain it positive values via a softplus function. We apply ReLU activation, Dropout (Srivastava et al., 2014), and LayerNorm (Ba et al., 2016) after each INR layer. The dimension of Fourier embedding layer of INR is defined independently of the size of other layers. Here we specify the total size of the Fourier embedding layer where the number of dimensions for each Fourier frequency scale is computed as the size of the layer divided by the number of scales. We refer the reader to (Woo et al., 2023) for complete details of the model.
FimeFlow implementation and reproducibility. To compare our approach with another time-index model, TimeFlow (Naour et al., 2024), which only reports results on a subset of our benchmarks, we used the implementation opensourced by Naour et al. (2024). The original code loads data from pre-processed files and does not have an interface to read CSV files. We augmented the original implementation with Deep'Time data loaders to read data directly from CSV files and did our best to redefine the concepts of batches and epochs to align with TimeFlow code's interpretation. However, were unable to reproduce the results reported by the authors on the ECL and Traffic datascts. We shared our implementation with the authors and asked them to verify the correct of our reproduction. The authors confirmed that our implementation was correct and identified the normalization procedure as the source of the performance difference. Upon further investigation, we found that the discrepancy arose from TimeFlow's use of both the train and test data to compute statisties (i.e., mean and variance) for input data pre-processing. Following standard practices, this paper reports the results of TlmeFlow that were obtained while using train set statistics to normalize both train and test sets.
D Univariate forecasting
We evaluate our proposed approach in the unviariate setting where the model predicts the last variable of multivariate datasets. We follow the literature and use the Exchange and ETTm2 datasets for this
22

---

Published in Transactions on Machine Learning Research (01/2025)
Methods
DoepTime
DeepRRTime
Metrics
MSE
MAE
MSE
MAE
ETTm2
96
0.301510.0151 0.2009
0.1835.10.0047
0.2041-1..003
192
0.2336=0.0073
0.8153+0.0087
0.2303:+0.004
0.3120=0.0060
336
0.2824=0.000) 0.3473+0.0061
0.2787+0.0044
0.3434=0.0054
720
0.3834=0.0077 0.4145+0.0067
0.3634+0.0028
0.3942+0.0042
96
0.1551=0.0003
0.263810.0006
0.1544510.0005
0.2620=0.00099
ECL
192
0.1667=00006
0.2740:10.000
0.165910.000
0.2727-0.0006
336
0.1808=0.0005
0.2897+0.0007
0.1791 +0.0004
0.2870=0.0007
720
0.2150-0.0003
0.3197+0.0004
0.2146+00.0 +0.0005
0.3192-0.0006
Exchange
96
0.1750=0.0696
0.268310.0400
0.081110.0007
0.2052=0.0013
192
0. 0.1668=0.0106
0.3037
1.158210.0004
0.2927=0.0052
336
0.3114=0.0315
0,4162+100025
0. 0.259411.0009
0.3797=0.0044
720
0.6655-0.193)
0.5937+0.0888
+6116800.0583
0.5320-0.0291
96
1.4174=0.0005
0.295110.0007
0.4172-10.0007
0.2937=0.0004
Traffic
192
0.4246=0.0005
0.297310.0004
0.4229
0.2055=0.0003
336
0.4385=0.0000 0007
0,305410.0007
0.4354+10.000
0.3005=0.0005
720
0.1759=0.0011
0.3250+0.0011
0.40330+000018
0.3223-0.00M
Weather
96
0.1766110.0112
0.2423=0.0016
0.1778=0.0011
0.243110.0003
192
0.2168:10.001 000117
0.2758=0.0019
0.2184 0006
0,2772+100007
336
0.2620=0.0016
0.3146::00012
0.2607=10.002
0.3127=0.0028
720
0.3183=000008
0,3596:0.0012
0.3168:100011
0.3570=0.0023
2.5713=001236
1.1205++0.025
2.3953::01274
1.0760=0.0193
36
2.3090=0.1166
1.0469#000430
2.2915
1.0338=0.0173
48
2.3526=0.1322
1.0326=0.0294
2.34461:0.488
1.0395+0.0407
60
2.3287+0.0833
1.0235=0.0283
2.3414#0.1340
1.0397++00041
Table 9: Comparison of DeepRRTime with Deep'Time on multivariate benchmarks for long sequence time- series forecasting with 50% of lookback values missing. Best results are highlighted in bold. The table reports mean and standard deviation over 10 random network initializations.
Methods
DipRRRTime
DospTime
N-HFTS
ETSSorre
FRDformer
N-BRATS
DoeAR
Prophet
ARIMA
GP
Metries
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.071
0.182
0.072
0.104
0.006
0.185
0.080
0.212
0.063
0.180
0.082
0.219
0.800
0:237
0.257
0.466
01:11
0.362
0.125
0.273
ETTa2
192
0.006
0.292
0.096
0.251
0.087
0.223
0.150
0.302
0.102
0.245
0.120
0.268
0.154
0.310
0.312
0.485
0.261
0.306
0.154
0.307
338
0.121
0.265
0.120
0.264
0.106
0.251
0.175
0,334
0.130
0.279
0.226
0.370
0.277
0.331
0.474
0.317
0.318
0,189
0.338
720
0.177
0.327
0.178
0.325
0.157
0.812
0.224
0.379
0.178
0.325
0.188
0.338
0.332
0.468
0.534
0.583
0.306
0.487
0.318
0.421
Exccaange
96
0.086
0.226
0.066
0.225
0.03
0.223
0.099
0.230
0.131
0.254
0.156
0.259
0.117
0.515
0.825
0.762
0.112
0.245
0.165
0.311
0.174
0.174
0.363
0.277
0.420
0.689
0.304
0.649
0.817
396
0.802
0.445
0.308
0.452
0.370
0.486
0.421
0.487
0.426
0.511
0.611
0.605
1.331
0.962
1.394
0.988
0.736
0.508
0.506
0.502
720
0.536
0.731
0.865
0.752
0.726
0.569
1.114
0.807
1.162
0.832
1.111
0.860
1,800
1.181
3.258
1.566
1.871
0.055
1.002
0.786
Table 10: Univariate forecasting benchmarks on long sequence time-scries forecasting. highlighted in bold, and second best, results are underlined.
Best results are
experiment (Liu et al., 2022; Challu et al., 2023; Woo al., 2023). We compare our approach with both time- index models including DeepTime, Prophet (Taylor & Letham, 2018), Gaussian Processes (GP) (Rasmussen, 2004), and with historical-value models including N-HITS (Challu et al., 2023), ETSFormer (Woo et al., 2022), FEDformer (Zhou et al., 2022), DecpAR (Salinas et al., 2020), and ARIMA (Anderson & Kendall, 1976). We note that the baseline models Prophet, DeepAR, and ARIMA are strictly univariate. We set the lookback multiplier to μ === 1 for this experiment and report the results and report results in Table 10. We observe that our approach achieves the best performance across the basclines on 4 out of 16 metrics and scores as the second one on more metrics. At the same time, N-HiTS outperforms our method on all the metries on the ETTm2 dataset which we believe to be explained by its inductive bias for smoother data.
23

---

Published in Transactions on Machine Learning Research (01/2025)
H 96
H 192
H 336
H 720
1.2
1.0
MSS
MSE
1.0
MSB
0.8
MSB
0.8
0.6
0.6
0.4
20
40
20
40
10
20
Forecast frequency multippier, V
Forecast frequency muttippierr P
Forerast frequency meltiplier, P
Forecast frequency multiplier, P
0.0
Ay= 1.0
A 10.0
5--50.0
Figure 7: Performance of our regularized method with on the test-time interpolation setting on the ET datasel (Aa corresponds DeepTime). The shadow areas report standard deviations over 10 network initializations. We observe that DeepRRTime achieves significant improvements when forecasting at an integer frequency v higher at test time as compared the frequency observed at train time.
Training
Inference
Peak memory (GB)
Time
Peak
Time
Dataset/ Horizon
Model
epoch (s) J
memory (CB3)
epoch [s) L
DeepRiTTime PatekTST
Exchange /96
0.449
3.88
0.153
1.45
Exchange /720
DeepRRTine PatchTST
2.98
1.88
0.774
0.257
0.491
3.59
0.208
1.41
ETTm22/96 /96
DeepRRTime
PatchTST
1.05
6.39
0.448
1.43
ETTTm2/720
DeepRITine
1.48
8.13
1.32
2,8
PatchTST
1.7
13.51
0.556
5.24
DeeRRTine
Traflie/96
PatekTST
OOM
N/A
N/A
N/A
Table 11: We measure the wall-clock time per epoch and peak memorr usage in both train and inference modes when using PatchTST and DeepRRTime on a single NVIDIA Tesla GPU (16GB). We observe that DeepRRTTime is consistently faster than PatchTST in terms of both training and evaluation time. We also found DeepRRTime to more memory efficient on average, however some exceptions exist such as Exchange/ 720. At the same time, we encountered GPU out-of-memoot error for PatchTST all forecast horizons of Traffic while there such issue for Deme..TTime.
24

---

Published in Transactions on Machine Learning Research (01/2025)
missing lookbock vallus
5055 misinn lookbock values
ST
DispRiTTime
ParchTST (replace with
PotchTST (liscar interpciation)
DeephiiTimee
W
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.165100.011
0.25
0.185620.000
0.2086211.1002
0:22181000000
0,090000000 0211
0.00095000155
0.183520.0047
0.284120.0063
0.2220000000
029035000007
00201200:0022
0.300400.3022
0.508+00025
0.7245 0096
0.5994-0.388
0-880)0000441
0.2808-0.0048
0.3020=000000
336
0.2702:0 0008
0:0280-000007
0.276410..00 0025
0.337810.00000
0005770000477
0.7372:0.00 0077
0.5265:0:0417
0:17611000100
0.2787=0.0044
0.315420.0054
730
0.365-00.0013
0:3657-00006
0.368 +0.0057
0.5070+0.9046
10430+000016
0.7378-0.010
0.5748-0.0686
0.5026+0.0266
0.8634+0.0028
0.5042-0.0042
Table 12: Comparison of DeepRRTiime with PatchTST on ETTm2 with 50% of lookback values missing, We explore two techniques to enable PatchTST forecasting with missing values: (a) replacing missing values with 0, (b) linear interpolation. While we observe a significant drop performance for PatchTST with both techniques, DeepRRTime remains robust on this cahllenging setting which highlights its advantage over historical-valuee models. Best results with the missing values are highlighted in bold. The table reports means and standard deviations 10 network initializations.
25

---

Published in Transactions on Machine Learning Research (01/2025)
H = 96
H 192
=
=20
10a
10-1
10a
5x 10
MSE
MSE
6x 10
MSB
MSE
x 10-
x
x 10
x 10
50
75
50
75
99
75
99
99
Missing lookback values, %
Missing lockback values, %
Missing lookback values, %6
Missing looksback values,
Dimp
DeepRTTimee/
ETTm2
H 96
H 192
H 336
== 720
x
4.2 x 10
4x 10-1
4x 1x10011
4 10-1
3.8
MSD
3x-10-
MSB
xxx
MSB
MSS
3.6 x 10
1
x.1x100
2x-10-1
3.2 10
25
70
580
25
50
75
99
25
50
70
900
25
50
75
99
Missing looksback v values, %
Missing lookback values, %
Missing lookback values, %
Missing look values,
DiepTime
DeepRRTimel(M (A)
(b) Weather
H = 96
H = 192
H = 336
H = 720
6x
60
6x 10
6x 10
MSE
4x 10
x0
x 10-
MSB
4xx 10
3x 10
x
10
10
2x 10
10
2x 10
2 10
75
75
99
75
75
99
Missing lookback values, %
Missing look back values, %%
Missing lookback values, %
Missing lookback back vallues, 56
Dime
DeepRRTine<A
(c) ECL.
= 96
H = 192
H 336
H = 720
10a
10a
MSE
MSS
6 10
10
10
6x 10
xx 10
4 1x00
xx 10-2
25
50
75
199
25
50
75
90
25
50
75
99
25
50
75
90
Missing lnokkback values, %
Missing lookbark valuos, %
Missing lookback values, %
Missing lookback values, 36
DiepTime
DeepRTTime\A==))
(d) Traffic
Figure 8: Plots of mean squared error (MSE) of DeepRRTime and Deep'Time function of missing lookback values percentage for different forecast horizons on ETTm2 (top), Weather (middle) and ECL (bottom) dataset. The shadow areas report standard deviations over 10 network initializations. For the ETTm2 dataset, the performance of DeepTime deteriorates more significantly for higher missing rates than our model. For the bigger datasets (i.e., Weather, ECL, and Traffic), the performances of DeepRRTime and Deep are very similar. We skip this experiment for ILI due to the shorter forecast horizons used for this dataset.
26

---

Gooad
DegpThee
Desphtttte
Pash
Suatplle custiber 101
Sumple cusminer
Sumple sumdder 28:
Sassale susslls 500
Sumple sumiin 801
(ASE-IVE) 0.3262
DISE 2.2154
MSER22T 0,3399
MSRE 0.1774
MARE,D/E 0.3005
MSSCDIIITT 0.3004
MSSEDILRTT 6.772 DESE1PaHSTTT
AISEDRUITT: 16:0
MSEDDRITT: 0.3011
MSEE 0.30m
MSD PTSTT
S0SEOP (
200
22
1.
3.5
000
300
110
320
100
110
300
200
010
-455
-35
-306
-375
-1.
-100
-18
110
200
300
22
210
200
120
3.4
240
100
18
-00
-1.0
3.70
Wh
2:7
Vmw
25-
2.00
3.0
2.2
100
1100
100
170
3.3
35.
160
26
360
700
100
100
0/25
4/67-
3.0
02-
3.8
-05
2.70
2.00
3.70
3.00
3.50
100
340
300
200
200
600
120
200
320
220
110
Figure 9: Forccasting Plots on Exchange: We visually compare between DecpTime, DecpRRTime and PSt The rows of a column denote the 8 different time-variates and all plots within the same column are taken from the same time-period. The sample-IDs for the visualisation are randomly sampled and indicated at the top of each column along with the mean-squared error (MSE) of forecasts for each method. More specifically, the MSE at the top of each column the MSE averaged over all 8 plots in that column. several cases, observe that the DeepRRTime
27


Reviews:
[{"review_id": "o9Th2FoHnb", "reviewer": "Reviewer_kRWA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The following significant strong aspects of this study are highlighted:\n1. The authors showcase in several experiments the **potential of time-index models as an alternative to historical-value models** that are often prone to overfitting/poor generalization in time series forecasting.\n2. The presentation of related work and contributions and the methodology part are **clearly written** and easy to follow.\n3. The main forecasting experiments are extended with setups **incorporating missing values and different train portions**, showing some interesting ways of testing models’ generalizability and robustness in forecasting beyond standard experimental setups.\n\nThe following weaknesses associated with this work can be easily identified:\n- **Incremental Methodological Contribution:** **[W1]** The technical contribution of this work remains limited to extending DeepTime with a regularization term designed for part of ridge regression. It is unclear whether this term could be generally used for other time series forecasting architectures (mainly time-index). **[W2]** Additionally, since ridge regression is a widely used technique in machine learning, the related work section should include references to studies proposing regularization techniques (for standardizing and balancing the correlation of the basis elements or beyond).\n- **Issues with Experimental Evaluation:** **[W3]** Several more diverse historical-value state-of-the-art models for time series forecasting could be added as baselines in performance comparisons [1,2,3]. **[W4]** In the experiment with missing points, it is interesting that historical-value models are also evaluated in a fair way (using some simple interpolation method for missing points). **[W5]** Additionally, several methods have been proposed for modeling irregularly sampled / with missing values time series, such as Neural ODEs, RNN-based, and others [4,5], that could potentially outcompete most standard models in the missing points setup. **[W6]** There is also a discrepancy in the number of runs used for PatchTST and the DeepRRTime. The authors mention performing 10 runs for DeepRRTime, and the models they need to re-evaluate. However, the obtained results for PatchTST (for all datasets except the Exchange dataset) from the original paper are averaged over 5 runs. \n- **Studies on Computational Cost/Visualizations:** **[W7]** The authors mention that “time-index models are generally less computationally expensive, especially at inference time”, but if not mistaken, this is not accompanied by any time/memory cost comparison between the considered models. **[W8]** Plots on predictions are also missing, which could bring some light to the cases the the regularized DeepTime surpasses the baselines.\n\n1. Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M. (2023). itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625.\n2. Zhang, Y., & Yan, J. (2023, May). Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The eleventh international conference on learning representations.\n3. Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., & Long, M. (2022). Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186.\n4. Rubanova, Y., Chen, R. T., & Duvenaud, D. K. (2019). Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems, 32.\n5. Yalavarthi, V. K., Madhusudhanan, K., Scholz, R., Ahmed, N., Burchert, J., Jawed, S., ... & Schmidt-Thieme, L. (2024, March). GraFITi: Graphs for Forecasting Irregularly Sampled Time Series. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 15, pp. 16255-16263).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work focuses on **time-index models for time series forecasting**, which have recently shown quite competitive performance compared with the common historical-value models. More specifically, the authors propose a **regularization term** for the recently proposed DeepTime framework, whose time-index basis is built upon an implicit neural representation, and a meta-learning module generates predictions with ridge regression. The proposed regularization term forces the elements to be less mutually correlated *to boost the robustness of the ridge regression part*. Experimental evaluation on forecasting with benchmark datasets, especially with *synthetically generated missing values* in the loopback windows, shows improved performance for the proposed method compared to baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following aspects could significantly improve the quality of this work and the impact of the contribution:\n1. Based on **[W1]**, the authors should better justify the impact of the technical contribution beyond the chosen DeepTime architecture. They should consider updating the part of related work with some references to relevant topics studied in ridge regression to address **[W2]**.\n2. Authors should consider adding additional baselines for historical-value methods **[W3]** and for the experiment with missing points **[W4, W5]** to enable more fair performance comparisons. Based on **[W6]**, since the experimental evaluation (e.g., in the number of runs) followed by the authors differs from other baselines, they should evaluate performance with the same protocol for all methods.\n3. It is important that the computational cost of the meta-learning framework is captured along with other time/memory cost comparisons with the baselines **[W7]**. Plots on the predictions for the several setups could further enable qualitative comparisons of methods **[W8]**.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mvN87jXp6m", "reviewer": "Reviewer_V2ZN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n\nS1. The paper is clearly written and easy to follow, with a well-defined scope of contributions. Figure 1, in particular, is an excellent visual aid that enhances understanding. Moreover, the advantages of time-indexed models over traditional historical-value models for time series are clearly articulated and well-supported.\n\nS2. Although the improvement over DeepTime may seem incremental at first glance, the focus on scenarios with limited number of samples and missing values in the look-back window address challenges frequently encountered in real-world applications. The experiments effectively validate the proposed regularization approach, reinforcing the claims made in the introduction and demonstrating its practical relevance.\n\n\n### Weaknesses\n\nW1. The authors compare their approach to Gaussian and TimeFlow in Table 1, but I would have liked to see these methods included in the comparisons for Table 2 and Table 3 as well. If I am not mistaken, both of these time-index models are also capable of handling forecasting tasks with missing values in the look-back window, so a broader comparison would be valuable for evaluating performance under these conditions.\n\nW2. It would be beneficial to include some forecasting plots in the appendix, such as comparisons between DeepRRTime, DeepTime, GP, TimeFlow, and PatchTST. Visualization of these results would provide a clearer understanding of how the models perform in practice.\n\nW3. The inclusion of the Martingale baseline is nice to illustrate the complexity of the forecasting problem. However, it would be helpful to also include a simple repeat baseline where the look-back window is repeated over the forecast horizon. For example, for a horizon of 336, simply repeat the last 336 values. Adding this baseline could provide valuable additional insight into the difficulty of the forecasting task.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the challenge of time-indexed models for time series forecasting and introduces DeepRRTime, an enhanced version of the existing DeepTime model. DeepTime leverages Implicit Neural Representations (INRs) as a latent basis, which are then combined with a Ridge regressor through a meta-learning optimization. This combination enables the model to adapt to different look-back windows effectively.\n\nDeepRRTime improves upon DeepTime by incorporating covariance regularization over the INR basis representation (before the ridge regressor). Specifically, it adds a term to the loss function that encourages the off-diagonal elements of the covariance matrix to approach zero while pushing the diagonal elements toward one. The rationale is that Ridge regression can be more robust when operating on a basis of linearly uncorrelated vectors with unit norms.\n\nExperiments conducted on standard LTSF datasets demonstrate that DeepRRTime outperforms the original framework in scenarios with limited sample sizes and when the look-back window contains missing data. In traditional settings, where such challenges are absent, both methods exhibit comparable performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the section on weaknesses. I believe that the suggested improvements could significantly enhance the quality of the paper.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EngBBkckme", "reviewer": "Reviewer_ZUzk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths and Weaknesses\n+ Paper and method is easy to understand\n+ The regularization term is well motivated and its usefulness is intuitive\n+ The experiments are extensive: Normal setting, low data regime, different test frequencies etc\n\n- The results are to me not really promising: In Table 6, the enhancement do not look that large between DeepTime and DeepRRTime\n- DeepTime and DeepRRTime are most of the time beaten by PatchTST in Table1. Why use Time-Indices at all? \n- The computationally inexpensiveness has to be showed somehow. I mean, for Equation (4) you would have to compute ZZ^T which is naively L^2*d and then also solve the equation system\n\n# Are all Claims Proven?\n- To some extent: Looking at Table 6: That DeepRRTime beats normal DeepTime,especially for the smaller horizons, may not be significant as the stdvs indicate\n- Same problem I have with Table 2, the 50% missing experiment\n- In Table 4 (and the picture below) it is more convincing.\n\nAs I have to make a binary decision, I would, at the current stage, slightly lean towards a \"No\", but I am willing to change my decision depending on the outcome of my requested changes\n\n# How interesting is it to the Audience?\n\n- I would say it is somehow interesting. It is an easy to implement\n  regularization term which seems to help (to a slight extent / in the low\n  training-data regime). So especially when having low amount of training data,\n  one may would try to test DeepRRTime. However that Deep(RR)Time seems not be\n  fully competitive with PatchTST limits the importance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors build on the DeepTime model. DeepTime works by having\na neural network to infer hidden predictions at arbitrary historic and future\ntime-points and then apply ridge-regression on the hidden-representations of the\nlookback window. Then,  the neural network and the ridge regression top head can\nbe used to forecast for the future time-points.\n\nThe particular extension of this paper lies in a regularization term. The\nauthors propose to not only fit the ridge regression on the forecasting task,\nbut also on de-corellating the hidden representations. In detail, they add an\nauxillary loss that pushes the centralized covariance matrix of the hidden\nrepresentations towards the identity matrix.\n\nThe authors evaluate their method DeepRRTime on \"standard\" time-series\nforecasting, forecasting with missing values, in the small training data regime\nand in a setting where the frequency in the test data differs from the training\ndata.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Important:\n\n- Present empirical evidence, that your methods is indeed always very fast\n- I would like to see standard deviations for Table 2, so basically Figure 2 also for the other datasets, to see if your method is really superior when dealing with missing values.\n- Why should I use Time-Indices at all over, say, PatchTST? Can you compare it against PatchTST (or another regular sequential model) on the 50% missing value setting? in PatchTST, you could for example simply 0 the missing values or linearly interpolate them first. Or you compare against that on the 10% training data setting. Just an indication, that I should use Time-Indices at all in some settings.\n\n# Minor:\n\n- I would move theorem 2 into the main text, somewhere around section 4.2. When I first read the main text, I had the issue that 4 tells me \"We want to maximize the smallest and minimize the largest eigenvalue.\" And then at the beginning of 4.2 you say \"We do pushing G_theta against the Identity instead because it worked better.\" I was confused and asking myself why then even talk about the maximization and minimization of the eigenvalues at all? However, Theorem 2 and the discussion around it give the connection to understand that. Thus, they are crucial and should go into the main text. \"Similarly, the\nlargest eigenvalue of Gθ also cannot be very large when LCov(θ) is small.\" Can you elaborate on that in detail, as done in (10)-(12)?\n- I do not like calling the Z matrix the \"basis\". A basis of R^d is a very well-established term for a sequence of vectors in R^d, and its not how you use it here. I especially stumbled upon the first sentence in section 3.1 with respect to this regard. The part \"Z \\in R^D represents the values of the D-dimensional base\" sounds just wrong as a D-dimensional basis (in the sense of basis as always used in linear-algebra) would need D different vectors.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ae2gLXkseh", "reviewer": "Reviewer_BCmj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong Aspects\n\n1. The introduction of a simple yet effective regularization term to improve the robustness of ridge regression is a strong innovation. The approach is theoretically grounded and shows clear benefits in practical applications.\n2. The submission provides comprehensive experimental results across six real-world datasets, demonstrating the effectiveness of DeepRRTime. The improvements in performance over both DeepTime and other time-index models are consistent and significant, particularly on the challenging Exchange dataset.\n3. The robustness tests, including scenarios with missing data, reduced training datasets, and test-time interpolation, add depth to the evaluation and illustrate the versatility of the proposed method.\n4. The regularization technique is easy to implement and integrates smoothly with the existing DeepTime framework. This practicality is a valuable asset for deploying the model in real-world settings, where complex modifications can hinder adoption.\n5. The work is theoretically well-supported, leveraging insights from linear regression theory to justify the regularization. This grounding in theory enhances the credibility of the approach and explains why the improvements are observed.\n\nWeaker Elements\n\n1. The paper acknowledges that hyperparameter tuning was not extensively conducted, which might mean the reported performance does not fully capture the potential of DeepRRTime. Future work could focus on a more thorough exploration of parameters like the regularization coefficient $\\lambda_2$ and the lookback multiplier $\\mu$, perhaps integrating automated hyperparameter tuning methods to optimize performance further.\n2. While the six datasets used are diverse, the evaluation could benefit from additional datasets that exhibit different characteristics (e.g., datasets with higher frequency seasonal patterns, more non-stationary behaviors, or data from different application domains like finance or healthcare). Expanding the range of datasets would strengthen the claims of robustness and generalizability.\n3. The reliance on manually tuned parameters, such as $\\lambda_2$ and $\\mu$, could limit the scalability of the model. Automated techniques for dynamic parameter adjustment during training might improve usability in scenarios where these parameters are hard to determine or need to adapt based on the data.\n4. Although the regularization is described as computationally inexpensive, the need for fine-tuning across multiple datasets could introduce additional computational overhead, particularly when large-scale datasets or complex multivariate settings are involved. The authors might consider discussing strategies to mitigate such overheads, such as batch optimization or efficient regularization scheduling.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents DeepRRTime, a novel extension to the DeepTime framework for time-series forecasting, incorporating a covariance regularization technique.\n1. The authors propose a simple, computationally inexpensive regularizer that encourages the time-indexed basis learned by the model to be more unit-standardized and less mutually correlated. This regularization is designed to improve the conditioning of the ridge regression step, leading to more robust forecasts.\n2. DeepRRTime is evaluated on six real-world multivariate time-series benchmarks and is shown to outperform the original DeepTime and other time-index models. It demonstrates state-of-the-art results, especially on the challenging Exchange dataset, where previous models struggled due to its low signal-to-noise ratio and non-stationarity.\n3. DeepRRTime remains robust even when up to 90% of the lookback data is missing, outperforming DeepTime and other models.\n4. The regularizer helps maintain generalization when training data is limited, reducing the performance gap between reduced and full datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see Weaknesses", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While 2 reviewers leaning to Accept and 2 leaning to Reject, all reviewers answered to both questions required by TMLR policy for acceptance with "yes". After reading through the latest revision of the manuscript, the reviews and rebuttal, in my opinion the paper meets the criteria for acceptance. I find the novelty quite limited, but the theoretical insight worthy of the attention of the readers of TMLR.

I would ask the authors to extend the text by specifically acknowledging that there are other type of methods for sporadically observed data which was not tested in this work, like Neural ODEs. This was mentioned by the reviewers, and the authors decided that this comparison would be outside of the scope of the present work. I can live with this decision but I find it essential to mention in the manuscript the existence of these methods.
Decision: Accept (poster)