DeepTime: Deep Time-index Meta-learning for Non-stationary Time-series Forecasting
Authors: Gerald Woo
Chenghao Liu
Doyen Sahoo
Akshat Kumar
Steven Hoi
Venue: Submitted to ICLR 2023 (2023)
Keywords: time-series
forecasting
deep learning
implicit neural representation
meta-learning
time-index
non-stationary

Abstract: Advances in I.T. infrastructure has led to the collection of longer sequences of time-series. Such sequences are typically non-stationary, exhibiting distribution shifts over time -- a challenging scenario for the forecasting task, due to the problems of covariate shift, and conditional distribution shift. In this paper, we show that deep time-index models possess strong synergies with a meta-learning formulation of forecasting, displaying significant advantages over existing neural forecasting methods in tackling the problems arising from non-stationarity. These advantages include having a stronger smoothness prior, avoiding the problem of covariate shift, and having better sample efficiency. To this end, we propose DeepTime, a deep time-index model trained via meta-learning. Extensive experiments on real-world datasets in the long sequence time-series forecasting setting demonstrate that our approach achieves competitive results with state-of-the-art methods, and is highly efficient. Code is attached as supplementary material, and will be publicly released.

Under review as a conference paper at ICLR 2023
DEEPTIME: DEEP TIME-INDEX META-LEARNING FOR NON-STATIONARI TIME-SERIES FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
Advances in I.T. infrastructure has led the collection of longer sequences of time-series. Such sequences are typically non-stationary, exhibiting distribution shifts over time a challenging scenario for the forecasting task, due to the prob- lems of covariate shift, and conditional distribution shift. In this paper, we show that deep time-index models possess strong synergies with a meta-learning formu- lation of forecasting, displaying significant advantages over existing neural fore- casting methods in tackling the problems arising from non-stationarity. These advantages include having stronger smoothness prior, avoiding the problem of covariate shift, and having better sample efficiency. To this end, we propose Deep- Time, deep time-index model trained via meta-learning. Extensive experiments on real-world datasets in the long sequence time-series forecasting setting demon- strate that our approach achieves competitive results with state-of-the-an methods, and is highly efficient. Code is attached as supplementary material, and will be publicly released.
INTRODUCTION
Time-series forecasting has important applications across business and scientific domains, such as demand forecasting (Carbonneau et al. 2008), capacity planning and management (Kim) 2003). and anomaly detection (Laptev al.) (017). With the advances of I.T. infrastructure, time-series are collected over longer durations, and at higher sampling frequency. This has led to time-series spanning tens-of-thousands millions of time steps, on which we would like to perform forecasting on. Such datasets face the unique challenge of non-stationarity, where long sequences face distribution shifts over time, due factors like concept drift. This has practical implications on forecasting models, which face degradation in performance at test time (Kim et al. 2021 due covariate shift, and conditional distribution shift (see Appendix B for formal detinitions).
Table 1: Time-index models defined to be models whose predictions, Φt. are purely func- tions of the current time-index features, Tr, e.g. relative time-index (1, 2, 3, ...,, datetime fea- tures (minute-of-hourr week-of-day, etc.). Historical-value models, whose predictions of future time step(s), Dr+1, are explicit functions of past observations, (yr- ....:..)),, and optionally covariates, (Ze+1, 2t.1,.1..... which can include exogenous time-series or even datetime features.
Time-index Models
Historical-value Models
ÿ = f(T))
Út+1
f(y V:-1, Zt+1, .t....)
E.g.: DeepTime, Prophet, Gaussian process
E.g.: N-HiTS, Autoformer, DeepAR
In this work, we posit that deep time-index models exhibit strong synergies with a meta-learning formulation to tackle the problem of non-stationary forecasting, whereas existing neural forecasting methods, which are historical-valun models, are unable to take full advantage this formulation, and are still susceptible to the problem of covariate shift. In the following, models and their deep counterparts, highlighting how simple deep time-index models are unable to perform forecasting (i.e. extrapolate from historical training data). Yet, endowing them with meta-learning formulation solves this problem. Thereafter, we demonstrate the advantages of deep time-index meta-learning for non-stationary forecasting and how they alleviate the issues faced by historical-value models, which are namely: distribution shift, (ii) they avoid the problem of covariate shift, (iii) have stronger sample efficiency, and (iv) that ime-index models have a stronger smoothness prior.
discuss indexindex
(i) meta-learning is an effective solution for conditional

---

Under review as a conference paper at ICLR 2023
Time
Covariate Shift:
Pi Ot-
P2 P2
P3 Ve.Vv.---1o.
Conditional
P1 (++11le)...)
P2 P2
Pi P((eet1bee.....
Distrihution Shift:
Figure 1: Non-stationary time-series degrade model performance due to covariate shifts and con- ditional distributional shifts. Such behaviors can be modelled as locally stationary processes, by which contiguous segments are assumed to be stationary. Meta-learning takes advantage of this sumption to adapt to these locally stationary distributions. Yet, existing methods which model the conditional distribution, p(gr+1 (9t....). are still susceptible to covariate shifts since the meta model takes time-series values as input.
Reconstruction
Ground Truth
144
Forecast
1.2
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0
1000
200
300
400
500
600
700
1:00
200
300
4.00
500
6.0
700
(a) Without Meta-learning
(b) With Meta-learning (DeepTime)
Figure 2: (a) A naive deep time-index model. We visualize reconstruction of the historical training data, as well as the forecasts. As can be seen, it overfits to historical data and is unable to extrapolate. our This model corresponds (+Local) Tf our ablations. (b) DeepTime, our proposed approach, trained via a meta-learning formulation, successfully learns the appropriate function representation and is able to extrapolate. Visualized here is the last variable of the ETTm2 dataset.
Deep Time-index Models On the one hand, classical time-index methods (Taylor & Letham 2018 Corani et al. 2021 Ord et al. 2017) rely on predefined parametric representation functions ye f(T)) Et, Where E, represents idiosyncratic changes not accounted for by the model and I could be some polynomial function to represent trend, Fourier series to represent seasonality, or a composition of seasonal-tres -rrnnn components. While these functions are simple and easy to learn, the choice of representation function requires strong domain expertise
computationally heavy cross-validation. Furthermore, predefining the representation function is a strong assumption and may fail under distribution shifts. On the other hand, while deep time-index models (letting be a deep neural network) present a deceptively clear path to approximate the representation function in data-driven manner, deep time-index models too expressive. Trained via straightforward supervised learning on historical values without any inductive bias, they are unable to extrapolate to future time steps (visualized in Figure and meta-learning formulation is required to do this formulation has the added benefit of handling non- non-stationary forecasting.
Advantages of Deep Time-index Meta-learning Firstly, distribution shift of input statistics sharply degrade the prediction accuracy of deep learning models (Nado et al.) 2020). Historical- value models, which take past observations as imput, suffer from this an effect of covariate shift. Time-index models easily sidestep this problem since they take time-index features input. Next, meta-learning is an effective solution to tackle the problem of conditional distribution shift -nerby time steps are assumed to follow a locally stationary distribution (Dahlhaus) 2012) Vog( 2012) (see Figure [], considered to be a task. The base learner adapts to this locally stationary distribution, while the meta learner generalizes across various task distributions. In principle, historical-var models are able to take advantage the meta-learning formulation, however, they still suffer from the problem of covariate shift and sample efficiency issues. Time-index models are also able to achieve greater sample efficiency the meta-learning formulation. Like many existing state-of- the-art forecasting approaches, time-index models direct multi-step (DMS) approaches[] For a lookback window of length L and forecast horizon of length H, historical-value DMS model requires N L H time steps construct support set of size N. whereas time-index model only requires N time steps. Not only does this marked increase in sample efficiency mean that
'DSS methods directly predict the entire forecast horizon, and contrasted with iterative multi-step (IMS) methods. Further discussion DMS/IMS, and taxonomy of forecasting methods in Appendix C
2

---

Under review as a conference paper at ICLR 2023
time-index models can achieve improved task generalization enror bound (Appendix Q they also better able to adhere to the assumption of locally stationary distribution since using more time steps leads to the risk of non-stationary support set. Finally, time-index models have a stronger smoothness prior (Bengio et al. 2013), i.e. 1 R 1' = Ti R Tp = f(T:) R f(Te). whereas the complicated parameterization of historical-value models provide no such inductive biases.
To this end, we propose Deep'Time, a deep time-index model, endowed with a meta-learning formu- lation. We leverage implicit neural representations (INR) (Sitzmann et al.] 2020b) as our choice of deep time-index models, and also introduce novel concatenated Fourier features layer to efficiently learn high frequency patterns. The meta-learning formulation is instantiated as a closed-form ridge regressor (Bertinetto et al. (219)) to efficiently tackle the meta-learning formulation. Deep Time manages to overcome the limitations of naive deep time-index model by learning the appropriate inductive biases for extrapolation over the forecast horizon. It is also more effective than existing iistorical-value methods at non-stationary time-series forecasting by learning global meta model shared across tasks and performs adaptation on a locally stationary task, and also sidesteps the covariate shift problem. To summarize, the key contributions of our work are as follows:
We introduce a novel forecasting as meta-learning framework for deep time-index models, enabling them learn the appropriate representation function and tackle the problem of non- stationary forecasting. This is distinct from existing work which leverages meta-learning historical-valu models for adapting to new time-series datasets, where tasks are defined to be the entire time-series (Grazzi et al. (02T).
We propose TeepTime, leveraging an INR with concatenated Fourier features and closed- form ridge regressor achieve highly efficient forecasting model.
We conduct extensive experiments on the long sequence time-series forecasting (LSTF) datasets, demonstrating DeepTime to be extremely competitive. We perform ablation stud- ies better understand the contribution of each component of DeepTTime, and finally show that it is highly efficient in terms of runtime and memory.
2 DEEPTIME
Problem Formulation In time-series forecasting, we consider time-series dataset (2,,,2,.... DT), where y: E Imm is the m-dimension observation at time L. Given look- back window YI-La = : 9t-1 E R4xm of length I, the goal of forecasting to :1+ E "m The goal is to learn a time-index model. f R + RTm ff : Tr + Dr. where Te is a time-index feature, to quickly adapt to observations in the lookback window, (Ti-La, Yı-L:)), by minimizing reconstruction loss L : Rm x R + R. Then, we can query it over the forecast horizon to obtain forecasts, YE = [(T
construct point forecast over horizon of length H, Yu++H
In the following, we first describe our forecasting as meta-learning framework time-index mod- els. We emphasize that this formulation falls within the standard time-series forecasting problem and requires no extra information. Next, we further elaborate on our proposed model architecture, and how it uses differentiable closed-form ridge regression module to efficiently tackle forecasting as meta- letanng problem. Psuedocode of Deep is available in Appendix
2.1 FORECASTING AS META-LEARNING
In time-index meta-learning, each lookback window and forecast horizon pair, (Y,-L.t- L:t, Ye(+)) is a task. This task yields a single support and query set, which are the lookback win- dow and forecast horizon respectively. (Tete: Vtts), is an input-output sample, i.e. DS ((Te, yt),... 11....,(T+++111) (1++ 1 1++H-1)}.
Each time coordinate and time-series value pair,
= {(TT-LL (t-.-,.tt)........ (T-11, 91-1)). De itL
where Tt- = -+#-- is a [0, nooomaallee time-index. The time-index model, f, is parameterized by & and 6, the meta and base parameters respectively, and the bi-level optimization problem can be formalized as:
T T- H-1
o= arg min
Σ
Σ
C(((T+); Φ). yets)
(1)
$
t=++1
1=0
s.t. 0- = arg min
C( (145)
(2)
3

---

Under review as a conference paper at ICLR 2023
Task
Task M
Y'
Ridge Regressor
Time
Ridge Regressor
W
ang arg (t)
[as (ao
@
Random Fourter Peatures
46-1
Lookback
Forecasting as Meta-lcarning
Deep Time-index Model Architecture
Figure 3: Illustration of DeepTime. A time-series dataset can be split into M tasks as given in the problem formulation. For given task, the lookback window represents the support set, and the forecast horizon represents the query set. Io represents the meta model associated with the meta parameters. Ø is shared between the lookback window and forecast horizon. Inputs to 9é are not normalized due to notation constraints on this illustration. The ridge regressor performs the inner loop optimization, while outer loop optimization is performed over samples from the horizon. As illustrated in Figure 3b DeepTime has a simple overall architecture, comprising of random Fourier features layer, an MLP, and a ridge regressor.
Here, the outer summation in Equation ( over index represents each lookback-horizon window, corresponding to each task in meta-lcarning, and the inner summation over index j represents each sample in the query set, or equivalently, each time step in the forecast horizon. The summation in Equation (2) over index j represents each sample in the support set, or each time step in the lookback window. This is illustrated Figure Ba
To understand how our meta-lcarning formulation helps to learn an appropriate function represen- tation from data, we examine how the meta-learning process performs restriction on hypothe- sis class of the model f. The original hypothesis class of our model, or function representa- tion, HINR {f(r; A, Φ) l E Φ E $). is too large and provides no guarantees that train- ing the lookback window leads to good extrapolation. The meta-learning formulation allows hypothesis class of the representation function, from the space of all K-layered INRs, the space of K K-layered INRs conditioned the optimal meta parameters, HDeepTime = {f(T; 0, 6*) I E θ}. where the optimal meta parameters, ø", is the minimizer of forecasting loss (as specified in Equation (T). Given this hypothesis class, local adaptation is performed over Tiee given the lookback window, which is assumed to come from a locally stationary distribution, resolving the issue of non-stationarity.
DeepTime to restrict
2.2 MODEL ARCHITECTURE
Implicit Neural Representations The class of deep models which map coordinates to the value at that coordinate using a stack of multi-layer perceptrons (MLPs) known as INRs (Sitzmann al.) Tancik et al. 2020). We make use a of them as they are a natural fit for time-index models, map a time-index the value of the time-series that time-index. K-layered, ReLU (Nair &
2020b
Hinton 2010 INR is function fo R Rm which has the following form:
(0)
(k+1) = max(0, (K) (())), == K
fo(T) = (KK) + (K)
where T E RC is the time-index. Note that e = for our proposed approach as specified in Sec- the notation
tion 2.1 but are included. Tancik et al. (2020) introduced random Fourier features layer which allows INRs to fit to high frequency functions, by modifying 20) = 7TT) = [sin (2 BT), cos Br))" where each entry in B d/ is sampled from (O, o²) with is the hidden dimension size of the INR and o2 is the scale hyperparameter. [; is row-wise stacking operation.
IRC to allow for generalization cases where datetime features

---

Under review as a conference paper at ICLR 2023
Concatenated Fourier Features While the random Fourier features layer endows INRs with the ability to learn high frequency patterns, one major drawback is the need to perform a hyperparam- eter sweep for each task and dataset to avoid over underfitting. We overcome this limitation with a simple scheme of concatenating multiple Fourier basis functions with diverse scale param- eters, i.e. 7(T) = [sin(2 B1r), cos(2 (IT), ..... (2 B ; cos (2T: cos(2xBTT)) Bs where elements in Ba E Rd//2xc are sampled from (0,, and W E RdxSd
We perform an analysis in Section 3.3 and show that the performance of our proposed Concatenated Fourier Features (CFF) does not significantly deviate from the setting with the optimal scale parameter obtained from hyperparameter sweep.
Differentiable Closed-form Solvers One key aspect to tackling forecasting as a meta-learning problem is efficiency. opiimization-basee meta-learning approaches originally perform an expen- sive bi-level optimization procedure on the entire neural network model by backpropagating through inner gradient steps (Ravi & Larochelle 2017 Finn et al. (017). Since each forecast is now treated as an inner loop optimization problem, it needs to be sufficinnlly fast to be competitive with com- peting methods. We achieve this by restricting the inner loop optimization to only apply to the last layer of the INR. As a result, we can perform the inner loop optimization on this linear layer using the closed-form solution of ridge regressor for the our formulation is general, and any differentiable solver can be used instead (Bertinetto et al., 2019). This means that for a K K-laaered model, ø = {W(),
of mean squared error loss. We note that
(0), ((0) b (K-)1) b(K 1), }} are the meta parameters and 0 = { (W( } are the base parameters, following notation from Equation (3) Then let 9s : IR + d be the meta learner where (T(T) = (K)) For task with the corresponding bookackkhorizon pair, (Y,-L.t. L:t. Yc.+H), the support set features obtained from the meta learner is denoted Zi-L: = [go (T) ...... Io :o (T(-))]" E R. where [:: is a column- concatenation operation. The inner loop thus solves the optimization problem:
w!() arg min 1 W- Y 1 W
= (Z) L.t ZI-La + AI)
27_ LI
(4)
Now, let Zuct = [go(TT); :9o
E Hxd be the query set features. Then, our predictions are YE ZE:+# W (K)- This closed-form solution is differentiable, which en- ables gradient updates on the parameters of the meta learner, ø. A bias term can be included for the closed-form ridge regressor by appending scalar 1 to the feature vector go(T). The
end result of training DeepTime on a dataset is the restricted hypothesis class HDeepTime (9o-(7) (K( (K) E Rdxm). }. This is illustrated in Figure 3b
Some confusion regarding DeepTime's 's categorization as a time-index model may arise from the above simplified equation for predictions, since forecasts are now a function the lookback window due to the closed-form solution of ((K)* However, we highlight that DeepTTmee is meta-learning algorithm on top of deep lime-index model - it comprises a learning algorithm, A: H. x R 1. + H. specified in Equation (2) (the inner loop optimization step), and the deep time-index model itself, f H DeepTimee Thus, forecasts are of the form, DL+h A(f. Y, (T+b)) and as can be seen, while the inner loop optimization step function of past observations, the adapted time-index model it yields is purely function of time-index features. Further discussion can be found in Appendix D
3 EXPERIMENTS
We evaluate DeepTime on both synthetic datasets, and a variety of real-world data. We ask the following questions: (i) Is DeepTTime, trained family of functions following the same parametric form, able to perform extrapolation on unseen functions? (ii) How does Deep'Time compare to other forecasting models on real-world data? (iii) What are the key contributing factors to the good performance of DeepTime?
3.1 EXPERIMENTS ON SYNTHETIC DATA
We first consider Deep Time's ability to extrapolate the following functions specified by some parametric form: (i) the family of linear functions, y = ar + b, (ii) the family cubic functions, y ax +bx? +cr and (iii) sums of sinusoids, E, Aj spp( Parameters of the functions
5

---

Under review as a conference paper at ICLR 2023
Ground Forecast
15-
0-
50
100
152
200
220
300
200
620
50
120
250
200
250
300
250
630
20
200
330
200
250
200
202
400
(a) Linear
(b) Cubic
(c) Sum of Sinusoids
Figure 4: Predictions of Deep on three unseen functions for each function class. The orange line represents the split between lookback window and forecast horizon.
Table 2: Multivariate forecasting benchmark on long sequence time-series forecasting. Best results are highlighted in bold, and second best results are underlined.
Methods
DeepTime
NS Transfremer
N-HITS
FTSfomer
PRDiiemer
Autoformee
Informer
LagTran
CIP
Metnes
MSS
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAB
MSE
MAE
96
0.166
0.255
0.192
0.274
0.176
0.255
0.189
0.280
0.203
0.287
0.255
0.339
0.365
0.453
0.768
0.642
0,442
0.422
192
0.225
BL302
0.280
0:330
0245
0.385
0.253
0.119
0.769
0.328
0.281
0.340
0.533
0.563
0.989
0.757
0.605
0.505
336
0.277
0.336
0.334
0.361
0.255
0.366
0.314
0.357
0.325
0.366
0.339
0.372
1.363
0.887
1.334
0.872
0,731
0,569
720
0.383
0.409
0.417
0.133
0.401
0.426
0.414
0.413
0.421
0.415
0.422
0.419
3,379
1.388
3.048
1.328
0.959
0.609
96
0.137
0.238
0.169
0.273
0.147
0.249
0.187
0.304
0.183
0.297
0.201
0.317
0.274
0.368
0.258
0.357
0.513
0,588
ECL.
192
0.152
0.252
0.182
0.286
0167
0269
0.199
0.315
0.195
0.306
0.222
0.354
0.296
0.386
0.266
0.368
0.505
0,543
336
0.166
0.268
0.200
0.304
0.86
0.200
02112
0.329
0.212
0.313
0.231
0.338
0.300
0.394
0.280
0.380
0.612
0.614
0.201
0,340
0,345
0,343
0.254
0,361
0.373
0.439
0.283
0,376
0.652
0.633
Exchange
0.151
0.081
0.205
0.111
0.219
0.204
0.276
0.323
1.204
0.851
0,812
0,267
0.314
0,421
0.443
0.02%
0.426
0.161
0.524
1,622
0.447
0.556
0.663
1.092
0.769
0388
0.723
0.774
1,000
0.800
1.447
0.941
2.478
1.310
1.941
1.127
1.135
0.810
0.390
0.275
0.612
04422
02883
0.677
0,392
0,562
0.349
0.13
0.388
0.799
0.391
0.684
0.384
1.112
0.665
Trallie
0,402
0.278
0.633
0.99
0.399
0.562
0.365
0.66
0.382
0.69
0.379
0,390
1.133
0.671
336
0.415
0.288
0,618
0:788
0.445
0313
0.22
0,396
0.570
0.323
0.622
0.337
0.777
0.420
0.733
0.408
1.274
0.723
720
0.449
0.307
0.655
0.555
0.599
0.353
0.632
0.396
0.596
0.368
0.663
0.408
0.864
0.472
0,717
0.396
1,280
0,719
Weather
96
0.166
0.221
0.175
0.225
0.158
0.195
0.197
0.281
0.217
0.296
0.266
0.335
0.300
0.384
0.458
0.450
0,395
0.356
192
0.207
0.261
0.245
0.285
0211
0.247
0.237
0.112
0.276
0.356
0.307
0.367
0.598
0:544
0.658
0.589
0.450
0.398
336
0:251
0.296
0.321
0.338
0274
0.300
0.298
0.353
0.339
0,360
0.359
0.359
0.578
0.523
0.797
0.652
0.505
0.460
720
0.301
0.338
0.414
0.410
0.351
03355
0.352
0.388
0.403
0.428
0.419
0.419
1.059
0.741
0.889
0.675
0.498
0.450
24
2.425
1.086
2.294
0.915
1862
0.869
2.527
1.020
2.203
0.963
3.483
1.267
5.764
1.677
4,480
1.454
2.331
1.036
36
2.231
10088
1,825
0,548
2.071
0.969
2.615
1.007
2.272
0.976
3.103
1.148
4.755
1.467
4.799
1.467
2.107
1,002
2.230
1016
2.010
0.900
7.46
1.422
2.359
0.972
2.009
0.981
2.669
1.085
4.763
466
4,800
166
2.961
1.180
2.145
0.985
2.178
0.963
2.560
1.073
2.487
1.016
2.345
1.061
2.770
1.125
5.264
5.276
3.108
1,214
(i.e. a, b, d, Wj. (2)) are sampled randomly (further details in Appendix F to construct distinct tasks. total of 400 time steps are sampled, with a lookback window length of 200 and forecast horizon of 200. digoosrademssssratts that Deep'Time is able to perform extrapolation on unseen test functions/tasks after being trained via our meta- meta-lcarning formulation. It demonstrates an ability to approximate and adapt, based on the lookback window, linear and cubic polynomials, and even sums of sinusoids. Next, we evaluate Deep'Tmme on real-world datasets, against state-of-the-aa baselines.
3.2 EXPERIMENTS ON REAL-WORLD DATA
Experiments are performed on 6 real-world datasets Electricity Transformer Temperature (ETT), Electricity Consuming Load (ECL), Exchange, Traffic, Weather, and Influenza-like IlIness (ILI) with full details in Appendix G We evaluate the performance of our proposed approach using two metrics, the mean squared error (MSE) and mean absolute eтror (MAE) metrics. The datasets split into train, validation, and test sets chronologically, following a 70/10/20 split for all datasets except for ETTm2 which follows a 60/20/20 split, per convention. The univariate benchmark selects the last index of the multivariate dataset as the target variable, following previous work (Xu et al. 202T). Preprocessing on the data is performed by standardization based on train set statisties. Hyperparameter selection is performed on only one value, the lookback length multiplier, L = H=H which decides the length of the lookback window. We search through the values μ [1, 3,5,,, 5,7.9). and select the best value based on the validation loss. Further implementation details on Deep'Time are reported in Appendix [H] and detailed hyperparameters are reported in Appendix [ Reported results for Deep Time are averaged over three runs, and standard deviation is reported in Appendix
Results
We compare DeepTime to the following baselines for the multivariate setting, N-HITS Challu et al. 2022), ETSformer (Woo et al. 2022), Fedformer (Zhou et al. 2022) (we report the best score for each setting from the two variants they present). Autoformer (Xu et al.) (02T). In-
former (Zhou et al. 2021), LogTraas (Li et al. 2019), Non-stationary (NS) Transformer (Liu al.
6

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation study on variants of DeepTime. Starting from the original version, we add (+) or remove (-) some component from Deep'Time. RR stands for the differentiable closed-form ridge regressor, removing it refers to replacing this module with a simple linear layer trained via gradient descent across all training samples (i.e. without meta-learning formulation). Local refers to training an INR from scratch via gradient descent for each lookback window (RR is not used here, and there is no training phase). Datetime refers to datetime features. Further model details can be found in
Appendix 0.1
- RRR
+ Local Datetime
Methods
DeepTime
Datetime
RR
Datetime
+ Local
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.166
0.257
0.226
0.303
3.072
1.345
3.393
1.400
0.251
0.331
0.250
0.327
ETTm2
192
0.225
0.302
0.309
0.362
3.064
1.343
3.269
1.381
0.322
0.371
0.323
0.366
336
0.277
0.336
0.341
0.381
2.920
1.309
3,442
1.401
0.370
0.412
0.367
0.396
720
0.383
0.409
0.453
0.447
2.773
1.273
3,400
1.399
0.443
0.449
0.455
0.461
Table 4: Ablation study on backbone models. Deep'Time refers to our proposed approach, an INR with random Fourier features sampled from range of scales. MLP refers to replacing the random Fourier features with a linear map from input dimension to hidden dimension. SIREN refers to an INR with periodic activations as proposed by Sitzmaann et al. (20206). RNN refers to an autore- gressive recurrent neural network (inputs are the time-series values, yt). All approaches include differentiable closed ridge regressor. Further model details can be found in Appendix 0.2
Methods
DeepTime
MLP
SIREN
RNN
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.166
0.257
0.186
0.284
0.236
0.325
0.233
0.324
I:Tmm2
192
0.225
0.302
0.265
0.338
0.295
0.361
0.275
0.337
336
0.277
0.336
0.316
0.372
0.327
0.386
0.344
0.383
720
0.383
0.409
0.401
0.417
0.438
0.453
0.431
0.432
2022), and Gaussian Process (GP) (Rasmussen 2003). For the univariate setting, we include ad- ditional univariate forecasting models, N-BEATS (Oreshkin et al.) (200)), DeepAR (Salinas et al.] 2020), Prophet (Taylor & Letham 2018). and ARIMA. Baseline results are obtained from the re- spective papers. Table 2 and Table d (in Appendix for space) summarizes the multivariate and univariate forecasting results respectively. DeepTime achieves state-of-the-ar performance on 20 out of 24 settings in MSE, and 17 out of 24 settings in MAE on the multivariate benchmark, and also achieves competitive results on the univariate benchmark despite its simple architecture compared to the baselines comprising complex fully connected architectures and computationally intensive Transformer architectures.
3.3 ABLATION STUDIES
We perform ablation study to understand how various training schemes and input features affect the performance of DeepTime. Table 3 presents these results. First, we observe that our meta- learning formulation is a critical component to the success of DeepTTime. We note that Deep'Time without meta-learning may not be a meaningful baseline since the model outputs are always the same regardless of the input lookback window. Including datetime features helps alleviate this issue, yet we observe that the inclusion of datetime features generally lead to a degradation in performance. In the case of DeepTime, we observed that the inclusion of datetime features lead much lower training loss, but degradation in test performance - this is case of meta-learning memorization (Yin) al. 2020) due to the tasks becoming non-mutually exclusive (Rajendran et al. 2020). Finally, we observe that the meta-learning formulation is indeed superior to training model from scratch for each lookback window.
In Table we perform an ablation study on various backbone architectures, while retaining the differentiable closed-form ridge regressor. We observe a degradation when the random Fourier features layer is removed, due to the spectral bias problem which neural networks face (Rahaman et al. 2019 Tancik et al.) 2020). DeepTime outperforms the SIREN variant of INRs which is consistent with observations INR literature. Finally DeepTime outperforms the RNN variant which is the model proposed in Grazzi et al.) (202T). This is a direct comparison between IMS historical- value models and time-index models, and highlights the benefits of time-index models.
7

---

Under review as a conference paper at ICLR 2023
Table 5: Comparison of CFF against the optimal and pessimal scales obtained from the hyper- parameter sweep. We also calculate the change in performance between CFF and the optimal and pessimal scales, where a positive percentage refers to a CFF underperforming. and negative percent- age refers to CFF outperforming. calculated as % change (MSEcFF - MSE / M
CFF
Optimal Scale (% change)
Pessimal Scale (% change)
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
96
0.166
0.257
0. (1.20 (1.20%)
0.257 (-0.05%)
0.216 (-23.22%)
0.300 (-14.22%)
ET1m2
192
0.225
0.302
0.220 (1.87%)
0.301 (0.25%)
0.275 (-18.36%)
0.340 (-11.25%)
336
0.277
0.336
0.275 (0.70%)
0.336 (-0.22%)
0.340 (-18.68%)
0.375 (-10.57%)
720
0.383
0.409
0.364 (5.29%)
0.392 (4.48%)
0.424 (-9.67%)
0.430 (-4.95%)
0.12
0.10
ETSfarme [K-1]
Autafsemtr
9.08
0.06
Inforner M-BEATS
0.06
0.04
0.02
Looktack Window
Foreceal Hotizo
(a) Runtime Analysis
DeepTIMe
Menory (
ETSformer (K=11
Memon (30)
Autoformer
Informer
N HITS
96
168
336
720
1460
48
96
168 336
720
1440
Lookback Window
Forecast Horizon
(b) Memory Analysis
Figure 5: Computational efficiency benchmark on the ETTm2 multivariate dataset, on a batch size of 32. Runtime is measured for one iteration (forward backward pass). Left: Runtime/Memory usage as lookback length varies, horizon is fixed to 48. Right: Runtine/Memory usage as horizon length varies, lookback length is fixed to 48. Further model details can found Appendix P
Lastly, we perform a comparison between the optimal and pessimal scale hyperparameter for the vanilla random Fourier features layer, against our proposed CFF. We first report the results on each scale hyperparameter for the vanilla random Fourier features layer in Table 13 Appendix[N As with the other ablation studies, the results reported in Table 13 is based on performing hyperparameter sweep across lookback length multiplier, and selecting the optimal settings based on the validation set, and reporting the test set results. Then, the optimal and pessimal scales are simply the best and worst results based on Table [3] Table 5 shows that CFF achieves extremely low deviation from the optimal scale across all settings, yet retrains the upside avoiding this expensive hyperparameter tuning phase. We also observe that tuning the scale hyperparameter is extremely important, CFF obtains up to 23.22% improvement in MSE over the pessimal scale hyperparameter.
3.4 COMPUTATIONAL EFFICIENCY
Finally, we analyse DeepTime's 's efficiency in both runtime and memory usage, with respect both lookback window and forecast horizon lengths. The main bottleneck in computation for Deep'Time the matrix inversion operation the ridge regressor, canonically Onz²) complexity. This is major concern for Deep'Time as n is linked to the length of the lookback window. As mentioned in Bertinetto et al. (209). the Woodbury formulation,
+
used to alleviate the problem, leading to an Old3) complexity, where d is the hidden size hyper- parameter, fixed to some value (see Appendix D. Figure demonstrates that DeepTime is highly efficient, even when compared to efficient Transformer models, recently proposed for the long se- quence time-series forecasting task, well fully connected models.
8

---

Under review as a conference paper at ICLR 2023
4 RELATED WORK
Neural Forecasting Neural forecasting Benidis et al.] (2020) methods have seen great success in recent times. One related line of research are Transformer-based methods for LSTF (Li et al.) 2019 Zhou et al. 2021 Xu et al. 2021: Woo et al., 2022 Zhou et al. 2022) which aim to not only achieve high accuracy, but to overcome the vanilla attention's quadratic complexity. Fully connected methods (Oreshkin et al. 2020; Challu et al. 2022) have also shown success, Challu ct al. 2022) introducing hierarchical interpolation and multi-rate data sampling for the LSTF task. Meta-learning with the use of a differentiable closed-fore form solver has been explored in time-series forecasting (Grazzi et al., 202T), but for the meta- forecasting setting which adapts to new time-series datasets rather than to tackle non-stationarity. using an IMS historical-value backbone model.
Time-index Models Time-index models take as input time-index features such datetime features to predict the value of the time-series at that time step. They have been well explored as special case of regression analysis (Hyndman & Athanasopoulos 2018: Ord et al. (017). and many different predictors have been proposed for the classical setting.including linear, polynomial, and piecewise linear trends, and dummy variables indicating holidays. Of note, Fourier terms have been used to model periodicity, or seasonal patterns, and is also known as harmonic regression (Young et al. 1999). Prophet (Taylor & Letham| 2018) is popular classical approach which uses a structural time series formulation. specialized for business forecasting. Another classical approach of note are Gaussian Processes (Rasmussen) 2003; [Cranni et al. (2021) which are parammerrii models, often requiring complex kernel engineering. Gooffry & Gashler (2017) introduced an initial attempt at using time-index based neural networks to fit time-series for forecasting. Yet, their work is more reminiscent of classical methods, manually specifying periodic and non-periodic activation functions, analogous to the representation functions.
Implicit Neural Representations INRs have recently gained popularity in the area of neural ren- dering (Tewari et al. 202T). They parameterize a signal as a continuous function, mapping a coor- dinate to the value at that coordinate. A key finding was that positional encodings (Mildenhall et al.] 2020 Tancik et al.) 2020) are critical for ReL.U MI.Ps to learn high frequency details, while another line of work introduced periodic activations (Sitzmann et al.) 2020b). Meta-learning on via INRs have been explored for various data modalities, typically over images or for neural rendering tasks (Sitzmann et al.] 2020a Tancik et al.) [022 Dupont et al. 2021). using both hypernetworks and optimization- based approaches. Yüce et al. (2021) show that meta-learning on INRs is analogous to dictionary learning. In time-series,
& Shin (2022) explored using INRs for anomaly detection, opting to make use of periodic activations and temporal positional encodings.
5 DISCUSSION
In this paper, we proposed DeepTime, a deep time-index based model trained via a meta-learning formulation to automatically learn a representation function from time-series data, rather than man- ually defining the representation function as per classical methods. The meta-learning formulation further enables DeepTime to be utilized for non-stationary time-series by adapting to the locally sta- tionary distribution. Importantly, we use a closed-form ridge regressor to tackle the meta-learning formulation to ensure that predictions are computationally efficient. Our extensive empirical anal- ysis shows that DeepTime, while being a much simpler model architecture compared to prevailing state-of-the-as methods, achieves competitive performance across forecasting benchmarks on real world datasets. We perform substantial ablation studies to identify the key components contributing to the success of Deep'Time, and also show that it is highly efficient.
Limitations & Future Work Despite having verified DeepTime's s effectiveness, we expect some under-perfee peddor-pefo in cases where the lookback window contains significant anomalies, an abrupt change point which violates the locally stationary assumption. Next, while out of scope for our current work, limitation that DeepTime faces is that it does not consider holidays and events. We leave the consideration of such features as potential future direction, along with the incorporation of exogenous covariates and datetime features, whilst avoiding the incursion of the meta-learning memorization problem. Finally, time-index models are a natural fit for missing value imputation, as well as other time-series intelligence tasks for irregular time-series this is another interesting future direction to extend deep time-inder models towards.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended pac-bayes theory. In International Conference Machine Learning, pp. 205-214, PMLR, 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL
https://arxiss https///xxwv.oor//aaaae wwtp:/xwwwww.o./////////////////
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1825 2013.
Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Bernic Wang. Danielle Mad- dix, Caner Turkmen, Jan Gasthaus, Michael et al. Neural forecasting: Introduction and literature overview. arXiv preprint arXiv:2004. 10240, 2020.
David Salinas, Lorenzo Stella,
Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. In International Conference on Learning Representations, 2019. URL
httpps::/openrevieww net. wanee//forreeeeeeeeeeeeeeeeeeeee forum Hy
Real Carbonneau, Kevin Laframboise, and Rustam Vahidov. Application of machine learning tech- niques for supply chain demand forecasting. European Journal of Operational Research, 184(3): 1140-1154, 2008
Olivier Catoni. Pac-bayesian supervised classification: the thermodynamics of statistical learning. arXiv preprint arXiv:0712.02488, 2007.
Cristian Challu, Kin G Olivares, Boris Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201.12886 2022.
Guillaume Chevillon. Direct multi-step estimation and forecasting, Journal of Economic Surveys, 21(4):746-75,, 2007.
Giorgio Corani, Alessio Benavoli, and Marco Zaffalon. Time series forecasting with gaussian pro- cesses needs priors. In Joint European Conference on Machine Learning and Knowledge Discov- ery in Databases, pp. 103-117. Springer, 2021.
Rainer Dahlhaus. Locally stationary processes. In Handbook of statistics, volume 30, pp. 351-413. Elsevier, 2012.
Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang- Adarnn: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM Inter- national Conference on Information & Knowledge Management, pp. 402-411, 2021.
Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models distributions of func-
tions. arXiv preprini arXiv:2102.04777 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.
1126-1135. PMLR, 06-11 Aug 2017. URL https:t///// httpss///ccceeeengg.ooo .ateesm.rr..pe. wttp:/sss.........1...so
Finnl7a Soan17a.htm. html.
Zoubin Ghahramani and Geoffrey E Hinton. Variational learning for switching state-space models.
Neural computation, 12(4)8331-84, 2000.
Luke B Godfrey and Michael S Gashler. Neural decomposition of time-series data for effective
generalization. IEEE transactions on neural networks and learning systems, 29(7):993322985, 2017.
Riccardo Grazzi, Valentin Flunkert. David Salinas, Tim Januschowski. Matthias Seeger, and Cedric Archambeau. Meta-forecasting by combining global deeprepresentations with local adaptation. arXiv preprint arXiv:21/1.034/1 2021.
10

---

Under review as a conference paper at ICLR 2023
Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
Kyeong-Joong Jeong and Yong-Min Shin. Time-series series anomaly detection with implicit neural rep- resentation. arXiv preprint arXiv:2201.11550, 2022.
Kyoung-jae Kim. Financial time series forecasting using support vector machines. Neurocomputing, 55(1-2):307-319, 2003.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Ho Choi, and Jacgul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference Learning Representations, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1rXiv::412.6990 2014.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research Development Information Retrieval, pp. 95-104, 2018.
Nikolay Laptev, Jason Yosinski, Li Erran Li, and Slawek Smyl. Time-series extreme event forecast- ing with neural networks uber. In International conference machine learning, volume 34, pp. 1-5, 2017.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, ArXiv, abs//190..00 00235, 2019.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long, Non-stationary transformers: Exploring the stationarity in time series forecasting. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /
net htt.://eoooooomoodddddo- id-
Massimiliano Marcellino, James H Stock, and Mark W Watson. A comparison of direct and iterated multistep ar methods for forecasting macroeconomic time series. Journal of econometrics, 135 (1-2):499-5) 526, 2006.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' A'mour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
Keith Ord, Robert Alan Fildes, and Nikolaos Kourentzes. Principles of business forecatting Wessex Press Publishing Co., 2017.
Boris N. Oreshkin, Dmitri Cacoov,Nicolas pansion analysis for interpretable time series forecasting. In International Conference on Learn- ing Representations, 2020. URL http:htttt/://ereeeeeee
Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis
net. httts//ooommmmdmdreeeeeeee Y
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- ence on Machine Learning, pp. 5301-5310. PMLR, 2019.
Janarthanan Rajendran, Alexander Irpan, and Eric Jang. Meta-learning requires meta-augmentatiio.. Advances in Neural Information Processing Systems, 33:5705-5715, 2020.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
11

---

Under review as a conference paper at ICLR 2023
Sachin Ravi and H. Larochelle. Optimization as model for few-shot learning. In ICLR, 2017.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1191, 2020.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learming: From theory to algo- rithms. Cambridge university press, 2014.
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential neural processes. Ad- vances in Neural Information Processing Systems, 32, 2019.
Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein, Metasdf: Meta-learning signed distance functions. Advances in Neural Information Processing Systems, 13:10736-0017, 2020a.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7474, 2020b.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ily Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-1958 2014.
Souhaib Ben Taieb, Rob Hyndman, et al. Recursive and direct multi-step forecasting: the best of both worlds, volume 19. Citeseer, 2012.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances Neural Information Process- ing Systems, 33:7537-7547, 2020.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate-base neural representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Sean J Taylor and Benjamin Letham. Forecasting scale. The American Statistician, 72(1):37-45, 2018.
Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Yifan Wang, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Bruald Stephen Lombardi, et al. Ad- vances in neural rendering. arXiv preprint arXiv:2111.05849, 2021.
Michael Vogt. Nonparametric regression for locally stationary time series. The Annals of Statistics, 40(5):2601-2633. 2012.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting, arXiv preprint urXiv:2202.0138// 2022.
Jichui Xu, Jianmin Wang. Mingsheng Long. al. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 2021.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. In International Conference Learning Representations, 2020. URL.
https::/openzeview net/ forum? Bi1.ddddkkk1EEe
Peter C Young, Diego J Pedregal, and Wlodek Tych. Dynamic harmonic regression. Journal of forecasting, 18(6):369-394, 1999.
Gizem Yüce, Guillermo Ortiz-Jiménez, Beril Besbinar, and Pascal Frossard. A structured dictionary perspective implicit neural representations. arXiv preprint arXiv:21 2021.
12

---

Under review as a conference paper at ICLR 2023
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of AAAI, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Fre- quency enhanced decomposed transformer for long-term series forecasting. arXiv preprint arXiv:2201. 12740, 2022.
13

---

Under review as a conference paper at ICLR 2023
A EXTENDED RELATED WORK
Non-stationarity and Distribution Shift Switching state-space models (Ghahramani & Hinton 2000) generalizes and combines hidden Markov models and state-space models, where the dynam- ics in each regime is typically represented by a linear model (linear dynamic system) and switches between regimes, controlled by hidden transition probabilities of a Markov process. They can be applied to non-stationary time-series, but makes additional assumptions that the time-series has predefined number of regimes or generating distributions. Sequential Neural Processes (Singh et al. 2019) incorporates temporal state-transition model of stochastic processes, extending Neu- ral Process framework dynamic stochastic processes. Importantly, SNPs use a "black-box meta- meta-learning" approach. Fur- ther differences include that the standard SNP setting requires knowledge of task boundaries, and multiple support/query sets per task. Du et al. (2021) tackled temporal covariate shift by distri- bution matching, an approach popularly used in domain adaptation. They introduce the Temporal Distribution Characterization module which divides given time-series into regions with different disribution, and a Temporal Distribution Matching module reduces distribution mismatch in the time-series. Their approach is built top of RNN architecture. Kim al. (202T) introduced a learnable instance normalization method to tackle the covariate shift problem. Their approach is ad-hoc and can be attached to any existing architecture. Most relevant to our work is Non-stationary Transformers (Liu et al., 2022), which introduced an instance normalization method for the LSTF task. Rather than proposing a generic module, their approach is specialized for Transformer-baser architectures and performs normalization each layer to tackle the non-stationarity of intermediate representations, rather than just the inputs and outputs.
learning" approach, while DeepTTme uses "optimization-base
B NON-STATIONARITY AND DISTRIBUTION SHIFT
In our work, we tackle the problem non-stationarity time-series data, which has been a well explored problem in the context of classical semees amalysis. We map this problem to the modern setting deep learning for time-series forecasting. As mentioned in Section [ long time-series datasets collected due to the advances of I.T. infrastructure has been plagued by the problem of non- stationarity. In particular, we tackle the problems of covariate shift, and conditional distribution shift which arise from it.
Definition 1. (Covariate Shift) Given stochastic process {Yi)T= and let plyt, Bt-1..... Ht- L+1) be the unconditional joint distribution of a length L segment. The stochastic process said experience covariate shift if any two segments are drawn from different distributions, i.e. p(y,, Vt-1...., + p(ye, (o---.....V--L). t #
Definition 2. (Conditional Distribution Shift) Given a stochastic process {Y:)²= and let be the conditional distribution of Yo41 on length L segment of
p(1++1 (Mt, -o-1.....
previous time steps. The stochastic process is said to experience a conditional distribution shift if any two segments have different conditional distributions, i.e. p(yc+1 1le 00-1..... (t-+++) #
p 1BBe, 1 -t+1, t
14

---

Under review as a conference paper at ICLR 2023
C CATEGORIZATION OF FORECASTING METHODS
Table 6: Categorization of time-series forecasting methods over the dimensions of time-index vs historical-value methods, and DMS vs IMS methods.
Time-index
Historical-value
DeepTime
N-HiTS
Prophet
FEDformer
DMS
Gaussian process
ETSformer
Time-series regression
Autoformer
Informer
N-BEATS
DeepAR
IMS
ARIMA
ETS
Multi-step Forecasts
Forecasting over horizon (multiple time steps) be achieved via two
strategies, direct multi-step, or iterative multi-step (Marcellino et al. [0066] Chevillon
2007
Taieb
al. 2012). or even mixture of both, but this has been less explored:
Direct Multi-step (DMS): A DMS forecaster directly predicts forecasts for the entire hori- zon. For example, to achieve a multi-step forecast of H time steps, DMS forecaster simply outputs H values in single forward pass.
Iterative Multi-step (IMS): An IMS forecaster iteratively predicts one step ahead, and consumes this forecast to make subsequent prediction. This performed iteratively, until the desired length is achieved.
D FURTHER DISCUSSION ON DEEPTIME AS TIME-INDEX MODEL
We first reiterate our definitions of time-index and historical-value models from Section Time- index models models whose predictions are purely functions of current time-index features. To perform forecasting (i.e. make predictions over some forecast horizon), time-index models make the
predictions Dt+h f(T++h) for h 0, H 1. Historical-valee models predict the time series value future time step(s) as a function of past observations, and optionally, covariates.
Time-index Models
Historical-value Models
U. f(T))
D++1 f(V. V:-1..... Zt-1, žt.
Next, further discuss some subtleties of how time-index models interact with past observations. Astute readers may have noticed Deep to be a function of the past observations. In particular, that Equations 3 and (4) indicate that forecasts from DeepTime are in fact linear the lookback window. However, we highlight that this is not in contradiction with our definition of time-index and historical-valur models. Here, we differentiate between the model. and the learning algorithum, A. The learning algorithm A : H x RLxm + H. takes as input a model from the hypothesis class H and, past observations, returning model minimizing the loss function L. time-indes model is thus, still only function of time-index features, while the learning algorithm is function of past observations, i.e. fo E H. f Re + RT f A(foo Y,-L.). DeepTime as a forecaster, is deep time-index model endowed with a meta learning algorithm. In order to perform forecasting, it has to perform an inner loop optimization step defined by the learning algorithm, as highlighted in Equation (2). For the special learning algonithm reduces to a form which is linear the lookback window. Still, the deep time- index model is only function of time-index features.
where use the mosed-form ridge regressor, the inner loop
15

---

Under review as a conference paper at ICLR 2023
E DEEPTIME PSEUDOCODE
A PyTorch-Spip Style Pseudocode of Closed-Form Ridge Regressor
mm: mairix multiplication, di agonal; relurns the diagonal elements of matrix, add.; in-place addition linalg. solve computes the solution of square system of linear equations with unique solution.
X:
inputs, shape: (o.samples, n.dim)
# Y: targets, shape: (n_samples, n_out)
lambd: scalar value representing the regularization coefficient
n_samplea, n.dim x.ahape
term by concatenating
vector
ones torch. onessonnesff one cn_sampes, 1)
x cat([X, (Xx, ones], din--1)
n.samples m.dim: standard formulation
um (X.T, x)
A. d.diagona( (aa.aall..ddd..oottaaaaaa. (addd.(eorrtpluuseembb)))
mm(X.T,
weights = linalg. Solve B)
else:
Voodbury formulation
mm(X, X.T)
A. A.diagonal () (td.(toootllussaaaad)))
veights (X.T, sinalg-solves Y)) weights[:-1]] veights[-1:]
Algorithm PyTorch-Styy Pseudocode of DeepTIMe
rearrange: cinops style tensor operations
matrix multiplication
# input tine-series, shape: (lookback.len) multivariateddim)
# lookback.len: scalar value representing the length of the lookback window
horizon_lem, scalar value representing length of forecast horizon implicit neural representation
tidexindex index linspace(0, 1, lookback len + Norizon_len). # shape: (lookback len + horizon. len)
time_indez rearrange (time_index, 't -> 1') shape: (lookback.len horizon.len, 1) time_reprs inr
shape: (lookback_len horizon_len, hidden_din)
lookback_repp
horizon reprs tine reprs [-harizon [-prsn- lan:]
b [edgeerogressorcC (lookbao(oookkakkrrrgo x)
# w. shape - (hidden dim, multivariate dim), b.shape (1, miltivariate dim)
preds un (moorizon repre, +
preds
16

---

Under review as a conference paper at ICLR 2023
F SYNTHETIC DATA
The training set for each synthetic data experiment consists 1000 functions/tasks, while the test set contains 100 functions/tasks. We ensure that there is overlap between the train and test
Linear Samples are generated from the function = az + b for z E [-1,1]. This means that each function/task consists 400 evenly spaced points between -1 and The parameters of each function/task (i.e. b) are sampled from a normal distribution with mean O and standard deviation of 50, i.c. a, (0,500 502).
Cubic Samples generated from the function y ax. bu. + d for [-1, 1] for 400 points. Parameters of each task are sampled from a continuous uniform distribution with minimum value of -50 and maximum value of 50, i.e. a, b, c, ((500, 50).
Sums of sinusoids Sinusoids come from a fixed set of frequencies, generated by sampling ~ Ws.. (wi,..., Each function then
(0, 12x). We fix the size of this set to be five, i.e. sum of J sinusoids, where J E {1,2,3, 4,5} is randomly assigned. The function is thus = Ey-1 Aj sin(w,, P2)) for x E [0, 1J. where the amplitude and phase shifts freely chosen via Aj 4(001,5), P 240 x). but the frequency is decided by ry E (1,2,3,4,) 4,53 randomly select frequency from the set f2
G DATASETS
ETI Zhou et al. (021) Electricity Transformer Temperature provides measurements from an electricity transformer such as load and oil temperature. We use the E77m2 subset, consisting measurements at 15 minutes frequency.
ECI Electricity Consuming Load provides measurements of electricity consumption 321 households from 2012 to 2014. The data was collected at the 15 mintue level, but is aggregated hourly.
Exchange Lai et al. 2018 collection of daily exchange rates with USD of eight countries (Australia, United Kingdom, Canada, Switzerland, China, Japan, New Zealand, and Singapore) from 1990 to 2016.
Traffic dataset from the California Department of Transportation providing the hourly road oceu- pancy rates from 862 sensors in San Francisco Bay area freeways.
Weather provides measurements of 21 metcorological indicators such as air temperature, hu- midity, etc., every 10 minutes for the year of 2020 from the Weather Station of the Max Planck Biogeochemistry Institute in Jena, Germany.
IL Influenza-like Illness measures the weekly ratio of patients with ILI and the total number of patients, obtained by the Centers for Disease Control and Prevention of the United States between 2002 and 2021.
httess//eems.dd..cc..oo///
htrpa//www..beejeeenn.......
17

---

Under review as a conference paper at ICLR 2023
G.1 NON-STATIONARITY OF REAL-WORLD DATASETS
Table 7: Summary of real-world datasets, results of Chow test, and Augmented Dickey-Fuller (ADF) test. The statistical tests are performed on each dimension separately, since they are designed for univariate time-series. We report the number of dimensions which reject/fail to reject the null hy- pothesis, depending on which indicates non-stationarity. These are reported at significance levels of 0.1, 0.05, and 0.01. Larger values for the Chow test statistic indicate more non- non-stationarity. and larger (less negative) values for the ADF test statistic indicates more son-station.rity.
Dataset
ETTm2
ECL
Exchange
Traftic
Weather
ILI
Dimensions
321
8
862
21
Timesteps
57,600
26,304
7,588
17,544
52,696
966
Sampling Frequency
15 minutes
1 hour
day
I hour
10 minutes
week
Chow Test: dims reject null hypothesis @ 0.1
321
792
19
Chow Test: dims reject null hypothesis 0.05
320
771
19
Chow Test: dims reject null hypothesis 0.01
317
734
19
Chow Test: average Chow statistic
49.13
23.55
21.25
16.93
25.39
2.77
ADF Test: dims fail reject 0.1
139
807
ADF Test: # dims fail to reject @ 0.05
120
739
7
ADF Test: # dims fail to reject @ 0.01
103
657
6
ADF Test: average ADF test statistic
-4.48
-5.24
-2,40
-3.29
-4.49
-6.89
Real-world datasets used in long sequence time-series forecasting suffers from non-stationarity. We first verify this qualitatively by visualizing histogram values across some dimensions for each dataset in Figure 6 This simple visualization already gives us strong confirmation on the distribution mismatch between the training and testing phases. We further verify this quantitatively via two statistical tests, the Chow test, and Augmented Dickey-Fuller (ADF) test. The Chow test is test of whether the true coefficients in two linear regressions on different data sets are equal. Rejecting the null hypothesis of equality of regression coefficients in the two periods indicates that the train and test regions are generated from different distributions. The ADF test tests the null hypothesis that unit root is present in time series sample. Not rejecting the null hypothesis indicates that unit root is present, and thus non-stationary. Presented in Table 7 along with some dataset statistics, report the results of both tests and the number of dimensions which meet the criteria for non- stationarity (rejecting the null hypothesis for Chow test, and not rejecting the null hypothesis for the ADF test) over various significance levels. We observe that the real-world datasets exhibit high levels of non con-stationariyy across dimensions based on both tests.
18

---

Under review as a conference paper at ICLR 2023
trann
tran
trai
ETTm2
tmah
train
train
test
tost
ECL
Irain tost
Iriin
best
Exchange
train
train
train
Traffic
tmai text
trar text
train fest
Weather
Figure 6: Histograms of time-series values from each real-world dataset presented in experiments. Visualized are the histograms of the train and splits, verifying that even on the distribution of the marginals are different across time, signifying non-stationarity.
19

---

Under review as a conference paper at ICLR 2023
H DEEPTIME IMPLEMENTATION DETAILS
Optimization We train Deep'Time with the Adam optimizer (Kingma & Ba 2014) with learning rate scheduler following linear warm up and cosine annealing scheme. Gradient clipping by norm is applied. The ridge regressor regularization coefficient, X, is trained with a different, higher learn- ing rate than the rest of the meta parameters. We use early stopping based on the validation loss, with a fixed patience hyperparameter (number of epochs for which loss deteriorates before stopping). All experiments are performed on Nvidia A100 GPU.
Model The ridge regression regularization coefficient is a learnable parameter constrained to pos- itive values via a softplus function. We apply Dropout (Srivastava et al.. 2014), then LayerNorm (Ba et al.) 2016) after the ReLU activation function in each INR layer. The size of the random Fourier feature layer is set independently of the layer size, in which we define the total size of the random Fourier feature layer- the number of dimensions for each scale is divided equally.
I DEEPTIME HYPERPARAMETERS
Table 8: Hyperparameters used DeepTime.
Hyperparameter
Value
Epochs
50
Learning rate À learning rate
le-3
Optimization
1.0
Warm up epochs
5
Batch size
256
Early stopping patience
7
Max gradient norm
10.0
Layers
5
Layer size
256
Model
À initialization
0.0
Scales
[0.01, 0,,, 10,20,,22000 50, 100]
Fourier features size
4096
Dropout
0.1
Lookback length multiplier, μ
1 (1,3,5,7,9)
J UNIVARIATE FORECASTING BENCHMARK
Table 9: Univariate forecasting benchmark on long sequence time-series forecasting. Best results are highlighted in bold, and second best results are underlined.
N-BEATS
Micuco
MSE
SEAE
MSE
MAE
MSE
MAE
MIEE
MAAE
33SE
MAE
SISE
NEAE
MSE
MAN
MISE
MAAE
MSE
SCAE
MISE
MAE
3I5E
MAE
B3295
0.224
0187
(1123
IL:50
0.102
0.285
01115
(1256
0.261
0.120
11.85
BL110
0.851
0.261
0.406
0.154
0.337
3.66
B11.8
0.203
0.106
0.251
0.314
0130
0.279
11154
0.3
0.800
0.335
0.225
115/3
B1277
B443
0.311
0.824
0.377
0,488
0.399
03318
01185
0.157
0.225
01779
0178
0,325
03355
0.700
0.433
0.338
0.488
0.314
0.366
0.487
6318
6:866
0.225
0.09
0.333
0.59
0,330
0.111
0.284
0.341
0.299
0.591
0.65
0.156
0.299
0.417
0.515
0.325
0.762
0.112
0.365
0.165
0311
190
8.173
0.300
12:00
0.313
0.223
0.35
0271
0.420
0.273
0.565
1.183
0.12
0.603
0.665
01515
0.35
0,099
0374
0.302
0.46
0.549
0611
20

---

Under review as a conference paper at ICLR 2023
K DEEPTIME STANDARD DEVIATION
Table 10: DeepTime main benchmark results with standard deviation. Experiments are performed over three runs.
(a) Multivariate benchmark.
(b) Univariate benchmark.
Metrics
MSE (SD)
MAE (SD)
Metrics
MSE (SD)
MAE (SD)
96
1,66 (0.000)
0.257 (0.001)
96
0.065 (0.000)
0.186 (0.000)
ETTm2
192
0.225 (0.001)
0.302 (0.003)
BT7m2
192
0.096 (0.002)
0.234 (0.003)
336
0.277 (0.002)
0.336 (0.002)
336
0.138 (0.001)
0.285 (0.001)
720
0.383 (0.007)
0.409 (0.006)
720
0.186 (0.002)
0.338 (0.002)
96
0.137 (0.000)
0.238 (0.000)
96
0.086 (0.000)
0.226 (0.000)
Exchange
ECL
0.152 (0.000)
0.252 (0.000)
192
0.173 (0.004)
0.330 (0.003)
336
0.166 (0.000)
0.268 (0.000)
336
0.539 (0.066)
0.575 (0.027)
720
0.201 (0.000)
0.302 (0.000)
720
0.936 (0.222)
0.763 (0.075)
Exchange
96
0.081 (0.001)
0.205 (0.002)
192
0.151 (0.002)
0.284 (0.003)
336
0.314 (0.033)
0.412 (0.020)
720
0.856 (0.202)
0.663 (0.082)
96
0.390 (0.001)
0.275 (0.001)
Traffic
192
0.402 (0.000)
0.278 (0.000)
336
0.415 (0.000)
0.288 (0.001)
720
0.449 (0.000)
0.307 (0.000)
96
0.166 (0.001)
0.221 (0.002)
Weather
192
0.207 (0.000)
0.261 (0.000)
336
0.251 (0.000)
0.298 (0.001)
720
0.301 (0.001)
0.338 (0.001)
24
2.425 (0.058)
1.086 (0.027)
36
2.231 (0.087)
1.008 (0.011)
48
2,230 (0.144) 144)
1.016 (0.037)
60
2.143 (0.032)
0.985 (0.016)
21

---

Under review as a conference paper at ICLR 2023
L LOOKBACK LENGTH SENSITIVITY ANALYSIS
Table 11: Sensitivity analysis the lookback window length. Results presented the ETTm2 dataset across various values of the lookback length multiplier, μa. Best results are highlighted in bold.
Horizon
192
720
MSE
MAEE
MSE
MAE
MSE
MAE
MSE
MAE
0.192
0.287
0.255
0.332
0.294
0.354
0.383
0.409
0.172
0.264
0.228
0.304
0.277
0.336
0.371
0.403
0.259
0.225
0.302
0.275
0.389
0.420
7
0,166
0.257
0.223
0.300
0.279
0.343
0.440
0.451
0.258
0.223
0.301
0.285
0.350
0.409
0.434
M ADDITIONAL ABLATION STUDY
Table 12:
Additional ablation study on variants of DeepTime. Finetune refers to training INR via gradient descent for each lookback window on top of having a training phase. Full MAMI. refers performing the full meta-learning formulation the whole model rather than just the last layer, using gradient-based optimization.
Finetune Datetime
Methods
DeepTime
Finetune
Full MAML
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.166
0.257
3.028
1.328
3.242
1.365
0.235
0.326
E17m2
192
0.225
0.302
3.043
1.341
3.385
1.391
0.295
0.361
3.367
1.387
0.348
0.392
720
0.383
0.409
2.721
1.253
3.476
1.407
0.491
0.484
N RANDOM FOURIER FEATURES SCALE HYPERPARAMETER SENSITIVITY ANALYSIS
Table 13: Results from hyperparameter sweep the scale hyperparameter. Best scores are high- lighted bold, and worst scores are highlighted in bold red.
Hyperparam
Metrics
MSE
MAB
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
ET1m02
192
96
0.275
0.216
0.340
0.300
0.264
0.189
0.333
0.285
0.239
0.173
0.517
0.268
0.168
0.225
0.301
0.262
0.225
0.166
0.260
0.303
0.166
0.224
0.302
0.258
0.224
0.165
0.304
0.259
0.230
0.164
0.301
0.257
0.275
0.280
0.344
0.381
0.12
0.364
0.392
0.466
0.429
O ABLATION STUDIES DETAILS
In this section, we list more details on the models compared to in the ablation studies section. Unless otherwise stated, we perform the same hyperparameter tuning for all models in the ablation studies, and use the same standard hyperparameters such as number of layers, layer size, etc.
O.1 ABLATION STUDY ON VARIANTS OF DEEPTIME
RR Removing the ridge regressor module refers to replacing it with a simple linear layer, Linear Ed + "h where Linear(x) Wæ + b, dd W gm b e R" This corresponds to straight forward INR, which is trained across all lookback-horizon pairs in the dataset.
Local For models marked "Local", we similarly remove the ridge regressor module and replace with linear layer. Yet, the model is not trained across all lookback-horizon pairs in the dataset.
22

---

Under review as a conference paper at ICLR 2023
Instead, for each lookback-horizer pair in the validation/test set, fit the model the lookback window via gradient descent, and then perform prediction on the horizon to obtain the forecasts. A new model is trained from scratch for each (ookback-horizon window. We perform tuning an extra hyperparameter, the number of epochs to perform gradient descent, for which we search through {10, 20, 30, 40, 50).
As each dataset comes with timestamps for each observation, we are able to construct datetime features from these timestamps. We construct the following features:
Datetime Features
1. Quarter-of-year
2. Month-of-year
3. Week-of-year
4. Day-of-year
5. Day-of-month
6. Day-of-week
7. Hour-of-day
8. Minute-of-hour
9. Second-of-minute
Each feature is initially an integer value, e.g. month-of-year can take on values in {0, 1...., 11). 11), which we subsequently normalize [0. 1] range. Depending on the data sampling frequency, the appropriate features can be chosen. For the ETTm2 dataset, we used all features except second-of- minute since is sampled at 15 minute frequency.
O.2 ABLATION STUDY ON BACKBONE MODELS
For all models in this section, retain the differentiable closed-form ridge regressor, to identify the effects of the backbone model used.
MLP
The random Fourier features layer is mapping from coordinate space to latent space 7 R - Rd. To remove the effects of the random Fourier features layer, we simply replace with with linear map, Linear R RRd.
SIREN
We replace the random Fourier features backbone with the SIREN model which is intro- duced by Sitzmann et al. (2020b). In this model, periodical activation functions are used, i.c. sin(x), along with specified weight initialization scheme.
RNN We use a 2 layer LSTM with hidden size of 256. Inputs are observations, yt. in an IMS fashion, predicting the next time step, Y/+1-
P COMPUTATIONAL EFFICIENCY EXPERIMENTS DETAILS
Trans//n/Auto/ETS-formo We use model with 2 encoder and 2 decoder layers with a hidden size of 512, specified in their original papers.
use an N-BEATS model with stacks and layers (relatively small compared to 30 stacks and 4 layers used in their orignal papef"), with hidden size of 512. Note, N-BEATS a univariate model and values presented here are multiplied by factor of m to account for the multivariate data. Another dimension of comparison the number of parameters used in the model. Demonstrated in Table 14, fully connected models like N-BEATS, their number of parameters scales linearly with lookback window and forecast horizon length, while for Transformer-based and Deep- Time, the number of parameters remains constant.
N-BEATS
hhttps://githie com/E1 ementAl /N- c eleetricity//eeeeriin
23

---

Under review as a conference paper at ICLR 2023
N-HITS We use N-HiTS model with hyperparameters sugggested in their original paper (3 stacks, block in each stack, MLP layers, 512 hidden size). For the following hyperparam- eters which were not specified (subject to hyperparameter tuming). to 2, and the number of stack coefficients to [24, 12, 1]. Similar to N-BEATS, N-HiTS is univariate model, and values were multiplied by factor of to account for the multivariate data.
set the pooling kernel size
Table 14: Number of parameters in each model across various lookback window and forecast hori- lengths. The models were instantiated for the ETTm2 multivariate dataset (this affects the embedding and projection layers in Autoformer. Values for N-HiTS in this table are not multiplied by since global model (i.e. single univariate model is used for all dimensions of the time-series).
Methods
Autoformer
N-HiTS
DeepTime
48
10,535,943
927,942
1,314,561
96
10.535,943
1,038.678
1,314,561
Lookback
168
10,535,943
1,204,782
1,314,561
336
10,535,943
1,592,358
1,314,561
720
10,535,943
2,478,246
1,314,561
1440
10,535,943
4,139,286
1,314,561
2880
10,535,943
7,461,366
1,314,561
5760
10,535,943
14,105,526
1,314,561
48
10,535,943
927,942
1,314,561
96
10.535.943
955,644
1,314.561
168
10,535,943
997,197
1,314,561
Horizon
336
10,535,943
1,094,154
1,314,561
720
10,535,943
1,315,770
1,314,561
1440
10,535,943
1,731,300
1,314,561
2880
10,535,943
2,562,360
1,314,561
5760
10,535,943
4,224,480
1,314,561
24

---

Under review as a conference paper at ICLR 2023
Q GENERALIZATION BOUND FOR OUR META-LEARNING FRAMEWORK
In this section, we derive a meta-learning generalization bound for Deep'Time under the PAC-Baye framework (Shalev-Shward & Ben- 2014). Our formulation follows (Amit & Meir 2018) and assumes that all tasks share the same hypothesis space H. sample space 2 and loss function e H x 2 + [0, 1]. We observes n tasks in the form of sample sets D..... Dn. The number of samples in each task is H + L. Each dataset Dx is assumed to be generated i.i.d from an unknown sample distribution DT++. Each task's sample distribution D3 is i..d generated from an unknown meta distribution, E. Particularly, we have D₂ = (Z 2k...... 2k+H-11,, where 2 = (Ti, y:). Here, is the time coordinate, and ye is the time-series value. For any forecaster /t() parameterized by 6, we define the loss function E(ho, 21). We also define P as the prior distribution over H and Q as the posterior over H. for each task. In the meta-learning setting, we assume a hyper-prior P. which is a prior distribution over priors, observes a sequence of training tasks, and then outputs a distribution over priors, called hyper-posterior Q.
Theorem Q.1. Consider the Mern-ng framework, given the hyper-prior P, then for any hyper-posterior 2, any c11 02 > 0 and any ô (0, 1] with probability 1- 6 we have,
1 KI((I1Q) log
P
er (0) < (1---a)(
C102
ér(2. Da) +
1 e-
na1
[L + log oy
C1 (1 - e
2 21 (5)
n(H + L)e2
Proof. Our proof contains two steps. First, we bound the error within observed tasks due to ob- serving limited number of samples. Then we bound the error on the task environment level due to observing finite number of tasks. Both of the two steps utilize Catoni's 's classical PAC-Bayes bound (Catoni 2007) to measure the error. We give here the Catoni's 's classical PAC-Bayes bound.
Theorem Q.2. (Catoni's bound (Catoni 2007)) Let X be sample space, P(X) distribution over X, O a hypothesis space. Given a loss function ((,,X) ④ x X + [0. and a collection of M i..d random variables (X1..... Xxa sampled from P(X). Let π be prior distribution over hypothesis space. Then, for any 5 E (0, 1] and any real number c > 0, the following bound holds uniformly for all posterior distributions p over hypothesis space,
M
(T((p]TT) + log
P X,~P(X),8mP
E
1000 < 1- M
Σ m=1
E ((((0, 0mp
vp
Mc
2 1 8.
We first utilize Theorem Q.2 to bound the generalization error in each of the observed tasks. Let k be the index of task, we have the definition of expected error and empirical error as follows,
er(Q,
E PNO
E
E znDr
E(h,
(6)
(r,
E
E
1
Σ
2(h,2))
PNQ
H
j=k-L
Then, according to Theorem Q.2 for any oy ~ (0,1) and 02 > 0, we have
C2 (L(pl(T) + log bx
1
P er Dw)
C2 1 1- ér Dz)
2
8p.
(8)
1- (H + L)02
Next, we bound the error due to observing limited number of tasks from the environment. Simi- larly, we have the definition of expected task error as follows
(r)))= = E
E
E P~o
E E(h, 2) anD
D~E
DADRIL
k-Q(D.P)
=
E
E DNDRIL
D).
(9)
D~E
25

---

Under review as a conference paper at ICLR 2023
Then we have the definition of enror across the n tasks,
n
" n
E
E
E(h,2)
er Q, Dk).
(10)
n
P
hr.(De.P))
Dr
Then Thcorem Q.2 says that the following holds for any do ~ (0, 1] and c > 0, we have
KL(PI Q) + log
P er (r(()) c Σ er(Q.
a 1-e-a
21 6g.
(11)
nc1
Finally, by employing the union bound, we could bound the probability of the intersection of the events in Equation (11) and Equation @
For any 8 >> 0, set do o/2 and à 3 for 1, ,,..,,
n
(I((PI1) + log oz
er( Q) S
C1
ér(2.
(1 - e 00) 00)
n
1
nc1
KL(p][T) + log dz
0102
2 11 8. (12)
11 (1-
n(H + L)e2
Theorem Q.1 shows that the expected task generalization emror is bounded by the empirical multi- task error plus two complexity terms. The first term represents the complexity the environment, or equivalently, the time-series dataset, converging zero if observe infinitely long time-series (n + oc). The second term represents the complexity of the observed tasks, or equivalently, the lookback-horiin windows. This converges in each window (H + L - co).
when there sufficient number of time steps
26


Reviews:
[{"review_id": "nwPc1tG3Od", "reviewer": "Reviewer_comQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes DeepTime, a deep time-index model trained via meta-learning, with the motivations for stronger smoothness prior, avoiding the problem of covariate shift, and having better sample efficiency. Experimentation on various time series benchmarks are shown.", "summary_of_the_review": "Because of the issues above, mainly lack of convincing empirical validation appropriate for non-stationarity, the paper cannot be accepted in its current form in my opinion.", "strength_and_weaknesses": "Strengths\n\n- Forecasting as meta learning framework is an interesting and canonical idea. \n\n- Extensive ablation analyses\n\nWeaknesses\n\n- There is not sufficient explanation on what non-stationarity is. \n\n- The method is proposed for non-stationary time-series, but the experiments are not specifically obtained for non-stationary time-series data. The synthetic datasets are simple functions and most of the real-world benchmarks are not from applications where non-stationary dynamics are prominent, maybe except Exchange, on which the results are weaker. \n\n- It is unclear why Fourier features would be a good idea for non-stationary time series, as opposed to time-varying frequency representations. \n\n- There is no analysis on how much different datasets contain covariate or distribution shifts. \n\n- Some relevant baselines are skipped:\nhttps://arxiv.org/abs/2205.14415\nhttps://arxiv.org/abs/2205.13504\nhttps://arxiv.org/abs/1912.09363\n\n- For computational efficiency, the impact of parallelization is ignored.\n\n- The impact of lookback length is not sufficiently analyzed. \n\n- For a non-stationary dataset, training and validation distributions would have differences. How would it affect the hparam selection? This aspect is ignored. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WWK33a1m0NA", "reviewer": "Reviewer_i4vc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a meta-learning formulation for learning to adapt a deep time-index model to the look-back window. INR was the choice of time-index models, and meta-learning was formulated by using samples from look-back window as context set and forecasting horizon as query set. The goal of the meta-learning was mainly motivated for dealing with non-stationary time series. Experiments were conducted on synthetic as well as real datasets demonstrating performance improvement of forecasting over selected baselines.\n", "summary_of_the_review": "This paper tackles an important problem of time-series forecasting (i..e, non-stationary series) with an interesting solution (meta-learning of time-index models). The novelty however is unclear with respect to some existing works especially sequential neural processes. The choice or benefit of meta-learning method and the simplification of optimizing only the last layer of the INR, in comparison to alternative meta-learning method such as feed-forward model based method used in SNP that can bypass such simplifications, is not clear. The writing lacks many critical details on methodology and experiments, such as how are tasks handled during meta-training, whether the task boundary needs to be known, and how to address lookback window or forecasting horizon that includes task boundaries. It is also not clear how large a L is needed as the context set, and how baseline models utilized these context data at meta-test time. Overall it is an interesting idea, but can be improved in clarification of novelty and many methodological/experimental details. \n\n", "strength_and_weaknesses": "Strengths:\n\n1. Addressing non-stationary time-series by learning to adapt the time-series model is an important and interesting direction of research. \n2. The use of INR for the deep time-index model is interesting. \n3. The results demonstrated improvement over a good number of baselines used.\n\n\nWeakness:\n\n1. The presented work is heavily related to switching state-space models [1] and sequential neural processes [2]. In particular, it seems that it can be formulated as a special case of the SNP for learning y = f(t) where Bayesian meta-learning is learned to adapt f to observed frames from the lookback window for prediction in the forecasting horizon. Neither of these two works were discussed in the paper. Relations with these two works, and comparisons in terms of performance, should be provided. \n\n2. By formulating the meta-learning with optimization-based approaches, the model needs to be optimized to the query set before being used for forecasting. To mitigate this, the paper opted to restrict the optimization to only the last layer of the INR. The effect of this restriction should be demonstrated, empirically at least. In an ideal setting (where resource or time is not a constraint) where the full INR can be optimized to the lookback window each time, how does the performance look like, and how does it compare with the assumption of restricting the optimization to the last layer? Note that, with SNP, this was avoided by formulating the meta-learning with feedforward based approaches. This again stresses the need to compare with SNP.\n\n3. Details on meta-training is needed. It is not clear how the current tasks are defined -- by treating each pair of lookback window and forecasting horizon as a task, it means that the meta-training is looking at a large number of tasks and each task has only one set of context and query sets? Is the task boundary known assumed to be known ahead of time? If it is, it seems to be an unrealistic assumption. It it is not, this seems to be an unconventional setting of meta-learning -- Does the training follow the typical episodic training then? Further, how does the method apply if either the look-back window or the forecasting horizon falls within the transition of tasks boundaries? \n\n4. Experimental details are missing. It is not clear how the baseline models are trained. Since most of them do not use any meta-learning formulation, are they trained on the meta-training set? In that case, the comparison may not be fair as the presented model -- at meta-test time, actually is optimized to the look-back window (while the baseline models are simply applied to the look-back window without optimization). A fair comparison would be to fine-tune the baseline models (after training on the meta-training set) to the same samples (look-back window) used at meta-test time. If the argument is that these models cannot be fine-tuned this way (since they are history-value based models), at least in ablation study, the \"-RR\" version need to be fine-tuned at meta-test time.\n\n5. Since the look-back window represents context set in meta-learning, its size may have an important effect in \"optimizing\" the base model. The value of L in synthetic experiments seems to be large, and was not specified in real-data. Please add such details, and provide experimental evidence about the effect of the size of the context set L on meta-learning.\n\n6. While the paper was heavily motivated for better learning non-stationary time series, the experimental evaluation is limited in demonstrating how or whether the proposed solutions achieved the stated goal. In synthetic experiments, there lack details on how many tasks were used to meta-train the model and how many tasks were used in meta-testing. Since tasks/segments are generated with random sampling of the parameters, it'd be good to get a sense of the distance between the meta-train and meta-testing tasks. Finally, it was stated that \"A total of 400 time steps are sampled, with a lookback window length of 200 and forecast horizon of 200.\" In Appendix D, it was then stated that \"each function/task consists of 400 evenly spaced points\". So assuming each context-target set pairs are 400 time points, it was not clear how many such samples of length 400 were used. It was also not clear how does the forecasting work in such segment of 400. Does the model take 200 context samples and forecast for 200 context samples, and then it moves to the next window of 400? i.e., the task boundary is assumed to be known?\n\nIn real data experiments, it is understandable that the truth about the \"non-stationary\" nature of the data is not always available, but to the extent it's possible, it'd be desirable to say some level of analyses that link the model performance with the \"non-stationary\" nature of the underlying time series. \n\n\n\n7. The methodology is presented in a general fashion for forecasting m-dimensional observations. It'd be good to understand to what extend the value of m could be, i.e., what types of observations can be modeled by the presented method. Are we looking at multivariate data with relatively lower number of dimensions, or are we looking at image series?\n\n\n\n\n\n\n[1] Variational Learning for Switching State-Space Models, Zoubin Ghahramani and Geoffrey E. Hinton, Neural Computation 12, 831–864 (2000)\n[2] Sequential Neural Processes, Gautam Singh, Jaesik Yoon, Youngsung Son, Sungjin Ahn, NeurIPS 2019\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ME-L0GQTkz3", "reviewer": "Reviewer_NCFZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work developed light-weighted ts modeling, time-index meta learning scheme for time series forecasting.  ", "summary_of_the_review": "Need to better highlight the motivation on why Deep-time class is considered over historical based classes, and why meta learning helps with theoretical analysis or strong intuition. The writing can be enhanced significally especially in problem formulation. More solid experiments beyond multivariate modeling can be done to demonstrate the need of this method.", "strength_and_weaknesses": "## Strength\n- enhanced deep-time class of method via INR and \n- extensive multivariate real data experiments and ablation study\n\n\n## Weakness\n- clearly define what class of non-stationrity thei work target. It just generally mention covariate shift, conditional distribution shift, concept drift without formal definition. The figure 1 is not sufficient, except visuallizing locally-stationary (as high-level concept)\n- need to provide more direct evidence on 'whereas existing neural forecasting methods, which are historical-value models, are unable to take full advantage of this formulation,' in the introduction\n- need to clarify 'include having a stronger smoothness prior' in the abstract\n- In figure 2, need to specify the method of naive deep-time method. At the same time, usually, historical-value based model can capture such a seonsonalty well.\n- In section 2, what is the use of forecaster $h$?. the $\\mathcal{L}$ is poorly defined, seems to $R^m \\rightarrow R^m$ in eq (1) , which is different in problem formultation. \n- What is $\\Theta, \\Phi$? how do the author decide? is it depending on model types? any rule of thumb? it is too general and vague with too much degree of freedom and little intuition.\n- meta learning framework seesm to be nothing but a fitting some part of parameters like hyperaparemters in additional validation datasets. why this is specifically robust to non-stationary data?\n- why only applied for long-term and multivariate forecaster? what about medium-term forecasting and univariate (global) forecasters  like DeepAR, MQ-RNN, ConvTrans, etc.\n- lack of literature study on distribution shift, non-stationarity context. For example AdaRNN https://arxiv.org/abs/2108.04443", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "5VC8O8bHPb9", "reviewer": "Reviewer_3pqc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new deep learning model for forecasting under non-stationary conditions – utilising a network that builds global representations of random Fourier features and projects it using meta-learning framework that recalibrates the output layer with each window.", "summary_of_the_review": "I do think the idea and intuition behind the model is compelling, but would require improvements in experiments and benchmarks to be more convincing for me.", "strength_and_weaknesses": "Strengths\n---\n\nAs neural networks notoriously underperform in the presence of covariate shifts – which are common in non-stationary time series datasets – the creation of methods that allow neural networks to adapt to changing regimes is extremely important. The use of meta-learning here is an interesting approach, and the out-performance of other state-of-the-art transformers makes it a promising method.\n\nWeaknesses\n---\n\nHowever, I do have several concerns regarding experimental evaluation, in particular:\n\n1.\tLack of hyperparameter optimisation – while the paper does show out-performance across benchmarks, we note that all the hyperparameters of DeepTime and benchmarks have been pre-specified based on “suggested” settings. Given that optimal hyperparameters can vary largely across time-series datasets – for example smaller models in data-limited regimes vs larger models where data is abundant – the lack of hyperparameter optimisation does beg the question of whether the performance differences here are due to improperly selected hyperparams and not model improvements.\n\n2.\tAre improvements due to differences in features used? – While the use of random Fourier features helps to make the time index range bound (which would otherwise introduce its own train-test mismatch), the final output of the INR would essentially be a set of non-linear seasonal features which repeat over time. Out-performance hence could be due to these seasonal relationships being more prevalent in the datasets used. As such, it would be good to see 1) how properly tuned transformer/tabular models perform when given the same random Fourier features + observations, and 2) whether a rolling linear regression on random Fourier features is sufficient for performance.\n\n3.\tDoes the meta learner handle non-stationarity? – while promising in theory, it is unclear whether there are in fact concept drifts in the time series datasets used for benchmarking. Tests on simulated data that incorporates regime changes over time could be useful here, with performance changes/decays across regimes documented for all models.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject