Meta-Forecasting by combining Global Deep Representations with Local Adaptation
Authors: Riccardo Grazzi
Valentin Flunkert
David Salinas
Tim Januschowski
Matthias Seeger
Cedric Archambeau
Venue: ICLR 2022 Submitted (2022)
Keywords: time-series
meta-learning
closed-form
solvers

Abstract: While classical time series forecasting considers individual time series in isolation, recent advances based on deep learning showed that jointly learning from a large pool of related time series can boost the forecasting accuracy. However, the accuracy of these methods suffers greatly when modeling out-of-sample time series, significantly limiting their applicability compared to classical forecasting methods. To bridge this gap, we adopt a meta-learning view of the time series forecasting problem. We introduce a novel forecasting method, called Meta Global- Local Auto-Regression (Meta-GLAR), that adapts to each time series by learning in closed-form the mapping from the representations produced by a recurrent neural network (RNN) to one-step-ahead forecasts. Crucially, the parameters of the RNN are learned across multiple time series by backpropagating through the closed-form adaptation mechanism. In our extensive empirical evaluation we show that our method is competitive with the state-of-the-art in out-of-sample forecasting accuracy reported in earlier work.

Under review as a conference paper at ICLR 2022
META- BY COMBINING GLOBAL DEEP REPRESENTATIONS WITH LOCAL ADAPTATION
Anonymous authors Paper under double-bling review
ABSTRACT
While classical time series forecasting considers individual time series in isolation, recent advances based on deep learning showed that jointly learning from a large pool of related time series can boost the forecasting accuracy. However, the accuracy of these methods suffers greatly when modeling out-of-sample time series, significantly limiting their applicability compared to classical forecasting methods. To bridge this gap, we adopt a meta-learning view of the time series forecasting problem. We introduce novel forecasting method, called Meta Global- Local auto-Regression in closed-form the mapping from the representations produced by a recurrent neural network (RNN) to one-step-ahead forecasts. Crucially, the parameters of the RNN are learned across multiple time series by backpropagating through the closed-for moredd-frmm adaptation mechanism. In our extensive empirical evaluation we show that our method competitive with the state-of-the-art accuracy reported in earlier work.
(LA),, that adapts to each time series by learning
-ample forecasting
INTRODUCTION
Time series (TS) forecasting is of fundamental importance for various applications like marketing, customer/inventory management, and finance (Petropoulos et al., 2020). Classical examples are forecasting the number of daily sales of product over the next few weeks or the energy production needs in the next few hours. Accurate TS forecasting results in better down-stream decision making with potentially large monetary implications g....reegerrr et al. (2016); Faloutsos et al. (2019))
From a machine learning perspective, each TS represents forecasting task. Conventional approaches for TS forecasting have typically been local, i.e. each TS/task is modeled independently by a forecasting model with relatively few parameters (see [Hyndman and Athanasopoulos (2018) for an introductory overview). Despite the modest amount of data used train local TS forecasting models, they are effective in practice. They have only recently been outperformed by global deep learning strategies, which jointly train deep neural network on a large set of related TS/tasks. Global models are designed to work well on the set of TS they are trained on, but they perform poorly out-of-sample TS, Le. TS which are not present in the training set. For example, Oreshkin et al. (2020) show that DeepAR (Salnnas al. 2020) trained on the M4 dataset (Makridakis et al. (0200) performs poorly on the M3 dataset. Global- local approaches (Sen et al.] 2019 Smm]][220) Wang al. (019),, such the M4 competition winner (Smyl) 2020), exhibit a greater level of specialization as they learn parameters that are shared by all TS in the training set, as well as parameters specific to each TS. However, global-local models are still not able to handle -ammlle TS both types of parameters are learned jointly on large training set of related TS in a multi-task fashion.
In this work, we tackle the problem of out-of-sample TS forecasting by transferring knowledge from set of TS, called the source dataset, to another set of TS, called the target dataset. We assume that source and target datasets share some underlying structure which makes transfer learning possible, although they may contain TS from different domains. Our work can be seen as an instance of meta-learning (Schmidhuber) 1987 Ravi and Larochelle, 2016 Finn al. 2017). whose goal is to leverage pool of related tasks to learn how to adapt to new one with little data. We will refer to forecasting method performing well in this scenario as a mota-forecasting method Such models
This was called zero-shot transfer learning by Oreshkin al. 2020

---

Under review as a conference paper at ICLR 2022
could in principle be trained large set of TS and still produce accurate and fast predictions when applied to out-of-sample TS, potentially combining the inference speed and accuracy of deep learning models with the ease-of-use of classical local models.
Our moca-fsoreasting method produces one-step ahead forecasts by combining learned representations with differentiable closed-fome adaptation mechanism inspired by the few-shot image classification method proposed by |Bertinetto et al. (2018). Specifically, we propose a class of models which we call Meta Global-Locas Auto-Regressiv models (Meta-GLAR). Meta-GLAR models compute point forecasts for single TS in three steps. First, the TS is passed through representation model to obtain a representation for each time step. Second, local (i.e. TS TS-specific) linear model is learned in closed-form by solving a ridge regression problem mapping the representations to the observed fraction of the TS. Lastly: the local linear model is applied to the glohal representation to compute the final predictions. Crucially, forecasts are computed in the same way also during training, where we backpropagate through the closed-form adaptation step to learn the representation parameters globally, i.e. across multiple TS. Hence, we can learn representation which works well in combination with an efficient closed-form local adaptation.
We use the RNN backbone in DeepAR (Salinas et al. 2020) for the representation. However, one can transform any neural forecasting method in a meta- GLAR one by substituting the last global linear layer with the closed-form adaptation during both training and prediction. We also stress that, since our method adapts locally also during training, it is fundamentally different from fine-tuning the last layer on each TS, which we show perform significantly worse. This is in contrast recent results for few-shot image classification where fine-tuning the last layer is demonstrated to outperform many modern meta-learning methods (Tian al. 2020
Our main contributions can be summarized as follows.
We propose novel meta-learning method for TS forecasting that is suitable for out-of- sample TS forecasting. Meta-GLAR significantly improves the accuracy on out-of-sample TS forecasting (i.e., transfer setting) over global neural forecasting models such as DeepAR, which employs the same RNN backbone.
Our meta-forecasting method is competitive with classical local methods, for example beating the winner of the M3 competition, as well as NBEATS (Oreshkin et al. 2020), the state-of-the-arr method for out-of-sample TS forecasting, while having substantially fewer parameters than the latter method.
We perform an extensive ablation study which shows that the closed-form adaptation, the RNN backbone, and the use of iterated forecasts during training are needed to achieve the best performance. Furthermore we show that Meta-GL GLAR enjoys similar time and memory costs compared to global one-step ahead RNN with the same backbone architecture, while converging faster and achieving better accuracy.
RELATED WORK
Meta-learning has received considerable attention in recent years and several models have been developed primarily for few-shot image classification. Notable examples are Ravi and Larochelle (2016): Fini et al. (2017): [nell al. (2017); Nichol et al. (2018); (2018). These methods work by adapting to the task at hand before making predictions. Differently from fine-tuning, this adaptation is performed also during (meta) training procedure over large set of tasks to learn the (meta) parameters of the model. However, realistic datasets with the high number of tasks needed by meta-learning methods are rare, hence in commonly used benchmarks like mini-imagenet (Vinyals et al. 2016), each classification task is constructed by randomly selecting small set of classes and related images from a large single-task dataset. This construction is artificial and departs from real-world scenarios. Recently, the work by Tian et al. (2020) showed that training a neural network on the original single-task dataset and fine-tuning only the last layer on new tasks outperforms many modern meta-learning methods for few-shot image classification. By contrast, popular TS forecasting datasets like M4 fit more naturally into the learnig framework, since they already contain a large number of TS/tasks (up to 10sh for M4).
al. (2018); Bertinetto et al.
Our method relies on a differentiable closed-form solver to perform the local (or TS-specifcc) adaptation. Meta-learning is achieved by solving task-specific ridge regression problem that maps
2

---

Under review as a conference paper at ICLR 2022
deep representation to the target TS in closed-form, while the parameters of the representation learned by backpropagation through the solver. Aside from the original application in few-shot image classification (Bertinetto et al. 2018), differentiable closed-form solvers have been used for other few-shot problems like visual tracking (Zheng et al., 2019), video object segmentation (Liu et al. 2020), spoken intent recognition (Mittal et al. (020) and spatial regression (Iwata and Tanaka 2020
while we are not aware of any application in forecasting.
Meta-learning in the context TS forecasting has originally been synonym with model selection or combination of experts (see e.g. Collopy and Armstrong (1922); Lemke and Gabrys) (201)); Talagala al. (2018). This class of methods builds a meta-model which uses TS features to select the best performing model or the best combination of models to apply to a target TS. One drawback of these methods is that the features are usually manually designed from the data at hand and that the same set of features does not transfer well to other applications. [Laptev et al. (2018) train an LSTM neural network (Hochreiter and Schmidhuher (1997) on the source TS dataset and then its last layers are fine- tuned separately on each target TS. This approach overcomes the problems related to human designed features since the input of the network are just the previous TS observations. However, retraining the last layers of the network for each TS can be expensive, especially when dealing with a large number of TS. Additionally, their fine-tuning procedure requires the selection of hyperparameters like learning rate and number of steps of the optimizer. By contrast, our approach adapts only the last linear layer in closed-form, requiring a small increase in compute and no additional hyperparameters compared to standard neural forecasting model, while outperforming the simple fine tuning approach.
More recently, the NBEATS model has shown strong performance both in the standard (Oreshkin al. 2019) and in the meta-learning (Oreshkin et al. 2020) setting. This multi-step ahead method uses a residual architecture (He et al. (016) which takes past observations of a single TS as input and outputs point predictions over the whole forecast horizon. Thanks to the residual connections, a forward pass of the network allows it to implicitly adapt to the input out- of-sample TS. However, the final performance achieved using a large ensemble and the number of parameters of the model is quite large even when the residual blocks share the same parameters. Our method, although also using ensembles, achieves good accuracy on out-of-sample TS forecasting with significantly less parameters.
Finally, Twata and Kumagai 2020) consider the few-shot TS forecasting setting where each task is formed by a small group of closely related TS. Their method, which combines LSTMs and attention, uses the TS in the support set of the task to compute the one-step ahead forecasts for each TS in the query set. This is different from our approach, where do not exploit the other TS in the target dataset to compute predictions and we view each TS as a separate task. Our method can be extended to the case considered by [wata and Kumagai (2020) by performing the closed-form adaptation on all the TS in the task instead of just on one. We leave this for future work.
PROBLEM FORMULATION
We consider the setting studied by Oreshkin al. (2020), where a forecasting method can learn global parameters on a source TS dataset D& to produce accurate forecasts for an oft-of-sample TS which belongs to another. target TS dataset Dy. The model can only adapt locally to each TS in Dr. i.c. it cannot use information from the other TS in Dy. We view each TS as a task. Hence, our setting fits a meta-learning framework where Ds is the meta-training set and Dr is the meta-testing set.
We will denote single TS, as a tuple (z, x) where = [zi......2 2r] E IR are the (target) observations and x = [x]..... xr] E Txp is the matrix of covariates. We will denote with to E N the split point which divides the context window (or past) 21:ta = [=1,..., =t.). X1:to = [xx,...... ,Xo]] from the forecast horizon (or future) 2t +1.7 = [ 2T). Xto = [x
We also denote with H = T to the length of the forecast horizon. We view each TS as supervised learning task with training set {xxx,2) He and test set [xx, zt ) where an example is the covariates-targe pair (x,, 2.). We assume that the covariates vector X2 contain some of the previous observations 2t-1, 2...... as time-lagged values (or time-lags). Differently from standard supervised learning tasks, cannot assume that the examples are independent due to the temporal dependency.
The goal of TS forecasting is to compute predictions 2 for the observations in the forecast horizon 2tg 1:T using the covariates x and the observations in the context window Zizto In this work
+:7
3

---

Under review as a conference paper at ICLR 2022
focus the accuracy of the predictions which can be measured for example with the sMAPE metric (Hyndman and Kochler 2006)::
200
H
Foo+s
Elo+sl
MAPEE
[D]
H
26o+e + [26
4 META GLOBAL-LOCAL AUTO-REGRESSIVE FORECASTING
The forecasting model we prosose is non-linear, auto-regressive and computes one-step ahead forecasts, i.e. for each time step t the forecast horizon, the model outputs a point forecast as non-linear function of the covariates X1xt and the observations in the context Z1:to. The model generates forecasts over the forecast horizon using iterated (or recursive) forecasts (Salinas et al., 2020). Hence, the model will use new covariates vectors which contain previous forecasts in place of the missing time-lags for the time-steps in the horizon. These new vectors are constructed together with the forecasts in an iterative fashion from the first to the last point in the horizon. By contrast, multi-step ahead approaches like NBEATS compute forecasts over the whole horizon directly.
Starting from he 0 R the Meta Hlball-Local Regresssgee Mleta-GLAR) forecasting model computes point predictions as follows:
h; h(x, h O) V [1:T,
& : : he E (to :::
where he and & are respectively the d-dimensional representation and point forecast at time t. Here, is recurrent function providing an appropriate representation for TS. While in this work we choose to use the LSTM architecture used by DeepAR (Salinas et al. (020), our approach is not limited to it and could use other representations such as the one in Franceschi et al. (2019). C and Wopt are respectively the global and local parameters.
Estimating the local parameters. The local parameters Wopt are learned explicitly and in closed- form on the training set of a single TS/task, which contains the data in the context window, This is done by solving a ridge regression problem having as inputs and targets respectively the observations zit and the representation vectors h
Wopt arg min wERd
w h, =2)² + +
(4)
t=1
(h h1:t
71) h h1:t Zi:to:
(5)
where 7 E R+ is the regularization parameter. Aside from the training procedure, the key difference with a global neural forecasting model is that Wope is a local parameter. Albeit in closed-form, computing (5) requires either a d matrix inversion or solving a linear system and has cost of ((d Note, however, that d is usually small (e.g. 20-50) and the cost of computing the representation vectors h1:t. is usually far higher, especially for large to and when h an RNN.
Estimating the global parameters. Similarly to other neural forecasting models, the global parame- ters (e, 7) are trained instead by solving the following minimization problem on the source dataset Dx using a stochastic gradient method:
min
Σ
C(Zta+1:T, (1o0++11
(6)
(2,x)E23
where L is a regression loss suitable for forecasting (typical choices are (scaled) MAE or sMAPE) and DI is the training split of Ds. We note that the loss is computed over the test set of each TS/task in DY. which contains the data in the forecast horizon, not used to compute Wopt This prevents the overfitting of Wopt and allows to learn global parameters which generalize well to each TS/task.
We use iterated forecasts also during training even if the true target observations in the forecast horizon are available in this phase and show that this increases the performance, in line with Bengio et al. (2015), even without scheduled sampling. Dir is typically constructed by removing the last H observations and covariates from each TS in Dş- The last H observations are used to test the in-sample performance of the method on Ds-
4

---

Under review as a conference paper at ICLR 2022
Gradient Contributions. Recall that the predictions (t tu) given by & by. where Wopt the solution to the least squares problem for the past time points S to and thus depends on the past representations h1:to- Then, the gradient of the loss with respect to the global parameters O has two components:
dc
dc
dh,
dh, dWapt dh, h
do
=
Σ 1=to++1
de
W opt +
de
Hence, the first component is the direct gradient of the loss with respect the representations in the prediction range 1o+ The second component is an indirect contribution from the representations in the context h 1:to: which influence Wopt-
Since Wopt is obtained in closed-form and differentiable, we can compute the gradient of the loss using automatic differentiation. Replacing the Mean Squared Error in (4) with another loss, such as the one used in (6). usually leads to Wope not being in closed-form: However, if (4) is still a convex minimization problem or conic program, our approach still applies (Agrawal al., 2019).
e.g., by using iterative reweighted least squares. If instead 4 becomes non-convex, one could still backpropagate through few gradient descent updates as in Raghu et al. (2019).
Intuitively, the RNN is trained to generate time dependent (and TS specific) features h, that work well for the local linear forecasting model. In principle, this design allows the model both learn the common traits of the TS in Ds and to adapt to individual TS, combining the properties and benefits of both global and local models. Additionally, since Wopt is learned explicitly on each TS (also in 'D)) before computing predictions, the model should adapt better to potential dataset shifts between Ds and Dr. This is in contrast to other global-local models like the M4 competition winner (Smyl 2020). which learn the local parameters together with global ones only during training.
EXPERIMENTS
In all our experiments, train source dataset and measure predictive performance target dataset. We stress that when source and target dataset are not the same, the model computes the forecasts for each TS in the target dataset without access the other TS in the target dataset. This is different from domain adaptation settings where source and target dataset are available at training time. To compare with NBEATS, we use the same couree-target combinations reported by Oreshkin et al. (2020): we train on one of the M4 datasets and use datasets with same frequency as target datasets whenever possible (see Table in the Appendix). M4 contains a large number of diverse TS, thus there is higher chance for it to share some properties with other datasets. An ideal source dataset would contain a very diverse set time series from many real-world applications from different domains (and different frequencies).
Transfer from M4. In Table for both Meta-GIAR and DeepAR and for each frequency of the M4 dataset repeat the following model selection. We evaluate 200 random hyperparameter configurations and pick the 10 configurations with the lowest sMAPE on a subset of 8KK training TS. The hyperparameters for the random search are described in the Appendix (Table K). A single Meta- is competitive with Theta and ARIMA and always outperforms DeepAR. Ensembling can improve performance for both DeepAR and Meta-GLAR. An ensemble of 10 Meta-GLAR models Meta-GLAR-enns 10) outperforms NBEATS, which consists in an ensemble of 90 models, on ELECTR, as well DeepAR trained on the target dataset on ELECTR and M3. NBEATS is still superior on TRAFF and TOURISM while the gap is much closer on M3. As shown in Table our method has a substantially lower global parameter count than NBEATS: from 35 to 1000 times less parameters for the monthly frequency. We did not notice performance improvements of our method by increasing the number of layers or the hidden dimensions. This a common behavior of ooe-step-ahees RNN based forecasting models (see e.g. (Salinas et al. (2020)). whose accuracy does not increase much beyond certain model size. By contrast, NBEATS is a multi-step ahead residual network that computes sequence of predictions in a single forward pass and hence benefits from a larger number of parameters. More detailed results are in Appendix [.
GLAR model MMta-GGLAR-topll
Ablation analysis. In Figure I perform an ablation analysis to evaluate the benefits of different elements of our method Mleta+RNN+ITV These are the closed-form adaptation (Meta), the RNN network for the representation and the use of iterated forecast during training (ITF). We perform
5

---

Under review as a conference paper at ICLR 2022
Table 1: Out-of-sample TS forecasting. Global parameters of all models trained M4 except for the entries with target in the name, which are instead trained on the target dataset. We evaluate the 10 selected models both individually (top10), reporting mean 95% confidence interval of the performance metric, and as an ensemble which takes the median of the predictions (ens10). Best performance for each column is in bold and does not take into account the models with target the name since they use all the TS in the target to learn the global parameters.
* as reported in Salinas et (2200) SeeeepR-ttagget and Oreshkin et al. (2020) (all the others).
ELECTR (ND)
TRAFF (ND)
M3 (sMAPE)
TOURISM (MAPE)
Theta'
0.080
0.170
13.015
20.878
ARIMA"
0.067
0.145
14.005
20.959
DeepAR
0.070
0.170
12.671
19.276
NBEATS-tarre -target
0.067
0.114
12.374
18.523
DeepAR
0.150
0.360
14.767
24.787
NBEATS
0.090
0.150
12.441
18.828
DeepAR-top10
0,211 0.037
0.407 + 0.064
13.297
+ 0.101
23.920 + 0.631
DeepAR-ens 10
0,199
0.321
12.884
22.889
Meta-GLAR GLAR-top10
0.079 0.006
0.245 + 0.014
12.746
+ 0.048
21.254 + 0.247
Geta-GLAR--nsII
0.067
0.188
12.509
20.095
M4 monthly (trained M4 monthly)
M3 monthly (trained on M4 monthly)
14.4
13.2
14.2
SMAPE
13.0
sMAPE
14.0
13.8
12.8
13.8
12.6
13.4
Meta+ RNN+ITF
Meta+ Meta+FF+ITT FF+ITF
Meta+ Line!/F
Metar RNN
RNN
RNN-ITF
Meta+RNN-ITF FF+ITF Meta+ Meia-FF4ITT Meta*LinHITT
Meta+RNN RNN
RNNHITF
Figure 1: Ablation study for Meta-GLAR (Meta+RNN+HTF), Ablated elements the representation function, which can be Linear (Lin) feed-forwart (FF) or a recurrent (RNN) neural network, the dooeed-frrm adaptation from meta-learning (Meta) and iterated forecasts during training (ITF). Each box plot shows statistics for the 10 over 100 random search runs with lowest sMAPE computed on random training TS.
combinations of the following ablations. Replacing the closed-form adaptation with global linear layer (no Meta in the name). Using the observations in the horizon compute predictions during training instead of iterated forecasts (no ITF in the name). Replacing the RNN with stateless feed-forward model (FF) and linear (Lin) model. The results show that all the three ingredients are necessary to achieve the best performance. In addition, we note that in this scenario, using iterated forecasts during training without the closed-form adaptation (RNN+ITF) performs worse than the auto-regressive RNN model. Replacing the last global linear layer with the closed-fome foreddffom adaptation gives our model more "memory" even when we use a ferw-forward or lincar backbone, since the last linear layer is computed using past representations in the context, while autoregressive stateless model only uses the last representation the context. Hence, it is interesting to see that the RNN backbone performs the best even in this setting. Ablation the hourly frequency is in Appendix A.2
Comparison with fine-tuning. Recently, Tian et al. (2020) criticized meta-learning approaches in computer vision and achieved state-of-the-art results for few-shot image classification only by

---

Under review as a conference paper at ICLR 2022
24
RNN train
Meta-Gl Bett--LAAR train
Meta SaAt--GAARR train
NBEATS-SH train
22
RNN test
Meta-GLAR test
Meta-GLAR test
NBEATS-SH
15
18
14
16
14
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.5
2.0
2.5
3.0
training time (hours)
# of training examples (TS slices)
1e6
Figure 2: mean and max min (shaded regions) sMAPE over training computed on 8K random time series from the train (dashed lines) and test (solid lines) sets of the source dataset (M4 monthly). Results are for runs of each method where we only varied the seed controlling the network initialization and the training minibatches. All methods are optimized using the same loss function (sMAPE). Training consists in 25 ADAM steps with minibatch size 128 for Meta-GLAR and RNN and 3500 steps with minibatch size 1024 for NBEATS-SH.
Table 2: Number of global parameters of the single models for the monthly frequency, From the GluonTS implementation. NBEATS-SEE and NBEATS-NSS contain 30 residual blocks with (SH)
and without (NSH) shared weights as described in Oreshkin al. (2020).
Meta-GLAWRNN/DepAR NBEATS-SH NBEATS-NSH
2 25K
RS 880K
R3 25M
learning a good representation and then using with a local model at prediction time, avoiding any local learning or adaptation during training. We explored this approach by training a standard global RNN model on the source dataset. For prediction on the target dataset, we then kept the network weights fixed, but replaced the last layer with the closed-form adaption layer (with 7 1), corresponding to the setting of Tian et al. (2020) of fixed global representation trained without meta learning. This method reaches median sMAPE of 14.16, 15.40 (ITF) and 14.13, 15.74 (ITF) for M4 and M3 monthly respectively, which is far higher than all the other methods in Figure T While used the closed-form adaption layer in experiments, the solution is equivalent to training the last layer from scratch using SGD. We also tried a variant of this experiment where we use the regularization 7|| Wgloball || where Wglohal are the parameters of the global last layer. This is equivalent to gradient descent fine-tuning of the last layer starting from Wglobal- This variant reaches 14.86 and 16.06 median sMAPE respectively on M4 and M3 monthly (no ITF). These experiments show that in the forecasting setting it is important to meta-learn the representation parameters such that the local model works well, contrast to the results Tian et 2002))
Learning curves. In Figure we compare the performance during training of our method with that of NBEATS with shared weights (NBEATS-SHP SH) and the global RNN method from our ablation analysis. Meta-GLAR and RNN use the same model and training hyperparameters, while NBEATS uses an times larger minibatch size (the same used by Oreshkin et al. (2020)). The gap in performance is large at the beginning of the training. We believe this is due to the local closed-form adaptation, which allows to have greater performance even with random global parameters. We also note that the performance of
is more stable during training and that the training time (on CPU) is only 17% higher than that of the non-adaptive RNN. NBEATS is significantly faster to train: 1.38 hours for 300 epochs with batch size 1024. This is expectd since NBEATS computes all predictions in single forward pass and thus benefits more from parallel computation. The prediction time for Meta-GLAR
We computed the median on the 10 over 100 random search runs and with lowest sMAPE on subset of 8K training TS.
7

---

Under review as a conference paper at ICLR 2022
is only marginally higher than that of the non-adaptive RNN, i... it takes 3:21 minutes instead of 3:19 to forecast the full M4 test set, while NBEATS takes 3:15 minutes (we set the minibatch size to IK for all methods during prediction). Note also that the non-adaptive RNN training and prediction times similar to DeepAR since both have the same RNN backhone.
5.1 EXPERIMENT DETAILS
Benchmark datasets. We use five datasets in our experiment: ELECTR contains hourly time series of the electricity consumption of 370 customers (Dua et al. 2017). TRAFF time series measures hourly occupancy rate, between O and 1, of 963 San Francisco car lanes (Dua et al. (017). TOURISM collection of yearly, monthly and quarterly series of indicators related to tourism activities (Athanasopoulos et al. 2011). M3 and M4 are two collection of time series that were used in recent
forecasting competitions (Makridakis and Hibon 2000 Makridakis et al. 2020p. M3 comprises 645 yearly, 756 quarterly, 1428 monthly as well as 174 IS without time granularity (M3-OTHER). Finally M4 contains 100K time series in total of yearly, quarterly, monthly. weekly, daily and hourly granularities. Both M3 and M4 contain heterogeneous TS types from business, financial and economic domains, All datasets are accessed through the GluonTS library (Alexandrov et al., (020). Following Oreshkin al. (0200), to compare with previous result from the literature we report the commonly used metrics for each dataset: sMAPE for M4 and M3; ND, which corresponds to the P50 loss in Salinas et al. 2020), for ELECTR and TRAFF; and MAPE for TOURISM.
Baselines. We report the top baselines from Oreshkin et al. (200)) namely Theta (local) (Assi-
makopoulos and Nikolopoulos] 2000), auto-ARIMA (local), NBEATS and DeepAR (Salinas et al. 2020). We also report the results from DeepAR from [Salinas et al. (2020) and train and evaluate the DeepAR model using procedure similar our method. We recall that local methods do not require training on the source dataset and their parameters are estimated directly on each TS of the target dataset. Sine-ttning the whole network on each TS and other meta-leanan primarily in computer-vision like MAML (Finn et al. 2017). REPTILE (Nichol et al. 2018) or ANIL (Raghu et al. 2019) can in principle be applied to TS forecasting. However, all require an additional hyperparameter: the number of steps of gradient descent used to learn the local parameters, which can't be easily meta-learned. Furthermore, fine-tuning the whole network on each TS, MAML and REPTILE have as local parameters all the network weights, hence they would be significantly slower and occupy more memory than our method during prediction and, for MAML and REPTILE also during training. On the other hand, ANIL adapts only the last layer and thus should have similar time and memory cost than our method. Applying these methods to time-series forecasting is an interesting future work.
methods used
Models input/output. Like many RNN forecasing models, we provide previous observations as input to the models (time-lags). The lags for given time-frequency are detailed in Table(7of the appendix. In addition, we scale/descale the input time-lags and final output of the model by dividing/muliplyiing by a scaling coefficient which is the average of the absolute value of the observations in the context window | observation in TS as additional covariates. Forecasts below zero are set to zero, since all the datasets contain positive values. The NBEATS model applies a similar scaling but it takes as input to the residual network just the observations the context window, without using covariates. In our setting it is counterproductive to use the index of the TS as a covariate. However, handling cases where source and target datasets have other different covariates is an interesting future work.
We also use the log of the scaling coefficient and the "age", i.e. the distance from the first
Network architectures. Meta-GLAR employs an RNN network to compute the representation. This network is composed of two LSTM layers with hidden dimension between 20 and 50 (see Table followed by linear layer, which allows to control the dimension of the linear adapation layer independently of the hidden dimension of the RNN. The last LSTM layer has also a residual connection, i.e. the final output is the sum of the output and the input of the layer. Zoneout (Krueger let al. 2016), a form of dropout which works well for RNNs, is applied to both layers with rate 0.1. The same backbone is used for the representation of ReepARf and the RNN auto-regressive model in the ablation analysis, although without the final linear layer which would be redundant in this
Oreshkin et al. (2020) use an additional datasct called FRED as the source dataset. However, this dataset is not publicly available and can only be retrieved by crawling ever changing website over the course of days. Since this procedure does not result a reproducible datasct, we do not consider it here.
"This is a common practice, but becomes crucial when dealing with datascts coming from different domains. We used the default implementation of GluonTS.
8

---

Under review as a conference paper at ICLR 2022
case. For DeepAR and the auto-regressive RNN model, the hidden dimension of both LSTMs is the same and equal to the dimension of the representation. The feedforward network (FF in Figure I is composed by two dense layers both followed by ReLU with the same hidden size equal to the dimension of the output. For NBEATS (Figure 2 we used the implementation in Gluon'T' with sMAPE as loss function, 30 residual blocks with shared weights, and context length equal to 36.
Training procedure. Models are trained using a custom version of the Gluon'TS trainer which uses the ADAM (Kingma and Ba 2014) optimizer with weight decay and gradient clipping parameters set to 10- and 10 respectively. At the end of training, the final model is the averge of the last 5 checkpointed models (models are checkpointed every 50 ADAM steps). We do not use any early stopping strategy. The loss function used by our method is the sMAPE divided by 100. Learning rate, minibatch size, and number of steps are selected via random search. Minibatches for training are constructed by randomly selecting slices from the time series of the source dataset (Salinas et al. 2020). The slices are split into a context window that is used for fitting the linear adapation model and prediction window for which the fitted local model is used to generate predictions. The loss for backpropagation is computed on the prediction window. The context window plus prediction window may be longer than some of the shorter time series in the training or test dataset. In this case, the time series is left- seft-padee with Os to fill the context window and the corresponding time points are not included when fitting the local adaptation layer. This preprocessing is standard in Gluon'TS and is applied to all methods that we train. To make sure the model can handle such short time series at prediction time, we also include such cases from the training dataset. A hyperparameter controls the minimum number of observations in the context window.
Hardware. All models are trained and evaluated on a cloud ml.c5. 4xllage instance with 16 CPU cores and 32GB of RAM. Training and evaluating Meta-GLAR takes around 3-6 hours.
6 CONCLUSION AND FUTURE WORK
In this work we proposed a novel meta-learning approach for time series forecasting models called Meta-GLAR, Meta-GLAR is trained on a source dataset and can then generate accurate forecasts for new time series that may come from domain different from the one of the source dataset. This is achieved through a combination of a global deep representation and a local closed-form adaptation layer. Crucially, during training the local closed-form adaption is differentiable, such that the deep representations are learned across multiple time series in a meta-learning fashion, by backpropagating gradients through the solution of the closed-form solver at training time.
We evaluated Meta-GLAN on out-of-sample TS through an extensive empirical study. Results were competitive with NBEATS, the current slate-of-the-att method while requiring fewer parameters. Our model was also competitive with classical local methods (e.g. beating the winner of the M3 competition) and outperforms global LNNN-ased method with similar architecture.
While Neta-GLAR did not achieve state-of-the-ar results on all datasets. it introduces a novel approach to neural forecasting that draws from meta-learning and global-local forecasting. We showed that including differentiable local adaptation layer in neural forecasting models can improve forecast accuracy and transfer capabilities. This approach could be used in combination with other neural forecasting backbones such as transformers (Lim et al. 2019 Li ct al. 2019).
In contrast to recent results in computer vision (Tian et al., 2020), we showed that differentiable adaptation during training can perform well on forecasting problems. We believe that this is due to the different nature of forecasting datasets, which usually contain a large number of TS/tasks, while few-shot image classification benchmarks are constructed from large single-task dataset.
We also believe that the present work is an important step towards neural models that combine the ease-of-use of classical forecasting methods (directly used single time series), with the accuracy gains of deep learning models. However, the scope of applicability of meta-learning forecasting methods remains unclear. For example, further work is needed to remove the assumption that target and source dataset have the same frequency (e.g.. daily, monthly, weekly), or to deal with source and target dataset having different dataset or domain specific covariates.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
To ensure reproducibility of our experiments we evaluated our method on publicly available TS datasets and discussed in detail the training procedure, model parameters, preprocessing, and hardware used in Section 5.1 Furthermore, we state in the Appendix the time lags (Table 7 used by the model, the hyperparameter ranges for the the model selection procedure used in the majority of the experiments (Table g) and the source- surce-target pairs (Table 6). We also provide the source code for the experiments in the supplemetary material.
REFERENCES
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and Zico Kolter. Differentiable convex optimization layers, 2019.
Alexander Alexandrov, Konstantinos Benidis, Michael fohlle-SSchneider Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Türkmen, and Yuyang Wang. Gluonts: Probabilistic and neural time series modeling in python. Journal of Muchine Learning Research, 21(116):-66 2020. URL http:/mwr.rrrggpppp/ httpappppss////1apae
html
Vassilis Assimakopoulos and Konstantinos Nikolopoulos. The theta model: decomposition approach to forecasting. International journaa of forecasting. 16(4):521-55 530, 2000.
George Athanasopoulos, Rob J Hyndman, Haiyan Song, and Doris C Wu. The tourism forecasting competition. International Journal of Forecasting, 27(3):822-844, 2011.
Leee C Baker and Jeremy Howard. Winning methods for forecasting tourism time series. International Journal of Forecasting, 27(3):850-852, 2011.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks, 2015.
Luca Bertinetto, Joao Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. arXiv preprint arXiv: 2018.
Fred Collopy and J Scott Armstrong. Rule-based forecasting: Development and validation of an expert systems approach to combining time series extrapolations. Management science, 38(10): 1394-1414, 1992.
Dheeru Dua, Casey Graff, et al. Uci machine learning repository. 2017.
Christos Faloutsos, Jan Gasthaus, Tim Januschowski, and Yuyang Wang. Classical and contemporary approaches to big time series forecasting. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD '19 New York, NY, USA, 2019. ACM.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1135. PMLR, 2017.
Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable represen- tation learning for multivariate time series. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buch E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. htRR:tttss://// http:://///ppp..o.iiss/e. neur meuripi ips .pdf
cc/paper/201 22e.a//eSSScee222222222222322
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()) 1735 1780, 1997.
Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018
10

---

Under review as a conference paper at ICLR 2022
Rob J. Hyndman and Anne B. Koehler. Another look at measures of forecast accuracy. International Joumna of Forecasting, 22(():679-66) 2006. htll:htpp///ieeeeee reppc. html
Antfor/v2Zz wttp:////2222222111220
Tomoharu Iwata and Atsutoshi Kumagai. Few-shot learning for time-series forecasting. arXiv preprint arXiv:2009. 14379, 2020.
Tomoharu Iwata and Yusuke Tanaka. Few-shot learning for spatial regression. arXiv preprint arXiv:2010.04360, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
David Krueger, Tegan Maharaj. János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, and Chris Pal. Zoneout: Regularizing mns by randomly preserving hidden activations. arXiv preprint vrXiv:1606.01333 2016.
Nikolay Laptev, Jiafan Yu, and Ram Rajagopal. Reconstruction and regression loss for time-series transfer learning. In Proceedings of the Special Interest Group on Knowledge Discovery and Data Mining (SIGKDD) and the 4th Workshop on the Mining and LEarning from Time Series (MiLeTS), London, UK, volume 20, 2018.
Christiane Lemke and Bogdan Gabrys. Meta-leaming for time series forecasting and forecast combination. Neurocomputing, 73(10-12)2200-20116 2010.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series 2019.
forecasting. arXiv preprint arXiv:
Bryan Lim, Sercan O Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting, arXiv preprint arXiv: wrXiv:1922.09363 2019.
Yu Liu, Lingqiao Liu, Haokui Zhang, Hamid Rezatofighi, Qingsen Yan, and Ian Reid. Meta learning with differeeniablee closed-form solver for fast video object segmentation. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8439 8446, 2020. doi: 10.1 10991R004444433. 201.9444222099912222
Spyros Makridakis and Michèle Hibon. The m3-competition: results, conclusions and implications. International Journal of Forecasting, 16(4):451477, 00 2000, URL /ttttp://www.... wwweeciiieeeeeeecccccceeoeeeeeee com/ http:///nnc/eariee//////------/0 (bf
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assinakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54-74, 2020.
Ashish Mittal, Samarth Bharadwaj. Shreya Khare, Saneem Chemmengath, Karthik Sankaranarayanan, and Brian Kingsbury. Representation based meta-learning for few-shot spoken intent recognition. Proc. Interspeech 2020, pages 4283-4287, 2020.
Pablo Montero-Manso, George Athanasopoulos, Rob Hyndman, and Thiyanga S Talagala. Fforma: Feature-based forecast model averaging. International Journal of Forecasting, 36(1 36(1):86-99) 92, 2020.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint Xiv::/803.0299, 2018.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 2019.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. Meta-leamiing framework with applications to zero-shot time-series forecasting, arXiv preprint arXiv:2002.02887, 2020.
Fotios Petropoulos al. Forecasting: theory and practice. arXiv preprint
2020.
11

---

Under review as a conference paper at ICLR 2022
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Representations, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020.
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. on learning how to learn: The meta-meta-metta.. Diploma thesis, Technische Universitat Munchen, Germany, 1987. html
URL http:://wwww..iiiia..chaaaaeeeer
Matthias W Seeger, David Salinas, and Valentin Flunkert. Bayesian intermittent demand fore- casting for large inventories. In D. Lee, M. Sugiyama. U. Luxburg, Guyon, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Asso- ciates, Inc., 2016. URL
neurips /ttprrrppp.ccppppp////////////
Rajat Sen, Hsiang-Fu Yu, and Inderjit Dhillon. Think globally, act locally: deep neural network approach to high-dimensional time series forecasting. Proceedings of the 33rd International Conference Neural Information Processing Systems, pages 4837-4846, 2019.
Slawek Smyl. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting, International Journal of Forecasting, 36(1):75-85, 2020.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. arXiv preprint arXiv: 703 0775 2017.
Evangelos Spiliotis, Vassilios Assimakopoulos, and Konstantinos Nikolopoulos. Forecasting with a hybrid method utilizing data smoothing, variation of the theta method and shrinkage of seasonal factors. International Journal of Production Economics, 209:92 102, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208, 2018.
Thiyanga S Talagala, Rob J Hyndman, George Athanasopoulos, et al. Meta-learning how to forecast time series. Monash Econometrics and Business Statistics Working Papers, 6:18, 2018.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? arXiv preprint 1555 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-3638, 2016.
Yuyang Wang, Alex Smola, Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski. Deep factors for forecasting. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceed- ings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 6607-6617. PMLR, 09-15 Jun 2019.
Linyu Zheng, Ming Tang, Jinqiao Wang, and Hanqing Lu. Learning features with differentiable closed-form solver for tracking. arXiv preprint arXiv: 10414, 6, 2019.
12

---

Under review as a conference paper at ICLR 2022
Appendix
A ADDITIONAL RESULTS
A.1 TRANSFER FROM M4 (RESULTS BY FREQUENCY)
Table S contain the results for the models in Table divided by frequency for M3, TOURISM and M4, together with additional baselines. The rows of the tables are divided in 3 sections. The first one contains local models, the second one models with global parameters and the third one models with global parameters which we trained and evaluated.
Meta-GLAR outperforms DeepAR (both trained on M4 using the same random search and evaluation procedure) overall on ELECTR, TRAFF, TOURISM, and M3 and also on most subdatasets of TOURISM and M3. Our method achieves the state of the art on ELECTR, reaching the same ND of ARIMA and NBEATS trained on electricity. Meta-GLARR-en// outperforms all the local methods on M3 and is second only to NBEATS. On Tourism, our method performs better than Theta and ARIMA although falls behind the local method LeeCBaker, which is tailored tourism TS On TRAFF has higher ND than NBEATS and local methods but improves substantially over DeepAR trained on M4-HOURLY.
A.2 ABLATION ANALYSIS ON M4-HOURLY
In Figure Bl we report the results of the ablation analysis on the hourly frequency. Note that in this case, the source dataset, M4-HOURLY, has only 414 TS. We perform combinations of the following ablations. Replacing the closed- forredffomm adaptation with global linear layer (no Meta in the name). Replacing the closed-form adaptation with a global linear layer only at training time, which is equivalent to fine-tuning the last layer (ADA in the name). Using the observations the horizon compute predictions during training instead of iterated forecasts (no ITF in the name).
We note that our method outperforms the autoregressive RNN both for ELECTR and TRAFF. Comput- ing the adaptation only during prediction can be advantagcous on TRAFF but significantly degrades the performance on M4-HOURLY. Using iterated forecast during training is beneficial in most cases. We also report the statistics for all the random search runs of the ablations studies in Figure 4 Our methods performs the best most cases even with random hyperparameters.
B ADDITIONAL EXPERIMENTS DETAILS
B.1 ADDITIONAL BENCHMARK METHODS
We Include the following additional benchmarks methods for an improved comparison. LeeCBaker (Baker and Howard 2)11), competitive method for tourism TS EXP (Spiliotis al., 2019). the state of the art local method on M3. The M4 competition winner (M4 winner) (Smyl 2020) and the second best entry (Best MI/TS) (Montero-Manso al. 2020).
B.2 SOURCE TARGET PAIRS
In Table we describe the cource-target combinations that for all our experiments. The same combinations are considered by Oreshkin et al. (2020). We use the model trained on M4-QUARTERLY compute forecasts M3-OTHER because they have forecast horizons of the same length. ELECTR and TRAFF contain hourly data, hence they are matched with M4-HOURLY.
B.3 METRICS
To measure the performance of the methods, we use the following metrics.
HD
1
200
1zto+
2
sMAPE =
(8)
D
H
Ztotsl + 12t
(
13

---

Under review as a conference paper at ICLR 2022
Reo ti NDp E(Ea)eD Eisl
(9)
E(E,))DD Ei=l
1260ts
1
100
HD
Etots
Stotil
MAPED = D HD
(10)
|26++1
(2,a)ED
where DD the TS dataset (for M3, M4, TOURISM consider each frequency separately). Hp is the corresponding forecast horizon and [D] is the number of TS in the dataset.
We compute aggregate metries as follows.
sMAPEM: =
Σ DEM3
HD [D]
X
Σ
HD x sMAPE
(11)
DEMB
MAPE TOURISM
Σ
HD X [
Σ
HD MAPEd
(12)
DETOURISM
DETOURISM
where M3 (TOUR-YEAR, TOUR-MONTH).
[M3-YEARLY ..... M3-OTHER)
and
TOURISM
Table 3: sMAPE on M3. All models except local ones are trained on M4 except the ones with target in the name, which are instead trained on the target dataset.
M3
M3-YEARLY
M3-QUART
M3-MONTHLY
M3-OTHER
Theta" ARIMA" 14.005
13.015
16.900
8.960
13.850
4.410
17.730
10.260
14.810
5.060
EXP"
12.713
16.390
8.980
13.430
5.460
DeepAR-target
12.671
13.330
9.070
13.720
7.110
NBEATS-ttrget
12.374
15.930
8.840
13.110
4.240
DeepAR"
14.767
14.760
9.280
16.150
13.090
NBEATS
12.411
15.250
9.070
13.250
4.340
CeepAR-toplle
13.297
16.214
+ 0.180
9.429 + 0.065
14.235
+ 0.142
4.674 + 0.079
DeepAR-ens10
12.884
15.640
9.228
13.782
4.536
Meta-GLAR-tople
12.746
15.842
= 0.267
9.282
0.132
13.513
0.077
5.033
0.390
Meta-GLAR--ensl
12.509
15.305
8.970
13.335
4.848
Table MAPE on TOURISM. All models except local ones are trained on M4 except the ones with target in the name, which are instead trained the target dataset.
TOURISM
POUR-YEAR
TOUR-QUART
TOUR-MONTH
Theta" ARIMA" 20.959
20.878
23.450
16.150
22.110
28.030
16.230
21.130
LeeCBaker'
19.350
22.730
15.140
20.190
DeepAR-targer
19.276
21.140
15.820
20.180
NBEAATS-tageet
18.523
21.440
14.780
19.290
DeepAR"
21.787
21.510
22.010
26.640
NBEATS"
18.828
23.570
14.660
19.330
DeepAR-toop0
23.920 0.631
26.233 + 0.585
17.724
0.362
25.784 +: 0.967
DeepAR-ens
22.889
25.174
17,000
24.640
Meta-GIAR
21. 21.254 + 0.247
25.806
+ 0.906
18.072
+ 0.658
21.418
+ 0.390
Beta-GLAR-enne
20.095
24.469
16.802
20.343
14

---

Under review as a conference paper at ICLR 2022
Table 5: sMAPE on M4 in the non-transter setting. All models are trained on the target dataset.
M4-YEARLY
M4-QUARTERLY
M4-MONTHLY
M4-HOURLY
Best ML/TS"
13.528
9.733
12.639
M4 winner'
13.176
9.679
12. 12.126
DeepAR
12.362
10.822
10.705
NBEATS
12.913
9.213
12.024
DeepAR-top10
14.100
0.109
9.942
0.040
13. 160 + 0.150
8.982 + 0.174
DeepAR-ens
13.658
9.765
12.770
8.515
Meta-GLAP GLAR-top10
14.026 + 0.119
9.993 + 0.036
12.670 + 0.061
8.721 + 0.110
Meta-Gi Peta-GLAR-ensID
13.524
9.758
12.523
7.702
M4 hourly (trained M4 hourly)
electricity (trained M4 hourly)
0.40
16
0.35
0.30
SMAPE
14
0.25
12
0.20
0.15
10
0.10
Meta+RNNNITTE Meta+RNN
RNN-ADA ITE RNA+ADA
RNNHITF RNN
Meta+RNNHITE
Meia+RNN
RNN+ADA RNN+ADA
RNNHITF RNN
traffic (trained hourly)
0.6
ND
0.4
0.3
Meta+RRNNiITE RNNN
Meta+RNN
RNN+ADA TTE
RNN+ADA
RNN-TTF RNN
Figure 3: Ablation study for Meta-GLAR (Met++RNN+ITF) on M4-HOURLY, Ablated elements the closed-form adaptation from meta-learning (Meta), which also used only during prediction (ADA) and iterated forecasts during training (ITF), Each box plot show statistics for the 10 over 100 random search combinations with lowest sMAPE computed on 8K random training TS.
15

---

Under review as a conference paper at ICLR 2022
M4 monthly (trained on M4 monthly)
M3 monthly (trained on M4 monthly)
17
16
SMAPE
sMAPE
16
15
14
13
Meta+ MNN++
Meta+FF+ITE
Meta+LinMTTE
Meta+RNN RNN
RNN-ITF
Meta RNN+TTF
Meta+FF+IT
Meta+Lin+
Meia+RNN RNN
RNNHTTF
M4 hourly (trained M4 hourly)
electricity (trained on M4 hourly)
0.8
25.0
22.5
20.0
SMAPE
17.5
0.3
15.0
12.5
0.2
2008
10.0
Meta+RNN-ITE Mata-RNN
RNN+ ITEE ADA
RNN+
RNN-ITF
Meta Mola+RNN+4TT RNN+ITF
Meta+RNN
RNN+ADA-JTE RNN+ADA
RNNHITF RNN
traffic (trained hourly)
ND
0.4
8
0.2
Meta+RNN-ITT
Meta+RNN
RNN+AAD RNN+ADA
RNN-ITF RNN
Figure 4: Ablation study for Meta-GLAR (Met++RNN+ITFF) The box plots show statistics for all the runs of the random search (no model selection).
16

---

Under review as a conference paper at ICLR 2022
Table 6: Source-targets combination for DeepAR and Meta-GLAR. model per row is trained on the dataset the source column and tested on all the ones in the target column. The length of the forecast horizon follows each dataset preceded by slash. Note that the length of the forecast horizon can differ between source and target.
Source
Targets
M4-YEARLY/6
M3-YEARLY/6, TOUR-YEAR/4
M4-QUARTERLY/8
M3-QUART/8, M3-OTHER/8, TOUR-QUART/8
M4- M
M3- MI3-MONTHLY/I8, TOUR- TOUR-MONTH/24
M4-HOURLY/48
ELECTR/24, TRAFF/24
Table 7: Time lags used by Meta-GLAR, DeepAR and ablations for each time-frequency. These are the defaults in the GluonTS implementation of DeepAR
Frequency
Lags
Yearly
1, 1,2,,,45,6677
2,,,,,,5,6,7,.8.9111, 12,
Quarterly
Monthly
25, 35, 7,8, 51,51,,22 52, 103, 104, 105, 155, 156, 157]
1,2,3,4,5,6,,,111, 12, 12,
1,2,3,4,5,6,7 5,,,,,,,,,,,,,,,,,,, 12,
Weekly
1, 4,5, 6,8,8 13, 14, 15, 21, 22, 27, 28, 29, 30, 31, 56, 84,363, 365, 727, 728, 1091, 1092, 1093]
Daily
2. 3, 2,5,,, 6, 7, 24, 27,47,48 49,71, 72, 73, 95, 96, 97, 119, 120, 121, 143, 144, 145, 167, 168, 169,
Hourly
335, 336, 337, 503, 504, 505, 671, 672, 673, 719, 720, 721
Table 8: Hyperparameters for the random search. The first three entries are respectively the number of steps, minibatch size and learning rate used by the optimizer. Context mult multiplies the horizon length of the training dataset to give the final context length for the model. Representation dim the dimension of the input to the last linear layer of the model (which is the dimension of the representation). Min history length is the minimum number of observations that must be present in the context window of the training TS slices.
Name
Type
Values/Ranges
Number of steps
Categorical
{25K, 50K
Minibatch size Learning
Categorical
{32, 64, 128]
Float
[1
Context mult
Float
[0.3,
Representation dim
Integer
[20, 50
Min history length
Integer
24, 100]
17


Reviews:
[{"review_id": "TidzTaJoj9t", "reviewer": "Reviewer_zvPZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work develops a new method for out-of-sample time-series forecasting (that is, transfer setting). Instead of calling the method a meta-forecasting method, which seems problematic. It would be better to use the more standard and frequently used term of zero-shot transfer learning, which fits much better with the method proposed.\n\nMeta-GLAR is shown to have similar runtime and memory overhead compared to a global one-step ahead RNN with similar structure. However, it is unclear how it compares with the other baselines shown in Table 1? In fact, the only result showing training time I noticed, is in Figure 2 (left). However, this only compares Meta-GLAR to RNN. What about the other methods, especially NBEATS? I was hoping to see a table similar to Table 1, but with training time for all the baseline methods across all the datasets. That said, I would remove the last contribution mentioned, since it is only in terms of RNN, but not the actual state-of-the-art methods.  \n\nDeep Factors and the more recent Graph Deep Factors were both shown to outperform DeepAR, NBEATS, and the other baselines used. How do these methods compare to Meta-GLAR? It seems they would be trivial to use in this setting as well. Nevertheless, they should also be included as baselines and discussed appropriately since both utilize global and local components and can be trained with a large corpus of time-series. \n\nThe caption of Table 3 and Table 4 have a typo, “Table 3: sMAPE on M3. All All models”. The reproducibility details mentioned in the Appendix are good and easy to follow. \n", "summary_of_the_paper": "This work proposes a new forecasting method for jointly learning from a large pool of related time-series. The method called Meta Global-Local Auto-Regression (Meta-GLAR) learns the mapping from representations produced by an RNN to one-step ahead forecasts where the parameters are learned across multiple time-series through backpropagation. This work studies the zero-shot transfer learning problem and proposes a method for it. The method is somewhat incremental and evaluation has some issues, see below for details.\n", "summary_of_the_review": "The method proposed is interesting (but somewhat incremental) and the problem is important. However, there are several issues that need to be addressed. The method typically doesn’t perform well compared to the state-of-the-art as shown in Table 1. It is also strange why different metrics are used for different datasets, for instance, of the four datasets used, there are 3 different metrics used in the evaluation ND for Electr/Traff, SMAPE for M3, and MAPE for Tourism. It would have been better to report all 3 metrics for each dataset (or just show one of the metrics, and the others in the appendix). It currently seems to be cherry picked a bit. Furthermore, the training time for each baseline and dataset is missing, please include a table like Table 1 but with the runtime for each baseline and dataset. Please see above for other points. Overall, the contribution and novelty of this work is limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "CxpMyyTlXds", "reviewer": "Reviewer_Tvo1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n* The authors describe the problem setting for meta forecasting as well as the Meta-GLAR approach quite clearly. \n\n* The experiments conducted on Meta-GLAR are quite comprehensive (large hyper-parameter sweeps, comparisons with models directly trained on target TS). Furthemore, the experimental setup is described in sufficient detail along with supplementary code to ensure reproducibility.\n\n* The ablation study conducted to determine the importance of each component in the Meta-GLAR model is comprehensive and shows that each component (RNN, Meta closed-form adaptation, and iterated forecasts) in the model adds to the performance although it does not clarify why RNN with iterated forecasts leads to worse performance than an RNN baseline. \n\n## Weaknesses\n\n* The novelty of the approach itself is quite limited. The local model (*differentiable local adaptation layer*) is essentially a linear layer applied on top of the RNN in the context window. The Meta-GLAR approach could be (approximately) described as training an RNN on the full time series without training the linear layer in the forecast horizon, and then using the linear weights from the context window only during transfer to new tasks. \n\n* In Table 1, why did the authors chose to report three different metrics across the four datasets (ND on ELECTR and TRAFF, sMAPE on M3, MAPE on M4) in their comparisons? If all three metrics were calculated across all baselines and the authors' proposed models then these results need to be included (at least in the appendix). Since the authors have not reported their reasoning behind chosing different metrics for different datasets, it raises the suspicion that these comparisons might be cherry picked.\n\n* In their comparison with fine-tuning, did the authors try fine-tuning both the final layer and RNN? That seems like a fairer comparison versus keeping the RNN model fixed and only training the final layer. The authors used closed form solutions for the final layer which might get more expensive to compute if the RNN model wasn't fixed. A comparison with a global linear layer fine-tuned alongside the RNN model would provide a better picture of the importance of their local adaptive layer.\n\n* One broader question that this research raises is about the scope and relevance of the meta learning framework to TS forecasting. The framework lends itself very well to domains like computer vision due to relationship between visual tasks [1]. However, it is not immediately obvious why it would be relevant to time series forecasting where the underlying domains and granularities can vary widely. I'd like to see the authors comment on the scope and limits of this work in their claims:\n> Meta-GLAR is trained on a source dataset and can then generate accurate forecasts for new time series that may **differ significantly** from the source dataset\n\n\n## Errata/Clarifications\n\n*  Did the authors mean to say generalization in the Into (para 2)? \n> Global-local approaches ..., exhibit a greater level of **specialization** as they learn parameters that are shared by all TS in the training set.  \n* The reference regarding the use of closed form solvers in spatial regression seems incorrect in the Related Work (end of para 2)\n> ...and spatial regression (Iwata and Kumagai, 2020), while we are not aware of any application in forecasting.\n\n## References\n\n[1] Taskonomy: Disentangling Task Transfer Learning. Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, Silvio Savarese; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018,", "summary_of_the_paper": "The authors propose an autoregressive framework, Meta-GLAR, comprising of a global RNN model and a local linear model (learned using a closed form solution) for the task of meta-forecasting of time series (TS). The linear model is the task-specific model varies for each TS in the dataset, whereas the RNN model is the meta model that is shared across al time series. The authors perform comparisons with local models as well as NBEATS and DeepAR, and show that Meta-GLAR is competitive with some of them on particular metrics. They also perform an ablation study to show that each component in their model is important to the overall zero-shot transfer in TS forecasting.", "summary_of_the_review": "The problem of zero-shot transfer in time series forecasting is important but it is also necessary to define its scope. The method presented by the authors is novel but incremental. There are concerns regarding the reporting of results (see Weaknesses) which cast doubt on the model performance, which is already lower than NBEATS on some tasks. Overall the paper presents an interesting approach to performing meta-forecasting but it needs more detailed analysis in terms of its applicability as well as completeness in terms of reporting its results. I would be willing to consider an improvement in the score if these issues are addressed.  \n\n**Post-rebuttal:** My concerns regarding scope and cherry-picking have been sufficiently addressed in the revised submission and the authors' comments. However, I am still concerned about the novelty of the work as well as the extent of ablations. I have increased my overall score to 5 (marginally below the acceptance threshold) and provided additional feedback to the authors in comments.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "1dSBXzT9bGD", "reviewer": "Reviewer_FZbR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n---\n1.\tThe automatic output layer tuning they propose is an interesting idea. Although not explored in this paper, this could potentially be adopted to cope with concept drifts or structural breaks which are prevalent in non-stationary time series datasets.\n\nWeaknesses\n---\nHowever, I do have some concerns with the paper in its current form, which has a couple of areas that need to be addressed. \n1.\tThe authors use a couple of technical terms in non-standard ways, making the introduction slightly confusing:\n  * Out-of-sample -- in the time series forecasting domain, out-of-sample data typically refers to data forwards in time that is unavailable during training. Generalisation forwards in time is the primary purpose of training forecasting models, whether statistical or machine learning approaches. The authors use this to refer to generalisation performance on a different time series entirely, however – which depends critically on similarities between source and target tasks. For instance, it is not unreasonable for a forecasting model that predicts patient survival times to fail to predict retail sales.\n  * Multi-task learning – global forecasting models are typically not referred to as multi-task learning, as the goal is to achieve good forecasting performance across all entities and time steps, as measured by a single evaluation metric. See [1] for more on multi-task learning.\n2.\tSuitability of benchmarks – while local (statistical) models can be useful for **data-limited** settings, I am not sure these are suitable comparisons for zero-shot models which cannot access the target dataset before prediction time. Assuming some data is available for training, there are few global network options available for small data regimes (e.g. [2]), and comparisons to other standard transfer learning/domain adaptation techniques should be evaluated. Similarly, any differences vs NBEATS could have arisen from the choice of encoder -- a NBEATS style encoder should be used within the proposed meta-learning framework for a fair comparison.\n3.\tHow does the framework handle transfer learning between datasets with a different number of covariates? The utility of the approach would very limited if only univariate target inputs can be used, as a lot of useful information would also need to be discarded.\n4.\tComplexity of matrix inversion -- assuming that h(t) has dimensions d, inverting the dxd matrix typically is O(d^3). As it is possible for d > t (see optimal hidden state sizes in DeepAR and Temporal Fusion Transformer vs encoder length), this can be greater than the sequential matrix multiplications for the LSTM (O(d^2 t) I believe). As such, computational complexity may indeed be a concern in some scenarios for this layer.\n\nReferences\n1.\tCrawshaw. Multi-task learning with deep neural networks. arXiv:2009.09796.\n2.\tRangapuram et al. Deep State Space Models for Time Series Forecasting. NeurIPS 2018.\n", "summary_of_the_paper": "The authors propose a new meta-learning framework to tackle the zero-shot learning problem for time series data – through the combination of:\n1.\tA standard autoregressive architecture encoding historical information (e.g. DeepAR).\n2.\tAn adaptive linear output layer whose weights are calibrated in closed-form using encoder history.\n\nThis allows the model to be trained end-to-end on a source task, while automatically performing domain adaptation when applied to a new target task.\n", "summary_of_the_review": "While the raw idea has promise, many corrections/clarifications need to be made before it is ready for publication -- specifically related to the terminology used, whether inputs between datasets can differ, and the complexity claims of their proposed layer. In addition, more suitable benchmarks are required to fully evaluate performance claims.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an autoregressive framework that combines RNN and local linear component for the problem of meta-forecasting of time series. The linear model can domain-adapt to different time series while the RNN component is shared across series. Reviewers thought the problem was important, the paper was generally clear and the experiments extensive. However they found the significance to be limited and all took issue with some of the ways that the comparisons were done. FZbR also raised the issue of complexity of the matrix inversion component of the method.  I believe this paper does fall on the rejection side of the fence due to the issues of complexity, significance and evaluations. With some development, the paper could certainly be ready for acceptance.
Decision: Reject