Year: 2021

Abstract:
While classical time series forecasting considers individual time series in isolation, recent advances based on deep learning showed that jointly learning from a large pool of related time series can boost the forecasting accuracy. However, the accuracy of these methods suffers greatly when modeling out-of-sample time series, significantly limiting their applicability compared to classical forecasting methods. To bridge this gap, we adopt a meta-learning view of the time series forecasting problem. We introduce a novel forecasting method, called Meta GlobalLocal Auto-Regression (Meta-GLAR), that adapts to each time series by learning in closed-form the mapping from the representations produced by a recurrent neural network (RNN) to one-step-ahead forecasts. Crucially, the parameters of the RNN are learned across multiple time series by backpropagating through the closed-form adaptation mechanism. In our extensive empirical evaluation we show that our method is competitive with the state-of-the-art in out-of-sample forecasting accuracy reported in earlier work.

heading: 1 INTRODUCTION
text: Time series (TS) forecasting is of fundamental importance for various applications like marketing, customer/inventory management, and finance (Petropoulos et al., 2020). Classical examples are forecasting the number of daily sales of a product over the next few weeks or the energy production needs in the next few hours. Accurate TS forecasting results in better down-stream decision making with potentially large monetary implications (e.g., Seeger et al. (2016); Faloutsos et al. (2019)).
From a machine learning perspective, each TS represents a forecasting task. Conventional approaches for TS forecasting have typically been local, i.e. each TS/task is modeled independently by a forecasting model with relatively few parameters (see Hyndman and Athanasopoulos (2018) for an introductory overview). Despite the modest amount of data used to train local TS forecasting models, they are effective in practice. They have only recently been outperformed by global deep learning strategies, which jointly train a deep neural network on a large set of related TS/tasks. Global models are designed to work well on the set of TS they are trained on, but they perform poorly on out-of-sample TS, i.e. TS which are not present in the training set. For example, Oreshkin et al. (2020) show that DeepAR (Salinas et al., 2020) trained on the M4 dataset (Makridakis et al., 2020) performs poorly on the M3 dataset. Global-local approaches (Sen et al., 2019; Smyl, 2020; Wang et al., 2019), such as the M4 competition winner (Smyl, 2020), exhibit a greater level of specialization as they learn parameters that are shared by all TS in the training set, as well as parameters specific to each TS. However, global-local models are still not able to handle out-of-sample TS as both types of parameters are learned jointly on a large training set of related TS in a multi-task fashion.
In this work, we tackle the problem of out-of-sample TS forecasting by transferring knowledge from a set of TS, called the source dataset, to another set of TS, called the target dataset. We assume that source and target datasets share some underlying structure which makes transfer learning possible, although they may contain TS from different domains. Our work can be seen as an instance of meta-learning (Schmidhuber, 1987; Ravi and Larochelle, 2016; Finn et al., 2017), whose goal is to leverage a pool of related tasks to learn how to adapt to a new one with little data. We will refer to a forecasting method performing well in this scenario as a meta-forecasting method.1 Such models
1This was called zero-shot transfer learning by Oreshkin et al. (2020).
could in principle be trained on a large set of TS and still produce accurate and fast predictions when applied to out-of-sample TS, potentially combining the inference speed and accuracy of deep learning models with the ease-of-use of classical local models.
Our meta-forecasting method produces one-step ahead forecasts by combining learned representations with a differentiable closed-form adaptation mechanism inspired by the few-shot image classification method proposed by Bertinetto et al. (2018). Specifically, we propose a class of models which we call Meta Global-Local Auto-Regressive models (Meta-GLAR). Meta-GLAR models compute point forecasts for a single TS in three steps. First, the TS is passed through a representation model to obtain a representation for each time step. Second, a local (i.e. TS-specific) linear model is learned in closed-form by solving a ridge regression problem mapping the representations to the observed fraction of the TS. Lastly, the local linear model is applied to the global representation to compute the final predictions. Crucially, forecasts are computed in the same way also during training, where we backpropagate through the closed-form adaptation step to learn the representation parameters globally, i.e. across multiple TS. Hence, we can learn a representation which works well in combination with an efficient closed-form local adaptation.
We use the RNN backbone in DeepAR (Salinas et al., 2020) for the representation. However, one can transform any neural forecasting method in a meta-GLAR one by substituting the last global linear layer with the closed-form adaptation during both training and prediction. We also stress that, since our method adapts locally also during training, it is fundamentally different from fine-tuning the last layer on each TS, which we show to perform significantly worse. This is in contrast to recent results for few-shot image classification where fine-tuning the last layer is demonstrated to outperform many modern meta-learning methods (Tian et al., 2020).
Our main contributions can be summarized as follows.
• We propose a novel meta-learning method for TS forecasting that is suitable for out-ofsample TS forecasting. Meta-GLAR significantly improves the accuracy on out-of-sample TS forecasting (i.e., transfer setting) over a global neural forecasting models such as DeepAR, which employs the same RNN backbone.
• Our meta-forecasting method is competitive with classical local methods, for example beating the winner of the M3 competition, as well as NBEATS (Oreshkin et al., 2020), the state-of-the-art method for out-of-sample TS forecasting, while having substantially fewer parameters than the latter method.
• We perform an extensive ablation study which shows that the closed-form adaptation, the RNN backbone, and the use of iterated forecasts during training are needed to achieve the best performance. Furthermore we show that Meta-GLAR enjoys similar time and memory costs compared to a global one-step ahead RNN with the same backbone architecture, while converging faster and achieving better accuracy.

heading: 2 RELATED WORK
text: Meta-learning has received considerable attention in recent years and several models have been developed primarily for few-shot image classification. Notable examples are Ravi and Larochelle (2016); Finn et al. (2017); Snell et al. (2017); Nichol et al. (2018); Sung et al. (2018); Bertinetto et al. (2018). These methods work by adapting to the task at hand before making predictions. Differently from fine-tuning, this adaptation is performed also during a (meta) training procedure over a large set of tasks to learn the (meta) parameters of the model. However, realistic datasets with the high number of tasks needed by meta-learning methods are rare, hence in commonly used benchmarks like mini-imagenet (Vinyals et al., 2016), each classification task is constructed by randomly selecting a small set of classes and related images from a large single-task dataset. This construction is artificial and departs from real-world scenarios. Recently, the work by Tian et al. (2020) showed that training a neural network on the original single-task dataset and fine-tuning only the last layer on new tasks outperforms many modern meta-learning methods for few-shot image classification. By contrast, popular TS forecasting datasets like M4 fit more naturally into the meta-learning framework, since they already contain a large number of TS/tasks (up to 105 for M4).
Our method relies on a differentiable closed-form solver to perform the local (or TS-specific) adaptation. Meta-learning is achieved by solving a task-specific ridge regression problem that maps a
deep representation to the target TS in closed-form, while the parameters of the representation are learned by backpropagation through the solver. Aside from the original application in few-shot image classification (Bertinetto et al., 2018), differentiable closed-form solvers have been used for other few-shot problems like visual tracking (Zheng et al., 2019), video object segmentation (Liu et al., 2020), spoken intent recognition (Mittal et al., 2020) and spatial regression (Iwata and Tanaka, 2020), while we are not aware of any application in forecasting.
Meta-learning in the context of TS forecasting has originally been synonym with model selection or combination of experts (see e.g. Collopy and Armstrong (1992); Lemke and Gabrys (2010); Talagala et al. (2018)). This class of methods builds a meta-model which uses TS features to select the best performing model or the best combination of models to apply to a target TS. One drawback of these methods is that the features are usually manually designed from the data at hand and that the same set of features does not transfer well to other applications. Laptev et al. (2018) train an LSTM neural network Hochreiter and Schmidhuber (1997) on the source TS dataset and then its last layers are finetuned separately on each target TS. This approach overcomes the problems related to human designed features since the input of the network are just the previous TS observations. However, retraining the last layers of the network for each TS can be expensive, especially when dealing with a large number of TS. Additionally, their fine-tuning procedure requires the selection of hyperparameters like learning rate and number of steps of the optimizer. By contrast, our approach adapts only the last linear layer in closed-form, requiring a small increase in compute and no additional hyperparameters compared to a standard neural forecasting model, while outperforming the simple fine tuning approach.
More recently, the NBEATS model has shown strong performance both in the standard (Oreshkin et al., 2019) and in the meta-learning (Oreshkin et al., 2020) setting. This multi-step ahead method uses a residual architecture (He et al., 2016) which takes past observations of a single TS as input and outputs point predictions over the whole forecast horizon. Thanks to the residual connections, a forward pass of the network allows it to implicitly adapt to the input out-of-sample TS. However, the final performance is achieved using a large ensemble and the number of parameters of the model is quite large even when the residual blocks share the same parameters. Our method, although also using ensembles, achieves good accuracy on out-of-sample TS forecasting with significantly less parameters.
Finally, Iwata and Kumagai (2020) consider the few-shot TS forecasting setting where each task is formed by a small group of closely related TS. Their method, which combines LSTMs and attention, uses the TS in the support set of the task to compute the one-step ahead forecasts for each TS in the query set. This is different from our approach, where we do not exploit the other TS in the target dataset to compute predictions and we view each TS as a separate task. Our method can be extended to the case considered by Iwata and Kumagai (2020) by performing the closed-form adaptation on all the TS in the task instead of just on one. We leave this for future work.

heading: 3 PROBLEM FORMULATION
text: We consider the setting studied by Oreshkin et al. (2020), where a forecasting method can learn global parameters on a source TS dataset DS to produce accurate forecasts for an out-of-sample TS which belongs to another, target TS dataset DT . The model can only adapt locally to each TS in DT , i.e. it cannot use information from the other TS in DT . We view each TS as a task. Hence, our setting fits a meta-learning framework where DS is the meta-training set and DT is the meta-testing set. We will denote a single TS, as a tuple (z,x) where z = [z1, . . . , zT ] ∈ RT are the (target) observations and x = [x1, . . . ,xT ] ∈ RT×p is the matrix of covariates. We will denote with t0 ∈ N the split point which divides the context window (or past) z1:t0 = [z1, . . . , zt0 ], x1:t0 = [x1, . . . ,xt0 ] from the forecast horizon (or future) zt0+1:T = [zt0+1, . . . , zT ], xt0+1:T = [xt0+1, . . . ,xT ]. We also denote with H = T − t0 the length of the forecast horizon. We view each TS as a supervised learning task with training set {(xt, zt)}t0t=1 and test set {(xt, zt)}Tt=t0+1 where an example is the covariates-target pair (xt, zt). We assume that the covariates vector xt can contain some of the previous observations zt−1, zt−2, . . . as time-lagged values (or time-lags). Differently from standard supervised learning tasks, we cannot assume that the examples are independent due to the temporal dependency.
The goal of TS forecasting is to compute predictions ẑt0+1:T for the observations in the forecast horizon zt0+1:T using the covariates x and the observations in the context window z1:t0 . In this work
we focus on the accuracy of the predictions which can be measured for example with the sMAPE metric (Hyndman and Koehler, 2006):
sMAPE = 1 |D| 200 H H∑ i=1 |zt0+i − ẑt0+i| |zt0+i|+ |ẑt0+i| . (1)

heading: 4 META GLOBAL-LOCAL AUTO-REGRESSIVE FORECASTING
text: The forecasting model we prosose is non-linear, auto-regressive and computes one-step ahead forecasts, i.e. for each time step t in the forecast horizon, the model outputs a point forecast ẑt as a non-linear function of the covariates x1:t and the observations in the context z1:t0 . The model generates forecasts over the forecast horizon using iterated (or recursive) forecasts (Salinas et al., 2020). Hence, the model will use new covariates vectors which contain previous forecasts in place of the missing time-lags for the time-steps in the horizon. These new vectors are constructed together with the forecasts in an iterative fashion from the first to the last point in the horizon. By contrast, multi-step ahead approaches like NBEATS compute forecasts over the whole horizon directly.
Starting from h0 = 0 ∈ Rd, the Meta Global-Local Auto-Regressive (Meta-GLAR) forecasting model computes point predictions as follows:
ht := h(xt,ht−1; Θ), ∀t ∈ [1 : T ], (2) ẑt := w > optht, ∀t ∈ [t0 + 1 : T ], (3)
where ht and ẑt are respectively the d-dimensional representation and point forecast at time t. Here, h is a recurrent function providing an appropriate representation for TS. While in this work we choose to use the LSTM architecture used by DeepAR (Salinas et al., 2020), our approach is not limited to it and could use other representations such as the one in Franceschi et al. (2019). Θ and wopt are respectively the global and local parameters.
Estimating the local parameters. The local parameters wopt are learned explicitly and in closedform on the training set of a single TS/task, which contains the data in the context window. This is done by solving a ridge regression problem having as inputs and targets respectively the observations z1:t0 and the representation vectors h1:t0 :
wopt : = arg min w∈Rd t0∑ t=1 (w>ht − zt)2 + γ‖w‖2 (4)
= (h>1:t0h1:t0 + γI) −1h>1:t0z1:t0 , (5)
where γ ∈ R+ is the regularization parameter. Aside from the training procedure, the key difference with a global neural forecasting model is that wopt is a local parameter. Albeit in closed-form, computing (5) requires either a d × d matrix inversion or solving a linear system and has a cost of Ω(d2t0). Note, however, that d is usually small (e.g. 20-50) and the cost of computing the representation vectors h1:t0 is usually far higher, especially for large t0 and when h is an RNN.
Estimating the global parameters. Similarly to other neural forecasting models, the global parameters (Θ, γ) are trained instead by solving the following minimization problem on the source dataset DS using a stochastic gradient method:
min Θ,γ ∑ (z,x)∈DtrS L(ẑt0+1:T , zt0+1:T ), (6)
where L is a regression loss suitable for forecasting (typical choices are (scaled) MAE or sMAPE) and DtrS is the training split of DS . We note that the loss is computed over the test set of each TS/task in DtrS , which contains the data in the forecast horizon, not used to compute wopt. This prevents the overfitting of wopt and allows to learn global parameters which generalize well to each TS/task. We use iterated forecasts also during training even if the true target observations in the forecast horizon are available in this phase and show that this increases the performance, in line with Bengio et al. (2015), even without scheduled sampling. DtrS is typically constructed by removing the last H observations and covariates from each TS in DS . The last H observations are used to test the in-sample performance of the method on DS .
Gradient Contributions. Recall that the predictions (t > t0) are given by ẑt = w>optht, where wopt is the solution to the least squares problem for the past time points t ≤ t0 and thus depends on the past representations h1:t0 . Then, the gradient of the loss with respect to the global parameters Θ has two components:
dL dΘ = T∑ t=t0+1 dL dẑt [ dht dΘ wopt + t0∑ τ=1 dhτ dΘ dwopt dhτ ht ] . (7)
Hence, the first component is the direct gradient of the loss with respect to the representations in the prediction range ht0+1:T . The second component is an indirect contribution from the representations in the context h1:t0 , which influence wopt.
Since wopt is obtained in closed-form and differentiable, we can compute the gradient of the loss using automatic differentiation. Replacing the Mean Squared Error in (4) with another loss, such as the one used in (6), usually leads to wopt not being in closed-form. However, if (4) is still a convex minimization problem or conic program, our approach still applies (Agrawal et al., 2019), e.g., by using iterative reweighted least squares. If instead (4) becomes non-convex, one could still backpropagate through a few gradient descent updates as in Raghu et al. (2019).
Intuitively, the RNN is trained to generate time dependent (and TS specific) features ht that work well for the local linear forecasting model. In principle, this design allows the model both to learn the common traits of the TS in DS and to adapt to individual TS, combining the properties and benefits of both global and local models. Additionally, since wopt is learned explicitly on each TS (also in DT ) before computing predictions, the model should adapt better to potential dataset shifts between DS and DT . This is in contrast to other global-local models like the M4 competition winner (Smyl, 2020), which learn the local parameters together with global ones only during training.

heading: 5 EXPERIMENTS
text: In all our experiments, we train on a source dataset and measure predictive performance on a target dataset. We stress that when source and target dataset are not the same, the model computes the forecasts for each TS in the target dataset without access to the other TS in the target dataset. This is different from domain adaptation settings where source and target dataset are available at training time. To compare with NBEATS, we use the same source-target combinations reported by Oreshkin et al. (2020): we train on one of the M4 datasets and use datasets with same frequency as target datasets whenever possible (see Table 6 in the Appendix). M4 contains a large number of diverse TS, thus there is higher chance for it to share some properties with other datasets. An ideal source dataset would contain a very diverse set of time series from many real-world applications from different domains (and different frequencies).
Transfer from M4. In Table 1, for both Meta-GlAR and DeepAR and for each frequency of the M4 dataset we repeat the following model selection. We evaluate 200 random hyperparameter configurations and pick the 10 configurations with the lowest sMAPE on a subset of 8K training TS. The hyperparameters for the random search are described in the Appendix (Table 8). A single MetaGLAR model (Meta-GLAR-top10) is competitive with Theta and ARIMA and always outperforms DeepAR. Ensembling can improve performance for both DeepAR and Meta-GLAR. An ensemble of 10 Meta-GLAR models (Meta-GLAR-ens10) outperforms NBEATS, which consists in an ensemble of 90 models, on ELECTR, as well as DeepAR trained on the target dataset on ELECTR and M3. NBEATS is still superior on TRAFF and TOURISM while the gap is much closer on M3. As shown in Table 2, our method has a substantially lower global parameter count than NBEATS: from 35 to 1000 times less parameters for the monthly frequency. We did not notice performance improvements of our method by increasing the number of layers or the hidden dimensions. This is a common behavior of one-step-ahead RNN based forecasting models (see e.g. Salinas et al. (2020)), whose accuracy does not increase much beyond a certain model size. By contrast, NBEATS is a multi-step ahead residual network that computes a sequence of predictions in a single forward pass and hence benefits from a larger number of parameters. More detailed results are in Appendix A.1.
Ablation analysis. In Figure 1, we perform an ablation analysis to evaluate the benefits of different elements of our method (Meta+RNN+ITF). These are the closed-form adaptation (Meta), the RNN network for the representation and the use of iterated forecast during training (ITF). We perform
combinations of the following ablations. Replacing the closed-form adaptation with a global linear layer (no Meta in the name). Using the observations in the horizon to compute predictions during training instead of iterated forecasts (no ITF in the name). Replacing the RNN with a stateless feed-forward model (FF) and a linear (Lin) model. The results show that all the three ingredients are necessary to achieve the best performance. In addition, we note that in this scenario, using iterated forecasts during training without the closed-form adaptation (RNN+ITF) performs worse than the auto-regressive RNN model. Replacing the last global linear layer with the closed-form adaptation gives our model more “memory” even when we use a feed-forward or linear backbone, since the last linear layer is computed using past representations in the context, while an autoregressive stateless model only uses the last representation in the context. Hence, it is interesting to see that the RNN backbone performs the best even in this setting. Ablation on the hourly frequency is in Appendix A.2.
Comparison with fine-tuning. Recently, Tian et al. (2020) criticized meta-learning approaches in computer vision and achieved state-of-the-art results for few-shot image classification only by

heading: Meta-GLAR/RNN/DeepAR NBEATS-SH NBEATS-NSH
text: learning a good representation and then using it with a local model at prediction time, avoiding any local learning or adaptation during training. We explored this approach by training a standard global RNN model on the source dataset. For prediction on the target dataset, we then kept the network weights fixed, but replaced the last layer with the closed-form adaption layer (with γ = 1), corresponding to the setting of Tian et al. (2020) of a fixed global representation trained without meta learning. This method reaches a median sMAPE2 of 14.16, 15.40 (ITF) and 14.13, 15.74 (ITF) for M4 and M3 monthly respectively, which is far higher than all the other methods in Figure 1. While we used the closed-form adaption layer in our experiments, the solution is equivalent to training the last layer from scratch using SGD. We also tried a variant of this experiment where we use the regularization γ‖w − wglobal‖ where wglobal are the parameters of the global last layer. This is equivalent to gradient descent fine-tuning of the last layer starting from wglobal. This variant reaches 14.86 and 16.06 median sMAPE respectively on M4 and M3 monthly (no ITF). These experiments show that in the forecasting setting it is important to meta-learn the representation parameters such that the local model works well, in contrast to the results of Tian et al. (2020).
Learning curves. In Figure 2, we compare the performance during training of our method with that of NBEATS with shared weights (NBEATS-SH) and the global RNN method from our ablation analysis. Meta-GLAR and RNN use the same model and training hyperparameters, while NBEATS uses an 8 times larger minibatch size (the same used by Oreshkin et al. (2020)). The gap in performance is large at the beginning of the training. We believe this is due to the local closed-form adaptation, which allows to have greater performance even with random global parameters. We also note that the performance of Meta-GLAR is more stable during training and that the training time (on CPU) is only 17% higher than that of the non-adaptive RNN. NBEATS is significantly faster to train: 1.38 hours for 300 epochs with batch size 1024. This is expectd since NBEATS computes all predictions in a single forward pass and thus benefits more from parallel computation. The prediction time for Meta-GLAR
2We computed the median on the 10 over 100 random search runs and with lowest sMAPE on a subset of 8K training TS.
is only marginally higher than that of a the non-adaptive RNN, i.e. it takes 3:21 minutes instead of 3:19 to forecast the full M4 test set, while NBEATS takes 3:15 minutes (we set the minibatch size to 1K for all methods during prediction). Note also that the non-adaptive RNN training and prediction times similar to DeepAR since both have the same RNN backbone.

heading: 5.1 EXPERIMENT DETAILS
text: Benchmark datasets. We use five datasets in our experiments3. ELECTR contains hourly time series of the electricity consumption of 370 customers (Dua et al., 2017). TRAFF time series measures hourly occupancy rate, between 0 and 1, of 963 San Francisco car lanes (Dua et al., 2017). TOURISM is a collection of yearly, monthly and quarterly series of indicators related to tourism activities (Athanasopoulos et al., 2011). M3 and M4 are two collection of time series that were used in recent forecasting competitions (Makridakis and Hibon, 2000; Makridakis et al., 2020). M3 comprises 645 yearly, 756 quarterly, 1428 monthly as well as 174 TS without time granularity (M3-OTHER). Finally M4 contains 100K time series in total of yearly, quarterly, monthly, weekly, daily and hourly granularities. Both M3 and M4 contain heterogeneous TS types from business, financial and economic domains. All datasets are accessed through the GluonTS library (Alexandrov et al., 2020). Following Oreshkin et al. (2020), to compare with previous result from the literature we report the commonly used metrics for each dataset: sMAPE for M4 and M3; ND, which corresponds to the P50 loss in Salinas et al. (2020), for ELECTR and TRAFF; and MAPE for TOURISM.
Baselines. We report the top baselines from Oreshkin et al. (2020), namely Theta (local) (Assimakopoulos and Nikolopoulos, 2000), auto-ARIMA (local), NBEATS and DeepAR (Salinas et al., 2020). We also report the results from DeepAR from Salinas et al. (2020) and train and evaluate the DeepAR model using a procedure similar to our method. We recall that local methods do not require training on the source dataset and their parameters are estimated directly on each TS of the target dataset. Fine-tuning the whole network on each TS and other meta-learning methods used primarily in computer-vision like MAML (Finn et al., 2017), REPTILE (Nichol et al., 2018) or ANIL (Raghu et al., 2019) can in principle be applied to TS forecasting. However, all require an additional hyperparameter: the number of steps of gradient descent used to learn the local parameters, which can’t be easily meta-learned. Furthermore, fine-tuning the whole network on each TS, MAML and REPTILE have as local parameters all the network weights, hence they would be significantly slower and occupy more memory than our method during prediction and, for MAML and REPTILE also during training. On the other hand, ANIL adapts only the last layer and thus should have a similar time and memory cost than our method. Applying these methods to time-series forecasting is an interesting future work.
Models input/output. Like many RNN forecasing models, we provide previous observations as input to the models (time-lags). The lags for a given time-frequency are detailed in Table 7 of the appendix. In addition, we scale/descale the input time-lags and final output of the model by dividing/multiplying by a scaling coefficient which is the average of the absolute value of the observations in the context window.4 We also use the log of the scaling coefficient and the “age”, i.e. the distance from the first observation in TS as additional covariates. Forecasts below zero are set to zero, since all the datasets contain positive values. The NBEATS model applies a similar scaling but it takes as input to the residual network just the observations in the context window, without using covariates. In our setting it is counterproductive to use the index of the TS as a covariate. However, handling cases where source and target datasets have other different covariates is an interesting future work.
Network architectures. Meta-GLAR employs an RNN network to compute the representation. This network is composed of two LSTM layers with hidden dimension between 20 and 50 (see Table 8) followed by a linear layer, which allows us to control the dimension of the linear adapation layer independently of the hidden dimension of the RNN. The last LSTM layer has also a residual connection, i.e. the final output is the sum of the output and the input of the layer. Zoneout (Krueger et al., 2016), a form of dropout which works well for RNNs, is applied to both layers with rate 0.1. The same backbone is used for the representation of DeepAR5 and the RNN auto-regressive model in the ablation analysis, although without the final linear layer which would be redundant in this
3Oreshkin et al. (2020) use an additional dataset called FRED as the source dataset. However, this dataset is not publicly available and can only be retrieved by crawling an ever changing website over the course of days. Since this procedure does not result in a reproducible dataset, we do not consider it here.
4This is a common practice, but becomes crucial when dealing with datasets coming from different domains. 5We used the default implementation of GluonTS.
case. For DeepAR and the auto-regressive RNN model, the hidden dimension of both LSTMs is the same and equal to the dimension of the representation. The feedforward network (FF in Figure 1) is composed by two dense layers both followed by ReLU with the same hidden size equal to the dimension of the output. For NBEATS (Figure 2) we used the implementation in GluonTS with sMAPE as loss function, 30 residual blocks with shared weights, and context length equal to 36.
Training procedure. Models are trained using a custom version of the GluonTS trainer which uses the ADAM (Kingma and Ba, 2014) optimizer with weight decay and gradient clipping parameters set to 10−8 and 10 respectively. At the end of training, the final model is the averge of the last 5 checkpointed models (models are checkpointed every 50 ADAM steps). We do not use any early stopping strategy. The loss function used by our method is the sMAPE divided by 100. Learning rate, minibatch size, and number of steps are selected via random search. Minibatches for training are constructed by randomly selecting slices from the time series of the source dataset (Salinas et al., 2020). The slices are split into a context window that is used for fitting the linear adapation model and a prediction window for which the fitted local model is used to generate predictions. The loss for backpropagation is computed on the prediction window. The context window plus prediction window may be longer than some of the shorter time series in the training or test dataset. In this case, the time series is left-padded with 0s to fill the context window and the corresponding time points are not included when fitting the local adaptation layer. This preprocessing is standard in GluonTS and is applied to all methods that we train. To make sure the model can handle such short time series at prediction time, we also include such cases from the training dataset. A hyperparameter controls the minimum number of observations in the context window.
Hardware. All models are trained and evaluated on a cloud ml.c5.4xlarge instance with 16 CPU cores and 32GB of RAM. Training and evaluating Meta-GLAR takes around 3-6 hours.

heading: 6 CONCLUSION AND FUTURE WORK
text: In this work we proposed a novel meta-learning approach for time series forecasting models called Meta-GLAR. Meta-GLAR is trained on a source dataset and can then generate accurate forecasts for new time series that may come from a domain different from the one of the source dataset. This is achieved through a combination of a global deep representation and a local closed-form adaptation layer. Crucially, during training the local closed-form adaption is differentiable, such that the deep representations are learned across multiple time series in a meta-learning fashion, by backpropagating gradients through the solution of the closed-form solver at training time.
We evaluated Meta-GLAR on out-of-sample TS through an extensive empirical study. Results were competitive with NBEATS, the current state-of-the-art method while requiring fewer parameters. Our model was also competitive with classical local methods (e.g. beating the winner of the M3 competition) and outperforms a global RNN-based method with a similar architecture.
While Meta-GLAR did not achieve state-of-the-art results on all datasets, it introduces a novel approach to neural forecasting that draws from meta-learning and global-local forecasting. We showed that including a differentiable local adaptation layer in neural forecasting models can improve forecast accuracy and transfer capabilities. This approach could be used in combination with other neural forecasting backbones such as transformers (Lim et al., 2019; Li et al., 2019).
In contrast to recent results in computer vision (Tian et al., 2020), we showed that differentiable adaptation during training can perform well on forecasting problems. We believe that this is due to the different nature of forecasting datasets, which usually contain a large number of TS/tasks, while few-shot image classification benchmarks are constructed from a large single-task dataset.
We also believe that the present work is an important step towards neural models that combine the ease-of-use of classical forecasting methods (directly used on single time series), with the accuracy gains of deep learning models. However, the scope of applicability of meta-learning forecasting methods remains unclear. For example, further work is needed to remove the assumption that target and source dataset have the same frequency (e.g., daily, monthly, weekly), or to deal with source and target dataset having different dataset or domain specific covariates.

heading: 7 REPRODUCIBILITY STATEMENT
text: To ensure reproducibility of our experiments we evaluated our method on publicly available TS datasets and we discussed in detail the training procedure, model parameters, preprocessing, and hardware used in Section 5.1. Furthermore, we state in the Appendix the time lags (Table 7) used by the model, the hyperparameter ranges for the the model selection procedure used in the majority of the experiments (Table 8) and the source-target pairs (Table 6). We also provide the source code for the experiments in the supplemetary material.

heading: Appendix

heading: A ADDITIONAL RESULTS

heading: A.1 TRANSFER FROM M4 (RESULTS BY FREQUENCY)
text: Table 3-5 contain the results for the models in Table 1 divided by frequency for M3, TOURISM and M4, together with additional baselines. The rows of the tables are divided in 3 sections. The first one contains local models, the second one models with global parameters and the third one models with global parameters which we trained and evaluated.
Meta-GLAR outperforms DeepAR (both trained on M4 using the same random search and evaluation procedure) overall on ELECTR, TRAFF, TOURISM, and M3 and also on most subdatasets of TOURISM and M3. Our method achieves the state of the art on ELECTR, reaching the same ND of ARIMA and NBEATS trained on electricity. Meta-GLAR-ens10 outperforms all the local methods on M3 and is second only to NBEATS. On Tourism, our method performs better than Theta and ARIMA although it falls behind the local method LeeCBaker, which is tailored to tourism TS . On TRAFF Meta-GLAR has a higher ND than NBEATS and local methods but improves substantially over DeepAR trained on M4-HOURLY.

heading: A.2 ABLATION ANALYSIS ON M4-HOURLY
text: In Figure 3 we report the results of the ablation analysis on the hourly frequency. Note that in this case, the source dataset, M4-HOURLY, has only 414 TS. We perform combinations of the following ablations. Replacing the closed-form adaptation with a global linear layer (no Meta in the name). Replacing the closed-form adaptation with a global linear layer only at training time, which is equivalent to fine-tuning the last layer (ADA in the name). Using the observations in the horizon to compute predictions during training instead of iterated forecasts (no ITF in the name).
We note that our method outperforms the autoregressive RNN both for ELECTR and TRAFF. Computing the adaptation only during prediction can be advantageous on TRAFF but significantly degrades the performance on M4-HOURLY. Using iterated forecast during training is beneficial in most cases. We also report the statistics for all the random search runs of the ablations studies in Figure 4. Our methods performs the best in most cases even with random hyperparameters.

heading: B ADDITIONAL EXPERIMENTS DETAILS

heading: B.1 ADDITIONAL BENCHMARK METHODS
text: We Include the following additional benchmarks methods for an improved comparison. LeeCBaker (Baker and Howard, 2011), a competitive method for tourism TS . EXP (Spiliotis et al., 2019), the state of the art local method on M3. The M4 competition winner (M4 winner) (Smyl, 2020) and the second best entry (Best ML/TS) (Montero-Manso et al., 2020).

heading: B.2 SOURCE TARGET PAIRS
text: In Table 6 we describe the source-target combinations that we use for all our experiments. The same combinations are considered by Oreshkin et al. (2020). We use the model trained on M4-QUARTERLY to compute forecasts on M3-OTHER because they have forecast horizons of the same length. ELECTR and TRAFF contain hourly data, hence they are matched with M4-HOURLY.

heading: B.3 METRICS
text: To measure the performance of the methods, we use the following metrics.
sMAPED = 1 |D| ∑
(z,x)∈D
200
HD HD∑ i=1 |zt0+i − ẑt0+i| |zt0+i|+ |ẑt0+i|
(8)
NDD =
∑ (z,x)∈D ∑HD i=1 |zt0+i − ẑt0+i|∑
(z,x)∈D ∑HD i=1 |zt0+i|
(9)
MAPED = 1 |D| ∑
(z,x)∈D
100
HD HD∑ i=1 |zt0+i − ẑt0+i| |zt0+i|
(10)
where D is the TS dataset (for M3, M4, TOURISM we consider each frequency separately), HD is the corresponding forecast horizon and |D| is the number of TS in the dataset. We compute aggregate metrics as follows.
sMAPEM3 = ( ∑ D∈M3 HD × |D| )−1 × ∑ D∈M3 HD × sMAPED (11)
MAPETOURISM = ( ∑ D∈TOURISM HD × |D| )−1 × ∑ D∈TOURISM HD ×MAPED (12)
where M3 = {M3-YEARLY, . . . ,M3-OTHER} and TOURISM = {TOUR-YEAR, . . . , TOUR-MONTH}.

Review summaries:
The authors propose an autoregressive framework, Meta-GLAR, comprising of a global RNN model and a local linear model (learned using a closed form solution) for the task of meta-forecasting of time series (TS). The linear model is the task-specific model varies for each TS in the dataset, whereas the RNN model is the meta model that is shared across al time series. The authors perform comparisons with local models as well as NBEATS and DeepAR, and show that Meta-GLAR is competitive with some of them on particular metrics. They also perform an ablation study to show that each component in their model is important to the overall zero-shot transfer in TS forecasting.

The authors propose a new meta-learning framework to tackle the zero-shot learning problem for time series data – through the combination of: 1.	A standard autoregressive architecture encoding historical information (e.g. DeepAR). 2.	An adaptive linear output layer whose weights are calibrated in closed-form using encoder history.  This allows the model to be trained end-to-end on a source task, while automatically performing domain adaptation when applied to a new target task. 

This work proposes a new forecasting method for jointly learning from a large pool of related time-series. The method called Meta Global-Local Auto-Regression (Meta-GLAR) learns the mapping from representations produced by an RNN to one-step ahead forecasts where the parameters are learned across multiple time-series through backpropagation. This work studies the zero-shot transfer learning problem and proposes a method for it. The method is somewhat incremental and evaluation has some issues, see below for details.