Meta-Forecasting by combining Global Deep Representations with Local Adaptation

1 INTRODUCTION . Time series ( TS ) forecasting is of fundamental importance for various applications like marketing , customer/inventory management , and finance ( Petropoulos et al. , 2020 ) . Classical examples are forecasting the number of daily sales of a product over the next few weeks or the energy production needs in the next few hours . Accurate TS forecasting results in better down-stream decision making with potentially large monetary implications ( e.g. , Seeger et al . ( 2016 ) ; Faloutsos et al . ( 2019 ) ) . From a machine learning perspective , each TS represents a forecasting task . Conventional approaches for TS forecasting have typically been local , i.e . each TS/task is modeled independently by a forecasting model with relatively few parameters ( see Hyndman and Athanasopoulos ( 2018 ) for an introductory overview ) . Despite the modest amount of data used to train local TS forecasting models , they are effective in practice . They have only recently been outperformed by global deep learning strategies , which jointly train a deep neural network on a large set of related TS/tasks . Global models are designed to work well on the set of TS they are trained on , but they perform poorly on out-of-sample TS , i.e . TS which are not present in the training set . For example , Oreshkin et al . ( 2020 ) show that DeepAR ( Salinas et al. , 2020 ) trained on the M4 dataset ( Makridakis et al. , 2020 ) performs poorly on the M3 dataset . Global-local approaches ( Sen et al. , 2019 ; Smyl , 2020 ; Wang et al. , 2019 ) , such as the M4 competition winner ( Smyl , 2020 ) , exhibit a greater level of specialization as they learn parameters that are shared by all TS in the training set , as well as parameters specific to each TS . However , global-local models are still not able to handle out-of-sample TS as both types of parameters are learned jointly on a large training set of related TS in a multi-task fashion . In this work , we tackle the problem of out-of-sample TS forecasting by transferring knowledge from a set of TS , called the source dataset , to another set of TS , called the target dataset . We assume that source and target datasets share some underlying structure which makes transfer learning possible , although they may contain TS from different domains . Our work can be seen as an instance of meta-learning ( Schmidhuber , 1987 ; Ravi and Larochelle , 2016 ; Finn et al. , 2017 ) , whose goal is to leverage a pool of related tasks to learn how to adapt to a new one with little data . We will refer to a forecasting method performing well in this scenario as a meta-forecasting method.1 Such models 1This was called zero-shot transfer learning by Oreshkin et al . ( 2020 ) . could in principle be trained on a large set of TS and still produce accurate and fast predictions when applied to out-of-sample TS , potentially combining the inference speed and accuracy of deep learning models with the ease-of-use of classical local models . Our meta-forecasting method produces one-step ahead forecasts by combining learned representations with a differentiable closed-form adaptation mechanism inspired by the few-shot image classification method proposed by Bertinetto et al . ( 2018 ) . Specifically , we propose a class of models which we call Meta Global-Local Auto-Regressive models ( Meta-GLAR ) . Meta-GLAR models compute point forecasts for a single TS in three steps . First , the TS is passed through a representation model to obtain a representation for each time step . Second , a local ( i.e . TS-specific ) linear model is learned in closed-form by solving a ridge regression problem mapping the representations to the observed fraction of the TS . Lastly , the local linear model is applied to the global representation to compute the final predictions . Crucially , forecasts are computed in the same way also during training , where we backpropagate through the closed-form adaptation step to learn the representation parameters globally , i.e . across multiple TS . Hence , we can learn a representation which works well in combination with an efficient closed-form local adaptation . We use the RNN backbone in DeepAR ( Salinas et al. , 2020 ) for the representation . However , one can transform any neural forecasting method in a meta-GLAR one by substituting the last global linear layer with the closed-form adaptation during both training and prediction . We also stress that , since our method adapts locally also during training , it is fundamentally different from fine-tuning the last layer on each TS , which we show to perform significantly worse . This is in contrast to recent results for few-shot image classification where fine-tuning the last layer is demonstrated to outperform many modern meta-learning methods ( Tian et al. , 2020 ) . Our main contributions can be summarized as follows . • We propose a novel meta-learning method for TS forecasting that is suitable for out-ofsample TS forecasting . Meta-GLAR significantly improves the accuracy on out-of-sample TS forecasting ( i.e. , transfer setting ) over a global neural forecasting models such as DeepAR , which employs the same RNN backbone . • Our meta-forecasting method is competitive with classical local methods , for example beating the winner of the M3 competition , as well as NBEATS ( Oreshkin et al. , 2020 ) , the state-of-the-art method for out-of-sample TS forecasting , while having substantially fewer parameters than the latter method . • We perform an extensive ablation study which shows that the closed-form adaptation , the RNN backbone , and the use of iterated forecasts during training are needed to achieve the best performance . Furthermore we show that Meta-GLAR enjoys similar time and memory costs compared to a global one-step ahead RNN with the same backbone architecture , while converging faster and achieving better accuracy . 2 RELATED WORK . Meta-learning has received considerable attention in recent years and several models have been developed primarily for few-shot image classification . Notable examples are Ravi and Larochelle ( 2016 ) ; Finn et al . ( 2017 ) ; Snell et al . ( 2017 ) ; Nichol et al . ( 2018 ) ; Sung et al . ( 2018 ) ; Bertinetto et al . ( 2018 ) . These methods work by adapting to the task at hand before making predictions . Differently from fine-tuning , this adaptation is performed also during a ( meta ) training procedure over a large set of tasks to learn the ( meta ) parameters of the model . However , realistic datasets with the high number of tasks needed by meta-learning methods are rare , hence in commonly used benchmarks like mini-imagenet ( Vinyals et al. , 2016 ) , each classification task is constructed by randomly selecting a small set of classes and related images from a large single-task dataset . This construction is artificial and departs from real-world scenarios . Recently , the work by Tian et al . ( 2020 ) showed that training a neural network on the original single-task dataset and fine-tuning only the last layer on new tasks outperforms many modern meta-learning methods for few-shot image classification . By contrast , popular TS forecasting datasets like M4 fit more naturally into the meta-learning framework , since they already contain a large number of TS/tasks ( up to 105 for M4 ) . Our method relies on a differentiable closed-form solver to perform the local ( or TS-specific ) adaptation . Meta-learning is achieved by solving a task-specific ridge regression problem that maps a deep representation to the target TS in closed-form , while the parameters of the representation are learned by backpropagation through the solver . Aside from the original application in few-shot image classification ( Bertinetto et al. , 2018 ) , differentiable closed-form solvers have been used for other few-shot problems like visual tracking ( Zheng et al. , 2019 ) , video object segmentation ( Liu et al. , 2020 ) , spoken intent recognition ( Mittal et al. , 2020 ) and spatial regression ( Iwata and Tanaka , 2020 ) , while we are not aware of any application in forecasting . Meta-learning in the context of TS forecasting has originally been synonym with model selection or combination of experts ( see e.g . Collopy and Armstrong ( 1992 ) ; Lemke and Gabrys ( 2010 ) ; Talagala et al . ( 2018 ) ) . This class of methods builds a meta-model which uses TS features to select the best performing model or the best combination of models to apply to a target TS . One drawback of these methods is that the features are usually manually designed from the data at hand and that the same set of features does not transfer well to other applications . Laptev et al . ( 2018 ) train an LSTM neural network Hochreiter and Schmidhuber ( 1997 ) on the source TS dataset and then its last layers are finetuned separately on each target TS . This approach overcomes the problems related to human designed features since the input of the network are just the previous TS observations . However , retraining the last layers of the network for each TS can be expensive , especially when dealing with a large number of TS . Additionally , their fine-tuning procedure requires the selection of hyperparameters like learning rate and number of steps of the optimizer . By contrast , our approach adapts only the last linear layer in closed-form , requiring a small increase in compute and no additional hyperparameters compared to a standard neural forecasting model , while outperforming the simple fine tuning approach . More recently , the NBEATS model has shown strong performance both in the standard ( Oreshkin et al. , 2019 ) and in the meta-learning ( Oreshkin et al. , 2020 ) setting . This multi-step ahead method uses a residual architecture ( He et al. , 2016 ) which takes past observations of a single TS as input and outputs point predictions over the whole forecast horizon . Thanks to the residual connections , a forward pass of the network allows it to implicitly adapt to the input out-of-sample TS . However , the final performance is achieved using a large ensemble and the number of parameters of the model is quite large even when the residual blocks share the same parameters . Our method , although also using ensembles , achieves good accuracy on out-of-sample TS forecasting with significantly less parameters . Finally , Iwata and Kumagai ( 2020 ) consider the few-shot TS forecasting setting where each task is formed by a small group of closely related TS . Their method , which combines LSTMs and attention , uses the TS in the support set of the task to compute the one-step ahead forecasts for each TS in the query set . This is different from our approach , where we do not exploit the other TS in the target dataset to compute predictions and we view each TS as a separate task . Our method can be extended to the case considered by Iwata and Kumagai ( 2020 ) by performing the closed-form adaptation on all the TS in the task instead of just on one . We leave this for future work . 3 PROBLEM FORMULATION . We consider the setting studied by Oreshkin et al . ( 2020 ) , where a forecasting method can learn global parameters on a source TS dataset DS to produce accurate forecasts for an out-of-sample TS which belongs to another , target TS dataset DT . The model can only adapt locally to each TS in DT , i.e . it can not use information from the other TS in DT . We view each TS as a task . Hence , our setting fits a meta-learning framework where DS is the meta-training set and DT is the meta-testing set . We will denote a single TS , as a tuple ( z , x ) where z = [ z1 , . . . , zT ] ∈ RT are the ( target ) observations and x = [ x1 , . . . , xT ] ∈ RT×p is the matrix of covariates . We will denote with t0 ∈ N the split point which divides the context window ( or past ) z1 : t0 = [ z1 , . . . , zt0 ] , x1 : t0 = [ x1 , . . . , xt0 ] from the forecast horizon ( or future ) zt0+1 : T = [ zt0+1 , . . . , zT ] , xt0+1 : T = [ xt0+1 , . . . , xT ] . We also denote with H = T − t0 the length of the forecast horizon . We view each TS as a supervised learning task with training set { ( xt , zt ) } t0t=1 and test set { ( xt , zt ) } Tt=t0+1 where an example is the covariates-target pair ( xt , zt ) . We assume that the covariates vector xt can contain some of the previous observations zt−1 , zt−2 , . . . as time-lagged values ( or time-lags ) . Differently from standard supervised learning tasks , we can not assume that the examples are independent due to the temporal dependency . The goal of TS forecasting is to compute predictions ẑt0+1 : T for the observations in the forecast horizon zt0+1 : T using the covariates x and the observations in the context window z1 : t0 . In this work we focus on the accuracy of the predictions which can be measured for example with the sMAPE metric ( Hyndman and Koehler , 2006 ) : sMAPE = 1 |D| 200 H H∑ i=1 |zt0+i − ẑt0+i| |zt0+i|+ |ẑt0+i| . ( 1 )

Summary: The authors propose a new meta-learning framework to tackle the zero-shot learning problem for time series data – through the combination of: 1.	A standard autoregressive architecture encoding historical information (e.g. DeepAR). 2.	An adaptive linear output layer whose weights are calibrated in closed-form using encoder history.  This allows the model to be trained end-to-end on a source task, while automatically performing domain adaptation when applied to a new target task.