Automatic Forecasting via Meta-Learning

1 INTRODUCTION . Accurate time-series forecasting at scale is critical for several industrial domains such as cloud computing Poghosyan et al . ( 2021 ) , supply chain Abbasimehr et al . ( 2020 ) , energy Abdallah et al . ( 2021 ) , and finance Oreshkin et al . ( 2020 ) . Most of the current time-series forecasting solutions are built by experts and require significant manual effort in model construction , feature engineering , and hyper-parameter tuning Bergstra et al . ( 2011 ) . Hence , they do not scale while generating high-quality forecasts for a wide variety of applications . Moreover , there is no learning scheme that is uniformly better than all other learning schemes for all problem instances Wolpert ( 1996 ) ( see Appendix A ) . A naı̈ve approach would be , given a new dataset , we search over thousands of models to select the best forecasting model for the problem at hand . However , this approach is practically infeasible due to the untenable time burden for every new problem . In this work , we formulate the problem of automatic and fast selection of the best time-series forecasting model as a meta-learning problem . Our solution avoids the infeasible burden of first training each of the models and then evaluating each one to select the best model for a new unseen time-series dataset , or even a new time window within a non-stationary dataset . A practically important desideratum for any solution to this problem is that once the meta-learner L is trained in an offline manner using a large corpus of time-series data , then we can use it to quickly infer the best forecasting model . The quick inference requirement of this new problem , makes it challenging to solve , yet practically important . Our meta-learner L is trained on the models ’ performances on historical datasets and the time-series meta-features of these datasets . We emphasize that the new time-series forecasting model selection meta-learning problem introduced in this paper has several unique characteristics and challenges compared to previous related meta-learning problems , e.g. , Finn et al . ( 2017 ) ; Rusu et al . ( 2019 ) ; Wistuba et al . ( 2018 ) . First , existing time-series forecasting models have different designs and different assumptions around the characteristics of time-series ( e.g. , probabilistic , seasonal , traditional , etc. ) . Therefore , different models perform differently depending on the characteristics that each dataset exhibits . Thus , capturing the similarity among different datasets needs careful selection of representative time-series meta-features . Second , the new meta-learning approach should capture the temporal variations of the models ’ performances over different time windows of the dataset , i.e. , the best time-series forecasting model for time window wt is not necessarily the best model for a subsequent time window wt+k ( see Figures 5-6 in Appendix B ) . Third , the number of available time-series forecasting models is large and thus training each forecasting model and then evaluating the suitability of each in inference leads to an unacceptable time burden for most real-world scenarios . These challenges motivate the need for our proposed approach . To solve the problem of automatic time-series forecasting model selection , we propose a temporal meta-learning approach , called AUTOFORECAST that makes time-series forecasting model selection . The intuition of our approach is learning both models ’ performance evolution over time horizon for same dataset ( via our “ temporal meta-learner ” ) and task similarity across different datasets ( via our “ general meta-learner ” and proposing novel time-series meta-features that capture their characteristics ) . In particular , we train our meta-learner using a large model space which has over 320 forecasting models ( Section 5.1 ) . We also generate more than 800 meta-features that represent five different types of meta-features which are ( simple , statistical , information theoretic , spectral-based , and landmarker ) , which reflect various characteristics of the time-series dataset ( Section 3 ) . We also consider diverse datasets so our meta-learning model can be generalized on new tasks ( Table 1 ) and release the corpus of datasets , along with their meta-features and the performances across hundreds of models . Given a new ( unseen ) dataset , AUTOFORECAST automatically determines the best forecasting model among a large space of models , without the need to train and evaluate any of the different forecasting models on this new dataset . The experiments demonstrate the effectiveness of our proposed approach where we validate our meta-learning approach on both univariate and multivariate testbeds . In particular , we show the superiority of our approach over the state-of-the-art ( SOTA ) time series forecasting models Salinas et al . ( 2020 ) ; Wang et al . ( 2019 ) ; Taylor & Letham ( 2018 ) ; Montero-Manso et al . ( 2020 ) ; Liaw et al . ( 2002 ) and different meta-learning approaches Kadioglu et al . ( 2010 ) ; Nikolić et al . ( 2013 ) ; Zhao et al . ( 2020 ) . Across all datasets , AUTOFORECAST achieves 2X gain in selecting the best forecasting model . Moreover , AUTOFORECAST yields a significant reduction in inference time over the naı̈ve approach , i.e. , across all datasets , AUTOFORECAST has a 42X median inference time reduction . We find empirically that no single forecasting model triumphs in more than 0.7 % of the datasets , thus motivating the need for automatic model selection for new time-series datasets . Summary of Main Contributions . The key contributions of this work are as follows : 1 . Problem Formulation : We propose a novel meta-learning approach that predicts the best model for new ( unseen ) forecasting tasks where each model is composed of time-series specific parameters along with traditional meta-learning model components . 2 . Temporal Learning of Performances : We propose a time-series meta-learner that learns the models ’ performances evolution over time windows of the datasets . Our meta-learner has two main sub-learners — the time-series ( temporal ) meta-learner and the general meta-learner that are designed for different data types with different time dependencies . 3 . Specialized Meta-features for Time-series Forecasting : We design novel time-series landmarker meta-features to capture the unique characteristics of a time-series dataset toward effectively quantifying task similarity . 4 . Efficiency and Effectiveness : Given a new time-series dataset , AUTOFORECAST selects the best performing forecasting algorithm and its associated hyperparameters without requiring any model evaluations , while incurring negligible run-time overhead . Through extensive experiments on our benchmark testbeds , we show that selecting a model by AUTOFORECAST outperforms SOTA meta-learners and popular forecasting models . 5 . Benchmark Data : We release our meta-learning database corpus ( 348 datasets ) , performances of the 322 forecasting models , and meta-features for the community to access it for forecasting model selection and to build on it with new datasets and models.1 2 RELATED WORK . Meta-learning in Time-series Forecasting : There are few works that considered meta-learning for time-series analysis Hooshmand & Sharma ( 2019 ) ; Ribeiro et al . ( 2018 ) ; Pan et al . ( 2020 ) . The works Hooshmand & Sharma ( 2019 ) ; Ribeiro et al . ( 2018 ) applied a neural network time-series forecasting model trained on a source ( energy ) dataset and fine-tuned it on the target ( energy ) dataset . However , these works did not consider using meta-learning for the general problem of forecasting model selection that we consider in our current work . There also exist few works that have explored model selection problem using ensemble learning Ye & Dai ( 2018 ) ; Talagala 1Anonymous URL for our database : https : //drive.google.com/drive/folders/1K1w1Ida5Cr15b5Fhidax-i-fNpWZjvet et al . ( 2018 ) ; Vaiciukynas et al . ( 2020 ) . However , their problem domain of ensemble learning is different from our problem of model selection . That is due to the fact that ensemble learning constitutes building multiple models for the same task and does not in itself involve learning from prior experience on other tasks . In contrast to those works , AUTOFORECAST can select among any ( heterogeneous ) set of methods . Finally , there is a line of work that considered empirical analysis for performance estimation Bergmeir & Benı́tez ( 2012 ) ; Cerqueira et al . ( 2020 ) and model selection Cerqueira et al . ( 2021 ) in time-series forecasting . However , these works have several distinctions from our work which are the need of evaluating all forecasting models in inference and providing an analysis of the ranking ability of performance estimators without having a meta-learner that learns how to automatically select the best model . Few-shot Learning & Transfer Learning : Few-shot learning has been recently leveraged for automating machine learning pipeline Ravi & Larochelle ( 2016 ) ; Snell et al . ( 2017 ) ; Oreshkin et al . ( 2020 ) . In particular , the works Ravi & Larochelle ( 2016 ) ; Snell et al . ( 2017 ) investigated different problems outside the domain of time-series forecasting . The work Oreshkin et al . ( 2020 ) applied meta-learning for zero-shot univariate time series forecasting . However , that work has the limitations of focusing on solving the cold start problem ( i.e. , learning model parameter initialization that generalizes better to similar tasks ) which is different from our forecasting model selection problem , considering different models from the same N-BEATS architecture Oreshkin et al . ( 2019 ) , and tackling only univariate time-series datasets . We emphasize that our framework can use N-BEATS as one forecasting algorithm in our model space . Finally , there exist few works that applied transfer learning for time series classification ( TSC ) Fawaz et al . ( 2018 ) ; Abdallah et al . ( 2021 ) ; Wen & Keyes ( 2019 ) ; Narwariya et al . ( 2020 ) . These works however have two distinctions from our work . First , they transfer the learned network ’ s weights to another network that is also trained on a target dataset . Second , the TSC problem is different from our forecasting problem that we consider here . 3 PROBLEM FORMULATION . Meta-learning Components : We address the problem of model selection for time-series forecasting via the meta-learning approach . Our proposed meta-learner AUTOFORECAST depends on : • A collection of historical time-series forecasting datasets Dtrain = { D1 , D2 , · · · , Dn } , namely , a meta-train database , where n is the number of the historical datasets in Dtrain . Note thatDi ∈ Rni×vi , where ni is the number of observations of the datasetDi and vi is the number of variables inDi . • The forecasting models that define the model space , denoted asM = { M1 , M2 , · · · , Mm } , where m is the size of the model spaceM . • For each datasetDi ∈ Dtrain , we sample T windows fromDi , where each sample window wt from datasetDi has length |wt| ( smaller than the dataset length ) ; See Appenidx C.1 . Model Design : For our forecasting model selection problem , we define our model as follows . Definition 1 . A modelMi ∈M is given by the tupleMi = ( ai , hi , gi ( · ) ) , where ai is the forecasting algorithm , hi is the hyperparameter vector for the forecasting algorithm ai , and gi ( · ) : Rni×vi → Rni×vi is the time-series data representation . We emphasize that hi consists of hyper-parameters of the forecasting algorithm ( e.g. , number of RNN layers in DeepAR Salinas et al . ( 2020 ) ) and that gi ( · ) represents time-series different representations ( e.g. , the smoothing transformation Kalekar et al . ( 2004 ) ; see Table 9 in Appendix F ) . Using the defined meta-train database , the model space , and the sampling windows , we now introduce the performance tensor . Definition 2 . Given a meta-train databaseDtrain and a model spaceM , we define the performance tensor P ∈ RT×n×m as P = { P1 , P2 , · · · , PT } , where Pk = ( p i , j k ) ∈ Rn×m and the element p i , j k = Mj ( wk ( Di ) ) denotes the j th model Mj ’ s performance on the time window wk of the ith meta-train dataset Di.2 We thus denote pik = [ pi,1k · · · p i , m k ] as the performance vector of all models inM on time window wk ofDi . 2For each time window wt , we have the pair ( xt−|wt| : t−1 , yt : t+H ) , where the different models are trained on xt−|wt| : t−1 and evaluated on yt : t+H , where H is the forecasting horizon . The performance tensor represents the prior experience that the meta-learner will build upon to perform efficiently on the new unseen task ( time-series ) . This motivates us to define our problem . Definition 3 . Time-series forecasting model selection problem . Given a new input task ( dataset ) Dtest ( i.e. , unseen time-series forecasting task ) , the time-series forecasting model selection problem is then stated as follows : for each time windowwt inDtest , select the best model M̂t ∈M to employ on that window ofDtest . Formally , such selection problem is given by M̂t ∈ arg max Mj∈M Mj ( wt ( Dtest ) ) ∀t ∈ { 1 , 2 , . . . , T } . ( 1 ) Time-series Meta-Features : A key component of AUTOFORECAST is the extraction of meta-features that aims to capture the important characteristics of a time-series dataset . To achieve such a goal , we extract meta-features for each time-series dataset which we define formally next . Definition 4 . Given a time-series dataset Di , we define the meta-features tensor Fi = { F i1 , · · · , F iT } ∈ RT×d×vi , where the meta-features matrix F ik ∈ Rd×vi denotes the set of the meta features for the time window wk of the datasetDi , given by F ik , { ψ ( wk ( Di ) ) : ψ : R|wi|×vi → Rd×vi } , ( 2 ) where ψ ( · ) : R|wi|×vi → Rd×vi defines feature extraction module in AUTOFORECAST and d denotes the number of the meta-features.3 We summarize our notations in Table 7 ( Appendix C ) . Meta-Features Categories : The set of meta features in our work that capture the main characteristics of a dataset can be organized into five categories Vanschoren ( 2018 ) : simple ( general task properties ) , statistical ( properties of the underlying dataset distributions ) , information-theoretic ( entropy measures ) , spectral ( frequency domain properties ) , and landmarker ( forecasting models ’ attributes on the task ) features . The idea of the proposed landmarker features is to apply a few of the fast , easy-to-construct time-series forecasting models on a dataset and extract features from ( i ) the structure of the estimated forecasting model , and ( ii ) its output performance scores . The complete meta-features list in AUTOFORECAST are explained in Appendix E ( Table 8 ) .

Summary: The authors address the problem of selecting the best forecasting model for a time series dataset. Datasets are described by a large vector of meta features, then two regression models are learnt to predict the test loss of a forecasting model for such a dataset at a given cutoff time point (called window): one predicts directly on the meta features computed until the cutoff time point, the other predicts autoregressively starting from the first cutoff timepoint until the target cutoff time point is reached. In experiments on two meta datasets (called performance tensors), one univariate, one multivariate, it is shown that esp. the autoregressive model outperforms the best constant model (called global best) as well as some baselines.