Year: 2021

Abstract:
In this work, we develop techniques for fast automatic selection of the best forecasting model for a new unseen time-series dataset, without having to first train (or evaluate) all the models on the new time-series data to select the best one. In particular, we develop a forecasting meta-learning approach called AUTOFORECAST that allows for the quick inference of the best time-series forecasting model for an unseen dataset. Our approach learns both forecasting models performances over time horizon of same dataset and task similarity across different datasets. The experiments demonstrate the effectiveness of the approach over state-of-the-art (SOTA) single and ensemble methods and several SOTA meta-learners (adapted to our problem) in terms of selecting better forecasting models (i.e., 2X gain) for unseen tasks for univariate and multivariate testbeds.

heading: 1 INTRODUCTION
text: Accurate time-series forecasting at scale is critical for several industrial domains such as cloud computing Poghosyan et al. (2021), supply chain Abbasimehr et al. (2020), energy Abdallah et al. (2021), and finance Oreshkin et al. (2020). Most of the current time-series forecasting solutions are built by experts and require significant manual effort in model construction, feature engineering, and hyper-parameter tuning Bergstra et al. (2011). Hence, they do not scale while generating high-quality forecasts for a wide variety of applications. Moreover, there is no learning scheme that is uniformly better than all other learning schemes for all problem instances Wolpert (1996) (see Appendix A). A naı̈ve approach would be, given a new dataset, we search over thousands of models to select the best forecasting model for the problem at hand. However, this approach is practically infeasible due to the untenable time burden for every new problem.
In this work, we formulate the problem of automatic and fast selection of the best time-series forecasting model as a meta-learning problem. Our solution avoids the infeasible burden of first training each of the models and then evaluating each one to select the best model for a new unseen time-series dataset, or even a new time window within a non-stationary dataset. A practically important desideratum for any solution to this problem is that once the meta-learner L is trained in an offline manner using a large corpus of time-series data, then we can use it to quickly infer the best forecasting model. The quick inference requirement of this new problem, makes it challenging to solve, yet practically important. Our meta-learner L is trained on the models’ performances on historical datasets and the time-series meta-features of these datasets.
We emphasize that the new time-series forecasting model selection meta-learning problem introduced in this paper has several unique characteristics and challenges compared to previous related meta-learning problems, e.g., Finn et al. (2017); Rusu et al. (2019); Wistuba et al. (2018). First, existing time-series forecasting models have different designs and different assumptions around the characteristics of time-series (e.g., probabilistic, seasonal, traditional, etc.). Therefore, different models perform differently depending on the characteristics that each dataset exhibits. Thus, capturing the similarity among different datasets needs careful selection of representative time-series meta-features. Second, the new meta-learning approach should capture the temporal variations of the models’ performances over different time windows of the dataset, i.e., the best time-series forecasting model for time window wt is not necessarily the best model for a subsequent time window wt+k (see Figures 5-6 in Appendix B). Third, the number of available time-series forecasting models is large and thus training each forecasting model and then evaluating the suitability of each in inference leads to an unacceptable time burden for most real-world scenarios. These challenges motivate the need for our proposed approach.
To solve the problem of automatic time-series forecasting model selection, we propose a temporal meta-learning approach, called AUTOFORECAST that makes time-series forecasting model selection. The intuition of our approach is learning both models’ performance evolution over time horizon for same dataset (via our “temporal meta-learner”) and task similarity across different datasets (via our “general meta-learner” and proposing novel time-series meta-features that capture their characteristics). In particular, we train our meta-learner using a large model space which has over 320 forecasting models (Section 5.1). We also generate more than 800 meta-features that represent five different types of meta-features which are (simple, statistical, information theoretic, spectral-based, and landmarker), which reflect various characteristics of the time-series dataset (Section 3). We also consider diverse datasets so our meta-learning model can be generalized on new tasks (Table 1) and release the corpus of datasets, along with their meta-features and the performances across hundreds of models. Given a new (unseen) dataset, AUTOFORECAST automatically determines the best forecasting model among a large space of models, without the need to train and evaluate any of the different forecasting models on this new dataset.
The experiments demonstrate the effectiveness of our proposed approach where we validate our meta-learning approach on both univariate and multivariate testbeds. In particular, we show the superiority of our approach over the state-of-the-art (SOTA) time series forecasting models Salinas et al. (2020); Wang et al. (2019); Taylor & Letham (2018); Montero-Manso et al. (2020); Liaw et al. (2002) and different meta-learning approaches Kadioglu et al. (2010); Nikolić et al. (2013); Zhao et al. (2020). Across all datasets, AUTOFORECAST achieves 2X gain in selecting the best forecasting model. Moreover, AUTOFORECAST yields a significant reduction in inference time over the naı̈ve approach, i.e., across all datasets, AUTOFORECAST has a 42X median inference time reduction. We find empirically that no single forecasting model triumphs in more than 0.7% of the datasets, thus motivating the need for automatic model selection for new time-series datasets.
Summary of Main Contributions. The key contributions of this work are as follows: 1. Problem Formulation: We propose a novel meta-learning approach that predicts the best model
for new (unseen) forecasting tasks where each model is composed of time-series specific parameters along with traditional meta-learning model components.
2. Temporal Learning of Performances: We propose a time-series meta-learner that learns the models’ performances evolution over time windows of the datasets. Our meta-learner has two main sub-learners — the time-series (temporal) meta-learner and the general meta-learner that are designed for different data types with different time dependencies.
3. Specialized Meta-features for Time-series Forecasting: We design novel time-series landmarker meta-features to capture the unique characteristics of a time-series dataset toward effectively quantifying task similarity.
4. Efficiency and Effectiveness: Given a new time-series dataset, AUTOFORECAST selects the best performing forecasting algorithm and its associated hyperparameters without requiring any model evaluations, while incurring negligible run-time overhead. Through extensive experiments on our benchmark testbeds, we show that selecting a model by AUTOFORECAST outperforms SOTA meta-learners and popular forecasting models.
5. Benchmark Data: We release our meta-learning database corpus (348 datasets), performances of the 322 forecasting models, and meta-features for the community to access it for forecasting model selection and to build on it with new datasets and models.1

heading: 2 RELATED WORK
text: Meta-learning in Time-series Forecasting: There are few works that considered meta-learning for time-series analysis Hooshmand & Sharma (2019); Ribeiro et al. (2018); Pan et al. (2020). The works Hooshmand & Sharma (2019); Ribeiro et al. (2018) applied a neural network time-series forecasting model trained on a source (energy) dataset and fine-tuned it on the target (energy) dataset. However, these works did not consider using meta-learning for the general problem of forecasting model selection that we consider in our current work. There also exist few works that have explored model selection problem using ensemble learning Ye & Dai (2018); Talagala
1Anonymous URL for our database: https://drive.google.com/drive/folders/1K1w1Ida5Cr15b5Fhidax-i-fNpWZjvet
et al. (2018); Vaiciukynas et al. (2020). However, their problem domain of ensemble learning is different from our problem of model selection. That is due to the fact that ensemble learning constitutes building multiple models for the same task and does not in itself involve learning from prior experience on other tasks. In contrast to those works, AUTOFORECAST can select among any (heterogeneous) set of methods. Finally, there is a line of work that considered empirical analysis for performance estimation Bergmeir & Benı́tez (2012); Cerqueira et al. (2020) and model selection Cerqueira et al. (2021) in time-series forecasting. However, these works have several distinctions from our work which are the need of evaluating all forecasting models in inference and providing an analysis of the ranking ability of performance estimators without having a meta-learner that learns how to automatically select the best model.
Few-shot Learning & Transfer Learning: Few-shot learning has been recently leveraged for automating machine learning pipeline Ravi & Larochelle (2016); Snell et al. (2017); Oreshkin et al. (2020). In particular, the works Ravi & Larochelle (2016); Snell et al. (2017) investigated different problems outside the domain of time-series forecasting. The work Oreshkin et al. (2020) applied meta-learning for zero-shot univariate time series forecasting. However, that work has the limitations of focusing on solving the cold start problem (i.e., learning model parameter initialization that generalizes better to similar tasks) which is different from our forecasting model selection problem, considering different models from the same N-BEATS architecture Oreshkin et al. (2019), and tackling only univariate time-series datasets. We emphasize that our framework can use N-BEATS as one forecasting algorithm in our model space. Finally, there exist few works that applied transfer learning for time series classification (TSC) Fawaz et al. (2018); Abdallah et al. (2021); Wen & Keyes (2019); Narwariya et al. (2020). These works however have two distinctions from our work. First, they transfer the learned network’s weights to another network that is also trained on a target dataset. Second, the TSC problem is different from our forecasting problem that we consider here.

heading: 3 PROBLEM FORMULATION
text: Meta-learning Components: We address the problem of model selection for time-series forecasting via the meta-learning approach. Our proposed meta-learner AUTOFORECAST depends on:
• A collection of historical time-series forecasting datasets Dtrain = {D1,D2, · · · ,Dn}, namely, a meta-train database, where n is the number of the historical datasets in Dtrain. Note thatDi ∈ Rni×vi , where ni is the number of observations of the datasetDi and vi is the number of variables inDi.
• The forecasting models that define the model space, denoted asM = {M1,M2, · · · ,Mm}, where m is the size of the model spaceM.
• For each datasetDi ∈ Dtrain, we sample T windows fromDi, where each sample window wt from datasetDi has length |wt| ( smaller than the dataset length); See Appenidx C.1.
Model Design: For our forecasting model selection problem, we define our model as follows. Definition 1. A modelMi ∈M is given by the tupleMi = (ai,hi, gi(·)),where ai is the forecasting algorithm, hi is the hyperparameter vector for the forecasting algorithm ai, and gi(·) : Rni×vi → Rni×vi is the time-series data representation.
We emphasize that hi consists of hyper-parameters of the forecasting algorithm (e.g., number of RNN layers in DeepAR Salinas et al. (2020)) and that gi(·) represents time-series different representations (e.g., the smoothing transformation Kalekar et al. (2004); see Table 9 in Appendix F).
Using the defined meta-train database, the model space, and the sampling windows, we now introduce the performance tensor. Definition 2. Given a meta-train databaseDtrain and a model spaceM, we define the performance tensor P ∈ RT×n×m as P = {P1,P2, · · · ,PT }, where Pk = (p i,j k ) ∈ Rn×m and the element p i,j k = Mj(wk(Di)) denotes the j
th model Mj’s performance on the time window wk of the ith meta-train dataset Di.2 We thus denote pik =[ pi,1k · · · p i,m k ] as the performance vector of all models inM on time window wk ofDi.
2For each time window wt, we have the pair (xt−|wt|:t−1,yt:t+H), where the different models are trained on xt−|wt|:t−1 and evaluated on yt:t+H , where H is the forecasting horizon.
The performance tensor represents the prior experience that the meta-learner will build upon to perform efficiently on the new unseen task (time-series). This motivates us to define our problem. Definition 3. Time-series forecasting model selection problem. Given a new input task (dataset) Dtest (i.e., unseen time-series forecasting task), the time-series forecasting model selection problem is then stated as follows: for each time windowwt inDtest, select the best model M̂t ∈M to employ on that window ofDtest. Formally, such selection problem is given by
M̂t ∈ arg max Mj∈M Mj(wt(Dtest)) ∀t ∈ {1, 2, . . . , T}. (1)
Time-series Meta-Features: A key component of AUTOFORECAST is the extraction of meta-features that aims to capture the important characteristics of a time-series dataset. To achieve such a goal, we extract meta-features for each time-series dataset which we define formally next.
Definition 4. Given a time-series dataset Di, we define the meta-features tensor Fi = {F i1 , · · · ,F iT } ∈ RT×d×vi , where the meta-features matrix F ik ∈ Rd×vi denotes the set of the meta features for the time window wk of the datasetDi, given by
F ik , {ψ(wk(Di)) :ψ : R|wi|×vi → Rd×vi}, (2)
where ψ(·) : R|wi|×vi → Rd×vi defines feature extraction module in AUTOFORECAST and d denotes the number of the meta-features.3 We summarize our notations in Table 7 (Appendix C).
Meta-Features Categories: The set of meta features in our work that capture the main characteristics of a dataset can be organized into five categories Vanschoren (2018): simple (general task properties), statistical (properties of the underlying dataset distributions), information-theoretic (entropy measures), spectral (frequency domain properties), and landmarker (forecasting models’ attributes on the task) features. The idea of the proposed landmarker features is to apply a few of the fast, easy-to-construct time-series forecasting models on a dataset and extract features from (i) the structure of the estimated forecasting model, and (ii) its output performance scores. The complete meta-features list in AUTOFORECAST are explained in Appendix E (Table 8).

heading: 4 AUTOFORECAST SOLUTION
text: AUTOFORECAST consists of two-phases: offline training of the meta-learner and online inference that aims at selecting the appropriate model at test time. We argue that running time of the offline training phase is not critical since it is done only once. On the contrary, forecasting model selection for a new time-series dataset should incur small run-time overhead since it is critical for quick selection of the forecasting model. We now explain our meta-learning approach and its components.

heading: 4.1 META-LEARNING OBJECTIVE AND (OFFLINE) TRAINING
text: We show the overview of the major components of AUTOFORECAST in Figure 1. We highlight the components transferred from offline to online stage (model selection) in blue; namely, meta-feature extractors ψ, feature embedding, time-series meta-learner Θ, and general meta-learner Φ. The meta-learner L has three main inputs; the performance tensor P, the meta-features tensor F, and the loss function. In the offline training of the meta-learner L, it learns two components Θ and Φ, where the former captures the temporal relationship between the meta-features of the consecutive time windows within the same dataset and the evolution of the performances of the models on these windows (i.e., the best candidate models on those time-windows). On the other hand, the general meta-learner Φ predicts the best model for each task (window) without taking into account the temporal relationship among different time windows within the same dataset.
Rationale for Having both General and Time-series Meta-learners: The rationale of having both meta-learners is the fact that the temporal dependency among different time windows depends on the dataset type. Some datasets have strong temporal dependency which would be predicted efficiently by the time-series meta-learner Θ while others datasets would have weak temporal dependence among different windows performances in which the general meta-learner Φ is expected to perform better. We show the results of such different datasets for our two testbeds in Table 12-13. 3Note that we do feature-embedding (PCA), as shown in Figure 1, to get the final meta-features tensor Fi.
General Meta-learner Φ: We propose multi-output regression model for training our general meta-learner Φ in Figure 1. From running all the models inM on different time windows wt with t ∈ {1, . . . , T} for all datasets in the meta-train database Dtrain, we collect a set of N = T × n distinct training samples (F it ,p i t), with t ∈ [1, T ] and i ∈ [1, n]. Recall that F it ∈ Rd×vi is the meta-feature matrix of the window wt of dataset Di ∈ Dtrain and that pit ∈ R|M| (performance vector on window wt ofDi). Thus, the multi-output regression model is given by
p̂it = Φ ( F it ,β ) ; t ∈ [1, T ], i ∈ [1, n], (3)
where Φ denotes the regression function (e.g., linear, NN) and β are the unknown regression parameters. Thus, the general meta-learner’s objective, denoted by the loss function LΦ, is given by
LΦ = T∑ t=1 n∑ i=1 L(p̂it,p i t), (4)
where L is the loss metric (e.g., MSE, MAPE, etc). Therefore, Φ learns the mapping between the meta-features of a time window in a dataset and the corresponding best model in the model space.
LSTM-based Time-series Meta-learner Θ: The goal of the time-series meta-learner Θ (in Figure 1) is to learn how the models’ performances evolve with the time-series meta-feature matrices over time. For such purpose, we propose time-series multi-regression model to learn such performance evolution. For any dataset Di, given the time-series meta-feature matrices F i1 , . . . ,F i t−1,F i t and the history of the performance vectors p i 1, . . . ,p i t−1, we aim to predict performance vector pit of current time window wt. The time-series regression equation would be
p̂it = Θ ( F i1 ,F i 2 , . . . ,F i t−1,F i t ,p i 1,p i 2, . . . ,p i t−1 ) , i ∈ [1, n], t ∈ [1, T ],
where Θ denotes the time-series regression function, and p̂it is the predicted performance vector.
We adapt long-short term memory (LSTM) inputs for our time-series meta-learner Θ. We denoteXt as the input at the time window wt which is given by Xt = [ F i1 ,p i 1,F i 2 ,p i 2, · · · ,F it−1,pit−1,F it ] . The predicted LSTM’s output denoted by p̂it is a function ofXt; detailed equations of such relation between p̂it and Xt in the LSTM are provided in Appendix D. During training, Θ learns the parametersWf , bf ,Wl, bl,Wo, bo,Wu, bu,Wv, bv which are the weights and biases of the forget, input, and output layers and cell updates, respectively. Thus, the objective of the LSTM time-series meta-learner Θ, denoted by LΘ, would be given by
LΘ = T∑ t=1 n∑ i=1 L(p̂it,p i t). (5)
We emphasize that Θ learns temporal relationships between the history of the meta-features and performance vectors over time windows and the corresponding current best model in model space.
Meta-learner Objective: Having established the two main components — the general meta-learner Φ and the time-series meta-learner Θ) — we now define the objective of our meta-learner L, which is given by a linear combination of the two components as follows:
min β,Wf ,Wl,Wo,bf ,bl,bo,Wu,bu,Wv,bt aLΦ(F,P) + bLΘ(F,P). (6)
The meta-learner L learns jointly general meta-learner Φ (equation 4) and time-series meta-learner Θ (equation 5) given meta-learner inputs, the performance tensor P and the meta-features tensor F. By definition, this meta-learner L optimizes the loss over all datasets and all time windows.

heading: 4.2 ONLINE INFERENCE AND MODEL SELECTION
text: In the online mode of AUTOFORECAST, we aim to make use of the trained meta-leaner L to quickly infer the best model for the current task. Given a new time-series dataset Dtest, AUTOFORECAST first computes the corresponding meta-features tensor by F̂test = ψ(Dtest). Those time-series meta-features are then embedded (using PCA) to obtain the final meta-features tensor Ftest. Then, in the model inference, as shown in Figure 1, the model set performances are predicted for each available model in M. The model M̂t with the lowest predicted error score by L on the time window wt of Dtest is chosen as the selected model for that window wt. Now, we explain such model selection process for each time window across the time windows w0, w1, . . . , wT of Dtest as follows. For the first window (w0), the inference is given by M̂0 ∈ arg minM̄∈M L(F test0 ). For any other window wt (with t > 0), the time-series meta-learner Θ inference depends on the history of the meta-features and the history of the models’ performances as follows M̂Θt ∈ arg minM̄∈MΘ(F test0 , . . . ,F testt−1 ,F testt , p̂test0 , . . . , p̂testt−1). On the other hand, the general meta-learner Φ inference depends on the predicted (regression) output on the meta-features of current time window where M̂Φt ∈ arg minM̄∈MΦ(F testt ). Thus, the final selected model is given by
M̂t ∈ arg min M̄∈{M̂Φt ,M̂Θt } p̂testt (M̄). (7)
We emphasize that tie between models can happen in online inference (i.e., two or more models can have an identical predicted performance). We built upon the several tie breaking techniques that have been examined in the literature Kalousis (2002); Brazdil et al. (2008), but usually such a choice does not have a strong influence on the performance of a meta-learning system. For making the decision between the model selected by the general meta-learner Φ and that selected by the time-series meta-learner Θ, we choose the model with the least predicted error score (illustrated in equation 7). We present the time complexity analysis of AUTOFORECAST’s inference in Appendix H.

heading: 5 EXPERIMENTS
text: We evaluate AUTOFORECAST by designing experiments to answer the following research questions:
• Does employing AUTOFORECAST for time-series forecasting model selection yield improved performance, as compared to no model selection, as well as other selection techniques (such as meta-learners adapted from the AutoML domain)?
• How much reduction in inference time does AUTOFORECAST give over the naı̈ve method? • How does performance change with different datasets with different temporal dependence?

heading: 5.1 EXPERIMENTAL SETUP
text: Models and Performance Collection: By pairing seven SOTA time-series forecasting algorithms (which are DeepAR Salinas et al. (2020), Deep Factors Wang et al. (2019), Prophet Taylor & Letham (2018), Seasonal Naive Montero-Manso et al. (2020), Gaussian Process Yan et al. (2009), Vector Auto Regression Lewis & Reinsel (1985), and Random Forest Regressor Liaw et al. (2002)) and their corresponding hyperparameters, and using different data representation methods, we compose a model set M with 322 unique models (see Table 9 for the complete list). For our testbeds, we first generate the performance tensor P, by evaluating the models fromM against the benchmark datasets in each testbed. For consistency, all models are built using the GluonTS Alexandrov et al.
(2020), Scikit-learn Pedregosa et al. (2011), and Statsmodels Seabold & Perktold (2010) Python libraries on an Intel i7 @2.60 GHz, 16GB RAM, 8-core workstation.
Meta-Train Testbed: Meta-learning works if the new task can leverage prior knowledge. We thus create two testbeds with 348 public forecasting datasets. In particular, we collect 308 univariate time-series datasets for the first testbed (Table 10 in Appendix G) and 40 multivariate datasets with 317 time-series for the second testbed (Table 11 in Appendix G). In total, we have 625 time-series in our testbeds. We refer to Table 1 for a summary of the statistics of the corpus of time-series data we used. For each dataset in the testbeds, we use different time windows selected randomly from the dataset, where each time window has a length of 16 (i.e., |wt| = 16 ∀t ∈ {1, . . . , T}). We emphasize that our approach has no restriction on the length of the time window.
Evaluation: For evaluating AUTOFORECAST, we split each testbed into 5 folds for cross-validation. For each fold, after training the meta-learning approach, we use it to infer the best forecasting model for the new unseen test datasets. Finally, we take the average performance of these five folds. We mainly compare the Hit-at-k accuracy (which indicates whether the selected model is within the actual top-k models), of AUTOFORECAST against different meta-learners baselines. We also compare the performance of selected model by AUTOFORECAST against the performance of the selected model by other meta-learners. This is measured by the mean square error (MSE) and the average rank (for each testbed, the meta-learners are first ranked by the corresponding forecasting MSE under the selected model for each dataset and then the rank is averaged across all datasets).
Time-series Meta-learner Setup: For our time-series meta-learner Θ explained in Section 4, we used LSTM with 4 layers where each layer has 50 units. The training was with 50 epochs with the Adam optimizer with a batch size of 25 and dropout rate of 0.2 to prevent over-fitting. The evaluation of the effect of such parameters on Θ’s performance is shown in Appendix J.3.

heading: 5.2 BASELINES
text: We adapt leading methods from algorithm selection, and include additional baselines by creating a variant of the proposed AUTOFORECAST (marked with TSL). All the baselines can be organized into the following categories (the detailed description of each baseline is given in Appendix I).
No model selection: This category always employs either the same single model or the ensemble of all the models, e.g., Random Forest (RF) Liaw et al. (2002).
Simple meta-learners: Meta-learners in this category pick the generally well-performing forecasting model, globally or locally (via clustering or KNN): Global Best (GB), ISAC Kadioglu et al. (2010), and ARGOSMART (AS) Nikolić et al. (2013).
Optimization-based meta-learners: These Meta-learners learn task similarities via optimizing performance estimates on meta-features: Multi-layer Perceptron (MLP), and AUTOFORECAST-TSL; a variant in which meta-learner L consists only of time-series learner Θ.

heading: 5.3 RESULTS
text: Now, we show our results to answer the aforementioned evaluation questions.

heading: 5.3.1 UNIVARIATE TESTBED RESULTS
text: To investigate the impact of the train/test similarity on meta-learning performance, we build the univariate testbed that consists of 308 datasets (Appendix G Table 10) with diverse datastes.
Superiority of AUTOFORECAST compared to all baseline methods w.r.t. the Hit-at-k, average rank, and MSE: The different results are provided in Table 2-4 where the best result for every testbed is highlighted in bold. We observe that AUTOFORECAST outperforms previous SOTA meta-learning methods adapted to our problem. For example, AUTOFORECAST has 79.20%, 171.86%, 423.17%, 375.61%, and 51.59% higher Hit-at-10 accuracy than GB, AS , ISAC, MLP, and AUTOFORECAST-TSL, respectively. Full evaluation of univariate testbed is in Appendix J.
Statistical Significance of AUTOFORECAST: To compare two methods statistically, we use the pairwise Wilcoxon rank test on performances (i.e., MSE of selected models) across datasets (significance level p < 0.05). Table 5 shows that AUTOFORECAST is significantly better than most of the baseline meta-learners, i.e., including GB (9.07 × 10−5), AS (1.07 × 10−37) and AUTOFORECAST-TSL (8.16× 10−15). Appendix J.2 has full statistical significance tests. Meta-learners perform better than methods without model selection: As Table 14 (Appendix J) shows, the meta-learners outperforms almost all models with no model selection. In particular, three meta-learners (Global Best, ISAC, AUTOFORECAST) significantly outperform the baseline time-series forecasting models. For instance, AUTOFORECAST respectively has 92.58%, 84.39%, 88.20%, 87.14%, 83.48%, 98.45%, and 95.75% lower MSE over Seasonal Naive, DeepAR, Deep Factors, Random Forest, Prophet, Gaussian Process, and VAR, respectively. These results signify the benefits of using meta-learning for model selection, specifically using AUTOFORECAST.
Optimization-based meta learners generally perform better than simple meta learners: Two of the top-3 meta learners by average rank and MSE (AUTOFORECAST and AUTOFORECAST-TSL) are all optimization-based and significantly outperform simple meta-learners such as ISAC and AS as shown in Table 2 and Table 4. The interpretation is that simple meta-learners weigh meta-features equally for task similarity, whereas optimization-based methods learn which meta-features matter (e.g., time-series regression on meta-features in AUTOFORECAST-TSL), leading to better results.
Dataset-wise Performance: We present the detailed performances for each dataset by comparing AUTOFORECAST with all baseline methods in Table 12. It is noted these results are averaged across the different time windows for each dataset. The results shows that AUTOFORECAST achieves the best MSE and average rank among all meta-learners. We note that AUTOFORECAST and AUTOFORECAST-TSL have same performance for datasets with higher temporal dependency.

heading: 5.3.2 MULTIVARIATE TESTBED RESULTS
text: In this testbed, we simulate the case when there are similar meta-train tasks to the test task, where we choose variables from the multivariate for training and the rest in testing. We build the multivariate testbed that consists of 40 datasets (Table 11 in Appendix G).
For the Multivariate testbed, AUTOFORECAST still outperforms all baseline methods w.r.t. average rank, MSE, and Hit-at-k accuracy as shown in Tables 2-4. Moreover, Figure 8 (Appendix J) shows that for the pool of multivariate datasets (across all time windows), AUTOFORECAST gives a gain of 2X in selecting better models as compared to other meta-learning baselines. Dataset-wise performance for the datasets in the multivariate testbed is shown in Table 13 in Appendix J. AUTOFORECAST has the lowest average MSE on most of the multivariate datasets.
Figure 2: Boxplot of AUTOFORECAST inference time for univariate testbed (in sec.). AUTOFORECAST takes less than 1.7 sec for most datasets (median = 1.65).
Statistical Significance of AUTOFORECAST: Table 5 shows that for the Multivariate testbed, AUTOFORECAST is also significantly better than most of the baseline meta-learners, i.e., including AS (3.94×10−7), MLP (0.0004) and AUTOFORECAST-TSL (0.0025) and has no significance from GB and ISAC. For the full statistical significance results for all pairs see Table 15 in Appendix J.2.

heading: 5.3.3 RUNTIME ANALYSIS
text: Inference run time statistics of AUTOFORECAST: Figure 2 shows that AUTOFORECAST (meta-feature generation and model selection) takes less than 1.7 seconds on most time series datasets. Moreover, Figure 3 shows that AUTOFORECAST has significant reduction in inference time compared to the naı̈ve approach, median is 42X on both testbeds (see also Appendix J.5).
Comparing AUTOFORECAST with baselines: In terms of inference, Table 6 shows that most of the meta-learners are fast taking only a few seconds to infer the best forecasting model. Finally, we compare the training cost of AUTOFORECAST against baseline meta-learners. Table 16 shows that AUTOFORECAST has comparable computational training cost. While the training process is offline and done only once and hence, not as important as the inference time cost, this experiment reassures us that our better model selection performance does not entail a prohibitive training cost.

heading: 6 CONCLUSION
text: We introduced a meta-learning approach to automate the process of time-series forecasting by automatically inferring the best time-series model on an unseen dataset, without needing exhaustive evaluation of all existing models on this dataset. The problem arises because there are many possible forecasting models with their associated hyperparameters, and different choices are optimal for different datasets. Further, even within one non-stationary dataset, different models are appropriate for different time windows. Our proposed solution AUTOFORECAST is a meta-learner, trained on an extensive pool of historical time-series forecasting datasets and models. To effectively capture task similarity, we designed novel problem-specific meta-features. Extensive experiments on two large testbeds showed that AUTOFORECAST significantly improves time-series forecasting model selection over directly using some of the most popular models as well as several SOTA meta-learners. We showed that AUTOFORECAST gives a significant improvement in the inference time compared to naı̈ve approaches. We release the benchmark data for the community to contribute new datasets and models to stimulate further advances on automating time-series forecasting.
Reproducibility Statement: To further research into the important problem introduced in our work, we have publicly released our benchmark data to to enable others reproduce our work. In particular, we are publicly releasing, with this submission, our meta-learning database corpus of 348 datasets, containing 625 time series in all, performances of the 322 forecasting models, and meta-features for the datasets. This resource will hopefully encourage the community to standardize efforts at benchmarking time series forecasting model selection. We also encourage the community to expand this resource by contributing their new datasets and models. The anonymized website with our database is: https://drive.google.com/drive/ folders/1K1w1Ida5Cr15b5Fhidax-i-fNpWZjvet. The details of each dataset in the two testbeds, univariate (308 datasets) and multivariate (40 datasets) are explained in Appendix G (Table 10 and Table 11) and the complete list of meta-features is presented in Appendix E (Table 8).
The performance collection is difficult and time-consuming to obtain, yet fundamental for researchers to begin studying the important problem formulated in our work, and developing better approaches to solve it. Here, we give the result of that time consuming process of collecting the performance of each of the 322 forecasting models on all the datasets (in the above link). In particular, we collect this for the performance metrics of MSE, training time, and inference time. This serves as the training data and ground truth evaluation data for AUTOFORECAST and all other competing protocols. We refer the reader to Appendix F in which we provide complete description of the algorithms, libraries, and hyper-parameter values used for this collection process.

heading: APPENDIX
text: A VALIDATION OF NO-SINGLE-MODEL HYPOTHESIS ON OUR TESTBEDS
Figure 4 shows a validation of the hypothesis that there is no unique single model that works well on all datasets, where we show a histogram of the best models probability distribution across the datasets of the training testbed where different datasets have different (best) models. That motivates the need for our framework, AUTOFORECAST, for automating model selection via meta-learning.
B VARIATION OF BEST MODEL ACROSS TIME WINDOWS
Figure 5 validate the no-free lunch theorem across different datasets (vertically) and across different time windows within the same dataset (Horizontally). Figure 6 shows the aggregate statistics on all datasets of the univariate testbed. This contradicts the claims that one forecasting algorithm can work best for different datasets and motivates the need for an effective approach for learning such both dimensions for selecting the best model for new tasks, which we propose in our current work.

heading: C SUMMARY OF NOTATION
text: We summarize the notation used in this paper in Table 7.

heading: C.1 WINDOW NOTATION IN AUTOFORECAST
text: We now reemphasize the notation of time window we used in Section 4. In AUTOFORECAST, the time window represents a sequence of time observations in the time series. In particular, wt denotes the time window index, and |wt| is the length of that time window (e.g., |w10| = 16 means that the 10th time window of the dataset has a length of 16 observations). Notice that for each time window wt, we have the pair (xt−|wt|:t−1, yt:t+H), where the different models are trained on xt−|wt|:t−1 and
evaluated on yt:t+H , where H is the forecasting horizon. For example, for single-step forecasting on a window wt with length 16, we would train the forecasting model on the first 15 observation points of the window and do forecasting on the 16th observation. Second, we emphasize that we can train a forecasting model on windows with any length, but the forecasting horizon may be limited by the forecasting model parameters.

heading: D ADDITIONAL DETAILS OF TIME-SERIES META-LEARNER
text: We now detail the equations of our LSTM-based time-series meta-learner Θ. The LSTM cell at time t has two recurrent features, denoted by hit and c i t, called the hidden state and the cell state, respectively. The LSTM cell consists of three layers (forget gate layer, input gate layer, and output gate layer). The activation of those layers is given by
f it = σ ( Wf · [hit−1,Xt] + bf ) , (8)
lit = σ ( Wl · [hit−1,Xt] + bl ) , (9)
oit = σ ( Wo · [hit−1,Xt] + bo ) . (10)
where Wf ,Wl,Wo and bf , bl, bo ∈ Rm denote the weights matrices and the biases of the three layers, respectively. These are the parameters to be learned during the training of the time-series meta-learner. The forget gate f it controls how much of the current cell state we should forget, the input gate lit controls how much of the cell update is added to the cell state, and the output gate o i controls how much of the modified cell state should leave the cell and become the next hidden state. Moreover, the cell update uit is constructed with a tanh activation function as follows.
uit = tanh ( Wu · [hit−1,Xt] + bu ) , (11)
whereWu and bu ∈ Rm are further weight parameters to be learned. Thus, the new cell and hidden states at time t are given by
cit = f i t · cit−1 + lit · uit (12)
ĥit = o i t · tanh(cit) (13)
Finally, the output equations of the LSTM cell are given by
Vit = Wvĥ i t + bv (14)
p̂it = Softmax(V i t), (15)
where Wv and bv ∈ Rm are weight parameters to be learned. This gives the relationship between input Xt and predicted performances output p̂it. For ease of notation, we considered a single layer in the above analysis . However, we have used multiple layers with several units (see Section 5).

heading: E TIME-SERIES META FEATURES
text: There are prior works that generated standard time-series features Franceschi et al. (2019), tsfresh Christ et al. (2018) (that we used for generating part of our meta-features). We now provide details of our meta-features and their different categories.

heading: E.1 META-FEATURES CATEGORIES:
text: For each dataset, we generate a meta-feature vector that consists of more than 800 meta-features. This meta-features vector includes different components that capture various properties of the time-series dataset, e.g., the statistical features (number of crossings, count of observations above/below the mean value, quantiles, etc.), the data trend (linear, non-linear, dynamics, etc.), data interdependence (lag autocorrelation, difference features, etc.), information-theoretic features that are typically based on entropy measures, and the frequency (FFT, wavelet, etc.). Our meta-features vector also includes landmarker features, which are problem-specific, and aim to capture the unique characteristics of a dataset. The idea is to apply a few of the fast, easy-to-construct time-series forecasting models on a dataset and extract features from (i) the structure of the estimated forecasting model, and (ii) its output performance scores.
Multivariate meta-features: For this time-series dataset type, we have two types (explained in Appendix G). For both types, we generate a single meta-features vector of the multivariate dataset by averaging the meta-features vectors of all variables within the dataset.

heading: E.2 COMPLETE LIST OF FEATURES
text: We summarize the meta-features used by AUTOFORECAST in Table 8. When applicable, we provide the formula for computing the meta-feature(s) and corresponding variants we used for generating the meta-features, and the corresponding number of features. Some are based on Vanschoren (2018). Specifically, our meta-features can be categorized into (1) simple features, (2) statistical features, (3) information-theoretic features, (4) Spectral features, and (5) landmarker features. Broadly speaking, the statistical features captures statistical properties of the underlying data distributions; e.g., min, max, variance, skewness, covariance, etc. of the features and feature combinations. The information-theoretic features capture information-theoretic underlying characteristics in the time-series; e.g., entropy, trend, non-linearity, change statistics, etc. Most of those meta-features have been commonly used in the AutoML literature Vanschoren (2018). To the best of our knowledge, we emphasize that our landmarker meta-features (detailed below) are novel and that some components of the spectral meta-features have not been used in any related work.

heading: E.3 LANDMARKER META-FEATURE GENERATION
text: In addition to simple, statistical, and information-theoretic meta-features, we use three time-series forecasting landmarker algorithms for computing forecasting-specific landmarker meta-features, Auto Regression Lewis & Reinsel (1985), Random Forest Liaw et al. (2002), and Bayesian Ridge Regression Shi et al. (2016) to capture landmarker characteristics of a time-series dataset.
We now provide a quick overview of each algorithm and then discuss how we are using them for building meta-features. The algorithms are executed with the default parameter.
Auto Regression Lewis & Reinsel (1985): In this landamarker feature, we fits the unconditional maximum likelihood of an autoregressive process. The k parameter represents the maximum lag of the process
Xt = ϕ0 + k∑ i=1 ϕiXt−i + εt
Then, we extract the coefficients ϕi whose index i ∈ {0, · · · , k}. Random Forest Liaw et al. (2002): is a tree-based ensemble method that builds a collection of base trees using the subsampled unlabeled data, splitting on (randomly selected) features as nodes. Random Forest grows internal nodes until the terminal leaves contain only one sample or the predefined max depth is reached.
For Random Forest, we use the balance of base trees (i.e., depth of trees and number of leaves per tree) and additional information (e.g., feature importance of each base tree). It is noted that feature importance information is available for each base tree—we therefore analyze the statistic of mean and max of base tree feature importance. The following information of base trees are used:
• Tree depth: min, max, mean, std, skewness, and kurtosis • Number of leaves: min, max, mean, std, skewness, and kurtosis • Mean of base tree feature importance: min, max, mean, std, skewness, and kurtosis • Max of base tree feature importance: min, max, mean, std, skewness, and kurtosis • Out-of-bag estimate score.
Bayesian Ridge Regression Shi et al. (2016): it estimates a Gaussian probabilistic model of the regression problem using Bayesian regression. The prior for the coefficient is given by a spherical Gaussian distribution.
For Bayesian Ridge Regression, we use the following information of the probabilistic model (fitted using the dataset) as landmarker features:
• Mean of the distribution • Log marginal Likelihood score • The precision of the estimated weights • The noise precision • The actual number of iterations to achieve the stopping criterion

heading: F MODEL SPACE
text: We now provide details on the model space used to study the meta-learning problem formulated in our work. Recall that a model in the context of our problem is a time-series forecasting algorithm and the hyperparameters used.
DeepAR Salinas et al. (2020): DeepAR experiments are using the model implementation provided by GluonTS version 1.7. We did grid search on different values of number of cells and the number of RNN layers hyperparameters of DeepAR since the defaults provided in GluonTS would often lead to apparently suboptimal performance on many of the datasets. The training parameters for each dataset are described in Table 9. All other parameters are defaults of gluonts.model.deepar.DeepAREstimator.
Deep Factors Wang et al. (2019): Deep Factors experiments are using the model implementation provided by GluonTS version 1.7. We did grid search over the number of units per hidden layer for the global RNN model and the number of global factors hyperparameters of Deep Factors. The training parameters for each dataset are described in Table 9. All other parameters are defaults of gluonts.model.deep factor.DeepFactorEstimator.

heading: Forecasting Algorithm HyperParameter 1 HyperParameter 2 Data Representation Total
text: Prophet Taylor & Letham (2018): Prophet experiments are using the model python implementation provided by Facebook (fbprophet) version 0.7.1. We did grid search over the change point prior scale and the seasonality prior scale hyperparameters of Prophet. The training parameters for each dataset are described in Table 9. All other parameters are defaults of fbprophet.Prophet.
Seasonal Naive Montero-Manso et al. (2020): Seasonal Naive experiments are using the model implementation provided by GluonTS version 1.7. We did grid search over the length of seasonality pattern, since it is different unknown for each, dataset hyperparameter of Seasonal Naive. The training parameters for each dataset are described in Table 9. All other parameters are defaults of gluonts.model.seasonal naive.SeasonalNaivePredictor.
Gaussian Process Yan et al. (2009): Gaussian Process experiments are using the model implementation provided by GluonTS version 1.7. We did grid search over the cardinality of the time-series and the maximum number of iterations for jitter to iteratively make the matrix positive definite hyperparameter of Gaussian Proces. The training parameters for each dataset are described in Table 9. All other parameters are defaults of gluonts.model.gp forecaster.GaussianProcessEstimator.
Vector Auto Regression Lewis & Reinsel (1985): Vector Auto Regression experiments are using the model implementation provided by statsmodels python library version 0.12.2. We did grid search over the loss covariance type and the trend hyperparameter of Vector Auto Regression. The training parameters for each dataset are described in Table 9. All other parameters are defaults of statsmodels.tsa.var model.
Random Forest Liaw et al. (2002): Random Forest models’ experiments are using the model implementation provided by sklearn python library version 0.24.2. We did grid search over the the number of estimators (trees) and the max depth (i.e., the longest path between the root node and the leaf node in a tree) hyperparameter of Random Forest. The training parameters for each dataset are described in Table 9. All other parameters are defaults of sklearn.ensemble.RandomForestRegressor.
Exponential Smoothing Data Representation Kalekar et al. (2004): We use the exponential smoothing from the statsmodels python library version 0.12.2. We optimized the smoothing level for the exponential smoothing (which controls the weights given for the history samples). All other parameters are defaults of statsmodels.tsa.api.ExponentialSmoothing.

heading: G DATASET DESCRIPTION

heading: G.1 DATASET TYPES IN AUTOFORECAST
text: In our work, we consider two general types of time-series datasets depending on the number of the variables vi in the time-series dataset. Now, we detail the different dataset types.
Univariate Time-series Datasets: This kind of datasets is the traditional single time-series which usually consists of single variable that need to be predicted (this variable is likely numerical, but can also be categorical in rare cases). Formally, a time-series datasetDi is single-variate if vi = 1.
Multivariate Time-series Datasets: In our work, we consider two main types (subcategories) of the Multivariate datasets that we define below. We define the Multivariate Homogeneous dataset as the multivariate dataset that consists of multiple time-series in which each time-series represents the same metric (e.g., collection of r time-series representing CPU usage of r different machines). In this type, vi = r (the number of different time-series for that same measurement). Also, we define the Multivariate Heterogeneous dataset as multivariate time-series where each time-series column represents a different measurement (e.g., wind speed, humidity, temperature). In this type, vi = |columns(Di)| (the number of the different variables for the time-series dataset).

heading: G.2 DATASET TESTBEDS IN AUTOFORECAST
text: Dataset Sources: Our testbeds are built to simulate the testbed when meta-train comes from many different distributions. This diversity makes the meta-learning model learn from such diversity. Model selection on test data can thus benefit from the prior experience on the train set. For this purpose, we use benchmark datasets from Kaggle Kaggle (2021), Adobe real traces, and other open source repositories. In particular, the Adobe trace datasets records CPU and Memory usage for 50 different services running in Adobe production clusters collected for 15 days from May 1 to May 15 in 2021. Such traces are shared for the first time in our current work.
Testbeds Summary: In short, we have collected 308 univariate datasets from such different sources. The details of the datasets (i.e., the dataset name, and the number of points in the dataset) are shown in Table 10. We also collected 40 multivariate datasets, where each dataset can have different number of variables and different type. For instance, Adobe service CPU and memory 15 days utilization is a Homogeneous multivariate dataset that has r = 100. Most of the datasets are from different application domains (e.g., finance, IoT, energy, storage, etc.). The details of the datasets (i.e., the dataset name, the variate name, and the number of points for each variate the dataset) are detailed in Table 11. For robustness, for each testbed, we split its datasets into 5 folds for cross-validation. We build the train/test testbed by each time selecting four folds from the datasets for training and the remaining fifth fold for testing. We re-emphasize that we released the datastes for the community for future usage (link is in Section 1).

heading: H DISCUSSION ON INFERENCE TIME COMPLEXITY
text: Recall that the number of meta-features is d, the number of models is m. The time complexity for the inference part of the general meta-learner Φ is O(d). On the other hand, the time complexity of the time-series LSTM meta-learner Θ is given by O(|Xt|), where |Xt| is the length of the input sequence. Thus, AUTOFORECAST’s inference time is given by max{O(d),O(|Xt|)}. We emphasize that the inference time of the naı̈ve method is much larger since it is given byO( ∑m i=1 Ii), where Ii is the inference time of forecasting algorithm ai. The aggregate statistics for reduction in inference time across both testbeds is shown in Appendix J.5. Moreover, Figure 7 shows that AUTOFORECAST yields significant reduction in inference time (68X reduction) over the naı̈ve approach on the Adobe trace CPU and Memory 15 days usage dataset.

heading: I BASELINES DETAILS
text: Meta-learning has recently received a significant attention for automating ML pipelines for a variety of different problems outside the domain of time-series forecasting including supervised learning Feurer et al. (2015); Wistuba et al. (2018), classification and regression Finn et al. (2017); Rusu et al. (2019), unsupervised learning Abdulrahman et al. (2018), outlier detection Zhao et al. (2020) and other applications Mittal et al. (2020); Vinyals et al. (2016).
In particular, meta-learning has been leveraged for such automation by designing models for new tasks based on prior experience Vanschoren (2018); Yao et al. (2018); Raghu et al. (2020).

heading: I.1 BASELINE CATEGORIES
text: We now detail each such baseline. Most of the methods proposed in these works can be categorized into the following three main categories.4
No model selection: This category always employs either the same single model or the ensemble of all the models:
• Random Forest (RF) Liaw et al. (2002): is a SOTA tree ensemble that combines the predictions made by many decision trees into a single model. In prediction, the random forest regression model takes the average of all the individual decision tree estimates.
Simple meta-learners: Meta-learners in this category pick the generally well-performing forecasting model, globally or locally:
• Global Best (GB): is the simplest meta-learner that selects the forecasting model with the largest average performance across all train datasets (across all time windows), without using any meta-features.
• ISAC Kadioglu et al. (2010): clusters the meta-train datasets based on meta-features. Given a new test time-series dataset, it identifies its closest cluster and selects the best model with largest average performance on the cluster’s datasets.
• ARGOSMART (AS) Nikolić et al. (2013): finds the closest meta-train time-series dataset (1NN) to a given test time-series dataset, based on meta-feature similarity, and selects the model with the best performance on the 1NN dataset.
Optimization-based meta-learners: Meta-learners in this category learn meta-feature by task similarities toward optimizing performance estimates:
• Multi-layer Perceptron (MLP): Given the meta-train datasets and selected time window, the MLP regressor directly maps the meta-features onto model performances by regression. However, such baseline does not learn temporal dependence within datasets.
• AUTOFORECAST-TSL: is a variant in which the meta-learner L consists only of the time-series learner Θ.

heading: I.2 USAGE OF META-LEARNING IN AUTOFORECAST
text: We emphasize that we use the term “meta-learning” in the context of traditional principle of meta-learning which is building upon prior experience on a set of historical tasks to “do better” on a new task. We build the experience across different datasets using our “general meta-learner” and build experience on the sequential dependence among the same dataset using the “LSTM time-series meta-learner”. We also capture task similarity between a new input task (dataset) and historical datasets using the “meta-features”.
We also emphasize that our proposed method is faster compared to gradient-descent-based bi-level meta-learners, e.g., on our univariate tesbed N-Beats has significantly slower training (average = 3600 seconds) and inference time (average = 101 seconds) compared to AUTOFORECAST (average = 670 seconds for training and 1.13 seconds for inference). We emphasize that enhancing gradient-descent-based bi-level meta-learners for our new problem is an avenue for future work.
4As will be shown in Table 14, we emphasize that we did a model selection on seven popular time series forecasting models as well (including the recent works DeepAR Salinas et al. (2020), DeepFactors Wang et al. (2019), and Prophet Taylor & Letham (2018)) as baselines where we chose the model with the best average performance across all windows for each dataset from all model variants.
Under review as a conference paper at ICLR 2022
Inference Time  Example:  322 forecasting models
 Our approach yields significant reduction in inference time (68X reduction)  Naïve approach: Doing inference on all models and select the best candidate

heading: J EXTENDED RESULTS

heading: J.1 FULL EVALUATION ON DATASETS PER TESTBED
text: Univariate Testbed: We now present the full evaluation of univariate testbed. Such evaluation is shown in Table 12 (we show one fold in that table in the interest of space). We note similar performances for the rest of the folds (as reflected in the average MSE and average rank for all datasets shown in Table 3-4). AUTOFORECAST outperforms the meta-learner baselines for most datasets. Moreover, AUTOFORECAST has the lowest average MSE, and the lowest average rank.
Multivariate Testbed: We here present the full evaluation of multivariate testbed. Such evaluation is shown in Table 13. It is clear that AUTOFORECAST ouperforms the meta-learner baselines for most datasets (best for 28 out of the 40 datasets). Again, we note that AUTOFORECAST has the lowest average MSE, and the lowest average rank. In particular, AUTOFORECAST (MSE = 0.12) gives a gain of 50% over the best baseline (MSE = 0.26) on the Adobe CPU Mem 15d dataset.
Meta-learners perform better than methods without model selection: Table 14 shows that meta-learners outperforms almost all models with no model selection. In particular, three meta-learners (Global Best, ISAC, AUTOFORECAST) significantly outperform the baseline time-series forecasting models. For instance, Global Best respectively has 79.42%, 56.71%, 67.28%, 64.32%, 54.19%,95.73%, and 88.21% lower MSE over Seasonal Naive, DeepAR, Deep Factors, Random Forest, Prophet, Gaussian Process, and VAR. These results signify the benefits of model selection (specifically using AUTOFORECAST).
Prophet and DeepAR have the best performance across the baseline forecasting algorithms: Table 14 shows that Prophet model has the best average MSE across the baseline forecasting algorithms. Notably, DeepAR has the second best average MSE across them.
Gain in Selecting better models using AUTOFORECAST: Figure 8 shows that across the pool of the multivariate testbed (across all time windows), AUTOFORECAST has superiority in selecting better models compared to the baseline meta-learners (2X gain in selecting better forecasting models).

heading: J.2 STATISTICAL SIGNIFICANCE OF AUTOFORECAST
text: We now show the pairwise statistical test results between every pair of methods by Wilcoxon signed rank test in Table 15. Statistically better method shown in bold (both marked bold if no significance). We re-emphasize that we use the pairwise Wilcoxon signed rank test on performances (MSE) across datasets (significance level p < 0.05). In the left, univariate testbed is shown where AUTOFORECAST is statistically significantly better than most baselines including GB, AS, and AUTOFORECAST-TSL. In the right, multivariate testbed is shown where AUTOFORECAST is statistically significantly better than AS, MLP, and AUTOFORECAST-TSL.
We next turn our attention to the tuning of the time-series meta-learner and the timing of AUTOFORECAST, respectively. In particular, we show both the computational cost and the inference time for AUTOFORECAST and compare it with our different baselines.

heading: J.3 TUNING OF TIME-SERIES META-LEARNER Θ
text: We show the effect of different hyper-parameters used in the training of the time-series meta-learner Θ on the performance (in terms of MSE under the selected model by Θ). For searching on each parameter, we fix the other parameters on their best values. In the interest of space, we show full details for the univariate testbed. Figure 9 shows the effect of the number of the LSTM layers of the time-series meta-learner. We observe that LSTM with 4 layers gives the best performance (lowest MSE). Second, Figure 10 shows the effect of the number of the units in the LSTM layer of the time-series meta-learner. We observe that LSTM with 50 units per layer gives the best performance (lowest MSE). Third, Figure 11 shows the effect of the training batch size of the time-series meta-learner. A batch with size = 25 instances gives the best performance (lowest MSE). Finally, Figure 12 illustrates the effect of the number of training epochs of the time-series meta-learner. We note that training with 50 epochs gives the best performance (lowest MSE). We have also used dropout rate of 0.2 to prevent over-fitting.

heading: J.4 COMPARING COMPUTATIONAL COST OF AUTOFORECAST
text: Table 16 shows that AUTOFORECAST has comparable computational training cost compared to the other meta-learners baselines. We re-emphasize that the running time of such offline training phase is less critical since it is done only once. However, this experiment shows that our better time series model selection performance does not entail a prohibitive training cost.
J.5 INFERENCE RUN TIME STATISTICS
Comparison between AUTOFORECAST and naı̈ve approach: We compare the inference time that AUTOFORECAST versus the inference time of the naı̈ve approach, i.e., doing inference over all possible models. We first show the aggregate statistics for the gain across all datasets for each testbed. Figure 3 illustrates the reduction in inference time for using AUTOFORECAST over the naı̈ve approach for both univariate and multivariate testbeds. In particular, AUTOFORECAST gives a median gain of at least 42X over the naı̈ve approach for both testbeds.
Dataset-wise inference time comparison: Next, we pick several random groups of 10 datasets each and show the time (in seconds) that AUTOFORECAST takes versus the time the naı̈ve approach takes for forecasting model selection. Figures 13-14 shows such comparison for the univariate testbed. It is noted the higher reduction of inference time for larger datasets (i.e., with more data points).
AARIVINT AMAZON ALIGN AMERICAN apple ARCTIC AROSA.1 ASKEW.1 BOISE.1 brent_spot
Dataset Name
0
10
20
30
40
50
60
70
80
90
In fe
re n
c e
T im
e (
s e
c o
n d
s )
AutoForecast Naive
Figure 13: Inference time comparison between AUTOFORECAST and naı̈ve approach. ASKEW8 ASKEW9 ATT AZUSA bank BAYDU BEARDS CPI BIGCONEgdp_japan Dataset Name
0
20
40
60
80
100
120
140
In fe
re n
c e
T im
e (
s e
c o
n d
s )
AutoForecast Naive
Figure 14: Inference time comparison between AUTOFORECAST and naı̈ve approach.
Multivariate testbed: We also observe that AUTOFORECAST (meta-feature generation and model selection) takes less than 1.2 second on most time series datasets for multivariate testbed.
Time overhead of AUTOFORECAST relative to training of selected model: Figure 15 shows that it incurs only negligible overhead relative to actual training of the selected model (with median = 0.1%). Similarly, AUTOFORECAST incurs only negligible overhead relative to actual training of the selected model for the multivariate testbed (with median = 0.3%) (Figure is omitted for multivariate testbed). This shows that AUTOFORECAST is lightweight, incurring small selection time overhead.
K INTERPRETABLE FEATURES OF DATASETS
Now, we show some of the interpretable input feature values for all datasets in the univariate testbed. In particular, we show mean, median, variance, skeweness, kutosis, absolute energy, and benford correlation for each of the individual time series in that testbed. Such features are shown in Figures 16-23. Note that time-series datasets have standard normalization before feature extraction. Such features values show the diversity in the datasets, where each dataset has different characteristics. We emphasize that in the interest of space, we show sample of these interpretable features.
0 50 100 150 200 250 300 Dataset Index
0.0
0.2
0.4
0.6 0.8 M ea n
Figure 16: Mean 0 50 100 150 200 250 300 Dataset Index
0.0
0.2
0.4
0.6
0.8
1.0
M ed
ia n
Figure 17: Median
Figure 20: Kurtosis 0 50 100 150 200 250 300 Dataset Index
Figure 21: Absolute Energy
0 50 100 150 200 250 300 Dataset Index
−0.075
−0.050
−0.025
0.000
0.025
0.050
0.075
Be nf or d Co
rre la tio
n
Figure 22: Benford Correlation 0 50 100 150 200 250 300 Dataset Index
0.0
0.2
0.4
0.6
0.8
1.0
Du pl ica
te s
Figure 23: Existence of duplicates (binary)

Review summaries:
This paper proposes a meta learning approach for time series. Specifically, the approach learns to select the best model and hyper-parameters for unseen test datasets, when trained on performance data of various models applied to training datasets. The approach utilizes various dataset specific features in order to make the predictions. Experimental results show an improved performance over other meta learning baselines. The corpus of time series datasets that was used for the experiments, is also provided as supplementary material.

This paper formulates the problem of automatic and fast selection of the best time-series forecasting model as a meta-learning problem.    Specifically, an AUTOFORECAST model consisting of a temporal meta-learner and a general meta-learner are proposed.     A large benchmark and a collection of models are established to evaluate the problem.     

The authors address the problem of selecting the best forecasting model for a time series dataset. Datasets are described by a large vector of meta features, then two regression models are learnt to predict the test loss of a forecasting model for such a dataset at a given cutoff time point (called window): one predicts directly on the meta features computed until the cutoff time point, the other predicts autoregressively starting from the first cutoff timepoint until the target cutoff time point is reached. In experiments on two meta datasets (called performance tensors), one univariate, one multivariate, it is shown that esp. the autoregressive model outperforms the best constant model (called global best) as well as some baselines. 

This paper present a novel approach to addressing a fundamental problem in forecasting research: given the plethora of methods available and no clearly superior approach overall, which forecasting method should be chosen for a new dataset/forecasting task combination. The authors propose a combination of a static (not taking the time evolution of accuracy into account) meta-learning model with one that takes the evolution of accuracy over time into account. They apply this methodology using a number of well-chosen forecasting methods on public data sets. Some of these (e.g., Adobe traces), I hadn't see before, and their discovery may be an important contribution in itself.