Automatic Forecasting via Meta-Learning
Authors: Mustafa Abdallah
Ryan Rossi
Kanak Mahadik
Sungchul Kim
Handong Zhao
Haoliang Wang
Saurabh Bagchi
Venue: ICLR 2022 Submitted (2022)
Keywords: Forecasting Model Selection
Time-series Forecasting
Meta-features

Abstract: In this work, we develop techniques for fast automatic selection of the best forecasting model for a new unseen time-series dataset, without having to first train (or evaluate) all the models on the new time-series data to select the best one. In particular, we develop a forecasting meta-learning approach called AutoForecast that allows for the quick inference of the best time-series forecasting model for an unseen dataset. Our approach learns both forecasting models performances over time horizon  of  same  dataset  and  task  similarity  across different datasets. The experiments demonstrate the effectiveness of the approach over state-of-the-art (SOTA) single and ensemble methods and several SOTA meta-learners (adapted to our problem) in terms of selecting better forecasting models (i.e., 2X gain) for unseen tasks for univariate and multivariate testbeds. 


Under review as a conference paper at ICLR 2022
AUTOMATIC FORECASTING VIA META-LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
In this work, develop techniques for fast automatic selection of the best forecasting model for new unseen time-series dataset, without having to first train (or evaluate) all the models on the new time-series data select the best one. In particular, we develop a forecasting meta-learning approach called AUTOFORECAST that allows for the quick inference of the best time-series forecasting model for an unseen dataset. Our approach learns both forecasting models performances over time horizon of same dataset and task similarity across different datasets. The experiments demonstrate the effectiveness of the approach over state-of-the-art (SOTA) single and ensemble methods and several SOTA meta-learnrss (adapted to our problem) in terms of selecting better forecasting models (i.e., 2X gain) for unseen tasks for univariate and multivariate testbeds.
INTRODUCTION
Accurate time-series forecasting scale is critical for several industrial domains such cloud computing Poghosyan et al. (2021), supply chain Abbasimehr et al. (2020), energy Abdallah et al. (2021), and finance Oreshkin et al. (2020). Most of the current time-series forecasting solutions are built by experts and require significant manual effort in model construction, feature engineering, and hyper-parameter tuning Bergstra et al. (2011). Hence, they do not scale while generating high-quality forecasts for a wide variety of applications. Moreover, there is no learning scheme that uniformly better than all other learning schemes for all problem instances Wolpert (1996) (see Appendix A). A naïve approach would be, given new dataset, we search over thousands of models to select the best forecasting model for the problem hand. However, this approach is practically infeasible due to the untenable time burden for every new problem.
In this work, we formulate the problem of automatic and fast selection of the best time-series forecasting model as meta-learning problem. Our solution avoids the infeasible burden of first training each of the models and then evaluating each one to select the best model for new unseen time-series dataset, even new time window within a non-stationary dataset. A practically important desideratum for any solution to this problem is that once the meta-learner L is trained in offline manner using a large corpus of time-series data, then best forecasting model. The quick inference requirement of this new problem, makes it challenging to solve, yet practically important. Our meta-learnrr L is trained on the models" performances on historical datasets and the time-series meta-features of these datasets.
it to quickly infer the
We emphasize that the introduced in this paper has several unique characteristics and challenges compared to previous related meta-learning problems, e.g., Finn al. (2017); Rusu et al. (2019); Wistuba al. (2018). First, existing time-series forecasting models have different designs and different assumptions around the characteristics of time-series (e.g., probabilistic, seasonal, traditional, etc.). Therefore, different models perform differently depending on the characteristics that each dataset exhibits. Thus, capturing the similarity among different datasets needs careful selection of representative time-series meta-features. Second, the new meta-learning approach should capture the temporal variations of the models' performances over different time windows the dataset, i.c., the best time-series forecasting model for time window w, is not necessarily the best model for subsequent time window Wi+k (see Figures 5-6 in Appendix B). Third, the number of available time-series forecasting models is large and thus training each forecasting model and then evaluating the suitability of each in inference leads to an unacceptable time burden for most real-worlh scenarios. These challenges motivate the need for our proposed approach.
time-series forecasting model selection meta-learning problem

---

Under review as a conference paper at ICLR 2022
To solve the problem of automatic time-series forecasting model selection, we propose temporal meta-learning approach, called AUTOFORECAST that makes time-series forecasting model selection. The intuition of our approach is learning both models' performance evolution over time horizon for same dataset (via our "temporal meta-learner") and task similarity across different datasets (via our "general meta-learner" and proposing novel time-series meta-features that capture their characteristics). In particular, we train our meta-learner using a large model space which has over 320 forecasting models (Section 5.1). We also generate more than 800 meta-features that represent five different types of meta-features which are (simple, statistical, information theoretic, spectral- spectral-based, and landmarker), which reflect various characteristics of the time-series dataset (Section 3). We also consider diverse datasets so meta-leaminn model can be generalized on new tasks (Table 1) and release the corpus of datasets, along with their meta-features and the performances across hundreds of models. Given new (unseen) dataset, AUTOFORECAST automatically determines the best forecasting model among a large space of models, without the need to train and evaluate any of the different forecasting models on this new dataset.
The experiments demonstrate the effectiveness of our proposed approach where we validate our meta-learning approach on both univariate and multivariate testbeds. In particular, show the superiority of our approach over the state-of oftth--r-the-rt (SOTA) time series forecasting models Salinas al. (2020); Wang et al. (2019); Taylor & Letham (2018); Montero-Manss et al. (2020); Liaw al. (2002) and different meta-learning approaches Kadioglu et al. (2010); Nikolić et al. (2013); Zhao al. (2020). Across all datasets, AUTOFORECAST achieves 2X gain in selecting the best forecasting model. Moreover, AUTOFORECAST yields a significant reduction in inference time over the naïve approach, i.e., across all datasets, AUTOFORECAST has a 42X median inference time reduction. We find empirically that no single forecasting model triumphs in more than 0.7% of the datasets, thus motivating the need for automatic model selection for new time-series datasets.
Summary of Main Contributions. The key contributions of this work are as follows:
1. Problem Formulation:
We propose novel meta-learning approach that predicts the best model for new (unseen) forecasting tasks where each model is composed of time-series specific parameters along with traditional meta-learning model components.
2. Temporal Learning of Performances: We propose a time-series meta-learner that learns the models' performances evolution over time windows of the datasets. Our meta-learner has two main sub-learners - the time-series (temporal) meta-learner and the general meta-learner that are designed for different data types with different time dependencies.
3. Specialized Meta-features for Time-series Forecasting: We design novel time-series landmarker meta-features to capture the unique characteristics of a time-series dataset toward effectively quantifying task similarity.
4. Efficiency and Effectiveness: Given a new time-series dataset, AUTOFORECAST selects the best performing forecasting algorithm and its associated hyperparameters without requiring any model evaluations, while incurring negligible run-time overhead. Through extensive experiments on our benchmark testbeds, we show that selecting model by AUTOFORECAST outperforms SOTA meta-learners and popular forecasting models.
5. Benchmark Data: We release our meta-learning database corpus (348 datasets), performances of
the 322 forecasting models, and meta-features for the community access it for forecasting model selection and to build on it with new datasets and models.
2 RELATED WORK
Meta-learning in Time-series Forecasting: There are few works that considered meta-learning for time-series analysis Hooshmand & Sharma (2019); Ribeiro et al. (2018); Pan et al. (2020). The works Hooshmand & Sharma (2019); Ribeiro al. (2018) applied neural network time-series forecasting model trained on a source (energy) dataset and fine-tuned it on the target (energy) dataset. However, these works did not consider using meta-learning for the general problem of forecasting model selection that we consider in our current work. There also exist few works that have explored model selection problem using ensemble learning Ye & Dai (2018); Talagala
AAnoyymous URL for our database: https://drs wwwi/eivveeeeeeeee ive googl e
1k 1 da55rr 1555t i dax- FNpWxjvet
2

---

Under review as a conference paper at ICLR 2022
et al. (2018); Vaiciukynas al. (2020). However, their problem domain of ensemble learning is different from our problem of model selection. That is due to the fact that ensemble learning constitutes building multiple models for the same task and does not in itself involve learning from prior experience on other tasks. In contrast to those works, AUTOFORECAST can select among any (heterogeneous) set of methods. Finally, there is line of work that considered empirical analysis for performance estimation Bergmeir & Benítez (2012); Cerqueira et al. (2020) and model selection Cerqueira et al. (2021) in time-series forecasting. However, these works have several distinctions from our work which are the need of evaluating all forecasting models in inference and providing an analysis of the ranking ability of performance estimators without having a meta-leamer that learns how to automatically select the best model.
Few-shot learning has been recently leveraged for automating machine learning pipeline Ravi & Larochelle (2016); Snell et al. (2017); Oreshkin et al. (2020). In particular, the works Ravi & Larochelle (2016); Snell et al. (2017) investigated different problems outside the domain of time-series forecasting. The work Oreshkin et al. (2020) applied meta-learning for zero-shot univariate time series forecasting. However, that work has the limitations of focusing solving the cold start problem (i.e., learning model parameter initialization that generalizes better to similar tasks) which is different from our forecasting model selection problem, considering different models from the same N-BEATS architecture Oreshkin et al. (2019), and tackling only univariate time-series datasets. We emphasize that our framework can use N-BEATS as one forecasting algorithm in our model space. Finally, there exist few works that applied transfer learning for time series classification (TSC) Fawaz et al. (2018); Abdallah et al. (2021); Wen & Keyes (2019); Narwariya et al. (2020). These works however have two distinctions from our work. First, they transfer the learned network's weights to another network that is also trained on target dataset. Second, the TSC problem is different from our forecasting problem that we consider here.
Few-shot Learning& Transfer Learning:
3 PROBLEM FORMULATION
Meta-learning Components:
We address the problem of model selection for time-series forecasting via the meta-learning approach. Our proposed meta-learner AUTOFORECAST depends on:
A collection of historical time-series forecasting datasets Dtrain = {Di, D2, D.). namely, meta-train database, where is the number of the historical datasets in Dtrain- Note that D, E R" where ni; is the number of observations of the dataset D; and U, is the number of variables in D:.
The forecasting models that define the model space, denoted as M = {M1, M2. ... Ammh where m is the size of the model space M.
For each dataset D; E Dtrain. we sample T windows from D., where each sample window We from dataset D, has length |w,| smaller than the dataset length); See Appenidx C.I.
Model Design: For our forecasting model selection problem, we define our model as follows. Definition 1. A model M; E M is given by the tuple M; = (a,, hi, ge(-)), where 0s is the forecasting algorithm, h, is the hyperparameter vector for the forecasting algorithm at and g((-) pa,x., > Ra4x% is the time-series data representation.
We emphasize that h, consists of hyper-parameters of the forecasting algorithm (e.g.. number of RNN layers in DeepAR Salinas et al. (2020)) and that gx(-) represents time-series different representations (e.g., the smoothing transformation Kalekar et al. (2004); see Table 9 in Appendix F).
Using the defined meta-train database, the model space, and the sampling windows, we now introduce the performance tensor.
Definition 2. Given a mela-train database Derain and model space M, we define the performance lensor P E xxxxx
as
P (P.. where Pt = (wkt) E R and the element pk i. = Mf(wx(D.) denotes the jth model M, 's performance on the time window wx of the ith meta-train dataset D We thus denote pi. =
Pr).
[pt] Pk ph as the performance vector of all models in M on time window wi of D..
Forr each time window We, we have the pair (x dd Vt++H). where the different models are trained on 1 and evaluated on yue+ Ht, where H is the forecasting horizon.
3

---

Under review as a conference paper at ICLR 2022
The performance tensor represents the prior experience that the meta-learner will build upon to perform efficiently on the new unseen task (time-serie),, This motivates us to define our problem.
Definition 3. Time-series forecasting model selection problem. Given new input task (dataset) Diest (i.e., unseen time-series forecasting lask), the time-seriex forecasting model selection problem is then stated as follows: for each time window we in Drest, select the best model Mr E -M to employ on that window of D Desst- Formally, such selection problem is given by
Mr arg max (Drest)) VI 21, T).
M,EM
Time-series Meta-Features: A key component of AUTOFORECAST is the extraction of meta-features that aims to capture the important characteristics of a time-series dataset. To achieve such goal, extract meta-features for each time-series dataset which we define formally next.
Definition 4. Given a time-series dataset D; we define the meta-features tensor F; = (Fi. Fi} E RTxxxe, where the meta-features matrix F E Rdxx; denotes the set of the meta features for the time window Wi of the dataset D;, given by
F (b(W (D:)) ap. Rle//x + pdxe..
(2)
where v-()
|las/xx Rdx defines feature extraction module in AUTOFORECAST and d denotes the number of the meta-features." We summarize our notations in Table 7 (Appendix C).
Feta-Features Categories: The set of meta features in our work that capture the main characteristics of a dataset can be organized into five categories Vanschoren (2018): simple (general task properties), statistical (properties of the underlying dataset distributions), information-theorede (entropy measures), spectral (frequency domain properties), and landmarker (forecasting models' attributes on the task) features. The idea of the proposed landmarker features is to apply a few of the fast, easy-to-construc time-series forecasting models on dataset and extract features from (i) the structure of the estimated forecasting model, and (ii) its output performance scores. The complete meta-features list in AUTOFORECAST are explained in Appendix E (Table 8).
4 AUTOFORECAST SOLUTION
AUTOFORECAST consists of two-phases: offline training of the meta-learner and online inference that aims at selecting the appropriate model at test time. We argue that running time of the offline training phase is not critical since it is done only once. On the contrary, forecasting model selection for a new time-series dataset should incur small run-time overhead since is critical for quick selection of the forecasting model. We now explain our meta-learning approach and its components.
4.1 META- ORJECTIVE AND (OFFLINE) TRAINING
We show the overview of the major components of AUTOFORECAST in Figure 1. We highlight the components transferred from offline to online stage (model selection) in blue; namely, meta-feature extractors vú, feature embedding, time-series meta-leamner O and general meta-learner $ The meta-learner L has three main inputs; the performance tensor P. the meta-features tensor F, and the loss function. In the offline training of the meta-learner L, it learns two components O and 4, where the former captures the temporal relationship between the meta-features of the consecutive time windows within the same dataset and the evolution of the performances of the models these windows (i.e., the best candidate models on those windww.. On the other hand, the general meta-learner & predicts the best model for each task (window) without taking into account the temporal relationship among different time windows within the same dataset.
Rationale for Having both General and Time-series Meta-learners: The rationale of having both meta-learners is the fact that the temporal dependency among different time windows depends on the dataset type. Some datasets have strong temporal dependency which would be predicted efficiently by the time-series meta-learner O while others datasets would have weak temporal dependence among different windows performances which the general meta-learner is expected to perform better. We show the results of such different datasets for our two testbeds in Table 12-13.
Note that we do feature-emheddingg (PCA), shown in Figure 1, to get the final meta-features tensor F.
4

---

Under review as a conference paper at ICLR 2022
Performance Tensor [P..Py.....P)) Pr] (2Xnkm)
Mata-llarner (L)
Nea-lamar
reatures
Regesskel
Training Datasets Press (0s, S2
Vate furtane
Eebadding (PEA)
orias Segensioo)
(Offline) Meta-leamer(s) Training
(Online) Forecasting Model Selection
Testing
feelodkiing (PCA)
Ooekk infensse
Pint
Figure I: An overview of AUTOFORECAST; components that transfer from offline online (model selection) phase shown in blue. Given the two main inputs; the performance tensor P and the meta-features tensor F, the meta-learner learns two main components; general meta-learner ($) and time-series meta-learner (9).
General Meta-learner Φ: We propose multi-output regression model for training our general meta-lcarner T in Figure 1. From running all the models in M on different time windows w, with t E (1..... T) for all datasets in the meta-train database Derain, we collect a set of N 7 n distinct training samples (F); pt), with t E T] and [], Recall that F; E RRdx, is the meta-feature matrix of the window w; of dataset D; E Dirain and that pỘ E RIM (performance vector on window W2 of D,). Thus, the multi-output regression model given by
pi = ():
; ;); [1, [I,
where + denotes the regression function (e.g., linear, NN) and ,3 are the unknown regression parameters. Thus, the general meta-learner's objective, denoted by the loss function Lę. is given by
L
MPt.pi)
where is the loss metric (e.g., MSE, MAPE, etc). Therefore, & learns the mapping between the meta-features of a time window in dataset and the corresponding best model in the model space.
LSsMd Time-series Meta-learner O: The goal of the time-series meta-learner O (in Figure 1) is learn how the models' performances evolve with the time-series meta-feature matrices over time. For such purpose, we propose time-series multi-regression model to learn such performance evolution. For any dataset D;, given the time-series meta-feature matrices Fi..... F Pi and the history of the performance vectors pi...., .i....,P'. Pi-1, we aim to predict performance vector pi of current time window wt. The time-series regression equation would be .i.....,p...... p + (Fi Fr Fi..., Fi Fr pi ... Pi-1) i [L,n], [1,7],
where O denotes the time-series regression function, and pł is the predicted performance vector.
We adapt long-short term memory (LSTM) inputs for our time-series meta-learner ④ We denote X, as the input at the time window w, which is given by X, = [Fi, F.. The predicted LSTM's output denoted by p is a function of X,; detailed equations of such relation between of and X, in the LSTM are provided in Appendix D. During training.
learns parameters W, y, by. Wi, bi, W.. bo W., b,. W.. b, which are the weights and biases of the forget, input, and output layers and cell updates, respectively. Thus, the objective of the LSTM time-series meta-learner θ, denoted by Le. would be given by
L ΣΣLPE L(pi,
1=1
We emphasize that O learns temporal relationships between the history of the meta-feature and performance vectors over time windows and the corresponding current best model in model space.
5

---

Under review as a conference paper at ICLR 2022
Meta-learner Objective: Having established the two main components the general meta-learner & and the time-series meta-learner θ) - we now define the objective of our meta- leatnee L. which is given by a linear combination of the two components as follows:
min 3. w. W.,b bi b,, W..bb,WW W..b.
aL. (F.P) (F.
(6)
The meta leannee learns jointly general meta-learner + (equation 4) and time-series meta-learner @ (equation 5) given meta- meta-learner inputs, the performance tensor P and the meta-features tensor F. By definition, this meta-learner optimizes the loss over all datasets and all time windows.
4.2 ONLINE INFERENCE AND MODEL SELECTION
In the online mode of AUTOFORECAST, we aim make use of the trained meta-leaner L. to quickly infer the best model for the current task. Given a new time-series dataset D Dtests AUTOFORECAST first computes the corresponding meta-features tensor by Frest = (teestt) Those time-series meta-features are then embedded (using PCA) to obtain the final meta-features tensor Frest- Then, in the model inference, as shown in Figure I, the model set performances are predicted for each available model in MM. The model Mt with the lowest predicted error score by C the time window W, of Dreat is chosen as the selected model for that window wi- Now, we explain such model selection process for each time window across the time windows wo, wI,..., WT of Dreat as follows. For the first window (wa), the inference is given by Mo E arg min M C( (( For any other window wi (with > 0), the time-series meta-learner θ inference depends on the history of the meta-features and the history of the models' performances as follows M, E arg M. (FFestt ...... F/eat Frest plast ...... p/==() On the other hand, the general meta-learner + inference depends on the predicted (regression) output on the meta-features of current time window where MiP E arg min MEM D(F(est) Thus, the final selected model is given by
M, E
arg min
plest
We emphasize that tie between models can happen in online inference (i.e., two have an identical predicted performance). We built upon the several tie breaking techniques that have been examined in the literature Kalousis (2002); Brazdil et (2008), but usually such a choice does not have a strong influence on the performance of a meta-learning system. For making the decision between the model selected by the general meta-learner & and that selected by the time-series meta-learner O, we choose the model with the least predicted error score (illustrated in equation 7). We present the time complexity analysis of AUTOFORECAST'S inference Appendix H.
models
EXPERIMENTS
We evaluate AUTOFORECAST by designing experiments answer the following research questions:
Does employing AUTOFORECAST for time-series forecasting model selection yield improved performance, as compared to no model selection, as well as other selection techniques (such meta-learners adapted from the AutoML domain)?
How much reduction in inference time does AUTOFORECAST give over the naïve method?
How does performance change with different datasets with different temporal dependence?
5.1 EXPERIMENTAL SETUP
Models and Performance Collection: By pairing seven SOTA time-series forecasting algorithms (whichare are DeepAR Salinas et al. (2020), Deep Factors Wang et al. (2019), Prophet Taylor & Letham (2018), Seasonal Naive Montero-Manso et al. (2020), Gaussian Process Yan et al. (2009), Vector Auto Regression Lewis & Reinsel (1985), and Random Forest Regressor Liaw al. (2002)) and their corresponding hyperparameters, and using different data representation methods, we compose model set M with 322 unique models (see Table for the complete list). For our testbeds, first generate the performance tensor P. by evaluating the models from M against the benchmark datasets in each testbed. For consistency, all models are built using the GluonTS Alexandrov et al.

---

Under review as a conference paper at ICLR 2022
Table 1:
Time-serie dataset corpus statistics and properties.
# time-series darasets (D) overall time-series (across all datasets) of multivariate time series datnsets
# of univariate time-series series datasets
348
625
40
308
(2020), Scikit-learn Pedregosa et al. (2011), and Statsmodels Seabold & Perktold (2010) Python libraries on an Intel @2.60 20 GHz, 16GB RAM, 8-core workstation.
Meta-Train Testbed: Meta-learning works if the new task can leverage prior knowledge. We thus create two testbeds with 348 public forecasting datasets. In particular, we collect 308 univariate time-series datasets for the first testbed (Table 10 in Appendix G) and 40 multivariate datasets with 317 time-series for the second testbed (Table 11 in Appendix G). In total, we have 625 time-series in our testbeds. We refer to Table for summary of the statistics of the corpus of time-series data we used. For each dataset in the testbeds, we use different time windows selected randomly from (.....,T)) We
the dataset, where each time window has a length of 16 (i.e.. lw, = 16 Vt emphasize that our approach has no restriction on the length of the time window.
For evaluating AUTOFORECAST, split each testbed into 5 folds for cross- cross-validanl validation. For each fold, after training the meta-learning approach, we use it to infer the best forecasting model for the new unseen test datasets. Finally, we take the average performance of these five folds. We mainly compare the Hit-at-k: accuracy (which indicates whether the selected model is within the actual top-k models), of AUTOFORECAST against different meta-learners baselines. We also compare the performance of selected model by AUTOFORECAS selected model by other meta-learners. This is measured by the mean square error (MSE) and the average rank (for each testbed, the meta-learners are first ranked by the corresponding forecasting MSE under the selected model for each dataset and then the rank is averaged across all datasets).
Evaluation:
against the performance of the
For our time-series meta-learner O explained in Section 4, we used LSTM with 4 layers where each layer has 50 units. The training was with 50 epochs with the Adam optimizer with batch size of 25 and dropout rate of 0.2 to prevent over-fitting The evaluation of the effect of such parameters on O's performance is shown in Appendix J.3.
Time-series Meta-learner Setup:
5.2 BASELINES
We adapt leading methods from algorithm selection, and include additional baselines by creating variant of the proposed AUTOFORECAST (marked with TSL). All the baselines can be organized into the following categories (the detailed description of each baseline given Appendix I).
No model selection: This category always employs either the same single model or the ensemble of all the models, e.g., Random Forest (RF) Liaw et al. (2002).
Simple meta-learners: Meta-learners in this category pick the generally well-performing forecasting model, globally or locally (via clustering or KNN): Global Best (GB), ISAC Kadioglu et al. (2010), and ARGOSMART (AS) Nikolić et al. (2013).
Optimization-based meta-learners: These Meta-learners learn task similarities via optimizing performance estimates on meta-features: Multi-layer Perceptron (MLP), and AUTOFÖRECAST-TSLL a variant which meta-learner consists only of time-series learner O.
5.3
RESULTS
Now, we show our results to answer the aforementioned evaluation questions.
5.3.1
UNIVARIATE TESTBED RESULTS
To investigate the impact of the train/test similarity on meta-learning performance, we build the univariate testbed that consists of 308 datasets (Appendix G Table 10) with diverse datastes.
Superiority of AUTOFORECAST compared to all baseline methods w.rt. the Hit-at-k, average rank, and MSE: The different results are provided in Table 2-4 where the best result for every testbed is highlighted in bold. We observe that AUTOFORECAST outperforms previous SOTA meta-learning methods adapted to our problem. For example, AUTOFORECAST has 79.20%, 171.86%. 423. 375.61% %, and 51.59% higher Hit-at-10 accuracy than GB, AS ISAC, MLP, and AUTOFORECAST-ISSS respectively. Full evaluation of univariate testhed is in Appendix J.
7

---

Under review as a conference paper at ICLR 2022
Table 2: Hit-at-k Accuracy (the higher the better) comparison of AUTOFORECAST against the different baseline meta-learners for both univariate and multivariate testbeds. AUTOFORECAST outperforms baselines for both testheds.
Dataset Testbed
Global Best
AS
ISAC
MLP
AUTOFORECAST-TSL
AUTOFORECAST
Univariate
2,46
2.15
0.82
0.62
2.67
3,95
7.18
4.92
2.67
1.13
9.04
14.57
11.97
7.89
4.10
4.51
14.15
21.45
37.40
28.00
11.45
22.25
35,28
52.05
Multivariate
6,78
2.26
4.19
0.43
5.16
5.87
12.18
4.73
5.69
1.51
9.03
13.86
16.21
9.03
7.31
4.06
11.39
20.91
41.72
24.73
14.64
20.86
35,06
51.67
Table 3: Average rank (the lower the better) comparison of AUTOFORECAST against the different baseline meta-learners for both testbeds. AUTOFORECAST outperforms all baselines.
Datasel Testbed
Global Best
AS
ISAC
MLP
AUTOSORECAT AUTOFORECAST
Univariate 2.3161
2,7965
2,9096
3.7072
2.5202
2.0571
Multivariate 2.3191
3.0851
2.3191
3.8723
2.3404
1.3191
Table 4: Results for one-step ahead forecasting (MSE) for both testbeds. The selected model by AUTOFORECAST yields better performance (i.e., lower MSE) compared to baseline meta-learners
Datnect Testhed
Global Best
AS
ISAC
MLP
ArTOFURICAST-T AUTOFORRIASTT
Univariate
U.KEES ED199
IJIIES DUGG6
0.0071 = 00044
0.RG1 = 0.1186
ELDIRES = DOLI8
0.00256 0,0090
Multivariate
0.0066 0.0080
0.0139
0.0663
0.0046 + 0.0088
0.0121 + 0.2462
0.00541 0.0186
0.00124 24 0,0851
Statistical Significance of AUTOFORECAST: To compare two methods statistically, we use the pairwise Wilcoxon rank test on performances (i..., MSE of selected models) across datasets (significance level < 0.05). Table 5 shows that AUTOFORECAST is significantly better than most of the baseline meta-learners i.e., including GB (9.07 x 10-5), AS (1.07 x 10 10-37) and AUTOFORECAST-TSL (8.16 x 10-15), Appendix J.2 has full statistical significance tests.
Meta-learners perform better than methods without model selection: As Table 14 (Appendix J) shows, the meta-learners outperforms almost all models with no model selection. In particular, three meta-learners (Global Best, ISAC, AUTOFORECAST) significantly outperform the baseline time-series forecasting models. For instance. AUTOFORECAST respectively has 92.58%, 84.39%, 88. 88.20%, 87.14%, 83, 93.48% 98.45%, and 95.75% lower MSE over Scasonal Naive, DeepAR, Deep Factors, Random Forest, Prophet, Gaussian Process, and VAR, respectively. These results signify the benefits of using meta-learning for model selection, specifically using AUTOFORECAST.
aptimization-bassed meta learners generally perform better than simple meta learners: Two of the top-3 meta learners by average rank and MSE (AUTOFORECAST and AUTOFORECAST-TSLJ are all optimization-basee as shown in Table and Table 4. The interpretation that simple meta-learnen weigh meta-features equally for task similarity, whereas optimization beaaee methods learn which meta-features matter (e.g., time-series regression on meta-features in AUTOFORREAATT-TSL) leading to better results.
and significantly outperform simple meta-learners such ISAC and AS
We present the detailed performances for each dataset by comparing AUTOFORECAST with all baseline methods Table 12. is noted these results are averaged across the different time windows for each dataset. The results shows that AUTOFORECAST achieves the best MSE and average rank among all meta-learners. We note that AUTOFORECAST and AUTOFORECAST-TTSL have same performance for datasets with higher temporal dependency.
Dateset-wise Performance:
5.3.2 MUL.TVVRRIATE TESTRED RESULTS
In this testbed, we simulate the case when there are similar meta-train tasks to the test task, where we choose variables from the multivariate for training and the rest in testing. We build the multivariate testbed that consists of 40 datasets (Table 11 in Appendix G).
For the Multivariate testbed, AUTOFORECAST still outperforms all baseline methods w.nt. average rank, MSE, and Hit-at-k accuracy as shown in Tables 2-4. Moreover, Figure 8 (Appendix J) shows that for the pool of multivariate datasets (across all time windows), AUTOFORECAST gives gain of 2X in selecting better models as compared to other meta-learning baselines. Dataset-wise performance for the datasets in the multivariate testbed shown Table 13 in Appendix J. AUTOFORECAST has the lowest average MSE most of the multivariate datasets.
8

---

Under review as a conference paper at ICLR 2022
Table Pairwise statistical test results between AUTOFORECAST and baselines by Wilcoxon signed rank test. Statistically better method (p = 0.05) shown in bold (both marked bold if no significance). In (left), Univariate testbed is shown. In (right), Multivariate testbed shown. For both testbeds, AUTOFORECAST is statistically better than most of the baseline meta-learners.
Owrs
Baseline
p-value
Ours AUTOPORECAST
Baseline
p-valte
AUTOPORECAST
AS
1.0725
AUTOFORECAST
S. 3.8399
AUTOPORECAST
ISAC
0.1349
AUTOPORECAST
ISAC
0.8260
AUTOPORECAST
ML.P
B0687
AUTOPORECAST
MEP
0.0004
AUTOPORECAST
AUTOFORICASTTTS.
8.1053 10
AUTOFORECAST
AUTOFORECAST TSL 0.0025
Rodiddeni Higooge Tiee
1.60
1.65
1.70
1.75
1.80
Figure 2:
Boxplot of AUTOFORECAST inference time for univariate testbed (in sec.). AUTOFORECAST takes less than 1.7 sec for most datasets (median = 1.65).
Figure 3: The inference time reduction of AUTOFORECAST over the naïve approach. AUTOFORECAST gives median reduction of 42X over naive approach for both testbeds.
Table 6: Inference runtime performance (in seconds) for both univariate and multivariate testbeds.
Datasct Toshod
Global Best
AS
ISAC
MLP
AUTOFOORCAST-TSS.
AUToFoRRCAST
Univarlate D. 0.6250 1 0.0964
0.85aT
0.1438
10.2480 L 2,7182
1,2745 1 0.5198
0.7962 0.0355
1,6008 1 0.0401
Multivariate
0.4151 1 0.0005
1,3055
0.3500
7.007
L,229
1,161 1 0.2267
0,682 1 0.0072
1,199 1 0.1257
Statistical Significance of AUTOFORECAST: Table 5 shows that for the Multivariate testbed, AUTOFORECAST is also significantly better than most of the baseline meta-learners, i.e., including AS (3.94 x 10-"), MLP (0.0004) and AUTOFORECAST-TSL (0.0025) and has no significance from GB and ISAC. For the full statistical significance results for all pairs see Table 15 in Appendix J.2.
5.3.3 RUNTIME ANALYSIS
Inference run time statistics of AUTOFORECAST: Figure 2 shows that AUTOFORECAST (meta-feature generation and model selection) takes less than 1.7 seconds on most time series datasets. Moreover, Figure 3 shows that AUTOFORECAST has significant reduction in inference time compared to the naive approach, median is 42X on both testbeds (see also Appendix J.5).
Comparing AUTOFORECAST with baselines: In tenms of inference, Table 6 shows that most of the meta-learners are fast taking only a few seconds to infer the best forecasting model. Finally, we compare the training cost of AUTOFORECAST against baseline meta-learners. Table 16 shows that AUTOFORECAST has comparable computational training cost. While the training process is offline and done only once and hence, not important as the inference time cost, this experiment reassures us that our better model selection performance does not entail prohibitive training cost.
6 CONCLUSION
We introduced a meta-learning approach to automate the process of time-series forecasting by automatically inferring the best time-series model on unseen dataset, without needing exhaustive evaluation of all existing models on this dataset. The problem arises because there are many possible forecasting models with their associated hyperparameters, and different choices are optimal for different datasets. Further, even within one non non-stationary dataset, different models are appropriate for different time windows. Our proposed solution AUTOFORECAST is a meta-learner, trained on extensive pool of historical time-series forecasting datasets and models. To effectively capture task similarity, we designed novel problem-specific meta-features. Extensive experiments on two large testbeds showed that AUTOFORECAST significantly improves time-series forecasting model selection over directly using some of the most popular models as well as several SOTA meta-learners. We showed that AUTOFORECAST gives a significant improvement in the inference time compared to naive approaches. We release the benchmark data for the community to contribute new datasets and models to stimulate further advances automating time-series forecasting.

---

Under review as a conference paper at ICLR 2022
Reproducibility Statement: To further research into the important problem introduced our work, we have publicly released our benchmark data to to enable others reproduce our work. In particular, we are publicly releasing, with this submission, our meta-learning database corpus of 348 datasets, containing 625 time series in all, performances of the 322 forecasting models, and meta-features for the datasets. This resource will hopefully encourage the community standardize efforts at benchmarking time series forecasting model selection. We also encourage the community to expand this resource by contributing their new datasets and models. The anonymized website with our database is: https:/ /drivediie.///e///eee/e/eeeeeeee google. com/ /rive// folders/ 1KK Ida5Cr1 5b5Fhidax- (ntpx- [Nwwzjveetttt The details of each dataset in the two testbeds, univariate (308 datasets) and multivariate (40 datasets) are explained in Appendix G (Table 10 and Table II) and the complete list of meta-features is presented in Appendix E (Table 8).
The performance collection is difficult and time-consuming researchers to begin studying the important problem formulated in our work, and developing better approaches to solve performance of each of the 322 forecasting models on all the datasets (in the above link). In particular, we collect this for the performance metrics of MSE, training time, and inference time. This serves the training data and ground truth evaluation data for AUTOFORECASS and all other competing protocols. We refer the reader to Appendix in which we provide complete description of the algorithms, libraries, and hyper-parameter values used for this collection process.
obtain, yet fundamental for
Here, we give the result of that time consuming process of collecting the
REFERENCES
Hossein Abbasimehr, Mostafa Shabani, and Mohsen Yousefi. An optimized model using 1stm network for demand forecasting. Computers & industrial engineering, 143:106435, 2020.
Mustafa Abdallah, Wo Jae Lee, Nithin Raghunathan, Charilaos Mousoulis, John W Sutherland, and Saurabh Bagchi. Anomaly detection through transfer learning in agriculture and manufacturing iot systems. arXiv preprint arXiv:2102.058// 2021.
Salisu Mamman Abdulrahman, Pavel Brazdil, Jan N. Rijn, and Joaquin Vanschoren. Speeding up algorithm selection using average ranking and active testing by introducing runtime. Mach. doi: 10.
Learn., 107(1):79-108, 2018
URL https: /to.doiorg///o//oooo////o
10. 100.100//110 0994 -687-887-888
Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider. Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C Maddix, Syama Sundar Rangapuram, David Salinas, Jasper Schulz, et al. Gluonts: Probabilistic and neural time series modeling in python. J. Mach. Learn, Res., 21 (1(166)1-66 2020.
Christoph Bergmeir and José M Benítez. On the use of cross-validation for time series predictor evaluation. Information Sciences, 191:192-213, 2012.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameere optimization. In 25th annual conference on neural information processing systems (NIPS 201 I), volume 24. Neural Information Processing Systems Foundation, 2011.
Pavel Brazdil, Christophe Giraud Carrier, Carlos Soares, and Ricardo Vilalta. Metalearning: Applications to data mining. Springer Science & Business Media, 2008.
Vitor Cerqueira, Luis Torgo, and Igor Mozetič. Evaluating time series forecasting models: An empirical study on performance estimation methods. Machine Learning, 109(1))1997-2208, 2020.
Vitor Cerqueira, Luis Torgo, and Carlos Soares. Model selection for time series forecasting: Empirical analysis of different estimators. arXiv preprint arXiv:2 /04.002044 2021.
Maximilian Christ, Nils Braun, Julius Neuffer, and Andreas W Kempa-Liehr. Time series feature extraction on basis scalable hypothesis tests (tsfresh-a python package). Neurocomputing. 307: 307: 72-77, 2018.
10

---

Under review as a conference paper at ICLR 2022
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alaie Alain Muller. Transfer learning for time series classification. In 2018 IEEE international conference on big data (Big Data), pp. 1367-1376. IEEE, 2018.
Matthias Feurer, Jost Springenberg, and Frank Hutter. Initializing bayesian hyperparameter optimization via meta-learning In Proceedings of the AAAI Conference on Artificial Intelligecce, volume 29, 2015.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Jean- Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. arXiv preprint arXiv: 10738, 2019.
Ali Hooshmand and Ratnesh Sharma. Energy predictive models with limited data using transfer learning. In Proceedings of the Tenth ACM International Conference on Fulure Energy Systems, pp. 12-16, 2019.
Serdar Kadioglu, Yuri Malitsky, Meinolf Sellmann, and Kevin Tiemny. Isac-instance-spccinn algorithm configuration. In ECAI, volume 215, pp. 751-756. Citeseer, 2010.
Kaggle. Time Series Forecasting Datasets. https: //www. http://wwwww.weeeee/e/oo/o com cecom/eaarees sea rch ?q=t
series+forecast ienestinngtt inniiinnnnnnn 3Adatasetss, Stddttttsttttiggni.ttttttttt... 2021. [Online; accessed 21-May-2021].
Prajakta Kalckar et al. Time series forecasting using holtewinters exponential smoothing. Kanwal Rekhi school of information Technology, 4329008(13): 1-33 2004.
Alexandros Kalousis. Algorithum selection via meta-learning. PhD thesis, University of Geneva, 2002.
Richard Lewis and Gregory C Reinsel. Prediction of multivariate time series by autoregressive model fitting. Journal of multivariate analysis, 16(3):393 411, 1985.
Andy Liaw, Matthew Wiener, et al. Classification and regression by randomforest. R news, 2(3): 18-22, 2002.
Gaurav Mittal, Chang Liu, Nikolaos Karianakis, Victor Fragoso, Mei Chen, and Yun Fu. Hyperstar: Task-aware hyperparameters for deep networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8736 8745, 2020.
Pablo Montero-Manse George Athanasopoulos, Rob Hyndman, and Thiyanga S Talagala. Fforma: Feature- based forecast model averaging. International Journal of Forecasting, 36(1):86 36(1)) 92, 2020.
Jyoti Narwariya, Pankaj Malhotra. Lovekesh Vig. Gautam Shroff, and TV Vishnu. Meta-learning for few-shot time series classification. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pp. 28- 36. 2020.
Mladen Nikolić, Filip Marić, and Predrag Janičié. Simple algorithm portfolio for sat. Artificial Intelligence Review, 40(4):457-46) 2013.
Boris Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint 17X1v:1905.10437, 2019.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. Meta-learning framework with applications to zero-shot time-series forecasting. arXiv:2002.028877 2020.
Zheyi Pan, Wentao Zhang, Yuxuan Liang, Weinan Zhang, Yong Yu, Junbo Zhang, and Yu Zheng. Spatio-temporae meta learning for urban traffic prediction. IEEE Transactions on Knowledge and Data Engineering, pp. 1-1, 2020. doi: 10.1 10.100//0DD...20029955
Fabian Pedregosa, Gaël Varoquaux. Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Seikit-leam: Machine learning in python. the Journal of machine Learning research, 12:2825-2830. 2011.
11

---

Under review as a conference paper at ICLR 2022
Arnak Poghosyan, Ashot Harutyunyan, Naira Grigoryan, Clement Pang, George Oganesyan, Sirak Ghazaryan, and Narek Hovhannisyan. An enterprise time series forecasting system for cloud applications using transfer learning. Sensors, 21(5):1590, 2021.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectivvneess of maml. In International Conference Learning Representations, 2020. URL httpt:ttt//// /ttp:ps:///perreeeeee revieee ew. net. to///orrmm/// (it.rrunnnnirrkkkkkk.r..
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Mauro Ribeiro, Katarina Grolinger, Hany ElYamany, Wilson A Higashino, and Miriam AM Capretz. Transfer learning with seasonal and trend adjustment for cross-building energy forecasting. Energy and Buildings, 165:352-363, 2018.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference Learning Reprexentations, 2019. Utthttps: www.reeeewwww....m.. net forum? id-BJgk1hAcK7,
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020.
Skipper Seabold and Josef Perktold. Statsmodels: Econometric and statistical modeling with python. In Proceedings of the 9th Python in Science Conference, volume 57, pp. 61. Austin, TX, 2010.
Qi Shi, Mohamed Abdel-Aty, and Jaeyoung Lee. A bayesian ridge regression analysis of congestion's 's impact on urban expressway safety. Accident Analysis & Prevention, 88:124-137, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: /ptp:////cccceeeessoo/oo mew.eeiiiinnnnnnnni..........iee neurips. wwwtarisscccpppppeee cctpappr//011// Paapeee.-p....p..........
Thiyanga S Talagala, Rob J Hyndman, George Athanasopoulos, et al. Meta-learning how to forecast time series. Monash Econometrics and Business Statistics Working Pupers, 6:18, 2018.
Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 72(1):37-45, 2018.
Evaldas Vaiciukynas, Paulius Danenas, Vilius Kontrimas, and Rimantas Butleris. Meta-learning for time series forecasting ensemble. arXiv preprint arXiv:2011.10545 2020.
Joaquin Vanschoren. Meta-learning: A survey, arXiv preprint arXiv: 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL httpss:/prooeeednngg neurips httpppp.ccppppppeeeeee htcc//pppr/221////// .df..
Yuyang Wang, Alex Smola. Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski. Deep factors for forecasting. In International Conference on Machine Learning, pp. 6607-6617. PMLR, 2019.
Tailai Wen and Roy Keyes. Time series anomaly detection using convolutional neural networks and transfer learning. arXiv preprint /905.
090../955...88 /3628, 2019.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Scalable gaussian process-baat based transfer surrogates for hyperparameter optimization. Machine Learning, 107(1):43-78, 2018.
12

---

Under review as a conference paper at ICLR 2022
David H Wolpert. The lack of priori distinctions between learning algorithms. Neural computation, 877)13441-330, 1996.
Weizhong Yan, Hai Qiu, and Ya Xue. Gaussian process for long-term time-series forecasting. In 2009 International Joint Conference on Neural Networks, pp. 3420-3427, IEEE, 2009.
Quanming Yao, Mengshuo Wang, Yuqiang Chen, Wenyuan Dai, Yu-Feng Li, Wei-Wei Tu, Qiang Yang, and Yang Yu. Taking human out of learning applications: A survey on automated machine learning. XrXiv preprint 1810. 13306, 2018.
Rui Ye and Qun Dai. A novel transfer learning framework for time series forecasting. Knowledge-Bodd Based Systems, 156:74- 99, 2018.
Yue Zhao, Ryan A Rossi, and Leman Akoglu. Automating outlier detection via meta-learning. arXiv preprint wwX.v:2009.10606, 2020.
13

---

Under review as a conference paper at ICLR 2022
APPENDIX
A VALIDATION OF NO-SINGLE-MODEL HYPOTHESIS ON OUR TESTBEDS
Figure shows a validation of the hypothesis that there is no unique single model that works well on all datasets, where we show histogram of the best models probability distribution across the datasets of the training testbed where different datasets have different (best) models. That motivates the need for our framework, AUTOFORECAST, for automating model selection via meta-learning.
Training
0.007
0.005
0.004
Froxs/alty
0.002
0.001
100
355
200
250
500
Forecauting Model Index
Figure 4: A histogram of the best forecasting model's probability distribution across the datasets of the training testbed. We observe that different datasets have different best models and no single model triumphs across all datasets. This motivates our automatic model selection problem.
B VARIATION OF BEST MODEL ACROSS TIME WINDOWS
Figure validate the no-free lunch theorem across different datasets (vertically) and across different time windows within the same dataset (Horizontally). Figure shows the aggregate statistics on all datasets of the univariate testbed. This contradicts the claims that one forecasting algorithm can work best for different datasets and motivates the need for an effective approach for learning such both dimensions for selecting the best model for new tasks, which we propose in our current work.
Procueed NASDAG-
Prophet cossesiligg poice 8.11)
Regnasiion
Desp
Semscnel
ooh 55 Sepen111
DospAt
Randors Rarent Regwosar
Pali
(yyesut)
Advic 15 den...
DeepAR
DespAN (n Eoluek eauya mngra_Jageraa
DespAR tnaumeh rejure Invensi)
stagert:: GAZeeerips
mo.mn (nyerrd)
Figure 5: The best forecasting model for each time window (across three consecutive time windows) for different datasets. We observe that different time windows have different (best) models.
C SUMMARY OF NOTATION
We summarize the notation used in this paper in Table 7.
C.1 WINDOW NOTATION IN AUTOFORECAST
We now reemphasize the notation of time window used in Section 4. In AUTOFORECAST, the time window represents a sequence of time observations in the time series. particular, wa denotes the time window index, and |wyl is the length of that time window (e.g.. |woo 16 means that the 10th time window of the dataset has a length of 16 observations). Notice that for each time window
wi,, have the pair (2i lwy lat 1t-1: 1, Mt-ttr --)) where the different models are trained Z /t/l:t111 and
14

---

Under review as a conference paper at ICLR 2022
Namber Bast Modet Sintwrtie
Sono Foresasing Medat and Sano Hyper porometerss
Same reresasing Medsa and bitteent Hypersuraeeten Diftrent Forecasing Models
200
150
50
Figure 6: The aggregate statisties for similarity in the best forecasting model across three consecutive time windows for univariate testbed. Most different time windows have different (best) models.
Table 7: A summary of our notation.
Symbol
Description
Dreain
Meta-train datahase
Meta-train databuse size observations
No. of variables in D,
Model space forecasting model Size of the model space Hyperparameeerr vector of Number of time windows
M,
m
Window length
Performance
Performance matrix of window une
Model M, performance oa time window on Models performances vector on time window for D. represcatation
ed-)
F
Meta-features tensor of D.
Meta- Meature matrix window D;
e())
Meta-feattuee extraction module
d
Number of meta-features
C
Akte-lcamerr
General Meta-learner
Time-series Meta-leamner
Mo. Mr
Selected model for time window my by and 49, respectively
Selected model window en inference AUtoFoRECASt
evaluated on Yeur H. where H is the forecasting horizon. For example, for single-step forecasting on window Wt with length 16. we would train the forecasting model on the first 15 observation points of the window and do forecasting on the 16th observation. Second, we emphasize that we can train a forecasting model on windows with any length, but the forecasting horizon may be limited by the forecasting model parameters.
D ADDITIONAL DETAILS OF TIME-SERIES META-LEARNE
We detail the equations of our LSTM-based time-series meta-learner O The LSTM cell time t has two recurrent features, denoted by h and ct, called the hidden state and the cell state, respectively. The LSTM cell consists of three layers (forget gate layer, input gate layer, and output gate layer). The activation of those layers is given by
ft (W, h
X,) by)
(8)
! WW: [ X,]
+ bi).
(9)
of = a (W. [h] b.)
(10)
where W,, Wi, W. and by. b, b, E R denote the weights matrices and the biases of the three layers, respectively. These are the parameters to be learned during the training of the time-series meta-learner. The forget gate fü controls how much of the current cell state we should forget, the input gate I controls how much the cell update added the cell state, and the output gate o controls how much of the modified cell state should leave the cell and become the next hidden state. Moreover, cell update u is constructed with a tanh activation function as follows.
n tanh (W.. [hi X.) + bw).
(11)
15

---

Under review as a conference paper at ICLR 2022
where W, and by R are further weight parameters to be learned. Thus, the new cell and hidden states at time are given by
cj fi c/11 +4-
(12)
h of tanh(c))
(13)
Finally, the output equations of the LSTM cell are given by
V W.h b.
(14)
pi Softmax(Vi))
(15)
where W. and b, E TR are weight parameters to learned. This gives the relationship between input X₁ and predicted performances output pi. For ease of notation, considered single layer in the above analysis. However, we have used multiple layers with several units (see Section 5).
E TIME-SERIES META FEATURES
There are prior works that generated standard time-series features Franceschi et al. (2019), tsfresh Christ et al. (2018) (that we used for generating part of our meta-features). We now provide details of our meta meta-reatures and their different categories.
E. FETA-REATURES CATEGORIES:
For each dataset, generate meta-feature vector that consists of more than 800 meta-features. This meta-features vector includes different components that capture various properties of the time-series dataset, e.g., the statistical features (number of crossings, count of observations above/below the mean value, quantiles, etc.), the data trend (linear, non-linear, dynamics, etc.), data interdependence (lag autocorrelation, difference features, etc.), information-thhorreee theoretic features that are typically based on entropy measures, and the frequency (FFT, wavelet, etc.). Our meta-features vector also includes landmarker features, which problem-specifie, and aim capture the unique characteristics of dataset. The idea is to apply few of the fast, casy-to-construct time-series forecasting models on dataset and extract features from (i) the structure of the estimated forecasting model, and (ii) its output performance scores.
Multivariate meta-features: For this time-series dataset type, we have two types (explained in Appendix G). For both types, we generate a single meta-feature vector of the multivariate dataset by averaging the meta-features vectors of all variables within the dataset.
E.2 COMPLETE LIST OF FEATURES
We summarize the meta-features used by AUTOFORECAST in Table 8. When applicable, we provide the formula for computing the meta-feature(s) and corresponding variants we used for generating the meta-features, and the corresponding number of features. Some are based Vanschoren (2018). Specifically, our meta-features can be categorized into (1) simple features, (2) statistical features, (3) informatioonthhoreee features, (4) Spectral features, and (5) landmarker features. Broadly speaking, the statistical features captures statistical properties of the underlying data distributions; e.g., min, max, variance, skewness, covariance, etc. of the features and feature combinations. The informationn-theoreie features capture information-theoreti underlying characteristics in the time-series; e.g., entropy, trend, non-linearity, change statistics, etc. Most of those featuressuses have been commonly used in the AutoML literature Vanschoren (2018). To the best of our knowledge, we emphasize that our landmarker meta-features (detailed below) are novel and that some components of the spectral meta-features have not been used in any related work.
E.3 LANDMARKER META-FEATURE GENERATION
In addition to simple, statistical, and information-theorett meta-features, we use three time-series forecasting landmarker algorithms for computing precasting-speciin landmarker meta-featuree Auto Regression Lewis & Reinsel (1985), Random Forest Liaw et al. (2002), and Bayesian Ridge Regression Shi et al. (2016) capture landmarker characteristics of time-series dataset.
16

---

Under review as a conference paper at ICLR 2022
We now provide quick overview of each algorithm and then discuss how using them for building mettafeetures. The algorithms are executed with the default parameter.
Auto Regression Lewis & Reinsel (1985): In this landamarker feature, we fits the unconditional maximum likelihood an autoregressive process. The k parameter represents the maximum lag of the process
X; = Po Σ pix, +
==1
Then, extract the coefficients 6i whose index (0..... ,k)..
Random Forest Liaw et al. (2002): is a tree-based ensemble method that builds a collection of base trees using the subsampled unlabeled data, splitting on (randomly selected) features as nodes. Random Forest grows internal nodes until the terminal leaves contain only one sample or the predefined max depth is reached.
For Random Forest, we use the balance of hase trees (i.e., depth of trees and number of leaves per tree) and additional information (e.g., feature importance of each base tree). It is noted that feature importance information is available for each base tree-we therefore analyze the statistic of mean and max of base tree feature importance. The following information of base trees are used:
Tree depth: min, max, mean, std, skewness, and kurtosis
Number of leaves: min, max, mean, std, skewness, and kurtosis
Mean base tree feature importance: min, max, mean, std, skewness, and kurtosis
Max of base tree feature importance: min, max, mean, std, skewness, and kurtosis
Out-of-bag estimate score.
Bayesian Ridge Regression Shi et al. (2016): it estimates Gaussian probabilistic model of the regression problem using Bayesian regression. The prior for the coefficient is given by a spherical Gaussian distribution.
For Bayesian Ridge Regression, we use the following information of the probabilistic model (fitted using the dataset) as landmarker features:
Mean of the distribution
Log marginal Likelihood score
The precision of the estimated weights
The noise precision
The actual number of iterations achieve the stopping criterion
F
MODEL SPACE
We now provide details on the model space used study the meta-learning problem formulated in our work. Recall that a model in the context of our problem is a time-series forecasting algorithm and the hyperparameters used.
DeepAR Salinas et al. (2020): DeepAR experiments are using the model implementation provided by GluonTS version 1.7. We did grid search on different values of number of cells and the number of RNN layers hyperparameters of DeepAR since the defaults provided in GluonTS would often lead to apparently suboptimal performance on many of the datasets. The training parameters for each dataset are described in Table 9. All other parameters are defaults of
Deep Factors Wang et al. (2019): Deep Factors experiments are using the model implementation provided by GluonTS version 1.7. We did grid search over the number of units per hidden layer for the global RNN model and the number of global factors hyperparameters of Deep Factors. The training parameters for each dataset are described Table 9. All other parameters are defaults of gluonts. model.deode factor. SewooDDeeeFaatottssinnee
17

---

Under review as a conference paper at ICLR 2022
Table 8: Time-series meta-features for characterizing an arbitrary time-series dataset. We extracts a comprehensive number of meta-features. The extracted meta-features are five categories: simple, statistical, information-theoreee best of our knowledge, we emphasize that our landmarker meta-features are novel and that some components of spectral meta-features have not been used in any related work.
meta-features, spectral, and landmarker meta-features. To the
Nume
Property
Variants
Feature
SpaalSeatiillty
Norber Tase waisbles
NiA
Lag Kupocmmasutee
Feauurz Intnkpanina
Lag Paniel Autooereaaion
Feaure Intrrppatii
acljus P.c. H.DC, (US)] max depticaios
15
Standard Devizoos Rarge
2r r(mss min)
Disgersios
Masimum
Data Rangs
Poris
phaks-aupprtt www.ee.onittm..........
Revcomeeez Statiates
Dass Raago
Midiaa
Cosceatrtion
mis me
Mean
Dinccttrttion
Natiuae
Disgersion
55d dey baniiee somm
Cver
Figerria
dul 913
Digersioa
dit-qqaaniles
Mass Quaniles
Count below Meas
Suiiiics
mis.mas.t7. mith mas.7.
Coast About Moa
PIX p]
Suiiiics
Number of Crosings
lxx
Stisiics
2010 sross oue crisk, tsiss coos, root lop usi
Fesm nommiliy
exmple Jomn
nommilit
comple,,ooo
Syrtizir
ReversiLAAyyman Asymuaee
E'Y''Y" (0X) ((N) X²]
Reveral Asyrinetry
lag.L.
Abullue Sam Change
Dilievenes
che, als sateesheeee
Charge Quaetils
Corike Quantiee
Dutteeecce
t. R. Sa in
60)
Entrags
Regularty
agpros, Jemmpss samipeeemmopy
Trea)
Iee chunks
Trexal
("poalee" : "realae" "intereep""
"stder")
Nep-
Fishill Cocaiicaas
Cocils
Wordltt Transfom Cocflsicats
Mexicun lat wunciet
Tine
"wadlu", "voefl""
Fei Tharker Trandom Caasiiiiets
IVTX) FFT:X)
Ieaquraay
("hod" "treg" 'agg (rarice)
201
Polai Fist Fo Transoou Coedlillests
Fequeecy
""hb", "ungle scetric
200
Abseluee Energe
Spectial
Energy chunks crosi.pe spect dets
Special Rendaariy
prinppy
Regrasion Cocfisiens
Saz Appendsx
Lastmarkr
de (L...r,10)
10
(7) Randon Fooed
Nurber Loaves
Appendx See Appands
Lasimarke
rakk, siis, mean sid
skew, kustosi
depis
Maar of Boee The hourr Impertioonn
Lantrorker
min mor, skrw kamois mure min menn, vid, skrw and karodds Nix.
Mis of Row The Pratum Imporaaee
Sox Appords
Landrooker
Out of bug cslinne soore
Set Appezdxx
Lastmakke
Bayesion
Meas Destibbtiion
Appendx
Lastmarker
NGA
Lay Marginal Likelibaod
Saz Appenda
Lastmarker
NIA
Eatiatted Woghte Precision
Sec Appzedis
Landmeker
NIA
Batimated Noiso Procision
See Appezdix Appendi
Lastrorker
NIA
Stepping
Table 9: Time-Series Forecasting Model Space. See hyperparameter definitions for various algorithms from GluonTS (Alexandrov et al., 2020) and statsmodels (Seabold & Perktold, 2010).
Forecusting Algoeithm
HyperParamctee 1 numuralk = (102003,,0050
HyperParameter 2
Data Representation
Total
DesppA
Joyers = (123345)
Fxgurmesshing Rew
90
Deep//ocioo
hidden 10,0000
facors -13.015550
[xp smooeting Ri
30
Prophit
changepooll
[0000, GOT, 02.02.
1essonalee 1001,
100
Exp snoochingg Ri
30
NAA Jin_Jern_ 15,2055
Exp
16
50
Repozion
"HCI"
Eaguameeeiing
Regresoor
nLAciniiooos (002500005000000
mat. Jepdi 125,
1xp snposineiiingg
Prophet Taylor & Letham (2018): Prophet experiments are using the model python implementation provided by Facebook (fbprophet) version 0.7.1. We did grid search over the change point prior scale and the seasonality prior scale hyperparameters of Prophet. The training parameters for each dataset are described Table 9. All other parameters are defaults of Spprophet.Prophed.
Seasonal Naive Montero-Manso et al. (2020): Seasonal Naive experiments are using the model implementation provided by GluonTS version 1.7. We did grid search over the length of seasonality pattern, since it is different unknown for each, dataset hyperparameter of Seasonal Naive. The training parameters for each dataset are described in Table 9. All other parameters are defaults of model.
Gaussian Process Yan et al. (2009): Gaussian Process experiments are using the model implementation provided by GluonTS version 1.7. We did grid search over the cardinality of the time-series and the maximum number of iterations for jitter to iteratively make the matrix positive definite hyperparameter of Gaussian Proces. The training parameters for each dataset are described in Table 9. All other parameters are defaults of gluonts.model.go
18

---

Under review as a conference paper at ICLR 2022
Vector Auto Regression Lewis & Reinsel (1985): Vector Auto Regression experiments using the model implementation provided by statsmodels python library version 0.12.2. We did grid search over the loss covariance type and the trend hyperparameter of Vector Auto Regression. The training parameters for each dataset are described Table 9. All other parameters are defaults of statsmodels.tss var var_model.
Liaw et al. (2002): Random Forest models' experiments are using the model implementation provided by sklearn python library version 0.24.2. We did grid search over the the number of estimators (trees) and the max_depth (i.e., the longest path between the root node and the leaf node in tree) hyperparameter of Random Forest. The training parameters for each dataset are described in Table 9. All other parameters are defaults of sklearn, www.emeenmmmbl..mm haw.bie.nnnoooooooeeerreeee
Random Forest
Exponential Smoothing Data Representation Kalekar al. (2004): We use the exponential smoothing from the statsmodels python library version 0.12.2. We optimized the smoothing level for the exponential smoothing (which controls the weights given for the history samples). All other parameters are defaults of statsmodels. statsmoiels.ssaa...xppoeeiiiiii wwp.p.EEpppeeniiammmttiiiig
G DATASET DESCRIPTION
G.1 DATASET TYPES IN AUTOFORECAST
In our work, consider two general types of time-series datasets depending on the number of the variables U; in the time-series dataset. Now, we detail the different dataset types.
Univariate Time-series Datasets: This kind of datasets is the traditional single time-series which usually consists of single variable that need to be predicted (this variable is likely numerical, but can also be categorical in rare cases). Formally, time-series dataset D; is single-variate if v; 1.
Multivariate Time-series Datasets: In our work, we consider two main types (subcategories) of the Multivariate datasets that define below. We define the Multivariate Homogeneous dataset as the multivariate dataset that consists of multiple time-series in which each time-series represents the same metric (e.g., collection of time-series representing CPU usage of different machines). In this type, t; = (the number of different time-series for that same measurement). Also, we define the Multivariate Heterogeneous dataset as multivariate time-series where each time-series column represents different measurement (e.g., wind speed, humidity, temperature). In this type, 0 ccolurmss D:)l (the number of the different variables for the time-series dataset).
G.2 DATASET TESTBEDS IN AUTOFORECAST
Dataset Sources: Our testbeds are built to simulate the testbed when meta-train comes from many different distributions. This diversity makes the meta-learning model learn from such diversity. Model selection on test data can thus benefit from the prior experience the train set. For this purpose, we use benchmark datasets from Kaggle Kaggle (2021), Adobe real traces, and other open source repositories. In particular, the Adobe trace datasets records CPU and Memory usage for 50 different services running in Adobe production clusters collected for 15 days from May to May 15 in 2021. Such traces are shared for the first time in our current work.
Testbeds Summary: In short, we have collected 308 univariate datasets from such different sources. The details of the datasets (i.e., the dataset name, and the number of points in the dataset) are shown in Table 10. We also collected 40 multivariate datasets, where each dataset can have different number of variables and different type. For instance, Adobe service CPU and memory 15 days utilization is Homogeneous multivariate dataset that has r 100. Most of the datasets are from different application domains (e.g., finance, loT, energy, storage, etc.). The details of the datasets (i.e., the dataset name, the variate name, and the number of points for each variate the dataset) are detailed in Table 11. For robustness, for each testbed, we split its datasets into folds for cross-vass gross-validation. We build the train/test testbed by each time selecting four folds from the datasets for training and the remaining fifth fold for testing. We re-emphasize that we released the datastes for the community for future usage (link is in Section 1).
19

---

Under review as a conference paper at ICLR 2022
Table 10: Univariate Time-series dataset corpus description and details. The details of the datasets (i.e., the dataset name, and number of points in the dataset) are shown.
Time-scries Name
# pts
Time-scries Name
# pls
Timee-crris Name
# pls
Time-scrios Name
# pls
Time-scrios Name
# pls
OLDMANII
1461
ALLI
97
NONEMERG.
319
IV.I
44
HOPEDALE.
101
WOLE.1
71
FRASER.1
946
EGDEMAN.2
100
[BM_I
S7
LACSTIIN,I
1440
HARROR.1
SERIESK3
PORKH
99
construction
ESPANOL
668
RIOTITTE.
372
adp_soattin
MINIMUM.I
848
TURTLE.
672
OLDMAN.1
1420
THAMES.1
71
SERIESE,
100
spS00o_picce
123
SP500.1
99
US.1
100
Y.I
BIGCONE,1
LACSTRRL
FEEDHLI
SERIESG.1
FISHER.1
1470
GLOBWARM.I
129
TBILLS.
102
CCPLI
102
CAMPITO.1
5405
JOE3
1294
CN.1
44
SERIESTY:
307
bank
581
PRECIPT
1096
GNPR.1
busiav
MISINAB.I
JUDITHLI
TEMPER
RING.1
LAKEVIEW.1
544
DAILYSAP1
3333
GEODUCK.1
97
HALSEY.1
108
DAL.I
70
SALESX.1
93
AMAZON
55
IP.I
111
well log
675
PCRGNP1
Stor
137
FEEDLII
95
DAILYIBM.I
3333
SERIESC1
228
FRNCHB.1
MAD.I
552
UN.1
81
JAMES.1
600
CPL.
288
LYNDPIN
136
OZONE.1
228
GRANT.I
151
SUNSPOTS.
289
KIEWALI
72
NEUMUNAS.
132
YD.I
CHICKNYC.I
498
NILEIJI
75
robocalls
52
AMERICAN.I
R
USMLI
P
ENGINES.1
NIAGARA.1
1861
YULELII
106
SERIESA.1
2000
ASKEW4.1
660
homeruns
118
SKUNK
71
stocks.price
560
PRGNPI
82
OLDMANE
1507
OOSTANAU
816
DANUBE.1
LACSTISN.1
EMERGING.1
CAFFEONE.1
us population
PLSUPER.
104
CIG.3
138
SIMAR4
818
SUNSPTMO.1
2820
ENGLISH
660
SERIESKT
BOISE
588
HREEDMAN.1
58
ASKEW3.1
708
global o022
104
WHEATT
E
VATNSD.1
SERIESIX
QBIRTHS.
MARTEN.1
FOOD.1
178
CONSUMI1
147
FEED
95
EXSHAW.1
506
WHITEMIN.I
1164
centralia
15
TSEOIL I
361
BRYCE.
625
MADISON.1
4.56
apple
622
GOLDH.1
MUSKRATI
71
BAYDU.
358
jok_aasengerrs
468
SFSKYKOM.1
456
SERIESD.1
312
LOGISTIC
200
VELMON
86
PEAS.
768
OTTER.L.I
71
METALS.I
178
BIRTHS.1
59
CURRENT.I
468
ASKEWS.1
108
ASKEW13.1
372
ELBE.1
300
FRNCHA.1
70
PLHURON
104
TOTAL.1
319
unemployment.nl
BEARDS.1
RIOGRAND.
occupancy
COLUM.I
EMP1
81
RGNE1
mil_line lines
37
PIPER.
348
GNPN.1
85
SPIRITS.3
254
NIGERIA.1
123
MBOULDER.I
588
deblircland
21
IPL,
85
CIGB.2
GOLDL11
CMINEF11
NILE2.1
ozone
ASKEW7.1
600
GLOBTRI
136
gdp
58
TIOGA.1
661
SERIESB.I
385
G.1
46
MCKEN
chiktren
301
YULE2,1
107
ISH66.1
163
GOTA.1
lotJempp
PPHII..1
WOODS.1
629
DIWERK.1
SAUGEEN.1
1403
USH.1
100
mike
100
co2.camdda
215
NAVAJO.1
700
TRADE_1
178
GRUEN.1 I
53
PACK.2
344
bee waggle 6
609
ARCTIC.1
66
whin_eemp
6074
shanghai Jicense
205
WG.1
71
NYSE.1
87
AZUSA.1
180
SERIESL.2
549
SOY.1
FORTALEZI
150
NMAGNETI
752
AROSA.1
480
FURNAS.DAT
576
LAKEMICILI
115
MCKENZIE.1
600
ROCKY.1
122
SNOW
S4
GUELPH.1
gdp arggnnnnnnn
SOYL.I
99
SAUGEENP.I
1412
PORK,I
99
coal employ
ASKEWIA.1
TRINITY.1
OGDEN.I
TPMON.1
2976
1.1
DRATHS.1
319
SSASK.1
780
SOYH.I
99
SAUGEENTI
1412
SKIRTS.J
M.1
85
OKAK.1
109
EAGLECOL.1
858
ASKEW10.1
WOLVEREN.1
SCHOLES.
STOOHNS.
SERIESL2
616
NILEMON.1
910
BOXHULI
48
YEAR.
208
BOXHUN.
217
CORN.
76
GNP]
SUMMER,
208
BLUME
64
NARAMATA, I
S15
HURON.1
USM2.1
CRYER.I
247
MITOUIS.1
MEASLNYC.J
534
scabbllt
192
ASKEW12.1
456
NILE.
75
ALIGN.1
55
USL.
100
ELECUS.1
51
SERIESB2.1
271
brent spot
500
measles
991
RWG.1
SNAKE_1
REDDEER
396
MUMPS.1
S34
CANFIRE
71
CMINER.1
528
CLEARWAT.
600
bitcoin
774
NYWATER. I
71
ASKEW15.1
452
ASKEW1
264
NECHES.
564
RACOON.1
71
ASKEWS.1
4.6
CD.I
44
RAPPAHAN.I
600
AARIVINT.
213
sdp japan
58
WBDELAWA.I
540
TPYRI
248
VEL.1
TRANEQ.1
NINEMILE
771
HEBRON.1
NAMAKAN.I
DELLI
655
PLMICH.I
104
SERIESM.2
300
MINK.I
71
man_ing
376
SUNSPTI
261
CO2.1
192
PAPER.2
320
FEATHER.1
708
FLOW.1
468
FISHERT.I
PREC.1
JOKULSALI
HBCO.1
WINTER.1
NAIN.1
109
CMINET.
S28
WATERQ.1
147
I
402
BND.1
71
LYNX.1
154
GOLD.I
97
Igo. passengers
468
FIGEON.1
636
USM3.1
398
AISHERPT
IRONSUA
SWATH-R1
HANKOU.1
DVLJ
470
RICHELU.
468
DJ.I
157
U.I
85
H DISCUSSION ON INFERENCE TIME COMPLEXITY
Recall that the number of meta-features is d, the number of models is m. The time complexity for the inference part of the general meta-learner & (d). On the other hand, the time complexity of the time-series LSTM meta-learner θ is given by O(X,,), where [X,] is the length of the input sequence. Thus, AUTOFORECAST'S inference time is given by max(O O()x,)). We emphasize that the inference time of the naïve method is much larger since it is given by 0 4t), where I, is the inference time of forecasting algorithm ag. The aggregate statistics for reduction in inference time across both testbeds is shown in Appendix J.5. Moreover, Figure 7 shows that AUTOFORECAST yields significant reduction in inference time (68X reduction) over the naive approach the Adobe trace CPU and Memory 15 days usage dataset.
20

---

Under review as a conference paper at ICLR 2022
Table 11: Multivariate Time-series dataset corpus description and details (i.e., the dataset name, the variate name, and the number of points for each variate of the dataset).
EST
1521
RIC
Sap Procared SAP
Procoeed
1964
Precased
S&7
MSF
1964
Pricosed
547
2934
Procesal
ROC_j
1964
12
2006
F2
2006
vonc.edl
16
250
112
3506
JEC
1506
vede
710
2506
weigeRiioesp.. it
10131
19735
on
R14,3
19735
Tranncions
D11
F30
Sake Tranicher
F1
Trasssbes
Trassahes
(9)
911
216
Sike.
p07
52
R63370
136
XAROOEETT 363370
ST7
96
(W (66 1776
S18
5515
5013
Xarwokk Banggpa
B1
Bu couge
cu
(M) cap. prrclien
cur og
Bikk Bonkhlgg
New an umags
baay roa .000
X senie._20i
1230
lacy rapu 3.00
136
kady FRg 1 3400
170
Adde,
eagend- sudpe
2016
AdBu.Ct: Max
2016
Adkla Max
eapetti OAhau
ABBucTT Mes.
STGD
16d
15g
5016
A640,CTC Mos
2016
Addocct: Mox
sugill Glandrau
S0
5TT andipe
2016
s44c.Cn: Man ihc
Sute
Aa4c (TL. Mon 1ng
5m
Man 1nd
Site
Adke CTC, Mas isc
sagetil-baat anday
J442 CTV. Man 1og
medpe--chhiillaeee andecs
5014
Adkic CTC
Mox 156
2014
sade (9) Mes 150
N016
1680 Miee 146
polgee Dodaddesolg
Mex 146
pentpet Pilltookps
2011
yreell PuSphiokk pookge
2016
prenti
predell Philbaetee
am
an1
2016
ASbu.TT Mes. (5c
5016
Mos 154
5016
156
2014
A040.C9: Max
201
201
2016
Mee (54
digros Dhewwer
2016
4485r1 (es.rrt-Min Mox
301
2016
M4
Aaa: CTC, Mas ise
2016
4682 (TL. Mas iss
2016
Adko CT. Mon 156
201e
Asde CTC Mas ise
2016
Ad84 CTV Mas 156
eapeil Ouartl-
2016
Adas CTC Mot 156
2016
Adde CTC Mas itc
2016
Adde CTC Mos 156
eagiiti-Q Goetl---eertteenn
S016
Adds CBC Mot 156
2016
Aggnil-
2m
11 Mon
ams
2016
vegenti Stuset-
ASecLit: Man
2m1
AseciT Man
AaR-E Mas 1o1
2010
201
2016
A84a. Meo.
Mey 156
degalll gal
2010
cagall griiieodii
dagalll Oheudion
2016
podias 18 po
2016
206
uggill gloodoows
2016
2554
XLF
104
Poap 11715
Kl
3554
SEF
254
25.4
qully 2.930
q483) anrd
133
100p 100
35/6
keey 2.011
2011
ka0y.rn..1 (00
2671
baor mon- 350
kady 1pN
137
laor rokk (30
1215
sacy 1824 130
kasy (80)
314
1esr.0n___10 1.31
544
109
Tracaoe NASD
EITDS
1964
NA5D
EMAA
1964
Tracssed NADD
FCID
7954
Praceed
XA5D
BUR
1964
NA5D
1964
Pracaeed NA5D
DMA 200
3954
5163
5143
133
131
135
135
des 788
5143
Ucx
$145
Dax
1143
TXXXF
1964
Unn
pio
Proe Uhll
1964
Pro
Bock
Procese
1901
Rrecoue B0S5
Beza
1964
Peccond 31555
ALD
Prsccue BLY5
AAP.
2014
Procrsed BUSS
oopperd
1984
2 2115
CTT
1964
Preeoue 2155
AM224
1904
Prscosed
1001
T3E
Goki
195.
Procered J013
1964
GRAXX
Procecid
1961
Neor mak ax1
344
kacy raN 230
T15
kazy ripu 2 20)
X
beer ros 3
545
5034
504
5034
Netwook
capaity perticead
504
fui- man-nagg-
CTU
caos
B634
I BASELINES DETAILS
Meta-learning has recently received a significant attention for automating ML pipelines for variety of different problems outside the domain of time-series forecasting including supervised learning Feurer et al. (2015): Wistuba et al. (2018), classification and regression Finn et al. (2017); Rusu et al. (2019), unsupervised learning Abdulrahman et al. (2018), outlier detection Zhao et al. (2020) and other applications Mittal et al. (2020); Vinyals et al. (2016).
21

---

Under review as a conference paper at ICLR 2022
In particular, meta-learning has been leveraged for such automation by designing models for new tasks based on prior experience Vanschoren (2018); Yao et al. (2018); Raghu et al. (2020).
1.1 BASELINE CATEGORIES
We now detail each such baseline. Most of the methods proposed in these works can be categorized into the following three main categories."
No model selection: This category always employs either the same single model or the ensemble of all the models:
Random Forest (RF) Liaw et al. (2002): is SOTA tree ensemble that combines the predictions made by many decision trees into single model. In prediction, the random forest regression model takes the average of all the individual decision tree estimates.
Simple meta-learners:
Meta-leamnrss in this category pick the generally well-performing forecasting model, globally or locally:
Global Best (GB): the simplest meta-leamer that selects the forecasting model with the largest average performance across all train datasets (across all time windows), without using any meta-features.
ISAC Kadioglu et al. (2010): clusters the meta-train datasets based on meta-features. Given new fest time-series dataset, it identifies its closest cluster and selects the best model with largest average performance on the cluster's datasets.
ARGOSMART (AS) Nikolić et al. (2013): finds the closest meta-train time-series dataset (INN) given test time-series dataset, based on meta-feature similarity, and selects the model with the best performance on the INN dataset.
Optimization-base meta-learners: Meta-learners in this category learn meta-feature by task similarities toward optimizing performance estimates:
Multi-layer Perceptron (MLP): Given the meta-train datasets and selected time window, the MLP regressor directly maps the meta-fcatures onto model performances by regression. However, such baseline does not learn temporal dependence within datasets.
AUTOFORECAST-TSL: variant in which the meta-learner L consists only of the time-series learner θ
1.2 USAGE OF META LEARNING IN AUTOFORECAST
We emphasize that we use the term "meta-learning" in the context of traditional principle of meta-learning which is building upon prior experience set of historical tasks to "do better" a new task. We build the experience across different datasets using our "general meta-learner" and build experience the sequential dependence among the same dataset using the "LSTM time-series meta-lcarner". We also capture task similarity between a new input task (dataset) and historical datasets using the "meta-features".
We also emphasize that our proposed method is faster compared to gradient-desceet-baeed bi-level meta-learners, e.g., on our univariate tesbed N-Beats has significantly slower training (average 3600 seconds) and inference time (average 101 seconds) compared to AUTOFORECAST (average 670 seconds for training and 1.13 seconds for inference). We emphasize that enhancing gradient-descent-baee bi-level meta-learners for our new problem is avenue for future work.
As will be shown in Table 14, we emphasize that we did a model selection on seven popular time series forecasting models as well (including the recent works DeepAR Salinas et al. (2020). DeepFactors Wang et al. (2019), and Prophet Taylor & Letham (2018)) as baselines where we chose the model with the best average performance across all windows for each dataset from all model variants.
22

---

Under review as a conference paper at ICLR 2022
Inference Time Comparison on Adobe CPU/MEM
170
Inference Time
80
Naive
112.72
40
1.65
Model Selection Methods
Figure 7: comparison of inference time between AUTOFORECAST and Naive approach (running inference on all 322 models and selecting the best candidate) on Adobe CPU/Mem trace 15 days. AUTOFORECAST yields significant reduction in inference time (68X reduction).
mode salodias (orkt 3i0 toc
400
300
No. of bottnee
08 154C
A5
WLP
AutoFueccadyy AusoForccust
Wets-leaming Methed
Figure 8: The number of the best model selections by each meta-learning approach for multivariate testbed. AUTOFORECAST has 2X gain in selecting best model compared to other baselines.
EXTENDED RESULTS
J.1 FULL EVALUATION ON DATASETS PER TESTBED
Univariate Testbed: We now present the full evaluation of univariate testbed. Such evaluation shown in Table 12 (we show one fold in that table in the interest of space). We note similar performances for the rest of the folds (as reflected in the average MSE and average rank for all datasets shown in Table 3-4). AUTOFORECAST outperforms the meta-learner baselines for most datasets. Moreover, AUTOFORECAST has the lowest average MSE, and the lowest average rank.
Multivariate Testhed: We here present the full evaluation of multivariate testbed. Such evaluation is shown in Table 13. is clear that AUTOFORECAST ouperforms the meta-learner baselines for that AUTOFORECAST has the
of the 40 datasets). Again, lowest average MSE, and the lowest average rank. In particular, AUTOFORECAST (MSE = 0.12) gives gain of 50% over the best baseline (MSE 0.26) on the Adobe CPU Mem 15d dataset.
most datasets (best for 28
Meta-learners perform better than methods without model selection: Table 14 shows that meta-learners outperforms almost models with model selection. In particular, three meta-leamnrs (Global Best, ISAC, AUTOFORECAST) significantly outperform the baseline time-series forecasting models. For instance, Global Best respectively has 79.42%. 56.7 56.71%, 1,, 67.28% 64.32%, 54. $4.19%.95... 19%.995.3.. 95.7%%, and 88.21% % lower MSE over Seasonal Naive, DeepAR, Deep Factors, Random Forest, Prophet, Gaussian Process, and VAR. These results signify the benefits of model selection (specifically using AUTOFORECAST).
Prophet and DeepAR have the best performance across the baseline forecasting algorithms: Table 14 shows that Prophet model has the best average MSE across the baseline forecasting algorithms. Notably, DeepAR has the second best average MSE across them.
Gain in Selecting better models using AUTOFORECASTI Figure 8 shows that across the pool of the multivariate testbed (across all time windows), AUTOFORECAST has superiority in selecting better models compared to the baseline meta-learners (2X gain in selecting better forecasting models).
23

---

Under review as a conference paper at ICLR 2022
Table 12: Method evaluation in Univariate testbed (one fold is shown in the interest of space). The most performing (lowest MSE) method is highlighted in bold. The rank is provided in parenthesis (lower ranks denote better performance). AUTOFORECAST achieves the best average MSE and average rank among all meta-learners.
Ditiser
Girball Rest
ISAC
ML.P
AUTOFORICAST-TS.
AUTOFORECAST
FEATHER.
01005299 (4)
0,009966 (3)
0103663
0.013679 (5)
0.000716
0.000716 (I)
FURNAS.DAT
0.017221 (4)
0,081947 (5)
0.085216 (6)
0.004338 (3)
0,000177 (1)
0.000177 (1)
EGGS.1
0,0015 (3)
0,000732
0.006797 (5)
0,02572
0.0001922
0.0000%
NYSE.1
0.411447 (4)
0.0114333 (1)
0:035966 (2)
0.109728 (6)
0.083661 (5)
0.036996 (3)
apple
0.000071 (3)
0,000727 (5)
0.000256 (4)
0.000842 (6)
0,000038 (1)
0.000038 (1)
STJOHNS.I
0.015178 (4)
0,152463
0,000281 (1)
0.51043
0.010065
0,009741 (2)
SERIESML2
0.00011 (3)
0.000931
0.000081 (2)
0.000254 (6)
0.000197
0.000184 (4)
ALIGN.I
0.116145 (5)
0.020837 (1)
0. 115548444 (6)
0.10843 (4)
0.071:56 (3)
0.0319977 (2)
LYNX.I
0.029034 (3)
0,036677 (4)
0,204602 (5)
0,076905 (6)
0,021614 (2)
0,012252 (1)
ARCTIC.I
0.002078 (4)
0.013534
0.001891 (3)
0.029793 (6)
0.00L175
0.0006
RHINE.I
0.000992 (1)
0.030022 (6)
0.026165 (5)
0.011251 (3)
0.013101 (4)
000088966 (2)
ASKEW15.1
0.002255 (5)
0,00036
0.001733 (4)
0.884906 (6)
0.000881 (3)
0,000182 (1)
MEASIBBLL11
0.000007 (2)
0,012992
0.000666 (3)
0,773299
0.000158
0.000002
REDDEER.1
0.091479 (3)
0.031401 (2)
0.1343477 (4)
0,910554 (5)
0.044403 (1)
0.044033 (1)
rail lines
0.000043 (I)
0,105375 (5)
0.001864 (4)
0.1923 (6)
0.000289 (2)
0.000289 (2)
deta_temp_dev
0.00111 (2)
0,000145
0.000996 (1)
0,03142
0.0000335
0,0003355
DELLI
0.005722 (3)
0,008785 (4)
0,013155 (5)
0.196153 (6)
0.00209
0.00209 (1)
TEMPER.I
0.0155079 (4)
0.011044 (3)
0.015079 (5)
0.489135 (6)
0.006356 (1)
O. 0.006356 (1)
SERIESB.1
0.005421 (3)
0,0321
0.005424 (3)
0,006113 (5)
0,000104 (1)
0,000104 (I)
LACSTJRA.1
2.022971 (5)
0,01
0.016164 (4)
0,261073 (6)
0.010394
0.010394
Ozone
0.0044442 (4)
0.19911 (5)
0.000799 (2)
0.191908 (6)
0.0059935 (3)
0,0917066 (1)
RAPPAHANII
0.006714 (3)
0,0141444
0.006714 (3)
0,048011 (6)
0.001641 (1)
0,001641 (1)
HBCO.1
0.077075 (5)
0.014459
(5)
0,021299 (4)
0.000486
0.000486
CURRENT.I
0.005603 (I)
0,007555 (2)
0.031418 (8)
0,0509233 (6)
0.010159 (4)
0.00879 (3)
ASKEW7.1
0 18159 (5)
0,003655
18199 (5)
0,080627 (4)
0,08322 (3)
0.012513 (2)
SIMAR4.1
0.05864
0.010436
0.05864
0.097847
0.0000933
0,0000933
NYWATER.1
0,000024 (I)
0 168 (3)
0,000024 (1)
0,108533 (6)
0.021238 (4)
0,021238 (4)
CONSUMI
0.006701 (3)
0.192148 (5)
0.006701 (3)
0.410414 (6)
0.003062 (2)
0.0019411 (i)
children_pee per_poman
0
0,000027 (1)
68767
0.013115
0.013:115
AMERICAN.1
D D29565 (4)
0.0061722 172 (3)
0.099565 (4)
0,27694 (6)
0.000867 (1)
0,000867 (1)
SERIESL2
0 (I)
0.021 (3)
0,000311 (1)
0,867712 (6)
0.0531 (4)
0.0531 (4)
FRNCHB.1
0 (4)
0,1038
0.032671 (4)
O 0.235725 (6)
0.013221 (3)
0,008845 (I)
OTTER
0,000697 (I)
0,154446
0.003607 (4)
0,228931 (6)
0.001445
0.001383 (2)
IRM.1
D.D005066 (3)
0,0070022 (1)
0.029243 (2)
1. 1.192122 (6)
0.181984 (5)
11053301 (4)
IXI
0.055671 (4)
0,01878
0.053671 (4)
0,179534 (6)
0,000081 (1)
0,000081 (I)
HANKOUII
0.002255 (4)
0,000199
(5)
0,427704
0.001
0.001 (2)
ESPANOLA.1
0.107704 (5)
0.063331 (3)
O. 1.107704 (8)
0.069477 (4)
0.0386766 (2)
0.0L3953 (1)
NEUMUNAS.1
0.019752 (2)
0,05087 (5)
0.019752 (2)
0.0398799 (4)
0.0749466 (6)
0.018099 (1)
NINEMILK.1
0.034372
0,150897
(2)
0.338648
0.084953
0,021218
NAVAJO.1
0.025011 (3)
0.045677 (4)
0.053738 (5)
0.121633 (6)
0.005994 (1)
0.005094 (1)
BLUME.1
0.002218 (2)
0,107086 (6)
0,001929 (1)
0.049793 (5)
0.003823 (3)
0.003519 (3)
NILE2.1
0
152
0.00137
0.193211
0.054451
0,01142022
LOGISTIC.1
0.064411 (3)
0.280174 (6)
0.080491 (3)
0.16654 (5)
0.000003 (1)
0,00003 (1)
Y.1
D.010648 (3)
0.00421 (2)
0.002002 (1)
0. 0.722457 (6)
0.065461 (5)
0.06 1986 (4)
WEDELAWA.I
0.050875 (4)
0,017199
0.050875 (4)
0.09-1832 (6)
0,003633
0,003033 (1)
AMAZON.2
0.000339 (2)
0,001309
0.000156 (1)
0,119018 (6)
0.051773
0.051773 (4)
CN.1
D115912
0,779448
0.0122008
0,007669
0.001442
0,09967 (1)
ASKEW141
0.021909 (4)
0.000364
0.021909 (4)
0,046782 (6)
0,001908 (2)
0 (2)
LACSTJINA
0.000059 (1)
0.043367
0.029783 (5)
0.024873 (6)
0.001409
0.001409 (2)
CD.I
0.064928 (1)
0.139645 (5)
0.041752 (4)
0.476946 (6)
0.010701 (2)
0,0107011 (2)
FISHERTI
0.023112 (3)
0,117186 (6)
0.033638 (4)
0.08318 (5)
0.009069 (1)
0,008869 (1)
RGNEI
0.000171
04034122 12
0.000009 (1)
0.079975
0.01095
(3)
AROSA.1
0,007555042 (1)
0.010614663 (2)
0,242362866 (5)
0,285212298 (6)
0.0180-633 (3)
0.0180-163
U.I
0.005295366 (4)
0.000199081 (1)
0.085691031 (5)
0.427703736 (6)
0.000999522 (2)
O 0.00199952 (2)
RACDON.1
0.001 346976 (3)
0.860859973
3.001366976 (3)
0.293946600 (6)
0.00633665
0.00N6 36665
END.
0.00047181
0.111166889
0.0000889 (1)
0.043800188
0.0024-16083
0.000256424 (2)
Average
0.0065 (2.4693)
0,0158 (3. (3.3663)
0.0071 (2,5742)
0.0351 (4.5742)
0.00463 (2.8019)
0.00256 (2.0571)
STD
0.028074
0.063828
0,043992
0.274294
0.028135
0/0209966
J.2 STATISTICAL SIGNIFICANCE OF AUTOFORECAST
We now show the pairwise statistical test results between every pair of methods by Wilcoxon signed rank test in Table 15. Statistically better method shown in bold (both marked bold if no significance). We re-emphasize that we use the pairwise Wilcoxon signed rank test on performances (MSE) across datasets (significance level p < 0.05). In the left, univariate testbed is shown where AUTOFORECAST is statistically significantly better than most baselines including GB, AS, and AUTOFORECAST--SLL In the right, multivariate testbed is shown where AUTOFORECAST is statistically significantly better than AS, MLP, and AUTOFORECAST-TSL
We next turn our attention to the tuning of the time-series meta-learner and the timing of AUTOFORECAST respectiell.. In particular, we show both the computational cost and the inference time for AUTOFORECAST and compare it with our different baselines.
24

---

Under review as a conference paper at ICLR 2022
Table 13: Method evaluation in multivariate testbed (average MSE). The most performing (lowest MSE) method is highlighted in bold. The rank is provided in parenthesis (lower ranks denote better performance). AUTOFORECAST achieves the best average MSE and average rank among all meta-learners. In particular, it has the best performance (lowest average MSE) on 28 datasets out of the 40 multivariate datasets and has comparable performance for remaining datasets.
Daizset
Global Best
A$
ISAC
MLP
AUTOPORECAST TSL
ALTOFORECAST
Processed S&P
0.137342 (2)
0.455675 (5)
0.137347 (2)
4.536663 (6)
0.158537 (4)
I,071178 (1)
conehr
0,780519 (5)
0.364615 (3)
0.70019915
0. 0,765257 (6)
0.189151 (2)
0,042093 (1)
Oce Train
0.000106
0.66462
0,000106
0.717224
0.305732
0,072843
Scanline
0008111 (2)
0.627566 (5)
1.292029
0.064343 (4)
0J005702
knoy 3.30
0.009803 (1)
0.176418 (6)
0.0098033 (1)
0.04145 (3)
0.10648 (5)
0,043316 (4)
energy data complete
0.273006 (3)
0,298254 (5)
0.273006 (3)
3.02911 (6)
0.145331 (2)
0,065071 (1)
Adobe Service,CUU... CPMMee. Mem.
0.261722 (2)
0,813373 (5)
0 0.261722
10.782728 (5)
0.581017 (4)
0.120063
1 1_340
0.009342
0.080522 (3)
0.083422
0.239649
0.001921
0,000E374
lown-clectricity
0.001711 (3)
0.052244 (5)
0.001111 (3)
0.667998
0,000001 (1)
0,000001
knoy mpu. 1 .00
0.077215 (4)
0.020874 (3)
0.072215 (4)
0.101818 (6)
0,000009 (1)
0,000009 (1)
knoy.mpu. 2.pp.mpu_2.400
0. 0.132552 (4)
0.00615 (1)
0. 0.132552 (4)
2 (6)
0.011195 (3)
0.00656 (2)
Occeppancy
0,000097 (1)
0.669766 (4)
0.000997 (1)
2 (6)
0.253453 (5)
0,02259) (3)
eighthr
0,131161 (2)
0.286562 (5)
0.131161
4,974266
0.151916
0
knoy mpp. mpw....0000
0.024336 (4)
0.00892 (3)
0,433366
0.083823
0.001408 (2)
0,000618
qualily.contrul .control
0,028198 (4)
0.077566 (3)
0,028198 (4)
303693 (6)
0.025724 (2)
0.006722 (1)
knoy __po_1_500
0 (4)
0.118844 (8)
0,022X32 (4)
0.063941 (6)
0.0000224 (2)
1.000005 (1)
knoy mmpu.3 3.100
0.088206 (1)
0014191 (3)
0.0082006 (1)
0,022998 (4)
0.025675 (5)
0,025675 (5)
knoy-mpp..1
0,023268
0.018866 (4)
0
0.001753
0,001025
0,000034
knoy_mmpn.2_100
0.056477
0.091133
0.056477
0,009588
0.008588
knoy. mupu m_pu_2_500
0.042663 (3)
4550295 (8)
0.043633
0
0,042508 (2)
0,001608 (1)
Sales, Trancactions
0.068063 (2)
0.122557 (5)
0.008063 (2)
0,57930 (6)
0.0747998 (4)
0.005779 (1)
cn2-concentration
0.001174 (1)
0.044724 (5)
0.001174 (1)
0.612426 (6)
0.005400 (4)
0:001976 (3)
Anoy.mpu.11100
0.00652 (2)
0.123459 (6)
0,00552 (2)
0.061915
0.006806 (4)
0,002683 (1)
Processed NASD
0.087896
0.255123 (5)
0.037896
0.6712377
0.117004 (4)
0,6220199
knoy mpu. 3
(3)
0,103497 (5)
0,081679
0,0239722 (2)
0,0059022 (1)
knoy mpu 2 320
0.006278 (3)
0.014435 (2)
0.036278 (3)
0.1927777 (6)
0,099011 (5)
0.01057 (1)
knoy 2.332
0,000959 (1)
0.009084 (4)
0,009959 (1)
1.172323 (6)
0.025441 (5)
0:001284 (3)
us-mmplyment
0,005489 (1)
0.578479 (5)
0,005489
4 724255 (6)
0,046471 (4)
0:000717 (3)
knoy mpn 3.00
0.034633
0.131345 (6)
053463
0.004766
0.008725
0.000013
knoy mpu mpp.2.pu.0..
0.021158 (3)
0.021852 (5)
0.021158
0.696677
0,002555 (2)
0.00048S
fost-duraae--1
0.113182 (2)
0,69599 (5)
0.113182 (2)
2 765069 (6)
0.307727 (4)
0,032013 (1)
knoy 11mp_ppo_16600
0.103141 (5)
0014411 (3)
0.103141 (5)
0.034072 (4)
0.0023555 (2)
1.000359 (1)
muoy_mup_33.20 3,200
0.01 (4)
0.066988 (3)
0.01 1019 (4)
01463735 (6)
0L0029EE (2)
1,002038 (1)
Processed DJ]
0,002731
0.406639 (5)
0.002731
6.420626
0.124
0:027849
Procoserd.je RSS
0.109793 (3)
059900 (5)
0.90799
2.403557
0.072741
B,126666
knoy mmpp_3_6000
0.067199 (3)
0.24021 (6)
0.005719
0.069626 (1)
0.011999 (5)
0.00S0RS (2)
knoy mppu 2. 600
0.079722 (4)
0.31116611 (3)
0.072992 (4)
1,204111 (6)
0,005216 (1)
0.005266 (1)
Processed NYSE
0.099206 (3)
0,50898 (5)
0,099/05 (3)
2.794788 (6)
0.072811 (2)
0,029246 (1)
Averuge STD
0.0584 (2.3191)
0,1683 (3,0851)
0.0584 (2.3191)
1,698 (3.8723)
0,065 (2.3404)
0,0163 (1.)111
0.1252
0.2295
0.1252
2.3421
0.1068
0,0271
Table 14: Results for one-step ahead forecasting (Average MSE across all datasets) for both testbeds. The seven SOTA forecasting models have worse performance (higher MSE) compared to most of the meta-learners. For each SOTA, we chose the model with the best average performance from all model variants. Moreover, AUTOFORECAST has the best performance for both testbeds.
Testhed
Seasonal Naive
DeepAR
Deep Factors
Random Forest
Prophet
Gaussian Process
VAR
Univariate 0.0345
0,0164
0.0217
0.0199
0.0155
0,1661
0.0602
Multivariate
0.0149
0,0085
0,0135
0.0071
0.0065
0,2576
0.9865
Global Best
ISAC
MLP
AUTOFORECAST-TSL
AUTOFORECAST
Univariate
0.0065
0,0158
0.0071
0.0351
0.00463
0.00256
Multivariate
0.0050
0.0139
0.0046
0.0121
0.00541
0.00124
Table 15: Pairwise statistical test results between every pair of methods by Wilcoxon signed rank test. Statistically better method shown in bold (both marked bold if no significance). In the left, univariate testbed is shown where AUTOFORECAST is statistically significantly better than GB, AS. and FTT In the right, multivariate testbed is shown where AUTOFORECAST is statistically significantly better than AS, MLP, and AUTOFORECAST-TS..
Oues
Rascline
p-value 0.0712 x 10
Oues AUTOPORECAST
Baseline
p-valne
AUTORORECAST
CB
GB
1.1l
AUTOPORECAST
1,0726 10
AUTOPORECAST
3,9380 10-
AUTOPORECAST
ISAC
0.1329
AUTOPORECAST AUTOFORECAST
0.8230
AUTOPORECAST
MLP
8.0657
MLP
0.0004
ATTOPORECAST
ALTOFFRRCAST-TS.
8.1693 x 10
AUTOFORECAST
AUTOFORECASE: TSL
0J00125
AUTOFORECASE.TSL
GB
2.2611 x 10
ALTOPORRCASTTISI
GB
0.00254
AUTOPORECASE.SI.
1,0780 11
AUTOPORECAST:SSI
AS
5.8013 10
AITORORFCASTETSL
ISAC
2.3843 x
ALTORORRCASTTTS!!
1.5585
AUTOFORECASE.TSL
MLP
1.1658 x 21
AUTOFORECAST-TSL
MLP
3.4572 x 10
GB
AS
9,0852 x 110
GB GR
AS
33399 x 10-7
GR
ISAC
0.0522
ISAC
0.5240
4.5689 10
GB
MLP
0.0014
AS
ISAC
1,7822 x 10 37
AS
ISAC
1,2217 x 10-
AS
ML.
4.4658 x 10
AS
MLP
6.6612 x 10-
ISAC
MLP
2.2062 x 10
ISAC
MLP
3.7789 x 10
25

---

show full Under review as a conference paper at ICLR 2022
2 004
1 104
0.84
0.044
Number Laysrs
Figure The performance of the time-series meta-learner @ vs. number of the LSTM layers of @ for the univariate testbed. LSTM with 4 layers gives the best performance (lowest MSE).
Figure 10: The performance the time-series meta-learner @ vs. number of units per layer of e for the univariate testbed. LSTM with 50 units per layer gives the best performance.
0422
0:04
005
0.62
20
326
Figure 11: The performance of the time-series meta-learner O vs. the training batch size of e for the univariate testbed. batch with size 25 instances gives the best performance.
Figure 12: The performance of the time-series meta-learner O vs. number of training epochs of @ for the univariate testbed. Training with 50 epochs gives the best performance.
J.3 TUNING OF TIME-SERIES META-LEARNER +
We show the effect of different hyper-parameter, used in the training of the time-series meta-learner @ on the performance (in terms of MSE under the selected model by +). For searching on each parameter, we fix the other parameters details for the univariate testbed. Figure 9 shows the effect of the number of the LSTM layers of the time-series meta-learner. We observe that LSTM with layers gives the best performance (lowest MSE). Second, Figure 10 shows the effect the number of the units in the LSTM layer of the time-series meta-learner. We observe that LSTM with 50 units per layer gives the best performance (lowest MSF). Third, Figure 11 shows the effect of the training batch size of the time-series meta-learner. MSE). Finally, Figure 12 illustrates the effect of the number of training epochs of the time-series meta-learner. We note that training with 50 epochs gives the best performance (lowest MSE). We have also used dropout rate of 0.2 to prevent over-fitting.
their best values. In the interest of space,
batch with size 25 instances gives the best performance (lowest
J.4 COMPARING COMPUTATIONAL COST OF AUTOFORECAST
Table 16 shows that AUTOFORECAST has comparable computational training cost compared to other meta-leamers baselines. We re-emphasize that the running time of such offline training phase less critical since it is done only once. However, this experiment shows that our better time series model selection performance does not entail prohibitive training cost.
Table 16: Computational cost for training (in seconds) for both univariate and multivariate testbeds. AUTOFORE has comparable computational cost compared to other meta-learners baselines.
Testhed
Univariate
N/A
308 008.0001 1968
27% 3083 de 58000000
2008 = 373.5715
354 S091 GeLs
670 5855 54G5
Multivrrite
NA
194.3877 39.7401
182. 4753 de 34.3233
411 9337 44.4
178. 1978 1800S
376.2953 40.0155
26

---

Under review as a conference paper at ICLR 2022
J.5 INFERENCE RUN TIME STATISTICS
Comparison between AUTOFORECAST and naive approach: We compare the inference time that AUTOFORECAST versus the inference time of the naive approach, i.e., doing inference over all possible models. We first show the aggregate statistics for the gain across all datasets for each testbed. Figure 3 illustrates the reduction in inference time for using AUTOFORECAST over the naive approach for both univariate and multivariate testbeds. In particular, AUTOFORECAST gives a median gain of at least 42X over the naive approach for both testbeds.
Dataset-wise inference time comparison: Next, we pick several random groups of 10 datasets each and show the time (in seconds) that AUTOFORECAST takes versus the time the naive approach takes for forecasting model selection. Figures 13-14 shows such comparison for the univariate testbed. It noted the higher reduction inference time for larger datasets (i.e., with more data points).
Horewo Tree legor re
Usiandt
Dstanot Name
Figure 13: Inference time comparison between Figure 14: Inference time comparison between AUTOFORECAST and naïve approach.
AUTOFORECAST and naïve approach.
Multivariate testbed: We also observe that AUTOFORECAST (meta-fcature generation and model selection) takes less than 1.2 second on most time series datasets for multivariate testbed.
Figure 15 shows that incurs only negligible overhead relative to actual training of the selected model (with median 0.1 0.1%). Similarly, AUTOFORECAST incurs only negligible overhead relative to actual training of the selected model for the multivariate testbed (with median 0.3%) (Figure is omitted for multivariate testbed). This shows that AUTOFORECAST is lightweight, incurring small selection time overhead.
Time overhead of AUTOFORECAST relative to training of selected model:
HB 400100m10 HDM 0-00+
0%
0.1%
1%
10%
50%
100%
Figure 15: Boxplot of time ACTOFORECAST takes relative training of selected model in univariate testbed. AUTOFORECAST incurs negligible overhead, (median = 0.1%).
K INTERPRETABLE FEATURES OF DATASETS
Now, we show some of the interpretable input feature values for all datasets in the univariate testbed. In particular, we show mean, median, variance, skeweness, kutosis, absolute energy, and benford correlation for each of the individual time series in that testbed. Such features shown in Figures 16-23. Note that time-series datasets have standard normalization before feature extraction. Such features values show the diversity in the datasets, where each dataset has different characteristics. We emphasize that in the interest of space, we show sample of these interpretable features.
27

---

Under review as a conference paper at ICLR 2022
1.0
0.0
300
200
200
350
1.00
130
200
230
Dittarett Indeex
Datasen Indes
Figure 16: Mean
Figure 17: Median
0.125
Valunos
0.050
Figure 18: Variance
Figure 19: Skeweness
17.5
125
100
Korben
5.-
-25
59
100
159
200
250
300
1.00
150
200
230
300
Dataset Indiex
Dalasee: Indixx
Figure 20: Kurtosis
Figure 21: Absolute Energy
Dentord CurMaton
0.025
0.000
-0.050
-0.075
Dxitasen
Figure 22: Benford Correlation
Figure 23: Existence of duplicates (binary)
28


Reviews:
[{"review_id": "cSL-CD-KFyi", "reviewer": "Reviewer_ZkNj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- The overall problem and method are novel and interesting. Model selection for time-series forecasting is a new setting and the proposed AUTOFORECAST is reasonable to solve the problem in an efficient and effective way.   \n- The constructed benchmark is large and comprehensive, which serves as a good testbed for the proposed problem and method. \n\n## Weaknesses\n- Several modules are named with \"meta\", but it is unclear to me where the meta procedure happens, e.g. inner-loop, outer-loop. \n- The overall formulation and writing are somehow hard to follow. Since this is a relatively new problem, clear formulation and problem definition are important for the audience. \n- It seems the baselines in Sec 5.2 are relatively old. \n\n### Detailed Comments \nAs mentioned above, I am not aware why the modules should be named \"meta-xxx\". There is no meta-learning procedure applied to learn these modules. For meta-learning, during training, a meta-model and meta-parameters are learned in a bi-level optimization manner. During testing, the task-specific parameters can be adapted to perform the target tasks.    \nFigure 1 is hard to see clearly without extensive zoom-in. \nThere are several meta-learning works for time-series forecasting missing, e.g. [R1, R2]. \n\n\n[R1] Pan, Zheyi, et al. \"Spatio-temporal meta learning for urban traffic prediction.\" IEEE Transactions on Knowledge and Data Engineering (2020).   \n[R2] Narwariya, Jyoti, et al. \"Meta-learning for few-shot time series classification.\" Proceedings of the 7th ACM IKDD CoDS and 25th COMAD. 2020. 28-36.", "summary_of_the_paper": "This paper formulates the problem of automatic and fast selection of the best time-series forecasting model as a meta-learning problem.   \nSpecifically, an AUTOFORECAST model consisting of a temporal meta-learner and a general meta-learner are proposed.   \n\nA large benchmark and a collection of models are established to evaluate the problem.    \n", "summary_of_the_review": "Overall, I think the proposed problem is novel and new. The proposed method is reasonable to solve the problem.   \nHowever, there are issues that need to be addressed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_rEyzremBim", "reviewer": "Reviewer_cQ6U", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Once the meta dataset has been computed, the problem\nbasically becomes a cost-sensitive classification problem,\nwhere the goal is to predict the right forecasting model (class)\nwhere wrong ones are penalized according to how much\nexcess loss they will cause. But none of the baselines is\nfollowing this simple setup. \n\nThe baselines seem to be very weak: they are consistently\nworse than the constant model that always predicts the same\nforecasting model. More challing baselines would make\nthe experiments more interesting.\n\nThe notation of the paper is not easy to follow.\n- It is not clearly defined what the windows are (beginning of\n  sect. 3).\n- Why are the metafeatures F_t^i in \\R^{d\\times v_i} (before\n  eq. 3), i.e., we have d metafeatures per channel (v_i)?\n  Some of the features in table 8 seem not to depend on\n  the channel, e.g., window length and landmark features.\n  - If they depend on the number of channels v_i, how can\n    the regression model then generalize over different datasets\n    with different numbers of channels?\n- eq.2: what is \"w_i\" ? \n", "summary_of_the_paper": "The authors address the problem of selecting the best\nforecasting model for a time series dataset. Datasets\nare described by a large vector of meta features, then\ntwo regression models are learnt to predict the test loss\nof a forecasting model for such a dataset at a given\ncutoff time point (called window): one predicts directly\non the meta features computed until the cutoff time point,\nthe other predicts autoregressively starting from the\nfirst cutoff timepoint until the target cutoff time\npoint is reached. In experiments on two meta datasets\n(called performance tensors), one univariate, one multivariate,\nit is shown that esp. the autoregressive model outperforms\nthe best constant model (called global best) as well as\nsome baselines.\n", "summary_of_the_review": "strengths:\ns1. interesting set of t.s. metafeatures.\ns2. interesting meta dataset for t.s. forecasting.\n\nweaknesses:\nw1. problem is basically a cost-sensitive classification problem,\n    but the loss does not reflect this.\nw2. baselines seem to be very weak.\nw3. complicated notation.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VZHM0Omvs_w", "reviewer": "Reviewer_M5ZB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- This paper introduces a novel approach for meta learning for time series datasets. While meta learning has been widely studied for various types of datasets, it has not been properly applied to time series. This paper takes one of the first steps in applying meta learning to time series.\n- While there exist a multitude of time series datasets, it is an immense effort to collect all of them together under a single corpus. This paper introduces a corpus constructed from Kaggle and other open source repositories and provide it as a part of the supplementary material.\n- Extensive experimental evaluation shows improved performance of the proposed algorithm over other meta-learning baselines.\n\n**Weaknesses**\n- Meta-learning performance can often be affected by the diversity of the training datasets. Yet, there is no discussion about the diversity of the time series in the corpus. It will be interesting to see some of the interpretable input feature values (for example, window length, mean, variance, etc) for each of the individual time series (or aggregated statistics).\n- The sources of the collected datasets are also not mentioned. It is important that meta learning datasets be collected from a diverse set of sources in order to be useful. While there is a reasonable amount of diversity within Kaggle datasets, Adobe datasets maybe quite similar to each other.\n- Some of the experimental details and results are unclear or inconsistent.\n  - Page 7: Hit-at-k is not clearly defined.\n  - Page 8: The p-values right below Table 4 do not match the numbers in Table 5. Similarly for the numbers in Page 9 right below Table 6. The meaning of these numbers is not clear.\n  - Table 4 presents the mean MSE over all datasets. However, the datasets can be quite diverse in scales as is also reflected in the variances (which are larger than the mean values itself). A normalized MSE metric seems more apt as a metric in this case.\n  - Table 5 results are not strong enough. However, I do not see this as a major drawback of the paper.\n\n## Update after rebuttal:\nThanks for the response. The diversity of the datasets is unfortunately not clear from the paper. The presentation of the paper needs to be improved greatly. In particular Tables 10 and 11 are unreadable. Moreover, I count only 23 multivariate datasets from Table 11. It can be much more readable if instead of the raw time series IDs, they are actually organized into categories (like finance, energy, etc). While I don't mind having the variable names in the table as well, a shorter table summarizing the diversity in the data will be much more useful. While this paper addresses an interesting problem, it needs significant improvements. After reading the other reviews and responses, I will be keeping the same score.", "summary_of_the_paper": "This paper proposes a meta learning approach for time series. Specifically, the approach learns to select the best model and hyper-parameters for unseen test datasets, when trained on performance data of various models applied to training datasets. The approach utilizes various dataset specific features in order to make the predictions. Experimental results show an improved performance over other meta learning baselines. The corpus of time series datasets that was used for the experiments, is also provided as supplementary material.", "summary_of_the_review": "This paper makes some significant contribution towards a meta learning approach for time series forecasting, including releasing a dataset. However, due the issues in the experimental evaluation/results, and the dataset collection process, I am leaning towards a reject. I am willing to increase my score if my concerns are satisfactorily addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "58UqKSm7a56", "reviewer": "Reviewer_sRFp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As a caveat, I'm not an expert in meta-learning, so I cannot comment expertly on the contributions of the paper in the meta-learning space.\n\nOverall, the paper proposes an interesting and practically relevant problem. Unfortunately, I found it hard to read from a language point of view and I'm missing important details without which I have a hard time to follow the paper. For example: what actually is a model in the way that the authors use it? I understand that the hyperparameters are fixed (so each M_i in the model \"space\" has different hyper parameters but may be from the same model family) but are the weights/parameters as well? If the hyperparameters are fixed, then the meta-learner basically is a combination of model-selector with HPO -- in this case, we should probably compare to an HPO method in the experiments (I understand that typically HPO methods wouldn't be able to produce results \"quick\" enough, but they would provide a natural upper bound on performance). \n\nIf weights/parameters are not fixed in each M_i not, where are they re-trained? Or are they fine-tuned only? How do we transfer a trained model from one forecasting task to another when important details differ (e.g., length of the forecast horizon, length of the history)? What actually is a forecasting task in the authors' understanding? \n\nI understand that space is limited and not all of this can be mentioned, but then, there are details offered about LSTMs which can be considered standard and shorted (on page 5).\n\ndetailed review/questions:\n- page 1: in what sense is the burden \"infeasible\"? Is it computationally too expensive? Further onwards, one desideratum is that the selection should be \"quick\" -- isn't a even more important feature that the selected method is accurate? \n\n- it's unclear to me how you run forecasting models on different length forecast horizons w_t in the meta-learning training data set. How are these models trained? Some of those forecasting models that you employ required a fixed-length horizon.\n\n- page 3: v_i is the number of *variables* in D_i. Variables is an odd name here. Probably \"number of time series\" or \"items\" is more appropriate. Or am I missing something? \n\n- why is the model space called a \"space\"? Isn't this just a \"set\"? \n\n- Also, in terms of exposition, it's unclear to me what exactly a model is until Definition 3 -- but by then this has already been extensively used. So, I would ask to move Definition 3 ahead of the other definition. As it, I'm left wondering in Definition 2 whether a \"model\" is a fully specified model (e.g., the weights in the neural network are already fixed? are hyper-parameters fixed)? In Definition 2, it's implicit that it's a fully trained model, but that's not what seems to be written in Definition 3. It's also unclear to me what a forecasting \"algorithm\" is. A \"method\" would make some sense to me, but \"algorithm\" evokes associations around which algorithm to use when estimating the free parameters of the model -- surely that isn't meant here.\n\n- why are the covariates of a data set called meta-features? That may be standard meta-learning terminology, but to me it reads odd. You're simply extracting features from the time series, I can't seem to find the \"meta\" nature here (apart for that the meta-learner uses them?) Also, there are standard time series features like tsfresh, catch22 around or funkier things by Francheschi et all (NeurIPS 2019) which should at least be mentioned.\n\n- page 5: paragraph starting \"General Meta-learner...\" what does from *running* all models actually mean? Are you re-training then doing inference? This is a bit too condensed/sloppily written for me to actually understand what's going on.\n\n- paragraph starting \"LSTM-based Time-series...\" I don't understand how we construct F^i_j and p^i_j when on page 3 it sounded as if the forecasting windows w_t are sampled randomly (random time points and random lengths -- I also don't understand how we train the models to arrive at the performance tensors by the way -- do we re-train for every different w_t?). I'm missing the connect between this random-looking construct of w_t and the systematically constructed F^i_j and p^i_j . Also, why is this called a \"multi-regression\" model? Isn't this just a forecasting model again where we have co-variates (F) available? \n\n- page 6: seasonal naive is surely not a SOTA method, but rather a baseline. Overall, the selection of methods makes sense to me, but not for the reason (being SOTA) listed there. Instead, they represent a large selection of methods available for which I would laude the authors. If state of the art was the primary reason, a transformer-based method (TFT for example) and NBEATS should also complement DeepAR. As far as I know GluonTS includes some of these models in reasonable implementations.\n\n- it would be helpful to define clearly what the characteristics of a \"forecasting task\" exactly are. I assume: length of the forecasting horizon/window, maybe also length of history.\n\n- the data set selection for training the meta-learners doesn't contain some standard forecasting benchmark data sets (e.g., M4, M5) which are collected in the Monash repository (which appeared concurrently with the present paper). This selection by the authors should be justified further. As is, it seems low in the number of time series overall, in particular when deep learning based methods are used for which one would expect a larger data hunger. Adobe real traces (e.g., Appendix G2) are mentioned, but there isn't a pointer to literature for them. If the authors are publishing this data set for the first time, this should be mentioned more clearly as a contribution.\n\n- Appendix A, Figure 4. I understand what the authors are trying to achieve here, but as is, the figure doesn't help my understanding. For example, some further grouping would make it easier. Where on the x-axis are the random forest models? Maybe there's not a single dominant model (again, the authors need to define more clearly what a model is), but maybe a model family? ", "summary_of_the_paper": "This paper present a novel approach to addressing a fundamental problem in forecasting research: given the plethora of methods available and no clearly superior approach overall, which forecasting method should be chosen for a new dataset/forecasting task combination. The authors propose a combination of a static (not taking the time evolution of accuracy into account) meta-learning model with one that takes the evolution of accuracy over time into account. They apply this methodology using a number of well-chosen forecasting methods on public data sets. Some of these (e.g., Adobe traces), I hadn't see before, and their discovery may be an important contribution in itself.", "summary_of_the_review": "While I think that the problem that the authors address in this paper is of great practical importance and has plenty of scientific challenges which attention, I cannot recommend the present paper for acceptance. I base this primarily on the exposition of the work which leaves too many too important details open. This inhibits my understand of the paper (and the appendices do not shed light on those details when I checked for it).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this work the authors consider the automatic selection of time-series forecasting model (and hyperparameters) based on historical data. It adopts a conventional feature-based meta-learning approach. Experimental results show an improved performance over the considered baselines.

The reviewers appreciated the clarifications provided by the authors, but a number of concerns were unresolved. For instance, questions remained regarding the dataset collection, the baselines against which the proposed method was compared to (which were considered too weak) and the large number of missing details in the presentation of the method. Based on this the reviewers concluded that the paper could not be accepted in its current form and would require a major revision.
Decision: Reject