Multi-Objective Model Selection for Time Series Forecasting
Authors: Oliver Borchert
David Salinas
Valentin Flunkert
Tim Januschowski
Stephan Günnemann
Venue: ICLR 2022 Submitted (2022)
Keywords: time series
forecasting
model selection
multiobjective optimization
transfer-learning
tabular dataset.

Abstract: Research on time series forecasting has predominantly focused on developing methods that improve accuracy. However, other criteria such as training time or latency are critical in many real-world applications. We therefore address the question of how to choose an appropriate forecasting model for a given dataset among the plethora of available forecasting methods when accuracy is only one of many criteria. For this, our contributions are two-fold. First, we present a comprehensive benchmark, evaluating 7 classical and 6 deep learning forecasting methods on 44 heterogeneous, publicly available datasets. The benchmark code is open-sourced along with evaluations and forecasts for all methods. These evaluations enable us to answer open questions such as the amount of data required for deep learning models to outperform classical ones. Second, we leverage the benchmark evaluations to learn good defaults that consider multiple objectives such as accuracy and latency. By learning a mapping from forecasting models to performance metrics, we show that our method ParetoSelect is able to accurately select models from the Pareto front — alleviating the need to train or evaluate many forecasting models for model selection. To the best of our knowledge, ParetoSelect constitutes the first method to learn default models in a multi-objective setting.

Under review as a conference paper at ICLR 2022
MULTI-OBIECTIVE MODEL SELECTION FOR TIME SE- RIES FORECASTING
Anonymous authors Paper under double-bling review
ABSTRACT
Research time series forecasting has predominantly focused developing methods that improve accuracy. However, other criteria such as training time latency are critical in many real-world applications. We therefore address the question of how to choose an appropriate forecasting model for a given dataset among the plethora of available forecasting methods when accuracy is only one of many criteria. For this, our contributions are two-fold. First, we present a comprehensive benchmark, evaluating 7 classical and deep learning forecasting methods on 44 heterogeneous, publicly available datasets. The benchmark code is open-sourced along with evaluations and forecasts for all methods. These eval- uations enable us to answer open questions such as the amount of data required for deep learning models to outperform classical ones. Second, we leverage the benchmark evaluations learn good defaults that consider multiple objectives such as accuracy and latency. By learning a mapping from forecasting models performance metrics, show that our method PARETOSELECT is able ac- curately select models from the Pareto front alleviating the need to train or evaluate many forecasting models for model selection. To the best of our knowl- edge, PARETOSELECT constitutes the first method to learn default models in a multi-objective setting.
INTRODUCTION
For decades, businesses have been using time series forecasting drive strategic decision-making (Simchi-Levi.ed et all 2013 Hyndman & Athanasopoulos, 201& Petropoulos et all, 2020). Analysts leverage forecasts gauge resource requirements, retailers forecast future product demand to opti- mize their supply chains, cloud providers predict future web traffic to scale server fleets, and in the energy sector, forecasting plays crucial role e.g. to predict load and energy prices. In domains like these and many other, more precise predictions directly translate into an increase in profit. Thus, no surprise that research on forecasting methods has historically focused on improving accuracy.
In addition to more classical local forecasting methods which fit a model per time series, global forecasting models such deep learning and tree-based models have demonstrated state-of-the-art forecasting accuracy ctall 2017: Oreshkin et all 2019; Salmas ef all 2020at Smyl, ((2))) when sufficient training data available (Makridakis all 20S8 lanuschowski call 2020). This research has led to large variety of different forecasting models and hyperparameter choices (Hutter et all, 2019), where different models exhibit vastly different characteristics, including accuracy, training time, model size, and inference latency.
While this variety of forecasting models is great resource, introduces challenging questions. On the one hand, researchers would like to understand patterns in the performance of different models and to benchmark new models against existing methods. On the other hand, practitioners interested in understanding which model performs best for a particular dataset application.
In this work, we address both of these problems: we release one of the most comprehensive publicly available evaluations of forecasting models across 44 datasets. Using this benchmark dataset, we develop a novel method for learning good defaults for forecasting models on previously unseen adopt multi-objective perspective that allows to select forecasting models that are simultaneously accurate and satisfy constraints such as inference latency, training time, model size.
datasets. Importantly,

---

Under review as a conference paper at ICLR 2022
The main contributions of this work can be summarized follows:
We release the evaluations of 13 forecasting methods on 44 public datasets with respect to mul- tiple performance criteria (different forecast accuracy metrics, inference latency, training time, and model size). This constitutes, by far, the most comprehensive publicly available evaluation of forecasting methods. Those evaluations can be leveraged, for example, to assess the relative performance of future forecasting methods with little effort.
As an example application of this benchmark, we use the data to perform a statistical analysis which shows that only a few thousands observations are required for deep learning methods to outperform classical methods. In addition, we investigate the benefit of ensembling forecasting models.
We propose novel method that can leverage offline evaluations to learn good default models for unseen datasets. Default models are selected to optimize for multiple objectives.
We introduce a technique for ensembling models in a multi-objecttvv setting where the resulting ensemble is not only highly accurate but also optimized for other objectives, such as low inference latency.
RELATED WORK
Time Series Forecasting. Time series forecasting has seen a recent surge in attention by the aca- demic community that has started to reflect its relevance in business applications. Traditionally, univariate, so-called local models that consider time series individually have dominated (Hyndman & Athanasopoulos 2018). However, in modern applications. methods that learn globally across set of time series can be more accurate (Oreshkin all 201% Salinas et all, 2020a: Lim all 2021: Montero- Manso & Hyndman, 202I in particular, methods that rely deep learning. With considerable choice of models available, it is unclear which methods should perform best on which forecasting dataset, unlike in other machine learning domains where dominant approaches exist.
In domains such as computer vision or neural architecture search (NAS), offline computations have been harvested and leveraged successfully to perform extensive model comparisons or compare different NAS strategies (Ying et all 2019: Dong & Yang, 20211 Phsterer ct all (002)) However, to the best of our knowledge, no comprehensive set of model evaluations has been released in the realm of forecasting. Consequently, some questions remain open: how much data is required for global deep learning methods outperform classical local methods such as ARIMA or ETS (Hyndman & Athanasopoulos, 2013) What is the impact accuracy when ensembling different forecasting models? While some recent methods incorporate ensembling (Oreshkin et all 2019; Jeon Seong, 2021), comparisons are often made against individual models which may cloud the benefit of the method proposed versus the sole benefit of ensembling.
Learning Default Models. Finding the best model set of hyperparameters often performed via Bayesian optimization given its theoretical regret guarantees (Srimvas et all, 2012). However, even with early-stopping techniques (Golovin all 2017: Lretal 2017) practitioners often restrict the search to single model due to the large cost of training many models. One technique to dras- tically speed up the model/hyperparameeer search is reuse offline evaluations of related datasets via transfer learning. For instance, Wistuba et all (2015); Winkelmolen et all (2020): Phsterer et all (202I) leverage offline evaluations to alleviate the need for training many models by learning a small list of n good defaults that provide small joint eттor when evaluated on all datasets. This approach bears a lot of similarity with ours. Key differences are that we consider multiple objec- tives and model ensembles. In the time series domain, Shah all (202I) also considers the task of automatically choosing from a large pool of forecasting models the one performing best on partic- ular dataset but also only optimizes for single objective and relies potentially expensive model training for model selection.
3 BENCHMARKING FORECASTING METHODS
In this section, we formally introduce the problem of time series forecasting and subsequently pro- vide overview of the benchmark evaluations that release. By evaluating 13 forecasting meth-
2

---

Under review as a conference paper at ICLR 2022
ods along with different hyperparameter choices all 44 benchmark datasets and for two random seeds, the benchmark comprises 4,708 training runs and amasses over TiB of forecast data. At the end of this section, demonstrate how this benchmark data can be used to perform an extensive comparison of contemporary forecasting methods.
3.1 TIME SERIES FORECASTING
The goal of time series forecasting predict the future values of one more time series based historical observations. Formally, we consider a set = {>1-2) Ji=1 of K univariate, equally-spaced
(0) denotes the vector observations for the i-th time series in the time interval a S t S b. In probabilistic time series forecasting, model then estimates the probability distribution across the forecast horizon т, i.e. the distribution over the future values, from the historical observations:
time series lengths T; where =2.h = (za #a+1.
(f)
( 11:
Models typically approximate the joint distribution of Eq. (I) via Monte Carlo sampling (Salinas at 20208) learn directly predict set of distribution quantiles for multiple time steps using quantile regression (Wen et all, 2017: Lim et all 202T).
3.2
METHOSS
We consider models in total that can be distinguished into 7 local methods (estimating parameters individually from each time series) and 6 global methods (learning from all available time series jointly). Details about models can found Appendix A
Local Methods. We use Seasonal Naive (Hyndman & Athanasopoulos, 20T)) and NPTS (Ranga- puram et anz 2021) as simple, non-parametric baselines. Further, we consider ARIMA and ETS H Hyn dman & Athanasopoulos, 2018) well-known statistical methods and STL-AR (alagal 202T)
and Theta (Assimakopoulos & Nikolopoulos, 20000) as inexpensive alternatives. Lastly, we include
Propher (Taylor & Letham, 2018), an interpretable model that has received plenty of attention.
Global Methods. All global methods that we use are deep learning models, namely: Simple Feed- forward (Alexandrov all (020), MQ-CNN and MQ-RNN (Wen all 2017).. DeepAR (Salinas at 2020a), N-BEATS (Oreshkin et all 2019), and TFT (Lam et all 202 For each model (ex- cept MQ-RNN), we consider three hyperparameter settings: this includes the default set provided by their implementation as well as hyperparameter sets that roughly halve and double the default model capacity (see Table in the appendix). Additionally, we consider three different context lengths that govern the length of the time series that predictions are conditioned on.
Model Training. Model training is only required for deep learning models since parametric lo- cal methods estimate parameters at prediction time. Deep learning models are trained for fixed duration depending on the size of the dataset. Further details can be found in Appendix I
3.3 DATASETS
Our benchmark provides 44 heterogeneous public datasets in total. The datasets greatly differ in the number time series (from to R 107,00,,) their mean length (from R22 to s=500,000, their frequency (minutely, hourly, daily, weekly, monthly, quarterly, yearly), and the forecast horizon (from to 60). Thus, we expect them to cover a wide range of datasets encountered in practice.
Datasets taken from various forecasting competitions, the UCI Dua & Grafil 2017),, and the Monash time series forecasting repository (Godahewa et all 2021).. Dataset sources, descriptions, basic statisties, and explanation of the data preparation procedure can be found in Appendix B.
3.4 METRICS
To measure the accuracy of forecasting models, we employ the normalized continuous ranked prob- ability score (nCRPS) (Matheson & Winkled [176; Gnerting & Rattery, 20077 whose definition
3

---

Under review as a conference paper at ICLR 2022
Table 1: The average performance of various forecasting methods across all datasets with respect to relative latency (compared to Seasonal Naïve) and nCRPS. The table shows classical methods (top), deep learning methods (middle, left), hyper-ensembles using all hyperparameter configurations of deep learning models (middle, right), and across within each group displayed in bold, best values across all models starred.
ensembles (bottom). Best values
Rel. Latency
nCRPS Rank
ARIMA
31100.33 + 11739.26
13.86 5.83
ETS
1183.26 + 321.88
15.45 :: 6.87
NPTS
133.64 = 20.82
17.82 = 7.35
Prophet
2635.38 393.48
17.82 5.52
Seasonal Naïve
*1.00 +
19.86
STL-AR
76. 1. 7.44
15.57 1 6.73
Theta
25.15 1: 1.55
15.59 1: 6.19
DeepAR
21.55 2.33 310.62 = 34.26
9,91 = 5.79 6.73 5.24
MQ-CNN
1,78 4 0.15 17.45 48.48 4: 1.60
15.86 1: 5.75/ 10..5//130000 13.07 5.98
MQ-RNN
2.01 1: 0.18/ 6.53 0.63
22.07 /4.30/2//.02
4.92
N-BEATS
3.33 4 34,29 4:
18.32 # 3.45/ 16.59 =
Simple Feedforward
1.57 + 0.07 14.12
+ 0.63
12.32 4.24 / # 4.16
TFT
4.66 + 0.31 / 44.08 +: 3.06
8.70 = 4.71 / + 4.61
Constrained Ensemble (1 ms)
2.29 + 0.16
13.77 + 5.01
Constrained Ensemble (5 ms)
11.99 1: 1.09
9.45 : 5.20
Constrained Ensemble (10 ms)
24.20 1: 2.18
8.30 :: 4.97
Constrained Ensemble (50 ms)
107.68 9.44
6.57 ::
Constrained Ensemble (100 ms)
157.75 = 14.86
5.55 =
Unconstrained Ensemble
196.41 = 20.44
*4.36 = 3.37
is given in Appendix A The benchmark dataset that release contains four additional forecast. accuracy metrics (MAPE, sMAPE, NRMSE, ND). Besides forecasting accuracy metrics, we store inference latency, model size (i.e. number of parameters) and training time for deep learning mod- els. Latency is measured by dividing the time taken to generate predictions for all test time series in the dataset by their number.
3.5 BENCHMARK ANALYSIS
Having outlined the benchmark, we now want to demonstrate the usefulness of the evaluations for research. For this, we want to analyze how local (classical) and global (deep learning) forecast- ing methods compare against each other. In fact, this comparison has been the object of heated discussions in the forecasting community (Makridakis all 20T8: Lanuschowski et all 2020)
order to compare methods, we consider their latency and accuracy (in terms of nCRPS) across all datasets. Table II shows each method's relative latency compared to Seasonal Naïve as well as the the methods' nCRPS rank As far as latency is concerned, global methods, once trained, allow generate forecasts considerably faster than local methods (except for Seasonal Naïve). The reason is simple: once trained, they simply need to run a forward pass. In contrast, the implementations for the local methods chosen here do not differentiate between training and inference, but rather estimate their parameters at prediction time. Across datasets, the relative latency of all considered deep learning models improves between 15% and 95% upon the latency of the fastest local method (excluding Seasonal Naïve). As far accuracy is concerned, complex deep learning models - namely DeepAR and TFT - generally perform best across the benchmark datasets. Except for MQ-RNN, global methods compare favorably against local methods.
Method Comparison.
As a measure of relative model stability, Table additionally shows the standard deviation of nCRPS rank across all benchmark datasets. Not only is TFT the method which performs best average, its rank is also comparatively consistent compared to other methods. Especially,
Ranks computed all methods and ensembles, including the constrained ensembles of Section E.
4

---

Under review as a conference paper at ICLR 2022
0.8
20%
Rel Improvement
-20%
-00%
10'
10 10"
10
10"
10"
10f
10"
10"
Number Observations
Number of Observations
Figure The p-values of the null hypothesis that statistical methods perform equal to or bet- ter than deep learning methods, evaluated for datasets of different sizes. The red line displays the significance level = 5%.
Figure 2: The relative improvement of the best deep learning method versus the best classical method on all benchmark datasets. The red line is fitted on the visualized datapoints via linear re- gression in the log-space.
Table 01 also indicates the performance of ensembles obtained by averaging the predictions of 9 (n = 3 for MQ-RNN) deep learning models trained with different hyperparameters (hyper- ensembles) as done for N-BEATS (Oreshkin ct al 2019 or in M5 competition's 's third-ranking method (Jeon Scong, POZI). While generally bearing a significant cost in latency, the benefit of ensembling is significant. For instance, the Simple Feedforward hyper-ensemble yields a com- petitive model that outperforms DeepAR in terms of both accuracy and latency. Lastly, DeepAR and TFT hyper-ensembles result in the most competitive ensembles.
Comparison of Classical and Deep Learning Methods. We further conduct a statistical analysis compare the classical time series models with the deep learning models listed in Section 3.21 For this, we construct the null hypothesis that the accuracy of classical models is equal to or better than the accuracy of deep learning methods. This is a claim put forward, for example, by Makridkiss (2018). More formally, we write Ho Celass < Qdeep where Zelnss and Qteep describe the distribution of nCRPS values of classical and deep learning models, respectively. Samples of the distributions are derived from the respective single-mode evaluations in our benchmark. Qetass and Queep are continuous and belong to unknown families of distributions with unequal variances. Hence, we choose the nonparametric two-sample Kolmogorov-Smirno test (Gitboons & Chakraborti, 20033 to compute the p-value, i.e. the probability of Hg holding true, on each individual dataset.
We evaluate the null hypothesis Ho all 44 benchmark datasets and plot the p-value for the dif- ferent dataset sizes in Figure MI At a significance level of a = 5%, Ho can be rejected for 30 out of 44 datasets. In fact, deep learning models exhibit competitive performance for almost all datasets and regularly outperform classical methods. Further, they tend to perform comparatively better with increasing dataset size even if for some large datasets, classical models indistinguishable and, nonetheless, show competitive performance for small datasets. The latter is of particular inter- since it contradicts tribal knowledge that large datasets needed to outperform local/classical methods and disputes the claim presented by Makridakis et all (20T8) since only a few thousands observations seem sufficient to outperform the classical models considered.
These statements are further supported by Figure 2 where we compare the best deep learning model Jeeep against the best classical model Telass on each dataset. We compute the relative improvement nCRPS(# 1. Except for few outliers, the best deep learning model outperforms all classical models 40 out of 44 datasets (see Appendix B for further analysis). Fitting linear regression model on the relative improvements shows that the improvement by using deep learning models is only slightly amplified as the datasets grow in size.
given by deep learning models as uCRPS(
We note, that we conflate deep learning and global models in our above discussion. Montero Manso Hyndman (2(2TT) show the general superiority of global models over local models both theoretically and empirically and our results confirm their findings further.
4 LEARNING MULTI-OBJECTIV DEFAULTS
The benchmark introduced in the previous section outlined how to acquire offline evaluations of forecasting methods various datasets. However, in the presence of multiple conflicting objectives
5

---

Under review as a conference paper at ICLR 2022
such accuracy and latency, is not clear how could choose the best models. In this section, first formalize the problem of learning good defaults from these evaluations for a single objective. Afterwards, formally introduce multi-objective optimization and show how learning defaults can be extended to account for multiple objectives. In the end, we report experimental results obtained by using method.
4.1 PREREQUISITES
In our setting, we consider an objective function f X' + R for given task (in our case, task a dataset). The objective function maps any time series model x X to m objectives (such as nCRPS, latency, etc.) that ought to be minimized. In addition, we assume that model evaluations on T different but related tasks with objective functions f()), ( are available. The set of offline model evaluations is then given as
f(())
D U (((P.D0))) j=1
with y (x)) R
where N, evaluations are available for task and y, denotes the evaluation of az on task j.
Learning Defaults. To learn set of n good default models for unseen datasets in the single- objective setting, Phsttrer atall (2021) propose to pick the models that minimize the joint error ob- tained when evaluating them on all datasets. This amount pick the set of models {21...., Fn C X' which minimize the following objective:
min CX
Σ f=1
fl Isisn
min
where k denotes single fixed objective (for instance, the classification error). Since the problem is NP-complete, greedy heuristic with provable approximation guarantee is used: it iteratively adds the model minimizing Eq. B to the current selection. However, this approach only works for single objective of choice f& rather than taking all objectives f (such accuracy and latency) into account. In addition, it does not easily support the selection of ensembles since objectives can ensembled: while accuracy will likely increase by small
interact in differently when models amount, the latencies of ensemble members need to be added up. Taking into account all possible ensembles of given size n would also blow up the combinatoric search when optimizing Eq. R.
Multi-Objective Optimization. In multi- optimization, there generally exists no singular solution. Thus, when considering the objective function f. there exists no single optimal model, but rather a set of optimal models (Emmerich & Deutz, 2018). A model model x' E X' (x <x x') if fa(x) S f((x') for all and there exists some i such that f((x)) < f((x). The set of all non-dominated solutions (i.e. models) is denoted the Pareto front Py (X) whose members are commonly referred to as Pareto-optimal solutions:
E X' dominates another
Py (X) {x E X -e' : x
(4)
To quantify the quality of set {ar,- ..... . F. C X of selected models, we use the hypervolume error (Zitzler & Tucle, 1998; & Yad 2011)). Given set of points Y C 1 and reference point
r E RTt we first define the hypervolume as the Lebesgue measure A of the dominated space:
H()') A (la "
ppp
q A/ S r})
In turn, the hypervolume error E of the set S is defined as the difference in hypervolume compared to the true Pareto front Py (Z)):
E(S) = H ((f (z) | I - (f. 20
(6)
Thus, the hypervolume error (S) reaches its minimum when S 3 Py (Z²). To account for different scales in the optimization objectives, we further standardize all objectives by quantile normalization
(Holstad et all 2003) such that they follow uniform distribution (1 1) this is feasible X' is finite. While the normalization makes our comparisons robust to monotonic transformations of the
objectives (Bnois et all 2020), it also allows to choose the reference point for Eq. 1 as l

---

Under review as a conference paper at ICLR 2022
Learning Multi-Objective Defaults. Using the hypervolume error, extend the minimiza- tion problem of Eq. B to learn defaults in the multi-objective setting. For this, we propose the following minimization problem:
I Σ E(((z1....,..1 1...i........
min CC
In words, we seek to find set complementary model configurations far..... Un C X that provide good approximation of the Pareto front Prcn (2) across tasks j E (1,..., }..
4.2 PARETOSELECT
We now introduce PARETOSELECT which tackles the minimization of Eq. M. On a high-level, PARETOSELECT works as follows: first, it fits a parametric surrogate model fo that predicts the performances of model configurations by leveraging the evaluations of D. Then, it uses the surrogate fo to estimate the objectives for all models in X' and applies the non-dominated sorting algorithm on the objectives to select a list of default models.
The surrogate model fs is trained by attempting to correctly rank model configurations within each task in the available offline evaluations D. For each model x E X, fo outputs a vector ÿ E IR" for optimization objectives. Models then be ranked for each objective independently. In order to fit the data D, we use listwise ranking (Xia el 2008) and apply linear discounting focus on correctly identifying the top configurations similar to Zügner et all (2020)). The corresponding loss function can be found in Appendix D
For the parametric surrogate model fo. choose simple MLP similar Salinas et all (20200) We use two hidden layers of size 32 with LeakyRel.U activations after each hidden layer and ap- ply weight decay of 0.01 as regularization. Importantly, do not tune these hyperparameters. Predictive performances for every model can eventually be computed as
(8) We note that for minimizing Eq. D. either regression or ranking objectives can be used: since we apply quantile normalization to compute the hypervolume, the magnitude of the values predicted by fo is irrelevant. However, we find the ranking setting to be superior to regression for finding good default models (see Appendix Di for more details).
y (fo(z) E X}}
Multi-Objective Sorting. The "best" configurations are then determined by applying the non- dominated sorting algorithm (NDSA) - an extension of sorting to the multi-dimensional setting (Snnivas & DeE, 1994; Emmerich Deutz 2018) - to the predictive performances D. Figure provides illustration of the sorting procedure. NDSA can be described
two-level procedure.
First, partitions the configurations to be sorted in multiple layers by iteratively computing the Pareto fronts, then it applies a sorting function sort to rank the configurations in each layer:
X1 = sort (Prg (X)), X = sort Pro
U jst
X
(9)
We choose sort to compute an 6-net (Clarkson. 2006) for which there exists a simple greedy algo- rithm. When sorting set X, the c-net sorting operation yields an order sort = [......, x- The first element æ is chosen randomly from and ri+1 is defined iteratively as the item which is farthest from the current selection in terms of Euclidean distance, formally:
E++1 = argmax min [] fxz))ll
(10)
The final ordering is eventually obtained by concatenating the sorted layers. While previous work followed different approaches for the sort operation in the NDSA algorithm (Srinvass DeR. 1992; Deb all 20022), we leverage the c-net since Salinas all (212T) highlighted its theoretical guarantees and Schmucker all (2021) provided empirical evidence for its good performance.
The pseudo-code of PARETOSELECT is given in Algorithm II in the appendix. In the where predictions of the surrogate are error-free. the recommendations of the algorithm can be guaranteed to be perfect with zero hypervolume error.

---

Under review as a conference paper at ICLR 2022
Layer
Layer 2
Layer 3
Layer 4
23 17
Objective
15
14
27
16g
18
Objective #2
Objective #2
Objective #2
Objective #2
Figure 3:
Illustration of non-dominated sorting. The layers show the partitioning of the data in Pareto fronts. The numbers depict the overall rank by computing the e-net within each layer.
0.9
Random
4-18
DespA
Model-Free
ARIMA
MQ CNN
Hapervolume Erree
0.7
Single-Objective
3-10"
0.6
NPTS
N-BEATS
Parcto Select
Prophet
Simple Foeeforwand
0.5
Seasonal Naice
TFT
0.4
2-18
STL-AR AR
ParetoSelect
0.2
0.0
10"
In
10'
Number Models
Lateney
Figure 4: Hypervolume error when iteratively choosing mod- els via different model selection approaches. Errors are averaged over all 44 benchmark datasets and five random seeds.
Figure 5: Visualization of forecast latency and nCRPS for mod- els trained and evaluated on the "M4 Yearly" dataset. Each color describes family of models with dots representing different hy- perparameter configurations and training times (checkpoints for deep learning models). Crosses show the default hyperparame- ter configuration and the maximum training time for the dataset.
Proposition 1. Assume that n 2 IPf and for all x E X, folo) f(x). Then, E((Z1,...... xn = where Ti,..., En are the first models selected by method.
Proof. By Eq. E. the first & recommendations of Pareto Select such that {a1, (.1,..., 3kk Pj (X) where k- = iPr. (x)) Since for all X, fo(x) f(x), we get that Pj, (X) = Py (2) It follows that for any < n, Py (X) = fæ1,.... x C {z1,..., ..,) and consequently e((#1,...,.. II,) 0 since the hypervolume error of any set of points containing the Pareto front is
4.3 RESULTS
In order to evaluate our approach, we leverage the data collected with the benchmark presented in Section E. For evaluation, we perform after the other as the test dataset and estimate the parameters 0 of our parametric surrogate model fo on the remaining 43 datasets. The multi-objective setting in the following experiments aims to minimize latency as well nCRPS.
where we use each dataset one
In Figure A. we report the average hypervolume obtained for PARETOSELECT and several baselines, test datasets. PARETOSELECT pieks the top elements of the non-dominated sort on the surrogate's 'ss predictive performances. As baselines, we consider a variant of PARETOSBLECT which considers the nCRPS as the single objective
averaged over
the approach of Phistered atal (202T) which greedily picks models to minimize the joint nCRPS on the training data (Greedy Model-Free), and random search over the entire model space size | = 247 (Random). The hypervolume obtained by PARETOSELECT with 10 default models is small (ss 0.06) compared to the best baseline that reaches x 0.20. Further, random search requires a total of 27 models to be on par with the hypervolume of only 10 models selected via method.
8

---

Under review as a conference paper at ICLR 2022
+10%
x
DeepAR
Rel. n.RPS Improvement
+50%
Ensemble Pareto Front
MQO-NNN
Constra ised Enssebble
x
N BEATS
+00%
Unsoosttrimoo Ensemble
x
Simple Foodfoeward
Models
+30%
120%
ms
ms
25
+10%
5000 ms
100 ms
2:50 ms
-0%
-100%
-20%
Rel. Latency Improvement
Figure 6: Comparison of individual forecasting models, hyper-ensembles and latency-constrainn ensembles. The axes show the average relative latency and nCRPS compared the unconstrained ensemble. Results are averaged across all 44 benchmark datasets. Light crosses show default config- urations of deep learning models, bold crosses show their corresponding hyper-ensembles. Classical models are not colored as most can only be found (far) beyond the range of the plot.
Figure B depicts the top 10 recommendations produced by PARETOSELECT when predicting default models on a single dataset. The default models cover the true Pareto front well: our method is able to pick models with low latency, low nCRPS, and good trade-off between these two objectives.
4.4 MULTI-OBIECTIVE ENSEMBLING
As shown in Section T. ensembling models yields significant gains in accuracy but comes at the cost of considerable latency. Since massively large ensemble model is of no practical use, it is critical to be able to select ensemble with latency that is acceptable for an application. Thus, we consider ensembles under different latency constraints.
For latency constraint we build ensemble with n < 10 members 21,..., Tn by iteratively picking the model with best nCRPS such that the ensemble latency constrain is still met. To gen- erate ensemble predictions, all 10-quantiles {O.1..... 0.9} for each time series and time step independently. Albeit the perfor- mances of the base models might differ significantly, our experiments showed that more compli- cated, non-uniform weighting schemes yielded less accurate forecasts on average. This aligns with current literature showing that simple averaging is often most effective (Petropoulos all 2020).
combine the forecasts of the base models 21,..., In by averaging
Figure H shows the performance of different latency-constrained ensembles, relative to the perfor- mance of an unconstrained ensemble that picks the n 10 models with lowest predicted nCRPS. As expected, higher latency constraints e allows the ensemble to perform better. For e 2 100 ms, the constrained ensemble recovers the Pareto front and outperforms all individual models r E X' as well as ensembles. The small gap to the Pareto front for constraints e < 50 ms can be attributed to multi- ple effects. First, the nCRPS values predicted by the surrogate fo are not optimal. Second, models may not satisfy c on every dataset although they have a lower latency on average (e.g. the default 2.5 ms on 17/44 datasets). Table II additionally reports performance
TFT configuration violates metries of altency-constrainet ensembles: for instance, the ensemble for e = 10 ms outperforms all individual base models in terms of average nCRPS rank while the ensemble for 100 ms outperforms all hyper-ensembles as well.
CONCLUSION
In this work, we have presented (i) a new benchmark for evaluating forecasting methods in terms of multiple metrics and (ii) PARETOŠELECT, new algorithm that can learn default model config- urations in a multi-objective setting. In future work, we will consider applying those techniques to other domains such computer vision where offline evaluations of multiple datasets were recently made available by Dong & Yang (2020). Additionally, future research may extend the presented predictive surrogate model to the of ensembles: this would enable PARETOSELECT be used directly to select members for ensembles of any latency.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY
Public datasets and method implementations were used to ensure the reproducibility of our experi- ments. In Appendix A we provide all hyperparameters used for training the benchmark forecasting models via GluonTS. The source and processing of each dataset as well as the procedure for gen- erating data splits detailed in Appendix BI We also provide details on the surrogate model used and comparison with other choices in Appendix DI The pseudocode of PARETOSELECT is given in Appendix E a source of truth, we share the entire code used to run the benchmark and its evaluation with the submission. This code well the generated evaluations will be released with the paper.
REFERENCES
AEMO. Market Data NEMWeb, 2020. URL /ttp::/wwt: nemweb. com. com.au7)
Alexander Alexandrov, Konstantinos Benidis, Michael Sehheee,, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Türkmen, and Yuyang Wang. GluonTS: Probabilistic and Neural Time Series Modeling in Python. Journal of Machine Learning Research, 21(116):1-6, 2020.
Vassilis Assimakopoulos and Konstantinos Nikolopoulos. The Theta Model: A Decomposition Approach to Forecasting. International Journal of Forecasting, 16(4):521-530, 2000.
George Athanasopoulos, Rob J Hyndman, Haiyan Song, and Doris C Wu. The tourism forecasting competition. International Journal of Forecusting, 27(3):822-844, 2011.
Ron Bekkerman, Hang Zhang, and Jeong- Yoong-YYonnie Lee. KDD Cup 2018, 2018. URL https: //WW:
Mickaël Binois, Victor Picheny, Patrick Taillandier, and Abderrahmane Habbal. The kalai- smorodinsky solution for many-objective bayesian optimization. Mach. Learn. Res., 21(150): 1-42, 2020.
Benjamin M Bolstad, Rafael A Irizarry, Magnus Astrand, and Terence P. Speed. comparison of normalization methods for high density oligonucleotide array data based on variance and bias. Bioinformatics, 19(2):185-193 2003.
Caltrans. Caltrans performance measurement system, 2020. URL sttps://pent dot dov/
Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems. arXiv preprint arXiv:1512.01274. 2015.
City of Melbourne. 2017.
Pedestrian counting system - monthly (counts per hour),
URL
Pedest.rian-Com ng - oo hoop::/ccooutttt----ooorr------.
Kenneth L Clarkson. Nearest-Neighbor Searching and Metric Space Dimensions. Nearest-Neighhor Methods for Learning and Vision: Theory and Practice, pp. 15-59, 2006.
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A Fast and Elitist Multi- objective Genetic Algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2): 182-197, 2002.
Ensheng Dong, Hongru Du, and Lauren Gardner. An interactive web-based dashboard to track COVID-19 in real time. The Lancet Infectious Diseases, 20(5):533-534, 2020.
Xuanyi Dong and Yi Yang. Nas-bench- 201:: Extending the scope reproducible neural architecture search, 2020.
10

---

Under review as a conference paper at ICLR 2022
Dheeru Dua and Casey Graff. UCI Machine Learning Repository, 2017. URL http:://acchitt ics. uci edu/m1
Michael TM Emmerich and André H Deutz. A Tutorial Multiobjectivee Optimization: Funda-
mentals and Evolutionary Methods. Natural Computing, 17(3):585-600 2018.
Corporación Favorita. Corporación Favorita Grocery Sales Forecasting, 2017. URL https://
WWw. www. httpee/ccmcc/aottttaarooooooccee sosery-cattsss-
Andrew Flowers and Reuben Fischer-Baum. Uber TLC FOIL Response, 2015. URL https:
Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic Forecasting with Spline Quantile Func- tion RNNs. In International Conference on Artificial Intelligence and Statistics, pp. 1901-1910, 2019.
Jean Dickinson Gibbons and Subhabrata Chakraborti. Nonparametric Statistical Inference. Marcel Dekker, 2003.
Tilmann Gneiting and Adrian Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477):359-378 2007.
Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero- Manso. Monash Time Series Forecasting Archive. arXiv preprint arXiv:2105.06644 2021.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John EI- liot Karro, and D. Sculley (eds.). Google Vizier: A Service for Black-Box Op- timization, 2017. URL nttp: /tt:://www..o kda. org http://rgg/aaa//aeeeeee
Recruit Holdings. Recruit Restaurant Visitor Forecasting, 2017. URL https: WWW kaggle.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated Machine Learning: Methods, Systems, Challenges. Springer Nature, 2019.
Rob Hyndman, George Athanasopoulos, Christoph Bergmeir, Gabriel Caceres, Leanne Chhay, Mitchell Elara-Wild Fotios Petropoulos, Slava Razbash, Earo Wang, and Farah Yasmeen, fore- cast: Forecasting functioon for time series and linear models, 2021. URL https://pke. cobthyyndnaannccm/fffrrsee
R package version 8.15.
Rob Hyndman. expsmooth: Data Sets from "Forecasting with Exponential Smoothing". 2015.
Rob J Hyndman and George Athanasopoulos. Forecasting: Principles and Practice, OTexts, 2018.
James M. Kilts Center. Dominick's Dataset, httppsss/www..cccoooooo........
Tim Januschowski, Jan Gasthaus, Yuyang Wang, David Salinas, Valentin Flunkert, Michael Bohlke- Schneider, and Laurent Callot. Criteria for classifying forecasting methods. International Journal of Forecasting, 36(1):167-177, 2020.
D. Jean-Michael.
Smart meters London, 2019.
URL https:/ www. kagg e.com
Londonl
Yunho Jeon and Sihyeon Seong. Robust recurrent network model for intermittent time-series fore- casting. International Journal of Forecasting, 2021.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv: 141: 69800 2014.
11

---

Under review as a conference paper at ICLR 2022
Guokun Lai, Wei- Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling Long- and Term Temporal Patterns with Deep Neural Networks. In International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 95-104, 2018.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A Novel Bandit- Based Approach to Hyperparameter Optimization. Journal of Machine Learning Research, 18(1):6765- 6816, 2017.
Miqing Li and Xin Yao. Quality Evaluation of Solution Sets in Multiobjective Optimisation: A Survey. ACM Computing Surveys, 52(2):1-38, 2019.
Bryan Lim, Sercan Ô Ank, Nicolas Loeff, and Tomas Pfister. Temporal Fusion Transformers for Interpretable Multi-Horizon Time Series Forecasting. International Journal of Forecasting. 2021.
Tie- Yan Liu. Learning to Rank for Information Retrieval. Foundations and Trends in Information Retrieval, 3(3):22533,, 2009.
Makridakis, E Spiliotis, and V Assimakopoulos. The M5 accuracy competition: Results, findings
and conclusions. International Journal of Forecasting, 2020a.
Spyros Makridakis and Michele Hibon. The M3-Competition: results, conclusions and implications. International Journal of Forecasting, 16(4))451-44) 2000.
Spyros Makridakis, Allan Andersen, Robert Carbone, Robert Fildes, Michele Hibon, Rudolf Lewandowski, Joseph Newton, Emanuel Parzen, and Robert Winkler. The Accuracy of Extrap- olation (lime Series) Methods: Results of a Forecasting Competit ion. Journal of Forecasting,
(2):111-153, 1982.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. Statistical and Machine Learning Forecasting Methods: Concerns and Ways Forward. PLOS One, 13(3), 2018.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1): 54-74, 2020b.
James E. Matheson and Robert L. Winkler. Scoring rules for continuous probability distributions. Management Science, 12110):0077 1976.
Michael W McCracken and Serena Ng. FRED-MD: A Monthly Database for Macroeconomic Re-
search. Journal of Business & Economic Statistics, 34(4):574-55)) 579, 2016.
Pablo Montero-Manso and Rob J Hyndman. Principles and Algorithms for Forecasting groups of Time Series: Locality and Globality. International Journal of Forecasting, 2021.
NYC Taxi and Limousine Commission. TLC Trip Record Data, 2015. URL REEPS: WWWL
Ser--p-record-oo datao page,
Mitchell O' O'Hara-Wiie Wild, Rob Hyndman, and Earo Wang. tsibbledata: Diverse Datasets for 'tsibble', 2021.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural Basis Expansion Analysis for Interpretable Time Series Forecasting. arXiv preprint arXiv: 10437, 2019.
Fotios Petropoulos, Danicle Apiletti, Vassilios Assimakopoulos, Mohamed Zied Babai, Devon K Barrow, Christoph Bergmeir, Ricardo Bessa, John E Boylan, Jethro Browell, Claudio Carnevale, et al. Forecasting: Theory and Practice. arXiv preprint arXiv:2012.0385. 2020.
Florian Pfisterer, Jan N. van Rijn, Philipp Probst, Andreas C. Müller, and Bernd Bischl. Learning Multiple Defaults for Machine Learning Algorithms. Proceedings of the Genetic and Evolu- fionary Computation Conference Companion, pp. 241-243 2021.
Syama Sundar Rangapuram, Jan Gasthaus, Lorenzo Stella, Valentin Flunkert, David Salinas, Bernie Wang, and Tim Januschowski. Deep Non- Por-Paramttiic Time Series Forecaster. to be published, 2021.
12

---

Under review as a conference paper at ICLR 2022
Rossmann.
Rossmann Store Sales, 2015.
URL hEEDS:/ kaqg 1 cww/cmmccccccc
David Salinas, Michael Bohlke-Schneiiee Laurent Callot, Roberto Medico, and Jan Gasthaus. High--iiensiioal Multivariate Forecasting with Low-Rank Gaussian Copula Processes. arXiv preprint 1XX.::111000000000 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks. International Journal of Forecasting, 36 13):1111-1191, 2020a.
David Salinas, Huibin Shen, and Valerio Perrone. A quantile-based approach for hyperparameter transfer learning. In Hal Daumé and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp- 8438 8448. PMLR, 13-18 Jul 2020b. URL https: /ttt:::/prrceedinne m D111
sa 1aa htmll
David Salinas, Valerio Perrone, Olivier Cruchant, and Cedric Archambeau. A Multi-Objective Per- spective Jointly Tuning Hardware and Hyperparameters. In ICLR Workshop on Neural Archi- tecture Search, 2021.
Robin Schmucker, Michele Donini, Muhammad Bilal Zafar, David Salinas, and Cédric Archam- beau. Multi-Objective Asynchronous Successive Halving. arXiv preprint arXiv:2106.12639 2021.
Syed Yousaf Shah, Dhaval Patel, Long Vu, Xuan-Hong Dang, Bei Chen, Peter Kirchner, Horst Samulowitz, David Wood, Gregory Bramble, Wesley M Gifford, al. AutoAI-TS: AutoAI for Time Series Forecasting, arXiv preprint arXiv:2102. 12347, 2021.
David Simchi-Levi, Xin Chen, and Julien Bramel. The Logic of Logisties: Theory, Algorithms, and Applications for Logistics Management. Springer, 2013.
Slawek Smyl. hybrid method of Exponential Smoothing and Recurrent Neural Networks for time series forecasting. International Journal of Forecasting, 36(1) (655)575-85, 2020.
Adam Sparks.
bomrang: Australian Govemment Bureau of Meteorology (BOM) Data
Client, 2021. URL https: / www rdocumentation httpa/ooo.oo//cccceee///eaaaaeee
versions
Nidamarthi Srinivas and Kalyanmoy Deb. Multiobjective Optimization Using Nondominated Sort- ing in Genetic Algorithms. Evolutionary Computation, 2(3):221-248, 1994.
Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Information- theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Transac- tions Information Theory, 5855) May 2012. ISSN 1557-9654. doi: 10.1109/tit. 2011.2182033. D wwT.:..11111TTTT 110.111111111111111111111111111. 2181.2
//d.:://dx..o
Martin Štěpnička and Michal Burda. On the Results and Observations of the Time Series Forecasting Competition CIF 2016. In IEEE International Conference on Fuzzy Systems, pp. 1-6, 2017.
Souhaib Ben Taieb, Gianluca Bontempi, Amir F Atiya, and Antti Sorjamaa. A review and com- parison of strategies for multi-step ahead time series forecasting based on the NN5 forecasting competition. Expert Systems with Applications, 39(8):7067-7083 2012
Thiyanga Talagala. STL-AR Method, ///
htmll
Sean J Taylor and Benjamin Letham. Forecasting at Scale. The American Statistician, 72(1):37-45 2018.
Walmart. Walmart Recruiting Store Sales Forecasting, 2014. URL https: www kaggle. com/ h/m/c/we wa mart recrui i no- sa - forecasting
13

---

Under review as a conference paper at ICLR 2022
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A Multi-Horizon Quantile Recurrent Forecaster. arXiv preprint arxiv:1711.11053 2017.
Fela Winkelmolen, Nikita Ivkin, Fuuurann Bozkurt, and Zohar Karnin. Practical and sample efficient zero-shot HPO. arXiv preprint arXiv:2007. 2020.
Martin Wistuba, Nicolas Schilling, and Lars Thimme, Learning hyperparameter optimiza- tion initializations. In 2015 IEEE International Conference on Data Science and Advanced Ana- lytics (DSAA), pp. 1-10, 2015. doi: 10.1 10.10990SAAA22117734811
Fen Xia, Tie- Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise Approach to Learning to Rank: Theory and Algorithm. In International Conference Machine learning, pp. 1192 1199, 2008.
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas- bench-101: Towards reproducible neural architecture search, 2019.
Yingchen Zhang. Solar Power Data for Integration Studies, 2006. URL nttps: www. nre Bov/
l.
Eckart Zitzler and Lothar Thiele. Multiobjective Optimization Using Evolutionary Algorithms- A Comparative Case Study. In International Conference on Parallel Problem Solving from Nature, pp. 292-301, 1998.
Daniel Zügner, Oliver Borchert, Amir Akbarnejad, and Stephan Günnemann. Adversarial attacks on graph neural networks: Perturbations and their patterns. ACM Transactions on Knowledge Discovery from Data, 14(5):1-31,2)) 2020.
14

---

Under review as a conference paper at ICLR 2022
Table 2: Overview of all models considered and their associated hyperparameters. The upper models do not require training. The lower six models are deep learning models. For all deep learning models, three context lengths considered well three hyperparameter configurations (except for MQ-RNN) that result in small, medium and large models, respectively. Hyperparameters that are not augmented keep their default values. The hyperparameters of the "default" models correspond to the medium size.
Context Lengths
Hyperparameters
ARIMA (Hyndman Athanasopoulog 20T
ETS (ynddmnn Athanasopoulog 2018)
NPTS (Rangapuram et all, 2021)
Prophet (Taylor & Letham 2018)
Seasonal Naïve (Hyndman & Athanasopoulos 2018)
STL-AR (Talagala, (021)
Theta (Assimakopoulos & Nnkolopoulos 2000
# Layers
# Cells
DeepAR (Salmass cal 2024
1,
20 40
# Filters
Kernel Sizes
MQ-CNN ee (
2, 4,
20
3, 3,
30
7, 3, 3
14, 7,
MQ-RNN (Wen 2017)
2, 4, 8
# Stacks
# Blocks
N-BEATS (Oreshkinellal (01))
1, 4
4
30
Hidden Dim
# Layers
Simple Feedforward
1,
30
40
Hidden Dim
# Heads
Temporal Fusion Transformer (Lim al 2021
1,
16
32
A MODELS
Table provides an overview over all benchmark models along with hyperparameters considered. In addition to three hyperparameter settings per deep learning model (except for MQ-RNN), three different context lengths are considered. These represent multiples of the forecast horizon and are thus detaset-dependent. Eventually, this results in 9 model configurations per deep learning model (and for MQ-RNN).
The implementation of all models are taken from GluonTS AAlxxnnrrov et all 2020). All deep learning models are implemented in MXNet (Chenettal 2015). ARIMA, ETS, STL-AR, and Theta are available in GluonTS as well but forward computations to the R forecast package (Hyndman
202D.
To measure the accuracy of forecasting models, employ the continuous ranked probability score (CRPS) (Matheson & Wmnkten, 1976; Gneiting & Raftery, 2007)). Given the quantile function ( of the forecast probability distribution in Eq. (I) for time series 2
Model Evaluation.
15

---

Under review as a conference paper at ICLR 2022
at time step t, the CRPS is defined as
CRPS ((r())-1, , [ Ao (1)(0))1,4 x()" da
(11)
where Aa is the quantile loss (or pinball loss) defined over quantile level 0 < a a
Aa (F-1, 2) = (ax I| (a)l) (a))
(12)
Here, I. is the indicator function. To compute aggregated score for model forecasting multiple time series and time steps, we compute normalized CRPS (nCRPS):
Σi CRPS
(F (7)
nCRPS (((E))))) Ji,e" :2)
(13)
Ex, (0)
We approximate the nCRPS using the 10-quantiles E {0.1, 0.2, ..2........0.9. While focus on nCRPS for the evaluations in this paper, the benchmark dataset that we release contains four additional forecast accuracy metries (MAPE, sMAPE, NRMSE, ND).
B DATASETS
Our benchmark provides 44 datasets which were obtained from multiple sources. 16 datasets were obtained though GluonTS (Alexandrov et all (2020). 24 datasets are taken from the Monash Time Series Forecasting Repository (Godahewa at 2021), and the remaining datasets were originally published as part of Kaggle forecasting competitions. Table 3 provides basic statistics about all datasets.
B.I DATASET DESCRIPTIONS
In the following, provide brief descriptions of all datasets and provide their sources. We link the original source (if possible) along with any work which pre-processed the data (if applicable). The datasets "Corporación Favorita", "Restaurant", "Rossmann", and "Walmart" which were obtained from Kaggle - are processed by ourselves.
Australian Electricity Demand (O Hara--WWldeee иоя Godahewa etll 2024) provides the 30- minute electricity demand of five Australian states (New South Wales, Qucensland, South Australia, Tasmania, Victoria). Time series range from January 2002 April 2015.
Bitcoin (Godahewa et all. 2021)) provides over dozen of potential influencers of the price of Bitcoin. Among others, these influencers are daily hash rate, block size, or mining difficulty. We exclude one time series from the original dataset as its absolute values are 2 10 Data is provided from January 2009 to July 2021.
CIF 2016 (Stěpnička Burda, 2017: Godahewa all, ()2 is the dataset from the "Computational Intelligence in Forecasting" competition in 2016. One third of the time series originate from the banking sector while the remaining two thirds are generated artificially. In the competition, the time series have two different prediction horizons, however, GluonTS forces us to adopt only a single horizon (for which we choose the most common one).
COVID Deaths (Dong et al] 2020: Godahewa et all, 2021) provides the daily COVID-19 death counts of various countries between January 22 and August 21 in the year 2020.
Car Parts (Hyndman, 2015; Godahewa et all, 2021) provides intermittent time series of car parts sold monthly by US car company between January 1998 and April 2002. Periods which no parts are sold have a value of zero.
Corporación Favorita (avoriial, 2017) contains the daily unit sales of over 4, 4,000 items 54 differ- ent stores of the supermarket company Corporación Favorita in Ecuador. Unit sales were manually set to 0 on days where there was no data available. Data ranges from January 2013 to August 2017. All covariates that provided in the Kaggle competition are discarded.
https:/ kagg com/
16

---

Under review as a conference paper at ICLR 2022
Table 3: Statistics of the benchmark datasets. Frequencies read as follows: MIN minutely, H hourly, daily, B business daily, W weekly, M - monthly, Q - quarterly, of observations calculated from the training data.
yearly. Number
Freq.
Horizon
Series
Avg. Length
Observations
Australian Electricity Demand
30 MIN
231,005
1,155,024
Bitcoin
D
30
17
4,134
70,279
CIF 2016
M
12
72
87
6,244
COVID Deaths
D
30
227
182
41,314
Car Parts
M
12
2,504
39
97,656
Corporación Favorita
D
171,091
1,089
186,310,518
Dominick
W
115,163
157
18,137,044
Electricity
H
24
321
21,044
6,755,124
Exchange Rate
B
30
6,071
48,568
Fred-MD
M
12
107
716
76,612
Hospital
M
12
767
72
55,224
KDD 2018
H
48
270
10,850
2,929, 404
London Smart Meters
30 MIN
48
5,559
29,903
166,231,248
MI Monthly
M
18
617
73
44,892
MI Quarterly
Q
203
8,320
MI Yearly
Y
181
3,429 19
M3 Monthly
M
18
1,428
99
141,858
M3 Other
Q
8
174
69
11,933
Q
M3 Quarterly
8
756
30,956
Y
M3 Yearly
645
14,449
M4 Daily
D
14
4,227
2,357
9,964,658
M4 Hourly
H
48
414
854
353,500
M4 Monthly
M
18
48,000
216
10,382,411
M4 Quarterly
Q
24,000
2,214, 108
M4 Weekly
W
13
359
1,022
366,912
M4 Yearly
Y
6
22,974
31
707,265
M5
D
28
30,490
1,885
57,473,650
NNS
D
56
111
735
81,585
Pedestrian Count
I
47,412
3,129, 178
Restaurant
D
39
823
321
263,817
Rideshare
H
48
2,304
493
1,135,872
Rossmann
D
48
1,115
864
963,689
San Francisco Traffic
862
17,496
15,081,552
Solar
137
7,009
960,233
Taxi
30 MIN
24
1,214
1,488
1,806,432
Temperature Rain
D
30
32,053
695
22,276,835
Tourism Monthly
M
24
366
275
100,496
Tourism Quarterly
Q
427
39,128
Tourism Yearly
Y
518
21
10,685
Vehicle Trips
D
30
314
100
31,361
Walmart
W
39
2,934
102
298,509
Weather
D
3,010
14,266
42,941,700
Wiki
9,535
762
7,265,670
Wind Farms
MIN
60
313
513,442
160,707,330
Dominick (lames M. Kilis Center. 2020 Godahewa et all 20211) contains time series with the weekly profits of numerous stock keeping units (SKUs). The data is obtained from the grocery store company Dominick's 's over period seven years.
Electricity (Dua & Graff, 20L2: Alexandrov all 2020 is comprised of the hourly electricity consumption (in kWh) of hundreds of households between January 2012 and June 2014.
Exchange Rate (L et all 2018: Alexandrov et all 2020) provides the daily exchange rates (on weekdays) between the US dollar and the currencies of eight countries (Australia, Great Britain, Canada, Switzerland, China, Japan, New Zealand, Singapore) in the period from 1990 to 2013.
17

---

Under review as a conference paper at ICLR 2022
Fred-MD (MCCracken & Ng. 2016; Godahewa at 20211 contains monthly time series of macro- economic indicators (e.g. interest rates, employment rates, ...) from the Federal Reserve Bank since January 1959 (until September 2019).
Hospital (Hyndman, 20T5: Godahewa et all 2021) provides the monthly patient counts for various products that are related medical problems between 2000 and 2007.
KDD 2018 (Bekkerman et all 2018: Godahewa et all 202T) is the dataset used by the KDD Cup 2018. It contains hourly time series with air quality levels (represented by various chemical com- pounds) measured by stations in Beijing and London between January 2017 and April 2018.
London Smart Meters MichneMichael, 2019 Godahewa et all (0221 was originally published on Kaggle and contains the half-hourly (electrical) energy consumption (in kWh) of thousands of households in the period from November 2011 February 2014.
MI (Makridkkis et all 1982: Godaheewa et all (021)) datasets are taken from the MI competition in 1982. Time series have varying start- and end dates and cover wide semantic spectrum of domains (demographics, micro, macro, industry).
M3 (Makrıdakis & Hibon, КCЛC royxandrov al. 2020) datasets are obtained from the M3 com- petition. Across frequencies, time series were collected from six different domains: demographics, micro, macro, industry, finance, and other. For the time series with no specified frequency, we assume quarterly frequency (as this is the default in GluonTTS).
M4 (Makridakis ala 20206 Alexandrov et 2020) datasets are taken from the prestigious M4 competition. The time series are collected from the same domains in the M3 competition.
M5 (Makridakis et all 2020; Alexandrov et all 2021) the dataset from the M5 competition and contains daily unit sales of 3,049 products across 10 Walmart stores located in California, Texas, and Wisconsin. While covariates are available (e.g. product prices or special events), none of the forecasting methods consider makes use of them. Sales data is provided from January 2011 to April 2016.
NNS (Taeeb al 2012: Godahewa at 2024)) data was used in the NN5 competition run in 2008. The time series provide daily amounts of cash withdrawals at dozens ATMs in the UK from March 1996 to May 1998.
Pedestrian Count (City of Melbourne, 2017: Godahewa et al, 2021) provides hourly pedestrian counts in Melbourne from May 2009 to May 2020. The counts captured by sensors scattered in the city.
Restaurant (Holdings, 201 provides the number of daily visitors of hundreds of restaurants in Japan between January 2016 and April 2017, as tracked by the AirREGI system. As dates for which restaurants closed do not provide visitor numbers, we impute zeros for these missing values. The Kaggle competition where this dataset is obtained from also provides data about reservations but this data remains unused for our purposes.
Rideshare (Godahewa all (021)) is comprised of hourly time series representing various attributes related services offered by Uber in Lyft New York. Time series include e.g. minimum, max- imum, and average price, or the maximum distance traveled - grouped by starting location and provider. Data is available for the period of month, starting late November 2018
Rossmann (Rossmann, 2015 provides daily sales counts at hundreds of different stores of the Ross- mann drug store chain. Just like for the M5 dataset, covariates are available (e.g. distance to com- petition, special events, or discounts) but not used by any method. Data is available from January 2013 until August 2015.
San Francisco Traffic (Caltrans, 2020 Godahewa et all 2021) contains hourly occupancy rates of freeways in the San Francisco Bay area. Data is available for two full years, starting from January 1, 2015.
Solar Zhang 2006: exxandrov et all ()2( is comprised of the hourly power consumption of dozens of photovoltaic power stations in the state of Alabama in the year 2006.
Taxi (NYC Taxi and Limousine Commission, 2015; Salinas et all 201%: Alexandrov et all 2020) contains the number of taxi rides in hundreds of locations around New York in 30 minute windows.
18

---

Under review as a conference paper at ICLR 2022
Training data contains data from January 2015 while test data contains data from January in the subsequent year.
Temperature Rain (Godahewa et al 20Z41 provides the daily temperature observations and rain forecasts gather from 422 weather stations across Australia. Data ranges from May 2015 through April 2017.
Godaheew et all 2021)) datasets come from a Kaggle fore- casting competition where time series were supplied by tourism bodies from Australia, Hong Kong, and New Zealand as well as academics. Monthly data ranges from 1979 through 2007, quarterly data from from 1975 through 2007, and yearly data from 1960 through 2008
Tourism (Athanasopoulos et al,
Vehicle Trips (Hlowers & Fischer-Baumt 20L5: Godahewa et all 202 contains the daily number of trips served by hundreds of for-hire vehicle (FHV) companies (such as Uber). Data ranges from January to September 2015.
Walmart (Walmart, 2014) contains data about Walmart store sales similarly to the M5 competition. Data is provided as the weekly sales (in USD) across 45 stores and 81 departments in different regions and ranges from February 2010 to November 2012. Just like for many of the other Kaggle competitions, covariates are available, yet unused.
Weather (Sparks, 2021; Godahewa el 2021) is comprised of daily data collected from hundreds of weather stations in Australia. Collected data is the amount of rain, the minimum and maximum temperature, and the amount solar radiation.
Wiki (Gasthaus et all, 2019 Alexandrov all, 20200) similarly provides daily pages views for several thousand Wikipedia pages in the period from January 2012 July 2014.
Wind Farms (AEMO, 2020; Godahewa et all, 2021) contains time series with the minutely power production of hundreds of wind farms Australia. Data is available for a period one year, starting in August 2019.
B.2 DATA PREPARATION
For all of the datasets in Table 3 except for "Taxi", we perform similar preprocessing. Let Z {=1,7; be the set of K univariate time series of lengths T-- Then, the follow- ing preprocessing steps:
1. Remove all time series which are constant to always be able to compute the MASE metric.
2, (z E Z -3ce ( = c- 1r.
(14)
2. Remove all time series which are too short. We exclude all time series of length T (p + 1)T where 7 is the prediction horizon and p is a dptaset-specific integer which governs over how many prediction lengths we want to perform testing". We are using p for all datasets but "Electricity" (p 7), "Exchange Rate" (p = 3), "Solar" (p 4), "Traffic" (p = 7) and "Wiki" (p- 3).
2g = (z (i) E 2, T, > (p + 1)r)
(15)
3. Split time series into training. validation and test data. For each time series, we split off the prediction horizon p- 1 times obtain time series for the test data. Then, we split off the prediction horizon another time to obtain a validation time series per training time series. Eventually, we construct the following sets:
(i) Zies U
(2) Pi(T Ba(T-(p-- 11)1) 1x0) E }3}
(16)
Zval
2 (1((Ti-p)) (2(0) {0)
(17)
Zunin
2 (1:(T:- 20) E {
(18)
AAssume = 2 and a time series E RT from a dataset with prediction horizon т. Then, we perform testing time series z!) 21:7 and 2 (2) -)( while 2 (3) 21:(T-20) is heing trained on.
19

---

Under review as a conference paper at ICLR 2022
For the "Taxi" dataset, we need to slightly augment the preprocessing to allow for discontinuities in the available time series. Essentially, each time series x E Z consists of an old part Sold and new part znew (data throughout January of two successive years). We then obtain the training data from Zold and construct the validation data by cutting the prediction horizon from Zold- The test dataset is constructed from nnew in bullet point 3 with p 56.
C TRAINING DETAILS
This section describes detail how the deep learning models from Table are trained on the datasets outlined in the previous section. As outlined in Section 3.2 parametric local methods directly timate their parameters at training time. While deep learning models are trained on Zuain such that models different checkpoints be compared using Zval, parameters of parametric local methods are estimated using Val--
C.I TRAINING TIME
For all deep learning models, we run training for a predefined duration depending on the dataset size. For total observations in the training data, we run training for hours where
d = with
min
max
loggo
(10000)
n
.
(19)
Training therefore always runs between one and eight hours (refer to Table El for training times on individual datasets). Even small datasets, train for hour ensure that the trained model receives sufficiently many gradient updates.
C.2 PROBABILISTIC FORECASTS
Forecasts of all methods are required to be probabilistic to store the 10-quantiles f0. 90.2.-.... Q0.9- While some methods forecast the quantiles directly (e.g. MQ-CNN, TFT), other methods provide samples of the output distribution (e.g. DeepAR) or point forecasts (e.g. N-BEATS). For sampling- based models, we simply compute the empirical 10-quantiles. For the point forecasts, we interpret forecasts as Dirac distributions centered around the forecasted value y. Thus, each quantile can be set to the value y.
C.3 LEARNING RATE SCHEEDLINGG
All deep learning models are trained using Adam optimizer (Kingma & Ba, 2014) with an initial learning rate of p 10 During training, we decrease the learning rate three times by factor of 2 using a linear schedule. When training for a duration we decrease the learning rate after
durations D = _ d 3d This results in final learning rate of X 1.25 10-4, The decay is always applied after the first batch exceeding one of the durations in D.
2
C.4 VALIDATION AND TESTING
During training, we save the model and compute its loss C as well as the nCRPS Q the validation data for total of 11 times. For model being trained for a duration t, we perform these computations after durations D H (KG each d E D, we run validation on Zyal after the first batch where the total training time exceeds d and compute the losses La and Qa-
(l,..., .....,. 9}} with intervals = {1-3 3- k E (O..... 4) For
Note that do not add the time taken to run validation to the training time to evaluate whether it exceeds We argue that validation can be run swiftly. If necessary, could also be run on a subset of all available time series for further speedup.
For testing, we choose of the 11 models that we saved during training. For each E I, we choose the model with the lowest nCRPS that was encountered up to d. Note that this may result in choosing the same model multiple times continued training did not yield any improvements on the validation data.
20

---

Under review as a conference paper at ICLR 2022
C.5 INFRASTRUCTURE AND TRAINING STATISTICS
All training jobs were scheduled on AWS Sagemaker using CPU-onlyml ml c5. 22l2xxll allggg instances (8 CPUs, 16 GiB memory). In few cases, we ran jobs on other instance types:
m1. .m5 2x1 arge instances (8 CPUs, 32 GiB memory) to counteract out-of-memory- issues. Used for N-BEATS on "Corporación Favorita" and "Weather", DeepAR "Cor- poración Favorita", and Prophet on "Wind Farms".
ml m5. 4xlarge instances (16 CPUs, 64 GiB memory) allow for even more memory- intensive models. Used for N-BEATS on "London Smart Meters".
m1 . cl. 1c55188xarger instances (72 CPUs, 144 GiB memory) speed up predictions by parallelization. Used for ARIMA on "Taxi""
In total, 4,708 training jobs. Including validation and testing, this amounted to roughly 684 days total training time (~3 hours and 30 minutes per training job). Parallelized over 200 instances, actual training time for our benchmark roughly 4 days and amounts approximately $7,300 compute costs on AWS.
D SURROGATE MODEL
In the following, we discuss the choice of the surrogate model. First, we outline why a parametric surrogate model is desirable. Then, we describe how model configurations are vectorized to be used by parametric surrogate models. Afterwards, list the ranking loss with linear discounting that we mention in Section !2 and eventually compare the ranking performance of different surrogate models.
D.1 CHOICE OF SURROGATE MODEL
Intuitively, a nonparametric surrogate model might appear to be optimal for ranking models. How- ever, a parametric surrogate has several advantages. First, it can interpolate between neighboring configurations, which is essential when the configurations differ between different datasets when the objective is noisy. Second, a parametric model allows for suggesting new configurations that e.g. maximize capacity under given constraint. Third, parametric surrogate model fo may also be conditioned the dataset by taking as input dataset features (such as number of time series or observations in the task hand) by concatenating features to the input vector of fo although do not consider this possibility here.
D.2 INPUT VECTORIZATION
When using XGBoost or an MLP as surrogate model for predicting model performance, models must be represented feature vectors. For this, proceed follows encode model configurations:
The model type (i.e. ARIMA, DeepAR, ....) is encoded via a one-hot encoding, yielding 1 13-dimensional vector.
Every model hyperparameter (across models) defines a new real-valued feature. Hyper- parameters that are shared among deep learning models (context length multiple, training time, learning rate are re-used across models. This results in 15 additional features which are all standardized to have zero mean and unit variance.
Inputs the MLP are, thus, 28-dimensional. Features that missing (e.g. all classical models do not provide a context length multiple) are imputed by the features' means (resulting in zeros due to the standardization).
Chttps://ane wtw.aaa.aaa... amazon. com/ saqemaker?
AIthough altered, this provides potential hyperparameter.
21

---

Under review as a conference paper at ICLR 2022
D.3 RANKING LOss WITH LINEAR DISCOUNTING
Section outlines that the MLP surrogate model fo is trained via listwise ranking using linear discounting, For training on the offline evaluations D, fo minimizes the loss via SGD, As ranking performed for each objective & E {L....,.} . and task {1...., ." independently, L can be
written as follows:
m
C(fo)
Cxy (fork
(20)
where Jo.k yields the surrogate model's outputs for objective & and Las(Sox) ) is defined as follows (where we ignore indices and in the formula for notational convenience):
N Σ i=1
N
Cus(Fox)
fo (Fr(O) + log Σ exp (Z)()))))) (1r11))
(21)
Here, N provides the number of available evaluations for task j and r(i) defines the index of the configuration that is ranked at position with respect to objective i among all configurations for task j. The ranks of the configurations 21.....22 UN with respect to an objective & can be computed provides the index for the "best" configuration and So outputs a smaller value for configurations that have a smaller rank (i.e.
from the corresponding evaluations Vi.k...., UN Note that rank
are "better").
D.4 COMPARISON OF DIFFERENT MODELS
The following compares the MLP surrogate trained via listwise ranking introduced in Section ! to other possible choices for the surrogate model. Much like the approach described in Section 2.3 we evaluate surrogate models by performing LOOCV: surrogates are evaluated on each of the 44 benchmark datasets one after the other, being trained on the remaining 43 datasets. Surrogates are evaluated with respect to different ranking metries that give an overview of the models' ability to rank the nCRPS values of model configurations on different datasets. As metrics for measuring the ranking performance on a single dataset, we consider the following which yield metrics between and 1:
Mean Reciprocal Rank (MRR)
LTTT (IT)):: Computed as one divided by the predicted
rank of the model with the lowest true nCRPS.
Precision @k (Lu.. 2009): The fraction of the models with the lowest true nCRPS cap- tured in the top k predictions.
Normalized Discounted Cumulative Gain (NDCG) (Li 2009): A measure for the over- all ranking performance. It first computes the discounted cumulative gain (DCG) by sum- 0.1 of the 10 models with the lowest true
ming the discounted relevance scores 1, nCRPS scores more specifically, a relevance score π()) is discounted by log, k + where k is the predicted rank of the model with true rank i. Then, it normalizes the DCG by using the ideal DCG (iDCG) to obtain the NDCG.
As comparison to the ranking MLP surrogate with linear discount (MLP Ranking Discounting). we first consider random surrogate which predicts nCRPS by sampling from (/(0, 1) baseline (Random). Then, we use nonparametrie model which predicts the nCRPS as the average among all training datasets (Nonparametric Regression) and one that predicts the average rank of the nCRPS across training datasets (Nonparametric Ranking). Additionally, we consider XGBoost surrogates which are trained via regression (XGBoost Regression) and pairwise ranking (XGBoost Ranking), respectively. Lastly, we consider an MLP with the same architecture but trained via regression instead of listwise ranking (MLP Regression) and one without any discounting (MLP Ranking).
Table 0 clearly shows that the MLP with listwise ranking and linear discounting markedly outper- forms all other choices for the surrogate model. Especially, it is much more capable of identifying
Gtooott does not provide a working version of listwise ranking.
22

---

Under review as a conference paper at ICLR 2022
Table 4: Comparison of different surrogate models with respect to their ability to rank model config- urations according to their nCRPS. Ranking metrics are averaged across all benchmark datasets and multiplied by 100. Metrics for non-deterministie surrogates (random and MLP) further averaged by running the entire evaluation over five random seeds. Best values for each metric are displayed in bold.
MRR
NDCG
Precision@5
Precision @ 10
Precision @I 20
Random
2.17
25.36
2.45
4.68
8.27
Nonparametric Regression
6.88
35.80
13.18
20.91
32.73
Nonparametrie Ranking
6.78
35.87
10.00
20.91
33.07
XGBoost Regression
5.68
35.34
13.18
22.05
32.61
XGBoost Ranking
6.64
35.63
12.27
21.59
32.84
MLP Regression
3.86
32.06
7.73
13.73
25.23
MIPP Ranking
5.00
34.59
10.27
21.00
33.34
MIP Ranking Discounting
16.41
43.74
23.82
28.32
34.39
the top configurations, stressed by the very high values of MRR and Precision@5 @5 compared to the other surrogate models. This can likely attributed to the linear discounting that encourages the model to focus ranking the top configurations correctly. Thus, the loss formulation is more stable in the presence of outliers.
E PARETOSELECT PSEUDO CODE
Algorithm 1: Overview of PARETOSELECT
Data: Time series models X. offline evaluations D, number of default models Result: set (ai...., ] C X' model defaults
function non coo__dmminated_oore (X. fo):
A |]
while X * do
Px.f. = [xE -3x' ExiT -s x}
Compute the Pareto front // Remove Pareto front
X = X \
extend (A, compute_eps" _on_net (Px fo))
return A
function compute epsilon net (X. fa):
[2))
// Choose and remove random element
while X' + do
D = {
// Mapping from remaining elements to minimum distances // Compute minimum distance to all chosen elements D[æ] = mineex (l)) fa(=')l2
for X' do
// Append the element furthest away from the current selection append (A, pop (X. argmax D(z))
return A
fe --t-tiinnsseee h_surrogatte_mmoee (D)
// Mulli-cbjjcttin sorting according to predictions
; mon mron_dominated_son (X' fo)
return Zi...., Fmin(o.
23

---

Under review as a conference paper at ICLR 2022
F ANALYSIS OF DATASETS CHARACTERISTICS
In this section, we try to answer the following question: are there some simple dataset characteristics that can help to know which model going perform best? In particular, we analyze in details the results of Section B.S where we showed that, on some datasets, (1) statistical methods outperformed deep learning methods and that (2) the performance of statistical and deep learning methods could not be distinguished.
We start by analyzing more in depth the four out 44 datasets where deep learning models do not outperform classical methods Figure 2 showed that the best deep learning method outper- forms the best classical methods on all but four of our benchmark datasets. On these four datasets ("KDD2018" "M3 Yearly", "M4 Quarterly". "Taxi"), the summary statistics differ wildly can be seen from Table both terms of the number of observations, the average length, and other characteristics. Further, the classical method that outperforms the best deep learning method is dif- ferent for each of these datasets ("KDD2018": NPTS, "M3 Yearly": Theta, "M4 Quarterly": ETS, "Taxi": STL-AR). Notably, this does not include the local method that performs best average (i.e. ARIMA, see Table II..
Figure showed that for 14 out of the 44 benchmark datasets, the performance of classical methods indistinguishable from the performance of deep learning methods. To further study these datasets, plot in Figure the correlation matrix of all &t benchmark methods' nCRPS ranks across the datasets (using only the default configuration for the deep learning models). Notably, for twelve datasets ("Exchange Rate" through "MI Yearly" on the y-axis), we observe very low correlation with a large number of datasets. On these datasets, all models perform similarly due to strong stochasticity (e.g. "Exchange Rate", "Bitcoin", "MI Quarterly". ...) or seasonality (e.g. "Tourism", "Ride share". ...) and predicting their ranking is therefore hard. Notably, 11 of these datasets overlap with the datasets where classical methods are indistinguishable from deep learning methods.
In line with these observations, we found that adding simple dataset grouping features such as do- main type (electricity, retail, finance) as an input for a surrogate model predicting the performance of forecasting methods does not have much effect. Our conclusion is that the ranks of the best performing models seem to be relatively hard to identify only from simple dataset grouping rules. Nonetheless, hope that by releasing the benchmark data, we help future research to identify dominant methods from dataset characteristics.
24

---

Under review as a conference paper at ICLR 2022
Yarty
Years
Tourfon
Other
Weikyy
Yours
Moethly
Quanetty
Monthly
Quartaly
Endrunee
Boopaa
Tounion Monthly
Biucis
Fred-DD
MI Moartly
CTT 300
Riderher
LDVID Deathe
Ni Hounty
Quanalty
Yeadse
Wealer
Ca Pats
Dociiti
San Freseico Traffle
Solar
NNS
Vidicle Trips
Coporziio
KDD
Wilmet
add Wasls
Yeals
NO Seetth
M4 Seaoi
LOODD Sari
kil Kain
Vende tre
deggntii Bain
Retakie Cual
stu
XXX
Contre
Sn
Figure 7: Correlation matrix of all benchmark methods' nCRPS ranks across all benchmark datasets. For the computation of the ranks, all classical methods and the default configuration of deep learning models are considered.
25


Reviews:
[{"review_id": "pJ8qPMv96_e", "reviewer": "Reviewer_FZ2f", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses the important question of a large-scale assessment of distributional time series forecasting algorithms, across a wide range of benchmark datasets, comparing both local methods (classical statistical techniques) with global ones (mostly deep learning approaches). Such a large-scale comparison is missing from the literature, and represents a gap in our current understanding of which method best apply to which context. The paper restricts its focus to univariate assessment in discrete time without covariates, which is a reasonable starting point. \n\nIn addition, the paper introduces a method to select hyperparameters and ensembles based on approximation of the Pareto front in a multi-objective setting. This approach is interesting and to the best of my knowledge is novel in a time series context.\n\nThe paper is reasonably well written and easy to follow, although the mathematical notation and writing style could be made more precise, as a few points are listed below. The authors are to be commended for providing full source code to reproduce experiments, and a cursory look suggests that the code is well written and easy to understand.\n\nThe main concern with the paper is that its focus is unclear as it tries to do both too much and too little:\n\n- If viewed as a contribution to the large-scale benchmarking of time series forecasting algorithms, it offers a starting point but falls far short in offering substantive analysis of the presented results, e.g. to convey strong conclusions as to the suitability of particular methods for particular datasets, e.g. regarding time series properties such as variability, intermittency, seasonalities, etc., and possibly grouping benchmark datasets by broad area of study (e.g. retail, economics, energy, ...).\n- If viewed as contributing the `ParetoSelect` algorithm, then this should come much earlier in the paper (in its current position towards the end, it feels like an afterthought), and deeper analysis of the results should be presented, along with a theoretical analysis if possible.\n\nAs written, the paper attempts to cover both objectives, but falls short of offering strong guidance or conclusions. It is therefore difficult to appreciate the paper's takeaways for the ICLR audience, and as such, it would require more work before recommending acceptance to the conference.\n\nDetailed comments:\n- Typographic remark: please put table captions above the table, and figure captions below the figure. (https://tex.stackexchange.com/questions/3243/why-should-a-table-caption-be-placed-above-the-table)\n- p. 3 \"three different context lengths that governs\" ==> govern\n- Section 3.4: since the forecast horizons and history length differ widely, isn't it better to use a *relative* latency measure compared to some simple benchmark (e.g. seasonal naïve)? Otherwise, the latency results are dominated by the datasets with the longest time series, regardless of the model performance per time step.\n- Section 3.5: The results reported in Table 1 include rank information across an unknown set of models: that's obviously more than 7+6=13 introduced in Section 3.2, and seems to include some form of ensembling. However, how the ensembles are constructed has not yet been introduced by that point in the paper. \n- p.5 [Comparison of classical & deep learning methods]: the paper should explain how the test statistic is constructed exactly. Does the test include only the raw deep learning methods, or it includes ensembles as well? Can anything be learned from the datasets for which local methods outperform global methods apart from the number of examples in the dataset?\n- p. 5: \"only a few thousands observations seem sufficient to outperform the classical models considered\": in Figure 1, there are datasets with >1e6 observations where local methods are indistinguishable from global methods, this is way more than \"a few thousands\". \n- p. 6 [top] it seems that $\\mathcal{Q}_\\mathrm{deep}$ and $\\mathcal{Q}_\\mathrm{class}$ represent different model results (best models of each class, respectively) than those given a few paragraphs above (set of all models within each class). It would help readability to have a more precise and coherent notation.\n- Section 4.1: it is not clear if the $M$ different tasks represent different datasets or different objectives. And why does eq. (5) evaluates on the $M$ related tasks only, but not on the main task? The notation should be tightened up.\n- Eq (12): it's not clear how sort is defined for vectors, and does not easily relate to the numbering of datapoints shown in Figure 3.\n- The bibliographic entries should be checked for completeness. For instance Pfisterer et al. (2021) and Winkelmolen et al. (2020) are incomplete.\n- p. 20: \"we set each quantile to the forecasted value\" ==> do you mean that $q_{0.5}$ receives all the probability mass and the others receive zero mass? ", "summary_of_the_paper": "The paper introduces a large-scale benchmark comparison of univariate distributional time series forecasting algorithms (7 local methods; 6 global methods, and for the latter, ensembles thereof) against 44 time series datasets that have been previously introduced in the literature. It also proposes an algorithm, ParetoSelect, to choose hyperparameter defaults that lie close to the Pareto frontier in a multi-objective evaluation (e.g. model accuracy and inference latency).", "summary_of_the_review": "The paper attempts a valuable contribution to the univariate discrete-time forecasting literature, but by attempting to cover too many things quite superficially, does not appear ready to recommend acceptance at ICLR in current form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QmPNAimcxj", "reviewer": "Reviewer_XSBL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- This is an excellent work comparing a large and diverse set of time series forecasting approach wrt to model performance and prediction latency.\n- The main contribution of this paper is empirical, presenting the latency and performance tradeoffs for various models. The study presented in this paper while not absolutely exhaustive, is still quite valuable to the community. Some ways to improve the study even further are suggested in the weaknesses section.\n- The main observation made in the paper is that deep learning models are competitive or even better than classical models, and the performance gap increases only slightly with exponentially more data. This is an important observation as it breaks a common myth in the time series community. Although this statement may be a little too strong since classical models may still outperform deep models when the number of data points is less than a thousand. The datasets considered in the paper have at least a 1000 points.\n- The experimental results seem convincing, and the datasets used in the paper are quite diverse and vary significantly in size, implying the robustness of the evaluation.\n\n**Weaknesses**\n- The choice of hyper-parameters (number of hidden layers, width, etc) for the various forecasting models has not been clearly mentioned. Were the models tuned in any way? or were default hyper-parameters used? This is the most significant weakness of this paper.\n- Classical models may still outperform deep models when the number of data points is less than a thousand. The datasets considered in the paper have at least a 1000 points.\n- It is unclear why a ranking model is used in place of a regression model. A more concrete justification should be provided for why ranking is equivalent to regression under quantile normalization.\n- The argument behind using a parametric model over a non-parametric model is not completely clear. Non-parametric models such as Gaussian processes can also make use of dataset features with the right choice of a kernel. Although, GPs are not necessarily the best choice for ranking.\n- The latency values in Table 1, do not include confidence intervals / stddevs (only required for the unconstrained models).\n\n\n**Update after rebuttal:**\nThanks for the response. Some of my concerns have been addressed. However, I do also agree with the other reviewers that the organization of the paper and the presentation of the results can be greatly improved and hence I am keeping the same score. Details below:\n- Hyper-parameters: Thanks for pointing to the appendix. Column 2 shows that different search sets for the context length have been used for different deep learning approaches. For a fair comparison, the same set of context lengths should be used since a larger context may provide an extra advantage  to the models.\n- Number of data-points: This point needs to be properly addressed in the paper as well and the claims need to be re-evaluated.\n- Ranking for model selection: My concerns have been fully addressed. It seems sufficiently well motivated to use parametric ranking models since the scales of the accuracy metrics can vary widely across problems. It is only feasible to rank the performance rather than predict the exact accuracy.\n- Contributions in the paper: I do agree with the other reviewers about the presentation of the contributions and the organization of the paper. The fact that this paper also claims to choose default hyper-parameters for various problems, missed my attention when I was reading the paper for the first time. While the results in the paper are quite interesting, the organization can be greatly improved.", "summary_of_the_paper": "This paper studies the tradeoff between primary model performance metrics such CRPS and CRPS rank vs other criteria such as inference latency. A surrogate model is proposed which ranks the various forecasting approaches and is used to predict the performance metrics and the prediction latency of the corresponding models. The surrogate model is used to sequentially evaluate the considered models and optimize the hyper-volume of the Pareto front. An interesting observation made in the paper is that classical models do not always beat deep methods even for relatively smaller datasets consisting of few thousand data-points. It is also shown that, as expected, there is a tradeoff between latency and performance.", "summary_of_the_review": "Overall this a great piece of work comparing tradeoffs between multiple performance objectives for time series forecasting models, including classical and deep models. Some interesting observations have been shown in the paper which are mostly well justified. The main weakness is that the choice of hyper-parameters has not been explained, raising the concern of whether better hyper-parameter choices can improve the baseline results. I am willing to increase my score once the concerns have been addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4dBNTEPfah7", "reviewer": "Reviewer_rQb3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**: \n\n1. Thorough empirical studies on both classical and state-of-the-art forecasting models on different datasets provide insights to researchers and practitioners on choosing appropriate models. The result itself is valuable: e.g., comparison between classical and NN-based methods, global and local models, etc.\n\n2. Learning models in a multi-objective setting is a common problem in many applications. Constructing the model configuration set based on the Pareto front is a natural solution and is proven to be effective over other baselines.\n\n**Weakness**: \n\n1. The overall structure of this paper can be improved. For example, the paper title is ''Multi-Objective Model Selection ...'', but the authors start discussing this topic in the second half of this paper with limited context. This can easily cause misunderstanding to readers what is the major focus of this work.\n\n2. Although the empirical study is an important component, there are still too many experimental details in the paper. It is better to elaborate on some employed methodologies. For example, instead of a plain citation, the authors could add brief introductions on the hypervolume error and $\\epsilon$-net.\n\n3. Since the property of each dataset varies a lot, the authors should also consider providing insights on which type of model is more suitable to datasets with certain properties, while the current paper aggregate the data information too much, and just provide general results on all datasets.\n\n**Other Questions**: \n\n1. Section 4.2, what is the reason that the parameters of MLP are not tuned?\n\n2. What are the layers in figure 3? Are these layers of MLP or others?\n\n3. For models in the Pareto front, how to finally trade off each objective in the experiments?\n\n4. Section 4.4, why are non-uniform weights not helpful in ensemble? Consider an extreme case where the model is just a naive average, then it is reasonable to assign less weight to this model compared with others. If the set of model candidates has already been very good, then the weight distribution should be close to uniform in the first place. Authors could add more clarifications on this matter.", "summary_of_the_paper": "This paper provides comprehensive evaluations of various time series forecasting models on 44 heterogeneous and public datasets. The obtained results, which are evaluated in multiple metrics, can serve as guidance for future researchers on choosing appropriate time series models for a given task. In addition, the authors have also studied how to obtain (or search) a set of default models that can provide good performance under single and multiple objective scenarios. For multi-objective optimization, a method called ParetoSelect is proposed to find the set of models that are in the Pareto front. Results have shown that ParetoSelect has outperformed other model selection methods.", "summary_of_the_review": "This paper has conducted a thorough empirical study on forecasting models, but it still needs some major modifications on paper structures and clarifications on methodologies before being accepted.\n\n## Update after Rebuttal\n\nWe thank the authors for their detailed response in addressing questions and updating their manuscripts. The analysis of dataset characteristics in Appendix F is interesting but it does not appear to provide useful information that can serve as an insight in the future: this is fine on my side since it is not easy to discover patterns on massive datasets. \nAlthough the authors have also addressed the minor questions, we still think the structure of the paper can be further improved to make the paper more readable. The authors need some non-trivial modifications before it can be accepted. We wish the authors good luck in future revisions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The premise of this paper is that the development of time series forecasting methods has traditionally focused on accuracy rather than other criteria such as training time or latency. This work presents a new benchmark, evaluating classical and deep learning-based methods on a number of public datasets. They also propose a technique, ParetoSelect that is able to select models from the Pareto front that can efficiently select models in a multi-objective setting. 

Reviewer XSBL liked the observation that classical methods do not always beat deep learning methods even for very small datasets. They thought that the empirical contribution was valuable and myth-breaking. They also commented that the evaluation was robust. Their main concerns were: inadequate description of hyperparameters, lack of evaluations on *really* small datasets, missing confidence measures for latency results. They also made some suggestions for improving clarity. The reviewers responded, pointing to a description of the hyperparameters in Table 2 of the appendix. They also responded to the reviewer’s comment about very small datasets and explained the advantage of the ranking loss. They made some small adjustments to the paper based on the clarity comments. 

Reviewer PZ2f also thought that the large scale comparison was valuable for the community. Overall they thought it was well written though could be improved w.r.t. Notation and writing style. They even inspected the code. Their primary concern was that the paper lacked focus and “tries to do too much and too little”. Is this a benchmarking effort of previous methods, or is the main contribution the ParetoSelect algorithm? This reviewer thought that due to its superficial coverage of too many things, and it wasn’t ready yet for publication at ICLR. The authors provided quite a comprehensive response to reviewer PZ2f and pointed to some minor improvements in the manuscript.

Reviewer rQb3, like the others, viewed the benchmark analysis as valuable. They thought that the ParetoSelect approach was “natural” and that it was shown to be effective over baselines. Like PZ2f they had some structural criticisms and pressed for more insights. 

Reviewers XSBL and rQb3 continued to engage in discussion through the AC-reviewer discussion phase. XSBL said that the authors’ response addressed some concerns yet raised others w.r.t. hyper-parameter selection. rQb3 updated their review after considering the author's response, feeling that minor concerns were addressed but the paper could still use further development. Overall, after considering the discussion I think that it’s been difficult for the authors to provide any patterns regarding which model performs best for which datasets. To me, a benchmark paper should provide some deeper insight and the paper appears to be struggling to do that. On the other hand, the study is comprehensive. The authors have argued in their response to all reviews, that their evaluation is at quite a different scale compared to other published time series model evaluations. I think that this benchmark paper can provide value to the community yet it could use further work: specifically the authors need to focus the paper and communicate clearer insights from the study.
Decision: Reject