Multi-Objective Model Selection for Time Series Forecasting

1 INTRODUCTION . For decades , businesses have been using time series forecasting to drive strategic decision-making ( Simchi-Levi et al. , 2013 ; Hyndman & Athanasopoulos , 2018 ; Petropoulos et al. , 2020 ) . Analysts leverage forecasts to gauge resource requirements , retailers forecast future product demand to optimize their supply chains , cloud providers predict future web traffic to scale server fleets , and in the energy sector , forecasting plays a crucial role e.g . to predict load and energy prices . In domains like these and many other , more precise predictions directly translate into an increase in profit . Thus , it is no surprise that research on forecasting methods has historically focused on improving accuracy . In addition to more classical local forecasting methods which fit a model per time series , global forecasting models such as deep learning and tree-based models have demonstrated state-of-the-art forecasting accuracy ( Wen et al. , 2017 ; Oreshkin et al. , 2019 ; Salinas et al. , 2020a ; Smyl , 2020 ) when sufficient training data is available ( Makridakis et al. , 2018 ; Januschowski et al. , 2020 ) . This research has led to a large variety of different forecasting models and hyperparameter choices ( Hutter et al. , 2019 ) , where different models exhibit vastly different characteristics , including accuracy , training time , model size , and inference latency . While this variety of forecasting models is a great resource , it introduces challenging questions . On the one hand , researchers would like to understand patterns in the performance of different models and to benchmark new models against existing methods . On the other hand , practitioners are interested in understanding which model performs best for a particular dataset or application . In this work , we address both of these problems : we release one of the most comprehensive publicly available evaluations of forecasting models across 44 datasets . Using this benchmark dataset , we develop a novel method for learning good defaults for forecasting models on previously unseen datasets . Importantly , we adopt a multi-objective perspective that allows us to select forecasting models that are simultaneously accurate and satisfy constraints such as inference latency , training time , or model size . The main contributions of this work can be summarized as follows : • We release the evaluations of 13 forecasting methods on 44 public datasets with respect to multiple performance criteria ( different forecast accuracy metrics , inference latency , training time , and model size ) . This constitutes , by far , the most comprehensive publicly available evaluation of forecasting methods . Those evaluations can be leveraged , for example , to assess the relative performance of future forecasting methods with little effort . • As an example application of this benchmark , we use the data to perform a statistical analysis which shows that only a few thousands observations are required for deep learning methods to outperform classical methods . In addition , we investigate the benefit of ensembling forecasting models . • We propose a novel method that can leverage offline evaluations to learn good default models for unseen datasets . Default models are selected to optimize for multiple objectives . • We introduce a technique for ensembling models in a multi-objective setting where the resulting ensemble is not only highly accurate but also optimized for other objectives , such as a low inference latency . 2 RELATED WORK . Time Series Forecasting . Time series forecasting has seen a recent surge in attention by the academic community that has started to reflect its relevance in business applications . Traditionally , univariate , so-called local models that consider time series individually have dominated ( Hyndman & Athanasopoulos , 2018 ) . However , in modern applications , methods that learn globally across a set of time series can be more accurate ( Oreshkin et al. , 2019 ; Salinas et al. , 2020a ; Lim et al. , 2021 ; Montero-Manso & Hyndman , 2021 ) — in particular , methods that rely on deep learning . With a considerable choice of models available , it is unclear which methods should perform best on which forecasting dataset , unlike in other machine learning domains where dominant approaches exist . In domains such as computer vision or neural architecture search ( NAS ) , offline computations have been harvested and leveraged successfully to perform extensive model comparisons or compare different NAS strategies ( Ying et al. , 2019 ; Dong & Yang , 2020 ; Pfisterer et al. , 2021 ) . However , to the best of our knowledge , no comprehensive set of model evaluations has been released in the realm of forecasting . Consequently , some questions remain open : how much data is required for global deep learning methods to outperform classical local methods such as ARIMA or ETS ( Hyndman & Athanasopoulos , 2018 ) ? What is the impact on accuracy when ensembling different forecasting models ? While some recent methods incorporate ensembling ( Oreshkin et al. , 2019 ; Jeon & Seong , 2021 ) , comparisons are often made against individual models which may cloud the benefit of the method proposed versus the sole benefit of ensembling . Learning Default Models . Finding the best model or set of hyperparameters is often performed via Bayesian optimization given its theoretical regret guarantees ( Srinivas et al. , 2012 ) . However , even with early-stopping techniques ( Golovin et al. , 2017 ; Li et al. , 2017 ) , practitioners often restrict the search to a single model due to the large cost of training many models . One technique to drastically speed up the model/hyperparameter search is to reuse offline evaluations of related datasets via transfer learning . For instance , Wistuba et al . ( 2015 ) ; Winkelmolen et al . ( 2020 ) ; Pfisterer et al . ( 2021 ) leverage offline evaluations to alleviate the need for training many models by learning a small list of n good defaults that provide a small joint error when evaluated on all datasets . This approach bears a lot of similarity with ours . Key differences are that we consider multiple objectives and model ensembles . In the time series domain , Shah et al . ( 2021 ) also considers the task of automatically choosing from a large pool of forecasting models the one performing best on a particular dataset but also only optimizes for a single objective and relies on potentially expensive model training for model selection . 3 BENCHMARKING FORECASTING METHODS . In this section , we formally introduce the problem of time series forecasting and subsequently provide an overview of the benchmark evaluations that we release . By evaluating 13 forecasting meth- ods along with different hyperparameter choices on all 44 benchmark datasets and for two random seeds , the benchmark comprises 4,708 training runs and amasses over 1 TiB of forecast data . At the end of this section , we demonstrate how this benchmark data can be used to perform an extensive comparison of contemporary forecasting methods . 3.1 TIME SERIES FORECASTING . The goal of time series forecasting is to predict the future values of one or more time series based on historical observations . Formally , we consider a set Z = { z ( i ) 1 : Ti } K i=1 of K univariate , equally-spaced time series of lengths Ti , where z ( i ) a : b = ( z ( i ) a , z ( i ) a+1 , . . . , z ( i ) b ) denotes the vector of observations for the i-th time series in the time interval a ≤ t ≤ b . In probabilistic time series forecasting , a model then estimates the probability distribution across the forecast horizon τ , i.e . the distribution over the τ future values , from the historical observations : P ( z ( i ) Ti+1 : Ti+τ ∣∣∣ z ( i ) 1 : Ti ) ( 1 ) Models typically approximate the joint distribution of Eq . ( 1 ) via Monte Carlo sampling ( Salinas et al. , 2020a ) or learn to directly predict a set of distribution quantiles for multiple time steps using quantile regression ( Wen et al. , 2017 ; Lim et al. , 2021 ) . 3.2 METHODS . We consider 13 models in total that can be distinguished into 7 local methods ( estimating parameters individually from each time series ) and 6 global methods ( learning from all available time series jointly ) . Details about models can be found in Appendix A . Local Methods . We use Seasonal Naïve ( Hyndman & Athanasopoulos , 2018 ) and NPTS ( Rangapuram et al. , 2021 ) as simple , non-parametric baselines . Further , we consider ARIMA and ETS ( Hyndman & Athanasopoulos , 2018 ) as well-known statistical methods and STL-AR ( Talagala , 2021 ) and Theta ( Assimakopoulos & Nikolopoulos , 2000 ) as inexpensive alternatives . Lastly , we include Prophet ( Taylor & Letham , 2018 ) , an interpretable model that has received plenty of attention . Global Methods . All global methods that we use are deep learning models , namely : Simple Feedforward ( Alexandrov et al. , 2020 ) , MQ-CNN and MQ-RNN ( Wen et al. , 2017 ) , DeepAR ( Salinas et al. , 2020a ) , N-BEATS ( Oreshkin et al. , 2019 ) , and TFT ( Lim et al. , 2021 ) . For each model ( except MQ-RNN ) , we consider three hyperparameter settings : this includes the default set provided by their implementation as well as hyperparameter sets that roughly halve and double the default model capacity ( see Table 2 in the appendix ) . Additionally , we consider three different context lengths that govern the length of the time series that predictions are conditioned on . Model Training . Model training is only required for deep learning models since parametric local methods estimate parameters at prediction time . Deep learning models are trained for a fixed duration depending on the size of the dataset . Further details can be found in Appendix C . 3.3 DATASETS . Our benchmark provides 44 heterogeneous public datasets in total . The datasets greatly differ in the number of time series ( from 8 to ≈170,000 ) , their mean length ( from ≈22 to ≈500,000 ) , their frequency ( minutely , hourly , daily , weekly , monthly , quarterly , yearly ) , and the forecast horizon ( from 6 to 60 ) . Thus , we expect them to cover a wide range of datasets encountered in practice . Datasets are taken from various forecasting competitions , the UCI ( Dua & Graff , 2017 ) , and the Monash time series forecasting repository ( Godahewa et al. , 2021 ) . Dataset sources , descriptions , basic statistics , and an explanation of the data preparation procedure can be found in Appendix B . 3.4 METRICS . To measure the accuracy of forecasting models , we employ the normalized continuous ranked probability score ( nCRPS ) ( Matheson & Winkler , 1976 ; Gneiting & Raftery , 2007 ) whose definition is given in Appendix A . The benchmark dataset that we release contains four additional forecast accuracy metrics ( MAPE , sMAPE , NRMSE , ND ) . Besides forecasting accuracy metrics , we store inference latency , model size ( i.e . number of parameters ) and training time for deep learning models . Latency is measured by dividing the time taken to generate predictions for all test time series in the dataset by their number .

Summary: This paper studies the tradeoff between primary model performance metrics such CRPS and CRPS rank vs other criteria such as inference latency. A surrogate model is proposed which ranks the various forecasting approaches and is used to predict the performance metrics and the prediction latency of the corresponding models. The surrogate model is used to sequentially evaluate the considered models and optimize the hyper-volume of the Pareto front. An interesting observation made in the paper is that classical models do not always beat deep methods even for relatively smaller datasets consisting of few thousand data-points. It is also shown that, as expected, there is a tradeoff between latency and performance.