Deep Probabilistic Time Series Forecasting over Long Horizons
Authors: Gregory Benton
Nate Gruver
Wesley Maddox
Andrew Gordon Wilson
Venue: Submitted to ICLR 2023 (2023)
Keywords: time series
neural networks
probabilistic forecasting

Abstract: Recent advances in neural network architectures for time series have led to significant improvements on deterministic forecasting metrics like mean squared error. We show that for many common benchmark datasets with deterministic evaluation metrics, intrinsic stochasticity is so significant that simply predicting summary statistics of the inputs outperforms many state-of-the-art methods, despite these simple forecasters capturing essentially no information from the noisy signals in the dataset. We demonstrate that using a probabilistic framework and moving away from deterministic evaluation acts as a simple fix for this apparent misalignment between good performance and poor understanding. With simple and scalable approaches for uncertainty representation we can adapt state-of-the-art architectures for point prediction to be excellent probabilistic forecasters, outperforming complex probabilistic methods constructed from deep generative models (DGMs) on popular benchmarks. Finally, we demonstrate that our simple adaptations to point predictors yield reliable probabilistic forecasts on many problems of practical significance, namely large and highly stochastic datasets of climatological and economic data.

Under review as a conference paper at ICLR 2023
DEEP PROBABILISTIC TIME SERIES FORECASTING OVER LONG HORIZONS
Anonymous authors Paper under double-bling review
ABSTRACT
Recent advances in neural network architectures for time series have led to signifi- cant improvements deterministic forecasting metrics like mean squared error. We show that for many common benchmark datasets with deterministic evaluation metries, intrinsic stochasticity is significant that simply predicting summary statistics of the inputs outperforms many state-of-the-art methods, despite these simple forecasters capturing essentially no information from the noisy signals in the dataset. We demonstrate that using a probabilistic framework and moving away from deterministic evaluation acts simple fix for this apparent misalignment between good performance and poor understanding. With simple and scalable ap- proaches for uncertainty representation we can adapt state-of-the of-the-art architectures for point prediction be excellent probabilistic forecasters, outperforming com- plex probabilistic methods constructed from deep generative models (DGMs) on popular benchmarks. Finally, we demonstrate that our simple adaptations to point predictors yield reliable probabilistic forecasts on many additional problems practical significance, namely large and highly stochastic datasets of climatological and economic data.
Exchange
ETTm2
Standardired Y
-0.2
i.
=0.4
Autaforme
Data
-0.50
-2.75
Emp rical
-1.0
50
300
150
200
1.00
150
200
50
1.00 3.50
200
Time
Figure 1: Example predictions exchange rate (left), ETTm2 (a sequence of electricity transformer temperature readings, center), and weather (right) for NHiTS (Challu et al., 2022b), Autoformer (Wu et al., 2021), and last value predictions, well as the historical standard deviation of the change from the last observed value. On the exchange (Lai et al., 2018) and ETTm2 (Zhou et al., 2021) datasets there is minimal structure be exploited except very short horizons, and forecasts tend to under-perfome simple baselines. On semi-structured datasets like weather, models can capture some overall structure, such NHiTS accurately predicting the final values in the forecasting window, but are still only on par with naive predictions. From these plots we see why probabilistic evaluation is necessary and point estimates are insufficient.
INTRODUCTION
Following deep learning's breakthroughs in sequence modeling for text and audio. significant research effort has sought to achieve comparable success in time series, where the unique challenges of data scarcity and long-range dependencies have created niche for creative architecture design. In rapid succession, many models and architectures have demonstrated improved point predictions on benchmarks adopted by the community (Challu et al., 2022b; Salinas et al., 2020; Wu al., 2021). These methods hold incredible promise for real impact on time series based decision making, especially in economic domains that require highly accurate long-term predictions.
While demonstrating steady improvements, however, research deep learning for point prediction frequently ignores a key but simple fact: the real world is complex and predicting the future accurately from past observations alone often impossible. In highly structured time series, such observed

---

Under review as a conference paper at ICLR 2023
Table 1: Multivariate results with varying prediction lengths. Bolded results indicate the best performing model, and italics the second best. In all cases simple statistics of the input data to the model are either the first or second best performing models terms of both MSE and MAE accuracy. Historical Inertia (HI) (Cui et al., 2021) was also introduced trivial baseline but has worse performance than our constants.
Models
Mean
Last Value
N-HITS
Autoformer
Informer
ARIMA
HI
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Weather ETTm2 Exchan.
96
0.1199 0.269
0.081
0.196
0.092
0.211
0.197 0.323 0.847
0.752
0.296
0.214
0.112 0.251
336
0,384
0.454
0.305
0.396
0.371
0.443
0,509
0,524
1.672
1.036
2.298
0.467
0.434 0,517
720
0.938
0.736
0.823
0,681
0.888
0.723
1.447
0.941
2.478
1.310
20.666
0.864
0.955 0.772
96
0.150
0.272
0.203
0.312
0.176
0.255
0.255
0.339
0.365
0,4536
0.225
0.301
0.158 0.261
336
0.205
0.313 0.270
0.361
0.295
0.346
0.339
0.372 1.363
0.887
0.370
0.386
0.379 0.438
720
0.261 0.350
0.335
0.401
0.401
0.426
0.422
0,419 3,379
1.388
0.478
0.445
0.446 0.477
96
102116
0.271 0.259
0.254
0.158
0.195
0.266
0.336 0.300
0.384
0.217
0.258
0.345 0.339
336
0.313
0.336
0.377
0.338
0.274
0.300
0.359
0.395 0.578
0.523
0.330
0.347
0.529 0.440
720
0.380 0.377 0.465
0.394
0.351
0.353
0.419 0.428 1.059
0.741
0.425
0,405
0.545 0.439
traffic, with strong periodicity from both daily and weekly cycles, we may be able to forecast point predictions with a high degree of accuracy. However, in stochastic time series which display only modest structure (e.g. periodicity seasonality), such as precipitation or wind speed patterns, we cannot hope to produce accurate predictions of specific future outcomes using only historic observations.
Researchers working in architecture design for timeseries frequently overlook intrinsic stochasticity in benchmark datasets. In Figure we show how even sophisticated methods struggle to forecast accurately on data with a low signal-to-noise ratio. In Table 1 we take this observation further and show that, shockingly, naive constant predictors outperform two state-of-the of-tte---tttttttt time series models (Lai et al., 2018: Zhou et al., 2021) on several widely reported MSE evaluations. The numbers shown are taken directly from Challu et al. (2022b), Wu et al. (2021), and Zhou al. (2021), but include new columns showing the performance of simply predicting either the mean or the last value of the observations. Notably, these datasets span many domains of practical interest (finance, energy, and climatology), and contain varying levels of structure and periodicity.
We present these surprising shortcomings of state-of-the-ar models in order to encourage adoption of more meaningful evaluations. Although these deep learning methods are extremely good at extracting and extrapolating trends and periodic structure far into the future, aspiring to predict a constant value in the best case reflects a misalignment. Meaningful comparisons require baseline that excels in both highly predictable and stochastic environments.
Keeping in mind the need for probabilistic, rather than deterministic frameworks for forecasting, we instead ask if the strong trend extrapolation performance of point prediction models is being overlooked in the probabilistic time series literature. For datasets that have both noise and structure (e.g. wind), we find that high performing point predictors, such as NHiTS, typically outperform the mean forecasts provided by state of the art probabilistic methods. In Figure 2 we show the cumulative mean squared error for forecasts produced by NHiTS and the mean prediction taken from several popular probabilistic frameworks on common benchmark datasets as well as real world data.' Motivated by improving probabilistic forecasting while retaining the strength in trend extrapolation found in some deterministic models, we explore adapting point predictors to the probabilistic setting rather than building new models entirely from the ground up.
In particular, we examine two baselines for constructing high performing probabilistic forecasters, built on quantile regression and heteroscedastic variance models (Section 3). These methods leverage advances in architecture design for time series modeling without succumbing to the pitfalls of point prediction. In Section 4, provide detailed comparison of our models with state-of-the-ar deep learning methods for probabilistic forecasting, demonstrating that recent advances in architecture design are directly relevant to uncertainty quantification. Finally, in Section 5, we demonstrate
"Models and datasets described in Sections 4 and 5 along with plats of cumulative CRPS scores.
2

---

Under review as a conference paper at ICLR 2023
Exc Exchange Rate
1.5
Traffic
1.30
Precipitation
2000
Wind Speed
NAS
Comunntive 10
400000
Model Type
NHTT
0.50
1000
TimeGred
Traselor
0.25
500
Deeple
NQ-MhN
200 Honzon
200
0.90
200
2.00
400
100
HG ONN
Horizon
Horiznn
Horizn
Horizon
Figure 2: Cumulative mean squared error with shade representing 2 standard deviations over 5 trials for both NHITS and the mean prediction from several popular probabilistic forecasting models. NHITS has better mean predictions than the probabilistic approaches in cemi-structured settings like precipitation and wind speed modeling, where there are still underlying trends caused by seasonality and daily effects. To build better probabilistic models we aim leverage the accurate trend capture abilities of models such as NHiTS.
our methods' exemplary performance on large and noisy datasets for climatology and financial forecasting.
We provide our code at http://aoooopoooosoooaccccccc/ct
2 RELATED WORK
Insights from classical methods. Prior to the first applications of machine learning in forecasting (De Gooijer and Hyndman, 2006), time series methods grew out of signal processing. Exponential smoothing (Brown, 1959), ARIMA (Box and Jenkins, 1968), state space models (Kalman, 1960; Shumway and Stoffer, 1982). and the (G)ARCH families of models (Engle, 1982) use simple filters or autoregressive models to extract trend components from noisy data. Many modern forecasting methods move beyond fixed parametric representations in order to better extract complex structure from data, but signal processing fundamentals continue to be guiding principles behind approaches that use filtering decompose inputs and construct predictions.
Deep decomposition approaches. Many recent deep learning architectures provide generaliza- tions of older scasonality accoumting approaches using decompositions. N-BEATS (Oreshkin et al., 2019), for example, times series, while N-HiTS (Challu et al., 2022b) combines this approach with multi-scale forecast- ing setup that uses several different networks predict at different scales. Similarly, Smyl (2020) uses RNNs with exponential smoothing for seasonality components, ultimately winning the M4 forecasting competition. Exponential smoothing has been key factor in many recent architectures. such as Autoformer (Wu et al., 2021) and ETSFormer (Woo et al., 2022), which both use smoothing modules as part of transformer-ing insprred architectures. Wen et al. (2017) use sequence-to-see -o-quuence RNN and CNN models in conjunction with quantile regression to produce a multi-horizon quantile forecasting model.
several layers removing trend and scasonality components from univariate
As we demonstrated above, these methods have a critical but easily resolvable shortcoming: they are constructed and evaluated without uncertainty in mind. In order to investigate the true impact of developments and deep decomposition methods, we draw on recent work in probabilistic forecasting.
Probabilistic forecasting. Salinas et al. (2020) proposed a deep autoregressive forecasting method with simple closed-form likelihood models. Other methods have combined deep autoregressive models with more complex conditional density estimators in an effort to better model the joint distribution over covariates. Salinas al. (2019), for example, extend the popular non-parametric Gaussian copula process volatility (GCPV) model of Wilson and Ghahramani (2010) for high- dimensional forecasting with RNNs. More recently, authors have turned to using normalizing flows (Rasul et al., 2020; de Bézenac et al., 2020) and denoising diffusion models (Rasul et al., 2021; Tashiro et al., 2021) as highly flexible conditional density estimators. In particular, Rasul et al. (2020) and Rasul et al. (2021) condition normalizing flow and diffusion models on the outputs of an RNN to construct joint model over all timesteps. In contrast, Tang and Matteson (202) 1) eschew conditioning on deterministic backbone and uses a fully probabilistic transformer architecture.
3

---

Under review as a conference paper at ICLR 2023
Future
Empirical Density
Empirical CDF
History
Cheap Simple
Less Cheap & Less Simple
Residual Gaussian
Quantile Regression
DGM
Bayesian
I (x) P()
Figure 3: In probabilistic forecasting form distribution possible future outcomes that most closely matches the empirical distribution at some future time point. Given a model architecture capable of mean trend extrapolation simultaneously predict many quantiles of the distribution, as in quantile regression. These approaches are inexpensive relative using a deep generative model (DGM) or the Bayesian posterior to quantifying uncertainty. We find, surprisingly, that residual Gaussians and quantile regression coupled with powerful point predictors often outperform more complex and expensive alternatives in the long horizon setting.
simply fit Gaussian distributions to the residuals,
While prior work in prohabilistic forecasting has focused on high-dimensional density estimation, we show that architecture design should be prioritized instead. Indeed, with extensive experiments, demonstrate that when given successful long-term predictor simple uncertainty representation is sufficient.
EXTENDING POINT FORECASTING METHODS
Prior work contains many methods for converting sequence models into probabilistic forecasters. We focus attention approaches that simple and computationallyy efficient (Figure 4): quantile regression (Chernozhukov et al., 2010) and two-stage heteroscedastie Gaussian regression (Nix and Weigend, 1994). While quantile regression provides more flexible forecast distribution that can asymmetric about the mean prediction, the Gaussian regression approach gives a closed form forecast which has practical use in applications like risk prediction.
Quantile Regression We first consider quantile regression (QR) for representing uncertainty because of its unique combination simplicity and flcxibility.. Unlike simple parametric families, QR can learn arbitrary distributions while maintaining efficient sampling. QR learns conditional order statistic, f+ (x), given a quantile level T E (0, 1) using the pinball loss (Chernozhukov et al., 2010):
L+(fr.y) (f+ v)( S f-(x))
(1)
With 0.5, this loss reduces to mean absolute error, which yields an estimate of the conditional median. If we take = 0.01 instead, the model yields an estimate of the smallest conditional percentile. To estimate full predictive distribution, for quantile levels {To. ..., Th) simultancously (Chung et al., 2021; Rodrigues and Percira, 2020; Tagasovska and Lopez-Paz, 2019). At test time,
train series of conditional estimates
interpolation to ensure monotonic quantiles
(Chernozhukov et al., 2010).
Quantile regression is popular instrument within the probabilistic forecasting community (Gouttes et al., 2021: Kan et al., 2022; Park et al., 2022; Romano et al., 2019), and we do not propose any novel extensions of it. Instead, show that simple UQ methods outperform more complex and computationally expensive techniques. When using quantile regression in conjunction with NHiTS we refer to the method NHiTS-QR.
Heteroscedastic Gaussian Regression Distinet from NHiTS-QR propose train neural network to output the variance parameter in Gaussian likelihood, analogous to earlier approaches (Bishop, 1995; Nix and Weigend, 1994; Rodrigues and Pereira, 2020). Here, assume het-
4

---

Under review as a conference paper at ICLR 2023
Prediction Time
Prediction Time
0.02
Model Type
0.03
NHITS-G
NHITS-QR
0.01
0.02
NHITS-5WAG
TimeGrad
TransMAF
100 200
300
400
500
200
400
600
Time Series Dimension
Prediction Horizon
(a) Inference time Dimension.
(h) Inference Time Prediction Horizon.
Figure 4: (a) Time in seconds to generate a single forecast as a function of the output dimensionality of the time series. (b) Time in seconds to generate single forecast function of prediction length. In both cases we see that the modified point predictor models (NHiTS-G and NHiTS-QU run much faster than either diffusion-base -asodd models for all but the longest highest dimensional forecasts.
Exchange
Solar
Electricity
Value
Targets Orignal Cutoff
200
400
600
800
200
400
600
8.0
200
400
600
800
Horizon
Horizon
Horizon
Figure 5: Many previous works on probabilistic time series forecasting have focused on short horizons, shown as the vertical dashed line. Here we extend the targets to more than 20 times the original length. By expanding both the forecast horizon and the test set size able to validate methods longer horizons as well as broader distribution of test points.
eroscedastic Gaussian observation model, Je ~ -N (f )e, (f(((()))))))) where f(.) is hierarchi- cal time series model as described in Challu et al. (2022a) and g(.) .) a simple multi-layer perceptron. We term this model WHliTS-Gaussian (NHiTS-G), To train this model, we use maximum likelihood two-step manner, first training the mean model, f()), using MSE loss, and then fixing that model and training the variance model, g(-). by maximizing the likelihood of the Gaussian observation model. Our approach parallels Model Agnostic Quantile Regression (MAQR) (Rodrigues and Pereira, 2020) and the heteroscedastic MLPs fit in Nix and Weigend (1994) and Bishop (1995), but with inductive biases particularly suited for probabilistic time series. While it strong assumption assume the marginal distribution each time point is Gaussian, this model serves as a simple baseline, and as we show in Section 4 is capable of outperforming more expensive methods.
BENCHMARK EXPERIMENTS
Dataset Description We consider multivariate time series datasets provided by the GluonTS time series library and used in several time series papers focused on uncertainty quantification (UQ) (Salinas et al., 2019; 2020; Tang and Matteson, 2021). There are notable shortcomings with this experimental setup, however, the default formulations predictions. For the exchange, solar, electricity, and traffic datasets the prediction horizon is only either 24 30 time steps, and the test sets comprised of more than seven testing forecast windows. These prediction horizons are less than 10% of the same evaluation sets as commonly used in the poo-bbbbaiilisttc time series literature, c.g. Challu al. (2022a); Cui al. (2021).
limited test sets and short horizon
To test on these datasets more fully and compare the performance of probabilistic forecasters in more challenging and realistic settings we modify the testing and training splits from leaving approximately 3% of the data as the test set to be 80% train, 10% validation, 10% test in order to assess the performance across set of long range horizons of 100's of steps into the future, with thousands of test points. We show examples of the expanded forecast windows in Figure 5, for further information on dataset setup, and additional experimental results see Appendix
Metrics For probabilistic forecasts, we compute CRPS (Gneiting and Raftery, 2007: Matheson and Winkler, 1976), summed over the dimensions of the time series, and averaged across the prediction
5

---

Under review as a conference paper at ICLR 2023
times. For single prediction, the CRPS score is defined against the estimated cumulative distribution function (CDF), F as
CRPS(F - ( ((=-v)0-))))) -()
where F(2) estimated as the empirical CDF produced by sampling from our forecasts. Given our forecast contains multivariate time series and is predicted out over multiple time steps, we compare using CRPSsum= first by summing over the dimensions of the time series, then by averaging over time points. The summation gives Fuum (4) and yt = E; Vt, where yt, is the dt dimension of time series y at time I, and the average yields CRPS = E., CRPS( (4): yt) CRPS is a proper scoring rule and takes into account both the sharpnes and accuracy of the prediction. CRPS can also be computed directly from the pinball loss (Eq. 1) as
CRPS(F, v) - 2 J. Lr(y, L=(u,F-(T))drr
the integral of the pinball loss over all quantiles (Gneiting and Raftery, 2007; Laio and Tamea, 2007).
Baselines We include several probabilistic time series methods as baselines for comparison. These methods cover a breadth of approaches to time series forecasting including autoregressive models (Salinas et al., 2020), sequence to sequence based quantile regression (MQ-RNN and MQ-CNN) (Wen et al., 2017), and diffusion models TransMAF and TimeGrad (Rasul et al., 2021; 2020). These models represent state of the art and are highly competitive approaches on the widely used GIuon'TS datasets seen in Rasul et al. (2021). Despite the strong performance shown by these methods for short horizon forecasting, at longer horizon predictions we see these established baselines suffer in performance, with the strong trend extrapolation provided by using an underlying deterministic model being critical to forecasting success.
4.1 PROBABILISTIC LONG TERM FORECASTING
While recent models such as TimeGrad (Rasul al., 2021), TransMAF (Rasul et al., 2020), and DeepAR (Salinas et al., 2020) have achieved sigmificant improvements in continuous ranked probabil- ity scores on benchmark datasets, these methods have primarily been explored in short forecasting settings with limited sized test sets. While the datasets typically contain thousands of observations in time, only the last few hundred are reserved for testing amounting in some cases to a mere 1% the data, and only in prediction intervals of 24 time steps, see Figure 13 for plots of the standard splits.
primary strength of decomposition based methods like Autoformer and NHiTS is their ability to produce long term forecasts, showing success with predictions out to 720 time steps in the future (Challu et al., 2022b; Wu et al., 2021). Our probabilistic forecasters build on these methods to be well suited for producing accurate long horizon mean forecasts, and ultimately strong probabilistic modeling due to the underlying strength of the point predictor.
Long Range Evaluation To more accurately gauge performance over long term forecasts with larger test sets, we change the GluonTS datasets to a 80% /20% train/test split, and predict on a horizons of {96, 192, 336} time steps. We show the results in Figure
While in highly stochastic settings with little trend to be extrapolated, such as the Exchange rate dataset, the advantages using models like NHiTS-G and NHiTS-QR are muted. However as the datasets exhibit more structure, such as the daily effects in electricity consumption and traffic congestion, or the seasonal effects in solar radiation, the improved trend extrapolation given by the underlying point predictor leads to stronger performance. This trend is more significantly pronounced at longer forecast horizons, where we see NHiTS-G and NHiTS-QU achieving state of the art in longer term forecasting for the electricity, traffic. and solar datasets.
Bayesian methods In probabilistic time series forecasting we are primarily concerned with rep- resenting epistemic uncertainty, uncertainty about the forecasts model is producing (rather than alcatoric, or irreducible, uncertainty). If our aim is to accurately represent our model uncer- tainty without altering the base architecture then Bayesian methods are natural choice (MacKay,

---

Under review as a conference paper at ICLR 2023
Prediction Length 96
Prediction Length 192
Prediction Length 336
Model Type
NHiTS-G
NHITS-O0 QR
CRPS
TimeGrad
DeepAR
0.2
MQ ANN
0.0
-1
Time
Exch.
Elec.
Traf.
Solar
Figure 6: Barplot Of CRPS scores for varying prediction lengths. As the forecast horizon increases, NHiTS approaches consistently outperform current state of the methods. Note, we used several arper-paraeetess for TransMAF, including those given in publicly available implementation here, but were not able to find stable training procedures for all datasets. Error bars represent #2 standard deviations over 5 random initializations, the y-axis is truncated for clarity.
Exchange
Solar
Traffic
Electricity
200
200
0.5
NHITS-G
Value
1.6
100
150
SWAG Samples Ensemble
1.4
100
0
50
50
50
50
Harizon
Horizon
Horizon
Horizon
Figure 7: Example forecasts produced by NHiTS-G in comparison to samples produced by SWAG and deep ensembles. In all cases the deep ensemble samples produce limited diversity and do not provide the uncertainty needed for probabilistic forecasting. SWAG samples tend to be noisy, and can fail to predict trends even when we confident of their existence traffic.
1992), and recent advances in approximate Bayesian neural networks (BNNs) have given us simple approaches to build models with out of the box probabilistic forecasts (Maddox et al., 2019).
In Figure 7, we show uncertainties learned by NHiTS- side by side with two different approximations of model uncertainty, Deep Ensembles (I Lakksminarayaaan et al., 2017) and SWAG (Maddox et al., 2019) for several datasets (Lai et al., 2018). On the traffic dataset, the base architecture is easily capable of producing accurate mean forecasts, leading to both ensembles and SWAG producing homogeneous forecasts that under-represent our true uncertainty. For the solar and traffic datasets are able forecast means that fit the cyclical nature of the data, but cannot predict peak solar radiation or traffic consistently leading to the need for uncertainty in our forecasts. In these cases SWAG overestimates model uncertainty and fails confidently predict the mean, when it is appropriate to do so (such as predicting low solar radiation at night).
We show the cumulative CRPS of both the adapted point forecasting methods, as well as SWAG and deep ensembles in Figure 8. Despite promising performance on short horizons, as the length of the forecast increases the performance of SWAG and deep ensembles degrades, and both NHiTS based models tend to perform much better at long term predictions. While Bayesian methods might also provide a simple mechanism for wrapping uncertainty around an effective base model, they carry serious computational costs at test time, because we need to do as many forward passes through the model as there are samples, as illustrated in Figure
Cumulative CRPS
1e7 Electricity
Exchange
1e6
Solar
Traffic
10000
Type NHTSS G
1.0
0.5
5000
MHrTs-OR QR
Ensemble
500
500
500
Figure 8: Cumulative CRPS of the NHiTS based methods with Bayesian baseline adaptations. While on short horizons SWAG and deep ensembles can provide reasonable forecast distributions, at longer horizons we find limited sample diversity or failure predict trends leading to reduced performance.
7

---

Under review as a conference paper at ICLR 2023
Model Type
Model Type
NHITS-G
CRPS
0.4
TimeSred
Callbration
0.75
NHITS QR
TimeGrad
0.2
TransWal
0.30
TransMaf
0.25
DeepAR
0.0
90--NN
MC-ANN
96
336
MQQ CNN
Length
Percentile
(a) Precipitation CRPS
(b) Precipitation Calibration
Figure 9: Left: CRPS scores for various prediction lengths in precipitation forecasting. Right: Calibration curves for forecasts of length 720. Both NHiTS G and NHiTS-QR achieve low CRPS, outperforming all competing methods at every time horizon. While NHiTS-G attains lower CRPS than NHiTS-QR, NHITS-QR produces highly calibrated forecasts. These metrics together suggest that methods like NHiTS-QR have practical relevance in climatological study, where accurate simulation a key area of interest.
5 NOISY, LONG-HORIZON DATASETS WITH RICH STRUCTURE
Climatology and finance are key areas of interest for probabilistic forecasting, they are stochastic domains where we cannot make confident predictions over long horizons, but retain exploitable structure like long term trends cyclical climatological patterns. The ability to estimate distributions of climatological quantities of interest such as precipitation and wind speed has broad impact in ecological impact study, insurance pricing models, architectural and civil engineering, and beyond. Similarly, the probabilistic financial forecasting are central to the applications market pricing, auto- mated trading, asset allocation. To demonstrate the efficacy of adapting deterministic models like NHiTS to probabilistic forecasting we examine predictive performance on two real world climatology datasets and historical prices from the component assets of the NASDAQ-100.
For these experiments we also consider calibration, as applications such as simulation and risk estimation demand that the probabilistic forecast is faithful representation of the ground truth distribution at all quantiles. For a time series and forecast time T, to compute the calibration at p. Cp. we compute the empirical quantile of the forecast 4r where PPrr (r) p. Cp is then the observed frequency yr Qr over all forecasts and time horizons, yielding Cp TK Ee >T Iy,, <gg. a predictor is calibrated, Cp s p for all p (0, 1), thus we can assess the overall calibration of predictor by plotting calibration against percentiles, which should be close to the identity line.
Precipitation Forecasting For precipitation forecasting we consider the United States Historical Climatology Network (USHCN) long-term daily climate records dataset (Menne et al., 2015). This dataset contains daily recorded precipitation values at approximately 1200 locations over the con- tinental United States with observations ranging back over 100 years. In order to more accurately consider scenario in which rather than a single observed event, we preprocess the data by averaging over day rolling windows.
forecasting typical precipitation for given day and location,
varying horizons, well calibration for forecasts made over 720 time steps. Both NHiTS-G and NHiTS-QR attain low CRPS at all time horizons, and achieve high calibration relative to competing methods. While diffusion-based TimeGrad provides calibration on par with NHiTS-QP it yields much higher CRPS, particularly at the longer horizons that we specifically interested in. MQ-CNN provides competitive performance in terms of CRPS for long forecasts, but gives very poor calibration, showing that the central quantiles of the forecast distribution are not well matched to the data.
In Figure 9b show CRPS performance
Wind Speed Forecasting For wind speed forecasting we use the United States Climate Reference Network (USCRN) dataset composed of wind speed observations taken 5 minute intervals at 154 spatial locations in the United States for the full 2021 calendar year (Diamond et al., 2013). Wind speed forecasting provides a particularly challenging problem for forecasting models, as there many cyclic trends in the data that relate to both seasonal and daily effects, but there are also gust-like effects that produce erratic localized behavior including random periods of either high or absent wind.
8

---

Under review as a conference paper at ICLR 2023
Model Type
Model Type
NHITS-G
CRP5
Timedred Transtor
Callbratto
0.75
NHITS QR
TimeGrad
No AM
Doog6t
0.50
DeepAR
TransMaf
NO-ChN
MC-ANN
96
336
0.00
MQQ CNN
Length
Percentile
(a) Wind Speed CRPS
(h) Wind Speed Calibration
Figure 10: Left:: CRPS scores for various prediction lengths wind speed forecasting. Right: Cali- bration curves for forecasts of length 720. For all time horizons NHiTS-G NHiTS-QR and TimeGrad achieve low CRPS, with NHiTS- and TimeGrad both producing well calibrated forecasts. Wind forecasting presents unique challenges due the stochastic nature of gust-like effects, and the ability of NHiTS-QR to produce calibrated forecasts indicates that we are accurately capturing both the frequency and magnitude of such effects.
Model Type
Model Type
1.0
KM 6
x
NATSiOn
Callbweerr
② 0.1
ThreCrad
Thunotal
Deeple
My
MO-1911
0.0
Model
My CKN
Percentile
MD.CW
(a) NASDAQ-100 CRPS
(h) NASDAQ-100 Calibration
Figure 11: Left: CRPS scores for prediction lengths of 200. Error bars represent one standard deviation from the mean over 5 trials, and the plot is truncated on the y-axis for clarity. Right: Calibration curves. Both variants of the probabilistic adaptation NHiTS achieve low CRPS values, however NHiTS-QR in particular achieves near perfect calibration. NHiTS-G underperforms in calibration as may be expected from the non-normality of securities prices.
Figure 10b includes both the CRPS and calibration comparison for the probabilistic NHiTS variant and competing methods. In wind speed forecasting TimeGrad performs approximately as well the adapted deterministic models in terms of both CRPS and calibration, but the remaining methods suffer both from higher CRPS scores and poor calibration.
Financial Data To demonstrate the performance of the probabilistic forecasting for real world financial data construct dataset containing hourly data from the component assets of the NASDAQ-100 for the two years leading up to September 2022. this setting we forecast 200 time steps into the future, which equates approximately a month of hourly predictions when accounting for trading hours. We show in Figure 11b that again both NHiTS-G and NHiTS-QR are among the best performing models, with NHiTS- SRR achieving nearly perfectly calibrated forecasts. Given the typically assumed log-normality of stock prices it is unsurprising that NHiTS-G is asymmetrically miscalibrated relative to its performance the climatology datasets.
CONCLUSION
Time series has rapidly become popular target of innovation in neural network architecture design, and we argue that this increased popularity must be accompanied by increased scrutiny of common baselines and evaluation methods. In particular, we demonstrate that probabilistic evaluation is critical for understanding the efficacy of new methods on highly stochastic data. Without probabilistic evaluation, many sophisticated methods can be matched or outperformed with naive predictors. In promoting probabilistic methods and evaluations, we also emphasize the importance of simple baselines for uncertainty quantification. From our experimental results, we conclude that architectural design can have significant impact on the quality of learned forecasts. Contrary to a perceived consensus in recent work, we find that elaborate uncertainty quantification techniques are less important than architecture design. We hope that work provides foundation for future methods in probabilistic forecasting, especially those that target long horizons and noisy datasets.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
We do not anticipate that this work should have any negative societal implications. On the contrary, improvements in time series forecasting should have primarily beneficial impacts. For example, improvements in weather forecasting, especially at higher frequencies should provide longer windows for evacuation from severe weather such as hurricanes and tornadoes. Similarly, improvements in time series forecasting should be extremely helpful in disease progression forecasting, enabling doctors and other practitioners to have better understanding of which patients need increased attention or different treatments (Jarrett et al., 2021). At the same time, over-reliance on machine learning models in quantitative finance settings (like in Section 5) could potentially lead financial shocks and decreased robustness to risk in the stock market, like was the case for several of the first usages of statistical learning in these settings (Jorion, 2000; MacKenzie and Spears, 2014).
REPRODUCIBILITY STATEMENT
Our experiments are easily reproducible with the scripts provided in our anonymized repo. The GluonTS and Autoformer benchmark datasets can be easily downloaded from each package's public code repo. The USHCN precipitation and wind speed datasets can be easily downloaded from public logs and used without data cleaning. The hourly NASDAQ-100 dataset can also be easily reproduced using public logs and used without data cleaning. The hyperparameters for our models included in the appendix.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Türkmen, and Yuyang Wang. Gluon'TS: Probabilistic and Neural Time Series Modeling Python. Journal of Machine Learning Research, 21(116):1 1, 2020. URL http://jmm wwwoooraaaeooo paapers/21/199 820 html.
Christopher M Bishop. Neural networks for pattern recognition. Oxford university press, 1995.
George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics), 17(2):11109,, 1968.
Robert Goodell Brown. Statistical forecasting for inventory control. McGraw/Hill, 1959.
Cristian Challu, Peihong Jiang, Ying Nian Wu, and Laurent Callot. Deep generative model with hierarchical latent factors for time series anomaly detection. arXiv preprint arXiv:2202.07586, 2022a.
Cristian Challu, Kin G Olivares, Boris Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting, arXiv preprint arXiv:2201. 12886, 2022b.
Victor Chernozhukov, Iván Fernández- Val, and Alfred Galichon. Quantile and probability curves without crossing. Econometrica, (8(3):1093-1122 2010.
Youngseog Chung, Willie Neiswanger, Ian Char, and Jeff Schneider. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification. Advances in Neural Information Processing Systems, 34, 2021.
Yue Cui, Jiandong Xie, and Kai Zheng. Historical inertia: neglected but powerful bascline for long sequence time-series forecasting. In Proceedings of the 30th ACM International Conference Information & Knowledge Management, pages 2965-2969, 2021.
Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke- Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. Advances in Neural Information Processing Systems, 33:2995-300 -3077 2020.
Jan G De Gooijer and Rob J Hyndman. 25 years of time series forecasting. International journal of forecasting, 22(3):443-477 2006.
Howard Diamond, Thomas Karl, Michael A Palecki, Bruce Baker, Jesse Bell, Ronald D Leeper, David R Easterling, Jay H Lawrimore, Tilden P Meyers, Michael R Helfert, et al. Us climate reference network after one decade of operations: Status and assessment. Bulletin of the American Meteorological Society, 94(4):485 4944)) 2013.
Robert F Engle. Autoregressive conditional heteroscedasticity with estimates of the variance of united kingdom inflation. Econometrica: Journal of the econometric society, pages 987-1007, 1982.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association. 102(477):359 378, 2007.
Adèle Gouttes, Kashif Rasul, Mateusz Koren, Johannes Stephan, and Tofigh Naghibi. Probabilistic time series forecasting with implicit quantile networks, arXiv preprint arXiv:2107.03443. 2021.
Daniel Jarrett, Jinsung Yoon, Ioana Bica, Zhaozhi Qian, Ari Ercole, and Mihaela van der Schaar. Clairvoyance: A pipeline toolkit for medical time series. In International Conference on Learning Representations, 2021.
Philippe Jorion. Risk management lessons from long-term capital management. European financial management, 6(3):277-300, 2000.
Rudolph Emil Kalman. A new approach linear filtering and prediction problems. 1960.

---

Under review as a conference paper at ICLR 2023
Kelvin Kan, Prançois-Xaviee Aubet, Tim Januschowski, Youngsuk Park, Konstantinos Benidis, Lars Ruthotto, and Jan Gasthaus. Multivariate quantile function forecaster, arXiv preprint 1iXxv:22022113116. 2022.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research Development Information Retrieval, pages 95-104, 2018.
Francesco Laio and Stefania Tamea. Verification tools for probabilistic forecasts of continuous hydrological variables. Hydrology and Earth System Sciences, 11(4::267-12277, 2007.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computa- tion, 4(3):448 472, 1992.
Donald MacKenzie and Taylor Spears. 'the formula that killed wall street': The gaussian copula and modelling practices in investment banking. Social Studies of Science, 4413)3393 2014.
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
James Matheson and Robert I. Winkler. Scoring rules for continuous probability distributions. Management science, 22 12)10):0887-109 1976.
MJ Menne, CN Williams Jr, and RS Vose. United states historical climatology network daily temperature, precipitation, and snow data. Carbon Dioxide Information Analysis Center, Oak Ridge National Laboratory, Oak Ridge, Tennessee, 2015.
David A Nix and Andreas S Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 ieee international conference on neural networks (ICNN'94), '94), volume 1, pages 55-60. IEEE, 1994.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv: 1905. 10437, 2019.
Youngsuk Park, Danielle Maddix, François- -avier Aubet, Kelvin Kan, Jan Gasthaus, and Yuyang Wang. Learning quantile functions without quantile crossing for distribution-fiee time series forecasting. In International Conference on Artificial Intelligence and Statistics, pages 8127-8150. PMLR, 2022.
Kashif Rasul, Abdul-Saboot Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. Mul- tivariate probabilistic time series forecasting via conditioned normalizing flows. arXiv preprint arXiv:2002.06 06 2020.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning, pages 8857-8868, PMLR, 2021.
Filipe Rodrigues and Francisco C Pereira. Beyond expectation: deep joint mean and quantile regression for spatiotemporal problems. IEEE Transactions on Neural Networks and Learning Systems, 3 31(12):5377-533 2020.
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019.
David Salinas, Michael Pohlke-Schneice Laurent Callot, Roberto Medico. and Jan Gasthaus. High- dimensional multivariate forecasting with low-rank gaussian copula processes. Advances in neural information processing systems, 32, 2019.
12

---

Under review as a conference paper at ICLR 2023
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020.
Robert H Shumway and David S Stoffer. An approach to time series smoothing and forecasting using the em algorithm. Journal of time series analysis, 3(4):253-264, 1982.
Slawek Smyl. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting, 36(1):75-2))) 2020.
Natasa Tagasovska and David Lopez-Paz. Single-model uncertaintics for deep learning. Advances in Neural Information Processing Systems, 32, 2019.
Binh Tang and David S Matteson. Probabilistic transformer for time series analysis. Advances in Neural Information Processing Systems, 14423592--23608- 2021.
Yusuke Tashrro, Jiaming Song. Yang Song. and Stefano Ermon. Csdi: Conditional score- based diffu- sion models for probabilistic time series imputation. Advances Neural Information Processing Systems, 34, 2021.
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11053, 2017.
Andrew G Wilson and Zoubin Ghahramani. Copula processes. Advances in Neural Information
Processing Systems, 23, 2010.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381, 2022.
Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 2021.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI, 2021.
13

---

Under review as a conference paper at ICLR 2023
Appendix
Table of Contents
A Limitations
14
B Experimental Details
14
Compute
B.2 GluonTS Splits
14
B.3 Data Pre-processing
15
B.4 NHiTS Hyperparameters
15
B.5 Univariate Experiment Details
15
C Further Experiments
15
Comparison with Conformal Quantile Regression
A LIMITATIONS
We see that our work, primarily empirical study, has several limitations:
Our empirical work focuses primarily on well-used benchmark datasets that could be quite distinct from other time series forecasting problems used in the literature. Thus, approaches that do not perform well here might perform better other tasks.
Our specific architecture is limited to forecasting only up to a pre-specified number of steps into the future.
Our training and evaluation procedure is quite distinct from earlier efforts for forecasting using online learning, preventing easy direct comparison with methods like ARIMA and Gaussian processes.
If the underlying time series is extremely non-stationary and the input time series is quite different from any previously observed data, we may still be unable to extrapolate.
B EXPERIMENTAL DETAILS
B.1 COMPUTE
Experiments were carried out across several computing platforms. All models were trained single GPU, Figures 1, 6, 7, were generated using single NVIDIA Titan RTX. The remaining figures were generating from experiments using single GPU larger cluster with mix of NVIDIA V100s, RTX 8000s, and A 100ss. Depending on hyperparameter setting and dataset, models take approximately 10-30 minutes train.
B.2 GLUONTS SPLITS
In Figure 13, display the training and testing splits for the six GluonTS datasets, with the dividing line shown in orange. Only on Wikipedia and Taxi are the training and testing splits especially large given the quantities of the data. Furthermore, the responses Wikipedia, solar, and electricity may not be well described as having Gaussian observation noise, as they roughly reflect counts or drop to zero at times (solar at night).
14

---

Under review as a conference paper at ICLR 2023
Exchange
Spler
Electricity
1.0
320
130
9.6
1832
2009
2012
Zerdopp
Jeati
2024022
20 407407
J0 aog
202400 J0c697
20140
WMPeeda
202e1
201264
20207
202210
202
207807
20/310
201
2000
2015-22-
201
2012
2nt
201
Figure 12: Training and test splits for GluonTS datasets. From top left down are exchange, solar, electricity, traffic, taxi, and wikipedia.
Hyperpuramettrr
hidlen dimension
(256, 512)
hidden layers
[2.3]
[01, Ma, 6a)
11
3.
8
[d1, d2,, da)
17
19
13
1)
6,
leamming rate
(10
10
weight decay
2 10
quantiles (NHITS-QR)
(0.01.0.102..33..
0..
Table 2: Hyperparameters for NHiTS on short probabilistic evaluations
B.3 DATA PRE-PROCESSING
Prior to training, all data were standardized using the Standa rdScall function from Scikit Learn. Except where specified for the datasets taken from the Gluon'SS library from Alexandrov et al. (2020), we use the same data splits as Wu et al. (2021). For the long horizon experiments in Section 4.1, we use the first 10% of the data as the validation set, the last 20% as the test set, and the remaining data as the training set.
B.4 NHITS HYPERPARAMETERS
Table 2 shows the hyperparameter space we searched for the GluonTS results in Section 4.1.
B.5 UNIVARLATE EXPERIMENT DETAILS
For the Autoformer and Transformer models, we used the public code Autoformer repo. We adapted the data loaders to work with GluonTS (Salinas al., 2019) datasets (which have history of 192 and prediction horizon of 24), and we searched over the common model hyperparameters from the repo, for both the Transformer and Autoformer architecture. For the univariate models, reduced the models' capacity (through embedding dimension) from 512 to 256 to avoid overfitting. We converted multivariate tim eseries univariate time series by sampling along the channels per batch and then flattening channels into the batch dimension. Models were trained for 20 epochs or until early stopping. We trained the NHiTS models with the hyperparameter space shown in Table 2.
C FURTHER EXPERIMENTS
C.1 COMPARISON WITH CONFORMAT. QUANTILE REGRESSION
Conformalized quantile regression (CQR) (Romano et al., 2019) is method for recalibrating quantile regression forecaster using conformal sets constructed from a validation (calibration) dataset. In Figure 13, we show a comparison of our two probabilistic forecasting methods with COR.
15

---

Under review as a conference paper at ICLR 2023
0.2
Traffic
0.2
Solar
Exchange Rate
0.06
Electricity
0.10
CRPS
0.0
0.00
0.00
336
Precipitation
0.75
Wind
NASDAQ
0.04
0.10
Model Type
0.50
NHiTS-G
0.02
0.25
0.05
NHITS QR
COR
0.00
0.00
0.00
Horizon
Horizon
Horizon
Figure 13: Comparing NHiTS-G and NHiTS- SRR (ours) with conformal quantile regression (CQR), a conformal prediction method that adjusts the uncertainty estimates using a calibration/validiie dataset. On the majority of the datasets conformal recalibration offers few benefits over methods. In the datasets the second row, CQR leads to significantly worse performance because constructing a compelling validation is challenging.
16


Reviews:
[{"review_id": "w82Pskub5G1", "reviewer": "Reviewer_5QvS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work considers long-term time series forecasting based on on deep probabilistic techniques. It shows that intrinsic stochasticity in data is often so remarkable that it poses notable challenges for the interpretation of many current deterministic methods. Importantly, the authors argue that uncertainty quantification is less important than neural network architecture design in many practical scenarios. The authors suggest a probabilistic evaluation framework that solves this challenge and improves prediction on common benchmarks and practical problems in climatology and economy.", "summary_of_the_review": "Overall this is an interesting and timely contribution of good technical quality. \n", "strength_and_weaknesses": "Strength\n\n+ intuitive representation of uncertainty through probabilistic treatment\n+ scalability of the method\n+ complements the current state of understanding in an innovative way\n+ relevance to a broad variety real application cases\n\nWeaknesses\n- Limitations and weaknesses of the method could be addressed in the manuscript in more detail as these are relevant for applications", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "odTR8owWlEB", "reviewer": "Reviewer_Cp5J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a new view of the probabilistic forecasting problem. The authors have shown some drawbacks of some current state-of-the-art probabilistic forecasting methods and propose that a simple model coupled with quantile loss or heteroscedastic Gaussian regression can often outperform more complex and expensive alternatives, particularly in the long horizon setting. The authors support their claim on multiple time series datasets from different domains and with distinct properties. They have also studied how the prediction horizon and time series dimension affect the inference time. Finally, they reach the conclusion that the architecture design of forecasting models should be prioritized over uncertainty quantification techniques.", "summary_of_the_review": "This paper is an empirical study of probabilistic forecasting models, while it lacks solid technical novelty and theoretical insights. There is not much innovation in experiments either. I think this paper will benefit from another round of revision.\n\n**Update after rebuttal**\n\nThe authors' responses have addressed some of my concerns, but I believe they are still not sufficient to significantly change my view of this paper. I would encourage authors to incorporate answers to the abovementioned questions in the next revision and discuss the underlying theoretical insights on why the methodology is performed.", "strength_and_weaknesses": "Strength:\nThe authors have conducted extensive empirical evaluations and conducted a comprehensive background review. The most inspiring part is the discussion on the impact of intrinsic stochasticity and the structure of data on forecasting models. Based on this property, the authors have divided the analysis into time series from different domains. This can serve as an interesting reference for the characteristics of time series datasets and provide insights on which model to choose.\n\nWeakness:\nThis paper is driven by pure empirical studies without theoretical insights or methodological innovation. The employed forecasting models and loss functions are all well-studied. The major work done by the authors is to combine them together and make a comparison on different datasets. Moreover, the authors don't provide a clear explanation of why the NHiTS is chosen as the base model and what is the intuition of its better performance than other complex methods. Lastly, I am also confused about how the authors reached the conclusion that architecture design should be prioritized over uncertainty quantification since the whole paper is comparing CRPS results across different methods. The authors apparently need a transition on this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jMYh1pBmZN", "reviewer": "Reviewer_RuZ4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach to provide a probabilistic forecast on top of any point forecast. Two post-processing approaches in particular are investigated, one fits a Gaussian noise by learning a condition heteroskedastic variance on top of an existing point forecast, the other fits a quantile-regression model. Experiments are conducted by applying this post-processing to an existing method (NHits) on 7 datasets against several baselines with an emphasis on long-term forecasting performance.", "summary_of_the_review": "Given the lack of novelty and lack of comparison with relevant approaches (conformal quantile prediction, likelihood directly fitted on top of previous models, [Hasson2021]), I recommend rejecting the paper in its current state.\n\nAdditional details:\n* p3: Salinas et al. (2020) does not consider only Gaussian noise model but also other likelihood (neg-binomial for instance)\n* p4: \"To train this model, we use maximum likelihood in a two-step manner, first training the mean model ... using MSE loss, and then ... maximizing the likelihood of the Gaussian observation.\" This seems to be the claimed novelty but a clear baseline would to fit both mean/variance or have a quantile regression on top of an existing architecture. This would have twice lower training runtime and latency and should probably be added in the comparisons (in addition to conformal prediction).\n* Figure 7: this figure is hard to read since you are plotting samples for some methods and confidence intervals for others. It would be much better to report calibration to allow to quantify the calibration of the probabilistic predictions as you are doing for wind/nasdaq\n* why reporting calibration only on wind/nasdaq on not on other datasets (electricity/traffic/etc)? I would recommend reporting calibration plots on all datasets, calibration errors would also be important to provide quantitative metrics\n* Figure 9: small comment calibration are typically plotted with equal axis as it is then easier to assess the quality of the fit\n", "strength_and_weaknesses": "The main strength of the paper are that it is clearly written and has clear motivation since providing accurate probabilistic forecast on top of point forecast has many relevant applications.\n\nThe main weaknesses are as follow.\n\nThe approach is very related to conformal prediction, see [Romano 2019] for instance who proposes to fit a quantile regression model that can be guaranteed to be calibrated or [Hasson2021] who already showed the benefit of the approach by demonstrating how the top point forecast of M5 could be made a top competitor to the uncertainty track with a post-processing. Sadly, those approaches are not mentioned nor compared while being directly similar to the method proposed.\n\nIn addition, the method proposed does not distinguish itself clearly from just fitting a Gaussian/Quantile regression on top of an existing architecture. Indeed, the main improvement reported in the experiments seems to come from using an architecture better suited for long term predictions (NHiTS). However, one could also fit the same model with a Gaussian likelihood or a quantile regression as the loss function of NHiTS and likely obtain similar results in term of CRPS and calibration. The main difference of the approach proposed is the two step estimation but it is not compared to the simpler approach consisting on just fitting a distribution on top of an existing model and also have drawbacks that are not mentioned (two training must be done and latency will likely be twice higher).\n\nTo clearly justify their contribution, the authors should probably show that their methods outperforms both non conformal prediction approaches and fitting Gaussian/quantile regression approaches (the first one because it is state-of-the-art and the standard approach, the second one because it is arguably simpler and faster than the proposed method by the authors). \n\n[Hasson 2021] Probabilistic Forecasting: A Level-Set Approach. Hasson et al. Neurips 2021.\n[Romano 2019] Conformalized Quantile Regression. Romano et al. Neurips 2019.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject