Conformal Time-series Forecasting

1 Introduction . Time-series forecasting tasks are central to a broad range of application domains , including stock price predictions [ 1 , 2 ] , service demand forecasting [ 3 , 4 ] , and medical prognoses [ 5–7 ] . Recurrent neural networks ( RNNs ) and their variants ( e.g. , LSTM , GRU , etc . ) constitute an instrumental class of models that are most commonly used to carry out time-series forecasting tasks [ 8 , 9 ] . These models , however , are usually used to issue point predictions—i.e. , singular estimates of the future values of a time-series . In many high-stakes applications—such as finance and medicine—these are not enough ; estimates of uncertainty are also required for accurate risk assessment and decision-making [ 10 ] . For example , clinical practitioners need to make treatment decisions accounting for all potential scenarios , where less likely scenarios may have graver consequences and require more care compared to the more likely scenarios [ 11 , 12 ] . While various methods for uncertainty estimation in standard feed-forward neural networks have been recently proposed [ 13–15 ] , equivalent methods for RNN-based time-series models are still under-explored . Existing solutions include Bayesian recurrent neural networks [ 16–18 ] , quantile regression models [ 3 , 19 ] , latent variable models with deep state-space architectures [ 6 , 20 ] , and post-hoc uncertainty estimates using bootstrapping , jackknife or other ensembling procedures [ 21– 23 ] . Each of these solutions has its own limitations : Bayesian models may be difficult to calibrate , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . quantile predictors may “ overfit ” their uncertainty estimates , and bootstrapping methods scale poorly for RNNs with large number of parameters . Almost all existing methods share at least one of the two major drawbacks : ( 1 ) they require substantial modifications to the underlying model architecture , and ( 2 ) they provide no theoretical guarantees on frequentist coverage , any of the exceptions being computationally intractable . We aim to address the above limitations by adapting conformal prediction ( CP ) [ 24 , 25 ] —a framework used to derive prediction intervals with guaranteed finite-sample frequentist coverage—to the timeseries forecasting setup . CP has originally been designed to construct prediction intervals for scalar targets ; on the other hand , observations and predictions in time-series forecasting involve temporally dependent , potentially multivariate sequences that are not , in general , directly comparable due to differences in observation lengths , irregular frequencies , non-stationarity , and other variations in temporal dynamics ( comparison between training points being a key step in CP ) . We extend CP to a novel , computationally efficient conformal forecasting framework that can leverage any underlying point forecasting model to produce multi-step prediction intervals with coverage guarantees across the prediction horizon . We focus on RNN-based conformal forecasting architectures , which we call conformal forecasting RNNs ( CF-RNNs ) , and explore their effectiveness in providing valid and efficient coverage intervals . 2 Related Work . Most previous work in the area of uncertainty quantification in deep learning focuses on feed-forward neural network models . Much less work has been done on uncertainty estimation for time-series models . In what follows , we discuss previous methods developed for uncertainty estimation for RNNs , which we also summarise in Table 1 . Bayesian RNNs [ 16–18 ] extend the ideas of Bayesian inference to RNN models , expressing the model ( epistemic1 ) uncertainty through distributions on model parameters [ 28 , 29 ] . Exact Bayesian inference quickly becomes infeasible , however ; various approximations based on Markov chain Monte Carlo [ 30–33 ] or variational inference [ 34–37 ] are needed . Bayesian neural networks depend on significant changes in the underlying model architecture ( at least doubling the number of parameters ) , and rely on a good choice of prior ( which may be challenging in practice ) . While simplifying techniques such as Monte Carlo dropout [ 26 ] ( with RNN-specific adaptation in Gal and Ghahramani [ 38 ] ) exist , they are often difficult to calibrate [ 21 ] . Quantile RNNs can be viewed as a deep neural network extension of quantile regression [ 39 ] for sequential data : instead of returning a series of point estimates across the prediction horizon , quantile RNNs learn the prediction intervals directly , with upper and lower bounds of the forecast as separate prediction targets . The standard approach to achieve this is to use the appropriate pinball loss function as the objective . While successful applications of this approach in time-series forecasting exist [ 3 ] , naively learning individual bounds may have problems such as quantile crossing ; more recent approaches Gasthaus et al . [ 19 ] resolve this by fitting the entire quantile function . Quantile RNNs are additionally at risk of quantile overfitting due to poor sample complexity [ 21 ] . Ensembles are based on the principle of training and combining multiple models , e.g . deep ensembles trained on different random initialisations [ 40 , 41 ] , or models retrained on partial datasets ( jackknife or bootstrap resampling-based RNNs , [ 22 , 21 ] ) . Deep neural network ensembles are in general not mathematically principled for uncertainty quantification [ 42 ] ; while resampling-based 1Contrast with aleatoric uncertainty of the data [ 12 ] . models resolve this and provide post-hoc frequentist coverage guarantees , they are instead limited in their time and space complexity . For example , exact inference on the state-of-the-art blockwise jackknife RNN ( BJ-RNN ) model [ 21 ] takes O ( P 3 ) time for P parameters , and even with simplifying approximations—which in turn deteriorate accuracy—the model does not scale beyond small datasets . Conformal prediction ( CP ) For a given significance level ( error rate ) α , the goal of CP [ 24 , 25 ] is to return a prediction region Γα that is guaranteed to contain the true value with probability of at least ( 1 − α ) . In regression problems ( such as time-series forecasting ) , CP is modified to work inductively using an additional calibration set and an underlying model—an approach called inductive conformal prediction ( ICP ) [ 43 , 44 ] . Little work has been done applying ( I ) CP methods for time-series forecasting ; the main challenge is that CP assumes exchangeability , where any permutation of the dataset observations is equiprobable . However , the time-steps within a time-series are inherently non-exchangeable due to temporal dependencies ( Figure 1 , left ) ; naively applying CP to derive forecast intervals from a given time-series is therefore not methodologically valid and lacks the validity guarantees . One notable exception is the EnbPI model [ 27 ] , which bypasses the exchangeability assumption ( introducing some others ) and uses an ensemble of bootstrapped estimators to provide approximately valid intervals . However , we argue that learning from a single time-series—while useful in cases where indeed only one time-series is available—may not be optimal in settings where datasets contain multiple time-series , the shared patterns of which could potentially be exploited ( Figure 1 , right ) . To the best of our knowledge , no existing method has applied CP to the latter forecasting setting ( despite it being more methodologically grounded ) ; yet the datasets of multiple time-series are increasingly common and useful [ 45 ] . 3 Conformal forecasting RNNs ( CF-RNNs ) . In this Section , we introduce the conformal forecasting RNN ( CF-RNN ) model . We start off by formalizing the multi-horizon time-series forecasting problem in Section 3.1 , and providing the necessary background on inductive conformal prediction ( ICP ) for regression tasks in Section 3.2 . We introduce the details of the conformal forecasting procedure in Section 3.3 . 3.1 Multi-horizon time-series forecasting . Let yt : t′ = ( yt , yt+1 . . . , yt′ ) be a time-series of d-dimensional observations yt , . . . , yt′ ∈ Rd that start at time step t and end at time step t′ . A multi-horizon time-series forecast predicts future values ŷ ( t′+1 ) : ( t′+H ) = ( ŷt′+1 , . . . , ŷt′+H ) ∈ RH×d , ( 1 ) given the history of observed values y1 : t′ , where H is the number of steps to be predicted ( the prediction horizon ) . For critical applications , we are interested in the uncertainty associated with the forecast—for each time step h in the prediction horizon , we would like to obtain prediction intervals of the form [ ŷLt+h , ŷ U t+h ] , h ∈ { 1 , . . . , H } , so that the ground truth value yt+h is contained in the interval with a sufficiently high probability . We fix a desired significance level ( or error rate ) α , such that the ground-truth values of the entire time-series trajectory are contained within the intervals ; i.e. , P [ yt+h ∈ [ ŷLt+h , ŷUt+h ] , ∀h ∈ { 1 , . . . , H } ] ≥ 1− α . ( 2 ) 3.2 Inductive conformal prediction ( ICP ) . Given a set of observations D = { ( x ( i ) , y ( i ) ) } li=1 and a new example x ( l+1 ) , the ICP procedure [ 24 , 46 , 47 ] returns a prediction interval Γα such that the property of validity is satisfied : Property 1 . ( Validity ) Under the exhangeability assumption , any conformal predictor will return the prediction region Γα ( x ( i ) ) such that the probability of error y ( l+1 ) /∈ Γα ( x ( l+1 ) ) is not greater than α. Alternatively : P [ y ( l+1 ) ∈ Γα ( x ( l+1 ) ) | D ] ≥ 1− α . ( 3 ) The conformal prediction framework is distribution-free ( i.e . it does not have any assumptions on the distribution of the underlying data D ) , and applies to any underlying predictive model as long as the exchangeability assumption is satisfied : Assumption 1 . ( Exchangeability ) In a dataset of l observations { ( x ( i ) , y ( i ) ) } li=1 , any of its l ! permutations are equiprobable . Note that independent identically distributed ( iid ) observations satisfy exchangeability . The inductive2 variant of CP operates by splitting the training set into the proper training set of size n and a calibration set of size m : D = Dtrain ∪ Dcal . The proper training set is used to train the underlying ( auxiliary ) model M , and the calibration set is used to obtain the nonconformity scores , which measure how unusual is the given example compared to previously observed data . While CP guarantees validity for any nonconformity score ( including a random number generator ) , the most commonly used nonconformity score in regression is of the form Ri = A ( D , ( x ( i ) , y ( i ) ) ) = ∆ ( M ( x ( i ) |D ) , y ( i ) ) , ( 4 ) where ∆ is some distance metric . While any choice for M is valid , the best architecture depends on the dataset and the problem . When ∆ ( ŷ , y ) = |ŷ − y| , the nonconformity score Ri = |ŷ ( i ) − y ( i ) | corresponds to the residual error between the prediction of the underlying model and the true label . The resulting empirical nonconformity score distribution { Ri } li=1 is used to compute a critical nonconformity score ε̂ , which corresponds to the d ( m+ 1 ) ( 1− α ) e-th smallest residual [ 48 ] . For a new example x ( l+1 ) , the prediction interval is then : Γα ( x ( l+1 ) ) = [ ŷ ( l+1 ) − ε̂ , ŷ ( l+1 ) + ε̂ ] , ( 5 ) with ŷ ( l+1 ) =M ( x ( l+1 ) ) .

Summary: This paper proposes a conformal prediction method (CoRNN) for uncertainty estimation in time series forecasting. CoRNN is a recurrent network-based multi-step forecaster, it can also achieve a valid finite sample coverage guarantee. Compared with other baselines, CoRNN is computationally efficient, does not require any modifications to the predictive model, and provides theoretical coverage guarantees. Empirical results on simulated and real-world datasets demonstrate the superiority of CoRNN in constructing confidence intervals for multi-step forecasts.