Conformal Time-series Forecasting

1 Introduction . Time-series forecasting tasks are central to a broad range of application domains , including stock price predictions [ 1 , 2 ] , service demand forecasting [ 3 , 4 ] , and medical prognoses [ 5–7 ] . Recurrent neural networks ( RNNs ) and their variants ( e.g. , LSTM , GRU , etc . ) constitute an instrumental class of models that are most commonly used to carry out time-series forecasting tasks [ 8 , 9 ] . These models , however , are usually used to issue point predictions—i.e. , singular estimates of the future values of a time-series . In many high-stakes applications—such as finance and medicine—these are not enough ; estimates of uncertainty are also required for accurate risk assessment and decision-making [ 10 ] . For example , clinical practitioners need to make treatment decisions accounting for all potential scenarios , where less likely scenarios may have graver consequences and require more care compared to the more likely scenarios [ 11 , 12 ] . While various methods for uncertainty estimation in standard feed-forward neural networks have been recently proposed [ 13–15 ] , equivalent methods for RNN-based time-series models are still under-explored . Existing solutions include Bayesian recurrent neural networks [ 16–18 ] , quantile regression models [ 3 , 19 ] , latent variable models with deep state-space architectures [ 6 , 20 ] , and post-hoc uncertainty estimates using bootstrapping , jackknife or other ensembling procedures [ 21– 23 ] . Each of these solutions has its own limitations : Bayesian models may be difficult to calibrate , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . quantile predictors may “ overfit ” their uncertainty estimates , and bootstrapping methods scale poorly for RNNs with large number of parameters . Almost all existing methods share at least one of the two major drawbacks : ( 1 ) they require substantial modifications to the underlying model architecture , and ( 2 ) they provide no theoretical guarantees on frequentist coverage , any of the exceptions being computationally intractable . We aim to address the above limitations by adapting conformal prediction ( CP ) [ 24 , 25 ] —a framework used to derive prediction intervals with guaranteed finite-sample frequentist coverage—to the timeseries forecasting setup . CP has originally been designed to construct prediction intervals for scalar targets ; on the other hand , observations and predictions in time-series forecasting involve temporally dependent , potentially multivariate sequences that are not , in general , directly comparable due to differences in observation lengths , irregular frequencies , non-stationarity , and other variations in temporal dynamics ( comparison between training points being a key step in CP ) . We extend CP to a novel , computationally efficient conformal forecasting framework that can leverage any underlying point forecasting model to produce multi-step prediction intervals with coverage guarantees across the prediction horizon . We focus on RNN-based conformal forecasting architectures , which we call conformal forecasting RNNs ( CF-RNNs ) , and explore their effectiveness in providing valid and efficient coverage intervals . 2 Related Work . Most previous work in the area of uncertainty quantification in deep learning focuses on feed-forward neural network models . Much less work has been done on uncertainty estimation for time-series models . In what follows , we discuss previous methods developed for uncertainty estimation for RNNs , which we also summarise in Table 1 . Bayesian RNNs [ 16–18 ] extend the ideas of Bayesian inference to RNN models , expressing the model ( epistemic1 ) uncertainty through distributions on model parameters [ 28 , 29 ] . Exact Bayesian inference quickly becomes infeasible , however ; various approximations based on Markov chain Monte Carlo [ 30–33 ] or variational inference [ 34–37 ] are needed . Bayesian neural networks depend on significant changes in the underlying model architecture ( at least doubling the number of parameters ) , and rely on a good choice of prior ( which may be challenging in practice ) . While simplifying techniques such as Monte Carlo dropout [ 26 ] ( with RNN-specific adaptation in Gal and Ghahramani [ 38 ] ) exist , they are often difficult to calibrate [ 21 ] . Quantile RNNs can be viewed as a deep neural network extension of quantile regression [ 39 ] for sequential data : instead of returning a series of point estimates across the prediction horizon , quantile RNNs learn the prediction intervals directly , with upper and lower bounds of the forecast as separate prediction targets . The standard approach to achieve this is to use the appropriate pinball loss function as the objective . While successful applications of this approach in time-series forecasting exist [ 3 ] , naively learning individual bounds may have problems such as quantile crossing ; more recent approaches Gasthaus et al . [ 19 ] resolve this by fitting the entire quantile function . Quantile RNNs are additionally at risk of quantile overfitting due to poor sample complexity [ 21 ] . Ensembles are based on the principle of training and combining multiple models , e.g . deep ensembles trained on different random initialisations [ 40 , 41 ] , or models retrained on partial datasets ( jackknife or bootstrap resampling-based RNNs , [ 22 , 21 ] ) . Deep neural network ensembles are in general not mathematically principled for uncertainty quantification [ 42 ] ; while resampling-based 1Contrast with aleatoric uncertainty of the data [ 12 ] . models resolve this and provide post-hoc frequentist coverage guarantees , they are instead limited in their time and space complexity . For example , exact inference on the state-of-the-art blockwise jackknife RNN ( BJ-RNN ) model [ 21 ] takes O ( P 3 ) time for P parameters , and even with simplifying approximations—which in turn deteriorate accuracy—the model does not scale beyond small datasets . Conformal prediction ( CP ) For a given significance level ( error rate ) α , the goal of CP [ 24 , 25 ] is to return a prediction region Γα that is guaranteed to contain the true value with probability of at least ( 1 − α ) . In regression problems ( such as time-series forecasting ) , CP is modified to work inductively using an additional calibration set and an underlying model—an approach called inductive conformal prediction ( ICP ) [ 43 , 44 ] . Little work has been done applying ( I ) CP methods for time-series forecasting ; the main challenge is that CP assumes exchangeability , where any permutation of the dataset observations is equiprobable . However , the time-steps within a time-series are inherently non-exchangeable due to temporal dependencies ( Figure 1 , left ) ; naively applying CP to derive forecast intervals from a given time-series is therefore not methodologically valid and lacks the validity guarantees . One notable exception is the EnbPI model [ 27 ] , which bypasses the exchangeability assumption ( introducing some others ) and uses an ensemble of bootstrapped estimators to provide approximately valid intervals . However , we argue that learning from a single time-series—while useful in cases where indeed only one time-series is available—may not be optimal in settings where datasets contain multiple time-series , the shared patterns of which could potentially be exploited ( Figure 1 , right ) . To the best of our knowledge , no existing method has applied CP to the latter forecasting setting ( despite it being more methodologically grounded ) ; yet the datasets of multiple time-series are increasingly common and useful [ 45 ] . 3 Conformal forecasting RNNs ( CF-RNNs ) . In this Section , we introduce the conformal forecasting RNN ( CF-RNN ) model . We start off by formalizing the multi-horizon time-series forecasting problem in Section 3.1 , and providing the necessary background on inductive conformal prediction ( ICP ) for regression tasks in Section 3.2 . We introduce the details of the conformal forecasting procedure in Section 3.3 . 3.1 Multi-horizon time-series forecasting . Let yt : t′ = ( yt , yt+1 . . . , yt′ ) be a time-series of d-dimensional observations yt , . . . , yt′ ∈ Rd that start at time step t and end at time step t′ . A multi-horizon time-series forecast predicts future values ŷ ( t′+1 ) : ( t′+H ) = ( ŷt′+1 , . . . , ŷt′+H ) ∈ RH×d , ( 1 ) given the history of observed values y1 : t′ , where H is the number of steps to be predicted ( the prediction horizon ) . For critical applications , we are interested in the uncertainty associated with the forecast—for each time step h in the prediction horizon , we would like to obtain prediction intervals of the form [ ŷLt+h , ŷ U t+h ] , h ∈ { 1 , . . . , H } , so that the ground truth value yt+h is contained in the interval with a sufficiently high probability . We fix a desired significance level ( or error rate ) α , such that the ground-truth values of the entire time-series trajectory are contained within the intervals ; i.e. , P [ yt+h ∈ [ ŷLt+h , ŷUt+h ] , ∀h ∈ { 1 , . . . , H } ] ≥ 1− α . ( 2 ) 3.2 Inductive conformal prediction ( ICP ) . Given a set of observations D = { ( x ( i ) , y ( i ) ) } li=1 and a new example x ( l+1 ) , the ICP procedure [ 24 , 46 , 47 ] returns a prediction interval Γα such that the property of validity is satisfied : Property 1 . ( Validity ) Under the exhangeability assumption , any conformal predictor will return the prediction region Γα ( x ( i ) ) such that the probability of error y ( l+1 ) /∈ Γα ( x ( l+1 ) ) is not greater than α. Alternatively : P [ y ( l+1 ) ∈ Γα ( x ( l+1 ) ) | D ] ≥ 1− α . ( 3 ) The conformal prediction framework is distribution-free ( i.e . it does not have any assumptions on the distribution of the underlying data D ) , and applies to any underlying predictive model as long as the exchangeability assumption is satisfied : Assumption 1 . ( Exchangeability ) In a dataset of l observations { ( x ( i ) , y ( i ) ) } li=1 , any of its l ! permutations are equiprobable . Note that independent identically distributed ( iid ) observations satisfy exchangeability . The inductive2 variant of CP operates by splitting the training set into the proper training set of size n and a calibration set of size m : D = Dtrain ∪ Dcal . The proper training set is used to train the underlying ( auxiliary ) model M , and the calibration set is used to obtain the nonconformity scores , which measure how unusual is the given example compared to previously observed data . While CP guarantees validity for any nonconformity score ( including a random number generator ) , the most commonly used nonconformity score in regression is of the form Ri = A ( D , ( x ( i ) , y ( i ) ) ) = ∆ ( M ( x ( i ) |D ) , y ( i ) ) , ( 4 ) where ∆ is some distance metric . While any choice for M is valid , the best architecture depends on the dataset and the problem . When ∆ ( ŷ , y ) = |ŷ − y| , the nonconformity score Ri = |ŷ ( i ) − y ( i ) | corresponds to the residual error between the prediction of the underlying model and the true label . The resulting empirical nonconformity score distribution { Ri } li=1 is used to compute a critical nonconformity score ε̂ , which corresponds to the d ( m+ 1 ) ( 1− α ) e-th smallest residual [ 48 ] . For a new example x ( l+1 ) , the prediction interval is then : Γα ( x ( l+1 ) ) = [ ŷ ( l+1 ) − ε̂ , ŷ ( l+1 ) + ε̂ ] , ( 5 ) with ŷ ( l+1 ) =M ( x ( l+1 ) ) .

Summary: The paper extends the conformal prediction framework, hitherto limited to models trained under the IID assumption, to the context of multi-step time series forecasting. It shows that under this framework, one can obtain reasonably well calibrated predictive intervals of the future trajectory of a time series, an important problem in decision theory.  Experiments with synthetic data and several healthcare datasets confirm the validity of the approach against baselines.