Conformal Time-series Forecasting
Authors: Kamilė Stankevičiūtė
Ahmed Alaa
Mihaela van der Schaar
Venue: NeurIPS 2021 Poster (2021)
Keywords: Conformal prediction
time-series forecasting
uncertainty estimation
confidence intervals

Abstract: Current approaches for multi-horizon time series forecasting using recurrent neural networks (RNNs) focus on issuing point estimates, which is insufficient for decision-making in critical application domains where an uncertainty estimate is also required. Existing approaches for uncertainty quantification in RNN-based time-series forecasts are limited as they may require significant alterations to the underlying model architecture, may be computationally complex, may be difficult to calibrate, may incur high sample complexity, and may not provide theoretical guarantees on frequentist coverage. In this paper, we extend the inductive conformal prediction framework to the time-series forecasting setup, and propose a lightweight algorithm to address all of the above limitations, providing uncertainty estimates with theoretical guarantees for any multi-horizon forecast predictor and any dataset with minimal exchangeability assumptions. We demonstrate the effectiveness of our approach by comparing it with existing benchmarks on a variety of synthetic and real-world datasets.

Conformal Time-Series Forecasting
Kamilè Stankevičiütë University of Oxford University of Cambridge ks830 cam ac uk
Ahmed M. Alaa
University of California, Los Angeles
ahmedmal aaaaaaalallllllea @uccelulle.eooo... edu
Mihaela van der Schaar
University of Cambridge
University of California, Los Angeles The Alan Turing Institute mv4 726 cam. ac
Abstract
Current approaches for (multi-horizon) time-series forecasting using recurrent neural networks (RNNs) focus on issuing point estimates, which are insufficient for informing cerision-mmaiigee in critical application domains wherein uncertainty estimates are also required. Existing methods for uncertainty quantification in RNN- based time-series forecasts are limited as they may require significant alterations to the underlying architecture, may be computationally complex, may be difficult to calibrate, may incur high sample complexity, and may not provide theoretical validity guarantees for the issued uncertainty intervals. In this work, we extend the inductive conformal prediction framework to the time-series forecasting setup, and propose lightweight uncertainty estimation procedure to address the above limitations. With minimal exchangcability assumptions, our approach provides uncertainty intervals with theoretical guarantees frequentist coverage for any multi-horizon forecast predictor and any dataset. We demonstrate the effectiveness of the conformal forecasting framework by comparing with existing baselines a variety of synthetic and real- world datasets.
Introduction
Time-series forecasting tasks central broad range of application domains, including stock price predictions [1, 2], service demand forecasting [3, 4]. and medical prognoses [5-7]. Recurrent neural networks (RNNs) and their variants (e.g.. LSTM, GRU, etc.) constitute an instrumental class of models that are most commonly used to carry out time-series forecasting tasks [8, 9]. These models, however, are usually used to issue point predictions i.e., singular estimates of the future values of a time-series. In many high-stakes applications such as finance and medicine these are not enough; estimates of uncertainty are also required for accurate risk assessment and decision- [10). For example, clinical practitioners need to make treatment decisions accounting for all potential scenarios, where less likely scenarios may have graver consequences and require more care compared to the more likely scenarios [11, 12].
While various methods for uncertainty estimation in standard feed-forward neural networks have been recently proposed [13-15], equivalent methods for RNN-based time-series models are still under-explored. Existing solutions include Bayesian recurrent neural networks [16-18], quantile regression models [3, 19], latent variable models with deep state-space architectures [6. 20], and post-hoc uncertainty estimates using bootstrapping, jackknife other ensembling procedures [21- 23]. Each of these solutions has its own limitations: Bayesian models may be difficult to calibrate,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

quantile predictors may "overfit" their uncertainty estimates, and bootstrapping methods scale poorly for RNNs with large number of parameters. Almost all existing methods share at least one of the two major drawbacks: (1) they require substantial modifications to the underlying model architecture, and (2) they provide no theoretical guarantees on frequentist coverage, any of the exceptions being computationally intractable.
We aim to address the above limitations by adapting conformal prediction (CP) [24, 25]-a framework used to derive prediction intervals with guaranteed finite-sample frequentist coverage-10 to the time- series forecasting setup. CP has originally been designed to construct prediction intervals for scalar targets: on the other hand, observations and predictions in time-series forecasting involve temporally dependent, potentially multivariate sequences that are not, in general, directly comparable due to differences in observation lengths, irregular frequencies, non-stationarity, and other variations in temporal dynamies (comparison between training points being a key step in CP). We extend CP to a novel, computationally efficient conformal forecasting framework that can leverage any underlying point forecasting model to produce multi-step prediction intervals with coverage guarantees across the prediction horizon. We focus on RNN-based conformal forecasting architectures, which we call conformal forecasting RNNs (CF-RNNs), and explore their effectiveness in providing valid and efficient coverage intervals.
2 Related Work
Most previous work in the area of uncertainty quantification in deep learning focuses on feed-forward neural network models. Much less work has been done uncertainty estimation for time-series models. In what follows, we discuss previous methods developed for uncertainty estimation for RNNs, which we also summarise in Table 1.
Table I: Overview of the most popular RNN-based probabilistic forecasting methods.
Time-series
Frequentist coverage
Method
Paradigm
Architecture
observations
Bayesian RNNs [16-18]
Bayesian Bayesian (approx.)
Built-in
Multiple
Monte Carlo drupout [26]
Built-in
Multiple
MQ- [3]. SQF-RNN [19]
Built-in
Multiple
BJ-RNN [21]
Frequentist
Post-hoc
Multiple
1 1- 2a
EnbPI [27]
Frequentist (approx.)
Ensemble
Single
1
CF-RNN (pruposed)
Frequentist
Post-hoc
Multiple
Bayesian RNNs [16-18] extend the ideas of Bayesian inference to RNN models, expressing the model (epistemic') uncertainty through distributions on model parameters [28, 29]. Exact Bayesian inference quickly becomes infeasible, however; various approximations based on Markov chain Monte Carlo [30-33] or variational inference [34-37] are needed. Bayesian neural networks depend on significant changes in the underlying model architecture (at least doubling the number of parameters), and rely on good choice of prior (which may be challenging in practice). While simplifying techniques such as Monte Carlo dropout [26] (with RNN-specific adaptation in Gal and Ghahramani [38]) exist, they are often difficult to calibrate [21].
Quantile RNNs can be viewed as a deep neural network extension of quantile regression [39] for sequential data: instead of returning series of point estimates across the prediction horizon, quantile RNNs learn the prediction intervals directly, with upper and lower bounds of the forecast as separate prediction targets. The standard approach to achieve this is to use the appropriate pinball loss function as the objective. While successful applications of this approach in time-series forecasting exist [3], naively learning individual bounds may have problems such quantile crossing; more recent approaches Gasthaus et al. [19] resolve this by fitting the entire quantile function. Quantile RNNs are additionally at risk of quantile overfitting due to poor sample complexity [21].
Ensembles are based on the principle of training and combining multiple models, e.g. deep en- sembles trained on different random initialisations [40, 41], or models retrained partial datasets (jackknife or bootstrap resampling- based RNNs, [22, 21J). Deep neural network ensembles are in general not mathematically principled for uncertainty quantification [42]; while resampling- based
CContrast with aleatoric uncertainty of the data [12].

---

independent (exchengeeble) observations
(21188)
(2r,
temporally dependent (non-exchangeable) observations
Figure Time-series observation paradigms. (Left) The dataset assumed to comprise single time- series, with observations being individual time-steps within the time-series. These observations are temporally dependent. (Right) The dataset consists of a sel of independent time-series, where the entire series is treated as an observation. Independence of time-series implies their exchangeability.
models resolve this and provide post-hoc frequentist coverage guarantees, they instead limited in their time and space complexity. For example, exact inference on the stat--ofthhe- art blockwise jackknife RNN (BJ-RNN) model [21] takes ((P)) time for P parameters, and even with simplifying approximations in turn deteriorate accuracy-thh the model does not scale beyond small datasets.
Conformal prediction (CP) For given significance level (error rate) a, the goal of CP [24. 25] to return a prediction region o that is guaranteed to contain the true value with probability of at least (1 a.. In regression problems (such as time-series forecasting), CP is modified to work inductively using an additional calibration set and an underlying model-at an approach called inductive conformal prediction (ICP) [43, 44). Little work has been done applying (I)CP methods for time-series forecasting; the main challenge is that CP assumes exchangeability where any permutation of the dataset observations is equiprobable. However, the time-steps within time-series are inherently non-exchangeable due to temporal dependencies (Figure 1, left); naively applying CP to derive forecast intervals from given time-series is therefore and lacks the validity guarantees. One notable exception is the EnbPI model [27], which bypasses the exchangeability assumption (introducing some others) and uses an ensemble of bootstrapped estimators to provide approximately valid intervals. However, we argue that learning from a single time-series wwile useful in cases where indeed only one time-series is available- mma not be optimal in settings where datasets contain multiple time-series, the shared patterns of which could potentially be exploited (Figure 1, right). To the best of our knowledge, no existing method has applied CP to the latter forecasting setting (despite it being more methodologically grounded); yet the datasets of multiple time-series are increasingly common and useful [45].
methodologically valid
Conformal forecasting RNNs (CF-RNNs)
In this Section, we introduce the conformal forecasting RNN (CF-RNN) model. We start off by formalizing the multi-horizon timeseeriss forecasting problem in Section 3.1, and providing the necessary background on inductive conformal prediction (ICP) for regression tasks in Section 3.2. We introduce the details of the conformal forecasting procedure in Section 3.3.
3.1 Multi-horizon time-series forecasting
Let your = (ye- 3t+1 Be) be time-series of d-dimensional observations .t...., Ye E IRd that start at time step and end at time step t'. A multi-horizon time-series forecast predicts future values
(0+1))++4) ($r41.... D0+H) E 1R
11 xxx

---

given the history of observed values Yi.e', where H is the number of steps to be predicted (the prediction horizon). For critical applications. we are interested in the uncertainty associated with the forecast-for each time step h in the prediction horizon, we would like to obtain prediction intervals of the form - o// M4thl: h E (1..... H. so that the ground truth value yt+hh is contained in the interval with a suffiientlyy high probability. We fix a desired significance level (or ervor rate) such that the ground-truth values of the entire time-series trajectory are contained within the intervals; i...
P (eth E We+ Rital, Vh E (1,..., H}] 2 21-a a.
3.2 Inductive conformal prediction (ICP)
Given a set of observations D - ((x(0, yy [ and a new example ((++1, the ICP procedure [24, 46, 47] returns prediction interval r such that the property of validity is satisfied:
Property 1. (Validity) Under the exhangeability assumption, any conformal predictor will return the prediction region (x) such that the probability of error ((+1) & r r (x((++)) is not greater than a. Alternatively:
Ply E (x((x())) 2-
The conformal prediction framework is distribution-free (i.e. it does not have any assumptions on the distribution of the underlying data D), and applies to any underlying predictive model as long as the exchangeability assumption is satisfied:
Assumption 1. (Exchangeability) In a dataset of observations (11))) any of its i! permutations are equiprobable. Note that independent identically distributed (iid) observations satissy exchangeability.
The inductive" variant of CP operates by splitting the training set into the proper training set of size and calibration set of size m: DD DDrriin U Daal. The proper training set is used to train the underlying (auxiliary) model M, and the calibration set is used to obtain the nonconformity scores, which measure how unusual is the given example compared to previously observed data. While CP guarantees validity for any nonconformity score (including a random number generator), the most commonly used nonconformity score regression of the form
R; = A(D, (x.)) (8)) A(M(x)))),
(4)
where A some distance metric. While any choice for M valid, the best architecture depends on the dataset and the problem. When A(j. y) 1/ y/. the nonconformity score R, = Ig(*) (*) corresponds to the residual error between the prediction of the underlying model and the true label.
The resulting empirical nonconformity score distribution [Ra) is used compute a critical nonconformity score é, which corresponds to the [(m + 1)(1 - a:) -th smallest residual [48]. For a
new example (1+1), the prediction interval is then:
(x(((+))) = [g((-1) - -E, ((+1)
- + E).
(5)
with g(++) Mx(++))
3.3 CF-RNN: ICP for multi-horizon RNNs
So far we have considered the case when the labels y E IR are scalar, but multi-horizon time-series forecasts return H (d-dimensional) values (in this work, we focus on 1; extending the results to multivariate time-series is left for future work). We extend the ICP framework to handle the
2 As opposed to the standard "transductive" setting, following the categorisation in Zeni et al. [48]. Alternative detinitions for "transductive" predictors exist; for further discussion Vovk [49].

---

(a) Construction of critical nonconformity
Forecasting time-series
prediction horizon
prediction time
1/s
prediction time
Bis
14
112
hig
3/3
time steps
Figure 2: CF-RNN uncertainty estimation procedure. (a) The calibration set is used to obtain the empirical distribution of nonconformity scores én. and its appropriate quantile is selected depending the desired target coverage level. (b) Critical nonconformity scores used oblain the prediction interval.
multi-horizon forecasting setup, while maintaining the validity of the resulting multi-horizon forecast intervals-we call this the conformal forecasting framework.
Let 'D be the set of exchangeable observations of the form (V1:T, HT+ where FırT is the time-series consisting of T observed steps, and +r+1:++ the H-step forecast. Note that the label yr+1T++ is now an H-dimensional value, in contrast with the scalar y value from before. Due to the sequential nature of the task, we will use an RNN as the underlying model M. We set M to produce multi-horizon forecasts directly (where at each time step t, all values of the H-step target Vt+1=+ are predicted the same time from a single embedding) rather than recursively (where single prediction is obtained at a time, and successive values are obtained by iteratively feeding them back into the RNN). We motivate our choice of the direct strategy by its robustness to error accumulation [50, 3]. and conditionally independent predictions given the state of M (which will be important for theoretical guarantees as discussed below). We now replace the single-dimensional nonconformity score defined earlier by its H-dimensional counterpart,
R, [1
( DEft
1itH 0itH!
where 1et1....y Di+H
(2 Since the H conditionally independent predictions are obtained from the same embedding, we apply Bonferroni correction to the critical calibration scores in order to maintain the desired eтror rate In particular, the original is divided by H, so that the critical nonconformity scores , E become the [m + 1)(1 ax/ H) -th smallest residuals in the corresponding nonconformity score distributions. The resulting set of prediction intervals is
((6)
therefore
I (1 (++1) (((t)) ...., I' ((tt))
where
IR
((+1) (1:)
With
(1+1)
(1+1) En, With + En
Wh E (1,...,,11 H!..
(8)
In summary, the conformal forecasting RNN (CF-RNN) model consists of an RNN issuing point forecasts, and conformal forecasting procedure derive the uncertainty. The entire procedure for constructing prediction intervals in CF-RNN is illustrated in Figure and summarized in Algorithm 1. Finally, we show the theoretical motivations behind our approach via the following Theorem, which provides validity for intervals obtained with the conformal forecasting procedure.
be the dataset of Theorem 1. (Conformal forecasting validity) Let D = { (vite Be4 )
(
i-1
exchangeable time-series observations and their H H-step forecasts obtained from the same underlying
5

---

probability distribution. Let M be the recurrent neural network predicting H H-step forecasts using the direct strategy. For any significance level a E [0, 1). the intervals obtained with the ICP-based conformal forecasting algorithm will have the evror rate of at most o;; alternatively,
P (Ah E (1, H). With (ūe+h En, Deth En.) 2 a.
The proof follows from conditional validity of ICP in Vovk [51] and Boole's inequality. The full statement and detailed proof is provided in Appendix A.
Algorithm Conformal forecasting RNN (CF-RNN)
1: Input: A trained model M producing H-step forecasts,
2:
calibration dataset D = { (WEi. M1-1:44)
(0)
target error rate a.
i=1"
3: Output: Critical nonconformity scores ,1,...,,1 ,,
4: Initialize E = ().... 0-
5: for 1 to m do
6: (+)
(w).))
D++1=+
7: for h = 1 to H do
8: En f EN U (1) #44 1-
9: end for end for
11: for h= to H do
12: (Bonferroni and finite sample correction)
13: En + [(m + 1)(1 a/ H)] 1-th smallest residual in en.
14: end for
return E1,-..,611.
For a new time-series example vi.:: M41444 <- (vi.).
17:
18: return intervals vē+1 + Ei,... V4 H + EH.
Experiments
In this section, we showcase the performance the conformal forecasting RNN (CF-RNN) model against three baselines: the frequentist blockwise jackknife RNN (BJ-RNN) [21], the multi-quantile RNN (MQ-RNN) [3], and the Monte Carlo tropout-based RNN (DP-RNN) [26]. We chose these baselines the most popular and representative examples of the different paradigms for uncertainty estimation (freguentist, quantile prediction and Bayesian uncertainty estimation for BJ-RNN, MQ- RNN and DP-RNN respectively). All architectures use LSTM as the underlying recurrent neural network, and are adapted to produce direct multi-horizon forecasts. We first present the performance of CF-RNNs on synthetic data with controlled properties. Since BJI-NNNs do not scale larger real datasets, we use smaller synthetic datasets to provide the comparison of BJ-RNNs with the other methods. Finally, we compare the performance of CF-RNNs with the remaining two baselines on three real-worke medical datasets. The code is available at github.coo//ccee www./amills///ooooomllll
4.1 Synthetic data
We first generate the synthetic time-series consisting of two components: the autoregressive process determining the trend of the time-series, and the noise process representing the inherent uncertainty of the dataset." For time-series of length T', this is expressed mathematically as:
Additional experiments showcasing performance on time-series with explicit seasonal (periodic) components of different frequencies are discussed Appendix B.2.

---

Σ k=0
a 24 + ce vk (T,
(10)
where (Hz,, 22). 0.9 is the memory parameter and G (0, o is the noise process. We consider five denendenn noise variance profiles, of 0.1fn and five static noise variance profiles of 0.1n, for n (1....,5.
4.2 Results
We train the models 2000 training sequences (with CF-RNNS splitting this dataset into 1000 true training and 1000 calibration sequences) for the two noise variance profiles. We aim to forecast prediction intervals for H future values yT+1:T+H H for a default coverage rate of 90% Here, 7' = 15 and H = 5. The RNN hyperparameters for the networks underlying the uncertainty estimation models
0.1).
fixed in order to ensure fair comparison, and largely follow those provided in previous work [21]. These are detailed in the Appendix B along with the time-series model parameters. Where possible," 4 repeat the experiments five times with a new randomly generated dataset, reporting the variation in empirical joint coverage over the different realisations.
Table 2: Comparison of joint coverages produced by CF-RNNs and competing baselines on autoregressive series with static or time-dependent noise profiles. Where possible, empirical joint coverages aggregated over repeated trials with randomly generated datasets.
Empirical joint coverage
Noise mode
CE-RNN
BJ-RNN
MQ-RNN
DP-RNN
92.8 + 0.8%
100%
65.0 4: 2.7%
5.4 + 0.5%
71 94.0
0.4%
100%
+: 3.4%
5.6 1.0%
Static of 0.1n
n 94.6 + 1.6%
100%
66.4 + 1.9%
5.0 + 0.9%
N
4
94.3 + 1,4%
100%
65.2 + 4.4%
4.7 + 1.0%
94.3 + 1.4%
100%
67.2 = 1.6%
4.2 = 1.0%
92.7 + 1.3%
99.4%
63.4 + 1.5%
2.5 + 1.1%
92.4 + 0.9%
100%
60.9 + 1.9%
0.4 + 0.2%
Time-depender of = 0.1tm
N 90.9 1.3%
100%
$7.2 1: 2.1%
0.2%
N =
90.6 1 1.2%
97.0%
57.1 1: 3.7%
0.0 1: 0.1%
71 91.1 + 0.7%
99.4%
58.6 + 2.1%
0.1 + 0.1%
Table 3: Prediction interval widths of CF-RNNs and competing haselines on synthetic datascts of autoregressive time-series with static or time-depenn dedendeenndt noise profiles. The mean and standard deviation are reported over all prediction horizons and random seeds.
Interval widths
Noise mode
CF-RNN
BJ-RNN
MQ-RNN
DP-RNN
16.45 3.69
98.45 25.95
25.95
9.47 + 1.99
2.82 # 0.33
Static of 0.1n
N
16.97 3.34
2.92 32.53
9.63 1.85
2.95 0.37
n =
17.12 4.550
35.82
9.72 1.92
2,77 177:0.37 0.37
n 17.34 3.77
33.83 2.49
9.71 1.80 1.80
2.87 + 0.35
16.97 +
51.23 3.21
9.84 1.99
2.85 0.38
2 =
19,80 3.61
27.09 = 1.16
11.50 1:50+10 1.66
3.01 - 0.35
25.74
104.85 5.68
15.45 1.68
3.15
Time-dependent a = 0.1
2 32.70 3.97
36.45 1.25
20.05 2.02
3.62 0.33
n 40.74 4.10
33.24 - 2.32
25.02 2.11
3.91 = 0.45
T 49.00 + 5.58
51.45 5.37
30.55 2.54
4.15 + 0.57
Tables 2 and compare the joint coverage uncertainty intervals of the models. Both CF-RNN and BJ-RNN empirically surpass the target joint coverage of 90% (a = 0.1) in both static and time- dependent noise settings, satisfying the finite-sample frequentist coverage guarantees as required.
"BJ-RNNs not retrained beyond single random seed due to limited resources.

---

HONDO
OFELA 3530
2014
Owicage
Figure 3: Trade-offs between the dataset size, joint coverage and interval widths. (Left and Middle) The relationship between the training dataset size, joint coverage rate (left) and average interval width (middle) for CF-RNN, MQ-RNN and DP-RNN baselines. (Right) The trade-off between the coverage and prediction horizon for a fixed prediction interval width in CF-RNN models with different types of underlying RNN.
Table additionally illustrates that CF-RNN intervals adapt to the properties of the temporal dynamies of the dataset: when the noise is static (and the time-series more predictable), CF-RNN prediction interval widths do not change much with increasing base variance; on the other hand, when the noise profile is time-dependent- that inherent noise of the time-series accumulates in addition to uncertainty of the model itself the average intervals get wider with increasing dataset uncertainty. The other frequentist baseline BJ-RNN has markedly wider intervals than those of CF-RNN. This might be important to maintain the perfect coverage; however, we argue that as long as the coverage rate surpasses target coverage, the intervals should be efficient (narrow) as possible to be the most informative for decision making. (Consider that infinite intervals would have perfect coverage but would not be informative.) In addition, BJ-RNNs take prohibitively long to compute (the reason for which they contain only single seed and will also be excluded from the comparisons on real data). Conversely, the ICP procedure only requires running the trained RNN model on a calibration set (where the model can be calibrated for any desired coverage simultaneously, with no additional computational cost), at which point adding uncertainty intervals to a prediction takes constant time. On the other hand, baselines following the alternative mnon-frequentist) paradigms- Baaegmss---RNNM MQ-RNN and OP-RNN-bon fail to achieve target coverage, sometimes reporting coverage rates low zero. For this reason, while the two models also come with narrower (more efficient) intervals, lack of coverage guarantees makes them less useful in high stakes real-world applications.
Experiments on the data with controlled properties provide additional insight on the trade-offs between the desired coverage rate and how far into the future can the predictions be reliably made. These trade-offs are shown in Figure 3 as applied to the datasets with time-dependent noise variance profile of 0.1t. The left and middle panels show the average performance of the CF-RNN, MQ-RNN and DP-RNN baselines depending on the training dataset size. CF-RNN the only model to achieve and maintain the required joint coverage rate with finite number of examples; additionally, with more data (larger calibration datasets), the distribution of nonconformity scores can be specified more accurately, so the width of the intervals decreases. Finally. in the panel the right fix the prediction interval width and for each horizon H compute the largest coverage level - aa maintained by CF-RNN. As shown in the Figure, low target coverage levels allow to make valid predictions far into the future, and ideal coverage levels can only be achieved with horizons near the prediction point. The overall trend is maintained for every recurrent neural network model M.
4.3 Experiments on real data
We now demonstrate the effectiveness of our procedure on real-world time-series. We train the proposed CF-RNN architecture as well as the MQ-RNN and DP-RNN baselines on three datasets summarised Table 4. For the first task, we use the data from the Medical Information Mart for Intensive Care (MIMIC-II) [52] dataset, where we forecast daily observations of white blood cell counts of varying lengths. For the second task, we use the elettroencephalography (EEG) dataset from the UCI machine learning repository [53], where EEG signals obtained from healthy subjects exposed to three types of visual stimuli. For the final task, we forecast daily COVID-19 cases within the United Kingdom local authority districts. All datasets are publicly available and the medical data is anonymised. We selected these datasets to represent variety of scenarios of real time-series: the numbers of available training instances span different orders of magnitude (from hundreds to tens of thousands). the datasets have varying observation sequence lengths, different stationarity properties (e-g. the COVID-19 time-series are synchronous-ere each time step representing the same point in time-the the others are not), different
forecast trajectories of downsampled
8

---

noise profiles (e.g. EEG signal data will inherently have higher frequencies than MIMIC-II IIIIIIIIIIIIIIIIII white blood cell count data), and different target prediction horizons. We note that COVID-19 dataset is especially challenging as the forecasts contain wave of infections and lockdowns, with the wave starting at different points for every region. Details on datasets and their preprocessing are provided in Appendix C.
Table 4: Dataset properties. The number in parentheses under the training example column indicates how many examples were used for training when a calibration set was required, such as in the case of CF-RNNs.
Dataset
Training sequences
Window length T
Prediction horizon
MIMIC-II [52]
3823 (2000)
[3,47]
EEG [53]
19200 (15360)
40
10
COVID-19 [54]
300 (200)
100
50
Table 5: Uncertainty forecasting model performance on three real-world datasets. Coverage refers to joint coverage (higher is better), and is averaged over the random splits of the dataset and training seeds. Prediction interval lengths (lower is better) are averaged over the prediction horizons and random seeds.
MIMIC-III
BEG
COVID-19
Model
Coverage
CI/PI lengths
Coverage
CI/PI lengths
Coverage
CI/PI lengths
DP-RNN
40.2 + 13.9%
3.59 + 0.90
3.3 + 0.7%
7.39 + 0.74
0.0 + 0.0%
61.18 + 32.37
MQ-RNN
89.3 + 1.2%
16.16 + 3,92
48.0 4.0%
21.39 2.36
15.0 5.9%
136.56 + 63.32
CF-RNN
94.0 1 1.2%
20.59 1 3.10
96.5 1.0%
61.86 1 18.02
89.7 5.3%
733.95 582.52
Performance of the models is summarised in Table 5. We note that the underlying LSTM model of the proposed CF-RNN architecture had the same hyperparameters as the competing baselines, yet fewer training instances (as some of the examples are used for the calibration procedure). Despite this, CF-RNN obtains the highest coverage for all datasets, and is the only model to empirically achieve the target joint coverage rates. While this seems to disproportionately affect the efficiency of CF-RNN intervals (as these the widest), the predictions are indeed reliable across the range of datasets and scenarios. On the other hand, the baseline models seem to have competitive coverage with better efficiency in some settings (e.g. MIMIC-II), yet revert to unreliable predictions in less certain scenarios (e.g. COVID-19). In other words, CF-RNN adapts its prediction interval widths to reliably match the required target coverage, increasing the width for unpredictable datasets.
Finally, we briefly explore the importance of and motivation behind the Bonferroni correction of the error rate in the CF-RNN calibration procedure. Table 6 shows that calibration scores without Bonferroni correction generally lead to poor joint coverage, even though independent coverage rates normally achieve the target coverage (most notable exception being the COVID-19 dataset selected for its forecasting difficulty).
Table 6: Bonferroni- corronccc and uncorrected empirical coverages of the CF-RNN model. Joint coverage is aggregated over the different random sceds; independent coverages present the range of observed values across all horizons and all random seeds.
MIMIC-111
EEG
COVID-19
Model
Joint
Independent
Joint
Independent
Joint
Independent
CF-RNN
94.0 1.2%
193.8%, 96.8%]
96.5 + 1.0%
[98.3%, 99.8%)
89.7 1: 5.3%
[87.5%, 100.0%
Uncorrected
89.0
1.4%
[89.0%. 91.4%]
59.4 2.4%
[85.5%, 91.6%]
55.5 + 8.0%
[77.5%, 98.8%]
5
Conclusion
In this paper, we extended the ICP framework to the multi- multi-horizon time-series forecasting problem, providing a lightweight algorithm with theoretical guarantees for frequentist coverage. Extending from the initial investigation presented in Appendix D, future work would focus on increasing the overall efficiency of prediction intervals by reducing their width, which would involve making them more adaptive to individual observations.

---

Acknowledgments
The authors would like to thank the reviewers for their helpful comments. This work was supported by AstraZeneca, the US Office of Naval Research (ONR) and the National Science Foundation (NSF, grant number 1722516).
References
[1] Sreelekshmy Selvin, R Vinayakumar, EA Gopalakrishnan, Vijay Krishna Menon, and KP Soman. Stock price prediction using Istm, rnn and con-sliding window model. 2017 International Conference on Advances in Compuling, Communications and Informatics (ICACCI), pages 1643-1647, IEEE, 2017.
[2] Wei Bao, Jun Yue, and Yulei Rao. A deep learning framework for financial time series using stacked autoencoders and long-short term memory. PloS one, 12(7):e01 80944, 2017.
[3] Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. multi- horizon quantile recurrent forecaster. arXiv preprint arXiv:/711.1105: 2017.
[4] Lingxue Zhu and Nikolay Laptev. Deep and confident prediction for time series at uber. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pages 103-110. IEEE, 2017.
[5] Bryan Lim, Ahmed Alaa, and Mihaela van der Schaar. Forecasting treatment responses over time using recurrent marginal structural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 7494-7504, 2018.
[6] Ahmed M Alaa and Mihaela van der Schaar. Attentive state-space modeling disease progres- sion. In Advances in Neural Information Processing Systems, pages 11334-11344, 2019.
[7] Benjamin Shickel, Patrick James Tighe, Azra Bihorac, and Parisa Rashidi. Deep ehr: survey of recent advances in deep learning techniques for electronic health record (ehr) analysis. IEEE journal of biomedical and health informatics, 22( 1589-1649,,, 2017.
[8] Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. Lstm neural networks for language mod- eling. In Thirteenth annual conference of the international speech communication association, 2012.
[9] Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods Natural Language Processing, pages 1700-1709, 2013.
[10] Christian Gollier. The economics of risk and uncertainty. Edward Elgar Publishing Limited, 2018.
[11] Michael W Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M Dai. Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference on Health, Inference, and Learning. 2020.
pages
[12] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? arXiv preprint arXiv: 2017.
[13] Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. simple baseline for bayesian uncertainty in deep learning. arXiv preprint 1XXX::992.002770 2019.
[14] Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pages 7047-7058, 2018.
[15] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances Neural Information Processing Systems (NeurIPS), pages 6402-6413, 2017.
10

---

[16] Meire Fortunato. Charles Blundell, and Oriol Vinyals. Bayesian recurrent neural networks. arXiv preprint arXiv:1704.002. 2017.
[17] Jen-T Ben-Tzung Chien and Yuan-Chu Ku. Bayesian recurrent neural network for language modeling. IEEE Iransactions on neural networks and learning systems, 27(2):361-374, 2015.
[18] Derrick T Mirikitani and Nikolay Nikolaev. Recursive bayesian recurrent neural networks for time-series modeling. IEEE Transactions on Neural Networks, 21(2):262 224,, 2009.
[19] Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic forecasting with spline quantile function mns. In The 22nd international conference on artificial intelligence and statistics, pages 1901-1910. PMLR, 2019.
[20] Syama Sundar Rangapuram, Matthias Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances' in neural information processing systems, 31:7785-7794, 2018.
[21] Ahmed Alaa and Mihaela van der Schaar. Frequentist uncertainty in recurrent neural networks via blockwise influence functions. In International Conference on Machine Learning, pages 175-190. PMLR, 2020.
[22] Ahmed Alaa and Mihaela van der Schaar. Discriminative jackknife: Quantifying uncertainty in deep learning via higher-order influence functions. In International Conference on Machine Learning, pages 165-174 PMLR, 2020.
[23] Chen Xu and Yao Xie. Conformal prediction interval for dynamic time-series. International Conference Machine Learning, 2021.
[24] Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic learning in random world. Springer Science & Business Media, 2005.
[25] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008
[26] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059, PMLR, 2016.
[27] Chen Xu and Yao Xie. Conformal prediction interval for dynamic time-series. In International Conference on Machine Learning, pages 11559-11569, PMLR, 2021.
[28] David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 4(3):448 472, 1992.
[29] Radford Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
[30] Radford M Neal. Bayesian learning via stochastic dynamics. Advances neural information processing systems, pages 475 1993.
[31] Radford M Neal al. Meme using hamiltonian dynamies. Handbook of markov chain monte carlo, 2)11):2, 2011.
[32] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-I1), pages 681-688, Citeseer, 2011.
[33] Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo, In International conference machine learning, pages 1683-1691, PMLR, 2014.
[34] Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.

---

[35] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[36] Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14(5), 2013.
[37] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pages 1613-1622. PMLR, 2015.
[38] Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. Advances in neural information processing systems, 29:1019-1027, 2016.
[39] Roger Koenker and Kevin F Hallock. Quantile regression. Journal of economic perspectives, 15(4)1143-156, 2001.
[40] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan, Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757, 2019.
[41] Xueheng Qiu, Le Zhang, Ye Ren, Ponnuthurai N Suganthan, and Gehan Amaratunga. En- semble deep learning for regression and time series forecasting. In 2014 IEEE symposium computational intelligence in ensemble learning (CIEL), pages 1-6. IEEE, 2014.
[42] Tim Pearce, Felix Leibfried, and Alexandra Brintrup. Uncertainty in neural networks: Approxi- mately bayesian ensembling. In International conference on artificial intelligence and statistics, pages 234-244 PMLR, 2020.
[43] Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I Jordan. Uncertainty sets for image classifiers using conformal prediction. International Conference on Learning Representutions, 2021.
[44] Harris Papadopoulos. Inductive conformal prediction: Theory and application neural net- works. In Tools in artificial intelligence Citeseer, 2008
[45] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020.
[46] Harris Papadopoulos and Haris Haralambous. Reliable prediction intervals with regression neural networks. Neural Networks, 2448):84 2011.
[47] Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In European Conference on Machine Learning, pages 345-356. Springer, 2002.
[48] Gianluca Zeni, Matteo Fontana, and Simone Vantini. Conformal prediction: unified review of theory and new challenges. arXiv preprint arXiv:2005.07972, 2020.
[49] Vladimir Vovk. Transductive conformal predictors. In IFIP International Conference on Artificial Intelligence Applications and Innovations, pages 348-360. Springer, 2013.
[50] Souhaib Ben Taieb and Amir F Atiya. bias and variance analysis for multistep-ahead time series forecasting. IEEE transactions on neural networks and learning systems, 27(1):22-76. 2015.
[51] Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference on machine learning, pages 475-490. PMLR, 2012.
[52] Alistair EW Johnson, Tom Pollard, Lu Shen, H Lehman Li-Wei Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[53] Catherine Blake. Uei repository of machine learning databases. http://www. ics. uci, edut" mlean//LReepsiiorr. html, 1998.
12

---

[54] Coronavirus (COVID-19) in the UK. https:://oornnviirss.deeee Accessed: 2021-05-25.
2021.
[55] J Durbin and SJ Koopman. Linear state space models. Time Series Analysis by State Space Methods, pages 43-75, 2012.
[56] Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in Neural Information Processing Systems, 2019.
13


Reviews:
[{"review_id": "fCvUNvFfDE5", "reviewer": "Reviewer_vdnC", "summary": "This paper uses conformal learning for inductive time series forecasting that can produce accurate uncertainties. The method have theoretical guarantee but is post-hoc.\n\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "How is the computation efficiency of this post-hoc method? How can it be generalized? This method needs an pre-determined p-value and should run multiple times if multiple p-values are used? This needs multiple times of the computation?\n\nA missing reference is [1] which presents that the conformal time series forecaster produce similar results compared with the baseline in terms of the pinball loss. This NeurIPS submission does not compare with other uq/calibration methods and please give some clarification on whether this NeurIPS submission share the same conclusion with [1]?\n\nIt can only be used in inductive setting. I am wondering whether  can use other units, such as CNN or transformer?\n\nDoes this method require an extra calibration/conformal dataset?\n\n[1] http://oa.upm.es/57817/1/TFM_JAKUB_KOWALCZEWSKI.pdf\n\n----POST REBUTTAL---\n\nAuthor rebuttal have resolved most of my questions and therefore I increase my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "erJobANOher", "reviewer": "Reviewer_Z5gg", "summary": "This work extend the idea of conformal prediction to time series setting successfully. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This work is the first practically applicable work that conformal prediction has been applied beyond the typical case of scalar target Y . \n\nQuality: The work is technically solid under the time series wise exchangability assumption authors made. The methodology is simple enough, basically apply conformal prediction individually/separately on each prediction time point. \n- However, I suspect that two claims in the beginning (in line 38-46) have been attained or not as they promised. For example, how this method handle different time length, frequency and asynchronosness that existing methods fails under regression setting? It would be great the claims are clearly demonstrated. \n- What if comparing with quantile forecaster? Spline Quantile Function by Gasthaus et al http://proceedings.mlr.press/v89/gasthaus19a/gasthaus19a.pdf does not suffer the overfitting (quantile crossing) the authors worry about. \n\nClarity/Questions: \n- This methodology seems not to be limited to RNN predictor. Any seq2seq forecaster should be applicable. Correct me if I am wrong. \n- One missing reference is 'Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data' https://arxiv.org/abs/1802.06300 which covers time series data\n- In many time series data exchangability often heavily breaks. For example, hierarchical dataset . In that case, direct quantile regression may work but this method may fail. In addition, in this case, Chen et al [26] and Chernozhukov et al (reference above) could be more relevant. \n\n\nSignificance: Even with several questions mentioned above to be resolved, this work is theoretically sound under the assumption and potential to be impactful to the practitioner, providing codes.  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "FZmGZllIOpz", "reviewer": "Reviewer_qsqN", "summary": "The paper extends the conformal prediction framework, hitherto limited to models trained under the IID assumption, to the context of multi-step time series forecasting. It shows that under this framework, one can obtain reasonably well calibrated predictive intervals of the future trajectory of a time series, an important problem in decision theory.  Experiments with synthetic data and several healthcare datasets confirm the validity of the approach against baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is in general well written and easy to follow. The main result appears novel and sound, and builds on a little-known method in ML that would benefit from greater visibility.\n\nSome specific questions appear in the detailed comments below. \n\nThe theoretical results appear ok, although I did not assess the proofs in depth. \n\nEven though experimental results can be strengthened (see comments below), they confirm the validity of the approach.\n\nAssuming that questions are satisfactorily answered, the paper would make a valuable contribution to the literature. I would recommend acceptance for publication.\n\nDetailed comments (by line or section):\n- 75: Even though naïve quantile regression suffers from quantile crossing, it is straightforward to prevent it in deep learning based architectures through reparametrization of the output layer.\n- Eq 1: if the observations are multidimensional with dimensionality $d$, then they will be in $R^{H \\times d}$, no?\n- Eq 7 is a bit hard to interpret: which examples in this calculation come from the calibation set, and which come from the training set?\n- Would Eq 8 need to be modified for non-symmetrical noise models (e.g. regression with a Box-Cox transformation of the dependent variable)? Could the critical nonconformity scores be computed separately for both tails of the residuals distribution?\n- Figure 2b, the illustration of the RNN makes it appear that the forecast over the prediction horizon is iterated instead of direct (as stated in the paper).\n- Section 4.2: why only include BJ-RNN as a baseline model for synthetic data, and not also MQ-RNN and DP-RNN? This would give a sense of their performance profile across the board.\n- 223: it feels awkward that CoRNN could have access to twice as much data as BJ-RNN. Can you arrange the experiments such that both have access to the same dataset size but use it differently to compute the coverage?\n- 242: coverage should be calibrated, not “higher”: if aiming for a 90% coverage, it is a disaster to get 100% (as shown in the width of the intervals for BJ-RNN). I suggest rephrasing to avoid suggesting that higher coverage can be “better” in some interpretations.\n- 248: why say CoRNN “is competitive” while it Pareto-dominates the baseline?\n- Section 4.3: are you also targeting $\\alpha = 0.9$ ?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The manuscript only cursorily discuss the limitations of the proposed approach. It does not address the societal impacts of the work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "14PliPxpRr-", "reviewer": "Reviewer_uEcc", "summary": "This paper proposes a conformal prediction method (CoRNN) for uncertainty estimation in time series forecasting. CoRNN is a recurrent network-based multi-step forecaster, it can also achieve a valid finite sample coverage guarantee. Compared with other baselines, CoRNN is computationally efficient, does not require any modifications to the predictive model, and provides theoretical coverage guarantees. Empirical results on simulated and real-world datasets demonstrate the superiority of CoRNN in constructing confidence intervals for multi-step forecasts.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses an important problem and proposes a reasonable solution. The main content is clear and well-written. All figures are nicely designed to facilitate understanding. Besides the above advantages, I have the following comments that might be worthwhile for a further discussion:\n\n1. There are many recent works in the conformal prediction literature, such as [1, 2] that can utilize local information to achieve adaptive interval length. CoRNN uses the basic split conformal prediction [3], which can only achieve fixed interval length (i.e., $2\\cdot \\hat{\\varepsilon}_h$) in principle. The authors could consider borrowing ideas from more recent works to build better solutions (stronger coverage with shorter interval).\n2. From my perspective, it is natural for an RNN model to use the recursive method for multi-step prediction. Despite the error accumulation, interim information can be used for future forecasts. However, I'm curious how to directly obtain forecasts in all steps at once, given the RNN can only be trained in a particular horizon.\n3. The authors bypass the exchangeability assumption in split conformal prediction by treating one time series as an independent data point (instead of a timestamp). But it seems like avoiding the main challenge (temporal dependency) of conformal time series forecasting problems, as discussed in EnbPI [4] and [5]. While this is acceptable for certain applications, the authors also need to specify how to deal with variable-length data.\n\nMinors:\n1. Although this is intuitive, I'm not sure the confidence interval length at each step is strictly monotonic w.r.t. forecasting horizon.\n2. In Eq (7), I wonder if the denominator should be $m+1$?\n3. It is better to waive the model dependency of this method, i.e., combine with any black-box forecasting models.\n4. Line 76, [6] has addressed the quantile crossing problem in training quantile RNN models for time series forecasting, the authors can briefly discuss this in related works.\n\nReferences:\n- [1] Romano et al, Conformalized Quantile Regression \n- [2] Guan et al, Conformal prediction with localization \n- [3] Lei et al, Distribution-Free Predictive Inference For Regression \n- [4] Xu et al, Conformal prediction interval for dynamic time-series \n- [5] Chernozhukov et al, Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data \n- [6] Han et al, Simultaneously Reconciled Quantile Forecasting of Hierarchically Related Time Series \n- [7] Gasthaus et al, Probabilistic Forecasting with Spline Quantile Function RNNs\n\n-----\n=====Update after Rebuttal=====\n\n\nI carefully went over the author's response; most of my concerns have been addressed. I am willing to increase my score. If this paper is made public, please be sure to include: \n- discussion on related works [e.g., 6, 7] that addressed quantile crossing problem for time series forecasting, and how this method is better than them\n- discussion on the difference between observing time stamps v.s. time-series, as well as the exchangeability assumption\n\nGood luck!", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No. The authors could briefly discuss potential problems encountered when deploying the proposed method in real applications.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper extends the conformal prediction methodology to the time series setting, with exchangeability assumed between time series. In doing so, the proposed approach can provide uncertainty estimates with coverage guarantees.

The reviewers agree that the paper tackles a relevant problem, proposes a reasonable solution, is well written and theoretically sound. While the reviewers pointed out some areas for improvements (additional related work, improvements to the empirical evaluation), the authors were able to alleviate all major concerns during the discussion period, making this a solid contribution with the potential to spark interesting follow-up work.
Decision: Accept (Poster)