Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a conformal prediction method (CoRNN) for uncertainty estimation in time series forecasting. CoRNN is a recurrent network-based multi-step forecaster, it can also achieve a valid finite sample coverage guarantee. Compared with other baselines, CoRNN is computationally efficient, does not require any modifications to the predictive model, and provides theoretical coverage guarantees. Empirical results on simulated and real-world datasets demonstrate the superiority of CoRNN in constructing confidence intervals for multi-step forecasts.
main_review: This paper addresses an important problem and proposes a reasonable solution. The main content is clear and well-written. All figures are nicely designed to facilitate understanding. Besides the above advantages, I have the following comments that might be worthwhile for a further discussion:

1. There are many recent works in the conformal prediction literature, such as [1, 2] that can utilize local information to achieve adaptive interval length. CoRNN uses the basic split conformal prediction [3], which can only achieve fixed interval length (i.e., $2\cdot \hat{\varepsilon}_h$) in principle. The authors could consider borrowing ideas from more recent works to build better solutions (stronger coverage with shorter interval).
2. From my perspective, it is natural for an RNN model to use the recursive method for multi-step prediction. Despite the error accumulation, interim information can be used for future forecasts. However, I'm curious how to directly obtain forecasts in all steps at once, given the RNN can only be trained in a particular horizon.
3. The authors bypass the exchangeability assumption in split conformal prediction by treating one time series as an independent data point (instead of a timestamp). But it seems like avoiding the main challenge (temporal dependency) of conformal time series forecasting problems, as discussed in EnbPI [4] and [5]. While this is acceptable for certain applications, the authors also need to specify how to deal with variable-length data.

Minors:
1. Although this is intuitive, I'm not sure the confidence interval length at each step is strictly monotonic w.r.t. forecasting horizon.
2. In Eq (7), I wonder if the denominator should be $m+1$?
3. It is better to waive the model dependency of this method, i.e., combine with any black-box forecasting models.
4. Line 76, [6] has addressed the quantile crossing problem in training quantile RNN models for time series forecasting, the authors can briefly discuss this in related works.

References:
- [1] Romano et al, Conformalized Quantile Regression 
- [2] Guan et al, Conformal prediction with localization 
- [3] Lei et al, Distribution-Free Predictive Inference For Regression 
- [4] Xu et al, Conformal prediction interval for dynamic time-series 
- [5] Chernozhukov et al, Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data 
- [6] Han et al, Simultaneously Reconciled Quantile Forecasting of Hierarchically Related Time Series 
- [7] Gasthaus et al, Probabilistic Forecasting with Spline Quantile Function RNNs

-----
=====Update after Rebuttal=====


I carefully went over the author's response; most of my concerns have been addressed. I am willing to increase my score. If this paper is made public, please be sure to include: 
- discussion on related works [e.g., 6, 7] that addressed quantile crossing problem for time series forecasting, and how this method is better than them
- discussion on the difference between observing time stamps v.s. time-series, as well as the exchangeability assumption

Good luck!
limitations_and_societal_impact: No. The authors could briefly discuss potential problems encountered when deploying the proposed method in real applications.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper extends the conformal prediction framework, hitherto limited to models trained under the IID assumption, to the context of multi-step time series forecasting. It shows that under this framework, one can obtain reasonably well calibrated predictive intervals of the future trajectory of a time series, an important problem in decision theory.  Experiments with synthetic data and several healthcare datasets confirm the validity of the approach against baselines.
main_review: The paper is in general well written and easy to follow. The main result appears novel and sound, and builds on a little-known method in ML that would benefit from greater visibility.

Some specific questions appear in the detailed comments below. 

The theoretical results appear ok, although I did not assess the proofs in depth. 

Even though experimental results can be strengthened (see comments below), they confirm the validity of the approach.

Assuming that questions are satisfactorily answered, the paper would make a valuable contribution to the literature. I would recommend acceptance for publication.

Detailed comments (by line or section):
- 75: Even though naïve quantile regression suffers from quantile crossing, it is straightforward to prevent it in deep learning based architectures through reparametrization of the output layer.
- Eq 1: if the observations are multidimensional with dimensionality $d$, then they will be in $R^{H \times d}$, no?
- Eq 7 is a bit hard to interpret: which examples in this calculation come from the calibation set, and which come from the training set?
- Would Eq 8 need to be modified for non-symmetrical noise models (e.g. regression with a Box-Cox transformation of the dependent variable)? Could the critical nonconformity scores be computed separately for both tails of the residuals distribution?
- Figure 2b, the illustration of the RNN makes it appear that the forecast over the prediction horizon is iterated instead of direct (as stated in the paper).
- Section 4.2: why only include BJ-RNN as a baseline model for synthetic data, and not also MQ-RNN and DP-RNN? This would give a sense of their performance profile across the board.
- 223: it feels awkward that CoRNN could have access to twice as much data as BJ-RNN. Can you arrange the experiments such that both have access to the same dataset size but use it differently to compute the coverage?
- 242: coverage should be calibrated, not “higher”: if aiming for a 90% coverage, it is a disaster to get 100% (as shown in the width of the intervals for BJ-RNN). I suggest rephrasing to avoid suggesting that higher coverage can be “better” in some interpretations.
- 248: why say CoRNN “is competitive” while it Pareto-dominates the baseline?
- Section 4.3: are you also targeting $\alpha = 0.9$ ?
limitations_and_societal_impact: The manuscript only cursorily discuss the limitations of the proposed approach. It does not address the societal impacts of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work extend the idea of conformal prediction to time series setting successfully. 
main_review: Originality: This work is the first practically applicable work that conformal prediction has been applied beyond the typical case of scalar target Y . 

Quality: The work is technically solid under the time series wise exchangability assumption authors made. The methodology is simple enough, basically apply conformal prediction individually/separately on each prediction time point. 
- However, I suspect that two claims in the beginning (in line 38-46) have been attained or not as they promised. For example, how this method handle different time length, frequency and asynchronosness that existing methods fails under regression setting? It would be great the claims are clearly demonstrated. 
- What if comparing with quantile forecaster? Spline Quantile Function by Gasthaus et al http://proceedings.mlr.press/v89/gasthaus19a/gasthaus19a.pdf does not suffer the overfitting (quantile crossing) the authors worry about. 

Clarity/Questions: 
- This methodology seems not to be limited to RNN predictor. Any seq2seq forecaster should be applicable. Correct me if I am wrong. 
- One missing reference is 'Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data' https://arxiv.org/abs/1802.06300 which covers time series data
- In many time series data exchangability often heavily breaks. For example, hierarchical dataset . In that case, direct quantile regression may work but this method may fail. In addition, in this case, Chen et al [26] and Chernozhukov et al (reference above) could be more relevant. 


Significance: Even with several questions mentioned above to be resolved, this work is theoretically sound under the assumption and potential to be impactful to the practitioner, providing codes.  


limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper uses conformal learning for inductive time series forecasting that can produce accurate uncertainties. The method have theoretical guarantee but is post-hoc.



main_review: How is the computation efficiency of this post-hoc method? How can it be generalized? This method needs an pre-determined p-value and should run multiple times if multiple p-values are used? This needs multiple times of the computation?

A missing reference is [1] which presents that the conformal time series forecaster produce similar results compared with the baseline in terms of the pinball loss. This NeurIPS submission does not compare with other uq/calibration methods and please give some clarification on whether this NeurIPS submission share the same conclusion with [1]?

It can only be used in inductive setting. I am wondering whether  can use other units, such as CNN or transformer?

Does this method require an extra calibration/conformal dataset?

[1] http://oa.upm.es/57817/1/TFM_JAKUB_KOWALCZEWSKI.pdf

----POST REBUTTAL---

Author rebuttal have resolved most of my questions and therefore I increase my score.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The paper extends the conformal prediction methodology to the time series setting, with exchangeability assumed between time series. In doing so, the proposed approach can provide uncertainty estimates with coverage guarantees.

The reviewers agree that the paper tackles a relevant problem, proposes a reasonable solution, is well written and theoretically sound. While the reviewers pointed out some areas for improvements (additional related work, improvements to the empirical evaluation), the authors were able to alleviate all major concerns during the discussion period, making this a solid contribution with the potential to spark interesting follow-up work.
Decision: Accept (Poster)