Error-quantified Conformal Inference for Time Series

Authors: Junxi Wu, Dongjian Hu, Yajie Bao, Shu-Tao Xia, Changliang Zou
Primary area: learning on time series and dynamical systems
Keywords: Time Series; Uncertainty Quantification; Conformal Prediction; Distribution Shift

Abstract:
Uncertainty quantification in time series prediction is challenging due to the temporal dependence and distribution shift on sequential data. Conformal prediction provides a pivotal and flexible instrument for assessing the uncertainty of machine learning models through prediction sets. Recently, a series of online conformal inference methods updated thresholds of prediction sets by performing online gradient descent on a sequence of quantile loss functions. A drawback of such methods is that they only use the information of revealed non-conformity scores via miscoverage indicators but ignore error quantification, namely the distance between the non-conformity score and the current threshold. To accurately leverage the dynamic of miscoverage error, we propose Error-quantified Conformal Inference (ECI) by smoothing the quantile loss function. ECI introduces a continuous and adaptive feedback scale with the miscoverage error, rather than simple binary feedback in existing methods. We establish a long-term coverage guarantee for ECI under arbitrary dependence and distribution shift. The extensive experimental results show that ECI can achieve valid miscoverage control and output tighter prediction sets than other baselines.

Decision: accept (poster)
Reviewer scores: 8, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730660918978

thread_type:
official_review

<thread_object>
creator: reviewer_yq3j

<content>
summary:
The author propose an extension over the Adaptive Conformal Inference method of Gibbs and Candes, then used in the time-series realm by Zaffran et al. by incorporating in the adaptation procedure information about the magnitude of the error in the miscoverage rate.
After having stated the method and analysed its performance with some real-world applications.

strengths:
The proposed extension is rather interesting, seems to provide reasonable results, and puts itself in a rather interesting area of conformal prediction.

weaknesses:
- "Conformal Inference for Time Series" is an extremely broad title, which gives the impression that the authors are the first in doing something in this which is, in fact, not true.

- I found the article written in a very rough and unrefined way: I have counted at least 10 instances of missing articles (the...). Moreover, the introduction contains claims that are either too bold, or require a deeper explanation. What does it mean, for instance, that "Bayesian recurrent neural networks or deep gaussian processes are difficult to calibrate?", or, why is it meaningful to say that "quantile regression models may overfit when estimating uncertainty"

- I am seriously concerned about the "stock market" application. It seems to me that the authors aim at forecasting log stock prices, which are known to be nonstationary. In fact, the usual approach in the econometric literature is to forecast returns (i.e. log differences) which instead, tend to be a stationary time series.

questions:
- The "adaptive" CI school is only one of the possible approache to deal with the absence of indipendence between observations in Conformal Inference, in fact, https://proceedings.mlr.press/v75/chernozhukov18a.html provide a fairly interesting approach, which would be interesting to see compared to the authors proposal.

- A very recent contribution by Oliveira et al. https://jmlr.org/papers/v25/23-1553.html shows that the contribution of adaptivity procedures to be negligible. How do the authors comment about this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713641610

thread_type:
official_review

<thread_object>
creator: reviewer_fwjo

<content>
summary:
This work introduces Error-quantified Conformal Inference (ECI), a scheme for providing prediction intervals for time series data. Standard conformal inference requires exchangeability for statistical coverage guarantees to hold but this is generally violated in the context of time series data. Other methods to provide prediction intervals exist but many do not incorporate the magnitude of miscoverage when providing updates for future intervals. ECI utilizes this information for its updates. The authors derive a coverage guarantee for their methodology and  provide empirical results on synthetic and real-world datasets to support their claims. Compared to baselines that do not incorporate the magnitude of error they find that they are able to provide narrower intervals.

strengths:
- General presentation of the problem is clear and this is an approach could be of interest to an active community
- Mathematical derivations are included
- Strong empirical results on synthetic and real-world datasets

weaknesses:
- Derivation on line 266 is difficult to read because it is inline
- Minor grammatical errors and occasional typos scattered throughout the manuscript
- See questions

questions:
- Do the proofs for the validity of EQ-integral and EQ-cutoff need to be adjusted from the proof of EQI?
- For Assumption 1 should it also be "$\forall t$"
- Line 159: "$(1-\alpha) (1 + t^{-1})$" is not necessarily in $[0, 1]$. For instance if $t=3$ and $\alpha = 0.001$. How can this be a sample quantile?
- The EQI intervals visually are much more jagged. Could this ever result in dangerous overfitting?
- The significance of Figure 2 is unclear to me and the caption does not provide much detail. Why is this important and what is going on in this figure?
- Would error magnitude information be more useful for better or worse performing base models $f$ relative to other methods (ACI, OGD, etc)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718845471

thread_type:
official_review

<thread_object>
creator: reviewer_f97d

<content>
summary:
This paper proposes Error-quantified Conformal Inference (ECI), based on the ACI approach, for online conformal inference with time series data. The key idea is to augment the online gradient descent with a smoothing term that reflects the extent of the gap between the non-conformity score and the threshold at each time step, moving beyond simple binary feedback. The main results include demonstrating coverage guarantees under a fixed learning rate, given bounded conditions for the smoothing function and non-conformity scores, and presenting a reasonable bound on the coverage gap in the case of well-designed adaptive learning rates.

strengths:
- While the methods are relatively straightforward, they are explained clearly and precisely.
- The use of smoothing feedback to actively incorporate the degree of miscoverage during the update is a compelling idea, allowing the prediction sets to adapt well to, e.g., potential distributional shifts in time series data.
- Theoretical results are based on reasonable assumptions.

weaknesses:
- The authors primarily select the Sigmoid function for smoothing. However, as illustrated in Figure 2, when the extent of miscoverage exceeds a certain level, the EQ term actually decreases (which is also the case for the Gaussian error function). While the authors justify this with the goal of ensuring robustness, the rationale seems somewhat unconvincing.
- In connection with the above, if the Sigmoid function is used, the parameter $c$ can be considered a hyperparameter. Also given that Theorem 1 imposes constraints on $c$ for coverage guarantees, a more in-depth discussion on its selection, beyond the brief mention in Appendix D, would be appreciated.
- The interpretation of the miscoverage bound presented in Theorem 2 is rather ambiguous. The choice of the learning rate is a crucial aspect, yet theoretical discussion on this remains superficial.
- Similarly, the experiments lack detailed discussion on how the learning rate, which is critical for the performance of each method, was chosen. The statement on Line 330, “we select the most appropriate range of learning rates $\eta$ for the respective datasets and present the best results in the tables,” is not only vague but could potentially raise concerns about data leakage. More transparency in the tuning process is needed.
- Additionally, the choices of $h$ and $w$ in the ECI-cutoff method are not discussed.

Overall, while the proposed method is straightforward and built on a solid idea, there is room for more detailed discussion regarding the rationales, interpretations of the results, and specific hyperparameter tuning.

Minor:
- Line 119: Please use \citep for Barber et al. (2021).
- Line 238: incorrect interval order
- Line 653: non-differential -> non-differentiable
- Are "coverage"s in, e.g., Figure 3, rolling averages?

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730942682329

thread_type:
official_review

<thread_object>
creator: reviewer_p71t

<content>
summary:
This paper focuses on uncertainty quantification for time-series prediction, where exchangeability does not hold due to distribution shift and sequential dependence of data-points. Online conformal inference methods can be utilized for this task. Recent body of work on online conformal inference methods uses online gradient descent on the quantile loss function to update the thresholds of prediction sets. However, the subgradients of quantile loss function, that update the threshold at every time step, only leverage information on whether the outcome at that time-step is contained in the prediction set. The update does not include information on the magnitude of miscoverage. This paper introduces a smoothed version of quantile loss function such that the subgradient carries this information. Thus, 'Error-quantified Conformal Inference' incorporates a smooth version of feedback to update the threshold that defines prediction sets. The paper also introduces variants of smoothed ECI feedback for uncertainty quantification in time-series. Theoretical results on coverage guarantees and averaged miscoverage error are obtained without any assumptions on the data-generating distribution. Experimental results demonstrate superior performance of the ECI method in terms of reduced width of prediction sets while maintaining good coverage in comparison with other important online conformal inference baselines.

strengths:
The idea of a smoothed version of quantile loss function to generate a more informative feedback for online gradient descent is simple and intuitive. It contributes meaningfully to improve uncertainty quantification in the online setting. 

The mathematical intuition for the idea is well-presented and contextualized within the literature on online conformal inference. 

The proposed technique produces consistent improvement in the width of prediction sets while maintaining coverage guarantees in a number of time series prediction tasks.

weaknesses:
The authors mention difficulties in calibrating the outcomes of complex machine learning models like transformer, along with other models like deep Gaussian processes. However, the proposed techniques were evaluated on simpler time series models. It would be helpful to have some results with more complex models or discuss how the method would impact them. 

It would be helpful to discuss the tradeoffs between different ECI variants in the context of experimental results. 

For the place where the conformal PID baseline beats the proposed method, it would be helpful to explain the scorecaster term in a couple of lines so that it is more obvious why this baseline could produce a superior performance.

questions:
For Table 3: Did the authors try to use a similar scorecaster term within the ECI update and check if that could improve over conformal PID in the Prophet model? 

Also, is there any intuition for why decay-OGD baseline beats ECI in Table 5 (Prophet model)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732098145635

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
**Main response**

We are grateful to the engaged reviewers, who took a clear interest in the paper and suggested ways to improve it. Thank you!

We hope to respond to the critical comments through the following extensive revisions:

**1.New experiments and detailed interpretations**

We have added new experiments of the Transformer and scorecaster in Appendix F and G.1, respectively. We also revamped Appendix D.1, D.2, and G.2 for detailed discussions on interpretations of the results, and specific hyperparameter tuning ($c$, $w$, and $\eta$, respectively).

**2.Interpretations of the EQ term**

We want to clarify that the EQ term was not intentionally designed but derived based on the idea of smoothed quantile loss.  Further, there are mainly two interpretations for the EQ function’s decreasing when it exceeds a certain level.

Firstly, when $s_t$ is significantly large (namely $Y_t$ is not predicted accurately), according to our method,  the learning rates $\eta_t=\eta \cdot \max_{t-w\leq i,j \leq t}|s_i-s_j|$ also become very large at several subsequent steps. Hence, inaccurate predictions lead to higher learning rates in future steps. In this case, we seek to adjust the interval width by increasing the learning rate, rather than utilizing the EQ term to exaggerate.   

Secondly, let’s consider the time series data with the existence of outliers, whose distribution deviates very far from the distribution of normal data. Whenever an outlier appears, it will cause a large error $s_t-q_t$. Note that the distribution of data before and after an outlier remains the same, overemphasizing this outlier can lead to drastic changes in the threshold of interval (for instance, the interval becomes very wide the day after the outlier, but narrows quickly on the third day,  causing excessive oscillations). We believe that overly fluctuating confidence intervals may not be very informative in the context of uncertainty quantification.

**3.Discussion on jags**


Actually, “jags” come from quick reactions to past mistakes. This is the common property of works based on adaptive procedure in conformal inference, and similar phenomenons can be found in ACI, Conformal PID, etc (see Figure 4). If our prediction set correctly covers $Y_t$, we narrow the next prediction interval by a small size (i.e. $\alpha-EQ$). On the contrary, once there is a miscoverage point, we widen the next prediction set by a relatively large value (i.e. $1-\alpha + EQ$) and make the several following points to be covered. Therefore, whenever we widen our interval (due to miscoverage) and then narrow it in the next steps (due to coverages), there is a jag.

Besides, the reason why SF-OGD and decay-OGD seem visually to be more “smooth” is due to the fact that their learning rate is decreasing and close to zero after several iterations, causing the length of predictive intervals to be constant (we can see the jags in the early stages of SF-OGD and decay-OGD in Figure 4). However, when the learning rate is close to zero, the methods cannot react to  sudden distribution shifts.


**4.Proof of variants**

Our proof of both Theorem 1 and Theorem 2 primarily lie in quantifying the EQ term $(s_t-q_t) \nabla f(s_t-q_t)$. Considering ECI-cutoff, all inequalities in the proof still hold after we replace it with the cutoff EQ term $(s_t-q_t) \nabla f(s_t-q_t)$. The assumptions made in Theorem 1 are easier to satisfy. Hence **the proof for ECI-cutoff does not need any adjustment**. 

When it turns to the ECI-integral case, ways of measuring inequalities in the original proof must be adjusted according to specific weights. The proof will be more cumbersome. Hence, similar to ACI and Conformal PID, we only prove the ECI form that implies the main mechanism of our method.

**5.Transformer**

The results of the transformers are shown in Appendix F (Tables 9 and 10) of our revision. Consistent with the results of other base models, **ECI and its variants achieve the best performance on five benchmark datasets.**

**6. Scorecaster**

We maintained the same settings to test the performance of ECI combined with the scorecaster (Theta model), and the results (see Table11) demonstrated that **ECI+scorecaster is consistently superior over Conformal PID across three base models**, thereby validating the effectiveness of the ECI update. Interestingly, for the worse-performing base model (Prophet), adding the scorecaster enhanced the performance of ECI, while the scorecaster tended to degrade performance when better-performing base models were used. This observation is also mentioned in Section 2.3 of the Conformal PID paper: “an aggressive scorecaster with little or no signal can actually hurt by adding variance to the new score sequence”.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Error-quantified Conformal Inference (ECI), an extension of online conformal inference that incorporates the magnitude of miscoverage errors into its update procedure through a smoothed quantile loss function. By leveraging this additional feedback, ECI adapts more effectively to distributional shifts in time series data, producing narrower prediction intervals while maintaining coverage guarantees. The authors provide theoretical results supporting the validity of ECI and its variants, as well as empirical evaluation on synthetic and real-world datasets.