Kernel-based Optimally Weighted Conformal Time-Series Prediction

Authors: Jonghyeok Lee, Chen Xu, Yao Xie
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Conformal prediction, Nonparametric kernel regression, Time series

Abstract:
Conformal prediction has been a popular distribution-free framework for uncertainty quantification. In this work, we present a novel conformal prediction method for time-series, which we call Kernel-based Optimally Weighted Conformal Prediction Intervals ($\texttt{KOWCPI}$). Specifically, $\texttt{KOWCPI}$ adapts the classic Reweighted Nadaraya-Watson (RNW) estimator for quantile regression on dependent data and learns optimal data-adaptive weights. Theoretically, we tackle the challenge of establishing a conditional coverage guarantee for non-exchangeable data under strong mixing conditions on the non-conformity scores. We demonstrate the superior performance of $\texttt{KOWCPI}$ on real time-series against state-of-the-art methods, where $\texttt{KOWCPI}$ achieves narrower confidence intervals without losing coverage.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729394695793

thread_type:
official_review

<thread_object>
creator: reviewer_wmcc

<content>
summary:
The paper presents a method to construct prediction intervals for time-series data, its main contribution being using RNW estimators for quantile regression. The paper builds upon a large body of work on the topic and offers a new algorithm under mild assumptions.

strengths:
- The paper is very well-written and easy to follow. 
- The algorithm that leverages the RNW estimator is innovative. 
- Theory section (as far as I have understood) is correct and well-argued.
- The experiment comparison is comprehensive (includes almost all the baselines in the space. thanks for doing all this work).

weaknesses:
**1. Differentiation from SPCI.**

One concern I have is that this approach feels very similar to SPCI both in terms of algorithm (using an estimator to model residuals) and guarantee / proof technique (relies on the convergence property of the estimator). In terms of experiment results, KOWCPI also does not significantly outperform SPCI (in figure 2(c), there is a significant chunk of time where SPCI is more valid). 

Can you add some more explanation on how your method is different from SPCI, in terms of theoretical guarantees, assumptions on data, and performance? It will not only improve clarity, but also help readers in practice know which method to choose for their specific applications. 

**2. Experiment results & analysis**

I would strongly encourage the authors to add more details to their experiment results, including: 

- Add standard deviations to table 1. 
- Explain the results in table 1. What are the rationale of highlighting the first row? You should also highlight other methods that achieve the same coverage, and explain  why lower width was not highlighted. 
- Add experiments for one or more values of $\alpha$ to demonstrate consistent performance.
- Provide some qualitative examples to show why/how the KOWCPI confidence bands are better. 
- Elaborate on figure 2(d), perhaps adding a further study/analysis on weights. The remark in line 494-496 is not very convincing when only one example is provided. 

If these concerns are addressed, I'm happy to increase my score.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729763013938

thread_type:
official_review

<thread_object>
creator: reviewer_75ps

<content>
summary:
This paper proposes a conformal prediction method for time-series data using the Reweighted Nadaraya-Watson estimator. The authors provide non-asymptotic marginal coverage gap in Proposition 4.1 and conditional coverage guarantees under strong mixing conditions in Theorem 4.9. Its efficiency is demonstrated through experiments on three real-world datasets.

strengths:
1. The paper is well-written and easy-to-follow.
2. The considered problem is important and well-motivated.
3. The proposed KOWCPI is efficient in various real datasets.

weaknesses:
1. Regarding the experiments, the author only provides real datasets to demonstrate the efficiency of marginal and conditional coverage without offering information about dependency. However, I believe it would be beneficial to include additional synthetic experiments to explore different dependency structures and more comprehensively assess the scope and validity of the methods.  

2. While a theoretical guarantee is provided, the bound heavily depends on the estimation of weights and the strong mixing assumptions.

questions:
1. Regarding the theoretical results, one of my concerns is could the author provide theoretical guarantees for other types of dependent time-series beyond strong mixing assumption or explain the difficulties under other dependent conditions?  

2. Related works and experiments: 
- Could the author provide more details about the benchmarks and related works to help readers unfamiliar with online conformal prediction understand why KOWCPI outperforms other methods? For example, providing a brief summary of the key differences between KOWCPI and each baseline method, highlighting why these differences might lead to improved performance may be helpful.  You can also include a table to compare the main features of each method if possible.
- Besides, the comparison methods in experiments seem to be incomplete. For example, the conformal PID in [1], the BCI in [2] and the decay-OGD in [3] should be considered. 

4. Optimal choice of window length $\omega$: 
 - Is it possible to design some strategy to determine optimal $\omega$ values that adapt to real-time changes in data volatility or distribution patterns, thereby further enhancing the KOWCPI? For example, using change point detection algorithms or online learning techniques to dynamically adjust the window length.

References:

[1] Angelopoulos A, Candes E, Tibshirani R J. Conformal PID control for time series prediction[J]. NeurIPS, 2023, 36.

[2] Yang Z, Candès E, Lei L. Bellman Conformal Inference: Calibrating Prediction Intervals For Time Series[J]. arXiv, 2024.

[3] Angelopoulos A, Barber R F, Bates S. Online conformal prediction with decaying step sizes. ICML, 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710140602

thread_type:
official_review

<thread_object>
creator: reviewer_lfjw

<content>
summary:
This article studies how to use the conformal prediction framework to establish prediction intervals for time series data.
The key point of the method is: to estimate the conditional distribution of future residuals using the Reweighted Nadaraya-Watson (RNW) method. Then establish the prediction interval for the future response with mean function and estimated distribution of residual. The experiments indicate that the method produces shorter and more efficient prediction intervals while meeting coverage requirements.

strengths:
- This article applies the conformal prediction and the RNW method to establish a prediction interval. Therefore, the proposed method is independent of specific time series models and has a broader range of applications.

- Instead of establishing prediction intervals using the quantiles $Q_{\alpha/2}$ and $Q_{1-\alpha/2}$. The method seeks a $\beta$ such that the distance between the quantiles $Q_{\alpha/2+\beta}$ and $Q_{1-\alpha/2+\beta}$ is minimized to form the prediction intervals. This approach is more effective for certain residual distributions.

- The authors consider time-series structures. In conditional coverage properties, the authors loosen the stationary condition only on strongly mixing residuals.

- The experiments indicate that the method produces shorter and more efficient prediction intervals while meeting coverage requirements.

weaknesses:
- The authors make a natural integration of the conformal prediction with the RNW method. However, the innovation of this approach is not particularly strong. The theoretical results primarily focus on the asymptotic accuracy of distribution estimation.

- When the dependence in the time series is strong, the dimension of $\tilde{X}$ may be large. In this case, when the data is limited, kernel estimates may not yield reliable results. Also, large $w$ leads to a low convergence rate of coverage. So it is better to include experimental results showing how coverage and interval width as $w$ increases for different sample sizes and datasets with varying degrees of temporal dependence.

-  The method establishes prediction intervals for stationary time series. Can the method be extended to handle local stationarity? Have authors considered techniques like differencing or detrending to handle non-stationarity? Additionally, please provide a more explicit discussion of the limitations of the proposed method for non-stationary data.

questions:
-  $T$ represents the number of past residuals used to give the estimation of residual at the next time point. Can the authors give particular analyses related to $T$, such as a sensitivity analysis showing how prediction interval performance changes with different values of $T$, or a discussion of how $T$ was chosen for each dataset? 

- The method involves two important parameters: the bandwidth $h$ and the window length $w$. While the paper discusses the settings for these parameters, it is unclear whether the experimental results are sensitive to them. Additional experimental results may be needed to clarify this.

- When the dimension of $\tilde{X}$ is large, the results from kernel estimation may not be robust. In this case, is it possible to train a function to compress $\tilde{X}$, aiming to extract effective information and reduce dimensionality for $\tilde{X}$?

- In classic time series models like the ARMA-GARCH model, can the choice of $w$ or $h$ be discussed more precisely in theoretical parts as a corollary?

- Can the authors provide some experimental results on non-stationary time series?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715381626

thread_type:
official_review

<thread_object>
creator: reviewer_e4wm

<content>
summary:
The authors introduce KOWCPI to address the difficulties that arise from employing conformal prediction in the time series setting. In order for standard conformal prediction procedures to have valid statistical coverage guarantees the data needs to be exchangeable. However, this requirement is violated in time series data. KOWCPI uses the Reweighted Nadaraya-Watson estimator for quantile regression in the presence of dependent data to learn the distribution of residuals. Using this information, intervals can be constructed that have been theoretically shown to provide conditional coverage and empirically have been found to be narrower than other baselines while maintaining desired coverage.

strengths:
- Authors provide a conditional coverage guarantee (much more meaningful than marginal coverage guarantee)
- Proofs are included and assumptions are clearly stated
- Strong coverage and efficiency results on intervals

weaknesses:
- Grammatical errors/typos scattered throughout
- Notation is rather dense. More plain language accompanying the mathematical formalism would help the reader
- Presentation of Algorithm 1 seems to be missing some pieces. In the required line should it be $(X_t, Y_t), t=1, 2, ..., T$? Variables $n, w$ are referenced but not explained earlier in the box. 
- Table 1 is a little bit confusing at first glance. Typically bold indicates "best" but in this case it only shows what the authors' method is
- See questions

questions:
- Is quantile crossing ever an issue in $\hat Q_\beta$?
- The form of the intervals is $\hat C^\alpha_{t-1} = [\hat f(X_T) + \hat Q_{\beta^*}(\tilde X_{n+1}) , \hat f(x) + \hat Q_{1 - \alpha + \beta^*}(\tilde X_{n+1})]$. Are there cases where both quantile estimates are positive/negative and possibly leading to invalid intervals?
- Are there ever discontinuities over time? (ie the quantile estimators dramatically jump around)
- Are there any diagnostics that can be used to assess the plausibility of the assumptions? 
- How robust is this method to violation of assumptions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732401853688

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common response to all reviewers

comment:
Dear Reviewers,

We sincerely appreciate the time and effort you dedicated to providing constructive feedback, as well as your recognition of our paper’s strengths. In the common response, we aim to address feedback points common among reviewers and highlight key aspects of our approach that we believe are important to emphasize for all reviewers. Full responses to each specific review are provided in the individual comments.

**Point 1. Strong mixing assumptions on residuals**

We first emphasize that we do **NOT** assume stationarity or strong-mixing of the original time series data $(X_t, Y_t)$. Instead, our theoretical results on conditional coverage rely on the stationarity and strong-mixing assumptions of the non-conformity scores (residuals). As clarified in Lines 320-323, this requirement is far less restrictive than assuming stationarity or strong mixing of the original series. It can be satisfied even when the data is clearly non-stationary or lacks mixing, e.g., a vector AR model with a time-dependent drift.

We also note that the Wind dataset used in our experiments is proposed to be non-stationary by the ADF test. As elaborated below on Point 3, we have also conducted additional experiments to further evaluate KOWCPI’s performance under non-stationary conditions.

**Point 2. Hyperparameter tuning**

Our theoretical results hold for any choice of window length (dimension of $\tilde X$) $w$, with a corresponding choice of bandwidth $h$ based on Assumption 4.5. Theoretically, optimal bandwidth selection is studied well in the literature of nonparametric estimation, and we utilized such theoretical choices of optimal bandwidth for constructing our theoretical results.

However, in practice, $w$ and $h$ become hyperparameters that require tuning. Bandwidth selection is critical for performance, and we have employed nonparametric AIC, a commonly used method in the literature for tuning the bandwidth of linear smoothers ([1]). In response to shared concerns about tuning $w$, we have suggested the implementation of an adaptive selection process for $w$, using a two-sample Kolmogorov-Smirnov test, motivated by the intention to exploit the distribution of residuals. Rather than relying on tuning via cross-validation to fix $w$ in prior, this can ensure $w$ to be chosen in a data-driven manner, with a minimal loss in performance in practice. Please refer to the newly added Appendix E in the revision for a more detailed discussion.

**Point 3. Additional experiments**

Furthermore, to address your shared recommendations on experiments, we conducted additional experiments, including:
- (Real data) Apple’s daily closing stock prices,
- (Synthetic data 1) A heteroskedastic mixture model of AR(1) + GARCH(1,1) given as
$$
	Y_t = 0.8Y_{t-1} + \sigma_t \epsilon_t + \xi_t, \quad \sigma_t^2 = 0.1 + 0.3Y_{t-1}^2 + 0.6 \sigma_{t-1}^2, \quad \epsilon_t \stackrel{iid} \sim N(0,1), \quad \xi_t \stackrel{iid} \sim N(0, 0.1^2),
$$
- (Synthetic data 2) A nonstationary time series with a seasonal trend given as
$$
	Y_t = \log(t’) \sin \left(\frac{2 \pi t’}{12} \right) (|\beta^T X_t| + |\beta^T X_t|^2 + |\beta^T X_t|^3)^{1/4} + \epsilon_t, \quad \epsilon_t \sim AR(1).
$$
We consistently observe the coverage validity of KOWCPI on these diverse datasets, and KOWCPI yields the shortest intervals on average. For further details, please refer to Tables A.1–A.3 and Appendices C and D, newly added to the rebuttal revision.

Additional experiments in individual responses include:
- Selection of $T$ and its impact on performance.
- Adaptive selection of $w$ using the two-sample Kolmogorov-Smirnov test.
- Real data experiments with different target coverage levels.

**Changes in rebuttal revision**

In the rebuttal revision, we have made the following changes to address reviewer feedback:
- (Lines 474-485 on experiment results) Added qualitative reasoning for superior performance of KOWCPI pertaining to the reweighting.
- Added detailed descriptions of the additional experiments in Appendices C-D.
- Added discussion and experiments about adaptive window selection in Appendix E.
- Removed misleading highlights in Table 1 and Figure 2.
- Added standard deviations for coverage and width terms in the experimental results, including Table 1.
- Revised Algorithm 1 for improved clarity.
- Included references to the suggested works ([2], [3]).
- Added plain text explanation for some notations.
- Corrected typos and grammar.

Changes in the revision are highlighted in blue.

[1] Cai, Z., Regression quantiles for time series. *Econometric Theory*, 18(1):169–192, 2002.

[2] Yang, Z., Candès, E., and Lei, L., Bellman conformal inference: Calibrating prediction intervals for time series. *arXiv preprint arXiv*:2402.05203, 2024.

[3] Angelopoulos, A. N., Barber, R. F., and Bates, S., Online conformal prediction with decaying step sizes. In *Proceedings of the 41st International Conference on Machine Learning*, 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work introduces a novel conformal prediction method for time-series, called Kernel-based Optimally Weighted Conformal Prediction Intervals (KOWCPI). KOWCPI adapts the Reweighted Nadaraya-Watson (RNW) estimator for quantile regression on dependent data and learns optimal data-adaptive weights. The paper addresses the challenge of establishing a conditional coverage guarantee for non-exchangeable data under strong mixing conditions on non-conformity scores. Experimental results show that KOWCPI outperforms state-of-the-art methods on real time-series, achieving narrower confidence intervals without compromising coverage. After the author response and author-reviewer discussions, this paper has received unanimous support from the reviewers. Therefore, I recommend acceptance.