Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper develops methods to construct predictive intervals on regression for cross-sectional time series data that come with valid coverage guarantees. The method developed in the paper focus on two types of validity: cross-sectional validity  where the randomness Is the randomness over time series, and longitudinal validity where the randomness is time series specific and is typically temporal dependent. While cross-sectional validity can be achieved following the standard conformal inference approach, achieving the longitudinal validity is the harder one. The paper suggests temporal quantile adjustment to achieve longitudinal validity via quantile budgeting (TQA-B) and error based adjustment (TQA-E). The authors give error bounds for coverage and also support their claim empirically.
strengths_and_weaknesses: Strengths:

1) The underlying problem studied in the paper is extremely relevant to the machine learning community.
2) The paper is supported both theoretically and experimentally showing efficacy of the methods developed.
3) Overall the paper is well written.
4) The ideas developed in the paper are novel.



questions: I do not have additional questions.
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper considers the task of predictive inference with panel data consisting of a time dimension and sample dimension. Two types of coverage guarantees are considered here--- longitudinal coverage and cross-section coverage. In contrast to previous works that focus on either of the coverage guarantees, this paper aims at achieving both (at least empirically). The authors leverage the conformal inference technique, adapting it to the temporal structure. Under certain conditions, the proposed method reserves the cross-section coverage ensured by the conformal inference method; by incorporating the temporal structure, the method(s) achieves better longitudinal coverage empirically.

strengths_and_weaknesses: Strength:
1. The paper is very well-written: there is a clear motivation, and a thorough introduction to the related works and their limitations; the proposed method is presented with explanations and intuitions. I enjoyed reading this paper.

2. The research question raised is interesting and can be relevant to practitioners.

3. The proposed method(s) achieves satisfactory empirical performance compared with competitors.

Weaknesses/room for improvement

1. The theoretical guarantee of the proposed method is a bit weak. Even for cross-section coverage, the guarantee (Theorem 4.1) holds under a fairly strong condition (as in the examples given in the remark after Theorem 4.1), although this is mitigated by the lower bound given in Theorem 4.3. There is no guarantee for longitudinal coverage (admittedly, this is impossible w/o further assumptions, but it might be worth understanding it under mild conditions).

2. It might be helpful to present simulation results. With synthetic data, it is possible to directly evaluate the validity of the proposed method and compare it with other competitors.
questions: 1. Line 35: is there a comma missing between $X_{i,t}$ and $Y_{i,t}$

2. Line 36: I assume that the prediction interval also depends on 
$X_{N+1, t+1}$ right? Should it be "Given data $[Z_{N+1,j}]^t_{j=1}$ and $X_{N+1, t+1}$"? 

3. Line 124:  should the "..." after $Z_{i,T}$ be removed?

4. Line 132: using both the terms "training data" and "training set"
could be confusing.

5. Line 208: is there any reason why $\beta$ is taken to be $0.8$?

6. Line 214: the notation $f(S_{i,:t};[S_{j,:t}]^{N+1}_{j=1})$
is confusing.

7. Line 222: is the quantity $\alpha^{\rm TQA-B}_{N+1,t}$ formally defined?

8. line 293: what does it mean to repeat the prediction 50 times? Is the randomness from sample splitting? If so, how are the time series selected?

9. Fig3: what is the coverage in the y-axis?

10. Appendix A1: $\hat{r}_{N+1,t} = \lceil (1- \alpha + \hat{\delta}_{N+1,t})(N+1) \rceil$?





limitations: The authors have thoroughly addressed the limitations of their work and potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Temporal Quantile Adjustment (TQA) is proposed for time-series data with a focus on constructing efficient and valid prediction intervals (PIs) for regressing the models with two distinct notions of coverage: cross-sectional coverage and longitudinal/temporal coverage. TQA adjusts the quantile to query in Conformal Prediction at each time t, accounting for both cross-sectional and longitudinal coverage in a theoretically-grounded manner. The theory part supports the validity of coverages. Two variants are also provided for generality of the TQA, following the nature of quantile adjustment.
strengths_and_weaknesses: Strength:
- Intends to resolve a critical challenge in conformal prediction, that is to construct efficient valid predictions from both cross-sectional and longitudinal perspectives.
- Clear writing and contribution to the Conformal Prediction problem
- Solid experiment results supporting the methodology in Appendix, Fig 8,9

Weakness:
- lack of guarantee or operational guidance for improving the coverages
questions: It is probably out of the scope of this paper, but would there be a method to deliver a target coverage when possible?
limitations:
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new approach called temporal quantile adjustments (TQA) that can improve longitudinal coverage and preserve cross-sectional coverage for the prediction interval built for regression on cross-sectional time series data.

The definitions of cross-sectional validity and longitudinal validity of a prediction interval are first presented in Eq (1) and (2). The existing split conformal method is reviewed and it can generate a cross-sectionally valid prediction interval. However, it cannot be directly applied to achieve  longitudinal validity as the exchangeability assumption is not satisfied.

The proposed TQA technique is introduced in section 4. Theorem 4.1 proved the coverage probability is lowered bounded by 1 - \alpha + E[\hat{\delta}_{N+1,t}]. Motivated by Theorem 4.1, TQA-B (quantile budgeting) is introduced in section 4.2 to make the E[\hat{\delta}_{N+1,t}] close to zero. TQA-B consists of 1) quantile prediction, and 2) budgeting. The error-based adjustment (TQA-E) is introduced to increase \delta_{t+1} if there are many errors in the past. 

Experimental results show that while existing conformal baseline approaches can achieve cross-sectional validity. TQA-E can achieve longitudinal validity besides cross-sectional validity.
strengths_and_weaknesses: Strengths
-----------
1) the major contribution of this work, i.e., improving longitudinal coverage and preserve cross-sectional coverage for the prediction interval built for regression on cross-sectional time series data, seems novel.
2) the proposed approach has solid theoretical guarantees.
3) experiment results support the major claim in the paper.

Weaknesses
---------------
I cannot find major weakness
questions: 1) In the definition of longitudinal validity (Eq. 2), the distribution on the RHS is Y_{N+1, t} | S_{N+1,:t-1}, shouldn't it be S_{N+1, t} | S_{N+1,:t-1} instead? Is S_{N+1, t} a random variable in this case?

2) can the author explain why the two-step approach in TQA-B can make E[\hat{\delta}] close to 0?

limitations: I didn't see major potential negative societal impact of this work.
review_title: null

Meta-review: metareview: In this paper, the authors propose the temporal quantile adjustments (TQA) that can improve longitudinal coverage and preserve cross-sectional coverage for the prediction interval built for regression on cross-sectional time series data. While previous works focus on either of the coverage guarantees, a major contribution of this paper is to achieve both. The research questions addressed in this paper are of critical importance for practitioners in relevant areas. The paper is well written. The presented approach has solid empirical support and decent theoretical guarantees. Including a simulation study to demonstrate the validity of the proposed approach under specific (and controlled) setup will further improve the paper.


confidence: Certain
Decision: Accept