Title: Sample-Conditional Coverage in Split-Conformal Prediction
Venue: NeurIPS 2025
Accepted: yes
Keywords: Conformal prediction

predictive inference

coverage
Abstract: We revisit the problem of constructing predictive confidence sets for which we wish to obtain some type of conditional validity. We provide new arguments showing how ``split conformal'' methods achieve near desired coverage levels with high probability, a guarantee conditional on the validation data rather than marginal over it. In addition, we directly consider (approximate) conditional coverage, where, e.g., conditional on a covariate $X$ belonging to some group of interest, we seek a guarantee that a predictive set covers the true outcome $Y$. We show that the natural method of performing quantile regression on a held-out (validation) dataset yields minimax optimal guarantees of coverage in these cases. Complementing these positive results, we also provide experimental evidence highlighting work that remains to develop computationally efficient valid predictive inference methods.


Peer reviews:
comments: SUMMARY:
This paper presents new theoretical results demonstrating that split conformal methods can achieve near-nominal coverage with high probability.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper offers a fresh perspective on sample-conditional coverage for split conformal prediction methods, supported by strong theoretical guarantees and clear exposition.

Weaknesses:
- The experimental results are not sufficiently intuitive or easy to interpret.

QUESTIONS:
1. In line 245, please clarify the statement: "the static thresholded sets have more variability in coverage on the random slices of the dataset." A more intuitive explanation of the experiments results, especially in relation to the theoretical results, would be very helpful for understanding how the theory connects to the experimental observations.

2. Aside from group-conditional coverage, are there other types of conditional coverage that split conformal prediction can guarantee?

LIMITATIONS:
Yes, limitations are discussed in the last section of paper.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits the problem of constructing predictive confidence sets, aiming to achieve near desired coverage levels with high probability, a guarantee conditional on the validation data rather than marginal over it. The authors show that the natural method of performing quantile regression on a held-out dataset yields minimax optimal guarantees of coverage. In addition, the authors specifically consider coverage conditional on a covariate $X$ belonging to some group of interest. Numerical experiments are provided that are consistent with the positive theoretical results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**: \nThis work investigates the theoretical analysis of conditional coverage properties within the framework of split conformal prediction, and the authors demonstrate rigorous results and establish tighter bounds with solid theoretical proofs. \n\n**Weaknesses**: \n1. The authors presented lots of theoretical results as well as prior findings in related works, though the presentation might be a little confusing and could be more focused and clearer to highlight the key contributions better. \n2. The authors focus on the sample-conditional coverage. A more thorough discussion of the underlying motivations could potentially enhance the applicability and practical value of this work. \n3. This work appears to focus primarily on theoretical contributions (tighter bounds), while it doesn't include a new algorithm or approach. \n4. The authors might consider providing more comprehensive elaboration on certain essential technical concepts. For example, the key results concerning minimax rate optimality may require additional explication for readers less familiar with this specialized framework.\n\nQUESTIONS:\n1. I would appreciate it if the authors could further clarify the motivations and key contributions of this work, as this would help readers better appreciate these rigorous theoretical results in this paper.\n2. I have some reservations regarding the experimental results. Could the authors clarify whether the demonstrated performance reflects sample-conditional coverage properties? From my perspective, these results appear to demonstrate group-conditional coverage properties rather than the claimed sample-conditional coverage. \n\nOther minor comments: \n● Typo: double appears in Line 136\n● Typo: $\\mathbb{S}^{d-1}$might be $\\mathbb{B}^{d-1}$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies Conformal Prediction under sample-conditional coverage, where the coverage holds with high probability conditional on the training and calibration data. The authors show that split conformal prediciton, including adaptive thresholds via quantile regression, can achieve approximate sample-conditional coverage. Theoretically results are established under standard i.i.d. assumptions, with minimax-optimal bounds in certain settings. The paper also includes a small empirical study to illustrate conditional coverage performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The theoretical results are solid and well-justified. Although i did not read all the proofs, they look good to me and the assumptions are reasonable.\n- Conditional coverage is a meaningful and realistic goal to study, and the motivation behind focusing on it in this work is well explained.\n- The presentation is mostly clear, and the theoretical contribution can be followed for readers familiar with the area.\n\nWeaknesses:\n- I don't quite get the idea of the empirical section. It is not connected to the theory. The setup is limited and does not clearly demonstrate the practical advantages of the proposed guarantees.\n- While the viewpoint of this paper is well motivated, the method is based on existing works and conformal tools, and the novelty lies mainly in pure analysis rather than methodology.\n- The paper may be hard to appreciate for readers unfamiliar with the conformal literature.\n\nTypos:\n- Line 136: duplicate \"appears\".\n- Line 250: typo \"slighlty\"\n- Line 261: typo \"downard\"\n- Line 398: typo \"shsared\"\n- The format of the reference part is inconsistent.\n\nQUESTIONS:\n- The theoretical guarantees for the threshold function depend on the VC dimension of the function class. In practice, if the model class is complex and the calibration sample is small, do the guarantees still hold meaningfully?\n\n- The theory assumes i.i.d. data. In practical applications (e.g., covariate shift or posterior drift), how robust is the proposed method to deviations from this assumptions? Could the sample-conditional perspective extend to covariate shift scenarios?\n\n- What does the empirical section actually validate? I don't see how this is related to the theoretical analysis.\n\nLIMITATIONS:\nNo. Please refer to the weaknesses and questions above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}

comments: SUMMARY:
This paper analyzes split conformal methods with confidence sets using adaptive thresholds of the form $\hat{C}(x) = \\{y | s(x, y) \leq \hat{h}(x)\\}$, where $\hat{h}$ is learnt via quantile regression on the calibration/validation data. It derives high-probability bounds for achieving approximate covariate-conditional coverage, while being conditioned on the calibration/validation data. Additionally, such methods are minimax rate optimal.

STRENGTHS AND WEAKNESSES:
__Strengths__

1.	Studying the conditional coverage of split conformal methods is an important research area. In practice, the calibration/validation is fixed, and covariate-conditional guarantees are ideal.

2.	The theoretical claims provide high-probability bounds for achieving approximate covariate-conditional coverage, while being conditioned on the calibration/validation data

3.	The paper is well written and has a good flow to it.

__Weaknesses__

1.	While the experimental results provide some evidence supporting the theoretical claims, is there a way to verify their correctness more concretely? For instance, by checking if the empirical coverage lies within the derived bounds?

QUESTIONS:
__Questions and Comments__

1.	Did the authors run evaluations on regression tasks as well?

2.	It is not easy to demarcate existing theoretical results from new ones. Doing so will help with the presentation of the paper.

3.	What is $S_{1}^{n}$ in line 52?

__Typos__

1.	The equation in the panel under line 106 has two equalities with 0.

2.	"The construction (5) appears appears to obtain better..." $\rightarrow$ "The construction (5) appears to obtain better..." (line 136)

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-22