Title: One Sample is Enough to Make Conformal Prediction Robust
Venue: NeurIPS 2025
Accepted: yes
Keywords: Conformal Prediction

Robust Conformal Prediction

Uncertainty Quantification
Abstract: For any black-box model, conformal prediction (CP) returns prediction *sets* guaranteed to include the true label with high adjustable probability. Robust CP (RCP) extends the guarantee to the worst case noise up to a pre-defined magnitude.
For RCP, a well-established approach is to use randomized smoothing since it is applicable to any black-box model and provides smaller sets compared to deterministic methods. However, smoothing-based robustness requires many model forward passes per each input which is computationally expensive.
We show that conformal prediction attains some robustness even with *a single forward pass on a randomly perturbed input*. Using any binary certificate we propose a single sample robust CP (RCP1). Our approach returns robust sets with smaller average set size compared to SOTA methods which use many (e.g. $\sim 100$) passes per input. 
Our key insight is to certify the conformal procedure itself rather than individual conformity scores. Our approach is agnostic to the task (classification and regression).  We further extend our approach to smoothing-based robust conformal risk control.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the high computational cost of current smoothing‐based robust conformal prediction (RCP) methods and shows that in fact a single noise‐augmented inference suffices. Their key insight is to “certify the conformal procedure itself” rather than individual model scores: by applying any binary robustness certificate (e.g. Gaussian or Laplace smoothing bounds for ℓ₂ or ℓ₁ threats) directly to the coverage probability of the vanilla conformal predictor, one can adjust the calibration quantile to restore a 1–α worst‐case coverage guarantee. This yields **RCP1**, a black‐box, score‐agnostic method that needs only one perturbed forward pass per test point.\n\n**Contributions**\n\n1. **Method**: Introduce RCP1, which uses a single randomly perturbed sample plus an off‐the‐shelf binary certificate to attain robust coverage, eliminating the need for costly Monte Carlo sampling.\n2. **Generality**: Works with any model, any conformity score (classification or regression), and any smoothing/distributional threat model (Gaussian, Laplace, uniform; ℓ₂, ℓ₁, etc.).\n3. **Extension**: Show how to convert smoothing‐based RCP1 into a robust *conformal risk control* procedure, demonstrated on semantic segmentation (Cityscapes) to control false‐negative rate.\n4. **Empirical validation**: On CIFAR-10 and ImageNet with both ResNets and ViTs, RCP1 matches the set‐size vs. coverage trade-off of state-of-the-art methods like BinCP (which use 64–128 samples) while being orders of magnitude faster (one vs. ∼100 inferences) (Fig. 1-middle) and achieving the same 1–α robust coverage under adversarial attacks (Fig. 1-left).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- This paper proposes RCP1, a novel robust conformal prediction method that requires only a single model inference per input, yet still guarantees the desired coverage under worst-case perturbations. The theoretical contribution is solid – the authors prove that by adding one instance of random noise to each example (both in calibration and at test time) and leveraging a robustness certificate, the conformal coverage guarantee can be maintained even for an adversarially perturbed input. In particular, they choose a slightly more conservative nominal coverage level $1-\\alpha'$ such that the certified lower bound on coverage under perturbations is at least $1-\\alpha$. This key insight – “certify the conformal procedure itself rather than individual scores” – is innovative and allows RCP1 to inherit some robustness from even a single noise-augmented forward pass.\n- The paper provides formal propositions (e.g. Proposition 1) and defers detailed proofs to the appendix, indicating a rigorous theoretical underpinning. All assumptions and theoretical results appear to be clearly stated (per the checklist), lending credibility to the claims (thought I did not have time to check all the details - I have read in detail the proof of proposition 1).\n- Empirically, the evaluation is thorough and convincing. The authors test RCP1 on multiple datasets and models, including CIFAR-10 and the large-scale ImageNet with both ResNet and Vision Transformer backbones, as well as a challenging semantic segmentation task for conformal risk control. The experiments demonstrate that RCP1 achieves robust prediction sets of comparable size to state-of-the-art multi-sample methods while using dramatically less computation. For example, on ImageNet, RCP1’s average prediction set size is similar to that of the prior BinCP method when BinCP uses about 64–128 Monte Carlo samples per input. This is a remarkable improvement in efficiency – RCP1 attains essentially the same set size and coverage as BinCP with ~100 forward passes, using only one forward pass. The paper quantifies computational gains: for instance, processing 5000 ImageNet images with a ViT model takes ~2.8 minutes with RCP1 versus ~5.9 hours with an equivalent BinCP (128-sample) approach – a huge speedup that underscores the practical value of the method. \nWeaknesses: \n- One concern is that RCP1’s reliance on a single random draw per input introduces higher variance in the prediction sets. The authors acknowledge this limitation: “RCP1 shows considerably more randomness in the prediction sets compared to BinCP”, due to the inherently random one-sample procedure. In other words, the exact set of labels returned by RCP1 can vary across different runs (or different noise instantiations) on the same input, more so than methods that effectively average over many samples. While the coverage probability is still guaranteed in expectation, this increased variability might be undesirable in certain applications that demand deterministic or highly stable predictions.\n- the paper does cover a lot of technical ground, which can make certain portions dense. A reader not already acquainted with randomized smoothing certificates might find it challenging to parse the certificate optimization (Equation 4) and related discussion without consulting the appendix. The authors rely on references (e.g. to Yang et al. [22]) for some of the certificate details, which is appropriate, but it means that part of the argument is somewhat terse in the main text. For example, the statement “the optimization in Eq. 4 is translation invariant... we can use a fixed set of canonical directions” hints at non-trivial results from prior work that are not fully explained. Similarly, Section 3.1 briefly mentions how to compute certified probabilities $c_{\\downarrow}[p,B]$ for various smoothing schemes by “small changes” to Yang et al.’s recipe, which could be hard to grasp without familiarity with [22].\n- While the idea is novel, it builds heavily on prior work, and in some respects could be seen as an incremental improvement. The paper itself references BinCP (Zargarbashi & Bojchevski, ICLR 2025) as a state-of-the-art robust CP method that introduced the concept of using a “single binary certificate” to reduce sample complexity. In fact, the authors of this submission appear to be extending their own previous work (BinCP) – the terminology and some methodology overlap (e.g. use of Clopper-Pearson confidence intervals, references [23] and [24] which are prior methods by presumably the same group). From that standpoint, RCP1’s novelty is mainly in eliminating the remaining Monte Carlo sampling that BinCP still required.\n- -A final point is that the method’s correctness assumes access to a reliable robustness certificate for the base model (e.g. the ability to lower-bound the probability $P(f(x+\\epsilon)$ predicts class $y$) for all $\\epsilon$ in a perturbation ball). The approach is only as good as these certificates – if a model cannot be certified for a meaningful radius (or if the certificate probability $c_{\\downarrow}$ is too low), the resulting prediction sets could become trivially large.\n\nQUESTIONS:\n- typo in the equation below line 121\n- Another slight clarity issue is that the paper introduces many acronyms (CP, RCP, RCP1, BinCP, CAS, etc.) and shorthand notations for different methods. While it does define them (often by citing the relevant references), a concise table or summary of methods might have helped the reader keep track. That said, the authors do consistently remind the reader of what each baseline is (e.g. noting that BinCP uses p-quantiles or that CAS uses CDF information), which mitigates confusion. \n- The writing could have been further improved with a bit more exposition on the certificate calculations and perhaps by explicitly contrasting RCP1’s procedure with standard conformal prediction in a step-by-step manner. These are minor suggestions; the core ideas and results are still clearly conveyed.\n- The method is described as agnostic to the setup (classification vs. regression), but all experiments are on image classification. It would be valuable for the authors to discuss or demonstrate the applicability of RCP1 in other settings – for example, conformal prediction for regression tasks or other data modalities. Even a brief discussion or a toy experiment (perhaps in the appendix or supplemental material) showing how robust conformal prediction works for a regression problem would strengthen the paper’s claim of generality\n- The proposed RCP1 method relies on the availability of a “binary certification” for model predictions (for example, the paper uses randomized smoothing to certify robustness of a classifier). A question arises about how practical and general this requirement is. The authors should clarify to what extent RCP1 depends on a specific certification technique and whether any robust model certificate can be used interchangeably\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
The paper introduces RCP1, a robust conformal prediction method that overcomes the computational hurdles present in smoothing-based RCP approaches. By leveraging randomized smoothing, the proposed method achieves robustness against adversarial perturbations using only a single forward pass per input. The experiments are conducted on CIFAR-10 and ImageNet for classification and cityscapes for risk control settings.

STRENGTHS AND WEAKNESSES:
Strengths

- The paper aims to address the computational bottleneck in existing smoothing-based robust conformal prediction methods, which is well-motivated.

- The integration of noise-augmented inference with robust conformal prediction offers an easy-to-implement solution.

- The paper is well written.

Weaknesses

- The robustness guarantee in RCP1 is marginal over the randomness introduced by the smoothing process. Coverage is only guaranteed on average over the added noise, rather than under worst-case scenarios as in prior work. Specifically, the guarantee depends on the type of added noise (e.g., Gaussian perturbations), and does not hold uniformly across all inputs or perturbations. This also weakens the comparison with baselines that aim to certify worst-case robustness, potentially leading to unfair or misleading results.

- The authors primarily focus on Gaussian smoothing, which limits the practicality of the approach, as real-world noise settings are often more complex and structured.

- The study heavily builds upon prior work, making its theoretical contribution insufficiently novel.

QUESTIONS:
See above.

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

comments: SUMMARY:
This paper uses the binary certificate technique to develop a single sample robust CP (RCP1), which reduces model forward passes in constructing the prediction set. The experiments show that RCP1 achieves robust prediction sets with comparable sizes to state-of-the-art methods while reducing computational overhead.

STRENGTHS AND WEAKNESSES:
**Strengths**

This paper introduces a new technique for robust CP under adversarial noise.

**Weaknesses**

**1. Notations are heavy, confusing, and not easy to follow.**

For example, from Lines 115 to 121, there are multiple definitions for $q$. In addition, Lines 80-85 are a recap of previous work [24], and the notations are not necessary here. Moreover, some notations are not explained here, such as $c$ with arrows. 

**2. The writing is not clear.**

The authors should provide a clean algorithm or definition for the proposed prediction set. After the first read, I didn't know what the final prediction set is, not to mention the implementation.

**3. Potential theoretical error.**

In Line 119, if $q$ is defined in Proposition 1, $P(S_{n+1} > q | E_+) \ge 1-\alpha$ seems wrong because given $E_1,...,E_{n+1}$, the quantile $q$ is not symmetric to $(X_1,Y_1),...,(X_{n+1},Y_{n+1})$.

QUESTIONS:
1. In Line 41, what does "vanilla CP combined with noise-augmented inference" mean?

2. What is the proposed prediction set and coverage guarantee?

3. How can you make sure the lower bound in Proposition 1 is greater than $1-\alpha$? If this lower bound is large, how can we say "conformal prediction combined with single randomized augmented inference already shows robustness"?

4. What is the form of $\mathcal{C}_r$ in Line 108?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

comments: SUMMARY:
The paper studies robust conformal prediction (RCP) by randomized smoothing. The authors show that with a single forward pass on randomly perturbed input can already attain robustness. Interestingly, the empirical findings support that RCP1 produces smaller set sizes compared to BinCP when sample rates are small. Additionally, RCP1 is agnostic to the distribution of inputs and scores.

STRENGTHS AND WEAKNESSES:
Strengths:

1.	The paper is well-written, clearly organized and easy to follow. 
2.	The theorems are well presented and are sound for the parts that I checked.
3.	The experiments indeed show favorable properties of RCP1 especially in terms of computational time, and comparable quality when sample size is small. In particular, the empirical findings support that RCP1 generate smaller prediction set size when sample rate is low.

Weakness:

1.	The comparison of RCP1 and previous methods using multiple samples is obscure. For example, a parallel comparison between RCP1 and BinCP can be beneficial.

QUESTIONS:
1.	Is the computation of the certified lower bound for RCP1 the main difficulty comparing to methods such as BinCP? If so, is the limitation on the choice of smoothing methods or assumptions?
2.	It is interesting to see when sample rate is low, RCP1 consistently produces smaller prediction size. Is it purely a consequence of randomness in sampling for BinCP by Monte Carlo? Can the authors provide more insights, or whether it is possible to improve prediction set size when sample size is moderate (e.g., between 1 to 100).

LIMITATIONS:
See "Strengths and Weakness".
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper claims that robust conformal prediction can be achieved by using a single sample for each example in the calibration set. \nRobust conformal inference is a method for providing a super-set of classes in a classification task that with high probability \n(on average over test points) includes the correct class, even when an adversary is allowed to locally perturb the feature vector. \nAs with (many) other robust conformal prediction, the key to robustness is adding random noise to the input before applying the score function, thereby negating the specific choice of perturbation. However, this cannot overcome the fact that the (smoothed) calibration set is no longer exchangeable with the (smoothed) perturbed feature vector, but rather only with the (smoothed) original vector. Methods that were proposed to bound this extra decay (or extra radius) used many samples (per calibration point) to estimate the post-smoothing mean score or the post-smoothing quantiles. This paper presents an argument that the multi-sample approach is not necessairy to maintain conformal coverage, and therefore reasonably-sized robust conformal set can be estimated with far fewer forward passes. This can make robust conformal prediction much more affordable even when using large prediction models. The experiments compare the conformal prediction levels under adversarial feature choice, and find that for small samples, the new sets are competitive in terms of size and coverage (though have higher variance).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nSignificance: \nFor ML algorithms based on large NN, the number of calls to the prediction function is often budgeted due to time, \ncompute or even monitory constrains. Therefore identifying a robust-conformal algorithm that can be run with considerably less \nfunction calls per sample with only limited impact on the efficiency (e.g. the adversarial radius) can be very important for adoption of these ideas.\n\nOriginality: \nI think the argument itself for providing much more efficient (in terms of samples) conformal sets is potentially a step forward for the field. \nIt is hard for me to assess how much is new in the argument or method due to issues of clarity in which I struggled with the paper. \n\nWeakness: \nClarity / Quality: \nI admit that I did not manage to understand the main theoretical argument of the paper.\nNamely, what are the method by which they claim to change methods that use multiple samples (to estimate lower bounds on the mean region or the quantiles of the distribution) into a method that uses a single sample. \n\nHere are the specific questions I did not have answers to, after multiple readings of this manuscript: \n- What are the main reason that other papers state for using multiple samples to estimate the bound-adjustment for robust Conf-Sets, \nand how does this paper elevate or pass over those reasons.  \n- Does the paper introduce \"only\" a new claim and proof, or also a new robust-conformal prediction algorithm (inc a new way to estimate the bounds c^{arrow down}). \n- Moreover, it is not clear what is the precise *algorithm* or bound that used to compute the Conformal Set and in what specific ways is it different from, say, Zargarbashi and Bojchevski bound or Cohen et al or other previous works. \n- Do the claims of success of a single sample based bound refer to the empirical or to the theoretical worst-case coverage (e.g. to which one of the curves in the middle of Figure 2). \n\nMore generally, I found it very hard to distinguish the trees from the forest: \nFor examples, in Prop 1 the papers runs through proof without explaining the ideas behind it, \nand uses in the statement of the result and the proof an important concept / notation (the lower certified bound c^{down-arrow}) that is  defined in Eq 4 almost one page below. \n\n[This could be my fault here; I am not an expert in the litrature of robust Conformal sets\nI am well read in conformal prediction generally, and have manged to read and mostly understand the main cited papers including Yang et al, Cohen et al, and Zargarbashi and Bojchevski; I failed on this paper. ]\n\nAlso, in clarity, it is not helpful that the full explanations of the figures is in the appendix; many of these figures where hard to understand without the longer version of the legend, and I did not know the longer legends are in the appendix until I went there to check.\n\nQUESTIONS:\nSee above, there are many things I did not understand upon reading the paper. \nI think this at least partly reflects on the clarity of the current writup. \nBut also I would like to better understand the answer to these questions, \nand clarifications, especially if they manifest in the text of the paper as well, could definitely result in the increase of scores.\n\nLIMITATIONS:\nI thought the discussion of the added variance in using one sample (discussed in two samples in the limitations) should be expanded;\n more generally, \nwhat do we lose when we only use a single sample?\nDoes it mean that our conformal coverage is less or more uniform \n(i.e. is the conditional coverage more or less similar across samples)?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-06"}