SAFER: Risk-Constrained Sample-then-Filter in Large Language Models
Authors: Qingni Wang
Yue Fan
Xin Eric Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Question answering
Calibration
Uncertainty Quantification
Primary area: generative models

Abstract: As large language models (LLMs) are increasingly deployed in risk-sensitive applications such as real-world open-ended question answering (QA), ensuring the trustworthiness of their outputs has become critical. Existing selective conformal prediction (SCP) methods provide statistical guarantees by constructing prediction sets with a constrained miscoverage rate for correct answers. However, prior works unrealistically assume that admissible answers for all instances can be obtained via finite sampling, even for open-ended QA scenarios that lack a fixed and finite solution space. To address this, we introduce a two-stage risk control framework comprising abstention-aware **SA**mpling and conformalized **F**ilt**ER**ing (SAFER). Firstly, on a held-out calibration set, SAFER calibrates a sampling budget within the maximum sampling cap, using the Clopper–Pearson exact method at a user-desired risk level (i.e., the maximum allowable miscoverage rate of the sampling sets). If the risk level cannot be satisfied within the cap, we abstain; otherwise, the calibrated sampling budget becomes the minimum requirements at test time. Then, we employ calibration instances where correct answers are attainable under the calibrated budget and apply the conformal risk control method to determine a statistically valid uncertainty threshold, which filters unreliable distractors from the candidate set for each test data point. In this stage, SAFER introduces an additional risk level to guide the calculation of the threshold, thereby controlling the risk of correct answers being excluded. We evaluate SAFER on three free-form QA datasets utilizing five popular LLMs, and demonstrate that it rigorously constrains two-stage miscoverage risks at test time. 
Furthermore, we show that SAFER is compatible with various task-specific admission criteria and calibration-test split ratios, highlighting its robustness and high data efficiency.

Published as a conference paper at ICLR 2026
SAFER: RISK-CONSTRAINED SAMPLE-THEN-FILTER IN LARGE LANGUAGE MODELS
Qingni Wang', Yue Fan', Xin Eric Wang! University of California, Santa Cruz
University California, Santa Barbara (qwang158, yfan71 @ucsc. edu, ericxwangeacsb.ed
Stage : Sample
Stage II: Filter
N Colibration [Original sumples)
N' B
Gencrate dates for
2 E
Qraa
Qzrqa)
Qr
LLM
Status
Calibration Subset (QA simples with admissible canoidates within 5 genenations)
Coverage
x
X
Abstention
Coverage
:
Uncertality Quentrficatin
Miscoverage
Uncertainty
Test Time
Semple Rt
Calibrated Sampling Size
Calibrated Uncertainty Threshold i
(OO)
Figure 1: Overview of SAFER's calibration and test-time process. In Stage I, we derive statistically valid minimum sample budget that can strictly control the test-time risk of the candidate of size & not covering correct answers. In Stage II, we employ the calibration instances, which can obtain admissible answers within & samples, to calibrate threshold i. This threshold filters out unreliable answers in the candidate set while still constraining the miscoverage risk of the final prediction set.
ABSTRACT
As large language models (LLMs) increasingly deployed in risk-sensitive ap- plications such as real-world open-ended question answering (QA), ensuring the trustworthiness of their outputs has become critical. Existing selective conformal prediction (SCP) methods provide statistical guarantees by constructing predic- sets with a constrained miscoverage rate for correct answers. However, prior works unrealistically assume that admissible answers for all instances can be ob- tained via finite sampling, even for open-ended QA scenarios that lack fixed and finite solution space. To address this, we introduce two-stage risk control framework comprising abstention-awant SAmpling and conformalized FiltERing (SAFER). Firstly, on held-out calibration set, SAFER calibrates a sampling bud- get within the maximum sampling cap. using the at dser-desired risk level (i.e., the maximum allowable miscoverage rate of the sampling sets). If the risk level cannot be satisfied within the cap, we abstain: otherwise, the calibrated sampling budget becomes the minimum requirements
exact method
"Corresooniigg author

---

Published as a conference paper at ICLR 2026
test time. Then, we employ calibration instances where correct answers are attain- able under the calibrated budget and apply the conformal risk control method to determine statistically valid uncertainty threshold, which filters unreliable dis- tractors from the candidate set for each test data point. In this stage, SAFER in- troduces additional risk level to guide the calculation of the threshold, thereby controlling the risk of correct answers being excluded. We evaluate SAFER on three frre-form QA datasets utilizing five popular LLMs, and demonstrate that it rigorously constrains two-stage miscoverage risks at test time. Furthermore, we show that SAFER is compatible with various task specific admission criteria and calibration-test split ratios, highlighting its robustness and high data efficiency. Our project page available at this link
INTRODUCTION
Recent advances in large language models (LLMs) have propelled substantial progress across vari- ous question answering (QA) tasks (Alawwad et al. 2025 Yang et al., 2025). Nevertheless, critical challenges persist-monn notably, the prevalence hallucinated content and miscalibration of con- fidence scores associated with model outputs (Huang et al., 2025; Cox et al. (2025). These issues compromise the reliability of LLMMs and constrain their deployment in real-world conversational systems (Guo et al., 2025; (inghal et al. 2025 Liu et al. [225aa] Li et al. [2025] 2024). Uncertainty
quantification (UQ) has emerged as a promising solution to these concerns, by providing quanti- tative indicators of predictive uncertainty (Duan et al.) 2024 Wang et al. [002cc 2025c). It can
trigger user-facing warnings the uncertainty exceeds a predefined threshold, thereby enhancing the trustworthiness of query results.
Split conformal prediction (SCP) is a distribution-free, model-agnostic framework that provides rig- orous UQ (Papadopoulos et al. 2002 Angelopoulos & Bates 2021 Angelopoulos et al. 2024a). It guarantees the coverage of ground-truth labels in classification tasks, assuming data exchangeabil- ity. Specifically, SCP defines nonconformity measure for each data point the residual between the model prediction and the ground-truth on a held-out calibration set, and then calibrates an uncer- tainty threshold under a user-specified risk level (rigorous uncertainty score), which guides test-time construction of prediction sets by selecting high-quality classes. The probability of each data-driven set failing to cover the true class controlled by the risk level (upper bound of risk). While recent work has effectively applied SCP to multiple-choice question answering (MCQA) (Ye et al. 2024), its application to open-ended settings remains limited due to two fundamental challenges: Open- ended QA lacks fixed output space, and we cannot ensure that for each given inquiry. the deployed
LLM is capable of generating admissible answers via finite sampling (Wang et al. 2024c 2025a]
Kaur et al. 2024). @ Due to the temperature settings of sampling hallucinations, the candidate (or sampling) set may contain irrelevant and unreliable answers (Wang et al.) 2025a d), which prevents them from being presented to users for downstream decision-making (Cresswell al. 2024 2025).
To resolve the dual challenges and control the miscoverage risk in open-ended QA tasks, we pro- pose two-stage calibration framework consisting of abstention-aware SAmpling and conformalized FiltERing (SAFER), as shown in Figure [] First, SAFER adopts learn-then-test (LTT) paradigm to derive the required sampling budget test time on the calibration set under user-specified risk level a (Angelopoulos et al. 2025). while developing an abstention mechanism (Yadkori et al.) 2024 Specifically, instead of assuming that every sampling set contains correct answers (Wang et al. 2024c; Kaur et al. 2024, Wang et al.) 2025a). SAFER imposes an upper bound M on the number of samples per question, and deems that if no admissible answers are obtained within M attempts, the model abstains from it. Then, we apply the Clopper-Pearson exact method (Clopper & Pearson| 1934, Wang et al., 2025b) to establish an upper confidence bound of the miscoverage rate at each sampling budget, and determine the minimum number of samples necessary for the test-time risk-the probability of failing to include admissible answers- to remain below a. If the calibration procedure shows that this risk bound cannot be satisfied even with M samples, SAFER abstains, ensuring that the framework never produces invalid prediction sets under unreachable risk levels.
Once the abstention-awan sampling stage secures the desired coverage (or abstains when unattain- able), the resulting candidate set undergoes conformalized filtering, which further prunes unreliable predictions and provides more efficient references for users. Specifically, SAFER first evaluates the
2

---

Published as a conference paper at ICLR 2026
uncertainty score of each sampled answer in the candidate set and removes those with scores above certain threshold. Since heuristic uncertainty measures cannot perfectly distinguish between correct leverage the conformal risk control (CRC) frame- work ( (Angelopoulos et al., 2024b) and introduce another risk level , to guide the calibration of the threshold, thereby controlling the risk of incorrectly excluding correct answers while removing unreliable answers from each candidate set. This two-stage design ensures that SAFER delivers reliable prediction sets while remaining robust to the inherent unpredictability of open-ended QA.
and incorrect answers (Wang et al. 2025c),
We evaluate SAFER on three open-ended QA benchmarks across five open-source LLMs. Experi- mental results demonstrate that SAFER validly estimates the upper bound of the miscoverage rate at each sampling budget and constrains the test-time risks under various user-specified risk levels (a). Through the conformalization of uncertainty guided by another risk level (3), SAFER validly filters out unreliable candidates and maintains the statistical rigor of prediction sets. Moreover, we vali- date the robustness of SAFER under four correctness evaluation criteria, as well as its efficiency in achieving risk control the test set using only small amount of calibration data. Our contributions can be summarized as follows:
We introduce SAFER as a novel risk control framework that provides user-specified cover- age of admissible answers in open-ended QA tasks.
We incorporate the abstention mechanism into the sampling stage and develop a more prac- tical method for calibrating the sampling budget.
We conformalize the uncertainty of admissible answers on the calibration set, and calibrate a threshold to filter out unreliable candidates within each sampling set, which maintains the statistical validity of the prediction sets and enhances predictive efficiency.
2 RELATED WORK
Uncertainty quantification. Informing users of the model's uncertainty score within the QA pro-
cess can enhance the reliability of downstream decision-makinn (Liu et al. 2025bb Xia et al. 2025
Shorinwa al. 2025). Previous studies have developed probabilistic frameworks (Hou al.
2025
Wang et al. (025c) like semantic entropy (SE))Kuhn
Kuhn et al. (Farquur et al. (2024) or prompted
models to express their uncertainty (Rathi et al. [202]] Xu et al., 2025). However, these approaches
are heuristic and lack statistical verification. SCP can transform any heuristic notion of uncertainty
from any model into a statistically rigorous one closed-ended tasks (Kumar et al. [023 Ye al.,
2024 Kostumov et al., 2024). Yet, due to the inherent definition of nonconformity in SCP, detailed in Appendix B applying it to open-ended tasks remains an open topic.
SCP in open-ended QA. In white-box scenarios, SE-SCP (Kaur et al. 2024 Kuhn et al., 2023) establishes the nonconformity based on the entropy derived from semantically clustered predictions. CLM (Quach et al., (024) utilizes the LTT framework (Angelopoulos et al. 2025) to identify param- eter configurations with rigorous statistical guarantees. Under black-box conditions, LofreeCP (Su) let al. 20244] Gupta et al.) (022) and ConU (Wang et al.) (022c) constructs nonconformity measures by employing Self-Consistency-Bass
(SCB) uncertainty estimates, such as sampling frequency. TRON (Wang et al.) 2025a) further employs sampling size calibration to determine the required number of samples at test time and then defines nonconformity scores based on frequency, thereby achieving two-stage risk control. However, existing research either assumes that every instance can yield correct answers via finite sampling or fails to consider that certain risk levels may be infeasible.
LTT (Angelopoulos & Bates 2021 Angelopoulos et al. 2025 2024a) is a mod- ular post-hoc calibration framework that retrames risk control as multiple hypothesis testing prob- lem: it first learns predictive model, then systematically searches over a low-dimensional param- eter space using calibration data, testing each candidate through valid concentration- based p-values and applying famely-wise error rate control to identify parameters that satisfy user-specified risk levels (Ni et al. 2025 Jung et al. 2025), thereby yielding finite-sample statistical guarantees inde- pendent of model architecture data distribution. In this work, draw the principles of LTT to analyze, on the calibration set, whether to abstain under a given risk level or to search for the minimal sampling size that can statistically guarantee the test-time coverage of admissible answers.
3

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
3.1 OVERVIEW
Following SCP-based frameworks, we hold out a set of N calibration samples Deat = {( yi ) H= where I; E X denotes the input question of the i-th calibration data and yf E 3 is the corresponding ground-truth answer. Let F X - y denote an LLM. For each question, we sample multiple (e.g., 8) candidate answers {ū,
{ 1 j==1- Let A : y x y + [0, 1] denote a task-specific relevance function that evaluates how well the generated answer matchs the ground-truth for a given input. An answer 8/)) E y is considered admissible only if A(g! yf) ) 2 λA, where Aa is a user-specified relevance threshold. For each sample, up M candidate answers are allowed be drawn. If
none of the M sampled answers is admissible (i.e., Vg E {0j 1:-1: A(G. v()) < Aa), we treat abstention, attributing this to the limitations of the LLM's capabilities. Moreover, to filter out unreliable candidates from each sampling set, we denote by U ) + [0. 1] an uncertainty measure
{o
that estimates the sentence entropy of the sampled answer QQach al. 2024: Duan et al. 2024).
An answer is retained only if its uncertainty score falls below a threshold 7.
Our goal is calibrate two parameters on the calibration set, s and f: the former guides the sampling process, while the latter filters out irrelevant answers from the sampled set, enabling the construction of prediction set C. for each test input Frest X that covering at least one admissible answer with high probability. Formally, given desired error tolerance we provide the following guarantee:
Pr (rV E (Ireet) A (Ú, Wiest) < AA) st) 21-6,
where 8 is the significance level (e.g., 0.05), and the outer probability controls for the sensitivity of our algorithm with respect to calibration data.
3.2 ABSTENTION- SAMPLING BUDGET CALIBRATION
We begin by calibrating a statistically valid sampling budget that guarantees the coverage of admis- sible answers. For a given sampling budget s, we first report the number of calibration data points for which the candidate sets of size fail to include any admissible answers:
N
A < Ann A(.
meat (5)
Σ
I Vi E
(2)
and then estimate the empirical miscoverage rate on the calibration set:
ical (s) meat (8)
(3)
N
We aim to establish an upper bound of the true (or system) miscoverage rate new test samples at sampling budget 8, denoted as R(s) == Pr(vý E C (x), , A v²) < Xa), through realizations observed on the calibration set. To achieve this goal, we leverage the Clopper-Pearson exact method, by constructing a high-probabilig (i.c., at least 8) exact upper confidence bound for R (s):
R+(ss = sup { R: (BiBi((N, < [Nffca 2 66.
(4)
which characterizes the largest system risk at a sampling budget of that does not contradict the empirical error rate fear (s) observed from the calibration set. That is, if the system risk R (s) exceeds the upper bound R (s), then observing an empirical error rate as low as (ea)(s) constitutes
low-probability event, and is considered statistically implausible at significance level 6. Hence, obtain the follow guarantee:
Pr(R(s)= & (s) 2 1-6.
(5)
A formal proof of Eq. 5 is provide Appendix
B
To rigorously ensure the high-probability coverage of admissible answers via sampling candidates, we calibrate by constraining the upper bound (e+(ss to fall below user-specified risk level a:
8 = inf {
M, -(((s)) (8) s sa..
(6)
4

---

Published as a conference paper at ICLR 2026
By construction, this choice of å minimizes the sampling cost subject to the risk control requirement. Note that Rt (M) > a, determine that, given the limited sampling, it is not possible control the miscoverage rate under the current risk level a, and thus abstain from making a prediction. For
a valid sampling budget 8, given that lt+ (8) 2 R (8) holds with high probability, we have
Pr(R(3) R (6) a)
8, $ E [].
We then obtain the test-time guarantee in the sampling stage:
Pr Vg E {0, )
(tst))
);=1, Viest) < <Aa) Sa, Se [1,
(8)
with probability at least 1 - å over the draw of the calibration set Deal.
3.3
CONFORMALIZED FILTERING
After determining the sampling budget 8, since each model's sampling space inevitably contains other unreliable answers, directly delivering the calibrated candidate set to users may not be suit- able for downstream decision-making due to these distractors. To address this issue while main- taining statistical rigor, SAFER introduces another risk level | and leverages the conformal risk control framework Angelopoulos et al. (2024b): it conformalizes the uncertainty scores reliable answers the calibration set, thereby calibrating statistically rigorous uncertainty threshold that is employed to filters out unreliable candidates from each test sample's candidate set at test time To maintain risk constraints,
to form the final prediction framework Angelopoulos et (2024b) to calibrate a statistically rigorous threshold to identify high- quality answers. Specifically, we first select, from the calibration Peal. all samples that contain at least one admissible answer within their s-sized sampling set to form the calibration subset:
employ the conformal risk control
D Deat (8) = { (zi, vi) E Deat 39 E 1,,0),, A(ū. (. 2 AA).
(9)
Here, we aim to ensure that under the same sampling budget & as set for test samples, each calibration data point includes at least one admissible answer in its candidate set. Then, this calibration subset of size N' = [Deat (8) can be utilized to calibrate the threshold that differentiates admissible answers from unreliable ones, subject to the other target risk level B.
For given threshold t, we construct a prediction set for each calibration sample in Deot($):
C, (x.) = { : (i) < {t) (Ii, v(") E (eat(S).
(10)
U calculatethe accumulation of the token-wise entropy over the whole sentence 9:
U Σ log p(zp (ck);
(11)
where p(zx (<c)) denotes the probability of generating 2 as the k-th token and ÿ<* refers the previously generated tokens within ÿ.
We then define a loss function as:
4(1) = VV A < 2a).
(12)
Let LN (t) 1 determine the uncertainty threshold as
NO EN (t) denote the average loss over all samples in the calibration subset. We
i - inf {
L' (t) + N'
{0) :3]]innt inf (::
(NN' + 1) -
LN (t)
(13)
N'
By applying in test-time filtering, we can ensure that if the s-sized candidate set of each test sample contains at least one admissible answer, the probability of the calibrated prediction set not covering admissible answers remains below B, formulated as:
Pr Vgg E Cj (Frest) A(ÿ, Viest) < <Aa) 53,, if 3 E (ū) j== Alū, Viest) 2 Aa. (14)
(test))
5

---

Published as a conference paper at ICLR 2026
LLaMA-3.1 388-nnitrctt
OpenChat-3355
Owen2 5-38 Instruct
Oweni2 5-78-insttur
Qwenn: 5-1 All-instruct
Empircal
0.05
10
20
15
15
10
20
Sampling Size
Sampling Size
Sampling 5
Sampling Stze
Sampling Size
Upper Bound of System Risk
Test-Time Emprrcal Risk
(a) TriviaQA.
LLoM0-J.1-00-imstruu
OpenChat-Ja 3.5
Owenz 3-30 Instruct
5w Instruct
Owen2 5-1-2-1------ 14B-instruct
Emprica faat
0.4-
0.2
0.1
10
15
100
15
20
10
15
20
10
20
15
20
Sampling Size
Sempling Size
Sempling Size
Sampling Size
Sempling Size
Upper Bound of System R sk
Thsti1mme Empirical Risk
(b) CoQA
Figure 2: Empirical miscoverage rates under different sampling budgets. The dashed lines denote the system miscoverage upper bounds derived via the Clopper- PPapssn exact method the calibration set, while the solid lines show the corresponding empirical miscoverage rates on the test set.
LLaMAr3.1 IiD-instruct
OpenChet:3.5
Owen2.5 Instruct
Owen2.5 7-8iintuett
Owen2 Instructt
0.5
6.15
0.15
0.15
0.15
0.10-
6.30
0.10
0.10
0.10
Empiniall
0.05
0.05
0.0s
0.55
0.56
0.05
0.55
6.00
6.35
0.15
0.00
0.55
6.10 Rigk Lovel
0.15
0.50
0.10 Risk Lovel
Risk Level
Risk Level
Risk Level
SAFER
IRON
Upper Bound
(a) TriviaQA.
LLaMA-3.1 3.1 88--nstruct
OpinChat-3.5
Owerdd 5-38-isttrctt
Gwenz 5-78-isstrurt
Qwen2 5-145--istt-
0.4
0.4
0.4
0.4
Emprical
02
0.
0.2
0.2
0.2
0.1
0.0
0.2
0,4
0.0
0.2
6A4
0.0
0.25
0.30
0.35
0.40
0.65
0.25
0.30
0.35
0.45
0.25
0.30
0.55
0.45
TRON
Upper Bound
(b) CoQA.
Figure 3: Comparison of TRON and SAFER on the control of test-time EER in the sampling stage.
Of course, in the absence of an admissible answer in the candidate set, the calibrated prediction set will inevitably lack one well.
Finally, we establish formal guarantees on the coverage of admissible answers:
Pr Pr(PPVV E Cy(Zres)), A(j, Viest) < AA) 3 2 1 - 8.
(15)
The complete proofs of Eq. and and Eq. are provided in Appendix B
4 EXPERIMENT
4.1 EXPERIMENT SETTINGS
Benchmarks. We evaluate SAFER on three open-ended QA benchmarks, categorized into closed-
book settings: TriviaQA (Joshi et al. 2017) and ScienceQA (Lu 2022), and open-book settings

---

Published as a conference paper at ICLR 2026
Table 1: Test-time EER results in the filtering stage on the TriviaQA 0.05) and CoQA (a 0.25) dataset at various risk levels (3).
(a) TriviaQA (a 0.05)
0.05
0.30
0.15
0.30
0.25
0.30
11.35
0.40
Upper (o oi)
D.0755
0.1925
0.3400
0.6755
0.335
0,5825
0.400
0:844-000137
0.1322-0:0100
0.1805000189
0.2265-0.019
0.2733002215
0:3255000211
03908=000198
041755000195
OpeaChat-3.5
2508901100012
0,1375:000:55
0,851120017
0.231820.115
0.200100211
0.32742000222
0,3722000211
0:4225::000144
Qwen2.5 5- Instrct
0.002200.0145
0,1288.0:0119
0.1757100190
0.2223.0.0204
122091000212
0.118610:2311
0,36310:0211
0.1121100216
Qwend: 5:78-Inimmet
008022-0.0136
0.1264-0.0158
0.1788001172
0.207700.094
0,203-00000
0.3184-00211
0:663-00022
5-14B Bastract
11077000100444
0,1253200146
0.1746200065
0,222360.0184
0,2025002122
0.316:20:0217
033122601217
04100
(b) CoQA (a 0.25)
0.05
0.20
0.15
0.20
0.25
0.30
0.35
0.40
Upper Bound 8
0.3875
0.3250
0.3625
0,4000
0,4375
0:4750
0,5125
0.5900
03347-00012
0.2743-0:0155
03130000171
0.3545-0.0068
0.9044-00166
04947-00212
04779-000222
OpeaChur-3.5
0:2773000119
0,2794200153
0:3201100189
0.359820.0196
0.400:001205
044040002224
0,402200222
0.5206::00224
Qwen2.5 5 30 Instrct
0.202510.0199
0::00100:0100
0.3391100222
0.3775000253
0.4159
0.4599000278
0,9022000278
0.5224100277
Qwen?: 5-78-Inemet
0.761-000149
0:3145-0:0881
0.0511000156
0.3912-0.0176
0:476-001755
0:660:001155
0.501500:000
0533300077
5-14B-Basttrect
0.271000.0177
0,5076400117
0334000068
0.851=00018
044245000156
0.46290000201
0,5035600213
0.59056000205
TriviaQA Dataset
COQA Dataset
14
12
12
10
Size
Size
8
2-
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
(B)
Lett-Prediction patternll, HighteSaaaptee Sampting
Liame-3.3
OpenChat--3.5
Owen2.5-10
Owen2.5.
Qwen225-148
Figure 4: Comparison of the (average) calibrated sampling budget in the sampling stage with the (average) prediction set size after filtering out unreliable answers in the second stage. Filtering compresses the calibrated budgets into tighter sets while maintaining risk control.
with passage context: CoQA (Reddy al. 2019). Additionally, we evaluate generalization on the multi-modal reasoning benchmark MM Yu al. (203)) in Appendix E.5 More details can be found in Appendix D.I
We consider three popular series of "off-the-sheie LLMS OpenChat (Wang et al. 2024a): OpenChat-3.5,, LLaMA (AI@ Meta 2024) LLLMA-3.1-880 3.11-88-Insstruct and Qwen (Yang ct al. 2024): Qwwen-2.5-3B-Instrrue Qwwn-2.5-7B-Innn Instruct, and Qwen-2.5-144 I-n-strtutt.
Backbone LLMs.
Correctness Evaluation. We deem candidate admissible if the task-specific relevance function R indicates agreement with the reference beyond a predefined threshold. We utilize sentence similar- ity (Wang et al.) 2024c 2025d) with a threshold 0.6 by default. Moreover, we consider Rouge-L score (Lin 2004 Duan et al. 2024), bi-entailment based on natural language inference (NLI) (Wang) et al. 2025c Kuhn et al. 2023), and LLLM-based semantic evaluation (Zhang et al. 2024) Details
of implementation are provided in Appendix D.2
Evaluation Metric.
We evaluate the statistical rigor of SAFER by checking whether the test-time
Empirical Error Rate (EER) (Angelopoulos & Bates 2021 Wang et al.) 2024c) 2025a is bounded by the user-specified risk level. In the sampling stage, EER represents the average miscoverage rate
7

---

Published as a conference paper at ICLR 2026
Upper
LLaM4k3.3 33--inttuct
OpenChat 3.5
Qwenn 5-Bi-ineruut
Owenn2 5-75-iratrus
Empiniaal
Owonz 14B-1sstuct
0.15
625
0.30
0.35
0.6G
0.65
9.70
0:66
6:80
(a) TriviaQA (closed-book).
Upper
Instrut
Emorical Srerr
Owen2 5-70 5-7-trrtttt
Owonz rrstrcctttt
0.2
0.0
0.20
625
030
0.55
6.50
0.8
0.50
6.70
0.79
0,5
0.00
RiSK
(b) CoQA (open-book).
Figure 5: Test-time EER results in the sampling stage with Rouge-L score as the correctness metric.
over all the candidate sets of size on the test set (size N,), denoted as
N.
EER
ve E A
tR
N
In the filtering stage, EER is computed on the prediction sets formed by answers selected from the sampling set according the calibrated uncertainty threshold, denoted
EER 1 N 1 lvg C,(, < tt)..
Σ
Hyperparameters. Following (Wang et al. 2025d b). we employ multinominal sampling for can- didate answers with the generating temperature of T.0. The generation length of free-form answers is fixed at 36 for TriviaQA and CoQA, and 24 for ScienceQA. We set the significance level 5 to 0.05. Moreover, we set the split ratio of the calibration and test amples to 0.5 by default. We provide detailed analysis of the impact of the sampling budget M in Appendix E.4
4.2 EMPIRICAL EVALUATIONS
We first verify the statistical validity of the system risk upper bound established in Eq, Autilizing the Clopper-Pearson exact method. As demonstrated Figure 2 across both the TriviaQA and CoQA datasets, the empirical miscoverage rates of five LLMs on the test set consistently remain below the corresponding confidence bounds under different sampling budgets. The results indicate that the sta- tistically rigorous sampling budget derived from the calibration set togethrr with the upper bound of its corresponding EER- effeeciivv carries over test time. Then, we can pre-compute a high- confidence upper bound on the test-set EER for given sampling budget using statistical methods on the calibration data, which establishes the foundation for calibrating the sampling budget.
We next discuss how, under various desired risk levels (a), SAFER controls the EER of the sampling stage. As presented in Figure 3 we compare our method with the previous TRON approach under conditions with abstention, i.e., we maintain the test data points that fail to obtain admissible answers via finite sampling (M samples in this case). can be observed that, at certain risk levels, the sampling budget derived by TRON fails to control the EER on the test set. For instance, when 0.03, TRON with the OpenChat-3.5 model on TriviaQA yields a test-time EER exceeding 0.06. In contrast, SAFER consistently maintains the EER below the corresponding risk level across
8

---

Published as a conference paper at ICLR 2026
Stage One (a=0.05)
Stage Two (a=0.05, (=0.05)
0.05
Uppe 3eanc-005
0.10
Upper Sean
0.04
Empirical Error Rate
0.08
0.03
0.06
Empirical
0.02
0.04
0.01
0.02
0.03
0.3 Split Sperchat-3.5
0.00
0.1
0.5
Solit
Owanz25 38-trstruct
Cwen2.5-77
2-w.55----m----- Instruct
Dwan223 14B-irttruct
Bound
Figure 6: Test-time EER results in both the sampling and filtering stages across various calibration- test split ratios on the TriviaQA dataset with sentence similarity as the correctness metric.
both datasets and all five LLMs. Since TRON does not account for cases where the model fails to obtain an admissible answer through limited sampling during calibration, the derived sampling budget cannot be effectively applied to the test set. As a result, some test data points that should be abstained from lead to errors exceeding the target risk level. SAFER effectively resolves this issue, making the sampling stage more rigorous and practical.
As mentioned, for the same question, LLMs may produce some irrelevant or incorrect answers in the candidate set due to hallucinations or the use of temperature sampling strategies (Zhu et al.) 2025] Hou et al. 2025). SAFER adapts the conformal risk control framework to address this. As illustrated in Table I although some correct answers may be filtered out due to the limitations of UQ (as no perfect method yet exists to fully distinguish correct from incorrect answers), SAFER still maintains control over the miscoverage rate of admissible answers in the final prediction sets on both the TriviaQA and CoQA datasets- joint upper bound of a, and 3. At that point, SAFER achieves two-stage guarantees, rigorously controlling test-time EER. Furthermore, as presented in Figure 4 given a fixed sampling budget under specified a, the filtering guided by the risk level substantially reduces the size of the final prediction set. For instance, when / is as small as 0.1, the average set size decreases from 7.9 to 5.5 on the TriviaQA dataset with the Llama-3. 1-8B model, and the prediction set size continues to shrink as 33 increases. At the same time, the test-time EER consistently remains below the upper bound. Results of two-stage risk control on ScienceQA are provided in Appendix E.2 While we primarily utilize entropy-basee uncertainty, our framework is metric-agnostoe We validate SAFER's effectiveness on black-box models (e.g., GPT-4o-mini) using consistency- -basd frequency scores in Appendix
that the EER consistently remains below the
4.3 SENSITIVITY ANALYSES
Sensitivity to Correctness Metric. SAFER applies to various correctness evaluation criteria. In this section, we discuss the case of using the Rouge-L score. As shown in Figure 5] SAFER still man- ages to keep the test-time EER within the target risk level on TriviaQA and CoQA. is worth noting that since Rouge-L. score evaluates the longest common subsequence of sentences, in the open-book setting, where the context provides the exact lexical form of the reference answer, relatively weaker LLMs can achieve lower risk levels on CoQA. By contrast, in the closed-book setting, semantically equivalent but lexically different answers may occur, making this alignment less straightforward. Results of risk control in the filtering stage on the TriviaQA and CoQA datasets using Rouge-L score, along with results under other evaluation metrics, are provided in Appendix E.
Sensitivity to Calibration-Test Split Ratio. SAFER calibrates the sampling budget and uncer- tainty threshold on the calibration set. As shown in Figure D when the split ratio between calibra- tion and test data decreases from 0.5 to 0.1, the EER on the test set consistently remains below the spec-iped upper bound. This demonstrates that SAFER can achieve risk control over a large number of test samples with only small amount of calibration data, highlighting its efficiency.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this paper, we propose a two-stage risk control framework, SAFER, which provides user-specified coverage of admissible answers in open-domain QA tasks. SAFER adopts the LTT paradigm, allow- ing that some instances may fail to yield a correct answer within finite sampling. It first determines the minimal sampling size that meets the -per-fped risk level. If the risk level cannot be satisfied even under the maximal sampling budget. SAFER abstains, which effectively ensures the statistical decision-making making, while overcoming the draw- backs of existing methods that rely on strong assumptions. SAFER further adapts the conformal risk control framework to uncertainty-thres -itheshotttt calibration, which, while maintaining risk control, effec- tively removes unreliable candidates from the sampled sets and produces prediction sets with higher predictive efficiency. We envision SAFER as a general-purpose framework to advance trustworthy, uncertainty aware decision-making in foundation models across diverse downstream tasks.
rigor of the prediction sets delivered for downstream decision
LIMITATION
Although SAFER addresses the control of miscoverage under conditions where correct answer cannot be obtained through limited sampling. bypassing the assumption that all problems for a deployed model can be correctly answered with finite sampling, and achieves two-stage miscoverage risk control, still relies on the assumption of data exchangeability. In the case of distribution shifts, SAFER would need further optimization ensure risk control the test set.
ETHICS STATEMENT
This work introduces SAFER, risk-constrained framework for uncertainty quantification in large language models. Our study does not involve human or animal subjects, personal or sensitive data, or any interventions that may raise ethical concerns. All datasets employed (TriviaQA, CoQA, and ScienceQA) are publicly available and widely used in the research community. We strictly adhere their original licenses and usage policies. While our framework aims to improve the reliability of open-ended question answering, in sensitive domains. By providing statistically rigorous guarantees, SAFER designed to enhance trustworthiness and mitigate harmful mispredictions, thus promoting responsible deployment of AI systems.
acknowledge the potential misuse of large language models
REPRODUCIBILITY STATEMENT
We have taken multiple steps to ensure reproducibility of our results.
Datasets.
We provide detailed descriptions of the datasets and experimental splits in Appendix
D.I
Algorithms and Proofs. The methodology is described with precise mathematical notation in Section 3. and all theoretical guarantees are proved in Appendix C
Hyperparameters. Evaluation hyperparameters, including sampling budgets, thresholds, and generation settings, are reported in Section 4.1
Code and Implementation. An anonymous code repository with implementation and instructions will be made available after the review process, once the paper is accepted and published.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Al@Meta. Llama 3 model card. 2024.
Hessa A Alawwad, Areej Alhothali, Usman Naseem, Ali Alkhathlan, and Amani Jamal. Enhancing textual textbook question answering with large language models and retrieval augmented genera- tion. Pattern Recognition, 2025.
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
Anastasios N Angelopoulos, Rina Foygel Barber, and Stephen Bates. Theoretical foundations of conformal prediction. arXiv preprint 1rX14 2024a.
Anastasios Angelopoulos, Stephen Bates, Emmanuel Candès, Michael I Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. The Annals of Applied Statistics, 2025.
Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Con- formal risk control. In The Twelfth International Conference Learning Representations, 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic risual-linguise tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185-24198, 2024.
Charles J Clopper and Egon S Pearson. The use of confidence or fiducial limits illustrated in the case the binomial. Biometrika, 1934.
Kyle Cox, Jiawei Xu, Yikun Han, Rong Xu, Tianhao Li, Chi-Yang Hsu, Tianlong Chen, Walter Gerych, and Ying Ding. Mapping from meaning: Addressing the miscalibration of prompt- sensitive language models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Jesse C Cresswell, Yi Sui, Bhargava Kumar, and Noël Vouitsis. Conformal prediction sets improve human decision making. In Proceedings of the 41st International Conference on Machine Learn- ing. pp. 9439-94444 9457, 2024.
Jesse C. Cresswell, Bhargava Kumar, Yi Sui, and Mouloud Belbahri. Conformal prediction sets can cause disparate impact. In The Thirteenth International Conference on Learning Representations, 2025.
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint 2501::::001.124455 2025.
Chirag Gupta, Arun K Kuchibhotla, and Aaditya Ramdas. Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recognition, 2022.
Pengcheng He, Xiaodong I Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert
with disentangled attention. In International Conference on Learning Representations, 2021.
Bairu Hou, Yang Zhang, Jacob Andreas, and Shiyu Chang. A probabilistic framework for 11m hallucination detection via belief tree propagation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, 2025.
11

---

Published as a conference paper at ICLR 2026
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions Information Systems, 2025.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.). Proceedings of the 55th Annual Meeting of the Association for Computational Linguis- tics (Volume I: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1
Jachun Jung, Faeze Brahman, and Yejin Choi. Trust or escalate: LLM judges with provable guaran- tees for human agreement. In The Thirteenth International Conference on Learning Representa- tions, 2025.
Ramneet Kaur, Colin Samplawski, Adam D. Cobb, Anirban Roy, Brian Matejek. Manoj Acharya, Daniel Elenius, Alexander Michael Berenbeim, John A. Pavlik, Nathaniel D. Bastian, and Susmit Jha. Addressing uncertainty in LLMs to enhance reliability in generative AI. In Neurips Safe Generative Al Workshop 2024, 2024.
Vasily Kostumov, Bulat Nutfullin, Oleg Pilipenko, and Eugene Ilyushin. Uncertainty-aware evalua- tion for vision-language models. arXiv preprint vrXiv:2402.14418 2024.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation natural language generation. In The Eleventh International Conference on Learming Representations, 2023.
Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and An- drew Beam. Conformal prediction with large language models for multi-choice question answer- ing, arXiv preprint arXiv:2305. 12...8884 18404, 2023.
Feiran Li, Qianqian Xu, Shilong Bao, Zhiyong Yang, Runmin Cong, Xiaochun Cao, and Qing- ming Huang. Size-invariance matters: Rethinking metries and losses for imbalanced multi-object salient object detection. In International Conference on Machine Learning, pp. 28989-29021. PMLR, 2024.
Feiran Li, Qianqian Xu, Shilong Bao, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. One image is worth a thousand words: A usability preservable text-image collaborative erasing frame- work. In Forty-second International Conference on Machine Learning, 2025.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 2004.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantifi- cation for black-box large language models. Transactions on Machine Learning Research, 2024.
Xiaohong Liu, Hao Liu, Guoxing Yang, Zeyu Jiang, Shuguang Cui, Zhaoze Zhang, Huan Wang. Liyuan Tao, Yongchang Sun, Zhu Song, et al. A generalist medical language model for disease diagnosis assistance. Nature medicine, 2025a.
Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. Uncertainty quan- tification and confidence calibration in large language models: survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, 2025b.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference Neural Information Processing Systems (NeurIPS), 2022.
Bo Ni. Yu Wang. Lu Cheng, Erik Blasch, and Tyler Derr Towards trustworthy knowledge graph reasoning: An uncertainty aware perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
12

---

Published as a conference paper at ICLR 2026
Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In European conference on machine learning, pp. 345-356. Springer, 2002.
Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. Conformal language modeling. In The Tweifth International Conference on Learning Representations, 2024.
Neil Rathi, Dan Jurafsky, and Kaitlyn Zhou. Humans overrely on overconfident language models, across languages. In Second Conference on Language Modeling, 2025.
Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266 2019. doi:
162/ta2/a La 00066..
Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z Ren, and Anirudha Majumdar. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. ACM Computing Surveys, 2025.
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, 2025.
Jiayuan Su, Jing Luo, Hongwei Wang, and Lu Cheng. API is enough: Conformal prediction for large language models without logit-access. In Findings of the Association for Computational Lingnistics: EMNLP 2024, 2024.
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Ad- vancing open-source language models with mixed-quality data. In The Twelfth International Conference on Learning Representations, 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing dision-language model's 's perception of the world at any resolution. arXiv preprint arXiv:2409.12111 12191, 2024b.
Qingni Wang, Tiantian Geng, Zhiyuan Wang, Teng Wang, Bo Fu, and Feng Zheng. Sample then identify: A general framework for risk control and assessment multimodal large language models. In The Thirteenth International Conference on Learning Representatiioss, 2025a.
Zhiyuan Wang, Jinhao Duan, Lu Cheng, Yue Zhang, Qingni Wang, Xiaoshuang Shi, Kaidi Xu, Heng Tao Shen, and Xiaofeng Zhu. ConU: Conformal uncertainty in large language models with correctness coverage guarantees. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024c.
Zhiyuan Wang, Jinhao Duan, Qingni Wang, Xiaofeng Zhu, et al. Coin: envertainty-guarddinn se- lective question answering for foundation models with provable risk guarantees. arXiv preprint arXiv:2506.20178, 2025b.
Zhiyuan Wang, Jinhao Duan, Chenxi Yuan, Qingyu Chen, Tianlong Chen, Yue Zhang, Ren Wang, Xiaoshuang Shi, and Kaidi Xu. Word-sequence entropy: Towards uncertainty estimation in free- form medical question answering applications and beyond. Engineering Applications of Artificial Intelligence, 2025c.
Zhiyuan Wang, Qingni Wang, Yue Zhang, Tianlong Chen, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. SConU: Selective conformal uncertainty in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), 2025d.
Zhiqiu Xia, Jinxuan Xu, Yuqian Zhang, and Hang Liu. A survey of uncertainty estimation methods on large language models. In Findings of the Association for Computational Linguistics: ACL 2025, 2025.
13

---

Published as a conference paper at ICLR 2026
Zhengtao Xu, Tianqi Song, and Yi-Chieh Lee. Confronting verbalized uncertainty: Understanding how I1m's verbalized uncertainty influences users ai-assisted docision-makingg International Journal of Human- Compuute Studies, 2025.
Yasin Abbasi Yadkori, Iiaa Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, luliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, et al. Mitigating llm hallucinations via conformal abstention. arXiv preprint orXiv:2405.01566 2024.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng. Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Ke- Yang Chen, Kexin Yang. Mei Li, Min Xue, Na Ni. Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin. Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yunyang Wan, Yunfei Chu, Zeyu Cui, Zhenru Zhang, and Zhi-Wei Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025.
Fanghua Ye, Mingming Yang. Jianhui Pang, Longyue Wang, Derek Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking Iims via uncertainty quantification. Advances in Neural Information Processing Systems, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02444 2023.
Ruiyang Zhang, Hu Zhang, and Zhedong Zheng. Vl-uncertainty: Detecting hallucination large vision-language model via uncertainty estimation. arXiv preprint arXiv:2411.11990 2024.
King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, et al. Scaling test-time compute for Ilm agents. arXiv preprint arXiv:2506. 12222226..222.... 2025.
14

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS
We used LLMS only as general- writing assistants, specifically to check grammar, spelling. and style consistency. The LLMs did not contribute research ideation, experimental design, data analysis, or result interpretation. All substantive content and scientific contributions were developed solely by the authors.
B BACKGROUND OF SPLIT CONFORMAL PREDICTION IN CLASSIFICATION TASKS
We summarize Split Conformal Prediction (SCP) for multi-class classification with precise notation. Let Deat = {(x,, yi ) be an i.i.d. calibration set of size n, where v E [K]= (1,..., K} the ground-truth class, and let ʃ X A be a fixed pretrained classifier that maps an input x to class probabilities f(x) (f()). ........ (.....,,(()))))))) f(x)x) with Σ f(z)x
Define the nonconformity score for each calibration example as 8; = f(x:) (Ira)y; (one minus the predicted probability of the true class), and sort the scores in ascending order, $(1) I ... (()) Given target error rate
a E (0, 1), set the finite-sample (1 - a) quantile
8 ((++)(1-a)))- At test time, for input z, form the conformal prediction set
(x) { E [K] 1 [(x), <
Under exchangeability between Deal and the test point (X, Y*). SCP attains the marginal coverage guarantee
)C(*) [(n + 1)(1 a]1
> 1- a.
n
C PROOFS
In this section, we first provide a complete proof that the upper confidence bound f (() defined in Eq. satisfies Pr (s)) f (8) 2 1 - 5 defined in Eq. 5
Without loss of generality, we define A(s) as the empirical error rate variable on the calibration set with the sampling size s. We then define the CDF of Rss) under the true risk R(s) as
D(r R(s)) Pr Pr(
[(s) S R(s))
(16)
This CDF characterizes the likelihood that the empirical error rate is no greater than a specified risk threshold r, under the true distribution governed by R(s).
We then define the corresponding inverse CDF as D-1 (v) sup {r: (r R())) sp}
(17)
By the definition of Rt (s), it holds that
D (Feal (3) Ret (s) =d.
(18)
Since (6) sup fr: D (s)) 8}, if D (feat (s) [s))) < 6, then we have feot (s)
D-1 (6). This implies the following equivalence
{D (feat (s) R (s)) < + foal (s) < D-1 (8)}
+ (Pr (D (foml (s) R < 6) < Pr (Feat (s) < D < (6))
Considering that D (r R (s)) is monotonic decreasing
(s), i.e., larger R (s) leads to smaller
D R (x)). if R (8) > R+ (s), then D (Feat (s) R (x)) < 8. This implies
(s) > Rt (s) - D (foal (s) R (s)) < 6)
- Pr Pr R(s) > Ret (s) < Pr (D (foat (s) R()) < 6)
15

---

Published as a conference paper at ICLR 2026
Thus,
Pr R( ((())) Ret (8) Pr ((()) Rtt (s)
(D (Foal (s) R8 (s)) < <,), Pr (Feat (8) D (6))
(19)
2 - 8
demonstrating that R+ (s) is the upper endpoint of the Clopper-Pearsen exact confidence interval for the R (s). This completes the proof.
Next, prove the theoretical soundness of Eq. 14 and Eq. 15
By the exchangeability of N' calibration data points and the given test instances, we have leest (t) ~ Uniform (14((), ".,,"",,,n ,Iv (t)))
Given that (f) S 1 (E {0, 1)), we obtain B(Itest (f) = I (i)
N'41
4())
N' +1
N' (f) + Least
(20)
N' #1
N' L' +1
N' +1
aa.
Then, if the sampling of size & contains admissible answers, i.e.,
3i {Ø, (oftest)) j--1 A(Ú. Viest) 2 AA
we guarantee the error rate of the calibrated prediction of the data point failing to cover admissible answers
Pr Vg Cg (Irest) A(Ú, Viest) < A1)) = =E(1 (
)a))
= (lhestt (1)
(21)
S 3
1 Since Pr(VV- E (offeott) }ー=1, A Vist) AA) satisfied with probability least 1 achieve two-stage risk control and guarantee the upper bound of the miscoverage rate: Pr(vů E Cj(Frest): A(j.viest) AAa)
Pr(vū E C₁ (Frest): A(ÿ, viest) <>Al 39 E { ); A(. Viest) 2 AA)+
Pr(vū E {1) (tett)))
A(g, Viest) < AA)
(22)
1 j==1"
< (1 at) (1 B)
aa aB
with probability at least å over the draw of the calibration set Deat. This completes the proof.
D DETAILS OF EXPERIMENTAL SETTINGS
D.1 DETAILS DATASETS
TriviaQA (Johhi et al. 2017) is large-scale open-domain QA dataset with about 650K ques- tion-and answer pairs, paired with automatically retrieved evidence from Wikipedia and the Web. We adopt the closed-bood book setting, evaluating models only questions and answers. For our experi- ments, we randomly sample 2000 data points from the validation set. Promot example shown in Figure 14
16

---

Published as a conference paper at ICLR 2026
CoQA (Redyy al. 2019) is conversational QA dataset with more than 127K question answer pairs from dialogues across seven domains. Answers are free-form spans grounded source pas- sages, and later questions depend strongly on dialogue history. We use CoQA in the open-book setting, where models have access to both the context. We evaluate on 2,000 instances sampled from the validation set. Prompt example is shown in T5
ScienceQA (uu et al. 2022) is a science-related open-domain QA dataset designed to evaluate the ability of models understand complex scientific text. It contains approximately 13,679 crowd- sourced science questions. In our experiments, we use the full validation set consisting of 1,000 questions. Prompt example shown Figure
MM Yu et al. 2023 is a comprehensive benchmark designed to evaluate the integrated ca- pabilities of large multimodal models. consists of 217 diverse image-text samples that require models to solve complex tasks by combining core vision-language skills. We utilize MMVet to assess the generalization of our framework to reasoning-intensive multi-modal scenarios. Prompt example is shown in Figure 17
D.2 DETAILS OF CORRECTNESS METRICS
Rouge-L deems a sampled answer as admissible if its longest common subsequence, regarding the ground truth answer, is larger than a threshold. Considering that there many semantically equiv- alent but differently phrased answers in open-ended QA tasks, we set the threshold of Rouge-L, as 0.3 by default (Duan et al. 2024). We also adopt bidirectional entailment approach to evaluate the correctness of free-form answers Kuhn et al. (202);; Farquhar et al. (2024); Lin et al. (2024); Wang al. (202cc). Specifically, we employ an off-the-sheli DeBERTa- large model He et (2021) as the Natural Language Inference (NLJ) classifier, which outputs logits over three semantic relation classes: entailment, neutral, and contradiction. An answer deemed correct if the classifier predicts entailment for both directions, i.e., when evaluated on (answer ground-truth) and (ground-truth answer). Moreover, following Zhang et al./ (2024), we employ Qwen-2.5-3B-Inss. to assess whether the answer matches the ground-trus truth, using the prompt template:
Ground truth: <sentence 1>.
Model answer: <sentence 2>> Please verify if the model answer matches the ground truth. Respond with either 'Correct' or 'Wrong' only.
E ADDITIONAL EXPERIMENTAL RESULTS
GENERALIZATION UNDER ALTERNATIVE EVALUATION CRITERIA
have already examined Rouge-L alternative correctness criterion the sampling stage on TriviaQA and CoQA, showing that the calibrated sampling procedure maintains bound-valid risk control. Here, extend the analysis the filtering stage across TriviaQA and CoQA. As shown in Tables 2b and 2a EER under Rouge-L evaluation remain strictly below the theoretical bound a + 8 a across models and risk levels, with EER increasing smoothly as 3 grows. These results confirm that SAFER consistently achieves valid risk control under Louge-L, demonstrating its robustness to task ssecifpecifcc evaluation metrics.
ROUGE-L Evaluation In the section
correctness, where candidate is deemed admissible if it is bidirectionally entailed with the reference answer at a threshold of 0.5. Figure 2 reports the sampling stage results across TriviaQA and CoQA. We observe that empirical error rates remain consistently below the upper bound a across risk levels. These results confirm that the abstention-awart sampling procedure generalizes robustly under entailment correctness. Ta- ble summarizes the filtering stage performance under the entailment criterion. Across all datasets and models, EER values remain strictly below the combined upper bound + B a& at each set- ting. This shows that SAFER maintains reliable two-stage guarantees under an entailment based definition of correctness.
Entailment Evaluation We next examine entailment-
17

---

Published as a conference paper at ICLR 2026
Upper
LLaM4k3.3 3i--inttuct
OpenChat 3.5
Qwenn2 5-Bi-ineruut
Empincall Erron
Owenn2 5-75-imatrut
Owonz 5-14B-rrstou
0.2
6-25
0:20
0.25
0.30
63
0.40
0.35
9.50
0.65
0.80
Risk Level
(a) TriviaQA
0.8
Upper
instrut
Emoiical Eror
Owen2 5-72 5-7---mttttt
Owanz t4eitrstuctt
0.0
0.30
0.35
040
0.5
0.0
0,5
0.5
085
6.73
0.50
(b) CoQA
Figure 7: Test-time EER results in the sampling stage with Entailment score as the correctness metric.
Table Test-time EER results in the filtering stage uitilize Rouge-L. score as correctness evaluation on the TriviaQA (a 0.5) and CoQA (a 0.5) dataset at various risk levels (3).
(a) TriviaQA (a = 0.5)
0.15
0.20
0.25
0,30
0.35
0.40
Upper Bound (e: + ad)
LLaMA 3.1 8B Instruct
0.1491 L10.02V9 022061101668 0.3122100206 0,385
0.84132000222
0.487220.0222
0.55401000441
0.5789100252
(b) CoQA (a 0.5)
Upper Bound (o 5 ad)
0.7
aMA-3 1-88-lestraet
01314+001157 0:002+00154 02750+00215 0.3889000033 0,3809 -0.067 0.431 40.227 04445-000253 0.5941+00254
Openchus-35
02053500166 0.2422000091 0.236620.119 0.3208000199 0.3591801199 0230366002006
LLM-Based Semantic Evaluation Following Zhang et al. 2024), We further examine an LLM- based semantic evaluation criterion, where admissibility is determined by semantic validation from Swen-2.5--B-Instrree This criterion is more flexible than similarity or entailment, as leverages the model's broader reasoning ability align outputs with human-like semantic evaluation. Figure reports the sampling stage results on TriviaQA and CoQA. Across all models and risk levels, the empirical eиrог remains strictly below the upper bound a. Table summarizes the filtering stage results on the same datasets. For all datasets and models, the empirical error rates remain well below the theoretical upper bound a + a3 at every setting, and track the bound closely with small, stable margin. Overall, these findings demonstrate that SAFER retains rigorous two-stage risk control even under an LLM-based semantic evaluation criterion, highlighting its adaptability to human-alignn correctness definitions.
E.2 EXPERIMENTAL RESULTS ON SCIENCEQA
This section supplements the experimental results and analysis across the sampling stage and tering stage ScienceQA under four correctness evaluation criteria: similarity-bane based matching.
18

---

Published as a conference paper at ICLR 2026
Upper
LLaM4k333 3i5-inttuctt
OpenChat 3.5
Owenn2 5-Bi-inerurt
Owenn2 5-75-tratrut
Empiniaal
Owonz (wen125-14B-rsttoe 14B-rsttuct
0:2
0.30
0.35
044
6.8
0.58
6.89
0.85
9.70
0.79
Risk Level
TriviaQA
Upper
Emorical Srerr
Owen2 5-72 5-7--mrattt
Owonz 11o0225-14B100 t44itrstucct
0.2
0.0
0.5
040
643
0.0
63
0.8
6.8
9.55
0.90
645
0.90
(b) CoQA
Figure 8: Test-time EER results in the sampling stage with LLM-based semantic evaluation as the correctness metric.
Table Test-time EER results in the filtering stage uitilize entailment correctness evaluation on the TriviaQA (a 0.5) and CoQA (a 0.5) dataset at various risk levels (5).
(a) TriviaQA (a = 0.5)
0.05
0.10
0.15
0.20
0.25
0,30
0.35
0.40
Upper Dound (o:
0.675
LIaMA-3 1-88-Iestract
0.1782000119
Openchur-35
0.003611000102
0.1399100138
0.1752100166
0,228820.112
0,2711200019
0.32740000199
0.3730100188
041571000212
Qwen2.5 Inttrett
00.06.10.0125
0.991100112
0.1431100176
0.0091
0,2130.0:0218
0,7741100227
0.3252100263
0.3829100006
Owen2.5
027761100210
0.3252100240
Qwen?. 5-14 ([B-Instruct
00646:100117
0,1211100665
0.1671100175
0.2128:0:000
0.299710:0007
0.30/510:0100
033251002266
0.3873100214
(b) CoQA (a 0.5)
8.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Upper Round (o + A w/)
0.775
0.625
0U675
LLaMA-3.1
10.091
Openchat-3.5
0.17 11711000109
0220010013
0.231100119
0.2755-00188
0,310210:0189
0.3661100088
0.363610022 0229
0.38361002229
Table 4: Test-time EER results in the filtering stage uitilize LLM as correctness evaluation on the TriviaQA (a 0.5) and CoQA (a 0.5) dataset at various risk levels (5).
(a) TriviaQA (a 0.5)
Upper Dound (o + 8 05)
0.525
0.53
0.575
3.6
0.625
0.65
0U675
0.7
LLaMA-3, I-St-lastract
Q17322001224
012809500119
0.3681=00236
04735000352
0,462200:144 0.53990012277
0.5754800023 014499900244
Openchat
0.0481
1100111
00842100122
0.123310015
0.171820.0192
0.230510:0214
0.26710.0000
0.3167100223
0.3621100241
(b) CoQA (a = 0.5)
Upper Dound (o + 3 ez)
0.525
0.575
366
0.625
0.68
0u675
0.7
LLMMA-3 1-St-llstrast
0144220001133 02100-00162
0.3633500193
0.3077000025
0.3755001041
040335001144
04477400257
010344800237
Openchat
0.1552100118
0.1891101111
02221100174
0.2066.0.008 0,2891000117
0.33661000144
01369110022
0.0811000251
ROUGE-L score, bi-entailment based on natural language inference (NLD), and LLM- based seman- tic evaluation.
19

---

Published as a conference paper at ICLR 2026
Upper
LLaMA.333 1ww--33--0---attree
OpenChot
Qwenn 5-384henurt
Denen2 5-754tammut
Omeon2 5-14-irst
0.03
0.7
0.13
0.5
0.1
Lival
Figure 9: Test-time EER results in the sampling stage with similarity score as the correctness metric on ScienceQA.
Upper Sound
0.2-
OpenChul-355
Ower2 38 instrct
Emprical Enve
Owen2 5-7-mmmtrutt
Owwe12.5 Beer2.--34-rrattuut
80
0.35
0.45
0:30
3.80
6.70
8.80
Nak Level
Figure 10: Test-time EER results in the sampling stage with Rouge-L score the correctness metric on ScienceQA.
Under similarity- baased evaluation, follow the same experimental setup described previously and adopt threshold of 0.6 to determine correctness on the ScienceQA dataset. We first calibrate the minimal sampling budget on the calibration split for each target risk level a and then assess on the test split. As shown in Figure the test-time EER stays strictly below for all models and all risk levels in the sampling stage. In the filtering stage, Table 5 reports that for every 3 the final EER controlled below + a.8.
Using ROUGE-L score as the correctness criterion and set thereshold 0.3, the same calibration protocol ensures that, for every a, test-time EER remains strictly below upper bound a for all models in the sampling stage, as shown in Figure 10 the filtering stage, when varying the risk level 8, the final EER remains below the theoretical bound a + 3 aB for every B, while the prediction set size decreases monotonically as 33 increases, as reported in Table 6
For bi-entailment correctness evaluation, follow the and set thereshold of 0.5, it is shown Figure [T in the sampling stage that across models and target levels, the test-time EER remains strictly below a. In the filtering stage, varying the risk level (3 keeps the final EER within the theoretical bound a + 3 a. for every 9 in Table 7
two-stage protocol described earlier
the judging model and follow the same experimental setup as in previous sections. The results are shown in Figure 12 and Table the target risk level a across all models, confirming that the abstention-awarc calibration procedure preserves valid control. Compared with similarity and Rouge-L metries, LLM judgement produces more semantically flexible assessments of correctness. In the filtering stage, varying the risk level demonstrates consistent behavior with thcoretical expectations. As reported in Table g for each the final EER is strictly bounded by + 3 a3. Among the evaluated models, Qwen2.5- 3B-Instruct as the judge yields the strongest calibration efficiency, resulting in smaller error rates. Across different correctness evaluations, SAFER provides uniform, model-agnostic risk control, meeting the corresponding upper bound across the sampling and filtering stages, demonstrating its robust abstention-aware calibration.
Under LLM-based correctness evaluation ScienceQA, we adopt Qwen2.5-3BB-Inss
the sampling stage, EER on the test split remains strictly below
20

---

Published as a conference paper at ICLR 2026
Upper
LLaM4k3.3 33--inttuuct
OpenChat 3.5
Empintaal Erroe
Owenn2 5-75-iratrus
Owonz 5- 14B-1rstrcct
0.2
6-55
0:20
0.25
8.30
0.49
0.35
9.58
0.79
0.79
Level
Figure 11: Test-time EER results in the sampling stage with entailment score as the correctness metric on ScienceQA.
Upper Sound
0.8
LLaMA 3.1 88 Instrutt
OpenChae:33.5
Oenen2 5-B5-tetrrrt 78-rrstrut
Empraa Ereo
0.6
0.4
025
0.35
040
045
0500
0:65
0390
Rak Level
Figure 12: Test-time EER results in the sampling stage with LLM-based semantic evaluation as the correctness metric on ScienceQA.
Table 5: Test-time EER results in the filtering stage uitilize similarity as correctness evaluation on the ScienceQA (a 0.05) dataset various risk levels (3).
0,10
0,15
Upper to 8 0.)
11 I-8--neeeccc
0.0768-0.0162
0. 0.1280000015
0:1734400252
0.2191-0.0075
02656+000289
05::55-000118
03369-000377
0.0100000041
Opendius.35
0.12480000255
026600002596
033122=0:002
5wen2 39
0.008510.0185
0.117900024
0,1657100244
0212440.0071
02621100000
0.307810.0229
03566100024
0,9081100034
Qwen2 Qwen25-7B-B8 5-7m-l
00.001
0.1220:0:0000
0,1718100216
0.2216:000076
02686100299
0.3:54:0.0055
03390:100000
0,4078
Qwen2 5-148-Irotruct
0.067240.0155
0.1163e0:217
0.169+00265
0.2140=0,0278
03655=00329
0.044+0.0311
Table Test-time EER results in the filtering stage uitilize Rouge-L. score as correctness evaluation on the ScienceQA (a 0.5) dataset at various risk levels (5).
0,85
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Upper Bound to o3
0,525
0.55
0,575
0.6
0.625
0.675
1LLMA-311-88--emn
0.00666000221
0.172200.099
022022000488
0.3100=0.0512
03777700050
04876-00487
0.5143+0.8477
Openchus-335 3.5
0,0481 20.0148
0.1409100264
0.1852-0.0290
0.22941000077
0.2723.0.1291
031724000199
0,362520.0328
Qwen2.5 5-30 Insruct
0.0284 10.0118 0.0011000119
0.0284
0.0901 100119
0.1239.0.0034 011566100250 0.1008
0.225210.0201
0.2638 10.0327
Table 7: Test-time EER results in the filtering stage uitilize entailment as correctness evaluation on the ScienceQA (a 0.5) dataset at various risk levels (3).
Upper Bound for o3)
LLaMA-3.1 3.1-A-----55 1D-------- Instrice
02118
0.2001100257
0.3321.0.0022
0.3922000336
0.0013000030
0.5012100004
0,5545.10.0555
Openchot-3.5
0.072100.0144
0. 011088000185
01105+00261
0.
0.1992000000
02246+00271
0.2855-0.0258
03322-000233
0.371+000300
5-38-Instmct
0.041:000014
1007%6201184
0.112520U227
0.1408:01.025
0.1873600281
0.2221-0.1275
022112000110
0,302250.1166
Qwen2.5 5 70 Insruct
0.0073.00.0127
00715L00183
0.1061
0.349610.0068
0.18701000011
0,2247.0.0011
02644100022
0.309110.0340
Qwen2 5-14B-Instrect
0.031740.0115
0061R-00163
00997400297
0.8377000066
01756600315
0.210000.031
03660+00067
0.3111-00058
E.3 VALIDATION ON BLACK-BOX MODELS
Crucially, the choice of uncertainty heuristic affects only the efficiency (prediction set size) rather
than the validity (risk control guarantee), as the conformal calibration remains distribution-free A
21

---

Published as a conference paper at ICLR 2026
Table 8: Test-time EER results in the filtering stage uitilize LLM as correctness evaluation on the ScienceQA (a = 0.5) dataset at various risk levels (3).
0.45
0.10
0.15
0.20
0.25
1.30
0.35
11.40
Upper Bound to 8 o.)
0.525
0.55
0,575
0.6
0.62
0.675
0.7
LLaMA Instrcct
0.006410:0240
0.18761000021
027801100222 0.3415.0.0800
0.384400000
0,446_000293
0.8551000086
0.524110.0222
Openchae-3.5
0:0476+0.0117
0,8800000288
01284500224
0.1719-0.044
02134400257
0,2907=0.019
0:2955+00121
0,3 0,3888000399
Qwen2 Qwen2.--88-nniiiee 5-38-he
0.0575
(10766000171
01101400197
0.1875=0.0221
0.1706000248 =0,009
0:2515+00018
0,292+400113
stronger heuristic yields smaller prediction sets, while weaker one results in larger sets to satisfy the same coverage requirement.
To demonstrate the applicabiliity of SAFER to black-box models where internal logits are inacces- sible, we evaluate GPT-4o-mini on the CoQA dataset. We employ a consistency-base frequency score the uncertainty heuristic U-),, defined 1 Frequency (j), where frequency denotes the proportion of semantically equivalent answers within the sampled set.
Table and Table 10 present the empirical results for Stage and Stage II, respectively. In the sampling stage, the empirical miscoverage rate consistently remains below the target risk level a. In the filtering stage, with a fixed 0.25, the final empirical risk is strictly bounded by the theoretical limit (a++3 3 - o:3) across all total risk budgets. These results confirm that SAFER maintains rigorous statistical guarantees even in black-box settings.
Table 9: Empirical risk (Mean + Std) of Stage on CoQA using GPT-4o-mini across varying feasi-
bility risk levels (a)).
Empirical Risk
0.25
0.2293 + 0,0173
0.30
0.2462 + 0.0229
0.35
0.2745 + 0.0234
0.40
0.2934 0.0110
0.45
0.2960 + 0.0036
0.50
0.2960 + 0.0036
0.55
0.2960 + 0.0036
Table 10:
Empirical risk (Mean Std) of Stage on CoQA using GPT-4o-mini with consistency- based frequency scores. Experiments use fixed 0.25 and varying total risk budgets.
+ B a3
Empirical Risk
0,2875
0.2293 + 0.0173
0.3250
0.2293 + 0.0173
0.3625
0.2293 + 0.0173
0,4000
0.2293 + 0.0173
0,4375
0.2674 + 0.0834
0,4750
0.3897 + 0.1116
0.5125
0.4691 0.0852
0,5500
0.5244 + 0,0376
0.5875
0.5570 0.0323
0.6250
0.5913 + 0.0388
E.4 SENSITIVITY ANALYSIS ON SAMPLING BUDGET
The maximum sampling cap M serves as a critical hyperparameter governing the trade-off between inference cost and the system's 's ability uncover admissible answers. We analyze the impact of M on the abstention rate using the CoQA dataset, comparing two models of different sizes: Qwen2.55- 14B Instructt and 2-
22

---

Published as a conference paper at ICLR 2026
As presented in Figure 13 increasing M significantly reduces the abstention rate for both mod- els, demonstrating the utility expanding the search space. For instance, with Qwen2.5-7B, the rate drops from 37.22% at .M to roughly 22.3% at M 19. However, this benefit exhibits diminishing returns, saturating as M approaches 20.
Furthermore, the results highlight the impact of model capability: the larger 14B model consistently achieves lower abstention rates compared to the 7B model under the same budget. This indicates that while increasing the sampling budget can compensate for model limitations extent, stronger underlying model fundamentally improves the feasibility of risk control.
E.5 GENERALIZATION TO MULTIMODAL REASONING
To assess the framework's ' generalization capabilities beyond text-only tasks, we extended our eval- uation to MMVet (Yu et al., 2023), challenging multi-modal benchmark that requires integrated visual recognition and reasoning capabilities. We evaluated two state-of-the-aa vision-language (Wang al. 2024b) and Inter VL2-8B (Chen al. 2024).
models: Qwen2-VL-7B-
The results confirm that SAFER maintains strict statistical validity in multi-modal settings. Table presents the results for Stage I (Sampling), where the empirical risk consistently tracks the target risk levels (a.). Table 12 shows the results for Stage (Filtering) using bi-entailment as the correctness 0.1). Across all target risk settings, the final empirical risk
metric (with fixed Stage risk remains strictly bounded by the theoretical limit (a + a.3).
Table 11:
Stage results on VM The empirical risk (Mean + Std) for both Qwen2-VL-7B and Intern VL2-8B consistently satisfies the user-specified risk level (a).
Risk Level (a)
Empirical Risk (Qwen2-VL-7B)
Empirical Riskv (InternVL2-88)
0.10
0.0819 + 0.0275
0.0917 + 0.0153
0.15
0.0835 + 0.0376
0.0901 + 0.0337
0.20
0.1182 = 0.0398
0.1252 0.0375
0,25
0.1438 + 0.0337
0.1582 + 0.0453
0.30
0.1484 0.0259
0.1882 0.0400
0.35
0.1484 + 0.0259
0.1955 + 0.0275
0.40
0.1484 + 0.0259
0.1955 + 0.0275
0.45
0.1484 + 0.0259
0.1955 + 0.0275
0.400
Owen? 5-148 Atseett Rate
Owerr 5 148 Uppan Bound
Onver Acatention
0355
0
0.25
Abstcritonn
0.00
0.75
0250
0.25
Figure 13: Abstention rates and (lopper-Pearseee upper bounds on the CoQA dataset for Qwen2.5- 14B and Qwen2.5-7B across varying sampling budgets (M).
23

---

Published as a conference paper at ICLR 2026
Table 12: Stage II results on MMVet using bi-entailment for correctness evaluation. With fixed
Stage risk a 0.1, the final empirical risk stays below the theoretical upper bound (a: + 3 a:3) across varying total risk levels.
Risk Level (a + 3 - a.3)
Empirical Risk (Qwen2-VL)
Empirical Risk VL2-8B)
0.145
0.114 + 0.031
0.113 + 0.025
0.190
0.152 0.041
0,153 + 0.040
0.235
0.190 + 0.045
0.182 + 0.046
0,280
0.239 0.051
0,236 = 0.046
0.325
0.287 + 0.058
0.273 = 0.057
0.370
0.334 0.052
0.327 + 0.055
TriviaQA
### System: This is a bot that correctly answers questions.
### User: In 1968, who did radical feminist Valerie Solanas shoot and wound as he entered his New York studio?
### Assistant: Andy Warhol
### User: What lake can be found on the border of Vermont and New York?
### Assistant: Lake Champlain
### User: Which competition was won by Nadiya Hussain in 2015?
### Assistant: The Great British Bake-Off
### User:
Who the behind The Chipmunks?
### Assistant:
Figure 14: prompt example in the TriviaQA Dataset.
24

---

Published as a conference paper at ICLR 2026
CoQA
### System: This is a bot that correctly answers questions. Once upon a time, in a barn near farm house, there lived little white kitten named Cotton. Cotton lived high up nice warm place above the barn where all of the farmer's horses slept. But Cotton wasn't t alone her little home above the barn, oh no. She shared her hay bed with her mommy and 5 other sisters. All of her sisters were cute and fluffy, like Cotton. But she was the only white one in the bunch. The rest of her sisters were all orange with beautiful white tiger stripes like Cotton's 's mommy. Being different made Cotton quite sad. She often wished she looked like the rest her family. So one day, when Cotton found a can of the old farmer's orange paint, she used it to paint herself like them. When her mommy and sisters found her they started laughing. "What are you doing, Cotton?!" "I only wanted to be more like you". Cotton's mommy rubbed her face Cotton's s and said "Oh Cotton, but your fur is pretty and special, like you. We would never want you to be any other way". And with that, Cotton's mommy picked her up and dropped her into big bucket of water. When Cotton came out she was herself again. Her sisters licked her face until Cotton's 's fur was all all dry. "Don't ever do that again, Cotton!" they all cried. "Next time you might mess up that pretty white fur of yours and we wouldn't 't want that!" Then Cotton thought, "I change my mind. like being special".
### User:
What color was Cotton?
### Assistant: white
### User:
Where did she live?
### Assistant:
Figure 15: prompt example in the CoQA Dataset.
ScienceQA
### System:
This is bot that correctly answers questions.
### User: What are cell walls of fungi made of?
### Assistant: chitin
### User:
What is the pattern of spacing among individuals within the boundaries of the population?
### Assistant: dispersion
Fragmentation with subsequent regeneration is method of what, exhibited by animals such as sea stars?
### User:
### Assistant: asexual reproduction
### User:
Who proposed the theory of evolution by natural selection?
### Assistant:
Figure 16: A prompt example in the ScienceQA Dataset.
MMVet
<image>>
What x in the equation?
NOTE: Provide only the final answer. Do not provide unrelated details.
Figure 17: prompt example the VM Dataset.
25


Reviews:
[{"review_id": "kxeCJbpIv3", "reviewer": "Reviewer_8PGK", "summary": "This paper introduces SAFER, a two-stage framework for controlling the risk of miscoverage in open-ended question answering (QA) with Large Language Models (LLMs).  The core contribution is a practical and theoretically-grounded method for providing rigorous coverage guarantees for LLM outputs in open-ended QA, particularly by formally integrating an abstention option.", "questions": "Questions for the authors:\n\nThe abstention mechanism is a key component. Could you provide more insight into the practical trade-offs involved in setting the maximum sampling cap (M)? How sensitive is the framework's performance to this hyperparameter, and what is the relationship between M, the risk level ω, and the resulting calibrated sample size?\nThe filtering stage relies on an uncertainty measure U, which is sentence entropy in this work. How dependent is SAFER's performance on the quality of this uncertainty measure? Have you explored how the two-stage guarantees hold up when using other uncertainty heuristics, especially those that might be less correlated with correctness?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "One of the main strengths of this work is its sound technical approach to a challenging problem. The formulation of a two-stage process that independently controls risk at both the sampling and filtering stages is elegant. The introduction of an explicit, statistically-grounded abstention mechanism is a significant step forward from prior methods that often assume an admissible answer is always attainable. This makes the framework more practical for real-world, risk-sensitive applications. The authors validate their method with experiments across multiple datasets, LLMs, and several different correctness evaluation metrics, demonstrating the robustness and data-efficiency of their framework.", "weaknesses": "A primary concern is the clarity of the novelty. While the application of abstention to this specific two-stage QA framework is new, the paper could better situate its contribution within the broader literature on conformal prediction and abstention, where similar concepts have been explored. The presentation is quite dense and assumes a high level of familiarity with conformal prediction, which may limit its accessibility to a wider audience. Furthermore, while the experiments are quite broad, they could be more rigorous. The analysis focuses on demonstrating that the empirical error rate is below the theoretical bounds, but a deeper analysis of the abstention cases or a more direct comparison with a wider set of non-conformal baselines would strengthen the claims of practical utility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5YtNXSQjiD", "reviewer": "Reviewer_mzfb", "summary": "This paper introduces SAFER, a two-stage framework for uncertainty quantification in open-ended question answering with LLMs. The framework consists of: (1) abstention-aware sampling that calibrates a minimum sampling budget using the Clopper-Pearson exact method, abstaining when risk constraints cannot be satisfied, and (2) conformalized filtering that removes unreliable candidates using conformal risk control. The authors provide theoretical guarantees that miscoverage probability is bounded by α + β - αβ and validate the approach on three QA datasets with five LLMs.", "questions": "1.What are the abstention rates across different datasets, models, and risk levels? How does abstention correlate with model capability or question difficulty?\n2. What is the wall-clock time cost compared to baseline sampling approaches? How does this scale with M and ŝ?\n3. Can you provide practical guidance or heuristics for selecting α and β for different applications? What are reasonable default values?\n4. How does SAFER perform when calibration and test distributions differ (e.g., temporal shift, domain shift)? This seems critical for real deployment.\n5. How does SAFER compare quantitatively to ConU, LofreeCP, and other recent methods on the same datasets?\n6. Have you evaluated alternative uncertainty measures? How much does performance depend on this choice?\n7. How does the framework handle questions with multiple semantically distinct but correct answers?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Unlike prior work (TRON, ConU), SAFER doesn't assume all instances can yield correct answers within finite sampling, incorporating a principled abstention mechanism that makes the framework more realistic for deployment.\n2. The paper provides formal statistical guarantees with detailed proofs, properly applying the Clopper-Pearson exact method for finite-sample bounds and extending conformal risk control to the filtering stage.\n3. The experimental validation is comprehensive\n4. The filtering stage substantially reduces prediction set sizes (e.g., 7.9 to 5.5 for β=0.1 on TriviaQA) while maintaining statistical validity, improving practical usability.", "weaknesses": "1. The main components (Clopper-Pearson bounds, conformal risk control) are existing techniques. The contribution is primarily in their combination with abstention handling rather than fundamental methodological innovation.\n2. Relies on exchangeability between calibration and test sets with no empirical evaluation of robustness to distribution shift\n3. Requires users to specify both α (sampling risk) and β (filtering risk). The combined bound α + β - αβ is less interpretable than a single risk parameter\n4. No principled guidance provided for parameter selection in practice\n5. What fraction of instances require abstention? This is crucial for assessing practical utility\n6. No wall-clock time comparisons or discussion of overhead\n7. Only compares against TRON; missing comparisons with ConU, LofreeCP, SE-SCP, and other recent conformal prediction methods for open-ended QA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KXbZW5jw97", "reviewer": "Reviewer_z2a5", "summary": "The paper proposers Sampling and conformalized filtering  that use conformal risk control into two stage framework. The first stage calibrates sampling budget for responses candidate set to contain admissible answers at miscoverage rate $\\alpha$. The second stage introduces second risk level $\\beta$ to control probability of incorrectly filtering out of all admissible responses to produce a prediction set. The final prediction set fails to cover an admissible answer with probability $1-delta$ over the calibration set with bound $\\alpha + \\beta - \\alpha\\beta$. Experiments on three QA datasets (TriviaQA, CoQA, ScienceQA) using five different LLMs demonstrate that SAFER validly controls the empirical error rate below the specified bounds.", "questions": "* Is stage I just straightfoward application of LTT and stage II just application of Conformal Risk Control? Can you clarify core contributions beyond just direct application of existing frameworks?\n* Is there bias in sequential calibration when stage II calibration is only performed on biased subset of admissible samples from stage I? This seems like form of conditional coverage and is not discussed. How do you justify that this biased selection step does not break exchangeability?\n* How to choose $M$ and why are ablation experiments not conducted on such a critical hyperparameter?\n* How to extend framework to black-box API-only models without logit access?\n* How does the quality of the uncertainty heuristic $U(u)$ affect the efficiency of the final prediction set size? \n* Can the 3D plots be visualized in a better way? \n* Can the sampling budget be made adaptive to improve efficiency based on example difficulty or some other criteria?\n* Can the two-stage  $\\alpha + \\beta - \\alpha\\beta$ be made tighter? \n* Could the calibration of $\\alpha$ and $\\beta$ be done jointly rather than separately? \n* How does Clopper-Pearson (classical statistical tool) compare to alternative calibration approaches to find confidence intervals?\n* How does this framework extend to when admissibility is fuzzy or difficult to measure exactly? \n* The framework requires the user to specify two risk levels, how should a user practically choose $\\alpha$ and $\\beta$ to target a specific overall risk level $\\epsilon$ between sampling and filtering?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The abstention aware sampling is mechanism is the main novelty. Should clarify and compare with Yadkori 2024. Is this just an application/extension of existing works?\n* The bound on the sampling risk looks to be formally derived and empirically validated across several LLMs and datasets. \n* The evaluation of different correctness criteria (sentence similarity, Rouge-L, NLI-based bi-entailment, and LLM-based semantic evaluation) add empirical support", "weaknesses": "* The paper is incremental in terms of technical contribution, combining existing LTT and conformal ideas with abstention-aware calibration.\n* The calibration for Stage 1 requires generating/evaluating $N\\timesM$ samples and evaluating these responses against the ground truth could be very computationally expensive.\n* The maximum sampling cap M is a key hyperparameter that is not explored in detail.\n* Although many related CP works are cited, the experiments ignores them completely and only compares against single TRON baseline.\n* Only QA datasets are used, no evaluation on reasoning, summarization, or multi-modal tasks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)