Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification
Authors: Kimia Hamidieh
Veronika Thost
Walter Gerych
Mikhail Yurochkin
Marzyeh Ghassemi
Venue: ICLR 2026 Poster (2026)
Keywords: Uncertainty Quantification
Language Models
Epistemic Uncertainty
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We show that epistemic uncertainty, estimated via cross-model semantic disagreement, complements aleatoric uncertainty and improves reliability in LLM predictions.

Abstract: Large language models (LLMs) often produce confident yet incorrect responses, and uncertainty quantification is one potential solution to more robust usage. Recent works routinely rely on self-consistency to estimate aleatoric uncertainty (AU), yet this proxy collapses when models are overconfident and produce the same incorrect answer across samples. We analyze this regime and show that cross-model semantic disagreement is higher on incorrect answers precisely when AU is low. Motivated by this, we introduce an epistemic uncertainty (EU) term that operates in the black-box access setting: EU uses only generated text from a small, scale-matched ensemble and is computed as the gap between inter-model and intra-model sequence-semantic similarity. We then define total uncertainty (TU) as the sum of AU and EU. In a comprehensive study across five 7-9B instruction-tuned models and ten long-form tasks, TU improves ranking calibration and selective abstention relative to AU, and EU reliably flags confident failures where AU is low. We further characterize when EU is most useful via agreement and complementarity diagnostics.

Published as a conference paper at ICLR 2026
COMPLEMENTING SELF-CONSISTENCY WITH CROSS-MODEL DISAGREEMENT FOR UNCERTAINTY QUANTIFICATION
Kimia Hamidieh "; Veronika Thost,, Walter Gerych Mikhail Yurochkin Marzyyh Ghassemi
MIT 2 MIT-IBM Watson Al Lab 3PI, IFM MBZUAI
ABSTRACT
Large language models (LLMs) often produce confident yet incorrect responses, and uncertainty quantification is one potential solution more robust usage. Recent works routinely rely on self-consistency to estimate aleatoric uncertainty (AU), yet this proxy collapses when models are overconfident and produce the same incorrect answer across samples. We analyze this regime and show that cross- model semantic disagreement is higher on incorrect answers precisely when AU is low. Motivated by this, we introduce an epistemic uncertainty (EU) term that operates in the black-box access setting: EU uses only generated text from a small, scale-matched ensemble and is computed the gap between inter-model and intra- model sequence-sent sementii similarity. We then define total uncertainty (TU) as the sum of AU and EU. In comprehensive study across five 7-9B instruction-tus -uuned models and ten long-form tasks, TU improves ranking calibration and selective abstention relative to AU, and EU reliably flags confident failures where AU is low. We further characterize when EU is most useful via agreement and complementarity diagnostics.
INTRODUCTION
Reliable uncertainty estimates are a prerequisite for deploying large language models (LLMss in high-stakes domains (Chakraborti et al., 2025). Many existing approaches for L.LM uncertainty estimation are based model's self-confidence (Xia et al., 2025; Vashurin et al., 2025; Shorinwa et al., 2024), such as by measuring response consistency under sampling (Kuhn et al., 2023a; Lin et al., 2023; Nikitin et al., 2024: Aichberger al., 2024) querying for verbalized uncertainty score (Lin et al., 2022). These metrics capture how internally confident a model in its prediction notion of predictive aleatoric uncertainty (AU). But this leaves important question unanswered: how confident should we be in the model? A model might be confident but wrong, such as responding with the same incorrect answer with high probability (see Figure 1). In these cases, methods that rely on self-consistency can fail (Kossen et al., 2024). To address this, we focus on estimating epistemic uncertainty (EU) uncertainty our choice of model which better reflects whether model's confidence is trustworthy for a given input.
Estimating EU requires evaluating a distribution of plausible models, which is prohibitively costly for LLMs, as training even one additional model adds significant overhead (Kirsch, 2024; Cottier et al., 2024). Recent shorteuts approximate EU in logit space (Ma et al., 2025), inject Bayesian noise during decoding (Liu et al., 2025; Gao et al., 2024), or rely on modefi disagreement (Xue et al., 2025), but each imposes strong task or prchitecture-specifi assumptions. We instead capitalize on the ecosystem of open-weight LLMs: sampling responses from small, scale-matched ensemble lets us estimate EU directly from cross-model semantic disagreement, without additional training. While prior work has shown that LLM ensembles improve accuracy (Lu et al., 2024; Dey et al., 2025; Chen et al., 2025; Shnitzer et al., 2023; lelanskyi et al.), their use for uncertainty quantification has not been systematically explored.
Correspondence to hamidich mit.cdu. Work done while MIT. Work done while at the MIT-IBM Watson AT Lab.

---

Published as a conference paper at ICLR 2026
Who was awarded the Oreanogrophy
Target: Annick Bricoud
Sample Respenses:
Society's Jerlov
Award in 20187
Avg. Accuracy
GDT-4o
Claude 3.7
Sonnet
GFT 40
0.589
0.703
0.641
09.5
Dr. Raymond C. Snith
Drv Kenneth J. Voss Kenneth Voss
Reymond Smith
Claude Sumnet
0.581
Reymond Snith
Kenneth Voss
Abateric AbRoc
Total AUROIC
Eplenmat AUROC
0.0
0.5
100
Self-coreinseeccy:: high
Accuracy
Inter-model
EU: high
disagrsement
Figure 1: (a) Two models confidently produce distinct, incorrect answers factual question, which results in low intra modal variability (AU) but high semantic disagreement across models (EU). (b) Total uncertainty (TU AU EU) effectively improves uncertainty calibration with correcness in terms of AUROC on S SimpleQA.
By enabling scalable estimation of EU from model outputs alone, we can combine with AU to obtain more robust measure of uncertainty. Total Uncertainty (TU). These two forms of uncertainty are complementary; AU reflects variability in a model's s own predictions, while EU measures divergence from other plausible models (Schweighofer et al., 2023). Together, they allow TU to account for both internal inconsistency and external disagreement.
We evaluate EU, AU and TU on two standard axes: ranking- based calibration (via AUROC) and selective prediction (via abstention under uncertainty thresholds), across a range of models and generation tasks. We conduct comprehensive experiments across five 7-9B parameter Instruction- tuned models (Yang et al., 2024; Jiang et al., 2023; Grattafiori et al., 2024; Granite Team, 2024), on ten long-form generation tasks spanning QA, summarization, translation, and math reasoning (Joshi al., 2017; Yang et al., 2018; Narayan et al., 2018). We also repeat these experiments for API models such as GPT-4e (Hurst et al., 2024) and Claude 3.7 Sonnet (Claude) on SimpleQA (Wei al., 2024). Our contributions are follows:
We diagnose the failure mode of self-consistency as an aleatoric proxy: it often collapses on confident errors, or prompts where the model produces the same wrong answer repeatedly.
We introduce epistemic term based on cross-model semantic disagreement within a small,
scale-matched ensemble, and show that reliably identifies confident but incorrect responses.
We conduct extensive empirical study across ten long-form generation tasks and five reference models, and show that TU consistently outperforms AU in both AUROC and selective abstention. We show that the proposed EU is most informative in tasks with unique correct answer, such as factual QA and translation.
RELATED WORKS
Aleatoric Uncertainty in LLMs. Existing approaches mainly focus AU, which captures response inconsistency or input ambiguity. Recent surveys provide extensive reviews of these methods (Xia al., 2025; Shorinwa et al., 2024; Vashurin al., 2025). Typical strategies involve sampling multiple responses per prompt and analyzing their semantic consistency, often through clustering or entropy- based metries (Lin al., 2023). In line with prior evaluations, we adopt degree-based semantic dispersion (Lin et al., 2023) as the AU baseline used throughout this paper.
peyesian-inspiree EU Estimation. A line of research employs Bayesian -nnhiirn----ieedd methods, such as adding noise to embeddings during generation to approximate uncertainty in model weights (Liu et al., 2025), sampling from a model with different temperatures (Gao et al.. 2024), or leverage entropy from decoding from different hidden states as proxy for uncertainty, which provides computationally efficient alternative to exhaustive sampling (Gao et al., 2025). Training ensembles explicitly, such as LoRA-basee methods (Wang al., 2023), demonstrate improved uncertainty calibration but incur significant computational costs. Another work (Ma et al., 2025) calculates AU and EU on token level by considering the LLM logits parameters of Dirichlet distribution and by applying other UQ methods subsequently. Unlike approaches that require logits or hidden states, whereas our estimator operates only on generated text (Eq. 3), which enables application black-bor models.
2

---

Published as a conference paper at ICLR 2026
Prompt-Based and Verifier-Basel EU Estimation. Prior works in iterative prompting (Abbasi Yad- kori et al., 2024; Johnson et al., 2024) estimate epistemic uncertainty by iteratively querying the same model, adding previous responses the later queries' prompts, and measuring probabilistic inconsistencies. However, these methods have limitations: gains over AU are mainly reported on multi-label data, with limited benefits on standard single-label QA (Abbasi Yadkori al., 2024), and some are evaluated only on synthetic data (Johnson et al., 2024). Xue et al. (2025) utilizes one verifier LLM and shows that inter-model disagreement in cases where we reach the performance bounds of self-consistency Their practical rule triggers cossssonsittency only at intermediate AU, whereas our evaluation, and (Simhi et al., 2025), indicates that low AU is especially prone to hallucination and is best complemented by EU (Sec. 5.1). Prior works are mostly limited to special kinds of question-answering data, do not account for the impact of the vast L.M model space on EU, and do not fully explore interactions between AU and EU, gaps our work addresses explicitly.
proxy for EU complements AU
LLM Ensembles. Our work builds upon classical uncertainty estimation, such as deep and dropout ensembles (Lakshminarayanan et al., 2017: 7: Gal & Ghahramani, 2016) and is closely related to LLM ensemble applications (Lu al., 2024). In particular, various recent works focus LLM collaborations (Dey et al., 2025), verifier LLMs (Lifshitz et al., 2025), and sampling from multiple LLMs (Chen et al., 2025). We study LLM ensembles from the viewpoint of uncertainty estimation.
3 QUANTIFYING PREDICTIVE UNCERTAINTY USING RESPONSE SIMILARITY
Let be the particular parameterization of an LLM, and let x be a prompt. Our goal is quantify the predictive uncertainty of w given x as input. As is standard, we categorize predictive uncertainty into two components: AU and EU (Hüllermeier & Waegeman, 2021). The aleatoric component captures the inherent unpredictability of the response to x under the model w, while the epistemic component captures our uncertainty in w being the correct parametrization to use when responding input x. We define total predictive uncertainty additively as the sum of the aleatoric and epistemic uncertainties.
3.1 ALEATORIC UNCERTAINTY VIA INTRA-MODEL RESPONSE SIMILARITY
Many recent works have proposed techniques to measure the randomness in LLM responses (Kuhn et al., 2023b; Lin et al., 2023; Liu et al., 2024). These techniques typically focus on measures of semuntic uncertainty, where uncertainty
defined as a function of how often an LLM produces semantically distinct outputs given the same input (Kuhn et al., 2023b; Lin et al., 2023). In particular, Lin et al. (2023) propose a measure equivalent to the following:
Ualeatoric (z; w) = (x.0)
- s(TT. rr))
(1)
where s(.,)) :) is a similarity metric for responses such as the cosine similarity in an embedding space. In essence, Equation corresponds to the expected similarity between two responses independently sampled from p(- w) the response distribution of w conditioned on æ. If responses typically have the same semantic meaning each other, meaning that the meaning of the response does not vary when resampled, then aalatorri (I; w) will be close to O, which means that there is little uncertainty in how w will respond to æ. If the model likely to produce semantically distinct responses for the same input, then Ustestoric (z:w) will be high, which means w has high uncertainty for x.
Equation| 1 captures the inherent uncertainty in the response to x given model w. However, w may not be the optimal model to use for x, and Equation 1 fails to capture the inherent uncertainty that comes from choosing w as our parameterization. There is thus a need to also capture the epistemic uncertainty that comes from our model choice.
3.2 EPISTEMIC UNCERTAINTY AS ANTER-MODEE RESPONSE SIMILARITY
Let w² represent a hypothetical "ideal" model, such that p(-la;=") = p(-[r); the distribution of responses from w" equals the true response distribution. We can thus quantify the epistemic
'hhi is equivalent to UDgg in Lin et al. (2023).
3

---

When the average similarity in responses between w and other sampled models matches the self- similarity of w's responses, then the semantic distribution of w matches the target distribution and the epistemic uncertainty is low. When there is a mismatch between the average similarity w's responses to the responses from the sampled models W compared to w's self- self-similarity, then there a disagreement in how models respond and the epistemic uncertainty is high. In Appendix A. Published as a conference paper at ICLR 2026
uncertainty of w divergence between and w": e.g., Uepistemie w) = D(w || w*) (Schweighofer et al., 2023). We define D as follows:
D(w 11=") = Eog mpt (ow- ( oy -pl p ( s( ,92
Beipp(((za Brg-p(( ((r)) x2))
cross-model similarity
self-smmilarity (1 At)
In effect, D(w |I w*) measures the difference between 1) the similarity of responses from w and w" (Io) ~p(-[x,w)) [s( 95 9)) and 2) the self-similarity of w (22p Bry Bry-p(-laww) ((rf,,r))] r))l). In the case where is optimal and equivalent to w", then D(w || w") will be O. When w produces responses that are semantically diverse from the ideal model's responses even after accounting for the diversity due to w's alcatoric uncertainty, then D(w || w*) will be high. In practice, we do not have access to the optimal model w". Instead, we can leverage a recent
technique (Schweighofer et al., 2023) and marginalize out w*. Let Pa be a distribution over models such that En [p(- x;; [)] = p(: [.). We can thus replace w" in Equation with an expectation over Pit, and define U Uppistemic (I, w) as:
U Uepistemic (I, Pa [Bor~ -pp an) Bof (oof-p(-laa0 (m. [>(0] }
(3)
+ lE EE ppl--p((laa 18 B ~p( (( r
provide detailed interpretation of D(w w*) as one-sided kernel discrepancy, establish its connections to variational inference, and show that it is upper bounded by total variation distance under mild conditions.
Desired Properties for Ω. Because the divergence ((w || ( is evaluated against samples drawn from surrogate distribution of models 12, its fidelity hinges on how well that ensemble of models approximates the (inaccessible) optimal distribution p(- w, Three criteria follow from the definition in Eq. 3:
(i) Support richness. Ω covers distinct yet plausible interpretations of models, rather than a narrow subset; otherwise the similaiiyy term in D may be artificially high, and D underestimates EU when predictions of models in Ω are different from w's predictions.
(ii) Non-collapsing diversity.
all members of Q are nearly identical (e-g- noise-perturbae versions of the same model), the ensemble average would be too close to hence the cross- model similarity term will be close to self-similarity and D may be small, even when the candidate predictor P.,. is mis-specified.
(iii) Calibrated weighting. Let Ps denote the mixing measure over models. For Eq. 3 to approach the ideal p(y z). each model should be weighted in proportion its posterior eredibility (e.g. uniform weights are appropriate only when validation risks are comparable).
Achieving Properties via Cross-Famill Models. A practical way to satisfy these criteria is to construct the surrogate ensemble Ω from models of similar architecture and scale, likely trained on overlapping or similar pre- pre-training datasets. Specifically, we populate Ω with 7-9B Transformer based models that share the same architecture class but are trained by different vendors. This setup ensures (i) support richness, as models differ in data pipelines, initializations, and alignment protocols, which results in diverse but plausible responses for the same input, that cover the ground-truth response set. These independently trained models also exhibit (ii) non-collapsing diversity, as their differences arise from different design choices, rather than perturee versions of a single model. Finally, because these models achieve similar validation performance, we adopt uniform weights in Pa. which satisfies the calibrated weighting (iii) requirement. Section 4 specifies the exact models used.
Total Predictive Uncertainty. We make the standard assumption that total predictive uncertainty can be obtained by adding aleatoric and epistemic predictive uncertainties (Hüllermeier & Waegeman,
4

---

Published as a conference paper at ICLR 2026
2021). Thus, define Utotal (2; w) as:
Usotal (2) = Uatentoric (z; w) + U Vepistemic (1; w)
Eis B ap(-) B of up (1, - sr)]
3.3 EMPIRICAL ESTIMATES OF UNCERTAINTY METRICS
For given input prompt v; we call the model whose uncertainty is being estimated the reference model w, and denote the set of models used compute epistemic uncertainty with respect to the reference as the auxiliary model set Q. Throughout the paper, we mainly focus on Cross-famlly auxiliary models: We estimate epistemic uncertainty by computing response divergence across an auxiliary set of models. To estimate uncertainty in practice, we proceed as follows:
Sample n responses from each model wi S2, and denote the set responses from w as R' = r2,,,2,.... r.y) and from w; as R₂ = {ri 22 ,..... , (t)
where |S]] m.
2. Approximate Alcatoric, Total, and Epistemic Uncertainty using these sampled responses.
Empirical Uncertainty Metrics
AU Ualeatoric
s(r r))]/n²
k=1j=1
TU Uwial
=
Σ
s(rk: /n?
k=
EU U Uepistemic
Unotal
U Ualentoric
Note that we assign uniform weights to different models in the auxiliary set, as we choose to use models of similar capabilities, and our estimate of AU is similar to the one from Lin et al. (2023). Also, observe that our evaluation shows that we can keep the overall number sampled responses at the magnitude used by self-consistency based methods while improving over those. More concretely, n' we choose n m when comparing to AU as standalone metric, where n' is the number of samples used for the latter.
EXPERIMENTAL SETUP
Models. In the main experiments, primarily focus on five instruction-tuned lan- guage models with approximately 7-9B parameters: Cemma-2-9B-28 (Team et al., 2024), Granite-3. 31-.33.-----sstttte (Grattafiori et al., 2024), Mistral -arraal--tt-nnttttttttta-o
(Granite Team, 2024), L1 amamm-3.1. 1.---3.-1----neeee (Jiang et al., 2023).
Qwen2.5-7R-TTe ruct (Yang et al., 2024). We compute the uncertainty measures from Sec- tion 3.3 and consider the models mentioned above as the set of auxiliary models. Appendix A.5, we also consider larger reference models. Unless otherwise noted, we compute TU by sampling 2 responses from each of the 5 models and similarly compute AU using 10 samples to keep the sampling budget the same across the two metrics.
Datasets. Our experiments cover a broad range of long-form generation tasks spanning ques- tion answering (QA), math reasoning, translation, and summarization. For QA, we include AmbigQA (Min et al., 2020) (open-domain QA with both anbiguous and unambiguous ques- tions), NQ-open (Kwiatkowski HotpotQA (Yang et al., 2018) (multi-hop QA requiring reasoning over multiple supporting docu- ments), CoQA (Reddy et al., 2019) (conversational QA with multiple turns), QASPER (Dasigi et al., 2021) (fact-based QA over long scientific papers), TriviaQA (Joshi et al., 2017) (QA based on styve questions), and TruthfuiQA (Lin et al., 2021) (QA to evaluate common misconcep- tions in models). For math reasoning, prompting. For language generation, we evaluate on the German to- English translation dataset WMT1 6-de-en (Bojar al., 2016) and the summarization benchmark X5um (Narayan et al., 2018).
al., 2019) (closed-boot QA derived from real user queries),
GSMBK (Cobbe et al., 2021) with tfain-of-thought
5

---

Published as a conference paper at ICLR 2026
Epistemic Uncertiinty
0.3
Correct
Epistemic Uncertainty
0.3
median 0.13
0.2
0.2
0.1
0.0
Low
Mid Aleatoric Bucket
Hlihh
0.0
Incorrect
Correct
(a) We stratify examples by AU (low, mid, high; 33% each) and compare the distribution EU for correct and incorrect generations. Incorrect responses show higher EU in the low-AU regime, but this separation weakens AU increases.
We isolate the most confident samples (lowest 5% AU) and find that incorrect generations have significantly higher EU than correct ones, which shows that EU effectively flags confidently wrong predictions.
Figure 2: Based on the distribution of EU across samples with different AU values, we find that EU separates incorrect from correct most strongly when AU low.
We additionally include SimpleQA (Wei et al., 2024), a factuality QA benehmark, with model responses generated by GPT-4o (Hurst et al., 2024) and Claude 3.7 Sonnet (Claude). Finally, we adapt tasks from the BBH multiple-chote choice benchmark (Suzgun al., 2022) to long-form format and add those evaluations to Appendix A.7.
Evaluation.
Correctness is defined per input-response pair using Meta -aaaalll- ama 3 3 -00-770--nnttte uncertainty estimation, j.M-ae-a-judge correctness evaluation has recently been shown to be the most reliable among the existing methods (Santilli et al., 2025). Following prior work (Lin et al., 2023; Kuhn et al., 2023a; Band et al., 2022), we evaluate the quality uncertainty by quantifying how well uncertainty scores separate correct from incorrect generations, using Area Under the ROC Curve (AUROC). Formally, AUROC corresponds to the probability that a randomly chosen incorrect response receives higher uncertainty score than a randomly chosen correct one.
judge (Appendix A.10). Note that, in the context of
We also evaluate effectiveness terms of selective prediction using Risk-Coverage Curves (Nadeem al., 2009), which measures how the error rate changes uncertain responses rejected. We further report standard summary metrics such as accuracy at 90% and 80% coverage (C@90 and C@) and the Area Under Risk-Coverage Curve (AURC), where lower better.
Baselines. As our primary aleatoric baseline, we use Lin et al. (2023)'s implementation of Aleatoric Uncertainty, which in practice similar Semantic Entropy (Kuhn et al., 2023a), and has been shown to perform well in recent benchmarks and surveys (Fadeeva et al., 2023; Vashurin et al., 2025). In our evaluation, we denote this baseline as Aleatoric or AU. We also experiment with noise-perturbed models (i.e., instead of models from different model families), similar to the approach of Liu al. (2025), see details in Appendix A.5.
5 RESULTS
5.1 EPISTEMIC UNCERTAINTY FLAGS CONFIDENT FAILURES OF ALEATORIC UNCERTAINTY
Language models are often applied to heterogeneous tasks, where model confidence does not always align with correctness (Zhou et al., 2024). To simulate such setting, we construct aggregated dataset by combining datasets mentioned Section 4, and analyze uncertainty trends on this pooled distribution. We particularly interested in identifying failure modes of AU.
In Figure 2a, we stratify examples by AU (low, mid, high) and compare EU across correct and incorrect responses. In the low-AU regime, incorrect responses exhibit higher EU than correct ones, which shows that EU is discriminative when aleatoric scores are overconfident. This separation diminishes in higher AU buckets, where both response groups become directly target this failure mode, we isolate the lowest 5% of AU scores and analyze EU by correctness. (Figure 2b). EU remains significantly higher for incorrect generations, which confirms that epistemic
uncertain. To more

---

Published as a conference paper at ICLR 2026
uncertainty flags confidently wrong outputs that aleatoric scores alone miss, which supports hypothesis of the complementary nature of scores this particular AU region.
This result contrasts with prior work, which treats low-AU predictions reliable and only incorporates cross-model comparisons when AU exceeds a threshold (Xue et al., 2025; Chen et al., 2025). Our findings reveal that this assumption overlooks critical failure mode: confidently wrong predictions with low AU. On the other hand, our observations validate findings about models being overconfident on HotpotQA (Ni et al., 2024) in that incorporating EU yields large improvements this dataset (see Figure 4 in Section 5.3). Per-dataset results are provided in Figure 18 in Appendix A.8.
5.2 EPISTEMIC UNCERTAINTY, AGREEMENT, AND DIVERSITY
We ask when similarity- EU most informative. To this end, we focus on the correctness of responses of different models and consider two metrics: Jaccard Agreement (or Redundancy) (J), which measures the overlap between predicted correct responses of the auxiliary models, used to quantify how redundant similar different predictions are; and Oracle Coverage Gain (also Complementarity) (G), the additional coverage (i.e., improvement in accuracy) obtained by an oracle that always chooses the correct model per example, over the best performing model. Exact definitions can be found in Appendix A.2.
Epistemic uncertainty does not always coincide with inter-model disagreement. Figure plots EU AUROC against the dataset-level statistics and G. We observe a positive correlation with redundancy (r +0.72, p 0.03) and a negative correlation with complementarity (r -0.72, p = 0.03), which is the opposite of the naive intuition that "more disagreement + higher epistemic utility."
The explanation lies in how EU is constructed: it grows with the divergence of generated answers, which arises in two distinct cases: (i) true EU on intrinsically hard questions where models do not know the answer, and (ii) the existence of many semantically different but correct responses (response noise). In complementary datasets (large G), each model specializes on different niches and consequently. EU is large even questions that an individual model answers correctly, because models in the auxiliary set return alternative (wrong) responses. In such cases, the misalignment with correctness drives AUROC down. Conversely, in redundant datasets (high J, low G), models converge to similar responses when correct (EU low) and, what we expect to be the usual case, still diverge when collectively wrong (EU high), which gives well-separated score and high AUROC. These observations characterize the cases in which our current EU estimator effective: tasks with single (or near-unique) correct answer, where models phrase that answer similarly yet generate diverse alternatives on the harder, unanswered inputs.
For example, WMT1 6 6-de-en and CoQA occupy the high-J, low-G corner of Figure 3; all models score above > 90% accuracy, so predictions are largely redundant and EU achieves its strongest
(r - 0.68, p 0.029)
(r--0.67, p 0.032)
Epistemic Uncertainty AUROC
Epistemic Uncertainty AUROC
WMTTl-deen
0.75
0.75
0.70
0.70
CADA
CorporIA Bivaza
0.65
0.65
0.60
0.60
DASPEENG-or N-upan soe
NQupen
0.55
0.55
SMEK
0.50
0.50
AnbigoA
0.45
0.45
04
0.6
0.8
0.05
0.10
0.15
0.20
0.25
Jaccard Agreement (Correct)
Oracle Coverage Gain
Figure 3: Epistemic uncertainty AUROC versus dataset level redundancy (J) and complementarity (G). Higher AUROC indicates better discrimination between correct and incorrect answers by EU.
7

---

Published as a conference paper at ICLR 2026
Alcatoric
0.5
ago
0140
Total
AUROC
122
ego
0g1
014
0.6
0.4
AmbigQA
CoQA
CSMSK
HotpolQA
NQ-open
QASPER
TriviaQA
TratffuQA
WMT16-de-en
XSum
Figure 4: Under matched sample budgets, TU (AU+EU) consistently shows higher AUROC than AU across datasets, with the largest gains on Hotpot QA, QA, CoQA, and WMT1 AMT16-de d-ddeea en (A >0.10). Bars show means over five 7-9B reference models with per-dataset dots.
discrimination. At the opposite extreme, XSum combines low accuracy with the largest G: models succeed on different inputs and can express many valid summaries, which inflates EU without improving ranking and thus lowering AUROC. Datasets such as HotpotQA and Tri viQA sit mid-range on both axes and have enough redundancy to suppress noise, but have sufficient diversity to expose disagreement and consequently produce the large TU gains in Figure 4.
5.3 TOTAL UNCERTAINTY IMPROVES CORRECTNESS CALIBRATION
Figure 4 reports the AUROC between negative uncertainty and correctness across datasets, and averaged over five 7-9B instruction-tun models mentioned in Section TU consistently improves over AU all benchmarks on average. The largest gains occur Hotpot (+0.15), CoQA (+0.14), and WMT16 6 de-enn en (+0.13), where models either disagree on complex multi-hop reasoning (AotpotQA) or achieve high overall accuracy (CoOA, WMT1 6-de-en), which allows EU to capture remaining errors.
Moderate improvements observed TriviaçA, and NQ-open, which exhibit balance of response redundancy and complementarity. In contrast, gains are more limited on fultf GSM8K (with thain-of-thoughe and QASPER, ER, where the presence of multiple valid stylistically diverse answers weakens the alignment between TU and correctness. These results align with the patterns described in Section 5.2: TU and EU are most effective when correct answers uniquely phrased and shared across models, while incorrect predictions remain diverse.
We also find that TU estimates consistently improve AUROC compared AU in GPT-4o (0.70 vs. 0.59), and Claude 3.7 Sonnet (0.58 vs. 0.53) on S Simples impleQ0, as shown in Figure 1. Figure 11 in Appendix A.4 shows the ROC curves on the combination of all datasets, where the relative ranking
0.8
6:23
AUROC
0,42
d,z
0
0.57
0.4
Total
Closenes Contrality
Alvatoric
Score
SC Based VC
SC+ VC
SemantieEntropp
Max Sequence Proh.
Mean Token Entropy
Max Token Prob.
Perploiity
Self Certainty
PTrae
Figure 5: Using - v0. 3 as the reference model, TU attains the best mean AUROC (0.72), and outperforms the strongest baseline (closeness centrality, 0.64) across almost all datasets. Per-task results appear in Table 3 in Appendix A.4.
8

---

Published as a conference paper at ICLR 2026
Size relative to reference (7B)
Size relative reference (7B)
0.07
0.3
0.14*
0,57
3,86
Auscllary Model Accuracy
Auxiliaay Model Acciracy
0.85
05
0.85
06
AUROC
0.4
AUROC
0.5
0.90
03
0.80
TU
AU
Auxiliary
Auxillary Acc.
0.75
0.75
002
Qwen2.5 Model Size (l)
Gemma3 Model Size (B)
Figure 6: We keep the reference model fixed as mistral-77 and vary the size of the single auxiliary model. TU achieves higher AUROC in comparison to AU, even in cases where size of the auxiliary model is lower than ( (x0. (x0.43) or roughly the same (x1) the reference model. The improvements are more significant with larger and more capable the auxiliary models on TriviaQA.
of data points across the whole dataset determines performance, and Table reports AUROC per model-dataset pair. We show that the improvement over AU is maintained in individual datasets, and in the combination of all datasets.
Comparison to Baselines. We compare TU against a number of baselines: Mean Token En- tropy (Fomicheva et al., 2020), Maximum Token Probability (Fadeeva et al., 2023), Maximum Sequence Probability (Fadeeva et al., 2023), Perplexity (Fomicheva et al., 2020), PTrue (Kada- vath et al., 2022), Self-Certainty (Kang et al., 2025), Semantic Entropy (Kuhn et al., 2023a), SC Seeff-Consistency) Score (Wang al., 2022; Manakul al., 2023), Closeness Centrality, SC + VC (Verbalized confidence), and SC based CV (Jiang et al., 2024). Figure 5 shows results for Mistral-77 Inst vC as the reference model across benchmarks and baselines. Ta- ble 3 in Appendix A.4 shows that TU outperforms the strongest baseline across almost all benchmarks.
Ablations. We further ablate the size the reference model Appendix A.5, and show that even in scenarios where the reference model is larger (and has higher accuracy) than models in the auxiliary set, TU still achieves higher AUROC than AU. Furthermore, show that AUROC improves with larger number of sampled responses in Appendix A.6.
5.4 TOTAL UNCERTAINTY IMPROVES SELECTIVE ABSTENTION
To evaluate whether uncertainty effectively distinguishes reliable responses from potential errors, we consider selective prediction, where models are allowed abstain from answering when uncertain.
Aggregated Risk Coverage (all datasets)
Risk Coverage (TriviaQA)
Aleatoric (AURC-0.250)
0.4
Aleaturic
Total (AURC-00211)
Bakk (Error)
0.3
Total
0.2
Risk (Error Rate)
0.25
0.0
0.20
Risk-Coverage (HotpotQA)
0.15
0.3
Rink (frror)
0.10
0.2
0.05
0.1
Aleatoric
0.00
Total
0.0
1.0
0.8
0.6
0.1
0.2
0.0
Coverage
Coverage
Figure 7: Risk-coverage analysis shows that TU consistently improves selective prediction across datasets and in aggregate.

---

Published as a conference paper at ICLR 2026
Risk- Coverage Tradeoff. Figure 7(a) shows the Risk-coverage curve for aleatoric and total uncertainty, aggregated across models mentioned in Section 4. Across all coverage levels and datasets, total uncertainty achieves the lowest risk, with single exception. This suggests that total uncertainty more effectively identifies unreliable predictions in comparison to AU.
To quantify this effect accuracy at fixed coverage levels (C@90, C@ and AURC (area under the risk-coverage curve) benchmarks and averaged over different models. In nearly all cases, total uncertainty achieves higher selective accuracy and in all cases lower AURC compared to aleatoric and EU alone. For example, on Hotpot and XSum, total uncertainty improves C@90 by over 1.5 points and reduces area under the coveraee curve (AURC L) by over 20%. These results confirm that TU yields better abstention behavior than AU EU alone.
Selective Accuracy and AURC.
precisely, Table reports selective
Table 1: Selective question answering performance of different uncertainty estimates.
C@90 (%) Epistemic
C@ Epistemic
AURC + Epistemic
Dataset
Aleatoric
Total
Aleatoric
Total
Aleatoric
Total
AmbigQA
56.0
52.4
56.2
59.5
53.2
60.2
0.325
0,456
0.278
CoQA
96.0
96.9
96.0
96.5
97.5
97.8
0.026
0.016
0.011
GSMBX
54.7
53.3
54.8
54.0
0.391
0,441
0.335
HotpotQA
64.9
66.2
66.9
66.2
67.2
69.2
0.304
0.257
0.206
NQ-open
53,8
51.3
53.1
57.2
52.8
57.5
0.388
0.484
0.323
QASPRR
38.0
36.9
39.1
39.5
37.5
40.8
0.533
0.602
0.503
Trivi aQA
64.0
60.4
64.0
69.0
61.8
70.2
0.254
0.343
0.208
TruthfulOA
73.8
74.2
75.0
75.0
73.0
0.220
0,251
0.195
WMT16-60 6-de-en
96.2
96.4
96.0
97.2
96.2
98.5
0.028
0.027
0.010
X8um
24.4
24.9
25.6
26.5
22.0
27.3
0.681
0.759
0.609
CONCLUSIONS
We propose that aleatoric and epistemic uncertainty capture complementary failure modes of language models: self-consistency methods reveal data ambiguity, while semantic disagreement across models uncovers uncertainty arising from model limitations. We operationalize this view by estimating TU as the combination of intra-model entropy and inter-model semantic divergence, using only black-box access to model outputs. We show that this combination effectively outperforms ceff-consistency- based methods across wide range of models and datasets in terms of different metrics. While this approach requires access to multiple comparable models, it reveals the limits of single-model uncertainty scores and offers a practical path toward more comprehensive uncertainty estimation.
Limitations. Our method relies on response-leve semantic similarity, which may underperform in tasks with many semantically distinct but correct answers, e.g., open-ended generation QA tasks where there are multiple distinct correct answers. In such cases, disagreement does not necessarily reflect uncertainty. Additionally, we focus specific form of AU; how best combine our EU estimator with other AU and EU estimators (c.g., token-level or logit-based methods) is left for future work. Moreover, the performance of TU depends on the model ensemble: If all surrogate models share similar pre-training data architectural biases, cross-model disagreement can underestimate true epistemic uncertainty. We examine this homogeneous-fail failure scenario in detail in Section 5.2. Finally, our evaluation hinges on a correctness judge; improvements in judge reliability will propagate to more precise AUROC and selective-risk estimates.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Yasin Abbasi Yadkori, Ilja Kuzborskij, András György, and Csaba Szepesvari. To believe or not to believe your Ilm: Iterative prompting for estimating epistemic uncertainty. Advances in Neural Information Processing Systems, 37:58077-581 17, 2024.
Lukas Aichberger, Kajetan Schweighofer, Mykyta Iclanskyi, and Sepp Hochreiter. Semantically diverse language generation for uncertainty estimation in language models. arXiv preprint arXiv:2406. 04306, 2024.
Neil Band, Tim GJ Rudner, Qixuan Feng, Angelos Filos, Zachary Nado, Michael Dusenberry, Ghassen Jerfel, Dustin Tran, and Yarin Gal. Benchmarking bayesian deep learning diabetic retinopathy detection tasks. arXiv preprint arXiv:2211. 2022.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yeetee Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Kochn, Varvara Logacheva, Christof Monz, al. Findings of the 2016 conference on machine translation (wmt16), In First conference on machine translation, pp. 131-198, Association for Computational Linguistics, 2016.
Tapabrata Chakraborti, Christopher RS Banerji, Ariane Marandon, Vicky Hellon, Robin Mitra, Brieuc Lehmann, Leandra Bräuninger, Sarah McGough, Cagatay Turkay, Alejandro F Frangi, et al. Personalized uncertainty quantification in artificial intelligence. Nature Machine Intelligence, 7(4): 522-530, 2025.
Jianhao Chen, Zishuo Xun, Bocheng Zhou, Han Qi, Qiaosheng Zhang, Yang Chen, Wei Hu, Yuzhong Qu, Wanli Ouyang, and Shuyue Hu. Do we truly need so many samples? multi-lim repeated sampling efficiently scale test-time compute. arXiv preprint arXiv:2504.00762 2025.
Claude. Claude 3.7 sonnet system card. URL httpp::///// //api. semant.iccch" CorpusID:27612236.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint TtV:2110.14468. 2021.
Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, Tamay Besiroglu, and David Owen.
The rising costs of training frontier ai models. arXiv preprint arXh::2405.21015 2024.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of informattion-seekingg questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011, 2021.
Prasenjit Dey, Srujana Merugu, and Sivaramakrishnan Kaveri. Uncertainty-awart fusion: An ensemble framework for mitigating hallucinations in large language models. arXiv preprint arXiv:2503.05757, 2025.
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyamin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, et al. Lm-polygraph: Uncertainty estimation for language models. arXiv preprint arXiv:2311.0738. 2023.
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8: 539-555, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Shiqi Gao, Tianxiang Gong, Zijie Lin, Runhua Xu, Haoyi Zhou, and Jianxin Li. Flue: Streamlined uncertainty estimation for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 16745-16753, 2025.

---

Published as a conference paper at ICLR 2026
Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Kamalika Das. Spuq: certurbation-baseer uncertainty quantification for large language models. arXiv preprint arXiv:2403.02500, 2024.
IBM Granite Team. Granite 3.0 language models, October 2024. URL httpttpp:///itub...... com/
ibm- granite/granttee 3. 0 Language language-mooeesse. models/.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
HW Chung Hou, Longpre, B Zoph, Y Tay, W Fedus, Li,, Wang, M Dehghani, S Brahma, and A Webson. Scaling cnatucttion-inetunne language models. arXiv preprint arXiv:2210.11416 2022.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learming, 1063)457-5506, 2021.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
Mykyta lelanskyi, Kajetan Schweighofer, Lukas Aichberger, and Sepp Hochreiter. Addressing pitfalls in the evaluation of uncertainty estimation methods for natural language generation. In ICLR Workshop: Quantify Uncertuinty and Hallucination Foundation Models: The Next Frontier in Reliable Al.
Albert Q Jiang, A Sablayrolles, A Mensch, C Bamford, D Singh Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b. arxiv. arXiv preprint arXiv:2310.06825, 10, 2023.
Mingjian Jiang, Yangjun Ruan, Prasanna Sattigeri, Salim Roukos, and Tatsunori Hashimoto. Graph-based uncertainty metrics for long-form language model generations. Advances in Neural Information Processing Systems, 17:32980-33006 2024.
Daniel Johnson, Daniel Tarlow, David Duvenaud, and Chris J Maddison. Experts don't cheat: learning what you don't 't know by predicting pairs. arXiv preprint arXiv:2402.08773, 2024.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: / /70.00355, 2017.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfiel-Doodds Nova DasSarma, Eli Tran-Johnson, al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.052211 2022.
Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainty. arXiv preprint arXiv:2502. 18581, 2025.
Andreas Kirsch. (implicit) ensembles of ensembles: Epistemic uncertainty collapse large models.
arXiv preprint (iXXx:240.002222 2024.
Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in llss. arXiv preprins Ttl::2406.159221 2024.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664 2023a.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. ICLR, 2023b.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
12

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Shalev Lifshitz, Sheila Mcllraith, and Yilun Du Multi-agent verification: Scaling test-time compute with multiple verifiers. arXiv preprint arXiv:2502.2037, 2025.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprini arXiv:2109.0095: 2021.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205. 14334, 2022.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantifica- tion for black-box large language models. arXiv preprint arXiv:2305.19187, 2023.
Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen. Uncertainty estimation and quantification for Ilms: A simple supervised approach. arXiv preprint urXiv:2404.15993, 2024.
Litian Liu, Reza Pourreza, Sunny Panchal, Apratim Bhattacharyya, Yao Qin, and Roland Memisevic. Enhancing hallucination detection through noise injection. arXiv preprint arXiv:2502.0377 2025.
Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang. Merge, ensemble, and cooperate! survey on collaborative strategies in the era of large language models. arXiv preprint arXivv24477.06i0 06089, 2024.
Huan Ma, Jingdong Chen, Guangyu Wang, and Changqing Zhang. Estimating Ilm uncertainty with logits. arXiv preprint arXiv:2502.00990. 2025.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-bos
hallucination detection for generative large language models. arXiv preprint wrXIv:2303.088%. 2023.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigga: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004. 10645, 2020.
Malik Sajjad Ahmed Nadeem, Jean-Daniel Zucker, and Blaise Hanezar. Accuracy-rejection curves (ares) for comparing classification methods with reject option. In Machine Learning in Systems Biology, pp. 65-81. PMLR, 2009.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization, arXiv preprint arXiv:/808s 08745, 2018.
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint PrXi::2008.0887, 2021.
Shiyu Ni, Keping Bi, Jiafeng Guo, and Xueqi Cheng. When do Ilms need retrieval augmentation? mitigating llms' overconfidence helps retrieval augmentation. arXiv preprint arXiv:2402. 1457, 2024.
Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. Kernel language entropy: Fine- fine-grained uncertainty quantification for Ilms from semantic similarities. Advances in Neural Information Processing Systems, 37:8901-8929, 2024.
Siva Reddy, Danqi Chen, and Christopher Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266 2019.
13

---

Published as a conference paper at ICLR 2026
Andrea Santilli, Adam Golinski, Michael Kirchhof, Federico Danieli, Amo Blaas, Miao Xiong. Luca Zappella, and Sinead Williamson. Revisiting uncertainty quantification evaluation in language models: Spurious interactions with response length bias results. arXiv preprint arXtv:2504.13677, 2025.
Kajetan Schweighofer, Lukas Aichberger. Mykyta lelanskyi, and Sepp Hochreiter. Introducing im- proved dnformation-thhoree measure of predictive uncertainty. arXiv preprint arXiv:2311.08309, 2023.
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets. arXiv preprint 0rXiv:2309.15789 2023.
Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z. Ren, and Anirudha Majumdar. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. arXiv preprint arXiv:2412.05560 2024.
Adi Simhi, Itay Itzhak, Fazl Barez, Gabriel Stanovsky, and Yonatan Belinkov. Trust me, i'm wrong: High-certainty hallucinations in llms. arXiv preprint arXiv:2502.1 22964, 2025.
Lintang Sutawika, Leo Gao, Hailey Schoelkopf. Stella Biderman, Jonathan Tow, Baber Abbasi, ben fattori, Charles Lovering, farzanehnakhaee77, Jason Phang, Anish Thite, Fazz, Aflah, Niklas Muennighoff, Thomas Wang, sdtblck, nopperl, gakada, ttyuntian, researcher2, Chris, Julen Etxaniz, Zdenčk Kasner, Khalid, Jeffrey Hsu, AndyZwei, Pawan Sasanka Ammanamanchi, Dirk Groeneveld, Major refactor, December 2023.
Ethan Smith, and Eric Tang. Eleuthera//mm-evvlaatinnnnnnnnn URL https::t//// // /ww.///iiioo/////////../ /00
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261. 2022.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.001 18, 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, al. Gemma technical report. urXiv preprint arXiv: 2503. 19786, 2025.
Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, et al. Benchmark ing uncertainty quantification methods for large language models with Im-polygraph. Transactions of the Association for Computational Linguistics, 13:220-248, 2025.
Xi Wang, Laurence Aitchison, and Maja Rudolph. Lora ensembles for large language model fine- tuning. arXiv preprint arXiv:2310.00033 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177 2022.
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024.
Zhiqiu Xia, Jinxuan Xu, Yuqian Zhang, and Hang Liu. A survey of uncertainty estimation methods on large language models. arXiv preprint arXiv:2503.00/72 2025.
Yihao Xue, Kristjan Greenewald, Youssef Mrouch, and Baharan Mirzasoleiman. Verify when uncertain: Beyond self-consistency in black box hallucination detection. arXiv preprint arXiv:2502. 15845, 2025.
14

---

Published as a conference paper at ICLR 2026
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXhv:2412.15115. 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arviv preprint arXhv:1809.09600 2018.
Lexin Zhou, Wout Schellaert, Fernando Martinez-Plumed, Yael Moros-Daval. Cèsar Ferri, and José Orraández-Orallo. Larger and more instructable language models become less reliable. Nature, 634(8033):61-68, 2024.
15

---

= k'r, the divergence in Eq. 2 can be Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 THEORETICAL INTERPRETATIONS OF EPISTEMIC UNCERTAINTY
Kernel and variational interpretation of D(w w*). Assume the similarity function s(.,-) is symmetric positive definite kernel k. Denote the predictive distributions by Po : p(. w) and Pw.. Their kernel mean embeddings the reproducing kernel Hilbert space (RKHS) H4 are
Pa = Pr [k((r:)) -)]
Ve = EquPP [k(g. :)
Using the reproducing property (k<rr, -) rewritten exactly as:
D(w || (1) (mw (w)HH ( HHu
(S)
Eq. 5 is the first two terms the squared maximum mean discrepancy (MMD):
MMD3 (Po. Pw-) = w Por llze. = || 11.4. - (Nww Mwr THe + +ll 112e. (Pwr How MH
D(w |l wt)
Thus D(w || w²) ) is a one-sided kernel discrepancy: measures how much the model's self-agreement exceeds its agreement with the ideal predictor, and it vanishes if and only if pw Mu (and, for characteristic kernels, iff Pa = Pw-).
Variatioonal-get Interpretation. Write classical KL KL(Pa IP. CE Ent, where Ent
Ey-Pal - log p(r x;w)] and CE Ep log 2)}. Replacing log with -k yields
D(w w²) E Er (k(r, [((r,))))
E Pa-Po Pu. kk g)
"negative kernel-entropy"
"kemel cross-entropy"
so the semantic variational gap between the model and the ideal distribution under the geometry induced by Minimizing D therefore projects Pa toward Pa- in RKHS while simultaneously penalizing model variability in Pq-
Lemma (Bound on Kernel-Based Divergence). Let Pw =p. () and Pa- - xx w²) be the predictive distributions of two language models. Let k((,,) be symmetric, bounded, positive-defiitte kernel such that 0 & ke(r r') I for all r,r'. Define the kernel-bass divergence
D(w (²) == Ey [k(r, r')] -Err (.lr,,,,,((.0 [k(r, (M(r.q)). 9)].
Then D is bounded above in absolute value by the total variation distance between Pa, and Pat: DD( l| PV.(P,,
where
TVD(Ps, Pw. Pw.):= 1 [p)()) Pw- (z)]d2.
Furthermore, the RKHS norm of the embeddings are equal, i.e., || || = I|ww-l then D(w (I O implies μ. = pwr. If characteristic, this further implies Po Pu..
Proof. Define the kernel-smoothed function = Ex-P. (k((r 2)]. Then we can write
D(w (') Ez-r, [[()) Rz-re- [f())]
By the definition of total variation distance and the fact that 0 < f(=) < 1 for all
D(w w) sup
EP [f] - Ep [f]] TVD Pw-).
11
For the second part, note that ((w w*) || (Mws Mw ). If ||ا| || || and D 0, then by Mauchy-Schwarz equality, we must have Ma = Mwn If the kernel is characteristic, then Muu = p implies P_, Pat.
16

---

Published as a conference paper at ICLR 2026
This result implies that D(w I|=") provides lower bound on distributional mismatch, and thus serves as a tractable proxy for epistemic uncertainty: it is provably small only when the model's 's predictive distribution aligns with that of the ensemble under the kernel geometry.
Under desired conditions mentioned in 3.2, and with a bounded characteristic kernel, D(w || w-)) = if, and only if, the predictive distribution P., is close the model set consensus shown above. Consequently D (i) flags cases where the model's intra-model similarity is high (AU is low) while its agreement with the ensemble is low, (ii) rewards calibrated agreement by attaining its minimum only within the support of the ensemble, and (iii) remains sample-efficient, as it only requires O(n²) kernel evaluations on generated responses per model, which cheaper than KL divergence evaluations or other approximations of EU.
A.2 AGREEMENT AND COVERAGE METRICS.
For every dataset we form the binary correctness matrix Cij which the correctness of response r( sampled from model to input i, and compute two coarse descriptors of cross-model behaviour:
Jaccard redundancy J The mean pairwise Jaccard index of the sets of correctly answered examples:
J
2
Σ
Sm n Skel
Sn = {i: Cim 1}
M(M -
Sm U Sk]
I5mckgM
High J/ means models succeed on the same inputs.
Oracle-gain diversity G - the additional coverage obtained by oracle that chooses the correct model per example:
M
N
G Aorale max Am
H
0
max
1 N
Σ
Cim
N
High G indicates that different models get different examples right.
A.3 ADDITIONAL EXPERIMENTAL SETUP
All experiments were conducted on two NVIDIA A100 80GB GPUs. We use the Im-evaluation-harmt
(Sutawika et al., 2023) codebase to generate responses from each model, and evaluate correctness using local vLLM (Kwon et al., 2023) server hosting Meta-Il ama -atta-3-770 ---70 Instruutt (Grattafiori al., 2024) the judge model. Following prior work (Lin et al., 2023). we compute correctness using only the first sampled response from each model. All evaluation conducted inference-only mode; no training or fine-tuning is performed.
For each dataset, we sample 10 responses per model for the first 100 prompts. AU is computed using all 10 responses. To match the sample budget with TU, the experiments in Section 5.3 and Section 5.4 report results using only 2 responses per model for computing both TU and epistemic uncertainty. We use a temperature of 0.7 and top-p 0.9 across all generations. For Simple@k model outputs are obtained from the OpenAl and Anthropic APIs.
measured using cosine distance over nally supported by (Hou et al., 2022; Ni et al., 2021) embeddings. For datasets not origi- we follow its prompt formatting conventions and include code for these additions in the supplementary material.
Semantic similarity between responses
A.4 ADDITIONAL RESULTS ON TOTAL UNCERTAINTY
Figure 8 reports the AUROC of aleatoric and total uncertainty across all madel-dataset pairs, and Fig- 9 shows the corresponding improvements from using TU over AU. Total uncertainty consistently improves correctness discrimination in nearly all cases, with the largest per-instance gains observed on HotpotQA, benchmark known for complex multi-hop reasoning.
17

---

Published as a conference paper at ICLR 2026
AUROC Results Per Model and Dataset. Figure 8 and Table 2 show AUROC of TU and AU in all model-dataset combinations, and Figure shows the improvement of total uncertainty over AU in AUROC. TU improves AUROC in nearly all model-dataset combinations, with the largest gains observed in Hotpot QA. Model performance substantially affects the magnitude of improvement. On datasets such XSum, where overall model accuracy is low, TU yields large improvements for weaker models but occasionally underperforms for the strongest ones (c.g., Llama and Qwen), potentially due to disagreement with less reliable auxiliary models. A similar pattern holds on GSM8K, where gains are concentrated among lower- swwer-performing models, while others benefit less.
In contrast, WMT16-de-en and CoOA show limited gains, likely due the high baseline accuracy (> 90%) of all reference models (see Fig. 10), where AU is already well-calibrated. Notably, TU corrects miscalibrated AU for specific models, such as Mi st ral on CoQA, where the base AU is anomalously low: On Truth Fu QA, which features open-ended questions with diverse valid answers, semantic disagreement does not reliably indicate epistemic uncertainty, which results in weaker improvements.
AmbigQA
C-QA
GSMSK
HeépoiQA
NQ-apen
8
AUROC F
AUBOC
2
8
00
Abotoci
QASPER
TrivisQA
TrunhfulQA
WMT16-deen
Sure Scod. Jvm
&
AUBDO
AURO
& 2 g
RR g 8
Litoue di foo
Hanall ( Chrea S-gh Auntest
Clamaa ai bodroet
Altira Lrth touroai-1t3
Cerec 77 bonkees
gero 2-00
Libee Bilbbrre Mltia w-w.8-788. Bovtrot
Llasu dB Soatoot Miluna tith besr borkest
graaie 1-4t Rh-qeepass
get 2.0b
Litena & didenttoo
Miniall Podessreee-t
Chron Prote
Figure 8: We show AUROC for each model separately to compare aleatoric and total uncertainty. TU consistently yields higher AUROC across models. We subsample 80% of the questions per dataset, 1000 times, to compute AUROC with 5% confidence intervals around the median AUROC value.
AmbigQ2A
COQA
GSMAK
HotporQA
NQQ-open
ALMECC
0.04
0.15
ALMOCC
0.00
0.00
0.5
DULS
-038
-0.00
QASPER
TriviaQA
TruthfulQA
WMT16-de-cn
XSum
Aistoa Aestraa---33 5-33
0002
D:00
0.04
Obeed By 2i-tstred
goutted D ieeeet
germaa--e 2-8--
AUROC:
0:00
0044
8.
002
0.01
Owied 3-18 Spoerer
Liossn's aitroor
Mitnal Wi-astrnntsstrrr....o Snsttran--113
ALMOC:
0002
Tiiman's Aiderrenn
MTROC
0n2s
Cheed 5.15 Spiterrr
peerse pecoaa---9b 3-8--l
Alsttd 1. Aesttee--ii Oeerd Ftb-tsttred
AUBOC
0:00
0.00
gremibe D inseeier
Figure 9: AUROC improvement obtained by adding EU AU across all samples per dataset,
measured as (Total- - Aleatoric).
18

---

Published as a conference paper at ICLR 2026
AmbigQA
CoQA
GSMSK
HotpotQA
NQ-open
R
AUROK
2
2
88
80
60
80
QASPER
TruchfulQA
WM
04
MUROC
F.
AUROC:
E E
8 24
B.
12
Dlecad Soarent
Oemaa S. batront
8
Liepuaa F-B Moarte
Quraa 2o-.5------ fienrt
greetios dh louttord
Likea's Eastdattrr
Mactad Bor.Beutttttttt
Quemato Dertar
gramtes faurent
Lieas rovr
Mastaa Petfeetfurree
Chomat www Peleerrene
predied bottbuttn
Davadd Jab Searent
Misrnd- bestretted
Qemma 5. Inttovt
gioroo 2a-yy glantee
Figure 10: Accuracy per model-dataset pair.
ROC Curves. Figure shows the ROC curve computed over the pooled set of model-dataset pairs. TU achieves higher AUROC (0.746 vs. 0.707), which shows improved seperation between correct and incorrect generations compared AU alone. Figure presents ROC curves for individual datascts. TU yields consistently better comparable performance across all tasks, with the largest gains observed on Hotpot QA, WMT --de de-en, earlier findings that TU is most effective tasks where models accurate but occasionally confidently wrong.
enn and CoQA. These improvements align with our
1.0
0.707)
Total
louz 0746)
0.8
powitiveros
0.2
0.0
0.4
1.0
false positive rate
Figure 11: ROC curves between aleatoric and total uncertainty aggregated across all models and datasets. Total uncertainty achieves higher AUROC, indicating better discrimination between correct and incorrect generations.
19

---

Published as a conference paper at ICLR 2026
Table 2: Uncertainty AUROC scores across models and benchmarks when different 7B/8B/9B
parameter models are used as auxiliary models. Total Uncertainty is better calibrated with correctness than Aleatoric Uncertainty.
Benchmark
Model
Accuracy
Aleatoric AUROC
Epistemic AUROC
Total AUROC
Llama 3.1 8B Instruct
0.62
0.764
0.52
0.753
Mistral-
0.58
0.716
0.506
0.768
AmbigQA
Qwen2.
0.42
0.645
0,639
0.75
gemma-2-9en -2-9b-it
0.52
0.81
0.447
0.833
granite 3.0 101-88--nnm instruct
0.644
0.488
0.93
0.788
0.797
0.845
0.95
0.458
0.80
0.819
CoQA
Swen2.5-78-Iastruse
0.97
0.466
0.567
0.595
gamma-2-bb-it
0.95
0.876
0.80
0,918
0.97
0.708
0.907
0.821
Llama 3.1 3.1-8B Instruct
0.69
0.727
0.346
0.626
Mistra -----Instruet-vo..
0.35
0.577
0.606
0.61
GSM8K
Qwen2. I--ttuuuttt
Instruct
0.61
0.524
0.623
0.656
gamma-2-9b-it
Sranite-3.0-88 -.0-tutinstrrct
0.60
0,726
0,472
0.727
0.65
0.546
0.662
0.7
Xigrrall-7--Intttru----
0.65
0.548
0.682
0.736
HotpotQA
Qwen2. E
0.71
0.483
0.681
0.679
gumma-2-9b-it
0.58
0.719
0.608
0.814
1ranite-3.0-10 instruct
0.58
0.615
0.634
0.736
3.
0.49
0.808
0.389
0.713
Mistra Mistral--7--1 17arra1-----nss
0.60
0.69
0.501
0.698
NQ-open
Qwen2. Bwen2.5-7B-Inatzue
0.37
0.462
0.696
0.703
gamma-2-9b-it
0.47
0.743
0.538
0.768
3raante--.0-8B-nnnsrree
0.30
0.487
0.714
distrral7BB-o S3-Innsrruee-O...
0.32
0.623
0.657
0,722
QASPER
Swen2.5-7--22 Iast-astuuttt
0.45
0.587
0.492
0.583
gemma-2-9b-it
0.37
0.588
0.509
0.596
granite -.
0.41
0.56
0.512
0.596
Llama-3. 3. Innsttt
0.65
0.744
0.562
0.776
Mistra --1--Instruut--oo.
0.54
0.796
01552
0.815
TriviaQA
0.5
0.811
0.668
0.9
gomma-2-9b-it
-ranite-3.0-bb-instte
0.64
0,733
0,575
0,784
11ama-3.1-8B -tttttttuttt
0.65
0.47
0.423
0.456
Mistral-7B --n-----nnttt--ddd
0.80
TruthfulQA
0.75
0.494
0.584
0,578
gemma-2-9b-ln
0.76
0.641
0.54
0.625
granite 3.0- instruct
0.75
0.532
0.556
0.548
L1ama-3. In9tutnst
0.95
0.691
0.695
0.859
Mistral -
0.91
0.808
0.835
0.897
WMT16-de-en
Swes2.5-7B-Insteeuct
0.832
0.883
gomma-2-9b-it
0.97
0.649
01666
0.811
0.95
0.755
0.493
0.884
1.1ama-3. T-ne-
Mistral 1ostral-7B--nnn Ins-rrstuur--
0.20
0,482
0.651
0.725
XSum
Qwen2. Instrrutt
0.819
0.31
cemma-2-9b-1s
0.16
0.598
0.565
0.631
granite 3.0 8b instruct
0.12
0.529
0.582
0.717
Comparison with Baselines. We compare TU against number baselines: Mean Token En- tropy (Fomicheva et al., 2020), Maximum Token Probability (Fadeeva et al., 2023), Maximum Sequence Probability (Fadeeva et al., 2023), Perplexity (Fomicheva al., 2020), PTruu (Kadavath et al., 2022), Self-Certainty (Kang et al., 2025), Semantic Entropy (Kuhn et al., 2023a), SC (Self- Consistency) Score (Wang et al., 2022; Manakul et al., 2023), SelfCheckGPT (Manakul et al., 2023), Kernel Language Entropy (Nikitin et al., 2024), Closeness Centrality, SC VC (Verbalized confi- dence), and SC based CV (Jiang et al., 2024). We use implementations from Fadeeva et al. (2023); Jiang al. (2024). AU baselines is the same (Lin al., 2023), but instead of using entailment with Deberta to compute response sequence similarity, we use a sent ence- T5-x1 model. Results provided in Table 3.
20

---

Published as a conference paper at ICLR 2026
Dweettiny Calhotion curre per ditace
Alsterk Ioar 0.77)
Alettrk Connrk--mn--11 1.75
AMttete 2.000
03
(lac-
1.00
C3
1o4d
glac 176)
00
Trul
1800-0030
00
Tral (80--1880)
00
Teal.
Use 0460)
0.00
6.3
0000
0.9
120
130
023
5.0
035
1.00
030
6.50
035
1.00
0.55
6.50
0.5
100
0.00
025
100
AmrborQA
TruthriaQ2s
M02eppen
Hotpett25
04
0.4
02
02
02
63
1001
T0cal
1.50)
00
Ronal
180C-0712
00
Trat
00
Test
(000)
077
120
2.50
0.5
3.0
2.00
0.7
1.00
8.00
0.75
1000
0.00
025
100
[ake posdiie cate
5i5e peslltiree rite
5i0e pestitire cate
Sile pesliive
Cakle pww.ppptteeeoooo
Figure 12: ROC curves comparing aleatoric and total uncertainty across individual datasets. TU achieves higher AUROC across most tasks, particularly Hotpot QA, WMT1 6 --de-edeeeeeee enn,, and CoQA, where models exhibit confident failures.
Table 3: Uncertainty AUROC scores across benchmarks for M1 wistrall a-ralllBB when different 7B/8B/9B parameter models are used as auxiliary models. Total Uncertainty is better calibrated with correctness than Aleatoric Uncertainty and other baselines.
AmilyyAA
CuQA
GSMEK
BoLpoLOA
NQ NQ-open
QASPER
TeivisQ8
TzuthfulCA
Average
Total
0.768
0.819
0.61
0.736
0.698
0.722
0.815
0.579
0.718
Aleatoric
0.716
0.458
0.577
0.548
0.69
0.623
0.796
0.528
0.617
Closeness Contrality
0.683
0.612
0.56
0.739
0.597
0.579
0.702
0.639
0.639
SC Score
0.649
0.553
0.551
0.711
0.616
0.567
0.673
0.603
0.61.
SC Based VC
0.658
0.483
0.555
0.677
0.63
0.568
0.705
0.602
0.61
SemanticEntropy
0.678
0.561
0,584
0.526
0.656
0.514
0.637
0.514
0.584
SC+ VC
0.671
0.411
0.544
0,51
0.641
0.581
0.731
0,581
0.584
Sequence
Mean Token Entropy
0.634
0.498
0.461
0.559
0.674
0.481
0.696
0.544
0.571
Token Entropy
0.654
0.502
0,465
0.558
0.672
0.481
0.542
0.57
Perplesity
0.661
0.513
0,462
0.548
0.652
0.464
0.672
0.527
0.562
Max Token Prob.
0.658
0.51
0.464
0.546
0.652
0.468
0.674
0.528
0.562
Self Certainty
0.569
0.51
0.459
0.553
0.598
0.484
0.586
0.549
PTrue
0.604
0.555
0.371
0.417
0.55
0.423
0.519
0.451
0.486
Sel[CheckGPT
Kernel Lang. Kat.
0.785
0.642
0.637
0.491
0.585
0.573
0.723
0.637
0.634
Correlation with BLEU and ROUGE. For the two generative tasks we additionally evaluate the correlation between uncertainty scores and standard automatic metrics (BLEU for translation, ROUGE for summarization). As shown in Table 4, both AU and TU show a moderate negative correlation with BLEU WMT1 6-de-en, consistent with higher uncertainty aligning with lower translation quality. On XSum, AU shows a positive correlation with ROUGE, while EU exhibits negative correlation (-0.213), which suggests that EU is better proxy for quality degradation in summarization than AU in this setting.
and XSum),
Table 4: Pearson correlation between uncertainty scores and automatic evaluation metrics (BLEU for WMT16 6-de en, ROUGE for XSum).
Dataset
Metric
Aleatoric
Epistemic
Total
W 6-de-en
BLEU
-0,319 +0.162
-0.115
-0.324 -0.056
XSum
ROUGE
-0.213
21

---

Published as a conference paper at ICLR 2026
A.5 THE EFFECT OF THE AUXILIARY MODEL SET ON TOTAL UNCERTAINTY
Ablating the Reference Model Size. We test how the quality of total uncertainty estimates depends on the capability of the reference model, whose uncertainty we aim estimate. We fix the auxiliary model set to pool of four 7-9B models mentioned in 4 that are not from the same family as the reference model, and vary the reference model's architecture and size. Figure 13 reports results on TriviaQA, using two model families (Gemma3 (Team et al., 2025) and Qwen2. 5 (Yang et al., 2024)) of various sizes. As the size of the reference model increases, both alcatoric and total uncertainty AUROC scores tend to decrease, but total uncertainty has consistently higher AUROC across different model sizes. This holds even when the reference model is substantially stronger than any model in the auxiliary set (e.g., Owen2. 5-B vs. 7-9B peers).
Unoertiinty Qwen2.3 Reference
Unoertabity Calbrrim Genma' Reference
1.0
-0
1eud Liwararr 2A0820
405
Tota Leconary 240800
-0
077
Quedd Kidd Fixe
Figure 13: We vary the size of the reference model while holding the auxiliary models fixed. TU achieves higher AUROC in comparison AU acorss different model sizes on TriviaQA.
Wuise-perturber Auxiliary Model Set. We consider noise-perturbed variants of the reference model itself as auxiliary model set, similar to Liu al. (2025). Specifically, we apply perturbation strategy in which we preserve the top-k singular vectors of each linear weight matrix and inject Gaussian noise into the remaining lower-rank subspace. This preserves dominant components of the model while allowing for controlled noise in its response distribution. Figure 14 in the appendix shows that the we sometimes obtain improvement TU-AU, but is overall lower than for the more diverse auxiliary model set from Figure 4.
1.0
Aleatoric
AUROC
0.8
0/
144
074
027
Total
140
021
080
0.4
AmbigQA
CoQA
GSMBK
HotpoiQA
NQ-open
QASPER
TriviaQA
TrathfolQA
WM de
XSum
Figure 14:
Uncertainty calibration for experiments where auxiliary model set for each model is consisted of multiple noise-perturbed models
A.6 ABLATIONS
Number of Auxiliary Models. We study how the size of the auxiliary model set affects the quality of total uncertainty estimates. For each reference model, we compute total uncertainty using n E {2, 3, 4, 5) models, where one model is fixed (the reference model) and the remaining n 1 are sampled from other model families. All methods use a fixed number of samples per model.
Figure 15 shows that total uncertainty improves monotonically as the number of auxiliary models increases. This holds across almost all tasks, with the largest gains typically occurring between 2
22

---

Published as a conference paper at ICLR 2026
and n = 3. In addition, observe that variance across rums decreases more models are added, which suggests a more calibrated uncertainty score can achieved from increasing the number of model in the auxiliary set. However, in all datasets, our multi- sample total uncertainty measure outperforms aleatoric uncertainty in AUROC, even when only one auxiliary model is used.
AmoryQA
CHQg
TeviagA
WM W/M111-deee
X0um
Toke Alanlone
Alararic
Irral
Irall
Takel
Alurtet::
Alraeetic
Abateex
et Acatbary
Auntlory
Aunikary Kary
Figure 15: We plot AUROC as function of the number of auxiliary models used to compute total uncertainty for Mistral-7B- Mistral--B-Innttrut-vvo. Total uncertainty improves with more models, and variance decreases.
Number of Samples for Uncertainty Estimation. We next investigate how the number of response samples per model affects the performance of uncertainty estimates. For each model in the auxiliary set, we vary the number of generations used in total uncertainty computation from 5 to 50, and compare against two baselines for aleatoric uncertainty: one computed using 5 samples and another using 10 samples, matching the regimes used in our main experiments.
As shown in Figure 16, AUROC for total uncertainty usually slightly increases with more samples, with diminishing returns beyond 30 samples in most tasks, Notably, TU consistently outperforms AU baselines across all datasets. These findings also reinforce the practicality of TU even under constrained budgets, as improvements are apparent with as few as 10 samples (n = 5 on the x-axis).
Ambric
GUANK
Teraqq
WM doan
Alastzzc Abokes (e-10)
25
S Nusb Sample
Nanter Samphi
Namber Sampila
Figure 16: AUROC total uncertainty as a function of the number of samples per model, Even with small number of samples, TU outperforms aleatoric baselines (5-sample and 10-sample variants). Gains saturate around 30-40 samples.
Similarity Backbone. To test the robustness of our results to the choice of similarity function, we repeated the analysis on TriviaςA using deberta-large--ae similarity) in addition to the original sentence-T5- performance trend (TU >> AU) remains consistent across both similarity measures, which confirms that our framework is not dependent on specific similarity backbone.
(entailment probability as embeddings. shown Table 5, the
Table 5: AUROC on TriviaQA using two different similarity backbones: cosine similarity over Sentence-T555 5- embeddings vs. entailment probability from deberta-large-mil TU consistently outperforms AU regardless of the similarity measure.
Embedding (sentence- -5-xl)
Entailment (deberta-large-mnll)
Model
AU
TU
AU
TU
Mistral--9--Instruct--o
0.796
0,815
0.788
0,885
Qwen2. 5-78-nit----
0.811
0.900
0.801
0.905
granite-3. --tee---..--bb-nsteeeeeee
0.733
0.784
0.740
0.821
0.744
0.776
0.752
0.810
genma genma-2-9h 22-9b-1- it
0.787
0.812
0.765
0.862
Weighted Total Uncertainty. Our formulation defines TU = AU + EU (Equation 4). In practice, one could also consider a weighted interpolation TU,- = U Ustemtrric + (1 - X) Uepistemie where À controls the relative contribution of each component. Note that do not assume access
23

---

Published as a conference paper at ICLR 2026
correctness labels uncertainty quantification time, À is not tuned hyperparameter. Table 6 reports AUROC across all benchmarks (averaged over five reference models) for different values of A. While specific values of À yield marginal gains individual datasets, the unweighted sum (TU) consistently outperforms AU across all benchmarks without requiring task-specific calibration.
Table 6: AUROC for weighted total uncertainty TU = A. AU + +1 X) EU across benchmarks, averaged over five reference models. Bold indicates the best score per dataset. The unweighted TU (= AU + EU) is competitive across all benchmarks without requiring tuning.
Metric
Ambiq0A
CooA
GSM8K
Rotpot
NQ-
QASPER
Trivia0A
Trathroloa
WMI16
XSum
AU
0.716
0.659
0.617
0,582
0.676
0.610
0.774
0.533
0,315
0.603
0.520
0.774
0.529
0.654
0.531
0.600
0.384
0.527
TU
0.753
0.800
0.651
0.733
0.727
0.642
0.817
0.557
0.384
0.707
A=0.1
0.556
0.792
0.549
0.676
0.565
0.548
0.648
0.546
0,384
0.543
A=0.2
0.599
0.803
0.575
0.698
0.699
0.386
0.568
A=0.3
0.661
0.825
0.597
0.725
0.646
0.602
0.752
0.562
0.386
0.612
A=0.4
0.714
0.818
0.628
0.744
0.689
0.631
0.795
0.563
0,383
0.679
A=0.5
0.753
0.800
0.651
0.733
0.727
0.642
0.817
0.557
0.384
0.707
A-0.6
0.764
0.777
0.648
0.701
0.739
0.639
0,825
0.549
0.379
0.678
0.0.7
0.754
0.751
0.639
0.662
0.731
0.635
0.823
0.547
0.365
0.649
0=8
0.742
0.726
0.6288
0.634
0.713
0.627
0.812
0.545
0.352
0.627
A-0.9
0.732
0.708
0.624
0.611
0.621
0.799
0.333
0.613
A.7 ADDITIONAL RESULTS ON MULTIPLE-CHOICE QA TASKS
To evaluate whether our findings extend beyond open-ended generation, we adapt subset of tasks from the Big-Bench Hard (BBH) (Suzgun et al., 2022) benchmark into long-form QA format with chain-of-thought answering. Specifically, we consider Boolean Express ions, Disambi guation QA, and Word Sorting, and prompt models to justify their answers rather than selecting from multiple choices directly. We then evaluate uncertainty scores over the full responses using the same semantic similarity pipeline as in our main experiments.
Table 7 reports AUROC scores for both AU and TU across models on these tasks. We observe that TU improves over AU in most cases, with the largest gains appearing when base model performance is Di low (e.g., Qwen2.5-7B Disambi ambiguat ion QA and Boolean Express onss. These results demonstrate that TU remains effective identifying incorrect generations even when the task is originally framed as multiple-choice, provided responses elicited in free-form.
Table 7: Uncertainty AUROC scores models and benchmarks when different 7B/8B/9B parameter models are used as auxiliary models. Total Uncertainty is better calibrated with correctness than Aleatoric Uncertainty.
Benchmark
Model
Accuracy
Aleatoric AUROC
AUROC
Llama 3, 1-1-8B Instruct
0.662
Mistral-7B- -B- v
0.84
0.746
0.735
BBH Fewshot Boolean Expressions
5-en2.5-7B-In-e gcrma-2-9b-it granite-3.0-8b-ine
0.53
0.744
0.909
0.593
0.9
0.659
0.658
1.lmm-3..1-8-IInstt
0.59
0.544
0.594
Mistral-7B-nnnszrett-
0.525
BBH Fewshot Disambiguation QA
50en2.557B-Innttruus
0.44
0.561
0.81
germa-2-9b-it
0.69
0.61
0.65
granite-3.0-BB-inns-
0.62
0.486
0.562
Llama-3. 1-1.--1-------tt-
0.69
0.476
0.512
Mistral-7B- -B--Inn
0.77
0.529
0.429
BBH Fewshot Word Sorting
Qwen2. 1wee2.5-7BB-1 5-7B-Instruct
0.44
0.587
0.645
gemma-2 2- 2-9b-15
0.96
0.475
0,576
granite 3. 10-----nneee instruct
0.58
0.578
0.485
24

---

Published as a conference paper at ICLR 2026
A.8 EPISTEMIC UNCERTAINTY ANALYSIS
Figure 17 disaggregates the trend shown in Figure 2a by model. For all five reference models, incorrect generations in the low-AU regime show consistently higher EU than correct ones, which reaffirms that EU captures confident failures missed by self-consistency. This separation weakens in mid- and high-AU buckets, where both correct and incorrect outputs tend be more uncertain. The consisteney of this pattern across different models highlights the effectiveness of EU in identifying unreliable predictions when AU alone low. Similarly, Figure shows a similar trend when results are disaggregated by dataset.
Mistral-7B-lnsttucc--..3
Qwen2.5-7B-nnttuul
granite-3 0-8b-instruct Llama-3.1-888 1-88-Instruct
gemma-2-9b-it
eptstemic uncerbaiity
0.3
correct
0.1
low
mid
high
low aleatoric bucket
mid
high
low
mid
high
low aleatoric bucket
mid
high
low
mid bucket
high
aleatoric
aleatoric
Figure 17: Distribution of EU across different levels of AU and correctness. Across all models, find that incorrect responses in the low-aleatoric regime are assigned higher EU than correct ones on average.
TriviaQA
CoQA
XSum
WMT16-de-en
GSM8K
epistemic uncertainty
0.3
0.0
AmbigQA
Truthfu\QA
NQ-open
QASPER
HolpoiQA
epistemi uncertaity
0.2
0.1
mid
high
low alestoric bucket
mid
high
low
mid
high
low
midl
high
Iow
mid
high
aloatoric bucket
aleatoric brccket
aleatoric bucket
alcatoric buccket
Figure 18: Distribution of EU across different levels of AU and correctness. Across all benchmarks, find that incorrect responses the low-aleatoric regime assigned higher EU than correct ones on average.
A.9 COMPUTATIONAL COST
We report full wall-clock measurements contextualize the overhead introduced by our TU ensemble relative to self-consistency baseline with the same total sample budget. All runs use TriviaQA on single Nvidia L40S (48 GB) unless noted. Decoding settings and similarity-orache calls are identical across settings; the observed gap comes from loading additional checkpoints.
Takeaways. 1) Token- generaaionn and similarity costs are matched between the self-consistency baseline (1 10 samples) and TU (5 2). 2) Sequential overhead is dominated by checkpoint loads; peak VRAM does not increase. 3) Parallel execution amortizes loading: with 5x parallelism, TU matches single-model wall-clock while retaining its AUROC and selective-abstention improvements.
25

---

Published as a conference paper at ICLR 2026
Regime
Sctting
Models x sumples
Per-model mean (6)
Total wall-clock (s)
Single-Cul MnggleGUUUa (sequentia))
Single model, 10 samples TU ensemble, models x samples
x 7-9B x 5 7-9B x
244.1 :: 58.7
244.1
Single GPU (sequential)
Single-GPU
78.3 12.9
391.7 R: 78
Multi-GPU (parallel, 5x)
TU ensemble, models x 2 samples
5.x 7-9B x
78.3 12.9
Table 8: Wall-clock on TriviaQA. Sequential TU is slower due to streaming four extra checkpoints; peak GPU memory unchanged. With S-way parallelism (one GPU per model), wall-clock returns ~78 while preserving TU's calibration and abstention gains.
In practice, small ensemble (2-5 auxiliaries, 1-2 samples each) provides most of the TU benefit under common deployment envelopes.
Computational overhead of loading models. We acknowledge the higher memory requirement of loading multiple models but emphasize that our comparison controls for total inference compute budget. Specifically, our main results compare AU (10 samples) against TU (2 samples x 5 models), which keeps the total tokens generated the same.
This overhead is tradeoff for better UQ, because increasing the sampling budget for the single-model baseline does not close the performance gap. As shown in Table 9, increasing the sampling budget for AU from 5 to 20 does not improve AUROC for AU. contrast, TU with just total samples (1 sample models) consistently outperforms AU with 20 samples.
Table 9: AUROC on TriviaQA under varying sampling budgets. Increasing AU samples does close the gap with TU.
Model
AU n=5
n=10
AU n=15
AU n=20
TU 1==1
TU n=2x5
T.1 1ma-331--8--Tssrrae
0.78
0.78
0.82
0.81
0.89
0.87
0.74
0.76
0.76
0.87
0.85
Qwen2.5-7B-Inste gemma-2-9t-it granite-3.0-8b-ae --bb--.s.tbb--nsruuctt
0.83
0.84
0.84
0.93
0.70
0.70
0.71
0.71
0.84
0.83
0.73
0.72
0.74
0.74
0.90
0.89
This confirms that the confident failures we found are systematic, and these models often have col- lapsed posteriors incorrect answers, oversampling cannot recover the uncertainty. The ensemble overhead provides qualitatively new information (cross-model disagreement) that is inaccessible to single models.
A.10 COMPUTING CORRECTNESS USING AN LLM JUDGE
We compute correctness scores using Meta-LI ama 3- OB-IInstrcc deployed via a local vLLM server. Each model prediction is evaluated independently against the gold answers using structured prompt that includes five few-shot examples, held fixed across all evaluations. The prompt instructs the judge assign correctness score from the discrete set (0.0, 0.1, based on the alignment between the predicted and gold answers, while explicitly ignoring the model's own knowledge.
O
The judge receives input: (1) the user-defined task question, (2) a list of gold answers, and (3) the model-ge genneatee answer. is instructed to output JSON object containing numerical score and a justification. The request is submitted using deterministic decoding (temperate re= O, max_tokent 20), and we employ up to three retries with truncated context in case of failures due to prompt length.
Correctness is evaluated using the first response generated by each model. The gold answers are passed verbatim, and no normalization is applied to either predictions references. By default, a prediction is considered correct its score exceeds 0.5. For tasks where differences should be penalized (c.g., summarization translation), we increase the threshold to 0.9 (specifically, for XSum and WMT16 6 -ene-en). These thresholds are applied during AUROC and selective prediction evaluations.
26

---

Published as a conference paper at ICLR 2026
Judge Evaluation. We assess the reliability of the LLM judge used to correctness against gold references. First, we perform a cross-judge check (Llama-3-76) --aa----7---nttrue vs. GPT-4o) and find high agreement both probabilities and binary labels. In 93/100 cases the judges produced identical correctness probabilities. In 5/100, probabilities differed but mapped to the same binary label under the 0.5 threshold, AUROC is unchanged. Only 2/100 items were hard conflicts, and manual inspection favored GPT-4o in both.
Second, We manually audited 100 Mistral-generated samples across TriviaQA and TruthfulQA and observed <6% human-judge disagreement. We also compare to rule-based matchers from in--evaluation-hareeee and find them brittle for free-form outputs, which motivagtes an LLM judge.
Prompt Format. We provide the prompt format used for the LLM Judge here.
I want you to act as a judge for how well model did answering a aser-defined task.
You will be provided with user-defined task that was given to the model, its golden (s), and the model's s answer. The context of the task may not be given here. Your task is to judge how correct the model's answer is based on the golden answer (s), without seeing the context of the task, and then give correctress score. The correctness score should be one of the below numbers: 0.0 (totally wrong), 0.1, 0.2, .... 1.0 (tetally right). brief just cation regarding how the model's answer conforms to or radt cts the golden answer (s). Your resporse must follow the format:
should also add
"correctness score": soorsscore, "justification": your_justification
Note that each of the golden answers is considered correct. Thus if the model's answer matches any the golden answers, should considered correct.
Example 1:
User-defined task Sandy bought million Safe Moon tokens. She has siblings. She wants to keep half of them to herself and divide the remaining tokens among her siblings. After splitting it up, how many more tokens will she have than any of her siblings?
Golden Answer
<answer 1> 375000
Model's Answer Sandy will have tokens than any sibling by 3/8 million.
Model Output:
"correctness "correetness_scozee"" sccores 1.0,
" "Justification": "The model's s answer 3/8 million equals 375, 000, which matches the gold answer exactly."
more examples)
Target Example:
User-defined task -- [QUESTION]
Golden Answer (a) <answer 1> :...l; <answer 2> [...]
Model's s Answer [MCDEL RESFONSE]
Model Output:
{
"correctness_scomt
"justification
27


Reviews:
[{"review_id": "MnlacBFN5K", "reviewer": "Reviewer_vtPS", "summary": "This paper addresses a critical limitation of existing uncertainty quantification (UQ) methods for large language models (LLMs): the failure of self-consistency-based aleatoric uncertainty (AU) to detect \"confident yet incorrect\" responses (where models repeatedly produce the same wrong answer, leading to low AU). To solve this, the authors propose three core contributions: 1) They identify that cross-model semantic disagreement is disproportionately high for incorrect answers when AU is low, a key empirical insight. 2) They introduce an epistemic uncertainty (EU) metric designed for black-box LLM access—EU is computed as the gap between inter-model and intra-model sequence-semantic similarity, using only outputs from a small, scale-matched ensemble (5×7–9B instruction-tuned models). 3) They define Total Uncertainty (TU) as the sum of AU and EU, and validate it across 10 long-form tasks (QA, summarization, translation, math reasoning) plus API models (GPT-4o, Claude 3.7 Sonnet). Experiments show TU outperforms AU in ranking calibration (AUROC) and selective abstention, while EU reliably flags confident failures missed by AU. The work also characterizes EU’s utility via agreement (Jaccard redundancy) and complementarity (oracle coverage gain) diagnostics, highlighting its effectiveness for tasks with unique correct answers (e.g., factual QA, translation).", "questions": "1. How do the performances of EU and TU change when adjusting the number of auxiliary models (e.g., 2 vs. 5) or their diversity (same-family vs. cross-family)? For instance, can an ensemble of 2 cross-family models achieve performance close to that of an ensemble of 5 models? This is crucial for the deployment of TU in resource-constrained scenarios .\n2. For tasks with multiple valid answers (such as XSum), can an improved version of TU be designed? For example, weighting EU by an \"answer uniqueness score\" (e.g., the number of correct answers for a single input) to avoid misclassifying differences between valid answers as uncertainty .\n3. Please supplement the comparison of AUROC and AURC between TU and methods such as Kernel Language Entropy (Nikitin et al., 2024) and Self-CheckGPT (Manakul et al., 2023) on the same tasks, so as to clarify the position of TU in the field .\n4. What specific types of problems do the \"confident failures\" flagged by EU belong to? Are they knowledge gaps (e.g., incorrect facts) or reasoning biases (e.g., flawed multi-hop logic)? A qualitative analysis can help readers understand the boundary of EU’s applicable scenarios .\n5. The paper verifies the effectiveness of TU when using 7–9B reference models. If the scale of the reference model is expanded (e.g., 70B+), will TU still be effective? For example, large-scale models may be closer to the ideal model ω*, so will the gain from EU decrease in this case ?\n\n\nIf you address my concerns, I will consider adjusting your score upward accordingly.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Proposes an EU estimation method for black-box scenarios, leveraging the ecological advantages of open-weight LLMs (scale-matched ensembles). This avoids the strong task/architecture dependencies of existing methods, demonstrating prominent originality.\n2. The method design is theoretically grounded (kernel divergence bounds), with rigorous experimental control of variables (sample budget, judge validation), ensuring high reproducibility of results.\n3. On tasks such as HotpotQA (multi-hop reasoning) and WMT16-de-en (translation), TU achieves an AUROC improvement of over 0.1, significantly enhancing LLM reliability and providing practical guidance for deployment in high-stakes scenarios.", "weaknesses": "1. The paper fixedly uses an ensemble of 5 7–9B models but fails to explore how ensemble size (e.g., 2–3 models) or diversity (same-family vs. cross-family) impacts TU. For example, is a small-scale ensemble still effective in resource-constrained scenarios? And what effects would a large parameter gap between ensemble models have? The lack of such analyses limits the practical application scope of TU.\n2. Although the paper notes that TU performs poorly on tasks with multiple valid answers, it does not propose solutions. For instance, could EU weights be dynamically adjusted via an \"answer uniqueness score\"? Or could TU calculation be optimized by incorporating task characteristics (e.g., summary relevance vs. QA factuality)? Current analysis only \"identifies the problem\" without advancing to \"solving the problem.\"\n3.  It only compares with traditional baselines (semantic entropy, perplexity) and fails to comprehensively benchmark against methods in the LLM UQ field (e.g., SelfCheckGPT), making it impossible to clearly demonstrate TU’s relative advantages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BdT7SK29rD", "reviewer": "Reviewer_KyTH", "summary": "This paper addresses unreliable uncertainty estimates in large language models by **complementing self-consistency (aleatoric uncertainty, AU)** with a black-box estimate of **epistemic uncertainty (EU)** derived from **cross-model semantic disagreement**. AU is measured as *intra-model* response similarity across samples; EU is defined as the gap between *inter-model* similarity (a small, scale-matched ensemble of open-weight models) and *intra-model* similarity, and **Total Uncertainty (TU) = AU + EU**. Using only generated text (no logits), this paper evaluates TU across five 7–9B instruction-tuned models on ten long-form tasks (QA, translation, summarization, math), and also on API models (GPT-4o, Claude). TU consistently improves **correctness calibration (AUROC)** and **selective abstention** over AU, and EU is particularly informative on **confident failures** where AU is low. Diagnostics relate EU’s usefulness to dataset redundancy vs. complementarity, clarifying when cross-model disagreement best predicts error.", "questions": "1. **Similarity backbone.** How sensitive are AU/EU/TU to the sentence encoder and to length normalization? Please report alternatives and length-controlled evaluations.  \n2. **Ensemble weighting.** Can EU improve by weighting auxiliary models by **posterior credibility/validation risk** rather than uniformly?  \n3. **Budget–performance trade-offs.** Provide wall-clock, memory, and FLOP profiles vs. *(m, n)* and show Pareto curves (accuracy/calibration vs. cost).  \n4. **Triggering EU.** Explore **adaptive triggering** (invoke cross-model sampling only under low AU or specific heuristics) to cut cost while keeping gains.  \n5. **Open-ended tasks.** For many-valid-answer settings, consider **reference-guided similarity**, entailment scoring, or diversity-aware TU that discounts benign variation.  \n6. **Judging robustness.** Verify results with multiple judges and human audits; report inter-judge agreement to strengthen calibration claims.  \n7. **Generality.** Test beyond 7–9B models and add domains like code or multi-modal tasks; study TU under retrieval-augmented generation and distribution shift.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Originality.** This paper proposes a **text-only, black-box EU** estimator via cross-model semantic disagreement and shows how it complements AU; the additive **TU** is simple and broadly applicable.  \n- **Quality.** Clear formulation (AU/EU/TU as intra-/inter-model similarities), principled ensemble construction (scale-matched, cross-family), and comprehensive experiments (10 tasks, multiple models, risk–coverage, AUROC, ablations on ensemble/sampling).  \n- **Clarity.** Well-motivated failure mode (overconfident, wrong, low-AU generations), intuitive figures, and diagnostics that explain **when** EU helps (unique-answer tasks; high redundancy).  \n- **Significance.** TU improves calibration and abstention across diverse models and tasks, including API models, without requiring logits or retraining—practical for black-box deployments.", "weaknesses": "- **Multiple-valid-answers regimes.** On open-ended tasks (e.g., abstractive summarization), semantic diversity among valid outputs can inflate EU and weaken AUROC; this paper acknowledges but does not fully resolve this.  \n- **Ensemble design & weighting.** Uniformly weighted, scale-matched ensembles are pragmatic but may bias EU; robustness to **ensemble composition**, size, and weighting (e.g., by validation risk) needs deeper study.  \n- **Labeling & metrics.** Reliance on LM-as-a-judge for correctness and a single sentence-embedding similarity can introduce bias (length/style effects); sensitivity analyses are limited.  \n- **Compute budget.** Although sample budgets are matched to AU, running **m models × n samples** adds latency/cost; end-to-end throughput/memory profiling is not fully reported.  \n- **Theory.** Additive TU is intuitive, but formal guarantees (e.g., bounds linking TU to error) are not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qh6ZTlbhXA", "reviewer": "Reviewer_iWhX", "summary": "This paper proposes a unified framework for estimating total uncertainty in large language models by decomposing it into aleatoric uncertainty and epistemic uncertainty. Instead of relying on output probabilities, the authors measure uncertainty through semantic similarity between generated responses, defining TU as the expected dissimilarity between samples from the same or different models.", "questions": "1. A weighted version of TU/EU, reflecting the posterior quality of each auxiliary model, could further strengthen the analysis.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Relevant and timely topic.\nThe paper tackles an important and underexplored problem. This is a meaningful direction for improving reliability and selective prediction in modern LLMs.\n2. Extensive and systematic experiments.\nThe authors evaluate the proposed method across diverse datasets, model scales, and uncertainty estimation tasks. The experimental coverage is broad and the reported metrics (AUROC, AURC, selective prediction) show consistent gains over standard AU-based baselines.\n3. Well-articulated discussion of limitations.\nThe paper clearly acknowledges the limitations of EU estimation (e.g., its reduced discriminative power in open-ended or multi-answer tasks) and provides thoughtful reasoning for these observations.", "weaknesses": "1. Limited methodological novelty.\nWhile the proposed TU = AU + EU framework is conceptually clear and practically useful, the mathematical formulation largely builds upon existing uncertainty decomposition ideas and replaces probability-based divergences with semantic similarity measures. The innovation is therefore more in engineering design than in theoretical advance.\n\n2. Lack of computational cost analysis.\nThe paper should provide an analysis of runtime and compute overhead. In particular, the multi-model sampling required for TU/EU estimation can be significantly more expensive than single-model AU estimation. Reporting inference latency, token usage, and compute normalization (e.g., equal-time or equal-token comparison) would make the evaluation more rigorous.\n\n3. Missing ablation on judge-model sensitivity.\nSince semantic similarity plays a central role, the conclusions may depend on the specific LLM used as the ``judger''. An additional experiment using alternative LLMs  as similarity evaluators would clarify how robust the TU/EU metrics are to the choice of judge model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RKHSeax2ln", "reviewer": "Reviewer_otCz", "summary": "This paper addresses a key failure mode of existing LLM uncertainty metrics like self-consistency, which primarily estimate Aleatoric Uncertainty (AU). These methods fail when models are \"confidently wrong\"—producing the same incorrect answer with high consistency. To overcome this, the authors introduce a complementary Epistemic Uncertainty (EU) term, which operates in a black-box setting by measuring the \"cross-model semantic disagreement\" within a small ensemble of different, scale-matched models. The study finds that this EU term reliably flags incorrect answers precisely when AU is low, effectively identifying these confident failures. By defining Total Uncertainty asTU = AU + EU, a comprehensive study—spanning five 7-9B models, GPT-4o, and Claude, across ten long-form tasks—demonstrates that TU significantly outperforms AU alone in distinguishing correct from incorrect responses.s", "questions": "1. Why is $TU = AU + EU$ the optimal combination? Was a simple additive sum found to be superior to a weighted sum?\n2. The paper motivates \"scale-matched\" ensembles , but ablations (Fig. 6, App. A.5)  suggest *stronger* auxiliary models yield better TU. Please clarify this discrepancy. Is scale-matching or ensemble strength the key factor?\n3. How does TU handle 'homogeneous failures,' where all ensemble models are confidently and *identically* wrong? In this case, EU would be zero, and TU would fail.\n4. Why was the evaluation for WMT16-de-en and XSum based *only* on the LLM-Judge, rather than also including standard metrics like BLEU or ROUGE?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The EU methodology is black-box and highly practical, as it relies solely on the model's generated text outputs rather than requiring access to internal logits or weights.\n2. The experimental design is rigorous. The study spans five different 7-9B open-source models and two top-tier API models (GPT-4o, Claude), evaluated on ten diverse long-form tasks using two standard metrics: AUROC (for ranking calibration) and AURC (for selective abstention).\n3. The method is effective, with the proposed Total Uncertainty (TU) metric demonstrating significant and consistent improvements over the Aleatoric Uncertainty (AU) baseline across nearly all tasks and models.", "weaknesses": "1. The method's performance is task-dependent. It is acknowledged to perform poorly on tasks that permit multiple semantically distinct yet correct answers (e.g., open-ended generation or summarization), where disagreement does not imply incorrectness.\n2. The approach is highly reliant on the diversity of the model ensemble. If all models in the ensemble share similar biases, they may fail homogeneously by confidently producing the same incorrect answer, rendering the EU metric ineffective.\n3. The use of an LLM-as-a-judge for evaluation lacks robustness. Any misjudgments or inherent biases from the judge model directly compromise the precision of the paper's core conclusion that TU outperforms AU.\n4. The method incurs substantial practical overhead due to the need to load and run multiple models. While the authors suggest amortization via 5x hardware parallelism, this solution itself represents a much higher hardware and deployment cost than the single-model baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)