CoCoA: A Minimum Bayes Risk Framework Bridging Confidence and Consistency for Uncertainty Quantification in LLMs
Authors: Roman Vashurin
Maiya Goloburda
Albina Ilina
Aleksandr Rubashevskii
Preslav Nakov
Artem Shelmanov
Maxim Panov
Venue: NeurIPS 2025 poster ()
Keywords: LLM
Large Language Model
Uncertainty Quantification
Minimum Bayes Risk
Primary area: deep_learning
TL;DR: A new method of uncertainty quantification for LLMs based on minimum Bayes risk framework combines model confidence with observed consistency.

Abstract: Uncertainty quantification for Large Language Models (LLMs) encompasses a diverse range of approaches, with two major families being particularly prominent: (i) information-based, which estimate model confidence from token-level probabilities, and (ii) consistency-based, which assess the semantic agreement among multiple outputs generated using repeated sampling. While several recent methods have sought to combine these two paradigms to improve uncertainty quantification performance, they often fail to consistently outperform simpler baselines. In this work, we revisit the foundations of uncertainty estimation through the lens of Minimum Bayes Risk decoding, establishing a direct link between uncertainty and the optimal decision-making process of LLMs. Building on these findings, we propose CoCoA, a unified framework that integrates model confidence with output consistency, yielding a family of efficient and robust uncertainty quantification methods. We evaluate CoCoA across diverse tasks, including question answering, abstractive text summarization, and machine translation, and demonstrate sizable improvements over state-of-the-art uncertainty quantification approaches.

CoCoA: A Minimum Bayes Risk Framework Bridging Confidence and Consistency for Uncertainty Quantification in LLMs
Roman Vashurin"
Maiya Goloburda
Albina Ilina
Aleksandr Rubashevski
Preslav Nakov
Artem Shelmanov
Maxim Panov
Mohamed bin Zayed University Artificial Intelligence (MBZUAI)
(Roman. Vashurin, Maiya. Goloburda, Paaam..anovveee jamrov@@mmzuaicccm ac..ac
Abstract
Uncertainty quantification for Large Language Models (LLMs) encompasses diverse range of approaches, with two major families being particularly prominent: (i) information-basee based, which estimate model confidence from token-level proba- bilities, and (ii) consistency-basee -bsed, which assess the semantic agreement among multiple outputs generated using repeated sampling. While several recent methods have sought to combine these two paradigms to improve uncertainty quantification performance, they often fail consistently outperform simpler baselines. In this work, we revisit the foundations of uncertainty estimation through the lens Minimum Bayes Risk decoding, establishing direct link between uncertainty and the optimal decision-making process of LLMs. Building on these findings, we propose CoCoA, unified framework that integrates model confidence with output consistency, yielding family of efficient and robust uncertainty quantification methods. We evaluate CoCoA across diverse tasks, including question answering, abstractive text summarization, and machine translation, and demonstrate sizable improvements over ofthe-of-the-rt uncertainty quantification approaches.
Introduction
Large Language Models (LLMs) have rev- olutionized natural language processing (NLP), enabling advances in information retrieval (Zhu et al., [025]. question an- swering (Kwiatkowski et al. [2019), ma- chine translation [Kocmi and Federmann 2023), and broad range of other NLP ap- plications. As these models become inte- gral part of our everyday life, ensuring the reliability of their outputs is crucial, espe- cially in high-stakes scenarios where errors can have serious consequences. One way to address this challenge is through uncer- tainty quantification (UQ), which focuses on estimating the confidence of model pre- dictions.
Tyoalt character which Shakespeare's plays?
Sampled answers
Probability
and juliet
Figure 1: Example of inconsistent probabilities assigned to semantically identical answers by an LLM,, demon- strating the limitation of relying solely on sequence- level information.
These authors contributed equally.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Evalusted sequesce y+
Fidby
a/ (x] 0.52
Alteruaivv snowers
Crociaaiolee = 0.022
Frompt: "Vadrref" a French
LLM
Fikday
Unmolp//s) 0.1
The Triday
0.16
Destusted sequancce
Toys4a
u(B-(z) 0.84
Pronept Which manuracher buide the 'Mvvnn rarge reviiuses??
Altemative srowers
Unmaaissle) - 0.739
LLM
Vesaell
0.65
Adton Narts
0.80
Figure 2: Illustration of our method: the LLM generates a response, evaluates the similarity to alternatives, computes the confidence, and finally combines the confidence with the similarity measure. High similarity to alternatives reduces the uncertainty, while low similarity keeps it high.
UQ for LLMs is rapidly advancing research area, with new UQ methods emerging each year. Most novel techniques are based on two fundamental approaches: (a) ceffommttion-theorett analysis or (b) assessment of output consistency.
cnoommtiiontthorretee methods quantify the confidence of model by analyzing the probability distributions it induces for predictions [Malinin and Gales 2021 Fomicheva et al. 2020). A key limitation of these methods that they cannot account for semantic variability across multiple possible responses for the same input. Specifically, the model may generate answers with the same meaning but with very different assigned probabilities; see Figure [ LLMs are trained to predict the next token in sequence based patterns observed in vast amounts of data, resulting in varying probabilities for semantically equivalent output sequences.
In contrast, consistency-bes -ased methods directly analyze the semantic relationship between the sampled outputs [Fomicheva al. 2020. Lin et al., 2024), capturing uncertainty as objective variability of meaning among the sampled outputs.
Information--heorrtic and consistency- based methods have complementary strengths. For this reason, recent state-of-th tate--ffthhe-rt methods aimed to unify these approaches [Kuhn et al. 2023 Duan et al. 2024]. We follow this direction and propose way of quantifying risk combination these basic measures, This results in family of efficient and robust UQ techniques. Our approach, illustrated in Figure 2 combines the strengths of both information-base and consistency-basee methods, providing more comprehensive and accurate assessment of uncertainty.
Daheim et 2025 that UQ in NLP tasks can be viewed through the lens of the Minimum Bayes Risk (MBR) framework. Following this, we formulate our methods as particular forms of risk functions under the MBR approach.
Recently, it has been argued [Wang and Holmes 2025
Our main contributions can be summarized as follows:
We propose new way of quantifying risk that combines information-theone and consistency-basee measures, and derive a family of Confidence and Consistency-based Approaches (CoCoA) uncertainty quantification in LLM
We show that the consistency component can be approximated by learned function trained on unlabeled held-out set with negligible loss of performance, eliminating the need for costly repeated sampling from the LLM. We call this variation of our method CoCoA Light.
We evaluate our approaches across a variety of NLP tasks, including question answering, summarization. and machine translation. Our experiments demonstrate sizable improve- ments in the reliability and the robustness of UQ compared to state-of-the-are methods.
Our code is available publicly at athttps://gihuu..co/ssta ml/llm uncertainty.com

---

2 Background
2.1 Language Model Decoding
LLMs define a probabilistic output distribution p(y x) for given input sequence x. The standard ways to obtain particular output y. given p(y x) include various variants of sampling y. ply x) and greedy decoding, where
y+ arg max ply x).
(1)
y
An alternative approach is to use Minimum Bayes Risk (MBR) decoding Kumar and Byrne
2004a
y+ arg min R(y x):
yEV
where y is the set of candidate sequences and Rly x) is a risk function:
R(y x) = Ryrxp(x) r(y,y')
(3)
with r(y.y') being a pairwise loss function.
The standard choice in the MBR literature is take r(y, y') -s(y, y'), where the so-called utility function y') represents some notion of similarity between generated sequences y and y'. Below we discuss how various decoding strategies and corresponding risk functions lead well-grounded definitions of uncertainty.
2.2 From Risk to Uncertainty
Generally, an uncertainty function U is mapping that quantifies the level of uncertainty associated with the output of a model y. conditioned on the input sequence x, which we denote as U(y x).
Bayesian uncertainty measures are well known to be strongly connected with minimum risks of various kinds [Xu and Raginsky 2022 Kotelevski et al., 2025]. Recently, this connection was revisited in the context of natural language generation [Wang and Holmes 2025 Daheim et al. (025). leading to the development of new MBR-based UQ methods. More precisely, one can consider the smallest achievable risk within y' or, equivalently, to maximum expected utility:
UMBR (y, x) min R(y x) -R(y= x),
yey
where y. is given by ( Interestingly, MBR-based uncertainty is applicable even to y consisting of single sequence y..
Below, we discuss how various existing uncertainty measures can be seen as particular approximations of minimum Bayes risks for various choices of utility functions. We also derive new uncertainty measures based on the MBR framework.
Single Sequence Information- Methods. Information- based methods estimate the uncertainty of the generated sequence by aggregating the uncertainty scores of individual tokens. Within the framework of MBR decoding, consider r'y, = 1 # y). In this case, the Bayes risk corresponds to the expected zero-one loss of decoding:
Roy1 (y x) Ey fy' # y) = ply xx.
(5)
This leads to one of the simplest information-base uncertainty measures, Sequence Probability (SP):
Usr(y= x) = p(y xx.
(6)
Although widely used as an uncertainty measure across various applications, including natural language generation, its connection to statistical risk has only recently been explored in the UQ
literature Kotelevskii et al. 2022 2025 AAihhberger et al. [2024].
Several other measures fall into this category, including Perplexity and Mean Token En- tropy [Fomicheva et al. 2020]; see Appendix Jppendix[D.]]tiann for details. While using only a single sample makes them computationally efficient, these techniques face three major challenges:
LLMs only give the probability of a specific answer, even though the same meaning can often be conveyed in multiple ways. To obtain a proper probability for the meaning of an answer, we need marginalize over its various possible rephrasings. However, this not feasible if we generate only single sample.
3

---

2. The reliability of such methods depends heavily on the underlying LLM being well- calibrated, which is a quality that hard define and harder to ensure.
3. These methods are point estimates that do not provide information about the shape of the output distribution.
Semantic Consistency- Based Methods. The aforementioned issues lead to the development of consistency-based methods based on repetitive sampling from the LLM. Consider that we have sampled a set of outputs { M i=='' where y()) ~ p(y x). Consistency-based UQ methods rely on
the diversity of the answers ylt) sampled from the LLM. The idea is that if the model outputs similar answers for the same prompt over and over again, it confident in its predictions; otherwise, is uncertain. These techniques do not require probability distribution estimated by LLM and can be applied in a black-box setting, where only the generated tokens are available. This case is quite common when LLMs are deployed as service and accessible through limited API.
Formally, consisteney-basee methods start from defining some similarity function syy E [0. between arbitrary LLM generations y and y'. The value s (y) = indicates the complete equivalence between y and y', and (y. 0 indicates that there is no similarity. The similarity could be computed in various ways. For instance, the Lexical Similarity method (Fomicheva et al.] 2020] relies on surface-form similarity, measuring the degree of word-level or phrase-level overlap between the generated texts. More advanced techniques propose various methods for taking into account the semantic similarity between the generated answers by means of hard or soft clustering [Ln ct al. 2024].
Finally, given M samples from the model, standard consistency- based methods compute a similarity matrix S. where 8y s(y ((y()) y()).. Then, various statisties of are computed in order to estimate more detail. Alternatively, one focus particular output y. and compute uncertainty measures based on its similarity S.i syy,,y()) with 2024 for more detail.
the uncertainty [Lin al.
2024| see Appendix(D.2)
other samples (*); see [Lin et al.
Interestingly, it was recently shown Wang and Holmes 2025 Daheim et al. 2025) that the latter approach has direct relation to MBR decoding. Building the MBR framework, one can define the utility function as r'y,y') = - s(y, y') and derive the following consistency- -based uncertainty score:
Ucons (y. x) = Eyr~pty s'y,,'']]
Then the Monte Carlo approximation of the minimum Bayes risk is given by
Σ Ü (y= 1x) M
1 M (1 -aa),
(8)
where 8.: = ((),
In our ablation study below, we show that such uncertainty measure reliably outperforms consistency-base measures that aggregate the pairwise similarities of all samples (see Appendix C.2).
A key strength of consistency- -bsed techniques is that, by generating multiple samples and analyzing their semantic similarity, they can estimate empirical probabilities over meanings rather than over individual answers. Their main drawback is that they discard the useful information that comes from the probability distribution represented by the LLM, including estimates of the probabilities of the specific answers.
3 CoCoA: Bridging Confidence and Consistency for Better Uncertainty Quantification
Both information-bas based and (semantic) consistency- based methods provide grounded and useful uncertainty quantification measures. There exist approaches that bridge the gap between information- based and consistency-based methods that show great promise but lack the fundamental base; see discussion in Section In what follows, we present a family of Confidence and Consistency-baver Approaches (CoCoA) for UQ offering a new way to merge information- and consistency- measures for uncertainty quantification in LLMs via the unifying MBR-basch framework.

---

If we only consider the consistency measure, e.g., given in 1 we miss the information that is contained in the model confidence. Thus, we want the resulting uncertainty measure to explicitly consider both the semantic consistency and the model-based uncertainty. Let us consider a risk of the following form:
r'y,y' x) u x) - syy,))))),
where
s syy [0, 1] is any utility function used in the standard MBR decoding (usually repre- senting semantic similarity);
"(y x) 2 is the model-based uncertainty measure for the output sequence y. For example, we use u(y x) -y x) (similarly to (6) or u(y x) - logp(y x).
Then, the corresponding Bayes risk is ReoCos (y x) a x) By'rn(xlb)) s(y, and for an output sequence y., the resulting uncertainty measure becomes
Ucacos (y- x) = uly, x) Ey'xp(yyx)) (1 sy,,
(10)
Finally, given a set of samples {y(1) ) i=11 we obtain an empirical uncertainty estimate:
M
Ococos 1x) = u(y x)
M Σ i=1 (1 11--88) 885) = u(yx x) Ucons(y x),
(11)
where S.z = s().,
The resulting uncertainty measure integrates both global (semantic) and local (model-specifiic uncertainty signals. It ensures that uncertainty is amplified for sequences y. that both intrinsically uncertain (high u(y- x)) and semantically inconsistent with respect to the other samples (high
M EE M (1 - se:)), while keeping it low for the opposite scenario; see Figure 2 for an example.
CoCoA Light. Computing the consistency-badd
uncertainty measure (y | x) within
UCocoAly x) requires multiple samples, whoch poses significant computational overhead. To address this, we propose to approximate the behavior of (x) with a learned function. Our method closely matches the original sampling- measure in quality while requiring only greedy output generation during inference, eliminating the need for additional sampling. Notably, this approximation can be learned without access to ground-truie truth labels, relying solely on the input data and the associated uncertainty values.
Thus, the CoCoA uncertainty measure that incorporates an approximation of the consistency uncer- tainty ÜL Jo (y. x) R Ü Ucons(X (y. x) can be defined follows:
Ulocoaay, x) = u(y. x) Ü (y+ xx)
(12)
We name this approach CoCoA Light. Below, we describe how this approximation can be obtained.
Assume access a held-out set of input sequences xy. j i...., n. We emphasize that this held-out set is not labeled. For each sequence, we extract the embeddings {e(y, | x,) the corresponding greedy model generations yj. The corresponding targets are the consistency uncertainty scores Uamslyy =1),j=1,..... 1,..,, n, computed via Monte Carlo estimation of the minimum Bayes risk; see
equation
lightweight auxiliary model, g(e(y= x)), trained in supervised fashion to map model embeddings to uncertainty scores. During inference, the model generates a greedy output Viest for test input Xist from which the embeddings e(Yiest Xiea) are obtained and passed to the auxiliary model to predict the uncertainty score:
i L (Yrest Xxex) (e Xiest))
(13)
We note that similar approach Kossen al. 2024 has been previously proposed to approximate semantic entropy using the hidden states from the model.
5

---

4 Related Work
In this section, we review existing approaches to uncertainty quantification, as well as prior work on Minimum Bayes Risk in application to natural language generation.
Information-based methods. These methods are one of the most commonly used. They quantify uncertainty by analyzing the probability distributions of the tokens within a given output. The simplest of these methods, Sequence Probability (SP), measures the probability of the sequence given a specific input. Perplexity is another common uncertainty measure: it gauges how well the model predicts each token in a sequence and is formally defined as the exponential of the mean negative log-likelihood of those tokens [Fomicheva et al. 2020). The Mean Token Entropy method evaluates the token-level predictions across the entire sequence by computing the average entropy of the token probability distributions at each position [Fomicheva et al., 2020]. While these methods provide useful uncertainty estimates, they do not address the broader uncertainty inherent in the generative tasks, as they rely on a single sequence and fail to capture the diversity of the possible outputs that the model could generate for the same input.
Consistency-Baid -Based Methods. These methods estimate uncertainty by generating multiple se- quences from the model's 's output distribution and analyzing the variability among the sampled outputs. These methods are particularly relevant to NLP tasks, as they account for scenarios in which multiple plausible outputs may exist for given input. Among the simplest consistency-ban bbassdd methods are the Number of Semantic Sets and the Sum of Eigenvalues of the Graph Laplacian [Lin et al., (024). which quantify uncertainty by how many distinet "meanings" the model produces. While effective at capturing global variation, they do not yield uncertainty scores for individual responses. To overcome this, the diagonal of the Degree Matrix was proposed to assess how similar each output is to the rest, enabling per-response uncertainty quantification (Lin et al. 2024). A related approach, Lexical Similarity, operates by calculating the average similarity of words or phrases across every pair of responses in the sample [Fomicheva et al. 2020).
Information- -aasee Methods with Repeated Sampling. More recent methods attempt to reconcile output consistency with information based signals-i.e they not only look how varied outputs are, but also how probable each output is under the model's 's own generation process. For instance, Semantic Entropy (Kuhn et al. 2023 groups outputs into semantically homogeneous clusters- capstris distinct "meanings" that might just differ in phrasing-ane calculates entropy across these clusters. SentenceSAR refines this idea by defining "relevance" as the sum of pairwise similarities between a given sentence and others weighted by each sentence's 's model-assigned generation prohability (Duan (ct al. 2024). SAR combines the sentence-level relevance of SentenceSAR with token-level probability adjustments, attempting more fine-grained balance of semantic and token-level uncertainty (Duan ct al. 2024]. Semantic Density [Qiu and Müikkulainen 2024] evaluates uncertainty by evaluating how densely a model's generated response is situated within the semantic space of all possible outputs, with lower density indicating higher uncertainty. One limitation of sampling- based approaches is their computational cost: generating multiple outputs from the model can be expensive. Recent work has sought to address this by learning to predict semantic uncertainty directly from hidden states, allowing for the approximation of semantic entropy without repeated sampling [Kossen et al., 2024). Finally, BSDetector [Chen and Mueller 2024] proposes an additive composition of self-reported confidence with observed consistency, however, it relies on ability of the model to assess its confidence in text form. Another limitation is that it requires selection of trade-off coefficient between confidence and consistency parts.
While these probability measures can provide deeper insights, they also sometimes under- perform in practice [Vashurin et al. 2025), especially when the model's probability estimates are unreliable, or when important nuances of semantic diversity are lost in the weighting process. Proper balancing raw output consistency with generation likelihood remains open problem.
Minimum Bayes Risk for LLMs. Minimum Bayes Risk (MBR) decoding, originally used in machine translation [Kumar and Byme/ 2004b), has recently been applied to LLMs by incorporating posterior over model parameters to enable uncertainty-awan Complementing this, a Bayesian framework for subjective uncertainty quantification was developed yielding improved calibration in generation tasks (Wang and Holmes, 2025).
generation [Daheim et al.) 2025

---

Experiments
In this section, we present the experimental setup, the results, and the ablations.
5.1 Experimental Setup
To evaluate the efffectiveness of our proposed framework, we extended the LM-Polygraph li- brary [Vashurin et al., 2023 Fadeeva et al. 2023]. Since it already includes tools for calculating other uncertainty scores, it provided convenient and efficient environment for setting up and running experiments. The primary objective of our experiments is to evaluate whether our method offers improved performance in key tasks such question answering (QA), text summarization (SUM), and machine translation (MT), compared to existing baselines.
Datasets. For QA, we selected diverse datasets to capture a variety of challenges: TriviaQA [Joshi et al. 2017], an open-domain factual QA dataset; CoQA [Reddy et al.]
al.] 2019), a conversational QA benchmark requiring multi-turn contextual understanding; MMLU [Hendrycks et al. 2021). multi-
task dataset spanning 57 topics to test broad knowledge; and GSM8k [Cobbe et al. 2021]. which focuses grade-schoo math problems requiring logical reasoning. For translation, we evaluated our method on WMT14 French-Englise [Bojar et al. 2014] and WMT19 Ceerman-Engliin [Barrault let al. 2019]. Finally, for summarization, used XSUM [Narayan et al. 2018). dataset of complex documents paired with concise abstractive summaries. For all datasets, we follow [Vashurin et al. 2023 in subset selection, prompt formatting, and few-shot example sourcing.
Models. We evaluated our method on the base versions of three open-weight language models: LLaMA 3.1 8B [ouurro et al. 2023), Mistral 7B [Tiang et al. 2023), and Fal- con 7B [Tem 2024]. The open- weight nature of these models enables direct access to token probabilities, which is crucial for implement- ing our UQ method. All experiments were con- ducted using the base variants. We additionally report results on the larger Gemma 3 12B-Base model [Team et al. 2025). with detailed analysis provided in Ap- pendix [
Oracle
Random
Accuracy
innstrctioontunee
PRR_norm
Rejection rale
Figure 3: Dndiiction-Reeectiin Ratio (PRR) Curve illustrating the quality of the non-rejected predic- tions as function of the rejection rate. Oracle represents the optimal rejection strategy, Random a random rejection, and UQ is rejection based on the evaluated uncertainty quantification method.
Similarity Function. To measure the simi- larity between two generations, we use the RoBERTa-large cross-encoder model, fine- tuned on the Semantic Textual Similarity bench- mark dataset [Liu et al., 2019 Reimers and Gurevych] 2019 Cer et al.) 2017]. J. This model is widely regarded as one of the most reliable
and commonly used approaches for evaluating sentence similarity. The cross-encoder processes two sequences jointly and directly outputs similarity score ranging from O to 1, providing a nuanced measure. Appendix conTtiinss comparative experiments with cross-encoder and other choices of the similarity function, substantiating this choice.
Baselines. We compare the performance of the proposed method against a diverse set of baselines and state-of-the-art UQ scores, including confidence-based consistency-based. and hybrid approaches. For information-basee approaches, we evaluate Sequence Probability (SP), Perplexity (PPL.), Mean Token Entropy (MTE), Monte Carlo Sequence Entropy (MCSE), and Monte Carlo Normalized Sequence Entropy (MCNSE). In the consistency-based category, we consider the Degree Matrix (DegMat) and the Sum of Eigenvalues of the Graph Laplacian (Eig Vaiaapaaala))ia) Finally, we include hybrid methods Semantic Entropy and SAR, as well as verbalized confidence method P(true) [Kadavath et al. 2022 All formulations for these baselines can be found in Appendix D Finally, we evaluate the performance of Üowns (Consistency) and Ô (Consistency Light) as an uncertainty measure.
7

---

Evaluation Measure. As our evaluation measure, we chose the Prediction Rejection Ratio (PRR), which measures the effectiveness of the uncertainty scores for identifying high-quality predic- tions (Malinin and Gales 2021]. PRR operates by progressively rejecting predictions with uncertainty scores above a threshold a and observing how the average quality of the remaining predictions changes (see Figure B It is calculated as the ratio of two areas: the area between the Prediction Rejection (PR) curves for the evaluated uncertainty score and a random baseline, and the area be- tween the oracle (the ideal uncertainty score that perfectly ranks instances by quality) and the random baseline. Formally, PRR is defined as follows:
AUCune AUUCmd
PRR
(14)
AUCoraele AUCmd
Higher PRR values indicate better alignment of uncertainty scores with prediction quality, approaching the performance of an oracle. To ensure practical applicability, we compute PRR only up to a rejection threshold of 50%, preventing cases where excessive rejection artificially inflates the quality measures.
For the QA datasets, we further report AUROC (see Appendix G
Quality Measures. PRR requires an appropriate quality measure for each specific task to effectively evaluate the model output. For question answerrng tasks, we use Accuracy to directly evaluate whether the generated answers match the ground truth in short-form QA tasks (e.g., MMLU), and we use the AlignScore between the correct answer and generated sequence for assessing the performance for long-form QA tasks [Zha et al. 2023). For summarization tasks,
AlignScore measure the alignment between the output summary and the input document. It serves as a quality indicator by evaluating the relevance and the overlap between the generated content and the source text. For translation tasks, we use COMET, as it captures both semantic adequacy and fluency, ensuring that translations are accurate and linguistically appropriate [Rei al. 2020).
To further improve the comprehensiveness of the reported results, we considered alternative choices of quality measures, and report the corresponding PRR scores in Appendix F
Generation Setup. We discuss the generation parameters, the decoding strategy, and the sample selection procedure in depth Appendix [ In short, we report the evaluation results in two distinct setups: greedy decoding and stochastic sampling with focus the most probable sequence among the generated outputs (best-sample). These two setups offer the highest-quality outputs and are the most reasonable generation approaches in practice.
CoCoA Light Details. We use a multilayer perceptron network as an auxiliary model. The features used are embeddings from the middle layers of the base LLM, which the most informative according to recent work [Chen et al. [024]. In particular, for Llama 3.1 8B and Mistral 7B models, take embeddings from the 16th layer, and for Falcon 3 7B from the 14th layer. More details the training are available in Appendix
5.2 Results
Table shows the PRR scores under the greedy generation setup (See Appendix B for Best Sample and MBR Sample). We report aggregated PRR for each type of task question answering, neural machine translation (NMT), and summarization (SUM) - by averaging the results across all relevant datasets (c.g.. TriviaQA, MMLU, CoQA, GSM8k for QA). This aggregated score provides a concise measure of the performance for each model for each task. Detailed results for each dataset separately can be found in Appendix
We can see that our CoCoA methods are the best across all tasks and models. They outperform existing consistency- based and hybrid state-of-the-ar approaches, like Semantic Entropy and SAR. In addition, the proposed CoCoA approach consistently surpasses the baseline UQ measures: for example, CoCoAppr. outperforms standard Perplexity, illustrating the advantage combining token- level confidence with semantic consistency. This pattern holds for other information- -aaed metrics as well, demonstrating that using the consistency between multiple sampled outputs reliably enhances uncertainty quantification.
8

---

Metric
LlamaSb-Ban
Mistral7b- Base
Falcon7hr Talson7b-Base
QA
NMT
SUM
QA
NMT
SUM
QA
NMT
SUM
MCSE
0.310
0.323
0.033
0.389
0.304
0.007
0.414
0.317
0.159
MCNSE
0,309
0,393
0.022
0,384
0.410
0.009
0.405
0.422
0,108
Semantic Entropy
0.356
0.343
0.033
0.423
0.327
0.008
0.439
0.348
0.164
DegMat
0,406
0,302
0.081
0.423
0.305
0.137
0.483
0.353
0.201
Eig Vallccaa
0.375
0.238
0.079
0.391
0.267
0.132
0.459
0.312
0.201
SAR
0,414
0.455
0.077
0.462
0,435
0.094
0.481
0.458
0,144
P(Truc)
-0.064
0.042
0.058
-0.029
0.075
0.179
0.118
0.155
-0.159
Consistency
0,437
0,421
0.024
0.471
0.392
0,051
0.494
0,416
0,226
Consistency Light
0.390
0.458
-0.022
0.444
0.387
-0.006
0.427
0.476
0.232
SP
0.409
0.399
0.328
0.475
0.383
0.287
0.475
0.356
0.201
CoCoAsP
0.451 f
0.519 f
0.378
0.509
0.497
0.330 f
0.511 f
0.505 f
0.257
CoCoAsp Light
0.449 1
0.502 f
0.358
0.503
0.480
0.309 f
0.492 f
0,514 1
0.242
PPL
0.381
0.386
0.369
0.424
0.427
0.204
0.456
0.450
0.155
CoCoApPL
0,454 1
0.481 f
0.387
0.494
0.472
0,286 1
0.523
0,508 f
0.229
CoCoApPL Light
0.445 f
0.487 f
0.382
0.479
0.480
0.260 f
0.512 f
0.528 f
0.234
MTE
0.353
0.382
0.357
0.417
0.438
0.182
0.456
0.473
0.152
CoCoANTE
0.447 f
0.478 f
0.380
0.492
0,469
0.288 f
0.527 f 0.508 1
0.228
CoCoANTE Light
0.428 f
0.494 f
0.372
0.475
0.482
0.254 f
0.507 f
0.533 1
0.234
Table 1: Results for Evaluated Sequence Greedy Sample: Mcan PRR across datasets for each task. The best-performing method is shown in bold, and the second-best is underscored. The arrows indicate improvement in CoCoA over the base version.
5.3 Ablations
Similarity Function. We investigate the impact of different similarity measures (see Appendix C.T On average, the cross-encoder provides strong performance. However, for summarization tasks, AlignScore yields better results, while in long-form QA, similarity based on Natural Language Inference (NLJ) sometimes outperforms the cross-encoder. We hypothesize that for generation with shorter outputs (1-2 sentences), any capable NLI or cross-encoder model is sufficient. For longer generations, approaches that estimate similarity over chunks and then aggregate scores (e.g.. AlignScore) may be more effective. Nevertheless, across all tasks, the cross-encoder achieves the best overall performance and, in scenarios where tuning is not feasible, serves as a strong default choice.
Alternative Formulations. The next section of our ablation study focuses on alternative forms of combining model confidence u(y= x) and consistency Ücens(y- x); see Appendix C.2 First, we consider an additive form of combining them:
ÜAdiitveCacua (y, xx) u(y, xx + Ü (y. x).
(15)
The results show that this additive formulation does not perform as well as the multiplicative one. The additive form tends to underemphasize the interaction between the components, which is critical for capturing the nuanced relationships between confidence and consistency.
We also consider an alternative formulation of the consistency term Doons(y: x), as the average of the full pairwise dissimilarity. In this formulation, Ô. (y. x) represents the average inconsistency across all samples rather than focusing solely on the dissimilarity of the evaluated sequence with the other samples. Our experiments demonstrate that this formulation is not very strong. By distributing the consistency computation across all samples, it loses focus on the specific sequence being evaluated.
Lastly, in Appendix 2 we also consider alternative formulations of the information -aased metric that do not rely on logarithmic transformations. While we primarily use logarithms due to their numerical stability, we explore an alternative approach by converting these values back to probabilities and analyzing their impact on uncertainty quantification. Our findings indicate that both formulations exhibit consistent performance and yield similar results. This suggests that while logarithmic transformations enhance numerical stability, the choice between log-based and probability- formulations does not affect much the overall performance.

---

6 Limitations
While our proposed CoCoA approach demonstrates robust empirical performance, several important considerations remain.
Task and Domain Dependency. CoCoA relies information- confidence score and semantic similarity function, whose effectiveness can vary across models, tasks, and domains. In open-ended tasks like creative generation, producing diverse outputs is expected; on the other hand, tasks requiring precise reasoning can be sensitive to subtle errors that generic similarity metrics may miss. Adapting these components to specific domains remains important direction for future work.
Limited Sample Size. CoCoA estimates consistency by sampling multiple outputs. Generating many samples is computationally expensive, and thus our experiments, e.g., most sampling- methods, use relatively small sets. While even few samples can yield meaningful estimates, they may miss the full diversity of model outputs for complex prompts.
Quality Metric. Finally, the CoCoA's performance assessment depends on quality metrics (e.g., COMET for machine translation, and Accuracy for QA) that may not capture every nuance of textual outputs. Further refining or extending quality metrics to account for deeper reasoning. factual faithfulness, and stylistic appropriateness would better align uncertainty scores with real-world perceptions model correctness.
7 Conclusion and Future Work
We presented CoCoA, a unified approach that integrates Confidence and Consistency for uncertainty quantification in LLMs. By combining confidence scores with semantic similarity between multiple sampled outputs, CoCoA offers more holistic view of uncertainty than either approach alone. In extensive evaluations on question answering, summarization, and translation, our approach outper- formed existing baselines and state-of-the-ant UQ methods. Moreover, CoCoA's flexible design allows easy adaptation a variety of tasks and settings.
Moving forward, several directions are open for further exploration. These include incorporating more adaptive sampling strategies that efficiently capture the model output space, refining the semantic similarity functions for domain-specific tasks, like code generation or commonsense reasoning.
References
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey. ACM Trans. Inf. Syst., 2025. ISSN 1046-8188, doi: 10.1145/3748304, URL
https://doi wrg//0..///0/ 1145/31444//
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wee Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slay Petrov. Natural Questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics. 7:452 466, 2019. doi: 10. 10.1tal/ 00276. URL
https://aclanthologr. 1ttp://ooooo..o//1999/1/90110 10267
Tom Koemi and Christian Federmann. Large language models state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193-203, Tampere, Finland, 2023. European Association for Machine eamt-1 1.19/
Translation.
Andrey Malinin and Mark J. F. Gales. Uncertainty estimation in autoregressive structured predic- tion. In Proceedings of the Ninth International Conference on Leamning Representations, Vienna, Austria, 2021. htwpl//s....ooooeeeeeeee/////e/s
zb507m
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539-555, 2020. doi: 10. 1662//2/ 00330. URL https: https://aaclanthooogyy.. .rg 2020.000..00 tacl-1.35
10

---

Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantifi- cation for black-box large language models. Transactions on Machine Learning Research, 2024.
URL https://openrevieet hwtpppeddddddwwdddeeeeee
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation natural language generation. Proceedings of the Eleventh International Conference on Learning Representations, Kigali, Rwanda, 2023. Open URL https net/ wttp:/iwwwnnet/aa/diiio/eeo
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5050-5063 Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/ 1/20/////22442///// wal-:22..acllooo.... long.l6..66.... URL https://aclaanthooogg.orrr wwgp://....g//22..//.// acl-acllonnnseell long. 276
Ziyu Wang and Christopher C Holmes. On subjective uncertainty quantification and calibration in natural language generation. In Proceedings of the 28th International Conference on Artificial Intelligence and Statistics, Mai Khao, Thailand, 2025. URL httphhtt////opeeeeieew..t
id=D4HqXujpKA
Nico Daheim, Clara Meister, Thomas Möllenhoff, and Iryna Gurevych. Uncertainty-aware decoding with minimum Bayes risk. In Proceedings of the Thirteenth International Conference Learning Representations, Singapore, 2025. URL
Shankar Kumar and William Byrne. Minimum Bayes-risk decoding for statistical machine transla- tion. In Proceedings of the Human Language Technology Conference of the North American
Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169- 176, Boston, Massachusetts, USA, 2004a. Association for Computational Linguistics. URL
https://accanntoooog..orrg//////
Aolin Xu and Maxim Raginsky. Minimum excess risk in Bayesian learning. IEEE Transactions on Information Theory, 6812)7793,, 2022. doi: 10. 109//TT..0022.1766656
Nikita Kotelevskii, Vladimir Kondratyev, Martin Takáč, Eric Moulines, and Maxim Panov. From risk to uncertainty: Generating predictive uncertainty measures via Bayesian estimation. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025.
Nikita Kotelevskii, Aleksandr Artemenkov, Kirill Fedyanin, Fedor Noskov, Alexander Fishkov, Artem Shelmanov, Artem Vazhentsev, Aleksandr Petiushko, and Maxim Panov. Nonpara- metric uncertainty quantification for single deterministic neural network. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 36308-36323 Curran Asso- ciates, Inc., 2022. URL U neurips.....p....pp..m wcppaper.ppppppppppepeeeeeee.. wwwpparaileee//aaaaaaaaaapppeeee fil.es//ppper//2///llle pdf
Lukas Aichberger, Kajetan Schweighofer, and Sepp Hochreiter. Rethinking uncertainty estimation in natural language generation. arXiv preprint 1rX1v:2412.15176, 2024.
Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in LLMs. arXiv preprint 1tXiv:2406.15927, 2024.
Xin Qiu and Risto Mikkulaanen. Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space. In Proceedings of the Thirty-eight Annual Conference on Neural Information Processing Systems, 2024.
Jiuhai Chen and Jonas Mueller. Quantifying uncertainty answers from any language model and enhancing their trustworthiness. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, edi- tors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5186 3186-2000, Bangkok, Thailand, August 2024. Association for Computational I Linguisties. doi: 10. 1/00 188. URL, https:ttp//// aclathologgy org/2024. acl aclalong.....o...ooaeeoooaoellll 283/

---

Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing. Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. Benchmarking uncertainty quantification methods for large language models with IM-Polygraph. Transactions of the Association for Computational Linguistics, 13:220-248, 2025.
Shankar Kumar and William Bymne Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169-176, Boston, Massachusetts,, USA, May May 7 2004b. Association for Computational Linguistics. URL https://aclanntholooy org/oog///. 10227
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. L.M-Polyrrahh: Uncertainty estimation for language models. In Yansong Feng and Els Lefever, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 446-4611 Singapore, December 2023, Association for Computational Linguistics. doi: 10. 12166553112222..3mmmmemmmmmmm https://aclanthologg...o
URL
www.:p2....mmmpmmeemmmm emnlp-demo- 417
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1..18633////P171114 URL
org/
P17-1147
Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019. doi: 10.1 10.1162/tacl_a a 00266. URL/htttss//accaatttoooo
org/ wttp:/rg////1111111 1016
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceedings of the 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021. URL https://openreviiew net rett//ooooddddd///11aa
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1iii:22100111666 14168
2021. ftttp:://arxiv.org/abaaa
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. Findings the 2014 Workshop on Statistical Machine Translation. In Proceedings of the Ninth Workshop Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. URL http://www haaryaannn..///ttooooo////////// http://///annnnooooo////////////
Loie Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference on machine translation (WMT19). In Ondfej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Mauthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor, editors, Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers. Day 1), pages 1-61, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W 13-0031. URL https:://ccaanthoooggy orgg W19-530171
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujüi, editors, Proceedings of the 2018 Conference
12

---

on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. 2018. Association for Computational Linguisties. doi: 10.18653/v1/dd1- 1206. URL h
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023. URL https://doi.orr
48550/arXiv. 2ww..
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0688 2023. URL https: org/10. 48550/8550rra aarvvv
2380.06825
Falcon-LLM Team. The Falcon family of open models, December 2024.
URL
https
//huggingface co/blog/falcon3
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot. Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky. Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng. Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang. Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimezak- Pluciíska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman. Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy. Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa. pingmei Xu, Piotr Stanczyk, Pouya Tafti. Rakesh Shivanna. Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Pöder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, lean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alck Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. http::tts:////rr http://..org/////55..3 19786
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint 10XXvv1197..111.
2019. URL/https://arxiv org/ /bwp/9/7.... 11692
Nils Reimers and Iryna Gurevych. Sentence-BERY Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the
13

---

2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-ICCNLP), pages 3982-3992 Hong Kong, China, November 2019, Association for Computational Linguisties. doi: 10.18653/v1/ 019-1410) http:/ttss//aacantttooooo.ooroge
Daniel Cer, Mona Diab, Eneko Agire,, lñigo Sopezo and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer. and David Jurgens, editors, Proceedings of the 1 11th International Workshop Semantic Evaluation (SemEval-2011 pages 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 10.18665/v1/SS17-200 URL httptttt/caaaattthhotte www.h//o...oo.ooo//ooooo/ /r7/
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. arXiv Prerpint, arXiv:2207 2,220222000.0221,,, 2022. doi: 10.48550kar21..2227..555 URL httpps://dot
1155///045555//raii 2207 05221
Yuheng Zha, Yichi Yang. Ruichen Li, and Zhiting Hu. Alignscore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1 11328-11345 2023. https/ //aclanthology..om 2023. acllaacllannl 634
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685-2702, Online, 2020. Association for Computational Linguistics. doi: 10. 111118/61112000 emnlp- main.mmin.333 URL /ttpps://allanthloooo
org/2020. smnlp-maing 213
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs' internal states retain the power of hallucination detection. Proceedinsg of the Twelfth International Conference on Learning Representations, 2024. URL /ttps:ttt///oooeereeiie net// Forum?id-2j12n21100z
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with disentangled attention. In Proceedings of the 9th International Conference on Learning Representations. OpenReview.net, 2021. URL XPZIaotutsD
Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz, editors, Proceedings of the Ninth Conference on Machine Translation, pages 492-504, Miami, Florida, USA, November 2024, Association for Computational Linguistics. URL httpss//aclanttoolgg.oor org/222 1.
14

---

1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction accurately describe the proposed family of Confidence and Consistency-based Approaches (CoCoA) to UQ. Theoretical contributions include new methods that merge information- and consistency-bed LLMs. Experimental validation of the new approaches is done across different NLP tasks: question answering, summarization, and translation. It aligns with the content presented in Appendix B).
measures for UQ in
the paper (Section 3 5.2
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section discusses limitations, including relying on an information- con- fidence score and semantic similarity function, having limited sample sets, and depending quality metric.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper presents theoretical formulations: uncertainty measures derived from Minimum Bayes Risk (MBR) and the definition of CoCoA and its approximated
variant CoCoA Light (Sections) 2.2 and 3p. However, we do not make any formal theoretical
statements in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 5.1 outlines the experimental setup, datasets and evaluation metrics. The description of datasets used for each task includes TriviaQA, CoQA, MMLU, GSM8K for Question Answering, WMT14 -rench-English, WMT19 German-English for machine translation, and XSUM for summarization. The models used: LLaMA 3.1 8B, Mistral 7B and Falcon 3 7B. The similarity function is specified as RoBERTa-large fine-tuned on the Semantic Textual Similarity benchmark. The evaluation metric, Prediction Rejection Ratio (PRR), is clearly defined, and task-specific quality measures such as Accuracy, AlignScore, and COMET are used. Furthermore, the training setup for the CoCoA Light method, including the use of multilayer perceptron and middle layer embeddings of the base LLM, is described. The appendix provides additional details on ablation studies, similarity metrics, and decoding strategies (Appendix C and E
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
16

---

(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: Code is open source and is provided as an extension to the LM-Polygraph library (Section[5..)
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings, including model architecture (base LLaMA 3.1 8B, Mistral 7B, Falcon 7B), embedding layer selection for auxiliary training (16th or 14th layer depending model), and the auxiliary model structure (multilayer perceptron) are described Section 5.1 and Appendix A
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
17

---

The full details can be provided either with the code, in appendix, or as supplemental
material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not provide errorbars as it operates in (almost) deterministic scemario: we operate with pre-trained models and there is no training involved whatsoever. The only randomization involved is the one related sampling multiple sequences for consistency-basee methods. The choice to not compute errorbars here is deliberate as our inference involves heavy computations and it is not possible to conduct main experiments multiple times due to that. However, we conducted experiments on 3 different LLMs and 7 datasets, which is equivalent 21 independent run, demonstrating the robustness and reliability of our findings. For this reason, we do not report error bars, as the results are stable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative етrог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix H discusses computational resources used to produce experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
18

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp:///eerrrps..ccpppp//ccccct
Question:
Answer: [Yes]
Justification: Our research does not involve human subjects or ethically sensitive applications. We believe it conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Section (icioon[[iiie discusses the importance of Uncertainty Quantification, highlighting its positive societal impacts (helps to reduce overreliance on automated systems and en- courages responsible LLM usage). We do not foresee any negative negative societal impact of this work.
Justification:
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly cites the original sources of models (e.g., LLaMA 3.1 8B, Mistral 7B, Falcon 3 7B), datasets (CoQA, TriviaQA, MMLU, GSM8K, XSUM, WMT14FrEn, WMT19DeEn), and tools (e.g., RoBERTa, LM-Polygraph). However, does not explicitly mention the specific licences and terms of use for these assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paper thcode. com/ coom/datsets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
The paper introduces a new family of uncertainty quantification methods called CoCoA (Confidence and Consistency-
Approaches) and an approximated variant, CoCoA Light. However, it does not release any new dataset, model, or software tool alongside the paper, and does not provide accompanying documentation or artifacts.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
20

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research does not involve crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research does not involve crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper centers its contributions on the use Large Language Models (LLMMs) as the core subject of study. It introduces new methods for uncertainty quantification in LLMs, specifically evaluating base models such as LLLMA 3.1 8B, Mistral 7B, and Falcon 3 7B. Apart from that LLMs were used solely aid in editing the final manuscript and writing some of the visualization code. Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non nta-standard components.
Please refer our LLM policy what should or should not be described.
for
21

---

A Decoding Strategy and Sample Selection
Modern LLMs are capable producing output using wide range of decoding strategies. and is not readily apparent which one to use foundation for UQ experiments. On top of that, when sampling multiple outputs stochastically, one has decide which sample to select for comparison with the target sequence and UQ purposes.
To facilitate the choice of decoding and sample selection strategies for our experiments, conducted an evaluation of model performance with different approaches to both. Table shows average values of corresponding quality metrics for all combinations of models and datasets. We considered approaches for the selection of output that subsequently is used to calculate the quality of generation:
Greedy decoding produces single output by selecting top-1 candidate token at each genera- tion step, thus no further selection of sample is needed.
Random sample corresponds to the case where random output selected among the numher of samples produced repeatedly prompting the model with the question. In practice we use first generated sample, highlighting model performance when stochastic decoding is done only once.
Most Probable sample selects the output with highest model-assigned probability among several sampled outputs.
MBR sample selects the output with highest average consistency with respect to other sampled outputs.
We note that selecting random sample from the model outputs incurs a significant drop in the quality of results several datasets, most prominently GSM8K. Based on these observations, evaluate the efficacy of UQ three setups: greedy decoding, stochastic sampling with focus on the most probable sample and MBR decoding.
Regardless of the way the main model response was obtained, responses that were used quantify consistency in (8) were generated via repeated stochastic sampling. In all the experiments, where stochastic sampling involved, was performed with temperature T 1.0, top-k equal to 50, and top-p equal to 1.0.
Sample
Metric
Greedy
Random Sample
Most Probable Sample
MBR Sample
Falcon7b-Bass
CoQA GSM8K
Align Score
0.793
0. 0.706
0.783
0.793
Accuracy
0.776
0.313
0.205
MMLU
Accuracy
0.715
0.638
0.715
TriviaQA
Align Score
0.557
00.473
0.568
0.559
WMT14F:E
Comet
0.867
0.833
0.857
0.855
WMT19DeEn
Comet
0.846
0.807
0.826
0.828
XSUM
Align Score
0.842
0.734
0.782
0.801
Llama8h-Bas
CoQA
Align Score
0.756
0.671
0.743
0.766
GSM8k
Accuracy
0.548
0.234
0.261
0.346
MMLU
Accuracy
0.570
0.368
0.577
0.469
TriviaQA
Align Score
0.686
0.625
0.687
WMT14FFER
Comet
0.863
0.819
0.852
0.844
WMT19DeEn
Comet
0.870
0.816
0.854
0.845
XSUM
Align Score
0.848
0.608
0.825
0.703
Mistral7b-Bac
CoQA
Align Score
0.793
0.695
0.777
0.790
GSMBK
Accuracy
0.382
0. 169
0.190
0.259
MMLU
Accuracy
0.632
0.552
0.632
0.612
TriviaQA
Align Score
0.743
0.655
0.750
0.739
Comet
0.863
0.812
0.830
0.841
WMT19DeEn
Comet
0.864
0.805
0.836
XSUM
Align Score
0.803
0.578
0.775
0.665
Table Base quality metrics for models for different evaluated sequence choice.
22

---

B Results Summary for Most Probable Sample and MBR Sample strategies
Tables 3 and 4 report the PRR scores under the Most Probable Sample and MBR Sample generation
setups, as discussed in Section 5.2 We observe that CoCoA-famil -amily methods, and their Supervised
versions, consistently improve base uncertainty estimators.
Metric
Hlasa8b-Base
Mistral7h-Base
Falcon7b-Base
QA
NMT
SUM
QA
NMT
SUM
QA
NMT
SUM
MCSE
0.356
0,380
0.192
0.453
0.406
0.162
0.460
0.409
0.128
MCNSE
0,380
0.429
0,186
0.466
0.489
0.196
0.530
0,424
0.153
Semantic Entropy
0.396
0.411
0.194
0,482
0.438
0.164
0.479
0.440
0.134
DegMat
0.422
0.342
0.191
0.465
0.425
0.205
0,543
0.386
0.177
Big Vallapaa
0.388
0.274
0.190
0.426
0.366
0.197
0.498
0.336
0.174
SAR
0.478
0,506
0.159
0.542
0.576
0.175
0.590
0.488
0.193
P(True)
-0,071
0.066
0.058
0.051
0.371
0.207
0,281
0,245
0,022
Consistency
0.535
0.536
0.030
0.572
0.689
0.071
0.626
0.571
0.282
SP 0.395
0.376
0.464
0.444
0.252
0.330
0.343
0.381
0.099
CoCoAsp
0.484
0.607
0.484
0.526 f
0.721
0.366
0.529
0.631
0.210
PPL
0.532
0.563
0.458
0.587
0.686
0.365
0.627
0.589
0.275
CoCoAPPE
0.571
0.617
0.450
0.613 f
0.745
0.372
0.647
0.648
0.310 1
MTE
0.476
0.469
0.449
0.559
0.637
0.350
0.602
0.492
0.186
CoCoAMTE
0.547
0.579
0.451 f
0.600 f
0.720
0.373
0.641
0.614 f
0.289 f
Table 3: Results for Evaluated Sequence Most Probable Sample: Mean PRR across datasets for each task. The best performing method is in bold, and the second-best is underscored. Arrows indicate improvement in CoCoA over the base version.
Metric
Llamašb-Base
Mistral7b-Bass
Falcon7b-Base
QA
NMT
SUM
QA
NMT
SUM
QA
NMT
SUM
MCSE
0.402
0.297
0.158
0.474
0.330
0.237
0.437
0.346
0,076
MCNSE
0.392
0.389
0.124
0.441
0.435
0.192
0.446
0.440
0.078
Semantic Entropy
0.443
0.316
0.159
0.507
0.352
0.236
0.458
0.373
0.077
DegMat
0.478
0.290
0.047
0.471
0,322
0.124
0.499
0.371
-0.018
Fig Vall.alaana
0.434
0.224
0.037
0.437
0.279
0.115
0.460
0.326
-0.022
SAR
0.509
0,456
0,187
0.532
0.448
0,266
0.531
0.478
0.083
P(True)
-0.049
0.094
-0.014
-0.044
0.113
0.079
0.160
0.178
-0.124
Consistency
0.504
0.365
0.235
0.518
0.367
0.284
0.538
0.410
0.099
SP
0.415
0.404
0.266
0.447
0.433
0.312
0.405
0.376
0.170
CoCoAsp
0,527
0.496 f
0.293
0.553 f
0,515
0.348 f
0,500 1
0.526 1
0.154
PPL
0.408
0.421
0.164
0.432
0.452
0.273
0.445
0.438
0.100
CoCoApPL
0.532
0.504 f
0.215
0.554 f
0.514
0.328 f
0.552
0.537 1
0.109
MTE
0.467
0.461
0.148
0.515
0.496
0.219
0.490
0.521
0.069
CoCоAuTH
0.542
0.479 1
0.201
0.574 1
0.486
0.290 1
0.562
0.527
0.081
Table 4: Results for Evaluated Sequence MBR Decoding: Mean PRR across datasets for each task. The best performing method is in bold, and the second-best is underscored. Arrows indicate improvement CoCoA over the base version.
23

---

Ablation
C.1 Choice of Similarity Function
For sample consistency estimation, one could come up with a variety of similarity functions y()."))
We perform a comparison the effectiveness of CoCoA-family methods using several such functions. We consider the following functions:
AlignScore (Zha al. 2023 with -lig model;
Rougel.
[Lnn
2004
NLI [He et al. 2021
based micrrsoffrdbbrrrr--rrrgmmmmmmm model;
CrossEncoder
Liu ct al. 2019
based on croos-ennodee//ttstrrrrrrr----- model.
Tables and 7 report these results. There exists a considerable variation of relative effectiveness of proposed methods with various similarity function choices, depending task at hand. We opt report all results in other sections with CrossEncoder--asee similarity as by itself provides significant improvement over baselines, and for consistency and case of comparison reasons. However, when applying these methods to particular task, we encourage users to select appropriate underlying similarity function for best results.
24

---

Dataset
Method
XSUM
WMT19DeEn WMT14FrEn
CoQA
TriviaQA
MMLU
GSM8K
Mistral7b-Base
CoCoAsp
AlignScore
0.334
0.293
0.445
0.354
0.655
0.466
0.550
RougeL.
0.289
0.358
0.546
0.369
0.649
0,466
0.536
NLI
0,308
0.313
0,477
0.403
0.677
0.470
0.568
CrossEncoder
0.327
0.397
0,595
0.381
0.671
0.466
0.505
CoCoAprL
AlignScore
0,307
0.308
0,489
0.373
0.666
0.466
0.536
RougeL
0.226
0.369
0.531
0.352
0.653
0.466
0.466
NI.I
0.233
0.316
0.501
0.376
0.682
0.470
0.480
Crossencoder
0.281
0.371
0.565
0.365
0.674
0.466
0.465
CoCoAMTR
AlignScore
0.302
0.299
0.477
0.366
0.664
0.450
0.555
Rougel
0.212
0.377
0.528
0.345
0.652
0.449
0.497
NLI
0.219
0.313
0.488
0.362
0.681
0.453
0.490
Erooser
0.282
0.368
0.560
0.351
0.673
0.448
0.486
Llama8b
CoCoAsp
AlignScore
0.367
0.331
0.452
0.308
0.596
0.484
0.401
Rougel.
0.336
0.393
0.545
0.321
0.563
0.474
0.375
NLI
0.344
0,352
0.467
0.364
0.606
0.478
0.419
Crossencoder
0,375
0,454
0.583
0.350
0.598
0.480
0.367
CoCoAprL
AlignScore
0,422
0,346
0.450
0.337
0.596
0.453
0.446
RougeL
0.370
0,408
0.486
0.319
0.552
0.441
0.418
NLI
0.374
0.354
0.438
0.348
0.600
0.446
0.409
Crossencoder
0.380
0.444
0.514
0.339
0.593
0.447
0.429
CoCoAMTE
AlignScore
0.419
0.340
0.438
0.339
0.605
0.411
0.459
RougeL
0.362
0.417
0,481
0.319
0.560
0.390
0.440
NLI
0,366
0,342
0,428
0.340
0.612
0.396
0.420
Crodeer
0.374
0.441
0.511
0.337
0.601
0.394
0.444
CoCoAsr
AlignScore
0,278
0.306
0.475
0.361
0.677
0.528
0.470
Rougel.
0.205
0.394
0.499
0.378
0.678
0.527
0.417
NLI
0.236
0.361
0.511
0.407
0,684
0.532
0.532
Erossenc
0,253
0.436
0.577
0.396
0.685
0.529
0.428
CoCoAppi
AlignScore
0.252
0.340
0.523
0,410
0.678
0.528
0.521
RougeL
0.170
0.409
0.537
0.389
0.668
0.527
0.439
NLI
0.193
0.364
0.531
0.408
0.680
0.532
0.499
Croseen
0.226
0.437
0.579
0.405
0.677
0.529
0.474
CoCoA
AlignScore
0.253
0.337
0,519
0.403
0.683
0.515
0.554
RougeL
0.170
0.426
0,540
0.382
0.673
0.514
0.472
NLI
0.190
0.364
0.525
0.398
0.687
0.521
0.514
Enoosen
0.223
0.438
0.575
0.395
0.685
0.517
0.505
Table 5: Comparison of PRRs CoCoA-family methods with different choices the similarity function. Main model response obtained by greedy decoding.
25

---

Dataset
Method
XSUM
WMT14F:En
WMT19DeEn
CoQA
TriviaQA
MMLU
GSM8K
Mistrallb-Bass
CoCoAMsP
AlignScore
0.393
0,448
0,491
0.399
0.626
0.467
0,476
RougeL.
0.344
0.602
0.597
0.420
0.622
0,466
0.538
NLI
0.340
0.615
0.604
0.445
0.651
0,470
0,456
CrossEncoder
0,366
0.712
0.730
0.430
0.644
0.466
0,562
CoCoA PPL
AlignScore
0.474
0.619
0.657
0,408
0.638
0.467
0,910
RougeL
0,362
0.710
0.717
0.391
0.627
0.466
0.950
NI.I
0.370
0.677
0.684
0.414
0.657
0.470
0.941
CrossEncoder
0.372
0.735
0.755
0.402
0.648
0.466
0.937
CoCoA
AlignScore
0.492
0,547
0.590
0.383
0.633
0.449
0.914
RougeL
0.355
0,695
0.684
0.366
0.624
0.448
0,959
NLI
0.364
0.656
0.658
0.387
0.656
0.453
0.918
CrossEncoder
0.373
0.708
0.732
0.373
0.645
0.447
0.935
LlamaSb- -Bas
CoCoAMsP
AlignScore
0.520
0.332
0.491
0.354
0.587
0.457
0.401
RougeL.
0.471
0.470
0.588
0.362
0.551
0.446
0.499
NLI
0.466
0.442
0.577
0.386
0.597
0.446
0.470
CrossEncoder
0,484
0.529
0.685
0.384
0.587
0.452
0.513
CoCoApPL
AlignScore
0.546
0.406
0,561
0.376
0.577
0.429
0.875
RougeL
0,452
0.518
0.639
0.352
0.532
0.417
0.931
NL.I
0.458
0.466
0.597
0.365
0.583
0.418
0.912
CroseEneoder
0.450
0.544
0.689
0.364
0.573
0.422
0.925
CoCoA MFT
AlignScore
0.561
0.325
0.497
0.365
0.589
0.380
0.821
RougeL
0.448
0.496
0.598
0.336
0.539
0.361
0,921
NLI
0.449
0.446
0,565
0.344
0,598
0.359
0.881
Croseen
0.451
0.520
0.638
0.346
0.582
0.363
0.900
Falcon baBas
CoCoAMsr
AlignScore
0.181
0.378
0.473
0.410
0.654
0.528
0.239
Rougel.
0.122
0,531
0.581
0.420
0.655
0.528
0.426
NLI
0.120
0.496
0.607
0.437
0.658
0.533
0,458
Croosen
0.210
0.564
0.698
0.428
0.659
0.530
0.498
CoCoApPL
AlignScore
0.384
0.454
0.586
0.440
0.648
0.528
0.994
RougeL
0.280
0.565
0.668
0.410
0.637
0.528
1.000
NLI
0.283
0.515
0.671
0.424
0.647
0.533
0.998
Croseen
0.310
0.579
0.717
0.415
0.644
0.530
1.000
CoCoAsTE
AlignScore
0.292
0.386
0.498
0.435
0.648
0.515
0,972
RougeL
0.222
0.545
0.607
0,400
0.633
0.515
0.998
NLI
0.201
0.498
0.636
0.415
0.645
0.521
0,987
CrossEncoddr 0.289
0.551
0.678
0.402
0.646
0.517
0.998
Table 6:
Comparison of PRRs of CoCoA-family methods with different choices similarity function. Main model response obtained by selecting the most probable sample.
26

---

Dataset
Method
XSUM
WM
WMT19DeEn
CoQA
TriviaQA
MMLU
GSM8K
Mistral7b-Bast
CoCoAMsP
AlignScore
0.287
0.326
0,460
0.312
0.575
0,515
0.747
RougeL.
0.2.99
0,420
0.562
0.304
0.502
0,515
0.750
NLI
0.278
0.375
0.506
0.350
0.589
0.512
0.766
CrossEncoder
0.295
0.441
0.589
0.336
0.597
0.514
0.766
CoCoAPPL
AlignScore
0.283
0.333
0.482
0.330
0.5%6
0.515
0.738
RougeL
0.219
0.427
0,561
0.300
0.517
0,515
0.744
NI.I
0.219
0,390
0.508
0.334
0.602
0.512
0.723
CrossEncoder
0.258
0.451
0.577
0.322
0.612
0.514
0.768
CoCoA
AlignScore
0.269
0,314
0.447
0.357
0.663
0.497
0.746
RougeL
0.181
0.446
0.571
0.327
0.622
0.497
0.786
NLI
0.182
0.370
0.482
0.348
0.686
0.499
0.704
CrossEncoder
0.236
0.419
0.553
0.337
0.680
0.494
0.783
LlamaSb-
CoCoAMsP
AlignScore
0.221
0.309
0.462
0.261
0.467
0.622
0.717
RougeL.
0.195
0.401
0.542
0.250
0.361
0.631
0.721
NLI
0.222
0.311
0.463
0.311
0,480
0,580
0.769
CrossEncoder
0.225
0.403
0.590
0.304
0.487
0.586
0.731
CoCoApPL
AlignScore
0.225
0.325
0.484
0.304
0.476
0.603
0.747
RougeL
0.169
0.416
0.537
0.274
0.370
0,612
0.758
NI.I
0.180
0.334
0.460
0.319
0.488
0.558
0.761
CroseEneoder
0.198
0.428
0.580
0.320
0.496
0.562
0.751
CoCoA MFTI
AlignScore
0.209
0.293
0.460
0.320
0.563
0.556
0.749
RougeL
0.148
0.436
0.559
0.314
0,497
0.587
0.785
NLI
0.156
0.298
0.445
0.340
0.592
0.460
0.740
CrossEncoder
0.189
0.398
0.561
0.345
0.587
0.472
0.765
Falcon Tb-Base
CoCoAMsP
AlignScore
0.279
0.307
0.473
0.312
0.594
0.574
0.460
Rougel.
0.202
0,424
0.487
0.307
0.566
0.574
0.447
NLI
0.231
0.362
0.508
0.355
0.618
0.577
0,531
Croosen
0.248
0.461
0.590
0.336
0.617
0.575
0.472
CoCoAppl
AlignScore
0.278
0.320
0.534
0.350
0.607
0.574
0,617
RougeL
0.190
0.428
0.556
0.333
0.571
0.574
0,632
NLI
0.216
0.376
0.549
0.368
0.623
0,578
0.661
Croseen
0.242
0.462
0.613
0.345
0.624
0.575
0.662
CoCoANTE
AlignScore
0.277
0.302
0.522
0,382
0.657
0.556
0.598
RougeL
0.175
0.472
0,573
0.364
0.642
0.556
0.617
NLI
0.196
0.359
0.530
0.400
0.671
0.562
0,639
CrossEncoddr 0.237
0.454
0.600
0.373
0.666
0.557
0.650
Table 7: Comparison of PRRs of CoCoA-family methods with different choices similarity function. Main model response obtained by MBR decoding.
27

---

C.2 Different Ways of Combining Confidence and Consistency
We justify the particular form of equation (II) by considering alternative ways combine sample- focused confidence with consistency estimation. Results are presented in Tables g particular, we investigate the performance of the additive approach (AdditiveCoCoA);
9 and I0 In
Udddiiveooooa (y+ x) - w(y, xx + Conns (y, x),
(16)
8 with the and the same multiplicative combination, replacing sample-focused dissimilarity from average of the full pairwise dissimilarity matrix (2
UpulSampecCocoa (y= x) u(y= x) (x))
(17)
In addition, we present results where we compute u(y. x) using probability- -based scores instead of log-likelihood, see equation (6). We refer this variant as UpotCoCia (y= x). While the results
generally similar, the oog-likelihood based formulation offers greater numerical stability. Therefore, we adopt it the default in the main paper.
evident that on average the multiplicative form proposed in equation (10) with both confidence and consistency terms focused on single sample is the better performing variant.
28

---

Dataset
Method
XSUM
WMT14FTED
WMT19DeLn CoQA
TriviaQA
MMLU
GSM8K
Mistral7b-Ban
AdditveCoCoAss
0.290
0.319
0.459
0.351
0.654
0,471
0.472
FullSampleCoCoAgr
0.319
0.385
0.590
0.357
0.668
0.467
0.505
ProbCoCoAsp
0.059
0,302
0,520
0.390
0.671
0.461
0.435
CoCoNsp
0.330
0.396
0.598
0.383
0.670
0,466
0.517
AdditveCoCoAppz
0.262
0.392
0.564
0.369
0.671
0.464
0.494
PullSumpeeCCoAApp
0.277
0.373
0,551
0.334
0.672
0,467
0,435
ProbCoCoAPPI.
0.297
0.369
0,566
0.373
0.674
0,464
0,475
CoCoApPL
0.286
0.375
0.568
0.369
0.674
0.466
0.467
AdditivCoCooArre
-0.279
-0,058
-0.072
0.098
0.312
0.079
0.187
FullSampleCoCoA
0.274
0.368
0.543
0.309
0.668
0.442
0.456
CoCoAMTE
0.288
0.374
0.564
0.355
0.673
0.447
0.491
Ilama8h Base
AdditiveCoCoAgp
0.330
0.345
0.462
0.301
0.566
0.502
0.326
FullSampleCoCoAgr
0.358
0.434
0.564
0.333
0.589
0.488
0.354
ProbCoCnAsp
0.031
0.405
0.471
0.612
0.461
0.368
CoCoAsp
0.456
0.582
0.349
0.485
0.372
AdditivCoCoAAprL
0.368
0,431
0.504
0.336
0.595
0,455
0.437
FullSampl:CoCoAppp
0.389
0.420
0.487
0.314
0.580
0.450
0.399
ProbCoCoAPPL
0.381
0.445
0,513
0.345
0.599
0.446
0.438
CoCoApPI.
0.387
0.448
0,514
0.338
0.593
0.452
0.433
Adde
-0.331
-0.042
-0.122
0.089
0.321
-0.122
0.117
FullSampl:CoCoA MTB
0.383
0.410
0.481
0.308
0.588
0.363
0.414
CoCoAMTR
0.380
0.446
0.511
0.337
0.601
0,402
0.447
Falcon Malconnb-Bain Base
AddiiveCoCoAgp
0.203
0.318
0.409
0.350
0.674
0.533
0.379
FullSumpleCoCoAgp
0.423
0.388
0.678
0.533
0.404
ProbCoCoAsP
0.367
0.515
0.416
0.680
0,526
0.426
CoCoAsp
0.257
0.433
0.578
0.396
0.684
0.529
0.436
AdditiveCoCoApPz
0.222
0.433
0,580
0.413
0.677
0.525
0.489
FullSampl:CoCoAPpp..
0.204
0.425
0.565
0.393
0.669
0.533
0.437
ProbCoCoAPPL
0.235
0.433
0.576
0.410
0.680
0.528
CoCoAPPL
0.229
0.436
0.406
0.529
AddiiveCoCoAurre
0.001
-0.103
10.606
0.114
0.041
0.138
0.221
FullSampleCoCoA.pr MrE
0.201
0.425
0.557
0.377
0.675
0.519
0,470
CoCoAMrE
0.228
0.439
0.577
0.395
0.685
0.517
0.510
Table 8: Comparison of PRRs of CoCoA-family methods with alternative formulations. Main model
response obtained via greedy decoding.
29

---

Dataset
Method
XSUM
WMT14FTED
WMT19DeLn
CoQA
TriviaQA
MMLU
GSM8K
Mistral7b-Bas
AdditveCoCoAss
0.333
0.239
0.310
0,406
0.631
0,472
0.311
FullSampleCoCoAgr
0.354
0.543
0.565
0.412
0.643
0.468
0.428
ProbCoCoAsp
0.076
0.684
0.721
0.428
0.643
0.464
0.846
CoCoNsp
0,366
0.712
0.730
0.430
0.644
0,466
0.562
AdditveCoCoAppz
0.368
0.737
0.751
0.406
0.644
0,465
0.939
PullSumpeeCCoAApe
0,383
0.714
0.723
0.379
0,649
0,468
0,933
ProbCoCoAPPI.
0.369
0.738
0.756
0,401
0.649
0,467
0,935
CoCoApPL
0.372
0.735
0.755
0.402
0.648
0.466
0,937
AdditivCoCoAArre
0.368
0.723
0.702
0.332
0.643
0.452
0,942
FullSampleCoCoA
0.380
0.661
0.653
0.331
0.643
0.442
0.929
CoCoAMTE
0.373
0.708
0.732
0.373
0.645
0.447
0.935
Ilama8h Base
AdditiveCoCoAgp
0.466
0,349
0.425
0.333
0.555
0.473
0.285
FullSampleCoCoAgp
0.476
0.462
0.619
0.363
0,574
0.464
0.379
ProbCoCnAsp
0.491
0.617
0.398
0.598
0.433
0.795
CoCoAsp
0,529
0.685
0.384
0.587
0.452
AdditivCoCoAApPL
0.454
0.536
0.673
0.358
0.575
0,425
0,923
FullSampl:CoCoAppp
0.459
0.525
0.649
0.343
0.556
0.430
0,914
ProbCoCoAPPL
0.438
0.547
0.689
0.364
0.574
0.419
0.923
CoCoApPI.
0.450
0.544
0.689
0.364
0.573
0.422
0.925
Adde
0.457
0.496
0.579
0.304
0.561
0.361
0.901
FullSampleCoCoA.yry MTB
0.455
0.464
0.577
0,313
0.563
0.341
0.878
CoCoAMTR
0.451
0.520
0.638
0.346
0.582
0.363
0,900
Palcon7b-Ja Base
AddiiveCoCoAgp
0.100
0.397
0.394
0.393
0.649
0.534
-0.156
FullSumpleCoCoAgp
0,531
0.607
0.416
0.654
0,533
ProbCoCoAsP
0.282
0.522
0.670
0.434
0.658
0,529
CoCoAsp
0.210
0.564
0.698
0.428
0.659
0.530
0.498
AdditiveCoCoApPz
0.297
0.582
0.706
0.417
0.643
0.526
1.000
FullSampl:CoCoAPpp..
0.297
0.560
0.670
0.405
0.641
0.533
1.000
ProbCoCoAPPL
0.587
0.414
0.648
0.531
CoCoAPPL
0.579
0.717
0.415
0.644
0,530
AddiiveCoCoAurre
0.253
0.554
0.634
0.383
0.630
0.523
0.997
FullSampleCoCoA..ag MrE
0.237
0,502
0.554
0.383
0.636
0.519
0.989
CoCoAMrE
0.289
0.551
0.678
0.402
0.646
0.517
0.998
Table 9: Comparison of PRRs of CoCoA-family methods with alternative formulations. Main model response obtained by selecting the most probable sample.
30

---

Dataset
Method
XSUM
WMT14FTEn
WMT19DeEn
CoQA
TriviaQA
MMLU
GSM8K
Mistral7b-Bas
AdditveCoCoAse
0.382
0.231
0,388
0.512
FullSampleCoCoAgp
0.287
0.447
0.591
0.318
0.570
0.516
0.748
ProbCoCoAsp
0.258
0.374
0.473
0.149
0.099
-0.265
0.710
CoCoAMTE
0.295
0.441
0.589
0.336
0.597
0.514
0.766
AddiiveCoCoApPL
0.234
0.430
0.560
0.255
0.517
0.507
PullSampleCCCoAppL
0.251
0.458
0.305
0.516
ProbCoCoAPPL.
0.094
-0.123
-0.143
-0,202
-0,044
-0.423
0.027
CoCoApPE
0.258
0.451
0.577
0.322
0.612
0.514
0.768
AdditiveCoCoAurre
0.201
0.468
0.585
0.296
0.669
0.501
0.760
FullSampleCoCoAurre
0.232
0.426
0.544
0.318
0.673
0.496
0.772
CoCoAMTE
0.419
0.337
0.494
0.783
Llama8h- BaaaaschBBac
Addit
0.197
0.345
0.465
0.177
0.287
0.571
0.676
FullSampleCoCoAgr
0.214
0.409
0,589
0.289
0.449
0.589
0.724
ProbCoCoAsP
0.196
0.337
0.449
0.102
0.053
0.434
0.673
CoCoAMTE
0.225
0,403
0,590
0.304
0.487
0.586
0.731
AddiiveCoCoAppr.
0.182
0.406
0.500
0.229
0.401
0.540
0.715
FullSampleCoCoAppe
0.186
0.433
0.563
0.304
0.461
0.565
0.746
ProbCoCoAPPL
0.101
-0.097
-0.152
-0.175
-0,367
0.099
0.053
CoCoApPE.
0.198
0.428
0.580
0.320
0.496
0.562
0.751
AdditveCoCoAurr
0.437
0.302
0,560
0.477
FullSamplcCoCoA MTE
0.174
0.400
0.543
0.334
0.568
0.475
0.762
CoCoAMTE
0.189
0.398
0.561
0,345
0,587
0.472
0.765
Falcon 7h-Base
AdditveCoCoAsp
0.188
0.356
0.401
0.228
0.488
0.578
0.368
FullSampeeCCooApp
0.237
0.468
0,586
0.325
0.593
0.576
0.453
ProbCoCoAsp
0.184
0.339
0.378
0.144
0.350
-0.345
0.364
CoCoAMTE
0.248
0.461
0.590
0.336
0.617
0.575
0,472
AdditveCoCoAPpí
0.206
0,404
0.565
0.271
0.537
0.572
0.625
FullSamp:coCoAApR
0.231
0.464
0.606
0.337
0.598
0.577
0.650
ProbCoCoAPPL
0.014
-0.167
-0.265
-0.187
-0.237
-0.416
-0.206
CoCoApPL.
0.242
0,462
0.613
0.345
0.624
0.575
0.662
AdditieCoCoAurre
0.191
0.488
0.612
0.353
0.644
0.564
0.617
FullSampleCoCoAArrr MIE
0.219
0.452
0.591
0.368
0.654
0.556
0.629
CoCoAMTR
0.237
0.454
0.600
0.373
0.666
0.557
0.650
Table 10: Comparison of PRRs of CoCoA-family methods with alternative formulations. Main model
response obtained by MBR decoding.
31

---

D Detailed Description of Uncertainty Quantification Methods
In this section, we provide detailed description of the uncertainty quantification methods used in this study.
D.1 Information-Baut Methods
Information based methods commonly used to estimate uncertainty by analyzing the probability distributions of tokens within a given output. These methods examine different levels of model generation, such the model's 's confidence in producing specific sequence, its ability to predict individual tokens at each generation step, and the variability in the token-level predictions across the sequence.
Sequence Probability (SP) is one of the simplest and most direct methods for estimating uncertainty. It measures the probability of the most likely output sequence given specific input. Thus, uncertainty quantified by calculating the probability of the sequence with the highest likelihood, under the assumption that the model is most confident in this output. In its simplest form it is given by (6). An equivalent (in terms of ordering of predictions for different inputs) but more numerically stable formulation is logarithmic. It is defined as:
Usp(y x) = - - ply x).
(18)
Perplexity (PPL) is another widely used metric for estimating uncertainty in language mod- els (Fomicheva al., 2020). It measures the model's confidence by evaluating the average likelihood of generating the sequence tokens:
UPpr(y x) - log p(y x).
(19)
Mean Token Entropy takes a broader view of uncertainty by considering the token-level predictions across the entire sequence [Fomicheva et al. [2020]. Instead of evaluating the model's confidence in a single output individual token predictions, Mean Token Entropy calculates the average entropy of the token probability distributions for each token in the sequence:
L
Uur (y x)
Σ
H(m Yetx x),
(20)
where H(y: y<t. x) is an entropy of the token distribution p(u y<t.)).
The TokenSAR method, introduced in (Duan et al. 2024), generalizes length-normane normalized log probabil- ity by computing weighted average of the negative log probabilities of generated tokens, where weights are based on token relevance to the overall text. Using a similarity function s(';)) and token relevance function Rr Ryk,, y. x) s(x Uy, x Uy \ Uk), the uncertainty estimate is calculated as:
UTokenSAR(y xx)= - Σ
Rr(m y, logp(ın l),,)),
(21)
1=1
where
Ry El_ RT(,,.
RT
(22)
This measure central for computing SAR uncertainty measure.
D.2
Consistency-Based Methods
Consistency-baat -based methods assess the uncertainty of a language model by evaluating the semantic consistency of its predictions across multiple outputs for the same prompt. The core idea is that semantically similar outputs indicate higher confidence, while diverse or conflicting outputs suggest greater uncertainty. Since language models can express the same meaning in different surface forms, these methods construct a semantic similarity matrix 5 = (siz), where each entry represents the degree of similarity between pairs of responses. By clustering responses into groups with equivalent meanings, these methods provide semantic measure of the model's 's consistency.
32

---

The work (Lin et al. 2024] offers two similarity measures to evaluate the similarity of sequences. The first is the Jaccard similarity, which treats sequences as sets of words and calculates the proportion of shared words to the total number of unique words in both sequences: s(y, y') ly n y'll/y U'l-
Natural Language Inference (NLJ) provides another method for computing similarity between sequences. We use the DeBERTa-large NLI model (He et al. [021], following (Kuhn et al. [003)] For each pair of sequences, an NLI model predicts two probabilities: Pentail (y, y'), indicating entailment, and Poontra (Y. y'), indicating contradiction. Similarity is then defined as either Sentail (y,
Pentail y') or Scontra (y, y')
Peontra (y.y)..
Among the simplest consistency- based approaches are the Number of Semantic Sets and the Sum of Eigenvalues of the Graph Laplacian (Lin et al., 2024]. Number of Semantic Sets estimates how many distinct "meanings" the model produces by clustering its outputs with an NLI model. The number of semantic sets is initially equal to the total number of generated answers, M. Two sentences grouped into the same cluster if the following conditions are satisfied: Pemail (y()))())) >
Poentra (y("), y(i)) and Pentail (y(+), . y()) Peootra (y(), y()) This computation is performed for all pairs of answers, and the final number of distinct clusters is denoted by UNumSemSees-
Sum of Eigenvalues of the Graph Laplacian examines global diversity: it constructs similarity matrix among the sampled outputs and computes a continuous uncertainty score from the eigenvalues of the Laplacian of that similarity graph. The work (Lin al. 2024] proposes computing an averaged similarity matrix S.j (s(y yy())=+ + ss .y ()2 The Laplacian for the matrix S is defined
as L = I - D- SD $, where D is a (diagonal) degree matrix with elements Di: Σm 8ij.
Consequently, the following formula is derived:
M UEgy(xx = Σ
max(0, 1- 1-(()))
(23)
1=1
Both Number of Semantic Sets and Sum of Eigenvalues of the Graph Laplacian effectively capture overall variation in generated text but cannot produce an individual uncertainty score for each output. To address this, the work (Lin et al. 22244) proposes to use the diagonal Degree Matrix D(x) which represents the total similarity of each answer with all others. The corrected trace of D(x) provides an average pairwise distance between answers, and uncertainty computed as:
DegMMat = 1 - trace (Dx))
(24)
D.3 Information-Based Methods with Repeated Sampling
The natural idea is to somehow benefit from having multiple samples from the model while using important information contained in the output probabilities estimated by an LLM. Below, we examine several approaches that have sought achieve this.
Averaging uncertainties.
We can compute the entropy on the sequence level E[- log x)] where the expectation is taken over the sequences y randomly generated from the distribution p(y x). Unfortunately, while for token level, we have an exact way of computing the entropy, for the sequence level, we need to adhere to some approximations. In practice, we can use Monte-Caria integration, i.c. generate several sequences y(), 1,..., M via random sampling and compute Monte Carlo Sequence Entropy (Kuhn al. 2023
UHS (x)
log (() (1) xx).
(25)
M
We can replace ((y x) with its length- normalied version ((y x) leading to a more reliable uncertainty measure in some cases.
While simple averaging represents a natural way to aggregate uncertainties, it has certain issues related to the nature of LLMs.. First of all, in the vast majority of applications, an LLM-based system should produce a single output y. for an input query. When we consider UNs (x) or other similar measure, we essentially perform averaging of uncertainties of different sequences, thus somewhat assessing the
33

---

uncertainty related to the entire generative distribution p x) for the imput x, but not for a particular generated sequence y.. This averaged uncertainty might not adequate for this particular sequence and, remarkably, often performs worse than the uncertainty u, = (y. x), which is related solely to the output y.. Moreover, although intuitive, this naïve aggregation method assumes that all outputs contribute equally to the final uncertainty estimate, regardless of their semantic relationships. This can lead to inconsistencies when semantically equivalent outputs have varying uncertainty scores or when outputs with low similarity are treated as equally important.
Semantically weighted averaging. Semantic Entropy (Kuhn et al. 2023 addresses the issue of generated sequences with similar meanings but differing probabilities according to the model, which can heavily influence the resulting entropy value (25). The method clusters generated sequences y(*), i i= ,....: M into semantically homogeneous groups Ck: k = ,...., K (where K s M) using a bi-directional entailment algorithm. Probabilities of sequences are averaged within each cluster. The entropy estimate is then defined as:
K Cal M
(xE = =- - log (x),,
(26)
where (x( Eyec, ply x) represents the aggregated probability for cluster Ck.
SentenceSAR DDann et al., 2024 enhances the probability of sentences that are more relevant. It uses sentence relevance measure s (k))) to evaluate the relevance of yO) with respect to (()), SentenceSAR is calculated as:
1 M
Σ log (p((()))) x) - Rs(x
U (x)
(27)
where t is temperature parameter used to control the scale of shifting to relevance, and
Rs(y x)
Σ sy(0), y(t) p()())()( xx).
(28)
The combination of SentenceSAR and TokenSAR results unified method called SAR [Duan et 2024]. In this approach, the generative prohability ply x) in the SentenceSAR formula is replaced with the token-shifted probability y' (y x) = exp T1--TToenSRRR, xx) creating comprehensive measure that integrates both sentence- and token-level adjustments.
The aggregation approaches like Semantic Entropy [Kuhn et al. 2023 or SAR DDann et al. 2024
can be unified into semantically-aware Generalized Monte Carlo uncertainty estimate, defined
Σ s (y y0)) (x) 1x))
UGMCU(x) (x)
(29)
Here, the inner summation aggregates sequence probabilities p x) weighted by their semantic similarity to the i-th output, and the outer summation averages these contributions across all samples. The function h(.) provides additional layer of flexibility, transforming the reweighted uncertainty scores, making the method generalized framework for uncertainty quantification.
Unfortunately, methods that fall under GMCU, while offering benefits, also inherit the aforementioned issues from both categories of methods. particular, the outer summation (29). similarly to the
case of simple Monte Carlo averaging, often fails to outperform the uncertainty (x U x)
of a single generated sequence y..
D.4 Verbalized Uncertainty
Verbalized uncertainty methods refer to approaches that prompt model to explicitly express its confidence. In our experiment, we utilize the P(True) method, implemented following the description by (Kadavath et al.) 2022). Specifically, the model is presented with the original question and its answer, and then prompted indicate whether the answer is True or False. We use the negative log-probability of the token "Truc" as the uncertainty score.
34

---

E Detailed Experimental Results
Tables and presented PRR scores averaged over datasets corresponding to each task. Here present expanded results for each dataset, see Tables 11 and 13
35

---

Dataset
Method
XSUM
WMT14FFEn
WMT19DeEn
CoQA
TriviaQA
MMLU
GSMSK
Mistral7b- Base
MCSE
0.007
0.257
0.350
0.247
0.496
0.337
0.475
MCNSE
0.009
0.342
0.478
0.238
0.540
0.356
0.401
Semantic Entropy
0.008
0.271
0.382
0.271
0.562
0.387
0.472
DegMat
0.137
0.229
0.382
0.336
0.646
0.410
0.299
Lappllannn
0.132
0.207
0.328
0.301
0.624
0.398
0.241
SAR
0.094
0.353
0.517
0.313
0.644
0,419
0.471
P(True)
0.179
0.118
0.033
-0.061
-0.128
0.068
0.005
Consistency Light
-0.006
0.317
0.457
0.306
0.597
0,432
0.441
Consistency
0.051
0.285
0,500
0.379
0.647
0.435
SP
0.287
0.315
0.451
0.326
0.628
0.474
0.471
CoCoAsP CoCoAsP Light
0.330
0.396
0.598
0.383
0.670
0,466
0.517
0.309
0.380
0.580
0.351
0.674
0.480
0.506
PPL
0.204
0.365
0.489
0.281
0.632
0.474
0,311
CoCoAppL
0.286
0.375
0.568
0.369
0.674
0.466
0.467
CoCoApPE Light
0.260
0.404
0.557
0.326
0.684
0.480
0.428
MTE
0.182
0.392
0.484
0.243
0.619
0.456
0.350
CoCoANTE
0.288
0.374
0,564
0.355
0.673
0.447
0.491
CoCoAMTE Light
0.254
0.415
0.550
0,303
0.672
0.465
0.458
Llama8b Base
MCSE
0.033
0.293
0.354
0.237
0.482
0.171
0.351
MCNSE
0.022
0.370
0.415
0.219
0.501
0.170
0.344
Semantic Entropy
0.033
0.297
0.389
0.272
0.549
0.229
0.375
Mat
0.081
0,250
0.355
0.353
0.622
0.342
0.309
Eig Vaplaapaaan
0.079
0.198
0.278
0.332
0,604
0.292
0.273
SAR
0.077
0.427
0.483
0.311
0,595
0.352
0.398
P(True)
0.058
0.047
0.037
-0.037
-0.066
-0.180
0,026
Consistency Light
-0.022
0.441
0.353
0.552
0.279
Consistency
0.024
0.389
0.453
0.375
0.614
0.391
0.368
SP
0.328
0.342
0.456
0.277
0.526
0.508
0.324
CoCoASP CoCoAsp Light
0.378
0.456
0.582
0.349
0.597
0.485
0,372
0.358
0.448
0.556
0.340
0.598
0.504
0.353
PPL.
0.369
0.351
0.422
0,253
0.507
0.461
0.303
CoCoApPL CoCoAppL Light
0.387
0.448
0.514
0.338
0.593
0,452
0.433
0.382
0.483
0.491
0.319
0.597
0.467
0.398
MTE
0.357
0.357
0.408
0,239
0.497
0.349
0.326
CoCoAMTE
0.380
0.446
0.511
0.337
0.601
0.401
0.447
CoCoAuTE Light
0.372
0.501
0.487
0.316
0.599
0.387
0.412
Falcon 7b-Base
MCSE
0.159
0.297
0.337
0.258
0.549
0.420
0.427
MCNSE
0.108
0.371
0.474
0.293
0.586
0.442
0.299
Semantic Entropy
0.164
0.307
0.389
0.294
0.581
0.463
0.418
DegMat
0.201
0.274
0.431
0.407
0.651
0,480
0.395
Vallapaaaa
0.201
0.229
0.394
0.381
0.645
0,454
0.358
SAR
0.144
0.398
0.517
0.381
0.649
0.508
0.387
P(True)
-0.159
0.175
0.135
0.036
0.275
0.027
0.133
Consistency Light
0.232
0.483
0.468
0.327
0.657
0.362
0.363
Consistency
0.226
0.337
0.496
0.408
0.656
0.485
0.426
SP
0.201
0.312
0.400
0.321
0.662
0,539
0.377
CoCoASP
0.257
0.433
0.578
0.396
0.684
0.529
0.436
CoCoAsP Light
0.242
0.463
0.566
0.349
0.691
0.522
0.405
PPL.
0.155
0.375
0.525
0.316
0.644
0,539
0.326
CoCoApPL
0.229
0.436
0.580
0.406
0.677
0.529
0.478
CoCoAppL Light
0.234
0.497
0.558
0.364
0.694
0.522
0.468
MTE
0.152
0,409
0.537
0.291
0.633
0.533
0.367
CoCoAMTE
0.228
0.439
0.577
0.395
0.685
0.517
0,510
CoCOAMTE Light
0.234
0.518
0.549
0.345
0.693
0.491
0.500
Table 11: Detailed experimental results. Main model response obtained by greedy decoding.
36

---

Dataset
Method
XSUM
WMT14FFEn
WMT19DeEn
CoQA
TriviaQA
MMLU
GSM8K
Mistral7b Base
MCSE
0.162
0.406
0.407
0.289
0.492
0.339
0.693
MCNSE
0.196
0.471
0.507
0.277
0.529
0.358
0.700
Semantic Entropy
0.164
0.434
0.442
0.312
0.554
0,389
0.675
DegMat
0,205
0.439
0,410
0.376
0.618
0.410
0.454
Vallaal.apllaian apllacian
0.197
0.388
0.344
0.342
0.600
0.399
0.361
SAR
0.175
0.563
0.590
0.347
0.620
0.421
0.780
P(True)
0,207
0.472
0.269
-0.058
-0.084
0.068
0.278
Consistency
0,071
0.670
0.708
0.614
0.423
0.846
SP
0.330
0.212
0.291
0.388
0.607
0.476
0.307
CoCOASP
0.366
0.712
0.730
0.430
0.644
0.466
0.562
PPL.
0.365
0.695
0.676
0.327
0.615
0.476
0.931
CoCoApPL
0.372
0.735
0.755
0.402
0.648
0.466
0.937
MTE
0.350
0.668
0.606
0.254
0.594
0.457
0.932
CoCoAmTE
0.373
0.708
0.732
0.373
0.645
0.447
0.935
IJamm8h-
Baasa Base
MCSE
0.192
0.366
0.395
0.258
0.465
0.158
0.545
MCNSE
0.186
0.377
0.480
0.239
0.484
0.165
0.631
Semantic Entropy
0.194
0.371
0,451
0.286
0.528
0.213
0.557
DegMat
0.191
0.274
0.409
0.366
0.606
0.320
0.396
Eig ValLallcaaa
0.190
0.216
0.333
0.339
0.587
0.274
0.351
SAR
0.159
0.441
0.571
0.327
0.578
0,340
0.667
P(True)
0.058
0.075
0.056
-0,011
-0.071
-0.120
-0.084
Consistency
0.030
0.473
0.598
0.394
0.600
0.353
0.793
SP
0.464
0.339
0.413
0.304
0.514
0.483
0.280
CoCoAsp
0.484
0.529
0.685
0.384
0.587
0.452
0.513
PPL
0.458
0,504
0.622
0.293
0.483
0.441
0.911
CoCoApPE.
0.450
0.544
0.689
0.364
0.573
0.422
0.924
MTE
0.449
0.437
0.501
0.238
0.458
0.326
0.883
CoCoANTE
0.451
0.520
0.638
0.345
0.582
0.363
0.899
Falcon7b Base
MCSE
0.128
0.399
0.419
0.285
0.535
0.421
0.598
MCNSE
0.153
0.395
0.452
0.318
0.588
0.443
0.771
Semantic Entropy
0.134
0.420
0.460
0.319
0.566
0.463
0.567
DegMat
0.177
0.350
0.422
0.422
0.637
0.480
0.633
Valpaaaanaa
0.174
0.289
0.382
0.393
0.622
0.454
0.522
SAR
0.193
0.455
0.521
0.385
0.642
0.509
0.826
P(True)
0.022
0.245
0.245
0,038
0.244
0.028
0.815
Consistency
0.282
0.491
0.651
0.416
0.627
0.484
0.979
SP
0.099
0.385
0.378
0.369
0.638
0.540
-0.175
CoCoAsp
0.210
0.564
0.698
0.428
0.659
0.530
0.498
0,275
0.541
0.353
1.000
CoCoAPPE
0.310
0.579
0.717
0.415
0.644
0,530
1.000
MTB
0.186
0.475
0.510
0.317
0.573
0.534
0.984
CoCoAuTE
0.289
0.551
0.678
0.402
0.646
0.517
0.998
Table 12: Detailed experimental results. Main model response obtained by selecting most probable candidate among stochastically sampled responses.
37

---

Dataset
Method
XSUM
WMT19DeEn WMT14FFn
CoQA
TriviaQA
MMLU
GSMSK
Mistral7b-Bas
MCSE
0.123
0.297
0.363
0.245
0,515
0.385
0.751
MCNSE
0.105
0,385
0,485
0.256
0.552
0.406
0.548
Semantic Entropy
0.127
0.310
0.395
0.272
0.580
0.439
0.735
DegMat
0.244
0.280
0.365
0.358
0.673
0.461
0.392
Vall apllacan
0.244
0.238
0.319
0.334
0.654
0.443
0.318
SAR
0.204
0.392
0.503
0.334
0.658
0.478
0.656
P(True)
0.079
0.142
0.084
-0.056
-0.084
0.047
-0,084
Consistency
0.211
0.297
0.437
0.355
0.659
0.458
0.603
SP
0.259
0,381
0,486
0,216
0.346
0,514
0.711
CoCoANTE
0.295
0.441
0.589
0.336
0.597
0,514
0.766
PPL
0.205
0.391
0,513
0.186
0.358
0,514
0.672
CoCoAPPL
0,258
0.451
0.577
0.322
0.612
0,514
0.768
MTE
0.156
0.443
0.550
0.234
0.598
0.508
0.717
CoCOANTE
0.236
0.419
0.553
0.337
0.680
0,494
0.783
Ilama8b- Base
MCSE
0.089
0,240
0.354
0.220
0.444
0.297
0.647
MCNSE
0.055
0,334
0,443
0,247
0.470
0.306
0.545
Semantic Entropy
0.089
0,250
0.382
0.245
0.514
0.361
0.650
DegMat
0.217
0,218
0.361
0.368
0.611
0.391
0.539
Eig Vilallaalla
0.217
0.167
0.280
0.347
0.592
0.305
0.492
SAR
0.125
0.393
0.520
0.345
0.578
0.472
0.642
P(Truc)
-0.014
0.098
0.091
-0,014
-0.054
-0.210
0.083
Consistency
0.146
0.290
0.440
0.387
0.603
0.428
0.599
SP
0.197
0.344
0.463
0.161
0,247
0.577
0.675
CoCOANTE
0.225
0.403
0.590
0.304
0.487
0.586
0.731
PPL.
0.168
0.382
0.460
0.161
0,253
0.549
0.668
PPL
0.198
0.428
0.580
0.320
0.496
0.562
0.751
MTE
0.130
0.412
0.509
0.245
0.466
0.459
0.699
CoCOANTE
0.189
0.398
0.561
0.345
0.587
0.472
0.765
Falcon7b Base
MCSE
0.188
0.331
0.362
0.274
0.560
0.455
0.460
MCNSE
0.126
0.387
0.494
0,306
0.605
0.479
0.396
Semantic Entropy
0.192
0.332
0.414
0.304
0,594
0.497
0.438
DegMat
0,246
0.271
0.470
0.392
0.672
0.516
0.415
Eig Lalaalacann
0,238
0,220
0.432
0.365
0.660
0.476
0.339
SAR
0.184
0.412
0.543
0.370
0.664
0,553
0.536
P(True)
-0.124
0.194
0.161
0.016
0.279
0.061
0,284
Consistency
0.235
0.323
0.498
0.376
0.665
0.523
0.588
SP
0.187
0.355
0.398
0.213
0.458
0.367
CoCoAsP
0,248
0.461
0,590
0.336
0.617
0.575
0.472
PPL
0.177
0.371
0.504
0.201
0.439
0.583
0.556
CoCoAPPL
0.242
0.462
0.613
0.345
0.624
0.575
0.662
MTEE
0.155
0.462
0.580
0.29)
0.588 0.666
0.565
().57
COCOANTE
0.237
0.454
0.600
0.373
0.557
0.650
Table 13: Detailed experimental results. Main model response obtained by MBR decoding.
38

---

Alternative Performance Metrics for PRR
The choice of PRR as a UQ quality metric of choice is dictated by its ability handle both continuous performance metrics, like Comet and AlignScore without the need for selecting arbitrary thresholds, as well as relative robustness to class imbalance in case of binary performance metric. However, PRR scores are calculated for a particular choice of underlying performance metric. For truly comprehensive evaluation, we perform the same evaluation as our main experimental run, but with alternative choice of performance metrics.
Tables 14 and 13 report PRRs for these metrics all models for NMT and QA tasks. MetricX [Juraska ct al. 2024] was used for NMT and aPT-as-a-judge (gpt-40-2024-08 datasets, The following the prompt used facilitate GPT QA scoring:
was used to score QA
You are a text evaluator. The model was asked the following question:
[question]
The 'Generated' text is a model's 's response. The 'Target' is the correct answer. If the generated answer correctly answers the question based on the target, return 1, If it is wrong, return 0. Respond ONLY with a single digit: or
Generated: (model output}
Target: (target sequence)
39

---

NMT (MetrieX)
QA (GPT-40)
Method
WMT14FFEn
WMT19DeEn
CoQA/Gpt
TriviaQ//ppt
MMLU/Gpt
GSMSk/Gpt
Mistral7b Basc
MCSE
0.127
0.212
0.226
0.478
0.338
0.453
MCNSE
0.267
0.417
0.246
0.535
0.357
0.388
Semantic Entropy
0.158
0.261
0,254
0.554
0.387
0.454
CaD
0.270
0.402
0.336
0.659
0.402
0,356
SAR
0.302
0,451
0.333
0.660
0.419
0,456
Semantic Density
0.291
0.366
0.056
0.688
0.335
0.195
SP
0.173
0.287
0.287
0.640
0.473
0.485
CoCoAsP
0.314
0.462
0.367
0.681
0.465
0.537
PPL
0.316
0.448
0.220
0.645
0.473
0.345
CoCoAPPZ
0.361
0.512
0.331
0.687
0.465
0.499
MTE
0.350
0.455
0.190
0.631
0.456
0.387
CoCOANTE
0.366
0.512
0.323
0.684
0.447
0.524
Llama8h
Jlama8h-Base Base
MCSE
0.145
0.181
0,200
0.486
0.158
0.332
MCNSE
0.319
0.361
0.183
0.530
0.170
0.324
Semantic Entropy
0.157
0.235
0.239
0.552
0.203
0.361
CEDegMat
0.287
0.367
0.350
0.621
0.299
0.300
SAR
0.354
0.420
0.302
0.610
0.319
0.382
Semantic Density
0,252
0.328
0.069
0.650
0.335
0,299
SP
0.176
0.274
0.228
0.564
0.469
0.313
CoCoAsp
0.329
0.451
0.332
0.625
0.449
0.365
PPL
0.327
0.388
0.217
0.549
0.462
0.309
CoCoAPPL
0.395
0.469
0.329
0.623
0.443
0.445
MTE
0.353
0.388
0.214
0.543
0.358
0.329
CoCoANTE
0.394
0.472
0.333
0.634
0.393
0.455
Falcon7b-Bann
MCSE
0.168
0.218
0.228
0.587
0.420
0.443
MCNSE
0.304
0.399
0.265
0.640
0,442
0.323
Semantic Entropy
0.201
0.276
0.262
0.623
0,463
0.448
CEDegMat
0.287
0,428
0.380
0.712
0.476
0,347
SAR
0.339
0.457
0.367
0.715
0.508
0.410
Semantic Density
0.323
0.422
0.076
0.746
0.396
0.349
SP
0.176
0.275
0.279
0.737
0.539
0.391
CoCoAsP
0.335
0.483
0.364
0.763
0.529
0.459
PPI.
0.322
0.454
0.245
0.722
0.539
0.341
CoCoArrz
0.394
0.531
0.345
0.754
0.529
0.513
MTE
0.370
0.455
0.228
0.707
0.533
0.385
CoCoAMTE
0.403
0.529
0.341
0.762
0.516
0.551
Table 14: PRRs for all models on QA and NMT tasks with alternative choice of performance metrics. Main model response obtained by greedy decoding.
40

---

NMT (MetrieX)
QA (GPT-40)
Method
WMT19DeEn WMT14FrEn
CoQA/Gpt
Triv
MMLU/Gpt
GSM8K/Gpt
Mistral7b Base
MCSE
0.305
0.324
0.244
0,464
0.339
0.392
MCNSE
0,360
0.466
0,252
0.516
0.358
0.188
Semantic Entropy
0.336
0.366
0.272
0.536
0.389
0,420
CaDegMat
0.468
0,485
0,339
0.623
0,404
0,239
SAR
0,455
0.527
0.336
0.628
0.421
0.298
Semantic Density
0.565
0.554
0.043
0.646
0.306
0.421
SP
0.169
0.252
0.311
0.608
0.474
0,460
CoCoAsP
0.607
0.623
0.376
0.647
0.466
0.550
PPL
0,510
0.552
0.265
0.619
0.474
0.096
CoCoApPL
0.620
0.667
0.346
0.655
0.466
0.091
MTE
0.487
0.499
0.211
0.457
0.167
CoCoANTR
0.581
0.638
0.324
0.653
0.447
0.151
Llama8b- Base
MCSE
0.284
0.271
0.191
0.451
0.155
0,302
MCNSE
0.354
0.419
0.197
0.493
0.171
0.251
Semantic Entropy
0,299
0.333
0.230
0.515
0.201
0.341
CaDegMat
0.343
0.451
0.369
0.598
0.301
0.241
SAR
0,412
0.497
0.315
0.584
0.323
0.311
Semantic Density
0.403
0,452
0.065
0.638
0.286
0.396
SP
0.247
0,289
0.225
0.523
0.467
0.332
CoCoAsT
0.473
0.568
0.346
0.597
0.440
0.418
PPL
0.437
0.517
0.238
0.498
0.459
0.185
CoCoAPPL
0,510
0.614
0.335
0.588
0.433
0.199
MTE
0.387
0.439
0.201
0.485
0.345
0.226
CoCoAMTE
0.495
0.589
0.330
0.604
0.372
0.237
Falcon7b-Bass
MCSE
0.269
0.341
0.261
0.553
0,421
0.159
MCNSE
0.325
0.403
0.312
0.620
0,443
0.064
Semantic Entropy
0.308
0.397
0.296
0.588
0.463
0.172
CEDegMat
0.381
0.478
0.398
0.686
0.477
0.083
SAR
0.389
0.483
0.398
0.685
0.509
0.152
Semantic Density
0.434
0.518
0.047
0.661
0.377
0.299
SP
0.269
0.311
0.340
0.683
0.540
0.104
CoCoAsP
0.465
0.621
0.407
0.705
0.530
0.300
PPI.
0.430
0.553
0.308
0.663
0.540
0.156
CoCoAPPL
0.526
0.655
0.376
0.691
0.530
0.128
MTE
0.418
0.452
0.275
0.615
0.535
0.054
CoCoAMTE
0.512
0.620
0.370
0.692
0.517
0.050
Table 15: PRRs for all models on QA and NMT tasks with alternative choice of performance metrics. Main model response obtained by selecting the most probable candidate among stochastically sampled responses.
41

---

G AUROC
We strongly believe that due to the considerations presented in Appendix F PRR is superior metric AUROC when comparing relative performance of UQ methods. However, we acknowledge that AUROC is widely used in modern UQ literature, so we to include AUROC results on QA datasets. All results here were obtained with GPT-as-a juddee
The results are presented in Tables and 17 correctness scoring.
CoQA/Gpt
TriviaQA/Gpt
MMLU/GPt
GSM8k/Gpt
Mistral7b Base
MCSE
0.642
0.784
0.737
0.716
MCNSE
0.653
0.806
0.744
0.684
Semantic Entropy
0.653
0.817
0.756
0.716
CHDegMat
0.690
0.859
0.759
0.682
SAR
0.694
0.862
0.770
0.717
Semantic Density
0.531
0.871
0.713
0.576
SP
0.674
0.849
0.794
0.684
CoCoAsP
0.708
0.867
0.790
0.729
PPL
0.636
0.855
0.794
0.628
CoCoApPE
0.690
0.871
0.790
0.714
MTE
0.620
0.849
0.787
0.640
CoCoANTE
0.687
0,870
0.783
0.727
Llama8b-Base
MCSE
0.635
0.779
0.620
0.705
MCNSE
0.630
0.799
0,629
0.689
Semantic Entropy
0.655
0.815
0.652
0.713
CRDegMat
0.708
0.845
0.703
0.691
SAR
0.694
0.842
0.715
0.724
Semantic Density
0.540
0.855
0.680
0.655
SP
0.647
0.816
0.787
0.669
CoCoASP
0.700
0.847
0.776
0.718
PPL
0.640
0.815
0.778
0.661
CoCoAPPE
0.699
0.848
0.770
0.745
MTE
0.633
0.817
0.726
0.667
CoCOAιTE
0.702
0.854
0.746
0.753
Falcon7b-Base
MCSE
0.648
0.807
0.770
0.766
MCNSE
0.660
0.831
0.779
0.709
Semantic Entropy
0.669
0.830
0.788
0.761
CEDegMat
0.718
0.870
0.784
0.722
SAR
0.713
0.870
0.805
0.757
Semantic Density
0.544
0.881
0.746
0.716
SP
0.681
0.866
0.820
0.731
CoCoAsP
0.719
0.884
0.815
0.774
PPL
0.645
0.860
0.820
0.700
CoCoApPL
0.705
0.882
0.815
0.800
MTE CoCoANTE
0.630
0.854
0.818 0.810
0.721
0.704
0.886
0.814
Table 16: decoding.
AUROC for all models on QA tasks. Main model response was obtained by greedy
42

---

CoQA/Gpt
TriviaQA/Gpt
MML.U/Gtt
GSM8k/Gpt
Mistral7b- Base
MOSE
0.650
0.777
0.738
0.676
MCNSE
0.656
0.798
0.745
0.602
Semantic Entropy
0.661
0.808
0.757
0.692
CaDegMat
0.693
0.845
0.760
0.625
SAR
0.695
0.849
0.771
0,643
Semantic Density
0.523
0.854
0.696
0.633
SP
0.684
0.836
0.795
0.663
CoCoAsP
0.713
0.853
0.790
0.720
PPL
0.657
0.843
0.795
0.592
CoCoApPL
0.697
0.858
0.790
0.615
MTE
0.627
0.838
0.788
0.606
CoCoAMTE
0.689
0.858
0.783
0.632
Llama8b
MCSE
0.632
0.763
0.617
0.653
MCNSE
0.633
0.783
0,629
0,629
Semantic Entropy
0.653
0,800
0,650
0.669
CHDegMat
0.716
0.836
0.703
0.633
SAR
0.698
0.831
0.715
0.659
Semantic Density
0.534
0.849
0.655
0.666
SP
0.647
0.801
0.787
0.627
CoCoASP
0.706
0.836
0.771
0.686
PPL
0.648
0.796
0.776
0.635
CoCoApPE
0.702
0.835
0.765
0.658
MTE
0.624
0.794
0.719
0.644
CoCoAиTE
0.701
0.842
0.736
0.673
Falcon7b-Ban
MCSE
0.662
0.797
0.770
0.589
MCNSE
0.826
0.779
0.571
Semantic Entropy
0.684
0,821
0.788
0.593
CEDegMat
0.726
0.863
0.785
0.580
SAR
0.727
0.862
0.806
0.611
Semantic Density
0.525
0.856
0.734
0.684
SP
0.706
0.850
0.820
0.555
CoCoASP
0.738
0.869
0.815
0.667
PPL
0.677
0.844
0.820
0.657
CoCoAPPE
0.719
0,866
0,815
0.657
MTE
0.654
0.825 0.867
0.818 0.810
0.603
CoCoAиre
0.717
0.628
Table 17: AUROC for all models on QA tasks. Main model response obtained by selecting the most probable candidate among stochastically sampled responses
H Computational Budget
Total available computational resources used to produce results in this paper amounted compute nodes each having 4xA100 40Gb GPUs. Total computational budget spent to produce results was around 400 GPU-days. Each individual combination of model and dataset amounted roughly to 16 GPU-days on average to obtain all model outputs and hidden states needed for computing the results.
43

---

I CoCoA Light Training Details
The model architecture consisted of simple multilayer perceptron (MLP) with single hidden layer:
Input dimension: equal to the embedding size of the corresponding base model (4096 for Mistra/7b- and Llama8b-Baa 3072 for Falcon7b-Base
Hidden dimension: 2048.
Output dimension: 4096.
Dropout: 0.1 applied between layers.
We trained the MLP on mean pooled hidden-layer embeddings extracted from the middle layer of an LLM. Table 18 details the size of train set for each model and dataset.
Dataset
LLaMA
Falcon
Mistral
CoQA
10,000
10,000
10,000
GSMSK
3,000
3,000
2,500
MMLU
1,461
1,461
1,461
TriviaQA
10,000
10,000
10,000
WMT14 Fr-En
6,000
6,000
6,000
WMT19 De-En
6,000
6,000
6,000
XSum
7,500
5,000
6,500
Table 18: Training set sizes (number of examples) for LlamaSb-Base Falcon7b-Base, and Mistral7b-
Base across several datasets.
Training was conducted for 20 epochs with the following hyperparameters:
Batch size: 4 (per device) with gradient accumulation of steps (effective batch size of 28).
Learning rate: 1 x 10-5 with AdamW optimizer.
Weight decay: 0.1.
Warmup ratio: 0.05.
Gradient clipping: 1.0.
44

---

Larger Model Experiments
We additionally evaluate our method using the Gemma 12B-Base model assess its performance on surge--ccale architecture. As shown in Table 19 CoCoA continues to achieve the best average performance among all methods.
Dataset
Method
XSUM
WMT14FFFn
WM
CoQA
TriviaQA
MMLU
GSM8k
MCSE
-0.014
0.277
0.408
0.262
0.556
0.449
0.386
MCNSE
-.031
0.393
0.442
0,280
0.595
0.468
0.344
Semantic Entropy
-0,011
0.273
0.430
0.287
0,621
0.531
0.407
DegMat
0.130
0.246
0.370
0.356
0.692
0.561
0.359
Laglllapa
0.130
0.195
0.307
0.323
0.667
0.526
0.312
SAR
0.087
0.429
0.494
0.335
0.688
0.582
0.418
P(True)
-0.022
0.012
-0.020
-0,006
0.195
0.046
0.145
Consistency Light
-0.006
0.328
0.483
0.345
0.709
0.408
0.421
Consistency
-0.163
0.452
0.496
0.311
0.657
0.293
0.387
MSP
0.257
0.315
0.499
0.319
0.672
0.630
0.335
CoCoAMSP
0.308
0.424
0.641
0.372
0.723
0.617
0.391
CoCoAMsP Light
0.287
0.416
0.356
0.720
0.613
0.360
PPL
0.303
0.369
0.451
0,288
0.681
0.630
0.275
CoCoApPL.
0.339
0.427
0.539
0.362
0.728
0.617
0.439
CoCoAppL Light
0.327
0.489
0.518
0.334
0.727
0.613
0.353
MTB
0.284
0.372
0.413
0.269
0.664
0.617
0.310
CoCoANTH
0.326
0.422
0.527
0.344
0.728
0.592
0.468
CoCoAιTE Light
0.308
0.501
0.498
0.313
0.719
0.565
0.386
Table 19: Detailed experimental results for Gemma 12B- 12B-Base model. Model response obtained by greedy decoding
Dataset
Method
XSUM
WMT14F:En
WMT19DeEn
CoQA
TriviaQA
MMLU
GSMSK
MCSE
0.051
0.346
0.447
0.273
0.531
0.447
0.495
MCNSE
0.088
0.441
0.456
0.319
0.580
0.465
0,533
Semantic Entropy
0.054
0.348
0.492
0.294
0.597
0.527
0.505
DegMat
0.165
0.294
0.372
0.366
0.659
0.558
0.457
Valaaaaaaaaa
0.164
0.228
0.306
0.327
0.634
0.522
0.415
SAR
0.109
0.474
0.526
0.353
0.661
0.577
0,593
P(True)
-0.029
0.037
-0.015
0.019
0.193
0.046
0.278
Consistency Light
0.076
0.478
0.631
0.373
0.671
0.443
0.744
MSP
0.188
0.353
0.466
0.361
0.646
0.624
0.223
CoCoAMSP
0.251
0.542
0.718
0.406
0.693
0.613
0.433
PPL
0.178
0.501
0,619
0.349
0.660
0.624
0.750
CoCoApPL
0.211
0.576
0.692
0.400
0.700
0.613
0.805
MTE
0.168
0.431
0.487
0.295
0.643
0.612
0.703
CoCoAMTE
0.212
0.546
0.641
0.369
0.702
0.591
0.796
Table 20: Detailed experimental results for Gemma 12B-Base model. Model response obtained by probable sample decoding
45

---

Dataset
Method
XSUM
WMT14FFEn
WMT19DeEn
CoQA
TriviaQA
MMLU
GSM8K
MCSE
0.120
0.290
0.361
0.271
0.544
0.487
0.493
MCNSE
0.126
0,366
0,439
0.308
0.597
0.497
0,464
Semantic Entropy
0.121
0.267
0.396
0.307
0.611
0.550
0.468
DegMat
0.195
0.222
0,318
0.399
0.684
0,571
0.373
Eig EagVlaaaaln
0.197
0.170
0,254
0.367
0.653
0.522
0.319
SAR
0.204
0.409
0,464
0.372
0.680
0.591
0.508
P(True)
0,030
0.043
0.018
0.014
0.079
0,313
Consistency Light
0.192
0,311
0.419
0.368
0.690
0.421
0.467
MSP
0.222
0.390
0.491
0.240
0.330
0.542
0.441
CoCOAMSP
0.282
0.452
0.598
0.347
0.624
0.574
0.498
PPL
0.187
0.400
0.526
0.183
0.344
0.542
0.468
CoCoAPPI.
0.270
0.476
0,587
0.337
0.638
0,574
0.557
MTE
0.139
0.425
0,515
0.251
0.628
0.623
0.498
CoCoANTE
0.230
0.451
0.528
0.361
0.712
0.610
0.575
Table 21: Detailed experimental results for Gemma 12B-Base model. Model response obtained by MBR decoding
46


Reviews:
[{"review_id": "mIg9pBDTYz", "reviewer": "Reviewer_zvVx", "summary": "The authors present a new framework called CoCoA for uncertainty quantification (UQ) of large language models (LLMs). The method is based on minimum Bayes risk (MBR). They start by showing how many existing “confidence” (information-based) and “consistency” (sampling-based) scores are different approximations of the same MBR objective.\n\nThen they introduce CoCoA - a method for MBR-derived scores by multiplying a sequence-level confidence term with semantic-inconsistency term. They also present CoCoA-Light, that learns to predict the consistency term from the model’s own hidden states, so it needs only a single greedy decode at test time. This can avoid the requirement to generate multiple samples for UQ.\n\nThe authors conduct experiments on 3 tasks (question answering, abstractive summarisation and machine translation) with 3 open-weight 7-8 B models (mistral, llama, falcon) and show that CoCoA and CoCoA-Light do better than baselines across PRR, AUROC and other metrics.\n\nOverall this is a very good submission.", "questions": "1. How will you use this approach for large black box models provided by the likes of OpenAI, Claude and Gemini. We can generate consistency scores. Can the confidence be approximated somewhat similar to what you have tried with CoCoA light? Assuming it is allowed under their terms, can small models be learned from such large black box models to provide confidence scores and used in this CoCoA setting?\n\n2. In Figure 1, you have illustrated the problem with model confidence. You tend to alleviate this problem by using it together with consistence measures. But is it possible that if the models are large and the generated text is small, the consistency based methods will also miss the variations in the possible outputs?\n\nWhat I mean is most of the samples look like romeo and juliet, and very few look like romeo & juliet or romeo and juliet,. This might happen if the samples are few and based on temperature sampling. In such a scenario how does MBR or CoCoA help?", "limitations": "Yes, the authors have listed 3 limitations which are quite thoughtful.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nS1.\tThe paper presents a clear theoretical unification.  While [10] had introduced uncertainty aware MBR, this paper re-derives disparate UQ scores as MBR risks and clarifies when and why they work.\n\nS2.\tThe solution presented seems relatively simple but effective metric.  The components of CoCoA are already familiar to the research community. So this method seems relatively easy to implement. I assume these methods are available in LM-Polygraph library, which the authors also seems to have used.\n\nS3.\tCoCoA-Light is very interesting. One of the main complaints about sampling techniques is the cost. Hence the solution provided in this paper to approximate using two MLPs and hence avoid sampling is very interesting.\n\nS4.\tThe paper does a very good job with the experiments. The authors experiment on 3 tasks (QA, summarization and translation), use 3 small sized models, use PRR besides other metrics. They also present useful ablations. Reporting AUROC in addition to PRR and performing experiments with other metrics (for PRR) shows thoroughness of experiments. \n\nS5.\tThe results of their experiments support the claims of the paper. CoCoA ppl performing better than baseline perplexity etc gives lot of confidence in this proposed approach.\n\nWeaknesses:\n\nW1.\tThere is some dependency on an external similarity model. They mention task and domain dependency in the limitations section. I wonder if the length of the generated text is also crucial. I might have missed it, but it would be good to comment on the effect of length of the generated output on CoCoA.\n\nW2.\tQA (especially long form), summarization and machine translation are somewhat similar tasks. Adding GSM8k is useful. It would be good to see how these methods work on code generation tasks.\n\nW3.\tMore details on the CoCoA light training will be useful. What is the size of the training set?", "was_revised": true, "final_justification": "I see that other reviewers have not considered this work as such a strong submission like I have. But I went through the discussions and I'm quite familiar with recent literature on this topic. I think the main framework formalizing the combining of consistency and confidence methods and the Cocoa lite variant are very strong contributions of this work. Hence I'm comfortable keeping my score of 6.", "rebuttal": "We thank the reviewer for their thoughtful comments and the time they invested. Please find below our responses to the concerns raised.\n\n>W1. There is some dependency on an external similarity model. They mention task and domain dependency in the limitations section. I wonder if the length of the generated text is also crucial. I might have missed it, but it would be good to comment on the effect of length of the generated output on CoCoA.\n\nWe absolutely agree that expected length of the generated sequence should be a major factor when selecting both particular confidence and consistency estimates. One has to be mindful whether this length falls into the domain that the similarity model was trained on. Most modern similarity models are trained on rather short sequences, so longer outputs might require special handling. On the other hand, from the confidence standpoint, some sort of importance weighting for individual token contribution might be needed.\n\n>W2. QA (especially long form), summarization and machine translation are somewhat similar tasks. Adding GSM8k is useful. It would be good to see how these methods work on code generation tasks.\n\nWe fully agree that code generation is an exciting domain to test UQ capabilities of LLMs. It would require a careful selection of the similarity measure for consistency estimation so we consider this to be out of scope for this submission. However, we definitely will pursue this in our future work.\n\n>W3. More details on the CoCoA light training will be useful. What is the size of the training set?\n\nTraining set sizes:\n\n\n| Dataset        | LLaMA | Falcon | Mistral |\n|----------------|-------|--------|---------|\n| CoQA           | 10,000 | 10,000 | 10,000  |\n| GSM8K          | 3,000  | 3,000  | 2,500   |\n| MMLU           | 1,461  | 1,461  | 1,461   |\n| TriviaQA       | 10,000 | 10,000 | 10,000  |\n| WMT14 Fr-En    | 6,000  | 6,000  | 6,000   |\n| WMT19 De-En    | 6,000  | 6,000  | 6,000   |\n| XSum           | 7,500  | 5,000  | 6,500   |\n\nThe average training set among datasets was around 5k examples. The architecture used was a simple MLP with one hidden layer. We have observed that optimal behavior was reached after only a couple of epochs, so probably the train set size can be further reduced with longer train time. We also highlight that these training examples don’t need to be labelled for correctness, only consistency among samples should be computed for each.\n\n>How will you use this approach for large black box models provided by the likes of OpenAI, Claude and Gemini. We can generate consistency scores. Can the confidence be approximated somewhat similar to what you have tried with CoCoA light? Assuming it is allowed under their terms, can small models be learned from such large black box models to provide confidence scores and used in this CoCoA setting?\n\nSome black box models give probabilities for top-1 tokens, which allows us to estimate confidence directly. For models which do not expose such information in their API, one could use empirical probability estimation from the relative frequencies of the outputs in the samples, or opt for verbalized approaches for confidence estimation, which are calibrated better for large models.\n\n>In Figure 1, you have illustrated the problem with model confidence. You tend to alleviate this problem by using it together with consistence measures. But is it possible that if the models are large and the generated text is small, the consistency based methods will also miss the variations in the possible outputs?\nWhat I mean is most of the samples look like romeo and juliet, and very few look like romeo & juliet or romeo and juliet,. This might happen if the samples are few and based on temperature sampling. In such a scenario how does MBR or CoCoA help?\n\nhttps://arxiv.org/abs/2406.15627 shows that for very large models, pure consistency estimation methods still perform well. This suggests that CoCoA would be a strong choice in such settings as well. We do agree, however, that the artifacts highlighted in Figure 1 are more prevalent in the outputs of smaller models. Conducting benchmarking with larger models would indeed be a valuable direction for future research."}, {"review_id": "HXx9E4nNSH", "reviewer": "Reviewer_Awk2", "summary": "This paper introduces CoCoA, a new framework for uncertainty quantification (UQ) in large language models. The method combines two key signals: the model's internal confidence (derived from token probabilities) and the semantic consistency of its outputs across multiple sampled generations. The authors frame this combination within a minimum Bayes risk framework. To address the computational cost of multiple generations, a more efficient variant, CoCoA Light, is proposed, which uses a lightweight predictor to estimate semantic inconsistency. The authors conduct experiments on various tasks, including question answering, summarization, and machine translation, showing that CoCoA outperforms existing UQ baselines in failure prediction.", "questions": "1.  How would you adapt the CoCoA framework for use with closed-source, black-box models (e.g., via the GPT-4 API) where direct access to token probabilities is not available?\n2.  The paper omits comparisons with verbalized confidence methods. Could you elaborate on the potential advantages or disadvantages of CoCoA relative to this line of work? \n3.  While the experiments show that a multiplicative combination works well, have you explored other aggregation functions (e.g., attention-based weighting)? \n4.  How the reasoning-models like QwQ-32B perform on the CoCoA framework?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1.  The paper provides a clear and well-grounded framework that unifies two important sources of uncertainty—model confidence and output consistency. The connection to the minimum Bayes risk framework provides a solid theoretical foundation.\n2.  The experiments are comprehensive, covering multiple tasks, datasets, and models. CoCoA demonstrates significant and consistent improvements over a variety of established UQ baselines on failure detection metrics like PRR and AUROC. And the CoCoA framework is presented as a plug-in module that can enhance different information-based uncertainty estimators, highlighting its flexibility and potential for broad applicability. \n\n**Weaknesses:**\n\n1.  **Limited Novelty:** The core idea of combining confidence and consistency feels somewhat incremental. The paper could be strengthened by a deeper investigation into the relationship and interaction between these two types of uncertainty, beyond the proposed multiplicative combination. The current insight into *why* this combination is so effective is not fully explored.\n2.  **Missing Baselines:** The evaluation does not include comparisons with \"verbalized confidence\" methods[1]. [1] demonstrates that the logits of the output of LLMs suffers from over-confident issue due to RLHF process. [2] proposes sampling and consistency-based confidence calibration methods to measure the intrinsic uncertainty of LLMs. These methods, which prompt the model to express its own uncertainty, are a relevant and important class of UQ techniques and would serve as a crucial point of comparison.\n3.  **Incomplete Evaluation Metrics:** The evaluation relies primarily on failure detection metrics (PRR, AUROC). While important, these do not measure confidence calibration. Including metrics like the Expected Calibration Error (ECE) would provide a more complete picture of the UQ quality.\n4.  **Limited Model Scale:** The experiments are conducted on models up to 8B parameters. The claims would be more convincing if the method's effectiveness were also demonstrated on larger, state-of-the-art models (e.g., Llama 3 70B, even with quantization). \n\n[1] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In EMNLP, 2023.\n\n[2] Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. In ICLR, 2023.", "was_revised": true, "final_justification": "My concerns are addressed.", "rebuttal": "We thank the reviewer for the insightful comments. Below, we would like to address the concerns raised by reviewer:\n\n>Limited Novelty: The core idea of combining confidence and consistency feels somewhat incremental. The paper could be strengthened by a deeper investigation into the relationship and interaction between these two types of uncertainty, beyond the proposed multiplicative combination. The current insight into why this combination is so effective is not fully explored.\n\n\n**On limited novelty**:  \nWhile our work builds upon established concepts of confidence and consistency, our contribution is threefold.\n\n(1) We focus on selective generation, where the decision to output a response is based on the confidence score of a specific candidate, rather than modeling uncertainty over a potential input. Specifically, we frame our approach as a variant of Minimum Bayes Risk Decoding, which defines a utility function that naturally favors responses that are both confident (according to the model) and consistent (with other sampled candidates).\n\n(2) We show that our MBR-based framework is flexible and performs well across various uncertainty measures. \n\n(3) We demonstrate that the consistency component can be learned, significantly reducing the computational cost associated with extensive sampling.\n\n**On discussing potential reasons for the best performance of multiplicative formulation**:\n\nWe agree that more discussion in the combination strategy is valuable. In Appendix C.2, we compare our multiplicative approach to additive and full-matrix variants. The additive form often lets one signal dominate (e.g., consistency gets overwhelmed by high-entropy outputs). Full-matrix averaging dilutes pairwise relationships by incorporating irrelevant comparisons. The multiplicative form works best because it naturally favours an output if it is both confident and semantically consistent. This enforces alignment between both criteria without introducing extra parameters or reweighting. \n\n> Missing Baselines: The evaluation does not include comparisons with \"verbalized confidence\" methods[1]\n\nWe do not focus on verbalized uncertainty methods, as prior work (https://openreview.net/pdf?id=gjeQKFxFpZ, https://aclanthology.org/2025.tacl-1.11/  ) has shown that LLMs tend to express high confidence regardless of actual correctness - particularly in smaller models, like 7-8b models used in our experiments. In such models these methods often yield poorly calibrated scores that do not reflect true reliability.\n\n> Incomplete Evaluation Metrics: The evaluation relies primarily on failure detection metrics (PRR, AUROC). While important, these do not measure confidence calibration. Including metrics like the Expected Calibration Error (ECE) would provide a more complete picture of the UQ quality.\n\nWhile calibration is an important aspect of uncertainty quantification (UQ), it assumes that uncertainty scores are interpretable as probabilities and that output correctness can be measured in binary terms. Neither of these assumptions holds for much of our experimental setup. Most UQ methods produce unbounded scores that lack a probabilistic interpretation, and many of the quality metrics we use - particularly in tasks like neural machine translation (NMT) and summarization - are continuous rather than binary. As a result, standard calibration metrics such as Expected Calibration Error (ECE) are not directly applicable.\n\nHowever, we adopted an approach to calibration proposed in (https://aclanthology.org/2025.tacl-1.11/) and fitted an isotonic regression to pairs of raw UQ scores and response quality values, and report here MSE between ground truth quality and prediction of the regression here:\n\n| Metric | Llama QA | Llama NMT | Llama SUM | Mistral QA | Mistral NMT | Mistral SUM | Falcon QA | Falcon NMT | Falcon SUM |\n|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n| MCSE | 0.177 | 0.005 | 0.074 | 0.159 | **0.004** | 0.099 | 0.140 | 0.005 | 0.059 |\n| MCNSE | 0.179 | 0.005 | 0.074 | 0.160 | **0.004** | 0.098 | 0.140 | **0.004** | 0.059 |\n| Semantic Entropy | 0.172 | 0.005 | 0.074 | 0.154 | **0.004** | 0.100 | 0.136 | 0.005 | 0.059 |\n| DegMat | 0.167 | 0.005 | 0.073 | 0.153 | **0.004** | 0.097 | 0.132 | 0.005 | 0.059 |\n| EigValLaplacian | 0.171 | 0.005 | 0.073 | 0.156 | **0.004** | 0.097 | 0.135 | 0.005 | 0.058 |\n| SAR | 0.164 | **0.004** | 0.073 | 0.150 | **0.004** | 0.098 | 0.131 | **0.004** | 0.059 |\n| Dissimilarity | 0.161 | **0.004** | 0.074 | 0.149 | **0.004** | 0.098 | 0.130 | **0.004** | 0.057 |\n| MSP | 0.165 | **0.004** | 0.066 | 0.151 | **0.004** | **0.088** | 0.131 | 0.005 | 0.058 |\n| CoCoA_MSP | **0.158** ↑ | **0.004** ↑ | **0.065** ↑ | **0.147** ↑ | **0.004** ↑ | **0.088** ↑ | 0.127 ↑ | **0.004** ↑ | **0.056** ↑ |\n| Perplexity | 0.170 | 0.005 | 0.066 | 0.155 | **0.004** | 0.089 | 0.135 | **0.004** | 0.058 |\n| CoCoA_Perplexity | 0.160 ↑ | **0.004** ↑ | **0.065** ↑ | 0.148 ↑ | **0.004** ↑ | 0.089 ↑ | 0.126 ↑ | **0.004** ↑ | 0.057 ↑ |\n| MeanTokenEntropy | 0.175 | 0.005 | 0.067 | 0.156 | **0.004** | **0.088** | 0.135 | **0.004** | 0.058 |\n| CoCoA_MeanTokenEntropy | 0.161 ↑ | **0.004** ↑ | **0.065** ↑ | 0.148 ↑ | **0.004** ↑ | 0.090 ↑ | **0.125** ↑ | **0.004** ↑ | 0.057 ↑ |\n\n\n*Results for Evaluated Sequence – Greedy Sample: Mean MSE between quality and calibrated UQ score across datasets for each task, lower values mean better calibration. The best performing method is in bold, and the second-best is underlined. Arrows indicate improvement in CoCoA over the base version.*\n\n\nCoCoA family methods still retain their edge over the baselines in most of the task-model pairs in this setting.\n\n>Limited Model Scale: The experiments are conducted on models up to 8B parameters. The claims would be more convincing if the method's effectiveness were also demonstrated on larger, state-of-the-art models (e.g., Llama 3 70B, even with quantization).\n\nWe appreciate the reviewer’s point regarding model size. Due to limited compute resources, our experiments focus on models up to 8B parameters (Similar to https://aclanthology.org/2025.tacl-1.11/, https://neurips.cc/virtual/2024/poster/97746, https://arxiv.org/abs/2503.05318, https://arxiv.org/abs/2406.04306 ).\n\nOur method does not rely on model-specific tuning and is compatible with quantized or cached generation setups, making it feasible to apply to larger models in future work. We agree that testing on 70B-scale models like LLaMA 3 would further strengthen our claims, and we plan to explore this direction in the future.\n\n>How would you adapt the CoCoA framework for use with closed-source, black-box models (e.g., via the GPT-4 API) where direct access to token probabilities is not available?\n\nWhile CoCoA relies on confidence estimates such as token-level probabilities, adaptation to black-box models like GPT-4 is still feasible:\nGPT-4 and other APIs expose log probabilities for generated outputs. This allows the computation of confidence-based metrics like Maximum Sequence Probability (MSP).\nIn cases where logprobs are unavailable, they can be estimated empirically from the sampled responses, or verbalized confidence can be used instead, which should perform well enough for larger models which are often used in a black-box setting.\n\n>The paper omits comparisons with verbalized confidence methods. Could you elaborate on the potential advantages or disadvantages of CoCoA relative to this line of work?\n\nWe do not focus on verbalized uncertainty methods, as prior work (https://openreview.net/pdf?id=gjeQKFxFpZ, https://aclanthology.org/2025.tacl-1.11/) has shown that LLMs tend to express high confidence regardless of actual correctness - particularly in smaller models. In such models these methods often yield poorly calibrated scores that do not reflect true reliability.\n\n> While the experiments show that a multiplicative combination works well, have you explored other aggregation functions (e.g., attention-based weighting)?\n\nIn Appendix C.2, we evaluated alternatives to the multiplicative combination-including additive scoring and full-sample averaging. The underperformance of these alternatives stems from their inability to balance both confidence and consistency signals effectively.\n\nWhile we did not experiment with attention‑based weighting of token logprobs (if that’s what is being referred to in the question), we stress that any confidence measure can be used in the CoCoA (CCP, TokenSAR, attention-based re-weighting, etc). We show that even using very basic approaches like perplexity and sequence probability can improve over the baseline methods.\n\n>How the reasoning-models like QwQ-32B perform on the CoCoA framework?\n\nWe have not evaluated CoCoA on reasoning models such as QwQ-32B, so we cannot make claims about their performance. However, if a long reasoning chain is considered as a part of the output, we hypothesize that more complicated measures of confidence and consistency are required. The reason for that is that over long sequences, simple aggregation of log probabilities can be drowned in noise and simple NLI similarity is not well-defined. In general, uncertainty quantification for reasoning setup and long-form outputs in general requires special considerations that we are planning to tackle in further work on CoCoA, but which are out of scope for this particular submission."}, {"review_id": "mqZ0FpLelz", "reviewer": "Reviewer_yPFe", "summary": "This paper introduces CoCoA, which is a framework that generalizes several prior works efforts in direction of semantic clustering from a standpoint of uncertainty estimation as risk prediction.\nThe decomposition consists of Consistency and Confidence scores.\nThe former corresponds to the plurality of possible predictions, while the latter corresponds to confidence of the model in the 'main predicted' sequence. \nThe authors further propose a computationally cheaper version of the measure, which is achieved by learning an estimate for Consistency score.\nThe method shows competitive performance on a variety of tasks.", "questions": "1. Would introducing weighing for the two terms into the AdditiveCoCoA improve its performance for specific tasks?\n2. How would the evaluation picture change if the PRR rejection cut-off was changed from it's 50% value?\n3. What is the theoretical justification for multiplicative CoCoA combination rather than additive?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 1, "originality": 3, "strengths_and_weaknesses": "#### Strengths\n1. **Overall paper quality:** the paper is well structured and written. \n2. **Originality:** CoCoA generalizes several prior works that deal with semantic invariances in uncertainty estimation for NLP. It advances the connections between uncertainty and prediction risks in the NLP domain.\n3. **Evaluation:** the authors perform extensive evaluation on datasets and models standard to the subfield. \n\n#### Weaknesses\n1. **Introducing a Framework Rather than a Specifc Method**: as is acknowledged by the authors in the limitations, the different versions of CoCoA excel at different tasks.\nAn entire framework of possible options is therefore presented that can accommodate many prior methods and has multiple design choices. \nThe method empirically speaking is largely a combination of two families of prior work methods, each performing better in different scenarios.\nThis erodes the methods empirical utility. \n2. **Treatment of risk combinations**: even though the authors attempt to derive their measure from a theoretically grounded standpoint of prediction risk, the resulting CoCoA measures are still quite arbitrary.\nFor instance, the authors do not provide a theoretical explanation for multiplicative combination of the confidence and consistency and instead settle for picking one with greater empirical performance.\nIf the Confidence and Consistency parts both correspond to some sort of risk, I would imagine them being additive rather than multiplicative. \n3. **Evaluation Nuances**:\n    - the authors discuss the advantages of PRR over AUROC (even though its not the focus of the paper), yet acknowledge that they have to cut the PRR evaluation at 50% rejection to avoid artifacts.\n    - in appendix G table 16 shows a less confident picture of the CoCoA performance when using the AUROC+Judge instead of PRR + Rouge/AlignScore.", "was_revised": true, "final_justification": "The paper was initially lacking some details and had a gap in between the theoretical and empirical part. The authors made some steps to bridge that gap during the rebuttal. I hope that all the promised material ends up being present in the final revision.", "rebuttal": "We appreciate the reviewer’s feedback and address their concerns below.\n\n>Introducing a Framework Rather than a Specifc Method: as is acknowledged by the authors in the limitations, the different versions of CoCoA excel at different tasks. An entire framework of possible options is therefore presented that can accommodate many prior methods and has multiple design choices. The method empirically speaking is largely a combination of two families of prior work methods, each performing better in different scenarios. This erodes the methods empirical utility.\n\nWhile it is true that our method combines two families of prior approaches, this is not a weakness but an intentional design choice. In the general case, a maximum sequence probability (MSP)-based combination yields strong, reliable performance. The framework makes it flexible: when time and computational resources are available, it can be further optimized to better suit the specifics of a task. This makes it not only effective in general settings but also adaptable to specialized use cases. In this sense, we view the introduction of a framework not as an erosion of empirical utility, but as a practical strength - offering a solid default with room for task-specific refinement.\n\n>Treatment of risk combinations: even though the authors attempt to derive their measure from a theoretically grounded standpoint of prediction risk, the resulting CoCoA measures are still quite arbitrary. For instance, the authors do not provide a theoretical explanation for multiplicative combination of the confidence and consistency and instead settle for picking one with greater empirical performance. If the Confidence and Consistency parts both correspond to some sort of risk, I would imagine them being additive rather than multiplicative.\n\nIn Appendix C.2, we compare our multiplicative formulation to additive and full-matrix consistency variants:\nAdditive combination often allows one signal to dominate, especially since some information-theoretic confidence scores (e.g., entropy) are unbounded while consistency is bounded in [0, 1]. This imbalance undermines the contribution of consistency and practically requires proper empirical selection of relative contribution of each source. This selection would be task-dependent and would generalize poorly to the OOD settings.\nOn the other hand, since consistency is bounded between 0 and 1, it serves as a natural scaling factor on confidence. This enforces mutual alignment between the two signals without requiring learned weights or tuning.\n\nWhile additive risk formulations are common in theory, in practice this setting involves differently scaled signals, and the multiplicative form has shown stronger empirical performance with a clear intuitive justification. That said, we agree that deeper theoretical analysis is a promising direction for future work.\n\n>Evaluation Nuances: the authors discuss the advantages of PRR over AUROC (even though its not the focus of the paper), yet acknowledge that they have to cut the PRR evaluation at 50% rejection to avoid artifacts.\n\nThe choice to cut off at 50% was deliberate, as it rarely makes sense to reject all of the answers in practical settings. By focusing on the 50% most uncertain cases, we target the region where rejection is most relevant and actionable. This provides a more realistic and interpretable measure of performance in scenarios where selective prediction is actually useful.\n\nFurthermore, due to diminishing sample size, the expected quality of the output on the remaining dataset is a heteroscedastic value wrt rejection rate, and extreme rejection can lead to very noisy estimates.\n\nThat being said, we report PRR up to 100% rejection here, and CoCoA-family methods retain their edge over the baselines even in this setup.\n\n| Metric | Llama QA | Llama NMT | Llama SUM | Mistral QA | Mistral NMT | Mistral SUM | Falcon QA | Falcon NMT | Falcon SUM |\n|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n| MCSE | 0.486 | 0.439 | 0.031 | 0.550 | 0.424 | 0.006 | 0.613 | 0.440 | 0.284 |\n| MCNSE | 0.468 | 0.449 | 0.087 | 0.541 | 0.465 | 0.037 | 0.599 | 0.478 | 0.211 |\n| Semantic Entropy | 0.528 | 0.474 | 0.029 | 0.581 | 0.462 | 0.014 | 0.631 | 0.478 | 0.301 |\n| DegMat | 0.539 | 0.374 | 0.165 | 0.566 | 0.377 | 0.224 | 0.642 | 0.426 | 0.363 |\n| EigValLaplacian | 0.515 | 0.360 | 0.158 | 0.546 | 0.367 | 0.214 | 0.627 | 0.414 | 0.363 |\n| SAR | 0.578 | 0.501 | 0.155 | 0.603 | 0.503 | 0.170 | 0.667 | 0.508 | 0.269 |\n| Consistency | 0.586 | 0.427 | 0.105 | 0.604 | 0.426 | 0.144 | 0.656 | 0.450 | **0.383** |\n| MSP | 0.544 | 0.495 | **0.584** | 0.552 | 0.494 | **0.559** | 0.637 | 0.487 | 0.290 |\n| `CoCoA_MSP` | **0.607** ↑ | **0.575** ↑ | 0.560 ↑ | **0.623** ↑ | **0.575** ↑ | 0.530 ↑ | 0.688 ↑ | **0.582** ↑ | 0.382 ↑ |\n| Perplexity | 0.491 | 0.434 | 0.543 | 0.518 | 0.463 | 0.520 | 0.601 | 0.493 | 0.229 |\n| `CoCoA_Perplexity` | 0.591 ↑ | 0.497 ↑ | 0.538 ↑ | 0.602 ↑ | 0.518 ↑ | 0.535 ↑ | 0.691 ↑ | 0.534 ↑ | 0.344 ↑ |\n| MeanTokenEntropy | 0.450 | 0.417 | 0.537 | 0.507 | 0.452 | 0.506 | 0.592 | 0.492 | 0.216 |\n| `CoCoA_MeanTokenEntropy` | 0.586 ↑ | 0.486 ↑ | 0.528 ↑ | 0.606 ↑ | 0.506 ↑ | 0.520 ↑ | **0.693** ↑ | 0.526 ↑ | 0.345 ↑ |\n\n*Results for Evaluated Sequence – Greedy Sample: Mean PRR up to full rejection across datasets for each task. The best performing method is in bold, and the second-best is underscored. Arrows indicate improvement in CoCoA over the base version.*\n\n> Would introducing weighing for the two terms into the AdditiveCoCoA improve its performance for specific tasks?\n\nYes, introducing learned or tuned weights could improve the performance of AdditiveCoCoA for specific tasks. However, adding weights would effectively turn the method into a supervised or task-specific approach, as these weights would need to be tuned on downstream validation sets to balance the contributions of each term. This tuning may yield better empirical performance but sacrifices the task-agnostic and unsupervised nature of the original CoCoA formulation.\nIn contrast, our multiplicative formulation avoids this trade-off by enforcing mutual agreement between the two signals without requiring tuning or additional parameters.\n\n>How would the evaluation picture change if the PRR rejection cut-off was changed from it's 50% value?\n\nWe report PRR up to 100% rejection here, and CoCoA-family methods retain their edge over the baselines even in this setup.\n\n>What is the theoretical justification for multiplicative CoCoA combination rather than additive?\n\nThe key justification lies in how consistency and confidence are treated as signals of uncertainty within the Minimum Bayes Risk (MBR) framework. Consistency is bounded between 0 and 1, while confidence metrics such as perplexity (PPL) or mean token entropy (MTE) are unbounded. As a result, an additive combination would require careful supervision and tuning of weights to ensure a balanced contribution from each signal. Without proper tuning, one signal could dominate or be underrepresented, reducing the robustness of the final score. By contrast, the multiplicative combination used in CoCoA avoids this issue, as it inherently normalizes the interaction between signals, preserving their relative informativeness without requiring task-specific weight calibration."}, {"review_id": "3veuUEF3mH", "reviewer": "Reviewer_aWz6", "summary": "This paper introduces CoCoA (Confidence and Consistency-based Approaches), a new framing of methods for uncertainty quantification (UQ) in large language models (LLMs), grounded in the Minimum Bayes Risk (MBR) framework. The goal is to unify information-based (model confidence) and consistency-based (semantic agreement between sampled outputs) UQ methods into a principled decision-making risk-based formulation. The authors also propose CoCoA Light, a learned approximation of the full CoCoA score that avoids repeated sampling using an additional held out set. They evaluate their methods on several datasets and compared with a few baseline methods.", "questions": "Please refers to my comments in the weakness section.", "limitations": "The authors mentioned the limitations but not much on the societal impact. I will encourage the authors to discuss this aspect in their rebuttal.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. Principled unification: The use of MBR as a unifying lens for both confidence and consistency is theoretically grounded (although reading line 47-49 and reference [9, 10] does not offer me the same hint, I wonder if the authors could elaborate for me). CoCoA makes this combination interpretable as a multiplicative interaction of model uncertainty and semantic inconsistency.\n\n2. Practical algorithm: CoCoA Light reduces the sampling cost by learning to approximate the consistency-based term from representations, making it more applicable for real-world applications.\n\n3. Empirical performance: Across diverse tasks and multiple open-source LLMs, CoCoA and CoCoA Light outperform good baselines.\n\n4. In general well-written and easy to follow.\n\n**Weaknesses**\n\n1. Limited novelty compared to prior works: While the MBR framing is clean, many recent UQ methods already combine confidence and consistency (e.g., SAR, Semantic Entropy). The main novelty lies in formalizing their combination via MBR. I would like to ask the authors to explain a bit more their motivation from [9] and [10], and what is the novelty of this formulation.\n\n2. Lack of theoretical justification: The main section for the CoCoA seems to build up a combination of model confidence and semantic inconsistency directly, without explaining the intuition and theoretical justification.\n\n3. Similarity function bottleneck: Results depend heavily on the semantic similarity function used. The paper acknowledges this but could further explore when and which score should be used for certain tasks and why (which is really helpful if we do not have an additional held out set, e.g. open-ended generation tasks). \n\n4. Measurement using PRR: While CoCoA improves PRR, it's less clear how well it calibrates model uncertainty. My understanding is this metric measures the ranking but not model uncertainty directly.\n\n5. Error bars missing: I am wondering why the variance is not reported in this paper.", "was_revised": true, "final_justification": "All details in the rebuttal history", "rebuttal": "We thank the reviewer for their valuable feedback and address the concerns raised below.\n\n>Limited novelty compared to prior works: While the MBR framing is clean, many recent UQ methods already combine confidence and consistency (e.g., SAR, Semantic Entropy). The main novelty lies in formalizing their combination via MBR. I would like to ask the authors to explain a bit more their motivation from [9] and [10], and what is the novelty of this formulation.\n\nWe acknowledge that there are existing methods which blend confidence and consistency. However, we argue that our method proposes several novel contributions:\n\nWhile existing approaches have explored combining confidence and consistency and they did serve as an inspiration for our work, they tend to aggregate both confidence and consistency signals over all the sampled outputs, which reduces sensitivity to the expected quality of the particular model output (https://arxiv.org/pdf/2305.19187).\n\nThis observation informs our design of CoCoA, which formalizes uncertainty estimation through a Minimum Bayes Risk (MBR) framework, interpreting uncertainty as expected risk based on both confidence and semantic consistency via explicit utility modeling. Within the MBR framework we introduced, semantic similarity and consistency are not used to reweight model confidence scores, but rather a separate signal integrated into the utility function.\n\nTo further improve efficiency, we introduce CoCoA Light-a supervised approximation of the consistency component-which enables uncertainty estimation without the need for repeated sampling at inference time, which is another limitation of existing hybrid approaches.\n\n>Lack of theoretical justification: The main section for the CoCoA seems to build up a combination of model confidence and semantic inconsistency directly, without explaining the intuition and theoretical justification.\n\n\nWhile we don’t have the full theoretical treatment for the choice of the loss function for CoCoA (which is often a heuristic choice in MBR framework), we can consider a case study of the default confidence choice (MSP) that leads to the strongest performance.\n\nIn this case, $U(y_*) = - \\log p(y_* | x) \\cdot \\frac{1}{M} \\sum_i (1 - s(y_*, y_i)$\n\nEquivalently, $U(y_*) = - \\log [ p(y_* | x)^{\\frac{1}{M} \\sum_i (1 - s(y_*, y_i))}] = - \\log [ p(y_* | x)^{ 1- \\frac{1}{M} \\sum_i s(y_*, y_i)}]$.\n\nThis formula is essentially a criterion to select the maximum probability sequence but tempered by a semantic consistency of this sequence with other samples. From the MBR perspective, it is a natural criterion that balances choosing the sequence with high probability and simultaneously from a part of the distribution with highly concentrated meaning (high average similarity).\n\n\n>Measurement using PRR: While CoCoA improves PRR, it's less clear how well it calibrates model uncertainty. My understanding is this metric measures the ranking but not model uncertainty directly.\n\nWhile calibration is indeed a very important topic in UQ, it requires uncertainty scores to be interpretable as probabilities and the output correctness measure to be binary. Neither is true for a large portion of our experimental setup. Most UQ methods output unbounded values which can not be interpreted as probabilities, while many quality metrics are continuous, especially in NMT and summarization tasks. So straightforward computation of ECE is not possible.\n\nHowever, we adopted an approach to calibration proposed in (https://arxiv.org/abs/2406.15627) and fitted an isotonic regression to pairs of raw ue scores and response quality, and report MSE between ground truth quality and prediction of the regression here:\n\n| Metric | Llama QA | Llama NMT | Llama SUM | Mistral QA | Mistral NMT | Mistral SUM | Falcon QA | Falcon NMT | Falcon SUM |\n|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n| MCSE | 0.177 | 0.005 | 0.074 | 0.159 | **0.004** | 0.099 | 0.140 | 0.005 | 0.059 |\n| MCNSE | 0.179 | 0.005 | 0.074 | 0.160 | **0.004** | 0.098 | 0.140 | **0.004** | 0.059 |\n| Semantic Entropy | 0.172 | 0.005 | 0.074 | 0.154 | **0.004** | 0.100 | 0.136 | 0.005 | 0.059 |\n| DegMat | 0.167 | 0.005 | 0.073 | 0.153 | **0.004** | 0.097 | 0.132 | 0.005 | 0.059 |\n| EigValLaplacian | 0.171 | 0.005 | 0.073 | 0.156 | **0.004** | 0.097 | 0.135 | 0.005 | 0.058 |\n| SAR | 0.164 | **0.004** | 0.073 | 0.150 | **0.004** | 0.098 | 0.131 | **0.004** | 0.059 |\n| Dissimilarity | 0.161 | **0.004** | 0.074 | 0.149 | **0.004** | 0.098 | 0.130 | **0.004** | 0.057 |\n| MSP | 0.165 | **0.004** | 0.066 | 0.151 | **0.004** | **0.088** | 0.131 | 0.005 | 0.058 |\n| CoCoA_MSP | **0.158** ↑ | **0.004** ↑ | **0.065** ↑ | **0.147** ↑ | **0.004** ↑ | **0.088** ↑ | 0.127 ↑ | **0.004** ↑ | **0.056** ↑ |\n| Perplexity | 0.170 | 0.005 | 0.066 | 0.155 | **0.004** | 0.089 | 0.135 | **0.004** | 0.058 |\n| CoCoA_Perplexity | 0.160 ↑ | **0.004** ↑ | **0.065** ↑ | 0.148 ↑ | **0.004** ↑ | 0.089 ↑ | 0.126 ↑ | **0.004** ↑ | 0.057 ↑ |\n| MeanTokenEntropy | 0.175 | 0.005 | 0.067 | 0.156 | **0.004** | **0.088** | 0.135 | **0.004** | 0.058 |\n| CoCoA_MeanTokenEntropy | 0.161 ↑ | **0.004** ↑ | **0.065** ↑ | 0.148 ↑ | **0.004** ↑ | 0.090 ↑ | **0.125** ↑ | **0.004** ↑ | 0.057 ↑ |\n\n*Results for Evaluated Sequence – Greedy Sample: Mean MSE between quality and calibrated UQ score across datasets for each task. The best performing method is in bold, and the second-best is underlined. Arrows indicate improvement in CoCoA over the base version.*\n\nCoCoA family methods still retain their edge over the baselines in most of the task-model pairs in this setting.\n\n>Similarity function bottleneck: Results depend heavily on the semantic similarity function used. The paper acknowledges this but could further explore when and which score should be used for certain tasks and why (which is really helpful if we do not have an additional held out set, e.g. open-ended generation tasks).\n\nThis is indeed an important point, we thank the reviewer for bringing this up. Indeed, the choice of the similarity function is the important step here. We hypothesize that for general natural language generation with outputs of limited length (1-2 sentences) any capable NLI/CrossEncoder model will suffice. For longer generations, models that perform similarity estimation on the chunks with further aggregation over the chunks (like AlignScore for example) can be a better choice. For special cases, like code generation, a domain-specific similarity function would probably be needed.\n\n>Error bars missing: I am wondering why the variance is not reported in this paper.\n\nThank you for pointing this out. We only ran a single instance of the experiment. Each run requires generating 2,000 outputs from the model, followed by extensive quality and other downstream evaluations. Due to the significant computational cost, repeating the experiment enough times to produce statistically meaningful error bars was not feasible within our resource constraints. We acknowledge this as a limitation and plan to address it in future work, resources permitting."}]

Author rebuttal:

Meta-review: # Summary
This paper introduces CoCoA, a new framework for uncertainty quantification (UQ) in large language models. The method combines two key signals: the model's internal confidence (derived from token probabilities) and the semantic consistency of its outputs across multiple sampled generations. The authors frame this combination within a minimum Bayes risk framework. To address the computational cost of multiple generations, a more efficient variant, CoCoA Light, is proposed, which uses a lightweight predictor to estimate semantic inconsistency. The authors conduct experiments on various tasks, including question answering, summarization, and machine translation, showing that CoCoA outperforms existing UQ baselines in failure prediction.

# Strengths
* The paper is well structured and written.
* The paper provides a clear and well-grounded framework that unifies two important sources of uncertainty—model confidence and output consistency. The connection to the minimum Bayes risk framework provides a solid theoretical foundation.
* The experiments are comprehensive, covering multiple tasks, datasets, and models.
* CoCoA Light reduces the sampling cost by learning to approximate the consistency-based term from representations, making it more applicable for real-world applications.

# Weaknesses
* The core idea of combining confidence and consistency feels somewhat incremental. While the MBR framing is clean, many recent UQ methods already combine confidence and consistency (e.g., SAR, Semantic Entropy). The main novelty lies in formalizing their combination via MBR.

# Overall

Following the author-reviewer discussion period, no major weaknesses or concerns remain. The reviewers generally agree that the paper is well-written and supported by extensive experiments across numerous tasks, datasets, and models. In particular, they view CoCoA Light as an important and novel contribution to the research community.

A slight reservation was noted regarding the novelty compared to previous work. While combining confidence and consistency methods is not novel per se, framing it as a combination via MBR is new. Additionally, CoCoA Light is recognized as a novel and efficient contribution.
Decision: Accept (poster)