Title: CoCoA: A Minimum Bayes Risk Framework Bridging Confidence and Consistency for Uncertainty Quantification in LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

Large Language Model

Uncertainty Quantification

Minimum Bayes Risk
Abstract: Uncertainty quantification for Large Language Models (LLMs) encompasses a diverse range of approaches, with two major families being particularly prominent: (i) information-based, which estimate model confidence from token-level probabilities, and (ii) consistency-based, which assess the semantic agreement among multiple outputs generated using repeated sampling. While several recent methods have sought to combine these two paradigms to improve uncertainty quantification performance, they often fail to consistently outperform simpler baselines. In this work, we revisit the foundations of uncertainty estimation through the lens of Minimum Bayes Risk decoding, establishing a direct link between uncertainty and the optimal decision-making process of LLMs. Building on these findings, we propose CoCoA, a unified framework that integrates model confidence with output consistency, yielding a family of efficient and robust uncertainty quantification methods. We evaluate CoCoA across diverse tasks, including question answering, abstractive text summarization, and machine translation, and demonstrate sizable improvements over state-of-the-art uncertainty quantification approaches.

CoCoA : A Minimum Bayes Risk Framework Bridging Confidence and Consistency for Uncertainty Quantification in LLMs

Abstract

Uncertainty quantification for Large Language Models (LLMs) encompasses a diverse range of approaches, with two major families being particularly prominent: (i) information-based, which estimate model confidence from token-level probabilities, and (ii) consistency-based, which assess the semantic agreement among multiple outputs generated using repeated sampling. While several recent methods have sought to combine these two paradigms to improve uncertainty quantification performance, they often fail to consistently outperform simpler baselines. In this work, we revisit the foundations of uncertainty estimation through the lens of Minimum Bayes Risk decoding, establishing a direct link between uncertainty and the optimal decision-making process of LLMs. Building on these findings, we propose CoCoA , a unified framework that integrates model confidence with output consistency, yielding a family of efficient and robust uncertainty quantification methods. We evaluate CoCoA across diverse tasks, including question answering, abstractive text summarization, and machine translation, and demonstrate sizable improvements over state-of-the-art uncertainty quantification approaches.

1 Introduction

Large Language Models (LLMs) have revolutionized natural language processing (NLP), enabling advances in information retrieval [Zhu et al., 2025], question answering [Kwiatkowski et al., 2019], machine translation [Kocmi and Federmann, 2023], and a broad range of other NLP applications. As these models become an integral part of our everyday life, ensuring the reliability of their outputs is crucial, especially in high-stakes scenarios where errors can have serious consequences. One way to address this challenge is through uncertainty quantification (UQ), which focuses on estimating the confidence of model predictions.

Question

Tybalt is a character in which of Shakespeare's plays?

romeo & juliet

romeo and juliet,

Sampled answers

Probability

0.687

0.045

0.030

CoCoA:

0.739

It's Friday

The day is Friday

Evaluated sequence

Alternative answers

LLM

0.08

0.15

Prompt:

"Vendredi" is French

for which day of the week?

Friday

CoCoA:

0.052

Vauxhall

Evaluated sequence

Alternative answers

LLM

0.85

0.89

Prompt:

Which motor

manufacturer builds the

'Movano' range of vans and

mini-buses?

Toyota

UQ for LLMs is a rapidly advancing research area, with new UQ methods emerging each year. Most novel techniques are based on two fundamental approaches: (a) information-theoretic analysis or (b) assessment of output consistency.

Information-theoretic methods quantify the confidence of a model by analyzing the probability distributions it induces for predictions [Malinin and Gales, 2021, Fomicheva et al., 2020]. A key limitation of these methods is that they cannot account for semantic variability across multiple possible responses for the same input. Specifically, the model may generate answers with the same meaning but with very different assigned probabilities; see Figure 1. LLMs are trained to predict the next token in a sequence based on patterns observed in vast amounts of data, resulting in varying probabilities for semantically equivalent output sequences.

In contrast, consistency-based methods directly analyze the semantic relationship between the sampled outputs [Fomicheva et al., 2020, Lin et al., 2024], capturing uncertainty as objective variability of meaning among the sampled outputs.

Information-theoretic and consistency-based methods have complementary strengths. For this reason, recent state-of-the-art methods aimed to unify these approaches [Kuhn et al., 2023, Duan et al., 2024]. We follow this direction and propose a way of quantifying risk as a combination of these basic measures. This results in a family of efficient and robust UQ techniques. Our approach, illustrated in Figure 2, combines the strengths of both information-based and consistency-based methods, providing a more comprehensive and accurate assessment of uncertainty.

Recently, it has been argued [Wang and Holmes, 2025, Daheim et al., 2025] that UQ in NLP tasks can be viewed through the lens of the Minimum Bayes Risk (MBR) framework. Following this, we formulate our methods as particular forms of risk functions under the MBR approach.

Our main contributions can be summarized as follows:

We propose a new way of quantifying risk that combines information-theoretic and consistency-based measures, and derive a family of Confidence and Consistency-based Approaches ( CoCoA ) to uncertainty quantification in LLMs 2 .

We show that the consistency component can be approximated by a learned function trained on unlabeled held-out set with a negligible loss of performance, eliminating the need for costly repeated sampling from the LLM. We call this variation of our method CoCoA Light.

We evaluate our approaches across a variety of NLP tasks, including question answering, summarization, and machine translation. Our experiments demonstrate sizable improvements in the reliability and the robustness of UQ compared to state-of-the-art methods.

2 Background

2.1 Language Model Decoding

LLMs define a probabilistic output distribution p ( y | x ) for a given input sequence x . The standard ways to obtain a particular output y ∗ given p ( y | x ) include various variants of sampling y ∗ ∼ p ( y | x ) and greedy decoding, where

An alternative approach is to use Minimum Bayes Risk (MBR) decoding [Kumar and Byrne, 2004a]:

where Y is the set of candidate sequences and R ( y | x ) is a risk function:

with r ( y , y ′ ) being a pairwise loss function.

The standard choice in the MBR literature is to take r ( y , y ′ ) ∝ -s ( y , y ′ ) , where the so-called utility function s ( y , y ′ ) represents some notion of similarity between generated sequences y and y ′ . Below we discuss how various decoding strategies and corresponding risk functions lead to well-grounded definitions of uncertainty.

2.2 From Risk to Uncertainty

Generally, an uncertainty function U is a mapping that quantifies the level of uncertainty associated with the output of a model y , conditioned on the input sequence x , which we denote as U ( y | x ) .

Bayesian uncertainty measures are well known to be strongly connected with minimum risks of various kinds [Xu and Raginsky, 2022, Kotelevskii et al., 2025]. Recently, this connection was revisited in the context of natural language generation [Wang and Holmes, 2025, Daheim et al., 2025], leading to the development of new MBR-based UQ methods. More precisely, one can consider the smallest achievable risk within Y or, equivalently, to maximum expected utility:

where y ∗ is given by (2). Interestingly, MBR-based uncertainty (4) is applicable even to Y consisting of a single sequence y ∗ .

Below, we discuss how various existing uncertainty measures can be seen as particular approximations of minimum Bayes risks for various choices of utility functions. We also derive new uncertainty measures based on the MBR framework.

̸

Single Sequence Information-Based Methods. Information-based methods estimate the uncertainty of the generated sequence by aggregating the uncertainty scores of individual tokens. Within the framework of MBR decoding, consider r ( y , y ′ ) = 1 { y ′ = y } . In this case, the Bayes risk corresponds to the expected zero-one loss of decoding:

̸

This leads to one of the simplest information-based uncertainty measures, Sequence Probability (SP) :

Although widely used as an uncertainty measure across various applications, including natural language generation, its connection to statistical risk has only recently been explored in the UQ literature [Kotelevskii et al., 2022, 2025, Aichberger et al., 2024].

Several other measures fall into this category, including Perplexity and Mean Token Entropy [Fomicheva et al., 2020]; see Appendix D.1 for details. While using only a single sample makes them computationally efficient, these techniques face three major challenges:

LLMs only give the probability of a specific answer, even though the same meaning can often be conveyed in multiple ways. To obtain a proper probability for the meaning of an answer, we need to marginalize over its various possible rephrasings. However, this is not feasible if we generate only a single sample.

The reliability of such methods depends heavily on the underlying LLM being wellcalibrated, which is a quality that is hard to define and harder to ensure.

These methods are point estimates that do not provide information about the shape of the output distribution.

Semantic Consistency-Based Methods. The aforementioned issues lead to the development of consistency-based methods based on repetitive sampling from the LLM. Consider that we have sampled a set of outputs { y ( i ) } M i =1 , where y ( i ) ∼ p ( y | x ) . Consistency-based UQ methods rely on the diversity of the answers y ( i ) sampled from the LLM. The idea is that if the model outputs similar answers for the same prompt over and over again, it is confident in its predictions; otherwise, it is uncertain. These techniques do not require a probability distribution estimated by an LLM and can be applied in a black-box setting, where only the generated tokens are available. This case is quite common when LLMs are deployed as a service and are accessible through a limited API.

Formally, consistency-based methods start from defining some similarity function s ( y , y ′ ) ∈ [0 , 1] between arbitrary LLM generations y and y ′ . The value s ( y , y ′ ) = 1 indicates the complete equivalence between y and y ′ , and s ( y , y ′ ) = 0 indicates that there is no similarity. The similarity could be computed in various ways. For instance, the Lexical Similarity method [Fomicheva et al., 2020] relies on surface-form similarity, measuring the degree of word-level or phrase-level overlap between the generated texts. More advanced techniques propose various methods for taking into account the semantic similarity between the generated answers by means of hard or soft clustering [Lin et al., 2024].

Finally, given M samples from the model, standard consistency-based methods compute a similarity matrix S , where s ij = s ( y ( i ) , y ( j ) ) . Then, various statistics of S are computed in order to estimate the uncertainty [Lin et al., 2024]; see Appendix D.2 for more detail. Alternatively, one can focus on a particular output y ∗ and compute uncertainty measures based on its similarity s ∗ i = s ( y ∗ , y ( i ) ) with other samples y ( i ) ; see [Lin et al., 2024] for more detail.

Interestingly, it was recently shown [Wang and Holmes, 2025, Daheim et al., 2025] that the latter approach has direct relation to MBR decoding. Building on the MBR framework, one can define the utility function as r ( y , y ′ ) = 1 -s ( y , y ′ ) and derive the following consistency-based uncertainty score:

Then the Monte Carlo approximation of the minimum Bayes risk is given by

In our ablation study below, we show that such an uncertainty measure reliably outperforms consistency-based measures that aggregate the pairwise similarities of all samples (see Appendix C.2).

A key strength of consistency-based techniques is that, by generating multiple samples and analyzing their semantic similarity, they can estimate empirical probabilities over meanings rather than over individual answers. Their main drawback is that they discard the useful information that comes from the probability distribution represented by the LLM, including estimates of the probabilities of the specific answers.

3 CoCoA: Bridging Confidence and Consistency for Better Uncertainty Quantification

Both information-based and (semantic) consistency-based methods provide grounded and useful uncertainty quantification measures. There exist approaches that bridge the gap between informationbased and consistency-based methods that show great promise but lack the fundamental base; see discussion in Section 4. In what follows, we present a family of Confidence and Consistency-based Approaches ( CoCoA ) for UQ, offering a new way to merge information- and consistency-based measures for uncertainty quantification in LLMs via the unifying MBR-based framework.

If we only consider the consistency measure, e.g., as given in (8), we miss the information that is contained in the model confidence. Thus, we want the resulting uncertainty measure to explicitly consider both the semantic consistency and the model-based uncertainty. Let us consider a risk of the following form:

where

s ( y , y ′ ) ∈ [0 , 1] is any utility function used in the standard MBR decoding (usually representing semantic similarity);

u ( y | x ) ≥ 0 is the model-based uncertainty measure for the output sequence y . For example, we can use u ( y | x ) = 1 -p ( y | x ) (similarly to (6)) or u ( y | x ) = -log p ( y | x ) .

Then, the corresponding Bayes risk is R CoCoA ( y | x ) = u ( y | x ) · E y ′ ∼ p ( y | x ) ( 1 -s ( y , y ′ ) ) and for an output sequence y ∗ , the resulting uncertainty measure becomes

Finally, given a set of samples { y ( i ) } M i =1 we obtain an empirical uncertainty estimate:

where s ∗ i = s ( y ∗ , y ( i ) ) .

The resulting uncertainty measure integrates both global (semantic) and local (model-specific) uncertainty signals. It ensures that uncertainty is amplified for sequences y ∗ that are both intrinsically uncertain (high u ( y ∗ | x ) ) and semantically inconsistent with respect to the other samples (high 1 M ∑ M i =1 (1 -s ∗ i ) ), while keeping it low for the opposite scenario; see Figure 2 for an example.

CoCoA Light. Computing the consistency-based uncertainty measure ̂ U cons ( y ∗ | x ) within ̂ U CoCoA ( y ∗ | x ) requires multiple samples, whoch poses significant computational overhead. To address this, we propose to approximate the behavior of ̂ U cons ( x ) with a learned function. Our method closely matches the original sampling-based measure in quality while requiring only greedy output generation during inference, eliminating the need for additional sampling. Notably, this approximation can be learned without access to ground-truth labels, relying solely on the input data and the associated uncertainty values.

Thus, the CoCoA uncertainty measure that incorporates an approximation of the consistency uncertainty ̂ U L cons ( y ∗ | x ) ≈ ̂ U cons ( y ∗ | x ) can be defined as follows:

We name this approach CoCoA Light . Below, we describe how this approximation can be obtained.

Assume access to a held-out set of input sequences x j , j = 1 , . . . , n . We emphasize that this held-out set is not labeled. For each sequence, we extract the embeddings { e ( y j | x j ) } n j =1 of the corresponding greedy model generations y j . The corresponding targets are the consistency uncertainty scores ̂ U cons ( y j | x j ) , j = 1 , . . . , n , computed via Monte Carlo estimation of the minimum Bayes risk; see equation (8).

A lightweight auxiliary model, ̂ g ( e ( y ∗ | x ) ) , is trained in a supervised fashion to map model embeddings to uncertainty scores. During inference, the model generates a greedy output y test for a test input x test, from which the embeddings e ( y test | x test ) are obtained and passed to the auxiliary model to predict the uncertainty score:

We note that a similar approach [Kossen et al., 2024] has been previously proposed to approximate semantic entropy using the hidden states from the model.

4 Related Work

In this section, we review existing approaches to uncertainty quantification, as well as prior work on Minimum Bayes Risk in application to natural language generation.

Information-based methods. These methods are one of the most commonly used. They quantify uncertainty by analyzing the probability distributions of the tokens within a given output. The simplest of these methods, Sequence Probability (SP) , measures the probability of the sequence given a specific input. Perplexity is another common uncertainty measure: it gauges how well the model predicts each token in a sequence and is formally defined as the exponential of the mean negative log-likelihood of those tokens [Fomicheva et al., 2020]. The Mean Token Entropy method evaluates the token-level predictions across the entire sequence by computing the average entropy of the token probability distributions at each position [Fomicheva et al., 2020]. While these methods provide useful uncertainty estimates, they do not address the broader uncertainty inherent in the generative tasks, as they rely on a single sequence and fail to capture the diversity of the possible outputs that the model could generate for the same input.

Consistency-Based Methods. These methods estimate uncertainty by generating multiple sequences from the model's output distribution and analyzing the variability among the sampled outputs. These methods are particularly relevant to NLP tasks, as they account for scenarios in which multiple plausible outputs may exist for a given input. Among the simplest consistency-based methods are the Number of Semantic Sets and the Sum of Eigenvalues of the Graph Laplacian [Lin et al., 2024], which quantify uncertainty by how many distinct 'meanings' the model produces. While effective at capturing global variation, they do not yield uncertainty scores for individual responses. To overcome this, the diagonal of the Degree Matrix was proposed to assess how similar each output is to the rest, enabling per-response uncertainty quantification [Lin et al., 2024]. A related approach, Lexical Similarity , operates by calculating the average similarity of words or phrases across every pair of responses in the sample [Fomicheva et al., 2020].

Information-Based Methods with Repeated Sampling. More recent methods attempt to reconcile output consistency with information-based signals-i.e., they not only look at how varied outputs are, but also how probable each output is under the model's own generation process. For instance, Semantic Entropy [Kuhn et al., 2023] groups outputs into semantically homogeneous clusters-capturing distinct 'meanings' that might just differ in phrasing-and calculates entropy across these clusters. SentenceSAR refines this idea by defining 'relevance' as the sum of pairwise similarities between a given sentence and others weighted by each sentence's model-assigned generation probability [Duan et al., 2024]. SAR combines the sentence-level relevance of SentenceSAR with token-level probability adjustments, attempting a more fine-grained balance of semantic and token-level uncertainty [Duan et al., 2024]. Semantic Density [Qiu and Miikkulainen, 2024] evaluates uncertainty by evaluating how densely a model's generated response is situated within the semantic space of all possible outputs, with lower density indicating higher uncertainty. One limitation of sampling-based approaches is their computational cost: generating multiple outputs from the model can be expensive. Recent work has sought to address this by learning to predict semantic uncertainty directly from hidden states, allowing for the approximation of semantic entropy without repeated sampling [Kossen et al., 2024]. Finally, BSDetector [Chen and Mueller, 2024] proposes an additive composition of self-reported confidence with observed consistency, however, it relies on ability of the model to assess its own confidence in text form. Another limitation is that it requires selection of a trade-off coefficient between confidence and consistency parts.

While these probability-weighted measures can provide deeper insights, they also sometimes underperform in practice [Vashurin et al., 2025], especially when the model's probability estimates are unreliable, or when important nuances of semantic diversity are lost in the weighting process. Proper balancing raw output consistency with generation likelihood remains an open problem.

Minimum Bayes Risk for LLMs. Minimum Bayes Risk (MBR) decoding, originally used in machine translation [Kumar and Byrne, 2004b], has recently been applied to LLMs by incorporating a posterior over model parameters to enable uncertainty-aware generation [Daheim et al., 2025]. Complementing this, a Bayesian framework for subjective uncertainty quantification was developed yielding improved calibration in generation tasks [Wang and Holmes, 2025].

5 Experiments

In this section, we present the experimental setup, the results, and the ablations.

5.1 Experimental Setup

To evaluate the effectiveness of our proposed framework, we extended the LM-Polygraph library [Vashurin et al., 2025, Fadeeva et al., 2023]. Since it already includes tools for calculating other uncertainty scores, it provided a convenient and efficient environment for setting up and running experiments. The primary objective of our experiments is to evaluate whether our method offers improved performance in key tasks such as question answering (QA), text summarization (SUM), and machine translation (MT), compared to existing baselines.

Datasets. For QA, we selected diverse datasets to capture a variety of challenges: TriviaQA [Joshi et al., 2017], an open-domain factual QA dataset; CoQA [Reddy et al., 2019], a conversational QA benchmark requiring multi-turn contextual understanding; MMLU [Hendrycks et al., 2021], a multitask dataset spanning 57 topics to test broad knowledge; and GSM8k [Cobbe et al., 2021], which focuses on grade-school math problems requiring logical reasoning. For translation, we evaluated our method on WMT14 French-English [Bojar et al., 2014] and WMT19 German-English [Barrault et al., 2019]. Finally, for summarization, we used XSUM [Narayan et al., 2018], a dataset of complex documents paired with concise abstractive summaries. For all datasets, we follow [Vashurin et al., 2025] in subset selection, prompt formatting, and few-shot example sourcing.

Models. We evaluated our method on the base versions of three open-weight language models: LLaMA 3.1 8B [Touvron et al., 2023], Mistral 7B [Jiang et al., 2023], and Falcon 3 7B [Team, 2024]. The open-weight nature of these models enables direct access to token probabilities, which is crucial for implementing our UQ method. All experiments were conducted using the base (non-instruction-tuned) variants. We additionally report results on the larger Gemma 3 12B-Base model [Team et al., 2025], with detailed analysis provided in Appendix J.

Similarity Function. To measure the similarity between two generations, we use the RoBERTa-large cross-encoder model, finetuned on the Semantic Textual Similarity benchmark dataset [Liu et al., 2019, Reimers and Gurevych, 2019, Cer et al., 2017]. This model is widely regarded as one of the most reliable

Oracle

UE

Random

PRR_norm =

0

Rejectionrate

1

and commonly used approaches for evaluating sentence similarity. The cross-encoder processes two sequences jointly and directly outputs a similarity score ranging from 0 to 1, providing a nuanced measure. Appendix C.1 contains comparative experiments with cross-encoder and other choices of the similarity function, substantiating this choice.

Baselines. We compare the performance of the proposed method against a diverse set of baselines and state-of-the-art UQ scores, including confidence-based, consistency-based, and hybrid approaches. For information-based approaches, we evaluate Sequence Probability (SP), Perplexity (PPL), Mean Token Entropy (MTE), Monte Carlo Sequence Entropy (MCSE), and Monte Carlo Normalized Sequence Entropy (MCNSE). In the consistency-based category, we consider the Degree Matrix (DegMat) and the Sum of Eigenvalues of the Graph Laplacian (EigValLaplacian). Finally, we include hybrid methods Semantic Entropy and SAR, as well as verbalized confidence method P(true) [Kadavath et al., 2022]. All formulations for these baselines can be found in Appendix D. Finally, we evaluate the performance of ̂ U cons (Consistency) and ̂ U L cons (Consistency Light) as an uncertainty measure.

Evaluation Measure. As our evaluation measure, we chose the Prediction Rejection Ratio (PRR) , which measures the effectiveness of the uncertainty scores for identifying high-quality predictions [Malinin and Gales, 2021]. PRR operates by progressively rejecting predictions with uncertainty scores above a threshold a and observing how the average quality of the remaining predictions changes (see Figure 3). It is calculated as the ratio of two areas: the area between the Prediction Rejection (PR) curves for the evaluated uncertainty score and a random baseline, and the area between the oracle (the ideal uncertainty score that perfectly ranks instances by quality) and the random baseline. Formally, PRR is defined as follows:

Higher PRR values indicate better alignment of uncertainty scores with prediction quality, approaching the performance of an oracle. To ensure practical applicability, we compute PRR only up to a rejection threshold of 50%, preventing cases where excessive rejection artificially inflates the quality measures.

For the QA datasets, we further report AUROC (see Appendix G).

Quality Measures. PRR requires an appropriate quality measure for each specific task to effectively evaluate the model output. For question-answering tasks, we use Accuracy to directly evaluate whether the generated answers match the ground truth in short-form QA tasks (e.g., MMLU), and we use the AlignScore between the correct answer and generated sequence for assessing the performance for long-form QA tasks [Zha et al., 2023]. For summarization tasks, we use AlignScore to measure the alignment between the output summary and the input document. It serves as a quality indicator by evaluating the relevance and the overlap between the generated content and the source text. For translation tasks, we use COMET , as it captures both semantic adequacy and fluency, ensuring that translations are accurate and linguistically appropriate [Rei et al., 2020].

To further improve the comprehensiveness of the reported results, we considered alternative choices of quality measures, and report the corresponding PRR scores in Appendix F.

Generation Setup. We discuss the generation parameters, the decoding strategy, and the sample selection procedure in depth in Appendix A. In short, we report the evaluation results in two distinct setups: greedy decoding and stochastic sampling with focus on the most probable sequence among the generated outputs ( best-sample ). These two setups offer the highest-quality outputs and are the most reasonable generation approaches in practice.

CoCoA Light Details. We use a multilayer perceptron network as an auxiliary model. The features used are embeddings from the middle layers of the base LLM, which are the most informative according to recent work [Chen et al., 2024]. In particular, for Llama 3.1 8B and Mistral 7B models, we take embeddings from the 16th layer, and for Falcon 3 7B from the 14th layer. More details on the training are available in Appendix I.

5.2 Results

Table 1 shows the PRR scores under the greedy generation setup (See Appendix B for Best Sample and MBR Sample). We report aggregated PRR for each type of task - question answering, neural machine translation (NMT), and summarization (SUM) - by averaging the results across all relevant datasets (e.g., TriviaQA, MMLU, CoQA, GSM8k for QA). This aggregated score provides a concise measure of the performance for each model for each task. Detailed results for each dataset separately can be found in Appendix E.

We can see that our CoCoA methods are the best across all tasks and models. They outperform existing consistency-based and hybrid state-of-the-art approaches, like Semantic Entropy and SAR. In addition, the proposed CoCoA approach consistently surpasses the baseline UQ measures: for example, CoCoA PPL outperforms standard Perplexity, illustrating the advantage of combining tokenlevel confidence with semantic consistency. This pattern holds for other information-based metrics as well, demonstrating that using the consistency between multiple sampled outputs reliably enhances uncertainty quantification.

5.3 Ablations

Similarity Function. We investigate the impact of different similarity measures (see Appendix C.1). On average, the cross-encoder provides strong performance. However, for summarization tasks, AlignScore yields better results, while in long-form QA, similarity based on Natural Language Inference (NLI) sometimes outperforms the cross-encoder. We hypothesize that for generation with shorter outputs (1-2 sentences), any capable NLI or cross-encoder model is sufficient. For longer generations, approaches that estimate similarity over chunks and then aggregate scores (e.g., AlignScore) may be more effective. Nevertheless, across all tasks, the cross-encoder achieves the best overall performance and, in scenarios where tuning is not feasible, serves as a strong default choice.

Alternative Formulations. The next section of our ablation study focuses on alternative forms of combining model confidence u ( y ∗ | x ) and consistency ̂ U cons ( y ∗ | x ) ; see Appendix C.2. First, we consider an additive form of combining them:

The results show that this additive formulation does not perform as well as the multiplicative one. The additive form tends to underemphasize the interaction between the two components, which is critical for capturing the nuanced relationships between confidence and consistency.

We also consider an alternative formulation of the consistency term ̂ U cons ( y ∗ | x ) , as the average of the full pairwise dissimilarity. In this formulation, ̂ U cons ( y ∗ | x ) represents the average inconsistency across all samples rather than focusing solely on the dissimilarity of the evaluated sequence with the other samples. Our experiments demonstrate that this formulation is not very strong. By distributing the consistency computation across all samples, it loses focus on the specific sequence being evaluated.

Lastly, in Appendix C.2, we also consider alternative formulations of the information-based metric that do not rely on logarithmic transformations. While we primarily use logarithms due to their numerical stability, we explore an alternative approach by converting these values back to probabilities and analyzing their impact on uncertainty quantification. Our findings indicate that both formulations exhibit consistent performance and yield similar results. This suggests that while logarithmic transformations enhance numerical stability, the choice between log-based and probability-based formulations does not affect much the overall performance.

6 Limitations

While our proposed CoCoA approach demonstrates robust empirical performance, several important considerations remain.

Task and Domain Dependency. CoCoA relies on an information-based confidence score and a semantic similarity function, whose effectiveness can vary across models, tasks, and domains. In open-ended tasks like creative generation, producing diverse outputs is expected; on the other hand, tasks requiring precise reasoning can be sensitive to subtle errors that generic similarity metrics may miss. Adapting these components to specific domains remains an important direction for future work.

Limited Sample Size. CoCoA estimates consistency by sampling multiple outputs. Generating many samples is computationally expensive, and thus our experiments, e.g., most sampling-based methods, use relatively small sets. While even a few samples can yield meaningful estimates, they may miss the full diversity of model outputs for complex prompts.

Quality Metric. Finally, the CoCoA 's performance assessment depends on quality metrics (e.g., COMET for machine translation, and Accuracy for QA) that may not capture every nuance of textual outputs. Further refining or extending quality metrics to account for deeper reasoning, factual faithfulness, and stylistic appropriateness would better align uncertainty scores with real-world perceptions of model correctness.

7 Conclusion and Future Work

We presented CoCoA , a unified approach that integrates Co nfidence and Co nsistency for uncertainty quantification in LLMs. By combining confidence scores with semantic similarity between multiple sampled outputs, CoCoA offers a more holistic view of uncertainty than either approach alone. In extensive evaluations on question answering, summarization, and translation, our approach outperformed existing baselines and state-of-the-art UQ methods. Moreover, CoCoA 's flexible design allows easy adaptation to a variety of tasks and settings.

Moving forward, several directions are open for further exploration. These include incorporating more adaptive sampling strategies that efficiently capture the model output space, refining the semantic similarity functions for domain-specific tasks, like code generation or commonsense reasoning.

References

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey. ACM Trans. Inf. Syst., 2025. ISSN 1046-8188. doi: 10.1145/3748304. URL https://doi.org/10.1145/3748304 .

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026/ .

Tom Kocmi and Christian Federmann. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193-203, Tampere, Finland, 2023. European Association for Machine Translation. URL https://aclanthology.org/2023.eamt-1.19/ .

Andrey Malinin and Mark J. F. Gales. Uncertainty estimation in autoregressive structured prediction. In Proceedings of the Ninth International Conference on Learning Representations, Vienna, Austria, 2021. URL https://openreview.net/forum?id=jN5y-zb5Q7m .

Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539-555, 2020. doi: 10.1162/tacl_a_00330. URL https://aclanthology.org/2020.tacl-1.35 .

Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. Transactions on Machine Learning Research, 2024. URL https://openreview.net/pdf?id=DWkJCSxKU5 .

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In Proceedings of the Eleventh International Conference on Learning Representations, Kigali, Rwanda, 2023. OpenReview.net. URL https: //openreview.net/pdf?id=VD-AYtP0dve .

Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5050-5063, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.276. URL https://aclanthology.org/2024.acl-long.276 .

Ziyu Wang and Christopher C Holmes. On subjective uncertainty quantification and calibration in natural language generation. In Proceedings of the 28th International Conference on Artificial Intelligence and Statistics, Mai Khao, Thailand, 2025. URL https://openreview.net/forum? id=D4HqXujpKA .

Nico Daheim, Clara Meister, Thomas Möllenhoff, and Iryna Gurevych. Uncertainty-aware decoding with minimum Bayes risk. In Proceedings of the Thirteenth International Conference on Learning Representations, Singapore, 2025. URL https://openreview.net/forum?id=hPpyUv1XyQ .

Shankar Kumar and William Byrne. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169176, Boston, Massachusetts, USA, 2004a. Association for Computational Linguistics. URL https://aclanthology.org/N04-1022/ .

Aolin Xu and Maxim Raginsky. Minimum excess risk in Bayesian learning. IEEE Transactions on Information Theory, 68(12):7935-7955, 2022. doi: 10.1109/TIT.2022.3176056.

Nikita Kotelevskii, Vladimir Kondratyev, Martin Takáˇ c, Eric Moulines, and Maxim Panov. From risk to uncertainty: Generating predictive uncertainty measures via Bayesian estimation. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025.

Nikita Kotelevskii, Aleksandr Artemenkov, Kirill Fedyanin, Fedor Noskov, Alexander Fishkov, Artem Shelmanov, Artem Vazhentsev, Aleksandr Petiushko, and Maxim Panov. Nonparametric uncertainty quantification for single deterministic neural network. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 36308-36323. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ eb7389b039655fc5c53b11d4a6fa11bc-Paper-Conference.pdf .

Lukas Aichberger, Kajetan Schweighofer, and Sepp Hochreiter. Rethinking uncertainty estimation in natural language generation. arXiv preprint arXiv:2412.15176, 2024.

Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in LLMs. arXiv preprint arXiv:2406.15927, 2024.

Xin Qiu and Risto Miikkulainen. Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space. In Proceedings of the Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.

Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5186-5200, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.283. URL https://aclanthology. org/2024.acl-long.283/ .

Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. Benchmarking uncertainty quantification methods for large language models with LM-Polygraph. Transactions of the Association for Computational Linguistics, 13:220-248, 2025.

Shankar Kumar and William Byrne. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169-176, Boston, Massachusetts, USA, May 2 - May 7 2004b. Association for Computational Linguistics. URL https://aclanthology.org/N04-1022/ .

Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. LM-Polygraph: Uncertainty estimation for language models. In Yansong Feng and Els Lefever, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 446-461, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-demo.41. URL https://aclanthology.org/2023.emnlp-demo.41/ .

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology.org/ P17-1147 .

Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019. doi: 10.1162/tacl_a_00266. URL https://aclanthology.org/Q19-1016 .

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceedings of the 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ .

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168 .

Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. Findings of the 2014 Workshop on Statistical Machine Translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. URL http://www. aclweb.org/anthology/W/W14/W14-3302 .

Loïc Barrault, Ondˇ rej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference on machine translation (WMT19). In Ondˇ rej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor, editors, Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-5301. URL https://aclanthology.org/ W19-5301/ .

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference

on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/d18-1206. URL https: //doi.org/10.18653/v1/d18-1206 .

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. URL https://doi.org/10. 48550/arXiv.2307.09288 .

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. URL https://doi.org/10.48550/arXiv. 2310.06825 .

Falcon-LLM Team. The Falcon 3 family of open models, December 2024. URL https: //huggingface.co/blog/falcon3 .

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, JanThorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna KlimczakPluci´ nska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786 .

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019. URL https://arxiv.org/abs/1907.11692 .

Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERTnetworks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the

2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/ D19-1410. URL https://aclanthology.org/D19-1410/ .

Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens, editors, Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/S17-2001. URL https://aclanthology.org/S17-2001/ .

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. arXiv Prerpint, arXiv:2207.05221, 2022. doi: 10.48550/arXiv.2207.05221. URL https://doi.org/10.48550/arXiv.2207.05221 .

Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. Alignscore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328-11348, 2023. URL https: //aclanthology.org/2023.acl-long.634 .

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685-2702, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. emnlp-main.213. URL https://aclanthology.org/2020.emnlp-main.213 .

Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs' internal states retain the power of hallucination detection. In Proceedinsg of the Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/ forum?id=Zj12nzlQbz .

Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013/ .

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with disentangled attention. In Proceedings of the 9th International Conference on Learning Representations. OpenReview.net, 2021. URL https://openreview.net/forum?id= XPZIaotutsD .

Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz, editors, Proceedings of the Ninth Conference on Machine Translation, pages 492-504, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.wmt-1.35 .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately describe the proposed family of Confidence and Consistency-based Approaches (CoCoA) to UQ. Theoretical contributions include new methods that merge information- and consistency-based measures for UQ in LLMs. Experimental validation of the new approaches is done across different NLP tasks: question answering, summarization, and translation. It aligns with the content presented in the paper (Section 3, 5.2, Appendix B).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 6 discusses limitations, including relying on an information-based confidence score and a semantic similarity function, having limited sample sets, and depending on a quality metric.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper presents theoretical formulations: uncertainty measures derived from Minimum Bayes Risk (MBR) and the definition of CoCoA and its approximated variant CoCoA Light (Sections 2.2 and 3). However, we do not make any formal theoretical statements in this work.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 5.1 outlines the experimental setup, datasets and evaluation metrics. The description of datasets used for each task includes TriviaQA, CoQA, MMLU, GSM8k for Question Answering, WMT14 French-English, WMT19 German-English for machine translation, and XSUM for summarization. The models used: LLaMA 3.1 8B, Mistral 7B and Falcon 3 7B. The similarity function is specified as RoBERTa-large fine-tuned on the Semantic Textual Similarity benchmark. The evaluation metric, Prediction Rejection Ratio (PRR), is clearly defined, and task-specific quality measures such as Accuracy, AlignScore, and COMET are used. Furthermore, the training setup for the CoCoA Light method, including the use of multilayer perceptron and middle layer embeddings of the base LLM, is described. The appendix provides additional details on ablation studies, similarity metrics, and decoding strategies (Appendix C and E).

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is open source and is provided as an extension to the LM-Polygraph library (Section 5.1).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental settings, including model architecture (base LLaMA 3.1 8B, Mistral 7B, Falcon 3 7B), embedding layer selection for auxiliary training (16th or 14th layer depending on model), and the auxiliary model structure (multilayer perceptron) are described in Section 5.1 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not provide errorbars as it operates in (almost) deterministic scenario: we operate with pre-trained models and there is no training involved whatsoever. The only randomization involved is the one related so sampling multiple sequences for consistency-based methods. The choice to not compute errorbars here is deliberate as our inference involves heavy computations and it is not possible to conduct main experiments multiple times due to that. However, we conducted experiments on 3 different LLMs and 7 datasets, which is equivalent to 21 independent run, demonstrating the robustness and reliability of our findings. For this reason, we do not report error bars, as the results are stable.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix H discusses computational resources used to produce experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research does not involve human subjects or ethically sensitive applications. We believe it conforms the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Section 1 discusses the importance of Uncertainty Quantification, highlighting its positive societal impacts (helps to reduce overreliance on automated systems and encourages a more responsible LLM usage). We do not foresee any negative negative societal impact of this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: N/A.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper properly cites the original sources of models (e.g., LLaMA 3.1 8B, Mistral 7B, Falcon 3 7B), datasets (CoQA, TriviaQA, MMLU, GSM8k, XSUM, WMT14FrEn, WMT19DeEn), and tools (e.g., RoBERTa, LM-Polygraph). However, it does not explicitly mention the specific licences and terms of use for these assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: The paper introduces a new family of uncertainty quantification methods called CoCoA (Confidence and Consistency-based Approaches) and an approximated variant, CoCoA Light. However, it does not release any new dataset, model, or a software tool alongside the paper, and does not provide accompanying documentation or artifacts.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The research does not involve crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The research does not involve crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: The paper centers its contributions on the use of Large Language Models (LLMs) as the core subject of study. It introduces new methods for uncertainty quantification in LLMs, specifically evaluating base models such as LLaMA 3.1 8B, Mistral 7B, and Falcon 3 7B. Apart from that LLMs were used solely to aid in editing the final manuscript and writing some of the visualization code.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Decoding Strategy and Sample Selection

Modern LLMs are capable of producing output using a wide range of decoding strategies, and it is not readily apparent which one to use as a foundation for UQ experiments. On top of that, when sampling multiple outputs stochastically, one has to decide which sample to select for comparison with the target sequence and UQ purposes.

To facilitate the choice of decoding and sample selection strategies for our experiments, we conducted an evaluation of model performance with different approaches to both. Table 2 shows average values of corresponding quality metrics for all combinations of models and datasets. We considered 4 approaches for the selection of output that subsequently is used to calculate the quality of generation:

Greedy decoding produces single output by selecting top-1 candidate token at each generation step, thus no further selection of sample is needed.

Random sample corresponds to the case where random output is selected among the number of samples produced by repeatedly prompting the model with the same question. In practice we use first generated sample, highlighting model performance when stochastic decoding is done only once.

Most Probable sample selects the output with highest model-assigned probability among several sampled outputs.

MBR sample selects the output with highest average consistency with respect to other sampled outputs.

We note that selecting a random sample from the model outputs incurs a significant drop in the quality of results on several datasets, most prominently on GSM8k. Based on these observations, we evaluate the efficacy of UQ in three setups: greedy decoding, stochastic sampling with a focus on the most probable sample and MBR decoding.

Regardless of the way the main model response was obtained, responses that were used to quantify consistency in (8) were generated via repeated stochastic sampling. In all the experiments, where stochastic sampling is involved, it was performed with temperature T = 1 . 0 , top-k equal to 50, and top-p equal to 1 . 0 .

B Results Summary for Most Probable Sample and MBR Sample strategies

Tables 3 and 4 report the PRR scores under the Most Probable Sample and MBR Sample generation setups, as discussed in Section 5.2. We observe that CoCoA-family methods, and their Supervised versions, consistently improve base uncertainty estimators.

C Ablation

C.1 Choice of Similarity Function

For sample consistency estimation, one could come up with a variety of similarity functions s ( y , y ′ ) . We perform a comparison of the effectiveness of CoCoA -family methods using several such functions. We consider the following functions:

AlignScore [Zha et al., 2023] with AlignScore-large model;

RougeL [Lin, 2004];

NLI [He et al., 2021] based on microsoft/deberta-large-mnli model;

CrossEncoder [Liu et al., 2019] based on cross-encoder/stsb-roberta-large model.

Tables 5, 6 and 7 report these results. There exists a considerable variation of relative effectiveness of proposed methods with various similarity function choices, depending on a task at hand. We opt to report all results in other sections with CrossEncoder-based similarity as it by itself provides a significant improvement over baselines, and for consistency and ease of comparison reasons. However, when applying these methods to a particular task, we encourage users to select appropriate underlying similarity function for best results.

C.2 Different Ways of Combining Confidence and Consistency

We justify the particular form of equation (11) by considering alternative ways to combine samplefocused confidence with consistency estimation. Results are presented in Tables 8, 9 and 10. In particular, we investigate the performance of the additive approach (AdditiveCoCoA):

and the same multiplicative combination, replacing sample-focused dissimilarity from (8) with the average of the full pairwise dissimilarity matrix (24):

In addition, we present results where we compute u ( y ∗ | x ) using probability-based scores instead of log-likelihood, see equation (6). We refer to this variant as U ProbCoCoA ( y ∗ | x ) . While the results are generally similar, the log-likelihood-based formulation offers greater numerical stability. Therefore, we adopt it as the default in the main paper.

It is evident that on average the multiplicative form proposed in equation (10) with both confidence and consistency terms focused on a single sample is the better performing variant.

D Detailed Description of Uncertainty Quantification Methods

In this section, we provide a detailed description of the uncertainty quantification methods used in this study.

D.1 Information-Based Methods

Information-based methods are commonly used to estimate uncertainty by analyzing the probability distributions of tokens within a given output. These methods examine different levels of model generation, such as the model's confidence in producing a specific sequence, its ability to predict individual tokens at each generation step, and the variability in the token-level predictions across the sequence.

Sequence Probability (SP) is one of the simplest and most direct methods for estimating uncertainty. It measures the probability of the most likely output sequence given a specific input. Thus, uncertainty is quantified by calculating the probability of the sequence with the highest likelihood, under the assumption that the model is most confident in this output. In its simplest form it is given by (6). An equivalent (in terms of ordering of predictions for different inputs) but more numerically stable formulation is logarithmic. It is defined as:

Perplexity (PPL) is another widely used metric for estimating uncertainty in language models [Fomicheva et al., 2020]. It measures the model's confidence by evaluating the average likelihood of generating the sequence tokens:

Mean Token Entropy takes a broader view of uncertainty by considering the token-level predictions across the entire sequence [Fomicheva et al., 2020]. Instead of evaluating the model's confidence in a single output or individual token predictions, Mean Token Entropy calculates the average entropy of the token probability distributions for each token in the sequence:

where H ( y l | y <l , x ) is an entropy of the token distribution p ( y l | y <l , x ) .

The TokenSAR method, introduced in [Duan et al., 2024], generalizes length-normalized log probability by computing a weighted average of the negative log probabilities of generated tokens, where weights are based on token relevance to the overall text. Using a similarity function s ( · , · ) and token relevance function R T ( y k , y , x ) = 1 -s ( x ∪ y , x ∪ y \ y k ) , the uncertainty estimate is calculated as:

where

This measure is central for computing SAR uncertainty measure.

D.2 Consistency-Based Methods

Consistency-based methods assess the uncertainty of a language model by evaluating the semantic consistency of its predictions across multiple outputs for the same prompt. The core idea is that semantically similar outputs indicate higher confidence, while diverse or conflicting outputs suggest greater uncertainty. Since language models can express the same meaning in different surface forms, these methods construct a semantic similarity matrix S = ( s ij ) , where each entry represents the degree of similarity between pairs of responses. By clustering responses into groups with equivalent meanings, these methods provide a semantic measure of the model's consistency.

The work [Lin et al., 2024] offers two similarity measures to evaluate the similarity of sequences. The first is the Jaccard similarity, which treats sequences as sets of words and calculates the proportion of shared words to the total number of unique words in both sequences: s ( y , y ′ ) = | y ∩ y ′ | / | y ∪ y ′ | .

Natural Language Inference (NLI) provides another method for computing similarity between sequences. We use the DeBERTa-large NLI model [He et al., 2021], following [Kuhn et al., 2023]. For each pair of sequences, an NLI model predicts two probabilities: p entail ( y , y ′ ) , indicating entailment, and p contra ( y , y ′ ) , indicating contradiction. Similarity is then defined as either s entail ( y , y ′ ) = p entail ( y , y ′ ) or s contra ( y , y ′ ) = 1 -p contra ( y , y ′ ) .

Among the simplest consistency-based approaches are the Number of Semantic Sets and the Sum of Eigenvalues of the Graph Laplacian [Lin et al., 2024]. Number of Semantic Sets estimates how many distinct 'meanings' the model produces by clustering its outputs with an NLI model. The number of semantic sets is initially equal to the total number of generated answers, M . Two sentences are grouped into the same cluster if the following conditions are satisfied: p entail ( y ( i ) , y ( j ) ) > p contra ( y ( i ) , y ( j ) ) and p entail ( y ( j ) , y ( i ) ) > p contra ( y ( j ) , y ( i ) ) . This computation is performed for all pairs of answers, and the final number of distinct clusters is denoted by U NumSemSets.

Sum of Eigenvalues of the Graph Laplacian examines global diversity: it constructs a similarity matrix among the sampled outputs and computes a continuous uncertainty score from the eigenvalues of the Laplacian of that similarity graph. The work [Lin et al., 2024] proposes computing an averaged similarity matrix as s ij = ( s ( y ( i ) , y ( j ) ) + s ( y ( j ) , y ( i ) )) / 2 . The Laplacian for the matrix S is defined as L = I -D -1 2 SD -1 2 , where D is a (diagonal) degree matrix with elements D ii = ∑ M j =1 s ij . Consequently, the following formula is derived:

Both Number of Semantic Sets and Sum of Eigenvalues of the Graph Laplacian effectively capture overall variation in generated text but cannot produce an individual uncertainty score for each output. To address this, the work [Lin et al., 2024] proposes to use the diagonal Degree Matrix D ( x ) which represents the total similarity of each answer with all others. The corrected trace of D ( x ) provides an average pairwise distance between answers, and uncertainty is computed as:

D.3 Information-Based Methods with Repeated Sampling

The natural idea is to somehow benefit from having multiple samples from the model while using important information contained in the output probabilities estimated by an LLM. Below, we examine several approaches that have sought to achieve this.

Averaging uncertainties.

We can compute the entropy on the sequence level E [ -log p ( y | x ) ] , where the expectation is taken over the sequences y randomly generated from the distribution p ( y | x ) . Unfortunately, while for token level, we have an exact way of computing the entropy, for the sequence level, we need to adhere to some approximations. In practice, we can use Monte-Carlo integration, i.e. generate several sequences y ( i ) , i = 1 , . . . , M via random sampling and compute Monte Carlo Sequence Entropy [Kuhn et al., 2023]:

We can replace p ( y ( i ) | x ) with its length-normalized version ¯ p ( y ( i ) | x ) leading to a more reliable uncertainty measure in some cases.

While simple averaging represents a natural way to aggregate uncertainties, it has certain issues related to the nature of LLMs. First of all, in the vast majority of applications, an LLM-based system should produce a single output y ∗ for an input query. When we consider U H S ( x ) or other similar measure, we essentially perform averaging of uncertainties of different sequences, thus somewhat assessing the

uncertainty related to the entire generative distribution p ( y | x ) for the input x , but not for a particular generated sequence y ∗ . This averaged uncertainty might not be adequate for this particular sequence and, remarkably, often performs worse than the uncertainty u ∗ = U ( y ∗ | x ) , which is related solely to the output y ∗ . Moreover, although intuitive, this naïve aggregation method assumes that all outputs contribute equally to the final uncertainty estimate, regardless of their semantic relationships. This can lead to inconsistencies when semantically equivalent outputs have varying uncertainty scores or when outputs with low similarity are treated as equally important.

Semantically weighted averaging. Semantic Entropy [Kuhn et al., 2023] addresses the issue of generated sequences with similar meanings but differing probabilities according to the model, which can heavily influence the resulting entropy value (25). The method clusters generated sequences y ( i ) , i = 1 , . . . , M into semantically homogeneous groups C k , k = 1 , . . . , K (where K ≤ M ) using a bi-directional entailment algorithm. Probabilities of sequences are averaged within each cluster. The entropy estimate is then defined as:

where ˆ p k ( x ) = ∑ y ∈C k p ( y | x ) represents the aggregated probability for cluster C k .

SentenceSAR [Duan et al., 2024] enhances the probability of sentences that are more relevant. It uses a sentence relevance measure s ( y ( j ) , y ( k ) ) to evaluate the relevance of y ( j ) with respect to y ( k ) . SentenceSAR is calculated as:

where t is a temperature parameter used to control the scale of shifting to relevance, and

̸

The combination of SentenceSAR and TokenSAR results in a unified method called SAR [Duan et al., 2024]. In this approach, the generative probability p ( y | x ) in the SentenceSAR formula is replaced with the token-shifted probability p ′ ( y | x ) = exp { -TokenSAR( y , x ) } , creating a comprehensive measure that integrates both sentence- and token-level adjustments.

The aggregation approaches like Semantic Entropy [Kuhn et al., 2023] or SAR [Duan et al., 2024] can be unified into a semantically-aware Generalized Monte Carlo uncertainty estimate, defined as

Here, the inner summation aggregates sequence probabilities p ( y ( j ) | x ) weighted by their semantic similarity to the i -th output, and the outer summation averages these contributions across all samples. The function h ( · ) provides an additional layer of flexibility, transforming the reweighted uncertainty scores, making the method a generalized framework for uncertainty quantification.

Unfortunately, methods that fall under GMCU, while offering benefits, also inherit the aforementioned issues from both categories of methods. In particular, the outer summation in (29), similarly to the case of simple Monte Carlo averaging, often fails to outperform the uncertainty U ∗ ( x ) = U ( y ∗ | x ) of a single generated sequence y ∗ .

D.4 Verbalized Uncertainty

Verbalized uncertainty methods refer to approaches that prompt a model to explicitly express its confidence. In our experiment, we utilize the P(True) method, implemented following the description by [Kadavath et al., 2022]. Specifically, the model is presented with the original question and its answer, and then prompted to indicate whether the answer is True or False. We use the negative log-probability of the token 'True' as the uncertainty score.

E Detailed Experimental Results

Tables 3 and 1 presented PRR scores averaged over datasets corresponding to each task. Here we present expanded results for each dataset, see Tables 11, 12 and 13.

F Alternative Performance Metrics for PRR

The choice of PRR as a UQ quality metric of choice is dictated by its ability to handle both continuous performance metrics, like Comet and AlignScore without the need for selecting arbitrary thresholds, as well as relative robustness to class imbalance in case of binary performance metric. However, PRR scores are calculated for a particular choice of underlying performance metric. For a truly comprehensive evaluation, we perform the same evaluation as in our main experimental run, but with alternative choice of performance metrics.

Tables 14 and 15 report PRRs for these metrics on all models for NMT and QA tasks. MetricX [Juraska et al., 2024] was used for NMT and GPT-as-a-judge ( gpt-4o-2024-08-06 ) was used to score QA datasets. The following is the prompt used to facilitate GPT QA scoring:

G AUROC

We strongly believe that due to the considerations presented in Appendix F, PRR is a superior metric to AUROC when comparing relative performance of UQ methods. However, we acknowledge that AUROC is widely used in modern UQ literature, so we opt to include AUROC results on QA datasets. The results are presented in Tables 16 and 17. All results here were obtained with GPT-as-a-judge correctness scoring.

H Computational Budget

Total available computational resources used to produce results in this paper amounted to 12 compute nodes each having 4xA100 40Gb GPUs. Total computational budget spent to produce results was around 400 GPU-days. Each individual combination of model and dataset amounted roughly to 16 GPU-days on average to obtain all model outputs and hidden states needed for computing the results.

I CoCoA Light Training Details

The model architecture consisted of a simple multilayer perceptron (MLP) with a single hidden layer:

Input dimension: equal to the embedding size of the corresponding base model (4096 for Mistral7b-Base and Llama8b-Base, 3072 for Falcon7b-Base).

Hidden dimension: 2048.

Output dimension: 4096.

Dropout:

0.1 applied between layers.

We trained the MLP on mean pooled hidden-layer embeddings extracted from the middle layer of an LLM. Table 18 details the size of a train set for each model and dataset.

Training was conducted for 20 epochs with the following hyperparameters:

Batch size: 4 (per device) with gradient accumulation of 7 steps (effective batch size of 28).

Learning rate: 1 × 10 - 5

Weight decay: 0.1.

Warmup ratio: 0.05.

Gradient clipping: 1.0.

J Larger Model Experiments

We additionally evaluate our method using the Gemma 3 12B-Base model to assess its performance on a larger-scale architecture. As shown in Table 19, CoCoA continues to achieve the best average performance among all methods.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoCoA (Confidence and Consistency-based Approaches), a new framing of methods for uncertainty quantification (UQ) in large language models (LLMs), grounded in the Minimum Bayes Risk (MBR) framework. The goal is to unify information-based (model confidence) and consistency-based (semantic agreement between sampled outputs) UQ methods into a principled decision-making risk-based formulation. The authors also propose CoCoA Light, a learned approximation of the full CoCoA score that avoids repeated sampling using an additional held out set. They evaluate their methods on several datasets and compared with a few baseline methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Principled unification: The use of MBR as a unifying lens for both confidence and consistency is theoretically grounded (although reading line 47-49 and reference [9, 10] does not offer me the same hint, I wonder if the authors could elaborate for me). CoCoA makes this combination interpretable as a multiplicative interaction of model uncertainty and semantic inconsistency.\n\n2. Practical algorithm: CoCoA Light reduces the sampling cost by learning to approximate the consistency-based term from representations, making it more applicable for real-world applications.\n\n3. Empirical performance: Across diverse tasks and multiple open-source LLMs, CoCoA and CoCoA Light outperform good baselines.\n\n4. In general well-written and easy to follow.\n\n**Weaknesses**\n\n1. Limited novelty compared to prior works: While the MBR framing is clean, many recent UQ methods already combine confidence and consistency (e.g., SAR, Semantic Entropy). The main novelty lies in formalizing their combination via MBR. I would like to ask the authors to explain a bit more their motivation from [9] and [10], and what is the novelty of this formulation.\n\n2. Lack of theoretical justification: The main section for the CoCoA seems to build up a combination of model confidence and semantic inconsistency directly, without explaining the intuition and theoretical justification.\n\n3. Similarity function bottleneck: Results depend heavily on the semantic similarity function used. The paper acknowledges this but could further explore when and which score should be used for certain tasks and why (which is really helpful if we do not have an additional held out set, e.g. open-ended generation tasks). \n\n4. Measurement using PRR: While CoCoA improves PRR, it's less clear how well it calibrates model uncertainty. My understanding is this metric measures the ranking but not model uncertainty directly.\n\n5. Error bars missing: I am wondering why the variance is not reported in this paper.\n\nQUESTIONS:\nPlease refers to my comments in the weakness section.\n\nLIMITATIONS:\nThe authors mentioned the limitations but not much on the societal impact. I will encourage the authors to discuss this aspect in their rebuttal.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoCoA, which is a framework that generalizes several prior works efforts in direction of semantic clustering from a standpoint of uncertainty estimation as risk prediction.\nThe decomposition consists of Consistency and Confidence scores.\nThe former corresponds to the plurality of possible predictions, while the latter corresponds to confidence of the model in the 'main predicted' sequence. \nThe authors further propose a computationally cheaper version of the measure, which is achieved by learning an estimate for Consistency score.\nThe method shows competitive performance on a variety of tasks.\n\nSTRENGTHS AND WEAKNESSES:\n#### Strengths\n1. **Overall paper quality:** the paper is well structured and written. \n2. **Originality:** CoCoA generalizes several prior works that deal with semantic invariances in uncertainty estimation for NLP. It advances the connections between uncertainty and prediction risks in the NLP domain.\n3. **Evaluation:** the authors perform extensive evaluation on datasets and models standard to the subfield. \n\n#### Weaknesses\n1. **Introducing a Framework Rather than a Specifc Method**: as is acknowledged by the authors in the limitations, the different versions of CoCoA excel at different tasks.\nAn entire framework of possible options is therefore presented that can accommodate many prior methods and has multiple design choices. \nThe method empirically speaking is largely a combination of two families of prior work methods, each performing better in different scenarios.\nThis erodes the methods empirical utility. \n2. **Treatment of risk combinations**: even though the authors attempt to derive their measure from a theoretically grounded standpoint of prediction risk, the resulting CoCoA measures are still quite arbitrary.\nFor instance, the authors do not provide a theoretical explanation for multiplicative combination of the confidence and consistency and instead settle for picking one with greater empirical performance.\nIf the Confidence and Consistency parts both correspond to some sort of risk, I would imagine them being additive rather than multiplicative. \n3. **Evaluation Nuances**:\n    - the authors discuss the advantages of PRR over AUROC (even though its not the focus of the paper), yet acknowledge that they have to cut the PRR evaluation at 50% rejection to avoid artifacts.\n    - in appendix G table 16 shows a less confident picture of the CoCoA performance when using the AUROC+Judge instead of PRR + Rouge/AlignScore.\n\nQUESTIONS:\n1. Would introducing weighing for the two terms into the AdditiveCoCoA improve its performance for specific tasks?\n2. How would the evaluation picture change if the PRR rejection cut-off was changed from it's 50% value?\n3. What is the theoretical justification for multiplicative CoCoA combination rather than additive?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoCoA, a new framework for uncertainty quantification (UQ) in large language models. The method combines two key signals: the model's internal confidence (derived from token probabilities) and the semantic consistency of its outputs across multiple sampled generations. The authors frame this combination within a minimum Bayes risk framework. To address the computational cost of multiple generations, a more efficient variant, CoCoA Light, is proposed, which uses a lightweight predictor to estimate semantic inconsistency. The authors conduct experiments on various tasks, including question answering, summarization, and machine translation, showing that CoCoA outperforms existing UQ baselines in failure prediction.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1.  The paper provides a clear and well-grounded framework that unifies two important sources of uncertainty—model confidence and output consistency. The connection to the minimum Bayes risk framework provides a solid theoretical foundation.\n2.  The experiments are comprehensive, covering multiple tasks, datasets, and models. CoCoA demonstrates significant and consistent improvements over a variety of established UQ baselines on failure detection metrics like PRR and AUROC. And the CoCoA framework is presented as a plug-in module that can enhance different information-based uncertainty estimators, highlighting its flexibility and potential for broad applicability. \n\n**Weaknesses:**\n\n1.  **Limited Novelty:** The core idea of combining confidence and consistency feels somewhat incremental. The paper could be strengthened by a deeper investigation into the relationship and interaction between these two types of uncertainty, beyond the proposed multiplicative combination. The current insight into *why* this combination is so effective is not fully explored.\n2.  **Missing Baselines:** The evaluation does not include comparisons with \"verbalized confidence\" methods[1]. [1] demonstrates that the logits of the output of LLMs suffers from over-confident issue due to RLHF process. [2] proposes sampling and consistency-based confidence calibration methods to measure the intrinsic uncertainty of LLMs. These methods, which prompt the model to express its own uncertainty, are a relevant and important class of UQ techniques and would serve as a crucial point of comparison.\n3.  **Incomplete Evaluation Metrics:** The evaluation relies primarily on failure detection metrics (PRR, AUROC). While important, these do not measure confidence calibration. Including metrics like the Expected Calibration Error (ECE) would provide a more complete picture of the UQ quality.\n4.  **Limited Model Scale:** The experiments are conducted on models up to 8B parameters. The claims would be more convincing if the method's effectiveness were also demonstrated on larger, state-of-the-art models (e.g., Llama 3 70B, even with quantization). \n\n[1] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In EMNLP, 2023.\n\n[2] Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. In ICLR, 2023.\n\nQUESTIONS:\n1.  How would you adapt the CoCoA framework for use with closed-source, black-box models (e.g., via the GPT-4 API) where direct access to token probabilities is not available?\n2.  The paper omits comparisons with verbalized confidence methods. Could you elaborate on the potential advantages or disadvantages of CoCoA relative to this line of work? \n3.  While the experiments show that a multiplicative combination works well, have you explored other aggregation functions (e.g., attention-based weighting)? \n4.  How the reasoning-models like QwQ-32B perform on the CoCoA framework?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new framework called CoCoA for uncertainty quantification (UQ) of large language models (LLMs). The method is based on minimum Bayes risk (MBR). They start by showing how many existing “confidence” (information-based) and “consistency” (sampling-based) scores are different approximations of the same MBR objective.\n\nThen they introduce CoCoA - a method for MBR-derived scores by multiplying a sequence-level confidence term with semantic-inconsistency term. They also present CoCoA-Light, that learns to predict the consistency term from the model’s own hidden states, so it needs only a single greedy decode at test time. This can avoid the requirement to generate multiple samples for UQ.\n\nThe authors conduct experiments on 3 tasks (question answering, abstractive summarisation and machine translation) with 3 open-weight 7-8 B models (mistral, llama, falcon) and show that CoCoA and CoCoA-Light do better than baselines across PRR, AUROC and other metrics.\n\nOverall this is a very good submission.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nS1.\tThe paper presents a clear theoretical unification.  While [10] had introduced uncertainty aware MBR, this paper re-derives disparate UQ scores as MBR risks and clarifies when and why they work.\n\nS2.\tThe solution presented seems relatively simple but effective metric.  The components of CoCoA are already familiar to the research community. So this method seems relatively easy to implement. I assume these methods are available in LM-Polygraph library, which the authors also seems to have used.\n\nS3.\tCoCoA-Light is very interesting. One of the main complaints about sampling techniques is the cost. Hence the solution provided in this paper to approximate using two MLPs and hence avoid sampling is very interesting.\n\nS4.\tThe paper does a very good job with the experiments. The authors experiment on 3 tasks (QA, summarization and translation), use 3 small sized models, use PRR besides other metrics. They also present useful ablations. Reporting AUROC in addition to PRR and performing experiments with other metrics (for PRR) shows thoroughness of experiments. \n\nS5.\tThe results of their experiments support the claims of the paper. CoCoA ppl performing better than baseline perplexity etc gives lot of confidence in this proposed approach.\n\nWeaknesses:\n\nW1.\tThere is some dependency on an external similarity model. They mention task and domain dependency in the limitations section. I wonder if the length of the generated text is also crucial. I might have missed it, but it would be good to comment on the effect of length of the generated output on CoCoA.\n\nW2.\tQA (especially long form), summarization and machine translation are somewhat similar tasks. Adding GSM8k is useful. It would be good to see how these methods work on code generation tasks.\n\nW3.\tMore details on the CoCoA light training will be useful. What is the size of the training set?\n\nQUESTIONS:\n1. How will you use this approach for large black box models provided by the likes of OpenAI, Claude and Gemini. We can generate consistency scores. Can the confidence be approximated somewhat similar to what you have tried with CoCoA light? Assuming it is allowed under their terms, can small models be learned from such large black box models to provide confidence scores and used in this CoCoA setting?\n\n2. In Figure 1, you have illustrated the problem with model confidence. You tend to alleviate this problem by using it together with consistence measures. But is it possible that if the models are large and the generated text is small, the consistency based methods will also miss the variations in the possible outputs?\n\nWhat I mean is most of the samples look like romeo and juliet, and very few look like romeo & juliet or romeo and juliet,. This might happen if the samples are few and based on temperature sampling. In such a scenario how does MBR or CoCoA help?\n\nLIMITATIONS:\nYes, the authors have listed 3 limitations which are quite thoughtful.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-05"}