Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers
Authors: Dylan Bouchard
Mohit Singh Chauhan
Venue: Accepted by TMLR ()

Abstract: Hallucinations are a persistent problem with Large Language Models (LLMs). As these models become increasingly used in high-stakes domains, such as healthcare and finance, the need for effective hallucination detection is crucial. To this end, we outline a versatile framework for closed-book hallucination detection that practitioners can apply to real-world use cases. To achieve this, we adapt a variety of existing uncertainty quantification (UQ) techniques, including black-box UQ, white-box UQ, and LLM-as-a-Judge, transforming them as necessary into standardized response-level confidence scores ranging from 0 to 1. To enhance flexibility, we propose a tunable ensemble approach that incorporates any combination of the individual confidence scores. This approach enables practitioners to optimize the ensemble for a specific use case for improved performance. To streamline implementation, the full suite of scorers is offered in this paper's companion Python toolkit, \texttt{uqlm}. To evaluate the performance of the various scorers, we conduct an extensive set of experiments using several LLM question-answering benchmarks. We find that our tunable ensemble typically surpasses its individual components and outperforms existing hallucination detection methods. Our results demonstrate the benefits of customized hallucination detection strategies for improving the accuracy and reliability of LLMs.

Published in Transactions on Machine Learning Research (11/2025)
Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers
Dylan Bouchard CVS Health, Wellesley, MA
dylan. dgjonn.bocharrd@@cskeett..ooo
Mohit Singh Chauhan CVS Health, Wellesley, MA
Reviewed on OpenReview: https: // openreview. net/ met/ forum? id- WOFspd4
Abstract
Hallucinations are a persistent problem with Large Language Models (LLMs). As these mod- els become increasingly used high-stakes domains, such as healthcare and finance, the need for effective hallucination detection is crucial. To this end, we outline versatile framework for closed-book hallucination detection that practitioners can apply to real-world use cases. To achieve this, we adapt variety of existing uncertainty quantification (UQ) techniques, including black-box UQ, white-box UQ and LLM-as-a-Judge, transforming them as nec- essary into standardized response-level confidence scores ranging from 0 to 1. To enhance flexibility, we propose a tunable ensemble approach that incorporates any combination of the individual confidence scores. This approach enables practitioners to optimize the en- semble for specific use case for improved performance. To streamline implementation, the full suite of scorers is offered in this paper's companion Python toolkit, uqlm. To evaluate the performance of the various scorers, conduct extensive set of experiments using several LLM musstion-answerngg benchmarks. We find that our tunable ensemble typically surpasses its individual components and outperforms existing hallucination detection meth- ods. Our results demonstrate the benefits customized hallucination detection strategies for improving the accuracy and reliability of LLMs.
1 Introduction
Large language models (LLMs) are being increasingly used in production-level applications, often in high- stakes domains such healtheare or finance. Consequently, there is rising need to monitor these systems for the accuracy and factual correctness of model outputs. In these sensitive use cases, even minor errors pose serious safety risks while also leading to high financial costs and reputational damage. A particularly concerning risk for LLMs is hallucination, where LLM outputs sound plausible but contain content that is factually incorrect. Many studics have investigated hallucination risk for LLMs (see Huang et al. (2028) Tonmoy et al. 2024]; Shorinwa et al. (2024); /Huang al. (2024) for surveys of the literature). Even recent models, such as APen GPT-4.5, have been found to hallucinate as often as 37.1% on certain benchmarks
OpenAl 2025 underscoring the ongoing challenge of ensuring reliability in LLM outputs.
Hallucination detection methods typically involve comparing ground truth texts to generated content, com- paring source content to generated content, or quantifying uncertainty. Assossments that compare ground truth texts to generated content are lypically conducted pre-deployment in order to quantify hallucination risk an LLM for a particular use case. While important, this collection of techniques does not lend itself well to real-time evaluation and monitoring of systems alrcady deployed production. In contrast, tech- niques that compare source content generated content or quantify uncertainty can compute response-level scores generation time and hence can be used for real-time monitoring of production-level applications.

---

Published in Transactions on Machine Learning Research (11/2025)
Uncertainty quantification (UQ) techniques can be used for hallucination detection in a closed-book set- ting, meaning they do not require access to database of source content, ground truth texts, or internet access. These approaches are typically classified as either black-box UQ, white-box UQ, or LLM-as-a-Judge. Black-box UQ methods exploit the stochastic nature of LLMs and mcasure semantic consistency of multiple responses generaled from the same prompt. White-box UQ methods leverage token probabilities associated with the LLM outputs to compute uncertainty or confidence scores. L.M-as-a-Judge methods use one or more LLMs to evaluate the factual correctness of a question-answer concatenation.
In this paper, outline a versatile framework for generation-time, closed-book hallucination detection that practitioners can apply to real-world use cases. To achieve this, we adapt variety of existing black-box UQ, white-box UQ, and LLM-as-a-Judge methods, applying transformations as necessary to obtain standardized confidence scores that range from 0 to 1 For improved customization, we propose a tunable ensemble approach that incorporates any combination of the individual scorers. The ensemble output a simple weighted average of these individual components, where the weights can be tuned using a user-provided set of graded LLM responses. This approach enables practitioners lo optimize the ensemble for a speeific use case, leading to more accurate and reliable hallucination detection. Importantly, our ensemble is extensible, meaning practitioners can expand to include new components as rescarch on hallucination detection evolves.
We evaluate the full suite of UQ scorers on responses generated by four LLMs across various question-answer benchmarks, yielding several empirical insights. Most notably, our tunable ensemble generally outperforms its individual components for hallucination detection. However, the performance ranking of individual scor- ers varies by dataset, underscoring the value of tailoring methods to specific use cases. Among black-box UQ scorers, entailment-style approaches demonstrate superior performance in our comparisons. Further- more, gains from sampling additional responses shrink as the number of candidate responses rises, providing practical guidance for deployment. Lastly, a model's accuracy on a given dataset positively relates to its performance as a judge of other models' answers on that dataset.
Finally, this paper is complemented by our open-source Python package, uqlm, that provides ready-to-use implementations of all uncertainty quantification methods presented and evaluated in this work P uqlm enables practitioners to generate responses and obtain respouse-level confidence scores by providing prompts (i.c., the questions or tasks for the LLM) along with their chosen LLM. Our framework and toolkit provide researchers and developers a model-agnostic, user-friendly way to implement our suite of UQ-based scorers in real-world use cases, enabling more informed decisions around LLM outputs.
2 Related Work
Black- UQ Cole et al. 20233 propose evaluating similarity between an original response and candi- date responses using exact match- based metrics. In particular, they propose two metrics: repetition, which measures the proportion of candidate responses that match the original response, and diversity, which penal- izes a higher proportion of unique responses in the set of candidates. These metrics have the disadvantage of penalizing minor phrasing differences even if two responses have the same meaning. Text similarity metries assess response consistency in a less stringent manner. Manakul et al. 2023) propose using n--gaa-bbased evaluation to evaluate text similarity. Similar metrics such as ROUGE [Lin 2004), BLEU (Papineni et al. 2002), and METEOR (Banerjee & Lavie 2005) have also been proposed (hhoriwwa et al. 2024). These metrics, while widely adopted, have the disadvantage of being highly sensitive to token sequence orderings and often fail to detect semantic equivalence when two texts have different phrasing. Sentence embedding- based metrics such as cosine similarity (Qurashi et al., 2020),, computed using sentence transformer such as Sentence-Bert (Reimers & Gurevych 2019), have also been proposed (Shorinwa et al. 2024). These metrics have the advantage of being able to detect semantic similarity in pair of texts that are phrased differently. In similar vein, Manakul et al. 2023 propose using BERTScore (Zhang et al. 2020 based
We employ UQ-style signals pursuing uncertainty quantification as an end in itself. We do not separatc alcatoric and epistemic uncertainty. 2The uqlm repository can he found at htttp:://gtthbb.com com. www/cveeeeelllteee heslth/uqlm For a detailed description of the software, we refer the reader Rouchard et 2025
means produce responso-level confidence for hallucination detection, rather than
2

---

Published in Transactions on Machine Learning Research (11/2025)
the maximum cosine similarity of contextualized word embeddings between token pairs in two candidate texts.
Natural Language Inference (NLI) models are another popular method for evaluating similarity between an original response and candidate responses. These models classify a pair of texts as either entailment, con- tradiction, or neutral. Several studies propose using NL. estimates of 1 P(contradiction) or P(entailment) between the original response and a set of candidate responses to quantify uncertainty (Chen & Mueller 2023 Lin et al.) 2024). Zhang et al. (2024) follow similar approach but instead average across sentences and exclude P(neutral) from their calculations Other studies compute semantic entropy using NLI-based
clustering (Kuhn ct al. 2023 Kossen et al. 2024 Farquhar et al. 2024). Qiu & Miikkulainen (2024) estimate
density in semantic space for candidate responses.
White-Box UQ Manakul et al. (023) consider two scores for quantifying uncertainly with token prob- abilities: average negative log probability and maximum negative log probability. While these approaches effectively represent a mcasure of uncortainty, they lack case of interpretation, are unbounded, and are more useful for ranking than interpreting a standalone score. Fadeeva et al. (2024) consider perplexily, calculated the exponential of average negative log probability. Similar to average negative log probability, perplexity also has the disadvantage of being unbounded. They also consider response improbability, computed as the complement of the joint token probability of all tokens in the response. Although this metric is bounded and easy to interpret, it penalizes longer token sequences relative to semantically equivalent, shorter token sequences. Another popular metric is entropy, which considers token probabilities over all possible token choices in a pre-defined vocabulary (Malinin & Gales] 2021 Manakul et al.) 2023). Malinin & Gales (2021) also consider the geometric mean of token probabilities for response, which has the advantage of being bounded and easy to interpret.
LLM-as-a-Judge For uncertainty quantification, several studies concatenate qucstion-answer pair and ask an LLM to score or classify the answer's correctuess. Chen & Mueller (0023 propose using LLM for self-reftection certainty, where the same LLM is used to judge correctness of the response. Specifically, the LLM is asked to score the response as incorrect, uncertain, or correct, which map to scores of O, 0.5, and 1, respectively. Similarly. Kadavath et al. (2022) ask the same LLM to state P(Correct) given a question- answer concatenation. Xiongg et al. (2024) explore several variations of similar prompting stralegies for LLM self-evaluation. More complex variations such as multiple choice question answering generation (Manakul
et al. 2023), multi-LLM interaction (Cohen et al. 2023), and follow-up questions (Agrawal et al. 2024 have also been proposed.
Ensemble Approaches Chen & Mueller (023) propose a two-component ensemble for zero-resource hallucination known as BSDetector. The first component, known as observed consistency, computes a weighted average of two comparison scores between an original response and a set of candidate responses, one based on exact match, and another based on NLL-estimated contradiction probabilities. The second component is self-reflection cortainty, which uses the same LLM to judge correctness of the response. In their ensemble, response-level confidence scores are computed using weighted average of observed consistency and self-reftection certainty. Vergaa et al. (0244) propose using Panel of LLM evaluators (PoLL) to assess LLM responses. Rather than using single large LLM judge, their approach leverages panel of smaller LLMs. Their experiments find that PoLL outperforms large LLM judges, having less intra-model bias in the judgments.
3 "Averaging across sentences done to address long-form responses. Jiang al. (004)) also address long-form hallucination detection but follow graph-based approach instcad. FFr additional white-box uncortainty quantification techniques, we refer the reader to Ling et (2024]
Bakman al.
2024h: Guerreiro al. 20233] Zhang al. 20233 Varshney et. 2023 Luo al. 2023 Ren al. 20023 van der Poel
2022);
Wang et
2023)
3

---

Published in Transactions on Machine Learning Research (11/2025)
3 Hallucination Detection Methodology
3.1 Problem Statement
We aim to model the binary classification problem of whether an LLM response contains a hallucination,
which we define as any content that nonfactual. To this end, we define collection of binary classifiers, each of which map an LLM response Ys V, generated from prompt T₁, to a 'confidence score' between 0 and 1, where y is the set of possible LLM outputs. We denote hallucination classifier $ - -0,,1 1).
Given a classification threshold т, we denote binary hallucination predictions from the classifier as h {0, 1}. In particular, hallucination is predicted if the confidence score is less than the threshold т:
h(y::
7) -
I(((():)) < <),
(1)
where 8, and consequently h, are conditioned on context variables including the prompt x; and, when applicable, additional responses generated from x; Note that h(-) implies a hallucination is predicted. We denote the corresponding ground-truth label indicating whether the original response y; actually contains hallucination as h(y:i). The function h represents grading process that compares generated response Vi to correct reference answer of for the same prompt zy;
MyGDi - (1
y1 is incorrect with respect to y, herwise.
(2)
When the correct answer oi is known, direct comparison yields the most accurate hallucination label. In deployment, however, y is not available at generation time, so h cannot be used as real-time classifier. We therefore treat h as oracle ground truth used only offline for evaluation and, where applicable, tuning, Our framework approximates h with h using uncertainty-basel signals available at generation time, consistent with our closed-book setting and without access to y.-
We adapt techniques from the mncertaintt-quantiffctttio literature to compute response-level confidence for gencration-time hallucination detection. Acknowledging the distinction between uncertainty quantification and hallucination detection, between aleatoric and epistemic uncertainty. We transform and normalize outputs, if necessary, that each score lies in [0, 1] with higher values indicating greater confidence 5 Below, we detail these scorers.
study UQ-based confidence scorers for this purpose and do not distinguish
3.2 Black- Box UQ Scorers
Black-box UQ scorers exploit variation in LLM responses the same prompt to assess semantic consistency, For given prompt æ;, these approaches involve generating candidate responsos ý; {0A, ..., Fim } using non-zero temperature, from the same prompt and comparing these responses to the original response yi. We provide detailed descriptions of each below.
Exact Match Rate. For LLM tasks that have a unique, closed-form answer, exact match rate can be a useful hallucination detection approach. Under this approach, an indicator function used score pairwise comparisons between the original response and the candidate responses. Given original response yi and candidate responses yi, generated from prompt I,, exact match rate (EMR) is computed as follows:
1 (M = m Σ I(ys = Dt))
(3)
FFor notational simplicity, use n. placcholder here and explicitly write out conditioning variables in the definitions that follow. Many already output values 0, and therefore do require normalization.
4

---

Published in Transactions on Machine Learning Research (11/2025)
Non-Contradiction Probability. Non-contradiction probability (NCP) is similar, but less stringent approach. NCP, a component of the BSDetector approach proposed by Chen & Mueller (2023) also conducts pairwise comparison between the original response and each candidate response. In particular, an NLI model is used to classify each pair (y:. Byy) as entailment, neutral, or contradiction and contradiction probabilities are saved. NCP for original response yi is computed as the average NLI-based non-contradiction probability across pairings with all candidate responses:
NCP( = 1
m myv. Diy) + m(0r5.
(4)
m
2
Above, Ty, Mij) denotes the contradiction probability of (yis Day) estimated by the NLI model Following Chen & Mueller (2023) and Farquhar al. (2024); we model.
for our NLI
BERTScore. Another approach for measuring text similarity between two texts is BERTScore (Zhang
et al. 2020). Let a tokenized text sequence denoted as tt= (11,...1.) and the corresponding contextualizes word embeddings as E (ei,..,,zl where L is the number of tokens in the text. The BERTScore precision and recall scores between two tokenized texts t, t' are respectively defined follows:
BtrlP(t,t
Σ 1Et
max e e:;; BerlR(tt, () = tt' eet'
Σ
max e
(5)
[t
LEt
MEE
where e, respectively correspond to t, t'.
We compute our BERTScore confidence (BSC) as follows:
Bert yys De)) BertR Dig) BSC()a x;) = m BertP (v., Dij) + Bert Dug)
1
(6)
i.c. the average BERTScore Fl score across pairings of the original response with all candidate responses.
Normalized Cosine Similarity. Normalized cosine similarity (NCS) leverages sentence transformer to map LLM outputs to an embedding space and measure similarity using those sentence embeddings. Let V y - Rd denote the sentence transformer, where is the dimension of the embedding space. We define NCS as the average cosine similarity across pairings of the original response with all candidate responses, normalized by dividing by and adding 1
V(6) V(7 + NCS( = 2m (V
Normalized Semantic Negentropy. Semantic entropy (SE), proposed by Farqubar et al. (024)), exploits variation in multiple responses to compute measure of response volatility. The SE approach clusters responses by mutual entailment and, like the NCP scorer, relies on an NLI model. However, in contrast the aforementioned black-box UQ scorers, semantic entropy does not distinguish between an original response and candidate responses. Instead, it computes a single metric value on list of responses generated from the same prompt. We consider the discrete version of SE, defined as follows:
SE v., = Σ P(Clus, log P(Cly.. 3t).
(8)
CEC
where P(Clys 3:) denotes the probability randomly selected response E {8i, Di,.... Dim) belongs to cluster C, and C denotes the full set of clusters of {y:, ".., Dim To ensure that we have a normalized
usymmetric measure. Manakul al. (023) propose a one-directioall variant of NCP. We follow Chen & Mueller 2023 using the bidirectional formulation allow for more flexibility in detecting contradictions. "I1 token probabulttis of the LLM responses available, the values of P(Cly; Vv) can be instead estimated using mean token probability. However, unlike the discrete case, this version semantic entropy unbounded and hence does not lend itself well normalization.
We note that NLI
5

---

Published in Transactions on Machine Learning Research (11/2025)
confidence score with [0, 1] support and with higher values corresponding to higher confidence, implement the following normalization to arrive at Normalized Semanlic Negendropy (NSN):
SE(y: Vi. 23)
NSN(u. =1
(9)
log(m
where log(m + 1) is included to normalize the support.
3.3 White-Bor UQ Scorers
White-box UQ scorers leverage token probabilities of the LLM's generated response to quantify uncertainty. We define two white-box UQ scorers below.
Length-Normalized Token Probability. Let the tokenization of LLM response yi be denoted (l1,..,4t,) where L; denoles the number of tokens the response. Lenglh-normmalized token probability (LNTP) computes length-normalized analog of joint token probability:
LNTP(gAE.)
II Pt
(10)
KEW: where Pt denotes the token probability for token Note that this score is equivalent to the geometric mean of token probabilities for response y/-
Minimum Token Probability. Minimum token probability (MTP) uses the minimum among token probabilities for a given responses as a confidence score:
MTP(()a;; min Pt.
(11)
where f and Pt follow the same definitions as above.
3.4 LLM-as-a-Judge Scorers
We employ am LLM-as-a-Judge scorer (115 2T:) as an additional method for obtaining response-level con- fidence scores. In this approach, we concalenate a question-response pair and pass it to an LLM with a carefully constructed instruction prompt that directs the model to evaluate the correctness the response. We adapt our instruction prompt from Xiong al. (2024), instructing the LLM score responses on a 0-100 scale, where higher score indicates greater certainty that the provided response is correct. These scores are then normalized to 0-to-1 scale to maintain consistency with our other confidence scoring methods. The complete prompt template is provided in Appendix A
3.5 Ensemble Scorer
We introduce tunable ensemble approach for hallucination detection. Specifically, ensemble is weighted average of K binary classifiers: & 3 - [0, for = 1...., K. As several of our ensemble components exploit variation in LLM responses to the same prompt., our ensemble conditional on (yi, w), where w denote the ensemble weights. For original response yi, we can write our ensemble classifier as follows:
8(yn w) Σ wv (16; Vi,
(12)
where w (wi,..., WK) E = 1, and we E 10 for h: K
9 Although it not reflected in notation, the probability for given token conditional the precoding tokens. Note that although write each classifier be conditional the set candidate responses, of the classifiers depend only on the original response.

---

Published in Transactions on Machine Learning Research (11/2025)
Tuning the ensemble requires a sample of LLM responses Un to set of prompts, a set of K confidence scores for each response {8) (11; ys. 24), ...,)...888 8x (y:: y,, -;)) and corresponding binary hallucination indica- tors h .1),..... h(yni 2, Fn]] 1I Given a classification objective function, the ensemble weights w can be tuned with an optimization routine If the objective is threshol--agnosiic, the weights and threshold T can be luned sequentially. For a thresholdddepeenden objective, the weights and threshold can be tuned jointly. Because scorer performance depends on both the dataset and the underlying L.M,, the ensemble weights are tuned per use case (chosen LLM and dataset), and the resulting weights are intended for in-domain deployment. See Appendix B for more details on ensemble tuning.
4 Experiments
4.1 Experiment Setup
We conduct a series of experiments to assess the hallucination detection performance of the various scorers. To accomplish this, we leverage a set publicly available benchmark datasets that contain questions and answers. To ensure that our answer format has sufficient variation, we use two benchmarks with numerical answers (GSM8K (Cobbe al.) 2021) and SVAMP (Patel al. 2021)), two with multiple-choice answers (CSQA Talmor ct al. 2022) and A12-ARC (Clark et al. 2018) and two with open-ended text answers ( Mallen et al. 2023 and NQ-Open (Lee el al. 2019)).
We sample 1000 questions for each of the six benchmarks. For each question, we generate an original response and m = 15 candidate responses using four LLMs: GPT-4o (OpenAI), GPT-4o-mini (OpenAl). Gemini-2.5-Flash (Google), and 2emmini-2.5-Flahh--iitt (Googlep 13 Candidate responses are generated using a temperature of 1.0. For each response, we use the corresponding candidate responses generated by the same LLM to compute the full suite of black-box UQ scores. We extract the bog-probabilities from the generated responses to compute white-bax UQ scores. Lastly, compute LLM-as-a- auduee scores for each response using the same four LLMs. We evaluate the hallucination detection performance of all individual scorers as well as our ensemble scorer for each of the benchmarks using various metrics, with grading processes detailed in Appendix All scores are computed using this paper's companion toolkit, uqlm
4.2 Experiment Results
Threshold-Agnos Evaluation To assess the performance of the scorers as hallucination classifiers, we evaluate the performance of the confidence scores in Uhreshold-agnostic ic fashion 15 Under this setting, we use the AUROC-optimized ensemble weights and compute the ensemble's AUROC using 5-fold cross- validation 10 The final reported AUROC is obtained by averaging the AUROC values across the five holdout
sets.
Figure presents the AUROC scores for all scorers across the 24 LLM--dtaset scenarios, while Table ! high- lights the best-performing scorer for each scenario. The AUROC values for the scenario-spccific best scorers range from 0.729 for GPT-4o responses on NQ-Open (Ensemble) to 0.986 for GSM8K (Ensemble). Overall, the top-performing scorers for each scenario exhibit strong hallucination detection performance, with AUROC values greater than 0.8 for 19 out of 24 scenarios. However, some scorers perform no better than or only negligibly better than a random classifier in hallucination detection in certain scenarios, such as the GPT-4o-mini LLM judge for all four GSM8K scenarios.
Lite responses
Note that h(w;gg .n a.).... h( V. un) serve as 'ground truth' labels in the classification objective function. 12Wo use optuna (Akiba et al. 2019) for optimization with default settings (more details available here). We use large number of candidate responses (m 15) to ensure robust comparisons across black-box scorers. In practice,
fewer candidates can be used. For an experimental evaluation of the impact of m performance, refer to Appendix (
14Using al-standard-16 machine (16 vCPU, 8 core, 60 GB memory) with a single NVIDA T4 GPU attached, experiments took approximately 0.5-3 hours per LLM-dataset combination complete.
16 our experiments, we label 'cotrect' LLM responses
and 'incorrect' responses 0.
16 16scorer performance depends on both the dataset and the LLM, we tune the ensemble for specific L.M-dataset pair. Our experiments reflect this in-domain setup; we do not evaluate ont-of-ddistribution generalization across datasets or domains.
7

---

Published in Transactions on Machine Learning Research (11/2025)
Black Box UQ
White- Box UQ
LLMaas-a
Ensemble UQ
(a) NQ Open
(b) PopQA
(c) GSM8K
(d) SVAMP
CSQA
(f) A12-ARC
Figure 1: Scorer-Specific AUROC Scores for Hallucination Detection by LLM and Dataset (Higher is Better)
When comparing AUROC values across scorers, find ensemble scorer outperforms its individual components in 20 out of 24 scenarios, demonstrating the benefits of uso-caso-specific optimization. The rankings of individual scorers, however, vary significantly across scenarios, with LLM-as-e-Judge. black- box, and white- box methods achieving top non-ensemble performance in 11, 7, and scenarios respectively. NLIIbased scorers (NSN and NCP) exhibit strong performance, achieving the highest AUROC in out of 24 scenarios among black-box scorers. Among LLM judges, GPT-4o and Gemini-2.5-Flash consistently outperform their smaller counterparts (GPT-4o-miní and
with GPT-4o ranking
8

---

Published in Transactions on Machine Learning Research (11/2025)
the highest-performiin judge in 13 out of 24 scenarios and Gemini-2.3-Flash in 10 out of 24 scenarios. Notably, Gemini-2.5-Flash tended to be the highest performing L.M judge on the math benchmarks (6 out of 8 scenarios), and GPT-4o tended to be the highest performing judge on the short-answer benchmarks (6 out of 8 sccnarios). Finally, the two white-box scorers perform approximately equally, with very similar AUROC scores the vast majority of scenarios.
Table 1: Hallucination Detection AUROC (Higher is Better): Best-Performing Scorer by LLM and Dataset
Model
Metric
NQ-Open
PopQA
GSM8K
SVAMP
CSQA
A12-ARC
AUROC
0.749
0.826
0.771
0.851
0.800
0.975
Gem.-2.5-Flah
Best Scorer
Ensemble
Ensemble
Ensemble
Ensemble
Ensemble
LNTP
AUROC 0.806
0.880
0.986
0.968
0.840
0.965
Lems-L.---l
Best Scorer
Ensemble
Ensemble
Eusemble
Ensemble
Ensemble
LNTP
AUROC 0.729
0.798
0.979
0.938
0.844
0.860
GPT-4o
Best Scorer
Ensemble
Ensemble
Ensemble
Ensemble
Ensemble
LNTP
AUROC 0.778
0.880
0.982
0.930
0.822
0.935
GP'T-4o-Mini
Best Scorer
Ensemble
Ensemble
Ensemble
Ensemble
Ensemble
LNTP
Threshold-Opiimized Evaluation. We evaluate the various scorers using a threshold-depeendent metric (FI-score). To compute our ensemble scores in this setting, we jointly optimize the ensemble weights and threshold using Fi-score the objective function, outlined in Appendix B To ensure robust evaluations, compute the scorer-specific Fl-scores using 5-fold cross-validation For each individual scorer, select the F1-optimal threshold using grid search on the tuning set and compute F1-score on the holdout set. We report the final Fl-score for each scorer as the average across holdout sets.
Table 2: Hallucination Detection F1-Scores (Higher is Better): Best- Scorer by LLM and Dataset
Model
Metric
NQ-Open
PopQA
GSMSK
SVAMP
CSQA
A12-ARC
Precision
0.597
0.688
0.950
0.968
0.856
0.985
Recall
0.907
0.890
0.999
0.999
0.986
0.987
Cem.-2.5-Flashh
F1-Score
0.718
0.774
0.974
0.983
0.916
0.986
Best Scorer
Ensemble
Ensemble
Ensemble
Ensemble
Ensemble
NSN
Precision
0.587
0.680
0.960
0.977
0.863
0.968
Recall
0.838
0.864
0.980
0.976
0.973
0.997
Gem.-2.5-FI.-Lt
F1-Score
0.688
0.759
0.969
0.977
0.915
0.982
Best Scorer
Ensemble
Ensemble
Ensemble
Gem.-2.5-F1.
Ensemble
Gem.-2.5-Fi
Precision
0.649
0.727
0.958
0.978
0.853
0.987
Recall
0.917
0.880
0.987
0.990
0.972
1.000
GPT-4o
F1-Score
0.756
0.795
0.972
0.984
0.909
0.993
Best Scorer
Ensemble
Ensemble
Ensemble
Gem.-2.5-F1.
NCP
Gem.-2.5-Fi.
Precision
0.616
0.700
0.906
0.969
0.840
0.952
Recall
0.867
0.876
0.978
0.988
0.991
0.998
GPT--4-Mini
F1-Score
0.718
0.776
0.940
0.978
0.909
0.975
Best Scorer
Ensemble
Ensemble
Ensemble
Ensemble
Ensemble
Gem.-2.5-FL.
Figure 2 displays the Fl-scores for each scorer, while Table summarizes the precision, recall, and Fl metrics for the top-performing scorer across all 24 evaluation scenarios. The results are largely consistent with the AUROC experiments. The ensemble scorer outperforms its individual components in most scenarios, achieving highest Fl-score in 17 out 24 scenarios. Similar to the AUROC experiments, the NLI-based scorers (NSN and NCP) often outperform other black-box scorers (18 out of 24 sconarios), consistent with the findings of previous studies (Kuhn et al.) 20233 Manakul al.] 2023 Lin et al. 2024 Farquhar et al. 2024). Once again, the Demini-225-Flaas judge outperfoms other L.M judges the math benchmarks

---

Published in Transactions on Machine Learning Research (11/2025)
Black Box UQ
White- Box UQ
LLMaas-aJudee
Ensemble UQ
(a) NQ Open
(b) PopQA
EPTA
(c) GSM8K
(d) SVAMP
(e) CSQA
(f) A12-ARC
Figure 2: Scorer-Specifi F1-Scores for Hallucination Detection by LLM and Dataset (Higher is Better)
out of 8 scenarios), while the GPT-4o judge outperforms other LLM judges on the short answer benchmarks (6 out of scenarios). Interestingly the strong LLM judge performance of Gemini-2.5-Flash on the math benchmarks and GPT-4o on the short answer benchmarks are consistent with their respective strengths when generating answers as the original LLM, where they achieved highest accuracy among the four LLMs on those same benchmarks (see Figure 3). Lastly, the two wehite-box scorers perform approximately equally, in the AUROC experiments.
10

---

Published in Transactions on Machine Learning Research (11/2025)
Filtered Accuracy07. Lastly, we compute model accuracy the subset of LLM responses having con- fidence scores exceeding specified threshold т. We refer to this metric as Filtered Accuracy@T. Since the LLM accuracy depends on the choice of the threshold т, repeat the calculation for T = 0, 0... 0.9. Note that accuracy at T = 0 uses the full sample without score-based filtering.
Black Box Scorers
White, Box Scorers
Judges
Ensemble
Baseline LLM Accuracy
Gemini-2.5 2.5-hiish
Semmini-.5--llah--1
GPT--40
GPT-4o-mini
SVAMP
0.85
0.05
0.35
0.35
0.50
0.30
0.85
0.85
0.85
0.80
0.80
GSM8K
0.0
0.8
0.8
0.8
0.6
0.6-
0.2
0.4
0.6
0.8
0.0
02
0.0
0.2
0.5
0.8
A12-ARC
0:98
0.39
0.38
0.98
0.26
0.06
0.06
0.95
0.95
0.35
0.84
0.84
0.0
0.2
0.6
0.5
0.0
0.5
0.2
0.4
0.6
0.0
0.0
0.2
2.4
0.6
0.0
0.0
0.2
0.4
0.6
0.0
0.50
0.50
0.00
0.50
0.06
0.25
0.84
0.24
0.34
0.84
0.82
0.82
0.82
0.82
0.80
0.50
0.50
0.80
0.0
0.2
0.4
0.6
0.5
0.0
0.2
0.4
0.8
0.8
0.8
0.8
NO-Open
0.5
0.5
0.4
0.4
0.4
0.4
0.4
0.5
0.0
0.0
0.0
0.0
PopOA
0.6
0.6
0.
0.5
Figure 3: Filtered LLM Accuracy vs. Confidence Threshold (Top per Scorer Type)
Figure presents the Filtered Accuracy@7 for the highest performing white-box, black box, LLM-as-a-Judge, and ensemble scorers. Across all scenarios, response filtering with the highest-performing scorers leads to an approxinately monotonie increase in LLM accuracy as the threshold increases. For example, when filtering Gemini-2.55Flastt
responses to PopQA questions using the leading white-box scorer, accuracy improves dramatically from a bascline of 0.35 to 0.61 r = 0.6. Similarly, with GPT-4o responses on GSM8K, filtering with the ensemble scorer achieves 0.93 accuracy - 0.6, substantially higher than the bascline accuracy of 0.55.
11

---

Published in Transactions on Machine Learning Research (11/2025)
5 Discussion
Choosing Among Scorers. Choosing the right confidence scorer for LLM system depends several factors, including API support, latency requirements, LLM behavior, and the availability of graded datasets. If the API supports access to token probabilities in L.M generations, white-box scorers can be implemented without adding latency or generation costs 17 abilities, black-box scorers and LLM-as-a-Judge may be the only feasible options. When choosing among black-box and T.I.Mas----uuge scorers, latency requirements are a key consideration. For low-latency ap- plications, practitioners should avoid higher-lateney black-box scorers such as NLI-based scorers (NSN and NCP), opting instead for faster black-box scorers or LLM-as-a-Julge. If lateney is not concern, any of the black-box scorers may be suitable.
However, if the API does not provide access to token prob-
For LLM-as-a-Judge implementations, our findings reveal that an LLM's accuracy on a specific dataset positively relates lo ils ability to judge responses to questions from that same dataset, providing a practical criterion for judge selection. Finally, if graded dataset available, practitioners can tune an ensemble of various confidence scores to improve hallucination detoction, as detailed in Appendix B Our experiments demonstrate that a tuned ensemble can potentially provide more accurate confidence scores than individual scorers. By considering these factors and choosing the right confidence score, practitioners can improve the performance of their LLM system.
Using Confidence Scores. In practice, practitioners may wish to use confidence scores for various pur- poscs. First, practitioners can use our confidence scores for response filtering, where responses with low confidence are blocked, or 'targeted' human-in-Uhe-loop, where low-confidence responses are selected for manual review. Our experimental evaluations of Filtered Accuracy@7 demonstrate the efficacy of these ap- proaches, illustrating notable improvements in LLM accuracy when low-confidence responses are filtered out. Note that selecting a confidence threshold for flagging or blocking responses will depend on the scorer used, the dataset being evaluated, and stakeholder values (e.g., relative cost of false negatives vs. false positives).
Alternatively, confidence scores can be used for pre-deployment diagnostics, providing practitioners insights into the types of questions which their LLM is performing worst. Findings from this type of exploratory analysis can inform strategies for improvements, such as further prompt engineering. Overall, the scorers included in our framework and toolkit provide practitioners with an actionable way to improve response quality, optimize resource allocation, and mitigate risks.
We note two important ethical considerations related to using confidence scores. First, a confidence score reflects model uncertainty rather than ground-truth-base correctness, and high scores can induce over- reliance in high-risk settings. We therefore caution against. using confidence scores decision guarantees in domains such medicine, law, or finance without external human review and domain-specificc safeguards. Second, we encourage monitoring for distributional fairness and content diversity, including per-segment error-rate evaluation, and, where applicable, safeguards that prevent systematic removal of accurade responses for particular groups.
Limitations and Future Work. We note few important limitations of this work. First, although our experiments leverage six question-answering benchmark datasets spanning three types of questions, we acknowledge that our findings on scorer-specific performance may not generalize to other types of questions T8 For example, long-form generation (such as summarization) can mix true and false claims within a single output and may require claim-level decomposition to obtain useful uncertainty signals. Likewise, correctness in code generation depends syntax and exccution, black-box consistency measurements will likely require alternative methods.
Second, while we conduct experiments using four LLMs, performance of the various scorers may differ for other LLMs Note that for different LLMs, differences in Loken probability distributions will impact the
The white-box scorers cousider in this work require only single generation per prompt. However, sampling-bused
whito-box methods also exist. See, for example, Kuhn al. 2023 Scalena all 2025 Vashurin et all (2025b Qiu &
Mikkkulainen (2024p. for practical reasons, we selected benchmark datasets containing questions that could be graded computationally
12

---

Published in Transactions on Machine Learning Research (11/2025)
behavior of white-box scorers, and the degree of variation in responses to the same prompt will affect the performance of black-box scorers. L.L.aas-a-Judge performance may vary significantly depending on the choice of LLM and instruction prompts used.
Additional limitations relate to the ensemble approach. First, while the ensemble approach is designed to optimize in-domain performance, we have not evaluated out-of-distribution generalization, including cross- dataset transfer of learned weights; we leave this to future work. Second, our experiments consider only linear ensembles. Future work should examine non-linear ensembling and its impact on hallucination detection, e.g.: monotonic generalized additive models that retain interpretability while capturing modest non-linear effects, mixture-of-cxperts. and tree-based ensembles. These may outperform lincar averaging when scorer signals interact or exhibit non-linear relationships. Practical trade-offss to study include latency, overfituing risk on small graded sets, and the need for interpretability in high-stakes deployments. Third, tuning the ensemble requires a graded dataset. For tasks with trivial grading (e-g., arithmetic, multiple-choice, short- answer), where there are well-defined, easy-to-recognize correct answers and comparisons with generated text can be computed automatically, this can be done by sampling representative set of questions, generating responses, computing the selected UQ scores, and using the automatically derived labels to fit the weights. For tasks with non-trivial grading (c.g., summarization), practitioners can begin with a small human-graded dataset (e.g., a few hundred items); additional labels can then be added incrementally from production logs to further optimize the weights and improve ensemble performance.
6 Conclusions
In this paper, we detail framework for closed-book hallucination detection comprised of various black-box UQ, white-box UQ, and LLM-as-a-Judge scorers. To ensure standardized outpuls of the scorers, we transform and normalize scorers (if necessary) such that all outputs range from 0 to 1, with higher scores indicating greater confidence in an LLM response. These response-level confidence scores can be used for generation- time hallucination detection across a wide variety of LLM use cases. Additionally, we introduce a novel, ensemble- based approach that leverages an optimized weighted average of any combination of individual confidence scores. Importantly, the extensible nature of our ensemble means that practitioners can include additional scorers as new methods become available.
Our experimental al evaluation of UQ-based scorers offers clear guidance for practitioners. Ensemble approaches consistently outperform individual methods for hallucination detection, with our findings strongly supporting use-case-specific customization rather than ano-size-fits-al solutions. For those without token-probability access, NLI-based approaches typically provide the best black-box performance. Importantly, gains from sampling additional responses diminish the number of candidate responses increases, offering a practical deployment guideline that balances effectiveness with computational efficiency. Finally, our analysis revealed that model's accuracy on specific dataset positively relates to its ability to judge responses to questions from that same dataset, providing practical criterion for judge selection in evaluation frameworks.
The uqlm library implements all scorers presented and evaluated in this work. The repository is actively maintained. We welcome issues and pull requests and will continue to update integrations, add new scorers as rescarch advances, and refresh examples as model and provider APIs change.
Acknowledgements
We wish to thank David Skarbrevik, Picro Ferrante, Xue (Crystal) Gu, Blake Aber, Viren Bajaj, Ho-Kyoong Ra, Zeya Ahmad, Matthew Churgin, Saicharan Sirangi, and Erik Widman for their helpful suggestions.
Conflict of Interest
DB and MSC are employed and receive stock and equity from CVS Healthe Corporation.
13

---

Published in Transactions on Machine Learning Research (11/2025)
References
Ayush Agrawal, Mirac Suzgun, Lester Mackey, and Adam Tauman Kalai. Do language models know when they're re hallucinating references?, 2024. URL https://arxiv.. http:////aaxi.ooraabb////// wtwgorrg////aeeeeeeoeee 2305. 18248
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD interna- tional conference on knowledge discovery & data mining, pp. 2623 2631, 2019. doi: 10.1145/3292500. 3330701.
Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, and Salman Avestimehr. Mars: Meaning aware response scoring for uncertainty estimation in generative llms, 2024. URL
org/ /
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (eds.), Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Transla- tion and/or /or Summarization, pp. 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL https://acclanthologgy www.gy...gggoogee... org/w05-0009/
Dylan Bouchard, Mohit Singh Chauhan, David Skarbrevik, Ho-Kyeong Ra, Viren Bajaj, and Zeya Ahmad. Uqlm: A python package for uncertainty quantification in large language models, 2025. URL. https: //arxiv. wwt.ooraabbb/..001.111.1
Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answors from any language model and enhancing their trustworthiness, 2023. URL
16171
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the al2 reasoning challenge, 2018. URL
https://arxiv. wwwps//rggbbbbbbbbbebeee 1500bb0557133
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL
www.ooggg/////ee.eeo/ooooo.ee//e abs/ htbb///11....4.. www.aiivioo//aa/aeeleeoooaaaooot
Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. Lm vs Im: Detecting factual errors via cross examination, 2023. URL httpp:///axiv..c http///rxii...rr///////5// 13281
Jeremy R. Cole, Michael J. Q. Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. Selectively answering ambiguous questions, 2023. URL https://arxiv org/abs/ htgg/bb/2305..0 2305 14613.
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. Fact-checking the output of large language models via token-level uncertainty quan- 04696
tification, 2024. URL http:::://axxv.rrr//eee///
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625 630, Jun 2024. ISSN 1476-4687. doi: 10.1038/ $41156-004-00421100 URL http://pp.////...//////////////0
Google. URL.
Nuno M. Guerreiro, Flena Voita, and André F. T. Martins. Looking for a needle in a haystack: com- prehensive study of hallucinations in neural machine translation, 2023. URL htt.hhtps///// http//xxivv.rrrr abs/
05309
Hsiu- Yuan Huang, Yutong Yang, Zhaoxi Zhang, Sanwoo Lee, and Yunfang Wu. A survey of uncertainty estimation in llms: Theory meets practice, 2024. URL htttpp::/aarii.oor/bbbb//////// 15326
14

---

Published in Transactions on Machine Learning Research (11/2025)
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023. URL http:///aaxx..oorrraa///1.....2
Mingjian Jiang, Yangjun Ruan, Prasanna Sattigeri, Salim Roukos, and Tatsunori Hashimoto. Graph-based uncertainty metrics for long-form language model outputs, 2024. URL fttps://arxxv..rg// wtgp/bs////bbaaeeeeeee
20783
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Haddicld-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Gan- guli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know, 2022. URL https: /ttps://arxiv.orts http//aaa/22200000
Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa. Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in Ilms, 2024. URL https:://axxi.... /bbs/ 15927
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainly: Linguistic invariances for uncer- tainty estimation in natural language generation, 2023. URL attps://arxiv.org/abbb 09664
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering, 2019. URT. httpp://aarrv..oor///////0......
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Terl Summarization Branches Out, pp. 74 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https:
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models, 2024. URL https://arxiv. 0205.//2355.00 1.0...11111 19187
Chen Ling, Xujiang Zhao, Xuchao Zhang, Wei Cheng, Yanchi Liu, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Jie Ji, Guangji Bai, Liang Zhao, and Haifeng Chen. Uncertainty quantification for in-context learning of large language models, 2024. URL huttps://arxi
http/abb24402...1
Junyu Luo, Cao Xiao, and Fenglong Ma. Zerorresource hallucination prevention for large language models, 2023. URL
awtpr/9////////...om// abs 2209/ 220099026942444
Andrey Malinin and Mark Gales. Uncertainty estimation autoregressive structured prediction, 2021. URL
https://arxiv.ot wwtp://orrggaab///..oooo /t/////b222000276666
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. Tn Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Associalion for Compulational Linguisties (Volume 1: Long Papers), pp. 9802 9822, Toronto, 146..546. URL
Canada, July 2023. Association for Computational Linguistics. doi: https://aclanthool. wwwpaatthooooooooo////eo a:12:0233.1111.00
546/
Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models, 2023. URL
Op URL https://platforrs http::///ppatffroooopaaaa. openai www htt///dos/mmmoll
OP Introducing gpt-4.5 | openai, 2025. URL
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pp. 311-318, USA, 2002. Association for Computational Linguistics. doi: 10.3115/ 1073083.10731355 URL UR http::///aaa.orgg...1/////////// 107313
15

---

Published in Transactions on Machine Learning Research (11/2025)
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word wwtp://rr/ab///...02223200
problems?, 2021. URL
Xin Qiu and Risto Miikkulainen. Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space, 2024. URL https://axxii org/ ht////b//244/// 2405 13845
Abdul Wahab Qurashi, Violeta Holmes, and Anju P. Johnson. Document processing: Methods for semantic text similarity analysis. In 2020 International Conference on INnovations in Intelligent Sys Tems and /TTT:/ISSTAA99472000000002
Applications (INISTA), pp. 1 6, 2020. doi:
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks, 2019.
URL attpss///arxiv.org/bbbb
10084
Jie Ren, Yao Zhao, Tu Vu, Peter Liu, and Balaji Lalshminarayanan. Self-evaluation improves selective generation in large language models, 2023. URL http:://arxxi.orrg/bb//////
Daniel Scalena, Leonidas Zotos, Elisabetta Fersini, Malvina Nissim, and Ahmet Üstün. Eager: Entropy- /ttps///..o..//////111111/
aware generation for adaptive inferenco-time scaling, 2025. URL
Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z. Ren, and Anirudha Majumdar. A survey uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions, 2024. URL https://arxiv org
org/ /bs//ab/.////../2
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. Commonsensega 2.0: Exposing the limits ai through gamification, 2022. URL hhttps://arrt //arxvv. org/ hog/abb/2200..055
S. M Towhidul Islam Tonmoy, M Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models, 2024. URL
013113
Liam van der Poel, Ryan Cotterell, and Clara Meister. Mutual information alleviates hallucinations in abstractive summarization, 2022. URL httpps//arxxvv.or//bbeee
13210
Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. A stitch in time saves nine: Detecting and mitigating hallucinations of 1lms by validating low-confidence generation, 2023. URL
https://arxivv org/ //////b//// 2307 039887
Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. Benchmarking uncertainty quantification methods for large language models with im-oolygraph. Transactions of the Association for Computational Linguistics, 13:220 248, 2025a. ISSN 2307-3874 doi: 10.1162/tacl a 00737. URL http://dx. www.dooo............. 10. 1 /162/tacclccccc a 00737
Roman Vashurin, Maiya Goloburda, Albina Ilina, Aleksandr Rubashevskii, Preslav Nakov, Artem Shel- manov, and Maxim Panov. Uncertainly quantification for 1lms through minimum bayes risk: Bridging confidence and consistency, 2025b. URL htttp:://arxiv.rggee
Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorod- sky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating Ilm generations with a panel of diverse models, 2024. URL https:://a/at:t org/ abs/2404. 2404. 18796
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazcika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models, 2024. URL https: //arxiv wwt.:os//////////ooo 11698
16

---

Published in Transactions on Machine Learning Research (11/2025)
Xuezhi Wang, Jason Wei, Dale Schnurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. URL
https://arxiv.orr/abbel org/abs/2203 11171
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can 1lms express their uncertainty? an empirical evaluation of confidence clicitation in llms, 2024. URL https://arxiv.org/ abs/2306 13063
Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. Luq: Long-text uncertainty quantification for llms, 2024. URL fttpa::/aaxxs..rr//abb////
20279.
Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, and Luoyi Fu. Enhancing uncertainty- bbsed hallucination detection with stronger focus, 2023, URL http:://arxavv.or//bb/2231111111 13230
13230
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert, 2020. URL http://awww..r///bb//////0/.00.0
17

---

Published in Transactions on Machine Learning Research (11/2025)
A LLM-as-a-Judge Prompt
Our LLM-as-a-Judge scorer used the following instruction prompt:
Question: [question], Proposed Answer: [answer].
How likely is the above answer lo be correct? Analyze the answer and give your confidence in this answer between (lowest) and 100 (highest), with 100 being certain the answer is correct, and O being certain the answer is incorrect. THE CONFIDENCE RATING YOU PROVIDE MUST BE BETWEEN O and 100. ONLY RETURN YOUR NUMERICAL SCORE WITH NO SURROUNDING TEXT OR EXPLANATION.
# Example
## Data analyze
Question: Who was the first president of the United States?, Proposed Answer: Benjamin
Franklin.
## Your response
4 (highly certain the proposed answer is incorrect)
# Example
## Data to analyze
Question: What is 2+2?, Proposed Answer:
## Your response
99 (highly certain the proposed answer is correct)
To ensure normalized confidence score consistent with the other scorers, we normalize the value returned by the LLM judge be between and 1. The capitalization and repeated instructions, inspired by Wang et al. 2024), are included ensure the LLM correctly follows instructions.
B Ensemble Tuning
We outline method for tuning ensemble weights for improved hallucination detection accuracy. This approach allows for eustomizable component-inportaace that can be optimized for specific use case. In practice, tuning the ensemble weights requires having a 'graded' set of original LLM responses which indicate whether a hallucination present in each response
19 Given a set of n prompts, denoted as x:
41
T2
x
(13)
Tm
denote corresponding correct reference answers as y";
yi
y"
(14)
19Grading responses may accomplished computationally certain tasks, e-g. multiple choice questions. many cases, this will require human grader manually evaluate the responses.
However, in
18

---

Published in Transactions on Machine Learning Research (11/2025)
original LLM responses y,
1/2
y =
(15)
yr
and candidate responses across all prompls with the matrix Y:
Y1
U11
V12
9im
Y2
Ú21
D22
72m
Y
(16)
Vn
On1
Un2
From
Analogously, we denote the vectors of ensemble confidence scores, binary ensemble hallucination predictions, and corresponding ground truth values respectively
w)
8(y2: D2, w)
s(y; Y, =
(17)
8(yni Fns Inn w)
Myyi 81, 21, W,
h(v2: 22 T,
h(y; Y,K,w,r)
(18)
(yyn Yn: Ing w,
and
h(y1: v.
h(y2; vē x22)
h(y:y",x)
(19)
(yys yi v.n)
Modeling this problem as binary classification enables us to tune the weights of our ensemble classifier using standard classification objective functions. Following this approach, we consider two distinct strategies to tune ensemble weights Wi...., Wx: threshol--agnostic optimization and threshold-aware optimization.
Agnossicc-Agnooticc Weights Optimization. Our first ensemble tuning strategy uses a threshold- agnostic objective function for tuning the ensemble weights. Given set of prompts, corresponding original LLM responses and candidate responses, the optimal set of weights, w is the solution to the following problem:
W = arg max wEW
argmaxx(ssy;; S(s(y; h(y,",)
(20)
where
W ((11, wx): wk 1, wx 4 1,..,, K}
(21)
is the support of the ensemble weights and S is a threshold-agnostic classification performance metric, such as area under the receiver-operator characteristic curve (AUROC).
After optimizing the weights, we subsequently tune the threshold using a threshold-dependden objective function. Hence, the optimal threshold, 7", is the solution to the following optimization problem:
19

---

Published in Transactions on Machine Learning Research (11/2025)
arg Y, w), h(ys"" rE(0,1)
(22)
where threshold-dependent classification performance metric, such as F1-score.
Threshold-Awars Weights Optimization. Alternatively, practitioners may wish jointly optimize en- semble weights and classification threshold using the same objective. This type of optimization relies on a threshold-dependentt objective. We can write this optimization problem as follows:
max B(h(y;; h(y,) wEW,rE(0.1)
(23)
where terms follow the same definitions as above.
C
Response Grading
For each task-type type, the LLM is instructed to output its response in a specific form. We therefore instantiate h(.) with task-specific graders aligned that form. Unless noted, grading case-insensitive and trims whitespace 20
Math (numeric answer only). For math questions, the LLM provided the following instruction: "Return only the numerical answer with additional text." Let yi be the model output. Let int(y;) extract the leading integer substring from y/ if present; otherwise return Ø. Let y E Z be the correct integer answer.
0
if int(y:) Vi. otherwise.
1math (yi vi)
Non-numeric missing integer outputs are graded as incorrect.
Multiple choice (letter only). For multiple choice questions, the LLM is provided the following instruc- tion: "Return only the letter of the response with no additional text or explanation." Let. norm (1k)) e {A, B, C, D, E} denote response after normalization to uppercase and trimming. Let y E (, be the correct lotter.
{
if normme = y: otherwise.
hme (B4 )t)
Any invalid response based on the provided instruction, i.c. normme(ys) & (A, is graded as incorrect.
Short-answer (keyword match). Lastly, for short-answer questions, the LLM is instructed as follows: "Return only the answer as concisely as possible without providing an explanation." Let y = ........ vit be the set of acceptable answers Define normalization function norm that lowercases and trims yi- Let contains(u,, v) indicate that string contains string contiguous substring.
( i of y s.t. contains(norm...())) norm.a (v))),
h hul())y))
otherwise.
If output includes additional text, grading is based on the presence of any acceptable answer string,
20These automatic graders not be perfectly accurate in every instance; residual errors arise when model outputs deviate from the specified response format. Their effectiveness therefore depends instruction adherence, and they should be viewed practical proxies for answer-key bused factual correctness.
The short-answer datascts contain multiple versions of the same correct answer allow for phrasing flexibility. For example, acceptable answers the question "What Tobias Lindholm's occupation?" include ("film director". "movie director". "director", "motion picture director". "screenwriter". "scenarist", "writer". "screen writer", "script writer", "scriptwriter")
20

---

Published in Transactions on Machine Learning Research (11/2025)
D Additional Experiments: Number of Candidate Responses vs. Black-Bor Box UQ Performance
To investigate the effect of number candidate responses m on the performance black-box scorers, we re-compute all black-box confidence scores for m 1,3,5, 5, 10, 15 for each our 24 LLM-dataset scenarios. We compute the scorer-specifie AUROC value for each value of m and evaluate the impact of number of candidate responses hallucination detection performance. These results are depicted in Figure
NCP
NCS
EMR
BSC
Gemini-22 2.5- 3-Flash
Gemini-2.5 3-FFashhLiit
GPT-4e -P---iimini
GPT-4o
0.75
0..5
0.76
0.75
NQ-Open
8.00
0.63
0/00
0.65
0.99
0.55
0.96
0.55
15
0.00
01.50
0.00
0.80
PopQ4
0.70
0.75
0.70
0.70
0.700
0.70
0.70
6/01
0..00
0.00
0.80
GSM8K
0.55
13.5
SVAMP
0.6
0.6
0.0
A12-ARC
0.7
0.9
0.8
2.5
0.5
05-
0.5
0.8
0.5
CSQA
0.7
0.7
sumpied Essposses
sumpled resporeess
Number campled cospotees
Number respocaes
Figure& 4: Hallucination Detection AUROC by Number of Sampled Responses
21

---

Published in Transactions on Machine Learning Research (11/2025)
Overall, the results indicate that hallucination detection performance of the black-box scorers improves considerably with number of candidate responses m. For instance, hallucination detection AUROC of the various black-box scorers on GPT-4o responses on CSQA improve from 0.54-0.57 with m 1 to 0.75-0.8 with m = 15. In particular, these performance improvements occur approximately monotonically with diminishing returns higher m, consistent with findings from previous studies (Kuhn et al. 2023 Manakul ct al., 2023: Lin al.) 2024 Farquhar ct al. 2024). This general trend is consistent across all 24 scenarios, with only few exceptions, notably BSC for GPT-4o responses on GSM8k, and BSC, NCS, and EMR for GPT-4o responses on AI2-ARC.
22


Reviews:
[{"review_id": "b96OaoIs9t", "reviewer": "Reviewer_BYp9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of Hallucinations in Large Language Models (LLMs) by developing a comprehensive approach to uncertainty quantification (UQ). The work systematically investigates three distinct UQ techniques: black-box UQ, white-box UQ, and LLM-as-a-Judge, with each technique outputting a confidence score for the LLM's response. A key contribution is the introduction of an ensemble method that fuses these individual scores using a weighted average. The weights for this ensemble are specifically tuned to maximize the F1 score as a performance metric. The efficacy of this novel, combined method is thoroughly validated through extensive experiments on several standard LLM question-answering benchmarks.", "claims_and_evidence_yn": "No", "claims_explanation": "The claims of efficacy are currently not fully supported due to a critical dependency on an unverified ground truth function (h). All performance measurements, including the evaluation of individual techniques and the tuning of the ensemble's weights, rely on a process, h(y_i), used to \"grade\" LLM responses and determine the presence of a hallucination.\n\nThe paper lacks clarity regarding the precise nature and accuracy of this function h. Furthermore, if this function is already highly effective at predicting hallucinations, the authors must clarify the necessity of developing separate UQ techniques or the complex ensemble model to approximate a solution already provided by h. This crucial methodological step needs to be fully described and justified.", "audience_yn": "Yes", "audience_explanation": "Yes, the findings of this paper would be of significant interest to a substantial portion of TMLR's audience, as the work addresses one of the most critical and timely challenges in modern AI: Large Language Model (LLM) reliability and safety.", "requested_changes": "1. Justify and Detail the Ground Truth Function (h): The authors must fully describe the ground truth function (h) used to label hallucinations. This description must include a detailed justification for its accuracy and a clear, logical explanation for why this function cannot be directly employed as the primary hallucination predictor, thus necessitating the development of the UQ techniques and ensemble model.\n\n2. Integrate the Prompt (x) into the Classifier Input: The proposed hallucination classification or uncertainty quantification mechanism should be revised to include the input prompt (x) as an input, rather than relying solely on the LLM's output (y)", "broader_impact_concerns": "The work, while aimed at improving LLM reliability, introduces ethical concerns primarily related to the risk of misplaced trust and \"confidence laundering.\" By generating a seemingly objective, quantitative confidence score through Uncertainty Quantification (UQ), the ensemble method risks convincing users and automated systems to over-rely on the LLM's output, especially when the score is high. If the UQ system is imperfectly calibrated, a high confidence score could falsely legitimize a residual hallucination or bias, thereby elevating the operational risk in sensitive applications like finance, law, or medicine. Consequently, the authors must provide a Broader Impact Statement that explicitly addresses these risks, clarifies that the confidence score is a measure of model uncertainty and not a guarantee of factual truth, and strongly warns against deploying the system in high-stakes environments without external human review to counteract this potential erosion of critical oversight.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "icovUvr9Th", "reviewer": "Reviewer_DY7K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I consider this paper to make a useful contribution to addressing the challenge of Large Language Model (LLM) reliability in practical settings.\n\nA central aspect of the paper is its presentation of a framework for zero-resource hallucination detection at the time responses are generated by LLMs. This approach is relevant for real-time monitoring of LLM outputs in production, as it does not rely on external databases, ground truth texts, or internet access.\n\nWithin this framework, the authors have adapted various existing Uncertainty Quantification (UQ) techniques—including black-box UQ, white-box UQ, and LLM-as-a-Judge methods—and converted their outputs into standardized response-level confidence scores ranging from 0 to 1. This standardization helps in making different UQ methods more consistent and comparable.\n\nThe paper also introduces a tunable ensemble approach. This method allows for combining individual confidence scores through a weighted average, with the weights being adjustable based on a provided set of graded LLM responses. The intent here is to optimize hallucination detection for specific use cases. The ensemble is also designed to be extensible, allowing for the inclusion of new components as research progresses.\n\nFinally, the paper offers practical guidance for practitioners on selecting appropriate scorers, considering factors like API support for token probabilities, latency requirements, and the LLM's response behavior. This work is complemented by an open-source Python toolkit, which provides implementations of the UQ methods discussed.\n\nStrengths:\n• Practical Orientation: The focus on zero-resource, generation-time hallucination detection directly addresses a need for improving LLM reliability in deployed systems, particularly in sensitive domains.\n• Standardized Approach: The effort to standardize various UQ techniques into a 0-1 confidence score is a practical step, contributing to a more consistent way of assessing LLM output confidence.\n• Tunable Ensemble: The tunable ensemble approach is a notable feature. It generally demonstrates improved performance over individual scorers in the experiments and allows for some customization based on specific use case needs.\n• Empirical Testing and Accessibility: The paper includes experiments across different LLMs and benchmarks, providing some evidence for the framework's effectiveness. The provision of an open-source toolkit facilitates the adoption of these methods.\n\nWeaknesses:\n• Reliance on Existing UQ Concepts: While the framework and ensemble are presented as a synthesis, the underlying black-box, white-box, and LLM-as-a-Judge methods are adaptations of techniques previously described in literature.\n• Limited Generalization of Task Types: The experimental validation primarily uses question-answering benchmarks. The authors note that the findings on scorer performance may not directly extend to other tasks, such as summarization or longer-form text generation.\n• Ensemble Tuning Requires Graded Data: To effectively tune the ensemble, a \"graded\" dataset of LLM responses indicating hallucinations is necessary. This may still require manual effort in many real-world scenarios.\n• Scope of Ensemble Exploration: The experiments were limited to linear ensembles. Further exploration of more complex ensembling techniques could potentially yield additional insights or performance gains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "In my view, the authors have undertaken a structured evaluation, and their findings appear to back their assertions.\n\nFramework for Real-time, Zero-resource Detection: \nThe definitions provided for zero-resource (no external databases, ground truth, or internet access) and generation time (suitable for real-time monitoring) are clear. The experimental setup, where scores are computed on LLM outputs, implicitly confirms these operational parameters.\n\nAdaptation and Standardization of UQ Techniques: \nThe submission claims to adapt existing black-box, white-box, and LLM-as-a-Judge UQ techniques, normalizing them to a 0-1 confidence score. The paper explicitly details how various methods like Exact Match Rate, Length-Normalized Token Probability, and LLM-as-a-Judge scores are calculated and, if necessary, transformed or normalized to fit this. This methodical approach provides clear support for the standardization claim.\n\nTunable Ensemble Performance: \nA core claim is the proposal of a tunable ensemble approach that typically surpasses its individual components. The tuning methodology, involving threshold-agnostic or threshold-aware optimization with graded LLM responses, is clearly outlined. Empirically, the ensemble is shown to outperform its individual components in 18 out of 24 scenarios for AUROC and achieving highest F1-score in 16 out of 24 scenarios. This consistent performance across multiple scenarios, as presented in Tables 1 and 2, provides convincing evidence for its general superiority.\n\nPractical Guidance and Toolkit: The python code uploaded is readable and support the experiments.\n\nThe evidence is accurate as it stems from a detailed experimental setup with specified LLMs, datasets, and metrics. It is convincing due to the breadth of evaluation scenarios and the consistent trends observed. \n\nThe acknowledgment of limitations, such as focusing on question-answering tasks and linear ensembles, also adds to the paper's credibility.", "audience_yn": "Yes", "audience_explanation": "I would assess this as yes.\n\nThe paper addresses a recognized challenge in LLM development: hallucination, particularly relevant for models deployed in high-stakes fields like healthcare and finance. Its proposed framework offers a method for zero-resource, real-time hallucination detection, which is a practical concern for monitoring LLM systems in production environments.\n\nThe tunable ensemble approach for combining various uncertainty quantification techniques seems to generally outperform individual methods and can be customized for specific use cases, which could appeal to researchers interested in model reliability and optimization.\nThe availability of an open-source toolkit for implementing these methods also makes the work more accessible and actionable for the community", "requested_changes": "Here are my requested changes as feasible:\n\n1. Clarify the \"Zero-Resource\" Scope: Please elaborate slightly on how the \"zero-resource\" claim specifically applies to the LLM-as-a-Judge component. A brief clarification on whether the judge LLM's own knowledge base impacts this classification would be helpful.\n2. Enhance Guidance on Graded Data Acquisition: While the need for graded data for ensemble tuning is mentioned, expanding on practical strategies for obtaining or scaling these datasets, or discussing associated trade-offs, would make the tunable ensemble more accessible for real-world implementation.\n3. Explain Task Scope Limitations: For the acknowledged limitation regarding generalization to summarization or other long-form tasks, a brief explanation of why the findings might not extend (e.g., different hallucination types, signal changes) would be beneficial.\n4. Briefly Outline Future Ensemble Directions: Since only linear ensembles were explored, a short note on potential non-linear ensemble techniques for future work, and why they might offer further improvements, would be a valuable addition.\n\nThank you for your considerations here", "broader_impact_concerns": "The contribution here is largely positive in addressing LLM hallucination. Some general considerations though\n1. Bias Amplification: If the UQ scorers or underlying LLMs have biases, filtering low-confidence responses could unfairly block accurate information for certain groups, thus amplifying existing societal biases.\n2. Misinterpretation/Over-reliance: Confidence scores, especially in \"high-stakes domains,\" could be misinterpreted as absolute truth or lead to over-reliance, potentially diminishing critical human oversight.\n3. Information Access: Aggressive filtering based on confidence thresholds, which depend on \"stakeholder values,\" might inadvertently narrow the diversity of information an LLM provides, even if the intent is to block hallucinations.\n\nThese points highlight ethical implications that need explicit discussion and mitigation strategies.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sm1rRzQKOQ", "reviewer": "Reviewer_XZVQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a practical framework for detecting LLM hallucinations in real-time without needing ground truth data. Its primary contribution is adapting various existing Uncertainty Quantification (UQ) techniques - specifically black-box, white-box, and LLM-as-a-Judge methods - and standardizing them into a unified confidence score. The authors then propose a tunable ensemble that combines these individual scorers as a weighted average. This allows a user to optimize the hallucination detector for their specific LLM and data. The authors provide an open-source Python toolkit for this system and experimentally compare their customized ensemble approach to the other scoring methods.\n\nThe framework's key strengths are its practicality (it is \"zero-resource\"), its flexibility, and its accessibility via the provided toolkit. Moreover, they test the different approaches in a standardized and clean experimental setup, explaining the pros and cons of each method. The main weaknesses are that all experiments were limited to question-answering tasks (so the findings may not generalize to other tasks succh as summarization) and that only a simple ensemble method (linear combination) was implemented.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s central thesis is that its proposed tunable ensemble outperforms any single Uncertainty Quantification scorer. This claim is convincingly validated through extensive experiments, covering several different LLMs and several QA benchmarks. The results show that the ensemble achieved the highest hallucination detection performance in most of the scenarios.", "audience_yn": "No", "audience_explanation": "Researchers and practitioners who aim to deploy LLMs, and who are interested in understanding hallucination rates, can learn about the different approaches outlined in this paper from their original publications. The method suggested by this paper- tuning an ensemble - is not particularly novel. This practice is common in many other contexts, and applying it specifically to hallucination detection is a natural step that others have likely already considered or implemented.", "requested_changes": "Due to the fundamental lack of novelty, there are no feasible changes I can recommend.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)