Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification (2024)

Abstract: Large language models (LLMs) are notorious for hallucinating, i.e., producing erroneous claims in their output. Such hallucinations can be dangerous, as occasional factual inaccuracies in the generated text might be obscured by the rest of the output being generally factually correct, making it extremely hard for the users to spot them. Current services that leverage LLMs usually do not provide any means for detecting unreliable generations. Here, we aim to bridge this gap. In particular, we propose a novel fact-checking and hallucination detection pipeline based on token-level uncertainty quantification. Uncertainty scores leverage information encapsulated in the output of a neural network or its layers to detect unreliable predictions, and we show that they can be used to fact-check the atomic claims in the LLM output. Moreover, we present a novel tokenlevel uncertainty quantification method that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use. Our method Claim Conditioned Probability (CCP) measures only the uncertainty of a particular claim value expressed by the model. Experiments on the task of biography generation demonstrate strong improvements for CCP compared to the baselines for seven LLMs and four languages. Human evaluation reveals that the fact-checking pipeline based on uncertainty quantification is competitive with a fact-checking tool that leverages external knowledge.

heading: 1 Introduction
text: Large language models (LLMs) have become a ubiquitous and versatile tool for addressing a variety of natural language processing (NLP) tasks. People use these models for tasks including information search (Sun et al., 2023b), to ask medical
♢ Equal contribution
questions (Thirunavukarasu et al., 2023), or to generate new content (Sun et al., 2023a). Recently, there has been a notable shift in user behavior, indicating an increasing reliance on and trust in LLMs as primary information sources, often surpassing traditional channels. However, a significant challenge with the spread of these models is their tendency to produce “hallucinations”, i.e., factually incorrect generations that contain misleading information (Bang et al., 2023; Dale et al., 2023). This is a side-effect of the way modern LLMs are designed and trained (Kalai and Vempala, 2023).
LLM hallucinations are a major concern because the deceptive content at the surface level can be highly coherent and persuasive. Common examples include the creation of fictitious biographies or the assertion of unfounded claims. The danger is that a few occasional false claims might be easily obscured by a large number of factual statements, making it extremely hard for people to spot them. As hallucinations in LLM outputs are hard to eliminate completely, users of such systems could be informed via highlighting some potential caveats in the text, and this is where our approach can help.
Fact-checking is a research direction that addresses this problem. It is usually approached using complex systems that leverage external knowledge sources (Guo et al., 2022; Nakov et al., 2021; Wadden et al., 2020). This introduces problems related to the incomplete nature of such sources and notable overhead in terms of storing the knowledge. We argue that information about whether a generation is a hallucination is encapsulated in the model output itself, and can be extracted using uncertainty quantification (UQ) (Gal et al., 2016; Kotelevskii et al., 2022; Vazhentsev et al., 2022, 2023a). This avoids implementing complex and expensive factchecking systems that require additional computational overhead and rely on external resources.
Prior work has mainly focused on quantifica-
9367
tion of uncertainty for the whole generated text and been mostly limited to tasks such as machine translation (Malinin and Gales, 2020), question answering (Kuhn et al., 2023), and text summarization (van der Poel et al., 2022). However, the need for an uncertainty score for only a part of the generation substantially complicates the problem. We approach it by leveraging token-level uncertainty scores and aggregating them into claim-level scores. Moreover, we introduce a new token-level uncertainty score, namely claim-conditioned probability (CCP), which demonstrates confident improvements over several baselines for seven LLMs and four languages.
To the best of our knowledge, there is no previous work that has investigated the quality of claim-level UQ techniques for LLM generation. Therefore, for this purpose, we construct a novel benchmark based on fact-checking of biographies of individuals generated using a range of LLMs. Note that different LLMs produce different outputs, which generally have higher variability than, e.g., outputs in such tasks as machine translation or question answering. Therefore, we compare the predictions and uncertainty scores to the results of an automatic external fact-checking system FactScore (Min et al., 2023). Human evaluation verifies that our constructed benchmark based on FactScore can adequately evaluate the performance of the uncertainty scores.
Our contributions are as follows: • We propose a novel framework for fact-
checking LLM generations using token-level uncertainty quantification. We provide a procedure for efficiently estimating the uncertainty of atomic claims generated by a whitebox model and highlighting potentially deceptive fragments by mapping them back to the original response. • We propose a novel method for token-level uncertainty quantification that outperforms baselines and can be used as a plug-in in a factchecking framework. • We design a novel approach to evaluation of token-level UQ methods for white-box LLMs based on fact-checking, which can be applied to other white-box LLMs. • We provide an empirical and ablation analysis of the method for fact-checking of LLM generations, and find that the uncertainty scores we produce can help to spot claims with factual errors for seven LLMs over four languages:
English, Chinese, Arabic, and Russian. • The method is implemented as a part of the
LM-Polygraph library (Fadeeva et al., 2023). All the code and data for experiments is publicly available1.

heading: 2 Related Work

heading: 2.1 Fact-Checking LLM Generations and Detecting Hallucinations
text: The problem of hallucinations has made factchecking of LLM outputs a prominent topic in the research community and resulted in a surge of publications on the topic. Chern et al. (2023) present Factool – a task and domain agnostic framework for hallucination detection that leverages GPT for claim extraction and verification. Manakul et al. (2023) suggest to sample multiple outputs from black-box LLMs and evaluate how similar the sampled responses are using the external model. Varshney et al. (2023) detect LLM hallucinations by extracting key parts of output using an external model and estimating their uncertainty based on logits. The most uncertain parts are verified using an external knowledge source. Pan et al. (2023) propose to fact-check complex statements by decomposing them into simpler subtasks and generating reasoning programs to verify these statements. Min et al. (2023) present a methodology for evaluating long LLM-generated texts by decomposing them into simple atomic statements and further verifying them against some knowledge source. Several subsequent works further optimize components of the knowledge-based fact-checking pipelines (Wang et al., 2024a,b).
In contrast to previous work that leverages external knowledge sources for fact-checking (a database or another LLM), our work is the first to investigate token-level UQ methods for this task using the LLM and its outputs only.

heading: 2.2 Uncertainty Quantification of LLM Generations
text: UQ techniques for LLM generation can be classified into five major categories (Fadeeva et al., 2023). Information-based methods leverage the probability distribution of generated tokens and usually do not require any additional models. In this category, we can include methods such as perplexity (Fomicheva et al., 2020), mean token entropy (Fomicheva et al., 2020), point-wise mutual
1https://github.com/IINemo/lm-polygraph
CCP accurately identifies the incorrectly specified number of awards (in red ), whereas Maximum Probability erroneously highlights the claim that is actually correct.
information (PMI) (Takayama and Arase, 2019), and conditional PMI (van der Poel et al., 2022).
Another category of methods is based on density estimation of latent instance representations. A typical example in this category is Mahalanobis distance (Lee et al., 2018) and its various modifications (Ren et al., 2023; Vazhentsev et al., 2023b). The disadvantage of such methods is the need for access to the LLM training data in order to fit external density models, which is problematic for most general-purpose LLMs.
Ensembling and Monte Carlo dropout methods are based on the lexical diversity of multiple outputs sampled from one or multiple versions of LLMs for a single query (Malinin and Gales, 2020; Fomicheva et al., 2020). Their main drawback is that they require many predictions, which makes them too computationally and memory intensive for practical purposes. Additionally, it is difficult to apply these techniques to quantify uncertainty of text fragments such as claims, as different samples may seriously diverge.
It has recently been shown that LLMs can reflexively estimate the confidence of their generations simply by asking themselves about the truthfulness of their output (Kadavath et al., 2022). This can work better than analyzing the probability distribution for the original prediction, but requires a second pass of inference, feeding the original output as a part of the query.
Finally, there is a group of methods that leverages the diversity of meanings that the LLM generates for a given query. This group includes semantic entropy (Kuhn et al., 2023) and various scores based on the analysis of the similarity matrix between outputs (Lin et al., 2023).
UQ methods can also be classified into whitebox and black-box (Lin et al., 2023) approaches,
depending on the required access to LLM itself and its outputs. Black-box techniques do not require any other input except generated texts.
The method in our work can be attributed to the information-based group, and can be applied only to white-box LLMs because it requires access to the probability distribution of the generated tokens. Compared to other techniques, it offers a novel approach to post-processing the probability distribution and is specifically designed to quantify uncertainty of output fragments, such as atomic claims and individual words.

heading: 3 Fact-Checking Pipeline
text: The fact-checking pipeline (see Figure 4) starts with splitting a generated text into atomic claims, e.g. using a much smaller model fine-tuned for this particular task. For experimental evaluations in this work, we follow the FactScore approach (Min et al., 2023), where splitting is implemented via the OpenAI Chat API.
Each atomic claim is matched against the sequence of tokens in the original text with the corresponding probability distributions. Then we calculate token-level uncertainty scores and aggregate them into the claim-level uncertainty.
Finally, the claim-level uncertainty scores are compared against a threshold obtained on a validation set to determine whether the claim should be highlighted for the end-user as unreliable. Individual tokens can be a part of multiple atomic claims. If the token belongs to a reliable and unreliable claim at the same time, it is not highlighted. An example visualization is presented in Figure 1.

heading: 4 Uncertainty Quantification
text: In this section, we first provide background on common UQ methods that can be used at the token level,
then delve into our Claim-Conditioned Probability (CCP) token-level method, and finally describe how token-level uncertainties are aggregated into a claim-level score.
Autoregressive language models generate text token by token. In this work, we will operate on the level of words and without loss of generality suppose that the autoregressive distribution at each step generates the random word Xj ∼ P (· | x<j), where x<j is the text generated before the word at position j. We also denote by xj the generated word at position j and by x1:j = x<j ◦ xj a text composed of words at positions 1 to j. For example, in the case of greedy generation, xj = argmaxx P (x | x<j), where xj is the most probable realization of Xj . Let us also denote by C a set of indices of words corresponding to a particular atomic claim.

heading: 4.1 Claim-Level UQ Baselines
text: We note that for UQ to be practical, it needs to be fast. Therefore, we do not consider methods such as deep ensembles (Lakshminarayanan et al., 2017), due to their significant computational overhead.
Maximum Probability represents a basic approach to UQ, where we simply treat the probability of the most likely generation as a confidence score:
MP (C) = 1− ∏
j∈C P (xj | x<j). (1)
Perplexity is a common metric used to evaluate the performance of LLMs. Lower perplexity indicates that a model’s probability distribution better predicts a sample. It is computed as the average
negative log probability of generated tokens that belong to the claim C:
Perp(C) = exp ( − 1|C| ∑ j∈C logP (xj | x<j) ) . (2)
Maximum Entropy of a token in the claim:
Ent(C) = maxj∈C H(· | x<j), (3)
where H(· | x<j) is the entropy of the autoregressive distribution of the current token. Preliminary experiments indicated that simply getting the maximum of the token entropies in a claim noticeably outperforms other aggregation techniques like average or minimum. It is also generally a slightly better baseline than perplexity.
P(True), similar to (Kadavath et al., 2022), measures the uncertainty of the claim by asking the LLM itself whether the generated claim is true or not. The confidence is the probability of the first generated token y1 being equal to “True”:
P (True) = 1− P (y1 = “True”). (4)
While some work has reported that this technique outperforms other baselines, the big drawback is that one needs to run the original LLM twice.

heading: 4.2 Claim-Conditioned Probability
text: In this subsection, we propose a novel method for token- and claim-level uncertainty quantification.

heading: 4.2.1 Motivation and Theoretical Background
text: When an LLM generates an output, it faces various types of uncertainty reflected in the token distribution of the current generation step (see Figure 5 for
an example). We identify three distinct types of uncertainty:
(1) Claim type/order uncertainty: What claim to generate on the current step? For example, on the current step, an LLM might hesitate between generating a year of graduation of a person and a field of study. A different order of claims, missing claims, or different types of generated claims do not make produced text less factual. Therefore, when performing fact-checking, we should not take this type of uncertainty into account.
(2) Surface form uncertainty: What synonyms or hypernyms to use when generating a claim (e.g. “art” or “painting”)? Different surface forms also do not make the text less factual — they might change the style, but not the underlying meaning of the text. Therefore, this type of uncertainty is also not relevant for fact-checking.
(3) Claim uncertainty: What specific piece of information to relay for a particular claim type? For example, an LLM might not be sure which field of study to generate, producing a token distribution with multiple highly-probable variants, such as “painting”, “acting”, “sculpture”. Similarly, for the year of graduation, an LLM might produce a distribution with various potential years. This uncertainty is relevant for fact-checking, because if the model is not sure about the information it relays, there might be a high chance of a factual mistake.
Two out of the three types of uncertainty are irrelevant for fact-checking and only introduce noise into the final score. We propose a new UQ method that ignores the first two types of uncertainty and focuses only on the third one, namely Claim-Conditioned Probability (CCP): CCP (xj) = P ( Meaning(x1:j)|x<j ,ClaimType(x1:j) ) . (5)
Here, ClaimType(x1:j) represents a claim type of the generated sequence x1:j and Meaning(x1:j) is a function that maps xj into its meaning given the previous words in a sentence x<j , so that various surface forms with a similar meaning for xj are mapped to a single categorical variable.
Conditional probability can be rewritten using unconditional probabilities:
P ( Meaning(x1:j) | ClaimType(x1:j), x<j )
= P (Meaning(x1:j),ClaimType(x1:j) | x<j)
P (ClaimType(x1:j) | x<j) .
Assuming that each meaning in a word distribution can correspond to only a single claim type, the joint
probability is the same as the meaning probability P (Meaning(x1:j),ClaimType(x1:j) | x<j) = P (Meaning(x1:j) | x<j).
Meaning probability in turn sums from the probabilities of word alternatives xkj that correspond to the same meaning: P (Meaning(x1:j) | x<j) =∑
xkj∈M(xj) P (x k j | x<j), where we say that xkj ∈ M(xj) if Meaning(x1:j) = Meaning(x<j ◦ xkj ). In the same way, the probability of a claim type sums from probabilities of all meanings and transitionally from probabilities of words that correspond to the particular claim type: P ( ClaimType(x1:j) ) =∑
xlj∈CT (xj) P (x l j | x<j), where we denote by xlj ∈ CT (xj) an event such that ClaimType(x1:j) = ClaimType(x<j ◦ xlj). Therefore, equation (5) can be rewritten as follows:
CCP (xj) =
∑ xkj∈M(xj) P (x
k j | x<j)∑
xlj∈CT (xj) P (x l j | x<j)
. (6)
The meaning function and the construction of the set of words that belong to the same claim type can be implemented in various ways. We outline our approach in Section 4.2.2.
Previously-proposed UQ methods have partially accounted for some of the types of uncertainty described above. For example, semantic entropy (Kuhn et al., 2023) accounts for uncertainty in semantically-equivalent groups, which helps to alleviate the surface-form uncertainty. However, in our method, we additionally remove the impact of claim-type uncertainty.

heading: 4.2.2 Implementation
text: We implement CCP using NLI at the word level. We compare the original claim and the claim, in which the target word is replaced by its alternatives from the autoregressive distribution.
The distribution Xj at the position j is approximated by top-K alternatives {xkj }Kk=1 with x1j ≡ xj . We replace xj with its alternatives xkj and obtain new instances x<j ◦ xkj , k = 1, . . . ,K. Each new instance is compared against the original prediction x1:j = x<j◦xj using an NLI model. We define NLI(xkj , xj) := NLI(x<j ◦ xkj , x1:j), where NLI(x<j ◦ xkj , x1:j) means application of the NLI model to the text fragments x<j ◦ xkj and x1:j .
The outcome of the NLI procedure is one of three labels: entail (‘e’), contradict (‘c’), or neutral (‘n’). If the new instance entails the origi-
nal prediction NLI(xkj , xj) = ‘e’, then we consider x<j ◦ xkj has the same meaning with x1:j (xkj ∈ M(xj)) and corresponds to the same claim type (xkj ∈ CT (xj)).
If the new instance contradicts the original prediction NLI(xkj , xj) = ‘c’, then we consider x<j ◦ xkj has a different meaning with x1:j (xkj /∈ M(xj)), but corresponds to the same claim type (xkj ∈ CT (xj)). Otherwise, if the new instance is neutral w.r.t. the original prediction NLI(xkj , xj) = ‘n’, then we consider that x<j ◦ xkj does not correspond to the same claim type as x1:j (xkj /∈ CT (xj)). Thus, equation (6) for CCP can be written as follows:
CCPword(xj) =
∑ k:NLI(xkj ,xj)=‘e’
P (xkj | x<j) ∑
k:NLI(xkj ,xj)∈{‘e’,‘c’} P (xkj | x<j)
.
For practical considerations, we consider that CCP for function words is always equal to 1. In our experiments, we base this determination on the stop word list from NLTK (Bird and Loper, 2004).
We note that most transformer LLMs generate sub-word tokens instead of whole words. To obtain distributions for whole words, we generate one or multiple tokens using beam search with K beams.
To obtain CCP-based claim-level uncertainty, we simply take the product of CCPs of each word from the claim C:
CCPclaim(C) = 1− ∏
j∈C CCPword(xj). (7)
An example of calculating the CCP for a claim is presented in Figure 2. Other detailed examples of CCP calculation are available in Appendix B.

heading: 5 Benchmark for Evaluation of Claim-Level UQ Methods
text: We evaluate claim-level UQ techniques and their ability to spot hallucinations on the task of generating biographies. In relevant previous work (Manakul et al., 2023), the authors generate biographies with one LLM (GPT-3), manually annotate sentences for factuality, and quantify uncertainty of a different “proxy” model. Factuality labels are then used to evaluate the quality of uncertainty scores. We argue that such an approach based on a proxy model introduces a big discrepancy between the generated text and what a proxy LLM actually wants to generate, which results in biased UQ evaluation results. To make the evaluation as close
as possible to the real-world scenario, we allow unrestricted generation of biographies from LLMs.
Unrestricted generation complicates automatic evaluation of the fact-checking pipeline, because obtaining gold standard annotation requires manual annotation of all outputs from each model. Therefore, in addition to manual annotation, we annotate claims in generated texts automatically using FactScore – a fact-checking tool (Min et al., 2023), which has access to an external knowledge source. Using FactScore, enables completely automatic evaluation and allows us to scale up experiments.
We generate LLM responses in English, Chinese, Arabic, and Russian to 100 biography prompts. The typical biography prompt is Give me a biography for <person name> in different languages. The set of people was generated by asking GPT-4 to list the most famous people since 1900. The maximum generation length is set to 256 tokens. If the last sentence of the generation is unfinished (i.e. does not end with any punctuation), it is discarded. We generate responses for the following LLMs: (for English) Vicuna 13b (Zheng et al., 2023), Mistral 7b (Jiang et al., 2023), Jais 13b (Sengupta et al., 2023), and GPT-3.5-turbo (Ouyang et al., 2022); (for Chinese) Yi 6b (Young et al., 2024); (for Arabic) Jais 13b and GPT-4; (for Russian) Vikhr-instruct-0.2 7b (Nikolich et al., 2024).
We decompose the generated text into atomic claims using GPT-4. For each claim, we map all its words back to generated text to access the corresponding token logits. Not all claims perfectly match to the original response. For example, for Vicuna 13b around 5% of all claims do not successfully match because ChatGPT abstained to respond or outputted words not present in the original. We consider only successfully matched claims.
For English, we are able to perform annotation completely automatically: each atomic claim is classified by FactScore as supported or not supported. The underlying FactScore model is “retrieval+ChatGPT” with a dump of Wikipedia articles as an external knowledge source. We also manually annotated 100 claims from English biographies produced by the Vicuna 13b model, 183 claims from Arabic biographies, 146 claims from Russian biographies, and 1603 claims in Chinese. Each of the statements was checked by two annotators with access to the corresponding Wikipedia article. The final label is set to “supported” only if both annotators label it as so.
The statistics of the resulting datasets with
automatically-labeled claims in English are presented in Table 6, and the statistics of manually annotated datasets are presented in Table 7. The majority of claims in the model output are correct, with 6–29% of hallucinations. The automatic pipeline for evaluation of UQ methods using FactScore is illustrated in Figure 6 in Appendix C.

heading: 6 Experiments

heading: 6.1 Experimental Setup
text: Fact-checking of atomic claims is framed as a binary classification task, where uncertainty scores serve as predictors of non-factuality, and FactScore or human labels serve as ground truth. The evaluation metric is ROC-AUC and PR-AUC (unsupported claims as a positive class).
For the CCP method, NLI scores are calculated using the DeBERTa-large model (He et al., 2021) fine-tuned for this task2. The number of alternatives used in CCP is K = 10, except for GPT3.5-turbo and GPT-4, as the OpenAI API does not allow us to retrieve more than 5 alternatives from the token distribution. Details on hardware and computational resources spent for experiments can be found in Appendix F.

heading: 6.2 Results for English on the FactScore Annotation
text: The main results of the experiments with FactScore labels for English are presented in Tables 1 and 2.
2https://huggingface.co/microsoft/ deberta-large-mnli
The proposed CCP method outperforms all other UQ techniques for each of the considered LLMs, with only exception in the PR-AUC metric for the GPT-3.5-turbo model, where the P(True) approach exhibits the highest performance. The underperformance of CCP for GPT-3.5-turbo may be attributed to the limited number of token options and their associated logits available through the OpenAI API. The best overall improvement from CPP is obtained for Jais 13b, where it outperforms the closest competitor by 0.07 ROC-AUC and 0.09 PR-AUC.
We further analyze the performance by plotting ROC-AUC as a function of the number of considered sentences from the beginning of the generated text (see Figure 3). We note that the quality of each method decreases as the number of considered sentences increases. This may be due to the fact that the model tends to start the response with easy-toknow and hence reliable claims and as it generates more text, it has to produce more complex and less reliable statements, which is illustrated for the Vicuna 13b model in Figure 8 in the appendix. For the majority of cases, CCP outperforms other methods, except when we consider only the first two and the first five sentences generated by GPT-3.5-turbo.

heading: 6.3 Multilingual Results on Manual Annotation
text: Multilingual results based on manual annotation are presented in Tables 3 and 4 and in Figure 9. Using manual annotation for English, we can also evaluate the performance of FactScore itself. The accuracy of automatic annotation is 77.2%
and ROC-AUC is 0.72. A detailed analysis of FactScore mistakes is presented in Appendix C.3.
For human annotation, the performance of all UQ methods appears to be even slightly higher than for the labels obtained using FactScore (Table 3). Moreover, CCP outperforms FactScore itself by 0.06 ROC-AUC. These results demonstrate that in the task of detecting LLM hallucinations, UQ techniques can be a strong alternative to fact-checking tools with an external knowledge source.
For Chinese, Arabic, and Russian, CCP also outperforms the baselines. For the Chinese Yi 6b model, from Figure 9, we can see that the gap between CCP and the baselines is especially significant for the several first claims. When considering more claims, CCP still clearly outperforms the Maximum Probability baseline, but the P(True) baseline substantially reduces the gap. For Arabic and Jais, CCP outperforms the closest competitor by 0.05 ROC-AUC. On GPT-4 generations for Arabic, the metrics for all methods are low. We explain this observation by a small ratio of nonfactual claims in the GPT-4 output. For Russian generations with the Vikhr model, CCP confidently outperforms Maximum Probability, which is its closes competitor, by 0.05 ROC-AUC.

heading: 6.4 Ablation Studies
text: In this section, we analyze the influence of various CPP components on English biographies annotated with FactScore (Tables 9–12). Details of the experimental setup for each ablation study are presented in Appendix D.
(1) Aggregation of CCPword for obtaining CCPclaim. Besides the product of probabilities, we also tried the normalized product, minimum, and
average probability. All these approaches perform slightly worse than the product (see Table 9).
(2) NLI model. We investigate the influence of the specific NLI model on the performance of CCP. Table 10 shows that CCP’s effectiveness is not critically dependent on the complexity of the NLI model employed. Notably, even a relatively small model with 22M parameters maintains strong performance without any degradation.
(3) NLI context. We analyze what context is sufficient for NLI in CCP (Table 11). In addition to the standard variant in CCP, where we keep the claim that precedes the word in question, we experiment with a single target word without context and the whole sentence that precedes the target word. All variants demonstrate lower performance. No context results in a drop of 0.02 ROC-AUC, and longer contexts – of more than 0.07.
(4) Functional words handling. The results in Figure 12 show that excluding functional words in CCP helps to improve the performance by 0.03 ROC-AUC. This approach also slightly improves the maximum probability baseline, but its performance is still much lower.
(5) The number of alternatives K. Taking K = 5 alternatives instead of K = 10 in CCP decreases ROC-AUC by 0.02. Figure 7 also demonstrates that further decreasing K reduces the performance even more. When increasing K, the performance plateaus at K = 8.

heading: 6.5 Qualitative Analysis
text: In qualitative analysis of uncertainty scores for various generations and models, we note that the maximal probability baseline produces a lot more false positives than CCP. This happens because CCP ignores some types of uncertainty, focusing only on the claim uncertainty. In some cases, CCP also finds false claims overlooked by other methods because ignoring certain types of uncertainty also allows us to reduce the cut-off threshold used to mark claims. An example where we compare CCP with maximal probability is presented in Figure 12, and more examples can be found in Appendix G.

heading: 6.6 Computational Efficiency
text: To demonstrate the computational efficiency of CCP, we compare it to the fastest UQ method – Maximum Probability. Experiments were conducted using a dataset of 100 biographies and Mistral 7b (Jiang et al., 2023). To ensure a fair comparison, we focus solely on the runtime of gen-
erating biographies and calculating the respective uncertainty scores for each claim. Time spent on claim extraction and matching is excluded. The experiments utilized two 32GB V100 GPUs. Each biography was processed in a single batch.
MP does not introduce notable overhead over the generation process, as it only aggregates produced logits. CCP involves running an NLI model for each of 10 token candidates per token position, which introduces some overhead.
Table 5 presents the runtime comparison. We see that NLI does not make a substantial impact. Using the default microsoft/deberta-largemnli model (350M parameters) results in 8% increase of the runtime compared to MP. Using the smaller cross-encoder/nli-deberta-v3-xsmall model (22M parameters), which achieves comparable UQ performance, reduces the computational overhead to only 3%.

heading: 7 Conclusion
text: We presented a novel approach to fact-checking and hallucination detection based on token-level uncertainty quantification. According to human evaluation, our approach is competitive with FactScore, a fact-checking tool that leverages an external knowledge source: we achieve similar or better results with access to only LLM outputs.
We proposed a computationally efficient tokenlevel and claim-level UQ method, Claim Conditioned Probability that outperforms a number of baselines in fact-checking. In this method, we postprocess the word distribution to mitigate the impact of uncertainty related to the variability of surface forms and uncertainty about what claim type to generate on the current step. In the constructed benchmark, where we detect hallucinations in biographies, CCP outperforms other methods for seven LLMs, including GPT-3.5-turbo and GPT-4, and four languages. We also demonstrate that computational overhead of CCP might be as low as 3% of the LLM inference runtime.

heading: Limitations
text: While this work has been conducted according to experimental and methodological best practice, there are several potential limitations.
First, at the core of the approach lies a text entailment classifier. As it was originally pre-trained for a slightly different use-case, more careful analysis of its performance on diverse domains and genres should be carried out.
Second, the current implementation of the method makes use of OpenAI’s GPT models for text segmentation and extraction of atomic facts, similarly to FactScore, which may not be practical in real applications. Replacing these components with cheaper open models should be feasible in principle, but is left for future work.
Third, part of our experimental results rely on a human evaluation, which may be subjective. We tried to mitigate this by creating detailed instructions for the annotators, but a larger-scale study with larger overlap would further strengthen the results.
Fourth, our uncertainty quantification is based on tokens, where taking into account also larger units such as noun or verb phrases as basic units of analysis may be better motivated linguistically.
Fifth, in this work, we do not consider the calibration of CCP scores. We note that calibration on its own does not provide the information about the performance in the practical task that we are interested in – fact-checking. At the same time, CCP could be post-calibrated in the same way as any other probabilities or their surrogates.
Finally, our approach only detects potentially spurious generations. A prominent direction for future research is to modify the generation of an LLM to exclude such spans, while ensuring fluency of a generated text: a simple removal of uncertain claims may result in incoherent sentences.

heading: Ethical Considerations
text: We would like to caution that our method for factchecking is based on uncertainty quantification, and thus it is not bullet-proof, as it only reflects the internal model state of the LLM. Thus, it is of limited utility when it comes to claims that are beyond the time cutoff of the model. Moreover, the LLM might be trained on factually false data, which is beyond our control. It could be also tricked by variations in the prompt.
We also caution that our solution does not eliminate hallucinations; instead, it can be used to highlight risky parts of a text, for humans to take into account. Our intended use is towards raising awareness and promoting human-machine collaboration.
Finally, our method can be misused to unfairly moderate content. Thus, we ask researchers and potential users to exercise due caution.

heading: A Fact-Checking Pipeline

heading: Match Token Logits with

heading: B Example of CCP Calculation
text: Figure 5 demonstrates an example of the LLM generation process and CCP calculation process. CCP quantifies the Claim Uncertainty, not taking into account the Claim order and Surface form uncertainties. As a result, CCP produces better uncertainty scores than Maximum Probability.

heading: C Benchmark Construction Details

heading: C.1 Benchmark Construction Pipeline

heading: Wikipedia
text: Figure 4 presents the suggested general preparation pipeline of a factuality dataset for an arbitrary model and Fact-Checking benchmark. This pipeline was used to generate the biography dataset. Figure 6 presents the detailed pipeline of the biography dataset preparation, based on the general schema.
The prompt for the matching model to map atomic claims to the initial text is as follows: “Given the fact “fact”, identify the corresponding words in the original sentence “sent” that help derive this fact. Please list all words that are related to the fact, in the order they appear in the original sentence, each word separated by comma.”. The prompt for the model that partitions the initial generated text into individual atomic claims, as well as the prompt for the classification model, are taken similar to the paper by Min et al. (2023).

heading: C.2 Datasets and Statistics
text: Since FactScore only supports English, for Arabic, Chinese, and Russian, we generate biographies of
well-known people and annotate them only manually. We also manually annotate 100 English claims generated by Vicuna 13b. The statistics for the annotated datasets are presented in Table 7.
For Arabic, using GPT-4, we generate 100 biographies of people randomly selected from the list of the most visited websites in Arabic Wikipedia. The used Arabic prompt is the translation of: “Tell me the biography of {person name}”. To extract claims, we prompt GPT-4 in the following way: “Convert the following biography into Arabic atomic factual claims that can be verified, one claim per line. Biography is: {biography}”. Arabic biographies and claims are translated into English using Google Translate. It is worth mentioning that almost one-third of the names in the list of person names are foreign, e.g. Donald Trump, Messi, Isaac Newton, Pope Benedict XVI, etc. On average, GPT-4 generates 20 claims from each biography, and random two claims from each biography are verified manually (total = 200 claims).
For Jais 13b experiments, we use the same prompts used for GPT-4. We notice that the biographies generated by Jais 13b are much shorter than the ones generated by GPT-4 (almost half-length). Similarly, we use GPT-4 to extract claims from the generated biographies. On average, biographies generated by Jais 13b have nine claims. Jais 13b generates empty biographies for seven names (out of 100) with response messages like: “I am sorry! I cannot provide information about {name}”, or “What do you want to know exactly?”. Two random claims from each biography are verified manually (total = 186 claims).
For Chinese, we first prompt ChatGPT to generate a list of 100 famous people. Then use the same way as we have done in Arabic, but change the prompt to Chinese, to generate biographies and claims. We use Yi 6b to generate texts and GPT-4 to split them into atomic claims.
For Russian, we conduct a similar approach, prompting ChatGPT to generate a list of 100 famous people and checking the result to obtain representative personalities from different areas such as science, sport, literature, art, government activity, cinematography, heroes, etc. A balanced list of famous people in different professional categories was obtained. For these people, we generate biographies using the Vikhr 7b model (Nikolich et al., 2024).

heading: C.3 FactScore Annotation
text: The statistics of English data annotated using FactScore is presented in Table 6. Here we give examples of the operation of the FactScore automatic markup system, see the Table 8. We hypothesize that the causes of FactScore errors are related to model hallucination (true information is present in the knowledge source, but the model produces incorrect information), lack of context (an excerpt from a Wikipedia article cannot capture all the details), and difficulties with information interpretation (the desired information is present in the knowledge base, but is formulated in different words or in several sentences). All these cases leave room for further improvement of the pipeline. We have also given the results of the proposed CCP method on selected examples. It can be seen that the results of our algorithm are similar to the annotation from FactScore, while CCP does not use external information in calculating the scores.

heading: D Ablation Studies
text: In this section, we perform the ablation study of various CCP components.

heading: D.1 Aggregation
text: Table 9 presents the result of CCP on Vicuna 13b when applying 4 different kinds of aggregation:
CCPprod(F ) = 1− k∏
j=1
CCPword(xj),
CCPlen(F ) = 1− exp ( 1
k
k∑
j=1
logCCPword(xj)
) ,
CCPmin(F ) = 1− min j=1,...,k CCPword(xj),
CCPmean(F ) = 1− 1
k
k∑
j=1
CCPword(xj).
CCPprod shows the best results and is chosen for the final CCP formula.

heading: D.2 NLI Models
text: Table 10 presents the result of CCP on Vicuna 13b when using different NLI models. We selected fine-tuned NLI models from HuggingFace, encompassing a range of sizes from Microsoft’s DeBERTa model (He et al., 2021), a multilingual variant (Laurer et al., 2022), and a model from SentenceTransformers (Reimers and Gurevych, 2019).
Our findings demonstrate that the CCP performance exhibits minimal dependence on the specific NLI model chosen. Notably, even CCP utilizing a relatively small CrossEncoder model with only 22M parameters achieves the highest performance.

heading: D.3 NLI Context
text: Table 11 presents the result of CCP on Vicuna 13b when adding different contexts as the inputs of NLI model:
1. CCPno context calculates NLI using these 2 words as NLI model input;
2. CCPsent pref calculates NLI using the prefix in the sentence from model generation up to the current word and its alternative, as input to NLI model;
3. CCPclaim pref, uses all words in the sentence corresponding to the greedy word, which were matched to the current fact and which comes before the greedy word (see example on Figure 2).
The results for sentence prefix provides too wide context to the NLI model, which results in the NLI model focusing on the contexts instead of the greedy words and its alternatives, thus outputting lots of entailment classes. On the other hand, the method without context does not provide enough context to calculate NLI class with more quality. CCPclaim pref shows better results and is the one chosen for the final CCP formula.

heading: D.4 Functional Words Handling
text: Table 12 presents the result of CCP and Maximum Probability on Vicuna 13b when handling functional words differently. The 2 approaches tested are:
1. CCPconfident and MPconfident are the versions of CCP and MP baselines, which assigns the most confident word-level score of 1.0 to words from NLTK (Bird and Loper, 2004) stopwords list, which is the same as skipping these words form the list of matched words of length k;
2. CCPignore and MPignore handles functional words similar to any other words. MPignore is the baseline used in the main section.
CCPconfident shows better results and is the one chosen for final CCP formula. In contract, Maximum Probability performs worse if specifically handling functional words.

heading: D.5 Number of Alternatives
text: Figure 7 presents the CCP ROC-AUC results on the whole model generation, depending on the number of beams n to run the method with.

heading: E Additional Experimental Results
text: Here we show additional results related to the performance of our pipeline and CCP method. In the Figure 8 we show the dependence of the percentage of supported claims in the generated LLM response (in this case for Vicuna 13b) as a function of its length. As we can see, the LLM produces wrong claims when the generation length increases, which may be due to the generation of additional facts.
In Figure 9, we show the performance quality of our CCP method on Chinese data on the Yi 6b model. We see that, as for other English models, our method outperforms the alternatives over the entire generation length in terms of ROC-AUC.

heading: F Resources and Expenses
text: For a single run of data generation and UQ methods evaluation, we spent 12 days of Nvidia A100 GPU compute time. OpenAI API was used mainly for splitting and matching atomic claims, the total cost for all English models (Mistral 7b, Vicuna 13b, Jais 13b, GPT-3.5-turbo) was $40. The cost of generating English biographies with GPT-3.5-turbo was $13.

heading: G Examples of Results Obtained with CCP and the Maximal Probability Baseline
text: Figures 10, 11 and 12 show examples of the Claim-Conditioned Probability method compared with Maximum Probability. On all examples, our CCP method correctly detects the wrong facts, whereas Maximum Probability mistakenly highlights correct facts.

heading: Future Work
text: Replacing these components with cheaper open models should be feasible in principle, but is left for future work. Third, part of our experimental results rely on a human evaluation, which may be subjective. We tried to mitigate this by creating detailed instructions for the annotators, but a larger-scale study with larger overlap would further strengthen the results. Fourth, our uncertainty quantification is based on tokens, where taking into account also larger units such as noun or verb phrases as basic units of analysis may be better motivated linguistically. Fifth, in this work, we do not consider the calibration of CCP scores. We note that calibration on its own does not provide the information about the performance in the practical task that we are interested in – fact-checking. At the same time, CCP could be post-calibrated in the same way as any other probabilities or their surrogates. Finally, our approach only detects potentially spurious generations. A prominent direction for future research is to modify the generation of an LLM to exclude such spans, while ensuring fluency of a generated text: a simple removal of uncertain claims may result in incoherent sentences