Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators (2024)

Abstract: Large Language Models (LLMs) tend to be unreliable in the factuality of their answers. To address this problem, NLP researchers have proposed a range of techniques to estimate LLM’s confidence over facts. However, due to the lack of a systematic comparison, it is not clear how the different methods compare to one another. To fill this gap, we present a survey and empirical comparison of estimators of factual confidence. We define an experimental framework allowing for fair comparison, covering both fact-verification and question answering. Our experiments across a series of LLMs indicate that trained hidden-state probes provide the most reliable confidence estimates, albeit at the expense of requiring access to weights and training data. We also conduct a deeper assessment of factual confidence by measuring the consistency of model behavior under meaningpreserving variations in the input. We find that the confidence of LLMs is often unstable across semantically equivalent inputs, suggesting that there is much room for improvement of the stability of models’ parametric knowledge. Our code is available at https://github.com/amazonscience/factual-confidence-of-llms.

heading: 1 Introduction
text: A major problem of Large Language Models (LLMs) is that they do not always generate truthful information. Models can hallucinate by convincingly reporting information that is actually false or they are not confident about, or provide factual answers only when prompted in a certain way (Elazar et al., 2021; Lin et al., 2022b; Ji et al., 2023; Wang et al., 2023a; Luo et al., 2023). This behavior can be severely harmful, especially given the current explosion of LLM usage: a lack of truthfulness can lead to spread of misinformation and breaches to user trust (Weidinger et al., 2021; Bender et al.,
* Work conducted during an internship at AWS AI Labs.
2021; Evans et al., 2021; Tamkin et al., 2021). Having a reliable estimate of the model’s confidence over a fact—the degree to which it is expected to have accurate factual knowledge with respect to an input—is key for mitigating this problem (Geng et al., 2023; Tonmoy et al., 2024).
Recently, a number of methodologies for estimating the factual confidence of LLMs were proposed (Lin et al. 2022a; Burns et al. 2023; Kuhn et al. 2023; Azaria and Mitchell 2023; Pacchiardi et al. 2024, among others). However, none of them establishes a unified experimental framework to compare methods. This leaves open questions regarding how aligned the methods are in their estimates, and which are the most reliable ones across models. We aim to fill this gap by providing a
4554
survey of the current state-of-the-art for estimating the factual confidence in LLMs, and performing a systematic empirical comparison of the reliability and robustness of the existing methods.
We introduce an experimental framework, shown in Figure 1, enabling a fair comparison between methods across models and datasets. We adopt two distinct formulations for measuring factual confidence: (i) the probability of a statement to be true, noted P (True)—fact-verification (Thorne et al., 2018; Azaria and Mitchell, 2023), and (ii) the probability of yielding a truthful answer to a query, noted P (I know)—question answering (Kadavath et al., 2022; Yin et al., 2023). Additionally, we categorize the existing methods into five groups: trained probes, sequence probability, verbalization, surrogate token probability, and consistency-based.
Our experiments across eight publicly available LLMs indicate that prompting-based methods are less reliable than supervised-probing, although the latter requires training data and access to models’ weights. For instruction-tuned LLMs, verbalization and consistency-based methods are viable alternatives. Further, we argue that all methods for estimating factual confidence can ultimately lead to misleading conclusions if only tested on a single way of asserting a fact: An LLM may seem to know a fact given an input, but then contradict itself given an alternative phrasing of that fact (Elazar et al., 2021; Kassner et al., 2021; Lin et al., 2022b; Qi et al., 2023; Kuhn et al., 2023). In our experiments, we find evidence of such instability, suggesting that LLMs do not always encode facts based on abstractions over diverse input variations.
In summary, this paper provides the following contributions:
• A survey of the literature on LLM factual confidence estimation; • An experimental framework enabling a fair comparison across methods proposed in the literature; • Insights about the reliability and robustness of different types of methods, providing recommendations for NLP practitioners; • Insights about the consistency of factual confidence across semantically equivalent inputs.

heading: 2 Factual Confidence: Key Concepts

heading: 2.1 Definition of a Fact
text: We consider a fact to be a piece of information that accurately represents a world state.1A natural-
language statement is truthful—or factual—if its meaning reports a state of affairs that is supported by a true fact: e.g., “Paris is a city in France” is truthful as the city of Paris is indeed located in France. Facts and natural-language statements are not linked by a one-to-one relation: The same fact can be declared with multiple statements, varying on the surface level, but sharing the same meaning.
For this reason, one’s confidence in a fact should be consistent across meaning-preserving linguistic variations, such as paraphrases or translations of a statement: If we are certain that “Paris is a city in France” is true, we will not doubt that its paraphrase “Paris is a French city” or its translation in French (if we understand French) are also true.

heading: 2.2 Factual Confidence
text: We distinguish between two facets of factual confidence of LLMs, following Kadavath et al. (2022):
P (True) – shortened as P (T): the degree to which a model considers likely that a fact stated in the input is true. For example, “Paris is the capital of France” should get a high P (T) as it is truthful and is common knowledge, while “Sidney is the capital of France” should get a low P (T). To estimate P (T) scores we need to pass a statement in the input, which is evaluated in its truthfulness: this is in line with the setup of fact-verification (Thorne et al., 2018; Hardalov et al., 2022; Guo et al., 2022).
P (I Know) – shortened as P (IK): the degree to which a model considers likely that it will return the correct answer to an input querying about a fact. For instance, we can compute P (IK) in a QA setup passing a question as input—e.g., “What is the capital of France?”. If confident to know the true answer, P (IK) should be high; it should instead be low in case of uncertainty. In contrast to P (T), P (IK) is estimated without stating the fact in the input, but rather expecting a factual answer by the model complementing the query.
P (T) and P (IK) are both telling of the underlying factual confidence of an LLM. However, depending on the data format (e.g., statements vs. questions) or task of interest (e.g., factverification vs. QA) focusing on one of the measures is more suitable. Previous works introducing methods to estimate factual confidence have
1For simplicity, in this work, we restrict our focus to minimal, atomic facts, in the sense that they do not involve a combination of other facts; e.g., “The Louvre is in Paris” as opposed to “The Louvre is in Paris, which is in France”.
typically addressed only one of the two measures. However, as we demonstrate with our experimental framework, most methods can be adapted to estimate both P (T) and P (IK), although in practice they may not be equally reliable in each setup.

heading: 2.3 Robustness of Factual Knowledge
text: We work from the hypothesis previously voiced by Petroni et al. (2019) that a language model’s factual knowledge may stem from encoding facts in its weights (parametric memory) as an abstraction over the linguistic input in the training data.
Such human-like robustness and abstraction ability cannot however be taken for granted (Bender and Koller, 2020; Mitchell and Krakauer, 2023; Mahowald et al., 2024). Testing for consistency to meaning-preserving variations of an input is key to distinguishing whether a model has encoded a fact as an abstraction over linguistic forms, as opposed to memorizing statements asserting the fact (Carlini et al., 2023). For instance, if a model has a robust encoding in its parametric memory of what the capital of France is, it should provide the same answer to “What is the capital of France?”, “What is the name of the French capital city?” or any other rewording. Prior works already provided evidence that models may not always act consistently across semantically equivalent inputs (Elazar et al., 2021; Kassner et al., 2021; Ohmer et al., 2023; Qi et al., 2023). However, this has not yet been investigated in relation to the degree of factual confidence.

heading: 3 Factual Confidence: Survey of Methods
text: Based on a review of the research area, we identify five groups of existing methods to estimate factual confidence, which we discuss in the following subsections. In Table 1, we provide an overview of the functional differences among these methods.

heading: 3.1 Trained Probes
text: The methods in this group are based on probes which compute a transformation of a model’s internal representations. The probes can take as input the final layer or earlier layers, taking advantage of the latent compression stages of an LLM (Voita et al., 2019).
Azaria and Mitchell (2023) proposed to train multi-layered probes to extract factual confidence scores from hidden states, under the argument that such estimates are less subject to surface-level features—how a claim is phrased—and thus more
reliable. Their setup is in line with an estimate of P (T). Kadavath et al. (2022) adopted this method in a QA setup, estimating P (IK) using a trained value head on top of the final transformer layer. Breaking off from the constraints of supervised training, Burns et al. (2023) propose another version of the probe, which they train in an unsupervised manner, by maximizing distance between representations of contradicting answers on a Yes/No question dataset. They report performance only slightly lower than supervised alternatives—we therefore do not separate it in its own group. This method is worth considering in the specific case where a model’s layer outputs are available but reliable annotations are not available to train a probe.
We must emphasize that these methods have more strict requirements compared to the other groups, as: (i) they require having access to the model weights, and (ii) they need supervised training data, i.e., data with labels that reflect if the statements are true, for P (T), and data with labels that reflect whether the model will provide the correct answer to the question, for P (IK).

heading: 3.2 Sequence Probability
text: This family of methods use the averaged log probabilities, assigned to a sequence of output tokens, to estimate factual confidence. For this approach to work well, sequence probabilities need to be well calibrated (Guo et al., 2017) to indicate probability of correctness (Fomicheva et al., 2020; Xiong et al., 2024). In the context of factual knowledge, sequence probability has been applied both in cloze tasks and QA setups (Jiang et al., 2021; Yin et al., 2023), which corresponds to measuring P (IK).
A major limitation of the sequence probabilities is that they represent the confidence over how a claim is made (i.e., the probability of the generated response as a sequence of tokens—different surface realizations of the claim would have different probabilities), rather than the confidence about the claim itself (Lin et al., 2022a). Moreover, Gal and Ghahramani (2016) showed that the output sequence probabilities produce unreliable, overconfident estimates in general. Thereby, these methods are mainly used as a weak baseline for model confidence estimation.

heading: 3.3 Verbalization
text: In the verbalization (verbalized confidence) methods (Lin et al., 2022a; Xiong et al., 2024; Yin et al., 2023; Tian et al., 2023; Kadavath et al., 2022), the
model is directly prompted to report its confidence level (e.g., “How confident are you that the answer is correct?”). Here, differently from the other methods, the LLM generates a numeric confidence level as a sequence of output tokens. Lin et al. (2022a) and Tian et al. (2023) find that this method provides well-calibrated and surprisingly accurate estimates for highly capable instruction-tuned models like GPT-3 (Ouyang et al., 2022; Bai et al., 2022; OpenAI, 2023). Tian et al. (2023) extend the verbalization to “top-k” prompting, i.e., prompting the model for k answers, along with their estimated confidence. They also propose to use Chain-ofThought (CoT) and multi-step prompting, e.g., first providing an answer and then providing a measure of confidence that this answer is correct. In this work, we focus on the simplified non-CoT, k = 1 prompting as it shows competitive results to the more complex methods for many model-datasetmetric combinations.

heading: 3.4 Surrogate Token Probability
text: These methods, extensively studied by Kadavath et al. (2022); Xiong et al. (2024), can be considered a hybrid approach between the Sequence Probability (Section 3.3) and Verbalization (Section 3.2). The input prompt asks the model to provide as output specific tokens to report the factuality of the claim in the input; the probabilities assigned to those tokens are then used to determine the confidence level. This method can be adapted to measure both P (T) and P (IK) (Kadavath et al., 2022).

heading: 3.5 Output Consistency
text: Output consistency methods (also referred to as self-consistency) (Wang et al., 2023b) build on the assumption that the high confidence of a LLM leads to generating consistent outputs. Given a question or an incomplete statement, we sample multiple completions and take the inter-responses consistency as the confidence measure. If the model con-
sistently generates the same answer, the confidence is high. Conversely, if the model generates contradictory answers, the confidence is low. One limitation of this method is that, due to its dependence on completion generation, it can only be used to estimate P (IK) and not P (T).
Manakul et al. (2023) demonstrated the efficacy of this method when applied to factual knowledge, focusing on GPT models and using output consistency to validate model responses. Kuhn et al. (2023) argued that additional clustering of the outputs that are semantically equivalent as instances of the same answer is needed.

heading: 4 Methodology

heading: 4.1 Data
text: Currently, there is no standardized set of benchmarking datasets that are adopted across previous work. To fill this gap, we adapt two publicly available datasets, namely Lama T-REx (Petroni et al., 2019) and PopQA (Mallen et al., 2023), to test factual confidence in both fact-verification (P (T)) and QA (P (IK)) setups. We believe that establishing such a baseline setup is important both for researchers and practitioners.
4.1.1 P (T) in Fact Verification: Lama T-REx Lama T-REx (Petroni et al., 2019) is a relational dataset made of triplets extracted from Wikipedia <subject, relation, object>, (e.g., <Victor Hugo, was born in, France>). We use this dataset to create both true and false statements for estimating P (T). We create false versions of each factual statement, by randomly substituting the object in the triplet with one from the same relation (“Victor Hugo was born in China”). This ensures the right entity type and avoids grammatical errors.
There are 34K triplets (true statements) in the T-REx dataset. For each true statement we sample one false, creating a balanced set of 50/50 true/false
statements. Then, we take 80% of the examples, 27K of the positives and their corresponding negatives, 54K in total, for training (only used for trained probe). The rest, we use for analysis—6.8K T-REx true statements and an equal number of false statements, 13.6K in total.
4.1.2 P (IK) in QA: PopQA
The PopQA dataset (Mallen et al., 2023) consists of short questions and single entity answers (e.g., question: What is George Rankin’s occupation?, answer: Politician.). This dataset offers a set of synonymous phrases for each correct answer. This is an important feature, which makes the evaluation more robust to answer phrasing, and, in turn, lowers the risk of underestimating model’s correctness. Moreover, this dataset covers a broad range of entities, with varying degrees of popularity (estimated based on the number of Wikipedia page views). On one hand, this ensures the diversity of the target entities, and, on the other, it allows for further analysis between popularity and estimated confidence, which we leave for future work.
We use PopQA to test models’ factual confidence given a fact-related query, i.e., P (IK). The dataset contains 14K questions: we keep 80% (11K) for training, and 20% (2.8K) for testing. By definition (Section 2.2), the gold labels for P (IK) should indicate if the model outputs a correct answer. Ultimately, a model’s answer depends on the decoding strategy; in this work, for simplicity and clarity of interpretation, we use greedy decoding. If the answer is correct, we set the gold P (IK) to 1, else to 0 (more details in Section 4.2). As the labels depend on model correctness, the data will have varying proportions of positive labels across models, in our case, ranging from ∼11% to ∼27%.2

heading: 4.2 Scoring Methods Implementation
text: Below, we report the main specifics of our implementation of the methods (details in Appendix A).
4.2.1 Estimating P (T)
Given a statement, we compute P (T) as follows: Trained probe: As in Azaria and Mitchell (2023), we train a 3-layer fully connected feed-forward network for 10 epochs, passing as input the hidden states from the 24th transformer layer for each LLM (requires hyper-parameter optimization, later
2 The questions from PopQA are generally considered hard (ChatGPT: 30% accuracy, SelfRAG (Asai et al., 2024): 55%).
layers before the last one work better) and predicting whether a statement is true or false. Sequence Probability: Average log-probability of the statement’s tokens. Verbalization: Prompting for the confidence level that the statement is true (Appendix A). Surrogate Token Probability: Log-probability of the “Yes” token following a query on whether the statement is true.
4.3 Estimating P (IK)
Below, we describe how the P (IK) estimates for each method group are computed, based exclusively on the question. Trained Probe: We use the same approach as for P (T), but train the probes to predict whether the model’s greedy-generated answers will be truthful or not.3
Sequence Probability: Average log-probability of the question’s tokens.4
Verbalization: Prompting for the confidence level of knowing the answer to the question (see Appendix A for details). Surrogate Token Probability: Log-probability of “Yes” token following a query on knowing the answer to the question. Consistency: We prompt the model with the question and sample 10 responses with τ = 1. Then, we compute a matrix of pairwise NLI scores (Laurer et al., 2024) on all generations, and return an average.

heading: 4.4 Evaluating Scoring Methods
text: To evaluate the methods, we use the area under the precision-recall curve (AUPRC), as is common in related literature, e.g., Kadavath et al. 2022. Using a metric that considers various decision thresholds enables a robust comparison across methods. The higher AUPRC, the better ranking capability of the method, with cleaner separation between true/false statements or known/unknown facts. In an effort to make interpretation of AUPRC more intuitive for practical applications, we also report the precision and recall at K (Tables 5 and 6 in Appendix D).
3This is a simpler, less computationally expensive version of the approach of Kadavath et al. (2022), where multiple answers are sampled and the probe initially predicts a continuous score—proportion of correct answers in the sampled set.
4This implementation captures how surprised the model is by the question, which is linked with expected correctness.
Names Size Open Arch. Instruct
Falcon 40B Ë Dense Falcon Inst. 40B Ë Dense Ë Falcon 7B Ë Dense Falcon Inst. 7B Ë Dense Ë Mixtral 46.7B Ë SMoE Mixtral Inst. 46.7B Ë SMoE Ë Mistral 7B Ë Dense Mistral Inst. 7B Ë Dense Ë
Table 2: The models used in our experiments. Dense represents the usual transformer decoder architecture, while SMoE stands for Sparse Mixture of Experts (Shazeer et al., 2017), here made of 8 experts of 7B. Instruct. models have been instruction fine-tuned. Open models have publicly available weights.

heading: 4.5 Models
text: We study eight publicly available LLMs, with open access to weights. We consider models with different sizes (7B to 46.7B), architecture, and training paradigms (instruction-fine-tuned or not) from the Falcon (Almazrouei et al., 2023) and Mistral (Jiang et al., 2023, 2024) model families (see Table 2).

heading: 4.6 Paraphrasing and Translation
text: To test methods robustness and to disentangle confidence over a fact from confidence based on a specific wording, we generate semantically equivalent variants of statements/questions from Lama T-REx and PopQA (see Section 6). For each input, we generate 10 paraphrases by prompting Mixtral8x7B-Instruct-v0.1 (prompt and examples are in Appendix B). We remove repetitions and only keep paraphrases that are semantically equivalent5 to the original input. This results in an average of eight paraphrases per original input.
We also consider translation as another meaningpreserving transformation. Specifically, we translate the English examples to two languages from different language families (Romance and Slavic):6 (i) French—a high resource language to which all models except the 7B Mistral models have been explicitly exposed to during training, and (ii) Polish— a language on which we expect lower degree of competence (Falcon reports “limited capability” for Polish (Almazrouei et al., 2023), while Mistral models do not mention it at all). Finally, we
5To detect semantic equivalence, we test for entailment in both directions using an NLI model (Laurer et al., 2024)
6For translations we use Amazon Translate.
Figure 2: AUPRC scores on T-REx with both true and false statements; P (T).
manually verified the quality of a sample of 100 translations, finding them to be meaning preserving and vastly without errors.

heading: 5 Empirical Comparison of the Methods
text: 5.1 P (IK) on Lama T-REx
With each of the four methods, we derive estimates of factual confidence for all statements in the Lama T-REx test set, repeating the experiment for each LLM. We evaluate the reliability of a method by checking whether it yields P (T) scores that can effectively separate the true statements from the false statements, measured as AUPRC.
We report the results of this analysis in Figure 2. The trained probe method performs best, outperforming the sequence probability by an average AUPRC of .3. Of all methods and models, only the verbalized method is competitive to the supervised probe, and only for Mistral 7B instruct. Otherwise all methods perform at least .1 AUPRC below the trained probe. This result suggests that information about the expected truth value of a statement is better captured in deeper layers of the network, as opposed to the output scores.
While for trained probe and average sequence probability we note relatively small differences in AUPRC across models, for the verbalized and surrogate methods we see large variation. Concretely, instruction-tuned models always perform better than their counterparts. This is expected as both methods require to follow instructions in the prompt. Model size also seems to have an effect: all 40B+ models perform better than their 7B counter-parts, with the exception of Mistral-7BInstruct-v0.2. Information on the specific differences between versions of Mistral is not publicly available, making the interpretation of this result difficult. One possibility is that the second version of the model has better instruction-following capa-
bilities, but we cannot be sure (BehnamGhader et al. 2024 discusses conjectures on unusual properties of the Mistral family of models).
Finally, while the average sequence probability method performs consistently above chance (50%), it has overall poor performance in comparison to other methods. It only outperforms the other nontrained methods—verbalized and surrogate—for non-instruction-tuned models.
5.2 P (IK) on PopQA
P (IK) estimates the degree of a model’s confidence that its predicted answer will be correct. A good estimator of P (IK) would thus assign high scores to queries which the model answers correctly, and low scores to others. Following this reasoning, for P (IK) we compute the AUPRC scores using binary labels that encode whether the model’s answer (in our case, generated with greedy decoding) is correct. Note that this way of computing AUPRC— based on future correctness—provides a direct estimate of the method’s expected effectiveness for hallucination mitigation (when the method is used to automatically detect when the model should abstain from answering). In this scenario, a method is effective only if its estimates are actually predictive of the correctness of the model’s answers.
The results are reported in Figure 3. In this experiment we also study the Consistency method, which was omitted from P (T) results because, by design, it cannot be applied to an entire statement. Overall, P (IK) is harder to estimate than P (T), with lower AUPRC results: e.g., The best trained probe is 0.1 below in AUPRC for P (IK) than it is for P (T). This may be due to the complexity of the setup—in QA the confidence is estimated only based on a query, in contrast to fact-verification. But it may also be that the binary future correctness labels used for our AUPRC computation introduce some noise. E.g., the model may be genuinely uncertain
and still output the correct answer by chance. The trained probe method is again, by large, the most reliable across all models. With the exception of Falcon-40B instruct, the other methods perform close to or below chance (depending on the model’s label distribution, chance level varies between 0.11 and 0.27). This indicates that non-trained estimators are generally not reliable for P (IK) despite being frequently used in the literature. Within each method, we observe differences across models—up to a 40% margin. This can be linked to (i) whether a model is instruction-tuned (as noted for P (T)) and (ii) the model family—with more reliable scores for Mistral models than for Falcon models.

heading: 5.3 Generalization of the Trained Probe
text: The results above highlight the trained probe as the most reliable estimator for factual confidence— both for P (T) and P (IK). However, in those experiments we trained and evaluated the models within the same domain, which leaves open questions about the probe’s generalization capabilities. We address this gap by evaluating the model from 5.1, trained to estimate P (T) from Lama T-REx data, on the PopQA dataset converted to test for P (T). Specifically, we re-work the PopQA data for the fact-verification setup by turning question-answer pairs into (evenly distributed) true and false statements, using the template: “The answer to [QUESTION] is [ANSWER]”.7 We derive estimates for P (T) on such statements using the probes trained on Lama T-REx, and compute AUPRC (Table 3).
Going from in-domain to out-of-domain test data 7For true statements we use the gold answers from PopQA dataset. For false statements, we sample alternative answers from the same question class in the dataset; e.g., The answer to “In which country is Washington?” is “United States of America” vs. “South Korea”.
(Lama vs. PopQA), we observe AUPRC differences of min -.15 and max -.31. However, the scores remain in a high range of [.62, .81] indicating substantial generalization. The LLMs for which the probe retains the least and the most reliability are Mistral-7B and Falcon-40B-instruct, respectively. Interestingly, these are also the models getting the least and the most answers right on PopQA in the QA setup (14% and 23%). This suggests that the transferability of the probe may be affected by how challenging the out-of-domain dataset is to the model. In the next sections, we provide further evidence of probe generalization by looking at whether and to what extent the AUPRC is affected by input paraphrasing and translation.

heading: 6 Robustness to Linguistic Variations
text: In this section, we apply meaning-preserving linguistic variations to each input in order to: (i) Assess the robustness of methods. The expectation is that if a method is robust, it should produce equally reliable estimates (equally high AUPRC scores) across different input formulations (Section 6.1); (ii) Investigate the stability of an LLM’s encoding of facts. The expectation is that if a fact is well abstracted, the factual confidence should be invariant to semantics-preserving changes in the input (Section 6.2). We consider two types of input variation: paraphrases and translations.

heading: 6.1 Robustness of Methods
text: We study method robustness in both P (T) and P (IK), using the same setup as before; in particular, we do not retrain the trained probe and do not adapt the prompts in any way.8 To test robustness on paraphrases, we generate 10 different paraphrase sets—each holding different formulations of the original inputs—and compute AUPRC on each set. We observe that AUPRC remains stable for all methods (absolute variation between 5% and 10%), indicating they are robust to paraphrasing. The most affected method is the trained probe in the P (IK) setting, but even here we only note up to a standard deviation of 3 percentage points (for Mistral-7B). Full result tables are in Appendix C.
For translations, we compute a separate AUPRC on the French and Polish versions of T-REx. We find varying degrees of method transferability to new languages (above .5 for verbalized, up to .91
8Note this also applies to translations; i.e., the trained probe is trained on English data only and we use English prompts to query the model about French/Polish inputs.
for trained probe). All methods generalize to both French and Polish above chance, except for (i) verbalized confidence and (ii) surrogate logits when applied to Mistral models (see Figure 7 in the Appendix). Notably, the probes trained on English data remain to a large extent reliable (AUPRC for French: .73-.91; for Polish: .61-.91) on unseen languages—with 40B+ models and the instructiontuned Mistral demonstrating the highest transferability. This provides additional evidence for outof-domain generalization of trained probes (Section 5.3). In particular, the probes can extract scores that are discriminative of true and false statements also from hidden states computed from inputs in a different language than the one used at training. This suggests that the LLMs encode factual confidence in a similar way across languages, which is remarkable, given the differences in exposure to training data from the three languages.

heading: 6.2 Robustness of Facts Encoding in LLMs
text: We hypothesize that, to robustly learn facts and minimize hallucinations, a model has to build stable abstractions over different types of relevant evidence from the training data. We also expect that if the model has built such a robust representation of a fact, this would lead to equal confidence under equivalent formulations of that fact. Inconsistent confidence would in turn indicate excessive reliance on surface-level features.
Figure 4 shows how paraphrasing the input (∼8 paraphrases per input causes changes in the trained
probe P (T) estimates across the T-REx dataset.9 Specifically, we present the distribution of the standard deviation of confidence scores, across different facts. The amount of variation is not stable across facts. While for each model, we observe a large amount of facts for which there is little-to-no variation in the score (indicating a stable encoding), we also note many facts for which different wordings lead to strongly varying degrees of confidence (up to .5 standard deviation, with very few cases at .7). This indicates inconsistent LLM behavior, with excessive sensitivity to how a claim is worded. Out of all models, Falcon-7B-instruct appears to have developed the least stable encoding, with the standard deviation distribution shifted towards higher values. On the other extreme, Mistral-7b-v0 appears to be the model showing less variation.
To test robustness of factual knowledge across languages, we compare the distributions of P (T) scores over the same facts using the Spearman correlation analysis (for language pairs) and the Friedman test (for language groups). Analysis reveals high correlations (Spearman’s ρ > .7; full results in Table 4 in Appendix) between factual confidence scores on all language pairs for the 40B+ models. In particular, we note the highest correlations (in the .87-.92 range) for Falcon 40B models, which points to highly robust multilingual behavior. However, the Friedman tests reveal that for all models, the differences across the distributions are statistically significant (p-values close to 0); i.e., the differences in scores across the languages are not close enough to be coming from the same population. Given those results, we conclude that while there is a link between the confidence scores across the languages, this is not fully systematic.

heading: 7 Discussion & Conclusion
text: In this paper, we compare existing methods to estimate LLMs factual confidence. Obtaining reliable estimates can benefit LLMs applications, by anticipating potential hallucinations and limiting the non-factual information output by a model (Tonmoy et al., 2024; Evans et al., 2021). However, if not reliable, such estimates can be counterproductive, as they would introduce errors and negatively affect user-model interactions.
Our experiments across eight LLMs demonstrate that the trained probe method is the most reliable
9We focus on the trained probe since it was the best performing method.
estimator of LLM factual confidence. It works well for both fact-verification (P (T)) and Question Answering (P (IK)) across all 8 tested models, indicating that its reliability is likely to generalize to other LLMs. Moreover, we show that it generalizes to out-of-domain data: (i) when a model trained on T-REx is applied to an unseen dataset (PopQA, Section 5.3), and (ii) in a cross-lingual transfer setting (Section 6.1). We must note, though, that the fine-tuning nature of the method clearly puts it at an advantage over the other zero-shot probing methods. Moreover, applying the trained probe method comes with strong requirements: (i) access to model weights—not always provided by proprietary LLMs, and (ii) need of supervised data. If these requirements cannot be met, but the model is instruction-tuned (Ouyang et al., 2022) we recommend estimating P (T) with verbalized confidence or surrogate probabilities. The other methods under study, especially if applied to non-instructiontuned LLMs, are not consistently reliable.
Our results highlight the need for more research to develop reliable estimators that can be applied to black-box models, with inaccessible internal representations. We expect the reliability gap between methods like verbalized confidence and trained probe to get smaller with increasingly powerful LLMs, especially in their ability to follow instructions. However, strong results of trained probe indicate that hidden states contain signal about factual confidence, and it is unclear whether this can ever be fully leveraged by the prompting approaches.
Besides the comparison among methods, we also provide insights on the stability of factual knowledge in LLMs (Petroni et al., 2019; Mitchell and Krakauer, 2023; Mahowald et al., 2024). We show that the factual confidence of an LLM is not always consistent under meaning-preserving variations of the input (paraphrases and translations). While the model may sometimes be sure that a fact is true or false, or that it knows the answer to a question, it may actually behave differently if we reformulate the statement or question. An interesting direction for future research is the exploration of training methods that teach an LLM to better disentangle facts from the diversity of forms they can be stated in, and ultimately exhibit better and more consistent factual knowledge. This would also contribute to increasing LLMs resistance to adversarial attacks (Madry et al., 2018), mitigating the generation of misinformation due to an incorrect sensitivity to input changes.

heading: Acknowledgments
text: We thank the anonymous reviewers for their helpful questions and comments, which have helped us improve the quality of the paper. We also want to thank Ionut Sorodoc, Neha Anna John, Phu Mon Htut and the entirety of the AWS Bedrock Responsible AI team for the time they took to support, discuss and improve this work. We thank Marco Baroni and Nathanaël Rakotonirina for helpful discussion during this project.
Matéo Mahaut is supported by the European Research Council (ERC) as part of a project in the European Union’s Horizon 2020 research and innovation program (grant agreement No. 101019291). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
Limitations
Given the extensive scope of this work (eight models, five methods and two facets of factual knowledge), we did not have the capacity to study more complex aspects of factual confidence, such as non-atomic facts, reasoning or in-context learning. While our results show that the trained probe is much stronger than other methods on T-REx and PopQA, there is no guarantee that this remains the case in more complex settings. Furthermore, methods themselves have limitations, making comparison use-case dependent. The trained probe method for example requires training data, and while we have tested for transfer capabilities in our simple atomic fact setup, Kadavath et al. (2022) have shown that there are limits to the kind of tasks this method can be transferred to. The same can be said of the sequence probability method, which in our experiments works better than both promptbased methods for non instruction fine-tuned models. While this method performs well on simple atomic facts, more complex sentences, or even simple but longer sentences could lead to weaker results. Furthermore, both prompt-based methods are sensitive to prompt-variations.
Ethics and Broader Impact
This work contributes to the wider goal of automatically reducing risk when using LLMs. We contribute to false statement detection and answer
confidence, leading to potential applications which can build trust in LLMs. None of the methods studied completely solve the issue of hallucination, or non-factual utterances of models, leaving a need for future works on the subject. While methods studied can work with models with 7B and 40B+ parameters, the deployment of those models requires specific infrastructure and is compute intensive.

heading: Appendix for “Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators”
text: A Implementation of scoring methods
A.1 Verbalized
For the verbalization method, we start with the prompt from Tian et al. (2023) and adapt it to the current tasks. Then, we filter out non-numerical outputs and normalize. Prompt used for P (T) experiments:
Provide your confidence level (on a scale of 1.0 to 10.0) that the following statement is correct .
The statement is: $statement
Confidence level:
Prompt used for P (IK) experiments:
Provide your confidence level (on a scale of 1.0 to 10.0) that you can answer the following question .
The question is: $question
Confidence level:
A.2 Surrogate token probabilities
For P (T) we use the prompt:
Q: Is the following statement true: $statement (Yes/No/Maybe)? A:
For P (IK) we use the prompt:
Q: Do you know the answer to the following question: $question (Yes/No/Maybe)? A:
A.2.1 Consistency The ground truth answer is generally a word, ranging between two and five tokens. We output 10 samples of up to 25 tokens using a temperature (τ ) of 1, leaving ample room for the ground truth to be generated in a sentence. Finally, we calculate the consistency score by computing the mean NLI (Laurer et al., 2024) score of these 10 samples.

heading: B Paraphrasing
text: Prompt used to generate paraphrases with Mixtral8x7B-Instruct-v0.1 (examples are provided in Table 7):
Given a sentence, generate paraphrases of it as follows:
- You can change and/or add words, and/or change the syntactic structure of the sentence;
- Make sure the new sentence does not add additional details with respect to the original.
- Make sure the new sentence does not omit any details with respect to the original.
- Make sure the new sentence is natural and plausible, in spite of the changes.
- Do not generate the original sentence or previously generated ones.
List your paraphrases as bulletpoint. Sentence: $sentence New sentences:
The variation of P (T) and P (IK) with paraphrases are provided in Figures 5 and 6.

heading: C Method Robustness to Variation
text: To measure the robustness of the methods towards linguistic variations, we randomly sample a paraphrase for every sentence in the original dataset, making ten sets of paraphrases of the same size.
We then compute AUPRC without changing the method in any way for the ten sets, and calculate the variance in results. The results are shown in Figures 5 and 6. All methods remain stable, and robust to paraphrases. The biggest variation occurs for the trained probe method, but are only of the order of 3 percentage points.
Table 4 shows the correlation between scores across different languages, and Figure 7 shows AUPRC of all four methods evaluated on the French and Polish versions of the Lama T-REx dataset.

heading: D Analysis of specific Precision and Recall
text: In Tables 5 and 6 respectively, we show results for precision at different recall thresholds for the two formations: P (T) on Lama T-REx and P (IK) on PopQA. Similar to our previous findings (see
Figures 2 and 3) the trained probe noticeably outperforms other methods, making the differences much more apparent, confirming that for P (IK) the Falcon 40B outperforms all other models, and all methods, expected trained probe, show low performance when applied with Falcon 7b Instruct. In constrast to the other methods, the precision of the verbalized confidence does not change for different recall thresholds, for both P (T) and P (IK), which suggests that we are reaching its limits to estimate factual confidence. We hypothesize that the emphverbalized confidence failing to disentangle correctly True sentences from False ones. This is also true, to some extent, for the surrogate token method and the sequence probability for P (IK). The trained probe, on the other hand, has better precision with lower recalls, which is also the case for the sequence probability for P (IK), as well as the surrogate token method in the same condition,
however only with bigger Falcon models. These results do push for additional work, as they point out that there remains substantial overlap when classifying True and False sentences (as well as successfully or unsuccessfully completed sentences) with a maximum of around 60% of sentences correctly classified, reaching a recall of 90 in both settings.

heading: Future Work
text: None of the methods studied completely solve the issue of hallucination, or non-factual utterances of models, leaving a need for future works on the subject. While methods studied can work with models with 7B and 40B+ parameters, the deployment of those models requires specific infrastructure and is compute intensive