Title: To Believe or Not to Believe Your LLM: Iterative Prompting for Estimating Epistemic Uncertainty
Venue: NeurIPS 2024
Accepted: yes
Keywords: uncertainty quantification

large language models
Abstract: We explore uncertainty quantification in large language models (LLMs), with the goal to identify when uncertainty in responses given a query is large. We simultaneously consider both epistemic and aleatoric uncertainties, where the former comes from the lack of knowledge about the ground truth (such as about facts or the language), and the latter comes from irreducible randomness (such as multiple possible answers). In particular, we derive an information-theoretic metric that allows to reliably detect when only epistemic uncertainty is large, in which case the output of the model is unreliable. This condition can be computed based solely on the output of the model obtained simply by some special iterative prompting based on the previous responses. Such quantification, for instance, allows to detect hallucinations (cases when epistemic uncertainty is high) in both single- and multi-answer responses. This is in contrast to many standard uncertainty quantification strategies (such as thresholding the log-likelihood of a response) where hallucinations in the multi-answer case cannot be detected. We conduct a series of experiments which demonstrate the advantage of our formulation. Further, our investigations shed some light on how the probabilities assigned to a given output by an LLM can be amplified by iterative prompting, which might be of independent interest.

To Believe or Not to Believe Your LLM: Iterative Prompting for Estimating Epistemic Uncertainty

Abstract

We explore uncertainty quantification in large language models (LLMs), with the goal to identify when uncertainty in responses given a query is large. We simultaneously consider both epistemic and aleatoric uncertainties, where the former comes from the lack of knowledge about the ground truth (such as about facts or the language), and the latter comes from irreducible randomness (such as multiple possible answers). In particular, we derive an information-theoretic metric that allows to reliably detect when only epistemic uncertainty is large, in which case the output of the model is unreliable. This condition can be computed based solely on the output of the model obtained simply by some special iterative prompting based on the previous responses. Such quantification, for instance, allows to detect hallucinations (cases when epistemic uncertainty is high) in both single- and multi-answer responses. This is in contrast to many standard uncertainty quantification strategies (such as thresholding the log-likelihood of a response) where hallucinations in the multi-answer case cannot be detected. We conduct a series of experiments which demonstrate the advantage of our formulation. Further, our investigations shed some light on how the probabilities assigned to a given output by an LLM can be amplified by iterative prompting, which might be of independent interest.

1 Introduction

Language models too occasionally suffer from hallucinations , or responses with low truthfulness, that do not match our own common or textbook knowledge (Bubeck et al., 2023; Gemini Team, Google, 2023). At the same time, since LLMs work by modeling a probability distribution over texts, it is natural to view the problem of truthfulness through the lens of statistical uncertainty. In this paper we explore uncertainty quantification in LLMs. We distinguish between two sources of uncertainty: epistemic and aleatoric (Wen et al., 2022; Osband et al., 2023; Johnson et al., 2024). Epistemic uncertainty arises from the lack of knowledge about the ground truth (e.g., facts or grammar in the language), stemming from various reasons such as insufficient amount of training data or model capacity. Aleatoric uncertainty comes from irreducible randomness in the prediction problem, such as multiple valid answers to the same query. Hence, truthfulness can be directly analyzed via looking at the epistemic uncertainty of a model in the sense that when the epistemic uncertainty is low, the model predictions must be close to the ground truth.

Rigorously identifying when (either) uncertainty is small 1 is notoriously hard, especially in deep neural networks (Blundell et al., 2015; Antorán et al., 2020). This is because we generally lack guarantees about learning the ground truth (consistency), or even a weaker guarantee about how large the variance of a learning algorithm is. At the same time, there exist many heuristic approaches for uncertainty quantification based on simply looking at the log-likelihood of responses (Kadavath et al., 2022), estimating entropy (Kuhn et al., 2023), ensembling (Lakshminarayanan et al., 2017b; Dwaracherla et al., 2023; Osband et al., 2023), or sometimes even more principled formulations, such as conformal prediction (Angelopoulos et al., 2023; Ravfogel et al., 2023; Yadkori et al., 2024) (which however come with strong assumptions).

To the best of our knowledge, a common limitation of these approaches is that they are only meaningful in problems where there exists a single correct response (e.g. label) as they aim for detecting if one response is dominant (or multiple responses with the same meaning), that is, if there is only little uncertainty in the prediction. On the other hand, when multiple responses are correct, that is, there is aleatoric uncertainty in the ground truth, simply estimating the amount of uncertainty in the LLM's output is insufficient, as the perfect (ground-truth) predictor may have large aleatoric uncertainty and no epistemic uncertainty, while a completely useless predictor may have large epistemic uncertainty only, but the total amount of uncertainty of the two predictors might be the same.

Contributions. In this paper we address the above problem directly, and design methods to decouple epistemic and aleatoric uncertainty , allowing us to effectively deal with multi-response queries. Rather than trying to quantify how small epistemic uncertainty can be, we aim to identify when only the epistemic uncertainty is large , in which case we can suspect that the response is hallucinated. 2

As a starting point we make a simple observation: If multiple responses are obtained to the same query from the ground truth (the language), they should be independent from each other, that is, in probabilistic interpretation, the joint distribution of these multiple responses, for a fixed query, must be a product distribution.

This observation can be used to measure how far the language model can be from the ground truth. The sequential model implemented by a language model allows us to construct a joint distribution over multiple responses, which is done through iterative prompting of an LLM based on its previous responses and the application of the chain rule of probability: first the model is asked to provide a response given a query, then to provide another response given the query and the first response, then a third one given the query and the first two responses, an so on. This is in contrast to some of the earlier works that approached decoupling epistemic and aleatoric uncertainty for classification problems by training the model with label pairs (or tuples) (Wen et al., 2022; Johnson et al., 2024).

So, if the response to a prompt containing the query and previous responses is insensitive to the previous responses, we have the desired independence and the LLM-derived joint distribution can be arbitrarily close to the ground truth. On the other hand, if the responses within the context heavily influence new responses from the model then, intuitively speaking, the LLM has low confidence about the knowledge stored in its parameters, and so the LLM-derived joint distribution cannot be close to the ground truth. As more responses are added to the prompt, this dependence can be made more apparent, allowing to detect epistemic uncertainty via our iterative prompting procedure .

Interestingly, as we will see in Section 3, we can force an LLM to provide a desired (possibly incorrect) response by adding this response repeatedly to the prompt. This phenomenon is then further investigated from the viewpoint of a transformer LLM architecture in Section 4.

The iterative prompting procedure then leads to the following main contributions:

(i) Based on the above iterative prompting procedure, we derive an information-theoretic metric of epistemic uncertainty in LLMs (Section 5), which quantifies the gap between the LLM-derived distribution over responses and the ground truth. This gap is insensitive to aleatoric uncertainty, allowing to quantify epistemic uncertainty even in cases where there are multiple valid responses.

(ii) We derive a computable lower bound on this metric, which turns out to be a mutual information (MI) of an LLM-derived joint distribution over responses, and propose a finite-sample estimator for it.

We prove that this finite-sample MI estimator sometimes suffers only a negligible error even though LLMs and their derived joint distributions are defined over potentially infinite supports (all possible strings in a language).

(iii) We discuss an algorithm for hallucination detection based on thresholding a finite-sample MI estimator, where the threshold is computed automatically through a calibration procedure. We show experimentally on closed-book open-domain question-answering benchmarks (such as TriviaQA, AmbigQA, and a dataset synthesized from WordNet) that when the data is mostly composed of either single-label or multi-label queries, our MI-based hallucination detection method surpasses a naive baseline (which is based on the likelihood of the response), and achieves essentially similar performance to that of a more advanced baseline which is based on the entropy of the output as a proxy for uncertainty. However, on datasets which contain both single- and multi-label samples at the same time, our method also significantly outperforms the entropy-based baseline, by achieving a much higher recall rate on samples with high output entropy while maintaining similar error rates.

(iv) Focusing on a single self-attention head, we identify a simple mechanistic explanation for how the model output can be changed through iterative prompting using previous responses, as discussed earlier. Suppose that the prompt is composed from a query and a repeated element (e.g., a possibly wrong answer). If the query lies within the space spanned by the large principal components of a key-query matrix product, then the output will be generated according to the knowledge extracted from the training data (now stored in a value matrix). On the other hand, if the query has little overlap with the large principal components, then the repeated element is likely to be copied from the prompt.

2 Preliminaries

Conditional distributions and prompting. Let X be the space of finite text sequences, that is X ⊂ Σ ∗ where Σ is a finite alphabet (and Σ ∗ = ⋃ ∞ n =1 Σ n ). Moreover, consider a family of conditional distributions P = { µ : X → [0 , 1] | ∑ x ∈X µ ( x | x ′ ) = 1 ∀ x ′ ∈ X} . In the following, we let P ∈ P be the ground-truth conditional probability distribution over text sequences (responses) given a prompt, and we let Q ∈ P be the learned language model. Given a fixed query x ∈ X and possible responses Y 1 , . . . , Y t , we define a family of prompts F = { F t : X → X | t ∈ N } , such that F t ( x, Y 1 , . . . , Y t ) is defined as:

Consider the following question: Q: x One answer to question Q is Y 1 . Another answer to question Q is Y 2 . [ . . . ] Another answer to question Q is Y t . Provide an answer to the following question:

Q: x . A:

Information-theoretic notions. Let µ, µ ′ be distributions supported on set Z = Z 1 ×··· × Z n where ( Z i ) i is a collection of countable sets. The entropy of a distribution µ is defined as H ( µ ) = ∑ z ∈Z µ ( z ) ln(1 /µ ( z )) . 3 If µ, µ ′ are such that µ ′ ( z ) = 0 only if µ ( z ) = 0 , we have a KullbackLeibler divergence between them defined as D KL ( µ, µ ′ ) = ∑ z ∈Z µ ( z ) ln( µ ( z ) /µ ′ ( z )) . For any z ∈ Z , we denote z \ i = ( z 1 , . . . , z i -1 , z i +1 , . . . , z n ) , and the marginal of the i th coordinate of µ is given by µ i ( z ) = ∑ z \ i ∈Z n -1 µ ( z ) . The product distribution of the marginals of µ is given by µ ⊗ ( z ) = ∏ n i =1 µ i ( z ) , and the mutual information of µ is defined as I ( µ ) = D KL ( µ, µ ⊗ ) .

3 Probability amplification by iteratively prompting

In this section we demonstrate that, as mentioned in the introduction, repeating possible responses several times in a prompt can have pronounced effects on the output of a language model. Consider x = 'What is the capital of the UK?' and Y 1 = · · · = Y t = 'Another answer to question Q is Paris. ' Here we can repeat the sentence 'Another answer to question Q is Paris. ' an arbitrary number of times. Although the number of repetitions changes the behavior of the LLM, the correct response maintains a significant probability: as Figure 1 shows, the conditional normalized probability 4 of

Q: What is the capital of the UK? A: London ( ≈ 1 . 0 ) and Paris ( 1 . 29 × 10 - 10 ).

Q: Who was the first US president? A: George Washington ( 0 . 999 ) and Abraham Lincoln ( 3 . 1 × 10 - 06 ).

Q: Who is the author of The Grapes of Wrath? A: John Steinbeck ( ≈ 1 . 0 ) and Ernest Hemingway ( 1 . 34 × 10 - 10 ).

Q: What is the largest country in the world? A: Russia ( 0 . 999 ) and United Kingdom ( 9 . 02 × 10 - 06 ).

1.00

1.00

1.0

0.10

0.99

0.98

0.8

0.08

0.98

0.96

0.6

0.06

0.97

0.94

0.4

0.04

0.96

0.92

0.2

0.02

0.95

0

20

40

60

80

100

20

40

60

80

100

0.0

0

20

40

60

80

100

0.00

0

20

40

60

08

100

0.8 -

0.6

0.4

0.2

0.0

0

2

4

6

8

Q: What is the national instrument of Ireland? A: The harp ( 0 . 936 ) and Uilleann pipes ( 0 . 063 ).

0.6

0.5

0.4

E°0

0.2

0.1

0.0

0

2

4

6

8

Q: Which actor became M in the Bond film Skyfall? A: Ralph Fiennes ( 0 . 651 ) and Judi Dench ( 0 . 348 ).

0.6

0.5

0.4

0.3

0.2

0.1

0.0

0

Z

4

6

8

Q: Which can last longer with out water a camel or a rat? A: A rat ( 0 . 538 ) and A camel ( 0 . 461 ).

0.08

0.06

0.04

Z0°0

0.00

0

6

8

Q: If Monday's child is fair of face what is Saturday's child? A: Work hard for a living ( 0 . 093 ) and Full of grace ( 0 . 906 ).

Q: Name a city in the UK A: London ( 0 . 958 ) and Manchester ( 0 . 041 ).

Q:

Name a yellow fruit

A:

(

0

.

715

) and

Lemon

(

0

.

284

).

Q: Name an alcoholic drink , A: Wine ( 0 . 685 ) and Beer ( 0 . 314 ).

1.0

1.0

1.0

1.0

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

0

8

Q: Name a ball game that is played by more than 5 players A: Volleyball ( 0 . 542 ) and Soccer ( 0 . 457 ).

the correct response, 'London' , reduces from approximately 1 to about 96% as we increase the number of repetitions of the incorrect response to 100. Figure 1 shows 3 more examples where, with initially low epistemic uncertainty in the response to the query (the aleatoric uncertainty is also low as we consider single-response queries), the correct response maintains a significant or non-negligible probability even in the presence of repetitions of incorrect information, while the probability of predicting the latter is increased.

Next, we consider a queries for which the model is more uncertain. For the prompt 'What is the national instrument of Ireland?' , we observe that responses 'The harp' and 'Uilleann pipes' both have significant probabilities (the first answer is the correct one). This time, by incorporating the incorrect response in the prompt multiple times, the probability of the correct answer quickly collapses to near zero, as shown in Figure 2, with significant epistemic uncertainty.

Finally, we consider multi-label queries for which the LLM confidently knows a correct answer. This time, by incorporating a potential response in the prompt, the probabilities of other correct answers stay relatively large. Figure 3 shows four such examples.

4 Explanation through the lens of in-context vs. in-weight learning

The sensitivity of the response of an LLM to extra in-context information, as observed above, can already be observed in a single attention head as explained next.

We consider an idealized attention mechanism as follows. Let Z ∈ R n × d ′ be an input matrix comprised of n semantic feature vectors each of dimension d ′ . Each row is meant to represent a complete statement (such as 'What is the capital of the UK?' or 'One answer to the question is Paris. ' , etc.) rather than a single token. Let X glyph[latticetop] ∈ R 1 × d ′ be the first row of Z , which represents the query of interest, such as 'What is the capital of the UK?' . Let E glyph[latticetop] ∈ R 1 × d ′ be a special vector indicating the end of the input. The matrix Z \ X , denoting the Z matrix without its first row, represents the in-context information.

We assume the ground-truth distribution P is such that a query vector is mapped to its response, but a statement is simply copied. For example, for V = 'What is the capital of the UK?' , P ( · | V ) would be a distribution with support on 'London' and its variations, while for V ′ = 'What is the capital of the UK? One answer to the question is Paris. ' , P ( · | V ′ ) returns the same distribution. We assume a parameter matrix W V is learned such that V glyph[latticetop] W V estimates P ( · | V ) for vector V .

Let W Q , W K , W V ∈ R d ′ × d be the query, key, and value matrices. A self-attention head with query X and context Z \ X is defined as

where the output of the softmax is a row vector of length n .

If X has appeared many times in the training data, then parameters W Q and W K could be learned such that E glyph[latticetop] W Q ( W K ) glyph[latticetop] X is large, that is, X is within the space spanned by the large principal components of the key-query matrix product. Then, no matter what in-context information appears in Z , the probability assigned to X will dominate the softmax, and we will have Softmax ( 1 √ d E glyph[latticetop] W Q ( ZW K ) glyph[latticetop] ) Z ≈ X glyph[latticetop] , and therefore f ( Z ; W Q , W K , W V ) ≈ P ( · | X ) .

Now, consider the case that X has not appeared many times in the training data, and vector Y is copied in many rows of Z . Then E glyph[latticetop] W Q ( W K ) glyph[latticetop] X could be small as X is not in the span of the large principal components of the key-query matrix product. Therefore f ( Z ; W Q , W K , W V ) ≈ Y since Softmax ( 1 √ d E glyph[latticetop] W Q ( ZW K ) glyph[latticetop] ) Z ≈ Y glyph[latticetop] . Even if X is in the span, repeating Y t times in Z would give a t -times increased total weight to Y inside the softmax, which can dominate the weight assigned to X when t is large enough, also resulting in Y as the answer.

5 Metric of epistemic uncertainty and its estimation

In this section we apply iterative prompting to estimate the epistemic uncertainty of a language model about responding to some query. The idea is to utilize the different behavior patterns observed in Section 3, which can be used to differentiate between two modes of high uncertainty: when the aleatoric uncertainty is high vs. when only the epistemic uncertainty is high. We then apply our new uncertainty metric to design a score-based hallucination detection algorithm.

We will first present the uncertainty metric and its estimate for a distribution defined on the direct outputs of an LLM; the changes needed to take semantic equivalences of language into account are deferred to Appendix A (Kuhn et al., 2023; Farquhar et al., 2024).

Our uncertainty metric, similarly to the ones considered by, e.g., Wen et al. (2022); Osband et al. (2023), is based on analyzing the joint distribution of responses: if multiple responses are sampled jointly according to the ground-truth distribution, they should be independent (as one instantiation of a response should not affect other responses). To make this notion precise, we start with defining a notion of the joint distribution over responses given a query, derived from the language model through the prompting mechanism F defined in Section 2:

Definition 5.1 (Pseudo joint distribution) . Given a family of prompt functions F , a conditional distribution µ ∈ P , and n ∈ N , we use notation ˜ · to denote a pseudo joint distribution defined as

The above is a pseudo joint distribution since the standard conditioning in the chain-rule is replaced with prompt functions of the conditioning variables. In the following we focus on ˜ Q derived from the LLM and ˜ P derived from the ground truth.

Remark 5.2 (Sampling from ˜ Q ) . Note that sampling from ˜ Q can be simply done through a chainrule-like procedure as can be seen from the above definition, that is, to have ( Y 1 , . . . , Y n ) ∼ ˜ Q we draw Y 1 ∼ Q ( · | F 0 ( x )) , Y 2 ∼ Q ( · | F 1 ( x, Y 1 )) , Y 3 ∼ Q ( · | F 2 ( x, Y 1 , Y 2 )) , and so on.

In the rest of the paper we drop subscripts in joint distributions and conditioning on query x (which is understood implicitly), for example, ˜ P ≡ ˜ P Y 1 ··· Y n | x .

For any query x ∈ X , let Y ˜ Q ( x ) denote the support of ˜ Q . We make the following assumption about the ground truth, which states that the model Q generates reasonable responses and that the distribution of such responses are independent according to the ground truth:

Assumption 5.3 (Ground truth independence assumption) . For any query x ∈ X , (i) there exists a sequence of valid responses Y ( x ) ⊂ X such that the ground-truth distribution satisfies

(ii) Y ˜ Q ( x ) ⊂ Y ( x ) , that is, the model Q generates reasonable responses.

Note that the above assumption is heavily dependent on our prompt construction and the assumption that Y 1 , . . . , Y t -1 are valid responses; without these the independence assumption would not hold, for example, if Y 1 , . . . , Y t were partial answers, such as a step of an algorithm or a part of a story, or would completely redefine the problem ( 'Disregard the previous question. Instead answer the following... ' ), because in such cases Y t might indeed depend on the previous outputs Y 1 , . . . , Y t -1 . Roughly speaking, the assumption tells that the response distribution is insensitive to a query based on previously sampled responses. For example, for query x = 'Capital of the UK:' , the probability of Y 2 = 'London' essentially does not change if a city is Y 1 = 'Paris' .

To measure epistemic uncertainty, we need to quantify how far the estimated pseudo joint distribution ˜ Q is from the ground truth ˜ P . One natural choice is the following definition:

Definition 5.4 (Epistemic uncertainty metric) . Given an input x ∈ X , we say that the epistemic uncertainty of ˜ Q is quantified by D KL ( ˜ Q, ˜ P ) .

Here D KL measures how well ˜ Q approximates ˜ P for a given query x . Namely, this metric determines if ˜ Q assigns a large probability to an event which has a small probability under ˜ P . In case of LLMs, this means the LLM generates a sequence that is unlikely in the typical usage of the language. Given an input x , we want to estimate the above hallucination metric, but we only have access to ˜ Q , and so computing it explicitly is impossible. However, next we show that under Assumption 5.3 we can lower bound D KL ( ˜ Q, ˜ P ) by a quantity which only depends on ˜ Q (the proof is given in Appendix E).

Theorem 5.5. For all pseudo joint distributions ˜ P and ˜ Q satisfying Assumption 5.3, D KL ( ˜ Q, ˜ P ) ≥ I ( ˜ Q ) .

The lower bound in the theorem holds uniformly for all ˜ P , and it is computable solely based on ˜ Q . This makes the bound applicable for decision making; in fact we chose to consider D KL ( ˜ Q, ˜ P ) as the measure of epistemic uncertainty (out of similar distance measures) since it admits this property.

glyph[negationslash]

Also, note that we have I ( ˜ Q ) = D KL ( ˜ Q, ˜ Q ⊗ ) , ˜ Q ⊗ = ∏ i ∑ y \ i ˜ Q ( y 1 , . . . , y i -1 , Y i , y i +1 , . . . , y n ) . In general ∑ y \ i ˜ Q ( y 1 , . . . , y i -1 , Y i , y i +1 , . . . , y n ) = ˜ Q ( Y i ) , because the independence assumption Assumption 5.3 does not necessarily (and, in practice, almost never) holds for Q .

Finally, a quantity related to D KL ( ˜ Q, ˜ P ) is D KL with arguments arranged in the opposite order, that is D KL ( ˜ P, ˜ Q ) which is a (query) conditional excess risk of the LLM-derived pseudo joint distribution ˜ Q , under the logarithmic loss. Controlling the excess risk (for instance, upper-bounding it) for various algorithms is one of the central questions in learning theory, however it is a much harder task than the one we consider here, because for the former we need to theoretically control all sources of errors (such as generalization, estimation, and approximation error).

1: Input :

µ ∈ M 1 ( X n ) . . . . . . . . . . . . . . . . . . . any (pseudo-) joint distribution over X n k ∈ N . . . . . . . . . . . . . . . . . . . . . . . . . . sample size

γ 1 , γ 2 ≥ 0 . . . . . . . . . . . . . . . . . . . . . . stabilization parameters (typically selected as 1 /k )

2: Independently sample tuples X 1 , . . . , X k ∼ µ ∈ M 1 ( X n )

3: Construct a set of indices of unique elements U = { i ∈ [ k ] : X i = X j ∀ j < i }

glyph[negationslash]

4: Construct empirical distributions: let Z = ∑ j ∈ U µ ( X j ) , and for all i ∈ U , compute

5: Compute estimate of MI ̂ I k ( γ 1 , γ 2 ) = ∑ i ∈ U ̂ µ ( X i ) ln ( ̂ µ ( X i )+ γ 1 ̂ µ ⊗ ( X i )+ γ 2 )

5.1 A computable lower bound on epistemic uncertainty

Theorem 5.5 gives a lower bound on the epistemic uncertainty by the mutual information. However, to compute the mutual information term, in practice we need to evaluate ˜ Q on its entire support, which is potentially infinite. Practically speaking, it is impossible to observe probabilities of all strings under the language model and so we must rely on a finite sample. Therefore, we replace ˜ Q with an empirical distribution with a finite support; in the following we show that the error induced by such an approximation is controlled. To estimate the MI we employ the method given in Algorithm 1; for generality it is presented for an arbitrary (pseudo) joint distribution µ , but we keep in mind that our case of interest is µ = ˜ Q . Note that most terms in the summations defining the product distribution ̂ µ ⊗ are zero (except the ones which correspond to the observed data). Adding γ 1 and γ 2 in the estimator ̂ I k ( γ 1 , γ 2 ) is intended to account for the total probability of missing observations, not included while constructing ̂ µ and ̂ µ ⊗ , making sure the estimate is bounded.

The bias introduced by ( γ 1 , γ 2 ) in the last equation allows us to rigorously bound the error in estimating I ( µ ) via ̂ I k ( γ 1 , γ 2 ) , which is explored next. In particular, in Theorem 5.6 we prove a high-probability lower bound on I ( µ ) in terms of ̂ I k . The core of controlling the estimation error is in accounting for the missing mass , or in other words, how much of µ we miss out by only observing a finite sample. In Appendix F, we present a more complete discussion and the proof of the bound on the estimation error for mutual information. Here we adapt this result to our particular case.

Define the missing mass as U k = ∑ x ∈X n µ ( x ) I { x glyph[negationslash]∈ { X 1 , . . . , X k } } . Using this quantity, we are ready to present a non-asymptotic bound on the estimation error, which depends on the estimator ̂ I k ( γ 1 , γ 2 ) , the expected missing mass, and the sample size:

Theorem 5.6. Suppose that ̂ I k ( γ 1 , γ 2 ) is given by Algorithm 1, and assume that X is finite. For γ 1 = 1 / ( k |X n | ) , and γ 2 satisfying γ 2 ≥ γ 1 + n (1 -Z ) , with probability at least 1 -δ , we have

Furthermore, given δ supp ∈ [0 , 1) , let ˜ X ⊆ X n such that µ ( ˜ X ) ≥ 1 -δ supp . Then, for γ 1 = 1 / ( k | ˜ X| ) , and γ 2 satisfying γ 2 ≥ γ 1 + n (1 -Z ) , with probability at least 1 -δ , we have

The theorem is a corollary of Theorem F.4 shown in Appendix F. Note that in Theorem 5.6 we consider two bounds. The first one is pessimistic in the sense that it does not expect that the samples carry much information about the support, and it is most suitable in situations where we expect µ to be spread out (uniformly) across its entire support. The price of not having samples covering the whole support in

this case is a factor n ln |X| appearing in the bound. For example, in case of a language model with 10 , 000 tokens, considering all possible strings of length T tokens yields n ln |X| = nT ln(10000) , and so I ( µ ) ≥ (1 -ε k ) ̂ I k ( γ 1 , γ 2 ) -(1 /k +(1 + nT ln(1 + k ln(10000))) ε k ) . Arguably, in practice, such situations are rare, as in natural languages we will not encounter all possible strings. To this end, we consider an optimistic scenario where the effective support of µ , denoted by ˜ X , is small with high probability. In this case, we can replace the size of the support for strings of length n , |X| n , in the first bound with the effective support size | ˜ X| , and we only pay essentially a factor ln(1 + k | ˜ X| ) instead of n ln(1 + k |X| ) . In case the effective sample size is only polynomial in n , this leads to an exponential reduction in n for the second term in the bounds. In fact, in Appendix F.4 we demonstrate some empirical evidence that on two question-answering benchmarks, | ˜ X| rarely exceeds ≈ 100 with µ ( ˜ X ) ≥ 0 . 95 , while sampling responses from an LLM given a query.

Next we consider sufficient conditions for the estimator to converge to the mutual information. In particular, using the first bound in the theorem, we have (hiding logarithmic factors) I ( µ ) = ˜ Ω((1 -E [ U k ]) ̂ I k ( γ 1 , γ 2 ) -E [ U k ]) as k →∞ . This tells us that the rate of estimation error is essentially controlled by the expected missing mass E [ U k ] , which, as we will see, converges to zero as k →∞ , however the decay can be very slow in general. For example, it is known that for a finite support of size N , E [ U k ] ≤ e -k N when k ≤ N and E [ U k ] ≤ N/ ( e k ) otherwise (Berend and Kontorovich, 2012). For countable distributions with entropy bounded by h , one has E [ U k ] ≤ h/ ln( k ) .

Despite these pessimistic bounds, in reality we expect the expected missing mass to be significantly smaller, especially when µ is heavy-tailed. It is well-known that natural languages (and many artificial ones) follow a Zipf distribution, where probability of each word (or a text piece) is proportional to 1 / freq(text) α for some exponent α > 1 , where freq() is a frequency of occurrence in the corpus (Piantadosi, 2014). Then, we expect that E [ U k ] should be much smaller than in such a case, since sampling from the tail of Zipf distribution is a rare event. To this end, in Appendix F we show that if ˜ Q is Zipf with exponent α > 1 , then for any free parameter β > 0 , E [ U k ] = O ( k -( α -1 α -β ) ) . Hence, the rate at which the expected missing mass vanishes can be very fast (potentially matching a concentration rate 1 / √ k for α = 2 ).

Finally in Appendix F.4 we present a data-dependent estimation of E [ U k ] based on a concentration inequality for a missing mass and repetitive sampling from LLM, in the context of Q/A datasets showing that the expected missing mass is highly concentrated close to 0 .

5.2 Score-based hallucination tests

Let ̂ I k ( γ, x ) ≡ ̂ I k ( γ ) computed as in Algorithm 1 for µ = ˜ Q , to emphasize the explicit dependence on the query x . The uncertainty estimate ̂ I k ( γ, x ) derived above can be used as a score indicating the strength of our belief that the LLM hallucinates for the given query x . Such a score can then be used to design abstention policies: if the response is deemed to be hallucinated, the system abstains from responding, while a response is provided otherwise. Score-based abstention methods usually compute a score chosen by the user (such as the response likelihood or the estimator ̂ I ( γ ) discussed earlier), and declare hallucination if the score is above or below a threshold, which is determined through calibration. To detect hallucinations successfully, the threshold can be adjusted through calibration on a given task using a hold-out (ground-truth) sample, see, for instance, the paper of Yadkori et al. (2024) where this calibration is discussed in detail.

Given our estimated lower bound on the epistemic uncertainty, we can define an abstention policy (a policy which decides when the LLM should abstain from prediction) as a λ ( x ) = 0 if ̂ I k ( γ 1 , γ 2 , x ) < λ and a λ ( x ) = 1 if ̂ I k ( γ 1 , γ 2 , x ) ≥ λ , where λ > 0 is a threshold parameter tuned on a hold-out sample of some particular task. This policy abstains ( a λ ( x ) = 1 ) when the epistemic uncertainty in the prediction (response) is large. When the policy does not abstain ( a λ ( x ) = 0 ), any prediction from ̂ Q can be served. In the experiments, we compare a number of scoring functions for detecting hallucinations, including ̂ I ( γ ) , the probability of the greedy (temperature zero) response, and an estimate of the entropy of the response distribution.

6 Experiments

In this section we evaluate our abstention method derived based on the MI estimate in Section 5.2 on a variety of closed-book open-domain question-answering tasks. In our experiments we either sweep through all abstention thresholds (Figure 4), or optimize the threshold on some calibration data, as explained in the description of the relevant experiment (Figure 5).

Language model. We used a Gemini 1.0 Pro model (Gemini Team, Google, 2023) to generate outputs and scores. Similar results were obtained with a - much smaller - Gemini 1.0 Nano-1 model, which are deferred to Appendix H.

Datasets. We consider three different datasets and their combinations: As base datasets, we consider (i) a random subset of 50 , 000 datapoints from the TriviaQA dataset (Joshi et al., 2017), and (ii) the entire AmbigQA dataset (with 12038 datapoints) (Min et al., 2020). These datasets mostly contain single-label queries, and only contain a few multi-label ones. 5 Moreover, we created a multi-label dataset based on the WordNet dataset (Fellbaum, 1998): We extracted all (6015) datapoints from WordNet at depth 4 or more of the physical_entity subtree. For each datapoint (entity, children) in WordNet, we constructed a query of the form 'Name a type of entity . ' and children are considered target labels.

Comparison of responses and computing the output distributions. We use the F1 score thresholded at 0 . 25 to decide if two text sequences match. Additional details are provided in Appendix G.

Baselines. We consider abstention policies based on four scoring methods: (i) the probability of the greedy response (denoted by T 0 ); (ii) the semantic-entropy method of Kuhn et al. (2023) whose score is the entropy of k = 10 generated samples (denoted by S.E.); (iii) our proposed mutual information score as defined in Section 5 (and denoted by M.I.) with the choices of k = 10 , n = 2 , and γ 1 = γ 2 = 0 (the latter choice approximates the case that the number of potential responses can be very large in which case the theoretical choice of γ 1 and γ 2 would be very small); (iv) the self-verification method of Kadavath et al. (2022) (denoted by S.V.). Additional details are provided in Appendix G.

Results. We consider the precision-recall (PR) trade-off for the various methods on the different datasets. Here, recall is the percentage of queries where the method does not abstain, and precision is the percentage of correct decisions among these queries. 6 Figure 4ab show PR-curves for the baselines and the proposed method on TriviaQA and AmbigQA. As can be seen, our method is better than the T 0 and S.V. baselines, but performs similarly to the S.E. method. This is because the TriviaQA and AmbigQA datasets contain mostly single-label queries, and therefore a first-order method such as S.E. is sufficient to detect hallucinations. The AmbigQA dataset contains a few multi-label queries, but upon closer inspection, we observe that the LLM has low entropy on most of these queries.Therefore, a first-order method can perform as well as our method on such queries. Our proposed method, as well as the baselines, make no mistakes on the WordNet dataset (as the prediction of the LLM is always correct), hence we omit those results. The S.V. baseline performs significantly worse than the other methods when the recall is not high (is below about 0.8).

The similar performance for the S.E. and M.I. methods shown in Figure 4ab is due to the fact that the LLM has low entropy on most multi-label queries. However, ideally, an LLM should have higher entropy on multi-label queries (which would demonstrate broader knowledge, not focusing on a single possible answer). To include such queries, we mix the TriviaQA and AmbigQA datasets with our WordNet-based dataset with 'truely' multi-label queries as constructed above. To enhance the intended effect, we filter our WordNet dataset by keeping only queries with entropy higher than 0 . 7 (approximately the entropy of the uniform distribution over two atoms). Then we have 842 remaining datapoints in WordNet. Note that when considered in isolation, both our proposed method and the semantic entropy method rarely make mistakes on this dataset. Then we create two new datasets by combining our 842 WordNet datapoints with 842 randomly selected datapoints from TriviaQA and AmbigQA, respectively, resulting in the TriviaQA+WordNet and AmbigQA+WordNet datasets. Figure 4cd show PR-curves for the S.E. and M.I. methods on these two combined datasets. Apart from low recall values, the performance of the S.E. method degrades noticeably with the addition

(a) TriviaQA

(b) AmbigQA

(c) TriviaQA+WordNet

(d) AmbigQA+WordNet

1.00

1.0

0.94

Precision

0.95

ecision

0.9

Precision

0.92

Precision

0.80

0.90

TO score

Pre

0.8

TO score

0.90

0.75

0.85

S.V. score

M.1. score

0.7

M.1. score

S.E.score

S.V. score

0.88

S.E.score

M.1. score

0.70

S.E.score

M.1. score

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.4

0.6

0.8

1.0

0.4

0.6

0.8

1.0

Recall

Recall

Recall

Recall

(a) TriviaQA+WordNet

(b) TriviaQA+WordNet

(c) AmbigQA+WordNet

(d) AmbigQA+WordNet

0.5

0.4

M.1.

0.6

0.5

M.1.

0.25

0.30

M.1.

0.8

0.7

M.I.

Recall

E°0

0.4

0.3

les

Re

0.20 -

0.15

0.6

0.5

0.4

0.2

0.2

0.10

°0

0.1

0.1

0.05

0.2

0.1

0.0

0.10.30.50.7

0.9

Entropy

1.1

1.3

ST

1.7

1.9

0.0

0.1

0.3

0.5

0.7

0.9

Entropy

1.1

1.3

1.5

1.7

1.9

0.00

0.1

S'OE'O

0.7

6°0

Entropy

1.1

E'T

1.5

1.7

1.9

0.0

0.1

E°0

0.5

0.7

0.9

Entropy

1.1

1.3

1.5

1.7

1.9

of extra multi-label data. This precision/recall curve might look somewhat strange (with precision sometimes increasing with recall); this is due to the fact that both methods are always correct on the large number of high-entropy WordNet queries, where the LLM's default predictions are correct.

The hardness with the combined datasets is that the predominantly single-label datasets (TriviaQA, AmbigQA) might need a different calibration threshold than the multi-label WordNet dataset, and this is better handled by our proposed method than by S.E. To better illustrate the improved abstention properties of our method, we examine how the two methods handle when the output of the LLM is diverse (i.e., has high entropy). In order to do this, we perform the following experiment: We create a calibration dataset by adding 500 random datapoints from the WordNet dataset to 500 random datapoints from TriviaQA, and another such random dataset for test. We determine the abstention thresholds on the calibration dataset for both the S.E. and the M.E. methods, 7 and measure the performance (error rate, i.e., 1 minus precision, and recall) of the resulting abstention policies on the test set. We repeat this process 10 times and report mean values and 95% confidence intervals with Gaussian approximation. We perform a similar evaluation process for mixtures of AmbigQA and WordNet datasets. Figure 5 show that while the S.E. method has similar recall and error rates to those of the proposed method on low-entropy queries, its recall values are much lower for queries with higher entropy, while the M.E. method makes only few mistakes on these queries.

References

Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L. Edelman. Distinguishing the knowable from the unknowable with language models, 2024. URL https://arxiv.org/abs/ 2402.03563 .

Anastasios N Angelopoulos, Stephen Bates, et al. Conformal prediction: A gentle introduction. Foundations and Trends R © in Machine Learning , 16(4):494-591, 2023.

Javier Antorán, James Allingham, and José Miguel Hernández-Lobato. Depth uncertainty in neural networks. Conference on Neural Information Processing Systems (NeurIPS) , 2020.

Amos Azaria and Tom Mitchell. The internal state of an LLM knows when its lying. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023.

Daniel Berend and Aryeh Kontorovich. The missing mass problem. Statistics & Probability Letters , 82(6):1102-1110, 2012.

Daniel Berend and Aryeh Kontorovich. On the concentration of the missing mass. Electronic Communications in Probability , 2013.

Daniel Berend, Aryeh Kontorovich, and Gil Zagdanski. The expected missing mass under an entropy constraint. Entropy , 19(7):315, 2017.

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learing (ICML) , 2015.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In International Conference on Learning Representations (ICLR) , 2023.

Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs' internal states retain the power of hallucination detection. In International Conference on Learning Representations (ICLR) , 2024a.

Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation. In Forty-first International Conference on Machine Learning , 2024b. URL https://openreview. net/forum?id=s3e8poX3kb .

Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation, 2023.

Jeremy R. Cole, Michael JQ Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. Selectively answering ambiguous questions. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2019.

Vikranth Dwaracherla, Zheng Wen, Ian Osband, Xiuyuan Lu, Seyed Mohammad Asghari, and Benjamin Van Roy. Ensembles for uncertainty estimation: Benefits of prior functions and bootstrapping. Transactions on Machine Learning Research (TMLR) , 2023. ISSN 2835-8856.

Farquhar, J. Kossen, L. Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature , 2024.

Christiane Fellbaum. WordNet: An electronic lexical database . MIT press, 1998.

Yarin Gal. Uncertainty in deep learning . PhD thesis, University of Cambridge, 2016.

Gemini Team, Google. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023. [Online; accessed 01-February-2024].

Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. Decomposing uncertainty for large language models through input clarification ensembling. In International Conference on Machine Learing (ICML) , 2024.

Mingjian Jiang, Yangjun Ruan, Sicong Huang, Saifei Liao, Silviu Pitis, Roger Grosse, and Jimmy Ba. Calibrating language models via augmented prompt ensembles. In Workshop on Challenges in Deployable Generative AI at International Conference on Machine Learning , 2024.

Daniel D. Johnson, Daniel Tarlow, David Duvenaud, and Chris J. Maddison. Experts don't cheat: Learning what you don't know by predicting pairs. arXiv preprint arXiv:2402.08733 , 2024.

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Transactions of the Association for Computational Linguistics (ACL) , pages 1601-1611, 2017.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, and et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 , 2022.

Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Transactions of the Association for Computational Linguistics (ACL) , 2020.

Satyapriya Krishna. On the intersection of self-correction and trust in language models, 2023. URL https://arxiv.org/abs/2311.02801 .

Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. Understanding the effects of iterative prompting on truthfulness. In Forty-first International Conference on Machine Learning , 2024. URL https://openreview.net/forum?id=KjazcKPMME .

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In International Conference on Learning Representations (ICLR) , 2023.

Philippe Laban, Lidiya Murakhovs'ka, Caiming Xiong, and Chien-Sheng Wu. Are you sure? challenging llms leads to performance drops in the flipflop experiment, 2024. URL https: //arxiv.org/abs/2311.08596 .

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Conference on Neural Information Processing Systems (NeurIPS) , volume 30, 2017a.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Conference on Neural Information Processing Systems (NeurIPS) , 2017b.

Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. Large language models with controllable working memory. In Transactions of the Association for Computational Linguistics (ACL) , 2023.

Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. Think twice before trusting: Self-detection for large language models through comprehensive answer reflection, 2024. URL https://arxiv.org/abs/2403.09972 .

Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187 , 2023.

Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations (ICLR) , 2020.

Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023.

Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. In Transactions of the Association for Computational Linguistics (ACL) , 2022.

Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020.

M. Neal. Bayesian learning for neural networks. Springer Science & Business Media , 2012.

Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering. arXiv preprint arXiv:2211.05655 , 2022.

Mesrob I Ohannessian and Munther A Dahleh. Distribution-dependent performance of the goodturing estimator for the missing mass. In 19th International Symposium on Mathematical Theory of Networks and Systems, MTNS , 2010.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Conference on Neural Information Processing Systems (NeurIPS) , volume 29, 2016.

Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. In Conference on Neural Information Processing Systems (NeurIPS) , 2023.

Steven T Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review , 21:1112-1130, 2014.

Yury Polyanskiy and Yihong Wu. Information theory: From coding to learning . Cambridge University Press, 2024.

Stephan Rabanser, Anvith Thudi, Kimia Hamidieh, Adam Dziedzic, and Nicolas Papernot. Selective classification via neural network training dynamics. arXiv preprint arXiv:2205.13532 , 2022.

Shauli Ravfogel, Yoav Goldberg, and Jacob Goldberger. Conformal nucleus sampling. In Transactions of the Association for Computational Linguistics (ACL) , pages 27-34. Association for Computational Linguistics, 2023.

Robert J Tibshirani and Bradley Efron. An introduction to the bootstrap. Monographs on statistics and applied probability , 57(1), 1993.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR) , 2022.

Ziyu Wang and Chris Holmes. On subjective uncertainty quantification and calibration in natural language generation, 2024. URL https://arxiv.org/abs/2406.05213 .

Zheng Wen, Ian Osband, Chao Qin, Xiuyuan Lu, Morteza Ibrahimi, Vikranth Dwaracherla, Mohammad Asghari, and Benjamin Van Roy. From predictions to decisions: The importance of joint predictive distributions. arXiv preprint arXiv:2107.09224 , 2022.

Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev. Mitigating llm hallucinations via conformal abstention. arXiv preprint arXiv:2405.01563 , 2024.

Fan Yin, Jayanth Srinivasa, and Kai-Wei Chang. Characterizing truthfulness in large language model generations with local intrinsic dimension. In Forty-first International Conference on Machine Learning , 2024. URL https://openreview.net/forum?id=7DbIyQlfaO .

Gal Yona, Roee Aharoni, and Mor Geva. Narrowing the knowledge evaluation gap: Open-domain question answering with multi-granularity answers. arXiv preprint arXiv:2401.04695 , 2024.

Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley Malin, and Sricharan Kumar. Sac3: Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023.

Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In In International Conference on Machine Learning , 2021.

Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. Knowing what LLMs do not know: A simple yet effective self-detection method. arXiv preprint arXiv:2310.17918 , 2024.

A Additional algorithms for taking semantic equivalences into account

Although Theorem 5.5 and Theorem 5.6 provide a lower bound for the divergence between the output distribution Q of the LLM and the ground truth P , these distributions ignore the semantic equivalences between texts, and hence they are less concentrated than their variants which only consider semantically different outputs. Given a semantic equivalence definition, similarly to Kuhn et al. (2023); Farquhar et al. (2024), we propose constructing new ground-truth distribution P ′ and LLM output distribution Q ′ , where we cluster together semantically equivalent texts, and the probability of a cluster is the sum of probabilities of all semantically equivalent texts in that cluster. We use a similarity function s to define semantic equivalences: two texts are considered equivalent if their similarity is greater than a given threshold τ . Our choices for similarity functions in the experiments are described in Section 6. We assume the similarity function and the similarity threshold induce a clustering of the space X , i.e., s ( Y, Y ′ ) ≥ τ for Y, Y ′ ∈ X if and only if they are in the same cluster.

In practice, rather than constructing the aforementioned distribution Q ′ explicitly, we can draw samples from Q ′ by sampling from Q and aggregating samples according to their clusters. The modified uncertainty-estimation algorithm is given in Algorithm 2 in Appendix A. The estimator is constructed using only (semantically) different elements in the sample (the indices of these representative elements are collected in S ), that is, we do not account for duplicate samples and we aggregate probabilities of samples that are lexically different but semantically equivalent. Algorithm 2 works with the aggregated probability distribution µ ′ = ˜ Q ′ (line 4) by summing over cumulative probabilities over clusters. Note that D KL ( µ ) ≥ D KL ( µ ′ ) by monotonicity property of KL-divergence (Polyanskiy and Wu, 2024, Theorem 2.16) (this is because µ ′ is defined on a smaller support). Therefore, Theorem 5.6 implicitly gives a bound on I ( µ ′ ) , and eventually we have I ( µ ) ≥ I ( µ ′ ) . More importantly, we can also directly apply Theorem 5.5 and Theorem 5.6 to the distributions P ′ and Q ′ and obtain that D KL ( ˜ Q ′ , ˜ P ′ ) ≥ I ( ˜ Q ′ ) , giving a lower bound on the much reduced (epistemic) uncertainty after taking semantic equivalences into account.

1: Input :

µ ∈ M 1 ( X n ) . . . . . . . . . . . . . . . . . . . any (pseudo-) joint distribution over X n k ∈ N . . . . . . . . . . . . . . . . . . . . . . . . . . sample size γ 1 , γ 2 ≥ 0 . . . . . . . . . . . . . . . . . . . . . . stabilization parameters (typically selected as 1 /k )

s : X n ×X n → R . . . . . . . . . . . . . . a similarity function

τ ∈ R . . . . . . . . . . . . . . . . . . . . . . . . . . a similarity threshold

2: Independently sample tuples X 1 , . . . , X k ∼ µ ∈ M 1 ( X n )

glyph[negationslash]

3: Construct a set of indices of unique elements U = { i ∈ [ k ] : X i = X j ∀ j < i }

4: Construct cluster centers S ⊂ U according to the similarity function such that for all i, t ∈ S , we have s ( X i , X t ) < τ and the cluster associated with X i is D ( i ) = { j ∈ U : s ( X i , X j ) ≥ τ } . Compute the aggregated probabilities: for all i ∈ S ,

5: Construct empirical distributions: for all i ∈ S ,

6: Compute estimate

1: Input :

µ ∈ M 1 ( X ) . . . . . . . . . . . . . . . . . . . . any distribution over

X

k ∈ N . . . . . . . . . . . . . . . . . . . . . . . . . . sample size

γ 1 , γ 2 ≥ 0 . . . . . . . . . . . . . . . . . . . . . . stabilization parameters (typically selected as

1 /k )

s : X × X → R . . . . . . . . . . . . . . . . . a similarity function

τ ∈ R . . . . . . . . . . . . . . . . . . . . . . . . . . a similarity threshold

2: Independently sample outputs X 1 , . . . , X k ∼ µ ∈ M 1 ( X )

3: Construct a set of indices of unique elements U = { i ∈ [ k ] : X i = X j ∀ j < i }

glyph[negationslash]

4: Construct cluster centers S ⊂ U according to the similarity function: for all i, t ∈ S , we have s ( X i , X t ) < τ and cluster associated with X i is D ( i ) = { j ∈ U : s ( X i , X j ) ≥ τ } . Aggregated probabilities: for all i, t ∈ S ,

5: Construct empirical distributions: for all i, t ∈ S ,

6: Compute estimate

B Implementation and usage examples of Algorithm 1 and Algorithm 2

In this section we present an implementation of Algorithm 1 and Algorithm 2 in Python with a simple usage example. In particular, the code given in Listing 1 generates a synthetic joint distribution over binary tuples with correlated elements (function create_synthetic_distribution ). Then, we compute an exact mutual information of the distribution (function compute_MI_exactly ) and use implementation of our estimator (function MI_estimator ) to estimate a mutual information. This is done for various sample sizes, number of random variables, and levels of correlation (a single experiment is implemented by run_experiment ) The results of these multiple experiments are eventually presented in as plots showing convergence of the estimate to the exact value of the mutual information. In practical applications, synthetic joint distribution (function create_synthetic_distribution ) can be replaced by an LLM-derived pseudo-joint distribution (see Definition 5.1). More detailed description of each function is given in Appendix B.1.

The example can be easily copied from listing.tex within https://arxiv.org/src/ 2406.02543 .

Listing 1: Implementation and usage examples of Algorithm 1 and Algorithm 2 on a synthetic joint distribution

C Usage example of Algorithm 3

Algorithm 3 is a slight modification of Algorithm 2 that we use in our experiments. We first explain Algorithm 3 via an example and then highlight the differences with Algorithm 2. In order to explain the implementation, we consider a running example with query x = 'What is the capital of the UK?' , F1 score as the similarity function s , similarity threshold τ = 0 . 25 , and number of samples k = 5 . Algorithm 3 also takes the LLM distribution Q as input µ = Q .

Given the query, in step (2) of Algorithm 3, we sample k outputs from Q . Let's assume these samples are X 1 = 'London' , X 2 = 'London' , X 3 = 'London, UK' , X 4 = 'Paris' , and X 5 = 'Berlin' . In step (3), we construct a set of indices of unique elements. In our example, we would have U = { 1 , 3 , 4 , 5 } . In step (4), we cluster responses and aggregate probabilities of each cluster. More precisely, if the F1 score of two responses is above 0.25, then they are in the same cluster. In our example, we have that F1 ( X 1 , X 3 ) > 0 . 666 > 0 . 25 and F1 ( X 1 , X 4 ) = F1 ( X 1 , X 5 ) = 0 , and therefore cluster centers are S = { 1 , 4 , 5 } . For query x , let's assume LLM probabilities are

Also assume conditional distributions are

and so on (we have omitted writing Q ( . | F 1 ( x, X 4 )) and Q ( . | F 1 ( x, X 5 )) ). Then after step (4), the aggregated probabilities are

and aggregated conditional probabilities are

and so on (we have similar aggregations for Q ′ ( . | F 1 ( x, X 4 )) and Q ′ ( . | F 1 ( x, X 5 )) ). Next, in step (5), we construct empirical estimates. We will have that Z = 0 . 85 and

For estimated conditional distributions, we will have Z 1 = 0 . 84 , and

and so on. The joint distribution ̂ Q ( ., . | x ) , the product of marginals ̂ Q ⊗ ( ., . | x ) , and the estimated mutual information ̂ I k are trivially obtained by the equations in steps (5) and (6).

Next, we highlight the differences between Algorithm 2 (with the choice of n = 2 ) and Algorithm 3. The input distribution to Algorithm 2 is the pseudo-joint distribution ˜ Q , while the input to Algorithm 3 is the LLM distribution Q . So in step (2) of Algorithm 2, each sample is a tuple such as ( 'London' , 'Paris' ), while a sample in step (2) of Algorithm 3 is an LLM output such as 'London' . Steps (3) and (4) of Algorithm 2 are similarly modified, and now the similarity function is defined over tuples.

D Related work

In this section we present an overview of the related literature.

D.1 Bayesian neural networks

In a Bayesian framework, we can estimate the epistemic uncertainty by the uncertainty in the posterior distribution (Neal, 2012; Gal, 2016; Wang and Holmes, 2024). Implementing a Bayesian neural network however can be very challenging.

D.2 Iterative prompting

A number of iterative prompting strategies are developed to improve the factuality of LLMs (Chen et al., 2023; Krishna, 2023; Laban et al., 2024). The idea is to follow-up the LLM response with another question such as 'Are you sure?' . Krishna et al. (2024) show that such strategies might in fact degrade LLM truthfulness due to a pattern of apologetic responses. Krishna et al. (2024) propose improved iterative prompting strategies, where instead of asking the LLM to re-think its response, the same question is posed again. They also propose strategies to collect more supporting facts and refine the final response accordingly. Li et al. (2024) propose an iterative prompting that instructs LLM to generate justifications for each answer before evaluating the correctness of the final answer. Different from these works, we assess hallucinations by measuring how LLM response changes with our iterative prompting scheme.

Perhaps the most related work to ours is the parallel and independent work of Ahdritz et al. (2024). Similar to us, Ahdritz et al. (2024) observe that in presence of high epistemic uncertainty, an LLM is more likely to copy the information provided in its context. For a given query, Ahdritz et al. (2024) propose considering topk completions of the model, and then computing the entropy of the model conditioned on an iterative prompt composed of the original query and each completion. The minimum of these entropies is considered as a measure of the epistemic uncertainty. The method as it is, might fail on single-response queries where the model has low uncertainty. Nevertheless, we can design a two-stage process where we only consider completions that have probability higher than certain threshold in the first stage, and then compute the entropy of the model conditioned on an iterative prompt composed of the original query and each candidate completion in the second stage. By a proper tuning of the threshold of the first stage, we can potentially avoid mis-classification of low-uncertainty single-response queries. Tuning this threshold, however, would introduce extra complications. In contrast, we propose a principled test using a mutual information score that is guaranteed to be a lower bound on the KL-divergence between the LLM and the ground-truth. Further, we provide a mechanistic explanations for why LLMs behave as described in the presence of high epistemic uncertainty.

D.3 Training models with pairs of responses

Wen et al. (2022); Osband et al. (2023); Johnson et al. (2024) show that we can decouple epistemic and aleatoric uncertainty if we train a model with paired observations.

We discuss the more recent work of Johnson et al. (2024) in more detail. The proposed approach first estimates a model ˆ p Y 1 ,Y 2 | x ( y 1 , y 2 | x ) over pairs using a training dataset of the form 'query, first observation, second observation'. At test time, for a prompt x and response y , Johnson et al. (2024) consider

as a measure of epistemic uncertainty. Assume that an equivalence class Φ (that maps a prompt to the set of equivalent prompts) is given, and let ν ( . | Φ( x )) be a distribution (say, uniform) over class Φ( x ) . If the trained model is second order calibrated with respect to the equivalence class and the

distribution ν , i.e.

then it follows from definitions that in the class associated with x ,

The quantity on the right-hand side is a measure of epistemic uncertainty. Notice that the equality states a coverage result, and it is not point-wise. Requiring the model to be second order calibrated is also a strong condition and ensuring it is highly non-trivial.

D.4 Epistemic neural nets

Ensemble methods are based on the classical idea of bootstrap for confidence estimation (Tibshirani and Efron, 1993), where multiple estimators for the regression function, each computed on a perturbed version of the data (e.g., by drawing samples from the empirical distribution over the data), are combined.

The empirical distribution of the resulting estimates is then used to construct confidence intervals. While many of these methods can be interpreted as sample-based approximations to Bayesian methods, model-hyperparameter selection (e.g., scale of perturbations, learning) for ensemble methods is typically done using a validation on holdout data (a subset of the training data). Many recent papers have studied ensemble methods in the context of deep learning and reinforcement learning (Osband et al., 2016; Lakshminarayanan et al., 2017a; Malinin and Gales, 2020). In the context of LLMs, the methods require training multiple language models, which is very expensive. Osband et al. (2023) introduces epistemic neural networks (epinets), which approximate ensemble methods by training a single network with an artificially injected (controlled) source of randomness. Rabanser et al. (2022) proposes to use intermediate model checkpoints to quantify the uncertainty of the final model in its responses. While these approaches aim to mimic the bootstrap procedure during prediction, their validity is not justified by theoretical considerations, and hence remain heuristic approximations.

D.5 Hallucination detection using first-order methods

First-order methods consider variance in the response distribution as a measure of hallucination (Kadavath et al., 2022; Cole et al., 2023; Manakul et al., 2023; Lin et al., 2023; Kuhn et al., 2023; Wang et al., 2022; Jiang et al., 2024; Zhang et al., 2023; Zhao et al., 2024; Yadkori et al., 2024). A common limitation of these approaches is that they are only applicable to prompts where there exists a single correct response, as they aim for detecting if one response (or multiple responses with the same meaning) is dominant. On the other hand, when multiple responses are correct, there is an aleatoric uncertainty in the ground truth: If an LLM correctly assigns non-negligible scores to multiple correct responses, most of these (if not all) will be declared as hallucination since, by design, only very few (typically at most one) responses can have scores higher than the threshold at the same time. Thus, hallucination detectors unaware of aleatoric uncertainty will invalidate most of the correct answers.

Yona et al. (2024) design a method that generates multiple responses, and then aggregates them into a single response at a (typically higher) granularity level where no further uncertainty (contradiction) is left compared to the generated responses. Although not a strictly first order method, it does not differentiate between aleatoric and epistemic uncertainty.

D.5.1 Asking language models to quantify uncertainty (self-verification)

Kadavath et al. (2022) propose to use LLM self-prompting to measure a model's uncertainty in its responses. More specifically, for a given query, a number of responses are generated, and then the model is queried if the responses are correct. For this query, the log-probability of 'True" is returned as a measure of uncertainty. Related approaches are studied by Mielke et al. (2022).

D.6 Uncertainty estimation based on sensitivity to contexts

Kassner and Schütze (2020); Zhao et al. (2021) show that an LLM's responses can be influenced by irrelevant contexts. Longpre et al. (2021); Neeman et al. (2022) study two sources of knowledge: parametric knowledge stored in the network weights, and contextual knowledge retrieved from external sources. They view reliance of the model on its parametric knowledge and ignoring relevant contextual information as hallucination. These works are mainly motivated by situations where the LLM's knowledge is outdated and it is instructed to use the (new) contextual information. Accordingly, they design strategies to prioritize contextual information over parametric knowledge. Longpre et al. (2021) also show that larger models are more likely to ignore in-context information in favor of in-weight information. They propose creating training data with modified contextual information so that the model learns to favor the contextual information. Neeman et al. (2022) propose to train a model that predicts two answers: one based on parametric knowledge and one based on contextual information.

Similarly to Neeman et al. (2022), Li et al. (2023) aims to design a mechanism such that the model's behavior is influenced more by relevant context than by its parametric knowledge (controllability), while the model is robust to irrelevant contexts (robustness). They improve controllability and robustness using finetuning.

Hou et al. (2024) study an approach to estimate model uncertainty due to ambiguity in a question. For a given question, their method generates multiple input clarification questions, and a new question is formed by augmenting the original question with each clarification question. The clarification questions are generated using an LLM with the aim of removing ambiguity in the question. This is different than the problem we study as the model can be uncertain about the answer even if the query itself has no ambiguity. For such queries, the method of Hou et al. (2024) might decide that no clarification is needed, and therefore there is no uncertainty.

D.7 Hallucination detection using internal states of LLMs

There are a number of papers that try to extract knowledge/truthfulness by inspecting hidden-layer activations of LLMs (Burns et al., 2023; Azaria and Mitchell, 2023; Chen et al., 2024a,b; Yin et al., 2024). Such methods clearly require access to the LLM's internal states, which is not always possible, and severely limits the applicability of these methods.

E Omitted proofs

Proof of Theorem 5.5. In the following we will use abbreviations

where each n -tuple y belongs to Y . Now,

Focusing on the last (cross-entropy) term

where in ( a ) we used the fact that entropy is no larger than cross-entropy. Thus,

F Estimation of mutual information and missing mass problem

In this section, we discuss how to estimate the mutual information from a finite sample, which may not cover the full distribution. To control the estimation error, we first introduce the concept of missing mass .

F.1 The missing mass problem

Let X be a countable set and suppose that X 1 , . . . , X k ∼ µ ∈ M 1 ( X n ) independently. In the following x is used as an element of X n rather than the query (as in Section 5). Then, the missing mass is defined as the random variable

Here we are primarily interested in two questions: (i) how quickly U k approaches the expected missing mass E U k , where it is not hard to see that

and (ii) we are also interested in giving an estimate for E U k given µ and k . The first question is answered by the following theorem:

Theorem F.1 (Concentration of a missing mass (Berend and Kontorovich, 2013)) . For any t > 0 , we have an upper-tail bound

and moreover for a universal constant c ≈ 7 . 6821 , we have an lower-tail bound

√

Notably U k exhibits a sub-gaussian concentration (i.e. 1 / k ), which is surprisingly fast. As we will see next, the main bulk of the error incurred for missing a subset of the support is hidden in E U k .

In particular, when X is finite with |X| = N , Berend and Kontorovich (2012) showed that

In the countably infinite X , we cannot generally have a non-trivial bound on E U k only in terms of n . In fact, Berend and Kontorovich (2012) show a bound that depends on µ which is expected to be finite for rapidly decaying atoms. Interestingly, when the entropy of µ is bounded, one has the following result (Berend et al., 2017):

Note that these estimates are very pessimistic, and in reality we expect the expected missing mass to be significantly smaller. Since natural (and many artificial) languages follow a Zipf distribution (Piantadosi, 2014), we expect that E [ U k ] should be much smaller than in the above cases, since sampling from the tail of a Zipf distribution is a rare event. In Appendix F.4 we show the following:

Corollary F.3 (Expected missing mass of Zipf distribution) . Consider distribution µ ( i ) = i -α /H ( α, N ) for i ∈ [ N ] , where α > 1 and H ( α, N ) = ∑ N i =1 i -α . Then, for any β > 0 ,

Proof. The statement followss by combining Lemma F.9 and Proposition F.10.

F.2 Estimating mutual information from the partial support

Our goal is to estimate

by only having access to X 1 , . . . , X k ∼ µ . Note that that the sample might cover only some part of the support of X and therefore we are facing a missing mass problem. In the following we consider estimator ̂ I k ( γ ) given by Algorithm 1.

In particular in Appendix F.3 we show the following

Theorem F.4. Fix ˜ X ⊆ X n . Fix γ 1 > 0 and suppose that γ 2 ≥ n (1 -Z ) + γ 1 . Then for any fixed δ ∈ (0 , 1) , with probability at least 1 -δ ,

where

In particular, Theorem F.4 implies the following:

Corollary F.5. Under conditions of Theorem F.4, there exists ( γ ∗ 1 , γ ∗ 2 ) ∈ (0 , 1) 2 such that

Note that, choosing any of the upper bounds on E U k discussed in Appendix F.1, we can see that Corollary F.5 implies asymptotic convergence in as a sense

F.3 Proof of Theorem F.4

The proof will heavily rely on the simple fact that

glyph[negationslash]

Recalling that S = { i ∈ [ k ] : X i = X j ∀ j < i } , this immediately implies the following connection between U k and the quantities used in Algorithm 1:

Proposition F.6. We have that

Recall that the product distribution of µ is defined as

Note that we use ∑ x \ i µ ( · · · ) instead of µ ( x i ) since these are not necessarily equal for some µ . Now, using the definitions of ̂ I k , ̂ µ , and ̂ µ ⊗ ,

Now we control each of the terms individually. To control ( i ) we will first need the fact that q ln(( q + γ 1 ) /p ) ≤ q ln( q/p ) + γ 1 for any q, p ∈ [0 , 1] , γ 1 > 0 . Note that this follows since

using that ln(1 + a ) ≤ a for a > -1 . Getting back to ( i ) , and using the aforementioned inequality, we get

Furthermore,

Next, observe that ( iii ) ≤ ln(1 /Z ) . Finally, term ( iv ) is controlled through the following fact shown at the end of this section:

Lemma F.7. Suppose that γ 1 ≥ 0 while γ 2 ≥ γ 1 + n (1 -Z ) . Then,

Putting everything together, we obtain

Finally, multiplying through by Z the entire inequality, and using the fact that Z ln(1 /Z ) ≤ 1 -Z , we get

To complete the proof we need to give a lower bound on Z . Note that Z = 1 -U k by the definition of Z and Proposition F.6, and so by Theorem F.1

Using this concentration bound together with the choices of γ (also setting δ supp = 0 for the first inequality in the main statement) completes the proof of Theorem F.4. glyph[square]

Proof of Lemma F.7. Observe that

Now, using that fact that

we arrive at

where to get ( a ) we used:

Proposition F.8. For any p 1 , . . . , p n ∈ [0 , 1] and a ≥ 0 , we have

Proof. The statement following by lower-bounding the left-hand side by its linearization in a (derivative at 0 ), while realizing that it is a convex function of a .

The above gives us that

and focusing on the case 1 -ξ ( x ) = 1 (otherwise both sides are 0 ) the above equals to

by setting γ 2 ≥ γ 1 + n (1 -Z ) .

F.4 Expected missing mass under Zipf distribution

Wewill rely on some machinery used by Ohannessian and Dahleh (2010) who established distributiondependent bounds on the expected missing mass. As before let µ be supported on a countable set. The accrual function is defined as

and moreover the accrual rates are defined as

We use the following result:

Lemma F.9 (Ohannessian and Dahleh, 2010, Theorem 1) . Let µ have lower and upper accrual rates 0 < ρ ≤ ρ < ∞ . Then for every β > 0 there exists k 0 such that for all k > k 0 we have:

or, equivalently, for every β > 0 we have that E [ U k ] is both Ω( k -( p + β ) ) and O ( k -( p -β ) ) . Proposition F.10. Consider the distribution µ ( v ) = i -α /H ( α, N ) for i ∈ [ N ] where α > 1 and H ( α, N ) = ∑ N i =1 i -α . Then, ρ = Ω( α -1 α ) as N →∞ .

Proof. The idea is to use Lemma F.9 to give an upper bound on the missing mass. Therefore, we need to establish a lower bound on ln F ( v ) . For now, abbreviate

First note that for some 1 ≤ u ≤ N

On the other hand,

So, and then

Data-dependent estimate of the expected missing mass We perform an experiment designed to give a data-dependent estimate of the expected missing mass E [ U k ] for some specific datasets. Clearly, we cannot simply apply a concentration bound discussed in Appendix F.1 since the complete support of the pseudo joint distribution derived from the LLM is unknown. To this end, we approximate it with a finite support driven by the language model itself. In particular, given a query we sample responses (at temperature 0 . 9 ) until their total probability mass reaches 0 . 95 or we reach 1000 responses per query. In case of TriviaQA, we performed 1233 queries in total. The mean and the median number of unique responses per query was eventually 118 . 3 and 22 , respectively. In case of the AmbigQA dataset, we performed 700 queries, while the mean and the median number of unique responses was 277 and 69 , respectively.

At this point, we denote the set of responses by ˜ X and let ˜ U k be the missing mass computed on ˜ X . Then, we have

which can be computed in practice. In Figure 6 we present our results in the form of empirical distributions of different quantities, where each observation corresponds to a single query. We compute the bounds for TriviaQA and AmbigQA datasets (see Section 6 for details about these datasets). From Figure 6 we can conclude that the expected missing mass for both datasets is very small: Both the missing mass computed on ˜ X and the resulting upper bound on E [ U k ] are concentrated close to 0 , while the cumulative probability of the approximate support ˜ X is close to 1 most of the time, showing that our approximations are meaningful.

0.2

0.4

0.6

0.8

1.0

Value of upper bound

0.0

0.2

0.4

0.6

0.8

1.0

Frequency

H.p. upper bound on

[

Uk

]

0.0

0.2

0.4

0.6

0.8

1.0

Value of

Uk

0.0

0.2

0.4

0.6

0.8

1.0

Frequency

Uk

0.2

0.4

0.6

0.8

1.0

Probability

0.0

0.2

0.4

0.6

0.8

1.0

Frequency

Total mass of sampled responses

TriviaQA dataset

0.2

0.4

0.6

0.8

1.0

Value of upper bound

0.0

0.2

0.4

0.6

0.8

1.0

Frequency

H.p. upper bound on

[

Uk

]

0.0

0.2

0.4

0.6

0.8

1.0

Value of

Uk

0.0

0.2

0.4

0.6

0.8

1.0

Frequency

Uk

0.2

0.4

0.6

0.8

1.0

Probability

0.0

0.2

0.4

0.6

0.8

1.0

Frequency

Total mass of sampled responses

AmbigQA dataset

G Additional experiments details

Comparison of responses and computing the output distributions. We use the F1 score 8 thresholded at 0 . 25 to decide if two text sequences match. When multiple responses are sampled, we approximate the output distribution of an LLM in a semantically meaningful way by collapsing matching responses into a single response: we sample k = 10 responses at temperature 0 . 9 for each query, and after eliminating repetitions, all those that match (according to the F1 score) are considered identical and their probabilities are aggregated. We only consider queries for which the greedy (temperature zero) and at least one of the random responses are shorter than 20 characters. This is because the F1 score (as a match function) and log-probabilities (as a measure of uncertainty) are less reliable for longer sequences. After this filtering, we are left with 38870 datapoints for TriviaQA, 5315 datapoints for AmbigQA, and 3296 datapoints for WordNet.

As shown in prior works (e.g. Kuhn et al. (2023); Yadkori et al. (2024)), we can use LLM selfprompting to obtain more reliable text comparisons specially for longer outputs. Such an approach however is computationally much more expensive.

Baselines. We consider abstention policies based on four scoring methods. The first three are as follows: (i) the probability of the greedy response (denoted by T 0 ); (ii) the semantic-entropy method of Kuhn et al. (2023) whose score is the entropy of k = 10 generated samples (denoted by S.E.). To calculate entropy, we first aggregate probabilities of equivalent responses and normalize the probabilities so that they sum to 1 (as described above); and (iii) our proposed mutual information score as defined in Section 5 (and denoted by M.I.) with the choices of k = 10 , n = 2 , and γ 1 = γ 2 = 0 (the latter choice approximates the case that the number of potential responses can be very large in which case the theoretical choice of γ 1 and γ 2 would be very small). To calculate the mutual information, as shown in Algorithm 3 (given in Appendix A), we first generate k = 10 random samples. Then for any response Y , we calculate the probability of all generated responses given the prompt F 1 ( x, Y ) . We construct estimates ̂ Q ( Y ) and ̂ Q ( Y ′ | Y ) by aggregating probabilities of equivalent responses, and normalizing the probabilities so that they sum to 1.

The calculation of the mutual information is slightly different than the algorithms presented in Algorithm 1 and Algorithm 2 and takes advantage of the available log-likelihood function in LLMs. Notice that the input µ in Algorithm 3 is the LLM's output distribution Q as opposed to being the pseudo joint distribution ˜ Q in Algorithm 1. Another difference is that the similarity function s now takes two texts as input (as opposed to taking two n -dimensional arrays of texts as inputs in Algorithm 2). As explained earlier, we use the F1 score as the similarity function and we use τ = 0 . 25 as the similarity threshold.

Each baseline also has a default choice which is taken when the relevant score is above a threshold, and hence the method does not abstain. For T 0 , the default choice is the greedy (temperature zero) response. For S.E., the default choice is the response with the highest (aggregate) probability among the generated random responses. For the M.I. method, the default choice is the sampled response with the highest probability according to the marginalized pseudo joint distribution.

We also consider a version of the self-verification method of Kadavath et al. (2022) (denoted by S.V.) that, for a query x , first finds Y 1 , the element with the largest (aggregated) probability (which is the default choice of S.E. method), and then calculates the probability of token 'True' (normalized for the two tokens 'True' and 'False' ) for the following query: 'Consider the following question: Q: x . One answer to question Q is Y 1 . Is the above answer to question Q correct? Answer True or False. A:' . The default choice of this baseline is the same as the default choice of the S.E. method. By this design, our intention is to construct a score that (unlike the first-order scores 9 we consider) is not sensitive to the size of the label set.

H Experiments with Gemini 1.0 Nano-1 model

In order to show that our findings are not limited to models of certain size, we conducted experiments with much smaller Gemini Nano model with 2B parameters. The results are shown in Figures 7 and 8.

(a) TriviaQA

(b) AmbigQA

(c) TriviaQA+WordNet

(d) AmbigQA+WordNet

0.48

0.46

0.23

0.65

0.60

0.55

0.60

Precision

0.44

ision

0.22

0.55

0.50

0.42

0.40

0.38

0.34

0.36

M.1. score

S.E.score

0.21

0.20

0.19

0.18

M.1. score

UOI

0.50

0.45

0.40

0.35

M.1. score

0.45

0.35

0.30

0.25

0.40

M.1. score

0.2

0.4

Recall

0.6

0.8

1.0

0.2

0.4

Recall

0.6

0.8

1.0

0.2

0.4

Recall

0.6

0.8

1.0

0.20

0.2

0.4

Recall

0.6

0.8

1.0

(a) TriviaQA+WordNet

(b) TriviaQA+WordNet

(c) AmbigQA+WordNet

(d) AmbigQA+WordNet

0.6

0.6

1.0

M.1.

M.1.

1.0

M.1.

M.1.

0.8

0.5

0.8

0.5

0.4

0.4

Recall

0.6

Error

0.3

Recall

0.6

Error

0.3

0.4

0.2

0.4

0.2

0.2

0.1

0.2

0.1

0.0

0.1 0.3 0.5 0.7 0.9 1.1 1.3 1.5 1.7 1.9

0.0

0.1 0.3 0.5 0.7 0.9 1.1 1.3 1.5 1.7 1.9

0.0

0.1 0.3 0.5 0.7 0.9 1.1 1.3 1.5 1.7 1.9

0.0

0.1 0.3 0.5 0.7 0.9 1.1 1.3 1.5 1.7 1.9

Entropy

Entropy

Entropy

Entropy

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Main claims made in the abstract and introduction accurately reflect the paper's scope, contributions, claims, and results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in the introduction.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For each theoretical result, the paper provides the full set of assumptions and proofs. Some proofs are included in appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, we disclose all the required information to reproduce our results in experimental section.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All the data is publicly available. Code is currently not available: our organizational policy forbids immediate publication of the source code without an internal review process. As the review process is long, it is unlikely that NeurIPS deadline could be met if we were to undergo a code review process.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details are included in experimental section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Confidence intervals are present in all relevant plots (with details therein).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This is provided in the experimental section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors followed the principles outlined in NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Broader impact statement is made at the beginning of the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work uses existing and publicly-available language models, so all the standard safeguards relevant models apply.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We correctly reference the datasets, the only external assets used in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our study does not involve human subjects or crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The research does not involve any human subjects or approval from IRB or equivalent.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is about uncertainty quantification in LLMs, specifically illustrating some results based on an assumption involving independence of the distribution for a correct output for a question given some “iterative prompting”. The proposed information-theoretic measure is intended to help measure epistemic uncertainty of LLM output.\n\nSTRENGTHS:\nI think the paper has several interesting ideas, and one of the strengths is the attempt to formalize epistemic uncertainty using information theoretic ideas. There is novelty in the approach, in my view, as well as potential for these ideas to be useful in practice.\n\nWEAKNESSES:\nI found it generally hard to understand various sections of the paper, even though I am generally familiar with literature in this space. I think the writing is somewhat unclear in a few places. For instance, what is the scope of the work? Exactly what kinds of tasks are the proposed methods suitable for? Why should the key assumption hold and why should one trust it from some anecdotal examples? \n\nSeveral statements do not seem fully explained or justified; I will mention a few later in my detailed comments. I’m also not sure that the paper does enough empirically to highlight the key ideas – if I’m mistaken, the authors should correct me. So, while I find the work interesting and potentially impactful, I feel there is room for improvement, particularly around the exposition. I am willing to revise my suggestion for the paper based on the discussion.\n\nQUESTIONS:\nSome comments and questions follow:\n\nThe title of the paper is too broad and could pretty much refer to any paper about uncertainty quantification in LLMs; I strongly recommend something more descriptive. One possibility is to use a sub-title such as “Iterative Prompting for Estimating Epistemic Uncertainty”.\n\nI don’t understand why recent references have been used to cite epistemic vs. aleatoric uncertainty; these are older ideas. Either cite something classic or remove them.\n\nThe paragraph starting on line 47 makes some factual errors. The prior work does not assume there is a single correct response. For instance, in the Kuhn et al. paper, all answers that meet a threshold on the Rouge-L score (as compared to some ground truth) are deemed to be correct. I suggest rewriting this paragraph.\n\nThe “contributions” part on page 2 is quite unclear, and some things only become clearer later. The authors should write this in plain language that is understandable at a high level, without getting into details. For instance, what is the “ground truth language”?  \n\nHallucinations are mentioned in the paper but never clearly defined or cited. I believe the notion here is different from that in other work.\n\nI recommend using title case for section headers.\n\nFig. 1 is really hard to read and has inconsistent y-scales across panels. Also, I see a drop in the probability in some panels. How is this consistent with what is mentioned in the text? Is it because it does not go to 0 quicker? Please clarify.\n\nI may be mistaken but I feel there may be something wrong with how Assumption 4.1 is written. It seems like Y_t must be a ground truth answer to the original question x (without the additional information in the prompt) but Y_1, through Y_t-1 can be any text. Is this true? The \n\nHow do we know that Assumption 4.1 is true? Is this based on the discussion in the preceding section? Does this depend on how long the ground truth response Y_t is? \n\nIs F_0(x) defined? Perhaps this is the case where the prompt is: “Provide an answer to the following question: Q: x. A:”\n\nI had a hard time figuring out what to make of the experiments. What is the main takeaway? When does the proposed approach perform better than baselines? There are other potential approaches (verbal and non-verbal, such as consistency-based approaches) that could be used as baselines. The P-R curves shown should mention coverage and accuracy; these are also known as accuracy-rejection curves.\n\nLIMITATIONS:\nLimitations do not seem to be discussed very much.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper, the authors address the challenge of distinguishing between epistemic and aleatoric uncertainty in large language models (LLMs). They develop methods to decouple these uncertainties, which is crucial for handling queries with multiple valid responses. Their approach involves iterative prompting of the LLM to generate responses and measuring the sensitivity of new responses to previous ones. This helps in identifying cases where epistemic uncertainty is high, indicating potential hallucinations by the model. They propose an information-theoretic metric to quantify epistemic uncertainty and derive a computable lower bound for this metric, which is validated through experiments on datasets like TriviaQA and AmbigQA.\n\nSTRENGTHS:\n1. The topic is important and interesting. \n2. The proposed iterative prompting strategy is practical and can be implemented easily.\n\nWEAKNESSES:\n1. The writing of this paper is poor and informal. The reviewer feels hard to understand the exact insights and the principles behind the paper. Please refer to Questions for a few of these confusions. The reviewer strongly recommends polishing the presentation carefully, especially those notations and definitions. \n2. The evaluation protocol is unclear to the reviewer. In Fig. 6, what is the meaning of the entropy in the x-axis? Also, some LLM UQ baselines are missing such as [1]\n\nReference:\n\n[1] Lin, Zhen, Shubhendu Trivedi, and Jimeng Sun. \"Generating with confidence: Uncertainty quantification for black-box large language models.\" arXiv preprint arXiv:2305.19187 (2023).\n\nQUESTIONS:\nSection 2:\n1.  \"Moreover, consider a family of conditional distributions P ... \", if P is a set of distributions, why \\mu is defined as a function? what is the meaning of \\mu(x|x')?\n\n2. what is \"ground-truth conditional probability distribution\"? do you mean p(Y|X) where (X, Y) are data and label?\n\n3. where the \"possible responses Y1, . . . , Yt\" come from? Are they sampled from LLM Q with different decoding strategy?\n\n4. what is the physical meaning of Z_i and also where does the second subscript i come from in (Z_i)_i?\n\nSince the explanation of Z_i, \\mu are missing, it is hard for the reviewer to identify how the \"information-theoretical notations\" contributes to this paper.\n\nSection 3:\n\n5. \"To obtain conditional normalized probabilities, we consider the probabilities of the two responses, and normalize them so that they add to \" it is unclear to the reviewer why this normalization is applied and how this is conducted. Please describe this procedure formally and justify the reason why conduct this.\n\n6. The x-axis and y-axis labels are missing in Fig 1, Fig. 2, and Fig. 3. \n\n7. How the proposed iterative prompting strategy (and its so-called \"Conditional normalized probability\") connected to epistemic uncertainty is missing. Although pieces of descriptions are mentioned in the Introduction, they are ambiguous and informal. Conventional definition of epistemic uncertainty is usually the model approximation error, e.g., p(\\theta|D) where D is the training data. However, the new constructed input, i.e., the iterative prompt, is assumed to be significantly different from the D and how this is capable of quantify epistemic uncertainty is confusing for the reviewer.\n\nLIMITATIONS:\nNo, the conclusion and limitations (and social impacts) are missing.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper considers both epistemic and aleatoric uncertainties and proposes a novel method to decouple them. This method employs iterative prompting based on its previous responses. Experiments demonstrate that the proposed approach effectively detects cases where only epistemic uncertainty is large for multi-label questions.

STRENGTHS:
1. Important problems: Aleatoric uncertainty is crucial for handling multi-label queries in practical applications.
2. Comprehensive theoretical analysis.
3. Demonstrates good performance in effectively dealing with multi-label queries by decoupling epistemic and aleatoric uncertainty.

WEAKNESSES:
1. **Limited application scope**: While I appreciate the method and performance of decoupling epistemic and aleatoric uncertainty for multi-label queries, it falls short in estimating uncertainty for single-label queries (Fig.5ab).

2. **Overlooked over-confidence issues**: The paper overlooks the problem of over-confidence, where the model produces low entropy among multiple responses despite providing incorrect answers.

3. **Limited dataset selection**: The paper filters the WordNet dataset, retaining only queries with entropy higher than 0.7. It claims that “both the proposed method and the semantic entropy method rarely make mistakes on this dataset, and therefore we are not adding any mistakes to either method.” This indicates that the selected dataset is relatively simple. However, as discussed in weakness 2, low entropy responses can still contain errors in more challenging datasets.

4. **Limited model selection**: The paper does not validate the effectiveness of the method across different types of models.

QUESTIONS:
N/A

LIMITATIONS:
The discussion on the limitations of this paper is insufficient.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an iterative prompt-based approach to uncertainty estimation. They make a model generate multiple answers, and estimate the probability of each, estimating uncertainty for each. They argue this method easily adapts to aleatoric and epistemic uncertainty, and can be applied to multiple choice question answering. They provide a mathematical framework to support their approach, arguing that a simple metric, Mutual Information between responses, can be a lower bound for uncertainty.\n\nSTRENGTHS:\nThe epistemic/aleatoric approach to understanding uncertainty is a clear theoretic framework that is helpful in the MQA stage. This is highly appliable as threshodling methodology is provided, and complete examples are provided. Interpretability notions are used to explain this behaviour, giving an insight on why this happens, and not only how.\n\nWEAKNESSES:\n1) Assumption 4.1 is, to my understanding, that the correct answer will have a probability independently of \"any\" context. Under this assumption uncertainty is observed when the model relies more on context. Later math is based on this assumption, which seems somewhat task specific.While this works in the controlled question answering setup, context dependence holds a lot more information(ex:linguistic dependencies, uncertainty regarding user intent...). This could make the resulting uncertainty metric reduce less useful in deployment to general usage. I think discussion of limitation to assumption 4.1 should be more throughly discussed.\n(It should be noted that 4.1/theorem 4.5 very clearly discusses the limitations of applying MI to infinite language)\n\n2) Prompt based methods have been shown to be very model reliant, size reliant and tuning reliant. From a reproductibility perspective, it is unclear if results on Gemini will reproduce on different architectures, or smaller models. Is this behaviour a skill that emerges at a model size? requires specific tuning? or is it inherent to language modelling?\nIn the same line of questioning, it is unclear to me how the prompt was chosen, and if this selection procedure needs to be repeated on a new model. If this probabilistic approach is universal to Language modeling? If so this would truly be an interesting step forward from previous works. Otherwise it is a limitation.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nLimitations are adressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}