Title: Estimating the Hallucination Rate of Generative AI
Venue: NeurIPS 2024
Accepted: yes
Keywords: Uncertainty Quantification

Large Language Models

Conditional Generative Models

Hallucination Prediction
Abstract: This paper presents a method for estimating the hallucination rate for in-context learning (ICL) with generative AI. In ICL, a conditional generative model (CGM) is prompted with a dataset and a prediction question and asked to generate a response. One interpretation of ICL assumes that the CGM computes the posterior predictive of an unknown Bayesian model, which implicitly defines a joint distribution over observable datasets and latent mechanisms. This joint distribution factorizes into two components: the model prior over mechanisms and the model likelihood of datasets given a mechanism. With this perspective, we define a \textit{hallucination} as a generated response to the prediction question with low model likelihood given the mechanism. We develop a new method that takes an ICL problem and estimates the probability that a CGM will generate a hallucination. Our method only requires generating prediction questions and responses from the CGM and evaluating its response log probability. We empirically evaluate our method using large language models for synthetic regression and natural language ICL tasks.

Estimating the Hallucination Rate of Generative AI

Abstract

This paper presents a method for estimating the hallucination rate for in-context learning (ICL) with generative AI. In ICL, a conditional generative model (CGM) is prompted with a dataset and a prediction question and asked to generate a response. One interpretation of ICL assumes that the CGM computes the posterior predictive of an unknown Bayesian model, which implicitly defines a joint distribution over observable datasets and latent mechanisms. This joint distribution factorizes into two components: the model prior over mechanisms and the model likelihood of datasets given a mechanism. With this perspective, we define a hallucination as a generated response to the prediction question with low model likelihood given the mechanism. We develop a new method that takes an ICL problem and estimates the probability that a CGM will generate a hallucination. Our method only requires generating prediction questions and responses from the CGM and evaluating its response log probability. We empirically evaluate our method using large language models for synthetic regression and natural language ICL tasks.

1 Introduction

This work presents a method to estimate the hallucination rate for in-context learning (ICL). In ICL, we feed a dataset to a conditional generative model (CGM) and ask it to make a prediction based on that dataset [1, 2]. This practice is useful because it allows us to use pre-trained models to solve problems they may not have been explicitly optimized for. For example, ICL can improve the prediction accuracy of large language models (LLMs) for benchmark tasks in math, translation, and time series prediction [3, 4]. However, understanding the errors-commonly termed hallucinations [5]-a particular ICL application might make is difficult.

We develop a new method that takes an ICL problem-a CGM, a dataset, and a prediction questionand estimates the probability that a model will generate a hallucination in response to the prediction question. Figure 1a shows an ICL problem with news snippets classified as World, Sports, Business, or Science [6]. The correct answer to the last snippet is Sports. Figure 1b displays responses generated by Llama-2-7B. Six of the twenty-four generated responses are incorrect, indicating that the model hallucinates at a rate of 25%.

Our approach takes a Bayesian perspective of ICL. This perspective assumes that the CGM generates samples from the posterior predictive distribution of an (unknown) Bayesian model that defines a joint distribution over observable data and latent mechanisms [7, 8]. We then define the posterior hallucination rate (PHR) under this model, which conditions on the observed data-in our case, this is the "context" provided. Finally, we show how to estimate the posterior hallucination rate using the predictive distribution of a CGM.

Related work. Hallucination prediction and mitigation is an active area of research [9-30]. This work is most closely related to a subset of methods based on uncertainty quantification [31-38]particularly, those methods that aim to predict hallucinations based on uncertainty about the meaning of generated responses [33, 34, 36]. Unlike those methods, the PHR does not require external information from auxiliary classifiers and is applicable beyond language tasks. Our approach is enabled by sampling ICL dataset completions from the predictive distribution, which was first explored in the context of sequential models by Fong et al. [39]. Extensions to various CGMs are an active area of research [40, 41]. Notably, Falck et al. [41] explores the hypothesis that in-context learning (ICL) performs Bayesian inference. They propose a similar method to ours for testing this hypothesis and present several ICL scenarios where it does not hold. However, their tests are stringent, and while ICL does not conform to Bayesian inference under these strict conditions, we demonstrate in our work that adopting a Bayesian perspective for ICL still offers significant practical benefits.

Contributions. In Section 2, we introduce the posterior hallucination rate for Bayesian CGMs, prove it is computable by sampling from the predictive distribution, and provide a finite-sample estimator for CGMs that only requires evaluating the log probabilities of responses. In Section 3, we empirically evaluate our methods. First, we study the PHR estimator with synthetic data to demonstrate that it can accurately predict the true hallucination rate. We then study our methods on natural language ICL problems with pre-trained CGMs from the Llama-2 [42] and Gemma-2 [43] model families. We demonstrate that the PHR estimator gives accurate estimates of the empirical error rate. We provide implementations of the method proposed and experiments used in this paper in https://github.com/blei-lab/phr.

2 The posterior hallucination rate and how to estimate it

In this section, we first review the fundamentals of conditional generative models (CGMs). Next, we define in-context learning (ICL) and discuss the Bayesian perspective. We then provide definitions for hallucinations and hallucination rates. Finally, we demonstrate how to estimate the hallucination rate given an ICL problem and a CGM.

Conditional generative models and in-context learning. A conditional generative model (CGM) is a sequential model of the form p θ (t | (t i ) m 1 ) where θ represents the parameters of the model, t represents an element in the domain of the distribution, and m is the number of elements in the sequence. If the support of each conditional distribution is over language tokens and the size of θ is large, these models are known as large language models (LLMs). We focus on LLMs, but our methods apply to many conditional generative models.

Implementing CGMs over sequences is often done using neural network architectures called Transformers [44]. The parameters θ are set by performing stochastic maximization of the model likelihood over a training dataset D = { (t j i ) m 1 } n j =1 , where i is the index over elements of a sequence, and j is the index over sequences. The resulting generative model approximates the distribution of token sequences in the training dataset D .

An in-context learning (ICL) problem is a tuple ( p θ , D n , x) containing a model p θ , a dataset D n , and a query x . The dataset, or "context," is a sequence of n examples, D n = (x i , y i ) n 1 = ((x 1 , y 1 ) , . . . , (x n , y n )) . For a new query, x , the model is prompted with the query x and context (x i , y i ) n 1 to generate a response according to p θ (y | x , (x i , y i ) n 1 ) .

ICL and Bayesian inference. One way to understand ICL is through Bayesian statistics [7, 8, 45]. This perspective associates each ICL problem with a latent mechanism f ⋆ defining the data generation process. Given f ⋆ , data (x i , y i ) are drawn independently from the distribution p ICL (x i , y i | f ⋆ ) . 2

Under this assumption, when considering all ICL problems simultaneously, the joint distribution of all query-response pairs can be expressed as a mixture of latent mechanisms f :

If we further assume that a pre-trained language model p θ approximates the posterior predictive distribution under this distribution p ICL:

it follows that doing ICL using an LLM can be seen as a form of implicit Bayesian inference over the latent mechanisms f [8].

Although Equation (1) can be justified by the definition of an ICL problem-as in Footnote 2 or by de Finetti-style arguments based on exchangeability [46]-Equation (2) must be assumed.

In this paper, we show how adopting this approach allows us to construct and estimate the posterior hallucination rate : the probability that a generated response y from p θ to a query x will be in an unlikely region according to the 'true' latent mechanism f ⋆ .

Before continuing, we provide a brief comment on notation. We use F , X , Y to emphasize when we refer to random variables, and f , x , y otherwise. For clarity, we use p θ when referring to the distribution of the CGM and p without a subscript when referring to the probability of the "true" data-generating process. The latter distribution can refer to p ICL or any other probability distribution that follows the factorization of Equation (1).

2.1 Hallucinations and the posterior hallucination rate

Using these ideas, we now define hallucinations and the hallucination rate.

First, imagine a setting where we observe the mechanism f ⋆ . For a query x , what values of y would we consider hallucinations? A straightforward idea is to characterize a hallucination as a value of y that is unlikely to be generated from p (y | x , f ⋆ ) . This choice motivates the following two definitions.

Definition 1. We define a (1ϵ )-likely set of f and x as any set A such that P(Y ∈ A | f , x) ≥ 1 -ϵ . Definition 2. For fixed (1ϵ )-likely sets A (f , x) , we call a value y a hallucination with respect to x and f , if y / ∈ A (f , x) .

As an intuitive example, assume that the actual generative model of (x i , y i ) n 1 is a Bayesian linear model with a known standard deviation σ . Specifically, f ∼ N (0 , I d ) , x i ∼ N (0 , I d ) , and y i ∼ N (f ⊤ x i , σ 2 ) , where f , x i ∈ R d and y i ∈ R . If ϵ = 0 . 05 , we could choose the interval between the 2 . 5 and 97 . 5 percentiles of the distribution N (f ⋆ ⊤ x , σ 2 ) as our (1ϵ )-likely set and call everything outside of this interval a hallucination.

In practice, we do not observe f ⋆ . Rather, we make predictions with p (y | x , D n ) . We ask: At what rate are we hallucinating when we make predictions? The answer is the true hallucination rate .

Definition 3. We define the true hallucination rate (THR) as the probability of sampling a hallucination given true mechanism f ⋆ and query x :

Where A (f ⋆ , x) is an (1 -ϵ ) -likely set of f ⋆ and x .

This value is higher when the posterior predictive p (y | x , D n ) places high probability in regions unlikely under p (y | x , f ⋆ ) , and, conversely, will be lower if the posterior predictive puts a lot of mass on areas likely under p (y | x , f ⋆ ) . We expect the value to be higher when we don't have enough examples and lower as the dataset size increases.

Of course, we do not observe the true mechanism f ⋆ . But the dataset ('context') D n provides evidence for it, as summarized in the posterior p (f | D n ) . With this distribution, we define the posterior hallucination rate , which is the focal point of this work.

Definition 4. We define the posterior hallucination rate (PHR) as

In linear regression, the PHR is the probability that y will land outside of the high probability interval of f and x when sampling f ∼ p (f | D n ) , the posterior over coefficients, and y ∼ p (y | D n , x) , the posterior predictive over responses.

Here, we remind the reader that directly calculating p (f | D n ) cannot be done using the CGM. We will address this problem in the next section.

As a final detail, we discuss how to construct (1ϵ )-likely sets. Ideally, we would like to define these sets so that evaluating y ∈ A (f , x) is easy. Otherwise, it would be hard to know if a response y is a hallucination. One way to do so is to define a statistic S that maps each value in Y to a value in R . We then define Q ϵ (f , x) as the ϵ quantile of this statistic under p (y | f , x) . Because it is a quantile,

Thus, A (f , x) = { y | S (y) ≥ Q ϵ (f , x) } is an (1ϵ )-likely set.

One convenient choice of S is log p (y | x , f) . Thus, moving forward, we let

and we replace all statements ✶ { y / ∈ A (f , x) } with ✶ { log p (y | f , x) < Q ϵ (f , x) } in the definitions of a hallucination, the true hallucination rate, and the posterior hallucination rate.

2.2 Calculating the posterior hallucination rate from predictive distributions

We want to calculate the posterior hallucination rate in Definition 4 using conditional generative models like LLMs. However, such models only provide an approximation to the predictive distribution p (x , y | D n ) rather than the posterior over mechanisms p (f | D n ) or the response likelihood p (y | x , f) . To overcome this, we propose Algorithm 1, which uses a CGM to estimate the PHR.

To start, we provide an intuitive explanation of Algorithm 1. A formal justification of how it produces an estimate of Definition 4 will follow. As input, Algorithm 1 receives an ICL problem, a budget for MCsamples M and K , and a maximum context length N . First, it samples N -n new query-response pairs according to the predictive distribution p θ (x N , y N , . . . , x n +1 , y n +1 | D n ) (Alg.1, lines 2-6). Intuitively, we can understand this as prompting the model to imagine future pairs it will receive. Next, it samples a new response y new from p (y | x , D n ) (Alg.2, line 9) and asks: Is this response likely for x given the task implied by D = (x 1 , y 1 , . . . , x N , y N ) (Alg.1, line 10)? If the model is confident about the task it is performing, the pair (x , y new ) will be coherent with (x i , y i ) N n +1 . If the model is uncertain, then y new will not be coherent with (x i , y i ) N n +1 . To determine coherence, we check whether y new is in the tails of log p θ (y i | D , x) (Algorithm 2, Lines 5 and 10). Finally, we average over K samples of y new and M samples of (x i , y i ) N n +1 . The result is an estimate of the PHR.

Building on this intuition, we present the theoretical justification for how this algorithm approximates Definition 4. This justification is based on Doob's theorem [47], which states that, as n → ∞ , drawing a value F from p (f) and evaluating h (F) is equivalent to first sampling (X i , Y i ) n 1 and then evaluating E [ h (F) | (X i , Y i ) n 1 ] . We state this result below.

Theorem 1 (Doob's Informal) . For F ∈ F , (X i , Y i ) ∈ X × Y , if (F , (X i , Y i ) ∞ 1 ) is distributed such that F ∼ p (f) and X i , Y i ∼ p (x , y | f) then, under general conditions, the posterior mean of h (F) given (X i , Y i ) ∞ 1 is almost surely equal to h (F) , as the number of samples goes to infinity. That is,

Theorem 1 helps us transform statements about h (f) to statements about E [ h (F) | (x i , y i ) n 1 ] , which only depends on (x i , y i ) n 1 . Thus, we can proceed without direct access to p (f | D n ) .

We now use this result to address the problem of estimating the posterior hallucination rate. The following theorem shows how to compute this rate without direct access to p (f | D n ) . We provide its proof in Appendix C.

Theorem 2 (PHR via Posterior Predictive) . Assume that the conditions of Theorem 1 hold for F and X , Y , then,

where Q ϵ ((x i , y i ) ∞ 1 , x) is the ϵ -quantile of lim N →∞ log p (Y | x , (x i , y i ) N 1 ) under the limiting distribution lim N →∞ p (Y | x , (x i , y i ) N 1 ) .

Theorem 2 suggests a natural finite approximation to the PHR where we clip all limits in the expression to a sufficiently large N . Using this approximation, the finite version of the true hallucination rate is

where Q ϵ ((x i , y i ) N 1 , x) is analogously defined by limiting the number of samples to N . The finite version of the posterior hallucination rate is

Finally, we derive an estimator for Equation (8) by replacing p with p θ and by using Monte Carlo to estimate the integrals and quantiles. Algorithm 1 and Algorithm 2 outline this procedure, where the former estimates Equation (8) and the latter estimates Equation (7).

The empirical evaluation below will demonstrate the effectiveness of these estimators, but it is important to consider the approximations involved. First, we apply the distributional approximation

p θ (y | x , D n ) ≈ p (y | x , D n ) . Second, we use Monte Carlo methods to estimate the quantiles and integrals in Algorithms 2 and 3. Third, we employ a truncation approximation by only generating N -n examples. Each of these approximations is a source of estimation error.

Lastly, while we present this methodology in the context of estimating the PHR, it is extendable to other measures of uncertainty. In Appendix D, we adapt these results to provide estimators for the mutual information between f and y (epistemic uncertainty) and the posterior average entropy of y (aleatoric uncertainty). These extensions provide a complementary framework for understanding uncertainty in generative models.

3 Empirical evaluation

To empirically evaluate the accuracy and applicability of the Posterior Hallucination Rate (PHR) estimator, we first examine whether the PHR estimator accurately predicts the True Hallucination Rate (THR). To do so, we design a synthetic regression experiment for which we can calculate the THR. For ICL regression tasks, the PHR is a reliable predictor of the THR and robust to the choice of ϵ parameter value. Moreover, we observe that the accuracy of the PHR estimator is higher for smaller ICL dataset sizes.

We then evaluate the PHR estimator on natural language ICL tasks using pre-trained large language models (LLMs). In this setting, calculating the THR is not feasible, so we investigate two alternative questions: (1) does the PHR estimator accurately predict the model hallucination rate (defined below in Section 3.2), and (2) can the PHR accurately predict the empirical error rate? The PHR estimator reliably predicts the model hallucination rate, regardless of model performance on all ICL natural language tasks. Moreover, the estimator remains robust to different ICL dataset sizes and settings of the ϵ parameter. Additionally, the PHR estimator accurately predicts the empirical error rate of generated responses when ϵ is set to values greater than 0.5 and the LLM can perform the task better than a random classifier.

3.1 Synthetic regression tasks

We implement a CGM trained on sequences of synthetic regression problems where the THR is available and compare the PHR estimates against the THR on new regression tasks.

n=2

n=2

n=100

n=100

2.5

1.0

2.5

1.0

2.0

0.8

2.0

0.8

1.5

1.5

1.0

0.6

1.0

Rate

0.6

>

0.5

>

0.5

0.4

0.4

0.0

0.0

0.5

0.2

0.5

0.2

1.0

1.0

1.5

2.01.51.00.5

0.0

S°0

1.0

1.5

2.0

0.0

0°0s0-0T-s'T-0-

0.5

1.0

1.5

2.0

1.5

2.01.51.00.5

0.0

0.5

1.0

1.5

2.0

0.0

2.01.51.00.5

0.0

0.5

1.0

1.5

2.0

X

X

Cl of pe(y |x,(x,y))

Cl of p(y | x,f*)

X

Hallucination

Correctresponse

ICL example

THR

PHR

E

Setup. We assume the true distribution of query-response pairs p ( D n ) is generated by the following factorization: ∫ ∏ n i =1 (∫ p (y i | x i , f) d P(x i ) ) d P(f) , where p (f) is the distribution of randomly drawn MLPs with He initialization, p (x i ) is the uniform distribution over [ -2 , 2] , and p (y | x , f) follows a normal distribution N (f(x) , 0 . 1) .

We implement the model p θ (y | x , D n ) with a setup similar to a Conditional Neural Process [48] and with an architecture close to Llama 2 [42] modified to model sequences of continuous variables. We train it using auto-regressive prediction, the standard for CGMs and LLMs. Training and test data are generated over non-overlapping sets of generated sequences. Example training datasets are shown in Figure 9a in Appendix F.1. Example datasets generated by the fit model when initialized with a single random query x are shown in Figure 9b in Appendix F.1. The full model and dataset details are given in Appendices E.1 and F.1, respectively.

In our experiments, we set ϵ = 0 . 05 so that a response y is considered a hallucination if it falls outside the 95% confidence interval of a given sampled distribution conditioned on x . To compute the THR, we use the ground truth function f that generated the dataset, replace p (y | x , D n ) with p θ (y | x , D n ) , and estimate the integral by sampling 2000 values from this latter distribution. This approach is reasonable, as we ultimately care about the true probability of hallucinations when using the model p θ to make predictions.

1.0

Context Length:2

ContextLength:10

ContextLength:50

TrueHallucinationRate

0.8

PosteriorHallucinationRate

Rate

1.0

Hallucination

0.8

0.6

β:0.990

0.6

p-val:0.000

0.4

MSE:0.006

0.4

R²:0.904

β: 0.942

β:0.986

β: 1.220

0.2

0.2

p-val:0.000

MSE: 0.011

p-val:0.000

MSE: 0.011

p-val:0.000

MSE:0.002

0.0

R2:0.865

R2:0.714

R2:0.305

0.0

0

25

50

75

100

125

150

175

200

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Num. Examples: n

PosteriorHallucinationRate

PosteriorHallucinationRate

PosteriorHallucinationRate

(a) PHR and THR vs.

n

Results. The first and third panels of Figure 2 show the model's generated outcomes for n = 2 and n = 100 , respectively. In these plots, the blue region represents the true (1ϵ )-likely set for the response distribution of a specific random ReLU neural network. The purple region depicts the model's (1ϵ )-likely set when conditioned on the blue data points and a query value x in the domain [-2, 2]. As more context examples are provided, the confidence intervals narrow, and responses y are increasingly likely to fall within the blue region.

The second and fourth panels of Figure 2 illustrate the THR and the PHR for x in the domain [-2, 2] for two settings of n . We set N -n to 100, M to 40, and K to 2000. On the left, dips in PHR and hallucination probability at x = 0 and x = 1 correspond with the ground truth in-context examples. In the right panel, with a larger n , both the PHR and hallucination probability remain low across all x values. Notably, the PHR and hallucination probability closely align throughout the domain. In Appendix G.1, we demonstrate that these findings hold across various values of the ϵ parameter.

In Figure 3a, we present the PHR and THR plotted against various context lengths, where each is averaged over 200 random test functions. For added interpretability, we report several metrics for evaluating the PHR as a linear predictor of the THR, including the mean squared error (MSE), the regression coefficient β , the p -value testing the null hypothesis that β = 0 , and the coefficient of determination ( R 2 ).

Although the PHR aligns well with the THR, it tends to underestimate it, particularly as the number of examples increases. This is also clear given the lower R 2 values as the context length increases. To investigate this further, Figure 3b visualizes individual PHR and THR predictions at different context lengths by plotting PHR values on the x-axis and the corresponding THR values on the y-axis. For small numbers of contextual examples, the PHR closely matches the THR, which is encouraging since accurately capturing the true probability of hallucination is crucial when few examples are available and errors are more likely.

Discussion. The results confirm that our method closely recovers the target estimand. However, the PHR estimator underestimates the THR. We do not believe this is due to the Monte Carlo or truncation approximations, as initial experiments with varying M , K , and N values exhibited the same pattern. Rather, we consider two main factors that likely contribute to this underestimation.

First, differences between the learned distribution p θ (y | x , D n ) and the true distribution p (y | x , D n ) likely contribute to the bias. This is seen in the slight discrepancies in the confidence intervals in the third panel of Figure 2. It may also be due to other mismatches in our assumptions, such as the lack of perfect exchangeability in the transformer architecture; even after training on permuted sequences, the architecture may not fully achieve exchangeability.

Second, even if the PHR estimate were entirely accurate, it would still differ from THR, as they represent distinct quantities. Therefore, some degree of bias is to be expected. We investigate this question further in Appendix G.1.2 where we compare our estimate with an analytic version of the PHR and find this problem is greatly reduced.

We leave it to future work to rigorously quantify these differences and develop methods that can more accurately align the learned and true distributions, particularly in scenarios with large context sizes.

3.2 Natural language tasks

Here we evaluate the posterior hallucination rate estimator on common natural language in-context learning tasks using the Llama-2 family of LLMs [42]. We also provide results for Gemma-2 9B in Appendix G.2.1. We evaluate the PHR by comparing it against the empirical error rate and the model hallucination rate (MHR), which we define below. These serve as proxies for the THR, which is no longer computable as we can't access f ⋆ for these examples.

SST2

Subjectivity

AGNews

MedicalQP

RTE

WNLI

0.5

β:1.327

β: 1.964

β:1.500

β:1.228

β: 1.735

β:1.898

p-val:0.000

p-val:0.000

p-val:0.000

p-val:0.000

p-val:0.000

p-val:0.000

0.4

MSE:0.005

MSE:0.027

MSE:0.017

MSE:0.001

MSE:0.018

MSE:0.017

R²:0.618

R²:0.621

R²:0.435

R²:0.914

R²:0.626

R²:0.657

Rate

0.3

R

0.2

0.1

0.0

5

10

n examples

15

20

25

30

10

n examples

15

20

25

30

5

10

nexamples

15

20

25

30

5

10

nexamples

15

20

25

30

90

n examples

15

20

25

30

10

n examples

15

20

25

30

Posterior Hallucination Rate

(a) Model Hallucination Rate (MHR) and Posterior Hallucination Rate (PHR) with ϵ = 0 . 05 as a function of the number of in-context examples. PHR closely follows MHR for all tasks.

SST2

Subjectivity

AGNews

Medical QP

RTE

WNLI

1.0

β:0.952

β:0.891

β:0.963

β:0.551

β:0.808

β:0.570

9'0

p-val:0.000

MSE:0.016

p-val:0.000

MSE:0.027

MSE:0.048

p-val:0.000

p-val:0.000

MSE: 0.020

p-val:0.000

MSE:0.025

p-val:0.000

MSE:0.029

R²:0.702

R²:0.616

R²:0.533

R²:0.222

R²:0.489

R²:0.281

Rate

0.6

0.4

0.2

0:0

5

90

nexamples

15

20

25

30

10

nexamples

t5

20

25

30

5

10

nexamples

15

20

25

30

5

10

n examples

15

20

25

30

5

90

nexamples

15

20

25

30

5

10

n examples

15

20

25

30

Empirical ErrorRate

PosteriorHallucination Rate

=

ErrorRandomGuessing

(b) Error Rate and PHR with ϵ = 0 . 75 . PHR matches the error rate for SST2, Subjective, and AG News. However, for more difficult tasks like Medical QP and entailment tasks (RTE and WNLI), PHR is less accurate, highlighting the limitations of Llama-2-7b.

Setup. We consider tasks defined by six datasets: Stanford Sentiment Treebank (SST2) [49] , Subjectivity [50] , AG News [6] , Medical QP [51] , RTE [52] , and WNLI [53] . We filter out queries of length longer than 116 tokens. Descriptions of all data sets and pre-processing are given in Appendix F.2.

To implement ICL for a given dataset, we sample a response-balanced training set of query/response pairs D n = (x i , y i ) n 1 . We generate a response y from the predictive distribution given by an LLM p θ (y | x , D n ) . We structure the prompt by adding strings to distinguish between inputs and labels. An example prompt from the Subjectivity dataset is shown in Appendix E.2.

Evaluation metrics. It is a challenge to assess the accuracy of the the posterior hallucination rate on LLM tasks as we only have a one ground truth response for each query instead of the ground truth f ⋆ .

One evaluation metric to consider is the empirical error rate,

̸

However, this metric only provides a partial picture as it does not account for the inherent variability over responses given a mechanism, which would not constitute a hallucination.

To get a more complete picture, we invoke Doob's theorem and note that, when (x i , y i ) ∼ p (x , y | f ⋆ ) and for sufficiently large N , we have

This motivates us to estimate the reference THR by approximating the distribution p (y | x , f ⋆ ) with p θ (y | x , D n ∪ D eval ) , where D eval is a set of additional (x i , y i ) examples from the same task that are not in D n . We call this estimate the model hallucination rate (MHR) as it is derived under the model p θ . Formally,

where y i ∼ p θ (y | x , D n ) , K is the number of response samples, and ̂ Q eval ϵ is the ϵ quantile of the samples { log p θ ( y j | x , D n ∪ D eval )} with y j ∼ p θ (y | x , D n ∪D eval ) . This value is only proposed for model evaluation. Importantly, it cannot be used to replace the PHR, as it depends on D eval , which is not available at test time. Intuitively, the MHR computes the probability that an answer y , generated by a model conditioned on D n , is considered a hallucination by a model conditioned on both D n and D eval .

For each task and context length n ∈ [2 , 4 , 8 , 16 , 32] , we sample 50 random training datasets D n , 50 evaluation datasets D eval , and 10 random test samples. We report the same metrics used in the synthetic experiments, but here we evaluate the PHR as a linear predictor of both the error rate and MHR. These metrics are evaluated over all 50 × 10 test samples for each context length.

Results. We report results for Llama-2-7b. We set N -n = 5 , M = 10 , and K = 50 . The top plots in Figure 4 show the MHR and estimated posterior hallucination rate against the number of in-context examples with ϵ = 0 . 05 . It shows that the posterior hallucination rate is a good estimator of the MHR. We show that this trend holds for alternative settings of ϵ in Figure 23 of Appendix G.2.2.

The bottom plots in Figure 4 show the empirical error rate and estimated posterior hallucination rate against the number of in-context examples with ϵ = 0 . 75 . The plots show that for the SST, Subjective and AG News tasks, the PHR follows the error rate closely, while it diverges for the Medical QP, RTE and WNLI tasks. We use a high value of epsilon so that only responses with very high probability are not taken to be hallucinations, aligning with our strict focus on correctness rather than modeling the full distribution. The impact of varying ϵ is explored in Figure 24 of Appendix G.

Discussion. The results highlight the usefulness of the PHR by demonstrating its ability to assess the model's uncertainty in language tasks and predict its error rate without requiring an evaluation dataset. Additionally, the discrepancy between the PHR and the error rate in Medical QP, RTE, and WNLI tasks can be attributed to the model's limited generalization ability in these tasks. This leads to an inadequate approximation of the posterior predictive; in other words, p θ (y | x , D n ) ̸≈ p (y | x , D n ) . This inadequacy is supported by Figure 4, which shows that in tasks where the PHR does not align with the error rate (green line), the model's performance is close to random guessing (dashed gray line). Hence the poor performance of the PHR in this scenarios is expected because our method relies on accurately approximating the posterior predictive and only accounts for hallucinations when the data distribution is properly modeled.

4 Conclusion

In this work, we have presented a new method for predicting the hallucination rate of in-context learning with conditional generative models. We provide a theoretical justification for our method. In synthetic experiments, we demonstrate that the PHR estimator yields accurate estimates of the actual probability of hallucination. With pre-trained LLMs that achieve non-trivial performance, we show that our method is valuable for predicting the error rate of natural language ICL tasks.

High-fidelity estimation of the PHR relies on two strong assumptions. The first is that the data of the in-context learning problem admits a de Finetti representation; the second is that the CGM p θ is a faithful estimate of the in-context learning distribution p ICL. While our results support the adoption of our method and these assumptions, divergences between p θ and p ICL may introduce inaccuracies, as we saw for natural language tasks where Llama-2-7B only performs as well as a random classifier. Falck et al. [41] also report instances where properties of the predictive distribution of a pre-trained LLM differ from those of the reference Bayesian posterior predictive for synthetic ICL tasks. Two directions for further research are to understand how these divergences affect the accuracy of the PHR and to identify the settings under which our assumptions break down.

Nevertheless, we are optimistic about future work that adopts Bayesian perspectives on LLMs and other CGMs, as we believe these approaches hold significant practical utility for understanding uncertainty and hallucinations in these models.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS , 33:1877-1901, 2020.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv:2301.00234 , 2022.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jeanbaptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530 , 2024.

Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv:2403.07815 , 2024.

Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In ACL , pages 1906-1919, Online, July 2020.

Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. NeurIPS , 2015.

Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do Bayesian inference. In ICLR , 2021.

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit Bayesian inference. In ICLR , 2021.

Philip Feldman, James R Foulds, and Shimei Pan. Trapping LLM hallucinations using tagged context prompts. arXiv:2306.06085 , 2023.

Shuo Zhang, Liangming Pan, Junzhou Zhao, and William Yang Wang. Mitigating language model hallucination with interactive question-knowledge alignment. arXiv:2305.13669 , 2023.

Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv:2302.12813 , 2023.

Nouha Dziri, Andrea Madotto, Osmar R Zaiane, and Avishek Joey Bose. Neural path hunter: Reducing hallucination in dialogue systems via path grounding. In EMNLP , pages 2197-2214, 2021.

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. RARR: Researching and revising what language models say, using language models. In ACL , pages 16477-16508, 2023.

Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. In ICLR , 2023.

Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. A stitch in time saves nine: Detecting and mitigating hallucinations of LLMs by validating low-confidence generation. arXiv:2307.03987 , 2023.

Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Read before generate! Faithful long form question answering with machine reading. In ACL , pages 744-756, 2022.

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In ICLR , 2024.

Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N Fung, Mohammad Shoeybi, and Bryan Catanzaro. Factuality enhanced language models for open-ended text generation. NeurIPS , 35:34586-34599, 2022.

Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) , pages 783-791, 2024.

Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. TACL , 10:857-872, 2022.

Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. TMLR , 2023.

Neil Band, Xuechen Li, Tengyu Ma, and Tatsunori Hashimoto. Linguistic calibration of language models. arXiv:2404.00474 , 2024.

Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2023.

Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it's lying. In EMNLP , pages 967-976, 2023.

Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In ICLR , 2022.

Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inferencetime intervention: Eliciting truthful answers from a language model. NeurIPS , 36, 2024.

Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering Llama 2 via contrastive activation addition. arXiv:2312.06681 , 2023.

Junyu Luo, Cao Xiao, and Fenglong Ma. Zero-resource hallucination prevention for large language models. CoRR , 2023.

Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. In ICLR , 2024.

Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason E Weston. Chain-of-verification reduces hallucination in large language models. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models , 2024.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with BERT. In ICLR , 2019.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv:2207.05221 , 2022.

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In ICLR , 2023.

Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. TMLR , 2024.

Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In ACL , pages 5186-5200, 2024.

Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xueying Zhang, Yu Wang, and Shizhu Liu. Halo: Estimation and reduction of hallucinations in open-source weak large language models. arXiv:2308.11764 , 2023.

Potsawee Manakul, Adian Liusie, and Mark JF Gales. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In EMNLP , 2023.

Jeremy R Cole, Michael JQ Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. Selectively answering ambiguous questions. EMNLP , 2023.

Edwin Fong, Chris Holmes, and Stephen G Walker. Martingale posterior distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology , 85(5):1357-1391, 2023.

Hyungi Lee, Eunggu Yun, Giung Nam, E Fong, and Juho Lee. Martingale posterior neural processes. In ICLR . ICLR, 2023.

Fabian Falck, Ziyu Wang, and Christopher C Holmes. Is in-context learning in large language models Bayesian? A martingale perspective. In ICML , 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 , 2023.

Gemma Team. Gemma, 2024.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 2017.

Naimeng Ye, Hanming Yang, Andrew Siah, and Hongseok Namkoong. Pre-training and in-context learning is Bayesian inference a la de Finetti. arXiv:2408.03307 , 2024.

Edwin Shields Hewitt and Leonard J. Savage. Symmetric measures on cartesian products. Transactions of the American Mathematical Society , 80:470-501, 1955.

Joseph L Doob. Application of the theory of martingales. Le calcul des probabilites et ses applications , pages 23-27, 1949.

Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In ICML , pages 1704-1713, 2018.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP , pages 1631-1642, 2013.

Sida Wang and Christopher D. Manning. Baselines and bigrams: simple, good sentiment and topic classification. In ACL , page 90-94, 2012.

Clara H McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain. Effective transfer learning for identifying similar questions: Matching user questions to covid19 faqs. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 3458-3465, 2020.

Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Joaquin Quiñonero-Candela, Ido Dagan, Bernardo Magnini, and Florence d'Alché Buc, editors, Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment , pages 177-190, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-33428-6.

Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Gerhard Brewka, Thomas Eiter, and Sheila A. McIlraith, editors, KR . AAAI Press, 2012. ISBN 978-1-57735-560-1.

Ferenc Huszár. Implicit Bayesian inference in large language models, 2023. Online; accessed 10-July-2023.

Michael Hahn and Navin Goyal. A theory of emergent in-context learning as implicit structure induction. arXiv:2303.07971 , 2023.

Hui Jiang. A latent space theory for emergent abilities in large language models. arXiv:2304.09960 , 2023.

Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? Bayesian model averaging, parameterization, and generalization. arXiv:2305.19420 , 2023.

Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In ICML , 2023.

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? Investigations with linear models. In ICLR , 2023.

Chi Han, Ziqi Wang, Han Zhao, and Heng Ji. In-context learning of large language models explained as kernel regression. arXiv:2305.12766 , 2023.

Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In EMNLP , 2023.

Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In ICLR , 2024.

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT3 ? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures , 2022.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL , 2022.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML , 2021.

Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relationships but is not conventional learning. In ICLR , 2024.

Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv:2303.03846 , 2023.

Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. In EMNLP , 2022.

Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning "learns" in-context: Disentangling task recognition and task learning. In ACL , 2023.

Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie CY Chan, Biao Zhang, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. In ICML 2024 Workshop on In-Context Learning , 2024.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv:2303.08774 , 2023.

Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the uncertainty estimation of large language models. arXiv:2307.01379 , 2023.

Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L Edelman. Distinguishing the knowable from the unknowable with language models. In ICML , 2024.

Daniel D Johnson, Daniel Tarlow, David Duvenaud, and Chris J Maddison. Experts don't cheat: Learning what you don't know by predicting pairs. arXiv:2402.08733 , 2024.

Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty-aware planning enhances information seeking in large language models. In ICLR 2024 Workshop on Large Language Model (LLM) Agents , 2024.

Hong Jun Jeon, Jason D Lee, Qi Lei, and Benjamin Van Roy. An information-theoretic analysis of in-context learning. In ICML , 2024.

Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Finetuning language models for factuality. In ICLR , 2024.

Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models , 2018.

Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In ICLR , 2019.

Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In ICML , 2022.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS , pages 24824-24837, 2022.

Lilian Weng. Extrinsic hallucinations in LLMs. lilianweng.github.io , Jul 2024. URL https: //lilianweng.github.io/posts/2024-07-07-hallucination/ .

Andrew Jesson, Nicolas Beltran-Velez, and David Blei. Can generative AI solve your in-context learning problem? A martingale perspective. Neurips Safe Generative AI Workshop , 2024.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.

Appendix

Contents

A Related works

Mechanisms and Capabilities of ICL. Several papers argue, both theoretically and through synthetic scenarios, that ICL can implement learning principles such as Bayesian inference and gradient descent [8, 54-59]. Evidence in actual pre-trained LLMs shows that ICL can be approximated as a kernel regression [60], and parametric approximations to ICL can be derived from the hidden state of the last input demonstration [61, 62]. Practical shortcomings of ICL include dependence on example order [63-66] and the impact of prediction preferences acquired during pre-training [65-69]. While future models might improve ICL performance [70], current limitations are clear and are thoroughly investigated in recent work by Falck et al. [41]. These results imply that ICL in real LLMs does not implement perfect Bayesian inference but suggest that LLM predictive uncertainty includes both epistemic and aleatoric components, and that LLMs can update uncertainties with new observations.

Uncertainties in LLMs. Our results are supported by evidence that predictive uncertainties of large language models are well-calibrated, even in scenarios requiring epistemic uncertainty [32, 71]. Relatedly, LLM uncertainties have been used to detect hallucinations in free-form generation settings such as question-answering [28-30, 32-38, 72]. Although not all these papers explicitly focus on LLM uncertainties, they rely on it implicitly by sampling multiple model completions for a query and quantifying the differences in meaning. Specifically, Kuhn et al. [33] highlight the challenge of isolating uncertainty over semantic meaning from uncertainty over syntax or lexis in free-form generation tasks. While these approaches do not disambiguate aleatoric and epistemic uncertainty, Ahdritz et al. [73], Johnson et al. [74] recently proposed methods to do so in CGMs. However, Ahdritz et al. [73] require access to two LLMs of different parameter counts, and Johnson et al. [74] do not apply their method to LLMs. Additionally, Hu et al. [75] show that Bayesian experimental design can turn LLMs into strategic question askers, and Jeon et al. [76] present a theoretic study on sources of errors in ICL.

Hallucinations in LLMs. In addition to approaches based on uncertainty, a variety of other strategies have been explored to detect or mitigate hallucinations in LLMs: retrieval-augmented generation [9-16], custom token sampling procedures [17-19], model fine-tuning to improve uncertainties [20-22] or reduce hallucinations outright [77], as well as learning to extract or steer truthfulness from hidden states [23-27].

Neural processes. Neural processes (NPs) [48, 78-80] are neural network-based non-parametric models trained over a collection of datasets. Similar to ICL, NPs take a collection of datapoints as input and amortize task learning in a single forward pass through the model. For instance, when datasets are drawn from a Gaussian process prior, NPs' predictive distributions closely approximate the true Bayesian posterior predictive for a given input dataset [7]. NPs have been used successfully for tasks requiring reliable uncertainty estimation, such as Bayesian optimization [40, 78, 80] or active feature acquisition [48]. Recently, Lee et al. [40] applied Doob's theorem to quantify uncertainties in neural processes.

Martingale Posterior The work most aligned with ours is [39], which introduces Martingale Posterior distributions. Their central idea is to focus on the posterior predictive, rather than the posterior itself, as the primary tool for expressing uncertainty. Similar to our approach, samples from the posterior predictive are used to estimate quantities of interest. Through repeated resampling and estimation of the predictive, a 'posterior' over the estimate is obtained. In particular, their predictive resampling algorithm can be seen as a generalization of Algorithm 1, although the motivation and use is different. Falck et al. [41] work towards formalizing this methodology for LLMs. They propose a set of statistical tests to estimate whether an LLM satisfies the Martingale property; these tests depend on being able to sample from the true Bayesian model that defines the posterior predictive distribution estimated by the LLM predictive distribution. In their evaluations using synthetic data and pre-trained LLMs, they find that violations of the Martingale property can occur. Moreover, they find that the fidelity of the LLM predictive distribution to the true Bayesian posterior predictive decreases as the length of dataset completions ( N -n ) increases. They also derive an epistemic uncertainty estimator based on the posterior covariance over mechanisms, which has connections to the posterior hallucination rate and the mutual information estimand we propose in Appendix D.

B Further discussion

B.1 Broader social impact

Positive social impact. In terms of misinformation, if users cannot distinguish between accurate information and hallucinations, they may spread misinformation unknowingly. This can damage the credibility of platforms that use LLMs and erode the trust users place in AI systems. Furthermore, as LLMs are increasingly adopted in high-risk sectors such as medicine and finance, hallucinated medical advice poses serious risks, potentially resulting in harmful health practices or delayed treatment. Similarly, inaccurate financial information could lead to poor investment decisions or significant financial loss.

Ethically, hallucinations can reinforce or propagate biases and stereotypes if the generated content reflects societal prejudices. This can perpetuate discrimination and inequality. Understanding when hallucinations are likely to occur is vital for holding developers and companies accountable for the content their models produce. Finally, many researchers use LLMs and other CGMs to generate or label data. Hallucinations in this setting can lead to false discoveries and wasted resources.

Being able to accurately predict hallucination rates for given tasks is essential for ensuring that AI systems contribute positively to society. It allows for maintaining trust, safety, ethical standards, and the overall integrity of information dissemination.

Negative social impact. When our model is being used as intended but gives incorrect results (i.e. produces a low estimate of probability of hallucination when the true probability is high), it could inadvertently be reinforcing biases present in hallucinations while increasing user trust in the outputs.

B.2 What kind of uncertainty does the posterior hallucination rate quantify?

Building trustworthy and effective ICL solutions requires understanding why and when incorrect or unexpected responses are generated by a CGM. The ICL literature provides two findings that suggest distinct sources of hallucination.

SST2

Subjectivity

AGNews

Medical QP

RTE

WNLI

1.0

0.8

Rate

0.6

Error

0.4

0.2

0.0

10

20

30

10

20

30

10

20

30

10

20

30

10

20

30

10

20

30

n examples

n examples

n examples

n examples

n examples

n examples

SST2

Subjectivity

AGNews

Medical QP

RTE

WNLI

3.0

Entropy

2.5

2.0

ResponseE

1.5

1.0

0.5

0.0

10

20

30

10

20

30

10

20

30

10

20

30

10

20

30

10

20

30

n examples

n examples

n examples

n examples

n examples

n examples

The first finding is that both the error rate and the prediction entropy decrease and saturate with an increasing number of examples [66]. This trend is illustrated in Figure 5 using the Llama-2-7b model [42] on a set of natural language ICL tasks. This finding indicates that one source of hallucination is an insufficient number of relevant in-context examples.

The second finding from the literature is that there are still many tasks for which ICL performs poorly. For example, the response accuracy of Gemini Pro 1.5 given several in-context examples from the American Mathematics Competition is only 37.2%, implying that the model would hallucinate an incorrect response at a rate of 62.8% [3]. We hypothesize that regardless of the number of relevant in-context examples, the model may lack the capacity to answer a specific user query from a complex or new domain accurately. This hypothesis is illustrated using Llama-2-7B by comparing the graphs of the first three tasks (SST2 [49], Subjective [50], and AG News [6]) to the graph of the WNLI task [53]) in Figure 5. While the error rate and response entropy for each of the first three tasks improve

significantly over random guessing with more examples, both measures appear to saturate near the random baseline for the WNLI task. The second source of hallucinations is then associated with whether a model has the capacity to factually answer queries for an ICL task.

This work focuses on the first source of hallucinations. That is, the posterior hallucination rate is concerned with estimating the rate of hallucinations that stem from a lack of relevant context, and not those that stem from the model not having the capability to solve the task. The predictive distribution encodes response variability coming from several sources, and this variability is closely tied to the ways in which a model can generate a hallucination. Below, we discuss these sources of response variability (or uncertainty) and their relations to hallucinations and the posterior hallucination rate.

Aleatoric Uncertainty. The (1 -ϵ ) -likely set defined by a given mechanism f ∗ reflects an irreducible component of response uncertainty. To illustrate the concept of irreducible (sometimes called 'aleatoric') response uncertainty, imagine a hypothetical and idealized LLM fit to a vast corpora containing many calculus examples so that it is capabable of integration. Consider,

If an LLM effectively models the mechanism associated with integration, then response variability will only be determined by the different ways the model can generate the correct response: perhaps, 18, eighteen, 3 3 3 --3 3 3 , 9+9 , 2 ∗ 9 , or XVIII, depending on the training data. Uncertainty reflecting the plurality of ways to communicate the same meaning is commonly referred to as syntactic uncertainty [33], which is considered irreducible in this particular example.

In language tasks, irreducible uncertainty does not need to be syntactic. Consider the same LLM and

In addition to the syntactically different ways to specify a particular function ( x 2 , x squared, x ∗ x , . . . ) and the corresponding result (18, eighteen, XVIII, . . . ), response variety also depends on the semantically different integrands that could fill the first blank ( x 3 , cos ( x ) , exp( x ) , 2 xy , . . . ) and the semantically different possible results. This additional variety is an instance of semantic uncertainty [33]. While high semantic uncertainty can be indicative of hallucinations, it is not a problem in this example because the imputation of any sensible function and answer could still be valid under the mechanism associated with integration datasets. That is, given Prompt 2 , uncertainty over semantically different functions is expected and even desirable.

This pair of examples illustrate an important insight; irreducible uncertainty is mechanism-relative . For example, it may be appropriate to equivocate irreducible and syntactic uncertainty in a simple question answering setting where the mechanism defines a (1 -ϵ ) -likely set over correct responses. However, in the second example we show that this decomposition is not always appropriate. In general, we consider aleatoric uncertainty to be associated with response variability induced under a specific mechanism f . That is, the variability of responses under the likelihood p (y | x , f) .

Special epistemic uncertainty. Now we provide examples to illustrate reducible uncertainty about the mechanism f . Returning to Prompt 1 , imagine that the user desires a response in terms of a reduced fraction. There are two obvious ways that the user could augment Prompt 1 to reduce the uncertainty over mechanisms yielding integer, word, fraction, etc. responses. (1) the user could simply replace 'fill in the blanks' with 'fill in the blank with a reduced fraction.' (2) the user could take an ICL approach and augment the prompt with a number of examples:

Prompt 3:

The first choice may result in reducing all uncertainty about which mechanism to sample responses according to. For the second choice, we can imagine a progressive reduction in uncertainty about the mechanism as more examples are added in-context. For example, if we were to see the two provided examples, we may still be uncertain about whether to respond with a number or a fraction, or whether to respond with any correct fraction or the reduced fraction. For example, given the context, a response of 54 3 would be as plausible as the desired 18 1 . It may not be until the prompt included an example like, 'the integral of x 3 with respect to x on the interval [2 , 4] is 60 / 1 ,' until all uncertainty about the mechanism is resolved. We suggest that this may explain the observed reduction in error rate and response entropy in a task like WNLI, where Llama-2-7B does not generalize effectively. That is, as we provide more in-context examples, the predictive distribution becomes more aligned with the set of acceptable responses, even if those responses may not be correct.

The preceding example illustrated a hallucination as a misaligned response; now, let us turn to an example of a non-factual response. Returning to Prompt 1 , imagine that the model generates the response 42. Why did it do this? A plausible answer could be that the model cannot do integration. We will touch on this possibility next, but first let's consider an equally interesting case. We know from few-shot and chain-of-thought prompting literature [14, 30, 81] that augmenting the context can have significant effects on ICL accuracy. For example, consider the hypothetical setting where the LLM has 'grokked' algebra, but only has the superficial capacity to output a number when completing definite integrals. Or perhaps the model has the capacity for integration, but the format of the examples and query is uncommon in the training corpora. In this case, the LLM may actually be capable of generating correct responses given some clever prompting. For example,

Prompt 4:

Label:

Again, as the prompt contains more examples that translate the form of the query into a suitable format, we can expect that the uncertainty about the answer will reduce. For conditional models in general, we will call this Special epistemic uncertainty , which could also be understood as in-context epistemic uncertainty. Both of these examples illustrate hallucinations that are due to insufficient context, which have been called in-context hallucinations by Weng [82]. In this paper we have focused on the case where increasing the number of in-context examples can resolve this uncertainty. We leave it to future work to understand more sophisticated prompt augmentation.

General epistemic uncertainty. Let's return to Prompt 1 , but this time imagine an LLM fit to a corpus not containing any examples from calculus or related mathematical fields. Or perhaps the LLM had finite capacity and is not capable of generating accurate answers to integrals. Then, what do we do when the model generates 'Dua Lipa' to an integration question? What do we do when the model outputs members of the set { 17 , 137 , Dua Lipa , Wednesday , . . . } ? We say that the response should have high General epistemic uncertainty because the LLM has not acquired the capacity to model the mechanism class, F , corresponding to integrals. In the example of Prompt 4 , imagine if there were no number of exemplars or no prompt augmentations that could induce a correct response.

(a) Special

(b) General

ContextLengthn=2

ContextLengthn=100

ContextLengthn=2

ContextLengthn=100

1.0

1.0

0.5

0.5

0.0

0.0

0.5

0.5

0

> -1.0

> -1.0

1.5

1.5

-1

2.0

2.0

Z-

2.5

2.5

3.0

3.0

2.01.51.00.5

0.0

0.5

1.0

1.5

2.0

2.01.51.00.5

0.0

0.5

1.0

1.5

2.0

2.01.51.00.5

0°0

0.5

1.0

1.5

2.0

2.01.51.00.5

0.0

0.5

1.0

1.5

2.0

Cl of pe(y | x, (xi, yi))

Cl of p(y |x, f*)

Correct response

Cl of p(y | x, (x, y))

Cl of p(y | x,f)

Correctresponse

Hallucination

In-context example

Hallucination

In-context example

When a condition generative model p θ is a good estimator of the true distribution p , the posterior hallucination rate-and mutual information quantity we propose in Appendix D-are designed to estimate special epistemic uncertainty. We leave the important work of estimating the posterior hallucination rate when the CGM is not a good estimator (under general epistemic uncertainty) and accounting for extrinsic hallucinations [82] to future work. Initial work toward this is presented by Jesson et al. [83].

C Proof of theorems in the main text

We begin by stating a lemma that will be useful throughout.

Lemma C.1. Assume that the conditions of Theorem 1 hold for F and (X , Y) , then for a fixed dataset and query D n , x and under a probability model where F ∼ p (f | D n ) and (X i , Y i ) ∼ p (x , y | F) , then almost surely

Proof. For a fixed y and x , let g (f) = p (y | x , F) and apply Doob's theorem on the probability model. Then it is the case that

where Equation (13) holds because x is independent of F . Taking logs at both sides and using the continuity of the logarithm, we obtain

and because this holds for all y , it must hold for the random variable Y .

Now we restate the main theorem with proof:

Theorem 3 (PHR via Posterior Predictive) . Assume that the conditions of Theorem 1 hold for F and X , Y , then,

✶ where Q ϵ ((x i , y i ) ∞ 1 , x) is the ϵ -quantile of lim N →∞ log p (Y | x , (x i , y i ) N 1 ) under the limiting distribution lim N →∞ p (Y | x , (x i , y i ) N 1 ) .

Proof. Define an alternative probability model such that F ∼ p (f | D n ) and (X i , Y i ) ∼ p (x , y | F) . Let p a , E a , Q a,ϵ and P a denote the relevant quantities computed with respect to this alternative probability model.

First, note that by expanding the definition of Q a,ϵ (f , x) under this new probability model

where we used the fact that Y ⊥ (X , Y) ∞ 1 | F , x in Equation (16). For simplicity, we will use p ( ·| (X i , Y i ) ∞ 1 ) to denote lim N →∞ p ( · | ( X i , Y i ) N 1 ) with similar conventions from other quantities. Now, applying Doob's to g (f) = P a (log p a (Y | x , (X i , Y i ) ∞ 1 ) ≤ q | f , x , (X i , Y i ) ∞ 1 ) we get that almost surely

where we used Doob's on Equation (19) and the tower property in Equation (20). Plugging this back in Equation (17) we obtain

To complete the proof, note that

where we changed the probability spaces from p to p a . This is justified because p a (y | x , f) = p (y | x , f , D n ) = p (y | x , f) where we used the independence of Y on D n once f is known, the fact that p a (f) = p (f | D n ) by definition, and the fact that for the quantile function Q a,ϵ (f , x) = Q ϵ (f , x) because

due again to the independence of Y on the dataset D n once f is known. Finally we have (abusing notation using D ∞ n +1 = (x , y) ∞ n +1 to refer to (x , y) ∞ n +1 in the original probability space but also (x i , y i ) ∞ 1 in the alternative probability space as they have the same distribution):

Where Equation (28) is justified by because (x , y) ∞ n +1 doesn't appear in the term inside, Equation (29) is justified by the arguments above and Lemma C.1, and Equation (31) is a result of marginalizing f out. The last thing to point out is that p a (y | x , (x , y) ∞ n +1 ) = p (y | x , (x i , y i ) ∞ 1 ) and

Using this fact in Equation (31) yields the theorem.

because

D Extensions to other measures of uncertainty

In the main paper, we focus on developing the posterior hallucination rate; however, there are other quantities that can also be used to predict model performance. In Bayesian machine learning, a commonly used quantity for this purpose is the posterior mutual information between F and Y , which is denoted by I(Y; F | x , D n ) . This is often interpreted as quantifying epistemic uncertainty. As is the case for other quantities presented in the paper, it is not possible to compute it directly in a CGM. Nevertheless, in this section we extend the results of the paper and demonstrate that by using a similar methodology to the one presented, it is possible to construct estimators for it.

D.1 Decomposing uncertainty

There are several ways to decompose total uncertainty about an answer into epistemic and aleatoric components. One such decomposition proceeds by defining uncertainty via entropy and then using a clever decomposition of mutual information to link various interpretable quantities. We use this decomposition throughout.

To elaborate, we define total predictive uncertainty as the entropy of the posterior predictive, denoted as H(Y | x , D n ) . We define aleatoric uncertainty-which represents the irreducible portion of uncertainty-as the expected entropy of the likelihood, denoted as E p (f |D n ) [H(Y | x , f)] . And finally, we define the difference between these two quantities as the epistemic uncertainty, which represents the reducible component of the uncertainty.

By defining epistemic uncertainty in this way, it turns out to be equivalent to I(Y; F | x , D n ) , as the following holds,

In this equation, we see that the total predictive uncertainty depends only on the predictive distribution, p (y | x , D n ) , which-under our assumptions-is the estimand of a CGM, p θ (y | x , (x i , y i ) n 1 ) . The aleatoric uncertainty, on the other hand, depends both on the likelihood function, p (y | x , f) , and the posterior over mechanisms, p (f | x , (x i , y i ) n 1 ) , which are only latently modeled by a CGM.

In the next section, we show how the aleatoric uncertainty can be expressed using only samples from p (x , y | (x i , y i ) n 1 ) , which enables its estimation with a CGM and, in turn, provides a practical estimator for the epistemic uncertainty of the model.

D.2 Aleatoric uncertainty for conditional generative models

We use a similar proof as in Appendix C.

Theorem 4. Assume that the conditions of Theorem 1 hold for F and (X , Y) . Then for a new independent x ,

Proof.

D.3 Estimators

Using the theorem in the previous section and similar approximations as for the PHR, we construct estimates of the aleatoric and epistemic uncertainty.

In particular, we can define a truncated approximation of the aleatoric uncertainty as

and N -n is a practical number of generated examples. In turn this model approximation for the aleatoric uncertainty allows us to define a finite model approximation for the epistemic uncertainty,

And finally, as in the main body of the paper, we can use Monte-Carlo estimation to construct practical estimates of these quantities. This is summarized below.

Total predictive uncertainty.

Aleatoric uncertainty.

Algorithm 3 ̂ H θ (Y | F , x , (x i , y i ) n 1 )

Require: Query x , context D n = (x i , y i ) n 1 , CGM p θ , number of context samples M , number of predictive uncertainty samples K , max context length N .

Epistemic Uncertainty.

Wenote that Algorithm 3 is very similar to Algorithm 1, with the key distinction being the replacement of the THR computation with the calculation of the average entropy of the likelihood. This similarity is not a coincidence, as both can be seen as specific instances of a more general predictive resampling algorithm presented by Fong et al. [39], albeit in a different context.

In Appendix G.2.2 and Appendix G.1.1 we provide some experiments comparing this epistemic uncertainty estimator against the PHR and find that both are correlated and thus represent similar information.

)

E Evaluation details

E.1 Synthetic tasks

We implement our neural process by modifying the Llama 2 architecture [42] to model sequences of continuous variables. We replace the tokenizer with a linear layer and the output categorical distribution with a Riemann distribution [7]. We train the model from random initialization on sequences of (x , y) pairs using a standard next token prediction objective and use the AdamW optimizer [84] with learning_rate = 0 . 0001 , β 1 = 0 . 9 , β 2 = 0 . 999 , ϵ = 1e -8 , and weight_decay = 1e -6 3 . We use a cosine learning rate schedule, with warmup of 2000 steps, and decay final learning rate down to 10% of the peak learning rate.

We define the (1ϵ )-likely set with ϵ = 0 . 05 such that a response y is a hallucination if it falls outside of the 95% confidence interval of a given sampled distribution conditioned on x . The data generating process is described in Appendix F.1.

E.2 Natural language ICL tasks

We consider tasks defined by six datasets: Stanford Sentiment Treebank (SST2) [49] : predict sentiment positive or negative ; Subjectivity [50] : predict review subjective or objective ; AG News [6] : predict article World , Sports , Business or Sci/Tech ; Medical QP [51] : predict medical question pairs as similar or dissimilar ; RTE [52] : predict two sentences as entailment or not entailment ; and WNLI [53] : predict sentence with pronoun replaced as entailment or not entailment . We replace the label Sci/Tech with Science for AG News and not entailment with not for RTE and WNLI. We filter out any queries of length longer than 116 tokens. Full descriptions of these datasets are given in Appendix F.2.

To implement ICL for a given dataset, we sample a response balanced training set of query/response pairs D train = (x i , y i ) n 1 . Each query in D train is prepended with the string Input: and appended with a new line . Each response is prepended with the Label: and appended with \n\n . Atest query x test is prepended with the Input: and appended with \nLabel: . These strings are concatenated together to form a prompt and we generate a response y from the predictive distribution given by a Llama-2 model p θ (y | x test , D train ) . An example prompt from the Subjectivity dataset is shown in Appendix E.2.

For our experiments in 3.2, we employ LLaMA-2 [42], a family of open source LLMs based on an auto-regressive transformer, pretrained on 2 trillion tokens with a context window of 4,096 tokens. We run LLaMA-2-7B as an unquantized model (16-bit).

To estimate the predictive distribution over responses, we sample n input/label pairs based on the given context length, create a prompt based on the context, and generate y samples. An example of a prompt from the Subjectivity dataset is set forth in Figure 7.

The y samples are generated as single tokens, since all labels for our datasets can be identified based on their first token. We set the temperature and top_p parameters to 1 to provide the greatest possible diversity and randomness to the label output.

For predictive resampling, we initialize the context by sampling n input/label pairs, generate N -n new context examples by producing prompts that include the updated context, and produce y samples from the cumulative context. Again, we set max_new_tokens = 200 , temperature = 1 and top_p = 0 . 9 to provide a high level diversity and randomness to the generated output. Examples of generated context pairs are set forth in Figure 8.

Using the transition scores from the model outputs, we compute the log likelihood needed for the posterior hallucination rate.

Input: the assasins force walter to drive their escape car . Label: objective

Input: vega and ulloa give strong performances as the leading lovers and there are some strong supporting turns , particularly from najwa nimri .

Label: subjective

Input: they decide that the path to true love is to purposely set each other up on '' extreme dates '' with the objects of their affections .

Label: objective

Input: '' maid in manhattan '' is a charmer , a pc '' pretty woman '' that ditches the odious prostitution theme for class commentary

Label: subjective

Input: each weekend they come back with nothing but a hangover Label: objective

Input: piccoli 's performance is amazing , yes , but the symbols of loss and denial and life-at-arm's-length in the film seem irritatingly transparent .

Label:

Input: pick any word , even the tiniest , and you will find writers arguing over its relative importance , its ' correct ' usage and how you pronounce it .

Label: objective

Input: janus' entry does a number of things the novel fails at : it tells the story of the novel and , better yet , it 's funny .

Label: subjective

Input: this is the kind of show that 's got the warm n' fuzzies all over , especially if you have a sick sense of humor .

Label: subjective

Input: even if you are an apple and orange man or woman who would be more comfortable at a state fair rodeo than in a silk dress , this should be on your list .

Label: objective

Input: nearly every adjective one could use to describe a movie theater , except expensive and first-run , can be used to describe this place .

Label: subjective

F Dataset details

F.1 Synthetic

(a) Training Datasets

(b) Generated Datasets

training dataset1

training dataset2

training dataset3

generated dataset1

generated dataset2

generated dataset3

2.5

1.25

2.5

>

1.0

2.0

1.5

0.25

0.75

0.50

1.00

2.0

1.0

0.5

0.00

0.5

0.5

0.0

training dataset4

0.25

0.50

training dataset5

2

training dataset6

generateddataset4

generateddataset5

0.5

0.0

generateddataset6

1.50

2.5

1.25

2.0

1.00

1.5

-1

>

0.75

0.50

1.0

0.2

0.4

0.25

0.25

0.00

0.5

0.0

0.0

0.2

Z-

Z

Z-

5'0

0.5

0.0

0.5

1.0

1.5

i

0.0

1.0

。

>

0.5

0.0

> -1.5

0.5

2.0

1.0

0.5

2.5

3.0

0.0

0.5

1.0

1.5

-3.5

1.5

2

X

X

-2

X

Queries x are sampled from a uniform distribution on [-2, 2]. Responses y are sampled from a normal distribution with mean µ (x) parameterized by a random ReLU neural network conditioned on x and constant standard deviation σ = 0 . 1 . We generate a set of 8000 sequences, each corresponding to a distinct random re-initialization of the neural network, with 2000 ( x , y ) examples each. Training and test data are generated over non-overlapping sets of generated sequences. Example training datasets are plotted as different colors in Figure 9.

F.2 Language

For our experiments in 3.2, we randomly sample context examples and test input/label pairs from the following datasets:

Stanford Sentiment Treebank (SST2) SST2 [49] is a corpus with fully labeled parse trees that allows for a complete analysis of the compositional effects of sentiment in language. The corpus consists of 11,855 single sentences extracted from movie reviews. It was parsed with the Stanford parser and includes a total of 215,154 unique phrases from those parse trees, each annotated by 3 human judges. Sentiments are classified as binary labels "positive" or "negative".

Subjectivity (Subj) The Subjectivity dataset [50] contains 5,000 movie review snippets from www.rottentomatoes.com labeled "subjective", and 5,000 sentences from plot summaries available from www.imdb.com labeled "objective". Selected sentences or snippets are at least ten words long and are drawn from movies released post-2001.

AG News The AG News dataset [6] contains 496,835 categorized news articles from more than 2,000 news sources. The 4 largest classes (World, Sports, Business, Sci/Tech) were chosen from this corpus to construct our dataset, including only the title and description fields.

Medical Questions Pairs (MQP) The MQP dataset [51] consists of 3,048 similar and dissimilar medical question pairs hand-generated and labeled by doctors based on patient-asked questions randomly sampled from HealthTap. Each question results in one positive question pair ("similar") that looks very different by superficial metrics, and a negative question pair ("different") that conversely look very similar, so as to ensure that the task is not trivial.

Recognizing Textual Entailment (RTE) The RTE dataset [52] comes from a series of annual textual entailment challenges. Examples are constructed based on news and Wikipedia text, and labeled as binary classifications based on whether or not there is entailment.

Winograd Schema Challenge (WNLI) The WNLI dataset [53] consists of 1,100 sentence pairs with ambiguous pronouns with different possible referents. The task is to determine whether the sentence with a substituted pronoun is entailed by the original sentence.

G Additional results

G.1 Synthetic

We extend the synthetic experiments in the paper in two ways. First, we provide additional results for the setup described in the main sections. Second, we provide additional results in a controlled experiment where we have access to analytic expressions for the posterior predictive.

G.1.1 Additional results with main paper setup

Figure 10 and Figure 11 offer different visualizations that support and extend the results in our main paper-specifically that the posterior hallucination rate (PHR) is an accurate predictor of the true hallucination rate (THR) under various ϵ settings and context lengths. Specifically, for different ϵ settings, Figure 10 displays scatter plots of the THR against the PHR, while Figure 11 plots the average THR and PHR against the number of in-context examples.

06

(a)

ϵ

= 0

.

02

(b)

ϵ

= 0

.

05

(c)

ϵ

= 0

.

1

(d)

ϵ

= 0

.

2

Context Length: 2

Context Length: 5

Context Length:10

Context Length:25

Context Length:50

Context Length:100

Rate

Hallucination

0.8

0.6

True

β: 0.958

β:0.993

β: 1.004

β: 1.067

β: 1.262

β: 1.717

0.0

R2: 0.873

R2: 0.826

R2: 0.740

R2: 0.492

R2: 0.312

R2: 0.257

Posterior Hallucination Rate

0.2

0.6

1.0

0.0

Posterior Hallucination Rate

0.2

1.0

Posterior Hallucination Rate

0.2

06.8

1.0

0.0

Posterior Hallucination Rate

1.0

Posterior Hallucination Rate

0.4

1.0

0.0

Posterior Hallucination Rate

0.2

0.6.8

1.0

Context Length: 2

Context Length:10

Context Length:25

Context Length: 50

Context Length: 100

Rate

True Hallucination

02

β: 0.942

β: 0.972

β: 0.986

β: 1.053

β: 1.220

β: 1.489

R2:0.865

R2: 0.810

R2:0.714

R2: 0.466

R²: 0.305

R2: 0.205

Posterior Hallucination Rate

0.2

0.8

1.0

Posterior Hallucination Rate

1.0

Posterior Hallucination Rate

0.6

0.8

1.0

Posterior Hallucination Rate

1.0

Posterior Hallucination Rate

Posterior Hallucination Rate

1.0

Context Length:2

Context Length:10

Context Length:25

Context Length:50

Context Length: 100

Rate

0.8

0.2

0.4

β: 0.938

β: 0.958

p-val:0.000

MSE:0.006

β: 1.003

MSE:0.003

β: 1.133

p-val:0.000

β: 1.270

p-val:0.000

R2: 0.861

R2: 0.801

R2: 0.698

R2: 0.462

R2: 0.315

R2: 0.216

0.0

Posterior Hallucination Rate

0.4

0.6

0.8

1.0

0.0

Posterior Hallucination Rate

0.2

0.4

0.6

1.0

Posterior Hallucination Rate

0.2

0.6

Posterior Hallucination Rate

0.2

0.4

0.8

1.0

Posterior Hallucination Rate

0.2

0.4

0.6

0.8

1.0

Posterior Hallucination Rate

0.6

Context Length: 2

Context Length:10

Context Length: 25

Context Length:50

Context Length: 100

eHallucination Rate

1.0

β: 0.879

β: 0.898

β: 0.904

β: 0.931

β: 1.008

β: 1.135

R2: 0.855

R2: 0.789

R2: 0.681

R2: 0.451

R²: 0.310

R2: 0.253

0.0

Posterior Hallucination Rate

0.2

0.4

0.6

0.8

1.0

0.0

Posterior Hallucination Rate

02

0.4

0.6

0.8

1.0

0.0

Posterior Hallucination Rate

0.2

0.4

0.6

0.8

1.0

0.0

Posterior Hallucination Rate

0.2

0.4

0.6

1.0

Posterior Hallucination Rate

0.4

0.6

0.8

1.0

Posterior Hallucination Rate

0.2

0.4

0.8

1.0

Context Length: 2

Context Length: 5

Context Length: 10

Context Length:25

Context Length: 50

Context Length: 100

Rate

e Hallucination

0.4

β: 0.878

β: 0.887

β: 0.879

β: 0.863

β: 0.902

β: 0.825

e

R2: 0.846

R2: 0.770

R2: 0.658

R2: 0.430

R2: 0.307

R2: 0.219

0.0

Posterior Hallucination Rate

0.2

0.4

0.6

0.8

1.0

0.0

Posterior Hallucination Rate

02

0.4

0.8

1.0

0.0

Posterior Hallucination Rate

0.2

0.4

0.8

1.0

0.0

Posterior Hallucination Rate

0.2

0.4

0.6

0.8

1.0

0.0

Posterior Hallucination Rate

0.2

0.4

0.6

0.8

1.0

0.0

Posterior Hallucination Rate

0.2

0.4

0.6

0.8

1.0

(e)

ϵ

= 0

.

5

(a)

ϵ

= 0

.

02

(b)

ϵ

= 0

.

05

(c)

ϵ

= 0

.

1

(d)

ϵ

= 0

.

2

(e)

ϵ

= 0

.

5

1.0

True Hallucination Rate

1.0

True Hallucination Rate

1.0

True Hallucination Rate

1.0

True Hallucination Rate

1.0

True Hallucination Rate

0.8

Posterior Hallucination Rate

0.8

Posterior Hallucination Rate

0.8

Posterior Hallucination Rate

0.8

Posterior Hallucination Rate

0.8

Posterior Hallucination Rate

0.6

β:1.008

0.6

β:0.990

0.6

β:0.967

0.6

β:0.913

0.6

B:0.872

0.4

p-val:0.000

MSE:0.006

0.4

p-val:0.000

MSE:0.006

0.4

p-val:0.000

MSE:0.006

0.4

p-val:0.000

MSE:0.005

0.4

p-val:0.000

MSE:0.003

R²:0.911

R²:0.904

R²:0.899

R²:0.891

0.2

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

0.0

0

25

50

Num. Examples: n

75

100

125

150

175

200

0

25

50

Num. Examples: n

75

100

t25

150

175

200

0

25

50

Num. Examples: n

75

100

f25

150

175

200

25

50

Num. Examples: n

75

900

125

150

175

200

0

25

50

Num. Examples: n

75

100

f25

150

175

200

as ˜ ϵ , used to calculate the PHR. Notably, we see that setting ˜ ϵ = 0 . 1 results in the PHR being a more accurate predictor of the THR as context length grows, as opposed to ˜ ϵ = 0 . 05 (where there is no misspecification). This reflects our observation that the PHR underestimates the THR, particularly for longer context lengths.

Context Length: 2

Context Length:10

Context Length:25

Context Length:50

Context Length:100

True Hallucination Rate

0.4

β:0.881

β:0.935

β:0.993

β: 1.205

β:1.624

β: 2.478

0.0

R2: 0.861

R2: 0.804

R2: 0.705

R2: 0.459

R2: 0.309

R2: 0.248

0.0

0.2

PHR:  = 0.02

0.4

0.6

0.2

PHR: = 0.02

0.6

0.8

I.M

0.2

PHR:  = 0.02

0.6

0.8

PHR:  = 0.02

0.4

0.8

1.0

0.2

PHR: = 0.02

0.6

0.8

1.0

0.2

PHR: = 0.02

0.4

0.8

Context Length: 2

Context Length: 10

Context Length:25

Context Length:50

Context Length:100

Rate

True Hallucination

β:0.942

β: 0.972

β:0.986

β:1.053

β: 1.489

0.0

R2: 0.865

R2:0.810

R2: 0.714

R2: 0.466

R²: 0.305

R2:0.205

0.0

0.2

PHR:  = 0.05

0.4

0.6

1.0

0.2

PHR:  = 0.05

0.6

0.8

0.2

PHR:  = 0.05

90

0.2

0.4

PHR:  = 0.05

0.8

0°1

0.0

0.2

PHR:  = 0.05

0.4

0.8

1.0

0.0

0.2

0.4

PHR:  = 0.05

0.8

1.0

Context Length: 2

Context Length:10

Context Length: 25

Context Length: 50

Context Length: 100

Rate

1.0

True Hallucination

0.6

β: 1.037

β: 1.042

β: 1.008

β:0.950

β: 0.972

β: 1.058

0.2

p-val:0.000

MSE:0.011

p-val:0.000

MSE:0.013

p-val:0.000

MSE:0.011

p-val:0.000

p-val:0.000

MSE:0.002

p-val:0.000

MSE:0.001

0.0

R2: 0.866

R2: 0.812

R2: 0.716

R2: 0.473

R2: 0.319

R2: 0.227

PHR:  = 0.1

0.4

0.6

80

PHR:  = 0.1

80

PHR:  = 0.1

0.8

Zr

PHR:  = 0.1

0.4

0.6

0.8

1.0

0.0

0.2

PHR:  = 0.1

0.4

0.6

0.0

0.2

PHR: = 0.1

0.4

0.6

0.8

0

Context Length: 2

Context Length:10

Context Length:25

Context Length:50

Context Length:100

Rate

.0

True Halluination F

0.6

0.4

β:1.155

β:1.137

β:1.060

β: 0.904

β: 0.821

β: 0.794

0.0

R2: 0.864

R2: 0.810

R2: 0.713

R2: 0.474

R2: 0.321

R²: 0.254

0.0

0.2

PHR:  = 0.2

0.4

0.6

0.8

1.0

0.0

0.2

PHR:  = 0.2

0.4

0.6

0.8

0.0

0.2

PHR: E= 0.2

0.4

0.6

0.8

1.0

0.0

0.2

PHR:  = 0.2

0.4

0.6

0.8

1.0

0.0

0.2

PHR:  = 0.2

0.4

0.6

0.8

1.0

0.0

0.2

PHR:  = 0.2

0.4

0.6

0.8

1.0

Context Length: 2

Context Length:10

Context Length: 25

Context Length:50

Context Length: 100

Rate

1.0

0.8

0.6

0.2

p-val:0.000

MSE:0.012

β:1.868

β:1.627

B:1.176

B0.899

β: 0.587

R2: 0.848

R2: 0.791

R2: 0.690

R2: 0.447

R2: 0.286

R2: 0.154

Z0

PHR:  = 0.5

0.4

0.6

0.8

1.0

0.0

0.2

PHR:  = 0.5

0.6

0.8

0.2

PHR:  = 0.5

0.4

0.6

0.8

0.0

0.2

PHR:  = 0.5

0.4

0.6

0.8

1.0

0.0

0.2

PHR:  = 0.5

0.4

0.6

0.8

1.0

Z0

PHR:  = 0.5

0.4

0.6

0.8

1.0

Mutual information. We also assess the mutual information (MI) estimator for epistemic uncertainty, derived in Appendix D and defined in Equation (37), to see how well it predicts the true hallucination rate (THR). Figure 13 shows that the MI estimates are significantly correlated with the THR, which indicates that the MI can also be an effective predictor of hallucinations. Figure 14 allows us to look deeper into the relationships between the THR, PHR, and MI. In comparing Figures 14a and 14b, we see that the PHR has a more linear relationship to the THR than the MI, which has a more sigmoidal relationship to the THR. Nevertheless, Figure 14c which plots the PHR against the MI shows that both are strongly correlated. Their correlation provides evidence that the PHR and MI encode related

information, and that the PHR is a measure of epistemic uncertainty, which is expected from its definition.

(a) MI and THR vs.

n

(b) Calibration: THR vs. MI against different numbers of context examples

1.0

Context Length:2

Context Length:10

Context Length:50

TrueHallucination Rate

0.8

Mutual Information

Rate

1.0

0.6

Hallucination

0.8

β:0.365

0.6

p-val:0.000

0.4

0.2

MSE:0.011

True

0.4

0.2

β: 0.273

p-val:0.000

p-val:0.000

β: 0.438

p-val:0.000

β:0.634

MSE: 0.017

MSE: 0.012

MSE:0.002

0.0

R2:0.785

R2:0.680

R2:0.319

0.0

0

25

50

75

100

125

150

175

200

0

1

2

3

0

1

2

3

0

1

2

3

Num. Examples: n

Mutual Information

Mutual Information

Mutual Information

(a) THR vs. PHR

(b) THR vs. mutual information

Context Length: 2

Context Length:10

Context Length: 25

Context Length: 50

Context Length:100

True Hallucination Rate

80

0.6

0.4

β:0.942

β: 0.972

β: 0.986

β: 1.053

β: 1.220

β: 1.489

0.2

p-val: 0.000

MSE:0.011

p-val: 0.000

MSE:0.013

p-val: 0.000

MSE:0.011

MSE:0.005

p-val:0.000

MSE:0.002

p-val; 0.000

p-val:0.000

MSE:0.001

0.0

R2: 0.865

R2: 0.810

R2: 0.714

R2: 0.466

R2: 0.305

R2: 0.205

0.0

Posterior Hallucination Rate

0.2

0.8

1.0

Posterior Hallucination Rate

0.8

1.0

0.0

Posterior Hallucination Rate

0.2

0.40.6

0.8

1.0

0.0

Posterior Hallucination Rate

0.2

0.8

Posterior Hallucination Rate

0.2

0.8

1.0

0.0

Posterior Hallucination Rate

0.2

1.0

Context Length: 2

Context Length:10

Context Length:25

Context Length:50

Context Length: 100

True Halluination Rate

0.8

0.2

β:0.273

p-val:0.000

β: 0.347

β: 0.438

β:0.585

β; 0.634

β: 0.341

MSE: 0.018

R2: 0.785

R2: 0.744

R2: 0.680

R2: 0.482

R2: 0.319

R2: 0.127

Mutual Information

Mutual Information

Mutual Information

2

Mutual Information

2

Mutual Information

Mutual Information

Context Length: 2

Context Length:10

Context Length:25

Context Length:50

Context Length:100

Rate

1.0

0.6

0.2

p-val:0.000

MSE:0.006

β: 0.292

p-val:0.000

MSE: 0.004

β: 0.359

β: 0.443

p-val:0.000

β: 0.534

0000

p-val:0.000

β: 0.475

β:0.207

0.0

R2:0.918

R2: 0.929

R²: 0.947

R2: 0.955

R2: 0.875

R2: 0.507

Mutual Information

2

Mutual Information

1

2

Mutual Information

2

Mutual Information

2

Mutual Information

1

2

Mutual Information

1

2

(c) PHR vs. mutual information

G.1.2 Additional results with an analytic posterior

Our method relies on the assumption that

is sufficiently accurate for our procedure to work. As discussed in Section 3.1, if this assumption is incorrect, our method may produce inaccurate results. However, in the experiments presented in the main body of the paper, we did not have an analytic posterior available to compare against, nor a way to compute ground-truth PHRs.

In this section, we investigate whether Equation (38) holds in a simple setting where we have an analytic posterior for comparison.

Setup. We use two simple synthetic datasets where it is possible to analytically compute the posterior p (y | x , D n ) .

Linear Regression : In the first setting, we use a simple one-dimensional linear regression. We assume f ∼ N (0 , I 2 ) , and each sequence of (x , y) is such that x ∼ N (0 , σ x ) and y ∼ N (f 1 +f 2 x , σ y ) , where σ x = 1 and σ y = 0 . 1 .

Polynomial Regression : In the second setting, we use linear regression on a polynomial basis. Specifically, f ∼ N (0 , I d +1 ) , x ∼ N (0 , σ x ) , and y ∼ N (f ⊤ ϕ (x) , σ y ) , where ϕ (x) ∈ R d +1 and ϕ (x) i = x i -1 . We use the same hyperparameters as in the linear regression setting and set d = 3 .

We then train a CGM using the same setup and architecture described in Section 3.1, with the only difference being that we train on sequences of length 100. For illustrative purposes, the left panel of Figure 15 displays the sequences used to train the model, while the right panel displays sequences generated by our model.

(a) Training Datasets

(b) Generated Datasets

training dataset1

10.0

7.5

trainingdataset2

training dataset3

0.0

5.0

-2.5

2.5

5.0

0.0

7.5

2.5

10.0

5.0

-12.5

-10.0

-7.5

-2

-15.0

S'-

10

trainingdataset4

trainingdataset5

training dataset6

generateddataset4

generateddataset5

generateddataset6

15

10

10

>

QT

generateddataset7

12

10

10

-15

X

X

We conduct two different experiments on these datasets.

Experiment 1 Setup. Calibration of resampling distribution: First, we examine the distribution produced by our resampling procedure. Specifically, we study the calibration of the distribution ˆ p θ (y | x , D n ) , where a sample y ∼ ˆ p θ (y | x , D n ) is generated by first sampling from p θ ((x i , y i ) N n +1 | D n ) , and then sampling from p θ (y | x , D N ) -where D N = D n ∪ (x i , y i ) N n +1 and N is defined as in Algorithm 1. Ideally, all three sampling procedures should align, and we would expect ˆ p θ (y | x , D n ) ≈ p θ (y | x , D n ) ≈ p (y | x , D n ) 4 . However, inaccuracies in the approximation may prevent this from being the case.

Weapproach this question from the perspective of calibration, as we believe it provides an interpretable metric for understanding the model's predictions and captures an important property that is essential for the PHR to function correctly.

Specifically, we ask whether the distribution produced by ˆ p θ (y | x , D n ) is calibrated in the sense that if Q q (x , D n ) is the q -th quantile of ˆ p θ (y | x , D n ) , then

✶ where the inner integral is taken with respect to the data distribution. We refer to this metric as the calibration error . If it were true that ˆ p θ (y | x , D n ) ≈ p (y | x , D n ) , then the inner integral would be zero for every q .

We estimate the value of this integral using a Monte Carlo estimate with 200 samples for Q q (x , D n ) and 200 samples for the integral, and then we evaluate over 200 values of q . Moreover, we set ( N -n ) to 30 to compute ˆ p θ .

Experiment 2 Setup. Model PHR vs Analytic PHR: For our second experiment, we compare our estimated PHR with an almost fully analytic counterpart. In particular, we compute the analytic PHR

by using an analytic expression for A (f , x) based on the quantiles of the normal distribution and estimate the integrals by sampling from the analytic posterior p (f | D n ) and the analytic posterior predictive p (y | x , D n ) . It is important to note that this differs from our evaluation in Section 3.1, where we compared the PHR to the THR and used p θ (y | x , D n ) instead of p (y | x , D n ) , as no analytic counterpart was available. We compute the estimate of the PHR given by the transformer model using Algorithm 1 with parameters

0.0

0.2

0.4

0.6

0.8

1.0

Predicted Quantile

Q

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of Samples Under

Q

Calibration Error: 0.023

Context Length: 1

Ideal Relationship

Actual Relationship

0.0

0.2

0.4

0.6

0.8

1.0

Predicted Quantile

Q

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of Samples Under

Q

Calibration Error: 0.019

Context Length: 2

Ideal Relationship

Actual Relationship

0.0

0.2

0.4

0.6

0.8

1.0

Predicted Quantile

Q

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of Samples Under

Q

Calibration Error: 0.035

Context Length: 8

Ideal Relationship

Actual Relationship

0.0

0.2

0.4

0.6

0.8

1.0

Predicted Quantile

Q

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of Samples Under

Q

Calibration Error: 0.026

Context Length: 1

Ideal Relationship

Actual Relationship

0.0

0.2

0.4

0.6

0.8

1.0

Predicted Quantile

Q

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of Samples Under

Q

Calibration Error: 0.029

Context Length: 2

Ideal Relationship

Actual Relationship

0.0

0.2

0.4

0.6

0.8

1.0

Predicted Quantile

Q

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of Samples Under

Q

Calibration Error: 0.025

Context Length: 8

Ideal Relationship

Actual Relationship

0.0

0.2

0.4

0.6

0.8

1.0

Estimated PHR

0.0

0.2

0.4

0.6

0.8

1.0

True PHR

Context Length: 1

Ideal

Estimated vs True PHR

0.0

0.2

0.4

0.6

0.8

1.0

Estimated PHR

0.0

0.2

0.4

0.6

0.8

1.0

True PHR

Context Length: 2

Ideal

Estimated vs True PHR

0.0

0.2

0.4

0.6

0.8

1.0

Estimated PHR

0.0

0.2

0.4

0.6

0.8

1.0

True PHR

Context Length: 8

Ideal

Estimated vs True PHR

0.0

0.2

0.4

0.6

0.8

1.0

Estimated PHR

0.0

0.2

0.4

0.6

0.8

1.0

True PHR

Context Length: 1

Ideal

Estimated vs True PHR

0.0

0.2

0.4

0.6

0.8

1.0

Estimated PHR

0.0

0.2

0.4

0.6

0.8

1.0

True PHR

Context Length: 2

Ideal

Estimated vs True PHR

0.0

0.2

0.4

0.6

0.8

1.0

Estimated PHR

0.0

0.2

0.4

0.6

0.8

1.0

True PHR

Context Length: 8

Ideal

Estimated vs True PHR

Results. Figure 16 and Figure 17 show the calibration curves according to the procedure described above in Experiment 1. Specifically, the x-axis represents the percentage of points that should lie below a given quantile if the distribution is calibrated, while the y-axis shows the actual number of points below that quantile. The calibration error is the area between the ideal calibration curve (blue) and the observed calibration curve (pink). Each panel in the figures represents the calibration with a different number of observed points in the context. In both figures, we observe that the calibration error is below 4%, independent of context length, which suggests that the distributions are well-calibrated.

The results for Expeirment 2 can be found in Figure 18 and Figure 19. They display the estimated PHR using the transformer model versus the estimate obtained from the analytic posterior. Each point in the graphs corresponds to a single test example; its x-coordinate represents the transformer-based estimate, while its y-coordinate corresponds to the analytic posterior estimate. If the transformer perfectly approximated the posterior and the number of Monte Carlo samples were sufficiently high, each point would lie on the x-y line. Most importantly, they show that the PHR estimate produced by the trained transformer is close to the estimate from the analytic posterior. The plots also indicate, as expected, that the PHR decreases as the context length increases.

Discussion. The results from both experiments suggest that the distribution p θ approximates p well, and that Algorithm 1 operates as expected. Specifically, we observe that the resampling mechanism used in Algorithm 1 leads to a calibrated distribution, and that the estimated PHR closely matches its analytic counterpart. This supports the hypothesis that the underestimation observed in Section 3.1 is due to the fact that the THR is not the PHR, and while these quantities are related, they are inherently distinct.

Although this certainly does not prove that p θ ≈ p for all ICL tasks, it provides further evidence of the applicability of the algorithm as-in these simple scenarios-we do not observe any significant failure modes, such as distribution drift due to resampling.

G.2 Language

G.2.1 Gemma-2

Here we evaluate the posterior hallucination rate estimator on the same in-context learning tasks used in the main body of the paper using Gemma-2 9B [43].

1.0

SST2

2.51.0

Subjectivity

2.51.0

AG News

2.51.0

MedicalQP

2.51.0

RTE

2.51.0

WNLI

2.5

0.8

-2.00.8-

2.00.8

2.00.8-

2.00.8

-2.00.8-

2.0

ResponseEntropy

Error Rate

0.6

1.50.6

1.50.6

1.50.6

1.50.6

1.50.6

1.5

0.4

1.00.4-

1.00.4-

1.00.4

1.00.4

1.00.4

1.0

0.2

0.50.2

0.50.2

0.50.2

0.50.2

-0.50.2-

0.5

0.0

5

n examples

10

15

20

25

30

0.00.0-

$

nexamples

10

15

20

25

30

-0.0

00

n examples

90

15

20

25

30

-0.0

0.0-

5

n examples

90

15

20

25

30

0.00.0-

5

n examples

10

15

20

25

30

-0.0

0.0-

n examples

90

15

20

25

30

0.0

Error Rate

Error Random Classifier

+ + +

Predictive Entropy

Entropy Random Classifier

Figure 20 shows the error rate (green) and response entropy (blue) of Gemma-2-9b. In general, both metrics decrease and saturate with longer context lengths. For all tasks, the model performs at least slightly better than random, and is particularly performant on SST2, Subjectivity, AG News and RTE. However, for Medical QP and WNLI, error rates are very close to random, indicating poor generalization.

SST2

Subjectivity

AGNews

MedicalQP

RTE

WNLI

1.0

β:0.831

β:1.023

β:0.847

β:0.652

β: 0.276

0.8

p-val:0.000

MSE:0.018

p-val:0.000

MSE:0.030

p-val:0.000

MSE:0.015

p-val:0.000

MSE:0.016

p-val:0.000

MSE:0.012

p-val:0.000

MSE:0.013

R²: 0.174

R²:0.130

R²: 0.270

R²:0.304

R²:0.062

R2:0.009

Rate

0.6

0.4

0.2

0:0

5

90

n examples

15

20

25

30

10

n examples

15

20

25

30

5

90

nexamples

15

20

25

30

5

10

nexamples

15

20

25

30

10

n examples

15

20

25

30

10

n examples

15

20

25

30

Posterior Hallucination Rate

Results. We report results for Gemma-2-9b. We set N -n = 5 , M = 10 , and K = 50 . Figure 21 plots the MHR and estimated posterior hallucination rate against the number of in-context examples with ϵ = 0 . 05 . As with the Llama-2 results presented in the main body of the paper, we see that the posterior hallucination rate is a good estimator of the MHR.

SST2

Subjectivity

AG News

Medical QP

RTE

WNLI

1.0

β:0.833

β:0.828

β:0.550

β:0.771

β:0.592

0.8

p-val:0.000

MSE:0.020

p-val:0.000

MSE:0.019

p-val:0.000

MSE: 0. 059

p-val:0.000

MSE: 0.021

p-val:0.000

MSE:0.026

p-val:0.000

MSE:0.047

R²:0.602

R²:0.681

R²:0.416

R²:0.208

R²:0.502

R²:0.190

0.6

0.4

0.2

0.0

$

90

nexamples

15

20

25

30

10

nexamples

15

20

25

30

5

90

nexamples

15

20

25

30

5

10

nexamples

15

20

25

30

5

90

n examples

15

20

25

30

5

90

nexamples

15

20

25

30

Empirical Error Rate

Posterior Hallucination Rate

Error Random Guessing

Figure 22 plots the empirical error rate and estimated posterior hallucination rate against the number of in-context examples. When ϵ is set to a high value (e.g., >0.5), the PHR closely aligns with the empirical error rate across all datasets, with the exception of WNLI, likely due to Gemma-2-9b's limited generalization to this task. Notably, it is surprising that the PHR tracks the empirical error rate well for the Medical QP dataset, given Gemma-2-9b's similarly weak generalization to this task. Besides this discrepancy, the results for Gemma corroborate those found for Llama-2 in the main paper.

G.2.2 Llama-2

Here we extend the natural language ICL task results of the main paper using the Llama-2 family of LLMs [42].

Results. We report results for Llama-2-7b with N -n = 5 , M = 10 , and K = 50 . Figure 23 in Appendix G shows that the posterior hallucination rate serves as a reliable estimator of the MHR across different ϵ settings. We further ablate the ϵ parameter against the empirical error rate and report results for each natural language ICL task in Figure 24 of Appendix G.

In Table 1, we report the results of an ablation study on SST2 that varies the number of generated examples, MC samples, y samples, and model parameters. We see that increasing the number of MCand y samples improves the R 2 scores for both the MHR and empirical error rate. In contrast, increasing the number of generated examples or model size alone results in a performance decline for this task.

In Figure 25, we compare the mutual information (MI) estimator with the error rate, MHR, and PHR across all tasks. Specifically, Figure 25a illustrates the relationship between MI and error rate, while Figure 25b illustrates the relationship between MI and MHR. Additionally, Figure 25c depicts the relationship between MI and PHR. The high R 2 values observed between the MI and PHR provide further evidence that both estimators capture the same underlying information.

(a) SST2

(b) Subjectivity

(c) AG News

(d) Medical QP

(e) RTE

Model Hallucination Probability

Posterior Hallucination Rate

= 0.02

= 0.05

= 0.1

= 0.2

= 0.5

= 0.667

= 0.75

1.0

β: 1.438

β: 1.303

β: 1.226

β: 1.163

β:1.040

β: 0.977

β:0.966

0.8

p-val:0.000

MSE: 0.004

p-val:0.000

MSE:0.005

p-val:0.000

MSE:0.006

p-val:0.000

MSE:0.009

p-val:0.000

MSE:0.008

p-val:0.000

MSE:0.005

p-val:0.000

MSE:0.005

0.6

R2: 0.574

R2: 0.606

R2: 0.636

R2: 0.628

R2: 0.821

R2:0.889

R2: 0.893

0.4

0.2

00

10

20

30

10

20

10

20

10

20

30

10

20

10

20

10

20

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

= 0.02

= 0.05

= 0.1

= 0.2

= 0.5

= 0.667

= 0.75

1.0

β: 2.165

β:1.953

β: 1.680

β:1.355

β: 1.079

B:1.003

β:0.965

0.8

p-val:0.000

MSE:0.031

p-val:0.000

MSE:0.032

p-val:0.000

MSE:0.027

p-val:0.000

MSE:0.025

p-val:0.000

MSE:0.025

p-val:0.000

MSE:0.016

p-val:0.000

MSE:0.016

0.6

R2: 0.461

R2: 0.594

R2: 0.704

R2: 0.731

R2:0.681

R2: 0.781

R2: 0.787

0.4

0.2

0.0

nexamples

10

20

30

nexamples

10

20

30

nexamples

10

20

nexamples

10

20

30

nexamples

10

OC

nexamples

10

20

30

nexamples

10

20

30

= 0.02

=0.05

= 0.1

= 0.5

=0.667

= 0.75

1.0

β:1.532

β: 1.499

β:1.398

β: 1.264

β: 1.082

β: 1.028

β: 1.009

0.8

p-val:0.000

MSE:0.010

p-val:0.000

MSE:0.015

p-val:0.000

MSE:0.016

p-val:0.000

MSE:0.020

p-val:0.000

MSE:0.022

p-val:0.000

MSE:0.018

p-val:0.000

MSE:0.017

0.6

R2: 0.360

R2: 0.444

R2: 0.545

R2:0.600

R2: 0.687

R2: 0.765

R2: 0.784

0.4

0.2

0.0

10

20

30

10

20

30

10

20

30

10

20

30

10

20

10

20

0C

10

20

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

Empirical Error Rate

Posterior Hallucination Rate

= 0.02

= 0.05

= 0.1

= 0.2

= 0.5

= 0.667

= 0.75

0.8

1.0

β: 1.409

p-val:0.000

MSE:0.002

β:1.228

p-val: 0.000

MSE:0.001

β: 1.095

p-val: 0.000

MSE:0.001

β:0.938

β:0.873

B:0.916

p-val:0.000

MSE:0.008

β:0.903

0.6

R2: 0.837

R2: 0.914

R2: 0.930

R2: 0.762

R2: 0.318

R2: 0.665

R2:0.677

0.4

0.2

0.0

10

20

30

10

20

30

10

20

30

10

20

30

10

20

10

20

30

10

20

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

= 0.02

= 0.05

= 0.1

= 0.2

= 0.5

= 0.667

= 0.75

1.0

β:1.725

β:1.735

β:1.577

β: 1.250

β:1.016

B:0.958

β:0.910

0.8

p-val:0.000

MSE:0.015

p-val:0.000

MSE:0.018

p-val:0.000

MSE:0.014

p-val:0.000

MSE:0.016

p-val:0.000

MSE:0.022

p-val:0.000

MSE:0.011

p-val:0.000

MSE:0.011

0.6

R2: 0.463

R2: 0.626

R2: 0.771

R2: 0.781

R2:0.588

R2: 0.750

R2: 0.757

0.4

0.2

00

10

20

10

20

10

20

10

20

10

10

20

10

20

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

= 0.02

= 0.05

= 0.1

= 0.2

= 0.5

= 0.667

= 0.75

1.0

β: 1.881

β: 1.898

β: 1.659

β:1.335

β: 1.120

β: 1.075

β:1.018

0.8

p-val:0.000

MSE:0.019

p-val:0.000

MSE: 0.017

p-val:0.000

MSE: 0.011

p-val:0.000

MSE:0.012

p-val:0.000

MSE: 0.025

p-val:0.000

MSE: 0.011

p-val:0.000

MSE:0.010

0.6

R2:0.452

R2: 0.657

R2: 0.813

R2: 0.824

R2: 0.571

R2:0.782

R2: 0.796

0.4

0.2

Q.0

10

20

30

10

20

30

10

20

30

10

20

30

10

20

10

20

10

20

nexamples

nexamples

nexamples

n examples

nexamples

nexamples

nexamples

(f) WNLI

(a) SST2

(b) Subjectivity

(c) AG News

(d) Medical QP

(e) RTE

(f) WNLI

Empirical Error Rate

Posterior Hallucination Rate

Error Random Guessing

= 0.02

= 0.05

= 0.1

= 0.2

= 0.5

= 0.667

= 0.75

1.0

β: 2.292

β: 1.765

β: 1.484

β: 1.261

β: 1.060

β: 0.952

β: 0.937

0.8

p-val:0.000

MSE:0.038

p-val:0.000

p-val:0.000

MSE:0.036

p-val:0.000

MSE:0.034

p-val:0.000

MSE:0.016

p-val:0.000

MSE:0.016

p-val:0.000

MSE:0.016

R2: 0.266

R2: 0.290

R2: 0.305

R2: 0.349

R2: 0.685

R2: 0.702

R2: 0.700

0.6

0.4

0.2

0.0

10

30

10

20

30

10

20

10

20

30

10

20

30

10

20

30

10

20

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

= 0.02

= 0.05

= 0.1

= 0.2

= 0.5

= 0.667

= 0.75

1.0

β: 2.192

β: 1.575

β: 1.202

β: 0.915

β: 0.972

β: 0.891

β:0.860

0.8

p-val:0.000

MSE:0.045

p-val:0.000

MSE:0.043

p-val:0.000

MSE:0.040

p-val:0.000

MSE: 0.041

p-val:0.000

MSE:0.029

p-val:0.000

MSE: 0.027

p-val:0.000

MSE:0.027

0.6

R2:0.363

R2: 0.397

R2: 0.430

R2: 0.425

R2: 0.594

R2:0.616

R2:0.624

0.4

0.2

00

10

20

30

10

20

30

10

20

10

20

30

10

OC

10

20

10

20

30

n examples

n examples

n examples

nexamples

n examples

n examples

n examples

= 0.02

= 0.05

= 0.1

= 0.5

=0.667

= 0.75

1.0

β: 2.229

β: 1.748

β: 1.552

β: 1.344

β: 1.094

β:0.963

β: 0.924

0.8

p-val:0.000

MSE-0:090

p-val:0.000

MSE-0:086

p-val:0.000

MSE:0:079

p-val:0.000

MSE:0:068

p-val:0.000

MSE:0:050

p-val:0.000

MSE-0:048

p-val:0.000

MSE-0:048

0.6

R2: 0.126

R2: 0.170

R2: 0.241

R2:0.341

R2: 0.514

R2:0.533

R2: 0.537

0.4

0.2

0.0

10

20

30

10

20

30

10

20

30

10

20

30

10

20

30

10

20

30

10

20

30

n examples

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

= 0.02

= 0.05

= 0.1

= 0.2

= 0.5

= 0.667

= 0.75

1.0

β: 0.884

β: 0.691

β:0.602

β: 0.486

β: 0.567

β: 0.551

β: 0.545

0.8

p-val:0.000

MSE:0.021

p-val:0.000

MSE:0.020

p-val:0.000

MSE:0.020

p-val:0.000

MSE:0.021

p-val:0.000

MSE:0.021

p-val:0.000

MSE:0.020

p-val:0.000

MSE:0.020

0.6

R2:0.185

R2: 0.197

R2:0.199

R2: 0.194

R2:0.222

R2: 0.227

0.4

0.2

0.0

10

20

30

10

20

30

10

20

30

10

20

10

20

30

10

20

10

20

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

= 0.02

= 0.05

= 0.1

= 0.2

= 0.5

= 0.667

= 0.75

1.0

β:1.735

β:1.282

β:0.993

β:0.749

β: 0.870

β:0.808

β:0.773

0.8

p-val:0.000

MSE:0.035

p-val:0.000

MSE:0.032

p-val:0.000

MSE:0.029

p-val:0.000

MSE:0.028

p-val:0.000

MSE:0.025

p-val:0.000

MSE:0.025

p-val:0.000

MSE:0.024

0.6

R2: 0.270

R2: 0.333

R2: 0.416

R2: 0.479

R2: 0.489

R2:0.499

0.4

0.2

0.0

10

20

30

10

20

30

10

20

30

10

20

30

10

20

10

20

10

20

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

= 0.02

= 0.05

= 0.1

= 0.2

= 0.5

=0.667

= 0.75

1.0

β: 1.408

β: 1.071

β:0.846

β: 0.654

β: 0.644

β: 0.570

β: 0.549

0.8

p-val:0.000

MSE: 0.031

p-val:0.000

MSE:0.029

p-val:0.000

MSE: 0.027

p-val:0.000

MSE:0.026

p-val:0.000

MSE:0.029

p-val:0.000

MSE:0.029

p-val:0.000

MSE:0.028

0.6

R2: 0.216

R2: 0.268

R2: 0.315

R2: 0.336

R2: 0.280

R2: 0.281

R2: 0.294

0.4

0.2

0.0

10

20

30

10

20

30

10

20

30

10

20

30

10

20

10

20

30

10

20

30

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

nexamples

(a) Mutual information and error rate vs.

n

examples.

(b) Mutual information and MHR vs.

n

examples.

SST2

Subjectivity

AG News

Medical QP

RTE

WNLI

B：0.262

β:0.235

β:0.231

β：0.129

β:0.176

β:0.146

p-val:0.000

p-val:0.000

p-val:0.000

p-val:0.000

p-val: 0.000

p-val:0.000

MSE: 0.040

R2:0.269

MSE: 0.051

MSE:0:087

R2:0.121

MSE: 0.021

MSE:0.033

R²:0.305

MSE: 0.031

R²:0.228

n examples

15

25

n examples

15

n examples

n examples

20

25

n examples

85

20

n examples

15

25

Empirical Error Rate

Mutual Information

Error RandomGuessing

SST2

Subjectivity

AG News

Medical QP

RTE

WNLI

aβ: 0.169

β:0.223

a:0.253

p-val:0.000

MSE: 0.007

MSE: 0.041

MSE: 0.019

MSE: 0.003

p-val:

MSE: 0.023

0.000

p-val:0.000

R2:0.460

R²:0.308

R²:0.502

R²:0.535

Rate

n examples

25

n examples

20

n examples

n examples

n examples

n examples

Mutual Information

SST2

Subjectivity

AG News

Medical QP

RTE

WNLI

β:0.131

β:0.150

β:0.131

β:0.187

β:0.131

β：0.133

p-val:0.000

p-val:0.000

p-val:0.000

p-val:0.000

p-val:0.000

p-val:0.000

MSE:0.001

R²:0.773

MSE:0.003

R²:0.727

MSE:0.002

R²:0.714

MSE:0.001

R²:0.908

MSE: 0.002

MSE:0.002

R²:0.810

Rate

n examples

20

n examples

n examples

nexamples

20

n examples

20

n examples

Posterior HallucinationRate

Mutual Information

H Computational requirements

For our experiments, we used an internal cluster made up of A100s and RTX 8000s, each with 40 to 48 GB of memory. Training the models for the synthetic experiments took approximately one day on a single machine. For the natural language experiments in the main paper, we ran 50 seeds per dataset, with each seed requiring between 20 minutes and 4 hours. Additionally, we conducted ablations over different values of M and N (as detailed in Appendix G). We also performed further experiments and developed models that required additional computational resources but were not included in the paper.

NeurIPS paper checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We back up the claims made in the abstract and introduction both through theoretical proofs in Section 2, and through experimental results in Section 3 and Appendix G.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations in Section 3.1, Section 4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide all assumptions and proof for Theorem 2 in Appendix C. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the full details needed to reproduce our experiments in Appendix E. Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a zip file containing the code at submission which includes instructions in the README.md, and will provide open source access on github after being published.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the full details needed to reproduce our experiments in Appendix E. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Standard errors are plotted in graphs where appropriate (except for Figure 3a, where the standard deviation is plotted instead because the standard errors were too small to show). In Figure 3b, we show the regression coefficinet, p-value, and R 2 for our calibration plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We describe the computational requirements of our experiments in Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, we went over the NeurIPS code of ethics and, to the best of our knowledge, conform with all the guidelines given.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss both the positive and negative social impacts of our work in Appendix B.1.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We have used publicly available datasets and models made available by huggingface and Meta. We have done our best to follow their guidelines for responsible use.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have done our best to cite original asset owners where applicable.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide open source code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a Bayesian interpretation of in-context learning. This interpretation enables us to calculate the hallucination rate. In other words, by considering in-context examples as observations, the posterior distribution can be computed and the hallucination rate is derived. Numerical experiments verify the practicality of the interpretation.\n\nSTRENGTHS:\n1. The hallucination is one of the most important problems of large language models. This paper explored a theoretical understanding of the problem in in-context learning.\n\n2. The idea to interpret the in-context examples as the observation, described in Section 2.1,  is original and interesting.\n\nWEAKNESSES:\nIn summary, I suggests that the authors clarify the focus of this paper either on the hallucination rate for NLP tasks or on the error rate for general in-context learning tasks.\n\n1. As discussed in L222, the proposed theory cannot be applied as is in NLP tasks and an approximated metric is required. Although the introduction of this paper focused on NLP tasks, the usefulness of the proposed theory in this area is limited, as reported in L265 with Figure 6.\n\n2. Hallucination is a complex phenomenon with a wide range of causes [1]. As mentioned in the above weakness, the effectiveness of this theory is limited to the synthetic regression task. In light of these, I am concerned whether it is appropriate to call the phenomenon examined in this paper a hallucination rate rather than an error rate.\n\n3. The claim of Section 2.2 is to use the cumulative probability of the token generation probability distribution as the confidence for token y. However, the use of cumulative probability as the confidence of token y is a common technique in NLP, for example in nucleus sampling [2]. To demonstrate the importance of the theory presented in this section, I consider it necessary to show its universality when other statistics S are used. Otherwise, I also suggest moving this section to the appendix as noted in the following weakness.\n\n4. L146 says `Algorithm 1 can be understood intuitively without appealing to any Bayesian arguments’. I agree with this statement, and I consider that the necessity of the presented theory depends largely on the discussion in Appendix B.1. Because the appendix is just supplementary material and cannot play a role in justifying the main claim, I suggest the authors should revise the structure of the paper.\n\n[1] Huang+, A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, arXiv:2311.05232.\n\n[2] Holtzman+, The Curious Case of Neural Text Degeneration, ICLR2020.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThis paper discussed broader social impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-30"}

comments: SUMMARY:
This paper presents a method for predicting the hallucination rate of in-context learning with conditional generative models.

STRENGTHS:
- NA

WEAKNESSES:
- Unclear how many queries would be require to validate the approach, as this will be very dependent of task, context, and LLM - this is clearly a missing element of the evaluation
- Evaluation on Llama 2 only - more is definitely require to validate the approach
- No identification of where the hallucination might come from
- Title is misleading - this is basically a hallucination identification method
- Non cost effective as this require multiple LLMs calls

QUESTIONS:
- Why not more LLMs used for experimentation?
- What could be the right number of LLM call needed?
- What parameters is the function of hallucinate rate? What are the optimal parameter? for which task? for which LLMs?

LIMITATIONS:
cf. weakness and questions
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the in-context learning setting of generative AIs, such as large language models, and proposes a new definition for hallucination. It introduces a novel metric, PHR, along with a corresponding estimation method. Unlike traditional metrics, the proposed metric accounts for label ambiguity resulting from unclear task specifications.\n\nSTRENGTHS:\n- The paper highlights a significant issue with using error rates to evaluate hallucination, that the ambiguity on labels caused by the lack of task specification has been overlooked.\n- The definition and algorithm of PHR are logically sound and make sense to me.\n\nWEAKNESSES:\n- The notations are messy and confusing.\n- - There is misuse and abuse of uppercase random variables and lowercase instantiations. For example, the distribution of  $F$  is referred to as  $p(f)$  in lines 87 and 121, while it is referred to as  $p(F)$  in line 124. This problem is exacerbated in Theorem 1, making it hard to read.\n- - The notations PHR/THR can refer to both the definitions and the algorithms, which adds to the confusion. For instance, in the sentence on line 174, “we evaluate the PHR in a setting where we know the true mechanism f* so that we can compare it directly against the true hallucination rate (THR),” PHR appears to refer to the algorithm, while THR seems to refer to Definition 3.\n- The experimental setup is difficult to understand.\n- - For example, the experiments in Section 3.1 are supposed to evaluate the proposed method in a setting where $f*$  is known. However, what $f*$  is and how THR is calculated are not explained. To understand this, at least the way $p(y|x, f*)$  is set should be specified. Given the unused space at the end of the paper, there is no reason for the authors not to elaborate on the experimental settings.\n- - In Section 3.2, the authors attempt to justify the use of PHR by introducing another proposed metric called MHR. It is unclear how MHR could serve as a gold metric. Even if MHR is a gold metric, why not just use MHR?\n- Typos: Line 31 \"the\" should be deleted.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nLimitations have been well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to estimate the hallucination rate for in-context learning (ICL) in a conditional generative model (CGM) from a Bayesian perspective. The authors assume the CGM samples from a posterior predictive distribution over a Bayesian model of a latent parameter and data. They define Posterior Hallucination Rate (PHR), along with True Hallucination Rate (THR) and Model Hallucination Rate (MHR), and provide theoretical proofs and empirical demonstrations to show that they can accurately estimate the true probability of hallucination.\n\nSTRENGTHS:\nS1. The paper tackles an important and well-known issue of hallucination in language models. By focusing on estimating the hallucination rate, the research contributes to improving the reliability and trustworthiness of outputs from CGMs. This makes the work highly relevant in the context of ongoing AI issues.\n\nS2. The paper introduces new metrics, particularly PHR, for future evaluation of model hallucinations. This can potentially advance methodology for assessing model output reliability.\n\nS3. The metrics are well-defined and the pseudo-algorithms help in understanding how the metrics are used.\n\nWEAKNESSES:\nW1. While the authors tested their methods on six datasets from various domains, the range of label content and types may be limited. This limitation could introduce confounding factors. For instance, tasks where one label can be a substring of another (e.g. “similar” vs “dissimilar”) might not fully represent the complexity of real-world scenarios.\n\nW2. The labels used in the datasets, such as “entailment” vs “not”, do not have a clear semantic meaning and relation like “positive” vs “negative” OR “subjective” vs “objective.” This absence of semantic relation might influence the ICL performance. Testing whether the semantic meaning of the labels affect performance and hallucination rate would strengthen the findings. A suggested approach is to use neutral labels such as letters (A,B,etc.) to determine if the lack of semantic meaning of the labels impacts the hallucination detection.\n\nW3. Related to W1, the datasets used in the experiments have either 2 or at most 4 categories. This limited variety raises questions about how the method performs with more complex classification tasks. For example, how would the method fare in English-to-French word translation, where the number of possible categories is significantly higher? Expanding the experiments to include tasks with a larger number of categories would provide a more comprehensive assessment of the method’s robustness.\n\nQUESTIONS:\nQ1. What constitutes an “in-capability” task and “out-of-capability” task?\n\nQ2. In Figure 10, why do the lines, especially for context length 50 graphs, stop earlier? What does that imply about the performance or behavior of the model at longer context lengths?\n\nQ3. The paper references Fong et al. [22] proposing Martingale Posterior distribution that uses posterior predictive. Are there any baseline results for this approximation that could be compared to PHR to evaluate which distribution better defined models?\n\nLIMITATIONS:\nThe authors have addressed the positive and negative societal impacts of their work. It would be beneficial to further discuss and address the dataset choices and potential limitations, such as label diversity and number of categories, to provide a comprehensive view of the conditions under which the method has been tested. This would enhance the understanding of the method’s applicability and generalizability ensuring a more thorough assessment of its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}