Estimating the Hallucination Rate of Generative AI

Authors: andrew  jesson
nicolas  beltran-velez
quentin  chu
sweta  karlekar
jannik  kossen
yarin  gal
john p cunningham
david  blei
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:27:09

Abstract: This paper presents a method for estimating the hallucination rate for in-context learning (ICL) with generative AI. In ICL, a conditional generative model (CGM) is prompted with a dataset and a prediction question and asked to generate a response. One interpretation of ICL assumes that the CGM computes the posterior predictive of an unknown Bayesian model, which implicitly defines a joint distribution over observable datasets and latent mechanisms. This joint distribution factorizes into two components: the model prior over mechanisms and the model likelihood of datasets given a mechanism. With this perspective, we define a \textit{hallucination} as a generated response to the prediction question with low model likelihood given the mechanism. We develop a new method that takes an ICL problem and estimates the probability that a CGM will generate a hallucination. Our method only requires generating prediction questions and responses from the CGM and evaluating its response log probability. We empirically evaluate our method using large language models for synthetic regression and natural language ICL tasks.

Main paper: 
 introduction: 
 This work presents a method to estimate the hallucination rate for in-context learning (ICL). In ICL, we feed a dataset to a conditional generative model (CGM) and ask it to make a prediction based on that dataset [1, 2] . This practice is useful because it allows us to use pre-trained models to solve problems they may not have been explicitly optimized for. For example, ICL can improve the prediction accuracy of large language models (LLMs) for benchmark tasks in math, translation, and time series prediction [3, 4] . However, understanding the errors-commonly termed hallucinations [5] -a particular ICL application might make is difficult.We develop a new method that takes an ICL problem-a CGM, a dataset, and a prediction questionand estimates the probability that a model will generate a hallucination in response to the prediction question. Figure 1a shows an ICL problem with news snippets classified as World, Sports, Business, or Science [6] . The correct answer to the last snippet is Sports. Figure 1b displays responses generated by Llama-2-7B. Six of the twenty-four generated responses are incorrect, indicating that the model hallucinates at a rate of 25%. Our approach takes a Bayesian perspective of ICL. This perspective assumes that the CGM generates samples from the posterior predictive distribution of an (unknown) Bayesian model that defines a joint distribution over observable data and latent mechanisms [7, 8] . We then define the posterior hallucination rate (PHR) under this model, which conditions on the observed data-in our case, this is the "context" provided. Finally, we show how to estimate the posterior hallucination rate using the predictive distribution of a CGM. The correct response, "Sports," is displayed in green. Wrong answers are in purple.Related work. Hallucination prediction and mitigation is an active area of research . This work is most closely related to a subset of methods based on uncertainty quantification [31] [32] [33] [34] [35] [36] [37] [38] particularly, those methods that aim to predict hallucinations based on uncertainty about the meaning of generated responses [33, 34, 36] . Unlike those methods, the PHR does not require external information from auxiliary classifiers and is applicable beyond language tasks. Our approach is enabled by sampling ICL dataset completions from the predictive distribution, which was first explored in the context of sequential models by Fong et al. [39] . Extensions to various CGMs are an active area of research [40, 41] . Notably, Falck et al. [41] explores the hypothesis that in-context learning (ICL) performs Bayesian inference. They propose a similar method to ours for testing this hypothesis and present several ICL scenarios where it does not hold. However, their tests are stringent, and while ICL does not conform to Bayesian inference under these strict conditions, we demonstrate in our work that adopting a Bayesian perspective for ICL still offers significant practical benefits.Contributions. In Section 2, we introduce the posterior hallucination rate for Bayesian CGMs, prove it is computable by sampling from the predictive distribution, and provide a finite-sample estimator for CGMs that only requires evaluating the log probabilities of responses. In Section 3, we empirically evaluate our methods. First, we study the PHR estimator with synthetic data to demonstrate that it can accurately predict the true hallucination rate. We then study our methods on natural language ICL problems with pre-trained CGMs from the Llama-2 [42] and Gemma-2 [43] model families. We demonstrate that the PHR estimator gives accurate estimates of the empirical error rate. We provide implementations of the method proposed and experiments used in this paper in https://github.com/blei-lab/phr.In the first setting, we use a simple one-dimensional linear regression. We assume f ∼ N (0, I 2 ), and each sequence of (x, y) is such that x ∼ N (0, σ x ) and y ∼ N (f 1 + f 2 x, σ y ), where σ x = 1 and σ y = 0.1. 
the posterior hallucination rate and how to estimate it: 
 In this section, we first review the fundamentals of conditional generative models (CGMs). Next, we define in-context learning (ICL) and discuss the Bayesian perspective. We then provide definitions for hallucinations and hallucination rates. Finally, we demonstrate how to estimate the hallucination rate given an ICL problem and a CGM.Conditional generative models and in-context learning. A conditional generative model (CGM) is a sequential model of the form p θ (t | (t i ) m 1 ) where θ represents the parameters of the model, t represents an element in the domain of the distribution, and m is the number of elements in the sequence. If the support of each conditional distribution is over language tokens and the size of θ is large, these models are known as large language models (LLMs). We focus on LLMs, but our methods apply to many conditional generative models.Implementing CGMs over sequences is often done using neural network architectures called Transformers [44] . The parameters θ are set by performing stochastic maximization of the model likelihood over a training dataset D = {(t j i ) m 1 } n j=1 , where i is the index over elements of a sequence, and j is the index over sequences. The resulting generative model approximates the distribution of token sequences in the training dataset D.An in-context learning (ICL) problem is a tuple (p θ , D n , x) containing a model p θ , a dataset D n , and a query x. The dataset, or "context," is a sequence of n examples, D n = (x i , y i ) n 1 = ((x 1 , y 1 ), . . . , (x n , y n )). For a new query, x, the model is prompted with the query x and context (x i , y i ) n 1 to generate a response according to p θ (y | x, (x i , y i ) n 1 ). ICL and Bayesian inference. One way to understand ICL is through Bayesian statistics [7, 8, 45] . This perspective associates each ICL problem with a latent mechanism f ⋆ defining the data generation process. Given f ⋆ , data (x i , y i ) are drawn independently from the distribution p ICL (x i , y i | f ⋆ ). 2Under this assumption, when considering all ICL problems simultaneously, the joint distribution of all query-response pairs can be expressed as a mixture of latent mechanisms f:EQUATIONIf we further assume that a pre-trained language model p θ approximates the posterior predictive distribution under this distribution p ICL :EQUATIONit follows that doing ICL using an LLM can be seen as a form of implicit Bayesian inference over the latent mechanisms f [8] .Although Equation ( 1) can be justified by the definition of an ICL problem-as in Footnote 2 or by de Finetti-style arguments based on exchangeability [46] -Equation ( 2) must be assumed.In this paper, we show how adopting this approach allows us to construct and estimate the posterior hallucination rate: the probability that a generated response y from p θ to a query x will be in an unlikely region according to the "true" latent mechanism f ⋆ .Before continuing, we provide a brief comment on notation. We use F, X, Y to emphasize when we refer to random variables, and f, x, y otherwise. For clarity, we use p θ when referring to the distribution of the CGM and p without a subscript when referring to the probability of the "true" data-generating process. The latter distribution can refer to p ICL or any other probability distribution that follows the factorization of Equation (1).Using these ideas, we now define hallucinations and the hallucination rate.First, imagine a setting where we observe the mechanism f ⋆ . For a query x, what values of y would we consider hallucinations? A straightforward idea is to characterize a hallucination as a value of y that is unlikely to be generated from p(y | x, f ⋆ ). This choice motivates the following two definitions. Definition 1. We define a (1-ϵ)-likely set of f and x as any set A such thatP(Y ∈ A | f, x) ≥ 1 -ϵ. Definition 2.For fixed (1-ϵ)-likely sets A(f, x), we call a value y a hallucination with respect to x and f, if y / ∈ A(f, x).As an intuitive example, assume that the actual generative model of (x i , y i ) n 1 is a Bayesian linear model with a known standard deviation σ. Specifically, f ∼ N (0, I d ), x i ∼ N (0, I d ), and y i ∼ N (f ⊤ x i , σ 2 ), where f, x i ∈ R d and y i ∈ R. If ϵ = 0.05, we could choose the interval between the 2.5 and 97.5 percentiles of the distribution N (f ⋆⊤ x, σ 2 ) as our (1-ϵ)-likely set and call everything outside of this interval a hallucination.In practice, we do not observe f ⋆ . Rather, we make predictions with p(y | x, D n ). We ask: At what rate are we hallucinating when we make predictions? The answer is the true hallucination rate. Definition 3. We define the true hallucination rate (THR) as the probability of sampling a hallucination given true mechanism f ⋆ and query x:EQUATION)Where A(f ⋆ , x) is an (1 -ϵ)-likely set of f ⋆ and x.This value is higher when the posterior predictive p(y | x, D n ) places high probability in regions unlikely under p(y | x, f ⋆ ), and, conversely, will be lower if the posterior predictive puts a lot of mass on areas likely under p(y | x, f ⋆ ). We expect the value to be higher when we don't have enough examples and lower as the dataset size increases.Of course, we do not observe the true mechanism f ⋆ . But the dataset ("context") D n provides evidence for it, as summarized in the posterior p(f | D n ). With this distribution, we define the posterior hallucination rate, which is the focal point of this work. Definition 4. We define the posterior hallucination rate (PHR) asEQUATIONIn linear regression, the PHR is the probability that y will land outside of the high probability interval of f and x when sampling f ∼ p(f | D n ), the posterior over coefficients, and y ∼ p(y | D n , x), the posterior predictive over responses.Here, we remind the reader that directly calculating p(f | D n ) cannot be done using the CGM. We will address this problem in the next section.As a final detail, we discuss how to construct (1-ϵ)-likely sets. Ideally, we would like to define these sets so that evaluating y ∈ A(f, x) is easy. Otherwise, it would be hard to know if a response y is a hallucination. One way to do so is to define a statistic S that maps each value in Y to a value in R.We then define Q ϵ (f, x) as the ϵ quantile of this statistic under p(y | f, x). Because it is a quantile,P (Y ∈ {y | S(y) ≥ Q ϵ (f, x)} | f, x) ≥ 1 -ϵ. Thus, A(f, x) = {y | S(y) ≥ Q ϵ (f, x)} is an (1-ϵ)-likely set.One convenient choice of S is log p(y | x, f). Thus, moving forward, we letEQUATIONand we replace all statements 1{y / ∈ A(f, x)} with 1{log p(y | f, x) < Q ϵ (f, x)} in the definitions of a hallucination, the true hallucination rate, and the posterior hallucination rate.We want to calculate the posterior hallucination rate in Definition 4 using conditional generative models like LLMs. However, such models only provide an approximation to the predictive distribution p(x, y | D n ) rather than the posterior over mechanisms p(f | D n ) or the response likelihood p(y | x, f).To overcome this, we propose Algorithm 1, which uses a CGM to estimate the PHR.To start, we provide an intuitive explanation of Algorithm 1. A formal justification of how it produces an estimate of Definition 4 will follow. As input, Algorithm 1 receives an ICL problem, a budget for MC samples M and K, and a maximum context length N . First, it samples N -n new query-response pairs according to the predictive distribution p θ (x N , y N , . . . ,x n+1 , y n+1 | D n ) (Alg.1, lines 2-6).Intuitively, we can understand this as prompting the model to imagine future pairs it will receive. Next, it samples a new response y new from p(y | x, D n ) (Alg.2, line 9) and asks: Is this response likely for x given the task implied by D = (x 1 , y 1 , . . . , x N , y N ) (Alg.1, line 10)? If the model is confident about the task it is performing, the pair (x, y new ) will be coherent with (x i , y i ) N n+1 . If the model is uncertain, then y new will not be coherent with (x i , y i ) N n+1 . To determine coherence, we check whether y new is in the tails of log p θ (y i | D, x) (Algorithm 2, Lines 5 and 10). Finally, we average over K samples of y new and M samples of (x i , y i ) N n+1 . The result is an estimate of the PHR. Building on this intuition, we present the theoretical justification for how this algorithm approximates Definition 4. This justification is based on Doob's theorem [47] , which states that, as n → ∞, drawing a value F from p(f) and evaluating h(F) is equivalent to first sampling(X i , Y i ) n 1 and then evaluating E[h(F) | (X i , Y i ) n 1 ]. We state this result below.Algorithm 1 PHR(x, D n , p θ , M, N, K)Require: Query x, context D n = (x i , y i ) n 1 , CGM p θ , number of context samples M , number of THR samples K, max context length N . 1: for i ← 1 to M do 2:// Sample imagined context 3:D ← D n 4: for j ← n + 1 to N do 5: (x j , y j ) ∼ p θ (x, y | D) 6: D ← D ∪ (x j , y j ) 7:// True hallucination rate 8: h ⋆ ϵ,N,i ← THR(x, D n , D, p θ , K) 9: return 1 M M i=1 h ⋆ ϵ,N,iy i ∼ p θ (y | x, D) 5: S ← S ∪ {log p θ (y i | x, D)} 6: Q ← ϵ quantile of Sy i ∼ p θ (y | x, D n ) 10: h ϵ,i ← 1 log p θ (y i | D, x) < Q 11: return 1 K K i=1 h ϵ,i Theorem 1 (Doob's Informal). For F ∈ F, (X i , Y i ) ∈ X × Y, if (F, (X i , Y i ) ∞ 1 ) is distributed such that F ∼ p(f) and X i , Y i ∼ p(x, y | f) then, under general conditions, the posterior mean of h(F) given (X i , Y i ) ∞1 is almost surely equal to h(F), as the number of samples goes to infinity. That is,lim n→∞ E[h(F) | (X i , Y i ) n 1 ] = h(F) a.s.Theorem 1 helps us transform statements about h(f) to statements aboutE[h(F) | (x i , y i ) n 1 ], which only depends on (x i , y i ) n1 . Thus, we can proceed without direct access to p(f | D n ). We now use this result to address the problem of estimating the posterior hallucination rate. The following theorem shows how to compute this rate without direct access to p(f | D n ). We provide its proof in Appendix C. Theorem 2 (PHR via Posterior Predictive). Assume that the conditions of Theorem 1 hold for F and X, Y, then,h ϵ (x) = 1 {log p(y | x, f) < Q ϵ (f, x)} dP(y | x, D n )dP(f | D n ) = 1 lim N →∞ log p(y | x, (x i , y i ) N 1 ) < Q ϵ ((x i , y i ) ∞ 1 , x) dP(y | x, D n )dP((x, y) ∞ n+1 | D n ),whereQ ϵ ((x i , y i ) ∞ 1 , x) is the ϵ-quantile of lim N →∞ log p(Y | x, (x i , y i ) N 1 ) under the limiting distribution lim N →∞ p(Y | x, (x i , y i ) N 1 ).Theorem 2 suggests a natural finite approximation to the PHR where we clip all limits in the expression to a sufficiently large N . Using this approximation, the finite version of the true hallucination rate isEQUATIONwhere Q ϵ ((x i , y i ) N 1 , x) is analogously defined by limiting the number of samples to N . The finite version of the posterior hallucination rate isEQUATIONFinally, we derive an estimator for Equation (8) by replacing p with p θ and by using Monte Carlo to estimate the integrals and quantiles. Algorithm 1 and Algorithm 2 outline this procedure, where the former estimates Equation ( 8) and the latter estimates Equation (7) .The empirical evaluation below will demonstrate the effectiveness of these estimators, but it is important to consider the approximations involved. First, we apply the distributional approximationp θ (y | x, D n ) ≈ p(y | x, D n ).Second, we use Monte Carlo methods to estimate the quantiles and integrals in Algorithms 2 and 3. Third, we employ a truncation approximation by only generating N -n examples. Each of these approximations is a source of estimation error.Lastly, while we present this methodology in the context of estimating the PHR, it is extendable to other measures of uncertainty. In Appendix D, we adapt these results to provide estimators for the mutual information between f and y (epistemic uncertainty) and the posterior average entropy of y (aleatoric uncertainty). These extensions provide a complementary framework for understanding uncertainty in generative models.In the second setting, we use linear regression on a polynomial basis. Specifically, f ∼ N (0, I d+1 ), x ∼ N (0, σ x ), and y ∼ N (f ⊤ ϕ(x), σ y ), where ϕ(x) ∈ R d+1 and ϕ(x) i = x i-1 . We use the same hyperparameters as in the linear regression setting and set d = 3.We then train a CGM using the same setup and architecture described in Section 3.1, with the only difference being that we train on sequences of length 100. For illustrative purposes, the left panel of Figure 15 displays the sequences used to train the model, while the right panel displays sequences generated by our model. We approach this question from the perspective of calibration, as we believe it provides an interpretable metric for understanding the model's predictions and captures an important property that is essential for the PHR to function correctly. Specifically, we ask whether the distribution produced by pθ (yEQUATIONwhere the inner integral is taken with respect to the data distribution. We refer to this metric as the calibration error. If it were true that pθ (y | x, D n ) ≈ p(y | x, D n ), then the inner integral would be zero for every q.We estimate the value of this integral using a Monte Carlo estimate with 200 samples for Q q (x, D n ) and 200 samples for the integral, and then we evaluate over 200 values of q. Moreover, we set (N -n) to 30 to compute pθ . Results. Figure 16 and Figure 17 show the calibration curves according to the procedure described above in Experiment 1. Specifically, the x-axis represents the percentage of points that should lie below a given quantile if the distribution is calibrated, while the y-axis shows the actual number of points below that quantile. The calibration error is the area between the ideal calibration curve (blue) and the observed calibration curve (pink). Each panel in the figures represents the calibration with a different number of observed points in the context. In both figures, we observe that the calibration error is below 4%, independent of context length, which suggests that the distributions are well-calibrated.The results for Expeirment 2 can be found in Figure 18 and Figure 19 . They display the estimated PHR using the transformer model versus the estimate obtained from the analytic posterior. Each point in the graphs corresponds to a single test example; its x-coordinate represents the transformer-based estimate, while its y-coordinate corresponds to the analytic posterior estimate. If the transformer perfectly approximated the posterior and the number of Monte Carlo samples were sufficiently high, each point would lie on the x-y line. Most importantly, they show that the PHR estimate produced by the trained transformer is close to the estimate from the analytic posterior. The plots also indicate, as expected, that the PHR decreases as the context length increases.Discussion. The results from both experiments suggest that the distribution p θ approximates p well, and that Algorithm 1 operates as expected. Specifically, we observe that the resampling mechanism used in Algorithm 1 leads to a calibrated distribution, and that the estimated PHR closely matches its analytic counterpart. This supports the hypothesis that the underestimation observed in Section 3.1 is due to the fact that the THR is not the PHR, and while these quantities are related, they are inherently distinct.Although this certainly does not prove that p θ ≈ p for all ICL tasks, it provides further evidence of the applicability of the algorithm as-in these simple scenarios-we do not observe any significant failure modes, such as distribution drift due to resampling. 
empirical evaluation: 
 To empirically evaluate the accuracy and applicability of the Posterior Hallucination Rate (PHR) estimator, we first examine whether the PHR estimator accurately predicts the True Hallucination Rate (THR). To do so, we design a synthetic regression experiment for which we can calculate the THR. For ICL regression tasks, the PHR is a reliable predictor of the THR and robust to the choice of ϵ parameter value. Moreover, we observe that the accuracy of the PHR estimator is higher for smaller ICL dataset sizes.We then evaluate the PHR estimator on natural language ICL tasks using pre-trained large language models (LLMs). In this setting, calculating the THR is not feasible, so we investigate two alternative questions: (1) does the PHR estimator accurately predict the model hallucination rate (defined below in Section 3.2), and (2) can the PHR accurately predict the empirical error rate? The PHR estimator reliably predicts the model hallucination rate, regardless of model performance on all ICL natural language tasks. Moreover, the estimator remains robust to different ICL dataset sizes and settings of the ϵ parameter. Additionally, the PHR estimator accurately predicts the empirical error rate of generated responses when ϵ is set to values greater than 0.5 and the LLM can perform the task better than a random classifier.We implement a CGM trained on sequences of synthetic regression problems where the THR is available and compare the PHR estimates against the THR on new regression tasks. We implement the model p θ (y | x, D n ) with a setup similar to a Conditional Neural Process [48] and with an architecture close to Llama 2 [42] modified to model sequences of continuous variables. We train it using auto-regressive prediction, the standard for CGMs and LLMs In our experiments, we set ϵ = 0.05 so that a response y is considered a hallucination if it falls outside the 95% confidence interval of a given sampled distribution conditioned on x. To compute the THR, we use the ground truth function f that generated the dataset, replace p(y | x, D n ) with p θ (y | x, D n ), and estimate the integral by sampling 2000 values from this latter distribution. This approach is reasonable, as we ultimately care about the true probability of hallucinations when using the model p θ to make predictions. In the right panel, with a larger n, both the PHR and hallucination probability remain low across all x values. Notably, the PHR and hallucination probability closely align throughout the domain. In Appendix G.1, we demonstrate that these findings hold across various values of the ϵ parameter.In Figure 3a , we present the PHR and THR plotted against various context lengths, where each is averaged over 200 random test functions. For added interpretability, we report several metrics for evaluating the PHR as a linear predictor of the THR, including the mean squared error (MSE), the regression coefficient β, the p-value testing the null hypothesis that β = 0, and the coefficient of determination (R 2 ).Although the PHR aligns well with the THR, it tends to underestimate it, particularly as the number of examples increases. This is also clear given the lower R 2 values as the context length increases. To investigate this further, Figure 3b visualizes individual PHR and THR predictions at different context lengths by plotting PHR values on the x-axis and the corresponding THR values on the y-axis. For small numbers of contextual examples, the PHR closely matches the THR, which is encouraging since accurately capturing the true probability of hallucination is crucial when few examples are available and errors are more likely.Discussion. The results confirm that our method closely recovers the target estimand. However, the PHR estimator underestimates the THR. We do not believe this is due to the Monte Carlo or truncation approximations, as initial experiments with varying M , K, and N values exhibited the same pattern. Rather, we consider two main factors that likely contribute to this underestimation.First, differences between the learned distribution p θ (y | x, D n ) and the true distribution p(y | x, D n ) likely contribute to the bias. This is seen in the slight discrepancies in the confidence intervals in the third panel of Figure 2 . It may also be due to other mismatches in our assumptions, such as the lack of perfect exchangeability in the transformer architecture; even after training on permuted sequences, the architecture may not fully achieve exchangeability.Second, even if the PHR estimate were entirely accurate, it would still differ from THR, as they represent distinct quantities. Therefore, some degree of bias is to be expected. We investigate this question further in Appendix G.1.2 where we compare our estimate with an analytic version of the PHR and find this problem is greatly reduced.We leave it to future work to rigorously quantify these differences and develop methods that can more accurately align the learned and true distributions, particularly in scenarios with large context sizes.Here we evaluate the posterior hallucination rate estimator on common natural language in-context learning tasks using the Llama-2 family of LLMs [42] . We also provide results for Gemma-2 9B in Appendix G.2.1. We evaluate the PHR by comparing it against the empirical error rate and the model hallucination rate (MHR), which we define below. These serve as proxies for the THR, which is no longer computable as we can't access f ⋆ for these examples. Setup. We consider tasks defined by six datasets: Stanford Sentiment Treebank (SST2) [49] , Subjectivity [50] , AG News [6] , Medical QP [51] , RTE [52] , and WNLI [53] . We filter out queries of length longer than 116 tokens. Descriptions of all data sets and pre-processing are given in Appendix F.2.To implement ICL for a given dataset, we sample a response-balanced training set of query/response pairsD n = (x i , y i ) n 1 .We generate a response y from the predictive distribution given by an LLM p θ (y | x, D n ). We structure the prompt by adding strings to distinguish between inputs and labels. An example prompt from the Subjectivity dataset is shown in Appendix E.2.Evaluation metrics. It is a challenge to assess the accuracy of the the posterior hallucination rate on LLM tasks as we only have a one ground truth response for each query instead of the ground truth f ⋆ .One evaluation metric to consider is the empirical error rate,EQUATIONHowever, this metric only provides a partial picture as it does not account for the inherent variability over responses given a mechanism, which would not constitute a hallucination.To get a more complete picture, we invoke Doob's theorem and note that, when(x i , y i ) ∼ p(x, y | f ⋆ )and for sufficiently large N , we haveEQUATIONThis motivates us to estimate the reference THR by approximating the distribution p(y| x, f ⋆ ) with p θ (y | x, D n ∪ D eval ), where D eval is a set of additional (x i , y i ) examples from the same task that are not in D n . We call this estimate the model hallucination rate (MHR) as it is derived under the model p θ . Formally,MHR(x; p θ , D n , D eval ) := 1 K K i=1 1 log p θ y i | x, D n ∪ D eval < Q eval ϵ ,where y i ∼ p θ (y | x, D n ), K is the number of response samples, and Q eval ϵ is the ϵ quantile of the samples log p θ y j | x, D n ∪ D eval with y j ∼ p θ (y | x, D n ∪ D eval ). This value is only proposed for model evaluation. Importantly, it cannot be used to replace the PHR, as it depends on D eval , which is not available at test time. Intuitively, the MHR computes the probability that an answer y, generated by a model conditioned on D n , is considered a hallucination by a model conditioned on both D n and D eval .For each task and context length n ∈ [2, 4, 8, 16, 32] , we sample 50 random training datasets D n , 50 evaluation datasets D eval , and 10 random test samples. We report the same metrics used in the synthetic experiments, but here we evaluate the PHR as a linear predictor of both the error rate and MHR. These metrics are evaluated over all 50 × 10 test samples for each context length.Results. We report results for Llama-2-7b. We set N -n = 5, M = 10, and K = 50. The top plots in Figure 4 show the MHR and estimated posterior hallucination rate against the number of in-context examples with ϵ = 0.05. It shows that the posterior hallucination rate is a good estimator of the MHR. We show that this trend holds for alternative settings of ϵ in Figure 23 of Appendix G.2.2.The bottom plots in Figure 4 show the empirical error rate and estimated posterior hallucination rate against the number of in-context examples with ϵ = 0.75. The plots show that for the SST, Subjective and AG News tasks, the PHR follows the error rate closely, while it diverges for the Medical QP, RTE and WNLI tasks. We use a high value of epsilon so that only responses with very high probability are not taken to be hallucinations, aligning with our strict focus on correctness rather than modeling the full distribution. The impact of varying ϵ is explored in Figure 24 of Appendix G.Discussion. The results highlight the usefulness of the PHR by demonstrating its ability to assess the model's uncertainty in language tasks and predict its error rate without requiring an evaluation dataset. Additionally, the discrepancy between the PHR and the error rate in Medical QP, RTE, and WNLI tasks can be attributed to the model's limited generalization ability in these tasks. This leads to an inadequate approximation of the posterior predictive; in other words,p θ (y | x, D n ) ̸ ≈ p(y | x, D n ).This inadequacy is supported by Figure 4 , which shows that in tasks where the PHR does not align with the error rate (green line), the model's performance is close to random guessing (dashed gray line). Hence the poor performance of the PHR in this scenarios is expected because our method relies on accurately approximating the posterior predictive and only accounts for hallucinations when the data distribution is properly modeled.Using the theorem in the previous section and similar approximations as for the PHR, we construct estimates of the aleatoric and epistemic uncertainty.In particular, we can define a truncated approximation of the aleatoric uncertainty asH θ (Y | F, x, (x i , y i ) n 1 ) := -H θ Y | x, (x i , y i ) N 1 dP θ ((x i , y i ) N n+1 | (x i , y i ) n 1 ),where,EQUATIONand N -n is a practical number of generated examples. In turn this model approximation for the aleatoric uncertainty allows us to define a finite model approximation for the epistemic uncertainty,EQUATIONAnd finally, as in the main body of the paper, we can use Monte-Carlo estimation to construct practical estimates of these quantities. This is summarized below.Total predictive uncertainty.EQUATIONAleatoric uncertainty.EQUATION) Algorithm 3 H θ (Y | F, x, (x i , y i ) n 1 ) Require: Query x, context D n = (x i , y i ) n 1 , CGM p θ , number of context samples M , number of predictive uncertainty samples K, max context length N . 1: for i ← 1 to M do 2:// Sample imagined context 3:D ← D n 4: for j ← n + 1 to N do 5: (x j , y j ) ∼ p θ (x, y | D) 6: D ← D ∪ (x j , y j ) 7://Total predictive uncertainty with D 8:EQUATIONWe note that Algorithm 3 is very similar to Algorithm 1, with the key distinction being the replacement of the THR computation with the calculation of the average entropy of the likelihood. This similarity is not a coincidence, as both can be seen as specific instances of a more general predictive resampling algorithm presented by Fong et al. [39] , albeit in a different context.In Appendix G.2.2 and Appendix G.1.1 we provide some experiments comparing this epistemic uncertainty estimator against the PHR and find that both are correlated and thus represent similar information.F Dataset details  
conclusion: 
 In this work, we have presented a new method for predicting the hallucination rate of in-context learning with conditional generative models. We provide a theoretical justification for our method. In synthetic experiments, we demonstrate that the PHR estimator yields accurate estimates of the actual probability of hallucination. With pre-trained LLMs that achieve non-trivial performance, we show that our method is valuable for predicting the error rate of natural language ICL tasks.High-fidelity estimation of the PHR relies on two strong assumptions. The first is that the data of the in-context learning problem admits a de Finetti representation; the second is that the CGM p θ is a faithful estimate of the in-context learning distribution p ICL . While our results support the adoption of our method and these assumptions, divergences between p θ and p ICL may introduce inaccuracies, as we saw for natural language tasks where Llama-2-7B only performs as well as a random classifier. Falck et al. [41] also report instances where properties of the predictive distribution of a pre-trained LLM differ from those of the reference Bayesian posterior predictive for synthetic ICL tasks. Two directions for further research are to understand how these divergences affect the accuracy of the PHR and to identify the settings under which our assumptions break down.Nevertheless, we are optimistic about future work that adopts Bayesian perspectives on LLMs and other CGMs, as we believe these approaches hold significant practical utility for understanding uncertainty and hallucinations in these models.  
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.[ 
 
 Appendix: Mechanisms and Capabilities of ICL. Several papers argue, both theoretically and through synthetic scenarios, that ICL can implement learning principles such as Bayesian inference and gradient descent [8, [54] [55] [56] [57] [58] [59] . Evidence in actual pre-trained LLMs shows that ICL can be approximated as a kernel regression [60] , and parametric approximations to ICL can be derived from the hidden state of the last input demonstration [61, 62] . Practical shortcomings of ICL include dependence on example order [63] [64] [65] [66] and the impact of prediction preferences acquired during pre-training [65] [66] [67] [68] [69] . While future models might improve ICL performance [70] , current limitations are clear and are thoroughly investigated in recent work by Falck et al. [41] . These results imply that ICL in real LLMs does not implement perfect Bayesian inference but suggest that LLM predictive uncertainty includes both epistemic and aleatoric components, and that LLMs can update uncertainties with new observations. Uncertainties in LLMs. Our results are supported by evidence that predictive uncertainties of large language models are well-calibrated, even in scenarios requiring epistemic uncertainty [32, 71] . Relatedly, LLM uncertainties have been used to detect hallucinations in free-form generation settings such as question-answering [28-30, 32-38, 72] . Although not all these papers explicitly focus on LLM uncertainties, they rely on it implicitly by sampling multiple model completions for a query and quantifying the differences in meaning. Specifically, Kuhn et al. [33] highlight the challenge of isolating uncertainty over semantic meaning from uncertainty over syntax or lexis in free-form generation tasks. While these approaches do not disambiguate aleatoric and epistemic uncertainty, Ahdritz et al. [73] , Johnson et al. [74] recently proposed methods to do so in CGMs. However, Ahdritz et al. [73] require access to two LLMs of different parameter counts, and Johnson et al. [74] do not apply their method to LLMs. Additionally, Hu et al. [75] show that Bayesian experimental design can turn LLMs into strategic question askers, and Jeon et al. [76] present a theoretic study on sources of errors in ICL.Hallucinations in LLMs. In addition to approaches based on uncertainty, a variety of other strategies have been explored to detect or mitigate hallucinations in LLMs: retrieval-augmented generation [9] [10] [11] [12] [13] [14] [15] [16] , custom token sampling procedures [17] [18] [19] , model fine-tuning to improve uncertainties [20] [21] [22] or reduce hallucinations outright [77] , as well as learning to extract or steer truthfulness from hidden states [23] [24] [25] [26] [27] .Neural processes. Neural processes (NPs) [48, [78] [79] [80] are neural network-based non-parametric models trained over a collection of datasets. Similar to ICL, NPs take a collection of datapoints as input and amortize task learning in a single forward pass through the model. For instance, when datasets are drawn from a Gaussian process prior, NPs' predictive distributions closely approximate the true Bayesian posterior predictive for a given input dataset [7] . NPs have been used successfully for tasks requiring reliable uncertainty estimation, such as Bayesian optimization [40, 78, 80] or active feature acquisition [48] . Recently, Lee et al. [40] applied Doob's theorem to quantify uncertainties in neural processes.The work most aligned with ours is [39] , which introduces Martingale Posterior distributions. Their central idea is to focus on the posterior predictive, rather than the posterior itself, as the primary tool for expressing uncertainty. Similar to our approach, samples from the posterior predictive are used to estimate quantities of interest. Through repeated resampling and estimation of the predictive, a "posterior" over the estimate is obtained. In particular, their predictive resampling algorithm can be seen as a generalization of Algorithm 1, although the motivation and use is different. Falck et al. [41] work towards formalizing this methodology for LLMs. They propose a set of statistical tests to estimate whether an LLM satisfies the Martingale property; these tests depend on being able to sample from the true Bayesian model that defines the posterior predictive distribution estimated by the LLM predictive distribution. In their evaluations using synthetic data and pre-trained LLMs, they find that violations of the Martingale property can occur. Moreover, they find that the fidelity of the LLM predictive distribution to the true Bayesian posterior predictive decreases as the length of dataset completions (N -n) increases. They also derive an epistemic uncertainty estimator based on the posterior covariance over mechanisms, which has connections to the posterior hallucination rate and the mutual information estimand we propose in Appendix D.B.1 Broader social impact Positive social impact. In terms of misinformation, if users cannot distinguish between accurate information and hallucinations, they may spread misinformation unknowingly. This can damage the credibility of platforms that use LLMs and erode the trust users place in AI systems. Furthermore, as LLMs are increasingly adopted in high-risk sectors such as medicine and finance, hallucinated medical advice poses serious risks, potentially resulting in harmful health practices or delayed treatment. Similarly, inaccurate financial information could lead to poor investment decisions or significant financial loss.Ethically, hallucinations can reinforce or propagate biases and stereotypes if the generated content reflects societal prejudices. This can perpetuate discrimination and inequality. Understanding when hallucinations are likely to occur is vital for holding developers and companies accountable for the content their models produce. Finally, many researchers use LLMs and other CGMs to generate or label data. Hallucinations in this setting can lead to false discoveries and wasted resources.Being able to accurately predict hallucination rates for given tasks is essential for ensuring that AI systems contribute positively to society. It allows for maintaining trust, safety, ethical standards, and the overall integrity of information dissemination.Negative social impact. When our model is being used as intended but gives incorrect results (i.e. produces a low estimate of probability of hallucination when the true probability is high), it could inadvertently be reinforcing biases present in hallucinations while increasing user trust in the outputs.Building trustworthy and effective ICL solutions requires understanding why and when incorrect or unexpected responses are generated by a CGM. The ICL literature provides two findings that suggest distinct sources of hallucination. The first finding is that both the error rate and the prediction entropy decrease and saturate with an increasing number of examples [66] . This trend is illustrated in Figure 5 using the Llama-2-7b model [42] on a set of natural language ICL tasks. This finding indicates that one source of hallucination is an insufficient number of relevant in-context examples.The second finding from the literature is that there are still many tasks for which ICL performs poorly. For example, the response accuracy of Gemini Pro 1.5 given several in-context examples from the American Mathematics Competition is only 37.2%, implying that the model would hallucinate an incorrect response at a rate of 62.8% [3] . We hypothesize that regardless of the number of relevant in-context examples, the model may lack the capacity to answer a specific user query from a complex or new domain accurately. This hypothesis is illustrated using Llama-2-7B by comparing the graphs of the first three tasks (SST2 [49] , Subjective [50] , and AG News [6] ) to the graph of the WNLI task [53] ) in Figure 5 . While the error rate and response entropy for each of the first three tasks improve significantly over random guessing with more examples, both measures appear to saturate near the random baseline for the WNLI task. The second source of hallucinations is then associated with whether a model has the capacity to factually answer queries for an ICL task.This work focuses on the first source of hallucinations. That is, the posterior hallucination rate is concerned with estimating the rate of hallucinations that stem from a lack of relevant context, and not those that stem from the model not having the capability to solve the task. The predictive distribution encodes response variability coming from several sources, and this variability is closely tied to the ways in which a model can generate a hallucination. Below, we discuss these sources of response variability (or uncertainty) and their relations to hallucinations and the posterior hallucination rate.Aleatoric Uncertainty. The (1 -ϵ)-likely set defined by a given mechanism f * reflects an irreducible component of response uncertainty. To illustrate the concept of irreducible (sometimes called "aleatoric") response uncertainty, imagine a hypothetical and idealized LLM fit to a vast corpora containing many calculus examples so that it is capabable of integration. Consider, Prompt 1: fill in the blanks: the integral of x^2 with respect to x on the interval [-3, 3] is ␣.If an LLM effectively models the mechanism associated with integration, then response variability will only be determined by the different ways the model can generate the correct response: perhaps, 18, eighteen, 3 3 3 --3 3 3 , 9 + 9, 2 * 9, or XVIII, depending on the training data. Uncertainty reflecting the plurality of ways to communicate the same meaning is commonly referred to as syntactic uncertainty [33] , which is considered irreducible in this particular example.In language tasks, irreducible uncertainty does not need to be syntactic. Consider the same LLM and Prompt 2: fill in the blanks: the integral of ␣, with respect to x on the interval [-3, 3] is ␣.In addition to the syntactically different ways to specify a particular function (x 2 , x squared, x * x, . . . ) and the corresponding result (18, eighteen, XVIII, . . . ), response variety also depends on the semantically different integrands that could fill the first blank (x 3 , cos (x), exp (x), 2xy, . . . ) and the semantically different possible results. This additional variety is an instance of semantic uncertainty [33] . While high semantic uncertainty can be indicative of hallucinations, it is not a problem in this example because the imputation of any sensible function and answer could still be valid under the mechanism associated with integration datasets. That is, given Prompt 2, uncertainty over semantically different functions is expected and even desirable. This pair of examples illustrate an important insight; irreducible uncertainty is mechanism-relative. For example, it may be appropriate to equivocate irreducible and syntactic uncertainty in a simple question answering setting where the mechanism defines a (1 -ϵ)-likely set over correct responses. However, in the second example we show that this decomposition is not always appropriate. In general, we consider aleatoric uncertainty to be associated with response variability induced under a specific mechanism f. That is, the variability of responses under the likelihood p(y | x, f). Special epistemic uncertainty. Now we provide examples to illustrate reducible uncertainty about the mechanism f. Returning to Prompt 1, imagine that the user desires a response in terms of a reduced fraction. There are two obvious ways that the user could augment Prompt 1 to reduce the uncertainty over mechanisms yielding integer, word, fraction, etc. responses. (1) the user could simply replace "fill in the blanks" with "fill in the blank with a reduced fraction." (2) the user could take an ICL approach and augment the prompt with a number of examples: The first choice may result in reducing all uncertainty about which mechanism to sample responses according to. For the second choice, we can imagine a progressive reduction in uncertainty about the mechanism as more examples are added in-context. For example, if we were to see the two provided examples, we may still be uncertain about whether to respond with a number or a fraction, or whether to respond with any correct fraction or the reduced fraction. For example, given the context, a response of 54 3 would be as plausible as the desired 18 1 . It may not be until the prompt included an example like, "the integral of x 3 with respect to x on the interval [2, 4] is 60/1," until all uncertainty about the mechanism is resolved. We suggest that this may explain the observed reduction in error rate and response entropy in a task like WNLI, where Llama-2-7B does not generalize effectively. That is, as we provide more in-context examples, the predictive distribution becomes more aligned with the set of acceptable responses, even if those responses may not be correct.The preceding example illustrated a hallucination as a misaligned response; now, let us turn to an example of a non-factual response. Returning to Prompt 1, imagine that the model generates the response 42. Why did it do this? A plausible answer could be that the model cannot do integration. We will touch on this possibility next, but first let's consider an equally interesting case. We know from few-shot and chain-of-thought prompting literature [14, 30, 81 ] that augmenting the context can have significant effects on ICL accuracy. For example, consider the hypothetical setting where the LLM has "grokked" algebra, but only has the superficial capacity to output a number when completing definite integrals. Or perhaps the model has the capacity for integration, but the format of the examples and query is uncommon in the training corpora. In this case, the LLM may actually be capable of generating correct responses given some clever prompting. For example, Again, as the prompt contains more examples that translate the form of the query into a suitable format, we can expect that the uncertainty about the answer will reduce. For conditional models in general, we will call this Special epistemic uncertainty, which could also be understood as in-context epistemic uncertainty. Both of these examples illustrate hallucinations that are due to insufficient context, which have been called in-context hallucinations by Weng [82] . In this paper we have focused on the case where increasing the number of in-context examples can resolve this uncertainty. We leave it to future work to understand more sophisticated prompt augmentation. General epistemic uncertainty. Let's return to Prompt 1, but this time imagine an LLM fit to a corpus not containing any examples from calculus or related mathematical fields. Or perhaps the LLM had finite capacity and is not capable of generating accurate answers to integrals. Then, what do we do when the model generates "Dua Lipa" to an integration question? What do we do when the model outputs members of the set {17, 137, Dua Lipa, Wednesday, . . . }? We say that the response should have high General epistemic uncertainty because the LLM has not acquired the capacity to model the mechanism class, F, corresponding to integrals. In the example of Prompt 4, imagine if there were no number of exemplars or no prompt augmentations that could induce a correct response. When a condition generative model p θ is a good estimator of the true distribution p, the posterior hallucination rate-and mutual information quantity we propose in Appendix D-are designed to estimate special epistemic uncertainty. We leave the important work of estimating the posterior hallucination rate when the CGM is not a good estimator (under general epistemic uncertainty) and accounting for extrinsic hallucinations [82] to future work. Initial work toward this is presented by Jesson et al. [83] .We begin by stating a lemma that will be useful throughout. Lemma C.1. Assume that the conditions of Theorem 1 hold for F and (X, Y), then for a fixed dataset and query D n , x and under a probability model whereF ∼ p(f | D n ) and (X i , Y i ) ∼ p(x, y | F), then almost surely log p(Y | x, F) = lim n→∞ log p (Y | x, (X, Y) n 1 ) .Proof. For a fixed y and x, let g(f) = p(y | x, F) and apply Doob's theorem on the probability model. Then it is the case thatEQUATION)where Equation ( 13) holds because x is independent of F. Taking logs at both sides and using the continuity of the logarithm, we obtain log g(F) = limn→∞ log p(y | x, (X i , Y i ) n 1 )and because this holds for all y, it must hold for the random variable Y.Theorem 3 (PHR via Posterior Predictive). Assume that the conditions of Theorem 1 hold for F and X, Y, then,h ϵ (x) = 1 {log p(y | x, f) < Q ϵ (f, x)} dP(y | x, D n )dP(f | D n ) = 1 lim N →∞ log p(y | x, (x i , y i ) N 1 ) < Q ϵ ((x i , y i ) ∞ 1 , x) dP(y | x, D n )dP((x, y) ∞ n+1 | D n ),whereQ ϵ ((x i , y i ) ∞ 1 , x) is the ϵ-quantile of lim N →∞ log p(Y | x, (x i , y i ) N 1 ) under the limiting distribution lim N →∞ p(Y | x, (x i , y i ) N 1 ).Proof. Define an alternative probability model such thatF ∼ p(f | D n ) and (X i , Y i ) ∼ p(x, y | F).Let p a , E a , Q a,ϵ and P a denote the relevant quantities computed with respect to this alternative probability model.First, note that by expanding the definition of Q a,ϵ (f, x) under this new probability modelEQUATION)where we used the fact that Y ⊥ (X, Y) ∞ 1 | F, x in Equation (16) . For simplicity, we will use p(•|(X i , Y i ) ∞ 1 ) to denote lim N →∞ p(• | X i , Y i ) N 1with similar conventions from other quantities. Now, applying Doob's to g(fEQUATION= limEQUATIONEQUATIONwhere we used Doob's on Equation ( 19) and the tower property in Equation (20) . Plugging this back in Equation ( 17) we obtainEQUATIONTo complete the proof, note thatEQUATION)where we changed the probability spaces from p to p a . This is justified becausep a (y | x, f) = p(y | x, f, D n ) = p(y | x, f)where we used the independence of Y on D n once f is known, the fact that p a (f) = p(f | D n ) by definition, and the fact that for the quantile functionEQUATION) due again to the independence of Y on the dataset D n once f is known. Finally we have (abusing notation using D ∞ n+1 = (x, y) ∞ n+1 to refer to (x, y) ∞ n+1 in the original probability space but also(x i , y i ) ∞1 in the alternative probability space as they have the same distribution):EQUATION=EQUATIONWhere Equation ( 28) is justified by because (x, y) ∞ n+1 doesn't appear in the term inside, Equation ( 29) is justified by the arguments above and Lemma C.1, and Equation ( 31) is a result of marginalizing f out. The last thing to point out is that p a (y | x, (x, y)∞ n+1 ) = p(y | x, (x i , y i ) ∞ 1 ) and Q a,ϵ ((x, y) ∞ n+1 , x) = Q ϵ ((x i , y i ) ∞ 1 , x) because P a log p a (Y | x, (x, y) ∞ n+1 ) ≤ q | x, (x, y) ∞ n+1 = P (log p(Y | x, (x i , y i ) ∞ 1 ) ≤ q | x, (x i , y i ) ∞ 1 )Using this fact in Equation ( 31) yields the theorem.In the main paper, we focus on developing the posterior hallucination rate; however, there are other quantities that can also be used to predict model performance. In Bayesian machine learning, a commonly used quantity for this purpose is the posterior mutual information between F and Y, which is denoted by I(Y; F | x, D n ). This is often interpreted as quantifying epistemic uncertainty. As is the case for other quantities presented in the paper, it is not possible to compute it directly in a CGM. Nevertheless, in this section we extend the results of the paper and demonstrate that by using a similar methodology to the one presented, it is possible to construct estimators for it.There are several ways to decompose total uncertainty about an answer into epistemic and aleatoric components. One such decomposition proceeds by defining uncertainty via entropy and then using a clever decomposition of mutual information to link various interpretable quantities. We use this decomposition throughout.To elaborate, we define total predictive uncertainty as the entropy of the posterior predictive, denoted as H(Y | x, D n ). We define aleatoric uncertainty-which represents the irreducible portion of uncertainty-as the expected entropy of the likelihood, denoted asE p(f|Dn) [H(Y | x, f)].And finally, we define the difference between these two quantities as the epistemic uncertainty, which represents the reducible component of the uncertainty.By defining epistemic uncertainty in this way, it turns out to be equivalent to I(Y; F | x, D n ), as the following holds,I(Y; F | x, D n ) = H(Y | x, D n ) -E p(f|Dn) [H(Y | x, f)] = -log p(y | x, D n )dP(y | x, D n ) + log p(y | x, f)dP(y | x, f)dP(f | D n ).In this equation, we see that the total predictive uncertainty depends only on the predictive distribution, p(y | x, D n ), which-under our assumptions-is the estimand of a CGM, p θ (y | x, (x i , y i ) n 1 ). The aleatoric uncertainty, on the other hand, depends both on the likelihood function, p(y | x, f), and the posterior over mechanisms, p(f | x, (x i , y i ) n 1 ), which are only latently modeled by a CGM. In the next section, we show how the aleatoric uncertainty can be expressed using only samples from p(x, y | (x i , y i ) n 1 ), which enables its estimation with a CGM and, in turn, provides a practical estimator for the epistemic uncertainty of the model.We use a similar proof as in Appendix C. Theorem 4. Assume that the conditions of Theorem 1 hold for F and (X, Y). Then for a new independent x,E p(f|Dn) [H(Y | x, f)] = E p(D ∞ n+1 |Dn) H(Y | x, D ∞ ) .Proof.E p(f|x,Dn) [H(Y | x, f)] = H(Y | x, f, D n )dP(f | x, D n ) = - log p(y | x, f, D n )dP(y | x, f, D n )dP(f | x, D n ) = - log p(y | x, f)dP(y | x, f)dP(f | x, D n ) = - log p(y | x, f)dP(y | x, f)dP(f, D ∞ n+1 | x, D n ) = - log p(y | x, D ∞ )dP(y | x, f)dP(f, D ∞ n+1 | x, D n ) By Lemma C.1 = - log p(y | x, D ∞ )dP(y | x, f, D ∞ n+1 )dP(f, D ∞ n+1 | x, D n ) By conditional independence = - log p(y | x, D ∞ )dP(y, f, D ∞ n+1 | x, D n ) = - log p(y | x, D ∞ )dP(y, D ∞ n+1 | x, D n ) By marginalizing f = - log p(y | x, D ∞ )dP(y | x, D ∞ )dP(D ∞ n+1 | x, D n ) = - log p(y | x, D ∞ )dP(y | x, D ∞ )dP(D ∞ n+1 | D n ) By independence of x = H(Y | x, D ∞ )dP(D ∞ n+1 | D n ) = E p(D ∞ n+1 |Dn) H(Y | x, D ∞ ) D.For our experiments in 3.2, we randomly sample context examples and test input/label pairs from the following datasets:Stanford Sentiment Treebank (SST2) SST2 [49] is a corpus with fully labeled parse trees that allows for a complete analysis of the compositional effects of sentiment in language. The corpus consists of 11,855 single sentences extracted from movie reviews. It was parsed with the Stanford parser and includes a total of 215,154 unique phrases from those parse trees, each annotated by 3 human judges. Sentiments are classified as binary labels "positive" or "negative".The Subjectivity dataset [50] contains 5,000 movie review snippets from www.rottentomatoes.com labeled "subjective", and 5,000 sentences from plot summaries available from www.imdb.com labeled "objective". Selected sentences or snippets are at least ten words long and are drawn from movies released post-2001.The AG News dataset [6] contains 496,835 categorized news articles from more than 2,000 news sources. The 4 largest classes (World, Sports, Business, Sci/Tech) were chosen from this corpus to construct our dataset, including only the title and description fields.Medical Questions Pairs (MQP) The MQP dataset [51] consists of 3,048 similar and dissimilar medical question pairs hand-generated and labeled by doctors based on patient-asked questions randomly sampled from HealthTap. Each question results in one positive question pair ("similar") that looks very different by superficial metrics, and a negative question pair ("different") that conversely look very similar, so as to ensure that the task is not trivial.Recognizing Textual Entailment (RTE) The RTE dataset [52] comes from a series of annual textual entailment challenges. Examples are constructed based on news and Wikipedia text, and labeled as binary classifications based on whether or not there is entailment.Winograd Schema Challenge (WNLI) The WNLI dataset [53] consists of 1,100 sentence pairs with ambiguous pronouns with different possible referents. The task is to determine whether the sentence with a substituted pronoun is entailed by the original sentence. as ε, used to calculate the PHR. Notably, we see that setting ε = 0.1 results in the PHR being a more accurate predictor of the THR as context length grows, as opposed to ε = 0.05 (where there is no misspecification). This reflects our observation that the PHR underestimates the THR, particularly for longer context lengths.Figure 12 : Synthetic data: Effect of misspecified ϵ. The true value is ϵ = 0.05 and is used to calculate the THR. We vary the value used in calculating the posterior hallucination rate, which we denote as ε here.Mutual information. We also assess the mutual information (MI) estimator for epistemic uncertainty, derived in Appendix D and defined in Equation (37) , to see how well it predicts the true hallucination rate (THR). Figure 13 shows that the MI estimates are significantly correlated with the THR, which indicates that the MI can also be an effective predictor of hallucinations. Figure 14 allows us to look deeper into the relationships between the THR, PHR, and MI. In comparing Figures 14a and 14b , we see that the PHR has a more linear relationship to the THR than the MI, which has a more sigmoidal relationship to the THR. Nevertheless, Figure 14c which plots the PHR against the MI shows that both are strongly correlated. Their correlation provides evidence that the PHR and MI encode related information, and that the PHR is a measure of epistemic uncertainty, which is expected from its definition. Our method relies on the assumption thatEQUATIONis sufficiently accurate for our procedure to work. As discussed in Section 3.1, if this assumption is incorrect, our method may produce inaccurate results. However, in the experiments presented in the main body of the paper, we did not have an analytic posterior available to compare against, nor a way to compute ground-truth PHRs.In this section, we investigate whether Equation (38) holds in a simple setting where we have an analytic posterior for comparison.Setup. We use two simple synthetic datasets where it is possible to analytically compute the posterior p(y | x, D n ).Here we evaluate the posterior hallucination rate estimator on the same in-context learning tasks used in the main body of the paper using Gemma-2 9B [43] . Figure 20 shows the error rate (green) and response entropy (blue) of Gemma-2-9b. In general, both metrics decrease and saturate with longer context lengths. For all tasks, the model performs at least slightly better than random, and is particularly performant on SST2, Subjectivity, AG News and RTE. However, for Medical QP and WNLI, error rates are very close to random, indicating poor generalization. Results. We report results for Gemma-2-9b. We set N -n = 5, M = 10, and K = 50. Besides this discrepancy, the results for Gemma corroborate those found for Llama-2 in the main paper.Here we extend the natural language ICL task results of the main paper using the Llama-2 family of LLMs [42] .Results. We report results for Llama-2-7b with N -n = 5, M = 10, and K = 50. Figure 23 in Appendix G shows that the posterior hallucination rate serves as a reliable estimator of the MHR across different ϵ settings. We further ablate the ϵ parameter against the empirical error rate and report results for each natural language ICL task in Figure 24 of Appendix G.In Table 1 , we report the results of an ablation study on SST2 that varies the number of generated examples, MC samples, y samples, and model parameters. We see that increasing the number of MC and y samples improves the R 2 scores for both the MHR and empirical error rate. In contrast, increasing the number of generated examples or model size alone results in a performance decline for this task.In Figure 25 , we compare the mutual information (MI) estimator with the error rate, MHR, and PHR across all tasks. Specifically, Figure 25a illustrates the relationship between MI and error rate, while Figure 25b illustrates the relationship between MI and MHR. Additionally, Figure 25c depicts the relationship between MI and PHR. The high R 2 values observed between the MI and PHR provide further evidence that both estimators capture the same underlying information. We see that there are significant correlations to either metric across tasks. The high R 2 between the mutual information and posterior hallucination rate is evidence that they quantify similar information.For our experiments, we used an internal cluster made up of A100s and RTX 8000s, each with 40 to 48 GB of memory. Training the models for the synthetic experiments took approximately one day on a single machine. For the natural language experiments in the main paper, we ran 50 seeds per dataset, with each seed requiring between 20 minutes and 4 hours. Additionally, we conducted ablations over different values of M and N (as detailed in Appendix G). We also performed further experiments and developed models that required additional computational resources but were not included in the paper.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.The existence of representations with such f variables is not a strong assumption. When there is a finite number of possible (x, y) pairs, as in LLMs, we can construct f representations by writing down a large vector where each element corresponds to a specific (x, y) pair, and fx,y represents the probability P(X = x, Y = y), as described in[45]. In other words, f is simply the probability distribution. If the support of the data is finite and (x, y) pairs are conditionally independent given the task, we can then assume the existence of the latent mechanisms f by identifying them directly with P(X = x, Y = y) for each task.This process is similar to the "prior fitted network" implementation of Müller et al.[7], but we require the conditional distribution of both queries x and responses y, where their implementation only models responses.In fact, with the true posterior we should have p(y | x, Dn) = p(y | x, Dn), where p(y | x, Dn) is drawn with the same sampling procedure as pθ (y | x, Dn). 


Decision: The paper proposes a method to estimate the hallucination rate for in-context learning (ICL) from a Bayesian perspective. It takes a conditional generative model (CGM), a dataset, and a prediction question, and estimates the probability of hallucination. The Bayesian interpretation assume the CGM samples from a posterior predictive distribution over a Bayesian model of a latent parameter and data. They define Posterior Hallucination Rate (PHR), along with True Hallucination Rate (THR) and Model Hallucination Rate (MHR), and provide theoretical proofs and empirical demonstrations to show that they can accurately estimate the true probability of hallucination. Numerical experiments verify the practicality of the interpretation.

Overall, the work is addressing a significant and timely problem, presents a technically sound solution, and introduces new metrics that would potentially help the whole community more reliably evaluate LMs from here on. 

Reviewers raised concerns about the limitation of the evaluations of their method, namely, the six datasets used are of relatively narrow scope, and more "synthetic" than real, which makes its applicability to real-world cases unclear. Two of the reviewers raised their scores during rebuttal. There's one extremely negative reviewer whose behavior is questionable, and for that reason I'd like to downweight their opinion.

Judging from the overall contribution, I tend to recommend accept.

Reviews: review: paper_summary: paper_summary: The paper proposes a method to estimate the hallucination rate for in-context learning (ICL) in a conditional generative model (CGM) from a Bayesian perspective. The authors assume the CGM samples from a posterior predictive distribution over a Bayesian model of a latent parameter and data. They define Posterior Hallucination Rate (PHR), along with True Hallucination Rate (THR) and Model Hallucination Rate (MHR), and provide theoretical proofs and empirical demonstrations to show that they can accurately estimate the true probability of hallucination.
strength_weakness: strength_weakness: W1. While the authors tested their methods on six datasets from various domains, the range of label content and types may be limited. This limitation could introduce confounding factors. For instance, tasks where one label can be a substring of another (e.g. “similar” vs “dissimilar”) might not fully represent the complexity of real-world scenarios.

W2. The labels used in the datasets, such as “entailment” vs “not”, do not have a clear semantic meaning and relation like “positive” vs “negative” OR “subjective” vs “objective.” This absence of semantic relation might influence the ICL performance. Testing whether the semantic meaning of the labels affect performance and hallucination rate would strengthen the findings. A suggested approach is to use neutral labels such as letters (A,B,etc.) to determine if the lack of semantic meaning of the labels impacts the hallucination detection.

W3. Related to W1, the datasets used in the experiments have either 2 or at most 4 categories. This limited variety raises questions about how the method performs with more complex classification tasks. For example, how would the method fare in English-to-French word translation, where the number of possible categories is significantly higher? Expanding the experiments to include tasks with a larger number of categories would provide a more comprehensive assessment of the method’s robustness.
questions: questions: Q1. What constitutes an “in-capability” task and “out-of-capability” task?

Q2. In Figure 10, why do the lines, especially for context length 50 graphs, stop earlier? What does that imply about the performance or behavior of the model at longer context lengths?

Q3. The paper references Fong et al. [22] proposing Martingale Posterior distribution that uses posterior predictive. Are there any baseline results for this approximation that could be compared to PHR to evaluate which distribution better defined models?
limitations: limitations: The authors have addressed the positive and negative societal impacts of their work. It would be beneficial to further discuss and address the dataset choices and potential limitations, such as label diversity and number of categories, to provide a comprehensive view of the conditions under which the method has been tested. This would enhance the understanding of the method’s applicability and generalizability ensuring a more thorough assessment of its limitations.
paper_summary: paper_summary: The paper focuses on the in-context learning setting of generative AIs, such as large language models, and proposes a new definition for hallucination. It introduces a novel metric, PHR, along with a corresponding estimation method. Unlike traditional metrics, the proposed metric accounts for label ambiguity resulting from unclear task specifications.
strength_weakness: strength_weakness: - The notations are messy and confusing.
- - There is misuse and abuse of uppercase random variables and lowercase instantiations. For example, the distribution of  $F$  is referred to as  $p(f)$  in lines 87 and 121, while it is referred to as  $p(F)$  in line 124. This problem is exacerbated in Theorem 1, making it hard to read.
- - The notations PHR/THR can refer to both the definitions and the algorithms, which adds to the confusion. For instance, in the sentence on line 174, “we evaluate the PHR in a setting where we know the true mechanism f* so that we can compare it directly against the true hallucination rate (THR),” PHR appears to refer to the algorithm, while THR seems to refer to Definition 3.
- The experimental setup is difficult to understand.
- - For example, the experiments in Section 3.1 are supposed to evaluate the proposed method in a setting where $f*$  is known. However, what $f*$  is and how THR is calculated are not explained. To understand this, at least the way $p(y|x, f*)$  is set should be specified. Given the unused space at the end of the paper, there is no reason for the authors not to elaborate on the experimental settings.
- - In Section 3.2, the authors attempt to justify the use of PHR by introducing another proposed metric called MHR. It is unclear how MHR could serve as a gold metric. Even if MHR is a gold metric, why not just use MHR?
- Typos: Line 31 "the" should be deleted.
questions: questions: Please refer to the weaknesses.
limitations: limitations: Limitations have been well discussed.
paper_summary: paper_summary: This paper presents a method for predicting the hallucination rate of in-context learning with conditional generative models.
strength_weakness: strength_weakness: - Unclear how many queries would be require to validate the approach, as this will be very dependent of task, context, and LLM - this is clearly a missing element of the evaluation
- Evaluation on Llama 2 only - more is definitely require to validate the approach
- No identification of where the hallucination might come from
- Title is misleading - this is basically a hallucination identification method
- Non cost effective as this require multiple LLMs calls
questions: questions: - Why not more LLMs used for experimentation?
- What could be the right number of LLM call needed?
- What parameters is the function of hallucinate rate? What are the optimal parameter? for which task? for which LLMs?
limitations: limitations: cf. weakness and questions
paper_summary: paper_summary: This paper presents a Bayesian interpretation of in-context learning. This interpretation enables us to calculate the hallucination rate. In other words, by considering in-context examples as observations, the posterior distribution can be computed and the hallucination rate is derived. Numerical experiments verify the practicality of the interpretation.
strength_weakness: strength_weakness: In summary, I suggests that the authors clarify the focus of this paper either on the hallucination rate for NLP tasks or on the error rate for general in-context learning tasks.

1. As discussed in L222, the proposed theory cannot be applied as is in NLP tasks and an approximated metric is required. Although the introduction of this paper focused on NLP tasks, the usefulness of the proposed theory in this area is limited, as reported in L265 with Figure 6.

2. Hallucination is a complex phenomenon with a wide range of causes [1]. As mentioned in the above weakness, the effectiveness of this theory is limited to the synthetic regression task. In light of these, I am concerned whether it is appropriate to call the phenomenon examined in this paper a hallucination rate rather than an error rate.

3. The claim of Section 2.2 is to use the cumulative probability of the token generation probability distribution as the confidence for token y. However, the use of cumulative probability as the confidence of token y is a common technique in NLP, for example in nucleus sampling [2]. To demonstrate the importance of the theory presented in this section, I consider it necessary to show its universality when other statistics S are used. Otherwise, I also suggest moving this section to the appendix as noted in the following weakness.

4. L146 says `Algorithm 1 can be understood intuitively without appealing to any Bayesian arguments’. I agree with this statement, and I consider that the necessity of the presented theory depends largely on the discussion in Appendix B.1. Because the appendix is just supplementary material and cannot play a role in justifying the main claim, I suggest the authors should revise the structure of the paper.

[1] Huang+, A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, arXiv:2311.05232.

[2] Holtzman+, The Curious Case of Neural Text Degeneration, ICLR2020.
questions: questions: See Weaknesses.
limitations: limitations: This paper discussed broader social impact.

Comments: comment: Thank you for the additional time you have taken to consider our rebuttal. We are pleased to have addressed your initial concerns regarding the empirical evaluation, limitations, and clarity. We are encouraged by and sincerely appreciate your decision to raise your score.
Thank you for your response. The authors have adequately addressed my initial concerns, and as a result, I have raised my score.
Dear Reviewer, thank you again for the time and effort you've dedicated to reviewing our work. Your insights and feedback have significantly contributed to improving the quality of our paper. 

We believe our responses address the concerns raised in your reviews and are committed to making the necessary revisions to clarify any uncertainties. As the discussion period is nearing its conclusion, if you find that any aspects of our responses require further clarification or discussion, we are eager to engage in constructive dialogue.
Dear Reviewer, thank you again for the time and effort you've dedicated to reviewing our work. Your insights and feedback have significantly contributed to improving the quality of our paper. 

We believe our responses address the concerns raised in your reviews and are committed to making the necessary revisions to clarify any uncertainties. As the discussion period is nearing its conclusion, if you find that any aspects of our responses require further clarification or discussion, we are eager to engage in constructive dialogue.
Thank you for taking the additional time to consider our rebuttal. As fellow reviewers, we empathize with your trepidation as we have also witnessed promised changes fail to manifest in camera-ready versions. The above clarifications are straightforward: minor changes in notational consistency/disambiguation and the inclusion of details as we have described. Ultimately, we can only ask for your trust that we will make these changes and respect your decision.

Regarding your concern about evaluation, we ask you to consider the additional language model experiments we have run in response to reviewers VGJP and 6dN5. For reviewer VGJP, in a more realistic setting where labels can take on multiple semantically equivalent values, we show that the PHR maintains its performance in predicting the error rate and MHR in [Figure Error](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/vgjp/w1_figure_error.png) and [Figure MHR](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/vgjp/w1_figure_mhr.png). And for reviewer 6dN5, we show that our methods also work for the Gemma-2 9B model, where this [figure](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/6dn5/figure1.png) indicates that the PHR can accurately predict the empirical error rate for Gemma-2 9B, and this [figure](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/6dn5/figure2.png) shows that the PHR accurately predicts MHR for Gemma-2 9B.
We sincerely appreciate the additional time you have taken to consider our rebuttal and your decision to raise your score. 

Regarding W2, we ask you to consider further the language model experiments we have run in response to reviewers VGJP and 6dN5. For reviewer VGJP, in a more realistic setting where labels can take on multiple semantically equivalent values, we show that the PHR maintains its performance in predicting the error rate and MHR in [Figure Error](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/vgjp/w1_figure_error.png) and [Figure MHR](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/vgjp/w1_figure_mhr.png). And for reviewer 6dN5, we show that our methods also work for the Gemma-2 9B model, where this [figure](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/6dn5/figure1.png) indicates that the PHR can accurately predict the empirical error rate for Gemma-2 9B, and this [figure](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/6dn5/figure2.png) shows that the PHR accurately predicts MHR for Gemma-2 9B.

Regarding W4, we will add a discussion summarizing the justification we present in the appendix in the main paper to improve the paper structure.
I appreciate the authors’ efforts in their rebuttal and their commitment to improving the presentation. However, since I cannot preview the modifications, I am unable to assess the extent of the enhancements. Additionally, I agree with the critique from other reviewers regarding the inadequate evaluation of the proposed method in non-synthetic scenarios. Although the proposed method's applicability to real-world cases remains unclear, I find the idea promising and interesting to me. Therefore, I would like to keep my scores unchanged.
Thank you for offering such a thorough response to my concerns. The misunderstandings about MHR have been clarified. I also agree that PHR can accurately track the error rate in the first three in Figure 6. For these reasons, I raised my score.

However, I did not raise the score further higher mainly because of W2 and W4. Regarding W2, the paper is largely devoted to experiments and discussions in the synthetic task and those in NLP are insufficient to explore the hallucination problem. Regarding W4, the justification of the contribution is described in the appendices.