Estimating the Hallucination Rate of Generative AI
Authors: Andrew Jesson
Nicolas Beltran-Velez
Quentin Chu
Sweta Karlekar
Jannik Kossen
Yarin Gal
John Patrick Cunningham
David Blei
Venue: NeurIPS 2024 poster (2024)
Keywords: Uncertainty Quantification
Large Language Models
Conditional Generative Models
Hallucination Prediction
Primary area: generative_models
TL;DR: A method for estimating the hallucination rate of generative AI for in-context learning

Abstract: This paper presents a method for estimating the hallucination rate for in-context learning (ICL) with generative AI. In ICL, a conditional generative model (CGM) is prompted with a dataset and a prediction question and asked to generate a response. One interpretation of ICL assumes that the CGM computes the posterior predictive of an unknown Bayesian model, which implicitly defines a joint distribution over observable datasets and latent mechanisms. This joint distribution factorizes into two components: the model prior over mechanisms and the model likelihood of datasets given a mechanism. With this perspective, we define a \textit{hallucination} as a generated response to the prediction question with low model likelihood given the mechanism. We develop a new method that takes an ICL problem and estimates the probability that a CGM will generate a hallucination. Our method only requires generating prediction questions and responses from the CGM and evaluating its response log probability. We empirically evaluate our method using large language models for synthetic regression and natural language ICL tasks.

Estimating the Hallucination Rate of Generative AI
Andrew Jesson"
Nicolas Beltran-Veler'
Quentin Chu
Sweta Karlekar*
Jannik Kossen
Yarin Gal
John P. Cunningham
David Bleit
Abstract
This paper presents method for estimating the hallucination rate for in-context learning (ICL) with generative AI. In ICL, a conditional generative model (CGM) is prompted with a dataset and a prediction question and asked to generate response. One interpretation of ICI. assumes that the CGM computes the posterior predictive of an unknown Bayesian model, which implicitly defines joint distribution over observable datasets and latent mechanisms. This joint distribution factorizes into components: the model prior over mechanisms and the model likelihood of datasets given mechanism. With this perspective, we define a hallucination as a generated response the prediction question with low model likelihood given the mechanism. We develop new method that takes an ICL problem and estimates the probability that CGM will generate hallucination. Our method only requires generating prediction questions and responses from the CGM and evaluating its response log probability. We empirically evaluate our method using large language models for synthetic regression and natural language ICI. tasks.
Introduction
This work presents method to estimate the hallucination rate for in-context learning (ICL), In ICL, we feed dataset conditional generative model (CGM) and ask it make prediction based on that dataset [1, 2]. This practice is useful because it allows us to use pre-trained models to solve problems they may have been explicitly optimized for. For example, ICL can improve the prediction accuracy of large language models (LLMs) for benchmark tasks in math, translation, and time series prediction [3, 4]. However, understanding the errors commsoly termed hallucinations [5]-a particular ICI. application might make is difficult.
We develop new method that takes ICL problem-a CGM, a dataset, and a prediction question- and estimates the probability that model will generate a hallucination in response to the prediction question. Figure la shows an ICL problem with news snippets classified as World, Sports, Business, or Science [6]. The correct answer to the last snippet is Sports. Figure 1 displays responses generated by Llama-2- --B.. Six of the twenty-four generated responses are incorrect, indicating that the model hallucinates at rate of 25%.
Our approach takes Bayesian perspective of ICI. This perspective assumes that the CGM generates samples from the posterior predictive distribution of an (unknown) Bayesian model that defines a joint distribution over observable data and latent mechanisms [7, 8). We then define the posterior hallucination rate (PHR) under this model, which conditions on the observed data-in our case, this the "context" provided. Finally, we show how estimate the posterior hallucination rate using the predictive distribution of a CGM.
"Denotes equal contribution. Correspondence to [adj2147, nh2838) @columbia.edu. Department Statis- tics, Columbia University. Department of Computer Science, Columbia University. & OATML, Department of Computer Science, University of Oxford.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input:
achedule TODAY'S GANES Division GREATER BOSTON --Arlington Malden...
Label:
Sports
Iaput: Putting Nature on the Pil1 Wildlife nanagers are looking to contraception as a way...
Label: Science
Iaput: Error Juigment The FBI alleges that veteran U.S. diplonat met with agents from...
Label: World
Iaput: Seara and Knart: to Marge After mach speculation, two diacount glants nove to create...
Label: Bueinese
Input: Defeat for GB canoeiets British canceiste lick Smith and Stuart Bovnan are out of...
Label:
(a) An in-context learning dataset.
Sports, 0lympics, Other, Sports, Sporta, Sporte, Sporte, Sports, Sports, Sporte, Sports, Athletice, Sporte, Entertainnent, Sporte, Sports, Sports, Sports, Olympice, Sports, Sports, Olympics, Sports, Sports
(b) Generated responses
An example of an in-context dataset and generated response examples for the last label. The correct response, "Sports," is displayed green. Wrong answers are purple.
Figure I:
Related work. Hallucination prediction and mitigation an active area of research [9-30]. This work is most closely related subset of methods based uncertainty quantification [31-38]- particularly, those methods that aim to predict hallucinations based on uncertainty about the meaning of generated responses [33, 34, 36). Unlike those methods, the PHR does not require external information from auxiliary classifiers and is applicable beyond language tasks. Our approach is enabled by sampling ICL dataset completions from the predictive distribution, which was first explored in the context of sequential models by Fong et al. [39]. Extensions to various CGMs are an active area of research [40, 41]. Notably, Falck al. [41] explores the hypothesis that in-context learning (ICL) performs Bayesian inference. They propose a similar method to ours for testing this hypothesis and present several ICL scenarios where it does not hold. However, their tests are stringent, and while ICL. does not conform to Bayesian inference under these strict conditions, we demonstrate in our work that adopting a Bayesian perspective for ICL still offers significant practical benefits.
Contributions. In Section 2, we introduce the posterior hallucination rate for Bayesian CGMs, prove it is computable by sampling from the predictive distribution, and provide finite-sample estimator for CGMs that only requires evaluating the log probabilities of responses. In Section 3, we empirically evaluate our methods. First, we study the PHR estimator with synthetic data to demonstrate that it can accurately predict the true hallucination rate. We then study our methods on natural language ICL problems with pre- pre-trained CGMs from the Llama-2 [42] and Gemma-2 [43] model families. We demonstrate that the PHR estimator gives accurate estimates of the empirical error rate. We provide implementations of the method proposed and experiments used in this paper in http::/github.coo/bblilaaaat
2 The posterior hallucination rate and how to estimate it
In this section, we first review the fundamentals of conditional generative models (CGMs). Next, we define in-context learning (ICL) and discuss the Bayesian perspective. We then provide definitions for hallucinations and hallucination rates. Finally, we demonstrate how to estimate the hallucination rate given ICL problem and CGM.
Conditional generative models and in-context learning. conditional generative model (CGM) is sequential model of the form po (1,))T) where 0 represents the parameters of the model, I represents an element in the domain the distribution, and m is the number of elements in the sequence. If the support of each conditional distribution is over language tokens and the size of 0 is large, these models are known as large language models (LLMss. We focus on LLMs, but our methods apply to many conditional generative models.
Implementing CGMs over sequences is often done using neural network architectures called Trans- formers [44]. The parameters O are set by performing stochastic maximization of the model likelihood over a training dataset D {(v)) I } Y where is the index over elements of a sequence, and the index over sequences. The resulting generative model approximates the distribution of token sequences in the training dataset D.

---

An in-context learning (ICL) problem is tuple (pe. D,. containing model Pe. dataset Dns and query x. The dataset, or "context," sequence of n examples, D, Vi)i (1),, (Xn; Yn)). For a new query, the model is prompted with the query and context (X4, to generate a response according to py x, (x)))
ICL and Bayesian inference. One way to understand ICL is through Bayesian statistics [7. 45]. This perspective associates each ICL problem with latent mechanism f* defining the data generation process. Given f*, data Yi) are drawn independently from the distribution PICL (x; y; f+
Under this assumption, when considering all ICL problems simultaneously. the joint distribution of all query-respone pairs can be expressed as a mixture of latent mechanisms f:
/ II
PICT. (ICI((D)) (Dn)
PiCt. (x4.s, Nz
2=1
If we further assume that pre-trained language model po approximates the posterior predictive distribution under this distribution PICL:
po(Nx+1 Xn+1, Dn) z PICL (yn+1 Xn+1, Dn)
(2)
f
(V xx+1, f) dPicL Dn),
(3)
it follows that doing ICL using an LLM can be seen as a form of implicit Bayesian inference over the latent mechanisms f [8].
Although Equation (1) can be justified by the definition of an ICL problem as in Footnote 2 or by de Finetti-style arguments based on exchangeability [46]-Equation (2) must be assumed.
In this paper, we show how adopting this approach allows us construct and estimate the posterior hallucination rate: the probability that a generated response from po to a query x will be in an unlikely region according the "true" latent mechanism [*
Before continuing, we provide a brief comment on notation. We use F,X, Y to emphasize when refer to random variables, and f, otherwise. For clarity, use pe when referring to the distribution of the CGM and p without a subscript when referring to the probability of the "true" data-generating process. The latter distribution can refer PICL or any other probability distribution that follows the factorization of Equation (1).
2.1 Hallucinations and the posterior hallucination rate
Using these ideas, we now define hallucinations and the hallucination rate.
First, imagine setting where observe the mechanism f*. For query x, what values of would we consider hallucinations? A straightforward idea is to characterize hallucination as a value of y that is unlikely to be generated from p(y x, ft). This choice motivates the following two definitions.
Definition 1. We define (1-c)-likely set of f and x as any set A such that P(Y A f 2 1 c.
Definition 2. For fixed (I-e) sets A(1, x), we call value y a hallucination with respect to x and f, i & x(f,x).
As an intuitive example, assume that the actual generative model of ve)) is Bayesian linear model with a known standard deviation a. Specifically, I ~ (0, Ta), NO, Ia), and y; ~ (f x,, o²), where f, E IRd and E IR. If e = 0.05, we could choose the interval between the 2.5 and 97.5 percentiles of the distribution A x, o2) as our (1-e)-likely set and call everything outside of this interval hallucination.
In practice we do not observe [* Rather, we make predictions with p(y x, D,.) We ask: At what rate are we hallucinating when we make predictions? The answer is the true hallucination rate.
The existence of representations with such variables is not a strong assumption. When there is a finite number of possible (x, y) pairs, as in LLMs, can construct representations by writing down large vector where each element corresponds to specific (x, y) pair, and fx.y represents the probability P(X x, Y y). as described in [45]. In other words, is simply the probability distribution. If the support of the data is finite and (x, y) pairs are conditionally independent given the task, we can then assume the existence of the latent mechanisms f by identifying them directly with P(X- x, Y y) for each task.
3

---

Definition 3. We define the true hallucination rate (THR) as the probability of sampling a hallucina- tion given true mechanism f* and query x:
" = / 1 & x(("x)) dPy DD,
(4)
Where A(f*, x) is an (1 (1-63 e)-ikeeyy set of and x.
This value is higher when the posterior predictive p(y x, Dn) places high probability in regions unlikely under p(y x, f*), and. conversely, will be lower if the posterior predictive puts a lot of mass on areas likely under p(y x. We expect the value to be higher when we don't 't have enough examples and lower as the dataset size increases.
Of course, we do not observe the true mechanism f*_ But the dataset ("context") Dn, provides evidence for it, as summarized in the posterior p(f D.)). With this distribution, define the posterior hallucination rate, which the focal point of this work.
Definition 4. We define the posterior hallucination rate (PHR) as
// 1 & x)) dP x, dP D,).
(x) = E [h, (F,x) D..]
(5)
In linear regression, the PHR is the probability that y will land outside of the high probability interval of and when sampling p(f DD,), the posterior over coefficients, and y p(y x), the posterior predictive over responses.
Here, remind the reader that directly calculating p(f) (D,) cannot be done using the CGM. We will address this problem in the next section.
As final detail, we discuss how to construct (1-e)-likely sets. Ideally, we would like to define these sets so that evaluating y A(f, x) is easy. Otherwise, it would be hard to know if a response y is a hallucination. One way to do so is to define a statistic S that maps each value in y to a value in RR We then define Qe x) as the e quantile of this statistic under p(y f,x). Because it is a quantile,
P(Y E (y S(y) 2 Qe x)} ff) 2 --c.
Thus, A(f,x) = I S(y) 2 (f, is an (l-e)-likely set.
One convenient choice of S is log ply x, f). Thus, moving forward, we let
A(f, {y logp(y x, 2 (f,
(6)
and we replace all statements [ & Af,, with 1 [logp(y f.x) < (f, in the definitions of hallucination, the true hallucination rate, and the posterior hallucination rate.
2.2 Calculating the posterior hallucination rate from predictive distributions
We want to calculate the posterior hallucination rate in Definition 4 using conditional generative models like LLMs. However, such models only provide an approximation to the predictive distribution p(x,y Dx)) rather than the posterior over mechanisms y( (D,) or the response likelihood p(y x, f). To overcome this, we propose Algorithm 1, which uses CGM to estimate the PHR.
To start, we provide an intuitive explanation of Algorithm 1. A formal justification of how produces an estimate of Definition 4 will follow. As input, Algorithm receives an ICL problem, a budget for MC samples M and K, and maximum context length N. First, samples N new query-respontt pairs according to the predictive distribution po (x yN.... Xx+1, Vn+1 D..) (Alg. 1, lines 2-6). Intuitively, we can understand this as prompting the model to imagine future pairs it will receive. Next, it samples new response Ynew from ply x, D,) (Alg.2, line 9) and asks: Is this response likely for given the task implied by D (x1; (xI:Y1,..., NN, (Alg. line 10)? If the model is confident about the task it is performing, the pair (x, Ynew)) will be coherent with (x,, yz) If the model is uncertain, then Ynew will not be coherent with (x;, (:) To determine coherence, check whether Ynew is in the tails of log (o D. x) (Algorithm 2, Lines 5 and 10). Finally, we average over K samples of ynew and M samples of (x, v+) The result is estimate of the PHR.
(Algorithm
Building on this intuition, we present the theoretical justification for how this algorithm approximates Definition 4. This justification is based on Doob's 's theorem [47], which states that, as -> oo, drawing value F from p(f) and evaluating ((FF) is equivalent to first sampling (X,, Y.) and then evaluating (F) (X,, Y. We state this result below.

---

Algorithm (x, Da, po. N, K)
Algorithm THR (x, Dn, D, po, K)
Require: Query x, context Dn (x,.ss).. CGM pø. number context samples M, number of THR samples K, max context length N.
Require: Query x, extended context D, original
context Dn,, CGM pe number of samples
K. //Estimate quantiles
2: S 6- 0
1:
for i 1 to M do
for 1 to K do
2:
// Sample imagined context
4:
( x, D)
3:
'D <- Dn
S {log po x, D)}
4:
for n
N do
6: Q < qqantille S
5:
(x), (x D)
// Frequency
hallucinations
6:
D ++ DU (x,,,))))
for i 1 to K do
7:
1/ True hallucination rate
9: Yi po x, Dr)
8:
N,, THR(x. D,, Po.
10: he, - logpoly; D,x) Q}
9:
return 1 D h N,
M
11: return
Σ
k -i=1 Ei=1be hes
Theorem 1 (Doob's Informal). F (X), X Xx
)) is distributed such that ~ p(f) and X,, Y p(x.y f) then, under general conditions, the posterior mean of h(F) given (X,, To is almost surely equal to h(F), the number of samples goes infinity. That is,
lim (F) (Xs, YO h(F) a.s.
Theorem helps us transform statements about I(() to statements about [B(((F) (xi, N:) which only depends on (xx, y.)f. Thus,
proceed without direct access to p(f Dx).
We now use this result address the problem of estimating the posterior hallucination rate. The following theorem shows how to compute this rate without direct access to p(f D,)) We provide its proof in Appendix
Theorem 2 (PHR via Posterior Predictive). Assume that the conditions of Theorem I hold for F and X, Y, then,
hx(x)= = // (logply |x,f) < (f( Dr)dP((I D..)
= -Iʃ i (1lm
lim logply x, (x) ((x,, y.) x) dP(y x, Dn)dd u) Dn);
where Qe((x, y:)f x) is the e-quantile of lim logp(Y x, (x).))) under the limiting distribution limn P(Y x, ).))".
Theorem 2 suggests natural finite approximation the PHR where we clip all limits the expression to sufficiently large N. Using this approximation, the finite version of the true hallucination rate is
f (logply | (x), < (x(( d)) x, Dn),
ha (Xx,, v
(7)
where Qe ((,, y.) x) is analogously defined by limiting the number of samples to N. The finite version of the posterior ballucination rate
(x) - (2x, x)) dP((x.)))) Dz))
(8)
Finally, derive estimator for Equation (8) by replacing with pe and by using Monte Carlo to estimate the integrals and quantiles. Algorithm and Algorithm outline this procedure, where the former estimates Equation (8) and the latter estimates Equation (7).
The empirical evaluation below will demonstrate the effectiveness of these estimators, but it important to consider the approximations involved. First, apply the distributional approximation
5

---

po(y x, Dn) R p(y x. Dx). Second, we use Monte Carlo methods to estimate the quantiles and integrals in Algorithms and 3. Third. we employ a truncation approximation by only generating N examples. Each of these approximations is source of estimation error.
Lastly, while we present this methodology in the context of estimating the PHR, it is extendable to other measures of uncertainty. In Appendix D, we adapt these results to provide estimators for the mutual information between f and y (epistemic uncertainty) and the posterior average entropy of y (aleatoric uncertainty). These extensions provide a complementary framework for understanding uncertainty in generative models.
3 Empirical evaluation
To empirically evaluate the accuracy and applicability of the Posterior Hallucination Rate (PHR) estimator, we first examine whether the PHR estimator accurately predicts the True Hallucination Rate (THR). To do so, we design a synthetic regression experiment for which we can calculate the THR. For ICL regression tasks, the PHR is a reliable predictor of the THR and robust to the choice of e parameter value, Moreover, we observe that the accuracy of the PHR estimator is higher for smaller ICL dataset sizes.
We then evaluate the PHR estimator on natural language ICL tasks using pre-trained large language models (LLMs). In this setting, calculating the THR is not feasible, so we investigate two alternative questions: (1) does the PHR estimator accurately predict the model hallucination rate (defined below in Section 3.2), and (2) can the PHR accurately predict the empirical error rate? The PHR estimator reliably predicts the model hallucination rate, regardless of model performance on all ICL natural language tasks. Moreover, the estimator remains robust to different ICL dataset sizes and settings of the parameter. Additionally, the PHR estimator accurately predicts the empirical error rate of generated responses when is set to values greater than 0.5 and the LLM can perform the task better than random classifier.
3.1 Synthetic regression tasks
We implement a CGM trained on sequences of synthetic regression problems where the THR is available and compare the PHR estimates against the THR on new regression tasks.
100
100
-34
Clof
Hullchaticn
THR
Figure 2: In the first and third panes, we see the neural process's generated outcomes for n 2 and n 100. The blue region is the true (l-e)-likely set, while the purple is the likely set when conditioned on the blue data points. The second and fourth panes are the corresponding measures of the PHR and THR across the domain.
Setup. We assume the true distribution of query-response pairs p(D,,) is generated by the following factorization: f IT (J p(y xi, I) (x.)) dP, where p(f) is the distribution of randomly drawn MLPs with He initialization, p(xx) is the uniform distribution over [-2, 2). and p(y x, follows a normal distribution A (f(()), 0.1).
We implement the model (y x, D.,) with setup similar to Conditional Neural Process [48] and with an architecture close to Llama 2 [42] modified to model sequences of continuous variables. We train it using auto-regressive prediction, the standard for CGMs and LLMs. Training and test data are generated over non-overlapping sets of generated sequences. Example training datasets are shown in Figure 9a in Appendix F.1. Example datasets generated by the fit model when initialized with single random query x are shown in Figure 9b in Appendix F. 1. The full model and dataset details given in Appendices E and F.I, respectively.

---

In our experiments, we set = 0.05 so that a response y is considered a hallucination if it falls outside the 95% confidence interval of a given sampled distribution conditioned on x. To compute the THR, we use the ground truth function f' that generated the dataset, replace p(y x, Dn)) with ( x, P,), and estimate the integral by sampling 2000 values from this latter distribution. This approach is reasonable, as we ultimately care about the true probability of hallucinations when using the model pe to make predictions.
Context Length: 2
Context Length:
Context Length:
Thue Halludiaaton Rate
Posterior Halludinaton Rate
True Hallsa2 Mollucteeett Rate
8: 6.990
34
1-3al: 0,386
PSE: 0,006
R": 6, 964
B 942
# 0 995
1 123
vod. 0.000
col 1 11000
and 0.000
0.002 R7:0,306
Ccamples 85
130
05
Posteror Halluchaatio
Postercrr Halluciation Rato
Posteror Halucination Rate
(a) PHR and THR vs. (b) Calibration: THR PHR against different numbers of context examples
Figure 3: Synthetic data: (a) Plot of the average THR and PHR as function of n. The THR and PHR follow each other and decrease as the number of contextual examples increases. (b) Plot of the estimated PHR vs the THR. Each point represents the predicted PHR and actual THR for a given instance. Perfect performance would have all points on the x = y. Results show the PHR closely approximates THR, but performance degrades with context length.
Results. The first and third panels of Figure 2 show the model's generated outcomes for n and n 100, respectively. In these plots, the blue region represents the true (1-c)-likely set for the response distribution of specific random ReL.U neural network. The purple region depicts the model's (1-e)-likely set when conditioned on the blue data points and a query value x in the domain [-2, 2]. As more context examples are provided, the confidence intervals narrow, and responses y are increasingly likely to fall within the blue region.
The second and fourth panels of Figure 2 illustrate the THR and the PHR for x in the domain [-2, 2] for two settings of n. We set N - to 100, M to 40, and K to 2000. On the left, dips in PHR and hallucination probability at x 0 and x 1 correspond with the ground truth in-context examples. In the right panel, with larger both the PHR and hallucination probability remain low across all x values. Notably, the PHR and hallucination probability closely align throughout the domain. In Appendix G.1, we demonstrate that these findings hold across various values of the parameter.
In Figure 3a, we present the PHR and THR plotted against various context lengths, where each is averaged over 200 random test functions. For added interpretability. we report several metrics for evaluating the PHR as a linear predictor of the THR, including the mean squared error (MSE), the regression coefficient B, the p-value testing the null hypothesis that 3 = O, and the coefficient of determination (R²).
Although the PHR aligns well with the THR, it tends to underestimate it, particularly as the number of examples increases. This is also clear given the lower R values the context length increases. To investigate this further, Figure 3b visualizes individual PHR and THR predictions at different context lengths by plotting PHR values the x-axis and the corresponding THR values on the y-axis. For small numbers of contextual examples, the PHR closely matches the THR, which is encouraging since accurately capturing the true probability of hallucination is crucial when few examples available and errors are more likely.
Discussion.
The results confirm that method closely recovers the target estimand. However, the PHR estimator underestimates the THR. We do not believe this is due to the Monte Carlo or truncation approximations, as initial experiments with varying M, K, and N values exhibited the same pattern. Rather, we consider two main factors that likely contribute to this underestimation.
First, differences between the learned distribution Po(y x, Dn) and the true distribution p(y x, D,)) likely contribute to the bias. This is seen in the slight discrepancies in the confidence intervals the third panel of Figure 2. It may also be due to other mismatches in our assumptions, such as the lack of perfect exchangeability the transformer architecture; even after training on permuted sequences, the architecture may not fully achieve exchangeability.
7

---

Second, even if the PHR estimate were entirely accurate, it would still differ from THR, as they represent distinct quantities. Therefore, some degree of bias is to be expected. We investigate this question further in Appendix G.1.2 where we compare our estimate with an analytic version of the PHR and find this problem is greatly reduced.
We leave it to future work to rigorously quantity these differences and develop methods that can more accurately align the learned and true distributions, particularly in scenarios with large context sizes.
3.2 Natural language tasks
Here we evaluate the posterior hallucination rate estimator on common natural language in-context learning tasks using the Llama-2 family of LLMs [42]. We also provide results for Gemma-2 9B in Appendix G.2.1. We evaluate the PHR by comparing it against the empirical error rate and the model hallucination rate (MHR), which we define below. These serve as proxies for the THR, which is no longer computable as we can't tt access f* for these examples.
Subgectiity
News
Modical
1:20
L
coamppls
coancios
ceampio
champles
(a) Model Hallucination Rate (MIR)) and Posterior Hallucination Rate (PHR) with 0.05 function of the number of in-context examples. PHR closely follows MHR for tasks.
Sabgectivtty
Nows
Medical
VNL
8.00
21 A.90
81 DM
31 A.M
0.550
3.21L
(b) Error Rate and PHR with 0.75. PHR matches the error rate for SST2, Subjective, and AG News. However, for more difficult tasks like Medical QP and entailment tasks (RTE and WNLI), PHR less accurate. highlighting the limitations of Ll7ma-2-7b.
Figure 4: Performance Metries for Llama-2-7b Across Tasks.
Setup. We consider tasks defined by six datasets: Stanford Sentiment Treebank (SST2) [49]. Subjec- tivity [50], AG News [6], Medical QP [51], RTE [52], and WNLJ [53]. We filter out queries of length longer than 116 tokens. Descriptions of all data sets and pre-processing are given in Appendix F.2.
To implement ICL for a given dataset, we sample a response- balanced training set of query/response pairs D,, = (x,, We generate response y from the predictive distribution given by an LLM Po(y x, Da). We structure the prompt by adding strings to distinguish between inputs and labels. An example prompt from the Subjectivity dataset is shown in Appendix E.2.
Evaluation metrics. It is a challenge to assess the accuracy of the the posterior hallucination rate on LLM tasks as we only have a one ground truth response for each query instead of the ground truth f*
One evaluation metric to consider is the empirical error rate,
K É(x,y:po: D.,) = Σ Hys Fy). yer - x, Da).
(9)
However, this metric only provides a partial picture as it does not account for the inherent variability over responses given a mechanism, which would not constitute a hallucination.
To get more complete picture, we invoke Doob's 's theorem and note that, when (x4,y:) ~ p(x,y (ft) and for sufficiently large N, we have
ply x, xxf") ply x, (x.) ply x, (24,))))) poly 1x, (xxY))))
(10)
8

---

This motivates us to estimate the reference THR by approximating the distribution ply x, f*) with Po (y x, Dn, D evaa)), where Deval a set of additional (x,,1) examples from the same task that are not in Dn, We call this estimate the model hallucination rate (MHR) as it is derived under the model po- Formally,
MHR(x: x: pe. Dns Deval)
1 K
Σ
{ log py: x, Dr Deval) Qeval }.
where Py Dn). K is the number of response samples, and Qeval is the c quantile of the samples (log Po x, Dr UDvaa)) with y; ~ po | x, Dn UDovat). This value is only proposed for model evaluation. Importantly, cannot be used replace the PHR, as it depends on Deval. which is not available at test time. Intuitively, the MHR computes the probability that an answer generated by a model conditioned on 2r,, considered a hallucination by model conditioned on both D,, and Deval.
For each task and context length [2, 4, 16, 32), we sample 50 random training datasets Da,, 50 evaluation datasets Deval, and 10 random test samples. We report the same metrics used in the synthetic experiments, but here evaluate the PHR as linear predictor of both the error rate and MHR. These metrics are evaluated over all 50 X 10 test samples for each context length.
Results. We report results for Llama-2-77. We set N n 5, M = 10, and K 50. The top plots in Figure show the MHR and estimated posterior hallucination rate against the number of in-context examples with e 0.05. It shows that the posterior hallucination rate is a good estimator of the MHR. We show that this trend holds for alternative settings of E in Figure 23 of Appendix G.2.2.
The bottom plots in Figure 4 show the empirical error rate and estimated posterior hallucination rate against the number of in-context examples with 0.75. The plots show that for the SST, Subjective and AG News tasks, the PHR follows the error rate closely, while it diverges for the Medical QP, RTE and WNLI tasks. We use high value of epsilon so that only responses with very high probability are not taken to be hallucinations, aligning with our strict focus on correctness rather than modeling the full distribution. The impact of varying is explored in Figure 24 of Appendix G.
The results highlight the usefulness of the PHR by demonstrating its ability to assess the model's uncertainty in language tasks and predict its error rate without requiring an evaluation dataset. Additionally, the discrepancy between the PHR and the eгror rate in Medical QP, RTE, and WNLI tasks can be attributed to the model's limited generalization ability in these tasks. This leads to an inadequate approximation of the posterior predictive; in other words, Po (y D,.) & p(y x, D,)). This inadequacy supported by Figure 4, which shows that in tasks where the PHR does not align with the error rate (green line), the model's 's performance is close to random guessing (dashed gray line). Hence the poor performance of the PHR in this scenarios is expected because our method relies on accurately approximating the posterior predictive and only accounts for hallucinations when the data distribution properly modeled.
Discussion.
4 Conclusion
In this work, we have presented new method for predicting the hallucination rate of in-context learning with conditional generative models. We provide a theoretical justification for our method. In synthetic experiments, we demonstrate that the PHR estimator yields accurate estimates of the actual probability of hallucination. With pre-trained LLMs that achieve non-trivial performance, we show that our method is valuable for predicting the error rate of natural language ICL tasks.
High-fidelity estimation of the PHR relies on two strong assumptions. The first is that the data of the in-context learning problem admits a de Finetti representation; the second is that the CGM pe is faithful estimate of the in-context learning distribution PICL. While our results support the adoption of our method and these assumptions, divergences between pe and PICL may introduce inaccuracies. as we saw for natural language tasks where Llama-2-78 only performs as well as a random classifier. Falck et al. [41] also report instances where properties of the predictive distribution of a pre-trained LLM differ from those of the reference Bayesian posterior predictive for synthetic ICL tasks. Two directions for further research are to understand how these divergences affect the accuracy of the PHR and to identify the settings under which our assumptions break down.

---

Nevertheless, we are optimistic about future work that adopts Bayesian perspectives on LLMs and other CGMs, as we believe these approaches hold significant practical utility for understanding uncertainty and hallucinations in these models.
5 Acknowledgements
This work is supported by the funds provided by the National Science Foundation and by DoD OUSD (R&E) under Cooperative Agreement PHY-2229929 (The NSF AI Institute for Artificial and Natural Intelligence). The authors would like to thank Amir Feder, Alessandro Grande, Achille Nazaret, Yookoon Park, Kathy Perez, Sebastian Salazar, Claudia Shi, Brian Trippe, Al Tucker, and Luhuan Wu for their reviews, feedback, and support.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 33:1877-1901, 2020.
[2] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. mmXiv:2301.002.. 2022.
[3] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions tokens of context. arXiv:2403.05530 2024.
[4] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv:2403.078155 2024.
[5] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality abstractive summarization. In ACL, pages 1906-1919, Online, July 2020.
[6] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. NewrIPS, 2015.
[7] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers do Bayesian inference. In ICLR, 2021.
[8] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit Bayesian inference. In ICLR, 2021.
[9] Philip Feldman, James Foulds, and Shimei Pan. Trapping LLM hallucinations using tagged context prompts. arXiv:2306..6600 2023.
[10] Shuo Zhang, Liangming Pan, Junzhou Zhao, and William Yang Wang. Mitigating language model hallucination with interactive question- drestion-knowledee alignment. arXiv:2305.13660 2023.
[11] Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv:2302. 2302:23302./2813. 2023.
[12] Nouha Dziri, Andrea Madotto, Osmar R Zaiane, and Avishek Joey Bose. Neural path hunter: Reducing hallucination in dialogue systems via path grounding. In EMNLP, pages 2197-2214, 2021.
[13] Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. RARR: Researching and revising what language models say, using language models. In ACL, pages 16477-16508, 2023.
[14] Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing- Chain-of-knowlldge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. In ICLR, 2023.
10

---

[15] Neeraj Varshney, Wenlin Yao. Hongming Zhang, Jianshu Chen, and Dong Yu. A stitch in time saves nine: Detecting and mitigating hallucinations of LLMs by validating low-confidence generation. AXXi:2307700388/ 2023.
[16] Dan Su, Xiaoguang Li, Jindi Zhang. Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Read before generate! Faithful long form question answering with machine reading. In ACL, pages 744-756, 2022.
[17] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In ICLR, 2024.
[18] Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, and Bryan Catanzaro. Factuality enhanced language models for open-ended text generation. NeurlPS, 35:34586 34599, 2022.
[19] Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 783-791, 2024.
[20] Sabrina Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau, Reducing conversational agents' overconfidence through linguistic calibration. TACL, 10:857-872, 2022.
[21] Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. TMLR, 2023.
[22] Neil Band, Xuechen Li, Tengyu Ma, and Tatsunori Hashimoto. Linguistic calibration of language models. arXiv:2404. AXXv:2444400447,, 2024.
[23] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2023.
[24] Amos Azaria and Tom Mitchell. The internal state of LLM knows when it's lying. In EMNLP, pages 967-976, 2023.
[25] Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In ICLR, 2022.
[26] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from language model. NeurlPS, 36, 2024.
[27] Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering Llama 2 via contrastive activation addition. arXiv:2312.0668/ 2023.
[28] Junyu Luo, Cao Xiao, and Fenglong Ma. Zero-resource hallucination prevention for large language models. CORR, 2023.
[29] Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. Self-contradictory hallucina- tions of large language models: Evaluation, detection and mitigation. In ICLR, 2024.
[30] Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason E Weston.
reduces hallucination in large language models. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models, 2024.
[31] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with BERT. In ICLR, 2019.
[32] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan. Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv:2207.0522 2022.
[33] Lorenz Kuhn, Yarin Gal, and Sehastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In ICLR, 2023.

---

[34] Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. TMLR, 2024.
[35] Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In ACL, pages 5186-5200, 2024.
[36] Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xueying Zhang, Yu Wang, and Shizhu Liu. Halo: Estimation and reduction of hallucinations in open-source weak large language models. arXiv:2308. 11700..1774. 2023.
[37] Potsawee Manakul, Adian Liusie, and Mark JF Gales. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In EMNLP, 2023.
[38] Jeremy R Cole, Michael JQ Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. Selectively answering ambiguous questions. EMNLP,, 2023.
[39] Edwin Fong, Chris Holmes, and Stephen G Walker. Martingale posterior distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology. 85(5)11357-1311, 2023.
[40] Hyungi Lee, Eunggu Yun, Giung Nam, Fong, and Juho Lee. Martingale posterior neural processes. In ICLR. ICLR, 2023.
[41] Fabian Falck, Ziyu Wang, and Christopher C Holmes. Is in-context learning in large language models Bayesian? A martingale perspective. In ICML, 2024.
[42] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09988 2023.
[43] Gemma Team. Gemma, 2024.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
[45] Naimeng Ye, Hanming Yang, Andrew Siah, and Hongseok Namkoong. Pre-training and in-context learning Bayesian inference de Finetti. arXiv:2408. 03300, 2024.
[46] Edwin Shields Hewitt and Leonard J. Savage. Symmetric measures on cartesian products. Transactions of the American Mathematical Society, 80:470-501, 1955.
[47] Joseph Doob. Application of the theory of martingales. Le calcul des probabilites et ses applications, pages 23-27, 1949.
[48] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Tch, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In ICML, pages 1704-1713, 2018
[49] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In EMNLP, pages 1631- 2013.
[50] Sida Wang and Christopher D. Manning. Baselines and bigrams: simple, good sentiment and topic classification. In ACI, page 90-94, 2012.
[51] Clara H McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain. Effective transfer learning for identifying similar questions: Matching user questions to covid- 19 faqs. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3458-3465, 2020.
[52] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Joaquin Siaihonero-Candela Ido Dagan, Bernardo Magnini, and Florence d' Alché Buc, editors, Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177-190, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-33428-.
12

---

[53] Hector J. Levesque, Ernsst Davis, and Leora Morgenstemn The winograd schema challenge. In Gerhard Brewka, Thomas Eiter, and Sheila A. Mcllraith, editors, KR. AAAI Press, 2012. ISBN 978-1-57735-560-1.
[54] Ferenc Huszár. Implicit Bayesian inference in large language models, 2023, Online: accessed 10-July-2023.
[55] Michael Hahn and Navin Goyal. A theory of emergent in-context learning as implicit structure induction. arXiv:2303.0797,, 2023.
[56] Hui Jiang. A latent space theory for emergent abilities in large language models. arXiv:2304. 09960, 2023.
[57] Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? Bayesian model averaging, parameterization, and generalization. arXiv:2305. 19420, 2023.
[58] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. ICML, 2023.
[59] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? Investigations with linear models. In ICLR, 2023.
[60] Chi Han, Ziqi Wang, Han Zhao, and Heng Ji. In-context learning of large language models explained as kernel regression. arXiv:2305. 12766, 2023.
[61] Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In EMNLP, 2023.
[62] Eric Todd, Millicent I. Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In ICLR, 2024.
[63] Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLlO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 2022.
[64] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACI., 2022.
[65] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.
[66] Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relationships but is not conventional learning. In ICLR, 2024.
[67] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv:2303.083: 03346 2023.
[68] Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. EMNLP, 2022.
[69] Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning "learns" in-context: Disentangling task recognition and task learning. In ACL, 2023.
[70] Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie CY Chan, Biao Zhang, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. ICML, 2024 Workshop In-Context Learning. 2024.
[71] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report.
13

---

[72] Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention relevance: Towards the uncertainty estimation of large language models. arXiv:2307.01379 2023.
[73] Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L Edelman. Distinguishing the knowable from the unknowable with language models. In JCML, 2024.
[74] Daniel D Johnson, Daniel Tarlow, David Duvenaud, and Chris Maddison. Experts don't t cheat: Learning what you don't 't know by predicting pairs. arXiv:2402.08733, 2024.
[75] Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng. Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty planning enhances information seeking in large language models. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.
[76] Hong Jun Jeon, Jason D Lee, Qi Lei, and Benjamin Van Roy. An information-theeorticc analysis of in-context learning. ICML., 2024.
[77] Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine- tuning language models for factuality. In ICLR, 2024.
[78] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018.
[79] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In ICLR, 2019.
[80] Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. ICML, 2022.
[81] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. NeurlPS, pages 24824-24833 2022.
[82] Lilian Weng. Extrinsic hallucinations in LLMs. illiawwengggithhh.in Jul 2024. URL https: //lilianweng. gwt.t.. http://pt///224-00000--aaaallaaa
[83] Andrew Jesson, Nicolas Beltran- Beltran-Velez, and David Blei. Can generative AI solve your in-context learning problem? A martingale perspective. Neurips Safe Generative Al Workshop, 2024.
[84] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
14

---

Appendix
Contents
A Related works
16
Further discussion
17
B.1 Broader social impact
17
B.2 What kind of uncertainty does the posterior hallucination rate quantify?
17
Proof of theorems in the main text
D Extensions to other measures of uncertainty
23
D.I Decomposing uncertainty...
23
D.2 Aleatoric uncertainty for conditional generative models
23
D.3 Estimators
24
Evaluation details
26
E.1 Synthetic tasks
26
E.2 Natural language ICL tasks
26
Dataset details
28
F. Synthetic
28
F22 Language
28
Additional results
30
G.1 Synthetic
30
G.2 Language
35
Computational requirements
15

---

A Related works
Mechanisms and Capabilities of ICL. Several papers argue, both theoretically and through synthetic scenarios, that ICL can implement learning principles such as Bayesian inference and gradient descent [8, 54. Evidence in actual pre-trained LLMs shows that ICL can be approximated as a kernel regression [60], and parametric approximations to ICL can be derived from the hidden state of the last input demonstration [61, 62]. Practical shortcomings of ICL include dependence on example order [63-66] and the impact of prediction preferences acquired during pre-training [65-69]. While future models might improve ICL. performance [70], current limitations are clear and are thoroughly investigated in recent work by Falck et al. [41]. These results imply that ICL in real LLMs does not implement perfect Bayesian inference but suggest that LLM predictive uncertainty includes both epistemic and alcatoric components, and that LLMs can update uncertainties with new observations.
Uncertainties in LLMs. Our results are supported by evidence that predictive uncertainties of large language models are well-calibrated, even in scenarios requiring epistemic uncertainty [32, 71]. Relatedly, LLM uncertainties have been used to detect hallucinations in free-form generation settings such as question-answeriinee
[28- 30, 32-38, 72]. Although not all these papers explicitly focus on LLM uncertainties, they rely on it implicitly by sampling multiple model completions for query and quantifying the differences in meaning. Specifically, Kuhn et al. [33] highlight the challenge of isolating uncertainty over semantic meaning from uncertainty over syntax or lexis in free-form generation tasks. While these approaches do not disambiguate alcatoric and epistemic uncertainty, Ahdritz et al. [73], Johnson et al. [74] recently proposed methods to do so in CGMs. However, Ahdritz et al. [73] require access to two LLMs of different parameter counts, and Johnson et al. [74] do not apply their method to LLMs. Additionally, Hu et al. [75] show that Bayesian experimental design can turn LLMs into strategic question askers, and Jeon et al. [76] present a theoretic study on sources of errors in ICL.
Hallucinations in LLMs. In addition to approaches based uncertainty, a variety of other strategies have been explored to detect or mitigate hallucinations in LLMss strieval-augmentt generation [9-16], custom token sampling procedures [17-19], model fine-tuning to improve uncertainties [20-22] or reduce hallucinations outright [77], as well as learning to extract or steer truthfulness from hidden states [23-27].
Neural processes. Neural processes (NPs) [48, 78-80] are neural network- based non- -arramamtti models trained over a collection of datasets. Similar to ICL, NPs take a collection of datapoints as input and amortize task learning in single forward pass through the model. For instance, when datasets are drawn from a Gaussian process prior, NPs' predictive distributions closely approximate the true Bayesian posterior predictive for given input dataset [7]. NPs have been used successfully for tasks requiring reliable uncertainty estimation, such as Bayesian optimization [40, 78, 80] or active feature acquisition [48]. Recently, Lee et al. [40] applied Doob's theorem to quantify uncertainties in neural processes.
Martingale Posterior
The work most aligned with ours is [39], which introduces Martingale Posterior distributions. Their central idea is to focus on the posterior predictive, rather than the posterior itself, as the primary tool for expressing uncertainty. Similar to our approach, samples from the posterior predictive are used to estimate quantities of interest. Through repeated resampling and estimation of the predictive, "posterior" over the estimate is obtained. In particular, their predictive resampling algorithm can be seen as a generalization of Algorithm 1, although the motivation and use is different. Falck et al. [41] work towards formalizing this methodology for L.MMs. They propose a set of statistical tests to estimate whether an L.LM satisfies the Martingale property; these tests depend on being able to sample from the true Bayesian model that defines the posterior predictive distribution estimated by the LLM predictive distribution. In their evaluations using synthetic data and pre-trained LLMs, they find that violations the Martingale property can occur. Moreover, they find that the fidelity of the LLM predictive distribution to the true Bayesian posterior predictive decreases as the length of dataset completions (N - n) increases. They also derive an epistemic uncertainty estimator based on the posterior covariance over mechanisms, which has connections to the posterior hallucination rate and the mutual information estimand we propose in Appendix D.
16

---

B Further discussion
B.1 Broader social impact
Positive social impact. In terms of misinformation, if users cannot distinguish between accurate information and hallucinations, they may spread misinformation unknowingly. This can damage the credibility of platforms that use LLMs and erode the trust users place in Al systems. Furthermore, as LLMs are increasingly adopted in high-risk sectors such as medicine and finance, hallucinated medical advice poses serious risks, potentially resulting in harmful health practices or delayed treatment. Similarly, inaccurate financial information could lead to poor investment decisions or significant financial loss.
Ethically, hallucinations can reinforce or propagate biases and stereotypes if the generated content reflects societal prejudices. This can perpetuate discrimination and inequality. Understanding when hallucinations are likely occur is vital for holding developers and companies accountable for the content their models produce. Finally, many researchers use LLMS and other CGMs to generate or label data. Hallucinations in this setting can lead to false discoveries and wasted resources.
Being able to accurately predict hallucination rates for given tasks essential for ensuring that AI systems contribute positively to society. It allows for maintaining trust, safety, ethical standards, and the overall integrity of information dissemination.
Negative social impact. When our model is being used as intended but gives incorrect results (i.e. produces a low estimate of probability of hallucination when the true probability is high), could inadvertently be reinforcing biases present in hallucinations while increasing user trust in the outputs.
B.2 What kind of uncertainty does the posterior hallucination rate quantify?
Building trustworthy and effective ICL solutions requires understanding why and when incorrect unexpected responses are generated by a CGM. The ICL literature provides two findings that suggest distinct sources of hallucination.
SGT2
Subjectivty
AG Nevs
Medical
WNL
examples
5512
Sutjectiviyy
AG News
Medical oF
RTE
WNEI
Rengrree 1
Figure 5: 2-77: Error Rate (Top green curves) and Response Entropy (Bottom blue curves) on LLM in in-context learning tasks. Grey dashed lines represent the error rate and entropy of a random classifier over the set of valid responses.
The first finding is that both the error rate and the prediction entropy decrease and saturate with an increasing number of examples [66]. This trend is illustrated in Figure 5 using the Llama-2-7b model [42] on a set of natural language ICL tasks. This finding indicates that one source of hallucination is an insufficient number of relevant in-context examples.
The second finding from the literature is that there are still many tasks for which ICL performs poorly. For example, the response accuracy of Gemini Pro 1.5 given several in-context examples from the American Mathematics Competition is only 37.2%, implying that the model would hallucinate an incorrect response at a rate of 62.8% [3]. We hypothesize that regardless of the number of relevant in-context examples, the model may lack the capacity to answer a specific user query from a complex or new domain accurately. This hypothesis is illustrated using Llama-2-7B by comparing the graphs of the first three tasks (SST2 [49], Subjective [50], and AG News [6]) to the graph of the WNLI task [53]) in Figure 5. While the error rate and response entropy for each of the first three tasks improve
17

---

significantly over random guessing with more examples, both measures appear to saturate near the random baseline for the WNLI task. The second source of hallucinations then associated with whether model has the capacity to factually answer queries for an ICI. task.
This work focuses the first source of hallucinations. That is, the posterior hallucination rate is concerned with estimating the rate of hallucinations that stem from lack of relevant context, and not those that stem from the model not having the capability to solve the task. The predictive distribution encodes response variability coming from several sources, and this variability is closely tied to the ways in which a model can generate hallucination. Below, we discuss these sourees of response variability (or uncertainty) and their relations to hallucinations and the posterior hallucination rate.
Aleatoric Uncertainty. The (1 (1-c) li-keel set defined by given mechanism f* reflects irreducible component of response uncertainty. To illustrate the concept of irreducible (sometimes called "aleatoric")) response uncertainty, imagine hypothetical and idealized LLM fit vast corpora containing many calculus examples so that it is capabable of integration. Consider,
Prompt 1:
fill in the blanks: the integral of x with respect to x on the interval [-3, 3] is u.
If an LLM effectively models the mechanism associated with integration, then response variability will only be determined by the different ways the model can generate the correct response: perhaps, 18, eighteen, x = 9 29, or XVIII, depending on the training data. Uncertainty reflecting the plurality of ways to communicate the same meaning is commonly referred to as syntactic uncertainty [33], which is considered irreducible in this particular example.
In language tasks, irreducible uncertainty does not need to be syntactic. Consider the same LLM and
Prompt 2:
fill in the blanks: the integral of with respect to x on the interval [-3, 3] is u.
In addition to the syntactically different ways to specify particular function (x", æ squared, x x, ...)) and the corresponding result (18, eighteen, XVIIII ...). response variety also depends the semantically different integrands that could fill the first blank (x", cos (z). exp (z), 2x.....) and the semantically different possible results. This additional variety is an instance of semantic uncertainty [33]. While high semantic uncertainty can be indicative of hallucinations, it is not a problem in this example because the imputation of any sensible function and answer could still be valid under the mechanism associated with integration datasets. That is, given Prompt 2, uncertainty over semantically different functions is expected and even desirable.
This pair of examples illustrate an important insight; irreducible uncertainty is mechanism-relative. For example, it may be appropriate to equivocate irreducible and syntactic uncertainty in a simple question answering setting where the mechanism defines a (1-c)-likee -iklik set over correct responses. However, in the second example general, we consider alcatoric uncertainty to be associated with response variability induced under a specific mechanism f. That is, the variability of responses under the likelihood p(y f).
show that this decomposition is not always appropriate. In
Special epistemic uncertainty. Now we provide examples to illustrate reducible uncertainty about the mechanism f. Returning to Prompt I, imagine that the user desires a response in terms of a reduced fraction. There are two obvious ways that the user could augment Prompt to reduce the uncertainty over mechanisms yielding integer, word, fraction, etc. responses. (1) the user could simply replace "fill in the blanks" with "fill in the blank with a reduced fraction." (2) the user could take an ICL approach and augment the prompt with number of examples:
Prompt 3:
18

---

Input: the integral of x with respect to x on the interval [-1, 6]
Label: $1frac(1296)
Input: the integral of x-6 with reapect to x on the interval [-2, 2]
Label: 0 / 1.
Input: the integral of x with respect to x on the interval [-3, 3]
Label:
The first choice may result in reducing all uncertainty about which mechanism to sample responses according to. For the second choice, we can imagine progressive reduction in uncertainty about the mechanism as more examples provided examples, we may still be uncertain about whether to respond with a number or a fraction, or whether respond with any correct fraction or the reduced fraction. For example, given the context, a response of 54 would be as plausible as the desired 18 It may not be until the prompt included an 3
added in-context. For example, if were to see the two
1
example like, "the integral of x with respect to I on the interval [2, 4] is 60/1," until all uncertainty about the mechanism is resolved. We suggest that this may explain the observed reduction in error rate and response entropy in a task like WNLI, where Llama-2-7B 2-mB does not generalize effectively. That is, as we provide more in-context examples, the predictive distribution becomes more aligned with the set of acceptable responses, even if those responses may not be correct.
The preceding example illustrated hallucination example of a non-factual response. Returning to Prompl 1, imagine that the model generates the response 42. Why did it do this? plausible answer could be that the model cannot do integration. We will touch on this possibility next, but first let's consider an equally interesting case. We know from few-shot and maain-of-thought prompting literature [14, 30, 81] that augmenting the context can have significant effects on ICL accuracy. For example, consider the hypothetical setting where the LLM has "grokked" algebra, but only has the superficial capacity to output number when completing definite integrals. Or perhaps the model has the capacity for integration, but the format of the examples and query uncommon in the training corpora. In this case, the LLM may actually be capable of generating correct responses given some clever prompting. For example,
misaligned response; now, let turn
Prompt 4:
Input: the integral of x33 with respect to on the interval [-1, 6]
Label: 6-4 / 4 -1-4 / 4 = 1295 4
Input: the integral of 3 with respect to the interval [2, 4]
Label: 4-4 4 2- 4 60 1.
Input: the integral of x66 with respect on the interval [-2, 2] Label: 2-7
-2 0
Input: the integral x22 with respect x on the interval [-3, 3]
Label:
Again, as the prompt contains more examples that translate the form of the query into a suitable format, expect that the uncertainty about the answer will reduce. For conditional models in general, we will call this Special epistemic uncertainty, which could also be understood as in-context epistemic uncertainty. Both of these examples illustrate hallucinations that due insufficient context, which have been called in-context hallucinations by Weng [82]. In this paper we have focused on the case where increasing the number of in-context examples can resolve this uncertainty. We leave it to future work to understand more sophisticated prompt augmentation.
19

---

General epistemic uncertainty. Let's return to Prompt 1, but this time imagine an LLM fit to corpus not containing any examples from calculus or related mathematical fields. Or perhaps the LLM had finite capacity and is not capable of generating accurate answers to integrals. Then, what do we when the model generates "Dua Lipa" to an integration question? What do we when the model outputs members of the set {17, 137, Dua Lipa, Wednesday, } We say that the response should have high General epistemic uncertainty because the LLM has not acquired the capacity to model the mechanism class, F, corresponding integrals. In the example of Prompt 4, imagine if there were no number of exemplars or no prompt augmentations that could induce a correct response.
Context Langth
Context 100
--2
Contast
100
(a) Special
(b) General
Figure 6: Comparing different sources of hallucination for regression models. Figure 6a illustrates hallucinations due to special epistemic uncertainty. As the context length increases, the predictive distribution concentrates around the true distribution and the model hallucinates less. Figure 6b illustrates hallucinations due to general epistemic uncertainty. Given data from -iito--distribution function, the predictive distribution may not cover the true distribution. Morcover, the model still hallucinates significantly even when special epistemic uncertainty is minimized the number of in-context examples is large.
When a condition generative model pe is a good estimator of the true distribution p. the posterior hallucination rate-and mutual information quantity we propose in Appendix D-are designed to estimate special epistemic uncertainty. We leave the important work of estimating the posterior hallucination rate when the CGM is not good estimator (under general epistemic uncertainty) and accounting for extrinsic hallucinations [82] future work. Initial work toward this is presented by Jesson et al. [83].
20

---

C Proof of theorems in the main text
We begin by stating lemma that will be useful throughout. Lemma C.1. Assume that the conditions of Theorem hold for F and (X, Y), then for a fixed dataset and query D X and under a probability model where F ~ p(f Dn) and (X,, Y,) p(x,y F), then almost surely
logp(Y x) lim logp (Y x. (X, Y)1).
n--00
Proof. For fixed y and x, let g(f) p(y x, F) and apply Doob's theorem on the probability model.
Then it is the case that
g(F)
lim
E
[ply x, F) (X,, Y.)"
(11)
n-+00 Prp(((D..)
/ ply xx,1x, (X,, Y.)")
lim n-+00
(12)
lim / ply f. x, (X,, Y,)")
(13)
n-+00
lim ply x, (X,, Y:)")
(14)
where Equation (13) holds because x is independent of F. Taking logs at both sides and using the continuity of the logarithm, we obtain
go) lim log x, (X,, Y.)
n-+00
and because this holds for all y, it must hold for the random variable Y.
Now we restate the main theorem with proof:
Theorem 3 (PHR via Posterior Predictive). Assume that the conditions of Theorem hold for F and X, Y, then,
= ff ply x,f) < Qe(f,x)) dP x, D,)ddPP(f D..)
(x(x)=
= =ʃʃ 1 { lim logp(y (M) < Qx vi)l x) } dP(r x, D.)dP witt DD,), -
=
where Qx v,,,x,,)))))))) is the e-quantile of lim logp(Y (x) under the limiting distribution 1im P(Y x, (x4,3.)17
Proof. Define an alternative probability model such that F~ p(f Dr)) and (X;, Y;) ~ p(x,y y F).
Let Pa. Eg, Qa,e and Pa denote the relevant quantities computed with respect this alternative probability model.
First, note that by expanding the definition of Qa,e (f, under this new probability model
Qa (F, x) = inf {q E R: < Pa (Y x, F)
< q F,x)}
(15)
(F,x) inf {q R < (logP (Y x, F) S q F,x, (X, Y)1
(16)
= = {q R: Pa
lim log Pa (Y x, (X,, Y. S q FX Y) No000
(17)
where we used the fact that Y 1 (X, Y)Po F,x x in Equation (16). For simplicity. we will use p(,,(X,, Y)) to denote 1om p(- (X,, Y.H) with similar conventions from other quantities. Now, applying Doob's g(f) P. (log Pa (Y x, (X,, Y.) s a f,x, (X, Y.) get that almost surely
Pa lY x, (Xi, Y.) S F,x, (X,, Y.)
(18)
= lim
E N F-p(()D.))
Pa (Y| x, (X,, Y.) S F (X,, Y.) (X,, Y.N
(19)
= lim Pa x, (X,, Y)) (X.,Y))
(20)
N-000
= (logPo (X, Y,) <
(21)
21

---

where we used Doob's on Equation (19) and the tower property in Equation (20). Plugging this back in Equation (17) we obtain
(F, inf (q E :R: PR:ceSPallopp xx (X,, Y.) s (X, Y)))
(22)
= Qa,(((X,, ,x),
(23)
To complete the proof, note that
If 1 logply | xx
< <x,f)< (c(t,)))dP(( dyy) xx P,,
(24)
-/ 1 logpaly x, Qu, (f. x) dP (f)dPly Dr)
(25)
where we changed the probability spaces from p Pa- This is justified because Pa x.f) = ply f, Dr) = p(y x, where we used the independence of Y on Dn once f is known, the fact that (f) p(f D,,) by definition, and the fact that for the quantile function Qa (f,xx) (f,
because
Pa Pl x, F) S q F,x) = (lo x, F) 9 F.
(26)
due again to the independence of Y on the dataset DD, once f is known. Finally we have (abusing notation using D = (x, y) to refer to yx, in the original probability space but also (x)) in the alternative probability space they have the same distribution):
// 1f Po | x, < Qu (f, (P (f) x, D,))
(x)
(27)
// 1 log x, < Qa, (f, x) (x, ( x,
(28)
// 1 logPaly x, D.t ) < Qa D x) ( x))P (f,
D
(29)
// 1f ly x, D 1 < Q (D n+ dP xx P.) (f,, D Pa))
(30)
// 1f log D < Qa, (D
x, Dr, (D n+1
Dx)
(31)
Where Equation (28) justified by because (xx, y) doesn't 't appear in the term inside, Equation (29) is justified by the arguments above and Lemma C.I, and Equation (31) is result of marginalizing f
out. The last thing to point out that (a x, (x, y) = p(y x, yx)f") and
(x.
(x x)
because
Pa (log Pa (Y x, (x,y) S q x, (x. (x
= P (logp(Y x, (x) S X, (x..
Using this fact in Equation (31) yields the theorem.
22

---

D Extensions to other measures of uncertainty
In the main paper, we focus on developing the posterior hallucination rate; however, there are other quantities that also be used predict model performance. In Bayesian machine learning, commonly used quantity for this purpose is the posterior mutual information between and Y. which denoted by I(Y; x. D.). This is often interpreted as quantifying epistemic uncertainty. As is the case for other quantities presented the paper, it not possible to compute it directly CGM. Nevertheless, in this section extend the results of the paper and demonstrate that by using similar methodology to the one presented, is possible to construct estimators for it.
D.1 Decomposing uncertainty
There are several ways to decompose total uncertainty about an answer into epistemic and aleatoric components. One such decomposition proceeds by defining uncertainty via entropy and then using a clever decomposition of mutual information to link various interpretable quantities. We use this decomposition throughout.
To elaborate, we define total predictive uncertainty as the entropy of the posterior predictive, denoted H(Y D,)) We define aleatoric uncertainty-whiie represents the irreducible portion of uncertainty- as the expected entropy the likelihood, denoted as Ep(f p,,) [H(Y x, f)]. And finally, define the difference between these two quantities the epistemic uncertainty, which represents the reducible component of the uncertainty.
By defining epistemic uncertainty in this way, it turns out to be equivalent to I(Y; x, D,), as the following holds,
T(YFF x, D,) =0)=HYYY H(Y Dn) p(f[D..)
E [H(Y x,
- - logp(s 1o, Dx,dd)PP( (x) +
+/ʃ logply x, f)d(( x, f)dP(f Dx).
In this equation, we see that the total predictive uncertainty depends only on the predictive distribution, p(y x, D,,,, which-under our assumptions-is the estimand of CGM, po(y x, (xx, yx)r). The aleatoric uncertainty, on the other hand, depends both on the likelihood function, p(y x, f), and the posterior mechanisms, p( x, (x,, Y.).), which only latently modeled by CGM.
In the next section, we show how the alcatoric uncertainty can be expressed using only samples from pxx,y (X4, yx)f). which enables its estimation with CGM and, in turn, provides practical estimator for the epistemic uncertainty of the model.
D.2 Aleatoric uncertainty for conditional generative models
We use a similar proof as in Appendix C.
Theorem 4. Assume that the conditions of Theorem 1 hold for F and (X, Y). Then for a new independent x,
E p([D.))
[[Y x,f)]
E (PMMII..
H(Y Doo)
23

---

Proof.
E D.()x.D.)
[ x, H(Y x,f, D2) x, Dn)
logp(y x,f, (n),Dr) D,)
// logp(y x,())P(y x, f)dP(f x, D.))
// logply (y x, ff,P 2 x, Dn))
// log
x, x.DeddP(y x, ff. D x, D,))
By Lemma C.1
/] logply
x, Doo
x, (xI,D D x, Dx)
By conditional independence
//
logply
(x, [ x, D,.)
//
logply xx D xx,
By marginalizing
//
logply xdDoy) D ddPPPP(D (D n-1 Dn)
// logply x, ddPo)) x, D.0 ddP n-1 Dn)
By independence of x
H(Y x, Doo )dP( Dz)
H(Y x, Doo)
pDD P,))
D.3 Estimators
Using the theorem in the previous section and similar approximations for the PHR, construct estimates the alcatoric and epistemic uncertainty.
In particular, we can define a truncated approximation of the aleatoric uncertainty
Ho F,x, (x.)))))) = / Ho (Y x, (x),):))) dPo((x,.y))) (+))),
where,
He (Y) x, (x,,))))) = flog py x, (x., (x xx, (x..)))
(33)
and N is practical number of generated examples. In this model approximation for the aleatoric uncertainty allows us to define a finite model approximation for the epistemic uncertainty,
LY;Y; x, (x) He Y x, (x,, y:) HY F,x, (x)))
(34)
And finally, as in the main body of the paper, we can use Monte-Carlo estimation to construct practical estimates of these quantities. This is summarized below.
Total predictive uncertainty.
AY x, (x).
Σ log Pogro x, (x),))))) Yi ~ poly I x, (x))
(35)
Aleatoric uncertainty.
Ho(Y F,x, (xx,)))) == See Algorithm 3.
(36)
24

---

Algorithm 3 HY F,X, (x))))))
Require: Query context Dn v.) CGM pe. number of context samples M, number of predictive uncertainty samples K, max context length N.
1: for i + 1 to M do Sample imagined context
2:
3:
D Dn
4:
for j +- n + 1 N do
5:
(x). ~ (x.y D)
6:
D D DU (x,.8j)
7:
//Total predictive uncertainty with D
8:
h, - Ej= logpoly; x, D) Vy poly x, D)
9: return M7 Ei= h
Epistemic Uncertainty.
ToYY;F (x6, = HY x, (x), Ho F,x, (x),
(37)
We note that Algorithm 3 is very similar to Algorithm 1, with the key distinction being the replacement of the THR computation with the calculation the average entropy of the likelihood. This similarity is not a coincidence, as both can be seen as specific instances of a more general predictive resampling algorithm presented by Fong al. [39], albeit different context.
In Appendix G.2.2 and Appendix G.1.1 we provide some experiments comparing this epistemic uncertainty estimator against the PHR and find that both correlated and thus represent similar information.
25

---

E Evaluation details
E.1 Synthetic tasks
We implement our neural process by modifying the Llama 2 architecture [42] to model sequences of continuous variables. We replace the tokenizer with linear layer and the output categorical distribu- tion with Riemann distribution [7]. We train the model from random initialization on sequences of (x,y) pairs using standard next token prediction objective and use the AdamW optimizer [84] with learning_ raterateee 0.0001, 31 = 0.9, B2 = 0.999, c = 1c-8, and
= 1e-6: 66. We use a cosine learning rate schedule, with warmup of 2000 steps, and decay final learning rate down to 10% of the peak learning rate.
We define the (1-r) (1-e)-likely with 0.05 such that response y a hallucination if it falls outside of the 95% confidence interval of a given sampled distribution conditioned on x. The data generating process is described in Appendix F.I.
E.2 Natural language ICL tasks
We consider tasks defined by six datasets: Stanford Sentiment Treebank (SST2) [49]: predict sentiment positive or negative; Subjectivity [50]: predict review subjective or objective; AG News [6]: predict article World, Sports, Business or Sci/Tech; Medical QP [51]: predict medical question pairs similar dissimilar; RTE [52]: predict sentences entailment not entailment; and WNLI [53]] predict sentence with pronoun replaced as entailment or not entailment. We replace the label Sci/Tech with Science for AG News and not entailment with not for RTE and WNLI. We filter out any queries of length longer than 116 tokens. Full descriptions of these datasets are given in Appendix F.2.
To implement ICL for a given dataset, we sample a response balanced training set of query/response pairs Durain (x, ys Each query in Dirain is prepended with the string Input: and appended with a new line Each response is prepended with the Label: and appended with \n A test query Xtest is prepended with the Input: and appended with \nLabel: These strings are concatenated together to form a prompt and we generate response y from the predictive distribution given by a Llama-2 model pe (y Ntest; Dersin). An example prompt from the Subjectivity dataset is shown in Appendix E.2.
For our experiments in 3.2, employ LLaMA-2 [42], family of open source LLMs based on an auto-regressive transformer, pretrained on 2 trillion tokens with a context window of 4,096 tokens. We run LLaMA-2-77 2-BB unquantized model (16-bit).
To estimate the predictive distribution over responses, we sample n input/label pairs based on the given context length, create prompt based on the context, and generate samples. An example of prompt from the Subjectivity dataset set forth in Figure
The y samples are generated as single tokens, since all labels for our datasets can be identified based on their first token. We set the temperature and top-p parameters to I to provide the greatest possible diversity and randomness to the label output.
For predictive resampling, initialize the context by sampling input/label pairs, generate N new context examples by producing prompts that include the updated context, and produce samples from the cumulative context. Again, we set max_ new_tokens top_p = 0.9 to provide a high level diversity and randomness to the generated output. Examples of generated context pairs are set forth in Figure 8.
200, temperature = 1 and
Using the transition scores from the model outputs, we compute the log likelihood needed for the posterior hallucination rate.
This process is similar to the "prior fitted network" implementation of Müller et al. [7], but require the conditional distribution of both queries and responses y, where their implementation only models responses.
26

---

the assasins force walter to drive their escape car Label: objective
Input:
Input: vega and ulloa give strong performances the leading lovers and there are some strong supporting turns. particularly from najwa nimri.
Label: subjective
Input: they decide that the path to true love is to purposely each other up on " extreme dates " with the objects of their affections
Label: objective
Input: maid manhattan " charmer pc " pretty woman that ditches the odious prostitution theme for class commentary Label: subjective
Input: each weekend they come back with nothing but hangover Label: objective
Input: piccoli 's performance is amazing yes but the symbols of loss and denial and life-at-arm aar--ammmssllenee s-length in the film seem irritatingly transparent
Label:
Figure Example prompt for language model
Input:
pick any word even the tiniest and you will find writers arguing over its relative importance its correct usage and how you pronounce it
Label: objective
Input: janus' entry does number of things the novel fails at : it tells the story of the novel and better yet it 's funny
Label: subjective
Input: this is the kind of show that 'g got the warm n' fuzzies all over especially if you have a sick sense of humor
Label: subjective
Input: even if you are an apple and orange man or woman who would be more comfortable at a state fair rodeo than in a silk dress this should be on your list
Label: objective
Input: nearly every adjective one could use to describe movie theater except expensive and first-run can used to describe this place Label: subjective
Figure 8: Example of generated context pairs
27

---

Dataset details
E.1 Synthetie
generated
datete::
generated dataset
training datagot
traihing dataset
training dataset
generated dataset
generated dataser:
generated dataset
training dataset
training dataset
training dataset
Pate kur
generated detwset
generated dataser
genernted dataset
(a) Training Datasets
(b) Generated Datasets
Figure 9: Training and generated datasets for the synthetic regression task.
Queries x are sampled from a uniform distribution on [-2, 2]. Responses y sampled from a normal distribution with mean p(x) parameterized by a random ReLU neural network conditioned on x and constant standard deviation 0.1. We generate set of 8000 sequences, each corresponding to distinct random re-initialization of the neural network, with 2000 (x, y) examples each. Training and test data generated non-overlapping sets of generated sequences. Example training datasets are plotted as different colors in Figure
F.2 Language
For our experiments in 3.2, we randomly sample context examples and test input/label pairs from the following datasets:
Stanford Sentiment Treebank (SST2) SST2 [49] is corpus with fully labeled parse trees that allows for a complete analysis of the compositional effects of sentiment language. The corpus consists of 1 11,855 single sentences extracted from movie reviews. It was parsed with the Stanford parser and includes total of 215,154 unique phrases from those parse trees, each annotated by human judges. Sentiments are classified as binary labels "positive" or "negative".
Subjectivity (Subj) The Subjectivity dataset [50] contains 5,000 movie review snippets from www. www.eettotoootos..oommmmm com labeled "subjective" ,, and 5,000 sentences from plot summaries available from www. imdb. com labeled "objective". Selected sentences or snippets are at least ten words long and are drawn from movies released p0st-2001.
AG News The AG News dataset [6] contains 496,835 categorized articles from more than 2,000 news sources. The largest classes (World, Sports, Business, Sci/Tech) were chosen from this corpus construct our dataset, including only the title and description fields.
Medical Questions Pairs (MQP) The MQP dataset [51] consists of 3,048 similar and dissimilar medical question pairs hand-generated and labeled by doctors based on patient-asked questions randomly sampled from HealthTTap. Each question results in one positive question pair (" ("similar") that looks very different by superficial metrics, and a negative question pair ("different") that conversely look very similar, as to ensure that the task is not trivial.
28

---

Recognizing Textual Entailment (RTE) The RTE dataset [52] comes from series of annual textual entailment challenges. Examples are constructed based on news and Wikipedia text, and labeled as binary classifications based on whether or not there is entailment.
Winograd Schema Challenge (WNLI) The WNLI dataset [53] consists of 1,100 sentence pairs with ambiguous pronouns with different possible referents. The task is to determine whether the sentence with substituted pronoun entailed by the original sentence.
29

---

G Additional results
G.1
Synthetic
We extend the synthetic experiments in the paper in two ways. First, provide additional results for the setup described in the main sections. Second, we provide additional results in a controlled experiment where we have access to analytic expressions for the posterior predictive.
G.1.1 Additional results with main paper setup
Figure 10 and Figure 11 offer different visualizations that support and extend the results in our main specf-lspc that the posterior hallucination rate (PHR) is an accurate predictor of the true hallucination rate (THR) under various settings and context lengths. Specifically, for different settings, Figure 10 displays scatter plots of the THR against the PHR, while Figure 11 plots the average THR and PHR against the number of in-context examples.
Coneetlangh
Condect
Contact Langt:
Coited
Contetllagg SC
Contedt Langitc 100
FAM EUR
15.3
Fod au
and au
Lox Lse
Pastddrr Inllactaatoo
Portrtor mutiraniinn
Porarer
Talkerasten Flats
Prolet: chilettkeeceeeeeeeeee flat
Posteror Ielladaation
(a)
0.02
Comeellangh
Cangen
Contact Langit:
Certed
Coneellangh SC
Donted Langit
F 1EQ
1.M MSL
2
MGC
Fow
Pandrii Hullede aicc Rate
Poste kx Hall Rall
Pucink Hulk, Raae
Poaknio Hilkainkk Rate
Posksi: Halkoande Rute
Paderse Holleuration Rub
(b)
0.05
Geneotlanghh
D XPN
PORO 183
21 IT
127 CoN Padeess Rub
Padeiix Rude
Pote Fale
Puciki Hulk, Rale
Prsknir Nilkeaade Hate
Poskei:
(c) 0.1
È BED
O CPR
J1 Ae
111 Dd
Pastetar Iralaciratt cosiitattccc Tate
Porartor Hallc Sa Fate
Porare Taauucaaaoonn natenn
Pretetor Ieltchrster Tlats
Pollet chl.crrteeeeeeee fatt
Poseenrr falladiatoon fade
(d)
0.2
Comertlangh
33:5
23.5 Edt
Taaledar wat.ccatttt Tate
Portator satonaat Fate
Porarbor Toabuc catinn
Prdertor Iellchrddr Tate
Poolet: cww..crrnioeeeee Faate
Pooteror ialladiatton Tade
(e) 0.5
Figure 10: Synthetic regression data ablation of the parameter. In the above scatter plots of the true hallucination rate against the posterior hallucination rate, we observe that the posterior hallucination rate is an accurate predictor of the true hallucination rate under various settings of the parameter value.
Figure 12 shows scatter plots of the THR vs. the PHR under misspecified values. The THR is calculated under 0.05. From top to bottom, we show charts for different epsilon values, denoted
30

---

pelicration
Halke mreoe
Inllccation Pice
Pristor hher
Pootete halloration fick
2.095
6-113
1.300
471 1.000
(a) c 0.02
(b) E 0.05
(C) = 0.1
(d) c = 0.2
(c) e = 0.5
Figure 11: Synthetic regression data ablation of the c parameter. Plotting the true and posterior hallucination rates (THR and PHR) against the number of in-context examples n. We observe that the PHR a good predictor of the THR across different settings of e. Further, we observe that the PHR is more accurate estimator for small values than for large values.
as é, used to calculate the PHR. Notably, we see that setting غ = 0.1 results in the PHR being more accurate predictor of the THR as context length grows, as opposed to ē = 0.05 (where there is no misspecification). This reflects our observation that the PHR underestimates the THR, particularly for longer context lengths.
Conget
1:00 P-R d 6.02
1:0
P-R Z 0.32
P-R 0.0
PAR 1.0.2
PHR 2.002
PHR 7 02
Comeellagh
X. MM
1,00
1:30
1/0
P-R 2.0.
P-R E00
PeR 3.05
PAR 1.0.0
PRR 1.0.0
PHR 1 05
Coneellagh
X1 11:01
A 1300
10M
PIF 01
21 2- 0.3
2IE 2-
PIR---00 0.2
P 0.
argt:11
Oenet
A 1155
J 1121
mil 0=
MFC E-02
2-
P P
argh:
Genet
2.50
210 0.
p1 0.5
2.5
Figure 12: Synthetic data: Effect of misspecified c. The true value is c = 0.05 and is used to calculate the THR. We vary the value used in calculating the posterior hallucination rate, which denote as ē here.
Mutual information. We also assess the mutual information (MI) estimator for epistemic uncertainty, derived in Appendix D and defined in Equation (37), to see how well it predicts the true hallucination rate (THR). Figure 13 shows that the MI estimates are significantly correlated with the THR, which indicates that the MI can also be an effective predictor of hallucinations. Figure 14 allows us to look deeper into the relationships between the THR, PHR, and MI. In comparing Figures 14a and 14b, we see that the PHR has more linear relationship to the THR than the MI, which has a more sigmoidal relationship to the THR. Nevertheless, Figure 1 4e which plots the PHR against the MI shows that both are strongly correlated. Their correlation provides evidence that the PHR and MI encode related
31

---

information, and that the PHR is a measure of epistemic uncertainty, which is expected from its definition.
bellctariinn Mituall Information
Context Length: 2
Context Length:
Context Length:
8: 6, 368
Trus Holll www.mmmeeiiia...
D=ya1 0.008
PSE: 0,011
8 073 vot- 0.000 MBE:
8 0 438
0 634
cal. 0.000 0.012
sul 1000
0.002
0.31.
Ccamples e5
130
Mutaal Information
Mutual Intormation
Mutual Information
(a) MI and THR vs. (b) Calibration: THR MI against different numbers of context examples
Figure 13: Synthetic data: (a) The THR and Mutual Information reduce significantly as the number of contextual examples n increases. (b) Calibration evaluation of the mutual information against the true probability of hallucination given é = 0.05.
Conmen 3 2
Corne Langht
Contert Langth
Contect Langhn
Contedt Lengh 50
Contert Lergth 108
2. OSe
C.M FOX
21 1100
L2
1,0
Postator Isllcinttin Rote
Postetor www.iiitiiiinnoooooooooooooo
Podtddor balcctddoo
14
Podtrior lnbuchntinnnan
in
Poste Hollciistinntaa
Postador lilke natton
(a) THR vs. PHR
Longht 2
Langht 5
Chldted Lenghh TO
Chebod Langen 2b
Darked Lenghh 50
Contim Langlt 108
2 0003
41
Jr 1.31
Manal Inboarice
Max Infoattoo
Maxi Inforadioo
Maxi intomaaice
Maxt Infomtioo
Max intoratioon
(b) THR vs. mutual information
Longht
Longht
Conted Lengt 10
Contert Langit 25
Langet 5D
Comtecl Conge 103
3000
Al
Masa Vnterati
Muxi Intomatoor
Visue inlomation
Waxt ioomaton
(c) PHR vs. mutual information
Figure 14: Synthetic data: visualizing and quantifying the relationship between the posterior halluci- nation rate and mutual information.
G.1.2 Additional results with an analytic posterior
Our method relies on the assumption that
p(y x, Dn) = po(y x, Da))
(38)
is sufficiently accurate for our procedure to work. As discussed in Section 3.1, if this assumption is incorrect, our method may produce inaccurate results. However, in the experiments presented in the main body of the paper, we did not have an analytic posterior available to compare against, nor a way to compute ground- -trnt PHRs.
In this section, we investigate whether Equation (38) holds in a simple setting where we have an analytic posterior for comparison.
Setup. We use two simple synthetic datasets where it possible analytically compute the posterior ply x, D.))
32

---

1. Linear Regression: In the first setting, we use simple one one-dimensional linear regression. We assume f ~ NV I2), and each sequence of (x, is such that x NO σ,)) and y ~ ((1 + f2x, ay), where o₂ = 1 and oy 0.1.
2. Polynomial Regression: In the second setting, we use linear regression on a polynomial basis. Specifically, f ~ (O0 x (N0,, oz), and y WN( o(x), Ty), where 6(x) E Rd+1 and d(x), x'_1. We use the same hyperparameters as in the linear regression setting and set d= 3.
We then train a CGM using the same setup and architecture described in Section 3.1, with the only difference being that we train on sequences of length 100. For illustrative purposes, the left panel of Figure 15 displays the sequences used to train the model, while the right panel displays sequences generated by our model.
training cataset
traningg dataset 2
training dataset
generated dataset
generated dataset
genarated dataset 3
30
training cataset 4
traning 5ataset 5
traniing cataset
getenated dataset
generaled delinee
generwted dataset
generated dataset
genersted dataset
penerated dataset
(a) Training Datasets
(b) Generated Datasets
Figure 15: Example training and model-generater datasets for the polynomial regression task.
We conduct two different experiments on these datasets.
Experiment Setup. Calibration of resampling distribution: First, we examine the distribution produced by our resampling procedure. Specifically, we study the calibration of the distribution Po(y x, D,), where sample ~ po xx P,)) is generated by first sampling from pe (xxx y yo) Dn). and then sampling from (y x, Dx -Dx)-where -here Dy Dr U yx) and N is defined as in Algorithm 1. Ideally, all three sampling procedures should align, and we would expect po(y x, Dn) z ( x, Dn) = p(y x, D,.) However, inaccuracies in the approximation may prevent this from being the case.
We approach this question from the perspective of calibration, as we believe it provides an interpretable metric for understanding the model's 's predictions and captures an important property that essential for the PHR to function correctly.
Specifically, we ask whether the distribution produced by (y x, D,,) is calibrated in the sense that if (x, Da) is the g-th quantile of po (y x, D..) then
L.
I 1l
(w)) Dn) - dg= = 0,
(39)
where the inner integral is taken with respect to the data distribution. We refer to this metric as the calibration error. If it were true that py x, D,,) x p(y x, D,), then the inner integral would be zero for every 4-
We estimate the value of this integral using a Monte Carlo estimate with 200 samples for (x D.) and 200 samples for the integral, and then we evaluate over 200 values q. Moreover, we set (N -n) to 30 to compute pa.
In fact, with the true posterior we should have ply x, P.) = ply xx D.), where ply x, D..) is drawn with the same sampling procedure as ýo(y D.).
33

---

Experiment 2 Setup. Model PHR vs Analytic PHR: For our second experiment, we compare our estimated PHR with an almost fully analytic counterpart. In particular, we compute the analytic PHR
/f 1 & A(f,x)) dP x, D.) dP Dn),
by using an analytic expression for A(f, x) based on the quantiles of the normal distribution and estimate the integrals by sampling from the analytic posterior p(f Px) and the analytic posterior predictive ply x. D.). It is important to note that this differs from our evaluation in Section 3. I, where we compared the PHR to the THR and used po x, Dn) instead of p(y x, Dn). ) as no
analytic counterpart was available. We compute the estimate of the PHR given by the transformer model using Algorithm with parameters
Context Length:
Context Length: 2
Context Length:
perconspee of Sampice Under
percentopee of Sample Under
Callbration Ener: 0.023
Call bolttraton Emer: 0,019
Callbration Ener: 0.035
Somnen de
Ideal Reatiorship
Ideal Re Reattorship Actual Relationship
Ideal Reatiorship
Actual Relationship
Actual Relationship
Predicted Quantile
Predicted Quantile Q
Predicted Quantile
Figure 16: Calibration curves for polynomial regression across different context lengths (1, 2, 8). The x-axis represents the expected quantile, and the y-axis shows the observed proportion of points below that quantile. The area between the ideal (blue) and observed (pink) curves reflects calibration error.
Context Length:
Context Length:
Context Length:
Undes
Calibration 0.026
Undes
Callbration Emor: 0.029
Reccos Samgles Undre
Calibration 0.025
Ideal Relatiorshpp Relationship
Relttorship
Ideal Relationship Relationship
Actual Relationship
Predicted Quantile
12
Predicted
04 Predicted Quantile
14
Calibration curves for linear regression across different context lengths (1, 2, 8). The x-axis represents the expected quantile, and the y-axis shows the observed proportion of points below that quantile. The area between the ideal (blue) and observed (pink) curves reflects calibration error.
Figure 17:
Context Length: 1
Context Length:
Context Length: 8
1.
TTu 51R
The PHR
0.2
Ideal
Ideal Estimated us Tue PHR
62
Ideal Estimated Tue PHR
Estimated Tue PHR
Estmated PHR
Estimated PHR
Estmated PHR
Figure 18: Estimated true PHR plots for polynomial regression across context lengths (1, 2. 8). Each point compares the transformer-bon PHR estimate (x-axis) with the analytic posterior estimate (y-axis), where points closer to the diagonal indicate better agreement.
34

---

Context Longth:
Context Length:
Context Length:
Ioial
Extimated us Thus PHR
Estimated
Thuc PAR
Trus PHR
Estimated vs True PHR
04
50
22
Estimated PHR
Figure 19: Estimated true PHR plots for linear regression across context lengths (1, 2. 8). Each point compares the transformer based PHR estimate (x-axis) with the analytic posterior estimate (y-axis), where points closer to the diagonal indicate better agreement.
Results. Figure 16 and Figure 17 show the calibration curves according to the procedure described above Experiment 1. Specifically, the x-axis represents the percentage of points that should lie below a given quantile if the distribution is calibrated, while the y-axis shows the actual number of points below that quantile. The calibration error is the area between the ideal calibration curve (blue) and the observed calibration curve (pink). Each panel in the figures represents the calibration with a different number of observed points the context. In both figures, we observe that the calibration error is below 4%, independent of context length, which suggests that the distributions well-calibrated.
The results for Expeirment 2 can be found in Figure 18 and Figure 19. They display the estimated PHR using the transformer model versus the estimate obtained from the analytic posterior. Each point in the graphs corresponds to a single test example; its x-coordinate represents the wansformer-based estimate, while its y-coordinate corresponds to the analytic posterior estimate. If the transformer perfectly approximated the posterior and the number of Monte Carlo samples were sufficiently high, each point would lie on the x-y line. Most importantly, they show that the PHR estimate produced by the trained transformer is close to the estimate from the analytic posterior. The plots also indicate, as expected, that the PHR decreases as the context length increases.
The results from both experiments suggest that the distribution approximates p well, and that Algorithm operates as expected. Specifically, we observe that the resampling mechanism used in Algorithm 1 leads to a calibrated distribution, and that the estimated PHR closely matches its analytic counterpart. This supports the hypothesis that the underestimation observed in Section 3. due to the fact that the THR is not the PHR, and while these quantities are related, they are inherently distinct.
Discussion.
Although this certainly does not prove that Pa s p for all ICL tasks, it provides further evidence of the applicability of the algorithm as-in these simple scenarios-we do not observe any significant failure modes, such as distribution drift due to resampling.
G.2 Language
G.2.1
Gemma-2
Here we evaluate the posterior ballucination rate estimator on the same in-context learning tasks used in the main body of the paper using Gemma-2 9B [43].
Subjectivity
AG Nevs
Medical on
WHLI
28
Dangpoooe Erergy
05
1100rrppee
soarrpies
Cya Rate
Figure 20: Gemma-2-9b: Error Rate (green curves) and Response Entropy (blue curves) on natural language ICL tasks. Grey dashed lines represent the error rate and entropy of a random classifier over the set of valid responses.
35

---

Figure 20 shows the error rate (green) and response entropy (blue) of Gemma-2-999 In general, both metries decrease and saturate with longer context lengths. For all tasks, the model performs least slightly better than random, and is particularly performant on SST2, Subjectivity, AG News and RTF. However, for Medical QP and WNL.I, error rates are very close to random, indicating poor generalization.
5512
Subgectivty
AC News
Medical OP
I1TC
WNU
0.50
2: A:7
1.23
6.371
PC 6:33
R11 0.0G
P1: 3.409
eoarrpies
Habe
Protener
Figure 21: MHR and PHR against number of in-context examples for Gemma-2-9b. We set 0.05, and the Posterior Hallucination Rate accurately tracks the Model Hallucination Probability for all tasks (SST2, Subjectivity, AG News, Medical QP, RTE, and WNLI).
Results. We report results for Gemma-2-9b. We set N n 5, M 10, and K 50. Figure 21 plots the MHR and estimated posterior hallucination rate against the number of in-context examples with 0.05. As with the Llama-2 results presented in the main body of the paper, we see that the posterior hallucination rate is good estimator of the MHR.
SST2
Subjectiity
AG News
Medical Op
WNU
3: 771
3. 2.59
exampies
Exaampps
osamples
exampess
exarplees
Enplical Croa Flata
Paanatoo Hallucration Tade
Evar Ransor Daesing
Figure 22: Error rate and PHR as a function of the number of in-context examples for Gemma-2-9b with 0.75.
Figure 22 plots the empirical error rate and estimated posterior hallucination rate against the number of in-context examples. When is set to high value (e.g., >0.5), the PHR closely aligns with the empirical error rate across all datasets, with the exception of WNLI, likely due to Gemma-2-9b limited generalization to this task. Notably, it is surprising that the PHR tracks the empirical error rate well for the Medical QP dataset, given Gemma-2-9b2 similarly weak generalization to this task. Besides this discrepancy, the results for Gemma corroborate those found for Llama-2 in the main paper.
G.2.2 Llama-2
Here we extend the natural language ICL task results of the main paper using the Llama-2 family of LLMs [42].
Results. We report results for Llama-2-7b with N - n = 5, M = 10, and K = 50. Figure 23 in Appendix G shows that the posterior hallucination rate serves as a reliable estimator of the MHR across different settings. We further ablate the e parameter against the empirical error rate and report results for each natural language ICL task in Figure 24 of Appendix G.
In Table 1, we report the results of an ablation study on SST2 that varies the number of generated examples, MC samples, y samples, and model parameters. We see that increasing the number of MC and y samples improves the R² scores for both the MHR and empirical error rate. In contrast, increasing the number of generated examples or model size alone results in performance decline for this task.
In Figure 25, we compare the mutual information (MI) estimator with the error rate, MHR, and PHR across all tasks. Specifically, Figure 25a illustrates the relationship between MI and error rate, while Figure 25b illustrates the relationship between MI and MHR. Additionally, Figure 25c depicts the relationship between MI and PHR. The high R² values observed between the MI and PHR provide further evidence that both estimators capture the same underlying information.
36

---

Table 1: SST2, Llama-2: Ablation Hyperparameters for the Posterior Hallucination Rate Estimator
MHR
Error Rate
# Generated
# MC Samples
#
samples
# Params
MAE
R²
MAE
R²
M
0.039
0.618
0.041
0.702
5
10
50
7B
0.045
0.611
0.045
0.705
10
10
50
7B
0.039
0.618
0.041
0.702
10
50
0.038
0.653
0.041
0.711
5
20
50
7B
0,039
0.618
0.041
0.702
5
10
50
7B
0.032
0.681
0.042
0.710
5
10
100
7B
0.039
0.618
0.041
0.702
5
10
50
7B
0.040
0.652
0.042
0.764
20
100
7B
0.037
0.689
0.044
0.719
10
20
100
7B
0.039
0.618
0.041
0.702
5
10
50
7B
0.033
0.607
0.041
0.669
10
13B
0.035
0.669
0.042
0.729
5
20
100
13B
37

---

Inllatiation Paranily
Pastwor
C-0.02
Bed 0.0
Del 3 093
A/ 0.000
All 0.05
4- 0.20
0.81
0.004
AE 0.00
03
oouapples
unampplo
Preamppes
Gunample
nusamppo
(a) SST2
5.02
3.02
x-02
0.5
8:203
and EARU
(nd 2.00)
(ivel BAM
2::0 651
p--0.50 1035
A" 6.70
6
A" 0.701
A" 6-722
68
d'uaamppe
unammppo
Phenamplex
Gunamplex
nuccmmppc
nusemppos
(b) Subjectivity
1.002
c-02
2-88 BA00
pvotl 2.00
prsat 3.000
vell 0.000
Prst 000
Fron 800
(2.0.000
0444
A". 0/42
0.000
0087
.P. 0.708
(1 D.754
exammppes
nexamples
examptes
examples
exammpes
osanpios
(c) AG News
D-aal BA00
D-M 3.000
G-ill 8.003
Duel 100
Eveil BOX
D/ME
0.
0.311
exxmpppeee
neszmrppee
examptes
exangggee
noxampiee
reccmppiss
recampppss
(d) Medical QP
5.03
3.05
0.
03
0.5
025
and EARU
nut 2800 6.T1
wi DARA
die 20 n.sd
A esumpppis
it Betamplo
Runampies
Nicumples
f1 socempplaa
(c)
c=0.02
1.0.0
0.
c=02
C=0.3
0.00
CH0255
351
1,755
USE EX01
MSE E117
ReE 3:11
WeE 8O1Z
MRS Bm11
Poee C.723
27-0-42
0,122
neexamples
noxamples
8 examppos
n nexzmiless
Roxamile
cxomppls
n osamplls
( WNLI
Figure 23: Ablating for the posterior hallucination rate estimator against the model probability of hallucination for Llama-2-7B.
38

---

Freirid Frem
Porteoor Siadion Rark
5.00
0:063
1:444 MDC 3830
1.291 MIE 533
1.000 MSC 0/0
0.00
USE 6403
MSD E.3T
MSC 0.310
soazpplex
á cnampppe
Ranamppo
nunample
1u plea
(a) SST2
0:02
1.09
2-2 150
2-1 375 RIEE 1.43
J- 1,20 FI-E: 3.84)
J- -0815 MAE I
J. ooorrs MaE 122
0.001 MH-E 122
0.000 B0--
44
examptes
exompppe
examptes
eexmmples
Roxamples
osampios
(b) Subjectivity
21 2.222
2-1 (4B
1.300 0.961
J. 1:344
0011 nistd
01002
exammpte
osomppes
examptos
exampies
ooammpee
exampies
osampios
(c) AG News
2- 25554 USE EX61
000911 KSB 5.000
D RSSE: 3.00
J. GA4D MAR DOD
3. 10.0577
0.045
#20mppte
easmmppe
esamptes
examppeee
Resampples
eecamples
exccmpppe
(d) Medical QP
J0 235
1000
tried 0.300
11000 Mit: 0.E
128
2.032
64
2:103700 370
2-00.00
64
Qunamppua
niccmplla
nocempppi
(c) RTE
400
17777
2-0.554
1.044 pvst 000
2.00 1.129
3.000
5.000
000
3:327
XP.
6" 0.300
0.200
0.204
64
neesamples
nexamples
nevamppto
nexamplos
nexamples
nexamples
oxamplos
( WNLI
Figure 24: Ablating e for the posterior hallucination rate estimator against the empirical error rate for Llama-2-7B.
39

---

16672
AO News
Medical
RIE
VHHL
(a) Mutual information and error rate vs. n examples.
Subjecttiiy
(b) Mutual information and MHR vs. n examples.
B573
Subjecttvity
AC News
Medical
RTE
WNU
1xscempoex
necktple
Fararoo Hallcriie Par
(c) Mutual information and PHR vs. n examples.
Figure 25: Comparing the mutual information estimator (a) the error rate, (b) the model hallucina- tion rate, and (c) the posterior hallucination rate using Llama-2-7B. We see that there are significant correlations either metric across tasks. The high R²2 between the mutual information and posterior hallucination rate is evidence that they quantify similar information.
40

---

H
Computational requirements
For our experiments. used internal cluster made up of A100s and RTX 8000s, each with 40 to 48 GB of memory. Training the models for the synthetic experiments took approximately one day on single machine. For the natural language experiments the main paper, we ran 50 seeds dataset, with each seed requiring between 20 minutes and 4 hours. Additionally, we conducted ablations over different values of and (as detailed in Appendix G). We also performed further experiments and developed models that required additional computational resources but were not included the paper.
41

---

NeurIPS paper checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We back up the claims made in the abstract and introduction both through the- oretical proofs Section 2, and through experimental results in Section and Appendix G.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section 3.1, Section 4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
42

---

Justification:
We provide all assumptions and proof for Theorem 2 in Appendix C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the full details needed reproduce our experiments in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
43

---

Answer: [Yes]
Justification; We provide a zip file containing the code at submission which includes instructions in the README.md, md, and will provide open source access github after being published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ cttpsscccppbcccccccccccccccccccc wtwp///gueeeeeeeeeee geggde////uge /Ctee//doeeubmmic ssi iasionnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the full details needed to reproduce our experiments in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Standard errors are plotted graphs where appropriate (except for Figure 3a, where the standard deviation is plotted instead because the standard errors were too small to show). In Figure 3b, we show the regression coefficinet, p-value, and R² for our calibration plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
44

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the computational requirements of our experiments in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp://eeeip....eee ct/ ceccchhiscciellll ines?
Answer: [Yes]
Justification: Yes, we went over the NeurIPS code ethics and, to the best our knowledge, conform with all the guidelines given.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both the positive and negative social impacts of our work in Appendix B.I.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
45

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We have used publicly available datasets and models made available by huggingface and Meta. We have done our best to follow their guidelines for responsible use.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have done our best to cite original asset owners where applicable.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
46

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide open source code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
47

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
48


Reviews:
[{"review_id": "j5WZwgyu7G", "reviewer": "Reviewer_VGJP", "summary": "The paper proposes a method to estimate the hallucination rate for in-context learning (ICL) in a conditional generative model (CGM) from a Bayesian perspective. The authors assume the CGM samples from a posterior predictive distribution over a Bayesian model of a latent parameter and data. They define Posterior Hallucination Rate (PHR), along with True Hallucination Rate (THR) and Model Hallucination Rate (MHR), and provide theoretical proofs and empirical demonstrations to show that they can accurately estimate the true probability of hallucination.", "questions": "Q1. What constitutes an “in-capability” task and “out-of-capability” task?\n\nQ2. In Figure 10, why do the lines, especially for context length 50 graphs, stop earlier? What does that imply about the performance or behavior of the model at longer context lengths?\n\nQ3. The paper references Fong et al. [22] proposing Martingale Posterior distribution that uses posterior predictive. Are there any baseline results for this approximation that could be compared to PHR to evaluate which distribution better defined models?", "limitations": "The authors have addressed the positive and negative societal impacts of their work. It would be beneficial to further discuss and address the dataset choices and potential limitations, such as label diversity and number of categories, to provide a comprehensive view of the conditions under which the method has been tested. This would enhance the understanding of the method’s applicability and generalizability ensuring a more thorough assessment of its limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1. The paper tackles an important and well-known issue of hallucination in language models. By focusing on estimating the hallucination rate, the research contributes to improving the reliability and trustworthiness of outputs from CGMs. This makes the work highly relevant in the context of ongoing AI issues.\n\nS2. The paper introduces new metrics, particularly PHR, for future evaluation of model hallucinations. This can potentially advance methodology for assessing model output reliability.\n\nS3. The metrics are well-defined and the pseudo-algorithms help in understanding how the metrics are used.", "weaknesses": "W1. While the authors tested their methods on six datasets from various domains, the range of label content and types may be limited. This limitation could introduce confounding factors. For instance, tasks where one label can be a substring of another (e.g. “similar” vs “dissimilar”) might not fully represent the complexity of real-world scenarios.\n\nW2. The labels used in the datasets, such as “entailment” vs “not”, do not have a clear semantic meaning and relation like “positive” vs “negative” OR “subjective” vs “objective.” This absence of semantic relation might influence the ICL performance. Testing whether the semantic meaning of the labels affect performance and hallucination rate would strengthen the findings. A suggested approach is to use neutral labels such as letters (A,B,etc.) to determine if the lack of semantic meaning of the labels impacts the hallucination detection.\n\nW3. Related to W1, the datasets used in the experiments have either 2 or at most 4 categories. This limited variety raises questions about how the method performs with more complex classification tasks. For example, how would the method fare in English-to-French word translation, where the number of possible categories is significantly higher? Expanding the experiments to include tasks with a larger number of categories would provide a more comprehensive assessment of the method’s robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the effort made in reviewing our work. We appreciate that you have identified several strengths and recognize that your main criticisms concern evaluation and clarity of concepts. We offer clarifications, have run additional experiments, and have updated our manuscript accordingly.\n\n**W1: While the authors tested their methods on six datasets from various domains, the range of label content and types may be limited.**\n\nWe recognize the importance of understanding the posterior hallucination rate for closer to real-world scenarios and offer additional experiments. Here, we represent task labels by a set of semantically similar responses. For example, the SST2 task has two labels, *1* and *0*. The valid responses for the *1* label are [positive, favorable, good]. The valid responses for the *0* label are [negative, unfavorable, bad]. The favorable/unfavorable dichotomy is analogous to your similar/dissimilar example, representing a more realistic setting where there may be a set of valid responses. In this setting, we show that the PHR maintains its performance in predicting the error rate and MHR in [Figure Error](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/vgjp/w1_figure_error.png) and [Figure MHR](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/vgjp/w1_figure_mhr.png).\n\n**W2: The labels ... such as “entailment” vs “not”, do not have a clear semantic meaning and relation like “positive” vs “negative” ...**\n\nWe have run the entailment tasks (RTE and WNLI) under two additional settings to understand the effects of semantic meaning. First, we changed the negative label from *not* to *not entailment* to enforce semantic difference. This change did not have a noticeable effect on the results for these tasks. Next, we took your suggestion and changed the negative label from *not entailment* to *A* and the positive label from *entailment* to *B*. This change resulted in a significant difference in the in-context learning dynamics. As can be seen in the [Figure Error](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/vgjp/w2_figure_error.png) and [Figure Entropy](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/vgjp/w2_figure_entropy.png), the semantic information given by using entailment as a positive label helps the RTE task converge to a non-trivial error rate as the number of in-context examples increases.\nThis change, in effect, makes the tasks even more out-of-capability. Thus, while interesting, we think it is out of scope for the primary evaluation of our methods.\n\n**W3/W4: ... experiments to include tasks with a larger number of categories would provide a more comprehensive assessment ... . ... discuss and address the dataset choices and potential limitations ...**\n\nWe agree that this work will offer insights but argue that by limiting our perspective to in-context learning, we enable a rigorous definition of hallucinations and hallucination rates. Moreover, we provide a firm foundation to generalize to more complicated and less structured natural language settings, which we want to pursue in future work. Please see our response to reviewer Vp93 for a more detailed argument.\n\n**Q1: What constitutes an “in-capability” task and “out-of-capability” task?**\n\nWe consider a task in-capability for an LLM if in-context learning leads to better-than-random accuracy as the number of in-context examples grows. In contrast, an out-of-capability task would only yield trivial predictions and thus hallucinate frequently. As a general example of this distinction for most LLMs, sentiment analysis may be a task that is in-capability, and theorem proving may be a task that is out-of-capability. We make this distinction because the model predictive distribution is not a good approximation of the ICL posterior predictive for out-of-capability tasks, and the requisite assumptions are not satisfied. \n\n**Q2: In Figure 10, why do the lines, especially for context length 50 graphs, stop earlier? What does that imply about the performance or behavior of the model at longer context lengths?**\n\nThe lines end earlier because the PHR and THR are generally lower for longer context lengths. This trend is sensible—if the model were a perfect estimator of the reference distribution, then both quantities would converge to epsilon as the context length (number of in-context examples) increases. However, as we discuss in lines 211-218, the model distribution is subject to estimation divergences, and the difficulty of modeling the shrinking discrepancies between the ICL and model conditional distributions increases with the number of in-context examples. It is valuable to note that although the PHR is underestimated and increases the estimator error, the precision of the model is higher in this setting.\n\n**Q3: Are there any baseline results for [the Martingale Posterior] that could be compared to ...?**\n\nThe martingale posterior is a generalization of standard Bayesian posteriors. It proposes a way of propagating uncertainty by specifying conditional distributions of the form $p(y_{n+1}| y_{1}, \\dots, y_n)$  and then sampling data in a way similar to our algorithm: sample $y_{n+1} \\sim p(y_{n+1}| y_{1}, \\dots, y_{n})$, then condition on this sample, and sample $y_{n+2} \\sim p(y_{n+2}| y_{1} \\dots, y_{n+1})$, and so on. Notably, the conditional distributions $p(y_{n+1} | y_1, \\dots, y_n)$ need not be posteriors in the Bayesian sense but simply valid conditionals. The samples for sufficiently large $n$ are understood as if they came from a posterior. Although this algorithm is related to what we do, they do not offer a measure corresponding to the PHR, so it is not directly applicable as a baseline. Moreover, they implement their method with Gaussian copulas, which does not apply to our setting.\n\nWe hope this response addresses your concerns and look forward to discussing any points you would like more elaboration on in the next phase.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2YICqcDxbY", "reviewer": "Reviewer_yYgH", "summary": "The paper focuses on the in-context learning setting of generative AIs, such as large language models, and proposes a new definition for hallucination. It introduces a novel metric, PHR, along with a corresponding estimation method. Unlike traditional metrics, the proposed metric accounts for label ambiguity resulting from unclear task specifications.", "questions": "Please refer to the weaknesses.", "limitations": "Limitations have been well discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper highlights a significant issue with using error rates to evaluate hallucination, that the ambiguity on labels caused by the lack of task specification has been overlooked.\n- The definition and algorithm of PHR are logically sound and make sense to me.", "weaknesses": "- The notations are messy and confusing.\n- - There is misuse and abuse of uppercase random variables and lowercase instantiations. For example, the distribution of  $F$  is referred to as  $p(f)$  in lines 87 and 121, while it is referred to as  $p(F)$  in line 124. This problem is exacerbated in Theorem 1, making it hard to read.\n- - The notations PHR/THR can refer to both the definitions and the algorithms, which adds to the confusion. For instance, in the sentence on line 174, “we evaluate the PHR in a setting where we know the true mechanism f* so that we can compare it directly against the true hallucination rate (THR),” PHR appears to refer to the algorithm, while THR seems to refer to Definition 3.\n- The experimental setup is difficult to understand.\n- - For example, the experiments in Section 3.1 are supposed to evaluate the proposed method in a setting where $f*$  is known. However, what $f*$  is and how THR is calculated are not explained. To understand this, at least the way $p(y|x, f*)$  is set should be specified. Given the unused space at the end of the paper, there is no reason for the authors not to elaborate on the experimental settings.\n- - In Section 3.2, the authors attempt to justify the use of PHR by introducing another proposed metric called MHR. It is unclear how MHR could serve as a gold metric. Even if MHR is a gold metric, why not just use MHR?\n- Typos: Line 31 \"the\" should be deleted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and attention. We appreciate that you have identified several strengths, and recognize that your main criticisms are concerned with clarity and missing details. We respond to each of your comments below and have updated our manuscript accordingly.\n\n**W1.1: There is misuse and abuse of uppercase random variables and lowercase instantiations.**\n\nWe thank the reviewer for their comment concerning the inconsistent notation for the prior over mechanisms $p(\\mathrm{f})$. Indeed, there is a typo and for consistency with the rest of the paper, it should read $\\mathrm{F} \\sim p(\\mathrm{f})$. As pointed out by the reviewer, our goal was to use uppercase to emphasize that a quantity is being treated as a random variable and lowercase when it is simply a number or a realized value. As an alternative, we propose to use lowercase letters throughout the paper (for example using lowercase in lines 88 and 87) and only to use uppercase when strictly necessary to emphasize the fact that we are dealing with a random variable (for example in theorem 1). We plan to correct the paper accordingly if you find this to be satisfactory, but we are open to any alternative suggestions you may have.\n\n**W1.2: The notations PHR/THR can refer to both the definitions and the algorithms**\n\nWe recognize that using the PHR notation for both the definitions and algorithms is a source of confusion. We will update the manuscript to use $\\widehat{\\text{PHR}}$ when referring to the algorithm and $\\text{PHR}$ to refer to the definition.\n\n**W2.1.1: \"... in Section 3.1 ... what $f^\\*$ is ... [is] not explained.\"**\n\nWe briefly explain the data-generating process in Appendix E.1 but agree that we can be more explicit and we will include a clear definition in the main text. First, $f^*$ is going to be defined by a random ReLU neural network: $f^*_w(x) = w_2^\\top\\text{ReLU}(w_1 x)$, where $w_1$ and $w_2$ are instances of the $d\\times1$ dimensional random variable $W \\sim \\mathcal{N}(0, 1)$. Queries $x$ are instances of the uniformly distributed random variable $X \\sim \\mathcal{U}(-2, 2)$. Responses $y$ are then instances of the normally distributed random variable $Y \\sim p(y \\mid x, f^*) := \\mathcal{N}(f^*_w(x), \\sigma^2)$, with $\\sigma = 0.1$. Example datasets are shown in Figure 8a of the Appendix, where each color corresponds to a different sample of the network weights $w = \\{w_1, w_2\\}$.\n\n**W2.1.2: \" ... in Section 3.1 ... how THR is calculated ... [is] not explained.\"**\n\nWhile the calculation is included in the attached code, we agree that it should be included in the paper. The quantiles of $p(y \\mid x, f^*) := \\mathcal{N}(f_w^*(x), \\sigma^2)$ are computed analytically by $Q_{\\frac{\\epsilon}{2}}(f^*, x) := f_w^*(x) + \\sigma \\sqrt{2}\\text{erf}^{-1}(2(\\frac{\\epsilon}{2}) - 1)$ and $Q_{1 - \\frac{\\epsilon}{2}}(f^*, x) := f_w^*(x) + \\sigma \\sqrt{2}\\text{erf}^{-1}(2(1 - \\frac{\\epsilon}{2}) - 1)$. True hallucinations are then counted as $y$ samples that are either less than $Q_{\\frac{\\epsilon}{2}}(f^*, x)$ or greater than $Q_{1 - \\frac{\\epsilon}{2}}(f^*, x)$. The THR is then estimated as the empirical average over the response $y$ samples for a given query $x$. For a specific $f^*$, we illustrate examples of such hallucinations in panes 1 and 3 of Figure 2, along with the $\\epsilon$ confidence intervals of $p(y \\mid x, f^*)$ as the shaded blue region.\n\n**W2.2.1: In Section 3.2, the authors attempt to justify the use of PHR by introducing another proposed metric called MHR.**\n\nThe MHR is not proposed to justify the PHR. Instead, it is a metric to evaluate whether the PHR is operating as expected when we do not have access to the true distribution and thus are unable to calculate the THR. It is to be considered along with the error rate. As we describe in lines 242-247, the error rate is grounded in human-labeled responses but does not capture the subtlety of the true conditional distribution of responses given a query and a set of in-context examples. The MHR assumes that the model predictive distribution is true and compares the distribution of responses to a query $x$ given the original in-context examples and a set of generated examples to the distribution of responses to a query $x$ given a set of additional true examples. In expectation, these two distributions should be equivalent if things are working correctly.\n\n**W2.2.2: It is unclear how MHR could serve as a gold metric.**\n\nWe would not call the MHR a \"gold\" metric, because it is decoupled from the true distribution of responses to a query, and instead assumes that the model distribution under additional examples is true. Because we cannot calculate the \"gold standard\" THR, we report results for both the error rate and MHR instead.\n\n**W2.2.3: Even if MHR is a gold metric, why not just use MHR?**\n\nIt does not make sense to use the MHR instead of the PHR because it assumes that you have more in-context examples than are available. This is fine for an evaluation metric, but not ok for a predictor of hallucinations.\n\n**W3: Typos: Line 31 \"the\" should be deleted.**\n\nThank you, we have addressed this typo.\n\nWe hope that this response addresses your concerns and look forward to discussing anything that may remain unclear.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WgxCu5QvPq", "reviewer": "Reviewer_6dN5", "summary": "This paper presents a method for predicting the hallucination rate of in-context learning with conditional generative models.", "questions": "- Why not more LLMs used for experimentation?\n- What could be the right number of LLM call needed?\n- What parameters is the function of hallucinate rate? What are the optimal parameter? for which task? for which LLMs?", "limitations": "cf. weakness and questions", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- NA", "weaknesses": "- Unclear how many queries would be require to validate the approach, as this will be very dependent of task, context, and LLM - this is clearly a missing element of the evaluation\n- Evaluation on Llama 2 only - more is definitely require to validate the approach\n- No identification of where the hallucination might come from\n- Title is misleading - this is basically a hallucination identification method\n- Non cost effective as this require multiple LLMs calls", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our work. It is disheartening that you did not find it appropriate to attribute strengths to our work. We hope to convince you of what we believe constitutes a significant positive contribution. We have addressed your concerns below and look forward to a fruitful discussion.\n\n**Q1: Why not more LLMs used for experimentation?**\n\nWe have also run Gemma-2 9B on the \"in-capability\" SST2, Subjectivity, and AG News tasks. In this [figure](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/6dn5/figure1.png), we show that the PHR can accurately predict the empirical error rate for Gemma-2 9B. In this [figure](https://anonymous.4open.science/r/figures-phr-rebuttal-E611/6dn5/figure2.png), we show that the PHR accurately predicts MHR for Gemma-2 9B.\n\nWe believe that these responses answer your concerns. If there are any remaining sources of concern, we are happy to discuss them further.\n\n**Q2: What could be the right number of LLM call needed?**\n\nUsing the notation in Algorithms 1 and 2, $N$ (the number of generated examples), $M$ (the number of Monte Carlo samples), and $K$ (the number of generated examples) are ideally as large as possible because increasing them will improve the accuracy of the PHR estimator. Considerations like the computational budget will influence the values chosen. In addition to these considerations, $N$ may need to be small enough so that the generated examples do not overflow the maximum context length seen during model training, which could result in non-sensical generations. The number of generated examples $K$ may also be influenced by the choice of $\\epsilon$. For example, evaluating the PHR at $\\epsilon=0.1$ would require at least $K=10$ response samples, whereas evaluation at $\\epsilon=0.02$ would need at least $K=50$ response samples.\n\nIn the context of our language model experiments, we have shown that a setting of $N=5$, $M=10$, and $K=50$ yields good results. Moreover, our ablations summarized in Table 1 of the Appendix do not give strong evidence that increasing these values (i.e. increasing computational cost) for these experiments leads to a significant gain in performance. We are happy to add this discussion to the main paper or the Appendix.\n\n**Q3: What parameters is the function of hallucinate rate? What are the optimal parameter? for which task? for which LLMs?**\n\nWe discuss the parameters $N$, $M$, and $K$ above. Now we consider the parameter $\\epsilon$. An intuitive way to think about the $\\epsilon$ parameter is from a decision-making perspective. Algorithmic decision-making system design includes error tolerances. For example, a chat application may be acceptable if it hallucinates only 5% or 10% of the time (i.e., it provides useful correct answers in 95% or 90% of user interactions). When the model accurately defines the task distribution, the PHR directly addresses such considerations. For example, if the tolerated rate of hallucination is 5%, then a PHR estimate greater than 0.05 could inform the deferral of a response. With this constraint, the system designer would choose $\\epsilon \\leq 0.05$ to estimate hallucination rates at least as small as 5%. Related to the above discussion, we can see that the computational cost grows with the tolerance stringency. If the tolerated hallucination rate is 10%, we need $K \\geq 10$; but if it is 1%, we need $K \\geq 100$ to resolve the desired rate. Safety-critical situations may justify the additional computational cost. \n\nIdeally, these considerations are independent of task and model. However, we are transparent about the accurate estimation of the PHR depending on the model defining a conditional distribution that closely approximates the in-context learning conditional distribution. We maintain that this is a significant and self-contained first step in estimating hallucination rates in general, and we leave it to future work to address the case when tasks are \"out-of-capability.\"\n\n**W5: Computational complexity**\n\nIt is true that our method, like other sampling-based methods, carries the computational cost of generating multiple responses to a given query and the cost of producing additional examples.\nWe contend that this additional computational cost is justifiable for at least the following reasons:\n\n1) The method holds for transformer models that define probabilities factorized as in equations (1-3). Model classes that satisfy this requirement exist, like conditional neural processes [24] and prior fitted networks [56], which are implementable with smaller Transformer models. \n\n2) We believe the method has scientific value. It helps determine if the uncertainty in an LLM's probabilities is due to a lack of in-context data or if it arises from inherently stochastic answers (i.e., the difference between reducible and irreducible uncertainty). We expect a highly entropic answer in both cases, but the latter would result in a low hallucination rate estimate, whereas the former would result in a high estimate. Future research could explore the scenarios where high entropy results in reducible or irreducible uncertainty. This analysis could clarify whether a model answers incorrectly due to genuine data ambiguity or insufficient in-context data. For such scientific inquiries, the computational constraints are less of a concern.\n\n3) Our paper provides a comprehensive underlying theory. Though our initial estimator of the PHR is computationally complex, more efficient methods exploiting clever solutions to the integrals presented could be developed in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UgY7hhz1m7", "reviewer": "Reviewer_Vp93", "summary": "This paper presents a Bayesian interpretation of in-context learning. This interpretation enables us to calculate the hallucination rate. In other words, by considering in-context examples as observations, the posterior distribution can be computed and the hallucination rate is derived. Numerical experiments verify the practicality of the interpretation.", "questions": "See Weaknesses.", "limitations": "This paper discussed broader social impact.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "1. The hallucination is one of the most important problems of large language models. This paper explored a theoretical understanding of the problem in in-context learning.\n\n2. The idea to interpret the in-context examples as the observation, described in Section 2.1,  is original and interesting.", "weaknesses": "In summary, I suggests that the authors clarify the focus of this paper either on the hallucination rate for NLP tasks or on the error rate for general in-context learning tasks.\n\n1. As discussed in L222, the proposed theory cannot be applied as is in NLP tasks and an approximated metric is required. Although the introduction of this paper focused on NLP tasks, the usefulness of the proposed theory in this area is limited, as reported in L265 with Figure 6.\n\n2. Hallucination is a complex phenomenon with a wide range of causes [1]. As mentioned in the above weakness, the effectiveness of this theory is limited to the synthetic regression task. In light of these, I am concerned whether it is appropriate to call the phenomenon examined in this paper a hallucination rate rather than an error rate.\n\n3. The claim of Section 2.2 is to use the cumulative probability of the token generation probability distribution as the confidence for token y. However, the use of cumulative probability as the confidence of token y is a common technique in NLP, for example in nucleus sampling [2]. To demonstrate the importance of the theory presented in this section, I consider it necessary to show its universality when other statistics S are used. Otherwise, I also suggest moving this section to the appendix as noted in the following weakness.\n\n4. L146 says `Algorithm 1 can be understood intuitively without appealing to any Bayesian arguments’. I agree with this statement, and I consider that the necessity of the presented theory depends largely on the discussion in Appendix B.1. Because the appendix is just supplementary material and cannot play a role in justifying the main claim, I suggest the authors should revise the structure of the paper.\n\n[1] Huang+, A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, arXiv:2311.05232.\n\n[2] Holtzman+, The Curious Case of Neural Text Degeneration, ICLR2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read and comment on our work. We are glad you found the ideas original and the contribution excellent. Although we have some disagreements about the weaknesses, we think your concerns are important and aim to clarify them below.\n\n**W1: As discussed in L222, the proposed theory cannot be applied as is in NLP tasks and an approximated metric is required.**\n\nWe appreciate the opportunity to clarify things here. The MHR *is not* an approximation of the PHR that enables the application of the theory to language tasks. Instead, it is an evaluation metric used to help assess whether the PHR estimator is working as intended when we don't have access to the ground truth data-generating process and thus cannot compute and compare against the THR. We discuss the shortcomings of the empirical error rate as an evaluation metric in L242-247 and offer the MHR as a complimentary evaluation metric to address those shortcomings. We present both results to serve as a proxy for the THR. We stress that the MHR is an evaluation metric and that the PHR, as presented, is directly applicable to NLP tasks.\n\n**W2.1: \"...the effectiveness of this theory is limited to the synthetic regression task...\"**\n\nIn light of our above clarification, we believe we have provided evidence that our theory is appropriate for NLP tasks. Notably, the PHR can accurately track the error rate for in-capability tasks, as shown in the first three panes of Figure 6. Moreover, when the model is assumed to define the in-context learning distribution, the PHR accurately tracks the MHR in all settings, as shown in Figure 5. Finally, there are non-NLP models where the theorem applies directly, for example, conditional neural processes [24] and prior fitted networks [56].\n\n**W2.2: \"Hallucination is a complex phenomenon ... [is it] appropriate to call the phenomenon examined ... a hallucination rate rather than an error rate.\"**\n\nWe agree that hallucination is a complex phenomenon. We believe that progress toward a holistic understanding of hallucination requires a principled deconstruction of the phenomenon into manageable steps. And we argue that one cause of hallucination is the *context* being \"underspecified,\" even if the *model* can perform a given task. For example, imagine asking, \"Who is the president?\" An LLM distribution over possible responses may include all the different ways to convey Joe Biden, Emmanuel Macron, the president of Microsoft, or a historical president of a fictional society. If your $f^*$ corresponds to semantic equivalents of \"Joe Biden,\" all other responses are considered hallucinations. This notion of hallucination is consistent with the recent Nature paper, \"Detecting hallucinations in large language models using semantic entropy.\" \n\nThe intriguing thing about such hallucinations is their resolution through improving the context. For example, asking, \"It is August 6th, 2024, who is the president of the United States?\" shrinks all uncertainty concerning *which* president. This example is already quite complex to start thinking about from mathematical first principles, so we distill the essence of it and tackle such hallucinations through the lens of in-context learning. This perspective allows us to take a rigorous mathematical approach to defining hallucinations and hallucination rates while still being able to study the phenomenon using the practical setting of LLMs. Moreover, it offers a firm foundation to start building back toward understanding this type of hallucination in less structured language tasks.\n\n**W3.2: To demonstrate the importance of the theory ... it [is] necessary to show its universality when other statistics S are used. Otherwise, I ... suggest moving this section to the appendix ... Algorithm 1 can be understood intuitively without appealing to any Bayesian arguments ...  the authors should revise the structure of the paper.**\n\nWe acknowledge these concerns but disagree that the paper should be restructured. \n\n1. The PHR, its theoretical justification, and its extension to modern conditional generative models are the primary contributions of this paper.\n\n2. Though we have not found a generalization of our result within a larger class of statistics, we have used similar techniques to derive an estimator for the mutual information to quantify uncertainty. Including the theory in the main text can inspire other researcher to use similar techniques in their algorithms.\n\n    Specifically, we can use a similar (but not identical) technique to derive an estimator for the expected entropy $\\mathbb{E}_{p(f|x, D_n)} \\left[ \\text{H}(Y \\mid f, x) \\right]$.\n\n    *Theorem: Assume that the conditions of Theorem 1 hold for $F$ and $(X, Y)$.* Then,\n$$\n\\mathbb{E} \\left[ \\text{H}(Y \\mid f, x) \\right] = \\mathbb{E} \\left[ \\lim_{N \\rightarrow \\infty} \\text{H}(Y \\mid (x_i, y_i)^N_{n+1}, x, D_n) \\right].\n$$\n\nwhere the first expectation is taken with respect to $p(f \\mid x, D_n)$ and the second with respect to $p((x_i, y_i)^\\infty_{n+1} \\mid D_n, x)$\n\n3. There are non-NLP models where the theory does apply directly, such as conditional neural processes [24]---which demand exchangeability from the stochastic process---or prior data-fitted networks [56].\n\n**W3.1: The claim of Section 2.2 is to use the cumulative probability of the token generation probability distribution as the confidence for token y ...**\n \nWe understand the author's concerns but respectfully disagree. We presented a rate using the CDF of probabilities because it is intuitive and widely used, which we believe strengthens and clarifies the theory.\n\nAdditionally, we emphasize that we use the CDF differently than in nucleus sampling. Rather than examining $p(y \\mid x, D)$ as done in nucleus sampling, we propose an algorithm for examining $p(y \\mid x, f)$ where $f$ is an implicit latent. Nucleus sampling mixes uncertainty about $p(f \\mid D)$ with uncertainty about $p(y \\mid x, f)$, whereas our method disentangles them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful and constructive feedback. We are particularly encouraged by the recognition of several strengths across different aspects of our paper, which we summarize below.

**Reviewer VGJP** acknowledges the significance of our work in addressing hallucination in language models, highlighting its relevance to current AI reliability concerns. They commend the introduction of the Posterior Hallucination Rate (PHR) for evaluating model output reliability, noting that this well-defined metric and accompanying pseudo-algorithms enhance comprehension and potential future applications in the field.

**Reviewer yYgH** appreciates our identification of the significant issue of label ambiguity in using error rates to evaluate hallucination, noting its often-overlooked impact due to the lack of task specification. They also find the definition and algorithm for the Posterior Hallucination Rate (PHR) to be logically sound and well-reasoned.

**Reviewer Vp93** recognizes the importance of addressing hallucination in large language models and commends our theoretical exploration of this issue for in-context learning. They find the interpretation of in-context examples as observations described in Section 2.1 to be original and intriguing.

These remarks are encouraging and affirm the core strengths of our theoretical, methodological, and empirical contributions. In response to the concerns and suggestions raised, we have reviewed our manuscript to address each point individually. We have made specific revisions and clarifications to refine our theoretical and methodological explanations. Each reviewer's feedback is considered to ensure our responses are thorough and reflective of the effort made to improve our manuscript's quality. Once again, we thank all reviewers for their valuable feedback. We look forward to discussing our responses with each of you further.

Meta-review: The paper proposes a method to estimate the hallucination rate for in-context learning (ICL) from a Bayesian perspective. It takes a conditional generative model (CGM), a dataset, and a prediction question, and estimates the probability of hallucination. The Bayesian interpretation assume the CGM samples from a posterior predictive distribution over a Bayesian model of a latent parameter and data. They define Posterior Hallucination Rate (PHR), along with True Hallucination Rate (THR) and Model Hallucination Rate (MHR), and provide theoretical proofs and empirical demonstrations to show that they can accurately estimate the true probability of hallucination. Numerical experiments verify the practicality of the interpretation.

Overall, the work is addressing a significant and timely problem, presents a technically sound solution, and introduces new metrics that would potentially help the whole community more reliably evaluate LMs from here on. 

Reviewers raised concerns about the limitation of the evaluations of their method, namely, the six datasets used are of relatively narrow scope, and more "synthetic" than real, which makes its applicability to real-world cases unclear. Two of the reviewers raised their scores during rebuttal. There's one extremely negative reviewer whose behavior is questionable, and for that reason I'd like to downweight their opinion.

Judging from the overall contribution, I tend to recommend accept.
Decision: Accept (poster)