Hallucination Detection for Generative Large Language Models by Bayesian Sequential Estimation
Authors: Xiaohua Wang
Yuliang Yan
Longtao Huang
Xiaoqing Zheng
Xuanjing Huang
Venue: EMNLP 2023 Main (2023)
Keywords: Hallucination dectection
fact checking
Bayesian sequential estimation
generative large language models
TL;DR: A Bayesian-based method for hallucination detection in LLMs.

Abstract: Large Language Models (LLMs) have made remarkable advancements in the field of natural language generation. However, the propensity of LLMs to generate inaccurate or non-factual content, termed "hallucinations", remains a significant challenge. Current hallucination detection methods often necessitate the retrieval of great numbers of relevant evidence, thereby increasing response times. We introduce a unique framework that leverages statistical decision theory and Bayesian sequential analysis to optimize the trade-off between costs and benefits during the hallucination detection process. This approach does not require a predetermined number of observations. Instead, the analysis proceeds in a sequential manner, enabling an expeditious decision towards "belief" or "disbelief" through a stop-or-continue strategy. Extensive experiments reveal that this novel framework surpasses existing methods in both efficiency and precision of hallucination detection. Furthermore, it requires fewer retrieval steps on average, thus decreasing response times.

Hallucination Detection for Generative Large Language Models by Bayesian Sequential Estimation
Xiaohua Wang 12, Yuliang Yan 1.2 Longtao Huang", Xiaoqing 2
1.2,,, Xuanjing Huang 1.2
School of Computer Science, Fudan University, Shanghai, China
Shanghai Key Laboratory of Intelligent Information Processing AAlibaba Group
(xiaohuawang22, y1yan21 } fudan, edu. cn
(zhengxa, xjhuang]ee @ē fuda.udd.... edu. cn
Abstract
Classifirr
Facilty
Large Language Models (LLMs) have made re- markable advancements in the field of natural language generation. However, the propensity of LLMs to generate inaccurate or non-factual content, termed "hallucinations", remains a sig- nificant challenge. Current hallucination detec- tion methods often necessitate the retrieval of great numbers of relevant evidence, thereby in- creasing response times. We introduce unique framework that leverages statistical decision theory and Bayesian sequential analysis op- timize the trade-off between costs and bene- fits during the hallucination detection process. This approach does not require a predetermined number of observations. Instead, the analysis proceeds in a sequential manner, enabling an expeditious decision towards "belief" or "dishe- lief" through a stop-or-contimee strategy. Exten- sive experiments reveal that this novel frame- work surpasses existing methods in both effi- ciency and precision of hallucination detection. Furthermore, requires fewer retrieval steps on average, thus decreasing response times'.
Prkc mek Retiane onnidor:
Decenen Ouroeciaal whifeer Intikoe coditical crakawe su Biyedie seguestil
Eviieeao
Halloctmatinn
Classiferr
Boyesian Rink
Decision
Hallucination
Figure I: Evidence- hallucination detection based on Bayesian sequencial analysis. Prior works often re- trieve a predetermined number of documents and over- looked the variabilityy among different inputs. In our framework, (1) we first retrieve one document from the search engine; (2) External evidence and the claim generated by the LLM are input into the hallucination classifier to calculate the veracity score; Then we make decision on whether to stop observing and make veracity judgment based on the available evidence or to continue observing additional evidence. This decision is made based on the Bayesian risk decision and sequential analysis. (3) stopping and making a determination carries higher risk, choose continue observing additional evidence; (4) Otherwise, we choose to deter- mine the veracity of the claim.
1 Introduction
In the era of information overload and the prolif- eration of misleading or false information, auto- matic fact checking has become an essential tool for verifying the veracity of claims and combating misinformation. Large Language Models (LLMs), such as GPT-4 (OpenAI, 2023).PaLM (Chowdhery et al., 2022) and LLaMA (Touvron et al., 2023), have made significant advancements in the field of natural language generation (NLG). However, the inherent tendency of LLMs generate inaccurate non-factual content, commonly referred to as "hallucinations" (Bang et al., 2023; Ji et al., 2022), continues to present a significant challenge.
et al., 2023), they assume that LLMs are capable of producing similar responses and contain consis- tent facts related to given concept. However, this method necessitates many samplings of the LLM. leading to tremendous costs in practical applica- tions. Another commonly employed strategy fact checking entails retrieving relevant evidence from various sources, such as texts (Thome et al., 2018; Augenstein et al., 2019), knowledge graphs (Kim et al., 2023) and the webs (Chen et al., 2023; Kamoi al., 2023) to validate challenge claims. These methods usually retrieve predetermined number of documents across diverse cases. How- ever, the effectiveness of these approaches heavily depends on the selection of the number of docu- ments to retrieve. While these methods can ensure
One previous work utilizes the sampled-based approach for detecting hallucination (Manakul
'Our cnde is available at https://giitee com/ stw.ang2//halllccinaattmeeeee

---

a certain level of consistency, in practical applica- tions, different claims may require varying amounts of external evidence for verification. This should be determined by the nature of the claim itself and the documents retrieved, rather than predetermined pa- rameters. The optimal number of external evidence sources needed to validate a claim should be dy- namically determined based on the specific context and characteristics of the claim, taking into account factors such as complexity, unbiguity, and the avail- ability of relevant information. This study aims to investigate the feasibility of gradually collecting relevant information and adopting a data-driven approach to document retrieval in the context of hallucination detection for large language models (LLMs) in real-world scenarios. By employing such an approach, we enhance retrieval effi- ciency and reduce unnecessary retrieval attempts.
framework outperforms sample-based methods in terms of efficiency and aceuracy in hallucination detection. the task of sentence-level hallucina- tion detection, our framework achieves comparable performance to the baseline approach. However, in the passage-level task, we observe an improvement of 6. 5.13% in terms of Spearman's 's correlation coef- ficient. Furthermore, across different sets of hyper- parameter configurations, our framework consis- tently reduces the number of retrieved documents compared to the approach with a fixed number of retrieved documents. These results indicate the superiority of our approach in detecting hallucina- tions with improved efficiency and precision, show- casing its potential for practical applications.
Related Work
2.1 Hallucination Detection for Large Language Models
In this paper, we propose novel framework that leverages Bayesian sequential analysis (Wetherill, 1961; Arrow al., 1949; Carlin et al., 1998) to en- hance the retrieval efficiency for detecting whether claims generated by LLMs are hallucinated or fac- tual in the wild. Recognizing that a single sentence may contain a mixture of factual and nonfactual information, we first decompose claim into sub- claims, each focusing on single entity and its associated attributes (Kamoi et al., 2023). In the subsequent stage, we employ a search engine to retrieve relevant documents for each subclaim in- dividually. These retrieved documents, in conjunc- tion with their corresponding subclaims, are then input into a classifier to estimate their veracity (Nie et al., 2020). As illustrated in Figure 1, we con- sider the retrieved documents as sequence and make informed decisions whether additional information should be retrieved for given case via Bayesian sequential analysis through a stop-or- continue strategy. Once the information gained enough to decide whether claims are ballucinations, we stop and make an evaluation. Otherwise, we retrieve additional documents and make a decision in the next step. Finally, we aggregate the assess- ments of the subclaims to determine the overall veracity of the input claim.
Several studies have explored the phenomenon of hallucination (Su et al., 2022; Lee et al., 2022; Dai et al., 2022) in large language models and have pro- posed various approaches for its causation and de- tection. Bang et al. (2023) conducted an evaluation of ChatGPT's hallucination on fact-checking test sets and knowledge- -aaeed QA tasks, highlighting the model's s susceptibility to extrinsic hallucination compared to intrinsic hallucination (Ji et al., 2022).
In terms of detection, Kadavath et al. (2022) and Mündler et al. (2023) conducted a study to investigate whether language models can assess the validity of their own claims, aiming to explore the self-evaluative capabilities of language models. Azaria and Mitchell (2023) employed the internal state of LLMs to determine the truthfulness state- ments by leveraging the hidden-layee activations as input to the classifier. Manakul et al. (2023) in- troduced sample-based detection method for hal- lucination detection. They emphasized that when language model is well-acquainted with a given concept, the sampled responses are likely to be similar and contain consistent facts. This method relies on analyzing the consistency of the generated samples to identify potential hallucinations. Unlike their approach, we employ an evidence -based fact- checking method, where we verify the correctness of claims based on external documents.
To evaluate the effectiveness of our framework, we conducted experiments on the dataset of Self- CheckGPT (Manakul et al., 2023), containing 1, 9088 sentences from 238 articles generated by GPT-3, each labeled with its corresponding verac- ity. The experimental results demonstrate that our
2.2 Evidence-based Fact Checking
Evidence-basee fact-checking has gained signifi- cant attention as an effective approach to combat

---

misinformation and ensure information accuracy. FEVER (Thorne al., 2018) utilized Wikipedia ev- idence for fact verification, while Augenstein et al. (2019) collected evidence from fact-checking web- sites and proposed method that combines veracity predictions and evidence ranking.
retrieving external documents. We treat the re- trieved documents as sequence and dynamically determine whether to continue or stop the retrieval process based on the accumulated evidence. This stop-or-continue strategy allows for efficient and effective information retrieval, optimizing the pro- cess by adapting to the evolving evidence and min- imizing unnecessary retrieval attempts.
Recent approaches include the use of knowledge graphs (KG) by Kim et al. (2023), where a classi- fier predicts relations and hops related to the claim, and related sequences are retrieved from the KG evidence. Kamoi et al. (2023) put forward dataset constructed using real-world claims and evidence. Their work focused enhancing the performance of entailment models by reducing the complexity of claims through a decomposition pro- cess. By breaking down the claims into simpler components, they aim to facilitate a more effective evaluation of entailment and improve the overall performance of the models. Chen et al. (2023) presented an automated pipeline for fact-checking real-world claims, retrieving raw evidence from the web. However, these methods retrieve a fixed number of documents for all instances, whereas our work focuses on adaptively retrieving a varying number documents for specific instances.
3 Method
In this paper, we propose a framework for detect- ing hallucinations of LLMs by leveraging external evidence.
Given claim generated by the LLMs, we first decompose it into subclaims that contain basic knowledge and proceed to evaluate the veracity of each subclaim individually.
Subsequently, we employ search engine trieve web documents for each subclaim external evidence. Rather than predefining the number of documents retrieve, we adopt a dynamic retrieval process that involves retrieving one document at one step and determining the need for additional retrieval based on the content of the retrieved docu- ment.
Then, utilize specific classification model to compute the entailment score between each doc- ument and subclaim which represents the extent how well the evidence supports a given claim. The entailment score is processed into features, and a Naive Bayes classifier (NBC) is applied to calcu- late the probability of the subclaim's veracity based on these features.
2.3 Bayesian Sequential Analysis
Sequential analysis (Wald, 1947) grounded de- cision theory and Bayesian inference (Gunst and Shcherbakova, 2008; Box and Tiao, 1973), pro- vides a valuable framework for making informed decisions based on accumulating evidence. The goal is to select decisions that minimize the ex- pected loss, considering the associated costs. The general approach Bayesian sequential decision problems involves backward induction (DeGroot, 2005; Berger, 2013), also known as stochastic dy- namic programming (Ross, 2014). However, as the complexity of the problem increases, this method can become computationally intensive. To address this issue, Cardillo and Fu (1968) proposed an ap- proximation by assuming that the subsequent stage is terminal, leading suboptimal solution. Ex- panding on the backward induction method, Brock- well and Kadane (2003) introduced a grid approx- imation approach. This technique estimates the expected loss at each decision time, significantly reducing computation time to linear complexity relative to the number of decision stages.
Next, we adopt a stop-or-conting strategy, em- ploying Bayesian risk decision to choose among three options: (1) stopping the retrieval process and classifying the subclaim as faculty, (2) stopping the retrieval process and classifying the subclaim as hallucination, (3) or continuing to retrieve addi- tional documents and make choice in the next step. After making the decision to stop the retrieval process and determine the veracity reaching the maximum retrieval times, we obtain the probability that the subclaim is faculty.
Finally, by aggregating the probabilities of the subclaims' veracity, we evaluate the overall verac- ity of the original claim and assess its potential hallucination.
In the following subsections, we discuss in more detail our method. In Section 3.1, we demonstrate the process of claim decomposition. In Section
In this paper, we utilize Bayesian sequential anal- ysis to implement a stop-or-continue strategy for

---

3.2, we describe details of using search engine to retrieve web documents. Section 3.3 explains how the entailment score is calculated and processed into features required by the Naive Bayes Classifier (NBC). Section 3.4, we provide specific de- scription of the stop-or-continee strategy's details. Finally, in Section 3.5, we elaborate on how the veracity probabilities of subclaims are aggregated.
Algorithm 1: Hallucination detection
Input: C
A claim generated by LLMs for hallucination detection;
m1 (0) Initial probability of C' being factual;
K
Maximum retrieval times;
CFA Cost of false alarm;
Cm Cost of miss;
Cretrieve: Cost of retrieving document;
Output: Pfactuai(C)) Probahility being
factual:
{C C CC. Cla
for 1 do
3.1 Claim Decomposition
3
" -
4
while &: do
Given a claim generated by LLMs, the direct ve- racity validation of the entire claim becomes chal- lenging due to the presence of multiple knowledge instances within the generated content. So we first decompose the original claim into subclaims, with each subelaim specifically focusing entity and its associated attributes. Kamoi al. (2023) show that similar method allows for a more fine-gra mne-grained analysis, enabling us to examine and evaluate each entity and its attributes individually. Simultancously, we replace pronouns sentences with their corresponding entities. This approach is implemented to enhance the retrieval of relevant evidence from web documents, making the search process more effective.
s
E" <- RetrieeveDocumen(((":
<- CalEntaaimenttFeatureeCC C²):
Ti + NBC(#T 1), fn):
Retop min((1 (n))Cus
(1 To(n)))PA):
Reontinac
Crstrieve + Brups (B (n + 1)):
single
10
if Rstop(m) <
Hoontsoue (n) then
break:
else end
n t n n
end
Pfactual = TI (i);
end
P Paactmal (C) = min P Prantuat
Return: Proctuat(C)
We employ zero-shot prompt-based approach using the GPT-3.5 (text-daviie text-davinci-0033 model to perform claim decomposition and the full prompt can be seen in Appendix A. For sentences that have not been successfully decomposed, we solely perform entity replacements for pronouns.
3.3 Entailment Score Calculation and Diseretization
We utilize specific classification model to calcu- late the entailment score between each document and subclaim which represents the extent to how well the evidence supports the given subclaim. A higher entailment score indicates stronger support from the external document for the corresponding subclaim. In this study, we use the DeBERTa-v3 (He et al., 2021) model fine-tuned on Natural Lan- guage Inference (NLI) dataset (Laurer et al., 2022) to perform two-label classification (entailment and not entailment). Let C refer to subclaim and E refer retrieved document, the entailment score sCC, E) is the probability of the entailment label obtained by inputting the claim and evidence into DeBERTa-v3. Therefore, sCC, E (0, 1) and (C, + 1 if the evidence completely sup- port the claim.
3.2 External Documents Retrieval
To validate the real-world efficacy of our frame- work, we utilize the subelaims derived from the previous decomposition step to retrieve external documents from search engines. We input the sub- claims as queries into the Bing Search API? to obtain URLs. We then utilize the newspaper 3k2 to extract web page content from the retrieved URLs. Note that the external documents are treated as a finite sequence with maximum length K and are retrieved one by one. Therefore, we don't pre- determine fixed number of retrieval times. We disregard web content that is inaccessible, such as PDF files or protected websites.
To overcome the limitation of the model's input length, we divide the document E into text spans {E,, E.). Each text span comprises m tokens, with a step size of (n < m) tokens. This means that we divide the document into text spans of length m tokens and move the segmentation
hhttp://wwe http://wwwwiiiiosoooe microsoft. con/ conm/en-us/bingge hompt//o/o//eess//////asss///// bing-weed www-wwewc--eeeeeeeeeeeeeeee Phttps: / thub. com com//codellcca//newssppee

---

window by n tokens for each step. We calculate the entailment score between each text span and the corresponding subclaim, and select the highest entailment score as the entailment score for the original document:
assumption of independence between the features. In our approach, we assume that the content gen- erated by the large language model (LLM) has an equal prior probability of being hallucinations or factual information and set #1 (1) = 0.5.
One of the difficulties in calculating the above probability is the conditional probability (01) and P(]n+1|00) We use a sampling method to compute this term required in Equation 4. We sampled set that consists factual claims and 8 nonfactual claims. For each claim, we re- trieved a piece of external evidence and computed the entailment features of the claim and evidence. Then we estimate P(fn-100) and P(J++1(0)) by counting the factual and nonfactual claims (plus smoothing term) that have the entailment feature equal to fn+1-
s(C, E) max(s((CCCCCcc E.))
(1)
This approach ensures that the document's over- all entailment score reflects the strongest evidence found within it.
Score Discretization
The entailment score is indeed continuous value ranging between 0 and 1. To simplify the repre- sentation of the document characteristics, we trans- form the entailment score into discrete entailment feature based on the assumption that two docu- ments with similar entailment scores possess the same feature:
Bayesian Sequential Analysis
For subcliam C, external document comes one by one in sequence. At step < K, we retrieve docu- ments {E*, E², EE'} and calculate the proba- bility that C is factual using Equation 4. Then we have three options:
f(C, = [00 (C,
(2)
where is the continuous value of entailment score and f E {0, 2, 8, 9} is the discrete entail- ment feature given claim C and evidence E. The entailment feature is then input into Naive Bayes Classifier (NBC) to calculate the veracity of the claim in Equation 4.
Decide 0 01: We stop observing additional evidence and determine the subclaim C is fac- tual.
Decide lo: We stop observing additional evidence and determine the subelaim C' is a hallucination.
3.4
Stop-or-Continue Retrieval Strategy
Given subcliam C that needs to be evaluated for veracity, and a finite sequence of retrieved external documents {E1, F2, ... _=K,, we use the entail- ment features of each document to access the verac- ity 0 of C. We denote 0 = On to represent that C is a hallucination and 0 = 02 if C' is factual. At time n < K, we use Ii (n) to represent the probability of C' being factual, given features f1, f2, fn:
Keep test: The previous evidence is not enough to evaluate the veracity of the sub- claim C and decide to retrieve additional doc- uments.
If we choose the first two options, we stop observ- ing additional external evidence. However, if we choose to continue with "keep test",, we retrieve additional document and make the choice among the three options again step n + 1. We assume that the document sequence is finite, and when the maximum step is reached, we no longer choose "keep test".
क (n) = P(0 = 01l
(3)
where the fi.n is the entailment features of Elm and C' calculated using Equation 2.
7i (n) the prior probabil- ity and calculate the posterior probability mi (n(n+) + based on ii (n) and fnt1:
At step + 1,
We make decisions based on the minimization of Bayesian risk. In this study, we consider three possible risk costs:
Ti (n(++)) = (I - (n) P(f++1
n (1(n)
8.)
(4) which comes by recursively applying Bayes' rule (Appendix B). This the probability that C' is fac- tual, given features fú, f2.... fn, futt, with an
CFAi Cost of false alarm, we declare 0 0-0 0g but C' is factual. In this condition, we incur a cost when mistakenly classifying factual in- formation as hallucinations.

---

Cm: Cost of miss, we declare 0 01 but C is hallucination. In this condition, we incur cost when mistakenly classifying hallucina- tions as factual information.
last, one must consider the probability of data values that have not yet been observed. This re- quires working the problem backward (from the max step K back to time 0), a process referred to as backward induction (DeGroot, 2005; Berger, 2013). However, computing Equation 6 is bit awkward, as we need to consider all discrete values for subsequent steps. The computation complexity that implements the backward induction solution increases exponentially in the number of backward steps. Therefore, we use the truncation approxima- tion made one-step-ahead (Cardillo and Fu, 1968) which leads to the suboptimal solution. In each step. the choice is made based on the assumption that the next step is the final step, meaning that only the first two options are considered. In this case, Equation 5 still hold, but R(n 1) is replaced by Rstop (n 1) in Equation 6:
Cretrieve: Cost of retrieve an external evi- dence. In this work, we assume that the cost of retrieving external evidence is equal for each retrieval.
We choose to stop only when the cost of stopping is less than the minimum expected cost of retrieving one more document. If we stop and declare 0, the cost due to "miss" would be (1(n) (n)CMr and the cost due to "false alarm" would be Tr(n)CrA Therefore, if we make the decision to stop at step n, the expected risk (ntop(n) is:
[n) min((1 - - (1 (n())CCrA)) (5)
where Rstop(p) (n) is the expected cost of choose the stop option. If the cost due to "miss" is smaller than the cost due to "false alarm", we declare 0 01 and determine that C is factual. Otherwise, we declare B 8g and determine that C' is a hallucination. we choose to retrieve an additional document and make a choice in the next step. because we do not know the retrieved document En+1 at step n, so we calculate the minimum expected risk:
Reontinuc (n) = Cretrieve +
Σ fnt1=0
Ratop(n 1))-(a))
(8)
By using this approach, the computational complex- ity decreases from exponential to linear. Specifi- cally, it reduces from O(10k) to O(K).
3.5 Aggregation
Finally, we calculate the original claim's probabil- ity of factual by considering the minimum proba- bility of factual among the subclaims:
Reontioue (n) Cretriesec
((n+1))P(fati) (6)
Pracuall (C) = min Practual (C*)
(9)
where Sn+1 is the entailment feature of En+1 and C. Reontinue the minimum expected risk to re- trieve an additional document and make a choice in the next step. is the overall minimum expected cost of the three options:
Where Pfactual is the probability of factual and the original claim C is decomposed into sub- claims {C", C² .......... .Ch.. This is based on the assumption that if any portion of the original claim is nonfactual, consider the whole sentence to be hallucination.
R(n)= min (eoontinue (m), (n))
(7)
Experiments
We use this recursive equation to obtain the op- timal solution. We stop and determine the prob- ability of C' being factual is TI (n) and evaluate the veracity of C' if this action results in minimum expected cost (n) Specifically, if Resoop (n) results in the cost of miss, we consider C' to be factual. Otherwise, we consider C' to be hallu- cination. On the other hand, if the measurement of (eomtinue yields the minimum expected cost, the process continues and an additional document F. is retrieved.
We conducted experiments on the hallucination detection classification task. First, we compared our framework with SelfCheckGPT to evaluate the effectiveness of our approach in hallucination de- tection at both the sentence-leved and passage-level. Next, we performed a comparative experiment us- ing fixed number of retrieved documents ex- ternal evidence to validate the efficiency of our framework in improving retrieval efficiency and reducing the number of external evidence. Finally, we conducted ablation study to evaluate the neces- sity of the claim decomposition step.
To obtain the optimal Bayesian decision in the nite sequential setting at any step other than the

---

Sentence-leved (AUC-PR)
Passage level (Corr.)
Method
Evidence Num
Nonfact
Factual
Acc
Pearson
Spearman
Soff-Detection
31.01
Score
20
81.96
44.23
58.18
55.90
SelICheckGPT
w/ QA w/Unigram (max)
20
84,26
48.14
61.07
59.29
20
85.63
58.47
64.71
64.91
Combination
60
87.33
61.83
69.05
67.77
C 14, CFa 24
3,03
82.42
57.01
80.24
71.37
Our Framework
Cs 28, CFa 96
6.22
86.45
61.96
82.39
81.18
Table Follow prior work (Manakul et al., 2023), we report the Area Under the Precision- Recall Curve (AUC-PR) the sentence-level and the Pearson and Spearman's 's rank correlation coefficient w.r.t human judgments at the passage-level. Additionally, we report the number of external evidence. For SelfCheckGPTT, "Evidence Num" represents the number of samples obtained from GPT-3, For our framework, "Evidence Num represents the average number of retrieved external documents. Furthermore, for our framework, also report the sentence-level accuracy. BERTScore, QA and Unigram are three metrics to evaluate the consistency of samples in SelfCheckGPT.
4.1 Experiment setup
correlation coefficient and Spearman's rank corre- lation coefficient w.r.t human judgments.
Dataset We use the dataset SelfCheckGPT (Manakul et al., 2023) which contains 1, 1,908 sen- tences from 298 article generated by GPT-3. Each sentence is labeled with one of the three verac- ity labels: "Accurate", "Minor Inaccurate" and "Major Inaccurate". In the experiments, "Ma- jor Inaccurate" and "Minor Inaccurate" sentences are labeled as mon-factual class, while sentences labeled as "Accurate" are considered factual class. Baseline SelfCheckGPT samples from GPT-3 multiple times and utilizing consistency between samples to identify ballucinations. We use Self- CheckGPT the baseline to evaluate the perfor- mance of our framework for hallucination detection sentence-level and passage-level. We also query ChatGPT itself to determine whether it constitutes hallucination as an additional baseline(Mündlee et al., 2023).
For our framework, we also report average search time and accuracy (ACC) for sentence-level detection tasks. To validate the advantage of our method in terms of sample cost, we report the num- ber of external evidence. For SelfCheckGPT, we consider the number of times they perform sam- pling from GPT-3 as the quantity of external evi- dence. For our framework, we report the average number of retrieved documents at the sentence- level. To validate our advantage in terms of sam- pling cost, we assume that SelfCheckGPT incurs the same cost for one GPT-3 sample as it does for searching external document.
4.2 Results of Hallucination Detection
The results of our experiments can be found in Table 1, which provides overview of the per- formance of our framework in hallucination de- tection. When the average number of retrieved documents per sentence is 3.05, our framework out- performs SelfCheckGPP with single consistency metric, which rely on 20 external evidence, in terms of passage-level performance. As the average num- ber of retrieved documents increased to 6.22, our framework shows slightly lower performance in Nonfact detection but uses less external evidence. In Factual detection and passage-level evaluation, we achieve significant improvements over the base- line. These results demonstrate the effectiveness of our framework hallucination detection.
Hype Hyperparameters Settings For calculation of the conditional probability of Equation 4, we used = 200 and smoothing term of 1 in the experi- ments reported here. In section 4.3, we adjust CFA and CM
framework for hyperparameter ex- periments. For the experiments conducted outside that section, we set the value of the three possible costs of Bayesian sequential analysis CrA = 24, Cs 14 and Cretrieve = 1. For the length text spans and the step size during document segmenta- tion, we set m = 400 and - 100. Additionally, we set the maximum retrieval time K- = 10.
Evaluation Metrics Following Sel/CheckGPT, we report AUC-PR for sentence-level detection tasks. For passage-level detection, we calculate the average score of all sentences passage to obtain the passage's veracity score. Passage-level ranking performances are measured by Pearson
4.3 Effectiveness for Reducing Retrieval Times
In this section, we conducted comparative experi- ments using a fixed number of retrieved documents.

---

The results obtained from the experiments are presented in Figure 2, which illustrates the rela- tionship between the average number of retrieved documents in our framework and the Spearman's correlation coefficient at the passage-level halluci- nation detection. Based on the results, we observe that as the average number of retrieved documents in our framework approaches the fixed number of retrievals, there is higher Spearman's correlation coefficient at the passage-level hallucination de- tection. This implies that our framework achieves better performance in hallucination detection with fewer retrievals compared to the fixed number retrievals. These findings indicate that our frame- work effectively reduces the number of retrievals re- quired while maintaining or even improving the per- formance of hallucination detection. This demon- strates the efficacy of our approach in enhancing retrieval efficiency and reducing search frequency in the context of hallucination detection tasks.
Sprrmnns Spramnnsanne Ramice
Nave Bayas Gassifiar (NBC)
Average
Maximum
Bayesian secue dal andlyss
The number of socuments retreveed
The performance of Bayesian sequential anal- ysis method on ranking passages (Spearman's 's) versus strategies with fixed number retrieval documents.
Figure 2:
For the case of using a fixed number of retrieved documents, we considered three methods to aggre- gate the features of these documents:
Sentence- (AUC-PR)
Naive Bayes Classifier (NBC): We treated the documents fixed number of features and used Naive Bayes classifier, as described in Eq.4, for hallucination detection.
Method
Nonfact
Factual
Acc
Decomposition
80.04
53.71
79.19
w Decomposition
82.42
57.01
80.24
Table
The performance at the sentence-level by directly using the original sentence for hallucination detection.
Average: For each document, compute entailment score and then average the scores of all the documents to obtain the veracity score of the claim.
4.4 The Effect of Claim Decomposition
In this section, we validate the contribution of the claim decomposition step to the overall per- formance. We consider the scenario where only the decomposed subclaims used for document trieval and directly perform hallucination detection on the original sentence.
Maximum: We calculated an entailment score for each document and selected the highest score as the veracity score for the sentence. This approach is based on the assumption that hallucinations usually lack any supporting ev- idence in external documents. Therefore, if there documents that validate sen- tence, we consider the sentence factual.
The results in Table demonstrate that when subclaims are only used for document retrieval, the performance of the framework decreases to some extent at both the sentence level and passage level. This indicates that directly determining the verac- ity of sentence that contains multiple pieces of knowledge is challenging. It highlights the neces- sity of the problem decomposition step.
The Cas and CFA hyperparameters represent the costs involved in making Bayesian risk deci- sions. Different values Cи and CFA can lead to different decisions by the strategy, for the same Ti (n) and Cretrieve, thereby influencing the aver- age number of retrieved documents. To evaluate the efffctiivness of our framework in improving re- trieval efficiency and reducing search frequency, we conduct comparative experiments using the fixed number of retireved documents. We randomly se- lect five sets of Cas and CFA values and compare the results with the fixed search times approach.
Conclusions
In this study, we propose a framework that utilizes Bayesian sequential analysis for detecting hallu- cinations in large language models (LLMs). We consider the retrieved documents as sequence and employ con-orr- strategy to make in-

---

formed decisions on whether retrieve additional information for given case. By applying Bayesian sequential analysis, our framework achieves com- petitive results on annotated GPT-3 responses, sur- passing sample-based methods and retrieval strate- gies with fixed number of search documents. Ad- ditionally, we validate the practical applicability of our framework by conducting searches for relevant documents the web, demonstrating the feasi- bility of our approach in corroborating the claims made by LLMs with real-world information.
James O Berger. 2013, Statistical decision theory and Bayesian analysis. Springer Science & Business Media.
G. E. P. Box and George C. Tiao. 1973. Bayesian infer- ence in statistical analysis. International Statistical Review, 43:242.
Anthony E Brockwell and Joseph B Kadane. 2003. A gridding method for bayesian sequential decision problems. Journal of Computational and Graphi- cal Statistics, 12(3):566-555
Gerald P Cardillo and King-Sin Sun Fu. 1968, On subop- timal sequential pattern recognition. IEEE Transac- lions on Computers, 100(8):789-792.
Limitations
In this study, the implemented strategy amalga- mates individual pieces of information extracted from search engine at each iterative stage. An inherent limitation of this methodology is the po- tentially extended duration required for inference. For future studies, it would be better to consider the integration of information drawn from multiple documents concurrently, which could significantly enhance the speed the inference process.
Bradley Carlin, Joseph Kadane, and Alan E Gelfand. 1998. Approaches for optimal sequential decision analysis in clinical trials. Biometrics, pages 975.
Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Dur- rett, and Eunsol Choi. 2023. Complex claim veri- fication with evidence retrieved in the wild. ArXiv, Abs/2305.11885
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinod- kumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan. Hyeontaek Lim. Barret Zoph, Alexander Spiridonov. Ryan Sepassi. David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz. Erica Moreira, Re- won Child, Oleksandr Polozav, Katherine Lee, Zong- wei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. ArXiv,
Acknowledgements
The authors would like to thank the anonymous reviewers for their valuable comments. This work was supported by National Natural Science Foun- dation of China (No. 62076068), and Shanghai Municipal Science and Technology Project (No.
21511102800)
References
Kenneth J Arrow, David Blackwell, and Meyer A Gir- shick. 1949. Bayes and minimax solutions of sequen- tial decision problems. Econometrica, Journal of the Econometric Society, pages 213-244.
Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. Multife: real-world multi-domain dataset for evidence- based fact checking of claims. In Conference on Empirical Methods Natural Language Processing.
Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. 2022. Plausible may not be faithful: Probing object hallucination in vision-language pre-training. In Conference of the European Chapter of the Asso- ciation for Computational Linguistics.
Amos Azaria and Tom M. Mitchell. 2023. The inter- nal state of an Ilm knows when its lying. ArXiv. abs/2304.13734.
Morris DeGroot. 2005. Optimal statistical decisions. John Wiley & Sons.
Yejin Bang, Samuel Cahyawijaya, Naycon Lee, Wen- liang Dai, Dan Su. Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hal- lucination, and interactivity. ArXiv, abs/2302.04023.
M. C. M. Gunst and O. Sheherbakova. 2008. Asymp- totic behavior of bayes estimators for hidden markov models with application to ion channels. Mathemati- cal Methods of Statistics, 17:342-356.

---

Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with gradientddieentangled embedding shar- ing. arXiv preprint wrXiv:2111.0954.
James Thome. Andreas Vlachos. Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: large-scale dataset for fact extraction and verification. ArXiv, abs/1803 603.05355.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang. Wenliang Dai, Andrea Madotto, and Pascale Fung. 2022. Survey of hallucination natural language generation. ACM Computing Surveys, 55:1 38.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur'elien elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Fili Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
Abraham Wald. 1947. Foundations of a general the- ory of sequential decision functions. Econometrica, Journal of the Econometric Society, pages 279-313.
Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. Wice: Real-world en- tailment for claims in wikipedia. arXiv preprint arX1v:2303.01452.
GB Wetherill. 1961. Bayesian sequential analysis. Biometrika, 48(3/4):281-292.
Jiho Kim, Sungjin Park, Yeonsu Kwon, Yohan Jo, James Thome, and Edward Choi. 2023. Factkg: Fact veri- fication via reasoning knowledge graphs. ArXiv, abs/2305.06590.
A Claim Decomposition Prompts
The zero-shot prompt we use to decompose the claim is shown Figure 3.
Moritz Laurer, W v Atteveldt, Andreu Casas, and
Kasper Welhers. 2022. Less annotating, more mussifying--ddleessingg the data scareity issue of su- pervised machine learning with deep transfer learning and bert-nli.
Fellow thes the the tripl dakmriter:
orignnal
promosm ndvering
Naycon Lee, Wei Ping. Peng Xu, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Factuality enhanced language models for open-ended text generation. ArXiv, 0hs/2206.006224.
them.
Step Decompase text into bast knowildge, requeing unch piace knowiddgg centain only and onevat that and
step 2: Lapres each decomponed prece knseeladge in natural langraage sentence.
Step Cutpo sextence in step
theorigh giri
Potsawee Manakul, Adian Liusie, and Mark John Fran-
Output the result in JSON lomnet, the key JRON is 'stell' Ske?? sky and sieyli, wlue cupt should
inipp
cis Gales. 2023. Selfcheckgpt: Zero-resource black- box hallucination detection for generative large lan- guage models. ArXiv, abs/2303.0882
miggnall seneeee in the oheorigi test the index the sentense the original
Outpet:
Niels Mündler, Jingxuan He, Slobodan Jenko, and Mar- tin Vechev. 2023. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. ArXiv, a5552
Figure 3: The zero-shot prompt for decompose the sen- Lence of the passage into subclaims.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela, 2020. Adversarial NLI: A new benchmark for natural language under- standing. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. As- sociation for Computational Linguistics.
B Derivation of Equation 4
Given a subcliam C' that needs to be evaluated for veracity, and finite sequence of retrieved externa documents {E', E² _K} }. we use the entail- ment features of each document access the verac- ity 0 of C. We denote hallucination and A, if C' is factual. time n < K, we use कi (n) to represent the probability of being factual, given features fi, f2. f2.-.... fu:
OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Og to represent that C is
Sheldon Ross. 2014. Introduction to stochastic dy- namic programming. Academic press.
Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022. Read before generate! faithful long form question answering with machine reading. ArXiv, abs/2203.00343.
T (n) = P(A -8i1ftsn)
(10)

---

where the fi:n is the entailment features of Eim and C calculated using Equation 2.
(m + 1)
(11)
P(0 = 01lfim41)
(12)
P(0r, fim+1)
(13)
P(fim41)
P(0s)P(f2n++1(11)
(14)
P(f [(:) + P(fim41 P(Bb)
P(8:)11221 P(
(15)
P(O)) P(f[(1) P(A))))))_((()))))))))() P P(N
P( 81)
P(f+1 (() 01) P(f (0) Im 0n)
(16)
P(f++1 (
P(4+1 [1)) P() P( 1 0o) P(Oo
(17)
P(()))(m++100)) 11) = (1 06) + ni (n)
(18)
Deriving Equation 15 from Equation 14 based on the assumption of independence between the fea- tures.
C Self-Detection Prompts
For querying ChatGPT itself to see whether it's hallucination, we use the prompt shown in Figure 4.
1 give you one statement, please conclude whether the statement nonfactual with Yes No.
Statement: [Tlyptthesis]
The zero-shot prompt for querying ChatGPT itself to see whether it's ballucination.
Figure 4:


Reviews:
[{"review_id": "UM2sRrmPOj", "reviewer": "Reviewer_6dtQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and valuable comments.\n\nQ1: It can be interesting to apply the method on simpler claims, that do not require the decomposition step; although the paper does try to evaluate the effect of this step.\n\nR1: We have considered your feedback regarding the necessity of decomposition for certain simpler claims. In response to your suggestion, we conducted a thorough examination of the impact of decomposition on simple claims. The results indicate that, when dealing with randomly selected 200 simple claims, there was no evidence of excessive decomposition. During our investigation, approximately 60% of instances remained in an undecomposed state due to their inherent simplicity, aligning with the predefined decomposition criteria specified by the prompt \" focusing on a single entity and its associated attributes \"\n\nHowever, it is worth noting that we did identify some instances where appropriate decomposition should have occurred but did not manifest despite its expected applicability. Our module exhibits a bias towards avoiding over-decomposition.\n\nIn Section 4.4 of our paper, we delve into an ablation study that demonstrates the critical role of the decomposition step within the proposed approach. This step significantly enhances performance by a significant margin of 3.3% in terms of accuracy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper applies an interesting technique to an important problem in the era of large LMs: claim verification and hallucination detection. The paper is clearly written, and provides a good introduction to the employed techniques for those without prior knowledge. ", "reasons_to_reject": "I do not see major issues with this paper. It can be interesting to apply the method on simpler claims, that do not require the decomposition step; although the paper does try to evaluate the effect of this step. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes  a framework for detecting hallucinations of LLMs by leveraging external evidence. Specifically, given some text generated by the LM, a number of supporting documents is retrieved via web search. The number of retrieved documents is chosen by Bayesian inference, allowing to tailor it to the degree of confidence at this stage. Specifically, at each stage it possible to decide whether to continue, stop and classify the claim as false, or stop and classifying it as true. The decision is based on features derived from a naive-bayes classifier predicting the veracity given the retrieved document and the claim generated by the LM. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mQOdX31V4h", "reviewer": "Reviewer_rSTF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and valuable comments.\n\nQ1: The entailment scores are still calculated using another deep learning model, which can still be silver-standard. It would be better to point out how reliable Deberta-v3-NLI is in terms of your task, with some expert annotations.\n\nR1: This is a very good suggestion. In response, we conducted a comprehensive evaluation of Deberta-v3-NLI as per your recommendation. Our evaluation encompassed multiple established natural language inference (NLI) datasets that are widely recognized in the field. In addition to these benchmark datasets, we created an additional dataset comprising 300 examples, designated as \"our_test\" This new dataset was created by randomly selecting 150 claims from our test set, and for each of these selected claims, we thoughtfully gathered both supporting and opposing texts. Note that Deberta-v3-NLI was not fine-tuned on the datasets by our team throughout this evaluation process. The accuracy of Deberta-v3-NLI is listed in the table below.\n| Dataset| mnli_test_m|mnli_test_mm|anli_test|anli_test_r3|ling_test|wanli_test|our_test|\n|:-----------:| :-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n| Accuracy|0.912|0.908|0.702|0.64|0.87|0.77|0.823| \n\nIt is crucial to highlight that we have taken into account the unreliability of Deberta-v3-NLI's predicted entailment scores within the framework of Bayesian sequential analysis. In our approach, these predicted scores serve as priors for the stop-or-continue retrieval strategy, which are then subject to gradual refinement through the accumulation of newly retrieved evidence, resulting in the establishment of more robust posterior probabilities. This integration of a Bayesian framework significantly boosts the performance on hallucination detection, effectively addressing the initial uncertainties associated with Deberta-v3-NLI's entailment predictions.\n\nQ2: 2.1 (baselines) seems weak. There should be many 2023 works for hallucination detection. Also there're only limited baselines in Table 1. What about some intuitive baselines like querying ChatGPT itself to see whether it's hallucination? This should be a strong one.\n\nR2: In the section of related work, we have discussed nearly all pertinent studies. Yes, there would be many 2023 works for hallucination detection. It is noteworthy that a considerable portion of these works became available either shortly after our submission or in the days immediately preceding it. \nTaking into account your valuable suggestion, we proceeded to execute an experiment based on the baseline you recommended. It employs ChatGPT itself to assess the presence of hallucination, akin to the method outlined in [1]. The experimental results are listed in the following table, and the results indicated by \"Self-detection\" highlight a notably inferior performance (drop about 0.5 in accuracy) by ChatGPT on hallucination detection. This finding aligns with a recent investigation [1], wherein the authors asserted that \"The latest LMs still cannot handle factuality directly without relying on grounded knowledge.\" \n\n|     Method  | Sentence-level Accuracy|\n|:-----------:| :-------------:|\n| Self-Detection|31.01|\n| Our framework|82.39|\n\nReference:\n[1] Niels Mündler, Jingxuan He, Slobodan Jenko, and Mar-tin T. Vechev. 2023. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. ArXiv, abs/2305.15852.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Good intuition to restrict the number of retrieved documents and use Bayesian risk decision.\n- Solid writing and introduction to the whole methodologies.\n- Experimental results seem promising. ", "reasons_to_reject": "- The entailment scores are still calculated using another deep learning model, which can still be silver-standard. It would be better to point out how reliable Deberta-v3-NLI is in terms of your task, with some expert annotations.\n- 2.1 seems weak. There should be many 2023 works for hallucination detection. Also there're only limited baselines in Table 1. What about some intuitive baselines like querying ChatGPT itself to see whether it's hallucination? This should be a strongo one.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper propose an LLM hallucination detection framework leveraging external evidence. The pipeline includes breaking LLM generations into sub-claims, dynamic evidence retrieval using search engine, a classification module to determine how likely the evidence supports the claims, coupled with a stop-or-continue strategy based on Bayesian risk decision. The authors conduct experiments on the SelfCheckGPT dataset and find that the detection accuracy can be increased a lot with fewer retrieved documents. Some ablations also demonstrate the effectiveness of Bayesian risk decision in thie pipeline. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fkcqEu9UDx", "reviewer": "Reviewer_o6Kk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and valuable comments.\n\nQ1: The proposed method is complex and might not be easy to reproduce. For example, the method relies on GPT-3.5, Bing Search API, DeBERTa-v3, and Naive Bayes Classifier. The sensitivity of the method to some key hyper-parameters, such as C_FA and C_M, is also not presented.\n\nR1: We are committed to releasing our source code and data to the public along with thorough and detailed documentation. We would like to emphasize that each of the components we present is indispensable in realizing effective hallucination detection within the framework of Bayesian sequential analysis, although there is flexibility in terms of their specific implementation.\n\nWe propose the following procedure to determine the values for the hyperparameters C_FA and C_M. Initially, a subset of k samples (where k = 3 in our experiments) is randomly selected, and subsequently, manual annotations are applied to capture the claims, evidence, and decisions from the stop-or-continue trial. Through this process, we refine the potential ranges within which C_FA and C_M values lie.\n\nThe following table show the means and variances of sentence-level AUC-PR and passage-level correlation. These metrics are evaluated across five distinct pairings of C_FA and C_M, selected randomly within their refined intervals (i.e., about several hundred pairs from C_FA = [10, 50] and C_M = [25, 100]). Our analysis reveals that the impact of the two hyperparameters on performance appears to be relatively insensitive when sampled randomly from the specified ranges, as determined by a limited number of initial samples. It is noteworthy that this study employed only three samples to establish the initial ranges for C_FA and C_M. However, it is important to note that in practical scenarios, increasing the number of samples for range determination leads to narrower intervals and consequently enhanced performance.\n\n|       | Evidence Num| Nonfact|Factual|Accuracy|Pearson|Spearman|\n|:-----------:| :-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n| $\\text{Average}_{\\text{var}}$|$\\text{5.97}_{\\text{0.06}}$|$\\text{86.37}_{\\text{0.13}}$|$\\text{62.35}_{\\text{0.07}}$|$\\text{82.19}_{\\text{0.31}}$|$\\text{80.80}_{\\text{0.11}}$|$\\text{73.65}_{\\text{0.57}}$|\n\nQ2: The complexity of the method might hinder the efficiency of the detection process. While the authors indicate that their method requires fewer document retrievals, it would be valuable to directly compare the detection time with alternative approaches to assess its efficiency in practice.\n\nR2: Incorporating the assessment of whether the outputs from a Language Model (LLM) are hallucinatory or not will undoubtedly introduce additional costs. These incremental costs primarily encompass both the frequency of LLM interactions (e.g., ChatGPT) and the number of webpages retrieved. The goal of this study is to optimize the trade-off between costs and benefits during the hallucination detection process within the framework of Bayesian sequential analysis. \n\nA closely related work, SelfCheckGPT (Manakul et al., 2023), necessitates an average of 60 interactions with ChatGPT for each input, as opposed to a solitary interaction required for the decomposition process in our approach. Furthermore, our method needs an average retrieval count of 6.22 webpages. The computational overhead associated with webpage retrieval is much lower compared to the expenditure involved in ChatGPT interactions. Therefore, our method reveals superior computational efficiency when contrasted with the baseline. In Section 4.3, we also show that our method achieves comparable performance with a reduced number of retrievals. This affirms the efficacy of our approach in achieving the desired outcomes in a resource-efficient manner.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents good experimental results.\n- The proposed Bayesian sequential analysis for hallucination detection is novel, and I haven't seen it before.", "reasons_to_reject": "- The proposed method is complex and might not be easy to reproduce. For example, the method relies on GPT-3.5, Bing Search API, DeBERTa-v3, and Naive Bayes Classifier. The sensitivity of the method to some key hyper-parameters, such as $C_M$ and $C_{FA}$, is also not presented.\n- The complexity of the method might hinder the efficiency of the detection process. While the authors indicate that their method requires fewer document retrievals, it would be valuable to directly compare the detection time with alternative approaches to assess its efficiency in practice.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper aims to tackle the hallucination detection problem. The authors first decompose a claim into subclaims using GPT-3.5. They then employ a search engine (Bing Search API and newspaper3k) to retrieve relevant documents for each subclaim individually. Subsequently, they calculate the subclaim's veracity using an NLI model (DeBERTa-v3). Next, they adopt a stop-or-continue strategy to decide whether to stop the evaluation of a subclaim or not. Finally, they evaluate the overall veracity of the original claim and assess its potential as a hallucination.\n\n----Post-rebuttal-----\n\nThe author has only partially addressed my concerns; I am still concerned about the comparison of detection times. I can only improve my soundness score to 3.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a Bayesian-based method for hallucination detection in LLMs. Reviewers agree on the merit of the proposal and mostly raise minor issues.
Decision: Accept-Main