SAC$^3$: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency
Authors: Jiaxin Zhang
Zhuohang Li
Kamalika Das
Bradley A. Malin
Sricharan Kumar
Venue: EMNLP 2023 Findings (2023)
Keywords: Hallucination
semantic consistency
blackbox
large language models
confidence

Abstract: Hallucination detection is a critical step toward understanding the trustworthiness of modern language models (LMs). To achieve this goal, we re-examine existing detection approaches based on the self-consistency of LMs and uncover two types of hallucinations resulting from 1) question-level and 2) model-level, which cannot be effectively identified through self-consistency check alone. Building upon this discovery, we propose a novel sampling-based method, i.e., semantic-aware cross-check consistency (SAC$^3$) that expands on the principle of self-consistency checking. Our SAC$^3$ approach incorporates additional mechanisms to detect both question-level and model-level hallucinations by leveraging advances including semantically equivalent question perturbation and cross-model response consistency checking. Through extensive and systematic empirical analysis, we demonstrate that SAC$^3$ outperforms the state of the art in detecting both non-factual and factual statements across multiple question-answering and open-domain generation benchmarks.

SAC Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency
Jiaxin Zhang', Zhuohang Li,, Kamalika Das Bradley Malin Sricharan Kumar
Intuii Al Research Vanderbill University VVanderbilt University Medical Center {jiaxin_zaang, kamal ika_das,, Jettharan_kumrrreenttut com (zhuohang. li, b. m.aalin@vvaderbbit...
Abstract
the output sequences. However, these metrics re- quire access token-level log probabilities, which are not available in commercial black-box LMs like ChatGPT or Bard that only offer limited API access. To overcome this limitation, recent studies explore sampling- based approaches for approxi- mating uncertainty estimation (Lin et al., 2023) through establishing connection between confi- dence and self-consistency (Manakul al., 2023; Mündler et al., 2023). The underlying premise of this principle is that LMs are more inclined generate consistent responses when high probabil- ities are assigned to tokens the answer, which, in turn, implies level of factuality. In contrast, inconsistent responses are more likely to contain hallucinations. To operationalize this concept, cur- rent approaches are designed sample multiple responses from the LMs for given question and then compose a hallucination score for each sen- tence based on the level of response consistency.
Hallucination detection is critical step toward understanding the trustworthiness of modern language models (LMs). To achieve this goal, we re-examine existing detection approaches based on the self-consistency of LMs and un- cover two types hallucinations resulting from 1) question-leve and 2) model-level, which cannot be effectively identified through self- consistency check alone. Building upon this discovery, we propose novel sampling-basee method, i.e., semantic-aware cross-check con- sistency (SAC3) that expands on the principle self coliiconsit checking. Our SAC ap- proach incorporates additional mechanisms detect both question-level and model-level hal- lucinations by leveraging advances including semantically equivalent question perturbation and cross- model response consistency check- ing. Through extensive and systematic empiri- cal analysis, demonstrate that SAC outper- forms the state of the art in detecting both non- factual and factual statements across multiple canssiinnnnswering and open-domain genera- tion benchmarks.'
Key Observations. this work, we investigate the relationship between the self-consistency LMs and the occurrence of hallucinations in a di- verse range of tasks. Our investigation indicates that while seff-inconsistency in LMs often coin- cides with hallucination, self-consistency does not necessarily guarantee factual answers, shown in Figure 1. Our findings challenge the notion that celf-consistency alone can serve as reliable indi- cator of veracity, as is demonstrated that LMs can exhibit various tiers of hallucination that elude de- tection through self-consistency checks. One such tier is question-level hallucination, where LMs con- sistently generate incorrect answers in response to specific questions (c.g.. Table 1). We reveal that by reformulating the questions, it is possible mitigate such instances of hallucinations. Addi- tionally, our work further reveals the existence of model-level hallucinations, whereby different LMs show discrepancies their propensity for halluci-
1 Introduction
Large-scale pre-trained language models (LMs) have demonstrated exceptional adaptability across a diverse array of natural language tasks that re- quire generating open-ended responses based on user prompt comprehension (Zhao et al., 2023). However, prominent LMs like GPT (Brown et al., 2020) and PaLM (Chowdhery et al., 2022), often exhibit tendency produce exceedingly confi- dent, yet erroneous, assertions commonly referred hallucinations. This phenomenon signifi- cantly impedes their applicability in domains where factual accuracy of utmost importance. Hallucinations can be detected through the assis- tance of metrics that capture the uncertainty about
All resources are available at http::://githul intuit/sac3.

---

Self-checking
Consictency
Inconsistency
Non-Factual
Consistency Factuality?
Inconsiatency = Non-factuallty?
I pi smaller than 3.2?
Was there ever US sonator that represented the Mississippi and whose alma mater was University Chicago
Answer
No. pi approximately
3.14159, which greater
X
Is pi smaller than 3.2?
Non-Factual
As October 2021, there has been US sonator that reprosented the state Mississippi and whose alma mater was University of Chicago,
Answer
No. (appproximatell equal to
X
14159) greater than 3.2.
Self Consistency Check?
who represented the state Mississippi and whose alma of Chicago, His namne was Thad Cochran
pl smaller
Answer
X
No, (n) not smaller than 3.2.
Yes Factual
University
X
Is pi smaller than 3.2?
Yes, that represented the state of Mississ ippi and whose alma was University of Chicago
Answer
No, (n) is greater than 3.2, ls
X
X
approsimataly 3.14159.
Figure I: Key observation: solely checking the self-consistency LLMs is not sufficient deciding factuality. Left: generated responses to the question may be consistent but non-factual. Right: generated responses may be inconsistent with the original answer that is factually correct.
observe cases where smaller LMs are capable of correctly answering questions for which larger LMs hallucinate. To- gether, these findings accentuate the need to con- sider model-specific characteristics when assessing the occurrence of hallucinations.
nation. Surprisingly,
others provide correct answers. Integrating these cross-checking extensions into our approach signif- icantly improves its ability to detect hallucinations that go beyond self-consistency, thereby providing a more comprehensive assessment of the presence of question-level and model-level ballucinations.
Effectiveness of SAC We demonstrate the effec- tiveness of SAC in multiple classification and open- domain generation QA datasets. In the classifica- tion QA tasks, our approach achieves a high AU- ROC score of 99.4% and 97. 97.0% respectively, which significantly outperforms the self-consistency base- line. the case of open-domain generation QA tasks, our approach achieves an AUROC score of 88.0% and 77.2%, which substantially improves upon the self-consistency baseline by a large mar- gin (+13.8%, and +6.7% respectively).
Proposed Approach. Motivated by these ob- servations, we introduce SAC new sampling- based approach utilizing semantic-aware cross- check consistency to improve the detection hal- lucinations in black-box LMs. An overview of our approach is provided in Fig. 2. To address question- level hallucination, we introduce mechanism that perturbs semantically equivalent questions eval- uate the consistency of LMs' responses across vari- ants of the same question. By examining the gener- ated answers these perturbed questions, we are able to identify cases where the LM consistently provides incorrect responses to specific question, which is indicative of a question-level hallucination. Furthermore, we address model-level hallucination by introducing cross model response consistency checking, which involves comparing the responses of different LMs the same of questions. By identifying discrepancies between the responses different models, we are able to pinpoint cases where certain models exhibit hallucinations while
2 Related Work
Hallucination in LMs. The issue of hallucina- tion in language models (LMs) has gained signifi- cant attention due to its negative impact perfor- mance and the risks it introduces in various natural language processing (NLP) tasks, such machine translation (Zhou et al., 2020), summarization (Cao et al., 2022), dialogue generation (Das et al., 2023), and question answering (Zhang et al., 2023a; Zheng

---

Method
Input
Model
Output
Consistoncy
Factuality
"No, 3691 is not prime number as 10
0: *Is 3691 prine number?"
Targa
divisible by and 13."
rumber divisible by and 13."
3691 is
Q 3681 prime number?"
Target
consistency
"No, 3691 is not prime number. It can be
Q: 3691 prine number?"
Target
divided evenly
and
Q1: *is the nunber 3691 prine?"
Target
"Yes, the number 3691 15 prime." "Yes, only factors 3891
factors of
Cross-Q consistoncy
3691 itself and one?"
Target
one."
Q3: "Can 3691 only be divided
"Yes, 3691 is prime number and can only divided by
Taggtt
and
Cross-M
Q: 3881
number?"
Verifier
"Yes, 3691 is prime number."
consistency
Q1: *Is the nunber 3681 prine?"
Verifier
"Yes."
Cross-M
only factors
Verilier
"Yes, the only factors of 3691 are and itself."
3691 itself and one?"
consistency
Q3: "Can 3691 only be divided
Verifier
"Yes, 3691 only divided by
by and 3691?"
Table 1: An illustrative example of seff-consistency, cross-question consistency, and cross-model consistency check. The original question and answer are "Is 3691 a prime number?" and "No, 3691 is not a prime number. divisible by and 13", respectively. Each row presents set of sampled QA pairs along with its consistency regarding the original answer, and the predicted factuality of the original answer.
et al., 2023b; Dhuliawala al., 2023). Recent sur- vey (Ji et al., 2023; Zhang et al., 2023c; Ye al., 2023) and evaluation benchmarks (Liu et al., 2021; Li et al., 2023a; Yang et al., 2023) have highlighted the importance of addressing this issuc. Previous research has explored hallucination evaluation us- ing confidence-basee approaches (Xiao and Wang, 2021; Varshney et al., 2023; Chen and Mueller, 2023) that require access to token-level log prob- ability (Kuhn et al., 2023; Cole et al., 2023) or supervised tuning (Agrawal et al., 2023; Li et al., 2023b) that relies on internal states of the LM. How- ever, these methods may not be applicable when only API access to the LM is available (Agrawal et al., 2023). Another approach involves retrieving knowledge from external databases tackle hallu- cinations (Ji et al., 2022; Zheng et al., 2023a; Peng al., 2023; Zhang al., 2023b). contrast to these studies, our work focuses on detecting halluci- nations in open-domain QA tasks using black-box LMs, without relying on external resources.
consistency, it becomes challenging regard LMs as reliable trustworthy systems. Recent studies employ self-consistency to detect hallucinations based on pretrained LMs (Manakul et al., 2023) and instruction-tune LMs (Mündler et al., 2023). Although these methods exhibit promising accu- racy several specific tasks, potential failures (Chen et al., 2023) of self-consistency looked in the current settings, as existing LMs frequently provide inconsistent responses to ques- tions (Mitchell et al., 2022) and factual knowledge inquiries (Elazar et al., 2021; Tam et al., 2023; Gekhman et al.. 2023). Our work addresses these concerns by introducing a cross-check consistency approach, aiming bridge the gap between self- consistency and factual assessment.
over-
3 Self-consistency Limitations in Factuality Assessment
The essential assumption of self-consistency in fac- tuality assessment that the LM has the knowl- edge of the concept, responses sampled from its output distribution, should be similar and consis- tent; conversely, the LM lacks corresponding knowledge, the sampled responses would contain hallucinated facts that are diverged and contradic- tory. Although this assumption may seem reason- able, does not always hold in practice (morc details provided in the Appendix A). Specifi- cally, we argue that solely checking the LM's self- consistency is insufficient for detecting hallucina- tion or verifying factuality under the following two circumstances:
Consistency Evaluation of LMs. An essential characteristic of logically valid intelligent systems is seff-consistency, which entails that no two state- ments provided by the system contradict each other. Self-consistency is defined by Elazar et al. (2021) the invariance of an LM's responses across var- ious types of semantics presttvving prompt trans- formations. This definition further enriched by multiple other consistency categories proposed by Jang et (2022). Wang et (2022) demonstrates that self-consistency can significantly enhance the chain of thought reasoning in LMs. Without self-

---

1. LMs may produce consistently hallucinated facts. We observe that for certain questions, LMs may out- put consistently wrong answers. For instance, as shown in Fig. 1, when prompted with the ques- tion "Is pi smaller than 3. 5.2?", ChatGPT consistently generates incorrect answers. In this case, where the generated responses are consistent but non- non-factual, solely relying self self-consistency checking of single model would yield false nega- tive hallucination detection results.
turbation. To achieve this, leverage advances in LLM prompting. Starting with queried input Zo, we acquire a set of k semantically equivalent inputs {œ1, 22.... Øz} through the prompt: "For the question [QUERIED QUESTION], provide k semantically equivalent questions".
To ensure the quality of the generated in- puts in this step, we further double-check the semantic equivalence between the generated in- puts {01, #2; .... .xf and the queried input 2o in pair-wise manner using the prompt "Are the following two inputs semantically equivalent? [QUERIED INPUT] [GENERATED INPUT]" and filtering out the inputs that do not share the same semantic mcaning as the original input. The complete prompt templates used in this work are provided in the Appendix B.
2. Even in cases when LMs generate factual state- ments in their original response, the stochastic sam- pled responses may lack veracity. For example. the original answer (Answer 1) of ChatGPT under zero temperature is correct regarding the senator search question as shown in Fig. I. However, when sampled with a higher temperature, ChatGPT gen- erates multiple incorrect responses (Answer 2 and Answer m). In this scenario, where the sampled responses are inconsistent and disagree with the original response which itself is factually correct, methods that rely solely on model self-checking would produce false positives.
4.2 Stage 2: Additional Verifier LM
Model-level Cross-check with
Let 8o denote the original response from target LM T based on a given query Zg. Our objective is to detect whether so is hallucinated by sampling responses from the predictive distribution of T. To avoid model-level hallucination, we introduce an additional verifier LM denoted as V for model-level cross-checking. We define the responses from both models as:
In summary, although the inconsistency of sam- pled responses has been empirically demonstrated to be correlated with hallucinated facts on certain tasks, in general, self-consistency is neither neces- sary nor sufficient to verify the veracity of large LMs' statements. Therefore, methods based solely on ceff-consistency checking may not be able to ac- curately detect hallucinations in complex QA and open-domain generation tasks, which motivates us to design a more reliable and robust factuality as- sessment method that extends this idea.
ST, = =T(a;)) Sv, = V(x;), j = 1,..., k, (1)
where ke is the length of the generated semanti- cally equivalent inputs (æ1, 22, mg3 in stage 1. To assess the factuality of zo the self-checking mechanism operates by drawing a set of n, stochas- tic response samples from the target LM: ST. = {s s....., .... 8% Similarly, we can apply the same self-checking mechanism to the verifier LM to generate another set of nm responses: Sy, dem ( 2 sy...., s }. To perform question-level cross cross-check, for each perturbed input 2t. we generate ng sampled response sequences ST, = Bg [0), 82, ./..... STe from the target LM T and ngm sampled responses Sy, from the verifier LM
4 SAC : Semantic-Aware Cross-check Consistency
This section describes the proposed semantic-aware cross-check consistency approach, high-level overview of which is provided in Fig. 1. Addi- tionally, an illustrative example of each component is presented in Table 1. Here, we walk through each component in detail.
4.1
Stage 1: Question-level Cross-checking via Semantically Equivalent Perturbations
( ..., S, ngm
Contrary to existing techniques that assess seman- tic equivalence through entailment or paraphrasing, our approach involves rephrasing the input query by generating alternative inputs that preserve semantic equivalence, i.e., semantically equivalent input per-
Finally, we collect the total sampled sets S (Sr, ST.
S by combining all samples drawn from self-checking and cross-checking, which will be used next for calculating a consis- tency score.

---

Question-level Cross-check
Model-ievll Cross-check
Response
QA Pair Consistency Check
Queried Question
Target LM
Response
Semantically Equal?
Samenicatly Equiviint Perlarattions
A
Response
Verifier LM
Candidate Questions
Response
Figure 2: Overview of the proposed semantic-aware cross-check consistency (SAC²) method.
4.3 Stage 3: Consistency Score Calculation
4.3.3 Question-level Consistency (SAC
Score
This stage uses the generated sample sets in all pre- vious stages to calculate numerical consistency score that captures the question-leve' and model- level cross-cheeking paradigm.
Besides self-checking the original question vo- SAC further assesses cross-check consistency of perturbed questions {01. (2 22, ...., .x... The corre- sponding QA pairs compose two-dimensional matrix, where each row corresponds to a perturbed question (k in total), and each column corresponds to a sampled response (ng in total);
4.3.1
Semantic-aware Consistency Check of QA Pairs
Most of the existing works mainly focus on exam- ining the consistency of LM outputs while ignoring the effect of the inputs. However, in QA tasks, it is important to consider both inputs and outputs when measuring semantic equivalence, as the same ques- tion can be rephrased in many ways. Although the answers to these questions (e.g., "no" and "yes") may not be lexically equivalent, the QA pairs as a whole can be semantically equivalent. In light of this, we propose to check the semantic consistency of the QA pairs instead of the answer only.
(t)
(01, Say
Pi'T
(3)
(a St (Kx. Sig)
Therefore, the question-level cross cross-checking con- sistency score Ssccoo can be obtained by
flg k C(Po, P'T.)) (4)
-
nlg
1=1 j=1
4.3.4 Model-level Consistency (SAC3-M & SAC Score
4.3.2 Self-checking Consisteney (SC²) Score
Let C(:-)) denote semantic equivalence check- ing operator that takes two QA pairs as inputs. The operator C returns "Yes" if the two QA pairs are semantically equivalent, and "No" otherwise. This operator should be reflexive, symmetric, and transitive. We implement the checking operator using an LM by leveraging the prompt: "Are the following two Question-Answering (QA) pairs semantically equivalent? [QA PAIR
In addition to the question-level score, a model- level cross-check score calculated by performing cross-model checking and cross-question checking using the verifier LM V. Specifically, for the orig- inal question zo. the model-level cross- cross-checking consistency score S is computed by
Z B -M Mmm.
1
Σ
Run
C(Po: P'y))
(5)
1] [QA PAIR 2]". We then map the best guess to numerical semantic equivalent score: (YYes" 0.0, "No" + 1.0). We use Po (æo, 80) denote the original QA pair. The self-checking score Zsca of the target LM T can be calculated by
where Psv, {(x0) 3/1);, ;,.,,, (Io, s Rmm) } is the
QA pairs generated by the verified LM V.
The cross-question consistency score on the ver- ifier LM is computed on the QA pairs produced by
1 Th. 1=1
n Σ C(Po. P's
V:
ZSC
(2)
(211
Rqm
where Psr (( 8//0;.... (mog +7)) repre-
Play;
(6)
(x S
S
sents the QA pairs generated in the self-checking scenario.
The cross-model cross- cross-question consistency score

---

can thus be obtained through
As for the generation QA tasks, we take questions from the following two open-domain QA datasets and generate answers using LLMs. Then we manu- ally annotate the factuality of the answers following previous work (Li et al., 2023a).
3 -MM
C(Po, Psv.). Pss.)
(7)
Mggmm
4.3.5 Final Score and Model Confidence
The different variants of SAC capture different aspects of the uncertainty about the original re- sponse and should complement each other. We thus consider combination of all variants includ- ing SAC -Q SA SAC* as the final score:
HotpotQA- We randomly sample 250 ex- amples from the training set of HotpotQA (Yang et al., 2018) and generate hallucinated answers drawn from gpt-3. 5-turbo Then manually annotate the answers by comparing the ground truth and knowledge.
- oo + X SM S (8)
NQ-open-halu: Natural Questions (NQ) (Lee et al., 2019) is more challenging open do- main QA benchmark (Kwiatkowski et al., 2019). We use the same setting HotpotQA-halu to create small-scale dataset that consists of 250 non-factual and factual examples with manual annotations.
where X is weight factor for the verifier LM. Un- less mentioned otherwise, we use À = by default in our experiments. In practice, the computa- tion of each component is independent, they can be computed in parallel reduce latency. The detection prediction is made by comparing the fi- nal score with a preset threshold. In addition to computed score, we also ask the target LM to gener- ate verbalized confidence score (Tian et al., 2023) along with its prediction when checking the seman- tic equivalence of QA pairs. More discussions are offered in the Appendix C.2.
Please find more relevant details about data annota- tions in the Appendix C.3.
Experiments
6.1 Experimental Setup
5 Data and Annotation
Evaluation Models. We use gpt-3.5-turbe 5-turbo from OpenAl as the target LM for our experi- ment. The verifier LM is chosen from the follow- ing two models: (1) Falcon-7biinstruct (Al- mazrouei et al., 2023): decoder-only model built by T1l that trained on 1,500B tokens of RefinedWeb (Penedo et al., 2023) and further enhanced using the curated corpora; and (2) Guanaco- an open-soune instruction- following models through QLoRA (Dettmers et al.. 2023) tuming of LLaMA (Touvron et al., 2023) base model on the OASSTI dataset.
We evaluate our hallucination detection approach on two categories of QA tasks, namely, classi- fication QA and generation QA, with each cat- egory containing two datasets. Following prior work (Zhang et al., 2023a), we use the following two binary classification datasets for evaluation on the classification QA task:
open-source causal
Prime number: this dataset contains 500 ques-
tions that query the primality of a randomly cho- sen prime number between 1,000 and 20,000, where the factual answer is always "Yes". The synthesized hallucinated answers are "No, it is not prime number".
Implementation Details. The evaluation con- ducted using Azure OpenAl API. When performing semantic perturbations and consistency checking. we set the temperature to 0.0 to get deterministic high-quality outputs. Given a specific input query, we generate k = 10 semantically equivalent inputs using the prompt described in Section 4.1. For the self-checking-bard based method SC² we follow prior work (Manakul et al., 2023) set the temperature to 1.0 and generate n, 10 stochastic samples. For SAC and SAC we set ng = nmm = to reduce computational cost. To further reduce the inference cost, we set n.m = by default and
Senator search: the dataset consists of 500 ques- tions that follow the following template: "Was there ever a US senator that represented the state of [US STATE NAME] and whose alma mater was [US COLLEGE NAME]?". The factual answer is always "No". We also generate hallucinated answers: "Yes, there was US senator that represented the state of [US STATE NAME] and whose alma mater was [US COLLEGE N.ME].".

---

combine -M with SAM report the model- level results. We use hallucination detection accu- racy and area under the ROC curve (AUROC) to evaluate the performance. In addition to the esti- mated hallucination score, we also show the verbal- ized probabilities (Tian et al., 2023) from the target LM for comparison. We execute all experiments on 8 NVIDIA V100 32G GPUs.
Prime number Confidence
Senator search
Method
Score
Score
Confidence
Sc" (got-3. S-turbo)
48.2
51.0
29.6
30.6
SAC"-Q (gpt-3.S-turbe
93.2
96.4
97.0
97.4
SACt-OM (Falcon-7b)
89.8
91.2
21.0
21.8
SACR-QM (Guanaco-235)
94.4
96.3
45.6
46.2
SAct-all (Falcon-7b)
97.8
98.0
84.6
85.6
SAC (Suanaco-33b)
99.4
99.4
86.7
Table 3: Accuracy on classification QA tasks with 100%
hallucinated samples (with the threshold set to 0.5).
Prime number
Senator search
Metbod
Score
Confidence
Score
Confidence
SC3 (gpt-3. Stttrr)-
65.9
67.5
56.1
53.1
Impact of Threshold. Since the choice of thresh- old has a significant impact on the detection accu- racy in the case where the dataset only contains pos- itive (hallucinated) samples, we further experiment with different thresholds and present the results in Fig. 3. We observe that SAC and SAC$-all with Guanaco- 3aa are more robust against large threshold values and outperform SC² in most cases,
SAC (gpt-3.5- t-trbo)
99.4
99.7
99.7
99.7
Table 2: AUROC on classification QA tasks with 50% hallucinated samples and 50% factual samples.
6.2
Evaluation Results
6.2.1
Classification QA
Balanced Dataset. We first experiment on bal- anced datasets with 50% hallucinated samples and 50% factual samples. Table 2 compares the de- tection performance of SC² and SAC*-Q in terms of AUROC and verbalized confidence score. We observe that self-checking (SC²) performs poorly on both datasets, with a low AUROC of 65.9% and 56.1%, respectively. Our question-level cross- checking (SAC"-Q) significantly outperforms the SC² bascline achieving 99% AUROC on both datasets and is in line with the verbalized confi- dence score, confirming the effectiveness of cross- checking.
Prime number
Senator search
100
160
80
80
Accuracy
60
60
40
40
20
20
0.4
0.5
0.6
0.7
0.0
0.4
0.5
0.6
0.7
0.8
Threshold
Threshold
SC igpt 3.5 turboo
SAC7 OM (Fa con!
-
SAC (Falcon))
SNC (Suenaco)
SAC* (Gueneco!
Figure 3: Impact of threshold on detection accuracy.
Why Does Self-checking Fail? To further un- derstand why self-checking methods fail to detect some hallucinated responses, we visualize the dis- tribution of consistency scores using histogram plots in Fig. 4. We observe that for SC², sig- nificant portion of hallucinated samples received highly consistent predictions. In other words, the target LM made consistently wrong predictions due to a lack of question and model diversity, which aligns with our analysis in Section 3. On the other hand, benefiting from the semantically equivalent question perturbation, SAC3-Q's scores are more spread out in the inconsistent region, which helps to improve the effectiveness of detecting hallucina- tions by choosing proper threshold.
Unbalanced Dataset. We further evaluate our method in a more challenging scenario where the dataset only contains hallucinated samples. Table 3 presents the accuracy of detecting hallucinated samples using a preset threshold of 0.5. In this case, the performance of self-check drops significantly 48.2% and 29.6 99.6% respectively. SAC still out- performs SC² by a large margin. The model-level cross-check with verifier LMs performs well in the prime number dataset but fails to accurately detect hallucination in the senator search dataset. This is because both verifier LMs refuse to answer a large portion of the questions this dataset due lack of sufficient information. By combin- ing the target LM, SAc3-all with Guanaco-33b achieves the highest detection accuracy compared to other baselines SC² (+51.2%), SAC (+6.2%) and SAC3-QM (+ 5.0%).
6.2.2 Open-domain Generation QA
Compared to the classification QA tasks, detect- ing hallucinations in open-domain generation QA

---

Prime number
Senator
the HotpotQA-halu task, where the verifier LM out- performs the target LM. On the contrary, in the NQ open task where the target LM shows competitive performance, a smaller weight would yield better results.
200
SC7
SAC-QQ
150
1.50
Froquency
100
100
50
HotpotQAAhalu
90
0
SAC --0
5AC5-all (haiconi
SAC (Gusnsco)
0.00
0.75
0.00
0.25
0,50
0.75
1.00
88
Hallucinstion score
Hallucination score
AUnOC
86
Figure 4: Histogram of hallucination score.
84
82
80
0.5
0.75
1.0
1.25
1.5
tasks is more challenging. As shown in Table 4, - exhibits better AUROC than SC (+7%) in both datasets. Compared to SAC SAC shows 6.7% improvement in the HotpotQA-baid dataset but is slightly worse in the NQ open dataset. SAC leverages the advantages of question- level and model-level cross-checking and achieves consistently good performance in both datasets.
Verifier LM weight factor
NQQ-pen-hallu
78
SAC -00
SAC' (talen!
SAC'all (Guencco)
AUnOC
78
77
76
0.5
1.0
1.25
er LM weight factor
Figure 5: Effect of verifier LM weight AUROC.
HospatQA halu
NQ open-halu halu
Method
Score
Conlidence
Score
Contidence
sc² 3.pt-3.5-turbo
74,2
77.0
70.5
72.7
SAC*-Q (apt-3.5-ture)
77.2
Effect of the Number of Perturbed Questions. methods is
SAC*-QM (Falcon-7b)
83.0
79.5
67.5
62.0
The performance of sampling- expected to improve as the sample size increases, at the cost of higher latency and computational cost. In Fig. 6, we study this trade-off by vary- ing the number of perturbed questions from 2 to 10. We observe that the performance of SAC increases as more question samples are used but the performance gain gradually diminishes after using more than 5 question samples. This suggests that in practice we could use 2-4 question samples to achieve reasonably good performance low computational cost.
SACh-OM (Guanaco- 33b)
72.7
72.7
sAc (Falcon-70 7b)
84.5
77.1
77.2
SAC'-all (Guanaco-33b)
87.0
86.8
77.2
77.8
Table 4: AUROC on open-domain generation QA tasks.
Effect of Verifier LM Weight. In our previous experiments, we assigned equal importance to the consistency score computed by the target and ver- ifier LM. However, typically, the target LM and the verifier LM have different architectures and scales such that the user may have different levels of trust in their output truthfulness. This differ- ence in trust be incorporated by introducing a weight X to the consistency score produced by the verifier LM. For instance, if the goal is de- tect hallucination a specialized domain and the verifier LM is a domain-specific model developed for this domain, we can assign large weight to its scores (e.g., À > 1.0). In the general case, where the verifier LM is small-sized open-source model, can apply small weight value (c.g., À < 1.0) discount the influence of the verifier LM in the final score. Fig. visualizes the effect of vari- ous weight factors on the generation QA datasets. We observe that SAC3-all with higher weight would result in a larger advantage over SAC in
HotpotQA-hall
NQ-oper-halu
100
90
AUROC
80
80
70
70
60
60
Number of perturbed questions
Number of perturbed questions
-o-
5C2
SAC7-o
SAC al (Falcon)
SAC all (Quanaco)
Figure 6: Performance of SAC with varying number of perturbed questions.
Efffet of the Model Type. Our proposed SAC framework does not restrict the type of LLM em- ployed and can be naturally extended to various

---

types of target LLMs. To verify this, in addi- tion to GPT-3.5 tutubb), we conduct experiments using GPT-4 (gpt-4) and PaLM (chat-bison) on the considered four datasets in the setting of the balanced dataset. The experimen- tal results of comparing the proposed SA with the SC² baseline are summarized in Table 5. We observe that the proposed SAC consistently out- performs the Sc baseline across all LLM variants.
according to specific tasks to maximize detection accuracy. We believe that our work is an important step towards building reliable LLMs.
Ethies Statement
This paper studies hallucination detection in LMs, which has significant broader impacts in the field of natural language processing (NLP) and helps to address ethical considerations regarding trust- worthiness and reliability. The research outcome may contribute to the development of more accu- rate and reliable LMs by mitigating the risks of misinformation and biased outputs and promoting accountability and trust in AI systems.
Prime Number
Senator Search
HorporQA halu
NQ-open halu
Method
Sca (gpt-3. 5-tuubb)
48.2
29.6
74.2
70.5
sca (spt-4)
38.3
18.4
79.7
76.3
sc* (chat-bison)
26.9
19.2
75.8
67.9
SAC (gat-3.5-turbo)
97.0
SAC'-Q (gnt-4)
91.1
61.6
87.2
82,9
SAC'-Q (chat-bison)
90.3
66.3
82.8
72.7
Limintations
Table 5: Accuracy of different LLMs (GPT-3.5, GPT-4, and PaL.M 2) on classification and generation QA tasks.
Our current experiments focus on the question- answering setting. Further research is needed to assess the generalizability of the proposed frame- work and the accuracy of semantic equivalence checks on more complex tasks such as conversa- tional or dialogue- would be interesting to investigate the efficiency- utility trade-off: we expect increasing sample sizes to improve detection accuracy but may introduce additional cost and latency. Speeding up the im- plementation through parallelization is also worth exploring.
Computational Cost. We monitor the computa- tional cost of our approach based on the number of model evaluations consumed by OpenAl API and open-source LLMs inference. Assuming the number of samples equals the number of perturbed questions, i.e., ns m'm n2 7lmm. the cost of SC² is ny API calls, and our SAC needs ne target LM calls plus 2 x ne verifier LM calls. Beyond the model evaluations, SAC may have addi- tional costs from question perturbations and seman- tic equivalence checking via prompting. Additional discussions can be found in the Appendix C.I.
prompting. Additionally,
References
Ayush Agrawal, Lester Mackey, and Adam Tauman Kalai. 2023. Do language models know when they're hallucinating references? arXiv preprint 2xXi:2305518825.
7 Conclusion and Discussion
We investigate the relationship between the self- consistency of LM and the factuality of the re- sponse and propose SAC robust hallucination detection approach for black-box LMs. Through extensive empirical analysis, our work highlights several findings. First, self-consistency checking alone is insufficient to effectively detect question- level and model-level hallucinations, where LMs generate consistently wrong responses to certain questions. Second, cross-checking between seman- tically equivalent questions can reduce the occur- rence of persistent hallucinations, potentially by re- ducing question-level ambiguity. Third, there exists model-level disparity in hallucinations, which we attribute to the inherent differences in LM capabili- ties originating from different training procedures and data. Thus, the verifier LM can be selected
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru. Merouane Debbah, Ftienne Goffinet, Daniel Hes- low, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-4OB: an open large language model with ofate-of-the-art performance.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Meng Cao. Yue Dong, and Jackie Chi Kit Cheung. 2022. Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 3340-3354

---

Angelica Chen, Jason Phang, Alicia Parrish, Vishakh Padmakumar, Chen Zhao, Samuel Bowman, and Kyunghyun Cho. 2023. Two failures of self- consistency in the multi-step reasoning of llms, arXiv preprint arXiv:2305. 14279.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023, Semantic uncertainty: Linguistic invariances for un- certainty estimation in natural language generation. arXiv preprint arXiv:2302.0900
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
Jiuhai Chen and Jonas Mueller. 2023. Quantifying un- certainty in answers from any language model via intrinsic and extrinsic confidence assessment. arXiv preprint wrXi:2338..6173
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019, Latent retrieval for weakly supervised open do- main question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages
Jeremy R Cole, Michael JQ Zhang, Daniel Gillick, Ju- lian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Fisenstein. 2023. Selectively answering ambiguous questions. arXiv preprint arXiv:2305. 14613.
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie. and Ji-Rong Wen. 2023a. Halueval: A large- scale hallucination evaluation benchmark for large language models. arXiv e-prints, pages arXiv-2305.
Souvik Das, Sougata Saha, and Rohini Srihari. 2023. Diving deep into modes of fact hallucinations in dia- logue systems. arXiv preprini arXiv:2301 04449..
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023b. Inference- time intervention: Eliciting truthful answers from a language model. arXiv preprint 2316:230033344
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized Iims. arXiv preprint arXiv:2305.14: 14314.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantifi- cation for black box large language models. arXiv preprint arXiv:2305.19187
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu,
Roberta Railcanu, Xian Li, Asli Celikyilmaz, and Ja- son Weston. 2023, Shain-of-verificatio reduces hal- lucination in large language models. arXiv preprint arXiv:2309.1 11955.
Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2021. A token-level reference-free hallucination detection benchmark for free-form preprint urXiv:2104.08704.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhi- lasha Ravichander, Eduard Hovy, Hinrich Schüizze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transac- tions of the Association for Computational Linguis- tics, 9:1012-1031.
generation. arXiv
Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. arXiv preprini arXiv:2303. 08896.
Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023. Trueteacher: Learn- ing factual consistency evaluation with large lan- guage models. arXiv preprint 2305v 11171.
Eric Mitchell, Joseph J Noh, Siyan Li. William S Arm- strong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher Manning. 2022. Enhancing self- consistency and performance of pre-trained language models through natural language inference. arXiv preprint arXiv:2211.1 18755.
Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. 2022. Becel: Benchmark for consis- tency evaluation of language models. Proceedings of the 29th International Conference on Computa- tional Linguistics, pages 3680-3696.
Niels Mündler, Jingxuan He, Slobodan Jenko, and Mar- Vechev. 2023. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. arXiv preprint arXiv:2305.15852.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang. Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei. and Julien Launay. 2023. The RefinedWeh dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint 2rXiv:2306.01116
Ziwei Ji, Zihan Liu, Nayeon Lee, Tiezheng Yu. Bryan Wilie, Min Zeng, and Pascale Fung. 2022. Rho: Reducing hallucination in open-domain dia- logues with knowledge grounding. arXiv preprint arXiv:2212.01588.

---

Baolin Peng. Michel Galley, Pengeheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang. Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.1 12813.
Shuo Zhang, Liangming Pan, Junzhou Zhao, and William Yang Wang. 2023b. Mitigating lan- guage model hallucination with interactive question-know alignment. arXiv preprint arXiv:2305. 13669.
Yue Zhang. Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023c. Siren's song the ocean: A survey on hallucination in large language models. arXiv preprint arXhv:3309.0121
Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, and Colin Raffel. 2023. Evalu- ating the factual consistency of large language mod- els through news summarization. In Findings of the Association for Computational Linguistics; ACL 2023, pages 2220-5255
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.1822
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023. Just ask for cali- bration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXivv2305.144 149975555...95. 14975.
Shen Zheng. Jie Huang. and Kevin Chen-Chure Chang. 2023a. Why does chatgpt fall short in providing truthful answers. ArXiv preprint, abs/2334.105133
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13977.
Shen Zheng, Jie Huang, and Kevin 2023b. Why does chatgpt fall short providing truthful answers? arXiv preprint arXiv:2304. 10513.
Chang.
Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Paco Guzman, Luke Zettlemoyer, and Marjan Ghazvininejad. 2020. Detecting hallucinated con- tent in conditional neural sequence generation. arXiv preprint arXiv:201 1.05993.
Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jian- shu Chen, and Dong Yu. 2023. A stitch in time saves nine: Detecting and mitigating hallucinations of Ilss by validating low-confidence generation. arXiv preprint arXivv23301 230700377.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi. Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 22x::2203.111777
Yijun Xiao and William Yang Wang. 2021. On halluci- nation and predictive uncertainty conditional lan- guage generation. arXiv preprint X 103. 15025.
Shiping Yang, Renliang Sun, and Xiaojun Wan. 2023, new benchmark and reverse validation method for passage-level ballucination detection, arXiv preprint arXiv:2310.06-988
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning, 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages
Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. 2023. Cognitive mirage: A review of hallucinations large language models. arXiv preprint 23X9:2230000677
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023a. How language model hallucinations can snowball. arXiv preprint arXiv:2305. 13534.

---

A Factuality Assessment of LLMs
larity using an additional pre-trained NLU model such as BERT (Manakul et al., 2023; Kuhn et al., 2023).
This section first introduces the background of LM factuality assessment and reviews the key compo- nents of existing black-box factuality assessment approaches.
B Complete Prompts
The prompt templates that we used are provided in Table 7, specifically the question perturbations ing semantically equivalent prompts and semantic equivalence checks between two QA pairs.
A.1 Background
Existing studies (Manakul et al., 2023) showed that the factuality of LMs' statements be as- sessed by capturing the output uncertainty. If permitted with complete white-box access, hal- lucinations of LMs can be detected through un- certainty metrics such as entropy, which mod- els the uncertainty of the output distribution as H(ss) - - - (<x log ((8,(<<))) for a se- quence of N tokens s { The insight is that factual sentences are likely to be composed of nigh-probability tokens and thus should have less entropy, whereas non-factur statements are likely be associated with lower probability and higher entropy.
C Additional Details and Discussions
C.1 Computational Cost
High computational cost is common research challenge faced in many areas of black-box LLM studies, including self-consistency (Wang et al., 2022) and self-check (Manakul ct al., 2023). Com- pared with these existing works, our framework enables model-level parallel execution of LLM in- ferences (either target or verifier LM) in the form of parallel API calls, which helps to mitigate the over- head. Moreover, may further employ more ad- vanced prompt strategies for semantic consistency checking that condense the pairwise comparisons to single inference call to substantially reduce the time complexity. Finally, our framework also of- fers the flexibility for the user to adjust the balance between computational cost and accuracy: users may choose to trade off accuracy for efficiency in scenarios where low overhead is more valued than high detection precision.
Although these methods provide an accurate es- timate of the LM's uncertainty, computation of such metrics involves token-level outpul probabil- ity distribution, which typically not accessible for prominent LMs like ChatGPT that are only available through black-box interactions.
A.2 Black-box Assessment via Checking Consistency in Sampled Responses
One natural way extend the above white-box factuality assessment methods restrictive black- box settings is to approximate the uncertainty metrics by sampling LM responses. Existing approaches (Manakul et al., 2023) mostly focus on assessing factuality through inspecting LM''s self-consistency. The typical process of self- consistency check is as follows. First, for given user-queried prompt æ, let the LM generate can- didate response with a low (default) tempera- ture. Next, to assess the factuality of the candidate statements, a higher temperature value used to stochastically sample a diverse set of supporting .....""!! responses {s] s?.... } from the LM. Finally,
To achieve more balanced trade-off, we pri- marily recommend two strategies: (1) Utilizing a smaller sample size, such as the number of ques- tion perturbations and self-evaluations. Our abla- tion experiments demonstrate that the performance of SAC improves with an increase in question samples, but the performance gain plateaus after exceeding five question samples. This indicates that using 2-4 question samples in practice could yield satisfactory results with minimal computa- tional expense. (2) Implementing an optimized prompt strategy. With the original input question, we prompt the GPT model to generate multiple paraphrases in single API call, i.e., ((1) for ques- tion paraphrasing. Although semantic consistency checking via pairwise method can be slightly bur- densome, we can devise an advanced prompt strat- egy parallelize the pairwise consistency check. i.e., through one API call instead of multiple calls, effectively reducing the inference complexity from (n) to O()). These strategies empower users
a hallucination score S E [0, 1] is calculated by measuring the consistency between the candidate response and the supporting responses: S - if the supporting responses agree with the candidate response and + if they disagree. The consis- tency between two responses is typically measured by checking the textual entailment or lexical simi-

---

Semantic Consistency
Method
Input
Model
Output
SCa
Q
Target
Yes. there a US senator that represented the state of North Dakota and whuse alma mater was MIT. His name was Quentin N. Burdick. Yes, there was a US senator who represented the state of North Dakota and whose alma mater was MIT. His name was Quentin N. Burdick.
Target
As an language model, do not have access to the most recent updates news. However, as of my last update, there no North Dakota US senator who graduated from MIT.
S
Q
Q1
Target
As an language model, not have access to real-time cal-time information current events. However,
Q2
Target
my last update, am aware of any US senators from North Dakota who were also MIT graduates.
SM
Verifier
No, there has never been a US senator who represented the state of North Dakota and attended MIT.
SAM
Q1
Verifier
No, North Dakota US senator did not have MIT listed as their alma mater.
Q2
Verifier
There are no known US senators from North Dakota who are also MIT graduates.
x
Table 6: Input Question (Q) and Answer (A): Q "Was there ever US senator that represented the state of North Dakota and whose alma mater was MIT? " A = "Yes, there was a US senator who represented the state of North Quentin Burdick." Semantically rephrased questions: QI "Did a North Dakota US senator have MIT listed as their alma mater? Q2 = "Were there any US senators from North Dakota who were also MIT graduates?
Dakota and whose alma mater was MIT. His name
=
Objective
Template
Semantic question
For the question IQUERIED QUESTIONI, provide S(k) semantically equivalent questions In Are the following two inputs semantically equivalent?
perturbation
[QUERIED INPUT] 'n (GENERTEED INPUT]
Semantic equivalence check
Are the following Question-Answering (QA) pairs semantically equivalent? Provide your best guess and the probability that is correct (0.0 1.0). Given ONLY the guess (Yes or No) and probability, other words explanation. example: 'n Guess: <most likely guess, as short as possible; not complete sentence, just the guess!>in Probubility: <the probability between 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever; just the probahility! 'n The first QA pair Q: 5(TE QUESTION) 'n A: $(THE ANSWER The sccond QA pair In Q: S/THE QUESTIÓN]
S(THR ANSWER
Table 7: Prompt templates for different objectives, including semantically equivalent perturbations of input questions and semantic equivalent check of QA pairs.
to optimally balance efficiency (light version) and accuracy (performance version), facilitating more informed decisions in practice.
cial for evaluation. In practice, the user may choose any open-sourced LLMs that have been aligned to follow instructions.
C.2 Semantic Consistency Checking
C.3 Data Annotations
The goal of our research is to provide flexible framework for the effective detection of hallucina- tions in black-box LLMs. Compared to previous approaches (Manakul et al., 2023) based on similar- ity metrics such as B BERTScore, employing LLM for semantic consistency checking not only offers better accuracy but also eliminates the need for in- volving additional models. We note that our design is not dependent on any specific type of LLM and the users are allowed to choose freely for each com- ponent. Specifically, we chose GPT as the instanti- ation in our experiments due to its aollamell ability to follow human instructions which cru-
For the two classification QA tasks (Prime Num- ber and Senator Search), we use the synthesized hallucinated answers following prior work (Zhang et al., 2023a). For the task of prime number, the factual/true answer is always "Yes", i.e., all the test- ing numbers are prime numbers. The synthesized hallucinated response is "No". The factual/true answer of the senator search is always "No" and the synthesized hallucinated answer is "Yes". Such an experimental setting on the binary classifica- tion tasks is realistic since we have verified that on these datasets most of the responses generated by gpt-3. 5- turbo are indeed "Yes" or "No" which

---

align with the synthesized response. For the generation QA tasks (HotpotQA-haal and NQ-Open-halu), we used answers generated by LLM (gpt-3. 5-ttrbbb) for experiments, which do not have pre-defined factual/non-factual labels. Therefore, we manually annotated these LLM- generated answers by comparing them with the ground truth.
We would like note that in practice, more versatile hallucination detection approach should be able evaluate the factuality of sample regard- less of its origin (c.g., synthesized or generated by itself other LLMs). In our framework, this is achieved through semantically equivalent question perturbation and cross-model response consistency checking. We plan to release the annotated datasets to facilitate future research.
D Additional Examples
Table provides another illustrative example to explain our methodology in the senator search dataset.


Reviews:
[{"review_id": "5tFhVmE9nq", "reviewer": "Reviewer_G3kV", "summary": "", "questions": "1. Line 527, should it be \"and the verifier LM is a domain specific model developed for this domain\"?\n2. For the collection of the two classification QA datasets, the hallucinated samples are synthesized or gpt-3.5-turbo generated?\n3. Will the annotated dataset open-source if this paper is accepted?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer G3kV\n\nThank you very much for the constructive comments and feedback. We are encouraged to hear that you found our paper well-written and our results exceptionally strong. Please find our point-by-point responses to your questions below. We are committed to fully addressing your concerns and are happy to provide additional clarification should you have any remaining questions.\n\n> **W1: The method requires a substantial number of API calls/LLM inference, encompassing question rewriting, response sampling, and semantic consistency checking, which is cost heavy.**\n\nThanks for the comment. High computational cost is a common research challenge faced in many areas of black-box LLM studies, including self-consistency [1] and self-check [2]. Compared with these existing works, our framework enables model-level parallel execution of LLM inferences (either target or verifier LM) in the form of parallel API calls, which helps to mitigate the overhead. Moreover, we may further employ more advanced prompt strategies for semantic consistency checking that condense the pairwise comparisons to a single inference call to substantially reduce the time complexity. Finally, our framework also offers the flexibility for the user to adjust the balance between computational cost and accuracy: users may choose to trade off accuracy for efficiency in scenarios where low overhead is more valued than high detection precision. We will clarify this.\n\n> **W2: The experimental results demonstrate remarkable performance gains of the method over the baseline in the classification QA task. However, the improvement in the generation QA task is not as pronounced. The authors should discuss the potential reasons for this phenomenon in the experimental section.**\n\nThank you for your observation. Our method shows superior performance on both tasks, while the improvement of classification QA is more significant. This is because the considered two classification QA tasks (i.e., Prime number and Senator search) are purposely crafted by prior work [3] to capture the worst scenario where the target LM such as GPT-3.5 is highly inclined to hallucinate and produces consistently wrong responses, which leaves more room for improvements. In contrast, the considered open-domain generation QA tasks are taken from general-purpose datasets that represent the average-case scenario. Although detecting hallucinations in the generative tasks is more challenging, our method still outperforms the baseline by a significant margin, namely achieving 13.8\\% and 6.7\\% improvement compared with the self-consistency baseline. We will add corresponding discussions.\n\n> **W3: It lacks experiments on whether the proposed method can be applicable to various types of LLMs as the target model other than gpt-3.5-turbo.**\n\nOur proposed framework does not restrict the type of LLM employed and can be naturally extended to various types of target LLMs. To verify this, in addition to GPT-3.5 (``gpt-3.5-turbo``), we have conducted additional experiments using GPT-4 (``gpt-4``) and PaLM 2 (``chat-bison``) on the considered four datasets. The experimental results of comparing the proposed SAC$^3$ with the SC$^2$ baseline are summarized in the following table. We observe that the proposed SAC$^3$ consistently outperforms the SC$^2$ baseline across all LLM variants.\n\n|       Methods           | Prime Number | Senator Search | HotpotQA-halu | NQ-open-halu |\n|-----------------------|:--------------:|:--------------:|:-------------:|:------------:|\n| SC$^2$ (GPT-3.5)    | 48.2         |      29.6      |      74.2     |     70.5     |\n| SC$^2$ (GPT-4)      | 38.3         |      18.4      |      79.7     |     76.3     |\n| SC$^2$ (PaLM 2)     | 26.9         |      19.2      |      75.8     |     67.9     |\n| SAC$^3$-Q (GPT-3.5) | 93.2         |      97.0      |      81.3     |     77.2     |\n| SAC$^3$-Q (GPT-4)   | 91.1         |      61.6      |      87.2     |     82.9     |\n| SAC$^3$-Q (PaLM 2)  | 90.3         |      66.3      |      82.8     |     72.7     |\n\n> **W4: It is unclear whether it is possible to employ open-source LLMs for semantic consistency checking, and whether the resulting performance decrease fall within an acceptable range.**\n\nThe goal of our research is to provide a flexible framework for the effective detection of hallucinations in black-box LLMs. Compared to previous approaches [2] based on similarity metrics such as BERTScore, employing LLM for semantic consistency checking not only offers better accuracy but also eliminates the need for involving additional models. We note that our design is not dependent on any specific type of LLM and the users are allowed to choose freely for each component. Specifically, we chose GPT as the instantiation in our experiments due to its well-acclaimed ability to follow human instructions which is crucial for evaluation. In practice, the user may choose any open-sourced LLMs that have been aligned to follow instructions.\n\n> **W5: The method proposed by the authors also encounters the same challenge as self-consistency, as depicted on the right side of Figure 1: even though the responses exhibit inconsistency among themselves, the response to be detected is actually factual.**\n\nThanks for the comment. We agree with the reviewer on that precisely detecting non-factual responses through inconsistency remains an open research problem and is difficult to address fully. However, compared to the existing approach that solely relies on model self-consistency, our proposed approach aims to mitigate this by weighing in the responses generated by multiple LMs with question perturbations, which, as verified by our extensive experiments, is more reliable and robust against various types of hallucinations and significantly outperforms the self-check baselines.\n\n>**Q1: Line 527, should it be \"and the verifier LM is a domain specific model developed for this domain\"?**\n\nThanks for your careful reading. We will correct this in our final revision. \n\n> **Q2: For the collection of the two classification QA datasets, the hallucinated samples are synthesized or gpt-3.5-turbo generated?**\n\nFor the two classification QA tasks (Prime Number and Senator Search), we use the synthesized hallucinated answers following prior work [3]. For the task of prime number, the factual/true answer is always \"Yes\", i.e., all the testing numbers are prime numbers. The synthesized hallucinated response is \"No\". The factual/true answer of the senator search is always \"No\" and the synthesized hallucinated answer is \"Yes\". Such an experimental setting on the binary classification tasks is realistic since we have verified that on these datasets most of the responses generated by ``gpt-3.5-turbo`` are indeed \"Yes\" or \"No\" which align with the synthesized response.\n\nFor the generation QA tasks (HotpotQA-halu and NQ-open-halu), we used answers generated by LLM (``gpt-3.5-turbo``) for experiments, which do not have pre-defined factual/non-factual labels. Therefore, we manually annotated these LLM-generated answers by comparing them with the ground truth (Section 5, lines 401-413).\n\nWe would like to note that in practice, a more versatile hallucination detection approach should be able to evaluate the factuality of a sample regardless of its origin (e.g., synthesized or generated by itself / other LLMs). In our framework, this is achieved through semantically equivalent question perturbation and cross-model response consistency checking.\n\n>**Q3: Will the annotated dataset open-source if this paper is accepted?**\n\nYes, we plan to release all the source code and annotated datasets (e.g., JSON files) for better reproducibility and also facilitate future research.\n\n**References**\n\n[1] Wang, Xuezhi, et al. \"Self-Consistency Improves Chain of Thought Reasoning in Language Models.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[2] Manakul, Potsawee, Adian Liusie, and Mark JF Gales. \"Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.\" arXiv preprint arXiv:2303.08896 (2023).\n\n[3] Zhang, Muru, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. \"How language model hallucinations can snowball.\" arXiv preprint arXiv:2305.13534 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well written.\n2. The proposed method can serve as a complementary approach to extend self-consistency methods.\n3. Exceptionally strong results were achieved on the two datasets for the classification QA task, with AUROC values approaching 1. Additionally, in the generation QA task, the method also outperformed the baseline, showcasing improved performance.", "reasons_to_reject": "1. The method requires a substantial number of API calls/LLM inference, encompassing question rewriting, response sampling, and semantic consistency checking, which is cost heavy.\n2. The experimental results demonstrate remarkable performance gains of the method over the baseline in the classification QA task. However, the improvement in the generation QA task is not as pronounced. The authors should discuss the potential reasons for this phenomenon in the experimental section.\n3. It lacks experiments on whether the proposed method can be applicable to various types of LLMs as the target model other than gpt-3.5-turbo.\n4. It is unclear whether it is possible to employ open-source LLMs for semantic consistency checking, and whether the resulting performance decrease fall within an acceptable range.\n5. The method proposed by the authors also encounters the same challenge as self-consistency, as depicted on the right side of Figure 1: even though the responses exhibit inconsistency among themselves, the response to be detected is actually factual.", "questions_to_authors": "1. Line 527, should it be \"and the verifier LM is a domain specific model developed for this domain\"?\n2. For the collection of the two classification QA datasets, the hallucinated samples are synthesized or gpt-3.5-turbo generated?\n3. Will the annotated dataset open-source if this paper is accepted?", "paper_topic_and_main_contributions": "This paper introduces a sampling-based hallucination detection method that addresses the limitations of traditional self-consistency-based detection methods. The specific contributions are as follows:\n1. The discovery of two types of hallucination that are challenging to detect through self-consistency: question-level hallucination and model-level hallucination.\n2. For these two types of hallucination, the authors propose SAC^3, an approach that calculates a weighted hallucination score through Question-level Cross-checking and Model-level Cross-checking.\n3. Significant performance improvements are achieved compared to self-consistency method on four datasets in the tasks of classification QA and generation QA.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FuR5VvB2ol", "reviewer": "Reviewer_eydx", "summary": "", "questions": "In Table 1 in the Cross-Q/M variants, the authors considered positive answers with different explanations as non-consistent. This does not align with the original self-consistency work, that stated that the consistency of the answered is determined by the final answer of the model, and not the CoT part of which, which in my opinion is somewhat equivalent to the “explanation part” of the exemplified answers. Could the authors please elaborate on why these answers were considered inconsistent?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer eydx\n\nThank you very much for your detailed review and valuable suggestions for improving our manuscript. We are glad to hear that you found our idea novel and our analysis thorough. Please find our point-by-point responses to your questions below. We are committed to fully addressing your concerns and are happy to provide additional clarification should you have any remaining questions.\n\n>**W1: The authors tested their strategy solely on the gpt-3.5-turbo model. Though their results seemed promising, I believe it is important to test their strategy on additional models, including at open-sourced models, to get a better picture of the strategy’s efficacy. Currently, with just one model being tested, this feels insufficient to fully assess the helpfulness of the author’s proposal.**\n\nOur proposed framework does not restrict the type of LLM employed and can be naturally extended to various types of target LLMs. To verify this, in addition to GPT-3.5 (``gpt-3.5-turbo``), we have conducted additional experiments using GPT-4 (``gpt-4``) and PaLM 2 (``chat-bison``) on the considered four datasets. The experimental results of comparing the proposed SAC$^3$ with the SC$^2$ baseline are summarized in the following table. We observe that the proposed SAC$^3$ consistently outperforms the SC$^2$ baseline across all LLM variants.\n\n|       Methods           | Prime Number | Senator Search | HotpotQA-halu | NQ-open-halu |\n|-----------------------|:--------------:|:--------------:|:-------------:|:------------:|\n| SC$^2$ (GPT-3.5)    | 48.2         |      29.6      |      74.2     |     70.5     |\n| SC$^2$ (GPT-4)      | 38.3         |      18.4      |      79.7     |     76.3     |\n| SC$^2$ (PaLM 2)     | 26.9         |      19.2      |      75.8     |     67.9     |\n| SAC$^3$-Q (GPT-3.5) | 93.2         |      97.0      |      81.3     |     77.2     |\n| SAC$^3$-Q (GPT-4)   | 91.1         |      61.6      |      87.2     |     82.9     |\n| SAC$^3$-Q (PaLM 2)  | 90.3         |      66.3      |      82.8     |     72.7     |\n\n\n>**W2: The paper mostly dealt with \"prompt-engineering\", which is not that novel. Despite the potential merits of the proposed prompting strategy, I believe this alone might not be sufficiently innovative. Perhaps with more analyses and perturbations, this work could have had a noticeable contribution.**\n\nWe would like to emphasize that our major contribution is to propose a novel and unified framework, SAC$^3$, that utilizes sampling-based semantic-aware cross-check consistency for effective hallucination detection in black-box LMs. SAC$^3$ is built upon a series of designs, including question perturbation, model perturbation, and semantic cross-check consistency. It is the combination of these design elements that enabled SAC$^3$ to achieve substantial improvements over the self-consistency baseline by a considerable margin (as high as 13.8%). The proposed prompting strategy is only one of the key components and is designed in place of the previous similarity-based consistency-checking techniques that failed to accurately assess semantic equivalency.\n\n>**Q: In Table 1 in the Cross-Q/M variants, the authors considered positive answers with different explanations as non-consistent. This does not align with the original self-consistency work, that stated that the consistency of the answered is determined by the final answer of the model, and not the CoT part of which, which in my opinion is somewhat equivalent to the “explanation part” of the exemplified answers. Could the authors please elaborate on why these answers were considered inconsistent?**\n\nIn this work, we considered the task of hallucination detection provided with only the (potentially hallucinated) target response and without CoT or explanation. Our approach achieves this through semantically equivalent question perturbation and cross-model response consistency checking.\n\nThe illustrative examples shown in Table 1 are regarding the input question ``Is 3691 a prime number?`` and the target response is ``No, 3691 is not a prime number. It is divisible by 7 and 13``, as mentioned in the caption.  We define the target question-answering (QA) pair by combining the input questions and the target response. The Cross-Q/M variants present a set of sampled QA pairs including perturbed questions and the corresponding answers. The semantic consistency between the target QA pair and sampled QA pairs:\n\n- Target QA pair:\n``Q: Is 3691 a prime number? A: No, 3691 is not a prime number. It is divisible by 7 and 13``\n\n\n- Sampled QA pair 1: \n``Q: Is the number 3691 prime? A: Yes``\n\n- Sampled QA pair 2: \n``Q: Are the only factors of 3691 itself and one? A: Yes, the only factors of 3691 are 1 and itself.``\n\n- Sampled QA pair 3: \n``Q: Can 3691 only be divided by 1 and 3691? A: Yes, 3691 can only be divided by 1 and 3691.``\n\nThe target QA pair is non-factual since 3691 is in fact a prime number, while the three sampled QA pairs are not semantically consistent with the target QA pair as they are factual.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis paper proposes a novel approach for identifying hallucinated responses, and for generating more trust-worthy responses.\n2.\tThe authors conduct very thorough analyses, that demonstrate the efficacy of their approach on the gpt-3.5-turbo model.", "reasons_to_reject": "1. The authors tested their strategy solely on the gpt-3.5-turbo model. Though their results seemed promising, I believe it is important to test their strategy on additional models, including at open-sourced models, to get a better picture of the strategy’s efficacy. Currently, with just one model being tested, this feels insufficient to fully assess the helpfulness of the author’s proposal.\n2. The paper mostly dealt with \"prompt-engineering\", which is not that novel. Despite the potential merits of the proposed prompting strategy, I believe this alone might not be sufficiently innovative. Perhaps with more analyses and perturbations, this work could have had a noticeable contribution.", "questions_to_authors": "In Table 1 in the Cross-Q/M variants, the authors considered positive answers with different explanations as non-consistent. This does not align with the original self-consistency work, that stated that the consistency of the answered is determined by the final answer of the model, and not the CoT part of which, which in my opinion is somewhat equivalent to the “explanation part” of the exemplified answers. Could the authors please elaborate on why these answers were considered inconsistent?", "paper_topic_and_main_contributions": "The paper focuses on the phenomenon of hallucinations in LLMs, and proposes a novel approach to verify the factuality of responses, that improves the self-consistency strategy. These improvements include: question-level improvements, where the question is replaced with semantically equivalent variants, which are generated and verified by an LLM (via designated prompts); and model-level improvements, where a verifier LLM is employed to also answer the question. In each of these improvements, the consistency of the answer is observed, namely across the different variants of the question and across the different models (and also the combination of question variants and models) to identify inconsistencies that could point to potential hallucinations.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dhltIdOUFB", "reviewer": "Reviewer_vZxH", "summary": "", "questions": "- I believe that the current evaluations only consider synthesized datasets where the non-factual and factual answers are predetermined. This means you manually determine whether the answers are correct and the ratio of correct and incorrect answers. This could potentially create a gap between the results of the current evaluation and the actual behavior of LLMs in real scenarios. Therefore, could you further evaluate the methods using real outputs from LLMs on randomly sampled questions. Please correct me if I have misunderstood anything.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer vZxH,\n\nThank you very much for your thoughtful comments. We are glad that you found the proposed method new and results promising (outperforms self-consistency by a large margin). Please find our point-by-point responses to your questions below. We are committed to fully addressing your concerns and are happy to provide additional clarification should you have any remaining questions.\n\n>**W1: The proposed method is computationally costly because it uses LLMs in multiple steps, such as paraphrasing the question, verifying semantic equivalency, obtaining the answer, and checking consistency.**\n\nThanks for the comment. High computational cost is a common research challenge faced in many areas of black-box LLM studies, including self-consistency [1] and self-check [2]. Compared with these existing works, our framework enables model-level parallel execution of LLM inferences (either target or verifier LM) in the form of parallel API calls, which helps to mitigate the overhead. Moreover, we may further employ more advanced prompt strategies for semantic consistency checking that condense the pairwise comparisons to a single inference call to substantially reduce the time complexity. Finally, our framework also offers the flexibility for the user to adjust the balance between computational cost and accuracy: users may choose to trade off accuracy for efficiency in scenarios where low overhead is more valued than high detection precision. We will clarify this.\n\n>**W2&Q: I believe that the current evaluations only consider synthesized datasets where the non-factual and factual answers are predetermined. This means you manually determine whether the answers are correct and the ratio of correct and incorrect answers. This could potentially create a gap between the results of the current evaluation and the actual behavior of LLMs in real scenarios. Therefore, could you further evaluate the methods using real outputs from LLMs on randomly sampled questions. Please correct me if I have misunderstood anything.**\n\nWe would like to clarify that our evaluations considered both synthesized outputs and real LLMs outputs. \n\nFor the generation QA tasks (HotpotQA-halu and NQ-open-halu), we used answers generated by LLM (``gpt-3.5-turbo``) for experiments, which do not have pre-defined factual/non-factual labels. Therefore, we manually annotated these LLM-generated answers by comparing them with the ground truth (Section 5, lines 401-413).\n\nFor the two classification QA tasks (Prime Number and Senator Search), we use the synthesized hallucinated answers following prior work [3]. For the task of prime number, the factual/true answer is always \"Yes\", i.e., all the testing numbers are prime numbers. The synthesized hallucinated response is \"No\". The factual/true answer of the senator search is always \"No\" and the synthesized hallucinated answer is \"Yes\". Such an experimental setting on the binary classification tasks is realistic since we have verified that on these datasets most of the responses generated by ``gpt-3.5-turbo`` are indeed \"Yes\" or \"No\" which align with the synthesized response.\n\n\n**References**\n\n[1] Wang, Xuezhi, et al. \"Self-Consistency Improves Chain of Thought Reasoning in Language Models.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[2] Manakul, Potsawee, Adian Liusie, and Mark JF Gales. \"Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.\" arXiv preprint arXiv:2303.08896 (2023).\n\n[3] Zhang, Muru, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. \"How language model hallucinations can snowball.\" arXiv preprint arXiv:2305.13534 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A new method based on question paraphrase and cross-model checking is proposed for hallucination detection.\n- The proposed method outperforms self-consistency by a large margin.", "reasons_to_reject": "- The proposed method is computationally costly because it uses LLMs in multiple steps, such as paraphrasing the question, verifying semantic equivalency, obtaining the answer, and checking consistency.\n- The evaluation all on synthesized datasets with pre-defined non-factual and factual answers, instead of real LLMs outputs.", "questions_to_authors": "- I believe that the current evaluations only consider synthesized datasets where the non-factual and factual answers are predetermined. This means you manually determine whether the answers are correct and the ratio of correct and incorrect answers. This could potentially create a gap between the results of the current evaluation and the actual behavior of LLMs in real scenarios. Therefore, could you further evaluate the methods using real outputs from LLMs on randomly sampled questions. Please correct me if I have misunderstood anything.", "paper_topic_and_main_contributions": "This paper presents a method called SAC^3, which aims to detect hallucinations using black-box LLMs. The proposed method starts by paraphrasing a question into multiple semantically equivalent questions. It then checks the consistency of the answers provided by the LLM to both the original question and the paraphrased questions to determine if the answers are factual. Additionally, this paper suggests incorporating the answers from other LLMs in the consistency checking process. The proposed method demonstrates better performance compared to the existing self-consistency approach on four datasets. The paper also investigates the impact of hyperparameters such as the number of paraphrased questions and the choice of threshold value.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to address two types of hallucinations due to question-level and model-level that are beyond the existing self-consistency detection method. Reviewers appreciate that this is a nice contribution to hallucination detection which demonstrates strong performances. The author also provides further results on GPT-4 and PaLM 2 in their rebuttal to show it can consistently deliver improvements. In addition to these new results, we strongly encourage the authors to take the feedback to improve the paper over computational costs, its usefulness for open-source LLMs, and the unresolved challenge of detecting non-factual responses through inconsistency.
Decision: Accept-Findings