A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection
Authors: Shiping Yang
Renliang Sun
Xiaojun Wan
Venue: EMNLP 2023 Findings (2023)
Keywords: hallucination detection; LLM
TL;DR: A new benchmark and zero-resource method for passage-level hallucination detection

Abstract: Large Language Models (LLMs) have shown their ability to collaborate effectively with humans in real-world scenarios. However, LLMs are apt to generate hallucinations, i.e., makeup incorrect text and unverified information, which can cause significant damage when deployed for mission-critical tasks.
In this paper, we propose a self-check approach based on reverse validation to detect factual errors automatically in a zero-resource fashion. To facilitate future studies and assess different methods, we construct a hallucination detection benchmark named PHD, which is generated by ChatGPT and annotated by human annotators. Contrasting previous studies of zero-resource hallucination detection, our method and benchmark concentrate on passage-level detection instead of sentence-level.
We empirically evaluate our method and existing zero-resource detection methods on two datasets. The experimental results demonstrate that the proposed method considerably outperforms the baselines while costing fewer tokens and less time. Furthermore, we manually analyze some hallucination cases that LLM failed to capture, revealing the shared limitation of zero-resource methods.

A New Benchmark and Reverse Validation Method for Passage-level
Hallucination Detection
Shiping Yang" Renliang Sun", Xiaojun Wan
Wangxuan Institute of Computer Technology, Peking University
Center for Data Science, Peking University
The MOE Key Laboratory of Computational Linguistics, Peking University yangshipingnipegmal.. com sunrenl sunrenliangokudggail.. com wanxiaojun@pku.. edu.
Abstract
informative responses. This hallucination issue sig- nificantly undermines public trust in LLMs and limits their deployment in certain critical and sen- sitive domains such as legal (Huang et al., 2023), medical (Singhal et al., 2023), and finance (Wu et 2023).
Large Language Models (LLMs) have shown their ability to collaborate effectively with humans real-world scenarios. However, LLMs apt generate hallucinations, i.., makeup incorrect text and unverified informa- tion, which can cause significant damage when deployed for mission-critical tasks. In this pa- per, we propose a self-check approach based on reverse validation detect factual errors auto- matically in zero-resource fashion. To facili- tate future studies and assess different methods, we construct hallucination detection bench- mark named PHD, which is generated by Chat- GPT and annotated by human annotators. Con- trasting previous studies of zero-resource hallu- cination detection, our method and benchmark concentrate on passage-level detection instead of sentence-level. We empirically evaluate our method and existing zero-resource detection methods datasets. The experimental results demonstrate that the proposed method considerably outperforms the baselines while costing fewer tokens and less time. Further- more, we manually analyze some hallucination cases that L.M failed capture, revealing shared limitation of zero-resee aerorresource methods.
Some recent studies have attempted to alleviate the hallucination of LLMs. They emphasized that knowledge gaps in LLMs hallucination (Petroni et al., 2019; Lee et al., 2022: Zheng et al., 2023; McKenna et al., 2023), which has inspired a retrieval strategy to mitigate halluci- nation through interacting with external knowledge bases (Guo et al., 2022; Peng et al., 2023). More- over, there is growing research in zero-resource hallucination detection (Manakul et al., 2023; Co- hen et al., 2023). However, in our view, previous disadvan- tages: (1) Relying on external retrieval modules. Retrieving external knowledge often has complex processes and notable delays. In addition, external databases may also be inaccessible. (2) Only fo- cusing on sentence-level hallucination detection. Real-world upplications often require passage-level hallucination detection rather than sentence-leve detection. This arises from the fact that LLMs tend to furnish users with comprehensive and informa- tive answers instead of single sentence.
primary cause of
studies suffer from the following
1 Introduction
Recent research has shown that Large Language Models (LLMs), like ChatGPT and GPT-4 (Ope- nAI, 2023), have achieved state-of-the-art across various NLG tasks (Bubeck et al., 2023; Kocoñ et al., 2023; Qin et al., 2023). Despite their re- markable capabilities in various aspects, LLMs still suffer from some inherent drawbacks, underper- forming in scenarios involving complex reasoning (Frieder et al., 2023; Chen et al.. 2023) and global learning (Li et al., 2023a).
In many scenarios, a judgment about the entire passage enough, which enables a quick deci- sion on whether to activate the retrieval module and generate a new response. It's highly ineffi- cient and costly to access the truthfulness of a re- sponse sentence by sentence. Therefore, exploring passage-level hallucination detection holds greater practical significance than exclusively concentrat- ing on sentence-level detection. Unfortunately, search on passage-level hallucination detection is still scarce: to the best of our knowledge, there is no available method and suitable benchmark cur- rently.
One prominent weakness of LLMMs is their ten- dency hallucinate when generating fluent and
"Equal contribution
Work done during internship

---

To facilitate research on passage-level halluci- nation detection, we first annotate and propose the PHD (Passage-level Hallucination Detection) benchmark, a new benchmark for the evaluation passage-level detection methods. We then repli- cate some recent sentence-level detection methods and adapt them to passage-level detection. Due to the poor performance of these methods our benchmark, propose zero-resource hallucina- tion detection method named Reverse Validation (RV). RV is specifically tailored for passage-level detection, which can detect whether a passage con- tains factual errors without access to token-level probability distribution and external knowledge. We empirically evaluate the RV method on the PHD benchmark and the WikiBio-GPT3 dataset (Manakul et al., 2023). The results show that the RV method significantly outperforms existing methods and baselines, demonstrating competitive performance, lower token costs, and faster response times. In addition, we conduct an ablation study to explore the robustness and compatibility of our approach for different LLMs. Finally, we reveal the shared limitation of zero-resource detection meth- ods by investigating bad cases.
hallucinations refer to output that cannot be con- firmed or contradicted by the source. According to a recent study (Bang et al., 2023), intrinsic hal- lucinations are rarely observed in LLMs such as ChatGPT, whereas extrinsic hallucinations tend to occur more frequently. Therefore, when referring to hallucinations in LLMs, we primarily focus on extrinsic hallucinations.
It is worth noting that extrinsic hallucination doe not always involve errors, as it can origi- nate from factually correct external information (Maynez et al., 2020; Thomson and Reiter, 2020). Such factual hallucination can be helpful because it recalls additional background knowledge from the parametric knowledge base of LLMs to generate informative responses. Hence, in recent work on LLMs, the term "Hallucination" only used to de- scribe the phenomenon where the model generates non-factual statements or fabricates unverifiable knowledge (Manakul et al., 2023; Peng et al., 2023; Li et al., 2023b; Cohen et al., 2023).
2.2 Mitigate Hallucinations
With the rise of ChatGPT, there has been an in- creased awareness of its tendency to generate plausible-looking statements that are unverifiable or factually incorrect. To alleviate this issue, there are many active endeavors to analyze the causes of hallucinations and mitigate the hallucination of LLMs.
We are committed to facilitating research on passage-level hallucination detection. Our contribu- tions can be summarized as follows: (1) We create a high-quality benchmark named PHD for the evalu- ation of passage-level hallucination detection meth- ods, and it will become challenging benchmark for hallucination detection. (2) We propose the re- verse validation method to detect passage-level hal- lucinations, which can be used in black-box mod- els and zero-resource fashion. (3) We demonstrate the effectiveness and robustness of the proposed method by comparing it with existing methods on two datasets. Data and code will be released at https://ght thub.
Zheng et al. (2023) hypothesized that the hallu- cinations may be attributed to knowledge gaps in LLMs, and several works have shown the promise of using retrieval knowledge to mitigate them (Lewis et al., 2020; Shuster et al., 2021; Peng et al., 2023). A natural idea is whether LLMs tect hallucinations in the outputs without any exter- nal knowledge, only invoking the retrieval module when hallucinations occur to shorten the response time. This cerorresourc method can also serve as an alternative solution in situations where external databases are inaccessible.
de-
2
Related Work
2.1
Hallucinations of LLMs
Hallucination has been studied in different natu- ral language generation tasks (Dziri et al., 2022; Gupta et al., 2021; Maynez et al., 2020; Raunak et al., 2021), and its definition may vary depending on the specific task. A recent survey categorized hallucinations into two classes (Ji et al., 2023), in- trinsic hallucinations and extrinsic hallucinations. Intrinsic hallucinations refer to generated output that contradicts the original content, while extrinsic
One approach to detect hallucination without external knowledge is by utilizing intrinsic uncer- tainty metrics, such as token probability and per- plexity. However, the token-level probability distri- bution is inaccessible for current LLMs like Chat- GPT.
Therefore, there a growing interest applying zero-resource hallucination detection for black-box models (Manakul et al., 2023; Cohen et al., 2023).

---

Our work is closely related to these works, but we focus on passage-level zero-resource hallucination detection instead of sentence-level.
model. By accessing the data volume of specific entity within the training data, we can readily find entities that LM might not be acquainted with. Al- though ChatGPT's training data is not accessible, publiely available information is that it includes the C4 corpus (Raffel et al., 2020) and a large amount of data crawled online. Consequently, we propose simple method to proxy the data volume of an entity in the training data through the number of related items that Google Search returns. Then, we create a dataset consisting of entities and their cor- responding proxies of data volume. We categorize the entities into three domains based on distinct data volumes: PIID-Low (<100K), PIID-Medium (100K-1M), and PHD-High (> (>IM), respectively. We randomly sample 100 entities from each of the above domains to construct our PHD bench- mark. Then, we prompt ChatGPT² to generate a Wikipedia article about each entity using the prompt Please write a brief Wikipedia for [Entity)- Following these procedures, we have successfully prepared the data for annotation in the next stage.
3 The PHD Benchmark
In this section, we describe the detailed information of our PHD Benchmark. Specifically, show how we select entities generate hallucination data in Section 3.1. Then, we give the annotation process of PHD benchmark in Section 3.2. Finally, we report statistical information about our PHD benchmark in Section 3.3.
3.1 Data Generation
A primary use case of LLMs is answering ques- tions and seeking factual knowledge about an en- tity. Wikipedia contains plenty of entities and corresponding introduction articles, which serve as a commonly used data source for construct- ing fact verification datasets (Thorne et al., 2018). Hence, we extract entities from the Wikipedia dump' to generate passages and employ the as- sociated Wikipedia articles as references at the fol- lowing annotation round.
3.2 Human Annotation
We choose ChatGPT, currently the most widely used and representative LLM, generate hallu- cination data. However, as one of the most power- ful LLMs, ChatGPT rarely generates hallucinations when responding to common questions, resulting in an insufficient number of hallucination samples included in the benchmark. Previous work has addressed this issue by instructing LLMs to gen- erate hallucinations intentionally (Li et al., 2023b) generate data on specific topics to which the model is prone hallucinate (Manakul al., 2023). However, a benchmark that can effectively evaluate the performance of hallucination detec- tion methods should contain entities from various domains and hallucinations generated by models naturally, which better match the hallucination sce- narios methods may encounter in practical usage. Therefore, we propose different strategy to gener- ate as many hallucinated samples possible in the benchmark.
Annotating the hallucination at the passage level is very challenging task for human labelers, which requires workers search for evidence to check each claim within the passage, often from long document and various websites. Therefore, we design a two-stage human annotation process and select reliable workers through strict criteria to en- sure the quality of PHD benchmark.
Labels Definition The definition of hallucina- tions has been introduced in detail in Section 2. Following the previous work, we define three labels used in our annotation process: factatul every claim in the passage is supported by evidence. (2) non-factual: the passage consists of non-factual or unverifiable information (even a minor error). (3) unverifiable: label only used in the first stage when there is insufficient evidence to reject or sup- port claims in the passage by using Wikipedia as a reference.
factual error often occurs when a model lacks sufficient knowledge (Zheng et al., 2023). This phenomenon inspires us to generate hallucinations by selecting entities that lack enough facts in the
Worker Requirements The requirements for workers who perform annotation are as follows: (1) had education experience in university with at least bachelor's degree; (2) passed the TOEFL or IELTS exam; (3) good at using search engines
'The English Wikipedia dumps are from https:// 2821 1820.. The version we used is 2021 -021-08-20, which is released hefore the dead- line for training data of ChatGPT.
archive. www.vve.rgrowowwwo wwwg///wwoad//eemmmomooooo mawikii-22208220
We set the temperature to 0.0 when generating data to reduce randomness.

---

look up reliable information; (4) passed the cor- responding Qualification Test (QT) designed for our task (more details below). These requirements are to ensure that workers are proficient in English reading comprehension and can perform the hallu- cination annotation task.
LLM's output: Procamation 10043 an executive order signed by former Uniedd States President Donald Trump on January 2021, just two days belor
ieem officw. prodemasion on indviduals from certain countries, indluding Brazi and Europe, that had bean pet pace due lo the DDVID-19 pandamk:. However, the proclamation cid iill reinciion on Insel from Chins, Iran, and other countras..
lnvoll restictions
Entily: Prodiamation 10043
Query
Qualification Test We designed Qualification Test (QT) to measure the worker's hallucination detection ability. Before the QT, workers were re- quired to read the task guide about how to annotate the hallucination. We provided a detailed expla- nation of the passage-level hallucination detection task. After workers took the QT, all submissions were manually checked to filter out workers who could not perform annotation correctly. Finally, ten candidates took the QT, and three of them passed the test and entered the following annotation stage.
Question:
What ezeczulive orchr
sigrad by former Unitd President
tnaeel redrictins un indivituds from carlain countties,
from China, Iran, and other counthies, jast two days belore of affra??
Answer: Exucation
19988
Antity-Asswer Match
Two-stage Annotation The annotation task was divided into two stages. In the first stage, quali- fied workers were asked to annotate hallucination using the corresponding Wikipedia articles as ref- erences. If workers fail to gather information from Wikipedia to support claim, they will annotate the passage as unverifiable. In the second stage, we let workers re-annotate unverifiable passages through access to the Internet. The content they browsed was obtained from the official website on the first page of Google search results to ensure the reliability of the evidence. When the second stage finished, all the passages were classified as factual or non-factual. Except for amnotating labels, workers were required to mark the corresponding content they believed to be incorrect or unverifiable. This additional information can provide help in the evaluation of fine-grained hallucination detection.
Result: This containe hallucmations.
Figure I: The overview process of the reverse validation (RV) method
The entities of the PHD-High domain rarely con- tains hallucinations in their passages, while nearly half the passages of entities in the PHD-Low domain hallucinate. The different distributions of hallucinated samples in three domains of PHD sup- port the previous hypothesis that hallucinations are commonly caused by knowledge gaps in LLMs (Zheng et al., 2023; Ji et al., 2023).
Factual
Nan-factual
PHD-High
86
14
MeD-MMedium
76
24
PHD-Low
60
40
Quality Control We added some fake examples to ensure the quality of the benchmark. Concretely, some fake examples annotated by researchers were assigned to workers in each annotation stage. We use Cohen's 's Kappa (k) to measure their agreement with the labels of researchers and obtain an average k-0.488 (0.80<=k<=1.00) showing perfect con- sistency. We also compute Fleiss' Kappa (Fleiss, 1971), which achieves 0.77.
Table I: The annotation result of PHD benchmark
4 Method
In this section, we propose a new method named Reverse Validation (RV). RV can be used to de- tect whether passages generated by LLMs contain hallucinations.
4.1 Reverse Validation Method
3.3 Statistics
Our method is motivated by the insight of un- derstanding language models as knowledge bases (Petroni al., 2019). In brief, we regard LLMs as huge databases that store entities and related knowledge, capturing factual errors by construct-
As we introduced above, the hallucination data was generated by ChatGPT and subsequently annotated by workers. Table 1 shows the final annotation result of the PHD benchmark.

---

ing a prompt to retrieve the corresponding entity from LLMs. We next describe the three stages of using this method, with the overall process illus- trated in Figure! 1.
Reverse Validation via Question Generation This variant constructs a query using question gen- eration (QG). We prompt LLMs to generate a ques- tion about the entity based on the content of the re- sponse. One key point of implementation that the entity name cannot appear in the question, which will lead to a label leakage.
Construct Query We extract the en- tity of user concern from the response and use the remaining information to construct a query.
Stage 1:
Reverse Validation via Entity Matching This variant constructs query by instructing the LLMs to perform the entity matching (EM) task directly. We rewrite the relevant information into list of requirements and then request the LLM return an entity that can match the requirements. Consid- ering that the model may return an entity that meets part of the requirements when it fails to locate a perfectly matched entity, we require the model to report the percentage of requirements the entity satisfies while returning the entity, This percent- age plays crucial role in the entity-answer match stage, as only a match exceeding 90 percent can be recognized as factual. Furthermore, we observe that L.MSs will point out the requirements that are not met by the returned entity, thereby enhancing the interpretability of our method.
Access Databases Then, We use the query construct in the first stage to access "databases" and get corresponding answer.
Stage 2:
Stage 3: Entity-Ansser Match Finally, we assess whether the returned answer matches the initial entity, If there is match, it indicates that the response about the entity is factual. Otherwise, the response contains hallucinated information.
Stages I and 2 are processes of reverse proba- bility modeling, which can be defined as follows: Given a user query Q, the process of generating responses can be formulated P (R(Q), where R denotes the response. This formula be fur- ther simplified P EI). where denotes the entity we extracted, and denotes the remaining information. The process of constructing a query to access LLMs can be denoted as (E I), which indicates generating the corresponding entity with hard constraints. After we complete the reverse modeling to obtain the corresponding entity, we validate it through entity- matching to reach final decision. Therefore, named our method Reverse Validation (RV).
For both variants, we use exact string match- ing strategy to implement the third stage of the Reverse Validation method.
5 Experimental Setup
In this section, we introduce the model, the datasets, the evaluation metrics, and the methods that are compared in the experiments.
The RV method works based on the fact that if a response contains hallucinations, it becomes an in- correct search condition when we reconstruct it as a query to access LLMs. Wrong retrieval conditions will lead to LLM generating a wrong entity or even failing to find the relevant entity. In contrast, if the retrieval is successful, it indicates that the entity information is stored in parameterized knowledge rather than a product of hallucinations.
We use stable ChatGPT version (gpt- 3.5.turbo-0301' and set the generation tempera- ture to 0.0 in our experiments, which ensures the reproducibility of results to the greatest extent.
Model
Datasets In addition to testing the PHD bench- mark, our evaluation also incorporates the syn- thetic WikiBio dataset (Manakul et al., 2023). This dataset comprises 238 passages generated by GPT3 and annotated at the sentence level, with WikiBio (Lebret et al., 2016) serving as the reference source. While the dataset lacks passage-level labels, we can aggregate sentence labels to derive pseudo- labels at the passage level. There are very few completely correct passages in this dataset. Conse- quently, we extract subset of the synthetic dataset,
4.2 Implementation of Variants
Recent success in prompting enables us to query LLMs with human language (Brown et al., 2020; Liu et al., 2023). Therefore, method can be im- plemented by designing suitable prompts. Specif- ically, We design two variants of our RV method, i.e., two different ways of constructing a query. The prompts used for each variant at every stage of the RV method are shown in Table 2.
'nttps://plaffomm openai. wwwpaai.comddccs...o.// cwt////////////s//////////c gpt-3-5

---

Stage
Reverse Validation via QG
Reverse Validation via EM
(1) Construct Query
will give you some information about en- all this information
tity. You should generate question, and the answer your question is the entity. not include entity your question. (Example) Entity: (Entity Information: (Rexponse) Question: <Ques- tion>
(Response) Please list all features of [Entity] which are mentioned above with numbers, do not include { |Earity) in your list. <Requirements>
(2) Access Databases
You should find an entity that conforms the following description: (<Requirements>] If you fail to find a perfect match, please say an entity that matches the requirements as much as possible. You need to give the percentage of the entity that meets requirements.
You should answer the following question short possible. {<Question>}
Table 2: Prompts used for each variant every stage of RV method. <Question> and <Requirements> are generated by models in the first stage.
consisting of 200 passages that are determined to be non-factual. In our experiment, we referred to this subset as the WikiBio-GPT3 dataset.
hallucinations the passage level, the "exam- iner" concluded that all passages were correct. Upon examining the interaction history be- tween the two models, we discovered that the "examinee" LM answered questions based on the passage's content rather than retrieving relevant knowledge from its parameters. To resolve this issue, revised the "examinee" LM's setup by modifying the original prompt " "llease answer the following questions re- garding your claim: (Questions]" to "Please answer the following questions: /Questions]" and restricted its access to the dialogue his- tory.
Evaluation Metries We evaluate how well the method detects passages that are non-factual using the portion of non-factual passages of the passages rejected by the detection method; Recall: the portion of passages rejected by the detection method out of all the non non-factual passages; F1: the harmonic mean precision and recall; Accuracy: the portion of passages rejected by the detection method out of all the passages. We only report Accuracy on the WikiBio-GPT3 dataset.
the following metrics:* Precision:
SelFCheckGPT via BERTScore A variant of SelfCheckGPT (Manakul et al., 2023), ing BERTScore (Zhang al., 2019) mea- sure the consistency between the response and stochastic samples. However, this method returns sentence score between 0.0 to 1.0 instead of label that indicates factual or non- factual. We derive passage-level scores by averaging the sentence-level scores. To align this method with our task, we then optimize threshold over the WikiBio-GPT3 dataset. If passage obtains a score higher than the thresh- old, it will categorized as non-factual. We only report SelfCheckGPT via BERTScore on the PHD benchmark.
Considering that there is no method for passage-level hallucination detection, we replicate the existing zero-resource detection methods and modify them for passage-level detec- tion. We also add two variations of our approach.
Compared Methods
All False: A straightforward approach that assigns the label "non-factual" to all the pas- sages as a baseline.
LMvsLM revised: LMvsLM, cited as the cur- rent leading method for detecting hallucina- tions at the sentence level (Cohen et al., 2023), is inspired by truth- mechanisms in the legal field. It establishes multi-tuma interac- tion between an "examiner" LM and "exam- inee" LM to capture contradictory statements. Following the setting in their experiments, we employed ChatGPT to fulfill different roles within the LMvsLM framework. However, when we applied this method directly to detect
Recent research has demonstrated the capability of LLMss iden- tify the presence of hallucinated information in various natural language generation tasks,
Zero-shot Detection:
'We use the implementation of SelfCheckGPT re- leased hy the author at: https://githup wwt. patp///etwwee/////m selfcheckgpt
'The detection method will "rejects" passage when iden- tifying passage containing hallucinations.

---

even in a zero-shot manner (Li et al., 2023b; Cohen al., 2023). We revised their instruc- tions slightly to perform our passage-level de- tection task. We use the following prompt: want you to act as a claim judger. Given claim about an entity, your objective is to determine if the provided claim contains non-factual hallucinated information. You should give your judgment based world knowledge, and answer with factual or non- factual. (Claim).
Validation via EM is better than Reverse Vali- dation via QG, indicating that the percentage of requirements fulfilled by an entity can be helpful to make a better decision in the Entity-Answer Match stage.
While both variants of our method exhibit good performance in the PHD-LOW and domains, they all struggle when it comes the PHD-High domain. This phenomenon suggests that there might be an implicit relationship between the difficulty of hallucination detection and the data volume of entities in the training data.
Reverse Validation via Question Genera- tion (QG): A variant of reverse valida- tion method, using question generation to con- struct query access the parametric Knowl- edge base of LLMs.
In Table 4, we present the accuracy of all meth- ods when evaluated on the WikiBio-GPT3 dataset. To measure whether method is sensitive fine- grained hallucinations, we also report the mini- mum value of the hallucination ratio among the passages that have been rejected. Using this met- ric, we find that the two variants of our reverse validation method can detect fine-grained halluci- nations, even when the passages only contain one non-factual sentence. In contrast, LMvsLM re- vised fail to identify non-factual passage with hallucination ratio below 60 percent.
Reverse Validation via Entity Matching (EM): A variant of our reverse validation method, instructing LLMs to perform the en- tity matching task to construct a query to ac- cess the parametric Knowledge base of LLMs.
Please refer to Section 4.2 for the concrete im- plementation of two variants.
Last, we calculate the average token cost and time delay for each method on the PHD bench- mark. The results shown in Table demonstrate that our methods have significant advantages in terms of token cost and response latency, which can be applied to production-level systems.
6 Results
6.1
Main Results
Table 3 shows the result of our experiments on the PHD benchmark.
6.2 Ablation Study Results
After revising the setting of LMvsLM to make it suitable for passage-level hallucination detection, this method achieve the highest Precision score among all the baselines. Nevertheless, it receives a lower score terms of Recall.
We perform ablation study to understand the im- portance of the backbone models for RV method. Therefore, we employ Llama-2-7b-chat-hh an open-source LLM, as the backbone model and con- duct experiments 7 on the PHD benchmark. Results are reported Table 6.
In order to test SeIfCheckGPT via BERTScore on the PHD benchmark, we optimize classifi- cation threshold for hallucination detection using KikiBio-GPTT dataset. However, the domain shift issue caused by different sources of two datasets (generated by different L.Ms)) makes the optimal threshold difficult to generalize to the PHD bench- mark, which result in the poor performance Self- GPTckGPT via BERTScore. Interestingly, we ob- serve that Zero-shot Detection completely fails to detect hallucination at passage-level, categorizing all the passages as factual labels.
It is worth noting that the ablation experiments using Llama-2-7h-chat-hf are not conducted in self-check scenarios since the PHD benchmark generated by ChatGPT. This change in the experimental setup considcrably impairs the perfor- mance of LMvsLM revised and SeIPCheckGPT via BERTScore.
For LMvsLM revised, the "examinee" Llama2 might provide answers that contradict the claim
"The hallucination ratio can calculated by dividing the count of non-factual sentences the passage by the total number of sentences in the passage. We calculate for each passage in the WikiBio-GPT3 dataset.
Across all the domains of PHD, two variants of our method outperform the basclines, often by a large margin. Notably, the performance of Reverse
"We set the temperature to 0.1, top to 0.05, top_k to 1.

---

PHD
PHD-LOW
PHD-Median
PHD PHD-High
F1
P
R
F1
F1
P
F1
P
R
All False
41.3
26.0
100
57.1
40.0
100
38.7
24.0
100
24.6
14.0
100
M revised
25.5
75.0
15.4
25.5
85.7
15.0
34.5
100
20.8
11.1
25.0
7.1
SeliCheckGPT via BERTScore
41.1
26.0
97.4
56.1
39.4
97.5
39.7
24.7
100
23.6
13.5
92.8
Zero-shot Detection
0.0
0.0
0.0
0.0
0.0
0.0
Reverse Validation via QG
40.7
73.1
58.6
44.7
85.0
53.3
44.4
33.3
25.5
50.0
Reverse Validation EM
59.1
48.0
76.9
63.6
50.7
64.2
58.6
70.8
41.9
31,0
64.3
Table 3: Fl scores, Precision (P), and Recall (R) for baselines and two variants of our RV method on the PHD benchmark. We also report the metrics on three subdomains of PHD benchmark. We use Bold to mark the best result and underline the second-best result.
generated by ChatGPT due knowledge gaps be- tween the two models. Consequently, the "exam- iner" classifies most claims as non-factual.
Accuracy
Min Hal. Ratio
LMvsLM revised
29.8%
60.0%
Zero-shot Detect.
0%
N/A
As sampling-based approach, SeIfCheckGPT via BERTScore using BERTSccore measure the consistency between the response and stochastic samples. When different model employed for sampling, the generated samples can diverge sig- nificantly from the original response, resulting in this method devolving into an "All false" baseline. Therefore, this method is only suitable for self- check scenarios.
RV via QG
74.2%
10.0%
RV via EM
67.2%
10.0%
Table 4: The result our experiments on WikiBio- GPT3 dataset. Min Hal. Ratio means the minimum Hallucination Ratio among the passages that have been rejected.
Token Cost
Time Delay (sec.)
RV via QG RV via EM
297.3
2.65
Both ChatGPT and Llama-2-7b-chatte fail detect passage-level hallucinations a zero- shot setting. This phenomenon demonstrates that passage-level detection is more challenging than sentence-level detection for LLMs, particularly when the hallucination is generated by models themselves.
510.4
7.25
LMvsLM rev.
5974.5
29.48
SelfCheckGPT.
435.7
122.4
Table 5: Average token cost and time delay of different methods on the PHD benchmark. We use SelfCheck- GPT. to represent SelfCheckGPT via BERTScore.
Unlike ChatGPT, Llama-2-7b-chat-nf is smaller and less powerful LLM, and one would expect degraded performance with Llama-2-7b- chat-hf as the backbone model. The results of RV via EM align with our expectations. However, we observe that the performance of RV via QG re- mains relatively stable when the backbone model is changed, which indicates the robustness of this variant.
understanding of the failure cases.
7.1 False Negative
Table provides two instances of false negatives, which were identified as hallucinations mistakenly by Reverse Validation via EM.
For the entity "Maximilian II Emanuel", LLM answers abbreviation rather than its full name. In another case, the model provides general entity "Jordan" instead more specific entity "History of Jordan". These situations are very common in our method. However, our automatic matching pro- cess, which relies on exact string matching, catego- rizes all these cases as non-factual due to minor string discrepancy.
Contrasting with other baselines, our methods are effective not only in self-checking scenarios but also in situations using other LLMs. We can even achieve competitive performance using a small LLM like Llama-2-7b-chat-ht. In summary, our reverse validation method shows strong robustness and compatibility across different settings.
7 Case Study and Analysis
The poor performance of exact string matching is the primary factor causing false negatives. order to address this issue, we employ LLM to perform fuzzy matching, using the prompt Please
We manually analyzed the false negative and false positive examples of our method to give a better

---

PHD
PHD-LOW
PHD-Medium
PHD-High
F1
R
F1
P
R
F1
R
F1
P
All False
41.3
26.0
100
57.1
40.0
100
38.7
24.0
100
24.6
14.0
100
M. revised
41.5
26.3
98.7
57.6
40.4
100
40.0
25.0
100
23.2
13.3
92.9
SeliChcekGPT via BERTScore
41.3
26.0
100
57.1
40.0
100
38.7
24.0
100
24.6
14.0
100
Zero-shot Detection
2.5
1.3
0.0
0.0
13.3
Reverse Validation QG
50.9
35.5
89.7
58.0
41.8
95.0
52.9
36.5
95.8
31.6
20.9
Reverse Validation EM
50.0
34.7
89.7
42.4
97.5
44.2
87.5
37.7
25.6
Table 6: Ablation study results on the PHD benchmark. We use Bold to mark the best result and underline the second- best result.
identify whether the above answer refers to (En-
Entity Maximilian II Emanuel
Model Answers
The entity that matches the requirements 100% is Max Emanuel, Elector of Bavaria. Jordan matches the require- 100% performance
tity/. Nevertheless, we have observed that this ap- proach significantly impairs the recall metric while improving the precision metric.
History of Jordan
The aforementioned unalysis demonstrates that the precision our approach can further improve if better automatic matching method can be avail- able.
Table 7:
Two cases False Negative. Model Answers the answer get in stage of RV via EM. We use red to highlight the mismatched entity.
Entity
None-Factuse Part
7.2 False Positive
National
The National Hockey League (NHL) professional ice hockey Icague in North Amer-
Hockey
Table presents two cases false positives, which our method mistakenly identified as factual. Due to the page limit, we only showed the non-factual part in the table.
League
comprising 31 teams: 24 the United States and
Canada
Lacrimo
They have released studio albums and are known their dramatic and emotional live performance
We check the non-factual part that has been marked by the annotator to analyze why our method failed to detect the hallucination in these cases. In the case of "National Hockey League", " a team joined the league in 2021, causing a change in the number of league teams. Similarly, the band Lacrimosa released new album in 2021, changing the total number of albums. However, the train- ing data, which serves the knowledge source of LLMs, does not update when changes happen the real world. Therefore, when we construct a query using outdated information, our reverse validation method may mistakenly categorize the case as factual due to successful access to the corre- sponding entity in the parametric knowledge base of LLMs.
Table Two cases of False Positive. We red highlight the non-factual information.
from outdated data rather than inadequate data, the model exhibits a high level of confidence in its responses, and almost no inconsistencies occur.
Conclusion
In this paper. we are committed to facilitating re- search on passage-level hallucination detection. We introduced the PHD benchmark, a new dataset for evaluating passage-level hallucination. Then, proposed new zero-resource hallucination de- tection method and demonstrated its effectiveness and robustness by comparing with existing meth- ods on two datasets. Finally, we manually analyzed the failure cases and revealed the shared limitation of zero-resse resoures methods.
This type of hallucination caused by outdated data also explains why hallucinations still occur in the PHD- PHD-Higg domain with adequate data. Unfor- tunately, existing zero-resource hallucination detec- tion methods incapable of detecting this specific type of hallucination. This limitation is inherent for zero-resource hallucination detection methods since their fundamental principle is detect hal- lucinations by capturing inconsistencies LLMss. However, when the knowledge gap in LLMs arises
Limitations
An inherent drawback of our Reverse Validation method its inability to identify a specific type of

---

hallucination caused by outdated data. This is also a common flaw in current zero-resource detection methods. In addition, our approach is specifically tailored for passage-level hallucination detection and may not suitable for sentence-level detection.
Roi Cohen, May Hamri, Mor Geva, and Amir Glober- son. 2023. Lm vs Im: Detecting factual errors via cross examination. arXiv preprint arXiv:2305. 133211.
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022. Evaluating attribution in dialogue sys- tems: The begin benchmark. Transactions of the Association for Computational Linguistics, 10:1066- 1083.
Ethics Statement
Our dataset was generated by ChatGPT and then annotated by hired workers. There are no intel- lectual property disputes for our data source. All annotations are collected exclusively from work- ers employ, and they adhere the relevant code of ethics. Furthermore, we pay annotators ap- proximately $8 per hour, which is above the local minimum wage standard.
Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin. 76(5):378.
Simon Frieder. Luca Pinchetti, Ryan-Rhys Grif- fiths, Tommaso Salvatori, Thomas Lukasiewiez. Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023. Mathematical capabilities of chatgpt. arXiv preprint arXiv:2301.13867
Acknowledgements
Zhijiang Guo, Michael Schlichtkrul,, and Andreas Vla- chos. 2022. A survey automated fack-che Transactions of the Association for Computational Linguistics, 10:178-206.
This work was supported by National Key R&D Program of China (2021 YFFO90155)))) National Science Foundation of China (No.62161 State Key Laboratory of Media Convergence Pro- duction Technology and Systems and Key Lab- oratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We appreciate the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.
Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2021. Dialfact: A benchmark for fact-checking in dialogue. arXiv preprint 10.02221
Quzhe Huang. Mingxu Tao, Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023. Lawyer Ilama technical report. arXiv preprint 23x.:23005.5066
Ziwei Ji. Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38.
References
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, al. 2023. A multi- task, multilingual, multimodal evaluation chatgpt on reasoming, ballucination, and interactivity, arXiv preprint 2302.2200000222
Jan Kocoñ, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydio. Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. 2023. Chatgpt: Jack of all trades, master of none. Information Fusion, page 101861.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with ap- plication to the biography domain. arXiv preprint arXiv: 1603.07771.
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lec, Yin Tat Lec, Yuanzhi Li, Scott Lund- berg, al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint 12221.2303.12712
Nayeon Lee, Wei Ping. Peng Xu, Mostofa Patwary, Pas- cale Fung. Mohammad Shoeybi, and Bryan Catan- zaro. 2022. Factuality enhanced language models for open-ended text generation. Advances in Neural Information Processing Systems, 15:34586-34599
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-angmented generation for knowledge-intensive nlp tasks. Advances Neu- ral Information Processing Systems, 33:9459-9474.
Wenhu Chen, Ming Yin. Max Ku, Elaine Wan, Xueguang Ma, Jianyu Xu, Tony Xia, Xinyi Wang, and Pan Lu. 2023. Theoremqa: A theorem- driven question answering dataset. arXiv preprint arXiv:2305. 12524.

---

Dawei Li, Hengyuan Zhang, Yanran Li, and Shiping Yang. 2023a. Multi-level contrastive learning for script-based character understanding.
the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1172-1183 Online. Association for Computational Linguistics.
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian- Yun Nic, and Ji-Rong Wen. 2023b. Halueval: large- scale hallucination evaluation benchmark for large language models. arXiv e-prints, pages ar Xivv-22005
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784-3803, Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55()))1335.
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023. Towards expert-level medical question an- swering with large language models. arXiv preprint 17Xiv:3305.0617.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. arXiv preprint arXiv:2303. 08896..
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguisties.
Craig Thomson and Ehud Reiter. 2020. A gold standard methodology for evaluating accuracy in data-to-text systems. In Proceedings of the 13th International Conference on Natural Language Generation, pages 158-168.
James
Thorne, Andreas Vlachos, Christos Christocdoulppoulss, and Arpit Mittal. 2018. Fever: scage dataset for fact extraction and verification. arXiv preprint arXiv:1803.055
Nick McKenna, Tianyi Li, Liang Cheng, Moham- mad Javad Hosseini, Mark Johnson, and Mark Steed- man. 2023. Sources hallucination by large lan- guage models inference tasks. arXiv preprint arXiv:2305.14555 14552..
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kam- badur, David Rosenberg. and Gideon Mann. 2023. Bloombergggt: large language model for finance. arXiv preprint arXiv:2303.0 230..1564..
OpenAl. 2023. Gpt-4 technical report.
Baolin Peng. Michel Galley, Pengeheng He, Hao Cheng. Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint 12811
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.0967
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 2463-2477
Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. 2023. Why does chatgpt fall short in answering ques- tions faithfully? arXiv preprint arXiv:2304. 10513.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? arXiv preprint wxXv:2302.064777
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020, Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, (1(5555485-
Vikas Raunak, Arul Menezes, and Marein Junezys- Dowmunt. 2021. The curious case hallucinations in neural machine translation. In Proceedings of


Reviews:
[{"review_id": "G8mIx6YMSI", "reviewer": "Reviewer_KooH", "summary": "", "questions": "1. What is the number and domain of entities extracted from the Wikipedia dump?\n2. How to conduct the qualification test and what is the labeling task? What content is shown to human labelers and what is been output?\n3. What is the filtering criteria to select the final qualified workers?\n4. An illustrative process of the annotation process might be helpful to understand the method.\n5. In the Stage 1 of reverse validation method, how to determine and extract the entity that is questioned? In the Stage 3, the matching method is exact match or fuzzy match?\n6. From the results in Table 3, the precision and recall of proposed method are much lower than other baselines (which contrasts with the result analysis) but get the best F score. Can you explain which score is more important and why does this phenomenon occur? \n7. The WikiBio dataset is post-processed by aggregating sentence labels. Does it lead to an unfair comparison between the proposed method to previous methods? And is it possible to aggregate into incorrect passage label?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and valuable questions. We appreciate the opportunity to answer your question about our paper.\n\n**For Question1:**\\\nWe give the version of Wikipedia dumps in the footnote of page 3, which includes 6,361,315 entities. Due to Google Search's limitations, we created a dataset of 30,000 items consisting of entities and their corresponding proxies of data volume. Our PHD Benchmark was constructed based on this dataset (Lines 224-230). We randomly sample entities to ensure the diversity of our benchmark (covering diverse domains like biology, astronomy, and geography). In addition, you can redirect to the entity's Wiki page through the link we gave in the dataset.\n\n**For Question2:**\\\nWe distributed questionnaires on the \"Wenjuanxing\" (a platform providing functions equivalent to Amazon Mechanical Turk) to conduct the qualification test (QT). The labeling task that annotators do in the QT is the same as the task they will do in the annotation stage (described in Lines 277-294), i.e. annotating the hallucination at the passage level.\n\nWe defined the labels that should be outputted by human annotators in the \"Labels Definition\" part (Lines 244-254) and described the content and tool that annotators used in the \"Two-stage Annotation\" part (Lines 277-294). \nWe list the input and output of human annotators in the following to clarify your confusion.\n```\nInput: Passage generated by LLM; Wikipedia articles as references; Browser(only used in the second stage)\nOutput: Label; Evidence (the content they believed to be incorrect or unverifiable)\n```\n\n**For Question3:**\n\nWe describe our criteria for annotators in the \"Worker Requirements\" part (Lines 255-265). We filter out unqualified annotators through their accuracy on QT to select the final qualified annotators (Lines 272-274). Questions in QT had been annotated by researchers in advance, filtering by this criteria can ensure the annotators passed QT have a high agreement with researchers\n\n**For Question4:**\\\nThanks for your suggestions. An illustrative picture of the annotation process will make it easy for the community to follow and refer to our annotation process. We will add it in the camera-ready version.\n\n**For Question5:**\n\nIn Stage 1 of the reverse validation method, you can prompt the LLM to extract the entity in a zero-shot style. We give the template as follows: \n```\nWhich entity is the following passage mainly describing？ Only extract the entity name.\nPassage: {Passage}\n```\nIn practical application scenarios, we can directly extract the entity from the user's query to save token costs.\n```\nWhich entity is the following query mainly asking？ Only extract the entity name. \nQuery: {query}\n```\nIn Stage 3, the matching method is an exact string match (Lines 401-403). We also discuss the drawbacks of this matching method in Section 7.1. The Precision of our approach can further improve if a better automatic matching strategy can be employed. \n \n**For Question6:**\n\nThe F1 score is the harmonic mean of Precision and Recall, which can indicate the performance of the method more accurately. Even a bad method can easily attain a high score in either Precision or Recall (e.g. \"All False\" Baseline), but they can't achieve high scores in both simultaneously. In summary, F1 score is more important.\\\nIn Table 3, the two variants of our method achieve a trade-off between P and R compared to other baselines. Therefore, the proposed method gets the best F score. We firmly believe there is no inconsistency between experimental results and analysis.\n\n**For Question7:**\n\nAll of the baselines are tested on the WikiBio dataset with the aggregating labels. Our strategy for aggregating labels is described below: *a passage is labeled as a hallucination if one of its sentences (or more) is a hallucination in the original dataset.*\\\nThere is no possibility of aggregating into incorrect passage labels if the annotation of the original dataset is correct. Therefore, our comparisons on WikiBio dataset are fair.\n\nWe also want to reply to your **reasons to reject**.\\\nRegarding the **missing details**, we have carefully addressed your questions and will include them, along with the missing references, in the camera-ready version. For your **third rejection reason**. We believe our answer to question 6 could resolve your concerns about the **inconsistency between experimental results and analysis**.\n\nThanks again for your time and valuable comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. New resources. This paper presents a new hallucination detection dataset, which might be helpful for future work to evaluate LLMs. Besides, the construction process can inspire following studies in this field.\n2. New method. This paper proposes a hallucination detection method at the level of passage, which is different from the previous work at the sentence level.", "reasons_to_reject": "1. Missing details about the process of automatic data generation and human annotation.\n2. Missing details about the reverse validation method.\n3. Inconsistence between experimental results and analysis.", "questions_to_authors": "1. What is the number and domain of entities extracted from the Wikipedia dump?\n2. How to conduct the qualification test and what is the labeling task? What content is shown to human labelers and what is been output?\n3. What is the filtering criteria to select the final qualified workers?\n4. An illustrative process of the annotation process might be helpful to understand the method.\n5. In the Stage 1 of reverse validation method, how to determine and extract the entity that is questioned? In the Stage 3, the matching method is exact match or fuzzy match?\n6. From the results in Table 3, the precision and recall of proposed method are much lower than other baselines (which contrasts with the result analysis) but get the best F score. Can you explain which score is more important and why does this phenomenon occur? \n7. The WikiBio dataset is post-processed by aggregating sentence labels. Does it lead to an unfair comparison between the proposed method to previous methods? And is it possible to aggregate into incorrect passage label?", "paper_topic_and_main_contributions": "This paper proposes a new hallucination detection benchmark, named PHD. Correspondingly, the authors propose a self-check approach\nbased on reverse validation to detect factual errors automatically at the passage level.\n\nThe main contributions of this work are twofold. First, the proposed benchmark focuses on entities from Wikipedia compared to previous work. Second, the self-check hallucination detection method is performed at the passage level.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 324, Language -> language"}, "extra_text": {"Missing_References": "Some important work about large language models and hallucination:\n1. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., ... & Wen, J. R. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223.\n2. McKenna, N., Li, T., Cheng, L., Hosseini, M. J., Johnson, M., & Steedman, M. (2023). Sources of Hallucination by Large Language Models on Inference Tasks. arXiv preprint arXiv:2305.14552."}}, {"review_id": "mOnzWZztuC", "reviewer": "Reviewer_JYbz", "summary": "", "questions": "- If we can adapt sentence-level hallucination methods for passage-level hallucination methods, why do we need passage-level hallucination methods? Specifically, while the RV method’s numbers are convincing, it would be interesting to know the ***motivation*** for what makes the two problems distinct and worth studying separately.\n- Is it possible to have an analysis of how many questions do not have key information in the RV-QG method based on a sample set? Identifying when and how often the model fails could inform future research in this direction.\n- How is the “Access Database” step performed? Is it by querying the same LLM?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for investing your time and effort in the evaluation of our work. We appreciate the opportunity to answer your question about our paper.\n\n**Question1:Why do we need passage-level hallucination methods?**\n\nIn fact, we have adapted some current leading sentence-level methods for passage-level hallucination detection in our experiment (LMvsLM and Zero-shot Detection). We have analyzed why these methods demonstrate a complete inability to detect passage-level hallucinations in Lines 450-458 and Lines 512-519 separately. Therefore, we need methods specifically designed for passage-level hallucination detection.\\\nThanks for your constructive suggestion. We want to use a paragraph to further clarify **our motivation for studying hallucination detection on the passage level**. \n```\nReal-world applications often require passage-level hallucination detection rather than sentence-level detection. This arises from the fact that LLMs tend to furnish users with comprehensive and informative answers instead of a single sentence (Lines 56-60). Indeed, when assessing the truthfulness of a response, it is highly inefficient and costly to perform hallucination detection on each sentence within the passage [1]. In many scenarios, a judgment about the entire passage is enough, which enables a quick decision on whether to activate the retrieval module (Lines 149-155) and generate a new response. Therefore, exploring passage-level hallucination detection holds greater practical significance than exclusively concentrating on sentence-level detection. Unfortunately, the existing research mainly focuses on sentence-level detection, proposing methods and datasets that are crafted for sentence-level detection. \n```\n**[1]: We calculate the token cost and time delay for our method and other sentence-level methods.**\\\nThe results are shown in the following table.\n\n|                            | Token Cost | Time Cost(second) |\n|----------------------------|------------|-----------|\n|            RV-EM           |    510.4   |    7.25  |\n|            RV-QG           |    297.3   |    2.65   |\n|       LMvsLM       |   5974.5   |   29.48   |\n| SelfCheckGPT via BertScore |    435.7   |   122.4   |\n\nTherefore, We firmly believe **it is meaningful to study passage-level hallucination detection** and we will refine our motivation in the camera-ready version.\n\n**Question2: Analysis of how many questions lost information in the RV-QG**\n\nIn fact, our RV-EM variants (Lines 384-400) have resolved the question of missing key information by listing all features of the entity mentioned in response.\nWe follow your suggestion to perform an analysis of how many questions lost information in the RV-QG method and how many bad cases are caused by missing key information.\\\nWe conducted an analysis by randomly selecting 30 samples from the output of the RV-QG method. The findings revealed that out of the 30 samples, **26 exhibited a certain degree of information loss**, while **9 samples contained incorrect labels**. Then, we analyzed **the causes of error** for these 9 examples and showed the result in the following table.\n\n| Matching Errors | Lost Key Inf. | Others |\n|:---------------:|:-------------:|:------:|\n|        3        |       4       |    2   |\n\nThe analysis results clearly highlight that the primary factor influencing the performance of the RV-QG method is the loss of key information. This finding aligns with the conclusion we presented in Lines 533-535. While information loss is a common phenomenon within this method, it's important to note that **errors only emerge when substantial or essential information is missing**.\n\n**Question3: Is the “Access Database” step performed by querying the same LLM?**\n\nYes. We use the prompt we constructed in Stage 1 to query the same LLM (Lines 335-337). Our method is a **self-check** method based on reverse validation (Line 8).\\\nNevertheless, you can also query another LLM to detect the hallucination. We follow the suggestion of reviewer 2PhM to add experiments using the latest open-source LLM **Llama-2-7b-chat-hf**. The Experimental results demonstrate that even with the utilization of a smaller LLM (**Llama-2-7b-chat-hf**) for detecting hallucinations generated by ChatGPT, our method still attains competitive performance.\n\n| (F1/P/R) |      PHD       |    PHD_Low   |   PHD_Medium   |    PHD_High    |\n|:--------:|:--------------:|:------------:|:--------------:|:--------------:|\n|   RV-EM  |  50/34.7/89.7  | 59/42.4/97.5 | 44.2/29.6/87.5 | 37.7/25.6/71.4 |\n|   RV-QG  | 50.9/35.5/89.7 |  58/41.8/95  | 52.9/36.5/95.8 | 31.6/20.9/64.3 |\n\n\nRegarding **the transparency of the collection procedure**, we will add the following ethical statement to clarify it in the camera-ready version. \n```\nOur dataset was generated by ChatGPT and then annotated by hired workers. There are no intellectual property disputes for our data source. All annotations are collected from workers we employ and adhere to the relevant code of ethics. We pay annotators approximately $8 per hour,  which is above the local minimum wage standard\n```\nOnce again, we express our appreciation for the valuable feedback provided by you. We will add the missing section and analysis in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper addresses a highly relevant problem in the current era, where language models are being considered for production-level systems to prevent the dissemination of misinformation.\n- The data curation procedure employed by the authors is fairly convincing, and the reported annotator agreement scores indicate that the collected dataset is of high quality.\n- The proposed RV method also outperforms baselines by a large margin on the proposed benchmark and serves as a useful baseline for future work to build upon.", "reasons_to_reject": "Lack of transparency on collection procedure: A primary contribution of this paper is the PHD benchmark for detecting hallucinations. However, the payment structure and templates used by the author(s) have not been disclosed in this document or an ethical statement.", "questions_to_authors": "- If we can adapt sentence-level hallucination methods for passage-level hallucination methods, why do we need passage-level hallucination methods? Specifically, while the RV method’s numbers are convincing, it would be interesting to know the ***motivation*** for what makes the two problems distinct and worth studying separately.\n- Is it possible to have an analysis of how many questions do not have key information in the RV-QG method based on a sample set? Identifying when and how often the model fails could inform future research in this direction.\n- How is the “Access Database” step performed? Is it by querying the same LLM?", "paper_topic_and_main_contributions": "Current LLMs are prone to generating hallucinatory content that can be detrimental when dealing with critical tasks. This paper addresses the question of detecting such hallucinations in passage-level content generated by LLMs. Due to the lack of available resources to study this problem, the author(s) introduce the PHD dataset, a dataset of passages generated by ChatGPT that have been annotated based on the presence of hallucinatory content. To detect hallucinations in generated passages, the author(s) propose the Reverse Validation (RV) method (with two different implementations). Their best RV method can outperform baselines by at least 10% in terms of F1 on the various splits of PHD.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 520: we revised → revising"}, "extra_text": {}}, {"review_id": "Igd9ULx8wL", "reviewer": "Reviewer_2PhM", "summary": "", "questions": "1) You mention a very high annotator agreement in the \"Quality Control\" paragraph. \nDid you calculate it for the whole dataset or just a small subset of fake examples? If it's the latter, can you please explain why?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for investing your time and effort in the evaluation of our work. We appreciate the opportunity to answer your concerns about our paper.\n\n**For your concern about the reproducibility of our results:** \\\nWe use a stable ChatGPT version in our experiment (gpt-3.5-turbo-0301 **[1]**) and set the temperature to 0, which ensures the reproducibility of our results to the greatest extent.  \n\nAlthough we use a stable API version to ensure reproducibility, we still follow your suggestions to add experiments using the latest open-source LLM Llama-2-7b-chat-hf. Due to the time limit, we only report the result of our method, other baselines' results will be reported in the camera-ready version.\n\n| (F1/P/R) |      PHD       |    PHD_Low   |   PHD_Medium   |    PHD_High    |\n|:--------:|:--------------:|:------------:|:--------------:|:--------------:|\n|   RV-EM  |  50/34.7/89.7  | 59/42.4/97.5 | 44.2/29.6/87.5 | 37.7/25.6/71.4 |\n|   RV-QG  | 50.9/35.5/89.7 |  58/41.8/95  | 52.9/36.5/95.8 | 31.6/20.9/64.3 |\n\n\nThe experiment result shows that our method is generalizable for different LLMs. We can even achieve a competitive performance using a small LLM. Codes and the hyperparameters we used in the experiments also will be released to the community to ensure the reproducibility of our results.\n\n*[1] Official document about the API version:\nhttps://platform.openai.com/docs/models/gpt-3-5, \nwhich introduces gpt-3.5-turbo-0301 as \"Snapshot of gpt-3.5-turbo from March 1st 2023. Unlike gpt-3.5-turbo, this model will not receive updates, and will be deprecated on June 13th 2024 at the earliest.\"*\n\n**For your concern about why the LMvsLM baseline failed:** \\\nWe also observe the same phenomenon that ChatGPT's performance varies when we perform math reasoning tasks using different versions of ChatGPT API. \n\nHowever, the failure of LMvsLM is not caused by the degraded performance of ChatGPT. A version issue of ChatGPT would not result in a total inability for passage-level hallucination detection. According to our experiments and findings, the \"examinee\" LM tends to answer questions based on the passage content when it meets a long context (Lines 454-458). Contradictory statements will not occur in this situation. Therefore, the original LMvsLM method completely fails to detect passage-level hallucinations. \n\nWe have resolved this issue successfully by restricting the \"examinee\" LM's access to the dialogue history (Lines 454-465). In brief, the revised version of \"examinee\" LLM can't see the passage content in dialogue history and only answers questions by retrieving relevant knowledge from its parameters.\n\n**For your concern about the limitation of our method:** \\\nWe will discuss it in the \"limitation\" section.  \n\nAlthough our method detects hallucinations by constructing a prompt to retrieve the corresponding entity from LLM (Lines 325-331), it can detect any non-factual knowledge about entities, not just entities themself.  In addition, a primary use case of LLMs is answering questions and seeking factual knowledge about an entity (Lines 179-180). Therefore, the application of our method is general and broad in real-world scenarios.\n\n**For your concern about the calculation of the annotator agreement:** \\\nWe only calculate the annotator agreement on our fake examples (Lines 295-303) due to the high cost of human annotation. If we want to calculate it for the whole dataset, we should annotate each example with three labels, incurring a cost three times higher. It needs to be emphasized that the quality of our dataset is guaranteed as annotators are selected through rigorous criteria and qualification tests. In addition, the community can easily scale the size of the dataset at a low cost by following our annotation process.\n\nThanks again for your time and valuable comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The authors stated that they will share their code and datasets, which is much appreciated. \n2) The paper is well-written and easy to follow. \n3) The dataset construction is very natural, making it highly useful for truly understanding the nature of LLM's hallucinations. \n4) The method is well-motivated and makes sense. \n5) The results look promising, showing that both of the authors' method suggested variants are superior to their suggested baselines.", "reasons_to_reject": "1) My main concern is the exclusive use of ChatGPT, a black-box model, in the experimental setting. \nI am well aware of the authors' intention to study the behaviour of black-box models, but IMO, a community model is needed here (you can use it as a black box for keeping the main story intact) for the research community to be able to reproduce (at least some of) your results. \nRecent studies showed that ChatGPT performance dramatically varies over time [1] in some tasks, which might lead to inconsistency in your main results. The first question that popped into my head when the LMvsLM baseline failed to perform is \"Maybe ChatGPT is worse now than it was when they tested it?\"\nI am very open to discussing this issue, maybe I'm missing something? Please justify your choice.\n\n[1] https://arxiv.org/pdf/2307.09009.pdf\n\n2) My minor concern is that I believe your method and dataset are very specific for entity-base retrieval, but I guess you can add it to the \"limitation\" section.", "questions_to_authors": "1) You mention a very high annotator agreement in the \"Quality Control\" paragraph. \nDid you calculate it for the whole dataset or just a small subset of fake examples? If it's the latter, can you please explain why?", "paper_topic_and_main_contributions": "The authors present two major contributions to the field of hallucination mitigation:\n1) A paragraph-level dataset for zero-shot hallucination detection.\n2) A method for hallucination detection named reverse validation.\n\nUsing an additional dataset, the WikiBIo-GPT3, they show their method is superior to other passage-level hallucination methods (most of them are reformed sentence-level methods).", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) Line 308 - of *the* PHD benchmark \n2) Line 309 - The entit*ies* of *the* PHD-high domain... *the* PHD-low domain..."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new benchmark for passage-level hallucination detection. The dataset consists of paragraphs generated by ChatGPT describing rare entities, which are annotated as being factual or not. It also presents a method for tackling the task, which essentially converts the paragraph into a question or list of entity requirements and tests whether the LM regenerates the original entity in response. Experiments show it outperforms previous approaches while running faster (the latter provided in the rebuttal). 

Reviewers appreciated the dataset construction process (KooH/2PhM) and data curation (JYbz). Reviewers also appreciated the strong results of the new method. However, reviewers had some concerns about the data collection procedure regarding some missing details (KooH), lack of payment information or ethical statement (JYbz), and possible reproducibility issues with using a non-open-source model (2PhM). In the rebuttal, authors provided more information on the benchmark, point out they are using a stable ChatGPT version, and present some preliminary results with LLaMA. The proposed method seems a bit specific to the benchmark rather than being useful for detecting hallucination more broadly, but does work effectively. Overall, I think the new dataset is timely and covers an important if narrow kind of hallucination.
Decision: Accept-Findings