DelucionQA: Detecting Hallucinations in Domain-specific Question Answering
Authors: Mobashir Sadat
Zhengyu Zhou
Lukas Lange
Jun Araki
Arsalan Gundroo
Bingqing Wang
Rakesh R Menon
Md Rizwan Parvez
Zhe Feng
Venue: EMNLP 2023 Findings (2023)
Keywords: hallucination
LLM
large-language-model
natural-language-generation
question-answering

Abstract: Hallucination is a well-known phenomenon in text generated by large language models (LLMs). The existence of hallucinatory responses is found in almost all application scenarios e.g., summarization, question-answering (QA) etc. For applications requiring high reliability (e.g., customer-facing assistants), the potential existence of hallucination in LLM-generated text is a critical problem. The amount of hallucination can be reduced by leveraging information retrieval to provide relevant background information to the LLM. However, LLMs can still generate hallucinatory content for various reasons (e.g., prioritizing its parametric knowledge over the context, failure to capture the relevant information from the context, etc.). Detecting hallucinations through automated methods is thus paramount. To facilitate research in this direction, we introduce a sophisticated dataset, DelucionQA, that captures hallucinations made by retrieval-augmented LLMs for a domain-specific QA task. Furthermore, we propose a set of hallucination detection methods to serve as baselines for future works from the research community. Analysis and case study are also provided to share valuable insights on hallucination phenomena in the target scenario.

DELUCIONQA: Detecting Hallucinations in Domain-specific Question
Answering
Mobashir Sadat Zhengyu Zhou' Lukas Lange' Jun Araki Arsalan Gundroo
Bingqing Wang" Rakesh Menon 3* Md Rizwan Parvez² Zhe Feng"
Computer Science, University of Illinois Chicago
Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI)
UNC Chapel-Hill
msadat3@uic.ed edu Zhengyu. Zhou2eus bosch.ooocccooocco.oo com Lukas. boschh com
(Jun.Araki, Arsalan. Gundroo, Bingqing. Bangging.Wang]eeee. Jeus. wos.h.occooooooo com
rrmenonêcs. www.eeessuuceeee. edu [Md. Parvez, Zhe. Feng2]@us. bong2)hus.bosch.... bosch. com
Abstract
Question: How do activate the Uconnect voice recognition system in my Jeep 2023 Gladiator?
Hallucination is a well-known phenomenon in text generated by large language models (LLMss.. The existence of hallucinatory re- sponses is found almost all application sce- narios c.g., summarization. question-answering (QA) etc. For applications requiring high re- liability (c.g., customer-facing assistants), the potential existence of hallucination in L.LM- generated text is critical problem. The amount hallucination can reduced by leverag- ing information retrieval provide relevant background information to the LLM. However, LLMs can still generate hallucinatory content for various reasons (e.g., prioritizing its para- metric knowledge over the context, failure to capture the relevant information from the con- text, etc.). Detecting hallucinations through automated methods is thus paramount, To facilitate research in this direction, intro- duce sophisticated dataset, DELUCIONQA that captures hallucinations made by retrieval- augmented L.LMs for domain-specifie QA task. Furthermore, we propose a set of halluci- nation detection methods to serve as baselines for future works from the research community. Analysis and case study are also provided to share valuable insights hallucination phe- nomena in the target scenario.
Large Language
OWNERS MANIAL
Sturms
Model
The butern brseed achvtol Emeriian your Secpet
Context Retrieval
foop
complo
Press the VR button. Alternatively, you can also say
Answer:
aus and
"Hey UConnect" or "Hey Chrysler" to activate the system.
?
Hallucination Answer
Figure 1: Hallucination in text generated by LLMs.
(e.g., ChatGPT? Bard') still have number of weaknesses. Notably, the tendency generate hallucinatory (non-factual) statements a critical issue in LLMs that hinders their usage in applica- tions where reliability is a must-have feature (e.g., vehicle-repair assistant).
To elaborate, hallucination occurs because the models c.g., LLMs follow the principle of maxi- mum likelihood while generating the output text without considering the factuality. As a result, they end up generating outputs that are always highly probable but not necessarily factual. This is a long- standing problem in neural natural language gener- ation and it is pervasive in all types of generation tasks ranging from QA (Su et al., 2022) to abstrac- tive summarization (Falke et al., 2019; Maynez et al., 2020). With the recent technological ad- vances made by state-of-the-art L.MMs, hallucina- tion has been largely reduced compared to models from prior work. However, hallucinatory state- ments are still prevalent in text generated by LLMs.
1 Introduction
Large Language Models (LLMs) have brought breakthrough performances for many natural lan- guage processing tasks, especially for generation- related applications such as chatbots and question- answering (QA). However, -tate-of-the-art LLMs
Work done during internship Bosch Research North America.
hattss//ppenaiico//bllgaaaatggg htttps://bar..gooate
https:: httpa//cc...ooccceeeeerr////llle

---

The potential incorrect/hallucinatee content in the answers provided by LLM-based assistants raises significant liability concerns (e.g., vehicle damage) for the applications requiring high reliability.
models for 500 source texts taken from the XSum dataset (Narayan et al., 2018). Pagnoni al. (2021) proposes a benchmark named FRANK for under- standing factuality in abstractive summarization. The benchmark dataset is derived by generating 2,250 abstractive summaries from different models and then collecting human annotations related to their factuality. Wang et al. (2020) and Kryscinski et al. (2020) also construct smaller scale resources for hallucination in abstractive summarization.
To date, various approaches have been proposed reduce the degree of hallucination in QA tasks (Shuster et al., 2021). One effective way is lever- age information retrieval (IR). In particular, instead of directly feeding the question alone to the LLM for answer generation, an IR system first retrieves relevant domain-specific information for the given question to be used as context. Next, the ques- tion and the retrieved context are used together in prompt for an LLM to generate an answer. The re- trieved context aids the LLM in generating more reliable answer. However, even with IR and prompt engineering, hallucinations still occur from time to time, typically due to imperfect information re- trieval and/or over-reliance on knowledge acquired through pre-training. Detection and mitigation of hallucinatory content is thus an essential topic to widen the application of LLMs to domain-specific QA tasks.
For machine translation, Guerreiro et al. (2023) recently introduced a dataset containing 3,415 ex- amples containing annotations for different types of hallucination. Moreover, there have been sig- nificant explorations (Lee et al., 2018; Zhou et al., 2021; Müller and Sennrich, 2021) for addressing hallucination in machine translation.
Recently, several datasets have been proposed to study hallucination in general text generation (not specific to any tasks) by LLMs. Liu et al. (2022) presents HADES, token-level hallucina- tion detection benchmark for free-form text gen- eration. In contrast to abstractive summarization and machine translation, where the input text can be used as a reference for detecting hallucination, H DES is reference-free Manakul et al. (2023) released a dataset containing 1,908 sentences with annotations related to hallucination. The sentences are derived by prompting LLM to generate wikipedia articles about different concepts. Azaria and Mitchell (2023) attempts to automatically de- tect the truthfulness of statements generated by an LLM by using its activations to train a classifier. To this end, they build a dataset containing truc/- false human annotated labels from 6 different top- ics. Rashkin et al. (2023) presents Attributable to Identified Sources or AIS as framework for evalu- ating if the text generated by model is supported by external sources (i.e., not hallucinated).
In this work, we focus on addressing hallucina- tion by LLMs in domain-specifie QA where high reliability is often needed. Specifically, we present DELUCTONQA, large dataset for facilitating re- search on hallucination detection in IR-aided car manual QA systems. We also propose set of hal- lucination detection approaches, which may serve basclines for further improvement from the re- search community. Our best-performing baseline shows a Macro F1 of only 71. 71.09% on the test set of DELJUCIONAA illustrating the difficulty of the task and room for future improvement. Insights on the causes and types of hallucinations for the target scenario are provided in this paper. We hope this work paves the way for more future explorations in this direction, which can expand the scope of LLM usage to tasks with high-reliability requirement.
While all of these datasets are valuable resources for the exploration of hallucination detection by LLMs, none of them cater to customer- facing sce- narios where high reliability is essential
2 Related Work
To date, various resources (e.g., datasets/bench- marks) have been made available for studying hal- lucination detection. Most of these resources ad- dress the existence of hallucinatory content in ab- stractive summarization and machine translation. For example, Maynez al. (2020) construct dataset where each example contains the human annotations for faithfulness and factuality of the abstractive summaries generated by state-of-the. art
3 DELUCIONQA: A Dataset for Hallucination Detection
We aim develop a dataset that represents domain-specific QA task requiring high system reliability for customer satisfaction. An example of such a scenario is question answering over car manuals, where incorrect LLM output could cause

---

improvement to seek damage cars and endanger lives. However, such domain-specific scenarios pose challenges such as technical terms and language particularities being infrequent during the pre-training stage of LLMs resulting in their sub-optimal representation. To en- sure answer reliability and prevent hallucination in these settings, IR technology is usually employed to inject related context information into the LLM prompt. Despite this, hallucinations can still occur.
3.2 Context Retrieval for QA
For each question (e.g., "how do activate the Uconnect voice recognition system?"); we retrieve potentially relevant text snippets from the car man- ual using different IR methods, where the following retrieval methods are evaluated:
Retrieval Foundation: We adopt sparse and dense retrieval functionn as foundations for more advanced retrieval methods. For this, we use the Pyserini toolkit (Lin et al., 2021a), which provides reproducible information retrieval results with state- of- of-the-art art retrieval models. The sparse retrieval function is keyword-basek based tf-idf variant with Lucene In contrast, the dense retrieval function is based on FAISS index (Johnson al., 2019) with reproduced ColBERT transformer model (Lin et al., 2021b) for embedding and retrieval score. As the scope of the retrieved context affects the system efficiency as well as the answer quality, we index each document at different granularity level (document-level, section-level, and paragraph- level). This allows the retrieval module to make selections at the appropriate index level and use the context in good scope that fits the question.
The DELUCIONQA dataset seeks capture in- stances of hallucination in domain- specific QA af- ter using IR, facilitating research on hallucination detection in this scenario. For this, we select car- manual QA as a domain-specific representative task and, without loss of generality, create the dataset using the manual of Jeep's 2023 Gladiator model.
To collect the dataset, create set of ques- tions for the technical car repair manual domain and use QA system equipped with multiple re- trievers to generate various retrieval results for each question. We then generate various answers based on those retrieval results, resulting in (question, re- trieval result, answer) triples which are annotated by human annotators. The resulting dataset is of high quality and can be used to investigate halluci- nation detection/handling approaches for domain- specific QA tasks. We elaborate on the dataset curation processes in the following sections.
Top-k Ensemble Retrieval: As the sparse re- trieval is keyword-based, it is limited by the ex- act overlap of tokens and ignores potential syn- onyms and similar meanings of sentences. There- fore, in addition to the Pyserini approach, we im- plement ensemble retrieval function for the QA system, which combines the sparse and dense re- trieval methods. The ensemble approach follows a multi-stage retrieval paradigm where we first over- retrieve 5 times the amount of the desired retrieval results. After merging the score from both the dense and sparse retrieval modules, the system reranks and selects the top-k results. In this pa- per, experiment with k = 1 for the single-best element, and k 3 for slightly broader retrieval result set, which is to focus the top-ranked sults, and filter out the less relevant content.
3.1 Question Creation
We create a set of questions for QA on topics re- lated to the Jeep 2023 Gladiator in two steps. First, we use an LLM to generate candidate questions automatically from the domain-specifer data re- sources, in this case the official car manual. For this, we download the publicly available car manual in HTML format and split it into smaller chunks fol- lowing the HTML paragraph structure. The set of candidate questions is then generated with a multi- task T5 model (Raffel et al., 2020) fine-tuned for question generation and answering." Then, the au- thors manually filter the set of candidate questions, removing the questions that are not realistic and polishing the remaining questions (e.g.. rephrasing a question to what a real user may ask). Addition- ally, the authors augment the dataset with important questions missing from the automatic generations.
Adaptive Ensemble Retrieval: It does not guar- antee that the static document segmentation at dif- ferent granularity level would make the returned context to be in best scope that matches the ques- tions. Therefore, the adaptive ensemble retrieval approach is proposed,
'https://lucene www.:ee.eeppccc.o.aaaooo "https://hugginggfacc co/ ca/cattornii/ttt. honbertt-2-hppmmmmare
'https: /hugsss/g/ggggagaa ace. co..alalllaaaaa//////aaaccaaaaaa t5-small-qge hh-

---

for dynamic context range, based on the ensemble retrieval approach introduced above.
hurts the answer generation, as discovered in our preliminary study. The resultant 2038 examples go through our manual annotation procedure as described next.
The adaptive retrieval checks the retrieval score at different levels, from the top whole-document the paragraphs at the bottom, which seeks the appropriate scope within document that matches the input question, and adjusts the retrieval score based on the identified scope. We suppose the document-level score does not sufficiently disclose the precise relevance to the input question. When a low document-level score is calculated, we still dig deeper into the lower level in the structured docu- ment, match the relevant text chunks, and eventu- ally combine the relevant chunks, ignoring the less relevant part for the adjusted final retrieval score.
3.4 Human Annotations
For manually annotating the (Question, Context, Answer) triples constructed from the car-manual QA, we make of the Amazon Mechanical Turk (MTurk) platform and let crowd-source workers classify each answer sentence it is supported or conflicted by the given context, or if neither applies when context and answer are unrelated. We also ask the annotators to label whether the question is answerable based on the given context (i.e., if the context is relevant to the question), whether the generated answer contains the information quested in the question, and whether the system conveys an inability to answer the question. The annotation interface can be seen in Appendix B. We take the following steps to ensure high quality of the annotations:
3.3 Answer Generation
For generating the answers, we choose the OpenAI ChatGPT model as a representative LLM because it is arguably the most advanced, best performing, and widely accessible LLM at the time of writ- ing this paper. In particular, each retrieval result i.e., the context, is fed together with the question into the OpenAl ChatGPT model with certain prompt format to generate an answer, resulting in (Question, Context, Answer) triple. In this work, we use the widely available stable-version model to generate answers. As sometimes different retrieval methods may re- turn the same retrieval result for a question, we further adopt multiple prompting techniques to in- crease the variety in answers, which will further facilitate the investigation of hallucination. More specifically, for the sparse retrieval result, con- struct the prompt with a simple Q+A style prompt. We use more verbose prompt for the Top-1 and Top-3 ensemble retrieval results. Finally, we use chain-of oft-ooftthhhee thought prompting ( (Wei et al., 2022) for the adaptive ensemble retrieval and concatenate the search result and the question into single prompt. More details about the three prompting approaches can be found in Table in the Appendix.
Test Exercise Before present each annotator with the main questionnaire containing a set of three (Question, Context, Answer) triples, we give them a test exercise to complete. The test exercise contains a context and three answer sentences. For each answer sentence, we ask the annotator to de- tect if it is supportelcomflicteedeiiteee supported nor conflicted by the context. If the annotator an- swers all questions correctly for the test exercise, they moved the main questionnaire. The test exercise is aimed at evaluating the annotators' understanding English and their capability to complete the main questionnaire successfully.
MTurk Filters We further ensure that the an- notations completed by diligent workers by requiring them to be "Master" turkers with an ap- proval rating of at least 90% and having least 10,000 tasks approved. In addition, we require the annotators' location to be one of the native English- speaking countries or countries where English is very commonly spoken as a second language.
Using the question creation procedure described in Section 3. we generate 913 distinct questions that QA user may ask about the 2023 Jeep Glad- iator. Then, apply the four retrieval methods we study every question and let ChatGPT gen- erate an answer with the corresponding prompting method. With this, we obtained 3,662 (Question, Context, Answer) triples in total. We then filter out triples with more than 40 context sentences, which
Compensation We the compensation for the annotators at an hourly rate of S10 which is consid- erably higher than the US federal minimum wage of S7.25, in order to attract qualified turkers to accept our annotation task.

---

Metric
Supported
Neither
Split
#Ques
#Triples
#Hal
#Not Hal
PRECISION
97.7%
89.5%
TRAIN 513
1,151
392
759
RECALL
95.6%
94.4%
DEV
100
216
94
122
Fi
96.6%
91.9%
TEST
300
671
252
419
TOTAL
913
2,038
738
1,300
Table Agreement between sentence-level labels based majority-consensus of annotators and expert- annotated labels. Conflicted label is very rare and does not exist in the subset that used for this comparison.
Table 2: Number of unique questions, number triples and label distribution in each split of DELUCIONQA, Here, Ques: Question and Hal: Hallucinated.
cording to Landis and Koch (1977). However, there high label imbalance our dataset, as 77% of answer sentences were labeled as supported by the context. This limits the expressiveness of Krippen- dorff's ' alpha score, due to its chance adjustment. Therefore, we compute the agreement between the sentence-level labels based on majority con- sensus among annotators and sentence-level labels from the annotations performed by one expert an- notator for randomly chosen samples. Consid- ering the expert- annotated labels as ground truth, we compute the precision, recall and F1 -cooe for the sentence-level labels and report them in Table 1. Note that no conflicted label occurred in this set of samples. As we can see, there is substantial agreement between the expert-annotated labels and the labels from the annotator's 's majority consensus with 96.6% agreement for supported and 91.9% agreement for neither. This demonstrates the high quality of our human-labeled and aggregated labels of DELUCIONQA.
3.5 The DELUCIONQA Dataset
Based on the annotator responses, we assign a sentence-level label to each sentence in the answer based on majority vote from the annotators (sup- ported/conflicte/nneilee))) In rare cases (for 0.5% of all instances), there was no consensus among the three annotators, which was mainly caused by par- tially answerable questions according to the given contexts. For these examples, one expert annota- tor manually inspected the instance and assigned a corrected label.
Based on the sentence-level labels, we then as- sign the final example- label for the complete answer. If all sentences in the answer for a certain triple have supported labels, the triple is labeled Not Hallucinated. If there exists at least one sentence in the answer that has a neither label or a conflicted label, we assign Hallucinated label the triple. In addition, a binary true/false An- swerable and Does not answer label assigned to each example based on the annotator consensus on whether the context is relevant to the question and whether the LLM refuses to provide an an- swer (i.e., response implies "I don't know"). Note that if the context is irrelevant (Answerable false) and the LLM does not refuse provide an answer (Does not answer false), the example is labeled as Hallucinated. Therefore, even if there can be cases where the context irrelevant due to imper- fect retrieval methods or the information related to the question being unavailable in the car manual, they do not affect the quality of the labels in our dataset. Our resulting corpus of 2038 examples contains 738 triples labeled as Hallucinated and 1300 triples labeled as Not Hallucinated.
Data Splits As previously mentioned, gener- multiple triples using different retrieval settings for each question. Thus, the same question can occur multiple times different triples in our cu- rated dataset. In order to avoid data lcakage by having the same question in multiple splits of the dataset (train/develoommentttss), we divide the data at the question level into the usual standard splits. Specifically, out of the 913 unique questions in our dataset, we randomly assigned 513 questions our training set, 100 to the dev set, and 300 the test set. After splitting by question, we divide the triples according to their respective split. The num- ber of triples in each split is reported in Table 2. Note that we do not use the training set for the ex- periments in Section 4, as we explore unsupervised methods. We still provide fixed standard splits to ensure comparability of upcoming research works using this dataset.
Inter-Annotator Agreement To understand the quality of the labels, we follow two procedures. First, we compute the inter-annotator agreement for the sentence-level labels (supportedconfficteeeee ther) and observe moderate agreement among an- notators based on Krippendorff's s alpha (55.26) ac-
Unanswerable Questions In addition to the (Question, Context, Answer) triples of DELU-

---

Method
Hal
#Exmpl
#Tokens
Answer
Retrieval Result
SPARSE
39.2%
590
190.9 + 0.2
Sentence 1
Sentence 1'
ENSEMBLE TOP-1
28.9%
574
162.3 = 0.2
Sentence 2
Sentence 2'
ENSEMBLE T
23.2%
258
421.2 + 0.7
A ENSEMBLE
45.6%
616
232.5 0.3
Sentence 3'
Sentence 4'
Table The percentage of Hallucinated examples, total number of examples, and the average + standard devia- tions of the number of tokens in the contexts of triples retrieved by each retrieval method. Here, A Ensemble: Adaptive Ensemble.
Sentence 5
Santence 5'
Sentence 6'
Attention:
Hallucination detectedi
Sentence
The claim of Sentence la not supported the retrieval resuit. halludinate
Sentence
CIONQA, provide collection of 240 (Question, Context) tuples, for which the LLM refused pro- vide an answer, e.g., "I don't know the answer" or "There is no mention of X in the given context." These instances were filtered before the annotation process using a rule-based method and were not an- notated as previously described. Instead, one expert annotator manually checked each irrestion-cottxx pair whether the contexts contained the answer and the LLM failed answer or if the question was indeed not answerable based the given context. Out of the total of 240 instances, 20 were identified as answerable. This suggests that the models' ab- stention to answer can be considered accurate, as it is correct more than 90% of these cases. We will publish this extra dataset as DELUCIONQA UNANSWERABLE for future studies on this topic. Note that there can still be examples which are not answerable in the main DELUCIONQA dataset (in train/test/dev splits). These examples were unde- tected by our rule-based method (i.e., they imply "I don't know" more subtle manner).
Figure 2: Sentence-similarity based hallucination detec- tion approach.
scarch on udvanced retrieval methods, in particular when working with eetrieall-augmene LLMs.
Automatic Hallucination Detection
This section describes two automatic approaches to detect hallucinations in context-augmented QA. The evaluation of these approaches is reported in Section 5. Note that while we evaluate the pro- posed hallucination detection approaches using only DELJCIONNA,, they can be used for other IR-augmented QA tasks as well.
4.1 Sentence-Similarity-
Approach
We first propose approach based on sentence- level similarity. The basic idea is to determine whether each sentence in the answer is grounded on the retrieval result i.c., the context, as otherwise, the sentences are likely to hallucinated by the LLM. If there exists one or more sentences that are deemed as not grounded, the answer is predicted to be Hallucinated.
3.6 Distribution of Hallucination Cases
We hypothesize that the context retrieved will be more relevant to the question by retrieving with more advanced methods, which meanwhile makes the answer less likely to contain hallucinated con- tent. To test this hypothesis, we divide the triples in dataset based on the retrieval methods used for their respective context and report the percentage of Hallucinated examples for each method in Table 3 on the complete DELUCIONQA dataset.
The approach calculates two types of similarity measures: sentence-embeddingg based similarity and sentence overlapping ratio. We use both mea- sures detect whether sentence the answer is deemed similar to (i.e., grounded on) one or more sentences in the retricval result, as illustrated Figure 2, where the third sentence is labeled as hal- lucinated because is deemed as not being similar to any sentence in the retrieved context. The two types of similarity calculation evaluate the similar- ity between sentences (one in the answer, the other in the retrieval result) from different angles.
In general, one can observe that the percent- age of hallucination declines with more sophisti- cated retrieval methods (top methods in Table 3). Interestingly, the contexts retrieved by Adaptive- ensemble search results in the highest percentage Hallucinated answers despite being a stronger method, which highlights the necessity of future re-
The -eweeenceeeeeddinng--gee based similarity caleu- lates the cosine similarity of the embedding vectors generated by a language model for each of the two

---

for Sent_A answer:
keywords extr(question, astrieval_resuts onswer)
for sent_ R in retrieval.resut (aandddng_.tnsssnt__2 sent_R) T_1):
for keywords: k not in retrieval_ bollucinated.k.addgg r_hallucinated
similar [sent_A, sent_R] True
elif reta,,,, sent_R) T.2:
len(hallucinated..k) len(keywords)
sinilar [sent_A, sent_R] " True
If ruhellucinated T_3: hal halluctnation (dd(wer)))
elsa:
similar[sent.A. sent_R] False
If not similar[sent.A, sent_X]
Listing 2: Keyword-based hallucination detection.
sent_x retrieval_result: sallucintted.adddseee
Listing I: Similarity- hallucination detection that leverages embedding similarity and overlap ratio.
These two types of sentence similarity measures have complementary advantages. The embedding- based measure is suitable for a one-to-one map- ping, while the overlap-based similarity is advanta- geous for one-to-many and many-to-on mappings. We further combine them into hybrid hallucina- tion detection approach, as described in Listing 1. Specifically, if a sentence is deemed as not being similar by the embedding- brid approach checks it is deemed as similar by the overlap-based measure. If both measures deem the sentence to be not similar, the sentence is marked not supported (i.e., hallucinated).
sentences in focus. If the cosine similarity score larger than a certain threshold, the two sentences are deemed similar, that is, the sentence the an- swer is predicted to be grounded in the retrieval result.
The second type of similarity is calculated based on the sentence overlapping ratio. It is designed to specifically handle the situation of one-to-many or many-to-one mapping (i.c., one sentence in the answer may merge the information from multiple sentences in the retrieved context, and vice versa.). In this situation, for the focused answer sentence and context sentence, their sentence-embedding similarity score may be low even if the shorter sentence is mostly part of the other sentence, be- cause the longer sentence may contain significant amount of additional information. The sentence- overlapping-ratte method aims to catch such partial similarity in this situation. The method determines whether two sentences are similar using the follow- ing steps:
measure, the hy-
4.2
Approach
We also propose an approach that leverages key- word extraction (Firoozeh et al., 2020) for detecting hallucination. Here, the main idea of this approach is that given an answer generated by an LLM, if a significant portion of the keywords in the answer does not exist in the retrieved context, the LLM is deemed to have hallucinated the answer. The detailed algorithm of this approach can be seen in Listing 2.
Step 1: Conduct dynamic programming to compare the two sentences with the objective of max- imizing overlap, outputting optimal path that reveals the overlapped words.
4.3 Settings for Hallucination Experiments
In this work, we use the all-MiniLM--66-2 sen- tence transformer (Reimers and Gurevych, 2019) to calculate the embedding- based similarity between two sentences. Sentence-BERN is chosen here due to its high efficiency and low cost. The key- words for the feyword-extractionn-bnn are generated using the OpenAl ChatGPT model (gpt-3.5-turbe 0301). The thresholds for all hallucination detection basclines are tuned on the development set of DELUCIONQA using values among (0.1, 0.2, ..... 0.9). We found the following thresholds: Ti = 0.6 for COSINE, T2 0.1 for OVERLAP, (Ti 0.1, T3 0.9) for HYBRID and T3 0.2 for SEYWORD-MATCHE
Step 2:
Calculate an overlap length by accumulat- ing the number of tokens for those overlapped phrases that each contains more than either 4 words, where is the 30% of the shorter sentence's s length. The relatively short overlap phrases are not considered in order to avoid the influences of noises from common words such as "the".
approach
Step 3: Compute the sentence overlap ratio as the overlap length divided by the number of to- kens in the shorter sentence. If the ratio is larger than a threshold, the focused two sen- tences are deemed similar.
hhttps://huggiggs http:///gggnngaaac..eceeeeeeeeea all al1-MiniLN-L6-V2

---

Train
Dev
Test
Method
Hal
N-Hal
Overall
Hal
N-Hal
Overall
Hal
N-Hal
Overall
SIM-COSINE 63.18
74.73
70.03
72.45
77.12
74.78
63.84
73.55
69.45
SIM-OVERLAP
68.47
82.72
75.59
73.51
80.16
76.84
63.89
78.28
71.09
SIM-HYBRID
68.73
83.16
75.94
73.51
80.16
76.84
63.33
78.29
70.81
KEYWORD-MATCH
30.25
77.47
53.86
31.58
69.57
50.57
31.23
74.31
52.77
Table 4: Class-wise F1 scores (%) and overall Macro F1 scores (%) of the bascline hallucination detection methods on the three splits of DELUCIONQA. Here, Hal: Hallucinated, N-Hal: Not Hallucinated.
5 Results and Analyses
to map them because it only calculates the similar- ity scores the sentence level. SIM-OVERLAP is better suited capture these similarities because of its capability in one-to-many mappings. As a result, despite not using any deep semantic embeddings, SIM-OVERLAP performs better than SIM-COSINE, This indicates that DELUCCINQA contains sig- nificant number of one-to-many mappings among the answer and context sentences, and any future methods should incorporate the capability cap- ture these mappings.
This section will report the evaluation of the previ- ously introduced hallucination detection methods on our newly proposed DELUCIONQA dataset.
Hallucination Detection Performance We pro- vide a comparison of our hallucination detection algorithms for all three splits of the DELUCIONQA dataset in Table 4. The best-performing model SIM-OVERLAP achieves Macro F1 of only 71.15 % on the unseen test set. This indicates that DELU- CIONQA presents a challenging new task with sub- stantial room for future improvement. From Table 4, we also notice that there are fluctuations per- formance across the three splits (train/dev/test) of the dataset. Recall that while randomly dividing the data into multiple splits, we ensure that each unique question ends up in single split (to avoid data leakage). We believe that the observed fluctu- ations are mainly due to the diversity in questions across the resulting data splits, each containing disjoint subset of questions.
Case Study and Error Analysis
We observe that hallucination may occur due to var- ious reasons, including insufficient retrieval results, excessive retrieval results, and the LLM focusing on its learned knowledge instead. To facilitate the discussion, we list number of examples covering various types of hallucination in Table 5.
P1: Using learned knowledge instead of context. When the retriever in the QA system fails to fetch the correct information, sometimes the LLM can honestly admit that it does not know the answer, but often the LLM simply makes something up, leading to a hallucinated answer which looks prob- able. As shown in the first example in Table 5, the system fabricates two non-existing commands, "Hey Uconnect" and "Hey Chrysler", which will never work for Jeep car. Such commands are hallucinated probably because similar commands like "Hey Google" "Hi Siri" are frequently found in the LLM's pre-training data.
Similarity match keyword match The results show that the similarity-basee methods (showing Macro Fl scores from 69.45% to 71.09%) perform considerately better than the KEYWORD-MATE approach with 52.8% Macro F,. This indicates that simple keyword matching is not enough for halluci- nation detection, and more sophisticated methods to capture the similarities between the context and answer are necessary.
Embedding similarity vs sentence overlap Comparing the performances of the similarity- based approaches, we can see that SIM-OVERLAP outperforms both of the other methods. This can be mainly attributed to the advantage of SIM- OVERLAP to match answer and context in one- to-many or many-to-one way, in contrast to SIM- COSINE. That is, if answer sentence merges the information from multiple sentences from the con- text (or vire-versa), SIM-COSINE may not be able
P2: Unrelated information in retrieval results. Even if the retrieved result includes the correct con- text, the LLM may still hallucinate, likely due to the additional information presented in the retrieval result. For example, for the second question in Table 5, when using the Lucence Search setting, the generated 4-sentence answer is correct except for the last sentence. The last sentence made up because of excessive information in the retrieval

---

Question
Answer
to provide additional context aid the model in providing a factual answer, hallucinations can still occur due to various reasons. Therefore, it is cru- cial to develop automatic hallucination detection and handling methods to improve the reliability of LLM- question guestion-answering systems. To facili- tate research in this topic, we release our newly col- lected DELUCIONQA dataset consisting of 2,038 high-quality human-labeled hallucination ratings for question answering over cur manuals. Based on that, we propose a set of initial solutions for hallucination detection as baselines in this paper. Furthermore, we perform a qualitative analysis to provide insights into why hallucination occurs even after the LLM is provided retrieved-context rel- evant to the question. While DELUCIONQA is constructed from single domain (car manual) and single LLM (ChatGPT), we believe that the in- sights gained from (both by us and the research community in the future) will be valuable in ad- dressing hallucinations by other LLMs in the other domains as well. the future, we will diversify our study of hallucination detection to multiple LLMs and domains, and present more advanced hallucination detection/handling approaches.
Alternatvely, you also say "Hey UConnect" "Hey Chrysler" to activate the system...
How do activate the UConneet voice recognition system?
say "Hey UConnect" followed by voice command
The inspection also checks for any issues that may cause the MII. to be illuminated during normal vehicle operation....
What does emission inspection check?
The inspection also checks for fluid level, exhaust system, and cat- alytic converter maintenance.
in general. the range can be several hundred feet
How can use key fob to lock/unlock donr?
from distances up to 66 ft m) using the key fob...
Table Examples of answers generated for given ques- tions, with ballucinated parts in bold.
result about unrelated MIL functions. In this case, switching to a better retriever may not guarantee to solve the problem, as shown in the second an- swer of the question, which is obtained with our Top-3 Ensemble Search. This answer is largely the same, except that it hallucinates the last sentence in different way due different set of excessive information in the retrieval result.
Limitations
While our proposed dataset can facilitate research on the development of methods for both halluci- nation detection and hallucination handling, our baseline methods are designed only for hallucina- tion detection. Moreover, our dataset exclusively contains answers generated by ChatGPT. At this point, is not clear whether the methods developed based on our dataset will generalize well for other LLMs.
P1 and P2 can oceur jointly. Note that the LLM does not always hallucinate when the retrieval re- sult includes additional text. Use the third question example. While the LLM halluci- nates wrong distance due to insufficient retrieval results when using the Lucence Search setting (see the 1st answer), successfully answers the ques- tion when using the Adaptive Ensemble Search (see the 2nd answer). Although with this setting, the retrieval result contains 25 sentences in total, among which only 2 sentences are related to the answer, the LLM does not hallucinate anything from the large amount of excessive text in retrieval result in this case. This means whether or not an LLM hallucinates also depends on how strongly is biased towards the pre-learned knowledge.
in Table
Ethics Statement
Considering legal and ethical standards and valu- ing work effort, we set the compensation for our annotators at the rate of $10/hour, which is above US federal minimum hourly wage, $7.25/hour, at the time of our study.
Acknowledgements
7 Conclusion
We thank Paheli Bhattacharya for her contribution in generating the questions that we to construct our dataset. We also thank our anonymous review- ers for their valuable and constructive feedback, which is helpful to our research work.
This work addresses hallucination in text generated by LLMs in retrieval-augmented question answer- ing, focusing on a domain-specific scenario with high-reliability requirements. Our study demon- strates that even after leveraging IR technology

---

References
A token-level reference- free hallucination detection benchmark for free-form text generation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics Voolue I: Long Pa- pers), pages 6723-6737, Dublin, Ireland. Association for Computational Linguistics.
Amos Azaria and Tom Mitchell. 2023. The internal state of an IIm knows when its lying. arXiv preprint 137344
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Rank- ing generated summaries by correctness: An interest- ing but challenging application for natural language inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 2220, Florence, Italy. Association for Computational Linguistics.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. arXiv preprint WxXiv:2303.0886.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguisties.
Nazanin Firoozeh, Adeline Nazarenko, Fabrice Alizon, and Béatrice Daille. 2020. Keyword extraction: Is- sues and methods. Natural Language Engineering, 26(3):259-299 291.
Mathias Müller and Rico Sennrich. 2021. Understand- ing the properties of minimum Bayes risk decoding in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference Natural Language Processing (Vol- ume 1: Long Papers), pages 259-272, Online. Asso- ciation for Computational Linguistics.
Nuno M. Guerreiro, Elena Voita, and André Martins. 2023. Looking for needle in a haystack: A com- prehensive study of hallucinations in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 1059-1075, Dubrovnik, Croatia. Association for Computational Linguistics.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. JEEE Transactions on Big Data, 7(3):535-547 5477.
Shashi Narayan. Shay B. Cohen, and Mirella Lapata. 2018. Don't 't give me the details, just the summary! Topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346 Online, Association for Computa- tional Linguistics.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 I Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 48 4812-4829, Online. As- sociation for Computational Linguistics.
Richard Landis and Gary G Koch. 1977. The mea- surement of observer agreement for categorical data. biometrics, pages 159-174
Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fan- njiang, and David Sussillo, 2018. Hallucinations in neural machine translation.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21 11(5):5485-5551
Jimmy Lin, Xueguang Ma, Sheng-Chier Lin, Jheng- Hong Yang. Ronak Pradeep, and Rodrigo Nogueira. 2021a. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), ), pages 2356-2362.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Ture, and David Reitter. 2023. Measuring attribution in natural lan- guage generation models. Computational Linguistics. pages 1-64.
Sheng-Chie Chieh Lin, Hong Yang. and Jimmy Lin. 20. In-batch negatives for knowledge distilla- tion with tightly-coupedd teachers for dense retrieval. In Proceedings of the 6th Workshop on Represen- tation Learning for NLP (RepLANI.-220) pages 163-173.
Nils Reimers and Iryna Gurevych. 2019. Sentence-ber Sentence embeddings using siamese nert-ne. arXiv preprint arXiv: 1908. 10084.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings
Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2022.

---

of the Association for Computational Linguistics: EMNLP 2021, pages 3784-3805 Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022. Read before generate! faithful long form question answering with machine reading. arXiv preprint arXiv:2203.00343
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 5008-5020, Online. Asso- ciation for Computational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint 1rXfv:2201./1001.
Chunting Zhou, Graham Neubig. Jiatao Gu. Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad, 2021. Detecting hallucinated content in conditional neural sequence generation. In Find- ings of the Association for Computational Linguis- tics: ACL-IJCNLP 2021, pages 1904, Online. Association for Computational Linguistics.

---

A Prompts
The prompts for generating the answers given the question and the context from each triple can be seen in Table 6. For extracting the keywords from the answer in our keyword based hallucination de- tection method, we use the following prompt: <an- swer> Given the paragraph above, show the key- words in it:
B Annotation User Interface
We can see the user interface shown to the annota- tors. In Figure 3, we can see the test exercise that every annotator need to pass to be able move the main questions. This exercise is up to ensure that the annotators have good command over English and are qualified to complete the task. The main questionnaire shown in two parts in Figure and 5. The left side of the main question- naire shows the question, context and the answer sentences from each triple. The questions are then asked in the right side.

---

Excercise Questooss
Response:
R1: The Bulley Clarge Warr lig Light ilumii otee when ihe boibery sharging properly inlicatess peresiile prob em with the eleditice syslem rebiled comporentt. recommencc:: contact on outhortand doaler soon possibre.
white
and reoson
Contextt: Battery Charge Warning Light This wemiing light will Ilmmiintt when the bartery is not sharging property. y. Ifit stays while the engine Lunniing there may be matunnction charging This indicales possible em electiical system related
Based the Response and the Contaxz, answer the tollowing questions,
and
3 Context supports Rt.
Contest conficts with R1.
Conleol suppor conflids
What The relation on betweeen R2 and the Contesd?? R2.
confiicts With
Context ne notheer supports nor contlicss with
between
Coniest?
Contest supports
Contest confiicte with RS.
Unnteet thether contlicte
main queationnatre
Figure 3: Test exercise for wowd-workers.
Main Quesstionaalr
Question ssked the system:
Given the contest, quection and response, enawer the toilowinn questions.
Solect all that apply.
how
traller?
and resommended way measure re the waight of traller yutr Tully loaded traller on vehice scale O
Contect provided to help addross the quasstion:
Modea the connect may not contain the answer to the glown guaistion.
Dontect supports R1.
Oustial
Gross Trakee Weight (GTW; The GTW the weight trakeer plus the weight of
Conteat thet supports nor confiiss
corgo, consumatiles and equipreent (permanent or temporayy Issited in the to put your operation
recommendee vehicle scale.
relotion en
entrr welghti
must be supported score.?
the traller must be supported by scale, WARNNIG It gress trailer welght 2 200 (507 more, it inmemeeeded use weig beiggt-dinnbutnng titoh to stable rg velicle you standaad s-carrinn of you vot
O
Content supports R2.
O Dontect contlicss R2,
supports conficts
What the relottion between the Contest and WR:: This will give the Grose Thater
system following for response cantonaso hattol soquencey
(GTW), which
the weight of cargo.
consumaties ecuipment (permamen temporary) loaded traller
"laaed and ready operation" concition??
C. Conteat supports B3.
Next
Figure Main Questionnaire Part 1.
Questoonntree
Grocs Trallrr Weight (GTW) The CTW is the we ght of traller plus the wright of cargo. conoumatics and equipment pponmnont or tomponyy, y) loadod on
Contesc neither supports nor contictete R2.
meder Tracded roady opiratio conditon
way
will give you Weight (GT,,) which the weight of mollor paua the we of cargo,
reliation betwoen Comcest
your fuly loaded heleer a wiiddde sorie wnine weight the traller must be supported by the scale WWRNING: the gross tratle weight recommended
messure
consumit unmb es and equpment (permenent temmpoaayy loesed in the meller nt reseyy operehior O Contert supports
(B07 ensure
weighh-dtetrb.dee hitsh wergii-carying hito,, you
your you cou lase control of your vehicle and cause a colitiion.
O Contest conticts RS).
Contest netther supoorts conficts
systom gonerates following responce question given /The recconze centenced lettd oelow in sequence)
is quaittion anewerat bosed on contect i., contect serdenoeey?
tuddrr recommended way
weight maller put your loaded
* Loos The responso succeoofully cnewer the cueston baced on the contosi
sertences?
#2: The sillie weight im rallee muad Le suppooleed lnd by the marel
Yes
R3:
Gloes Thaker Weight (STW), which weight trailer
resparme, the sysdem iply that close know arrosen centxin
plue weight cargo, consumattles and equ ipmeentnt (permanent or temporayy
way??
looded in on the trailer in ficaeed and ready for operatin" cand ton.
Yes
Figure 5: Main Questionnaire Part 2.

---

Retrieval Method
Prompt Type
Prompt
Sparse
Q+A
("role": "system", "content": are helpful and kind Assistant.") ") ("mole: "user", "content": <context> "Inin" "Q: + <question>> "In" "A:")
Ensemble Top-1
Verbose
("role": "system", "content" "" are a helpful and kind AT Assistant.") ("role": "user", "content": "Given the context: \n" <context> + "In'n"
"Please answer the question:\n" + <question>") ("role": "system", "content": ; "You are a helpful and kind Al Assistant,") "] ("role": "user". "content": "Given the context: 'un" + <context> + "aln" "Please unswer the question:'n" <question>") ("role": "system", "content": "You are helpful and kind AT Assistant. If there is multiple answer/reasons depending the situation you ask for my situation and provide answers/reasons step-hy-step list.")
Ensemble Top-3
Verbose
Adaptive Finsebble
COT.
("role": "user", "content": <context> "wnl"" "Q: <question>> "In" "A:")
Table The prompts used to generated answers using the question and context generated with different retrieval methods. Here, COT stands for Chain-of-thoughh thought. <context> and <question> replaced by actual context and question of each example.


Reviews:
[{"review_id": "lR5V6nOr5D", "reviewer": "Reviewer_6jrh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and your feedback. We address your concerns below.\n\n>> Regarding limiting the dataset to car-manual QA and a single LLM\n(Reviewer comment: \"The insights from this paper are rather shallow due to the following reasons: 1) restricted to car-manual qa setting, 2) restricted to one model (ChatGPT), 3)... \")\n\nWe construct a dataset that facilitates research in the topic of hallucination detection in retrieval-augmented question answering for domains where high reliability is required. As one might expect, manually annotating labels for this task is quite time consuming and therefore, expensive. As such, we chose car-manual QA, which is a grounded real-world setting where erroneous answers may potentially result in hazardous outcome, as a representative domain and chose ChatGPT, which was arguably the most advanced, best performing and widely accessible LLM when we conducted the experiments (GPT-4 was not available at that time), as a representative model to generate the dataset. To our knowledge, there is no other existing dataset that contains the hallucination detection labels for retrieval augmented question answering. While our dataset is indeed constructed from a single domain and single LLM, the insights gained from it (both by us in the paper and in the future by the research community) will be valuable in addressing hallucinations by other LLMs in the other domains as well. Moreover, the hallucination detection methodologies proposed to serve as baselines for future research, which will be discussed more below, are independent of the car-manual QA and ChatGPT, being generally applicable for retrieval-augmented QA. Nevertheless, we may further extend the dataset to other LLM and/or domains in the future.\n\n>> Regarding the insights in the paper being shallow \n(Reviewer comment: \"The insights from this paper are rather shallow due to the following reasons: 1) …, 2) …, 3) “LLM using learned knowledge instead of context” is an insight that has been already discovered...\" )\n\nAs hallucination is not a new topic for LLMs, it is natural that some observations reported in this paper have already been mentioned in multiple previous works. However, one unique contribution of this paper is the provision of a dataset that enables systematic investigation of hallucination phenomena especially for retrieval-based QA, which is an essential task to enable safe application of LLMs to applications with high reliability requirements. Using the collected dataset, we provide a systematic overview of the occurrence of hallucinations for retrieval-based QA, which has not been covered in previous literature. In addition to those observations that confirm the findings reported in previous papers, the additional insights as well as the systematic view we provided are also valuable for researchers working in this area. For example, we point out that \"unrelated information in retrieval results\" may or may NOT result in hallucination in a generated answer, with concrete examples given in the case study.  We also point out that hallucinations also depend on how strongly the LLM is biased towards the pre-learned knowledge. The relation between the bias of pre-learned knowledge and the retrieved context could be a meaningful research direction to identify the reason for hallucination. We will make this clear in the revised version.\n\n\n>> Regarding the generalizability of hallucination detector trained on our dataset\n(Reviewer comment: \"Another potential usage of the collected dataset would be to: train a hallucination detector...\")\n\nKindly note that the goal of this paper is to provide a dataset to investigate hallucination for retrieval-based QA and the corresponding hallucination detection methodologies. We provide a set of hallucination detection approaches to serve as baselines for future hallucination detection research, which are designed to be generally applicable for retrieval-based QA, algorithm-wise being independent of the LLM used to generate the answer. If another LLM is used and/or another domain (not car-manual) is adopted for retrieval-based QA, the proposed approaches can be applied to the new application simply by tuning thresholds (i.e., T_1, T_2, T_3 in Listing 1&2) on the new development dataset. The generalizability of a hallucination detector trained on the collected dataset to other LLMs is out of the scope of this paper. Though it is an interesting topic that we will investigate in our future work.\n\n>> Regarding the novelty and reliability of the proposed hallucination detection methods\n(Reviewer comment: \"The proposed automatic hallucination detection methods (i.e., sentence-similarity-based approach and keyword-extraction-based approach) are not novel,...\")\n\nThe hallucination detection methods proposed in this paper are especially designed for retrieval-based QA with LLMs. To our knowledge, there has not been any prior algorithm reported in literature for this area.  If the reviewer could point out relevant literature(s) that raised the novelty concern, we would be happy to acknowledge and/or discuss the differences further.\n\nOur proposed methods (i.e., sentence-similarity-based approach and keyword-extraction-based approach) are independently developed for the focused area, i.e., retrieval-based QA with LLMs, based on two valid assumptions, respectively: 1) if an answer sentence is similar to one or more sentences in the retrieved context, then this answer sentence is not hallucinated, and 2) if a large portion of keywords in an answer do not show up in the retrieved context, then the whole answer is likely to be hallucinated. The developed algorithms are intuitive and honestly reflect the two assumptions. The experimental results provide valuable insights regarding these two assumptions on hallucination detection. And the sentence-similarity-based approaches, which do not require supervised training at all, achieve  ~70% macro F1 on testing data. This shows that the proposed approaches are solid and serve well as baselines for future hallucination-detection research.\n\n>> Presentation Improvements\n\nThank you for pointing out the typo. We will update the paper accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Hallucination is an important and timely problem that needs to be addressed. \n2. Collected dataset can be helpful for understanding the hallucinating behaviour of ChatGPT better, and can be used to also train a hallucination detection model to serve as an automatic metric for further work \n", "reasons_to_reject": "1. The insights from this paper are rather shallow due to the following reasons: 1) restricted to car-manual qa setting, 2) restricted to one model (ChatGPT), 3) “LLM using learned knowledge instead of context” is an insight that has been already discovered in multiple pieces of literature. “Unrelated information in retrieval results” causing LLM is also a rather expected behaviour. If given the wrong context as prompt/input, LLMs are expected to also have errors propagated.. \n2. Another potential usage of the collected dataset would be to: train a hallucination detector. However, since this dataset only has generations from only one LLM, it is unclear if the hallucination detector trained on this dataset will generalize to other LLMs. \n3. The proposed automatic hallucination detection methods (i.e., sentence-similarity-based approach and keyword-extraction-based approach) are not novel, and I'm not convinced that these are reliable ways to do so. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on the problem of hallucination in the car manual-based question-answering system. Authors created a new dataset (DelucionQA) by i) creating diverse questions based on the car manual, ii) running context retrieval for the created Q (explores 3 different retrieval models), and iii) using the ChatGPT model (gpt-3.5-turbo) to generate answers. Human annotation is collected for each triple (question, answer, context) to see if the answer is hallucinated or not. Using the collected dataset, the authors describe two automatic ways of detecting hallucination and provide some analysis of how hallucination manifests in ChatGPT responses. \n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 285 - “we aplly” —> “we apply” "}, "extra_text": {}}, {"review_id": "iK20elpI9E", "reviewer": "Reviewer_1Jdn", "summary": "", "questions": "In Figure 2 What if sentence3 is the summary of context or sentence3 entails the context, even though it is not sentence-similar to them?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We address your concerns below.\n\n>> Novelty of the method for building the dataset\n(Reviewer comment: \"The method to build a dataset is not novel.\")\n\nWe would like to note that the novelty in our paper comes more from the dataset itself, rather than its construction method. More specifically, the goal of this paper is to provide a dataset to investigate hallucination for retrieval-based QA and the corresponding hallucination detection methodologies. To our knowledge, there are no existing datasets that contain <retrieved context, question, answer> triples with their manually annotated hallucination labels. In addition to the hallucination labels, we will also release the “Answerability” labels indicating whether each question is answerable based on the given context. We believe that our dataset will be a valuable resource for the advance of research in the topic of hallucination detection. \n\n>> Regarding using sentence similarity for hallucination detection\n(Reviewer comment: \"I don’t think it’s reasonable that use sentence similarity...\")\n\nThe proposed sentence-similarity based hallucination detection approaches aim to detect those answer sentences that are not grounded on the retrieved context, that is, being hallucinated.  The approaches determine whether or not an answer sentence is grounded on one or more context sentences based on similarity (i.e., semantic similarity, partial token similarity).  The SIM-COSINE approach attempts to capture the semantic similarity between one answer sentence and one context sentence by measuring the cosine similarity in the embedding space, the SIM-OVERLAP method attempts to capture the one-to-many or many-to-one mapping between answer and context sentences based on the ratio of partial token overlap, and their hybrid (i.e., SIM-HYBRID) attempts to combine the complementary benefits of the two approaches.  Although these approaches do not cover many corner cases, they can handle a significant portion of hallucinations, achieving ~70% macro F1 on testing data. Therefore, they are suitable to serve as solid baselines for future research on hallucination detection for retrieval-based QA.\n\n>> Regarding the question “Figure 2 What if sentence3 is the summary of context or sentence3 entails the context, even though it is not sentence-similar to them?”\n\nIf an answer sentence (e.g.,sentence 3) is the summary of context or entails the context, it will be determined as similar to one of the context sentences if the partial token overlap ratio (i.e., overlapping_ratio()) is larger than the threshold T_2 or if their embedding similarity (i.e., embedding_sim()) is larger than the threshold T_1. Otherwise, if it is determined as dissimilar to any context sentence, it will be labeled as hallucinated. \n\nIf two sentences (one from context and the other from answer) are dissimilar at the surface-level (i.e., low overlap of tokens) despite being supported by each other, the token overlap (SIM-OVERLAP) based method will fail to make the correct prediction. However, if these two sentences are indeed supported by each other, they will be semantically similar and therefore, they will be close to each other in the embedding space despite a possible low overlap of tokens. Since our SIM-COSINE method measures the similarity based on the closeness of sentences in the embedding space, it should be able to capture the fact that these two sentences are semantically similar and make the correct prediction. \n\nNevertheless, as discussed previously, the hallucination detection approaches we proposed to serve as preliminary baseline methods do not well cover corner cases yet.  Thus, one future research direction of hallucination detection can be the reduction of such missed corner cases. \n\n>> Regarding Missing References\n\nThank you for pointing out the missing references. We will update the paper accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Hallucination in LLM is getting more and more attention and this problem\n    \n2. The dataset caters to customer-facing scenarios where high reliability is essential.", "reasons_to_reject": "1. The method to build a dataset is not novel.\n    \n2. I don’t think it’s reasonable that use sentence similarity between each sentence of generation and context to detect hallucination.", "questions_to_authors": "In Figure 2 What if sentence3 is the summary of context or sentence3 entails the context, even though it is not sentence-similar to them?", "paper_topic_and_main_contributions": "1. introduce a dataset, DelucionQA, that captures hallucinations made by retrieval-augmented LLMs for a domain-specific QA task (technical car repair manual domain)\n    \n    1. A multi-task T5 model generates a set of candidate questions. manually filter questions\n        \n    2. Retrieve context via adopting sparse and dense retrieval functions as foundations for their more advanced retrieval methods.\n        \n    3. gpt-3.5-turbo-0301 model to generate answers.\n        \n2. propose a set of hallucination detection methods to serve as baselines\n    \n3. Analysis and case study", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Rashkin, Hannah, et al. \"Measuring attribution in natural language generation models.\" Computational Linguistics (2023): 1-66.\n\n2. Ji, Ziwei, et al. \"Survey of hallucination in natural language generation.\" ACM Computing Surveys 55.12 (2023): 1-38."}}, {"review_id": "Sv9rfr1RqX", "reviewer": "Reviewer_Q4DD", "summary": "", "questions": "1. In Table 4 results. Why the F1 socres for the same methods are so different among different data split (train val trest)? Does it mean the dataset is not consist enough among different split?\n2. There are many domain specific QA datasets that provide context, question and answer, especially in the medical domain (such as PubMedQA). What's the difference between them and this work? They even provide the gold answer rather than the ChatGPT answer.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper. We address your concerns below.\n\n>> Regarding relevance of the retrieved context \n(Reviewer comment: \"The Context for each question is retrieved automatically without manual verification,...\")\n\nIndeed, our information retrieval (IR) module can retrieve context that is irrelevant or only partially relevant to the question. Moreover, it is also possible that the model generates a factual answer despite being given an irrelevant context. However, in this study, we focus on hallucination phenomena, especially for real-world domain-specific QA equipped with state-of-the-art retrieval  methods and state-of-the-art LLM.  Thus, instead of labeling correct/wrong, we define/label hallucination as whether all content in an answer is supported by the context. If the context is irrelevant, the expected behavior is that the model generates an answer that implies “I don’t know.” Therefore, even though the dataset potentially contains examples where the context is irrelevant or partially relevant, they do not lead to erroneous labels. \n\n>> Regarding gold answer for the question\n(Reviewer comment: \"The dataset does not provide a gold answer for the question,...\")\n\nThe objective in building our dataset is to facilitate the research on detecting hallucinations occurring in answers generated by retrieval-augmented LLMs. While having ground truth answers can result in additional use cases of the dataset, we believe they are unnecessary for hallucination investigation/detection.\n\n>> Regarding the question on F1 scores being different for different data splits\n(Reviewer question: \"In Table 4 results. Why the F1 socres for the same methods are so different among different data split...\")\n\nWe follow the same annotation strategy for all <question, context, answer> triples in our dataset. After we obtain the hallucination labels for all examples, we randomly divide the dataset into the three splits at the question level to avoid any data leakage by having the same question in multiple splits. In this way, the characteristics of the resulting splits should be consistent. Nevertheless, as we report in Table 2, distribution of the Hallucinated vs. Not Hallucinated classes in train, test and dev splits end up different from each other, due to the diversity of the questions across the splits. For example, the percentage of hallucinated examples in the dev set is 43.51% whereas it is 34.40% and 37.70% in the train and test sets, respectively. This difference in class distribution could be a reason behind the fluctuations of the performance of the proposed methods. Also, the size of our dataset is relatively small compared to other tasks (e.g., those used in sentiment analysis) and the proposed approaches only tune a small number of hyperparameters. These also contribute to the observed fluctuations on the performance of the proposed methods.\n\n>> Regarding the question on the difference between our dataset and other domain-specific QA datasets\n(Reviewer question: \"There are many domain specific QA datasets that provide context, question and answer...\")\n\nThe task in the existing QA datasets (e.g., PubMedQA) is to generate the answer based on the context and the question. As a result, the answer in each example is written by humans (i.e., gold answer). In contrast, the task presented by our dataset is to detect hallucination in the answers generated by LLMs. As such, the answers in our dataset are not gold answers (not written by humans). Rather, the hallucination labels (Hallucinated or Not Hallucinated) are gold labels assigned based on human annotation. We will make sure to clearly explain the differences between the existing QA datasets and our dataset in the paper. \n\n>> Presentation Improvements\n\nThank you for pointing out the typo. We will update the paper accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to understand. \n2. The proposed dataset is carefully designed and annoted by human.\n3.  The proposed baseline for hallucination detection approaches are partially effective (Similarity based approaches achieved a overall performance higher than 70% ).\n4. ", "reasons_to_reject": "1. The Context for each question is retrieved automatically without manual verification, which could lead some errors in the dataset (all context of a specific question is not relevant to the question, The generated answer is correct but can not verified by the context because the context does not provide the all relevant information)\n2. The dataset does not provide a gold answer for the question, which could limited the use case of the proposed dataset.", "questions_to_authors": "1. In Table 4 results. Why the F1 socres for the same methods are so different among different data split (train val trest)? Does it mean the dataset is not consist enough among different split?\n2. There are many domain specific QA datasets that provide context, question and answer, especially in the medical domain (such as PubMedQA). What's the difference between them and this work? They even provide the gold answer rather than the ChatGPT answer.", "paper_topic_and_main_contributions": "The paper proposed DelucionQA, a QA dataset based on manual of Jeep’s 2023 Gladiator model, and evaluated the sentnece level hallucination of the ChatGPT generated answer. The paper also provides two baselines for automaticly hallucination evaluation.  \nThe main contributions:  \n1. The paper proposed the DelucionQA, a QA dataset that provides (Question, Context, and ChatGPT generated answer).\n2. The paper did a sentence level human evaluation of hallucination for the ChatGPT generated answer, which can boost further analysis of the  LLMs' hallucination.\n3. The paper introduce several baseline approaches (Sentence-Similarity-based Approach, Keyword-Extraction-based Approach) for hallucination detection.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 285 \"aplly\"\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on hallucination detection in the domain of car manual-based QA. The paper introduces a new dataset, DelucionQA, for this task based on the manual of Jeep's 2023 Gladiator model. They use ChatGPT to generate answers and manually verified hallucination in answers based on the question, answer, and context. The authors then use the proposed dataset to develop hallucination detection systems.

Pros:
1. They introduce a new dataset for hallucination detection.
2. The paper benchmarks the proposed dataset with several methods. 

Cons:
1. Limited generalizability. The data is only based on the manual of Jeep's 2023 Gladiator model and answers from the ChatGPT model. Reviewers point out that it is unclear how well the proposed dataset can generalize to different domains and LLMs (Language Model Models).
2. Lack of reliability in hallucination detection models. Reviewers point out that it is a reasonable approach to detect hallucination by measuring the similarity between the model's answer and context. More sophisticated methods and analyses are expected.
Decision: Accept-Findings