Enhancing Hallucination Detection through Perturbation-Based Synthetic Data Generation in System Responses (2024)

Abstract: Detecting hallucinations in large language model (LLM) outputs is pivotal, yet traditional fine-tuning for this classification task is impeded by the expensive and quickly outdated annotation process, especially across numerous vertical domains and in the face of rapid LLM advancements. In this study, we introduce an approach that automatically generates both faithful and hallucinated outputs by rewriting system responses. Experimental findings demonstrate that a T5-base model, fine-tuned on our generated dataset, surpasses state-of-theart zero-shot detectors and existing synthetic generation methods in both accuracy and latency, indicating efficacy of our approach.

heading: 1 Introduction
text: Large Language Models (LLMs) tend to produce hallucinations, wherein the generated text either contradicts the given source knowledge (intrinsic hallucination) or cannot be verified against it (extrinsic hallucination) (Maynez et al., 2020; Rawte et al., 2023). Despite the burgeoning enthusiasm for deploying Generative AI and LLMs in real-world applications, the issue of hallucinations poses significant concerns for downstream users. Consequently, the detection of hallucinations is paramount in enhancing the safety of LLM applications and in fostering trust among users of these technologies.
An effective hallucination detection system should be accurate, fast, and affordable. Costeffectiveness is crucial because every check for hallucinations adds extra cost to the use of large language models (LLMs), which may already be substantially high. Moreover, the system must possess the flexibility to adapt to the rapidly evolving landscape of LLMs. As shown in Table 1, newer iterations of LLMs generally exhibit enhanced capabilities in mitigating hallucinations, thereby escalating the complexity of the detection challenge.
Unfortunately, many current methodologies are either i) costly in terms of compute (Liu et al., 2023; Manakul et al., 2023b) or ii) depend on out-ofdomain/external resources such as QA (Honovich et al., 2021; Fabbri et al., 2022) or NLI annotation (Laban et al., 2022; Honovich et al., 2022), potentially compromising performance.
In this study, we introduce a simple yet effective approach for automatically generating synthetic annotations to train hallucination detectors. Figure 1 shows an overview of our approach. The core of our method involves prompting a rewriting LLM to transform a given system response from the target LLM into both faithful and hallucinated versions, respectively. This technique distinguishes itself from existing methods (Gupta et al., 2021; Das et al., 2022b; Li et al., 2023a; Dziri et al., 2022a) in three significant ways. First, unlike traditional methods that rely on human-annotated examples of faithfulness, our strategy is entirely automated, eliminating need for manual annotation. Second, by directly altering responses from the target LLM, our trained detector aligns more closely with the response distribution of the target LLM, facilitating seamless adaptation to new LLMs. Lastly, while previous approaches require predefined information about the types of hallucinations for their generation process, our method operates without such assumptions. This allows
13321
for the creation of a broader spectrum of hallucination types, enhancing the coverage and diversity of generated hallucinations.
Our experimental evaluations span two hallucination detection datasets, OpenDialKG (Moon et al., 2019) and BEGIN (Dziri et al., 2022b), where a T5base model, fine-tuned with our novel data generation approach, significantly surpasses GPT-4 based methods in performance while achieving a tenfold increase in speed. Further analysis of the generated hallucinations uncovers previously unreported patterns, such as "adding attributes", expanding the discourse on hallucination beyond existing literature. Our code and data will be available at https: //github.com/asappresearch/halugen.

heading: 2 Methodology
text: In this section, we detail our methodology for generating synthetic hallucinations that closely mimic those encountered in real-world applications of Large Language Models (LLMs). Prior approaches to hallucination generation have primarily relied on rewriting human-authored texts (Das et al., 2022b; Li et al., 2023a) or introducing perturbations to the knowledge source (Gupta et al., 2021; Dziri et al., 2022a; Zhang et al., 2023). However, these methods often yield outputs that diverge significantly from those produced by LLM systems, leading to a substantial discrepancy between the synthetic hallucinations and the genuine hallucinations observed
in practice. To address this gap, our approach involves prompting a rewriting LLM to perturb the responses of the LLM system itself, rather than those written by humans. This strategy draws inspiration from the “Minor perturbation” technique described by Lucas et al. (2023), adapted to our context to ensure the synthetic hallucinations closely align with the expected data in real-world deployments.
To effectively train a hallucination detector, it is imperative to have access to both hallucinated and faithful responses. Unlike previous studies, where human-curated outputs served as the benchmark for faithful system outputs (Das et al., 2022b; Li et al., 2023a; Dziri et al., 2022a), the responses obtained directly from the target LLM system may contain a considerable proportion of non-faithful responses. To overcome this challenge, we employ the rewriting LLM to adjust the system’s responses in a manner that promotes the generation of faithful outputs. The specific prompts utilized for inducing both hallucination and faithfulness are presented in Appendix §A. It is important to note that our process for generating hallucinations did not involve biasing the system with predefined categories of hallucination within the prompt, ensuring a more authentic and unbiased generation process.1 For the rewriting LLM, we selected GPT-4 2 due to its robust capabilities in text rewriting (Madaan et al., 2024). Leveraging a powerful rewriting LLM like GPT-4 enables the exploration of a wider array of hallucination categories, thereby enhancing the coverage of hallucinations that are likely to be encountered in real-world scenarios.

heading: 3 Experiments

heading: 3.1 Datasets
text: OpenDialKG is a dialogue dataset that was adopted by HaluEval (Li et al., 2023a), a recent benchmark for hallucination detection. OpenDialKG features human-generated dialogues exclusively with supporting knowledge sources from Freebase (Bollacker et al., 2008). In order to leverage the dataset for hallucination detection, we simulate a chatbot system by employing GPT-4 to generate responses grounded in both the provided knowledge and the preceding dialogue context. The
1These prompts have been designed with versatility in mind, allowing for straightforward adaptation to other NLP tasks such as question answering and summarization. However, our current investigation is focused exclusively on knowledge-grounded dialogues.
2We use gpt-4-1106-preview for our experiments.
specifics of the prompt template utilized for this simulation are detailed in Appendix B. To create a evaluation set on the generated responses, we employ Amazon Mechanical Turk annotators to evaluate whether the responses from the simulated chatbot system were fully supported by the dialogue history and the provided knowledge (for detailed annotation guidelines and interface, refer to Appendix D). Our collection (OpenDialKG-Eval) comprises 402 annotated responses. We designated responses with high-confidence labels as our test set and utilized the remainder for development purposes, resulting in 312 test responses and 90 for development. More details of OpenDialKG-Eval can be found in Appendix E. In addition, we simulate another 2000 responses from OpenDialKG for synthetic generation purpose.
BEGIN is a knowledge-grounded dialog dataset featuring 12k responses from four dialogue systems distributed over 3 document-scale knowledge domains – Wizard of Wikipedia (Dinan et al., 2018), TopicalChat (Gopalakrishnan et al., 2023) and DoG (Zhou et al., 2018) — all with mean knowledge snippets longer than OpenDialKG. In addition, there are three response categories in BEGIN: Fully attributable, Not fully attributable, Generic. Generic category refers to response that are vague and do not provide any new information. Therefore, in addition to faithful and hallucination generation, we also ask LLM to generate responses under "Generic" category. The detailed prompt can be found in Appendix A Table 9. Since BEGIN only released the Dev and Test split, we adopt 1,228 system responses from Dev for both synthetic generation and development while reporting results on Test split.

heading: 3.2 Baselines
text: Zero-shot Detection We compare with SelfCheckGPT (Manakul et al., 2023a), a consistencybased approach which samples system responses multiple times in temperature 1.0 and then leverage scores from NLI or QA to measure whether the target response is consistent with these samples.
Another baseline is G-Eval (Liu et al., 2023), which prompts GPT-4 with an annotation-rubric style prompt describing target variable and furthermore draws multiple samples at a higher temperature; emulating diverse multi-annotation by humans. Since both G-Eval and SelfCheckGPT can only output scores between 0 and 1 and BEGIN data has three output categories, we compare GPT-
4 (Internal), our self-devised zero-shot detector, which prompts GPT-4 with an intuitive prompt to enable three-way outputs(Appendix§F) and does greedy decodes to generate a binary/ternary answer.
The last zero-shot baseline we compare with is SCALE (Lattimer et al., 2023), NLI-based approach which first decomposes the supporting context into chunks, calculate NLI scores on the chunk level using FlanT5 (Chung et al., 2022), then use the maximum score as the final prediction of factual consistency.
Detection with End-to-end Finetuning We use T5-base, an encoder-decoder LM with 223M parameters, as the base model of the detector and fine-tune it on multiple synthetic datasets.3 We make our best efforts to conduct apple-to-apple comparison among different synthetic data. On OpenDialKG-Eval benchmark, we compare with FADE (Das et al., 2022b) and HaluEval (Li et al., 2023a), where we adopted their existing synthetic hallucinations as negative and human written responses from OpenDialKG as positive data for training. On BEGIN dataset, we compare with AugWOW (Gupta et al., 2021) and BEGIN-Adv. (Dziri et al., 2022a), both are synthetic generation baselines and their performances on BEGIN has been reported by (Dziri et al., 2022a). For more details of these synthetic data generation baselines, please refer to Section 5 3.3 Results
Table 2 shows the performance of hallucination detection and latency per response on OpenDialKG-Eval. Latencies are profiled over AWS g5.xlarge instances with no batching sae for G-Eval which requires OpenAI API access. From
3For more experimental details, please refer to Appendix H.
the results, our approach not only out-performs T5 detectors finetuned over previous hallucination generation baselines, but more interestingly, it out-performs state-of-the-art zero-shot detection methods. Besides performance, finetuned models achieve significantly lower latency than all zeroshot baselines. We also show the results on BEGIN data. The results can be found in Table 3, where similar observation can be found.
Lastly, average cost per synthetic response generation is 0.008 USD on OpenDialKG and 0.006 USD on BEGIN, using gpt-4-1106-preview. In comparison, average cost of human annotation per example for OpenDialKG-Eval is 0.20 USD.

heading: 3.4 Ablation Study
text: To analyze the significance of both hallucination and faithful response generation, we conduct an ablation study to replace one of the generation using system response. Results are shown in Table 4. Results show that both categories of synthetic data are necessary to effectively fine-tune the detector.

heading: 4 Hallucination Pattern Analysis

heading: 4.1 Hallucination Pattern Analysis
text: Previous work usually predefined hallucination patterns such as replacing or swapping entities (Das et al., 2022a; Li et al., 2023a). We randomly sample 144 hallucinations generated by our method over OpenDialKG dataset, and manually annotate these into a taxonomy of 6 distinct pattern-driven
categories characterizing the pattern surfaced in the hallucination, further described in Appendix §C.
Pattern distributions are both listed in Table 5 and Figure 2. From the pattern distribution, it is interesting to see that our method has fewer hallucinations from entity replacing/swapping, the most dominant hallucination type is adding unverifiable attributes to an entity. This indicates that our methods generate responses which conform tighter to the real hallucination distribution in contrast to prior approaches. The KL Divergence between the categorical pattern distribution of our method and the system response based distribution is 0.3395, compared to the much greater 0.6706 (and 1.52) between the distribution of HaluEval (and FADE) vs the latter. 4.2 Quality Analysis of Synthetic Data Generation To more closely evaluate the effectiveness of rewriting, we did human annotation over 100 randomly sampled system responses along with our synthetically generated responses based on these system responses. Table 6 shows the portion of faithful data within each type of responses:
Upon reviewing the annotations, our method
demonstrates a significant reduction in the generation of unfaithful responses (hallucinations) compared to system outputs. Our approach also yields a higher number of faithful responses compared to the baseline system outputs, aligning with our objectives. Specifically, out of 59 instances of hallucinations identified in system outputs, our faithful generator converts 11 into faithful responses, achieving a conversion rate of approximately 19%. Conversely, among the 41 faithful responses generated by the system, our method inadvertently transformed only one into a hallucination.

heading: 5 Related Work
text: Research on generating synthetic annotations for hallucination detection has explored various strategies. Some approaches, like FADE (Das et al., 2022b) and HaluEval (Li et al., 2023a), manipulate human-written texts by altering entities or applying predefined hallucination criteria, respectively. These methods only focus on introducing hallucinations and ignore faithfulness augmentation, assuming human-generated content to be inherently accurate, which maybe untrue. Other studies focus on modifying the knowledge source before response generation. AugWow (Gupta et al., 2021) introduces hallucinations by using irrelevant or no evidence, while BEGIN-Adv (Dziri et al., 2022a) alters subjects, objects, named entities, or verbs in the source material, prompting a GPT2-based system (Radford et al., 2019) for response regeneration. These techniques, however, might lead to predictable hallucination patterns due to their reliance on predefined rules.
More recently, ICD (Zhang et al., 2023) mitigates LLM hallucinations by finetuning a model on non-factual samples, aiming to down-weight factually weak predictions. Despite its novelty, the reliance on entity perturbation for generating nonfactual samples could limit the coverage of detected hallucinations. HaluEval-Wild (Zhu et al., 2024) aims to evaluate LLM hallucination in human-LLM interactions. Their approach first collects challeng-
ing user queries which can lead to hallucinated LLM responses. Faithful reference responses are generated using GPT4 with retrieval augmentation. While the generated data is challenging, it is not obvious how to adapt the approach for customized tasks. Interestingly, Li et al. (2023b) observed that the effectiveness of the LLM-generated synthetic data in supporting model training is negatively correlated with the subjectivity of the target task.

heading: 6 Conclusions
text: In this work, we aim to address the prevalent challenge of training data for hallucination detection being either unavailable or expensive to curate. We hypothesize that this can be addressed via a framework that automatically synthesizes both hallucinated and faithful responses using a prompt-based method. Our experimental results on two datasets verify effectiveness of our approach and show it compares favourably against several baselines, including those using prompt-based synthesis.

heading: 7 Limitations
text: In this work, the quality of the synthetically generated data is partially determined by the capability of prompted LLM. However, this issue is not severe since our goal is to facilitate the fine-tuning process of the hallucination detection model rather than using the data for evaluation.
In addition, note that hallucinations generated in this work, may still be different from those exist unintentionally in system outputs. One promising future work is to explore unintentional hallucination/faithful output generation, such as evaluating output faithfulness via sampling, or perturbing the prompt so that it becomes more or less likely to induce hallucinations.
Lastly, since we are encouraging the LLM to generate hallucinations, there is a risk of introducing misinformation into the real world data, which is also a common issue for large language model generation in general. We encourage people to follow policies and strategies with regarding to data sourcing, fact checking, etc. in order to mitigate such issue.

heading: A Prompt Template for Synthetic Response Generation
text: Table 7 and Table 8 include the prompt templates to generate hallucinated responses and faithful responses.
For BEGIN dataset, we also create a prompt to generate "Generic" responses, as shown in Table 9

heading: B Prompt Template for Simulating Chatbot on OpenDialKG
text: Table 10 contains the prompt template that we use to prompt GPT-4 for system responses on OpenDialKG.

heading: C Rubric/Typology for Qualitative Annotation
text: For the qualitative annotation in Table 4 of the main body, we use the rough definitions/guidelines below. We formulate these types based on prior work on hallucination and hallucination typology such as FRANK.
• Type No 1 : Adding attribute to entity, or adding new value to a known entity.
• Type No 2 : Changing or misspecifying the relation between two entities, or interchanging and swapping their roles w.r.t the same relation.
• Type No 3 : Adding new entities in place of an existing entity, or even otherwise, and mentioning any information about them leaving aside one that purely expresses a noinformation stance
• Type No 4 : Mistakenly claiming knowledge or committing to action about something that the model doesnt really know or cannot act upon
• Type No 5 : A genuine error in the logic and inference beyond just new entities, misattributed or swapped roles and relations.
• Type No 6: Definitely none of the above, it is something else

heading: D AMT Annotation Guidelines, Setup and Template
text: This section describes the AMT annotation guidelines for OpenDialKG-Eval.
A snapshot of the template instructions as seen for an actual example can be viewed in Figure 3. Furthermore, we enclose the complete annotation template [including rules and illustrative examples in its contents] in the form of a single .html file included in the Supplementary Materials along with this submission.
Annotators were restricted to be from Anglophone countries (USA, UK, Australia and New Zealand) to ensure a good likelihood of them being native speakers. Further, annotators were restricted to be from among those with a prior approval rate of atleast 98%.
Annotators were compensated fairly at a rate of 9.3$ per HIT per hour which is well over the minimum wage of 7.25$ per hour in the U.S.A as per Department of Labour estimates for 2023.
We also provide due warning to the annotators not to even inadvertently share any PII or personal information and this is in no way required for our task. We also assure them that time taken etc [nothing beyond the task pertinent annotation] will be used or shared. The disclaimer we include in the template is "Important Disclaimer: Please avoid sharing any personal details or information including PII or demographics anywhere in this study. We will also not be sharing how much time you took to solve this, or what your individual experience profile was. We will merely be using the judgements made about aspects of generated output in relation to input. No other data implicitly or explicitly collected will be shared."

heading: E Statistics of OpenDialKG-Eval
text: OpenDialKG-Eval comprises 180 faithful generations and 132 hallucinations, while the development set contains 39 faithful generations and 51 hallucinations. The inter-annotator agreement, measured by Cohen’s, stands at 0.583, indicating a moderate level of agreement according to Fleiss’s guidelines (Fleiss et al., 1981) on interpreting Cohen Kappa magnitude.
For the annotation process, we utilized a scale ranging from -2 to +2, excluding 0. Here, -2 represents strong hallucination, and +2 signifies strong faithfulness. Based on this scale, instances scoring greater than 1 or less than -1 were allocated to the test set, with the remaining instances assigned to the development set.

heading: F Prompt for Motivating Zero-Shot Detector Experiment in Intro Table 1
text: Prompts are shown in Table 11.

heading: G Prompt for GPT-4 (Internal) Zeroshot Approach
text: Prompts are shown in Table 12

heading: H Experimental Details
text: During finetuning, we use batchsize = 4, apply AdamW gradient descent (Loshchilov and Hutter, 2018) and tune learning rates from the range of [1e-3, 1e-4, 1e-5]. We run 5 epochs for OpenDialKG and 1 epoch over the BEGIN dataset. We evaluate our model, HaluEval-based and FADEbased finetuned models using Dev set, choose the best performing learning rate and report the performances on test set. In addition, we adopt Low Rank Adaptation (Hu et al., 2022) with r=16, α=32,
and target_modules=["q", "v"] during optimization. Our experiments are base on
BEGIN-Adv. has 8k unreleased data for training, while there are only 1.2k data in BEGIN dev that we can leverage for fine-tuning. In order to generate similar amount of training data, we generate 3 synthetic responses per category for each example in BEGIN Dev set. We adopt temperature 0.5 to avoid repeat generation.
Wherever pertinent, we provide mean results over two random runs.

heading: I Additional Arguments for Cost Efficiency of our Method
text: We re-emphasize some key additional points here regarding relative cost efficiency of our approach:
1. Inference Time Efficiency: Our findings, presented in Table 2 and Table 3, demonstrate that a fine-tuned T5-base classifier offers significantly lower latency compared to other base-
line methods. This is crucial for applications where high latency is not viable.
2. An additional pragmatic aspect bolstering our relative cost efficiency angle in the long term is the steadily decreasing per-token costs of API-based colossal LLMs like GPT-4, driven by widening adoption, faster accelerators, better compression/distillation and quantization etc. Specifically for GPT-4, we have in the past year seen three major price decrease events, each by a factor of 2+, on June 13th, November 11th and Jan 25th, causing a price decrease of over 8 fold. In contrast, human annotation costs in dollar terms rise albeit very gradually.

heading: J Out of Domain Evaluation
text: We conducted additional experiments to evaluate our model’s performance in out-of-distribution scenario. Specifically, we utilized the best-performing T5 models, which were finetuned using our dataset generated from OpenDialKG, and applied the model to the BEGIN test set. And we compared with our internal GPT-4 baseline.
Given that the BEGIN dataset features significantly longer knowledge contexts in natural language text, as opposed to the list of triplets from the
Freebase knowledge graph found in OpenDialKG, this setup simulates a substantial out-of-distribution evaluation. To adapt our binary classifiers for use with BEGIN, we collapse the “generic” and “not fully attributable” labels into a single “hallucination” category, resulting in a modified binarized test set. The results are as follows in Table 13, which indicates that our approach leads to signifantly better performance in comparison to a naive random baseline, and shows competitive performance with GPT-4 based zero-shot detection in out-of-distribution scenario, emphasizing its generalization ability.

heading: Future Work
text: One promising future work is to explore unintentional hallucination/faithful output generation, such as evaluating output faithfulness via sampling, or perturbing the prompt so that it becomes more or less likely to induce hallucinations. Lastly, since we are encouraging the LLM to generate hallucinations, there is a risk of introducing misinformation into the real world data, which is also a common issue for large language model generation in general. We encourage people to follow policies and strategies with regarding to data sourcing, fact checking, etc. in order to mitigate such issue