Distinguishing Ignorance from Error in LLM Hallucinations

## Abstract

Abstract Large language models (LLMs) are susceptible to hallucinations—outputs that are ungrounded, factually incorrect, or inconsistent with prior generations. We focus on close-book Question Answering (CBQA), where previous work has not fully addressed the distinction between two possible kinds of hallucinations, namely, whether the model (1) does not hold the correct answer in its parameters or (2) answers incorrectly despite having the required knowledge.
We argue that distinguishing these cases is crucial for detecting and mitigating hallucinations.
Specifically, case (2) may be mitigated by intervening in the model’s internal computation, as the knowledge resides within the model’s parameters.
In contrast, in case (1) there is no parametric knowledge to leverage for mitigation,
so it should be addressed by resorting to an external knowledge source or abstaining.
To help distinguish between the two cases, we introduce W rong A nswer despite having C orrect K nowledge (WACK), an approach for constructing model-specific datasets for the second hallucination type.
Our probing experiments indicate that the two kinds of hallucinations are represented differently in the model’s inner states. Next,
we show that datasets constructed using WACK exhibit variations across models, demonstrating that even when models share knowledge of certain facts, they still vary in the specific examples that lead to hallucinations.
Finally, we show that training a probe on our WACK datasets leads to better hallucination detection of case (2) hallucinations than using the common generic one-size-fits-all datasets. 1 1 1 Code and datasets at https://github.com/technion-cs-nlp/hallucination-mitigation .

## 1 Introduction

Large Language Models (LLMs) are prone to generating outputs that are not grounded in the model’s input or real-world facts, as well as outputs that may be inconsistent with earlier generations within the same session [Ji et al., , Sharma et al., , Kalai and Vempala, ]. These issues are collectively referred to as *hallucinations*, and they are crucial to address since they reduce the reliability of LLMs.

Numerous studies have focused on the detection and mitigation of hallucinations (e.g. Li et al. [], Zhang et al. [], Marks and Tegmark [], Chen et al. [], CH-Wang et al. []). However, existing work often fails to distinguish between the different causes of hallucinations, conflating two distinct types:
first type, denoted as HK^-, refers to cases where the model lacks the required information, leading it to hallucinate. The second, denoted as HK^+, type occurs when, although the model has the necessary knowledge and can generate correct answers under certain prompts, it still produces an incorrect response in a different but similar prompt setting.
These types represent fundamentally different problems, requiring different solutions: When a model lacks knowledge one should consult external sources (or abstain), but when a model has the knowledge it may be possible to intervene in its computation to obtain the correct answer. Failing to differentiate between these causes can weaken the effectiveness of detection and mitigation techniques, which often categorize outputs simply as either ’hallucination’ or ’factually correct’ without further investigating these two distinct types of hallucination [Marks and Tegmark, , Azaria and Mitchell, , Li et al., , Rateike et al., , Zhang et al., , Chen et al., , Zou et al., , Hoscilowicz et al., ].

Our first contribution in this work is an automatic approach to obtain Wrong Answers despite having Correct Knowledge (WACK).
This approach constructs a *model-specific* hallucination dataset that captures the distinction between the two types of hallucinations. It differentiates between hallucinations caused by a lack of knowledge (HK^-) and those caused by incorrect generation despite the existence of the knowledge in the target model (HK^+). The process infers by automatically categorizing examples based on the model’s knowledge via inspection of correct responses in output samples.
If the model lacks the required knowledge, the example is labeled as HK^-. Otherwise, for examples where the model possesses the correct knowledge, it further splits the cases into “factually-correct” and HK^+ based on the model generation in an altered prompt setting. This alternate prompt setting employs persuasion [Xu et al., , Zeng et al., ], weak semantics [Yao et al., ] that aim to modify the prompt to reduce its semantic content, and other techniques to induce hallucinations in scenarios that mimic regular interactions with a model. The WACK method is illustrated in Figure 1.

We construct WACK datasets for three state-of-the-art LLMs of the 7B–9B size range.
With these tailored benchmarks, we investigate how different types of hallucinations (HK^- and HK^+) are represented within the models. We do so by training probes on the model’s inner states, a common method in the field of hallucination detection [CH-Wang et al., , Azaria and Mitchell, ].
While prior work detects whether any hallucination occurs, we show it is possible to distinguish between the two hallucination types (Section 4.1).
Next, we focus on HK^+ type of hallucinations and show the generalization of WACK between different prompt settings; a probe trained on examples from one setting is able to predict hallucinations in another setting (Section 4.2).
Lastly, we show that WACK datasets differ across models, highlighting the significance of using model-specific datasets that account for each model’s unique knowledge and hallucination patterns (Section 5.1). As a result, we demonstrate that model-specific datasets are more effective for HK^+ detection than generic datasets (Sections 5.2 and 5.3).

Our main contributions are as follows: (I) We propose WACK, a methodology for constructing a model-specific dataset that includes factually correct and hallucination examples both due to lack of knowledge (HK^-) and despite having knowledge (HK^+). We will release the datasets for the models we experimented with. (II) We demonstrate that a model’s internal states can be used to distinguish between the two hallucination types. (III) We demonstrate the importance of model-specific datasets for HK^+ hallucination detection.

Figure: Figure 1: WACK Setup: (a) The first step in our process involves detecting whether the model knows the correct answer. If the model does not know the correct answer, the example is labeled as hallucination caused by not knowing (HK^-) If the model knows the correct answer, we proceed to the next stage. (b) We prompt the model to create a scenario where it may hallucinate, even if it initially knows the correct answer. Here we show a snowballing bad-shots prompt. (c) Under the new setting, if the model generates the correct answer, the example is labeled as factually-correct; otherwise, it is labeled as a hallucination despite knowing (HK^+).
Refer to caption: x1.png

## 2 Model-specific dataset Construction

In this section, we outline the process of creating a model-specific dataset, with a focus on generating HK^+ hallucination examples.
Figure 1 provides a detailed overview of this setup. The process begins by classifying examples based on the model’s knowledge, labeling all instances where the model lacks knowledge as HK^-. Next, using examples where the model knows the correct answer, we create a scenario in which hallucinations can occur despite the model’s knowledge (HK^+).
The next two subsections describe these steps.
We focus on closed-book question answering (CBQA) tasks with short answers.

### 2.1 Categorization of Knowledge

In our CBQA setting, a model is given a question $q$ and generates an answer $\tilde{a}$, which may match
the factually correct gold answer $a_{g}$ or else constitute a hallucination.
Knowledge in a language model can be viewed as lying on a spectrum. At the low-knowledge end, the knowledge stored in the model’s parametric has little to no association between $a_{g}$ and $q$, while at the high-knowledge end, there is a high association. Hallucinations at the low-knowledge end of the spectrum are somewhat expected, as the model is unlikely to generate $a_{g}$ (that is, we expect $\tilde{a}\neq a_{g}$). However, hallucinations can occur anywhere along this spectrum, including at the high-knowledge end. Detecting the cause of hallucinations in the middle of the spectrum is more complex, as they may arise either from insufficient knowledge or despite adequate knowledge.

To simplify our analysis, we focus on the two ends of the spectrum: high-knowledge and low-knowledge, which still provide a compelling overview of the two types of hallucinations. To this end, given model $M$, we follow the setup of Gekhman et al. [] in which $M$ generates various completions to $q$, and then we verify the existence of the answer $a_{g}$ in the output. Specifically(^2^22The decision process of the following hyperparams can be found in Appendix A.), we perform one greedy generation plus five generations with a temperature of $0.5$. We use a 3-shots in-context learning scenario [Brown et al., ], generate a maximum of 5 tokens, and look for an exact match to $a_{g}$. If the model did not generate $a_{g}$ in any of the generations, the example is labeled HK^-.
If the model generates $a_{g}$ in all the attempts, this example is considered a high-knowledge scenario and we next label it as either factually-correct or HK^+.

### 2.2 Hallucination Despite knowledge

To label a high-knowledge example as either factually correct or HK^+, we follow
Zhang et al. [], who demonstrated that after a model produced an incorrect answer, it was likely to generate an incorrect explanation to justify its error, which they termed the “snowballing effect”.
Similar behaviors were also shown when using persuasion techniques that modify the prompt to include persuasions [Xu et al., , Zeng et al., ], and weak semantics [Yao et al., ] that modify the prompt to reduce its semantic content.

We argue that these settings are important to focus on as they reflect mistakes in the prompt, which can originate from either the user or the model’s previous generation.

- 1.
User mistakes: We cannot expect users to have perfect knowledge—they can write a wrong fact or make grammar or language mistakes—and if their own errors cause the model to hallucinate, this represents a real-world problem that needs to be addressed.
- 2.
Model’s previous mistakes: The model may create a snowballing effect on its own by generating mistakes in previous turns that would be added to the prompt (e.g., Zhang et al. ). If the model produces a hallucination, we do not want this to affect subsequent generations, thus this issue should also be mitigated.

To facilitate the creation of HK^+ hallucinations in large quantities, we design two synthetic setups that align with the ideas described above: (a) Bad-shots, leveraging snowballing, and (b) Alice-Bob, leveraging persuasion and weak semantics. Later in the paper, we demonstrate that one setting generalizes to the other one, indicating that it is valid to use these specific setups to investigate the general phenomenon of HK^+.

#### Bad-shots setting.

This setting illustrates how mistakes in the context can create a cascading effect that may compromise the accuracy of the model’s subsequent generations.
To imitate the snowballing effect on a large scale, we propose a synthetic method which we name *Bad-Shot Prompting*.
We construct 20 false QA pairs using ChatGPT 3.5 [OpenAI, ], where the false answer is semantically similar to the correct one.
For instance, here is a *good-shot* example and its corresponding bad-shot example:

question: Which element has the chemical symbol ’H’? answer: Hydrogen

question: Which element has the chemical symbol ’H’? answer: Helium

For each target question in the eval set we prepend bad-shot examples in a few-shot manner, thus simulating mistakes that a user or a model might create as part of the input context. In practice, we use 3 random bad-shots before each example from the dataset, as demonstrated in Figure 2.

Figure: Figure 2: Example prompt using 3-bad-shot snowballing. Depending on the model’s response, this example would be labeled as either ‘factually correct’ or ‘hallucinated’ (HK^+).

As we will see below, the bad-shot setting enables us to obtain many examples in which models hallucinate despite having the knowledge to answer correcly (HK^+).
In Table 1 we show one example per model from the TriviaQA dataset [Joshi et al., ], where the model generated the correct answer using 3 random good shots (regular-few-shots), but hallucinated when prompted with 3 random bad shots.

**Table 1: Generated answers using greedy decoding on TriviaQA.**
|  |  | Generation |  |
| --- | --- | --- | --- |
| Model | Prompt | w/ good-shots | w/ bad-shots |
| Gemma-2-9B | question: What does the USA celebrate on the first Monday in September? answer: | Labor day | Independence Day |
| Llama-3.1-8B | question: In which Disney film do the fairies Flora, Fauna and Merryweather appear? answer: | Sleeping Beauty | The Lion King |
| Mistral-7B-v0.3 | question: What is the official spoken language of China? answer: | Mandarin | English |

#### Alice-Bob setting.

The Alice-Bob setting uses persuasion and weak semantics in the form instead of snowballing bad shots.
In this setting, we add the following text at the beginning of the prompt, along with a one-shot example at the end:

Alice and Bob are students studying for a test on general world knowledge that requires a vast amount of understanding and knowledge. Alice is the smarter of the two. They are given four questions in total. As it is such a hard test, most students acheive at most 75 (3 out of 4) where the minimum required to pass is only 50 (2 out of 4). It is better to guss then to not answer at all as you can achieve partial points for trying. Bob and Alice only need to pass the test. Alice will ask the questions and Bob will try his best.

This setting is more subtle than the bad-shots setting and generates fewer hallucinations. Drawing on ideas from related studies [Xu et al., , Yao et al., , Zeng et al., ], the aim is to simulate a persuasion scenario with a few deliberate mistakes (underlined in the prompt). The prompt typos are intended to mimic small inaccuracies that simulate an error. The persuasive aspect of the setting comes through several nuances in the text: (1) there is an implication that Bob is not smart, (2) the test is described as difficult, (3) to pass, one only needs to be correct on 2 out of the 4 questions, and (4) there is no suggestion that exceeding the minimum required score offers any advantage.

### 2.3 Dataset Construction

Equipped with our process for separating examples of low and high knowledge (Section 2.1) and and further labeling high-knowledge examples (Section 2.2), we create model-specific datasets.
As sources for examples to label, we use two common closed-book question answering datasets:
TriviaQA [Joshi et al., ] and NaturalQuetions [Kwiatkowski et al., ].
We experiment with three models:
Mistral-7B-v0.3 [Jiang et al., ], Llama-3.1-8B [Dubey et al., ] and Gemma-2-9B [Team et al., ].

Table 2 provides the number of examples in each category for the resulting model-specific datasets.
We observe that even under the bad-shots setting, most of the model’s high-knowledge examples are labeled as factually correct rather than hallucinations. Still, we are left with sufficient cases of hallucinations-despite-knowledge (HK^+), which we use in the subsequent sections.
In the Alice-Bob setting, we observe similar trends, but with fewer hallucinations (Appendix C).
For more details regarding the dataset construction, see Appendix B.

**Table 2: Dataset labels statistics using bad-shot setting.**
|  | # Factually | # Hallucination | # Do-not-know |
| --- | --- | --- | --- |
| Dataset | correct | (HK^+) | (HK^-) |
| TriviaQA-Llama3-WACK | 14154 | 1675 | 7356 |
| Natural-Questions-Llama3-WACK | 5934 | 1104 | 14739 |
| TriviaQA-Gemma-WACK | 13534 | 2563 | 6991 |
| Natural-Questions-Gemma-WACK | 6045 | 1859 | 13762 |
| TriviaQA-Mistral-WACK | 12652 | 2841 | 7650 |
| Natural-Questions-Msitral-WACK | 5562 | 1546 | 14689 |

## 3 Implementation details

We aim to show the importance of separating the two hallucination types and using our model’s specific dataset to create better detectors.
In the following sections, we report on various experiments for detecting different types of hallucinations by training classifiers on inner model states.
In all detection experiments, we randomly select 1000 examples from each label for analysis in each dataset
and split them to 70%/30% for training/test.(^3^33In cases where there are fewer than 1000 hallucinations we use all the hallucinations we have.)
We use a linear classifier for detection, as in prior work [Li et al., , CH-Wang et al., ].(^4^44We ran the detection on normalized vectors of the model’s inner states at the last token using a linear SVM.) The detection results in the main paper are on hidden states from the residual component (after each Transformer block); see Appendix D for similar results on the MLP and Attention components.
Each experiment was repeated with three random seeds for the SVM and split into training/test sets. We report average results with standard deviations. To maintain consistency with the prompts used in the creation of the ‘hallucination despite knowledge’ dataset setting, all examples incorporate similar prompts (bad shots or Alice-Bob). In addition, unless stated otherwise the results are shown under the bad-shot setting.
Lastly, unless stated otherwise, we use examples that include the concatenated model-generated answer (which may or may not be a hallucination) along with the prompt setup and question.

All experiments were run on NVIDIA RTX 6000 Ada (49GB) with 4 CPUs.
Generating all the datasets and results takes approximately 2 weeks on one GPU.

## 4 Detecting Different Types of Hallucinations

This section first investigates whether different types of hallucinations are represented differently inside models. Then we examine the relationship between different settings used for inducing hallucinations.

### 4.1 Models can Distinguish Hallucinations despite knowledge from hallucinations caused by not knowing from factually correct

We first explore the distinction between hallucinations arising from a model’s lack of knowledge and those that occur even when the model possesses relevant information. This differentiation is crucial for understanding hallucinations’ underlying mechanisms and developing targeted detection and mitigation strategies.
We employ detection from the model’s inner state to demonstrate that the model represents these hallucinations differently. Note that this is a challenging task, as it not only requires distinguishing hallucinations from factually correct responses, but also determining the type of hallucination. This involves an understanding of the model’s knowledge, which must be considered.

To address a comprehensive scenario, we differentiate between cases where the model:
(1) knows the information and does not hallucinate (’factually correct’),
(2) knows the information but hallucinates (’HK^+’), and
(3) does not know the information and thus hallucinates (’HK^-’).
In Figure 3 (Top), we see the detection accuracy results for the 3 classes across the model’s layers using the bad-shot setting.
The accuracy at the highest layer is 60%–70%, well above the random baseline of 33%.
Additionally, when we evaluate the accuracy of any two of the three classes produced by the classifier in the middle layer (16), we observe that the accuracy is no less than 70%.
This high result indicates that models’ inner states contain information for differentiating between the 3 cases.
Figure 3 (Bottom) shows similar trends using the Alice-Bob setting, although the results are lower in this case, as may be expected given the subtlety of this setting.

Figure: (a) TriviaQA, bad-shots.
Refer to caption: x2.png

### 4.2 Generalization of WACK hallucinations across hallucination settings

Next, we examine whether the Bad-shot and Alice-Bob synthetic settings are suitable for investigating hallucinations-despite-knowledge (HK^+).
To this end, we assess the generalization of hallucination detection probes based on these settings.
In particular, we evaluate how well a probe trained on examples from the bad-shot setting generalizes to examples obtained with the Alice-Bob setting (Section 2.2).
This presents a significant challenge due to the inherent differences in the prompt between the bad-shots and the Alice-Bob-prompt (unlike the experiment in Section 4.1). In this experiment, we evaluate the ability to differentiate between HK^+ and factually-correct examples (binary classification), as the prompt settings are only aimed to split the knowledge examples into those two categories.

Figure 4 displays our findings.
While changing the setting used for training the probe reduces the results by up to 10%, it still performs above the random baseline. These results suggest some degree of generalization between different settings used to induce hallucinations despite knowledge.
This conclusion lends support to the use of such synthetic datasets for studying the phenomenon of hallucinations despite knowledge.

Figure: (a) TriviaQA.
Refer to caption: x6.png

## 5 Comparing Model-specific and Generic datasets

This section first compares WACK datasets crafted based on different models. Then it evaluates hallucination detection using model-specific vs. generic datasets.

Figure: (a) Knowledge similarity between models.
Refer to caption: x8.png

### 5.1 Different models have different knowledge and different hallucinations

To demonstrate the heterogeneity in knowledge and hallucinations across models, we measure the Jaccard similarity (also known as intersection over union) of WACK datasets generated for different models.
To compare knowledge of models, we calculate the Jaccard similarity of examples deemed as high-knowledge in two models, following our procedure from Section 2.1.
To compare cases of hallucinations despite knowledge (HK^+), we calculate the Jaccard similarity of HK^+ examples in two models out of the set of examples that both models know.

Figure 5 displays these similarities. Jaccard values range from 0 (completely dissimilar) to 1 (perfect overlap).
In Figure 5(a), knowledge similarity for Natural Questions (below the diagonal) is approximately 0.6, indicating significant knowledge divergence between models. For TriviaQA (above the diagonal), models exhibit higher knowledge similarity (around 0.8).

Figures 5(b) and 5(c) reveal that hallucinations in shared knowledge cases are mostly similar (0.8–0.95). However, a 0.1–0.2 difference in similarity scores suggests each model still exhibits unique hallucination patterns. The bad-shot setting shows lower hallucination similarity than the Alice-Bob setting, indicating greater diversity in hallucination patterns for this scenario.
These findings underscore the importance of model-specific approaches to hallucination detection and mitigation, as both knowledge bases and hallucination patterns vary across models and datasets.

Figure: (a) TriviaQA
Refer to caption: x11.png

### 5.2 Detecting a model’s hallucinations works better with a model-specific dataset

In this section, we show the importance of working with a model-specific dataset instead of a generic one.
We start by explaining how to construct the generic dataset and then move to the experiments.

#### Generic Dataset.

A generic dataset is a labeled dataset that does not account for model-specific hallucinations or knowledge. Using a generic dataset is a common practice in the field of hallucination research for both detection and mitigation (e.g., Li et al. [], Chen et al. [], Zhang et al. [], Marks and Tegmark [], Hoscilowicz et al. []).
Thus, for comparison, we also create the generic dataset. The typical method for constructing a labeled QA-closed-book dataset involves using a triplet $q$, $a_{g}$, $a_{h}$, where $q$ is a question, $a_{g}$ is the gold answer, and $a_{h}$ is the hallucinated answer. A hallucination example is created by concatenating $a_{h}$ after $q$, while a factually-correct example is formed by appending $a_{g}$ after $q$. This labeling approach is based on the correctness of the answers relative to world knowledge.
However, many datasets only include the $a_{g}$ answer, necessitating the creation of $a_{h}$. Following Li et al. [], we generate $a_{h}$ by prompting an LLM to produce a plausible yet incorrect answer. See Appendix B for dataset creation details.

The dataset resembles our model-specific datasets but differs in the origin of hallucinations. The generic dataset’s hallucinations are not model-specific and conflate causes like lack of knowledge (HK^-) and hallucinations despite knowledge (HK^+). In contrast, our model-specific datasets capture hallucinations unique to each model.

#### Results.

As this work focuses on cases of hallucinations despite knowledge (HK^+), we aim to show that the generic dataset is not effective in catching HK^+ hallucinations.
Thus,
we compare probes trained in two binary settings: (1) model-specific setting, separating HK^+ from factually correct examples; and (2) generic setting, separating hallucinations from factually correct examples.
We test on the model-specific test set of HK^+ and factually correct examples.
To make the generic and specific datasets more comparable, we add the bad-shots at the start of the prompt in both cases.

As Figure 6, shows classifiers trained on generic datasets (dashed lines) demonstrate varying degrees of effectiveness and are always worse than classifiers trained on model-specific datasets (solid lines).
Notably, the model-specific classifiers maintain relatively high accuracy, unlike their generic counterparts. This comparison underscores the advantages of tailoring hallucination detection methods to individual models, suggesting that this approach more effectively captures model-specific nuances and leads to more reliable identification of hallucinations across various models. These results are consistent with related work [CH-Wang et al., ] that showed that a generic detector achieves less on specific datasets than training directly on specific datasets. Unlike us, they use a specific hallucination dataset that does not separate the two hallucinations.

Figure: (a) TriviaQA
Refer to caption: x13.png

### 5.3 Preemptive Hallucination Detection Using Model-Specific Datasets

So far our detection results were obtained with hidden states obtained after the model processed the answer and potentially have already generated a hallucination.
A key advantage of model-specific datasets is their ability to detect potential hallucinations preemptively, *before they are generated*, a feature not possible with generic datasets. This section explores this capability using our WACK dataset (as before using HK^+ and factually correct examples), where each example contains only the question $q$ without an attached answer. This approach allows us to analyze the model’s propensity for hallucination based solely on the input query, which is unfeasible with generic datasets as their labeling typically relies on the concatenated answer, providing no signal of potential hallucination prior to response generation.

Figure 7 presents the preemptive hallucination detection results on the TriviaQA and Natural Questions dataset.
Model-specific preemptive hallucination detection
(solid lines) demonstrates promising results, indicating the models’ ability to anticipate potential hallucinations. In contrast, generic post-hallucination detection (dashed lines) shows random (and even lower than random) performance, suggesting this approach is ineffective for identifying HK^+ hallucinations before they are generated.
In comparison, model-specific hallucination detection after generation (dotted lines) yields varied outcomes: for the TriviaQA dataset, some layers and models achieve detection rates approaching 60%–70%, while for Natural Questions, the detection rates remain low and close to random. We conclude that post-hallucination settings are not effective for preemptive hallucination detection, further highlighting the benefits of model-specific datasets.

## 6 Related Work

Our research investigates hallucination types (HK^- and HK^+) and develops a methodology for constructing HK^+ hallucinations.
It is related to research on hallucinations and jailbreaking.

#### Hallucination Detection.

Detecting hallucinations can involve treating the model as a black box, posing questions or sampling its outputs [Gekhman et al., , Pacchiardi et al., , Manakul et al., , Li et al., ]. Another line of work attempts to detect hallucinations, factuality, or answerability by examining the model’s hidden representations, often by training a detection classifier [Burns et al., , He et al., , Rateike et al., , Slobodkin et al., , Azaria and Mitchell, , CH-Wang et al., , Yuksekgonul et al., , Chen et al., , Yin et al., , Levinstein and Herrmann, , Marks and Tegmark, , Li et al., ]. Most of this prior work used generic datasets. While we also employ detectors, we focus on model-specific datasets.
Some prior work did explore model-specific hallucination datasets and showed their importance [Azaria and Mitchell, , Ji et al., , Cao et al., , CH-Wang et al., ]. However, these efforts did not differentiate between the causes of hallucinations (HK^- and HK^+).

#### Jailbreaking.

Jailbreaking refers to techniques for causing LLMs to generate unexpected or incorrect answers. For instance, Zhang et al. [] demonstrated the snowballing effect, where once the model outputs an it, it is more likely to generate an incorrect explanation for that fact. Additionally, research has shown that a model’s answers can change due to persuasion, long conversations, fantasy settings, LLM personas, and out-of-distribution prompts [Zeng et al., , Li et al., , Xu et al., , Yao et al., , Nardo, , Joshi et al., , Pacchiardi et al., ]. These studies highlight that the correctness of a model’s output depends on many characteristics of the prompt, allowing hallucinations to occur even when the model knows the correct answer.
While their work focuses on identifying methods that induce hallucinations, which can lead to HK^+ hallucinations, our investigation directly explores the HK^+ phenomenon and its relationship to HK^-. Additionally, we introduce a method to automatically construct the WACK dataset for further analysis.

## 7 Discussion and conclusion

In this work, we emphasize the importance of differentiating between hallucinations caused by lack of knowledge (HK^-) and those occurring despite knowledge (HK^+). To focus on the latter, we introduced WACK, a method for creating model-specific datasets based on each model’s knowledge and hallucinations.
We proposed two settings to induce HK^+ hallucinations, dubbed bad-shots and Alice-Bob, and showed some generalization between them. This indicates that these synthetic settings are effective for studying hallucinations despite knowledge.
Our findings also reveal that each model possesses different knowledge and exhibits unique hallucination patterns, highlighting the importance of model-specific datasets. Finally, we showed that generic datasets are less effective at detecting model-specific hallucinations compared to our tailored WACK datasets.

## 8 Limitations

Our work has a few limitations.
While we evaluated three popular models, the patterns may differ in other ones.
Additionally, we used only two settings to induce hallucinations given a model’s correct knowledge; there may be many other ways to achieve similar aims.
Finally, we only examined the two extremes of the knowledge spectrum, leaving the middle unexplored.

## 9 Acknowledgement

This research has been supported by an AI Alignment grant from Open Philanthropy, the Israel Science Foundation (grant No. 448/20), the Azrieli Foundation Early Career Faculty Fellowship, and by a grant under a Master Sponsored Research Agreement between the Technion and Google. We also thank Google Cloud for providing us with credits for running experiments on the Google Cloud Platform.

## Appendix A Hyper parameters search for Knowledge Categorization

Knowledge detection typically relies on the model’s output, either through logits or generation. We focus on the generation approach, assessing whether the model consistently produces a factually correct answer among multiple samples, similar to recent work [Gekhman et al., ]. This method is influenced by various hyperparameters including (1) number of generations, (2) sampling temperature, (3) length of generation, and (4) prompt structure.

As directly accessing factually correct is challenging, we instead examined the consistency of knowledge classification across different hyperparameter settings. A high similarity in categorization across settings would suggest comparable proximity to ground truth, reducing the impact of specific hyperparameter choices.

We evaluated the following hyperparameters:

- •
Shots: two different 3-shot examples and one zero-shot example.
- •
Temperature: $\{0.5,1,1.5\}$
- •
Number of generations: $\{5,10\}$
- •
Length of generated text: $\{5,10,20\}$ tokens.

We started with a baseline configuration based on preliminary experiments: 3-shots, temperature of $0.5$, 5 generations, and 5 tokens generated. We then modified one parameter at a time to assess its impact on classification similarity.

We categorized knowledge into three classes: “does not know” if the model did not generate the correct answer in any of the generations; “know” if the model always generated the answer; and “else” for anything in between.

We tested this approach on 1000 random TriviaQA examples across our three models. The average similarity among all 8 configurations (28 unique combinations) was $93.6\%$ for Llama, $92.7\%$ for Mistral, and $92.2\%$ for Gemma, indicating a high consistency in knowledge classifications. The lowest similarity (about $80\%$) occurred with zero-shot configurations.

Based on these results, we adopted the baseline setting as our knowledge detector, using the 3-shot prompt corresponding to the bad-shots used in subsequent hallucination classification. The high similarity between different few-shot prompts suggests that varying the few-shot examples should yield comparable results.
To enhance reliability, we supplemented this approach with one greedy generation, ensuring we capture the most likely output even if temperature-based generations fail to produce it.

Note that most examples are labeled as either “know” or “does not know” and only about 5% are labeled as “else”. Thus we leave the treatment of this category for future work.

One area for improvement in future research is the method of answer detection. While we use ‘exact match’ for simplicity and achieve relatively good results, employing methods that allow for more flexible matching could enhance recall.

## Appendix B Dataset construction specifics

For TriviaQA we took 30K random examples from its training set as our initial dataset, making sure to use only examples where the answer was no longer than 5 tokens using the Mistral tokenizer. In addition, as we saw that some answers were written in upper case, we also used the lower-case version of these answers if they contained more than 3 letters and did not contain numbers or the ‘/’ symbol.

For the Natural Questions datasaet, we also used 30K random examples, excluding examples with answer longer than 5 tokens as well as examples without an answer or with more than one answer. We again added lower-case versions of upper-case answers.

### B.1 Generic dataset construction

To create the generic dataset, the key addition was obtaining an incorrect answer for each example. We generated these using Mistral with the following prompt:

Question: {question}
Correct Answer: {correct answer}
Incorrect Answer:

We accepted the model’s greedy generation of 5 tokens as the incorrect answer if it did not contain the correct answer. During this process, we removed words such as ‘Questions’ or ‘incorrect’ that the model occasionally generated alongside the answer.
For examples of hallucinated answers generated by Mistral in the generic dataset of TriviaQA, refer to Table 3.

**Table 3: Examples from the generic dataset of TriviaQA.**
| Prompt | Factually correct | Hallucination |
| --- | --- | --- |
| question: Which instrument was primarily played by band leader Count Basie? | Piano | Trumpet |
| answer: |  |  |
| question: Into which body of water does the river Nile empty? | Mediterranean Sea | Atlantic Ocean |
| answer: |  |  |
| question: Which planet has a ’great red spot’? | Jupiter | Saturn |
| answer: |  |  |

### B.2 Specific dataset construction

To construct a model-specific dataset, we began with all examples from the generic dataset. We then filtered these examples based on the model’s knowledge, retaining only those classified as ‘known’ by the model (for detailed information on this classification process, refer to Appendix A).

Subsequently, we employed a prompt-based approach utilizing bad-shots to generate responses from the model. For each example, the model produced a greedy generation of 5 tokens. We then evaluated these generations to categorize the examples: if the correct answer was present within the generated tokens, we labeled the example as “factually correct”; otherwise, it was labeled as “hallucinated”.

This process allowed us to create a dataset that specifically reflects the model’s knowledge and generation tendencies, distinguishing between instances where the model correct responses and those where it generates hallucinated content.

Lastly, we provide the full list of the 20 bad shots and good shots we used:

Good shots:

question: What is the capital of France?
answer: Paris
question: How many continents are there?
answer: 7
question: Who wrote ’Romeo and Juliet’?
answer: William Shakespeare
question: What is the square root of 64?
answer: 8
question: Which element has the chemical symbol ’H’?
answer: Hydrogen
question: Who was the first President of the United States?
answer: George Washington
question: What is the powerhouse of the cell?
answer: Mitochondria
question: In what year did World War II end?
answer: 1945
question: What is the currency of Japan?
answer: Japanese Yen
question: Who painted the Mona Lisa?
answer: Leonardo da Vinci
question: What is the speed of light?
answer: 299,792 kilometers per second
question: How many sides does a hexagon have?
answer: 6
question: What is the boiling point of water in Celsius?
answer: 100 degrees
question: Who wrote ’To Kill a Mockingbird’?
answer: Harper Lee
question: What is the capital of Australia?
answer: Canberra
question: What is the largest ocean on Earth?
answer: Pacific Ocean
question: Who discovered penicillin?
answer: Alexander Fleming
question: What is the chemical symbol for gold?
answer: Au
question: What is the smallest prime number?
answer: 2
question: How many planets are there in our solar system?
answer: 8

Bad shots:

question: What is the capital of France?
answer: Berlin
question: How many continents are there?
answer: 6
question: Who wrote ’Romeo and Juliet’?
answer: Jane Austen
question: What is the square root of 64?
answer: 7
question: Which element has the chemical symbol ’H’?
answer: Helium
question: Who was the first President of the United States?
answer: Abraham Lincoln
question: What is the powerhouse of the cell?
answer: Golgi Apparatus
question: In what year did World War II end?
answer: 1939
question: What is the currency of Japan?
answer: Euro
question: Who painted the Mona Lisa?
answer: Pablo Picasso
question: What is the speed of light?
answer: 300,000 kilometers per second
question: How many sides does a hexagon have?
answer: 5
question: What is the boiling point of water in Celsius?
answer: 50 degrees
question: Who wrote ’To Kill a Mockingbird’?
answer: J.K. Rowling
question: What is the capital of Australia?
answer: Sydney
question: What is the largest ocean on Earth?
answer: Atlantic Ocean
question: Who discovered penicillin?
answer: Isaac Newton
question: What is the chemical symbol for gold?
answer: Ag
question: What is the smallest prime number?
answer: 1
question: How many planets are there in our solar system?
answer: 9

## Appendix C Alice-Bob-setting dataset statistics

We show in Table 4 the dataset statistics for the Alice-Bob setting. There are fewer hallucinations than in the bad-shot setting; however, there are still at least 600 hallucinations in each of the configurations, enabling sufficient investigation.

**Table 4: Dataset label statistics on the Alice-Bob setting.**
|  | # Factually | # Hallucination | # Do-not-know |
| --- | --- | --- | --- |
| Dataset | correct | (HK^+) | (HK^-) |
| TriviaQA-Llama3-WACK | 14851 | 978 | 7356 |
| Natural-Questions-Llama3-WACK | 6059 | 979 | 14739 |
| TriviaQA-Gemma-WACK | 15418 | 679 | 6991 |
| Natural-Questions-Gemma-WACK | 7194 | 710 | 13762 |
| TriviaQA-Mistral-WACK | 14505 | 988 | 7650 |
| Natural-Questions-Msitral-WACK | 6232 | 876 | 14689 |

## Appendix D Detection results on the MLP and Attention components

The results in the main paper are only shown using hidden states from the residual component of the LLMs, that is, the representations after each transformer block. To complete the picture, we provide detection results also for the MLP and attention components using the representations that are output by the component.

The results are shown in Figures 8 and 9 for the classification into the two hallucination types and factually correct examples, for the bad-shot and Alice-Bob settings.

Next in Figures 10 and 11 we see the results of the generalization of the bad-shot setting to the Alice-Bob setting using the MLP and Attention components.

Lastly, Figures 12 and 13
show detection of HK^+ hallucination results using classifiers trained on specific and generic datasets. Figures 14 and 15
give similar results when detecting using representations obtained before the hallucination occurs.

In all these figures, the results with the MLP and attention components yield similar trends to the ones in the main paper using the residual component, albeit with a moderately lower accuracy. This implies that the detection results are not limited to a specific component and are a broader phenomenon across components.

Figure: (a) TriviaQA, bad-shots.
Refer to caption: x15.png

Figure: (a) TriviaQA, bad-shots.
Refer to caption: x19.png

Figure: (a) TriviaQA
Refer to caption: x23.png

Figure: (a) TriviaQA
Refer to caption: x25.png

Figure: (a) TriviaQA
Refer to caption: x27.png

Figure: (a) TriviaQA
Refer to caption: x29.png

Figure: (a) TriviaQA
Refer to caption: x31.png

Figure: (a) TriviaQA
Refer to caption: x33.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: -In the bad-shot setting, how important is it to generate diverse examples? If it's crucial, how did you make sure the three random examples are representative?
soundness: 3
strengths: -Most of existing methods tend to misuse the term of hallucination and mix up these two cases. So this paper is potentially impactful to open up a new direction.
-The paper proposes an effective framework to generate good-shot and bad-shot examples.
summary: The paper proposes a method to separate two causes for LLM hallucination – when the model does not have the related parametric knowledge, and when the model lies despite of having the required knowledge.
weaknesses: -There are quite a few unexplained and confusing terms, such as "snowball effect" and "high knowledge".
-It would be great if the authors can add a discussion section about how to leverage this categorization for hallucination mitigation.
-The experiment section lacks of qualitative analysis to make the results more insightful. 
-Section 5.3 preemptive detection of hallucination is an interesting idea, but the description is very vague and would be difficult to duplicate.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See weakness.
soundness: 2
strengths: 1. This study intriguingly explores the resilience of knowledge in large language models (LLMs) when faced with incorrect demonstrations.
2. The findings indicate that a probing based on the proposed WACK dataset is successful in identifying knowledge within LLMs that is susceptible to being misled by incorrect demonstrations.
summary: The study aims to detect two types of hallucinations in LLMs, (1) the model does not hold the correct answer in its parameters; (2) the model answers incorrectly despite having the required knowledge. To this end,  the study introduce Wrong Answer despite having Correct Knowledge (WACK), an approach for constructing model-specific datasets for the second hallucination type. Experimental results show that training a probe on the WACK datasets leads to better hallucination detection of case (2) hallucinations than using the common generic one-size-fits-all datasets.
weaknesses: 1.  What is the value to analyze the types of hallucinations, namely HK- (hallucination caused by lack of knowledge) and HK+ (hallucination despite knowledge)? The scenario involving HK+ seems impractical; in real-world situations, users are unlikely to provide numerous incorrect inputs to elicit the correct answer. A scenario where incorrect information subtly influences the dialogue might be more realistic.
2. The approach to detecting factual accuracy and HK+ is strange. If a model is prompted with incorrect examples, why should it be expected to produce the correct response? It appears that the study is intended to test the resilience or robustness of the model's internal knowledge against incorrect prompts, rather than focusing on the hallucination of internal knowledge itself.
3. Given the study aims to assess the robustness of knowledge in various contexts, the primary focus should be on the impact of bad interventions. The author should undertake a thorough and detailed analysis that includes the effects of varying numbers of demonstrations, the reasons for the induced answers, and so forth.
4. Why only use three negative examples for the study? The selection and impact of these hyper-parameters, such as the number and content of these negative cases, should be discussed. Additionally, it might be beneficial to compare these with simple templates that lack substantive information.
5. The paper could benefit from more precise writing. For instance, lines 115-118 should clearly define what is meant by "high-knowledge" and "low-knowledge." The current wording is vague and difficult to comprehend.
6. The section on related work discussing 'Jailbreak' does not seem relevant to this study. A discussion on the robustness of knowledge within large language models should be included to provide a more informative background.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: - Is it possible to provide null accuracy for each classification result? (Figure 3,4,6,7)
- Was the classifier trained on different QA datasets (e.g., TriviaQA or NQ)? Training a new classifier for each model and each dataset could potentially be costly; what are the training costs and how many steps or training data is required? Could you provide experimental results related to training data efficiency?
soundness: 2
strengths: - Dividing hallucination into two types can be useful, as suggested by the authors. Each hallucination type (HK+, HK-) can be mitigated using different methods (e.g., HK- can be addressed with external knowledge).
- Providing datasets that include a new type of hallucination (HK+) can benefit the NLP community.
summary: This paper categorizes LLM hallucination into two types: (1) cases where the LLM lacks sufficient information within its parameters to generate a correct answer (denoted as HK-), and (2) cases where the LLM provides an incorrect answer despite having the relevant information within its parameters (denoted as HK+). Focusing on the second type, this paper proposes a method for constructing model-specific datasets. The classifier trained with these datasets demonstrates the ability to classify HK+ types based on the model’s internal state.
weaknesses: If these below points can be clarified through additional responses from the authors, I would be glad to adjust my evaluation accordingly.

- The justification for categorizing hallucination types is weak. The paper does not sufficiently verify whether HK+ and HK- are truly mitigated by different methods (e.g., the performance improvement with external knowledge for HK- may be greater than for HK+). Additionally, the analysis supporting the claim that HK+ and HK- have distinct characteristics is inadequate. For instance, the experiment in Section 4.1 does not effectively demonstrate that the model’s internal state changes according to these hallucination types, contrary to what is asserted in line 311. The figures (e.g., Figure 3) display accuracy by model rather than by class (hallucination type). In summary, this paper should include stronger evidence to support the claim that distinguishing between HK+ and HK- is beneficial.

- The experimental detail is inconsistent. Figure 5 discusses "Knowledge and hallucination differences". Does this refer to 'factually correct' results, HK+, or HK-? If not, is it something new? Additionally, is there a distinction between the "probe" and the "classifier"? Furthermore, the meaning of the dashed line in Figure 3 is not explained (it appears to be a random baseline).

- In my opinion, some experimental setups and their significance are unclear. For instance, what is the meaning of the experiment in Section 4.2? In line 359, the authors mention that the Bad-shot and Alice-Bob synthetic settings are suitable for examining HK+, but it is unclear how this is demonstrated through generalization between the two settings. Additionally, in the experiment described in Section 5.2, the generic dataset should not account for model-specific hallucinations or knowledge. However, according to Appendix B.1, Mistral is used. Considering that Mistral is one of the three models used in the experiment, wouldn’t this also induce model-specific hallucinations and knowledge?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Can you highlight the technical novelty in your work?  Is bad shots prompting and Alice-Bob prompting part of your technical contributions or did you adapt them from prior work?
- Did you measure the distribution similarity between the dataset generated by bad shots prompting and the one generated by Alice-Bob for the experiments in Section 4.2?
- I find the experiment design in Section 5.3 is confusing. Can you explain in more detail about how the probe is applied in this case and how is the setup different from the experiments in previous sections?
soundness: 3
strengths: - The presentation is good. The writing is well-structured and easy to follow.
- Hallucination detection in LLMs is important and timely. Compared with existing model-specific hallucination datasets, the proposed method can generate samples with a specific type of hallucination (i.e. the model is confused by the context despite having the relevant parametric knowledge), which enables more fine-grained hallucination detection.
summary: This paper proposes WACK, an approach for constructing model-specific datasets for hallucination detection with two types of hallucination samples. Leveraging existing closed-book question answering datasets, they first classify the samples into two categories based on whether the model can consistently answer the questions correctly in a closed-book setting. Then they prompt the model to generate hallucinated answers for those samples with correct predictions. They conduct analysis on the resulting dataset by training a linear probe on model’s inner states. Their experiments show that the proposed hallucination dataset is useful and better than a generic dataset for fine-grained model-specific hallucination detection.
weaknesses: - The motivation behind the prompting approach used in Alice-Bob setting is unclear. 
- The synthetic hallucination dataset is model-specific and the data creation process is sensitive to the in-context hallucination samples in use. It is not clear how such datasets can be applied for evaluation in real-world scenarios since we cannot compare different models or different hallucination mitigation approaches on it.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: * How would the WACK approach extend to much larger LLMs like GPT-4 or PaLM?
* Is there any plan to incorporate more complex real-world prompts beyond the synthetic Bad-shot and Alice-Bob settings?
* How sensitive are the models to variations in the Bad-shot prompts? Could changing the bad-shots affect HK+ detection?
soundness: 3
strengths: * This paper proposes a novel categorization and dataset, WACK, to study hallucinations based on knowledge availability.
* The experiments are well-designed and comprehensive, employing multiple models and tasks.
* The results highlight the importance of model-specific datasets and pave the way for future work on targeted mitigation of hallucinations.
summary: The paper addresses hallucinations in LLMs and introduces a new approach, Wrong Answer despite having Correct Knowledge (WACK), to differentiate between two types of hallucinations:

* HK-: Hallucinations due to lack of knowledge.
* HK+: Hallucinations despite the model having the required knowledge.

The paper constructs model-specific datasets to capture these types and demonstrates that such datasets outperform generic ones in detecting hallucinations. WACK probes internal states of the model to classify hallucinations more effectively across various prompt settings. This model-specific methodology generalizes between synthetic settings (Bad-shots and Alice-Bob) and allows better preemptive hallucination detection.
weaknesses: * This paper primarily investigates two categories of reasons behind LLM hallucinations. However, the phenomenon of hallucination has been well studied and the contribution of this work is limited. The authors should further build on their findings to better alleviate hallucinations.
* The study is limited to three models in the 7B-9B range, raising questions about generalizability to larger models and commercial LLMs, e.g. gpt-series and claude.
* While the synthetic setups (Bad-shots and Alice-Bob) are innovative, real-world scenarios might involve more complex interactions, which are not captured.
* The focus on extreme knowledge scenarios (high- and low-knowledge) omits middle-ground cases, which could also be important.