FactCheckmate: Preemptively Detecting and Mitigating Hallucinations in LMs

## Abstract

Abstract Language models (LMs) hallucinate.
We inquire:
Can we detect and mitigate hallucinations before they happen?
This work answers this research question in the positive, by showing that the internal representations of LMs provide rich signals that can be used for this purpose.
We introduce FactCheckmate , which preemptively detects hallucinations by learning a classifier that predicts whether the LM will hallucinate, based on the model’s hidden states produced over the inputs, before decoding begins.
If a hallucination is detected, FactCheckmate then intervenes, by adjusting the LM’s hidden states such that the model will produce more factual outputs. FactCheckmate provides fresh insights that the inner workings of LMs can be revealed by their hidden states.
Practically, both the detection and mitigation models in FactCheckmate are lightweight, adding little inference overhead; FactCheckmate proves a more efficient approach for mitigating hallucinations compared to many post-hoc alternatives.
We evaluate FactCheckmate over LMs of different scales and model families (including Llama, Mistral, and Gemma),
across a variety of QA datasets from different domains.
Our results demonstrate the effectiveness of leveraging internal representations for early hallucination detection and mitigation, achieving over 70% preemptive detection accuracy. On average, outputs generated by LMs with intervention are 34.4% more factual compared to those without intervention.
The average overhead difference in the inference time introduced by FactCheckmate is around 3.16 seconds.

## 1 Introduction

Language models (LMs) hallucinate, a phenomenon where they produce nonfactual or even misleading outputs that often appear plausible (Ji et al., ; Bang et al., ; Xu et al., ; Zhang et al., ; Li et al., ; Huang et al., ; Ye et al., ).
Extensive efforts have been devoted to mitigating their hallucination issues (Rawte et al., ; Zhou et al., ).
These approaches are mostly *reactive*, addressing hallucinations after they occur, and
often require resampling new outputs (Li et al., ; Manakul et al., ), substantially increasing the inference overhead.
In addition, these approaches treat the LM as a black box, while relying on external LMs for detecting hallucinations,
missing the opportunity to gain deeper insights into the internal workings of these models.

Recent findings by Azaria & Mitchell () and Burns et al. () show that probing the LMs’ representaions can effectively determine the factuality of their outputs.
Marks & Tegmark () observe that the hidden states produced by the middle layers of LMs over complete statements exhibit linear separability in binary factuality classification tasks.
However, these studies have a relatively narrow focus, primarily addressing hallucination detection in a *reactive* manner. A more thorough investigation is needed.

The key hypothesis of this paper is that, the LMs’ hidden states reveals valuable information about their internal working mechanisms, and provide signals that can be used to predict whether it is likely to hallucinate *before* it happens.
More formally, we propose FactCheckmate to answer the following research question (RQ):
*Can we preemptively predict and mitigate hallucinations with LMs’ internal representations?*
FactCheckmate learns a classifier that, taking the models’ hidden states over the inputs, predicts whether the model *is about to* hallucinate.
If a hallucination is detected, FactCheckmate intervenes, by adjusting the LM’s hidden states with a learned invervention model, and steering them towards producing more factual outputs (Figure [1](#S1.F1)).

Our experiments answer the RQ in the positive.
We evaluate FactCheckmate across three QA datasets from domains: NQ-open (Wikipedia; Lee et al., ), MMLU (STEM exam; Hendrycks et al., ), and MedMCQA (medical; Pal et al., ).
For all, FactCheckmate successfully predicts whether or not the LMs will hallucinate over 70% of the time, significantly outperforming a 50% random baseline.
This is achieved when the LMs have only seen the input questions and before decoding starts.
We observe consistent trends across LMs of different scales and familities, including Llama2 (7B and 13B; Touvron et al., ), Llama3/3.1-8B (Dubey et al., ) Mistral-7B (Jiang et al., ), and Gemma-7B (Team et al., ).
Furthermore, FactCheckmate’s intervention model can effectively improve the LMs’ outputs.
Using GPT-4o as a judge, which shows high agreement with human evaluations in our experiments, we find that on average, outputs generated by LMs with intervention are 34.4% more factual than those produced without intervention.
We also calculate the overhead in the inference time introduced by FactCheckmate, with an average increase of approximately 3.16 seconds, showing minimal impact on inference performance.

FactCheckmate reveals surprising insights into existing LMs, and can potentially lead to more profound understanding of their internal working. All code, data, and checkpoints for reproducing our findings will be released.

We start by presenting the FactCheckmate’s hallucination detection model and results in §[2](#S2),
followed by the intervention model and results in §[3](#S3).
Additional experiments and analysis are presented in §[4](#S4).

Figure: Figure 1: FactCheckmate Pipeline. A demonstration of how preemptive detection and subsequent mitigation work. As shown, at a layer $l$, the hidden states of only the prefix are aggregated and passed to the classifier $f_{\theta}$. Once hallucination is detected with classification probability $<\alpha$, $\mathbf{g}_{\phi}$ intervenes and adjusts the last token $\mathbf{h}^{(l)}_{N}$. This leads to a more factual output than before.
Refer to caption: /html/2410.02899/assets/figures/multicolor-diag.png

## 2 FactCheckmate Hallucination Detection

This section focuses on FactCheckmate’s preemptive hallucination classifier (§[2.1](#S2.SS1)) and experimental results (§[2.2](#S2.SS2)).

### 2.1 Preemptive Hallucination Detection with a Lightweight Classifier over Hidden States

##### Classifier.

FactCheckmate learns a binary classifier $f_{\theta}$ to preemptively detect hallucinations.
Parameterized by a learned two-layer $\operatorname{ReLU}$-MLP followed by a sigmoid function, $f_{\theta}$ takes as input the LM’s hidden states and outputs the probability that the LM *will* hallucinate.
More specifically, let $\{{\bm{h}}^{(l)}_{i}\}_{i=1}^{N}$ be a sequence of $N$ hidden states that the LM produces over the input of length $N$.
A $d$-dimensional vector ${\bm{h}}^{(l)}_{i}$ denotes the output of the feedforward network (FFN) of the $l$-th transformer layer, at the $i$-th token.
The classifier $f_{\theta}$ takes as input the average over $\{{\bm{h}}^{(l)}_{i}\}$ and produces a scalar between 0 and 1 indicating the probability that the LM will hallucinate in its response to the input:

$$ $\displaystyle f_{\theta}\Big{(}\{{\bm{h}}^{(l)}_{i}\}_{i=1}^{N}\Big{)}=\sigma\Bigg{(}\operatorname{ReLU-MLP}\Big{(}\frac{1}{N}\sum_{i=1}^{N}{\bm{h}}^{(l)}_{i}\Big{)}\Bigg{)}$ (1) $$

$l$ is empirically determined based on validation performance, and can vary by the LMs and datasets.
In general, $l$ tends to be the middle layers. More details about the best empirical layer for each LM can be found in Appendix [B.1](#A2.SS1)

We train a separate classifier tailored to each LM.(^1^11 Our preliminary experiments show that the hallucination classifier underperforms when applied to hidden states produced by a model different than that it is trained for.)
We consider LMs from different families of different scales, including Llama2 (7B and 13B; Touvron et al., ), Llama3/3.1-8B (Dubey et al., ) Mistral-7B (Jiang et al., ), and Gemma-7B (Team et al., ).

##### Data collection.

In order to train $f_{\theta}$, we need to collect a LM’s hidden states over the inputs, and the corresponding binary label indicating whether the LM will produce factual outputs.
We construct the training data on three datasets from different domains: NQ-open (Wikipedia; Lee et al., ), MMLU (STEM; Hendrycks et al., ), and MedMCQA (medical entrance exam; Pal et al., ).
NQ-open is a QA dataset and contains question and answer pairs.
MMLU and MedMCQA are multiple choice datasets, pairing each question with multiple options.
We convert MMLU and MedMCQA into a QA dataset by pairing each input question with the gold answer.

To collect the training data for LM $M$, we prompt $M$ with
few-shot demonstrations followed by a question, and then collect its hidden states over the inputs.
$M$’s output answers are checked against gold ones with the exact match (EM), following standard practice (Gao et al., ).
If the model’s output is wrong, we consider its associated hidden states *will* lead to a hallucination, and vice versa.
After producing hidden state and label pairs, we subsample the data to obtain balanced training data containing roughly the same amount of positive (will not hallucinate) and negative (will hallucinate) pairs.
In order to compare across different LMs, we create a shared test split across all LMs. Each LM have different training/validation splits.
Table [1](#S2.T1) summarizes the statistics of the datasets.

$f_{\theta}$ is trained with a cross-entropy loss on hidden state and label pairs.
Early stopping based on the validation accuracy is used.

**Table 1: Dataset splits and sizes for training the hallucination classifier $f_{\theta}$ over the LMs’ hidden states (§[2.1](#S2.SS1)).**
| Dataset | Total Size | Train (70%) | Validation (15%) | Test (15%) |
| --- | --- | --- | --- | --- |
| NQ-Open (Lee et al., ) | 12,000 | 8,400 | 1,800 | 1,800 |
| MMLU (Hendrycks et al., ) | 3,182 | 2,228 | 477 | 477 |
| MedMCQA (Pal et al., ) | 3,953 | 2,767 | 593 | 593 |

### 2.2 Results

Figure: Figure 2: An illustration of different settings used in the experiment. (Input + Output) are the hidden states of both the input and output. The subsequent hidden states in the experiment are taken by using only the input or by dropping the last $n$ tokens from the prefix.
Refer to caption: /html/2410.02899/assets/figures/input_illus.png

Table [2](#S2.T2) shows the hallucination detection test accuracy results.
We evaluate the hallucination detection performance using the same classifier $f_{\theta}$ on different inputs.
I indicates our *preemptive* classifier, that takes the LMs’ hidden states produced over the input questions only.
I+O indicates a *reactive* baseline, which sees the hidden states produced over a concatenation of the input questions and the LMs’ output answers.
It is, therefore, expected that I+O achieves better performance,
as it has access to more information.

Throughout all I settings across all LMs and datasets, $f_{\theta}$ achieves well above the 50% random guess baselines.
This confirms that LMs’ hidden states provide useful signal for predicting their hallucinations preemptively.

We further make the task more challenging for $f_{\theta}$, by feeding it with a prefix of the input questions.
The results are summarized in the Prefix columns.
Here, $-n$ indicates that $f_{\theta}$ sees the hidden states produced over a prefix not including the last $n$ tokens. Illustration of the different input settings for hallucination classification is shown in Figure [2](#S2.F2).

In some cases, using a prefix of $-n$ underperforms I, while for others their performance is comparable.
These results suggest that $f_{\theta}$ can often predict whether the LM is likely to hallucinate before it even finishes processing the input questions.

On MMLU, Mistral-7B behaves differently than others, and we are not able to produce a sufficiently large test split that is shared between it and others. Therefore, these results are excluded.

Table: Table 2: Hallucination detection test accuracy. I+O indicates a “reactive” baseline that classifies the LMs’ hidden states produced over both input questions and output answers, while I preemptively classifies hallucinations based on the hidden states over only the inputs. A prefix of $-n$ indicates that the classifier only sees a prefix of the input dropping the last $n$ tokens.

## 3 FactCheckmate Preemptive Hallucination Mitigation

This section focuses on using FactCheckmate to preemptively mitigate hallucinations,
including its intervention model (§[3.1](#S3.SS1)) and the experimental results (§[3.2](#S3.SS2)).

### 3.1 Training an intervention model

When $f_{\theta}$ detects that LM $M$ is about to hallucinate, FactCheckmate relies on an intervention model $\mathbf{g}_{\phi}$ to mitigate hallucinations preemptively.
Conditioning on ${\bm{h}}_{N}^{(l)}$, the LM’s last hidden state over the input, $\mathbf{g}_{\phi}$ generates a $d$-dimensional vector and adds it to ${\bm{h}}_{N}^{(l)}$, before the LM generates any output.

$$ $\widetilde{{\bm{h}}}_{N}^{(l)}={\bm{h}}_{N}^{(l)}+{\bm{g}}_{\phi}\Big{(}{\bm{h}}_{N}^{(l)}\Big{)}$ (2) $$

$\widetilde{{\bm{h}}}_{N}^{(l)}$ is then used in place of ${\bm{h}}_{N}^{(l)}$ for onward LM decoding.
The intervention is applied at the last hidden state of the input, as it aligns with the natural progression of decoding and targets the point where hallucinations are most likely to arise.

Intuitively, $\mathbf{g}_{\phi}$ is supposed to steer the LM’s hidden state towards a “target hidden state” ${\bm{h}}^{*(l)}$,
which is more likely to lead to a factual output.
When the LM answers the question correctly, no further modification is needed and ${\bm{h}}^{*(l)}={\bm{h}}_{N}^{(l)}$.
When the model answers the question incorrectly, we set the ${\bm{h}}^{*(l)}$ to the model’s final hidden state *over the input prompt followed by the gold answer.*
These target hidden states are paired with their corresponding inputs ${\bm{h}}_{N}^{(l)}$ to train $\mathbf{g}_{\phi}$.
We explore both a deterministic and a stochastic $\mathbf{g}_{\phi}$:

- •
The deterministic $\mathbf{g}_{\phi}$ is a three-layer ReLU-MLP. It trains by minimizing the mean squared error (MSE) between the adjusted hidden state $\widetilde{{\bm{h}}}_{N}^{(l)}$ and the target one ${\bm{h}}^{*(l)}$.
- •
The stochastic $\mathbf{g}_{\phi}$ treats the adjustment vector as a random variable of multivariate Gaussian.
It applies a reparameterization trick: ${\bm{g}}_{\phi}({\bm{h}}_{N}^{(l)})=\bm{\mu}({\bm{h}}_{N}^{(l)})+\bm{\epsilon}\odot\bm{\sigma}({\bm{h}}_{N}^{(l)})$ for training.
Two three-layer ReLU-MLPs are used to for $\bm{\mu}$ and $\bm{\sigma}$, with the first two layer shared.
Its training objective remains the same MSE loss.
One benefit of the stochastic $\mathbf{g}_{\phi}$ is allowing for sampling the adjustment vectors during inference, which we explore in the experiments.

### 3.2 Results

Figure [3](#S3.F3) summarizes the performance of FactCheckmate’s intervention model on on the NQ-open dataset, including both the deterministic and stochastic variants.
All LMs use the greedy decoding.
Following recent works (Raju et al., ; Chen et al., ), we employ GPT-4o (OpenAI et al., ) as the evaluator to assess for factuality.
Human evaluation performed by the authors indicate that there is a substantial agreement between GPT-4o and human judgement, with a Cohen’s Kappa of 0.6, justifying our choice of using GPT-4o as an automatic evaluation metric.
The specific prompt is provided in Appendix [A](#A1).

For the stochastic $\mathbf{g}_{\phi}$, we sample 1, 10, 20, and 30 different $\bm{\epsilon}$, and apply the interventions with $\mathbf{g}_{\phi}$;
we then use $f_{\theta}$ to select the intervened hidden state that leads to the highest probability by $f_{\theta}$,
which is then used for onward decoding.(^2^22A higher probability by $f_{\theta}$ indicates the hidden state is more likely to lead to a factual output.)

We apply the adjustment only to the first decoding step,
modifying ${{\bm{h}}}_{N}^{(l)}$ to $\widetilde{{\bm{h}}}_{N}^{(l)}$ when the classifier’s confidence $\alpha$ is less than or equal to $0.3$.
As shown in Figure [3](#S3.F3), the intervened LMs consistently outperform the base LMs, with a higher proportion of wins favoring the adjusted outputs, with the results varying depending on LM’s architecture. The deterministic intervention consistently achieves a win rate of at least 60% in all cases, while without interventions (Base), the LMs show significantly lower performance, with wins as low as 34%. On average, the winning rate of LMs with intervention across all models is 34.4% higher than that of the base LMs.

The results demonstrate that both deterministic and stochastic intervention models improve the factuality of LM’s outputs. These finding suggest that, we can mitigate the hallucination even before it shows up in the generation of the language model.

Figure: (a) Deterministic $\mathbf{g}_{\phi}$
Refer to caption: /html/2410.02899/assets/figures/mse_updated.png

We further compare to Duan et al. (),
a baseline that applies a Principal Component Analysis (PCA) based approach to engineer the hidden states to mitigate hallucinations in LMs, we refer to this baseline as *PCA*.
Figure [4](#S3.F4) compares the baseline methods, including *PCA* and a sampling-based decoding approach, which utilizes the hallucination classifier component of FactCheckmate (referred to as Sample-FactCheckmate-CLS).

For *PCA*, adjustments are applied at every generation step, following Duan et al. () with greedy decoding and evaluated against greedy-based decoding version of the non-intervened LM, this to eliminate any effects that sampling might cause.
Sample-FactCheckmate-CLS is evaluated against sample-based decoding version of the non-intervened LM, with the same random seed maintained for consistent comparison.
As shown, for most models, both baselines result in lower intervened win rates and constantly higher base wins, compared to FactCheckmate in Figure [3](#S3.F3).

Figure: (a) PCA
Refer to caption: /html/2410.02899/assets/figures/pca_updated.png

## 4 Additional Experiments

In the following section, we first evaluate the inference time overhead (§[4.1](#S4.SS1)).
Next, we conduct classification experiments to analyze performance across different modes of aggregation (§[4.3](#S4.SS3)) and investigate the role of word embedding layers (§[4](#S4.T4)).

### 4.1 Evaluating FactCheckmate Time Overheads

Both $f_{\theta}$ and $\mathbf{g}_{\phi}$ are lightweight and should incur minimal inference overhead.
We confirm this across three models: Llama-2-7B, Llama-3-8B, and Llama-3.1-8B. For each model, the average inference time was measured both with and without FactCheckmate over three runs, each processing 400 few-shot prompts.

**Table 3: Comparison of LMs inference time overheads over three runs per LM. The average difference in inference time is approximately 3.16 seconds, showing minimal impact on inference performance.**
| LM | LM Intervention State | Average Inference Time (s) | $\Delta$ (s) |
| --- | --- | --- | --- |
| Llama-2-7B | Base | 235.69 | - |
| Llama-2-7B | FactCheckmate | 237.91 | +2.22 |
| Llama3-8B | Base | 272.02 | - |
| Llama3-8B | FactCheckmate | 276.67 | +4.65 |
| Llama3.1-8B | Base | 272.84 | - |
| Llama3.1-8B | FactCheckmate | 275.46 | +2.62 |

The results are summarized in Table [3](#S4.T3).
The table shows that FactCheckmate introduces a negligible overhead to the inference process, preserving performance close to that of the baseline. We see that the result is consistent over models. This negligible overhead is a promising factor for scaling the experiments or integrating it into the existing LMs’ pipelines.

### 4.2 f θ subscript 𝑓 𝜃 f_{\theta} Classifies the Hidden States Rather than the Questions

Table: Table 4: Results for the word embedding layer of Llama-2-7b on MedMCQA dataset. (§[4](#S4.T4)). The figure shows classification accuracy of approximately 50%, indicating no influence of the question difficulty or type on the preemptive hallucination results shown in Table [2](#S2.T2).

One possible explanation for $f_{\theta}$’s strong preemptive hallucination detection performance is that it might be classifying the input questions rather than the LMs’ hidden states. It is true that more difficult questions could lead to a higher chances of hallucinations by the LMs. However, our results indicate that it is the LMs’ hidden states, rather than the questions themselves, that drive the success of $f_{\theta}$.

Table [4](#S4.T4) summarizes the test accuracies for an $f_{\theta}$ trained and tested on the word embedding layer of Llama-2-7B, before any contextualization by the LM.
Across the board, the accuracies are close to 50% random guess.
This confirms that, the model is not skewed towards favoring a certain type of question over another while doing the classification. The difficulty of the question is hence, not a contributing factor to the accuracy calculated by classifying the hidden states.

Table: Table 5: Comparison of hallucination classification across different modes of aggregation for the same layer and LM. Here we show the results for the Llama3-8B on layer 15. We see that the difference between I+O and I is the least when the mean is the mode of aggregation.

### 4.3 Preemptive hallucination detection across various modes of aggregation

We explore three modes for aggregating the hidden states before passing them to the classifier: mean pooling, max pooling, and taking the last token. We see that the mean pooling gives us the best accuracy as shown in Fig [5(a)](#A2.F5.sf1). To test how different modes of aggregation work for the preemptive experiments, we compare all the three modes. This is done across the same layer for a the same model. As shown in Table [5](#S4.T5), we see that the accuracy of the entire sentence (I+O) is similar for last token and mean pooling. However, the drop in the subsequent accuracies is the maximum when last token is used. The maximum accuracy for I is when mean pooling is used. Therefore, we use mean pooling as our mode of aggregation in all our experiments.

## 5 Related Work

Definitions.
In this work, we investigate the phenomenon of hallucinations in language models that generate responses based solely on their parametric knowledge, similar to  Azaria & Mitchell (). This contrasts with in-context generation scenarios where external knowledge sources are explicitly incorporated within the prompt. We adopt the refined taxonomy proposed by  Huang et al. (), categorizing hallucinations into Factuality or Faithfulness. FactCheckmate focuses its study on addressing Factuality hallucinations, which are further divided into factual inconsistencies and factual fabrications.

Hallucination Detection.
Hallucination remains a significant issue undermining the language model’s usefulness.
Existing research on hallucination detection has primarily focused on post-processing methods applied after the inference process is completed and often utilizing external knowledge sources for verification, as in  (Manakul et al., ; Li et al., ; Chern et al., ). For instance,  Gou et al. () introduce CRITIC, a framework that validates model outputs through tool interaction, and FACTSCORE proposed by  Min et al. (), is a fine-grained factual accuracy metric that breaks down generated content into atomic facts, assessing their accuracy by comparing them against reliable sources.

A recent promising line of research leverages the internal mechanics of language models to detect hallucinations. Works such as  (Burns et al., ; Azaria & Mitchell, ; Marks & Tegmark, ) are pioneering efforts to assess the truthfulness of outputs by examining the hidden states of language models. The work by Meng et al. () locates where factual associations are stored in GPT models. These studies have spurred further research into using LLMs’ internal representations in hallucination detection  (Chen et al., ; CH-Wang et al., ). For instance, the MIND framework, introduced by Su et al. (), generates training data in unsupervised approach for training hidden states based hallucination detectors. Duan et al. () conducts an experimental examination of the hidden states of LLMs when processing factual versus nonfactual responses. Following this line of research, FactCheckmate showcases the effectiveness of preemptive hallucination detection, i.e. identifying warning signals several tokens before the hallucinations actually occur, via solely exploiting the language model’s hidden states.

Hallucination Mitigation.
In the realm of hallucination mitigation at inference time, existing work has explored self-correction and automated feedback approaches, where the language model is prompted to fix its generation flaws, with or without leveraging feedback from the model itself or some external knowledge source, as detailed in (Pan et al., ; Dhuliawala et al., ; Ji et al., ).
A recent approach to mitigating hallucinations involves utilizing activation engineering (Subramani et al., ), first applied to hallucination mitigation by Duan et al. ().
FactCheckmate builds on these findings and explores additional activation engineering techniques to intervene and mitigate hallucinations during inference time. FactCheckmate is also related to inference-time approaches that utilize a scoring function to steer the LM toward desired behavior (Dathathri et al., ; Yang & Klein, ; Khalifa et al., ).

## 6 Conclusion

In conclusion, FactCheckmate demonstrates that the hidden states of language models encode rich information that can be used to predict hallucination preemptively, even before they appear in the generated output.
In FactCheckmate, leveraging this insight, we develop an intervention mechanism that steers the LM’s generation towards more factual outputs, once the hallucination is detected.
We achieve a preemptive hallucination detection accuracy of more than 70%, and an average of 34.4% more factual output by LMs supported by FactCheckmate, compared to the base LMs.
FactCheckmate empirically proves the significant potential of utilizing the internal working of LMs, through learning lightweight models for hallucination detection and mitigation, introducing only a negligible average overhead of 3.16 seconds to the inference time.

## 7 Limitations and Future work

We have only looked at the hidden states as an internal component for classification to predict the factuality of a sentence.
Exploring other LM’s internal components presents a potential direction for future work.
With our detection and intervention experiments, we see that different layers in the LM have varying effects.
The classifier $f_{\theta}$ and the intervention model $\mathbf{g}_{\phi}$, are sensitive to the hyperparameters selected.
To solve this, we want find out a more robust and consistent approach that is less sensitive to varying hyperparameters, to steer the generation of the model towards the truth.

Going ahead, we aim to build a pipeline that is more generalizable and is applicable to a variety of domains.
Expanding our evaluation to include diverse datasets with different distributions could provide valuable insights and a potential future direction for improving model generalizability.
FactCheckmate has shown promising results in question-answering tasks, and it would be interesting to extend its application to other tasks, including dialogue-based and long-form generation tasks.

#### Acknowledgments

This research used the Delta advanced computing and data resource which is supported by the National Science Foundation (award OAC 2005572) and the State of Illinois. Delta is a joint effort of the University of Illinois Urbana-Champaign and its National Center for Supercomputing Applications. Also, this research was supported in part by the Illinois Computes project which is supported by the University of Illinois Urbana-Champaign and the University of Illinois System.

## Appendix A Factual Assessment Prompt

To assess factual accuracy, we use GPT-4o (OpenAI et al., ) as the evaluator. To reduce stochasticity in the prompting process, we set the temperature to $1\times 10^{-14}$ and top_p to $1\times 10^{-17}$. The prompt used for evaluation is as follows:

## Appendix B Experiments for classification

### B.1 Hidden Representation Classification Analysis

Given the datasets and models described above, for every layer in a model we train a corresponding classifier on hidden states of that respective layer. We use three
modes for aggregating the hidden states before passing them to the classifier: mean pooling, max
pooling and taking the last token in the hidden states. Figure [5(b)](#A2.F5.sf2) illustrates the accuracy of hallucination detection of the classifiers for the entire sequence, using the mean token representation for aggregation. As shown, the accuracy across all evaluated models mostly exceeds 0.75, indicating a robust capability to identify hallucinations. This high level of performance underscores the efficacy of the hidden state representations in distinguishing factual accuracies within generated content. As seen in the figure, we also see that the accuracy peaks for the middle layers. The best performing layer per model per dataset is shown in Table [6](#A2.T6)
ur experiments also explore taking the elementwise max over hidden states, or taking the last one as the input to $f_{\theta}$,
and find they slightly underperform taking the average.

Therefore, for all models we calculate the test accuracy across all layers and all modes of aggregation. Quantitative results are shown in the first column of Table [2](#S2.T2). Given the three modes of aggregation, we see that mean pooling gives the best results in most cases. Figure [5(a)](#A2.F5.sf1) shows the test accuracy per layer per mode.

Setup: The classifier is trained using an Adam optimizer with a learning rate of $10^{-4}$ with a dropout rate of 0.1. We train all classifiers for 50 epochs.

Figure: (a) Test Accuracy by Layer for all modes for the model Mistral-7b
Refer to caption: /html/2410.02899/assets/figures/cls_plots/mistral_7b_v0.3_test_acc_by_layer.png

**Table 6: Best Performing layer per model and dataset**
| LM | NQ | MMLU | MedMCQA |
| --- | --- | --- | --- |
| Llama-2-7b-hf | 14 | 16 | 14 |
| Llama-2-13b-hf | 22 | 15 | 14 |
| Llama-3-8B | 15 | 17 | 11 |
| Llama3.1-8B | 23 | 14 | 15 |
| Mistral-7B | 13 | - | 12 |
| Gemma-7B | 17 | 17 | 18 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: 1. Both the detection and mitigation models are lightweight, resulting in minimal inference overhead, which is advantageous for practical applications.
2. The approach is evaluated across various large-scale models, including Llama, Mistral, and Gemma, demonstrating its broad applicability.
summary: This paper introduces a novel approach that predicts and mitigates hallucinations during the generation process by learning the internal representations of language models. This method is forward-looking in that it seeks to intervene before hallucinations occur.
weaknesses: 1.	The paper lacks an analysis of the generalizability of the learned classification network and intervention model. Specifically, it is unclear whether the trained classification and intervention models are generalizable across different large models and tasks. Given that the data collection for training was based on only three tasks, questions remain regarding the generalizability to other tasks. Is a new training dataset needed for additional tasks, or does the current model extend effectively?
2.	The dataset construction raises some issues or lacks clarity. For certain tasks, it may be straightforward to judge whether the generated output is correct. However, in the case of generative tasks—particularly when the output is lengthy—it becomes challenging to determine whether the output from the large model is accurate, and thereby to ascertain whether the label indicates hallucination. This aspect is not thoroughly addressed in the paper.
3.	For different large models, it is necessary to reconstruct training datasets and train distinct classifiers and intervention networks, making this process relatively complex. Although it may not increase inference time, the time required for dataset construction and model training is also significant and should not be overlooked.
4.	There is a lack of analysis regarding the structure of the classifier and intervention models used. Specifically, the classifier is implemented as a two-layer MLP, and the perturbation model as a three-layer MLP. Details such as the hidden dimensions of these MLPs, and the potential performance impact of adding or reducing layers, are not discussed. Moreover, it is unclear how alternative models, such as transformers, might affect performance.
5.	The paper does not provide specific details on the training setup for the classifier and intervention models, which creates challenges for reproducibility. In my view, training the classification network and intervention model should be central to this method, but there is limited discussion provided.
6.	The experiments presented are insufficient to substantiate the effectiveness of the proposed method. The experimental section primarily compares the base model, but numerous methods already exist for hallucination detection and mitigation in large models, such as PoLLMgraph, SELFCHECKGPT, and TruthX. These more advanced baselines are not included in the comparisons. Additionally, the number of datasets used appears limited, potentially insufficient to demonstrate broad effectiveness across various tasks and datasets. I recommend conducting experiments on a wider range of datasets to strengthen the validation.
7.	Clearly, this method, which relies on internal states, cannot be applied to black-box large models like GPT. This point should be included in the limitations section of the paper.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: - Can the authors elaborate on how hyperparameter sensitivity impacts the intervention model’s reliability?
- Is FACTCHECKMATE adaptable to more complex generative tasks, like dialogue or long-form generation?
soundness: 2
strengths: - Extensive experimental results across different models and datasets are robust and demonstrate effectiveness.
- Offers practical implications for real-world applications where factual accuracy is crucial, enhancing LM reliability.
summary: This paper introduces FACTCHECKMATE, a framework aimed at preemptively detecting and mitigating hallucinations in language models (LMs) by analyzing their hidden states before generating responses. The approach leverages lightweight classifiers to detect hallucinations early and an intervention model to adjust hidden states for improved factual accuracy.
weaknesses: - Writing needs improvement, including but not limited to the abstract and introduction
- Typos, e.g. Line 44 "representaions"
- determining the factuality through merely probing the LMs' representations is not novel as a methodology
- Limited exploration of other LM components beyond hidden states.
- Generalizability of results is uncertain for tasks beyond QA.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: **Question 1** Could the authors test the method on abstractive summarization tasks to demonstrate whether it performs well in open-book settings? This would help validate the method’s applicability across different types of hallucinations.

**Question 2** Could the authors verify the proposed method’s generality by evaluating its performance on dataset other than NQ-open? Understanding the method’s effectiveness across diverse datasets is essential for demonstrating its robustness.

**Question 3** Could the authors apply the proposed method to benchmarks like Alpaca-eval? It would be interesting to see how the intervention affects the model’s performance on nominal questions and whether there is any degradation in accuracy due to false positives.
soundness: 3
strengths: **Strength 1** The paper introduces a new approach to detecting hallucinations by leveraging the internal representations of LMs.

**Strength 2** The experimental design is solid, and the results effectively demonstrate the effectiveness of the proposed method.

**Strength 3** The paper is well-written and easy to follow, with clear explanations of the methodology and results.
summary: This paper explores the possibility of preemptively detecting and mitigating hallucinations in language models (LMs). The authors present FACTCHECKMATE, a method that learns to predict whether an LM will hallucinate based on the model's hidden states before decoding begins. If a hallucination is predicted, FACTCHECKMATE adjusts the LM's hidden states to produce more factual outputs. The method leverages the rich signals provided by the internal representations of LMs. Experimental results demonstrate that FACTCHECKMATE achieves over 70% preemptive detection accuracy and increases the factualness of LM outputs by an average of 34.4% compared to no intervention. The inference time overhead introduced by FACTCHECKMATE is approximately 3.16 seconds.
weaknesses: **Weakness 1** The paper focuses solely on close-book hallucinations, whereas many hallucinations occur in open-book settings, such as in abstractive summarization. Evaluating the method's effectiveness in handling open-book hallucinations would provide a more comprehensive understanding of its capabilities.

**Weakness 2** The evaluation of the proposed method's factuality is conducted on the NQ-open dataset, and the classifier used is also trained on the same dataset. It remains unclear whether the method can generalize to other datasets, which is crucial for demonstrating the robustness of the approach.

**Weakness 3**: There is no discussion regarding the potential impact of the proposed method on nominal (non-hallucinatory) questions. Since the classifier might have a false positive rate (FPR), it is important to understand how the intervention affects the performance on questions that do not contain hallucinations.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: See the weakness.
soundness: 2
strengths: Overall, the presentation is well-written and easy to follow.
summary: This paper presents an approach to detect and mitigate hallucinations in language models (LMs) before they occur. The authors introduce a system named FactCheckmate, which leverages the hidden states of LMs to predict potential hallucinations. This is achieved through a classifier that assesses whether the LM is likely to produce a hallucination based on the internal signals from its hidden states. When a hallucination is detected, FactCheckmate intervenes by adjusting the hidden states to steer the model towards generating more factual content.
weaknesses: 1. Although many methods for detecting and mitigating LLM hallucinations are outlined in the related work, the authors compare their approach with only one method. To convincingly demonstrate the superiority of their method, it would be prudent to include 3-4 baselines for both detection and mitigation aspects. Without this broader comparison, I cannot recognize the advantages of the authors' approach.

2. I appreciate the experiments conducted on different open-source model families, but there is limited information on the performance of the method on Llama2 Chat and Llama3 Instruct models. Furthermore, it's unclear how the method performs on larger models like Llama2 70B and Llama3 70B. This raises questions about the scalability and generalizability of the proposed approach across various model sizes.

3. While the authors compare their method, FactCheckMate, under random sampling conditions, its effectiveness significantly diminishes from previous levels above 60% to now below 50%. This indicates that FactCheckMate may not be as robust under varied sampling conditions.

4. The claim of a 3.16-second average time in the abstract lacks rigor. Details about the GPU and CPU environments where these measurements were taken are not provided. Additionally, the use of 400 few-shot prompts does not offer a comprehensive view of performance. It would be beneficial to see how the method performs under long-context scenarios to better understand its effectiveness.

5. Regarding the training of an intervention model, I can't find which dataset was used or discuss the hyperparameters involved in detail. More thorough discussion and transparency about the training conditions and parameters would enhance the credibility and reproducibility of the research.