ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability

## Abstract

Abstract Retrieval-Augmented Generation (RAG) models are designed to incorporate external knowledge, reducing hallucinations caused by insufficient parametric (internal) knowledge. However, even with accurate and relevant retrieved content, RAG models can still produce hallucinations by generating outputs that conflict with the retrieved information. Detecting such hallucinations requires disentangling how Large Language Models (LLMs) utilize external and parametric knowledge. Current detection methods often focus on one of these mechanisms or without decoupling their intertwined effects, making accurate detection difficult. In this paper, we investigate the internal mechanisms behind hallucinations in RAG scenarios. We discover hallucinations occur when the Knowledge FFNs in LLMs overemphasize parametric knowledge in the residual stream, while Copying Heads fail to effectively retain or integrate external knowledge from retrieved content. Based on these findings, we propose ReDeEP , a novel method that detects hallucinations by decoupling LLM’s utilization of external context and parametric knowledge. Our experiments show that ReDeEP significantly improves RAG hallucination detection accuracy. Additionally, we introduce AARF, which mitigates hallucinations by modulating the contributions of Knowledge FFNs and Copying Heads.

## 1 Introduction

Figure: Figure 1: Two examples of RAG where the retrieved document is correct but conflicts with parametric knowledge. The left example shows a correct response based on external knowledge, while the right example demonstrates hallucination despite accurate external context.
Refer to caption: /html/2410.11414/assets/x1.png

LLMs have made significant advancements in natural language processing tasks (Dubey et al., ; Achiam et al., ). However, they still face challenges with hallucinations, often generating factually inaccurate outputs (Huang et al., ). To mitigate this issue, many researchers have introduced Retrieval-Augmented Generation (RAG) models, which aim to improve the accuracy of LLM responses by incorporating relevant information retrieved from external knowledge bases (Shuster et al., ; Gao et al., ).

Despite the use of accurate and relevant retrieved context, RAG models may still produce statements that are either unsupported or contradict the retrieved information, a phenomenon we term RAG Hallucination (Niu et al., ; Magesh et al., ). Recent studies have examined the potential conflicts between the external context and the LLM’s parametric knowledge in RAG models (Xu et al., ). As shown in [Figure 1](#S1.F1), these conflicts can lead to hallucinations but do not always cause them. Therefore, it is important to distinguish RAG hallucination from Knowledge Conflict as a new research direction. *Our work focuses on detecting RAG hallucinations, specifically in cases where the retrieved external context is accurate and relevant.*

Figure: Figure 2: Causal perspectives on hallucination detection methods. (i): parametric knowledge is confounded by external context, (ii): external context is confounded by parametric knowledge, and (iii): mixes both without decoupling their contributions. (Ours): decouple these confounders using mechanistic interpretability, incorporating them as covariates to improve hallucination detection.
Refer to caption: /html/2410.11414/assets/x2.png

Existing hallucination detection methods can be categorized into three causal frameworks (Neuberg, ; Pearl, ), as illustrated in Figure [2](#S1.F2) (detailed introduction of methods see Appendix [H](#A8)): (i) Parametric Confounded by External: which relies on the LLM’s hidden states for hallucination detection, where the external context (E) serves as a confounder between parametric knowledge (P) and hallucinations (H). From a knowledge storage perspective (Geva et al., ), hidden states represent the result of querying the parametric knowledge (P) with external context (E), establishing a causal path from E to P (graph (a)). The presence of E as a confounder complicates the accurate prediction of hallucinations based on P alone (Chyzhyk et al., ). (ii) External Confounded by Parametric: which focuses on hallucination detection by leveraging external context and model responses. Here, parametric knowledge (P) is a confounder between the external context (E) and hallucinations (H), creating a causal link from P to E (graph (b)) due to the unavoidable presence of parametric knowledge in the response. (iii) Mixed Parametric and External: which combines both parametric and external knowledge directly, often using uncertainty or sampling techniques (e.g., token probability) to detect hallucinations (graph (c)). However, this mixing of E and P without decoupling their roles obscures their individual contributions (Bengio et al., ).

To address the challenges of hallucination detection in RAG models, we first leverage mechanistic interpretability (Ferrando et al., ; Elhage et al., ) to decouple the LLM’s utilization of parametric knowledge and external context. Specifically, we conduct an empirical study to explore the internal mechanisms behind hallucination generation in RAG scenarios. We introduce two metrics: the External Context Score, which uses attention heads to quantify the model’s utilization on external context, and the Parametric Knowledge Score, which is based on FFNs to evaluate LLM’s utilization of parametric knowledge (§ [3.1](#S3.SS1)). Correlation analysis and causal intervention reveals that hallucinations typically occur when Knowledge FFNs (from later LLM layers) over-add parametric knowledge into the residual stream, while Copying Heads (attention heads exhibiting copying behaviours) neglect the necessary external knowledge from retrieved content or LLM loses the information attended to by Copying Heads during the generation process (§ [3.2](#S3.SS2)).

Building on our causal analysis and mechanistic interpretability, we propose ReDeEP (Regressing Decoupled External context score and Parametric knowledge score) for detecting hallucinations in LLM-based RAG, which treat parametric knowledge (P) and external context (E) as covariates to solve the confounding problem (Kahlert et al., ) (see Ours in Figure [2](#S1.F2)). Additionally, we introduce AARF (Add Attention Reduce FFN), which mitigates hallucinations by modulating the contributions of Knowledge FFNs and Copying Heads in the residual stream (§ [4](#S4)). Experiments on RAGTruth and Dolly (AC) confirm that ReDeEP significantly outperforms existing detection methods, while AARF improves the truthfulness of LLaMA models (§ [5](#S5)).

## 2 Background and Related Works

### 2.1 Background

Our work is grounded in mechanistic interpretability (Ferrando et al., ; nostalgebraist, ; Meng et al., ; Elhage et al., ), which aims to explain how individual components of language models (LMs) contribute to predictions. In this study, we focus on transformer decoder-only architectures (GPT-like models) due to their widespread use (Achiam et al., ; Dubey et al., ). Transformers use residual connections, where each layer adds information from Attention Heads and Feed-Forward Networks (FFNs) to the hidden state via the residual stream, contributing to the final prediction (Elhage et al., ).

Attention Heads: Attention heads play a crucial role in contextualizing token representations by selectively attending to previous tokens and updating the residual stream (Ferrando & Voita, ; Clark et al., ; Wu et al., ). Notably, some attention heads, referred to as Copying Heads, have been shown to copy information from one token to another through their OV (output-value) circuits (Elhage et al., ). These heads can be identified by analyzing the positive eigenvalues of the OV matrix, which indicate copying behavior. Copying Heads contributes to preserving previously attended tokens in the residual stream, which is critical for external context utilization.

FFNs: FFN layers primarily function as knowledge storage in transformers (Geva et al., ). Each FFN layer transforms the hidden state by linearly combining key-value pairs, where keys encode specific knowledge and values represent the output of this knowledge. Research shows that FFNs are critical for the utilization of parametric knowledge within LLMs, enabling the model to retrieve and integrate stored information effectively for prediction (Dai et al., ).

Logit Lens: The $\operatorname{LogitLens}$ is a technique that decodes hidden states $\bm{x}^{l}$ directly into the vocabulary distribution using the LayerNorm and the unembedding matrix $\bm{W}_{U}$ of the LLM for interpretability (nostalgebraist, ):

$$ $\operatorname{LogitLens}\left(\bm{x}^{l}\right)=\operatorname{LayerNorm}(\bm{x}^{l})\bm{W}_{U}.$ (1) $$

By understanding the roles of Attention Heads (e.g., Copying Heads) and FFNs, we can better interpret the internal states of LLMs and identify the mechanisms behind hallucinations in RAG scenarios. Detailed background information can be found in Appendix [A](#A1).

### 2.2 Related Work

Hallucination of LLMs: LLMs often generate hallucinations—content inconsistent with real-world facts or inputs (Huang et al., ). As depicted in Figure [2](#S1.F2), although there has been extensive research on detecting hallucinations (Niu et al., ; Manakul et al., ; Han et al., ), few studies have concentrated on RAG hallucinations, particularly on the internal mechanisms driving these hallucinations. Inspired by research on Knowledge Conflicts (Xu et al., ), our work is the first to apply mechanistic interpretability to lens the internal mechanisms of RAG hallucinations from the perspectives of LLM’s utilization of external and parametric knowledge, leading to a more accurate detection method than previous approaches.

Mechanistic Interpretability: Mechanistic interpretability (Ferrando et al., ; Elhage et al., ) seeks to explain the internal processes of LLMs, enabling the interpretation of how individual model components contribute to the final prediction. Our work builds on insights into FFN layers, attention heads (Vaswani, ), residual streams (Elhage et al., ), and the logit lens (nostalgebraist, ) to analyze the internal mechanisms of LLMs when RAG hallucinations occur.

## 3 Empirical study

Our empirical study investigates how hallucinations in RAG models relate to the internal states of the LLM. Using Mechanistic Interpretability techniques, we focus on how the LLM’s use of external context and parametric knowledge contributes to hallucinations.

Figure: Figure 3: Expanded views of Unrolled LLMs’ Attention and FFN blocks. (a): The calculation process of the External Context Score and Parametric Knowledge Score. (b): Example of intervening on attention heads. (c): Example of intervening on FFN modules.
Refer to caption: /html/2410.11414/assets/x3.png

Experiment Setting:
We conduct experiments on the Llama2-7B-chat model (Touvron et al., ) using the training set of RAGTruth dataset (Niu et al., ), a high-quality, manually annotated dataset for RAG hallucinations (Details in Section [5.1](#S5.SS1)). Each data point in RAGTruth consists of a query $\mathbf{q}$, retrieved context $\mathbf{c}$, response $\mathbf{r}$, and the hallucination label $h$ (where 0 is truth and 1 is hallucination).
During generation, the input to the LLM $f$ is a sequence of tokens $\mathbf{t}=\left\langle t_{1},t_{2},\ldots,t_{n}\right\rangle$, including the query $\mathbf{q}=\left\langle t_{1},\ldots,t_{q}\right\rangle$, retrieved context $\mathbf{c}=\left\langle t_{q+1},\ldots,t_{c}\right\rangle$, and a partial generated response $\hat{\mathbf{r}}=\left\langle t_{c+1},\ldots,t_{n}\right\rangle$.

### 3.1 Metrics for LLMs’ Utilization of External Context and Parametric Knowledge

To quantify how LLMs use external context and parametric knowledge, we design two specific metrics, as shown in Figure [3](#S3.F3) (a):

External Context: Considering attention heads primarily function to retrieve relevant information (as discussed in Section [2.1](#S2.SS1)), we measure the LLM’s utilization of external context by assessing (1) whether attention heads focus on the correct context, and (2) whether the LLM effectively retains and utilizes this information during generation. To evaluate these aspects, we define the following metric based on the semantic difference between the external context attended by attention heads and the generated information:

For the last token $t_{n}$, the attention weights on the context are $\bm{a}^{l,h}_{n,q:c}$, where $\bm{a}_{n}^{l,h}$ is obtained from Equation [8](#A1.E8). We select the top $k\%$ tokens with the highest attention scores as attended tokens:

$$ $\mathcal{I}_{n}^{l,h}=\operatorname{arg\,top}_{k\%}(\bm{a}_{n,q:c}^{l,h}).$ (2) $$

Given that attention often shows high sparsity (Zhu et al., ; Zhang et al., ), with only a few tokens capturing most of the attention scores, we choose $k=10$ to ensure coverage of high-attention tokens and maintain token diversity.

Inspired by (Luo et al., ; Chen et al., ), which validated that the hidden states of LLM can serve as token semantic representations, we compute the token-level External Context Score (ECS) based on the cosine-similarity between the mean-pooling of the last layer hidden states of attended tokens and the hidden state of token $t_{n}$:

$$ $\mathcal{E}_{n}^{l,h}=\frac{\bm{e}\cdot\bm{x}^{L}_{n}}{\|\bm{e}\|\|\bm{x}^{L}_{n}\|},\quad\bm{e}=\frac{1}{|\mathcal{I}_{n}^{l,h}|}\sum_{j\in\mathcal{I}_{n}^{l,h}}\bm{x}^{L}_{j}.$ (3) $$

The response-level ECS is the average of token-level scores:

$$ $\mathcal{E}^{l,h}_{\mathbf{r}}=\frac{1}{|\mathbf{r}|}\sum_{t\in\mathbf{r}}\mathcal{E}^{l,h}_{t}.$ (4) $$

Parametric Knowledge: Considering FFNs store parametric knowledge, to assess how LLM use Parametric Knowledge (as discussed in Section [2.1](#S2.SS1)), we use the $\operatorname{LogitLens}$ to map residual stream states before (i.e., $\bm{x}_{n}^{\mathrm{mid},l}$, calculated from Equation [9](#A1.E9)) and after the FFN layer (i.e., $\bm{x}_{n}^{l}$, calculated from Equation [10](#A1.E10)) to vocabulary distributions. The difference in vocabulary distributions represents the parametric knowledge added by the FFN layer to the residual stream, which is measured by Jensen-Shannon divergence (JSD), gives the token-level Parametric Knowledge Score (PKS):

$$ $\mathcal{P}_{n}^{l}=\operatorname{JSD}\left(q(\bm{x}_{n}^{\mathrm{mid},l})\parallel q(\bm{x}_{n}^{l})\right),$ (5) $$

where $q(\bm{x})=\operatorname{softmax}(\operatorname{LogitLens}(\bm{x}))$.
The response-level PKS is the average of token-level scores:

$$ $\mathcal{P}^{l}_{\mathbf{r}}=\frac{1}{|\mathbf{r}|}\sum_{t\in\mathbf{r}}\mathcal{P}^{l}_{t}.$ (6) $$

Although these metrics may not be exact due to the complexity of LLMs, they serve as intuitive proxies that are sufficiently aligned with the understanding of existing works to analyze the LLM’s use of external context and parametric knowledge in relation to hallucinations, as explored in the following questions.

Figure: Figure 4: Relationship Between LLM Utilization of External Context, Parametric Knowledge, and Hallucinations. *Top* shows the internal mechanism of LLM’s utilization of external context and the occurrence of hallucinations, where the Pearson correlation coefficient between (c) and (a) is 0.41, and between (c) and (b) is 0.46, indicating correlations among them. *Bottom* illustrates the internal mechanism of LLM’s utilization of parametric knowledge and the occurrence of hallucinations, where (d) is scaled by $1e^{7}$.
Refer to caption: /html/2410.11414/assets/x4.png

### 3.2 Experiments

*RQ1: Relationship Between LLM Utilization of External Context, Parametric Knowledge, and Hallucinations*

(1) We first analyze the relationship between External Context and RAG Hallucinations:

ECS Differences between Truthful and Hallucinated Responses: To investigate the relationship between LLM utilization of external context and hallucinations, we compare the external context score $\mathcal{E}$ between truthful responses ($h=0$) and hallucinated responses ($h=1$). Specifically, we construct two subsets from the dataset: $\mathcal{D}^{H}$ for hallucinations ($h=1$) and $\mathcal{D}^{T}$ for truthful responses ($h=0$), and calculate the external context score difference for different attention heads:

$$ $\Delta\mathcal{E}^{l,h}=\mathcal{E}_{T}^{l,h}-\mathcal{E}_{H}^{l,h}=\frac{1}{|\mathcal{D}^{T}|}\sum_{\mathbf{r}\in\mathcal{D}^{T}}\mathcal{E}^{l,h}_{\mathbf{r}}-\frac{1}{|\mathcal{D}^{H}|}\sum_{\mathbf{r}\in\mathcal{D}^{H}}\mathcal{E}^{l,h}_{\mathbf{r}}.$ $$

Result: As shown in Figure [4](#S3.F4)(a), in Llama2-7B, 1006 out of 1024 attention heads show higher external context scores on the truthful dataset $\mathcal{D}^{T}$ compared to the hallucination dataset $\mathcal{D}^{H}$ (i.e., $\Delta\mathcal{E}^{l,h}>0$). Since the external context score represents the LLM’s utilization of external context through attention heads, we can conclude that, at a group level, LLMs utilize external context information less than truthful responses when generating hallucinations.

Correlation between ECS and Hallucination: To examine whether neglecting external context relates to RAG hallucinations, we analyzed the Pearson Correlation Coefficient (PCC) between the hallucination label and the external context score across data points in $\mathcal{D}$. Given the expected negative correlation, we inverted the hallucination label $h$ (denoted as $\bar{h}$) and used PCC to quantify the relationship between $\{\bar{h}_{i}\}_{i=1}^{N}$ and external context scores $\{\mathcal{E}_{i}\}_{i=1}^{N}$.

Result: As shown in Figure [4](#S3.F4)(b), most attention heads show negative correlation between external context scores and hallucination labels $h$. Since the external context score indicates LLMs’ utilization of external context, Figure [4](#S3.F4)(a) and (b) suggest that RAG hallucinations occur when the LLM inadequately leverages external context. Further analysis (Appendix [C](#A3)) shows that hallucinations stem primarily from the LLM losing information attended by attention heads during generation rather than attention heads neglecting external knowledge.

Relation between Copying Heads and Hallucination: We observed that the external context score $\mathcal{E}^{l,h}$ of certain attention heads correlates strongly with hallucinations, prompting further exploration of these heads’ characteristics. Inspired by the Copying Heads concept from Section [2.1](#S2.SS1), we examined the relationship between these heads and Copying Heads. The calculation process of each attention head’s copying head score $\mathcal{C}^{l,h}$ is shown in Appendix [B](#A2)).

Result: As shown in Figure [4](#S3.F4)(c), the correlation with the results in Figure [4](#S3.F4)(a) and (b) indicates that attention heads associated with hallucinations are often Copying Heads (PCC between (c) and (a) is 0.41, and (c) and (b) is 0.46). When these Copying Heads have low external context scores, they either fail to attend to the correct external context or, if attended, fail to retain and utilize this information effectively. This reduces the LLM’s copying ability and leads to hallucinations, explaining the negative correlation between these heads’ external context scores and the hallucination label $h$.

(2) Next, we analyze the relationship between Parametric Knowledge and RAG Hallucinations:

PKS Differences between Truth and Hallucination: We compare the Parametric Knowledge Score $\mathcal{P}$ across different layers when the LLM generates hallucinations versus truthful responses:

$$ $\Delta\mathcal{P}^{l}=\mathcal{P}_{H}^{l}-\mathcal{P}_{T}^{l}=\frac{1}{|\mathcal{D}^{H}|}\sum_{\mathbf{r}\in\mathcal{D}^{H}}\mathcal{P}^{l}_{\mathbf{r}}-\frac{1}{|\mathcal{D}^{T}|}\sum_{\mathbf{r}\in\mathcal{D}^{T}}\mathcal{P}^{l}_{\mathbf{r}}.$ $$

Result: As shown in Figure [4](#S3.F4)(d), parametric knowledge scores in the later layers of FFN modules are significantly higher in the hallucination dataset compared to the truthful dataset (i.e., $\Delta\mathcal{P}^{l}>0$). On average, across all layers, hallucination responses exhibit higher parametric knowledge scores than truthful ones.

Correlation between PKS and Hallucination: To further explore the relationship between parametric knowledge and hallucinations, we calculate the Pearson correlation between the hallucination label and parametric knowledge scores.

Result: As shown in Figure [4](#S3.F4)(e), parametric knowledge scores in the later layers’ FFN modules are positively correlated with the hallucination label $h$ and we define the FFN modules from later layers that show strong correlations with hallucinations as Knowledge FFNs. Since these scores represent the amount of parametric knowledge added to the residual stream, we conclude excessive addition of parametric knowledge by these Knowledge FFNs leads to hallucinations. This aligns with findings from LLM early exit studies (Chuang et al., ; Schuster et al., ): when external context provides sufficient information, shallow layers can generate truthful responses, but over-reliance on parametric knowledge from deeper layers can confuse the model, causing hallucinations.

Figure: Figure 5: (*Left*) Intervention Result for Attention Heads and FFNs. (*Right*) External Context Scores and Parametric Knowledge Scores (scaled by $1e^{5}$) comparing Truth & Known (where LLM knows the truthful answer) and Hallucination (where LLM is unknown about the answer and hallucinated).
Refer to caption: /html/2410.11414/assets/x5.png

*RQ2: Can the relationship identified in RQ1 be validated from a causal perspective?*

To validate the causal relationship between Copying Heads, Knowledge FFN modules, and RAG hallucinations identified in Section [3.2](#S3.SS2), we employ Causal Intervention (Ferrando et al., ) by intervening on attention heads and FFNs, where we applied noise to the attention scores and amplified the contributions of FFN modules to the residual stream (as shown in Figure [3](#S3.F3) (b) and (c)). We compare the Negative Log-Likelihood Loss (NLL) (PyTorch, ) difference for the experimental group (Copying Heads/Knowledge FFNs) and the control group (Other heads/FFNs) on truthful dataset $\mathcal{D}^{T}$. The detailed intervention procedures are provided in the Appendix [D](#A4).

Result: As shown in Figure [5](#S3.F5) (*Left*), the experimental group’s impact on NLL difference was significantly greater than that of the control group for both attention heads and FFN modules. These results, combined with findings from Section [4.3](#S4.SS3), validate parametric knowledge added by Knowledge FFNs and the ability of Copying Heads to retrieve relevant external knowledge and LLM effectively utilizes this information during generation, have a significant causal relationship with the RAG hallucinations.

*RQ3: Hallucination Behavior Analysis from the Parametric Knowledge Perspective*

In this section, we focus on parametric knowledge to analyze hallucination behavior when the LLM either knows or does not know the truthful answer. We conducted a comparison experiment using the LLM-known dataset $\widehat{\mathcal{D}}^{T}$ and the hallucination dataset $\mathcal{D}^{H}$ (For detailed analysis, see Appendix [E](#A5)).

Result: Our results in Figure [5](#S3.F5) (*Right*) show that when the LLM knows the truthful answer, Copying Heads more accurately capture and utilize external knowledge, and Knowledge FFNs add less parametric knowledge to the residual stream compared to hallucination scenarios, which also supports by (Wadhwa et al., ). The results support leveraging our Finding to detect RAG hallucination.

## 4 Methods

Building on our empirical findings (§ [3](#S3)), we propose ReDeEP (Regressing Decoupled External Context and Parametric Knowledge) to detect hallucinations in LLM-based retrieval-augmented generation (§ [4.1](#S4.SS1), § [4.2](#S4.SS2)), and AARF (Add Attention Reduce FFN) to mitigate hallucinations by reweighting the contributions of Knowledge FFNs and Copying Heads to the residual stream (§ [4.3](#S4.SS3)).

### 4.1 Token-level Hallucination Detection — ReDeEP (token)

Our empirical study identified RAG hallucinations as stemming from insufficient utilization of external context by Copying Heads (set $\mathcal{A}$) and excessive reliance on parametric knowledge by Knowledge FFNs (set $\mathcal{F}$). To address the confounding issues shown in Figure [2](#S1.F2), we developed a multivariate analysis approach that regresses decoupled External Context Score and Parametric Knowledge Score to predict hallucinations (Kahlert et al., ). For a response $\mathbf{r}$, the hallucination score ${\mathcal{H}}_{t}$ is:

$$ ${\mathcal{H}}_{t}(\mathbf{r})=\frac{1}{|\mathbf{r}|}\sum_{t\in\mathbf{r}}{\mathcal{H}}_{t}(t),\quad{\mathcal{H}}_{t}(t)=\sum_{l\in\mathcal{F}}\alpha\cdot\mathcal{P}^{l}_{t}-\sum_{l,h\in\mathcal{A}}\beta\cdot\mathcal{E}^{l,h}_{t},$ $$

where $\alpha$ and $\beta$ are regression coefficients for external context and parametric knowledge with $\alpha,\beta>0$, and this linear regression leverages the high Pearson correlation identified in § [3](#S3).

### 4.2 Chunk-level Hallucination Detection — ReDeEP (chunk)

As the Token-level Hallucination Detection computes scores for each token, it is computationally expensive and lacks full contextual consideration. To improve efficiency and accuracy, we propose Chunk-level Hallucination Detection as a more suitable method for RAG hallucination detection. Our approach is inspired by the common chunking operation in RAG Fan et al. (); Finardi et al. (), where the retrieved context $\mathbf{c}$ and the response $\mathbf{r}$ are divided into manageable segments $\left\langle\bm{\tilde{c}}_{i}\right\rangle_{i=1}^{N}$ and $\left\langle\bm{\tilde{r}}_{j}\right\rangle_{j=1}^{M}$.
For the chunk-level external context score $\hat{\mathcal{E}}^{l,h}$, we first calculate chunk-level attention weights $W_{i,j}^{l,h}=\operatorname{Mean-Pooling}\left(A_{\mathbf{\tilde{c}}_{i},\mathbf{\tilde{r}}_{j}}^{l,h}\right)$, where $A$ is the original token-level attention weight matrix, then determine the highest attention chunk pairs $(\mathbf{\tilde{c}},\mathbf{\tilde{r}})$. Using an embedding model ($\operatorname{emb}$), we compute the external context score for each chunk as follows:

$$ $\tilde{\mathcal{E}}^{l,h}_{\mathbf{r}}=\frac{1}{M}\sum_{\tilde{\mathbf{r}}\in\mathbf{r}}\tilde{\mathcal{E}}^{l,h}_{\tilde{\mathbf{r}}},\quad\tilde{\mathcal{E}}_{\mathbf{\tilde{r}}}^{l,h}=\frac{\operatorname{emb}(\mathbf{\tilde{r}})\cdot\operatorname{emb}(\mathbf{\tilde{c}})}{\|\operatorname{emb}(\mathbf{\tilde{r}})\|\|\operatorname{emb}(\mathbf{\tilde{c}})\|}.$ $$

For the chunk-level parametric knowledge score $\tilde{\mathcal{P}}^{l}$, we sum the token-level parametric knowledge scores for each chunk:

$$ $\tilde{\mathcal{P}}^{l}_{\mathbf{r}}=\frac{1}{M}\sum_{\tilde{\mathbf{r}}\in\mathbf{r}}\tilde{\mathcal{P}}^{l}_{\tilde{\mathbf{r}}},\quad\tilde{\mathcal{P}}^{l}_{\mathbf{\tilde{r}}}=\frac{1}{|\mathbf{\tilde{r}}|}\sum_{t\in\mathbf{\tilde{r}}}\mathcal{P}^{l}_{t}.$ $$

Finally, the Chunk-level Hallucination Detection score ${\mathcal{H}}_{c}(\mathbf{r})$ is defined as:

$$ ${\mathcal{H}}_{c}(\mathbf{r})=\sum_{l\in\mathcal{F}}\alpha\cdot\tilde{\mathcal{P}}^{l}_{\mathbf{r}}-\sum_{l,h\in\mathcal{A}}\beta\cdot\tilde{\mathcal{E}}^{l,h}_{\mathbf{r}}.$ $$

### 4.3 Truthful RAG Generation — AARF

Building on the above methods and the analysis in Appendix  [C](#A3), we propose Add Attention Reduce FFN (AARF) to reduce RAG hallucinations by intervening on attention heads and FFN modules without updating model parameters. AARF operates in two stages: (1) token-level hallucination detection and (2) reweighting the contributions of attention heads and FFN modules to the residual stream.

During the generation of token $t_{n}$, we compute the hallucination score ${\mathcal{H}}_{t}(t_{n})$. If ${\mathcal{H}}_{t}(t_{n})\leq\tau$, we proceed with the normal output computation $f(\mathbf{x})$ (see Equation [12](#A1.E12)). If $\mathcal{H}_{t}(t_{n})>\tau$, we adjust the weights of Copying Heads $\mathcal{A}$ and Knowledge FFN modules $\mathcal{F}$, shifting focus toward external context and reducing reliance on parametric knowledge:

$$ $f(\mathbf{x})=\sum_{l=1}^{L}\sum_{h=1}^{H}\widehat{\operatorname{Attn}}^{l,h}\left(\bm{X}_{\leq n}^{l-1}\right)\bm{W}_{U}+\sum_{l=1}^{L}\widehat{\operatorname{FFN}}^{l}\left(\bm{x}_{n}^{\mathrm{mid},l}\right)\bm{W}_{U}+\bm{x}_{n}\bm{W}_{U},$ $$

$$ $\widehat{\operatorname{Attn}}^{l,h}(\cdot)=\begin{cases}\alpha_{2}\cdot\operatorname{Attn}^{l,h}\left(\bm{X}_{\leq n}^{l-1}\right),&\text{if }(l,h)\in\mathcal{A},\\ \operatorname{Attn}^{l,h}\left(\bm{X}_{\leq n}^{l-1}\right),&\text{otherwise}\end{cases},\quad\widehat{\operatorname{FFN}}^{l}(\cdot)=\begin{cases}\beta_{2}\cdot\operatorname{FFN}^{l}\left(\bm{x}_{n}^{\mathrm{mid},l}\right),&\text{if }l\in\mathcal{F},\\ \operatorname{FFN}^{l}\left(\bm{x}_{n}^{\mathrm{mid},l}\right),&\text{otherwise}.\end{cases}$ $$

Here, $\alpha_{2}$ is a constant greater than 1 for amplifying attention head contributions, and $\beta_{2}$ is a constant between (0, 1) for reducing FFN contributions.

## 5 Experiments

**Table 1: Performance comparisons between ReDeEP and the baselines. The boldface represents the best performance, and the underline represents the second-best.**
| LLMs | Categories | Models | RAGTruth | Dolly (AC) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AUC | PCC | Acc. | Rec. | $\mathbf{F_{1}}$ | AUC | PCC | Acc. | Rec. | $\mathbf{F_{1}}$ |  |  |  |
| LLaMA2-7B | MPE | SelfCheckGPT | – | – | 0.5844 | 0.3584 | 0.4642 | – | – | 0.5300 | 0.1897 | 0.3188 |
| Perplexity | 0.5091 | -0.0027 | 0.5333 | 0.5190 | 0.6749 | 0.6825 | 0.2728 | 0.6363 | 0.7719 | 0.7097 |  |  |
| LN-Entropy | 0.5912 | 0.1262 | 0.5600 | 0.5383 | 0.6655 | 0.7001 | 0.2904 | 0.6162 | 0.7368 | 0.6772 |  |  |
| Energy | 0.5619 | 0.1119 | 0.5088 | 0.5057 | 0.6657 | 0.6074 | 0.2179 | 0.5656 | 0.6316 | 0.6261 |  |  |
| Focus | 0.6233 | 0.2100 | 0.5533 | 0.5309 | 0.6622 | 0.6783 | 0.3174 | 0.6262 | 0.5593 | 0.6534 |  |  |
| ECP | Prompt | – | – | 0.6700 | 0.7200 | 0.6720 | – | – | 0.6200 | 0.3965 | 0.5476 |  |
| LMvLM | – | – | 0.5946 | 0.7389 | 0.6473 | – | – | 0.6500 | 0.7759 | 0.7200 |  |  |
| ChainPoll | 0.6738 | 0.3563 | 0.6741 | 0.7832 | 0.7066 | 0.6593 | 0.3502 | 0.6200 | 0.4138 | 0.5581 |  |  |
| RAGAS | 0.7290 | 0.3865 | 0.6822 | 0.6327 | 0.6667 | 0.6648 | 0.2877 | 0.6500 | 0.5345 | 0.6392 |  |  |
| Trulens | 0.6510 | 0.1941 | 0.6422 | 0.6814 | 0.6567 | 0.7110 | 0.3198 | 0.6800 | 0.5517 | 0.6667 |  |  |
| RefCheck | 0.6912 | 0.2098 | 0.6467 | 0.6280 | 0.6736 | 0.6494 | 0.2494 | 0.6100 | 0.3966 | 0.5412 |  |  |
| P(True) | 0.7093 | 0.2360 | 0.5466 | 0.5194 | 0.5313 | 0.6011 | 0.1987 | 0.5444 | 0.6350 | 0.6509 |  |  |
| PCE | EigenScore | 0.6045 | 0.1559 | 0.5422 | 0.7469 | 0.6682 | 0.6786 | 0.2428 | 0.6596 | 0.7500 | 0.7241 |  |
| SEP | 0.7143 | 0.3355 | 0.6177 | 0.7477 | 0.6627 | 0.6067 | 0.2605 | 0.6060 | 0.6216 | 0.7023 |  |  |
| SAPLMA | 0.7037 | 0.3188 | 0.5155 | 0.5091 | 0.6726 | 0.5365 | 0.0179 | 0.5600 | 0.5714 | 0.7179 |  |  |
| ITI | 0.7161 | 0.3932 | 0.5667 | 0.5416 | 0.6745 | 0.5492 | 0.0442 | 0.5800 | 0.5816 | 0.6281 |  |  |
| Ours | ReDeEP(token) | 0.7325 | 0.3979 | 0.7067 | 0.6770 | 0.6986 | 0.6884 | 0.3266 | 0.6464 | 0.8070 | 0.7244 |  |
| ReDeEP(chunk) | 0.7458 | 0.4203 | 0.6822 | 0.8097 | 0.7190 | 0.7949 | 0.5136 | 0.7373 | 0.8245 | 0.7833 |  |  |
| LLaMA2-13B | MPE | SelfCheckGPT | – | – | 0.5844 | 0.3584 | 0.4642 | – | – | 0.5300 | 0.1897 | 0.3188 |
| Perplexity | 0.5091 | -0.0027 | 0.5333 | 0.5190 | 0.6749 | 0.6825 | 0.2728 | 0.6363 | 0.7719 | 0.7097 |  |  |
| LN-Entropy | 0.5912 | 0.1262 | 0.5600 | 0.5383 | 0.6655 | 0.7001 | 0.2904 | 0.6162 | 0.7368 | 0.6772 |  |  |
| Energy | 0.5619 | 0.1119 | 0.5088 | 0.5057 | 0.6657 | 0.6074 | 0.2179 | 0.5656 | 0.6316 | 0.6261 |  |  |
| Focus | 0.7888 | 0.4444 | 0.6000 | 0.6173 | 0.6977 | 0.7067 | 0.1643 | 0.5900 | 0.7333 | 0.6168 |  |  |
| ECP | Prompt | – | – | 0.7300 | 0.7000 | 0.6899 | – | – | 0.6700 | 0.4182 | 0.5823 |  |
| LMvLM | – | – | 0.5956 | 0.8357 | 0.6553 | – | – | 0.6300 | 0.7273 | 0.6838 |  |  |
| ChainPoll | 0.7414 | 0.4820 | 0.7378 | 0.7874 | 0.7342 | 0.7070 | 0.4758 | 0.6800 | 0.4364 | 0.6000 |  |  |
| RAGAS | 0.7541 | 0.4249 | 0.7000 | 0.6763 | 0.6747 | 0.6412 | 0.2840 | 0.6200 | 0.4182 | 0.5476 |  |  |
| Trulens | 0.7073 | 0.2791 | 0.6756 | 0.7729 | 0.6867 | 0.6521 | 0.2565 | 0.5700 | 0.3818 | 0.4941 |  |  |
| RefCheck | 0.7857 | 0.4104 | 0.7200 | 0.6800 | 0.7023 | 0.6626 | 0.2869 | 0.5700 | 0.2545 | 0.3944 |  |  |
| P(True) | 0.7998 | 0.3493 | 0.6266 | 0.5980 | 0.7032 | 0.6396 | 0.2009 | 0.5600 | 0.6180 | 0.5739 |  |  |
| PCE | EigenScore | 0.6640 | 0.2672 | 0.5267 | 0.6715 | 0.6637 | 0.7214 | 0.2948 | 0.6211 | 0.8181 | 0.7200 |  |
| SEP | 0.8089 | 0.5276 | 0.7288 | 0.6580 | 0.7159 | 0.7098 | 0.2823 | 0.6800 | 0.6545 | 0.6923 |  |  |
| SAPLMA | 0.8029 | 0.3956 | 0.5488 | 0.5053 | 0.6529 | 0.6053 | 0.2006 | 0.6000 | 0.6000 | 0.6923 |  |  |
| ITI | 0.8051 | 0.4771 | 0.6177 | 0.5519 | 0.6838 | 0.5511 | 0.0646 | 0.5200 | 0.5385 | 0.6712 |  |  |
| Ours | ReDeEP(token) | 0.8181 | 0.5478 | 0.7711 | 0.7440 | 0.7494 | 0.7226 | 0.3776 | 0.6465 | 0.8148 | 0.7154 |  |
| ReDeEP(chunk) | 0.8244 | 0.5566 | 0.7889 | 0.7198 | 0.7587 | 0.8420 | 0.5902 | 0.7070 | 0.8518 | 0.7603 |  |  |
| LLaMA3-8B | MPE | SelfCheckGPT | – | – | 0.4911 | 0.4111 | 0.5111 | – | – | 0.6800 | 0.2195 | 0.3600 |
| Perplexity | 0.6235 | 0.2100 | 0.6800 | 0.6537 | 0.6778 | 0.5924 | 0.1095 | 0.6200 | 0.3902 | 0.4571 |  |  |
| LN-Entropy | 0.7021 | 0.3451 | 0.6422 | 0.5596 | 0.6282 | 0.6011 | 0.1150 | 0.6100 | 0.5365 | 0.5301 |  |  |
| Energy | 0.5959 | 0.1393 | 0.5466 | 0.5514 | 0.6720 | 0.5014 | -0.0678 | 0.4300 | 0.4047 | 0.5440 |  |  |
| Focus | 0.6378 | 0.3079 | 0.4844 | 0.6688 | 0.6879 | 0.6177 | 0.1266 | 0.5900 | 0.6918 | 0.6874 |  |  |
| ECP | Prompt | – | – | 0.6400 | 0.4403 | 0.5691 | – | – | 0.6800 | 0.3902 | 0.5000 |  |
| LMvLM | – | – | 0.6222 | 0.5109 | 0.6986 | – | – | 0.5500 | 0.6341 | 0.5361 |  |  |
| ChainPoll | 0.6687 | 0.3693 | 0.6311 | 0.4486 | 0.5813 | 0.6114 | 0.2691 | 0.6600 | 0.3415 | 0.4516 |  |  |
| RAGAS | 0.6776 | 0.2349 | 0.5933 | 0.3909 | 0.5094 | 0.6870 | 0.3628 | 0.7100 | 0.8000 | 0.5246 |  |  |
| Trulens | 0.6464 | 0.1326 | 0.5867 | 0.3909 | 0.5053 | 0.7040 | 0.3352 | 0.7200 | 0.3659 | 0.5172 |  |  |
| RefCheck | 0.6014 | 0.0426 | 0.5511 | 0.3580 | 0.4628 | 0.5260 | -0.0089 | 0.5800 | 0.1951 | 0.2759 |  |  |
| P(True) | 0.6323 | 0.2189 | 0.6311 | 0.7083 | 0.6835 | 0.6871 | 0.3472 | 0.6500 | 0.5707 | 0.6573 |  |  |
| PCE | EigenScore | 0.6497 | 0.2120 | 0.6311 | 0.7078 | 0.6745 | 0.6612 | 0.2065 | 0.5584 | 0.7142 | 0.5952 |  |
| SEP | 0.7004 | 0.3713 | 0.6189 | 0.7333 | 0.6915 | 0.5159 | 0.0639 | 0.4500 | 0.6829 | 0.5385 |  |  |
| SAPLMA | 0.7092 | 0.4054 | 0.7133 | 0.5432 | 0.6718 | 0.5019 | -0.0327 | 0.4000 | 0.4040 | 0.5714 |  |  |
| ITI | 0.6534 | 0.3404 | 0.6600 | 0.6850 | 0.6933 | 0.5011 | 0.0024 | 0.5400 | 0.3091 | 0.4250 |  |  |
| Ours | ReDeEP(token) | 0.7522 | 0.4493 | 0.6533 | 0.7984 | 0.7132 | 0.6701 | 0.2421 | 0.6700 | 0.8293 | 0.6901 |  |
| ReDeEP(chunk) | 0.7285 | 0.3964 | 0.6288 | 0.7819 | 0.6947 | 0.7354 | 0.3652 | 0.7100 | 0.8392 | 0.7100 |  |  |

### 5.1 Settings

Data: We evaluate ReDePE and AARF on two public RAG hallucination datasets.
RAGTruth is the first high-quality, manually annotated RAG hallucination dataset. The data includes three RAG task types: Question Answering (QA), Data-to-Text Writing, and News Summarization.
Dolly (AC) is a dataset with Accurate Context obtained from (Hu et al., ), including tasks such as text summarization, closed-QA, and information extraction. More details of the data are in Appendix [G](#A7).

Baselines: We conduct experiments on three variants of LLaMA, including LLaMA2-7B-Chat, LLaMA2-13B-Chat, and LLaMA3-8B-Chat. For hallucination detection methods, we follow the classification of existing methods as shown in Figure [2](#S1.F2). We use (1) Parametric Confounded by External Methods (PCE), (2) External Confounded by Parametric Methods (ECP), and (3) Mixed Parametric and External Methods (MPE). For detailed baselines information, see Appendix [H](#A8). We used AUC, Pearson Correlation Coefficient (PCC), Accuracy (Acc.), Recall (Rec.), and $\text{F}_{1}$ as evaluation metrics for detection accuracy. Implementation details are provided in Appendix [I](#A9).

### 5.2 Experiments

RAG Hallucination Detection: As shown in Table [1](#S5.T1), ReDeEP consistently improves performance across two datasets, various backbone methods, and different metrics, validating its effectiveness in detecting RAG hallucinations. ReDeEP outperforms MPE methods, demonstrating that mechanistic interpretability effectively decouples the LLM’s utilization of external context and parametric knowledge, enabling more accurate detection of RAG hallucinations.
Additionally, ReDeEP surpasses both ECP and PCE methods by incorporating both the External Context Score and Parametric Knowledge Score as covariates in a multivariate regression approach, effectively addressing the confounding problem.
ReDeEP(chunk) generally outperforms ReDeEP(token) in most metrics, suggesting that chunk-level processing better preserves semantic integrity and improves detection performance. Further support for ReDeEP’s effectiveness is provided by the ablation study in Appendix [J](#A10), while the efficiency analysis in Appendix [K](#A11) confirms that ReDeEP achieves comparable time efficiency to the most efficient baselines.

Figure: Figure 6: Comparison between LLMs+AARF vs LLMs judged by GPT-4o.
Refer to caption: /html/2410.11414/assets/x7.png

Truthful RAG Generation:
We evaluated our hallucination reduction method AARF on both the RAGTruth and Dolly (AC) datasets, using GPT-4-o for automatic evaluation to assess the truthfulness (Prompt details can be found in Appendix [L](#A12)). Pairwise comparisons rated by GPT-4-o are shown in Figure [6](#S5.F6), demonstrating that AARF can reduce hallucinations to a certain extent compared to the baseline model. These results validate the effectiveness of our intervention experiments and confirm the findings presented in RQ2 of Section [3.2](#S3.SS2).

## 6 Conclusion

Detecting RAG hallucinations is critical for enhancing the security and reliability of RAG systems. In this work, we introduced ReDeEP, a novel method that detects RAG hallucinations by analyzing LLMs’ utilization of parametric knowledge and external context. Our empirical study shows that hallucinations arise from insufficient utilization of external context by Copying Heads and over-reliance on parametric knowledge by Knowledge FFN modules. These insights also guided the development of interventions to reduce hallucinations without updating model parameters. ReDeEP demonstrates significant performance improvements across the LLaMA family and RAG hallucination benchmarks, outperforming existing detection methods.

## Appendix A Full Background on Attention Heads, FFNs, and Logit Lens

The theoretical foundation of our work is grounded in research on mechanistic interpretability (Ferrando et al., ; nostalgebraist, ; Meng et al., ; Elhage et al., ), which seeks to explain the internal processes of language models (LMs) by interpreting how individual model components contribute to the final prediction. In this study, we focus on the transformer decoder-only architecture (also known as GPT-like) due to its widespread success and popularity (Achiam et al., ; Dubey et al., ). A decoder-only model $f$ consists of $L$ layers and operates on a sequence of embeddings $\mathbf{x}=\left\langle\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{n}\right\rangle$, which represent the tokens $\mathbf{t}=\left\langle t_{1},t_{2},\ldots,t_{n}\right\rangle$. Each embedding $\bm{x}\in\mathbb{R}^{d}$ is a row vector corresponding to a row of the embedding matrix $\bm{W}_{E}\in\mathbb{R}^{|\mathcal{V}|\times d}$, where $\mathcal{V}$ denotes the model’s vocabulary. The sequence $\mathbf{x}$ is represented as a matrix $\bm{X}^{0}\in\mathbb{R}^{n\times d}$ with the embeddings stacked as rows.

We interpret Transformers through the perspective of the residual stream (Elhage et al., ). Due to the residual connections in Transformers, each layer $l$ takes a hidden state $\bm{X}^{l-1}$ as input and adds information obtained from its Attention Heads and Feed-Forward Networks (FFNs) to the hidden state via the residual connection. In this context, the hidden state acts as a residual stream passed through the layers, with each attention and FFN contributing to the final prediction by adding information to the residual stream, resulting in the Residual Stream States. The final layer’s residual stream state is then projected into the vocabulary space using the Unembedding Matrix $\bm{W}_{U}\in\mathbb{R}^{d\times|\mathcal{V}|}$ and normalized via the softmax function to produce a probability distribution over the vocabulary, from which a new token is sampled.

The background knowledge for interpreting the contributions of each FFN and attention head to the model’s prediction is outlined as follows:

Attention Heads: Attention is crucial in Transformers for contextualizing token representations across layers. Each attention head selectively attends to previous positions, gathers information, and updates the current residual stream (Ferrando & Voita, ; Clark et al., ; Wu et al., ). The output of an attention layer is the sum of its attention heads. For each attention head:

$$ $\operatorname{Attn}^{l,h}\left(\bm{X}_{\leq i}^{l-1}\right)=\sum_{j\leq i}a_{i,j}^{l,h}\bm{x}_{j}^{l-1}\bm{W}_{V}^{l,h}\bm{W}_{O}^{l,h}=\sum_{j\leq i}a_{i,j}^{l,h}\bm{x}_{j}^{l-1}\bm{W}_{OV}^{l,h}$ (7) $$

where the learnable weight matrices $\bm{W}_{V}^{l,h}\in\mathbb{R}^{d\times d_{h}}$ and $\bm{W}_{O}^{l,h}\in\mathbb{R}^{d_{h}\times d}$ are combined into the OV matrix $\bm{W}_{OV}^{l,h}=\bm{W}_{V}^{l,h}\bm{W}_{O}^{l,h}\in\mathbb{R}^{d\times d}$, also referred to as the OV (output-value) circuit (Kobayashi et al., ). The attention weights for the current query $i$ to the previous tokens are computed as:

$$ $\bm{a}_{i}^{l,h}=\operatorname{softmax}\left(\frac{\bm{x}_{i}^{l-1}\bm{W}_{Q}^{l,h}\left(\bm{X}_{\leq i}^{l-1}\bm{W}_{K}^{l,h}\right)^{\top}}{\sqrt{d_{k}}}\right)=\operatorname{softmax}\left(\frac{\bm{x}_{i}^{l-1}\bm{W}_{QK}^{h}\bm{X}_{\leq i}^{l-1\top}}{\sqrt{d_{k}}}\right)$ (8) $$

where $\bm{W}_{Q}^{l,h}\in\mathbb{R}^{d\times d_{h}}$ and $\bm{W}_{K}^{l,h}\in\mathbb{R}^{d\times d_{h}}$ combine to form the QK (query-key) circuit (Elhage et al., ), $\bm{W}_{QK}^{h}=\bm{W}_{Q}^{h}\bm{W}_{K}^{h\top}\in\mathbb{R}^{d\times d}$. The QK circuit computes the attention weights, determining the positions that should be attended, while the OV circuit transfers and transforms the information from the attended positions into the current residual stream. The attention block output is the sum of individual attention heads, which is then added back into the residual stream:

$$ $\bm{x}_{i}^{\mathrm{mid},l}=\bm{x}_{i}^{l-1}+\sum_{h=1}^{H}\operatorname{Attn}^{l,h}\left(\bm{X}_{\leq i}^{l-1}\right)$ (9) $$

Previous research has shown that the primary role of attention layers in LLMs is implementing algorithms (Olsson et al., ; Ferrando et al., ). For instance, several attention heads in Transformer LMs have OV matrices exhibiting copying behavior; we refer to these heads as Copying Heads. Elhage et al. () propose using the number of positive real eigenvalues of the full OV circuit matrix $\bm{W}_{E}\bm{W}_{OV}\bm{W}_{U}$ as a summary statistic for detecting Copying Heads. Positive eigenvalues indicate that a linear combination of tokens contributes to an increase in the linear combination of logits of the same tokens.

FFN: Research has shown that the functionality of FFN layers lies in storing knowledge (Geva et al., ). Transformer FFN layers can be represented as a linear combination of vectors. Specifically, for an input vector $\bm{x}_{i}^{\mathrm{mid},l}\in\mathbb{R}^{d}$ drawn from the residual stream states, with FFN parameter matrices $\mathbf{K}^{l},\mathbf{V}^{l}\in\mathbb{R}^{d_{m}\times d}$, the FFN output can be expressed as:

$$ $\mathrm{FFN}^{l}\left(\bm{x}^{\text{mid},l}_{i}\right)=g\left(\bm{x}_{i}^{\mathrm{mid},l}(\mathbf{K}^{l})^{T}\right)\mathbf{V}^{l}=\sum_{i=1}^{d_{m}}f\left(\bm{x}_{i}^{\mathrm{mid},l}\cdot\bm{k}_{i}^{l}\right)\bm{v}_{i}^{l}=\sum_{i=1}^{d_{m}}m_{i}^{l}\bm{v}_{i}^{l}$ $$

$$ $\bm{x}^{l}_{i}=\bm{x}^{\text{mid},l}_{i}+\mathrm{FFN}^{l}\left(\bm{x}^{\text{mid},l}_{i}\right)$ (10) $$

where $g$ is the activation function. Thus, the FFN layer can be viewed as a linear combination of vectors: the multiplication of $\bm{x}_{i}^{\mathrm{mid},l}$ and the key vector $\bm{k}_{i}^{l}$ produces the coefficient $m_{i}^{l}$, which weights the corresponding value vector $\bm{v}_{i}^{l}$.

Logit Lens: The $\operatorname{LogitLens}$ is a technique that decodes hidden states $\bm{x}^{l}$ directly into the vocabulary distribution using the LayerNorm and the unembedding matrix of the LLM for interpretability (nostalgebraist, ):

$$ $\operatorname{LogitLens}\left(\bm{x}^{l}\right)=\operatorname{LayerNorm}(\bm{x}^{l})\bm{W}_{U}$ (11) $$

This approach has been validated in various studies as an effective method for interpreting LLMs’ weight matrices or hidden states (Hanna et al., ; Zhou et al., ; Yu et al., ).

The final output logits of the LLM can be expressed as:

$$ $f(\mathbf{x})=\left(\sum_{l=1}^{L}\sum_{h=1}^{H}\operatorname{Attn}^{l,h}\left(\bm{X}_{\leq n}^{l-1}\right)\bm{W}_{U}+\sum_{l=1}^{L}\operatorname{FFN}^{l}\left(\bm{x}_{n}^{\mathrm{mid},l}\right)\bm{W}_{U}+\bm{x}_{n}\bm{W}_{U}\right)$ (12) $$

## Appendix B Calculation of Copying Heads Score

The traditional method for detecting Copying Heads (Elhage et al., ) involves calculating the eigenvalues of the matrix $M=W_{U}W_{OV}^{h}W_{E}$ and assessing the proportion of positive eigenvalues, where $W_{E}$ is the Embedding Matrix, $W_{OV}$ is the OV Matrix and $W_{U}$ is the Unembedding Matrix.
However, the original Copying Heads identification method proposed by Elhage et al. () requires calculating eigenvalues of large matrices and using the ratio of positive eigenvalues to determine Copying Heads, which becomes computationally expensive for models with large hidden sizes. We propose using the trace of the matrix to estimate the ratio of positive eigenvalues, calibrated with the Gershgorin circle theorem (Bell, ), to obtain each attention head’s copying head score $\mathcal{C}^{l,h}$.
To simplify this, we estimate the proportion using the trace of the matrix and refine this estimation using the Gershgorin Circle Theorem (Bell, ).

- 1.
Trace-Based Estimation: The trace $\text{tr}(M)$ provides an indication of the distribution of positive and negative eigenvalues:
•
A positive trace suggests more positive than negative eigenvalues.
•
A negative trace suggests more negative than positive eigenvalues.
•
A trace near zero suggests a balance between positive and negative eigenvalues.
- 2.
Gershgorin Circle Theorem: To enhance the accuracy of our trace-based estimation, we employ the Gershgorin Circle Theorem, which provides an approximation of the eigenvalue distribution. For any $n\times n$ matrix $M=[a_{ij}]$, each eigenvalue of $M$ lies within at least one Gershgorin disk $D_{i}$:
$D_{i}=\{z\in\mathbb{C}:|z-a_{ii}|\leq R_{i}\}$
where $R_{i}=\sum_{j\neq i}|a_{ij}|$. Each disk is centered at the diagonal element $a_{ii}$ with a radius determined by the sum of the absolute values of the off-diagonal elements in the row. This theorem helps identify the regions in the complex plane where the eigenvalues are likely to be found, allowing us to approximate their distribution without direct computation.
- 3.
IQR-Based Outlier Detection for Boundary Points:
•
Collect boundary points $z+a_{ii}$ and $z-a_{ii}$ from each Gershgorin disk.
•
Calculate the first (Q1) and third quartiles (Q3) of these points, then determine the IQR as $\text{IQR}=Q3-Q1$ (Vinutha et al., ).
•
Identify outliers using bounds $Q1-1.5\times\text{IQR}$ and $Q3+1.5\times\text{IQR}$, counting points outside these limits.
- 4.
Copying Head Score Calculation: Combine rankings based on the number of detected outliers (ascending order) and the absolute value of the trace (descending order). Summing these ranks gives the Copying Head Score $\mathcal{C}^{l,h}$, reflecting the head’s tendency to behave as a Copying Head.

## Appendix C Dive into the Rationale Behind the External Context Score

In Section [3.1](#S3.SS1), we designed the external context score to measure two aspects: (1) whether the attention heads focus on the correct external context, and (2) if attending to the correct context, whether the LLM can effectively retain and utilize this information during the generation process. In this section, we aim to explore whether a low external context score is caused by attention heads focusing on the incorrect external context or by the LLM losing the information attended by the attention heads during the generation process.

To address this, we conducted experiments on the Llama2-7B model using the RAGTruth dataset. Specifically, in our validation experiments, we selected data from RAGTruth where LLaMA2-7B-Chat exhibited hallucinations. Using LangChain, a widely-used open-source toolkit, we applied the RecursiveCharacterTextSplitter to segment the input retrieved document into different spans. We then calculated whether the attention module’s attended span (by mean pooling the attention scores and selecting the input span with the highest score) during the generation of hallucination spans could identify the hallucination spans in the response. If the attention head successfully identified the correct span, this indicates that the attention mechanism focused on the correct external context, but the LLM did not effectively retain and utilize this information during the generation process.
This evaluation was based on GPT-4-o (from OpenAI) using the following prompt:

**Table 2: Proportion of data where Llama2-7B attention heads attend to the correct information.**
| Attention heads attend | Attention heads mis-attend |
| --- | --- |
| 77.5% | 22.5% |

From the results in Table [2](#A3.T2), we can see that a low external context score is mostly due to the LLM losing the information attended by the attention heads during the generation process. In most cases, the attention heads correctly attend to the appropriate external context.
This phenomena may be due to the presence of some Copy suppression heads in the LLM (McDougall et al., ), which may incorrectly suppress the information attended by the Copying head, resulting in the LM losing the information attended by the attention heads during the generation process.
This also validates the feasibility of our proposed AARF method in reducing hallucinations by increasing the output of copying heads in the residual stream.

## Appendix D Detailed Intervention Procedures

In this section, we provide the details of the intervention experiments for RQ2 from Section [3.2](#S3.SS2).

Attention Heads Intervention: As described in Figure [3](#S3.F3) (b), we applied noise to the attention scores $\bm{a}_{i}^{l,h}$ of the experimental group to evaluate their impact on hallucinations. Specifically, the attention scores were sampled from a standard normal distribution:

$$ $a_{i,j}^{l,h}\sim\mathcal{N}(0,1),\quad\tilde{\bm{a}}_{i}^{(l,h)}=\text{softmax}\left(\bm{a}^{(l,h)}\right).$ (13) $$

This approach simulates the removal of meaningful attention patterns, allowing us to assess how Copying Heads’ focus on external context impacts hallucinations.

FFN Modules Intervention: To investigate the role of Knowledge FFNs in hallucinations, we amplified the effect of the FFN modules by increasing their contribution to the residual stream tenfold ($k=10$). This intervention highlights the influence of parametric knowledge on the generation process.

Causal Matching: To reduce potential biases arising from the position of transformer layers or heads, we applied causal matching (Stuart, ). For attention heads, we matched the top 32 Copying Heads with the nearest non-experimental heads within the same layer. Similarly, we matched the top 5 Knowledge FFNs, identified as being most related to hallucinations, with the nearest FFN modules in adjacent layers. This matching process ensured that the comparison between the experimental and control groups was fair and focused on the specific roles of Copying Heads and Knowledge FFNs in hallucination generation.

## Appendix E Detailed analysis of RQ3: Hallucination Behavior analysis from parametric knowledge view

In the experiments (RQ1 and RQ2 in Section [3.2](#S3.SS2)), we analyzed the relationship between LLM-generated responses and the internal states of the model that lead to hallucinations. In this section, we shift our focus to parametric knowledge (since our setting assumes the external context is correct, there is no need for separate analysis from the external context perspective) to examine the two scenarios where the LLM’s internal memory either knows or does not know the truthful answer to the query. This analysis aims to validate whether our previous findings about the connection between LLM hallucinations and internal states are reasonable.

LLM Parametric Knowledge Unknown Truthful Answer
When the LLM’s parametric knowledge does not contain the truthful response, the model must rely on the retrieved context to generate a truthful answer. In this scenario, the Knowledge FFN module may over-add parametric knowledge to the residual stream, while Copying Heads may fail to attend to the correct external context or lose the attended external information during the generation process, leading to hallucinations. This phenomenon is consistent with our earlier findings, where Copying Heads neglect external context and Knowledge FFN modules excessively add parametric knowledge to the residual stream.

LLM Parametric Knowledge Known Truthful Answer
When the LLM’s parametric knowledge contains the truthful answer, RAG responses are typically truthful. To verify whether the model, in this case, relies more on external knowledge and less on parametric knowledge compared to when hallucinations occur, we designed a validation experiment. Specifically, we allowed the LLM to generate responses directly on the truthful dataset $\mathcal{D}^{T}$ without relying on retrieved documents to determine if the LLM could independently produce accurate answers. We used GPT-4-o (Achiam et al., ), along with the original truthful response, to evaluate whether the LLM-generated answers matched the expected ones, thus assessing if the LLM’s parametric knowledge could correctly answer independently (see prompt in Appendix [F](#A6)). These correct responses form the LLM-known dataset $\widehat{\mathcal{D}}^{T}$. We then analyzed the differences in external context scores and parametric knowledge scores for the LLM across $\widehat{\mathcal{D}}^{T}$ and $\mathcal{D}^{H}$.

Result: As shown in Figure [5](#S3.F5) (*Right*), when the LLM’s parametric knowledge knows the truthful answer, we observe that Copying Heads can more accurately capture external knowledge and effectively retain and utilize this information during the generation process, showing more stable performance compared to their behavior in the hallucination dataset. Although in scenarios where the LLM knows the truthful answer, the Knowledge FFN layers add less parametric knowledge to the residual stream than the hallucination dataset, this supports our earlier finding of the negative impact of excessive utilization of parametric knowledge by the Knowledge FFN module.

## Appendix F Prompt for Evaluating Parametric Knowledge

To assess whether the LLM’s parametric knowledge alone could provide accurate answers independently of retrieved documents, we used the following prompt in our validation experiment. The aim was to evaluate if the LLM-generated responses on the truthful dataset $\mathcal{D}^{T}$ matched the expected truthful answers. The prompt was designed to engage GPT-4-o (Achiam et al., ) as an evaluator to compare the LLM’s responses with the ground truth.

Prompt:

## Appendix G Details about RAG Hallucination Datasets

RAGTruth: The RAGTruth (Niu et al., ) dataset is the first high-quality, manually annotated RAG hallucination dataset. It is divided into three task types: Question Answering (QA), Data-to-Text Writing, and News Summarization. However, the dataset does not include hallucination annotations for responses generated by Llama3-8B. To address this, we employed three annotators with graduate-level qualifications to manually evaluate the presence of hallucinations in different LLM RAG responses. Each response was carefully assessed to determine whether it contained any hallucinations based on the accuracy and relevance of the retrieved and generated content.

Dolly (AC): The Dolly (AC) dataset is sourced from (Hu et al., ) and consists of tasks such as text summarization, closed-QA, and information extraction. Similar to RAGTruth, Dolly (AC) lacks hallucination annotations for responses generated by certain LLMs, particularly those without access to accurate context. To fill this gap, the same team of three qualified annotators was tasked with manually evaluating the RAG responses to determine if they contained hallucinations, focusing on the alignment between the generated responses and the external context.

In both cases, the manual annotation process involved cross-verifying the generated content with the provided external context to detect discrepancies or factual inconsistencies that would indicate hallucinations.

## Appendix H Details about Baseline Models

This section provides details on the baseline models and hallucination detection methods used in our experiments. We categorize the methods into three groups based on their approach to leveraging parametric knowledge and external context: Parametric Confounded by External (PCE), External Confounded by Parametric (ECP), and Mixed Parametric and External (MPE).

(1) Parametric Confounded by External (PCE):

- •
EigenScore: EigenScore measures the semantic consistency in the embedding space. Higher EigenScores suggest a higher likelihood of hallucinations, as they indicate greater semantic divergence (Chen et al., ).
- •
SEP: SEP (Semantic Entropy Probe) uses linear probes trained on the hidden states of LLMs to detect hallucinations by analyzing the semantic entropy of the tokens before generation (Han et al., ).
- •
SAPLMA: SAPLMA trains a classifier on LLM activation values to detect hallucinations. It captures internal signals from the LLM’s hidden layers to identify when the model might generate a hallucination (Azaria & Mitchell, ).
- •
ITI: Inference-Time Intervention (ITI) analyzes attention head activations and uses a binary classifier to predict hallucinations by studying the relationship between heads and task performance (Li et al., ).

(2) External Confounded by Parametric (ECP):

- •
Prompt: This method uses the prompts provided in the RAGTruth (Niu et al., ) to evaluate whether the LLM-generated responses are hallucinations by comparing them against the ground truth using GPT-4-o-mini (Niu et al., ).
- •
LMvLM: LMvLM employs a multi-turn interaction between two language models(GPT-4-o-mini vs. Backbone LLM) to discover inconsistencies by having them cross-examine each other’s responses (Cohen et al., ).
- •
ChainPoll: ChainPoll uses GPT-4-o-mini to determine if a completion contains hallucinations through a carefully designed prompt. The evaluation is repeated multiple times (typically five), and the final hallucination score is calculated as the ratio of ”yes” answers to the total number of responses (Friel & Sanyal, ).
- •
RAGAS: RAGAS checks the faithfulness of the generated response by breaking down sentences into shorter assertions and verifying each against the context, using GPT-4-o-mini to calculate a faithfulness score as the ratio of supported statements (Es et al., ).
- •
Trulens: Trulens assesses the overlap of information between the context and the generated response using GPT-4-o-mini, assigning a groundedness score between 0 and 10 based on the degree of overlap (Trulens, ).
- •
RefCheck: Similar to RAGAS, RefCheck extracts knowledge graphs from the generated responses and evaluates whether the knowledge graphs align with the external context (Hu et al., ).
- •
P(True): P(True) measures the uncertainty of the generated claim by querying the LLM itself on the truthfulness of its generated response. The confidence score is calculated as the probability of the first token being ”True” (Kadavath et al., ).

(3) Mixed Parametric and External (MPE):

- •
SelfCheckGPT: SelfCheckGPT uses a zero-resource, sampling-based approach where multiple reference responses are checked by GPT-4-o-mini for consistency with the generated answer (Manakul et al., ).
- •
LN-Entropy: Length-Normalized Entropy measures sequence-level uncertainty across multiple generations, using entropy normalized by sequence length to detect hallucinations (Malinin & Gales, ).
- •
Energy: Energy-based OOD detection identifies hallucinations by analyzing the uncertainty in the generated response using energy functions. Higher energy suggests a higher likelihood of hallucinations (Liu et al., ).
- •
Focus: Focus enhances uncertainty-based hallucination detection by focusing on key informative tokens, preceding words, and token properties, simulating human factuality checking (Zhang et al., ).
- •
Perplexity: This method uses the perplexity of the LLM-generated response to detect hallucinations. A higher perplexity indicates greater uncertainty and a higher likelihood of hallucinations (Ren et al., ).

## Appendix I Implementation Details.

We run all the experiments on machines equipped with NVIDIA V100 GPUs and 52-core Intel(R) Xeon(R) Gold 6230R CPUs at 2.10GHz. We utilize the Huggingface Transformers package to conduct experiments. During the decoding of responses from the language models, we employ greedy search to generate responses. The remaining parameters follow the models’ default settings. For RAGTruth, we use the validation set to select the hyperparameters. For Dolly (AC), we use two-fold validation to select the hyperparameters. For the baselines, we perform hyperparameter tuning within the range provided by the original works.
For ReDeEP(Chunk) on Dolly (AC), on Llama2-7B, we select the top-7 scoring Copying Head and top-3 FFN layers with $\alpha=1$ and $\beta=1.6$, as described in Section [3](#S3). On Llama2-13B, we select the top-11 scoring Copying Head and top-3 FFN layers with $\alpha=1$ and $\beta=0.2$. On Llama3-8B, we select the top-1 scoring Copying Head and top-1 FFN layers with $\alpha=1$ and $\beta=0.1$, as described in Section [3](#S3). For ReDeEP(Chunk) on RAGTruth, on Llama2-7B, we select the top-3 scoring Copying Head and top-4 FFN layers with $\alpha=1$ and $\beta=0.6$. On Llama2-13B, we select the top-9 scoring Copying Head and top-3 FFN layers with $\alpha=1$ and $\beta=1.8$. On Llama3-8B, we select the top-2 scoring Copying Head and top-5 FFN layers with $\alpha=1$ and $\beta=1.2$.
For ReDeEP(Token) on Dolly (AC), on Llama2-7B, we select the top-4 scoring Copying Head and top-3 FFN layers with $\alpha=1$ and $\beta=0.2$, as described in Section [3](#S3). On Llama2-13B, we select the top-4 scoring Copying Head and top-5 FFN layers with $\alpha=1$ and $\beta=0.6$. On Llama3-8B, we select the top-1 scoring Copying Head and top-1 FFN layers with $\alpha=1$ and $\beta=0.1$, as described in Section [3](#S3). For ReDeEP(Token) on RAGTruth, on Llama2-7B, we select the top-1 scoring Copying Head and top-10 FFN layers with $\alpha=1$ and $\beta=0.2$. On Llama2-13B, we select the top-2 scoring Copying Head and top-17 FFN layers with $\alpha=1$ and $\beta=0.6$. On Llama3-8B, we select the top-3 scoring Copying Head and top-30 FFN layers with $\alpha=1$ and $\beta=0.4$.
For AARF on RAGTruth, for Llama2-7B and Llama2-13B, we select $\alpha_{1}=5$, $\beta_{1}=0.2$, and $\tau=0.4$, while for Llama3-8B, we use $\alpha_{1}=5$, $\beta_{1}=0.2$, and $\tau=0.0$. On Dolly (AC), for Llama2-7B, Llama2-13B, and Llama3-8B, we select $\alpha_{1}=2$, $\beta_{1}=0.5$, and $\tau=0.6$.
As proposed in Section [4.2](#S4.SS2), ReDeEP (Chunk) requires segmenting the retrieved documents and responses from the benchmark. For this, we utilized LangChain (^1^11[https://www.langchain.com/](https://www.langchain.com/)), a popular open-source toolkit, and applied the RecursiveCharacterTextSplitter for the segmentation process.
We use BGE embeddings Xiao et al. () for ReDeEP(chunk), which is the sota embedding model.
The Llama2-7B can be downloaded from [https://huggingface.co/meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf).
The Llama2-13B can be downloaded from [https://huggingface.co/meta-llama/Llama-2-13b-chat-hf](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf).
The Llama3-8B can be downloaded from [https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct).
The BGE embeddings can be downloaded from [https://huggingface.co/BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5).

## Appendix J Ablation Study

**Table 3: Ablation Study of ReDeEP.**
| RAGTruth |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ReDeEP (Token) | AUC | PCC | ReDeEP (Chunk) | AUC | PCC |  |  |
| LLaMA2-7B | Only PKS | 0.6950 | 0.3327 | LLaMA2-7B | Only PKS | 0.6180 | 0.2103 |
| Only ECS | 0.7234 | 0.3779 | Only ECS | 0.7098 | 0.3944 |  |  |
| Full | 0.7325 | 0.3979 | Full | 0.7458 | 0.4203 |  |  |
| LLaMA2-13B | Only PKS | 0.7214 | 0.3682 | LLaMA2-13B | Only PKS | 0.6614 | 0.2566 |
| Only ECS | 0.8040 | 0.5201 | Only ECS | 0.7231 | 0.3922 |  |  |
| Full | 0.8181 | 0.5478 | Full | 0.8244 | 0.5566 |  |  |
| LLaMA3-8B | Only PKS | 0.6102 | 0.1085 | LLaMA3-8B | Only PKS | 0.6082 | 0.1695 |
| Only ECS | 0.7336 | 0.4312 | Only ECS | 0.6923 | 0.3102 |  |  |
| Full | 0.7522 | 0.4493 | Full | 0.7285 | 0.3964 |  |  |
| Dolly (AC) |  |  |  |  |  |  |  |
| ReDeEP (Token) | AUC | PCC | ReDeEP (Chunk) | AUC | PCC |  |  |
| LLaMA2-7B | Only PKS | 0.6671 | 0.2374 | LLaMA2-7B | Only PKS | 0.6383 | 0.2115 |
| Only ECS | 0.6629 | 0.2852 | Only ECS | 0.7552 | 0.4478 |  |  |
| Full | 0.6884 | 0.3266 | Full | 0.7949 | 0.5136 |  |  |
| LLaMA2-13B | Only PKS | 0.6639 | 0.2891 | LLaMA2-13B | Only PKS | 0.6790 | 0.2883 |
| Only ECS | 0.6856 | 0.3107 | Only ECS | 0.7383 | 0.4338 |  |  |
| Full | 0.7226 | 0.3776 | Full | 0.8420 | 0.5902 |  |  |
| LLaMA3-8B | Only PKS | 0.6329 | 0.2300 | LLaMA3-8B | Only PKS | 0.7334 | 0.3503 |
| Only ECS | 0.6105 | 0.1556 | Only ECS | 0.6166 | 0.2624 |  |  |
| Full | 0.6701 | 0.2421 | Full | 0.7354 | 0.3652 |  |  |

As shown in Table [3](#A10.T3), when performing RAG hallucination detection, using only the Parametric Knowledge Score (Only PKS) or only the External Context Score (Only ECS) does not achieve the same performance as the Full ReDeEP model. This validates the effectiveness of employing multivariate regression, where both PKS and ECS are used simultaneously as covariates. According to the analysis in  Section [1](#S1), using Only PKS or Only ECS introduces confounding issues, leading to a decrease in performance. This explains why both Only ECS and Only PKS yield lower results compared to Full ReDeEP.

## Appendix K Efficiency Analysis

Figure: Figure 7: Comparison of the detection efficiency of different hallucination detection methods on LLaMA-2-7B-Chat and LLaMA-2-13B-Chat.
Refer to caption: /html/2410.11414/assets/x9.png

As shown in Figure [7](#A11.F7), ReDeEP (chunk) is more efficient than ReDeEP (token), confirming the superior time efficiency of our proposed chunk-level hallucination detection method. Additionally, methods that do not rely on external models (e.g., ITI, ReDeEP, etc.) exhibit significantly higher efficiency compared to those using external models (e.g., RefCheck, Chainpoll, etc.).
ReDeEP is positioned at a high-efficiency level among hallucination detection models, validating its feasibility for industrial applications. Considering that the main time consumption of AARF lies in ReDeEP (token), the relatively low time consumption of ReDeEP (token) further validates the efficiency of AARF.

## Appendix L Prompt for truthfulness evaluation

In the Truthful RAG Generation Experiments, we used the following prompt to assess the truthfulness of RAG generation from AARF and baseline models:

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - Why LLama2-7B (smaller and older version than others) has better results on Dolly in terms of F1 or Accuracy in Table 1?
soundness: 3
strengths: - Authors provide a straightforward method to detect hallucinations in RAGs that does not require model fine-tuning.
- Empirical results provided by the authors look good.
summary: This paper proposes a method for detecting hallucinations of Retrieval Augmented Generation (RAG) models in the scenario when retrieved context is accurate and relevant. 

The authors hypothesize that hallucinations are caused by models ignoring the retrieved context and overemphasizing their parametric knowledge. To capture these concepts they introduce two auxilary scores: External Context Score (ECS) that reflects utilization of the retrieved context by the model, and Parametric Knowledge Score (PKS) that reflects utilization of the parametric knowledge. Hallucinations are then predicted by thresholding a hallucination score H which is computed as a weighted sum of ECS and PKS.

In addition to that, the authors propose a method to reduce hallucinations by suppressing outputs of attention heads that contribute to low ECS and outputs of fully-connected layers that contribute to high PKS.
weaknesses: ## Lack of justification for PKS and ECS

### No PKS justification
Although PKS is correlated with a hallucination label (line 319) there is still no guarantee that it is adding parametric knowledge. Since you do not provide any theoretical justification for this score, at least an empirical justification is needed. You can run a simple experiment: use LogitLensed outputs before FFN as final outputs and check whether it removes the parametric knowledge bias using some of the setups for that, for example, the one from [1] (they study it through the prism of the phenomenon you encounter in RQ3 and Appendix E).

### Questionable ECS justification
Contrary to the PKS the authors provided empirical justification for the ECS measuring model reliance on context, however, I find it not very convincing so far.

First of all, I do not see how the ratio of attention head attending vs mis-attending justifies ECS. It would make more sense to me if you provided such a ratio for mulitple different values of ECS and observed that the higher the ECS the more often a model attends.

Secondly, I am not sure that ratio of attending is computed correctly. As far as I understood for LLama-7B you take hallucinated response (which means that it contradicts external context) and the most attended span in external context. Then you ask gpt-4o to evaluate whether this span supports existence of a conflict in response or not. If that is the case, I do not understand why this experiment shows whether the model attends (the attention span contains part of the context needed for the correct answer) or mis-attends. If attention span supports the existence of a conflict in response it might still not be relevant for the correct response itself, which means a conflict exists but we can not call it a hallucination according to your definition (hallucination = response is contradicting the context or is not supported by it - line 72).

Please correct me if I misunderstood the experiment setting, what is meant by attending, or the way attending and mis-attending is computed.

## Too many hyperparameters
I am afraid that the proposed hallucination detection method is not applicable in practice as it requires a lot of manual hyperparameter tuning. According to the provided values, they all are different per dataset and model (see Appendix I). They include:

- top k % for ECS 
- top k % for PKS
- tau threshold for H - page 8 bottom
- alpha and beta for reweighting page 9 top
- chunk size for the chunked version of REDEEP

I suggest that the authors discuss strategies for automating hyperparameter selection or provide guidelines for choosing these parameters in real-world applications.

## Insufficient experiments

### Hallucination detection experiment
- For RagTruth dataset there exist baselines provided by the original paper [2] which perform better than all the baselines considered by you, could you please include them? E.g. Baseline LLama2-13B results fine-tuned on RagTruth have 78.7 F1, see Table 5 in [2] vs yours 78.3 in Table 1. I think the comparison makes a lot of sense since you tune many hyperparams using RagTruth validation dataset while you could simply fine-tune that baseline on the same data instead.
- Same comes for Dolly dataset, please include results for AlignScore and RepC-LE-nn-n2000-e2000 that have 84 and 86 accuracy correspondigly, while the best method provided by you scored 73.73 (LLama2-7B).
- Please also provide results for the Noisy Context split from Dolly [3] dataset because it better approximates realistic RAG application scenario. 

### Causal experiment

- First of all, I don’t see how a higher NLL difference for the experimental group than for the control group shows a causal relation between hallucinations occurrence and copying heads neglecting necessary knowledge, could you please elaborate?
- The experiment results are very noisy and it is hard to draw any conclusions from them, for example, boxplot of the experimental group is fully contained within the boxplot of the control group in Figure 5 (b). 
- It is not clear how many heads are within experimental and control groups, it can be the case that loss changes are bigger for the experimental group simply because it intervenes in more heads.

### Hallucination generation experiment

Prompt for truthfulness (Appendix L) creates bias, since GPT-4o knows which answer belongs to the baseline and which to AARF. It can influence its answers since usually in scientific papers named methods outperform baselines, which must have been the case on chatgpt training data as well and possibly created such a bias. 

Instead, it would be nice to see the results for prompts that contain anonymous names (e.g. model 1 and model 2 instead of baseline and AARF) to avoid the mentioned naming bias and have a randomly shuffled order of AARF and Baseline inputs before showing to GPT-4o to avoid positional bias.

### Lack of sensitivity experiments
Please provide sensitivity experiments to the numerous hyperparameters you introduced (see the section "Too many hyperparameters" for the hyperparameters)

## Unclear writing
- While being core concepts of the paper, Copying Heads (set A) Knowledge FFNs (set F) are not formally defined (line 381). I guess set A is built by taking top-k attention heads after sorting them by ECS while set B is built by taking top-k FFNs after sorting them by PKS, but I could not find it in text.
- Strange ordering equations, for example, Eq. 2 that defines an important part of ECS has an undefined value “a” which is only introduced in Appendix Eq. 8.

## Typos
455: REDEPE

## References

[1] Kortukov, E., Rubinstein, A., Nguyen, E., & Oh, S.J. (2024). Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documents.

[2] Wu, Y., Zhu, J., Xu, S., Shum, K., Niu, C., Zhong, R., Song, J., & Zhang, T. (2023). RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. Annual Meeting of the Association for Computational Linguistics.

[3] Hu, X., Ru, D., Qiu, L., Guo, Q., Zhang, T., Xu, Y., Luo, Y., Liu, P., Zhang, Y., & Zhang, Z. (2024). RefChecker: Reference-based Fine-grained Hallucination Checker and Benchmark for Large Language Models. ArXiv, abs/2405.14486.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: The paper offers valuable insights into how RAG-based LLMs produce hallucinated outputs. Building on these findings, it proposes a detection method and mitigation strategy grounded in this understanding. Presentation issues remain, particularly in the main figure explaining the method, yet the contribution is significant.
soundness: 4
strengths: Each step is thoughtfully motivated, with both conceptual reasoning and empirical validations in §3. The detection method shows effective results in Table 1, and the RAG truthfulness improves using AARF, as shown in Figure 6.
summary: Retrieval-Augmented Generation (RAG) models are still prone to hallucinations. This paper explores the internal mechanisms behind hallucinations in RAG settings. Building on these insights, the authors propose a hallucination detection method, ReDeEP, and a RAG truthfulness improvement method, AARF.
weaknesses: Figure 3 is problematic. The starting point and flow of the diagram are unclear, with too many arrows, making it hard to identify the main computational paths. An effective graphic would show one main data processing pipeline, which is missing here. Additionally, the quantities computed are not well-defined. Panels (b) and (c) add no extra information and could be removed without loss.

Otherwise, rather minor points:
- l.281: Please describe the number of hallucinations and non-hallucinations (h = 0 and h = 1) in the evaluation set.
- Pearson's Correlation in §3: Why measure Pearson’s correlation between ECS and hallucination labels (binary)? It would be more informative to report accuracy at a fixed threshold or detection metrics such as AUROC. Similarly, for PKS and hallucination, detection metrics like AUROC would be preferable.
- l.465: Could you clarify the criteria for selecting thresholds for accuracy, recall, and F1?

Even more nits:
- Use full names for FFN, ReDeEP, and AARF, at least in the abstract.
- In Figure 4(c), clarify what the colour bar values represent.
- Overall, font sizes in the figures are too small.
- Structure in §3.2 is difficult to follow. Stick to a standard structure using \section, \subsection, \subsubsection, \paragraph, etc., rather than introducing new hierarchies (boldface, underline, italics, numbering (1), (2), …).

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could you discuss more the trade-offs of your method? In particular thinking about real time settings? 

2. Have you tested your method on non-LLama models? Do you anticipate any challenges for different models? 

3. Could you provide some example outputs pre and post using AARF? Or can you speak to the effect AARF has on the coherence of the model’s output after AARF?
soundness: 3
strengths: 1. The development of the ECS and PKS metrics to understand the contributions external and internal knowledge have on the LLM's generation is a compelling and novel way to understand LLM outputs. 

2. They demonstrated great empirical validation by running extensive experiments across two datasets, three LLMs, and many baseline methods. 

3. They also introduce a method to curb hallucinations called AARF - which relates back to the introduced metrics nicely.
summary: This work proposes ReDeEP - a method to detect hallucinations by LLMs in retrieval augmented generation settings by using mechanistic interpretability. The authors introduce two novel metrics - (1) the External Context Score (ECS) and (2) Parametric Knowledge Score (PKS) to identify when hallucinations happen because of over reliance on internal knowledge or from the underuse of external information. The authors also introduce AARF (Add Attention Reduce FFN), which aims to adjust the weights of the attention heads, and feed forward layers to reduce hallucinations.  Their approach is empirically validated on standard benchmarks, demonstrating superior performance to existing methods.
weaknesses: 1. Performing this analysis at the token/chunk level might limit its practicality in real time or large scale settings - it would be nice to have a richer discussion of the trade-offs and real world feasibility. 

2. The experiments are extensive - however they are all with the LLama family of models - testing (even a much smaller set) on a different model would be informative. 

3. While the performance of AARF seems good (Figure 6) - it would be good to see some example outputs - its unclear how this could effect the model’s output in terms of coherence/writing in general.

Meta-review: This paper studies the potential hallucinations issue of Retrieval Augmented Generation (RAG) models in practice and proposes a hallucination detection method, ReDeEP, and a RAG truthfulness improvement method, AARF. Reviewers agreed that the paper is clearly written, and the ideas are well motivated, and experimental results sufficiently demonstrate the effectiveness of the proposed methods. Detecting hallucinations associate with RAG is definitely an important research topic, especially that RAG has been widely applied in many domains nowadays. This paper brings new insights and tools to the field.

Decision: Accept (Spotlight)