ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability
Authors: ZhongXiang Sun
Xiaoxue Zang
Kai Zheng
Jun Xu
Xiao Zhang
Weijie Yu
Yang Song
Han Li
Venue: ICLR 2025 Spotlight (2025)
Keywords: Retrieval-Augmented Generation Hallucination
Hallucination Detection
Mechanistic Interpretability
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose ReDeEP for detecting hallucinations in RAG models by decoupling external context and parametric knowledge, and AARF to reduce hallucinations by modulating the contributions of Knowledge FFNs and Copying Heads.

Abstract: Retrieval-Augmented Generation (RAG) models are designed to incorporate external knowledge, reducing hallucinations caused by insufficient parametric (internal) knowledge. However, even with accurate and relevant retrieved content, RAG models can still produce hallucinations by generating outputs that conflict with the retrieved information. Detecting such hallucinations requires disentangling how Large Language Models (LLMs) balance external and parametric knowledge. Current detection methods often focus on one of these mechanisms or without decoupling their intertwined effects, making accurate detection difficult. In this paper, we investigate the internal mechanisms behind hallucinations in RAG scenarios. We discover hallucinations occur when the **Knowledge FFNs** in LLMs overemphasize parametric knowledge in the residual stream, while **Copying Heads** fail to effectively retain or integrate external knowledge from retrieved content. Based on these findings, we propose **ReDeEP**, a novel method that detects hallucinations by decoupling LLM’s utilization of external context and parametric knowledge. Our experiments show that ReDeEP significantly improves RAG hallucination detection accuracy. Additionally, we introduce AARF, which mitigates hallucinations by modulating the contributions of Knowledge FFNs and Copying Heads.

Published as a conference paper at ICLR 2025
REDEEP: DETECTING
HALLUCINATION GENERATION
IN
VIA
MECHANISTIC INTERPRETABILITY
Zhongxiang Sun', Xiaoxue Zang", Kai Zheng", Jun Xu'; Xiao Zhang', Weijie Yu Yang Song 2, Han Li I Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
Kuaishou Technology Co., Ltd., Beijing, China
$ Sccooo of Information Technology and Management, University of International Business and Economics sun zanzhoggxi ang @ ruc.edu.en xxi c66 @ 126.com, zhengka i @ @kuaishou.com, cunxu@ruc.edu.oo zhangx8 8 @ruc.edu.cn, vu@uibe.edu.cn. y @ sonyis.me, 1 LihanO8 @ kuaishou.com
ABSTRACT
Retrieval-Aummented Generation (RAG) models are designed to incorporate ex- ternal knowledge, reducing hallucinations caused by insufficient parametric (in- ternal) knowledge. However, even with accurate and relevant retrieved content, RAG models still produce hallucinations by generating outputs that conflict with the retrieved information. Detecting such hallucinations requires disentan- gling how Large Language Models (LLMs) utilize external and parametric knowl- edge. Current detection methods often focus on one of these mechanisms or with- out decoupling their intertwined effects, making accurate detection difficult. In this paper, we investigate the internal mechanisms behind hallucinations in RAG scenarios. We discover hallucinations occur when the Knowledge Feedforward Neural Networks in LLMs overemphasize parametric knowledge in the residual stream, while Copying Heads fail to effectively retain or integrate external knowl- edge from retrieved content. Based on these findings, we propose ReDeEP, a novel method that detects hallucinations by decoupling LLM's utilization ex- ternal context and parametric knowledge. Our experiments show that ReDeEP significantly improves RAG hallucination detection accuracy. Additionally, we introduce AARF, which mitigates hallucinations by modulating the contributions of Knowledge FFNs and Copying Heads. The source code and dataset are avail- able at: https: / tttp://ttttttbtt thub com/ Jeryi Sun ReDEeR- -Stn/neDDne----.. ICLR
INTRODUCTION
LLMs have made significant advancements in natural language processing tasks (Dubey et al. 2024 Achiam al. 2023). However, they still face challenges with hallucinations, often generating fac- tually inaccurate outputs (Huang et al. (023). To mitigate this issue, many researchers have intro- duced Augmeetal-Anmeented Generation (RAG) models, which aim to improve the accuracy LLM responses by incorporating relevant information retrieved from external knowledge bases (Shuster
al. 2021 Gao et al. 2023).
Despite the use of accurate and relevant retrieved context, RAG models may still produce statements that are either unsupported or contradict the retrieved information, a phenomenon we term RAG Hallucination (Niu et al.) 2024 Magesh et al. 2024). Recent studies have examined the potential conflicts between the external context and the LLM's parametric knowledge in RAG models (Xu al. 2024). As shown in Figure I] these conflicts can lead hallucinations but do not always cause them. Therefore, it is important to distinguish RAG hallucination from Knowledge Conflict as a new research direction. Our work focuses detecting RAG hallucinations, specifically in cases where the retrieved external context is accurate and relevant.
* *Corresponding author. Work partially done at Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education.

---

Published as a conference paper at ICLR 2025
Reriave Doctmeet
Drro's replention Cung'ed moier, sacouccusfflty coerpected benging back 1935. farer curplex....
gasd parnooe 10 cltmoonds the betwoee modi
betwers 20
Quary
Chrah Chngre misi vacoaucully complexd world' fire ampie Bon Be fr side ofe raiee How raary gaons samples were boought bake?
alzsaonsd?
I'ccmetn knundedee booeoza and 32 wedka the best pietures
Paranceie lnselllgee
LLaMA2-708:
LLaMA2-70B:
brogght beak 1,55) gaars
Respotse
gender noveal utraarand, was until 17 wedo of prograncy in cturee accuraie roudts ber sitracordd, achodale appaiemmntt butwron webkeet pecpratacy pictaro
Chira's Cherg o- mesie exceosfully cellaneed (915.) gromn of sampppo fhom Be de the Moos.
Correct
Hallacination
Figure I: Two examples of RAG where the retrieved document is correct but conflicts with para- metric knowledge. The left example shows correct response based external knowledge, while the right example demonstrates hallucination despite accurate external context.
External Context
Parametric Knowledge
HE: Respoase Hallucination not
External contest
(9) conforndislll
Confounder:
Query
Confounder:
Decouple Regression
B
Causal graph
Causal gruph
Docrepk
II
Methods
Response
Methods
ITI
(ei)) Mis P
Prompe
SEP
Eiergy
RefCheck
SAPLMA
Methods
(Ours)
EigenScore
Porpleiity
ChainPoll
graph
UN-Fatropy
Figure 2: Causal perspectives on hallucination detection methods. (i): parametric knowledge is confounded by external context, (ii): external context is confounded by parametric knowledge, and (ii):: mixes both without decoupling their contributions. (Ours): decouple these confounders using mechanistic interpretability, incorporating them as covariates to improve hallucination detection.
Existing hallucination detection methods can be categorized into three causal frameworks (Neuberg 2003 Pearl 2009), illustrated in Figure ( (detailed introduction of methods see Appendix [ (1)
Parametric Confounded by External: which relies on the LLM's hidden states for hallucination detection, where the external context (E) serves as confounder between parametric knowledge (P)
and hallucinations (H). From knowledge storage perspective (Geva al., 2021), hidden states represent the result of querying the parametric knowledge (P) with external context (E), establishing causal path from E to P (graph (a)). The presence of E confounder complicates the accurate prediction of hallucinations based on P alone (Chyzhyk et al.] 20022.. (ii) External Confounded by Parametric: which focuses hallucination detection by leveraging external context and model responses. Here, parametric knowledge (P) is confounder between the external context (E) and hallucinations (H), creating causal link from P to (graph (b)) due the unavoidable presence of parametric knowledge in the response. (iii) Mixed Parametric and External: which combines both parametric and external knowledge directly, often using uncertainty or sampling techniques (e.g.. token probability) to detect hallucinations (graph (c)). However, this mixing of E and P without decoupling their roles obscures their individual contributions (Bengio et al.) 2013).
To address the challenges of hallucination detection in RAG models, we first leverage mechanis- tic interpretability (Ferrando et al. 2024, Elhage et al. 2021) to decouple the L.M'' utilization of parametric knowledge and external context. Specifically, ly, we conduct an empirical study to explore the internal mechanisms behind hallucination generation RAG scenarios. We introduce two met- rics: the External Context Score, which uses attention heads to quantify the model's utilization on external context, and the Parametric Knowledge Score, which is based on FFNs to evaluate LLM's utilization of parametric knowledge ($ 3.1.. Correlation analysis and causal intervention veals that hallucinations typically occur when Knowledge FENs (from later LLM layers) over-add parametric knowledge into the residual stream, while Copying Heads (attention heads exhibiting
2

---

Published as a conference paper at ICLR 2025
copying behaviours) neglect the necessary external knowledge from retrieved content LLM loses the information attended to by Copying Heads during the generation process ($ 3.2)
Building our causal analysis and mechanistic interpretability, propose ReDeEP (Regressing Decoupled External context score and Parametric knowledge score) for detecting hallucinations in LLM-based RAG, which treat parametric knowledge (P) and external context (E) as covariates to solve the confounding problem (Kahlert et al.) (017) (see Ours in Figure 2 Additionally, we introduce AARF (Add Attention Reduce FFN), which mitigates hallucinations by modulating the
contributions of Knowledge FFNs and Copying Heads in the residual stream ($ A.. Experiments
on RAGTruth and Dolly (AC) confirm that ReDeEP significantly outperforms existing detection methods, while AARF improves the truthfulness of LLaMA models (s
2 BACKGROUND AND RELATED WORKS
2.1 BACKGROUND
Our work is grounded in mechanistic interpretability (Ferrando et al., 2024 nostalgebraist [020] Meng et al. 2022; Elhage et al., (202I). which aims to explain how individual components of lan- guage models (LMs) contribute to predictions. In this study, we focus on transformer decoder-only architectures (GPT-like models) due to their widespread use (Achiam et al. 2023; Dubey et al.) 2024). Transformers use residual connections, where each layer adds information from Attention Heads and
Networky (FFNs) to the hidden state via the residual stream, contributing to the final prediction (Elhage et al. 2021).
Attention Heads: Attention heads play crucial role in contextualizing token representations by selectively attending to previous tokens and updating the residual stream (Ferrando & Voital [022]] Clark et al. 2019: Wu al. 2024). Notably, some attention heads, referred to Copying Heads, have been shown to copy information from one token to another through their OV (output-value) cir- cuits (Elhage et al., 2021). These heads can be identified by analyzing the positive eigenvalues of the OV matrix, which indicate copying behavior. Copying Heads contributes to preserving previously attended tokens in the residual stream, which is critical for external context utilization.
FFNs: FFN layers primarily function as knowledge storage in transformers (Geva et al. 2021) Each FFN layer transforms the hidden state by linearly combining key-value pairs, where keys encode specific knowledge and values represent the output of this knowledge. Research shows that FFNs are critical for the utilization of parametric knowledge within L.MMs, enabling the model to retrieve and integrate stored information effectively for prediction (Dai et al. 2022).
The LogitLens a technique that decodes hidden states æ² directly into the vocabulary distribution using the LayerNorm and the unembedding matrix Wt of the LLM for interpretabil- ity (nostalgebraist, 2020):
Logit Lens:
LogitLens (x') LayerNorm( We
By understanding the roles of Attention Heads (e.g.. Copying Heads) and FFNs, we can better interpret the internal states of L.MSs and identify the mechanisms behind hallucinations in RAG scenarios. Detailed background information can be found in Appendix A
2.2 RELATED WORK
Hallucination of LLMs: LLMs often generate hallucinations content inconsistent with real-world facts or inputs (Huang et al. 2023 Sun et al. 2025). As depicted in Figure 2 although there has been extensive research on detecting hallucinations (Niu al.] 2024: Manakul et al.) 2023 Han et al. 2024). few studies have concentrated on RAG hallucinations, particularly
the internal mechanisms driving these hallucinations. Inspired by research on Knowledge Conflicts (Xu et al.)
2024], our work is the first apply mechanistic interpretability to lens the internal mechanisms of RAG hallucinations from the perspectives of LLM's utilization of external and parametric knowl- edge, leading to more accurate detection method than previous approaches.
Mechanistic Interpretability; Mechanistic interpretability (Ferrando et al., 2024 Elhage et al,
2021) seeks explain the internal processes of L.LMs, enabling the interpretation of how individual
3

---

Published as a conference paper at ICLR 2025
External Context Score:
P(tslt,0)
(Step -Step2)
Step2: Calculate Cosine-Sinnnn Similarity (Eq.
Soltmax
Mean Pooling: (Eq. 3)
Wg
Unenbeddiing Marrix
Last Layer:
Last Layer Hidden State
FFN
Parametric Knowledge
Stepl: Identity
Attention Weights
Attnf-h
Seore
Altendded Tokens 2).
Residual Stream
After FFN
LogitLens
Softmax
FFN
JSD (Eq. 5)
Residual Stream
Refore FFN
LogitLens
Softmax
Embbedding Matrix:
W
Ingut Tokens:
Figure Expanded view of Unrolled LLMs' Attention and FFN blocks, illustrating the calculation processes for the External Context Score and Parametric Knowledge Score.
model components contribute to the final prediction. Our work builds insights into FFN layers, attention heads (Vaswani 2017), residual streams (Elhage et al. 202T), and the logit lens (nostalge-
braist 2020 to analyze the internal mechanisms of LLMs when RAG hallucinations occur.
3 EMPIRICAL STUDY
Our empirical study investigates how hallucinations in RAG models relate the internal states of the LLM. Using Mechanistic Interpretabilityy techniques, we focus on how the LLM's 's use of external context and parametric knowledge contributes to hallucinations.
We conduct experiments on the Llama2-7B-che model (Touvron et al. 2023) using the training set of RAGTruth dataset (Niu et al. 2024). high-quality, manually annotated dataset for RAG hallucinations (Details in Section 5.1p. Each data point in RAGTruth consists of query q, retrieved context c, response r, and the hallucination label h (where 0 is truth and hal- lucination). During generation, the input to the LLM is sequence of tokens including the query = (f1,...,ty fg), retrieved context = (fy+1,..., te), and a partial generated response (lo41....,ke (n))
Experiment Setting:
(f1, 12, fn),
3.1 METRICS FOR LLMs' UTILIZATION OF EXTERNAL CONTEXT AND PARAMETRIC KNOWLEDGE
To quantify how LLMs use external context and parametric knowledge, we design two specific metrics, shown in Figure (a):
External Context: Considering attention heads primarily function to retrieve relevant information
(as discussed in Section 2. measure the LLM's utilization of external context by assessing (1) whether attention heads focus on the correct context, and (2) whether the LLM effectively retains and utilizes this information during generation. To evaluate these aspects, define the following metric based on the semantic difference between the external context attended by attention heads and the generated information:
For the last token t,, the attention weights on the context are a where a is obtained from Equation We select the top 1% tokens with the highest attention scores as attended tokens:
I = arg top m.gee)
L.
(2)
2024 Zhang et al. 2024), with only few Given that attention often shows high sparsity (Zhu et al. tokens capturing most the attention scores, we choose 10 to ensure coverage of high-attention tokens and maintain token diversity.

---

Published as a conference paper at ICLR 2025
Difkmezze Crntest Fallacrntonn
Perowos
Creyrngg Riads Scors
035
025
039
0.5
Lapes
Differenoc Pamenctric Xnowledge Scores (Iblloctneionn Truih)
Peusont's Parmmetric Knowledge Soue Hellociastion Label
Avenge: 1.27
Comdetioo
83
62:
Differmnct in
-0.1
20
Layers
Layen
Figure 4: Relationship Between LLM Utilization of External Context, Parametric Knowledge, and Hallucinations. Top shows the internal mechanism of LLM'' utilization of external context and the occurrence of hallucinations, where the Pearson correlation coefficient between (c) and (a) 0.41, and between (c) and (b) 0.46, indicating correlations among them. Bottom illustrates the internal mechanism of LLM's utilization of parametric knowledge and the occurrence of hallucina- tions, where (d) scaled by le?.
Inspired by (Luo et al. 2024: Chen et al. 2024a), which validated that the hidden states of LLM can serve as token semantic representations, we compute the token-level External Context Score (ECS) based on the cosine-similane tokens and the hidden state token i,
between the mean-pooling of the last layer hidden states of attended
gl.h
e D
(3)
|ell
I
j
jEI!
The response-level ECS is the average of token-level scores:
el.b
(4)
iEr
Parametric Knowledge: Considering FFNs store parametric knowledge, to assess how LLM use Parametric Knowledge (as discussed in Section 2.1 we use the LogitLens to map residual stream states before (i.e., æ; a:.iid.i calculated from Equation 9 and after the FFN layer (i.e., 2i, calculated from Equation TT) to vocabulary distributions. The difference vocabulary distributions represents the parametric knowledge added by the FFN layer to the residual stream, which measured by Jensen-Shannon divergence (JSD), gives the token-level Parametric Knowledge Score (PKS):
P JSD (q(x.)id.)) || q(x%)),
(5)
where g(x:) softmax(LLgitLLon(((( The response-level PKS is the average token-level scores:
P
P/. TET
Although these metrics may not be exact due to the complexity of LLMs, they serve as intuitive proxies that sufficiently aligned with the understanding of existing works to analyze the LLM's use of external context and parametric knowledge in relation to hallucinations, as explored in the following questions.
5

---

Published as a conference paper at ICLR 2025
3.2 EXPERIMENTS
RQ1: Relationship Between LLM Utilization of External Context, Parametric Knowledge, and Hallucinations
(1) We first analyze the relationship between External Context and RAG Hallucinations:
To investigate the relationship between LLM utilization of external context and hallucinations, compare the external context score & between truthful responses (h = 0) and hallucinated responses (h = 1). Specifically, we construct subsets from the dataset: D for hallucinations (h = 1) and D for truthful responses (h = 0), and calculate the external context score difference for different attention heads:
ECS Differences between Truthful and Hallucinated Responses:
4
gt. ET
cl. EH
Σ RED
elh
Σ & PED#
[
IDH
Result: As shown Figure a). in Llama2-7B, 1006 out of 1024 attention heads show higher external context scores on the truthful dataset D compared to the hallucination dataset D (i.e., Ag > 0). Since the external context score represents the LLM's utilization of external con- text through attention heads,
conclude that, at group level, LLMs utilize external context information less than truthful responses when generating hallucinations.
Correlation between ECS and Hallucination: To examine whether neglecting external context relates to RAG hallucinations, we analyzed the Pearson Correlation Coefficient (PCC) between the hallucination label and the external context score across data points in D. Given the expected neg- ative correlation, we inverted the hallucination label h (denoted as h) and used PCC to quantify the relationship between { and external context scores {
Result: As shown in Figure bb), most attention heads show negative correlation between external context scores and hallucination labels h. Since the external context score indicates LLMs' uti- lization of external context, Figure 4[a) and (b) suggest that RAG hallucinations occur when the LLM inadequately leverages external context. Further analysis (Appendix (C) shows that hallucina- tions stem primarily from the LLM losing information attended by attention heads during generation rather than attention heads neglecting external knowledge.
Relation between Copying Heads and Hallucination: We observed that the external context score E/i of certain attention heads correlates strongly with hallucinations, prompting further exploration of these heads' characteristics. Inspired by the Copying Heads concept from Section 2.1 we amined the relationship between these heads and Copying Heads. The calculation process of each attention head's 's copying head score CL. is shown in Appendix (B.
Result: As shown in Figure c), the correlation with the results Figure and (b) indicates that attention heads associated with hallucinations are often Copying Heads (PCC between (c) and (a) 0.41, and (c) and (b) is 0.46). When these Copying Heads have low external context scores, they either fail to attend to the correct external context or, if attended, fail to retain and utilize this infor- mation effectively. This reduces the LLM's copying ability and leads to hallucinations, explaining the negative correlation between these heads' external context scores and the hallucination label h.
(2) Next, we analyze the relationship between Parametric Knowledge and RAG Hallucinations:
PKS Differences between Truth and Hallucination: We compare the Parametric Knowledge Score P across different layers when the LLM generates hallucinations versus truthful responses:
AP' PH -P-
Σ PEDM
Pt D
Σ P/. PED7
D
Result: As shown in Figure dd) parametric knowledge scores in the later layers FFN modules significantly higher in the hallucination dataset compared to the truthful dataset (i.e., AP' > 0). On average, across all layers, hallucination responses exhibit higher parametric knowledge scores than truthful ones.

---

Published as a conference paper at ICLR 2025
Attendinn Heads Interveation
FENa Intervention
Extemal Saars
Parsseer Knowledgt
0.25
Aog NLL Piffrreic
0.30
0.36
0.25
32.0
0.20
0.20
0.16
0.15
Experimntt
Contral
Gestol
(Capstag
Figure (Left) Intervention Result for Attention Heads and FFNs. (Right) External Context Scores and Parametric Knowledge Scores (scaled by 1e²) comparing Truth & Known (where LLM knows the truthful answer) and Hallucination (where LLM unknown about the answer and hallucinated).
Correlation between PKS and Hallucination: To further explore the relationship between para- metric knowledge and hallucinations, we calculate the Pearson correlation between the hallucination label and parametric knowledge scores.
Result: As shown in Figure 4e). parametric knowledge scores in the later layers' FFN modules positively correlated with the ballucination label h and we define the FFN modules from later layers that show strong correlations with hallucinations as Knowledge FFNs. Since these scores represent the amount of parametric knowledge added to the residual stream, we conclude excessive addition of parametric knowledge by these Knowledge FFNs leads to hallucinations. This aligns with findings from LLM early exit studies (Chuang et al. 2024 Schuster et al. 2022): when external context provides sufficient information, shallow layers can generate truthful responses, but over-reliance on parametric knowledge from deeper layers can confuse the model, causing hallucinations.
RQ2: Can the relationship identified in RQ1 be validated from a causal perspective?
To validate the causal relationship between Copying Heads, Knowledge FFN modules, and RAG hallucinations identified in Section 3.2 we employ Causal Intervention (Ferrando al., 2024) by intervening on attention heads and FFNs, where we applied noise to the attention scores and amplified the contributions
FFN modules to the residual stream. We compare the Negative Log-
Likelihood Loss (NLL) (Py (PyTorch 2023) difference for the experimental group (Copying Heads/- Knowledge FFNs) and the control group (Other heads/FFNs) on truthful dataset DT The detailed intervention procedures provided in the Appendix F
Result: As shown Figure (Left), the experimental group's 's impact on NLL difference was sig- nificantly greater than that of the control group for both attention heads and FFN modules. These results, combined with findings from Section 44.3 validate parametric knowledge added by Knowl- edge FFNs and the ability of Copying Heads retrieve relevant external knowledge and LLM effectively utilizes this information during generation, have a significant causal relationship with the RAG hallucinations.
Finding: The occurrence of RAG ballucinations is causally related to two primary factors: (1) while the Copying Heads may occasionally neglect necessary knowledge from the external context, more prominent cause is the LLM losing the Copying Heads retrieved information during the generation process [RQ1- RQ2 $ [. and (2) the Knowledge FFNs within LLM
excessively injecting parametric knowledge into the residual stream RQQ1-2 RQ2
D).
RQ3: Hallucination Behavior Analysis from the Parametric Knowledge Perspective
In this section, we focus on parametric knowledge to analyze hallucination behavior when the LLM either knows or does not know the truthful answer. We conducted comparison experiment using the LLM-known dataset D and the hallucination dataset H (For detailed analysis, Appendix F).
Result: Our results in Figure Right) show that when the LLM knows the truthful answer, Copying Heads more accurately capture and utilize external knowledge, and Knowledge FFNs add less para- metric knowledge to the residual stream compared to hallucination scenarios, which also supports by (Wadhwa al. 2024). The results support leveraging Finding detect RAG hallucination.

---

Published as a conference paper at ICLR 2025
4 METHODS
Building our empirical findings ($ 3). we propose ReDeEP (Regressing Decoupled External Context and Parametric Knowledge) detect hallucinations LLM-based aerieval-augmented generation ($ 4.1 $ 4.2), and AARF (Add Attention Reduce FFN) to mitigate hallucinations by
reweighting the contributions of Knowledge FFNs and Copying Heads to residual stream ($ 4.3
4.1 TOKEN-LEVEL HALLUCINATION DETECTION - REDEEP (TOKEN)
Our empirical study identified RAG hallucinations as stemming from insufficient utilization of ex- ternal context by Copying Heads (set A) and excessive reliance parametric knowledge by Knowl- edge FFNs F). To address the confounding issues shown in Figure we developed a multivari- ate analysis approach that regresses decoupled External Context Score and Parametric Knowledge Score to predict hallucinations (Kahlert et al. 2017). For response r, the hallucination score Hi is:
(r)
1 Σ Ht,,), r
H.(l) Σa- IET
a. Pt-
B.EA 1.EEA
where a and 3 are regression coefficients for external context and parametric knowledge with a, B and this linear regression leverages the high Pearson correlation identified in
4.2 CHUNK-LEVEL HALLUCINATION DETECTION - REDEEP (CHUNK)
As the Token-lever Hallucination Detection computes scores for each token, is computationally expensive and lacks full contextual consideration. To improve efficiency and accuracy, we propose Chunk-level Hallucination Detection as a more suitable method for RAG hallucination detection. Our approach is inspired by the common chunking operation in RAG Fan et al. (2024); FFnardi ct al. (2024), where the retrieved context c and the response r are divided into managcable segments (2)==1 and (Fz) For the chunk-level external context score Ćt. we first calculate chunk-level attention weights w Mean-Pooling weight matrix. then determine the highest attention chunk pairs (). Using embedding model (emb), we compute the external context score for each chunk as follows:
Ihh (A where A the original token-level attention
SLA
1
a
emb(F) emb(@) || emmb(F) emb(E)]l
M
FER
For the chunk-level parametric knowledge score Pl, we sum the token-level parametric knowledge scores for each chunk:
-ー110 M PER
M PH.
P
PL.
LEF
Finally, the Chunk-level Hallucination Detection score (T) is defined as:
Ha(r) = - a Σ B
ELA
lEF
LAKAA
4.3 TRUTHFUL RAG GENERATION AARF
Building on the above methods and the analysis in Appendix C propose Add Attention Reduce FEN (AARF) to reduce RAG hallucinations by intervening on attention heads and FFN modules without updating model parameters. AARF operates in two stages: (1) token-level hallucination detection and (2) reweighting the contributions attention heads and FFN modules to the residual stream.
8

---

Published as a conference paper at ICLR 2025
Table 1: Performance comparisons between ReDeEP and the baselines. The boldface represents the best performance, and the underline represents the second-best.
RAGTruth
Dolly (AC)
LLMs
Categories
Models
AUC
PCC
Ace.
Rec.
F1
AUC
PCC
Acc.
Rec.
SeliChekkGPT
0.5844
0,3584
0.4622
0.5300
0.1897
0.3188
Perplexity
0.5091
-0.0027
0.5333
0,5190
0.6749
0.6825
0,2728
0.6363
0.7719
0.7097
МPR
LN-Entropy
0.1262
0.6555
0.7001
0.7368
Energy
0.5619
0.1119
0.5088
0.5057
0.6657
0.6074
0,2179
0.5656
0.6316
0.6261
0.6233
0.2100
0.5533
0.6622
0.6783
0.3174
0.6262
0.5593
0.6834
Prompt
0.6700
0.7200
0.6720
0.6200
0.3965
0.5476
Llama2- 133BRR)
0.6350
0,7078
0.6750
0.6043
0.5918
0,6616
LMvLM
0.5946
0.7389
0.6473
0.6500
0.7759
0.7200
LLaMA2-78
ECP
ChainPoll
0.6738
0,3563
0.6741
0.7832
0. 0.7066
0.6593
0,3502
0.6200
0,4138
0.5581
0.7290
0.6667
0,6648
0,6500
0.5345
Trulens
0.6510
0.1941
0.6422
0,6814
0.6567
0.7110
0.3198
0.6800
0.5517
0.6667
RefCheck
0.6912
0.2098
0.6467
0.6736
0.6494
0.2494
0.6100
0.3966
0,5412
P(True)
0.7093
0,2360
0.5466
0.5194
0.5313
0.6011
0.1987
0.5444
0.6350
0.6509
EigenScore
0.6045
0.1559
0.5422
0,7469
0.6682
0.6786
0,2428
0.6596
0,7500
0.7241
SEP
0.3555
0.6177
0.6067
SAPLMA
0.7037
0.3188
0.5155
0,5091
0.6726
0.5365
0,0179
0.5600
0.5714
0.7179
0.5667
0.6745
0.5492
0.5800
Ours
ReDeEP(token)
0,7325
0.3979
0.7067
0.6770
0.6986
0.6884
0.3266
0.6464
0,8070
0.7244
ReDeEP(chunkk
0.7458
0.4203
0.6822
0.8097
0.7190
0.7949
0.5136
0.7373
0.8245
0.7833
SeliCheckGPT
0,5844
0,3584
0.4642
0.5300
0.1897
0.3188
Perplexity
0.6749
0.6825
0.2728
0.6363
MPR
LN En-ropy
0.5912
0.1262
0,5600
0,5183
0.6655
0.7001
0,2901
0.6162
0.7368
0.6772
Energy
0.5619
0.1119
0.5088
0,5057
0.6657
0.6074
0.2179
0,5656
0.6316
Rocus
0.7888
0.4444
0.0000
0,6173
0.6977
0.7067
0.1643
0.5900
0,7333
0.6168
Prompt
0.7300
0,7000
0.6899
0.6700
0.4182
0.5823
Llama2- 13B(2
0.7034
0,6839
0,7123
0.5545
0.6319
0.6664
L.vl M
0.5956
0.8357
0,6853
0.6300
0,7273
0.6838
LLaMA2-1BB
ECP
ChainPoll
0.7414
0.4820
0,7342
0,7070
0,4364
RAGAS
0,7541
0.4249
0.7000
0,6763
0.6747
0.6412
0,2840
0.6200
0.4182
0.5476
Trulens
0.6867
0.6521
0,5700
0.3818
0.4941
RefCheck
0.7857
0.4104
0.7200
0,6800
0.7023
0.6626
0,2869
0.5700
0,2545
0.3944
P(True)
0.7998
0.3493
0.6266
0,5980
0.7032
0.6396
0.2009
0.5600
0.6180
0.5739
EigenScore
0.6640
0.2672
0.5267
0.6715
0.6637
0.7214
0.2948
0.6211
0.8181
0.7200
SEP
0,8089
0.5276
0.7288
0.6580
0.7159
0.7098
0.2823
0.6800
0.6545
0.6923
PCE
SAPLMA
0.6529
ITI
0.8051
0.4771
0.6177
0.5519
0.6838
0.5511
0.0646
0.5200
0.5385
0.6712
Ours
ReDeeP(token)
0.8181
0.5478
0.7711
0.7440
0.7494
0.7226
0.3776
0.6465
0.8148
0.7154
ReDeEP(chunk)
0.8244
0,5566
0,7889
0.7198
0.7587
0.8420
0.5902
0,7070
0.8518
0,7603
During the generation of token f, we compute the hallucination score (t,,t,) ((,) S
proceed with the normal output computation fxx (see Equation 12). If He > т, we adjust the weights of Copying Heads A and Knowledge FFN modules F. shifting focus toward external context and reducing reliance on parametric knowledge:
f(x)
Attn
(x(-1) Sn
WU
Σ FFN (xiiid..)) Wo + æ, WU,
02 Attn
x1-1 Sn
if (1,h) E,,
FFN ()) [32 FFN' (amiid)), 'FFN' (xmiid,))
iflET otherwise.
Attn (-)
Ah
(x1-1 Sn
otherwise
Here, a22 is a constant greater than for amplifying attention head contributions, and 32 is constant between (0, 1) for reducing FFN contributions.
EXPERIMENTS
5.1 SETTINGS
Data: We evaluate ReDeEP and AARF on two public RAG hallucination datasets. RAGTruth is the first high-quality, manually annotated RAG hallucination dataset. The data includes three RAG task

---

Published as a conference paper at ICLR 2025
RAGTruth Comparison
Dolly(AC) Comparison
Llmm--1-8
170
183
97
Llmm-1148i
19
66
Liama 128-
160
178
112
Liama 13B
13
Wit:
Win
Llama-2-7B
155
Tie
185
110
Llame 2-7B
28
Tie
54
18
lifo
200
900
400
201
60
100
Number of Examples
Number of Examples
Figure 6: Comparison between LLMs+AARE vs LLMs judged by GPT-4o.
types: Question Answering (QA), Data-to-Text Writing, and News Summarization. Dolly (AC) is dataset with Accurate Context obtained from (Hu et al., 2024), including tasks such as text summarization, closed-QA, and information extraction. More details of the data are in Appendix H
Baselines: We conduct experiments on three variants of LLaMA, including LLaMA2-7B- LLaMA2-13B-Chan For hallucination detection methods, we follow the classification of existing methods as shown in Figure 2 We use (1) Parametric Confounded by External Methods (PCE), (2) External Confounded by Parametric Methods (ECP), and (3) Mixed Parametric and External Methods (MPE). For detailed baselines information, Appendix[] We used AUC, Pearson Corre- lation Coefficient (PCC), Accuracy (Acc.), Recall (Rec.), and F1 as evaluation metrics for detection accuracy. Implementation details are provided in Appendix
and
5.2 EXPERIMENTS
RAG Hallucination Detection: As shown in T ReDeEP consistently improves performance across two datasets, various backbone methods, and different metrics, validating its effectiveness in detecting RAG hallueinations. ReDeEP outperforms MPE methods, demonstrating that mecha- nistic interpretability effectively decouples the LLM's utilization of external context and parametric knowledge, enabling more accurate detection of RAG hallucinations. Additionally, ReDeEP sur- passes both ECP and PCE methods by incorporating both the External Context Score and Para- metric Knowledge Score as covariates in a multivariate regression approach, effectively addressing the confounding problem. ReDeeFP(chunk) generally outperforms ReDeFE((token) in most metrics, suggesting that chunk-level processing better preserves semantic integrity and improves detection performance. Further support for ReDeEP's effectiveness is provided by the ablation study in Ap- pendix K while the efficiency analysis in Appendix L confirms that ReDeEP achieves comparable time efficienyy the most efficient baselines.
Truthful RAG Generation: We evaluated our hallucination reduction method AARF on both the RAGTruth and Dolly (AC) datasets, using GPT-4-o for automatic evaluation assess the truth- fulness (Prompt details can be found in Appendix [.. Pairwise comparisons rated by GPT-4-o are shown in Figure demonstrating that AARF can reduce hallucinations certain extent compared to the baseline model. These results validate the effectiveness of our intervention experiments and confirm the findings presented RQ2 of Section 3.2
CONCLUSION
Detecting RAG hallucinations is critical for enhaneing the security and reliability of RAG systems. In this work, we introduced ReDeEP, a novel method that detects RAG hallucinations by analyzing LLMs' utilization of parametric knowledge and external context. Our empirical study shows that hallucinations arise from insufficient utilization of external context by Copying Heads and over- reliance on parametric knowledge by Knowledge FFN modules. These insights also guided the development of interventions to reduce hallucinations without updating model parameters. ReDeEP demonstrates significant performance improvements across the LLaMA family and RAG hallucina- tion benchmarks, outperforming existing detection methods.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was funded by the National Key R&D Program of China (2023YF11008004) the Na- tional Natural Science Foundation of China (62472426), fund for building world-class universities (disciplines) of Renmin University of China. Work partially done at Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education. Supported by Kuaishou Technology. Supported by the Outstanding Innovative Talents Cultivation Funded Pro- grams 2025 of Renmin University China.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 967-976 2023.
Howard E Bell. Gershgorin's 's theorem and the zeros of polynomials. The American Mathematical Monthly, 72(3):292-295, 1965.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3588):1798-1888, 2013.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMSs internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, 2024a. httthttts///ppenreeee
net/forum?id- 2j12nz10b2
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. Inside: Lims' internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, 2024b.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality large language models. In The Twelfth International Conference on Learning Representations, 2024. TtLh:tts:: httphst:/ppeeeeiiees met/forumm www.foowuunddddhooooooeooe Thooorummid-T
Darya Chyzhyk, Gaël Varoquaux, Michael Milham, and Bertrand Thirion. How to remove control confounds in predictive models, with applications to brain biomarkers. GigaScience, 1 11giia0014, 2022.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look an analysis of BERT's attention. In Tal Linzen, Grzegorz Chrupata, Yonatan Belinkov, and Dieuwke Hupkes (eds.), Proceedings of the 2019 ACI. Workshop BlackboxNI.P Analyzing and Interpreting Neural Networks for NLP, pp. 276-2866666662212222 Florence, Italy, August 2019, Association for Computational Linguistics. doi: 10. MEI18865////// 19-4828. URL sttps::/aclantthen org /11 4828
Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. Lm Im: Detecting factual errors via cross examination. arXiv preprint arXiv:2305. 132881, 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8493-8502, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2Xivv2240.21778. 2024.
11

---

Published as a conference paper at ICLR 2025
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly. Nova DasSarma, Dawn Drain, Deep Gan- guli, Zac Hatfield-Dodde Danny Hernandez. Andy Jones. Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. www.eiiicccti....
URL
index. html
Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. RAGAs: Automated eval- uation of retrieval augmented generation. In Nikolaos Aletras and Orphee De Clereq (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pp. 158. St. Julians, Malta, March 2024. Association for Computational Linguistics. WtLLttps::// http:ppp//c..anoooooooo/o/o//// org/ 2024 eacl c demo
Wengi Fan, Yujuan Ding, Liangho Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting Imss: Towards retrieval-augmente large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6491-6501, 2024.
Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale. arXiv preprint arXiv:2403.00824 2024.
Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. A primer the inner workings of transformer- based language models. arXiv preprint mrXiv:2455.00200 2024.
Paulo Finardi, Leonardo Avila, Rodrigo Castaldoni, Pedro Gengo, Celio Larcher, Marcos Piau, Pablo Costa, and Vinicius Caridá. The chronicles of rag: The retriever, the chunk and the genera- tor. arviv preprint Aiv:22240 .07883, 2024.
Robert Friel and Atindriyo Sanyal. Chainpoll: A high efficacy method for 11m hallucination detec- tion. arXiv preprint arXiv:23 10. 10.11444, 2023.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval- auggenttd generation for large language models: A survey. arXiv preprint Ar 10997, 2023.
Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.
2021.
Jiatong Han, Jannik Kossen, Muhammed Razzak, Lisa Schut, Shreshth A Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in 1lms. In ICML 2024 Workshop on Foundation Models in the Wild, 2024.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36. 2024.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Xiangkun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, and Zheng Zhang. Refchecker: Reference-based grainer hallucination checker and benchmark for large language models. arXiv preprint arXiv:2405. 14486, 2024.
Lei Huang, Weijiang Yu. Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng. Bing Qin, al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint Xi 1.05232, 2023.
Jajje Jin. Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. Flashrag: A modular toolkit for efficient retrieval-augmented generation research. arXiv preprint arXiv:2405.13576 2024.
12

---

Published as a conference paper at ICLR 2025
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield- Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language mod- els (mostly) know what they know. arXiv preprint arXiv:2207. 00221, 2022.
Johnny Kahlert, Sigrid Bjerge Gribsholt, Henrik Gammelager, Olaf M Dekkers, and George Luta. Control of confounding in the analysis phase overview for clinicians. Clinical epidemiology, pp. 195-204, 2017.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Incorporating residual and normalization layers into analysis masked language models. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 4547-4568, 2021.
Evgenii Kortukov, Alexander Rubinstein, Elisa Nguyen, and Scong Joon Oh. Studying large lan- guage model behaviors under context-memory conflicts with real documents. In First Conference on Language Modeling.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466 0663-66,222 2019.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenherg. Inference-time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detec- tion. Advances in neural information processing systems, 33:21464 21475, 2020.
Kun Luo, Minghao Qin, Zheng Liu, Shitao Xiao, Jun Zhao, and Kang Liu. Large language models as foundations for next-gen dense retrieval: A comprehensive empirical assessment, 2024. URL
12194
Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D Manning, and Daniel E
Ho. Hallucination-Fiee free? assessing the reliability of leading ai legal research tools. arXiv preprint
arXiv:2405.20362, 2024.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650, 2020.
Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box halluci- nation detection for generative large language models. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 9004 9017, 2023.
Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy sup- pression: Comprehensively understanding an attention head. arXiv preprint arXiv:2310.04625, 2023.
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 15:17359-17372. 2022.
Leland Gerson Neuberg. Causality: models, reasoning, and inference, by judea pearl, cambridge university press, 2000. Econometric Theory, 19(4):675-685 2003.
Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. RAGTruth: A hallucination corpus for developing trustworthy retrieval- augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, August 2024. As- sociation for Computational Linguistics.
nostalgebraist.
Interpreting GPT: the logit lens. AI Alignment Forum, 2020. http://www...eppee////////////aa
URL
13

---

Published as a conference paper at ICLR 2025
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint 2tt.::22209.1199.... 2022.
J Pearl. Causality, Cambridge university press, 2009.
PyTorch. torch.nn.nllloss. https://pytorch.
http//////////aaa////////a/////a nn nn. N...LSS html 2023. Accessed: 2023-09-20.
P Rajpurkar. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint
arXiv:1606.05250, 2016.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshninarayanan, and Peter Liu. models. In The Eleventh International Conference on Learning Representations, 2022.
detection and selective generation for conditional language
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022. URL https::/pppnreviiew net/ Sarte/oorunte Eorum? wtt.:rrmnnddd1L1113333333
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmenta- tion reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3784-3803, 2021.
Elizabeth Stuart. Matching methods for causal inference: A review and a look forward. Statistical science: review journal of the Institute of Mathematical Statistics, 25(1):1, 2010.
ZhongXiang Sun, Zihua Si, Xiaoxue Zang, Kai Zheng, Yang Song, Xiao Zhang, and Jun Xu. Largepig for hallucination-ffee query generation: Your large language model is secretly a pointer generator. In THE WEB CONFERENCE 2025, 2025. URL /ttps::/cpenrrview forumm?dd-yywd0eeyar
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09220
2023.
Trulens. Trulens: Evaluate and track I1m applications, 2024. URL org/
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
HP Vinutha, B Poornima, and BM Sagar. Detection of outliers using interquartile range technique from intrusion dataset. In Information and decision sciences: Proceedings of the 6th international conference on ficta, pp. 511-518. Springer, 2018
Hitesh Wadhwa, Rahul Seetharaman, Somyaa Aggarwal, Reshmi Ghosh, Samyadeep Basu, Soundararajan Srinivasan, Wenlong Zhao, Shreyas Chaudhari, and Ehsan Aghazadeh. From rags to rich parameters: Probing how language models utilize externl knowledge over parametric information for factual queries. arXiv preprint arXiv:2406. 12824. 2024.
Wenhao WU, Yizhong Wang, Guangxuan Xiao, Hao Peng. and Yao Fu. Retrieval head mechanisti- cally explains long-context factuality. arXiv preprint arXiv:2404.15574 2024.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:2309.07597, 2023.
Rongwu Xu, Zehan Qi, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. Knowledge conflicts for Ims: survey. arXiv preprint arXiv:2403.088: .00000000331, 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, 2018
14

---

Published as a conference paper at ICLR 2025
Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, and Luoyi Fu. Enhancing uncertainty-basee hallucination detection with stronger focus. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 915-932, 2023.
Zhenyu Zhang, Ying Sheng. Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient gen- erative inference of large language models. Advances Neural Information Processing Systems, 36, 2024.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Yixin Cao, Yang Feng, and Deyi Xiong (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 400-410, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 1116653/1/2222...dddddmmsss https://aclanthologg. org/ 2024. nell-emos...
URL
Hanzhang Zhou, Zijian Feng, Zixiao Zhu, Junlang Qian, and Kezhi Mao. Unibias: Unveil- ing and mitigating IIm bias through internal attention and ffn manipulation. arXiv preprint arXiv:2405.2061 2024.
Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng, Xin Lv, Huanqi Cao, Xiao Chuanfu, Xingcheng Zhang, et al. Near-lossless acceleration of long context Ilm inference with adaptive structured sparse attention. arXiv preprint arXiv:2406. 15486, 2024.
A FULL BACKGROUND ON ATTENTION HEADS, FFNS, AND LOGIT LENS
The theoretical foundation of our work is grounded in research mechanistic interpretability (Fer- rando et al.) 2024 nostalgebraist, 2020 Meng et al. 2022 Elhage et al., 2021), which seeks to explain the internal processes of language models (LMs) by interpreting how individual model com- ponents contribute to the final prediction. In this study, we focus on the transformer decoder-only architecture (also known GPT-like) due to its widespread success and popularity (Achiam al.) 2023 Dubey et al. 2024). A decoder-only model f consists of L layers and operates on a sequence of embeddings x (x1, 22, ........ ... F.). which represent the tokens (f1, .2,....,a,, fn)- Each embed- ding æ E R2 is row vector corresponding
row of the embedding matrix We E [ where V denotes the model's vocabulary. The sequence is represented as a matrix X E Baxd with the embeddings stacked as rows.
We interpret Transformers through the perspective of the residual stream (Elhage et al. 2021 Due to the residual connections in Transformers, each layer / takes a hidden state X as input and adds information obtained from its Attention Heads and Feed-Forward Networks (FFNs) to the hidden state via the residual connection. In this context, the hidden state acts as a residual stream passed through the layers, with each attention and FFN contributing the final prediction by adding information to the residual stream, resulting in the Residual Stream States. The final layer's residual stream state is then projected into the vocabulary space using the Unembedding Matrix Wo [dx[V]] and normalized via the softmax function to produce probability distribution over the vocabulary, from which a new token is sampled.
The background knowledge for interpreting the contributions of each FFN and attention head to the model's 's prediction is outlined as follows:
Attention Heads: Attention is crucial in Transformers for contextualizing token representations
across layers. Each attention head selectively attends to previous positions, gathers information, and updates the current residual stream (Ferrando & Voital 2024 Clark et al., 2019: Wu et al.) 2024).
The output of an attention layer is the sum of its attention heads. For each attention head:
Attn 1.1
(x(=))=
(x(-1)
Σ J51
Lh w di
whh
Σ 151
1h with a
15

---

Published as a conference paper at ICLR 2025
where the learnable weight matrices W Rd and Woth RRdx are combined into the OV matrix W w E Rdxd, also referred to the OV (output-value) eircuit (Kobayashi
et al. 2021
The attention weights for the current query to the previous tokens are computed as:
w-hh
X Si
w.hk) K
w X
X!
2;
d softmax
softmax
(8)
dR
v dp
where Wi E Raxds and Wit Rdxdd combine form the QK (query-key) circuit (Elhage et al. 2021), W QK = W E Rdxd The QK circuit computes the attention weights, de- termining the positions that should be attended, while the OV circuit transfers and transforms the information from the attended positions into the current residual stream. The attention block output the sum of individual attention heads, which is then added back into the residual stream:
mid,l 2;
Σ At
(X(-1) Si
Previous research has shown that the primary role of attention layers in LLMs is implementing algorithms (Olsson et al., 2022; Ferrando et al.) 2024). For instance, several attention heads in Transformer LMs have OV matrices exhibiting copying behavior; we refer to these heads as Copy- ing Heads. Elhage et al. (202)) propose using the number of positive real eigenvalues of the full OV circuit matrix W g Wov We summary statistic for detecting Copying Heads. Positive eigenval- ues indicate that a linear combination of tokens contributes to an increase in the linear combination of logits of the same tokens.
FFN: Research has shown that the functionality FFN layers lies in storing knowledge (Geva et al. 2021]. Transformer FFN layers can be represented linear combination of vectors. Specifically,
for an input vector x midd E Fd drawn from the residual stream states, with FFN parameter matrices K' V [Rd xd. the FFN output can be expressed as:
dm
d. Σ mfo!
FFN
2:
(..mid.)) = (x.iidd(<k)))) v Σ
midd k!) o
x xi mid,
+ FFN (2:miid.))
(10)
where g is the activation function. Thus, the FFN layer can be viewed as a linear combination mid,l and the key vector ky produces the coefficient m/. which
of vectors: the multiplication of æ; weights the corresponding value vector v!-
The LegitLens a technique that decodes hidden states æ' directly into the vocabulary distribution using the LayerNorm and the unembedding matrix of the LLM for interpretability (nos- talgebraist 2020
Logit Lens:
LogitLens (a') LayerNorm(a (Wu
(11)
This approach has been validated in various studies as effective method for interpreting LLMs' weight matrices or hidden states (Hanna et al. 2024 Zhhu et al. 2024 Yu et al.) 2023).
The final output logits of the LLM can be expressed as:
L
H
f(x) =
Σ Σ Attn
(xl-1 Sn
We +
Σ FFN (2r Wu + x Wy
(12)
1=1
16

---

Published as a conference paper at ICLR 2025
B CALCULATION OF COPYING HEADS SCORE
The traditional method for detecting Copying Heads (Elhage et al. 2021 involves calculating the cigenvalues of the matrix M Wy Wy Wy and assessing the proportion positive eigenvalues, where WE is the Embedding Matrix, Wov is the OV Matrix and Wo is the Unembedding Ma- trix. However, the original Copying Heads identification method proposed by |Elhage et al. (202I) requires calculating eigenvalues of large matrices and using the ratio of positive eigenvalues to de- termine Copying Heads, which becomes computationally expensive for models with large hidden sizes. We propose using the trace of the matrix to estimate the ratio of positive eigenvalues, cali- brated with the Gershgorin circle theorem (Bell 1965), to obtain each attention head's 's copying head score CtA To simplify this, we estimate the proportion using the trace of the matrix and refine this estimation using the Gershgorin Circle Theorem (Bell 1965).
1. Trace-Based Estimation: The trace tr(M) provides an indication of the distribution of positive and negative cigenvalues:
A positive trace suggests more positive than negative eigenvalues.
A negative trace suggests more negative than positive eigenvalues.
A trace near zero suggests balance between positive and negative eigenvalues.
2. Gershgorin Circle Theorem:
To enhance the accuracy of our trace-based estimation,
employ the Gershgorin Circle Theorem, which provides an approximation of the eigenvalue distribution. For any n n matrix M [a.y] each eigenvalue of M lies within least one Gershgorin disk D,:
D; (2 e C: anl S R,)
where R, = Ljfts (asyl- Each disk is centered at the diagonal element Og with a radius determined by the sum the absolute values of the off-diagonal elements in the row. This theorem helps identify the regions in the complex plane where the eigenvalues are likely to be found, allowing us to approximate their distribution without direct computation.
3. -oR-Based Outlier Detection for Boundary Points:
Collect boundary points + ag and au from each Gershgorin disk.
Calculate the first (Q1) and third quartiles (Q3) of these points, then determine the IQR as IQR = Q3 Q1 (Vinutha al. 2018).
Identify outliers using bounds Q1 - 1.5 x IQR and Q3 + 1.5 x IQR, counting points outside these limits.
4. Copying Head Score Calculation: Combine rankings based on the number of detected outliers (ascending order) and the absolute value of the trace (descending order). Summing these ranks gives the Copying Head Score CLA.A reflecting the head's tendency to behave as Copying Head.
C DIVE INTO THE RATIONALE BEHIND THE EXTERNAL CONTEXT SCORE
In Section 3.1 we designed the external context score to measure two aspects: (1) whether the attention heads focus on the correct external context, and (2) if attending the correct context, whether the LLM can effectively retain and utilize this information during the generation process. In this section, we aim explore whether low external context score caused by attention heads focusing on the incorrect externl context or by the LLM losing the information attended by the attention heads during the generation process.
To address this, we firstly conducted experiments on the Llama2-7B model using the RAGTruth dataset. Specifically, in our validation experiments, we selected data from RAGTruth where LLaMA2-7B-CHI exhibited hallucinations. Using LangChain, widely-used open-source toolkit, we applied the ent spans. We then calculated whether the attention module's 's attended span (by mean pooling the attention scores and selecting the input span with the highest score) during the generation halluci- nation spans could identify the hallucination spans in the response. If the attention head successfully
to segment the input retrieved document into differ-
17

---

Published as a conference paper at ICLR 2025
identified the correct span, this indicates that the attention mechanism focused on the correct exter- nal context, but the LLM did not effectively retain and utilize this information during the generation process. This evaluation was based on GPT-4-o (from OpenAl) using the following prompt:
Prompt: {external context + query)
Respond: (response)
Conflict Span: [Conflict Span)
Conflict Type: (Conflict Type}
Reason: [Reason]
Given the following context information: (Attend Span), this support the existence of a conflict in the response? Please answer with "Yes" or "No" and give the reason on the newline.
Table 2: Proportion of data where Llama2-7B attention heads attend to the information that supports conflict.
Attention heads attend
Attention heads mis-attend
77.5%
22.5%
From the results in Table 2 we can see that low external context score is mostly due the LLM losing the information attended by the attention heads during the generation process. In most cases, the attention heads correctly attend to the appropriate external context. This phenomena may be due to the presence of some Copy suppression heads in the LLM (McDougall et al., (023), which may incorrectly suppress the information attended by the Copying head, resulting in the LM losing the information attended by the attention heads during the generation process. This also validates the feasibility of our proposed AARF method in reducing hallucinations by increasing the output of copying heads in the residual stream.
In addition, we aim to verify whether the attention module's attended span during the generation of hallucinated span contains the information needed in the truthful answer. Our verification data is sourced from RAGTruth, where LLaMA2-7B Chat exhibited hallucinations. For the same questions, extracted truthful answers provided by other models in the RAGTruth dataset as the ground truth. We filtered out validation data where no truthful answer exists and instances with PKS exceeding the overall average, minimize the influence of PKS. This evaluation was conducted using GPT-4-o (from OpenAl) with the following prompt:
Prompt: (external context query)
Truthful Respond: {Truthful Respond} Given the following context information: {Attend Span), check it includes relevant infor- mation found in the truthful answer.
Please answer with "Yes" or "No" and provide the reason on a new line.
Table Proportion of data where Llama2-7B attention heads attend the correct information.
Attention heads attend truthful information Attention heads mis-attend truthful information
65.2 %
34.8%
The experimental results shown in Table demonstrate that in most cases when hallucinations oceur during generation, the attention heads correctly attend the appropriate external context. This indi- cates that a low external context score is primarily due to the LLM losing the information attended by the attention heads during the generation process.
18

---

Published as a conference paper at ICLR 2025
Table 4: Parametric knowledge bias comparison of Llama2 7B with different layer modifications across datasets." denotes a divide-by-zent error during computation.
Dutaset
Liamaž 7B
Remove (32nd) Layer
16th Layer
Remove 1st FPN
QA Yang et al. 2018
+1.44%
+0,52% (200.92%)
-1.6%% (A3.03%))
+1 +3.33% (10.111)) +3.30% (A1.80%
SQuAD (Rajpurkar 2076
+1.24% (A3.85%)
+2.18%
NQ Kwiatkowski al. 2019
+4.45%
+2.55% (A1 (90%))
+2.16% (A2.299%)
D JUSTIFICATION FOR PARAMETRIC KNOWLEDGE SCORE
To further verify whether the Knowledge FFNs within LLMs excessively injecting parametric knowledge into the residual stream lead to hallucinations, we follow the experimental settings of Table 4 in (Kortukov al.).. Using the Llama2 7B model, we examined whether removing FFN layers reduces parametric knowledge bias. Specifically, in our experiments, we removed the Last (32nd) Layer FFN, the 16th Layer FFN, and the 1st Layer FFN, observing whether the parametric knowledge bias (Kortukov et al.) will decrease. We sampled the first 1 1,000 examples from the RAG dataset, which common setting RAG scenarios (Jin al.. 2024).
The experimental results are shown in Table where larger values indicate greater parametric knowledge bias. The results demonstrate that removing either lower-layer or -ayee---peerr FFNs reduces parametric knowledge bias, with mid-to-upper-layer FFN removal being more effec- tive. This aligns with our finding the paper that Knowledge FFNs primarily inject parametric knowledge in the mid-to-upper layers. On the HotpotQA dataset, removing the 16th Layer FFN from Llama2 7B even resulted in negative parametric knowledge bias. This further supports our conclusion that. in RAG scenarios equipped with accurate and relevant external knowledge, the Knowledge FFNs within LLMs excessively injecting parametric knowledge into the residual stream can lead to hallucinations.
E DETAILED INTERVENTION PROCEDURES
In this section, provide the details of the intervention experiments for RQ2 from Section 3.2 Our intervention experiments were conducted on the subset of RAGTruth data which contains fruthful answers. The NLL difference was used to measure the change in the negative log-likelibee lilkiiiihhood of the model's 's output for the truthful answer before and after the intervention. the NLL difference in the experimental group is larger than that in the control group, it indicates that the intervention in the experimental group is more likely to cause the model to generate non-truthful answers (i.e., hallucinated answers).
Attention Heads Intervention: As described in Figure 3 (b), we applied noise to the attention scores (.h of the experimental group to evaluate their impact on hallucinations. Specifically, the attention scores were sampled from a standard normal distribution:
Ih d ~ NO0, 1), a;
(L.b) softmax (a (1
(13)
This approach simulates the removal of meaningful attention patterns, allowing us to assess how Copying Heads' focus on external context impacts hallucinations.
FFN Modules Intervention: To investigate the role of Knowledge FFNs in hallucinations, we am- plified the effect of the FFN modules by increasing their contribution to the residual stream tenfold (k = 10). This intervention highlights the influence of parametric knowledge on the generation process.
To reduce potential biases arising from the position of transformer layers or heads, we applied causal matching (Stuari 2010)). For attention heads, we matched the top 32 Copy- ing Heads with the nearest non-experimental heads within the same layer. Similarly, we matched the top 5 Knowledge FFNs, identified as being most related to hallucinations, with the nearest FFN modules in adjacent layers. This matching process ensured that the comparison between the ex- perimental and control groups was fair and focused on the specific roles of Copying Heads and Knowledge FFNs in hallucination generation.
Causal Matching:
19

---

Published as a conference paper at ICLR 2025
F DETAILED ANALYSIS OF RQ3: HALLUCINATION BEHAVIOR ANALYSIS FROM PARAMETRIC KNOWLEDGE VIEW
In the experiments (RQ1 and RQ2 in Section 3.2) we analyzed the relationship between LLM- generated responses and the internal states of the model that lead to hallucinations. In this section, we shift our focus to parametric knowledge (since our setting assumes the external context is correct, there is no need for separate analysis from the external context perspective) to examine the two scenarios where the LLM's internal memory either knows or does not know the truthful answer to the query. This analysis aims validate whether our previous findings about the connection between LLM ballucinations and internal states are reasonable.
LLM Parametric Knowledge Unknown Truthful Answer When the LLM's parametric knowl- edge does not contain the truthful response, the model must rely on the retrieved context to generate a truthful answer. In this scenario, the Knowledge FFN module may over-add parametric knowledge to the residual stream, while Copying Heads may fail to attend to the correct external context or lose the attended external information during the generation process, leading to hallucinations. This phe- nomenon is consistent with our earlier findings, where Copying Heads neglect external context and Knowledge FFN modules excessively add parametric knowledge to the residual stream.
LLM Parametric Knowledge Known Truthful Answer When the LLM's parametric knowledge contains the truthful answer, RAG responses are typically truthful. To verify whether the model, in this case, relies more external knowledge and less on parametric knowledge compared to when hallucinations occur, we designed a validation experiment. Specifically, we allowed the LLM to generate responses directly the truthful dataset DD without relying on retrieved documents to determine if the LLM could independently produce accurate answers. We used GPT-4-o (Achiam al., 2023). along with the original truthful response, to evaluate whether the LLM ALM-generated an- swers matched the expected ones, thus assessing if the LLM's parametric knowledge could correctly answer independently (see prompt in Appendix G These correct responses form the I.LM- dataset Dr We then analyzed the differences in external context scores and parametric knowledge scores for the LLM across DT and DH.
Result: As shown in Figure 5 (Right), when the LLM's parametric knowledge knows the truth- ful answer, we observe that Copying Heads can more accurately capture external knowledge and effectively retain and utilize this information during the generation process, showing more stable performance compared to their behavior in the hallucination dataset. Although in scenarios where the LLM knows the truthful answer, the Knowledge FFN layers add less parametric knowledge to the residual stream than the hallucination dataset, this supports our earlier finding of the negative impact of excessive utilization of parametric knowledge by the Knowledge FFN module.
G PROMPT FOR EVALUATING PARAMETRIC KNOWLEDGE
You are an AI evaluator tasked with assessing the accuracy and relevance of an Al-generated response. Here are the details:
1. Al-generated response:
[LLM-GennrateddRessponn
2. Expected response: {Ground Truth)
3. Query that prompted the response: [Query)
Evaluate if the Al-generated response accurately and comprchensively addresses the query and aligns with the expected response, If the Al-generated response aligns well with the expected response, output "yes". If it does not align, output "no". Only output "yes" or "no".
To assess whether the LLM's parametric knowledge alone could provide accurate answers indepen- dently of retrieved documents, we used the following prompt in our validation experiment. The aim was to evaluate if the LLM-generate responses the truthful dataset DT matched the expected truthful answers. The prompt was designed to engage GPT-4-o (Achiam et al. 2023) as an evaluator to compare the L.M''s responses with the ground truth.
20

---

Published as a conference paper at ICLR 2025
H DETAILS ABOUT RAG HALLUCINATION DATASETS
RAGTruth:
The RAGTruth (Niu et al. 2024) dataset is the first high-quality, manually annotated RAG hallucination dataset. It is divided into three task types: Question Answering (QA), Data- to-Text Writing, and News Summarization. However, the dataset does not include hallucination annotations for responses generated by Llama3-8B. To address this, we employed three annotators with graduate-level qualifications to manually evaluate the presence of hallucinations in different LLM RAG responses. Each response was carefully assessed to determine whether it contained any hallucinations based on the accuracy and relevance of the retrieved and generated content.
Dolly (AC): The Dolly (AC) dataset is sourced from (Hu et al., 2024) and consists of tasks such text summarization, closed-QA, and information extraction. Similar to RAGTruth, Dolly (AC) lacks hallucination annotations for responses generated by certain LLMs, particularly those without access accurate context. To fill this gap, the same team of three qualified annotators tasked with manually evaluating the RAG responses to determine if they contained hallucinations, focusing on the alignment between the generated responses and the external context.
In both cases, the manual annotation process involved cross-verilying the generated content with the provided external context detect discrepancies or factual inconsistencies that would indicate hallucinations. We consider some data that fails to gain consensus as low- low-quality data which can be filtered out.
I DETAILS ABOUT BASELINE MODELS
This section provides details on the baseline models and hallucination detection methods used in our experiments. We categorize the methods into three groups based on their approach to leveraging parametric knowledge and external context: Parametric Confounded by External (PCE), External Confounded by Parametric (ECP), and Mixed Parametric and External (MPE).
(1) Parametric Confounded by External (PCE):
EigenScore measures the semantic consistency in the embedding space. Higher EigenScores suggest a higher likelihood of hallucinations, as they indicate greater semantic divergence (Chen et al. 2024b).
EigenScore:
SEP: SEP (Semantic Entropy Probe) uses linear probes trained on the hidden states of LLMs to detect hallucinations by analyzing the semantic entropy of the tokens before gen- eration (Han et al.) 2024).
SAPLMA: SAPLMA trains classifier LLM activation values to detect hallucinations. It captures internal signals from the LLM's hidden layers to identify when the model might generate hallucination (Azaria & Mitchell 2023).
ITI: Inference- Intervention (ITI) analyzes attention head activations and uses binary classifier to predict hallucinations by studying the relationship between heads and task performance (Li et al. 2024).
(2) External Confounded by Parametric (ECP):
This method the prompts provided in the RAGTruth (Nuu et al. 2024) to evaluate whether the L.L.-generated responses are hallucinations by comparing them against the ground truth using GPT-4-o-mini (Niu et al.) 2024).
Prompt:
Llama2-13B(LR) Fine-tune the Llama2-13B model using LoRA (Hu al.) with the training data from RAGTruth for hallucination detection. The LoRA parameter settings are based on the paper (Zheng et al. 2024).
LMvLM: LMvLM employs a multi-turn interaction between two language models(GPT- 4-o-mini vs. Backbone LLM) to discover inconsistencies by having them cross-examine each other's responses (Cohen et al. 2023).
ChainPoll: ChainPoll uses GPT-4-o-mini to determine completion contains halluci- nations through carefully designed prompt. The evaluation is repeated multiple times
21

---

Published as a conference paper at ICLR 2025
(typically five), and the final hallucination score is calculated as the ratio of "yes" answers to the total number of responses (Friel & Sanyal 2023).
RAGAS: RAGAS checks the faithfulness of the generated response by breaking down
sentences into shorter assertions and verifying each against the context, using GPT-4-0- mini to calculate a faithfulness score as the ratio of supported statements (Es et al. 2024).
Trulens assesses the overlap of information between the context and the generated response using GPT-4-o-mini, assigning groundedness score between 0 and 10 based on the degree of overlap (Trulens 2024).
Trulens:
RefCheck: Similar to RAGAS, RefCheck extracts knowledge graphs from the generated responses and evaluates whether the knowledge graphs align with the external context (Hu et al. 2024).
P(True): P(True) measures the uncertainty of the generated claim by querying the LLM itself on the truthfulness of its generated response. 'he confidence score calculated as the probability of the first token being "True" (Kadavath al. 2022)
(3) Mixed Parametric and External (MPE):
SelfCheckGPT: SeIPCheckGPT uses zero-resource, sampling- based approach where multiple reference responses are checked by GPT-4-o-mini for consistency with the gener- ated answer (Manakul al. 2023).
LN-Entropy:
Length- Normali-or Entropy measures sequence-level uncertainty across multiple generations, using entropy normalized by sequence length to detect hallucina-
tions (Malinin & Gales 2020).
Energy-based OOD detection identifies hallucinations by analyzing the uncer- tainty in the generated response using energy functions. Higher energy suggests higher likelihood of hallucinations (Liu et al. 2020).
Energy:
Focus: Focus enhances uncertainty- -baed hallucination detection by focusing on key infor- mative tokens, preceding words, and token properties, simulating human factuality check- ing Zhhng et al. 2023).
Perplexity: This method uses the perplexity the L.M--genertted response to detect hal- lucinations. A higher perplexity indicates greater uncertainty and higher likelihood of hallucinations (Ren et al. 2022).
J IMPLEMENTATION DETAILS.
We run all the experiments on machines equipped with NVIDIA V100 GPUs and 52-core Intel(R) Xeon(R) Gold 6230R CPUs 2. 10G We utilize the Huggingface Transformers package to conduct experiments. During the decoding of responses from the language models, we employ greedy search to generate responses. The remaining parameters follow the models' default settings. For RAGTruth, use the validation set to select the hyperparameters. For Dolly (AC), we two-fold validation to select the hyperparameters. For the baselines, we perform hyperparameter tuning within the range provided by the original works. For experiments with randomness, them three times to calculate the average. For ReDeEP(Chunk) on Dolly (AC), on Llama2-7B, we 1.6, as described in Section 3 On Llama2-13B. we select the top- scoring Copying Head and top-3 3 FFN layers with and
select the top-7 scoring Copying Head and top-3 FFN layers with a 1 and
0.2. On Llama3-8B. we select the t scoring Copying Head and top-1 FFN layers with a = 1 and B = 0.1, as described in Section For ReDeEP(Chunk) on RAGTruth, on Llama2- 7B, we select the top-3 scoring Copying Head and top-4 FFN layers with a = and , 0.6. On A.amm2--13B we select the top-9 scoring Copying Head and top-3 FFN layers with a = 1 and 1.8. On Llama3-8B, we select the top-2 scoring Copying Head and top-5 FFN layers with Dolly (AC), on Llama2-7B, we select the top-4 scoring
and B 1.2. For ReDeEP(Token) Copying Head and top-3 FEN layers with a = 1 and 3 = 0.2, as described in Section On Llama2- 13B, select the top-4 scoring Copying Head and top-5 FFN layers with
and 8 0.6.
On Llama3-8B, we select the top-1 scoring Copying Head and top-1 1 FFN layers with a = 1 and = 0.1, described in Section 3 For ReDeeEP((Tkke) RAGTruth, on Llama2-7B, select
22

---

Published as a conference paper at ICLR 2025
the top-1 scoring Copying Head and top-10 FFN layers with 1 and B 0.2. On Llama2-
13B, we select the top-2 scoring Copying Head and top-17 FFN layers with a = 1 and = 0.6. select the top-3 scoring Copying Head and top-30 FFN layers with
1 and | = 0.4. For AARF on RAGTruth, for Llama2-7B and I. Llama2-13B, we select 01 = 5. 91 = 0.2, 0.0. On Dolly (AC), for Llama2-7B, Llama2-13B. and Llama3-8B, we select Q1 = 2. 31 = 0.5, and T = 0.6. As proposed in Section 4.2 ReDeEP (Chunk) requires segmenting the retrieved documents and responses from the benchmark. For this, we utilized LangChain SecursieCharacte//eettppttee for the segmentation process. We use BGE embeddings Xiao et al.) (2023) for ReDeeEP(chunk), which is the Sota embedding model. The thresholds for the prediction of ReDeeE(tokee)) and ReDDeE((chunk) are shown Table 5]
On Llama3-8B,
and т 0.4, while for Llama3-8B, we use 01 5, f1 0.2, and
popular open-source toolkit, and applied the
For the top K for attention heads (AA) and top K for FFNs (F), we performed an initial ranking of the correlations between the ECS of different attention heads and the PKS of FFN layers with hallucinations the validation set. Then, we employed greedy grid search method. For the top-K attention heads and top-K FFN layers, was searched within the range of 32]. For the remaining two parameters, we used two approaches: the first was to treat them trainable parameters and regress on the validation set, and the second, adopted in this paper, was to directly perform grid search. Specifically, we first fixed one parameter, a, to and then tuned (3. We found that B could be adjusted within the range of (0, with a step size of 0.1. I. For text chunking, we utilized LangChain's based on characters. Since this is not the focus of our research, we set the maximum chunk size the commonly used 256 in LangChain, with chunk overlap of 20, For AARF, the parameters are relatively simple. We used grid search for in the range of (0, 1) with a step size of 0.1. For the weight 022, grid search was performed within the values [1, 2, 5, 10). and for 32, grid search was performed in the range of (0, 1) with step size of 0.1.
seeursivelyyppoe Betpppiityyhharaaee
which automatically splits text
The Llama2-7B can be downloaded from https: http://hgggggggggg 1.1ama-2 2- 7b- chat-hf / //huggingface..o. wtw.aaacc.c.//taato www.e/eeetaaaamaaaaaeiaaeaaaaaae
ct/.t/////ta.//// 1 aamaa
The 1.aama2-13B can be downloaded from https:
The Llama3-
8B can be downloaded from https:
http:::///ugggnggac.eee co cw/meta////////ooo.aa...a...a..a 10--1lamas aamma/ hhw.ggg...faaa.ee/aaaa////-aaaae
Mata-Llama- 3-0-mma---------ttt
The BGE embeddings can be downloaded from https:
Table 5: Threshold of ReDeEP(Token) and ReDeIP((Chun)) ReD across different models and dataset.
RAGTruth
Dolly
ReDeEP(Token)
ReDeEP(Chunk)
ReDeEE(Token)
ReDeEE(Chhun)
LLaMA2-7B
0.6
0.3
0.4
0.3
LLLMM2-13B
0.6
0.6
0.6
0.2
LLaMA3-8B
0.4
0.4
0.2
0.2
K ABLATION STUDY
As shown in Table 6. when performing RAG hallucination detection, using only the Parametric Knowledge Score (Only PKS) or only the External Context Score (Only ECS) does not achieve the same performance as the Full ReDeBP model. This validates the effectiveness of employing multivariate regression, where both PKS and ECS are used simultaneously as covariates. According to the analysis in Section I using Only PKS or Only ECS introduces confounding issues, leading to decrease in performance. This explains why both Only ECS and Only PKS yield lower results compared to Full ReDeHP.
L EFFICIENCY ANALYSIS
As shown in Figure 7 ReDeEP (chunk) is more efficient than ReDeEP (token), confirming the superior time efficiency of our proposed chunk-level hallucination detection method, Additionally,
https::/ htop:::/www... wwngchaiinnccom
23

---

Published as a conference paper at ICLR 2025
Table 6: Ablation Study ReDeEP.
RAGTruth
ReDeEP (Token)
AUC
PCC
ReDeEP (Chunk)
AUC
PCC
Only PKS
0.6950
0.3327
Only PKS
0.6180
0,2103
LLaMA2-7B
Only ECS
0.7234
0.3779
LLaMA2-7B
Only ECS
0.7098
0.3944
Full
0.7325
0.3979
Full
0.7458
0.4203
Only PKS
0.7214
0.3682
Only PKS
0.6614
0.2566
LLaMA2-13B
Only ECS
0.8040
0.5201
LLaMA2-13B
Only ECS Full
0.7231
0.3922
Full
0.8181
0.5478
0.8244
0.5566
Only PKS 0.6102
0.1085
Only PKS
0.6082
0,1695
LLaMA3-8B
Only ECS
0.7336
0.4312
LLaMA3-SB
Only ECS Full
0.6923
0.3102
Full
0.7522
0.4493
0.7285
0.3964
Dolly (AC)
ReDeEP (Token)
AUC
PCC
ReDeEP (Chunk)
AUC
PCC
Only PKS
0.6671
0.2374
Only PKS
0.6383
0.2115
LLaMA2-7B
Only ECS
0.6629
0.2852
LLaMA2-7B
Only ECS
0.7552
0.4478
Full
0.6884
0.3266
Full
0.7949
0.5136
Only PKS
0.6639
0.2891
Only PKS
0.6790
0,2883
LLaMA2-13B
Only ECS
0.6856
0.3107
LLaMA2-13B
Only ECS Full
0.7383
0.4338
Full
0.7226
0.3776
0.8420
0.5902
Only PKS 0.6329
0.2300
Only PKS
0.7334
0.3503
LLaMA3-8B
Only ECS
0.6105
0.1556
LLaMA3-8B
Only ECS
0.6166
0.2624
Full
0.6701
0.2421
Full
0.7354
0.3652
Hollucination Cumputationa 11..M42-7B--CC
Hallacination Compuiational LiMMA-2-13B-Cha
Ii 6:25
all
Sut2 out due Bidt 0.11
A
Pasders
Red
Rantal iff
Betadt salum
Proneed
Rerong
Medi
Payylnus
alsorg
Figure 7: Comparison of the detection efficiency of different hallucination detection methods on LLaMA-2-7B. Chaat and LLaMA-2-13B.
methods that do not rely on external models (e.g., ITT, ReDeBP, etc.) exhibit significantly higher efficiency compared to those using external models (e.g., RefCheck, Chainpoll, etc.). ReDeEP is positioned at high-efficiency level among hallucination detection models, validating its fcasibility for industrial applications. Considering that the main time consumption of AARF lies in ReDeEP (token), the relatively low time consumption of ReDeEP (token) further validates the efficiency of AARF.
M PROMPT FOR TRUTHFULNESS EVALUATION
In the Truthful RAG Generation Experiments, we used the following anonymous prompt to assess the truthfulness of RAG generation from AARF (A) and Baseline (B) models:
24

---

Published as a conference paper at ICLR 2025
You tasked with evaluating answers generated by different models: and B. Your goal is to determine which answer is more truthful based on the provided content and query. Please carefully compare the truthfulness of both answers. Follow the specific response format provided below.
Query: [String)
External Context: {String)
answer: [String]
B answer: [String]
Respond in the following format:
{{
"win model": "A (A's answer is more truthful) or B (B's answer is more truthful) or Tie (Both answers equally truthful. Generally not needed. Choose this only if no distinction can be made)",
"reason": "Provide the reason for selecting the winning model's answer as more truthful." }}
Savvevvity Arrilycs on 11M443-9
Senvitivity Anelysice LLMM3-22
Samediity Anclyss LLaMAR-7B
ABC
0.24
AUC
035-
677
ABC
6:72
10
65
13
20
Top-K Copying Bosts
Top- PUN Ligats
@ Viluo
Scnatlivity Amilyss L3aMA2-13B
Semitivity Analysia LLaMA2-138
Secenivity Ambhes on LLAMAN-173
ABC
1.50
630
0.79
6.70
0.82
0.04
6:24
20
Top K Copying Heads
Top & FFN Lapers
8 Vilue
Sundivily HLMASSE
Senuitisiiy Andiviic 1LM-88885
Sand divll Araiycs LLMASS-
MB0
ABC
ALC
6.72
0:73
6.1
1000
Tip-K Copying Hoads
Top-K PIN Lapats
B Vilue
Figure 8: Sensitivity Analysis of ReDDeEP(chunk..
N SENSITIVITY ANALYSIS
In this section, we conduct sensitivity analysis experiments for ReDeEP(TTken) and Re- DeEP(Chunk) based on the parameter ranges provided in Section J From the experimental results in Figure 8 and Figure 9 we observe that ReDeeP(Token) and ReDeEP(Chunk' exhibit an initial increase followed by decrease or stable trend for most parameters. The limited performance vari- ation range indicates that individual parameters have minimal impact on the overall performance, demonstrating the stability of our model. Furthermore, the sensitivity curves for the same hyperpa- rameters across different models are similar. The performance extrema for most hyperparameters appear within the same parameter range across different models, further showcasing the cross-model stability of ReDeEP.
O CASE STUDY FOR AARF
25

---

Published as a conference paper at ICLR 2025
Saniévity Analyis LLaMA2-7B
Sessibvily Analsia on LLaMAR-1B
Sanstivvry Analyee LLAMAA--1
6.73
ADC
AUC
6.72
ACI
6:73
072
6.72
20
Tep-K Copying Beads
Top- X PPN Layets
B Vidue
Socatiiiit Anotrss LLAMAC-13B
Semitvitty LLMMAE-33
Scrailiity Amiydi on L1aMA2-11B
ABC
ABC
032
6X1
6-1
0,20
0.5
Tip-K Capying Deads
Top-K PPN Layans
B Vilue
Seasitiiit Anplesis LLMMAJBB
LLeMAS-5B
0.72
ABC
AbC
0.70
0.72
6/6
SUN
Vilue
Figure 9: Sensitivity Analysis of ReDeEP(token).
In this section, analyze the results of a case study for AARF on LLaMA2-7B RAGTruth data. From the examples where is the AARF and the Baseline, we observe that AARF can enhance the comprehensiveness of the generation (Case One), more accurately attribute referenced content to the correct passage (Case Two), and reduce fabricated hallucinations (Case Three), all without affecting coherence writing quality. These results highlight the advantages of AARF in mitigating RAG hallucinations.
Example One:
Part One
Write an objective overview about the following local business based only on the provided structured data in the JSON format. You should include details and cover the information mentioned in the customers' review. The overview should be 100 200 words. Don't make up information. Structured data:
Instruction:
I'name': "Sam's To Go", 'address': '315 Meigs Rd, Ste I, "city": 'Santa Barbara', 'state":
'CA', 'categories': 'Sandwiches. Restaurants, American (New)'. 'hours': ('Monday': '10:0-2::0', 'Tuesday': "10:0-21:0', "Wednesday": '10:0-21:0', 'Thursday': '10:0-21:0',
'Friday': '10:0-21:0', 'Saturday': '10:0-21:0', 'Sunday': 110:0-19:30), 'attributes':
"'BusinessParkingg" l'garage": False, 'street': False, 'validated': False, 'lot': True, 'valet': False), RestaurantsReservvtionn False, 'OutdoorSeating': True, "WiFi": 'free', 'Restau- mantsTakeOut" True, ResturannsGoodoorrroom
True, 'Music': None, 'Ambience": f'touristy': False. 'hipster': False, 'romantic': False. 'divey': False, 'intimate': False, 'trendy": False, 'upscale": False, 'classy': False, 'casual": False)], J. "business stars': 4.5, jeno'' ((reeview_stars" 5.0, 'review _date:: 22022-01 -03 00:05:08', 'review_text': "This was our first time Sams and it was delicious!! Excellent customer service, they were so sweet. We got the #11 Italian cold cuts, and the veggie sandwich #29 both were so good! The bread daughter got the turkey sandwich and she the pickiest eater and she loved it! Can't wait to go back").
fresh and you could tell the ingredients
really quality! My
26

---

Published as a conference paper at ICLR 2025
Example One:
Part Two
('review stars': 5.0, 'review date': "2021-12-10 03:34:00', 'review text': "The couple who runs the place are so nice and funny, and the sandwiches are delicious. It doesn't t look fancy, but do yourself favor and try out. The Pastrami Avocado and Cheese was delicious. Warning, the medium size sandwich is BIG."), {"review_stan' 2.0, 'review_date": '2021- 11-28 21:43:45', "review_ext!" 'Ive been going to Samá for the past several years while visiting beautiful Santa Barbara from the San Francisco Bay area. I'e always enjoyed the sandwiches from Sams. Unfortunately, my wife and I just finished eating their turkey sand- wich and in all honesty, itwas just, Ok, more like disappointing... We were disappointed because appears the quality and taste of the sandwich has digressed to a "point of no re- turn"... The sandwich was sloppily made, poorly wrapped, and importantly, the man behind the counter (usual guy) seems to have gotten cantankerous over time. He chastised me for ordering both white and wheat bread sandwiches the way he wanted, getting perturbed for asking for olives in my sandwich and, he had smart alec bad attitude demeanor about him poor customer service.
Suffice it to say, we will not be going back to Sams. There are too many good delis in the Cliffs Dr. area to subjected to poor quality sandwiches and an unpleasant customer experience... }}} Overview:
A's Response: Sam's To Go is a popular sandwich shop located in Santa Barbara, California. With rating of 4.5 stars, the business offers variety of delicious sandwiches, including Italian cold cuts, veggie sandwiches, and more. The atmosphere described casual and welcoming, with free WiFi and outdoor seating available. Customers have praised the excellent customer service and high-quality ingredients, with one reviewer noting that the bread always fresh and of high quality. However, negative experience with the service, finding the staff to be unfriendly and the sandwich to be poorly made. Overall, Sam's To Go great option for sandwich lovers looking for quick and satisfying meal in cozy setting.
customer had
B's Response: Sam's To Go is a popular sandwich shop located in Santa Barbara, California. With rating of 4.5 stars, the business offers a variety of delicious sandwiches, including Italian cold cuts, veggie sandwiches, and more. The atmosphere is casual and welcoming, with outdoor seating available. Customers have praised the excellent customer service and high-quality ingredients used in their sandwiches. One reviewer noted that the sandwiches are made fresh and with quality ingredients, while another appreciated the reviewer had disappointing experience with their
friendly and funny staff. However, turkey sandwich, finding to be poorly made and lacking in taste. Despite this, Sam's To Go remains a popular choice for sandwich lovers in the area.
GPT4-o Judge Result: Model A's answer provides a more comprehensive overview of both positive and negative customer experiences while maintaining the core information from the structured data. It accurately describes the atmosphere, the quality of the sandwiches, and includes specific feedback about customer service, which reflects the nuances in the reviews better than Model Model B lacks some details on the specific issues raised in the negative review, making A's response more truthful and balanced.
27

---

Published as a conference paper at ICLR 2025
Example Two
Briefly answer the following question: how boil potatoes for easy peeling Bear in mind that your response should be strictly based on the following three passages: passage 1:Boil until tender, then transfer minutes to cook, seconds to peel each cooked potato) And now, an epic demonstration of How Peel 2 Pounds of Potatoes in Under A Minute using a drill, (clean) toilet brush, bucket, and water hose. GAME. CHANGER.
Instruction:
ice water bath. When cool, peel off skin. (15
passage 2:This versatile medium-staree spud has a delicious, buttery flavor. It cooks up tender and holds its shape, but it also has just enough starch to contribute creaminess without making potato salad mushy. Its skin can be tough and papery, so we recommend peeling them first.
passage 3:Especially if you are peeling a crap load of potatoes. Here is the easiest way (or at least the easiest way I've discovered) to peel cooked potato. Step 1.) Bring pot of water to a boil. Step 2.) With a knife, score your potatoes all the way around the center. Step 3.) Place your potatoes in the pot and boil for about 15 minutes so. Step 4.) Prepare a bowl of ice water and place your potatoes in the water. Step 5.) Swish it around for a few seconds just enough that it's cool enough handle. Step 6.) Remove your potatoe from the water and remove the skin.
A' Response: Based on the provided passages, the best way to boil potatoes for easy peeling to:
1. Boil them until tender, then transfer them to an ice water bath to cool down quickly (as shown in passage 1). 2. Peel the potatoes as soon as they are cooked, as the skin can become tough and papery if left to sit (as stated in passage 2). 3. Use drill, toilet brush, bucket, and water hose to peel 2 pounds of potatoes in under a minute (as demonstrated passage 3).
B's Response: Based on the provided passages, the best way to boil potatoes for easy peeling to:
1. Boil them until tender, then transfer them to an ice water bath to cool down quickly (as shown in passage 1). 2. Peel the potatoes soon they cooked, the skin become tough and papery if left to sit (as mentioned in passage 2). 3. Boiling them, scoring the skin, cooling them in ice water, and then effortlessly removing the skin. (as demonstrated in passage 3).
GPT4-o Judge Result: A's answer accurately summarizes and organizes the critical steps for boiling potatoes for easy peeling, referencing the instructions from the passages sequen- tially. It correctly includes the process of boiling, cooling in an ice water bath, and peeling.
28

---

Published as a conference paper at ICLR 2025
Example Three:
Part One
Summarize the following news within 193 words: We kind. To walk into court and show remorse, tell us that they are really nice people who only did it because they feared for their lives, they were temporarity insane. Even though those things may not be true, too often they are accepted as legitimate excuses for murder. Well, Aaron Hernandez, the former New England Patriots tight end and now convicted murderer, made none of those excuses. He walked into court with an air of bravado, his head held high like the $40 million, NFL superstar he was just a couple of years ago. Several times he was even caught winking at his fiancee, Shayanna Jenkins, during the trial. And we didn't like his swagger. Hernandez didn't offer plausible alibi. He didn't look ashamed remorseful. He never wept. His own attorney, James Sultan, admitted that Hernandez "witnessed" the killing of Odin Lloyd, "committed by somebody he knew," but said his client did not commit the crime. Even before the guilty verdict came down Wednesday, for many - at least those covering the trial audacity seemed to be the defendant's biggest crime. "The Arrogance of Aaron Hernandez," New Yorker headline accused. don't 't get it. What does it matter that Hernandez was arrogant in court, or walked with too much swagger, even smiled at his girl? Like it or not, that is who he is. The evidence is what matters. And for once, this time it appears the jury carefully considered the damning mountain of circumstantial evidence against this defendant and came the right decision: guilty of first- degree murder in the 2013 slaying of Odin Lloyd. Hernandez, 25, was sentenced to life without the possibility of parole. He was also found guilty of unlawful possession firearm and unlawful possession of ammunition. It was the right decision. Still, it's hard not to feel sadness over such a senseless waste of life for both Lloyd and now Hernandez, who were once friends. Football was the best thing Hernandez ever had, especially after he father died suddenly after a routine hernia surgery in 2006. Hernandez was 16. After that his life got complicated. At 17, Hernandez went off to the University of Florida and seemed headed for greatness. There he won the John Mackey Award as the nation's 's best tight end and led the team in receiving during its 2009 Bowl Championship Series win. But off the field, his life was to beginning to unravel. Trouble started: bar fights, reports of marijuana use and failed drug tests. He was even questioned in relation to a shooting after fight at a Gainesville, Florida, nightclub. By the time he was drafted in 2010 by the New England Patriots, Hernandez had already been labeled a "troubled player." But neither the Patriots nor the NFL has anything to be ashamed of in how they handled the Hernandez case. They did everything right this time. Hours after Hernandez was charged with murder, he was let go from the team. And his coaches didn't 't spend any time trying to convince the public that their star tight end was a decent, family-loving guy, as the league has done too many times in the past when its players got in trouble. This time justice worked. There's 's no reason to sronddgueess what went wrong, to ask "How could a star NFLer be a murderer?" The NFL for the most part does great job vetting its players, and certainly Hernandez is anomaly in the league, where despite the ugly headlines, the overwhelming majority of players are upstanding, law-abiding citizens. And Patriots bashers (me included) would be mistaken to try to find fault with the team for drafting Hernandez despite his troubled past. No one really knows what evil lurks the hearts of others, even those closest to us. Not the mothers whose sons go off and commit schoolyard killing sprees, or wives whose husbands gun down innocent people, and certainly not employers who are mostly concerned about performance on the job. There was no way to predict Hernandez would end up a murderer. He was guy who had all the talent and opportunity in the world, but still went wrong. Hernandez himself may have explained it best he was being taken out of the courtroom: According to law enforcement source close to the case, Hernandez told officers escorting him, "'Hey man, I'm going miss you guys. ... I don't t need any luck any more.' He gave you the impression, 'It's kinda like no big deal. is what it is.' output:
Instruction:
killers to be
a
29

---

Published as a conference paper at ICLR 2025
Example Three:
Part Two
Aaron Hernandez, former New England Patriots tight end, convicted of first-degree murder and sentenced to life in prison without the possibility of parole. Despite his arrogant demeanor in court, the jury found him guilty of killing Odin Lloyd in 2013. Hernandez showed no remorse and offered no excuses for his actions, and his own attorney admitted that he witnessed the killing but did commit it. The case highlights the problem of excusing violent behavior by athletes and celebrities, and the need for more thorough vetting of NFL players. The NFL did the right thing by releasing Hernandez from the team hours after he was charged, and there is no reason to second-gus -ueess their decision. The case is reminder that evil can hide in plain sight, and that even those with the best of opportunities can still go wrong.
A's Response:
B's Response: Aaron Hernandez, former New England Patriots tight end, was convicted of first-degree murder and sentenced to life in prison without parole. Despite his $40 million NFL career and seemingly charmed life, Hernandez showed remorse in court and was found guilty of killing Odin Lloyd in 2013. The jury deliberated for less than a day before delivering the guilty verdict. Hernandez's defense team argued that he was not the killer, but the evidence against him was overwhelming. The case has raised questions about the NFL's vetting process and how it handles players with troubled pasts. Many left feeling saddened by the senseless waste of life, as both Lloyd and Hernandez had promising futures that were cut short.
GPT4-o Judge Result: Model A's answer focuses on the key elements of the case, including Hernandez's ss conviction, sentencing, and lack of remorse, while also addressing the concerns about the societal tendency to excuse violent behavior and the NFL's response. accurately captures the overall context of the situation, emphasizing the responsibility of the judicial process and the moral implications of Hernandez's actions, which aligns more closely with the original content's 's message.
30


Reviews:
[{"review_id": "vROxjzSxZk", "reviewer": "Reviewer_LdfE", "summary": "This paper proposes a method for detecting hallucinations of Retrieval Augmented Generation (RAG) models in the scenario when retrieved context is accurate and relevant. \n\nThe authors hypothesize that hallucinations are caused by models ignoring the retrieved context and overemphasizing their parametric knowledge. To capture these concepts they introduce two auxilary scores: External Context Score (ECS) that reflects utilization of the retrieved context by the model, and Parametric Knowledge Score (PKS) that reflects utilization of the parametric knowledge. Hallucinations are then predicted by thresholding a hallucination score H which is computed as a weighted sum of ECS and PKS.\n\nIn addition to that, the authors propose a method to reduce hallucinations by suppressing outputs of attention heads that contribute to low ECS and outputs of fully-connected layers that contribute to high PKS.", "questions": "- Why LLama2-7B (smaller and older version than others) has better results on Dolly in terms of F1 or Accuracy in Table 1?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Authors provide a straightforward method to detect hallucinations in RAGs that does not require model fine-tuning.\n- Empirical results provided by the authors look good.", "weaknesses": "## Lack of justification for PKS and ECS\n\n### No PKS justification\nAlthough PKS is correlated with a hallucination label (line 319) there is still no guarantee that it is adding parametric knowledge. Since you do not provide any theoretical justification for this score, at least an empirical justification is needed. You can run a simple experiment: use LogitLensed outputs before FFN as final outputs and check whether it removes the parametric knowledge bias using some of the setups for that, for example, the one from [1] (they study it through the prism of the phenomenon you encounter in RQ3 and Appendix E).\n\n### Questionable ECS justification\nContrary to the PKS the authors provided empirical justification for the ECS measuring model reliance on context, however, I find it not very convincing so far.\n\nFirst of all, I do not see how the ratio of attention head attending vs mis-attending justifies ECS. It would make more sense to me if you provided such a ratio for mulitple different values of ECS and observed that the higher the ECS the more often a model attends.\n\nSecondly, I am not sure that ratio of attending is computed correctly. As far as I understood for LLama-7B you take hallucinated response (which means that it contradicts external context) and the most attended span in external context. Then you ask gpt-4o to evaluate whether this span supports existence of a conflict in response or not. If that is the case, I do not understand why this experiment shows whether the model attends (the attention span contains part of the context needed for the correct answer) or mis-attends. If attention span supports the existence of a conflict in response it might still not be relevant for the correct response itself, which means a conflict exists but we can not call it a hallucination according to your definition (hallucination = response is contradicting the context or is not supported by it - line 72).\n\nPlease correct me if I misunderstood the experiment setting, what is meant by attending, or the way attending and mis-attending is computed.\n\n## Too many hyperparameters\nI am afraid that the proposed hallucination detection method is not applicable in practice as it requires a lot of manual hyperparameter tuning. According to the provided values, they all are different per dataset and model (see Appendix I). They include:\n\n- top k % for ECS \n- top k % for PKS\n- tau threshold for H - page 8 bottom\n- alpha and beta for reweighting page 9 top\n- chunk size for the chunked version of REDEEP\n\nI suggest that the authors discuss strategies for automating hyperparameter selection or provide guidelines for choosing these parameters in real-world applications.\n\n## Insufficient experiments\n\n### Hallucination detection experiment\n- For RagTruth dataset there exist baselines provided by the original paper [2] which perform better than all the baselines considered by you, could you please include them? E.g. Baseline LLama2-13B results fine-tuned on RagTruth have 78.7 F1, see Table 5 in [2] vs yours 78.3 in Table 1. I think the comparison makes a lot of sense since you tune many hyperparams using RagTruth validation dataset while you could simply fine-tune that baseline on the same data instead.\n- Same comes for Dolly dataset, please include results for AlignScore and RepC-LE-nn-n2000-e2000 that have 84 and 86 accuracy correspondigly, while the best method provided by you scored 73.73 (LLama2-7B).\n- Please also provide results for the Noisy Context split from Dolly [3] dataset because it better approximates realistic RAG application scenario. \n\n### Causal experiment\n\n- First of all, I don’t see how a higher NLL difference for the experimental group than for the control group shows a causal relation between hallucinations occurrence and copying heads neglecting necessary knowledge, could you please elaborate?\n- The experiment results are very noisy and it is hard to draw any conclusions from them, for example, boxplot of the experimental group is fully contained within the boxplot of the control group in Figure 5 (b). \n- It is not clear how many heads are within experimental and control groups, it can be the case that loss changes are bigger for the experimental group simply because it intervenes in more heads.\n\n### Hallucination generation experiment\n\nPrompt for truthfulness (Appendix L) creates bias, since GPT-4o knows which answer belongs to the baseline and which to AARF. It can influence its answers since usually in scientific papers named methods outperform baselines, which must have been the case on chatgpt training data as well and possibly created such a bias. \n\nInstead, it would be nice to see the results for prompts that contain anonymous names (e.g. model 1 and model 2 instead of baseline and AARF) to avoid the mentioned naming bias and have a randomly shuffled order of AARF and Baseline inputs before showing to GPT-4o to avoid positional bias.\n\n### Lack of sensitivity experiments\nPlease provide sensitivity experiments to the numerous hyperparameters you introduced (see the section \"Too many hyperparameters\" for the hyperparameters)\n\n## Unclear writing\n- While being core concepts of the paper, Copying Heads (set A) Knowledge FFNs (set F) are not formally defined (line 381). I guess set A is built by taking top-k attention heads after sorting them by ECS while set B is built by taking top-k FFNs after sorting them by PKS, but I could not find it in text.\n- Strange ordering equations, for example, Eq. 2 that defines an important part of ECS has an undefined value “a” which is only introduced in Appendix Eq. 8.\n\n## Typos\n455: REDEPE\n\n## References\n\n[1] Kortukov, E., Rubinstein, A., Nguyen, E., & Oh, S.J. (2024). Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documents.\n\n[2] Wu, Y., Zhu, J., Xu, S., Shum, K., Niu, C., Zhong, R., Song, J., & Zhang, T. (2023). RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. Annual Meeting of the Association for Computational Linguistics.\n\n[3] Hu, X., Ru, D., Qiu, L., Guo, Q., Zhang, T., Xu, Y., Luo, Y., Liu, P., Zhang, Y., & Zhang, Z. (2024). RefChecker: Reference-based Fine-grained Hallucination Checker and Benchmark for Large Language Models. ArXiv, abs/2405.14486.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ao9nBbweuB", "reviewer": "Reviewer_JEtc", "summary": "Retrieval-Augmented Generation (RAG) models are still prone to hallucinations. This paper explores the internal mechanisms behind hallucinations in RAG settings. Building on these insights, the authors propose a hallucination detection method, ReDeEP, and a RAG truthfulness improvement method, AARF.", "questions": "The paper offers valuable insights into how RAG-based LLMs produce hallucinated outputs. Building on these findings, it proposes a detection method and mitigation strategy grounded in this understanding. Presentation issues remain, particularly in the main figure explaining the method, yet the contribution is significant.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Each step is thoughtfully motivated, with both conceptual reasoning and empirical validations in §3. The detection method shows effective results in Table 1, and the RAG truthfulness improves using AARF, as shown in Figure 6.", "weaknesses": "Figure 3 is problematic. The starting point and flow of the diagram are unclear, with too many arrows, making it hard to identify the main computational paths. An effective graphic would show one main data processing pipeline, which is missing here. Additionally, the quantities computed are not well-defined. Panels (b) and (c) add no extra information and could be removed without loss.\n\nOtherwise, rather minor points:\n- l.281: Please describe the number of hallucinations and non-hallucinations (h = 0 and h = 1) in the evaluation set.\n- Pearson's Correlation in §3: Why measure Pearson’s correlation between ECS and hallucination labels (binary)? It would be more informative to report accuracy at a fixed threshold or detection metrics such as AUROC. Similarly, for PKS and hallucination, detection metrics like AUROC would be preferable.\n- l.465: Could you clarify the criteria for selecting thresholds for accuracy, recall, and F1?\n\nEven more nits:\n- Use full names for FFN, ReDeEP, and AARF, at least in the abstract.\n- In Figure 4(c), clarify what the colour bar values represent.\n- Overall, font sizes in the figures are too small.\n- Structure in §3.2 is difficult to follow. Stick to a standard structure using \\section, \\subsection, \\subsubsection, \\paragraph, etc., rather than introducing new hierarchies (boldface, underline, italics, numbering (1), (2), …).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Picn0j6Fo9", "reviewer": "Reviewer_ixxW", "summary": "This work proposes ReDeEP - a method to detect hallucinations by LLMs in retrieval augmented generation settings by using mechanistic interpretability. The authors introduce two novel metrics - (1) the External Context Score (ECS) and (2) Parametric Knowledge Score (PKS) to identify when hallucinations happen because of over reliance on internal knowledge or from the underuse of external information. The authors also introduce AARF (Add Attention Reduce FFN), which aims to adjust the weights of the attention heads, and feed forward layers to reduce hallucinations.  Their approach is empirically validated on standard benchmarks, demonstrating superior performance to existing methods.", "questions": "1. Could you discuss more the trade-offs of your method? In particular thinking about real time settings? \n\n2. Have you tested your method on non-LLama models? Do you anticipate any challenges for different models? \n\n3. Could you provide some example outputs pre and post using AARF? Or can you speak to the effect AARF has on the coherence of the model’s output after AARF?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The development of the ECS and PKS metrics to understand the contributions external and internal knowledge have on the LLM's generation is a compelling and novel way to understand LLM outputs. \n\n2. They demonstrated great empirical validation by running extensive experiments across two datasets, three LLMs, and many baseline methods. \n\n3. They also introduce a method to curb hallucinations called AARF - which relates back to the introduced metrics nicely.", "weaknesses": "1. Performing this analysis at the token/chunk level might limit its practicality in real time or large scale settings - it would be nice to have a richer discussion of the trade-offs and real world feasibility. \n\n2. The experiments are extensive - however they are all with the LLama family of models - testing (even a much smaller set) on a different model would be informative. \n\n3. While the performance of AARF seems good (Figure 6) - it would be good to see some example outputs - its unclear how this could effect the model’s output in terms of coherence/writing in general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)