LUMINA: Detecting Hallucinations in RAG System with Context–Knowledge Signals
Authors: Samuel Yeh
Sharon Li
Tanwi Mallick
Venue: ICLR 2026 Poster (2026)
Keywords: Hallucination detection
Retrieval-augmented generation
Reliability of LLM
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Retrieval-Augmented Generation (RAG) aims to mitigate hallucinations in large language models (LLMs) by grounding responses in retrieved documents. Yet, RAG-based LLMs still hallucinate even when provided with correct and sufficient context. A growing line of work suggests that this stems from an imbalance between how models use external context and their internal knowledge, and several approaches have attempted to quantify these signals for hallucination detection. However, existing methods require extensive hyperparameter tuning, limiting their generalizability. We propose LUMINA, a novel framework that detects hallucinations in RAG systems through context--knowledge signals: external context utilization is quantified via distributional distance, while internal knowledge utilization is measured by tracking how predicted tokens evolve across transformer layers. We further introduce a framework for statistically validating these measurements. Experiments on common RAG hallucination benchmarks and four open-source LLMs show that LUMINA achieves consistently high AUROC and AUPRC scores, outperforming prior utilization-based methods by up to +13% AUROC on HalluRAG. Moreover, LUMINA remains robust under relaxed assumptions about retrieval quality and model matching, offering both effectiveness and practicality.

LUMINA: https://github.com/deeplearning-wisc/LUMINA

Published as a conference paper at ICLR 2026
LUMINA DETECTING HALLUCINATIONS IN RAG SYSTEM WITH KONTEXT-KNOWLEDG SIGNALS
Samuel Yeh 2 Sharon Li', Tanwi Mallick?
Department of Computer Science, University of Wisconsin-Nadison (samuelyeh, sharonli 1 e wisc.edu
Argonne National Laboratory tmallicke gov
ABSTRACT
Retrieval-Augmented Generation (RAG) aims to mitigate hallucinations in large language models (LLMs) by grounding responses in retrieved documents. Yet, RAG-based LLMs still hallucinate even when provided with correct and sufficient context. A growing line of work suggests that this stems from imbalance be- tween how models use external context and their internal knowledge, and several approaches have attempted to quantify these signals for hallucination detection. However, existing methods require extensive hyperparameter tuning, limiting their generalizability. We propose LUMINA, a novel framework that detects halluci- nations in RAG systems through contextknoowledge signals: external context utilization is quantified via distributional distance, while internal knowledge uti- lization is measured by tracking how predicted tokens evolve across transformer layers. We further introduce framework for statistically validating these mea- surements. Experiments common RAG hallucination benchmarks and four open-source LLMs show that LUMINA achieves consistently high AUROC and AUPRC scores, outperforming prior utilization-bast methods by up to +13% AUROC on HalluRAG. Moreover, LUMINA remains robust under relaxed assump- tions about retrieval quality and model matching, offering both effectiveness and practicality.
LUMINA: http::s//gtttt..o/////p/aaaaaaas
INTRODUCTION
Large language models (LLMs) are prone to hallucination, i.e., producing responses that are factually incorrect, nonsensical, or not grounded in the input or available data, while still appearing fluent and plausible (Luo al., 2024; Huang al., 2024; Park al., 2025). One commonly used strategy to mitigate hallucination is providing LLMs with relevant information retrieved from external knowledge bases, so-called Setrieval-Augmente Generation (RAG) (Shuster et al., 2021; Fan al., 2024; Gao et al., 2024). However, despite having sufficient and relevant retrieved documents, RAG systems still have chance to hallucinate and produce statements that are either unsupported or contradict the retrieved information (Niu et al., 2024; Ridder & Schilling, 2025).
Recent work has shown that such failures often arise from conflicts between LLM's internal knowledge and the retrieved external context (Xu et al., 2024). In these cases, models tend to over-rely on internal knowledge regardless of correctness, undermining factual reliability (Longpre et al., 2021; Li et al., 2023; Sun et al., 2025a; Yamin et al., 2025). Inspired by this observation, recent approaches attempt to quantify hallucinations in RAG (Sun et al., 2025b; Wang, 2025; Tao et al., 2025). However, cxisting methods rely on mechanistic interpretability heuristics-sur as selecting specific attention heads or transformer layers to achieve the optimal hallucination detection performance-vwe which require heavy hyperparameter tuning and often fail to generalize across models and datasets.
To overcome these limitations, propose LUMINA, new framework for detecting hallucinations in signals, namely the signals of external context utilization and
RAG system through internal knowledge utilization, shown in Figure Rather than targeting particular attention heads

---

Published as a conference paper at ICLR 2026
External Context Utilization (Sec. 3.1)
Internal Knowledge Utilization (Sec. 3.2)
Embedding Space
LLM
Next Token Prab
Information
LLM (pe)
(g.d.
Layer
Processing Rate
Retrieved Doc (d)
Maxium Meen Discrepency
Layer/ -1
XW
0.93
Query (q)
Arewer (a)
Random Doc (ef)
Layer 2
x XW
0.56
Weighted sum
LLM (po)
Pel (9,d",oe))
Layer
>1 0.32
The overview of LUMINA. For external context utilization, propose measure the maximum mean discrepancy between two next token probability distributions conditioned on different documents. For internal knowledge utilization, we introduce the idea of information processing rate by looking at the ratio of the most probable output token's 's probability across transformer layers and use determine the amount of utilized internal knowledge when generating the next token.
Figure 1:
or layers, LUMINA measures these signals in a layer-agnostic manner, requiring less hyperparameter tuning. Specifically, for external context utilization, measure the discrepancy between predictive distributions conditioned on retrieved documents vs. random documents. A larger discrepancy indicates that the LLM is more sensitive semantic changes in documents when generating the answer, implying higher reliance on the external context. For internal knowledge utilization, we track how the model's internal states and token predictions evolve across layers: the internal layers' predictions do not converge to the final output until later layers, it suggests more information is added during the layer-wise process, implying stronger reliance on internal knowledge. We further validate the soundness of our measurements through statistical hypothesis testing on verifiable implications, establishing a stronger link between the proposed scores and actual utilization.
We conduct extensive experiments on common RAG hallucination benchmarks and across four LLMs to evaluate the performance of LUMINA on hallucination detection. The results show that the hallucination score calculated with LUMINA outperforms existing methods by significant margin. For example, LUMINA achieves more than 0.9 AUROC on the HalluRAG datasets across models, with improvements of up to +13% over prior state-of state-of-the-art. the-art.. Importantly, the decomposition into external context utilization and internal knowledge utilization provides interpretable insights: hallucinations strongly associated with low external context scores and disproportionately high internal knowledge scores. We further demonstrate that LUMINA is robust across different retrieval settings. These results validate both the effectiveness and practicality of our framework.
Our key contributions are summarized as follows:
1. We propose LUMINA, novel approach to quantify utilization of external context and internal knowledge for RAG-based hallucination detection.
2. We propose framework to statistically validate LUMINA, showing that they align with the intended results.
3. We conduct extensive experiments and show that LUMINA outperforms both score-based and learning based methods in hallucination detection, establishing new state-of-the-ar
PRELIMINARIES
2.1 PROBLEM FORMULATION AND MOTIVATION
RAG systems aim to improve factuality by incorporating external documents into the generation process. In cases such as news summarization, information extraction given a json file, and question answering that requires information emerging after the model's 's release date, RAG is usually necessary because an LLM cannot rely solely on its internal knowledge to complete the task. However, such cases, hallucinations still occur when model over-relies on its internal parametric knowledge and under-utilizes the retrieved external context. We provide a formal definition below.
2

---

Published as a conference paper at ICLR 2026
Conjecture 1 (External context internal knowledge utilization). Let pa be RAG-based LLM that takes a query q and retrieved documents d as inputs to generate a response a. Assume d is relevant q and contains correct and sufficient information respond q. Denote En, (alq, dl, Ip (alq d) e be the signals of external context utilization and internal knowledge utilization of po. respectively, when generating a. The response is more likely to be hallucination if Ip (a|q d)
> Ep (alg, d).
Conjecture 1 is built principled intuition that, if a LLM requires external knowledge to complete a task and if retriever can provide the LLM sufficient external information, the LLM should utilize those external context and ground its reasoning ability those context. Therefore, response in this scenario will be considered less reliable if it disproportionally relies on the LLM's internal knowledge without sufficient amount of external knowledge utilization.
Definition 2.1 (Hallucination in an RAG system). Based on Conjecture we define hallucination scores at both the loken and response level. Specifically, for generated answer a = (a1,-.. (d1,...,ar) ,r)
with T tokens. let Eo (ael9 qq d, a<i), ) Ino (aelg, d, act) be the signals of external context utilization and internal knowledge utilization of pe when generating the token ar respectively. The token-level hallucination score of at is defined
He (arala d, acx)) == X- Imm act)
(ar d, act).
(1)
where À is hyperparameter. Similarly, the response-level hallucination score of the response defined as the average of the token-level hallucination scores, i.e.,
1 Ha q, d) = T
Σ H,laalg,d,az
(2)
In this paper, we focus on the core question: How quantify the utilization of external context and internal knowledge?
2.2 RELATED WORK
Prior works have attempted to quantify Epo (aclg,, d, act) and Ipo (axlg, d, act) using empirical met- rics (Sun al., 2025b; Wang, 2025). For example, Sun et al. (2025b) proposed ReDeEP, which measures external context utilization through cosine similarity between the generated token and to- kens in context that have high attention weights w.r.t. certain attention heads. For internal knowledge utilization, it measures the
(JS) divergence between the hidden states before/after the FFN layer of certain transformer layers. The success of ReDeEP on some RAG hallucination detection datasets validates the idea of Conjecture 1. Wang (2025) combine the idea of ReDeEP with semantic entropy probes (SEP) (Han et al., 2024). They quantified external context utilization by measuring the semantic correlation between the semantic entropy of the generated token and attended tokens in the context. For internal knowledge utilization, they measured the absolute difference between the semantic entropy corresponding to hidden states before and after the FFN layer.
Although these approaches effectively detect hallucinations in the RAG system, they have two major limitations. First, these approaches require selecting specific attention heads and transformer layers to compute the external context score and internal knowledge score. However, the selection process non-trivial and requires extensive hyperparameter tuning. In addition, these hyperparameters are dataset and model-specifie, limiting the generalizability across different datasets and models. Another limitation is that although these works demonstrated the correlation between their proposed scores and hallucination, they did not validate whether the scores truly reflect the utilization of external context and internal knowledge.
3 METHODOLOGY
Overview. To overcome the limitations of prior empirical approaches, we introduce LUMINA, a new framework for quantifying both external context and internal knowledge utilization. In Section 1 and Section 3.2, we formalize the quantification of the two signals, which will be combined to compute the final hallucination score. In Section 3.3, we propose to validate the soundness of LUMINA through extensive hypothesis testing, addressing the challenges of score validation in previous works.
3

---

Published as a conference paper at ICLR 2026
3.1 QUANTIFYING EXTERNAL CONTEXT UTILIZATION
To measure LLM's external context utilization, our key idea is to assess its sensitivity to semantic changes in the input documents. If the LLM effectively incorporates the external context generate a response, then replacing relevant documents with random ones should noticeably change the token probability distribution. Formally, we propose the following measurement:
Measurement (External context utilization). Let be LLM -MM-geeertted answer to query with retrieved documents d as input, Assume d is relevant to q and contains correct and sufficient information to respond to q- Let d' be subset of random documents irrelevant q. The model's predictive distribution over tokens induces two (approximated) distributions over embeddings:
P(E.) = po 4 d, act), Q(E.) - po(v q,,d, de))).
where each token v E V in the vocabulary space is associated with an embedding E, E RDP Then, the degree to which the model uses external context for generating token a₂ reflected the divergence between the two distributions conditioned on d versus d':
(arlg, d, acx) == A(P,Q)
where P R2 is a distance function between two probability distributions.
Note that we adopt P(E.) and ((E.) proxies approximate the ground truth embedding distribution, as it is challenging to estimate it over the high-dimensional vector space. We instantiate A with Maximum Mean Discrepancy (MMD). which measures the distance of two probability distributions by mapping them into a Reproducing Kernel Hilbert Space.
Definition 3.1 (Maximum Mean Discrepancy (Gretton et al., 2012)). Given a positive semi- definde kernel function k; the squared MMD between two probability distributions P and Q defined as
(P Q) = (AAA-~Pl((A,, A') + E BB B, B')] 2EA-P,B-Q(K(A, B)}, (5)
where A, are i.i.d. vectors randomly sampled from P and B, B' are sampled from Q.
This metric provides us with a non-parametric and LLM-agnostic way to quantify the utilization of external context, making it generalizable to different models and datasets.
By rewriting MMD with P and Q we defined in Eq. (3) over token embeddings, we obtain:
Eyo (arlar act)== = Σ ((E.))(E)) E.)+
Σ Q(E.)((E)) E.)
a
(6)
Σ (E.,)) E.).
We adopt the cosine kernel:
ETE. E.
ET
Rece Ens (Eus Ex)
1
E. E E,.1
Note that the cosine kernel acts equivalent to computing cosine similarity between two token em- beddings, which commonly used to measure the semantic similarity of two pieces of text. In Section 4.4, we experiment with alternative kemnels such as the Gaussian kernel, and we show that our method is not sensitive to the choice of kernels.
3.2 QUANTIFYING INTERNAL KNOWLEDGE UTILIZATION
To quantify the utilization of internal knowledge, we focus the signals in internal states of an LLM. Specifically, transformer-based autoregressive LLM has multiple layers, through which information is gradually added into residual stream that flows from the input layer to the output layer, shaping the output token representation and probability distribution (Geva et al., 2022). Studies have found that by projecting the hidden state of each layer the token representation space, what an LLM believes after the process each layer (nostalgebraist, 2020). In addition, via logit lens (nostalgebraist, 2020), studies have identified the saturation event in LLM, Lee, the top-k
interpret
4

---

Published as a conference paper at ICLR 2026
prediction of the LLM remains constant in all subsequent layers after certain layer called the k-th saturation layer (Geva et al., 2022; Lioubashevski et al., 2025).
Inspired by these observations, we propose metric that quantifies how actively the model updates its predictions across layers. Formally, we define the rate of information processing below.
Definition 3.2 (Information processing rate). Given an L.M po with L layers, which takes Tict as the input and generate the next token 2t, we denote Ft,1 == arg mäx, pa (vl( 3) as the most probable next token and 1a E I as the I-th layer hidden state when generating It. Let f: IRD + P be projection from a hidden state a probability distribution over the vocabulary V. The information processing rate of pe conditioned on Fet is defined as
Rya (x (=a====== = El=1
1 min
Lfthrdllea 13)
Polaralre))"
(8)
EV=1 H((
where H.)) is the entropy function, and f is the logit lens (nostalgebraist, 2020) that projects the hidden state of each layer to logits using the LayerNorm and the unembedding matrix W, iee.
LogitLens(1) (h) == LayerNorm(h) f(-) == Softmaax(ogiiLeens(
Specifically, Ry (z))) captures two key elements: (1) The numerator measures the extent to which each layer's prediction for the most probable token differs from the final output, weighted by layer
depth emphasize later-layer processing. When [f is small, it indicates the layer has
( yet converged to the final prediction, suggesting active information processing. (2) The denominator provides adaptive normalization based on each layer's 's prediction uncertainty (entropy), giving higher relative weight to layers that exhibit confident, decisive processing patterns. Given this definition, we attribute the utilization of internal knowledge the 1st information processing rate and propose the following measurement:
Measurement (Internal knowledge utilization). An LLM is considered to be more heavily utilizing its internal knowledge to generate a when it exhibits higher information processing rate. Specifically, we propose that the internal knowledge utilization of LLM to generate a, given q and d can be measured as
Ipo (aelg, d, act) == Ryw dd
(10)
3.3 STATISTICAL VALIDATION OF THE MEASUREMENT
In this section, we validate the soundness of our approach. Previous work such as Sun et al. (2025b) primarily verified whether their scores have causal relationship with hallucination but failed to show the relationship between the scores and actual external context/internal knowledge utilization. To address this, we directly assess whether our measurements capture the intended notion of utilization. Specifically, we derive verifiable implications that must hold if our proposed measurements are valid. We then use the proposed score to verify these implications with statistical hypothesis testing. the proposed score passes all tests, the score reflects the corresponding utilization.
External context utilization.
validate Measurement 1, we examine the following implications:
H1. If Measurement is valid, then Epg (arlg, d, a.ce) ) > Ep, (a Ø, a'ze). That is, generations with retrieved documents have stronger external context utilization than generations without.
H2. If Measurement 1 is valid, then Epg (at Isum; deum, (ett) > Epp (a:/iQA, IQA, AQ^, a.x). That is, summarization tasks should exhibit higher external context utilization than question answering.
Internal knowledge utilization. To validate Measurement
examine the following:
H3. If Mcasurement is valid, then po (g, Ø, a.x)) > R. (q, d, azr). That is, generating an answer without retrieved documents requires more internal knowledge than with retrieved documents.
H4. If Measurement is valid, then R/, (QDzT, dpzr, act) > R. (grum d azz)) In other words, data-to-text generation requires more internal knowledge than summarization.
To examine H1, we utilize data in the QA set of RAGTruth (Niu et al., 2024). We use the original data compute E (at (4, d, (.ce), and generate additional answers without providing retrieved documents
5

---

Published as a conference paper at ICLR 2026
Table 1: All the hypotheses pass the statistical tests. For HI, H2, H4, we report one-tailed I-statistic; for H3, we report paired-sample one-tailed t-statistic. All four implications reject their null hypothesis, validating the soundness of LUMINA. Note that the tests are run with 65k tokens and the magnitude of the t-statistic means how easy we can distinguish the two distributions. * 0.05; p < 0.01; *** < 0.001.
LLM
H1
H2
H
H4
Llama2-7B
79, *5
27.67**
101.20*
15.36***
LJama2-13B
73.49*4*
20.51 4+*
91.00***
7.71 48
LJama3-8B
94. 15.4
6.35+**
102.44**
15.85***
Mistral-7B
88. 78.70
6.21 ****
109.26***
9.69***
as a' to compute Epu (a/lg, 23, dzz)). For H2, we utilize the Summary and QA set of RAGTruth; for H4, the Summary and Texa set; and for H3, the entire RAGTrruth dataset. We test the hypotheses with four different instruction-tuned LLMs, including Llama2-(7B, 13B) (Llama Team, 2023), Llama3-8B (Llama Team, 2024), and Mistral-7B (Jiang et al., 2023). Results in Table indicate that all four implications reject their null hypothesis, validating our measurements for external context utilization and internal knowledge utilization.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Baselines. We compare LUMINA with baselines across 8 different hallucination detection strategies: (I) Uncertainty- which detects hallucination by estimating uncertainty via token-level proba- bility or entropy. Baselines of this category include Perplexity (Ren et al., 2023), LN-Entropy (Malinin & Gales, 2021), and Focus (Zhang et al., 2023). (2) Cross-sample consistency, which detects hallu- cination by sampling multiple responses for query and measuring their (logic/semantic) consistency. Approaches include SelfCKGPT (Manakul et al., 2023) and EigenScore (Chen et al., 2024). (3)
Verbalization, which detects hallucinations by prompting another LLM to score the correctness of the answer. Approaches include P(True) (Kadavath et al., 2022) and RefChecker (Hu et al., 2024). (4) Utilization of external context and internal knowledge, which decouples these two signals via findings in the study of mechanistic interpretability. Baseline of this category is ReDeEP (Sun et al., 2025b). Details of each baseline are introduced in Appendix B.
LLMs. To demonstrate the generalizability of LUMINA, we conduct experiments with four open- sourced LLMs, including Llama2- 13B]. Llama3-8B, and Mistral-7B. Specifically, each LLM used to detect hallucinations performance of proxy LLM setting, i.e., using one LLM to detect hallucinations in responses generated by another model, in Sec. 4.3. All LLMs are the instruction-tume version.
responses generated by the same model. We also report the
Datasets. Experiments are conducted on two representative RAG hallucination detection bench- marks: RAGTruth (Niu et al., 2024), the first high-quality RAG hallucination detection dataset, consisting of three types of RAG tasks, including question answering, data-to-text writing, and news summarization. HalluRAG (Ridder & Schilling, 2025), a dataset of free-form question answering in an RAG setting. Details of these datasets are introduced in Appendix C.
Evaluation metrics. We measure the performance with three metrics: AUROC, AUPRC, and Pearson's correlation coefficient (PCC). AUPRC captures precision-recall trade-offs, while AUROC evaluates the trade-offs between true and false positive rates. These metrics are threshold-agnostie and better suited for comparing scoring-based methods. We also report the optimal precision, recall, and Fl score (Precope; Recallopt, F lopps)) in Appendix E. where Flopp is the optimal Fl score among all possible threshold and Precopt and Recallopt are corresponding Precision and Recall.
Implementation details. We adopt À = 0.5 to compute Eq. (1) as ablations show that balancing the scores of external context and internal knowledge yields relatively strong performance (see Appendix E.3 for detailed ablations). Other implementation details and computational resources of LUMINA are reported in Appendix D and G, respectively.

---

Published as a conference paper at ICLR 2026
Table 2: LUMINA consistently achieves high performance across datasets and LLMs. The highest scores are set in bold. Note that HalluRAG dataset does not contain responses generated by Llama3-8B.
RAGTruth
HalluRAG
LLM
Approach
AUROC
PCC
AUPRC t
AUROC 1
PCC
AUPRC f
Perplexity
0.5103
-0.0118
0,4836
0,4610
-0.0673
0,2332
LN-Entropy
0.6964
0.3318
0.6615
0.9102
0.5133
0.6812
Focus
0.5633
0.0811
0,5386
0.5652
0.2415
0.3844
SelfCKGPT
0.4787
-0.0279
0.4859
0.4669
-0.0070
0.2377
Llama2-7B
EigenScore
0.5454
0,0717
0.5183
0.6720
0.2705
0,4470
P(True)
0.5197
0.0404
0.5334
0.5847
0.1143
0.2976
RefChecker
0.5869
0.1751
0.6827
0.4907
-0.0255
0.2750
ReDeEP
0.7273
0.3859
0.6971
0.6771
0.1468
0.3378
LUMINA
0.7646
0.4546
0.7491
0.9153
0.6554
0.7572
Perplexity
0.4539
-0. 1020
0,3993
0.2548
-02366
0.0944
LN-Entropy
0.7677
0.4446
0,6838
0.7826
0.3262
0.3567
Focus
0,5451
0.0130
0.4603
0.6739
0.2563
0.3181
SelfCKGPT
0.4545
-0.0835
0.4106
0.7729
0,2640
0.3029
Llama2-13B
EigenScore
0.6329
0.2080
0.5202
0.7862
0.4250
0.4867
P(True)
0.7543
0.3821
0.7418
0.6914
0.2480
0.2146
RefChecker
0.6363
0.2723
0.6988
0.5670
0.1390
0.3169
ReDeEP
0.8055
0.5195
0.7792
0.7645
0.2705
0.3001
LUMINA
0.8569
0.6041
0,8436
0.9166
0.6044
0.8497
Perplexity
0.7130
0.3568
0.7183
LN-Entropy
0.7072
0.3500
0.7109
Focus
0,5258
0.0375
0.5380
SelCCKGPT
0.5339
0.0491
0,5550
Llama3-8B
EigenScore
0.6001
0.1774
0.5824
P(Truc)
0,5407
0.0928
0,5502
RefCheeker
0.5718
0.1494
0.6874
ReDeEP
0.7495
0.4458
0.7817
LUMINA
0.7446
0.4236
0.7874
Perplexity
0.6200
0.1463
0.6106
0.5362
-0.0264
0.1261
LN-Entropy
0.7607
0.4386
0.7377
0.9188
0.6076
0.7347
Focus
0.7803
0.4188
0.7647
0.8565
0.4318
0.4219
ScIfCKGPT
0,5680
0.0812
0.5698
0.8275
0.5552
0.6098
Mistral-7B
EigenScore
0.5642
0.1006
0.5637
0.8652
0.6411
0.7337
P(Truc)
0.7530
0.4334
0.7494
0.5899
0.0886
0.1771
RefChecker
0.6017
0.2047
0.7303
0.5065
0.0153
0.1784
ReDeEP
0.7615
0.4613
0.8133
0.7870
0.2611
0,3516
LUMINA
0.7685
0.4623
0.7942
0.9899
0.7529
0.9431
4.2
MAIN RESULTS
LUMINA achieves state-of-the-art performance. Table 2 summarizes the experimental compar- ison across methods. The results show that LUMINA has consistently high performance across datasets and LLMs. In particular, it almost always outperforms ReDeEP, the previous attempt of measuring the utilization of external context and internal knowledge to detect hallucinations. The gap between them is particularly large on the HalluRAG dataset. Noticeably, LUMINA achieves more than 0.9 AUROC on the HalluRAG dataset across models, outperforming the baselines by a substantial margin. We further conduct an error analysis to see when and why LUMINA fails. Specifically, sample 20 false-negative and 20 false-positive cases from the RAGTruth dataset, respectively, and qualitatively analyze the reason of errors. The result reveals that most of the errors stem from incorrect labels and low-quality retrieved documents of the dataset, suggesting potentially higher performance in a setting with high-quality data. The details of this analysis can be found in Appendix F.
7

---

Published as a conference paper at ICLR 2026
Noises in random doc (%)
0.75
0.75
0.74
0.74
0.84
0.83
0.80
0.81
0.79
0.78
0.78
0.78
0.79
0.79
3.79
0.79
retrieved
0.74
0.74
0.79
0.72
0.77
30
0.70
0.70
0.69
0.88
0.77
0.76
0.73
0.68
0.80
0.80
0.80
0.79
0.74
0.73
0.73
0.73
Llama2-7B
Llama2-138
Llama3-8B
Mistral-7B
0.76
Figure 2: Noises in context do not largely degrade the performance of LUMINA. We add ~ 30% noises the retrieved documents and random documents and evaluate the hallucination detection performance. The experiment is conducted on the RAGTruth dataset.
Comparison with supervised approach. We also compare LUMINA with SAPLMA (Azaria & Mitchell, 2023), a supervised approach that trained a binary classifier on the last token hidden states detect hallucination. Since method is unsupervised in nature and does not rely labeled data, the supervised baseline can be viewed as a performance upper bound. Results in Appendix E.2 show that LUMINA achieves a competitive performance against SAPLMA and even sometimes outperforms it, all without any training, highlighting both its supreme performance and case of deployment.
4.3 RELAXING ASSUMPTIONS
In Section 3, implicitly make assumptions: 1) perfect context assumption: assume the retrieved documents d are correct, sufficient, and relevant to the query. 2) same LLM assumption: we assume the LLM used compute the external context score and internal knowledge score the same as the L.M used to generate responses. These two assumptions are usually introduced in other hallucination detection works as well (Zhang et al., 2023; Sun al., 2025b). Unfortunately, they are often strong and have a significant impact the performance, limiting the usability of these methods (such as for open-sourced reder-geneeated responses only). this section, we investigate the performance of LUMINA when relaxing these two assumptions, showing the robustness of LUMINA.
Relaxing perfect context assumption. We relax this assumption by gradually injecting noise into the retrieved documents and random documents d'. Specifically, for the assumption retrieved documents, we randomly remove (0%, 10%, 20%, 30%) sentences from d. And for the assumption the random documents, shows the AUPRC of all noise injection combinations on the RAGTruth dataset. The result shows that except Llama2-13B, which has 0.1 performance drop after injecting noises, LUMINA with other LLMs yields stable performance. Furthermore, after removing sentences from retrieved documents, LUMINA with Llama3-8E 3-B even achieves higher AUPRC. These results demonstrate the robustness of LUMINA against context noises.
randomly add f0%, 10%, 20%, 30%} sentences from to d'. Figure
Relaxing the same LLM assumption. We relax this assumption by using different LLMs to compute the scores for response. Specifically, we use Llama2-7B. Llama2- 13B, Llama3-8B Mistral-7B, Qwen3-4B (Yang et al., 2025). Llama4-17B (MetaAI, 2025) to detect hallucination on the RAGTruth dataset, which contains responses generated by Llama2-7B, Llama2-13B, Llama2-70B. Llama3-8B, Mistral-7B, GPT-3.5, and GPT-4. Figure 3 shows AUROC across different generator- detector LLM pairs
The results show that the same model setting is not always necessary. Specifically, Llama2-7B achieves comparable or higher AUROC than Llama3-8B on Llama3-8B responses. Moreover, LUMINA with Llama2-7B 7B and Llama3-8B has stable performance across different generation LLMs. In addition, newer models, such Qwen3-4B and Llama4-178 also perform well across gener- ation LLMs. Overall, LUMINA demonstrates a plausible solution for generation LLM-agnostic hallucination detection, which is practical in real-world scenarios.
8

---

Published as a conference paper at ICLR 2026
Generation
1ama2-78
Laam22-338
Lama3-888
Mistral-78
Llmma2700
GPT-35
GPT-4
Liama2
Detection LLM
Ukmr22
Lanua 88
Matrel
Owar3 48
Lamma-i7tt
ofz sin ch als oãn alt off ain ch oár ch olz zix 62n alz x/n od
The "same LLM" setting is not essential for LUMINA to achieve the optimal perfor- mance. On the RAGTruth dataset, for each set of responses generated by the same LLM, we apply LUMINA with different base LLM to detect hallucination. Bars in more saturated shades indicate settings where the same LLM used for both generation and detection.
Figure 3:
4.4 ABLATION STUDY
Impact of kernel selection. We ablate on the selection of kernel & (Cosine, RBF o.5,
LLM
LlamaZ-78
RBF 1, RBF2, RBFa), where RBF
Llama2-138
RBF kernel, ie.. RBF, (Bn,
Figure shows the AUPRC of different kernels on the RAGTruth dataset. The results show that the optimal setting of the RBF kernel has similar performance to the cosine kernel, suggesting our external con- text score is insensitive to the kernel selection. We default to the cosine kernel it less de- pendent on hyperparameters, making it easy to use practice.
exp
1...F.112
AUPRI
6.70
Mistral-7B
Llema3-88
Kernel
RBF
Cosine
Figure 4: MMD with cosine kernel performs similarly or better than with RBF kernel.
Impact of external context & internal knowledge. Our final hallucination score is the combination of the external context score and internal knowledge score. To obtain more insights into how each component contributes to the final score, we ablate on the components by considering only the external context score and internal knowledge score. The right plot of Figure 5 shows that combining scores of external context and internal knowledge achieves the highest AUPRC on the RAGTruth dataset for every LLM. For example, on Llama2-13B, the combination leads to more than 10% improvement. This observation justifies the effectiveness of the hallucination Definition 2.1. In addition, the left plot of Figure 5 shows that response generated by Llama2- more likely to hallucination if has high internal knowledge score and a low external context score. This observation validates Conjecture and suggests that Eq. (1) does not imply an objective function that forces LLM only using external context to answer questions. Instead, it suggests that the internal knowledge utilization should be grounded in an external context to achieve a reliable tasks.
introduced in
generation, implying its potential for generalizing to reasoning-intent
Additional ablations. We also conduct other ablations, covering the selection of X in Eq (1), the impact of random documents d', and the contribution of two components the information processing rate. Please see Appendix E.3 and E.4 for more details.
5 CONCLUSION
In this paper, introduce LUMINA, a novel approach to quantify the utilization of external con- text and internal knowledge. These context-knowleet signals provide a principled way to assess how LLMs balance retrieved evidence against their own parametric knowledge during generation. Experimental results on common benchmarks across four LLMs demonstrate that LUMINA has a consistently high performance hallucination detection for RAG-based generations, outperforming

---

Published as a conference paper at ICLR 2026
Non- Nor-hhillciination
Hallucination
Extemal
LLM
ineenl
Combinston
Llama2-76
External Context Score
Esternal
Llama2-138
intemal
I Iama3 aR
Extemal
Mistral-78
Combinwioo
0.25
Exteral
intemal
Combination
0.00
0.00
0.25
0.50
0.75
0.90
0.58
0.22
0.70
Internal Knowledge Score
AUPRC
Figure 5: Combining scores of external context and internal knowledge boosts the hallucination detection performance. Left: 2D kernel density estimation (KDE) of the distribution of external context score and internal knowledge score Llama2- responses on the RAGTruth dataset. Right: Hallucination detection performance with external/internal score only, as well as the performance of their combination.
prior attempts of quantifying external context and internal knowledge utilization, and being com- petitive with supervised hallucination detection models. Analyses also show that LUMINA is robust against noise in retrieved documents and can be generalized to the proxy LLM setting, demonstrating its usability in real-world scenarios.
ETHICS STATEMENT
This work introduces LUMINA, novel way estimate the utilization of external context and internal knowledge when an LLM generates responses with the RAG setup. LUMINA significantly improves the performance of hallucination detection, which will help increase the reliability RAG systems in real-world deployments and reduce the risk of sharing misinformation. Through deeper analysis of LUMINA the future, researchers may better understand how LLMs utilize external context and internal knowledge to generate responses. Such findings will help the community design approaches to mitigate ballucinations and create a more reliable AI system.
REPRODUCIBILITY STATEMENT
We provide all details of the implementation of LUMINA in Appendix D, including the approximation of MMD, the selection kernel, and the choice of random documents for measuring external context score, as well as the calibration of internal knowledge score. In Sec. 4.1, we illustrate the experimental settings, including baselines, datasets, LLMs, and evaluation metrics. The details of baselines and datasets are further provided in Appendix B and C, respectively. Furthermore, provide the codebase of LUMINA at https: : / thub. com/ deepl campleerrnnggg www.gngwwww//wiieeeeeeeeeegoeeee Se//LLMMIAA.... These comprehensive reports will help future studies easily reproduce experiments.
ACKNOWLEDGEMENT
We thank Chongdae Oh and Seongheon Park for their valuable feedback on the draft. This work was supported by Laboratory Directed Research and Development funding from Argonne National Laboratory, provided by the Office Science, U.S. Department of Energy under Contract No. DE- AC02-06CHI 1357. Sharon Li acknowledges support from AFOSR Young Investigator Program under award number FA9550-23-1--0 13-1-18444 National Science Foundation under awards IIS-223 2237037 and NSS-2331669) Office of Naval Research under grant number 200014-23-1 1-2643,, Schmidt Sciences Foundation, Open Philanthropy, and Alfred P. Sloan Fellowship. The funders had no role in study design, data collection, data analysis interpretation, manuscript preparation.
10

---

Published as a conference paper at ICLR 2026
BIBLIOGRAPHY
Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023.
Chao Chen, Kai Liu, Ze Chen, Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and lieping Ye. INSIDE: LLMs' internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, 2024.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting Ilms: Towards retrieval- -aggeeene large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 24, pp. 4491- 2024. ISBN 9798400704901.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Serrieval-augmente generation for large language models: A survey. arXiv preprint arXiv:2312. 232::23/2.10997 2024.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 30-45, 2022.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723-773, 2012. ISSN 1533-7928.
Jiatong Han, Jannik Kossen, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in llms. In ICML 2024 Workshop on Foundation Models in the Wild, 2024.
Xiangkun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, and Zheng Zhang. Refchecker: Reference- based fine the-ggriied hallucination checker and benchmark for large language models. arXiv preprint arXiv:2405. 14486, 2024.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng. Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 2024. ISSN 1046-8188.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprint arXiv: 10.006825, 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac -oadi,, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know, arXiv preprint (tti:22207.0522222 2022.
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. Large language models with controllable working memory, In Findings of the Association for Computational Linguistics: ACL 2023, 2023.
Daria Lioubashevski, Tomer Schlank, Gabriel Stanovsky, and Ariel Goldstein. Looking beyond the 1o Transformers determine top tokens in order. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
AI @ Meta Llama Team. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
11

---

Published as a conference paper at ICLR 2026
AI @ Meta Llama Team. The Ilama herd of models. arXiv preprint arXiv:2407.21788 2024.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity- based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
Junliang Luo. Tianyu Li, Di Wu, Michael R. M. Jenkin, Steve Liu, and Gregory Dudek. Hallucination
detection and hallucination mitigation: An investigation. arXiv preprini arXiv:2401.08358, 2024.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations, 2021.
Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: Zero-resource black-box halluci- nation detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing. 2023.
MetaAl
https: http://ttttttt//ataaaaaaao..oee hotp://ggggiggaaaccooottttttteea /alamaaattaalaaaa/aaaaaaaaaaaaaa allama-- 4- Swwssstoottt-oo -1 Instruct, 2025.
Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. RAGTruth: A hallucination corpus for developing trustworthy reurieedl- language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers). 2024.
nostalgebraist. interpreting gpt: the logit lens, 2020. URL hww.ttttp////c /wwp://wwwwww//mm/ lesswrong com/ post Sestss/AcccssssDdaaavvesso wwwrrr/inneepeetiie
tept-the--ae -oao--ogitt--- lens.
Seongheon Park, Xuefeng Du, Min-Hsuan Ych, Haobo Wang, and Yixuan Li. Steer LLM latents for hallucination detection. In Forty-second International Conference on Machine Learning, 2025.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Salch, Balaji Lakshminarayanan. and Peter Liu. Out-of-distribution detection and selective generation for conditional language models. In The Eleventh International Conference Learning Representations, 2023.
Fabian Ridder and Malte Schilling. The hallurag dataset: Detecting dlosed hallucinations in rag applications using Ilm's internal states. arXiv preprint arXiv:2412.17055: 2025.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association Computational Linguisties: EMNLP 2021, 2021.
Kaiser Sun, Fan Bai, and Mark Dredze. What seen cannot be unseen: The disruptive effect of knowledge conflict on large language models. arXiv preprint arXiv:2506.06488, 2025a.
ZhongXiang Sun, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Weijie Yu, Yang Song, and Han Li. RedeEP: Detecting hallucination in strieval-augmennt generation via mechanistic interpretability. In The Thirteenth International Conference on Learning Representattions, 2025b.
Yufei Tao, Adam Hiatt, Rahul Seetharaman, and Ameeta Agrawal. for quantifying contextual grounding in large language models. arXiv preprint arXiv:2507.05442 2025.
Framework
Lei Wang. Seredeep: Hallucination detection in retrieval-augmenter models via semantic entropy and context-parameter fusion. arXiv preprint arXiv:2505.0 077528 2025.
Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. Knowledge conflicts for LLMs: A survey. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Khurram Yamin, Gaurav Ghosal, and Bryan Wilder. Lims struggle to perform counterfactual reasoning with parametric knowledge. arXiv preprint arXiv:2506.1577 15732, 2025.
12

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388. 2025.
Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, and Luoyi Fu. Enhancing uncertainty-basee hallucination detection with stronger focus. Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, 2023.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
CONTENTS
Broader Impacts
14
Details of Baselines
14
C Details of Datasets
15
D Implementation Details of LUMINA
15
E Additional Experimental Results
16
E.1 Evaluation with Other Metrics
16
E.2 Compare with supervised baselines...
17
E.3 Performance with Hyperparameter Tuning
17
E,4 Additional Ablation study
18
F Error Analysis
18
G Computational Resources
19
A BROADER IMPACTS
Beyond hallucination detection, LUMINA has broader impacts in interpretability and LLM under- standing. Specifically, our proposed score validation framework in Sec. 3.3 suggests novel way to empirically validate the finding of mechanistic interpretability, which can be used to highlight the soundness of proposed hypotheses. In addition, our proposed information processing rate in Sec. 3.2 presents new lens for examining the internal states L.LMs. Deeper investigation of this measure could help the community better characterize how LLMs reason and leverage internal knowledge, potentially leading to more reliable training and inference processes. While our experiments focus on using LUMINA for hallucination detection, its utility extends further. For instance, could inform the design of new training objectives or decoding algorithms aimed at mitigating hallucinations, ultimately making LLMs more reliable and trustworthy.
B DETAILS OF BASELINES
(1) Token-level uncertainty:
Perplexity:
This approach measured the perplexity of the generated response uncertainty and to detect hallucinations.
LN-Entropy: This approach measured sequence-level uncertainty with entropy normalized by sequence length. higher entropy indicates greater uncertainty and higher likelihood of hallucinations.
This approach used entropy and token probability based score, and calibrated it by focusing only on key informative tokens and propagating the score according to the attention weight.
Focus:
(2) Cross-sample consistency:
14

---

Published as a conference paper at ICLR 2026
SelfCKGPT:
This approach sampled multiple responses and used an NLI model to check the logistic consistency between the target generation and additional samples. In our experiment, we follow the setting of Manakul et al. (2023) to set the sample size 20.
Similar to SelfCKGPT, this approach sampled multiple responses and checked the semantic consistency between the additional samples and the target generation through measuring the cigenvalues of responses' covariance matrix. In our experiment, set the sample size as 20.
EigenScore:
(3) Verbalization:
P(True): This approach prompted an LLM with the generated answer and asked whether the LLM think the answer true. The approach then estimated the probability of the "Yes" generated by the LLM.
RefChecker:
This approach prompted an LLM to extract claims from generation, and prompted another L.M to verify the logical consistency between each claim and reference documents. In our experiment, we use dongyru Mi the model finetuned by Hu et al. (2024), to extract claims.
im-Ext ractor,
(4) Utilization of external context and internal knowledge:
For external context utilization, ReDeEP measured the cosine similarity between the generated token and topk attended tokens retrieved documents. For internal knowledge utilization, it measured the IS divergence of the vocabulary distributions between logit lens outputs before and after FFN layers in a Transformer. At the end, it weighted summed the two scores to obtain hallucination score.
ReDeEP:
C DETAILS OF DATASETS
The RAGTruth dataset is human annotated hallucination detection dataset, containing 15,090 training data and 2,700 testing data. Each data point consists of a query, retrieved documents, LLM-generated answer, and span-level hallucination annotation. The dataset covers three tasks, including summarization. data to text generation, and question answering. For each query-and- documents pair, RAGTruth provides answers generated by six different LLMs, including Llama2-7B, Llama2-13BB 13B. Llama2-70B, Mistral-7B, GPT-3.5, and GPT-4. In our experiment, we also utilize the extended test set provided by Sun al. (2025b), who curated and annotated Llama3-8B generated responses.
RAGTruth.
HalluRAG. HalluRAG is an LLM annotated hallucination detection dataset for question answering. Ridder & Schilling (2025) prompted GPT-4o to generate question given sentences from Wikipedia, then used Llama2-7B, Llama2-133 and Mistral-7B to generate answer for each question given the relevant Wikipedia article. The hallucination labels were assigned by GPT--o with Chain-of- Thought (CoT) prompt and verified by human. HalluRAG contains both answerable and unanswerable questions, while we only the answerable instances for evaluation.
D IMPLEMENTATION DETAILS OF LUMINA
For external context utilization, we measure MMD with Eq. (6), which requires summing over the combinations of the entire vocabulary. In practice we approximate it with the top 100 tokens to reduce the computational cost. To obtain Paxx', our experiment we treat the retrieved documents of another data point the d' of the target data point. In a real-world RAG system, d' can be obtained by selecting random documents from the data store or retrieving less relevant documents the query with retrieval model.
For internal knowledge utilization, Eq. (10) computes the first information process rate of generating a, based on the next token with the highest probability. However, due to the sampling process of generation, the generated token a, is not always the highest probability token. Thus, the internal knowledge used during the generation process may not fully apply to az. To take this factor into
15

---

Published as a conference paper at ICLR 2026
Table LUMINA consistently achieves balanced precision-recall trade-off and high F1 score across datasets and LLMs. We report the score of PrecOpt. Recallopt. and Fl for LUMINA and baselines on each dataset.
RAGTruth
HalluRAGi
LLM
Approach
PreCopt 1
Recallopt f
Flopt f
PreCopt 1
Recallope f
Flops f
Perplexity
0.5080
0.9867
0.6707
0.2531
1.0000
0,4040
LN-EEtropp
0.6303
0.7920
0.7020
0.7143
0.7500
0.7317
Focus
0.5276
0.9292
0.6731
0.3077
1.0000
0,4706
SeI[CKGPT
0,5125
1.0000
0.6777
0.2631
1.0000
0,4167
Llama2-7B
EigenScure
0.5201
0,9735
0.6780
0.4333
0.6500
0,5200
P(True)
0,5079
0,9956
0.6726
0,3065
0,9500
0.4634
RefChecker
0,5022
1.0000
0.6686
0.2532
1,0000
0,4040
ReDeEP
0.6898
0.7478
0.7176
0.4167
0,7500
0.5357
LUMINA
0.7131
0.7699
0.740M
0.7826
0.9000
0.8372
Perplexity
0,4926
0,9662
0.6525
0.1519
1,0000
0.2637
LN-Entropy
0.6602
0.8164
0.7300
0.5385
0.5833
0,5600
Focus
0.4938
0.9565
0.6513
0.5556
0.4167
0,4762
SelfCKGPT
0.4801
0.9903
0.6467
0.3056
0,9167
0,4583
Llama2-13B
EigenScure
0,5389
0,9034
0.6751
0.5833
0,5833
0,5833
P(True)
0.6957
0.6923
0.2449
1.0000
0.3934
RefChecker
0.4600
1.0000
0.6301
0.2727
0,2500
0,2609
ReDeEP
0.7772
0.7246
0.7500
0.4706
0.6667
0,5517
LUMINA
0.7816
0.7778
0.7797
1.0000
0,7500
0.8571
Perplexity
0.6369
0.8519
0.7289
LN-Entropy
0,5852
0.9465
0.7233
Focus
0.5571
0.9630
0.7059
SelfCKGPT
0.5657
0.9918
0.7205
Llama3-8B
EigenScore
0,5907
0,9383
0.7250
P(True)
0,5718
0.9342
0.7094
RefChecker
0.5400
1.0000
0.7013
ReDeFP
0.6621
0.7901
0.7205
LUMINA
0.6988
0.7449
0.7211
Perplexity
0.6187
0.9243
0.7412
0.1702
0,8000
0.2807
IN--Ettrppy
0.6890
0.9040
0.7820
0.8571
0.6000
0.7059
Focus
0.7175
0.9004
0.7986
0.7143
0.5000
0.5882
SeISCKGPT
0.5914
0.9920
0.7411
0.5385
0.7000
0.6087
Mistral-7B
EigenScore
0.5931
0.9522
0.7309
1,0000
0.5000
0.6667
P(True)
0.7030
0.8486
0.7690
0.3333
0,3000
0,3158
RefChecker
0,5578
1.0000
0.7161
0.1266
1.0000
0,2247
ReDeEP
0.6506
0.8640
0.7423
0.6250
0,5000
0.5556
LUMINA
0.6600
0.9320
0.7728
0.9000
0.9000
0,9000
account, we calibrate the internal knowledge score by the ratio of probability between the generated token and the highest probability token. In the end, the calibrated internal knowledge score of a, is defined as
Ipo (arlg, d, act) Po(a, q, d, act) po(as,1 lg, d, aze)
Ry Rg, acx))
(11)
E ADDITIONAL EXPERIMENTAL RESULTS
E. EVALUATION WITH OTHER METRICS
Table 3 shows the scores of Precopt. Recallopt- and F'lpt each dataset. The results show that LU- MINA consistently has balanced precision-recal recall trade-off, where the differences between Precopt and Recallopt
smaller than other baselines. Specifically, it achieves (Precopt- Recallopt) = (0.9. 0.9) on HalluRAG with Mistral-7B. This suggests that LUMINA does not over over-predict hallucina- tions achieve high F1 Opt score.
16

---

Published as a conference paper at ICLR 2026
Table 4: LUMINA achieves a competitive performance against supervised approaches. We report the score of AUROC (ROC), Pearson's correlation coefficient (PCC), and AUPRC (PRC) for LUMINA and baselines on each dataset. The highest scores are set in bold.
RAGTruth
HalluRAG
LLM
Approach
ROC f
PCC
PRC
ROC
PCC
PRC
SAPLMA
0.6508
0.2530
0.6446
0.8813
0.6710
0.8023
Llama2-7B
LUMINA
0.7646
0,4546
0.7491
0.9153
0.6554
0.7572
SAPLMA
0,8337
0,5623
0.8466
0.8925
0.8249
0.8647
Llama2-13B
LUMINA
0,8569
0.6041
0.8436
0.9166
0.6044
0.8497
SAPLMA
0.8073
0.5027
0.8164
0.9667
0.7920
0,9088
Mistral-7B
LUMINA
0.7685
0.4623
0.7942
0.9899
0.7529
0.9431
0.880
LLM
Llama2-7B
0.500
Llama2-13B
0.715
Llama3-8B
AUPRC
0.750
Mistral-7B
0.725
0,700
0.875
0.8
0.4
0.5
0.6
0.7
0.8
0.9
Figure 6: A good performance of LUMINA happens with a medium À value. We alter À in Eq. (1) control the weight of internal knowledge score and external context score and evaluate the resulted hallucination detection performance. We conduct the experiment on the RAGTruth dataset and report the AUPRC score.
E.2 COMPARE WITH SUPERVISED BASELINES
We further compare LUMINA with SAPLMA (Azaria & Mitchell, 2023), supervised approach that trained a MLP model over the internal hidden states of the last generated token to classify whether the generation is hallucination or not. Following the original paper, use hidden states at the 20th layer as input features SAPLMA. Result Table 4 shows that LUMINA has a competitive performance against SAPLMA and even sometimes outperforms it. Note that Table 4 doesn't show the result of Llama3-8B as the training set doesn't 't contain responses generated by Llama3-8B.
E.3 PERFORMANCE WITH HYPERPARAMETER TUNING
We evaluate the hallucination detection performance with À E 0..1, 0.4,0.. 0.6, 0.7, 0.8, 0.9). Figure shows the AUPRC different À on the RAGTruth dataset. The results show that the LUMINA achieves the optimal performance with varies À across LLMs. For Llama2-13B and MBstral-7B, setting = 0.5, i.e., the default setting. is the optimal. While for Llama2-7B and Llama3-8B, the optimal À is 0.2. However, for these two models, their performance only drops less than 0.025 when setting À = 0.5, suggesting that weighting internal knowledge and external context utilization equally is still good practice.
17

---

Published as a conference paper at ICLR 2026
Table Both components of information processing rate are important. We report the AUROC of each component, as well as the performance of their combination.
LLM
Layer weighting
Entropy normalization
Both
Llama2-7B
0.7023
0.7164
0.7652
Llama2-13B
0,8007
0.8433
0.8554
Llama3-8B
0,7512
0.7683
0.7697
Mistral-7B
0.6111
0.6723
0.7679
E.4 ADDITIONAL ABLATION STUDY
When implementing LUMINA, we approximate MMD with the top k tokens and set k 100, aiming to balance between computational cost and approximation eror. To test the impact of k, we ablate on {50, 100, 500) and evaluate on RAGTruth dataset. The result shows consistent AUROC across different k, with 0.02% difference, suggesting that LUMINA is insensitive to the choice of MMD approximation. Additionally, in cases where the computational power is limited, choosing 50 is also considerable.
Impact MMD approximation.
Impact of random documents. In Section 4.3, we study the impact of noises in the retrieved and random documents. To further examine the impact of different random documents on the performance, select 5 different random documents and use each of them to compute the hallucination score. Experiments on the RAGTruth dataset shows that the standard deviation across the 5 rounds with different random documents is < 0.0025, suggesting that LUMINA is very robust to the choice of random document.
Impact of the components of the information processing rate. Our proposed information pro- cessing rate consist of two components: layer weighting probability ratio (numerator) and entropy normalization (denominator). Table shows the AUROC of ablating these two components. The result shows that both components contribute to the overall performance, jusfirianceejusiiyyiinn our design choice.
F ERROR ANALYSIS
To analyze the failure of LUMINA, we sample 40 cases from the RAGTruth dataset that are (1) hallucinated with high-external context and low-internal knowledge scores (i.e., false negative) or (2) non-hallucinated with low-external context and high-internal knowledge scores (i.e., false positive). We qualitatively analyze these cases and categorize them into three groups:
(1) Incorrect labels. Sometimes LLMS generate fabricated content that is not sourced from the retrieved document (e.g., a detailed menu of a restaurant). However, these fabricated contents are sometimes not identified by human annotators. Also, human annotators sometimes misclassify semantically equivalent content as hallucination. In these cases, the provided labels are incorrect, and LUMINA indeed correctly detects hallucination.
(2) Generally low hallucination score for the summarization task. We observe that many false negative samples come from the summarization task. these cases, the LLM does generate content that contradicts the retrieved documents and has relatively high internal knowledge score. However, since most of the generated content is still grounded the retrieved documents, they usually have a high external score well, resulting in relatively low hallucination score. This observation suggests that different tasks might have different distributions of hallucination scores. A better practice is to independently evaluate the hallucination detection performance each task.
(3) Low quality of retrieved documents. For the false positive cases, we observe that many of them are due to the quality issue of the retrieved documents. These documents often contain only irrelevant information over them and respond with "unable to answer" or use its internal knowledge to generate answers with details and examples. This results in relatively high internal knowledge score and low external
too vague to concretely answer the query. Thus, the LLM has to reason
18

---

Published as a conference paper at ICLR 2026
Table The errors of LUMINA are mainly due incorrect labels, quality of retrieved docu- ments, and task-dependent biases. We report the proportion of each error type classified by GPT-5.
Error Type
Proportion
False Positive
Incorrect labels
32%
Low quality retrieved documents
24%
Others
44%
False Negative
Incorrect labels
16%
Low hallucination score for summarization task
64%
Others
20%
Table 7: LUMINA more effecient than ReDeEP. We report the average computational time (second/sample) for ReDeEP and LUMINA.
LLM
ReDeEP
LUMINA
Llama2-1 7B
0.86
0.69
Ilama2-1 17B
1.17
0.88
LJama3-8B
1.13
0.58
Mistral-7B
0.54
context score. To address this, a future direction can focus assessing whether the utilization of internal knowledge is necessary and correct, and using that calibrate the hallucination score.
We extend the еттоr analysis by sampling 50 false positive and 50 false negative cases, and prompting GPT-5 classify the reason for error. The result in Table shows that while there are edge cases that LUMINA can not handle correctly, many of the errors are due to incorrect labels and low quality of retrieved documents. For those edge cases, we observe that they usually happen when the internal knowledge and external context scores are close and when the task is more reasoning intensive. Thus, when deploying LUMINA, controlling the balance between internal knowledge score and external knowledge according to the task might be good practice to further increase the performance.
G COMPUTATIONAL RESOURCES
LUMINA lightweight and efficient approach, which requires only two forward passes to obtain the necessary information to compute external context and internal knowledge scores. As LUMINA does require generating multiple samples nor training. it is easy scale up a large amount of data. All the experiments of LUMINA are conducted on a single Nvidia H100 GPU. The execution time of computing both external context and internal knowledge scores varies depending on the length of the response. For responses around 1 150 tokens, the average computational time is less than second. In addition, while LUMINA requires two forward passes to compute the score, efficient than ReDeFP, shown in Table 7. We believe that it is because for the external context score, ReDeEP has to store the entire attention map for every layer and use that to select the top tokens from the external context. And for the internal knowledge score, ReDeEP has to apply the logit lens before and after FEN for each transformer layer. In contrast, our external context score only requires approximating MMD the output layer, and our internal knowledge the logit lens only once per layer. These design choices reduce the computational cost, making LUMINA much more efficient.
consistently more
needs applying
19


Reviews:
[{"review_id": "npfwigWV39", "reviewer": "Reviewer_VZPC", "summary": "This paper propose LUMINA, a framework that detects hallucinations in RAG systems with external and internal context\nutilization metrics. The external context utilization metric measure the maximum mean discrepancy between two next token probability distributions conditioned on retrieved documents and random documents. The internal knowledge utilization metric introduce the idea of information processing rate by looking at the ratio of the most probable output token’s probability across transformer layers and\nuse it to determine the amount of utilized internal knowledge when generating the next token. The experiments show that LUMINA outperforms both score-based and learning based methods in hallucination detection.", "questions": "1. If a model can use internal knowledge to solve many RAG queries, how should we rate this model under LUMINA?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper attributes the hallucination problem in RAG systems to LLMs' preference for utilizing internal knowledge versus external retrieval information. It argues that a good RAG system should pay more attention to external retrieval information, and designs two related metrics to quantify the ability to utilize external information and the ability to utilize internal knowledge. The idea is interesting and the metric design logic is sound.\n\n2. This paper presents a framework to statistically validate the metrics, showing the soundness of the evaluation approach.\n\n3. The experimental results show that the proposed evaluation approach works well on relevant benchmarks.", "weaknesses": "1. Although Section 3.2 describes the relationship between internal knowledge utilization and output token probability across transformer layers, it is not clear how the internal knowledge works in black-box LLMs. Could you provide some concrete examples demonstrating that the internal knowledge utilization metric effectively captures the overuse of the internal knowledge?\n\n2. The experiments are conducted on outdated models, with the most recent being Llama 3, which was released 1.5 years ago. Could you provide additional experimental results on newer models such as Qwen 3 or Llama 3.2 to demonstrate the consistent superiority of your approach?\n\n3. How about the evaluation resource cost of LUMINA? There should be a comparison between LUMINA and other related works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zDhdFh9Qz5", "reviewer": "Reviewer_VWtx", "summary": "This paper introduces LUMINA, a framework for detecting hallucinations in Retrieval-Augmented Generation (RAG) systems by jointly modeling external context utilization and internal knowledge utilization. The method assumes that if an LLM effectively leverages external documents, replacing them with random ones should significantly alter the model’s output distribution. Internal knowledge usage is estimated through layerwise token evolution, measured by changes in predicted token distributions across transformer layers. LUMINA integrates these two signals into a unified hallucination score and provides a statistical validation framework. Experimental results across common RAG hallucination benchmarks demonstrate consistent improvements in AUROC and AUPRC compared to prior utilization-based detectors, with robustness under noisy or mismatched retrieval settings.", "questions": "1.\tHow does LUMINA distinguish between cases where the model truly ignores external context and cases where external and internal knowledge simply overlap?\n2.\tCould the authors provide examples where correct answers yield low context-utilization scores, and explain how such cases are handled?\n3.\tHow might the proposed framework adapt to reasoning tasks that integrate both context and internal knowledge, such as multi-hop question answering?\n4.\tWhy does LUMINA remain robust under noisy retrievals, given that both proposed metrics are highly dependent on the correctness of external documents?\n5.\tWould integrating interpretability analyses (e.g., attention visualization or neuron probing) strengthen the causal link between the measured utilization signals and factual reliability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tClear motivation addressing persistent hallucinations even with correct retrievals.\n2.\tSimple yet effective framework combining two measurable utilization signals without model retraining.\n3.\tComprehensive experiments across multiple LLMs and benchmarks, with robust results under noise and model mismatch.\n4.\tPractical applicability — LUMINA is lightweight, reproducible, and generalizable to different RAG architectures.\n5.\tStatistical validation adds methodological rigor to empirical measurements, distinguishing the work from heuristic detectors.", "weaknesses": "1.\tAssumption vulnerability — The external-context utilization metric assumes that replacing relevant documents with random ones should change generation probabilities. This fails when the model already knows the correct answer from its parametric knowledge, leading to potential false negatives.\n2.\tLimited generalization to reasoning tasks — The internal-knowledge utilization signal may not handle complex reasoning cases where correct responses rely on both external and internal information, potentially misclassifying reasoning-based outputs as hallucinations.\n3.\tInsufficient explanation for robustness — While LUMINA shows stability under noisy retrievals, the paper does not clearly justify why its metrics remain reliable when external documents are irrelevant or partially correct.\n4.\tMissing theoretical grounding — The framework treats token-level utilization as a direct indicator of truthfulness, but lacks analysis connecting these signals to semantic correctness or factual alignment.\n5.\tIncomplete discussion of failure cases — The paper could better analyze when and why LUMINA misclassifies or underperforms, especially under low retrieval relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IjjbVk4wUX", "reviewer": "Reviewer_TLT8", "summary": "This paper proposes LUMINA, a framework for detecting hallucinations in Retrieval-Augmented Generation (RAG) systems by quantifying external context utilization and internal knowledge utilization. The authors measure external context utilization via Maximum Mean Discrepancy (MMD) between predictive distributions conditioned on retrieved vs. random documents, and internal knowledge utilization by tracking token prediction evolution across transformer layers (information processing rate). The method is validated through statistical hypothesis testing and demonstrates strong performance on RAGTruth and HalluRAG benchmarks across four open-source LLMs.", "questions": "1.\tCan you provide theoretical analysis or bounds on the MMD approximation error when using top-100 tokens?\n2.\tHow sensitive is the method to the choice of random documents d'? Have you tried adversarial selection?\n3.\tCan you ablate the components of Equation (8) to show the contribution of layer weighting vs. entropy normalization?\n4.\tWhat is the computational cost comparison (wall-clock time, memory) vs. baselines like ReDeEP?\n5.\tWhy is HalluRAG missing Llama3-8B results?\n6.\tCan you provide more quantitative error analysis beyond 20 sampled cases?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tNovel and principled approach: The decomposition of hallucination signals into external context and internal knowledge utilization is well-motivated. The use of MMD for context sensitivity and information processing rate for internal knowledge are creative and theoretically grounded.\n2.\tStatistical validation framework: The hypothesis testing approach (Section 3.3) to validate the proposed measurements is a significant contribution. This addresses a critical gap in prior work (e.g., ReDeEP) that lacked validation of whether their scores truly reflect utilization.\n3.\tComprehensive experiments: Testing across 4 LLMs, 2 datasets, multiple baselines, and extensive ablations demonstrates thoroughness.", "weaknesses": "1.\tLimited theoretical justification for Conjecture 1: The core assumption that hallucinations occur when I_pθ >> E_pθ is presented as a conjecture without formal proof. While empirically validated, stronger theoretical grounding (e.g., information-theoretic analysis) would strengthen the contribution.\n2.\tMMD approximation concerns: (1) The approximation using only top-100 tokens (Appendix D) may miss important distributional information; (2) No analysis of approximation error or sensitivity to this choice; (3) The choice seems arbitrary without justification.\n3.\tRandom document selection: The paper states d' is obtained from \"another data point\" but doesn't discuss: (1) How this choice affects results; (2) Whether document similarity between d and d' matters; (3) Potential biases from this selection strategy\n4.\tEvaluation limitations: (1) No comparison with more recent methods (paper only compares with ReDeEP from 2025); (2) Missing evaluation on longer-form generation tasks; (3) HalluRAG doesn't include Llama3-8B results (unexplained gap)\n5.\tComputational cost: While claimed to be \"lightweight\" (Appendix G), requiring two forward passes with full hidden states may still be expensive for very long contexts. No comparison of computational cost vs. baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)