Neural Message-Passing on Attention Graphs for Hallucination Detection
Authors: Fabrizio Frasca
Guy Bar-Shalom
Yftah Ziser
Haggai Maron
Venue: ICLR 2026 Poster (2026)
Keywords: hallucination detection
graph neural networks
LLMs
attention graphs
Primary area: learning on graphs and other geometries & topologies
TL;DR: We propose CHARM, a message-passing neural network that models LLM computations as attributed graphs to detect hallucinations; it integrates attention and activation signals, subsumes and outperforms prior methods across benchmarks and granularities.

Abstract: Large Language Models (LLMs) often generate incorrect or unsupported content, known as hallucinations. Existing detection methods rely on heuristics or simple models over isolated computational traces such as activations, or attention maps. We unify these signals by representing them as attributed graphs, where tokens are nodes, edges follow attentional flows, and both carry features from attention scores and activations. Our approach, CHARM, casts hallucination detection as a graph learning task and tackles it by applying GNNs over the above attributed graphs. We show that CHARM provably subsumes prior attention-based heuristics and, experimentally, it consistently outperforms other leading approaches across diverse benchmarks. Our results shed light on the relevant role played by the graph structure and on the benefits of combining computational traces,  whilst showing CHARM exhibits promising zero-shot performance on cross-dataset transfer.

Published as a conference paper at ICLR 2026
NEURAL MESSAGE-PASSING ON ATTENTION GRAPHS FOR HALLUCINATION DETECTION
Fabrizio Frasca*
Guy Bar-Shalom"
Technion
Technion guybs 98gma i com
Yftah Ziser
Haggai Maron
University of Groningen, NVIDIA Research
Technion, NVIDIA Research
ABSTRACT
Large Language Models (LLMs) often generate incorrect or unsupported content, known as hallucinations. Existing detection methods rely on heuristics or simple models over isolated computational traces such as activations, or attention maps. We unify these signals by representing them as attributed graphs, where tokens are nodes, edges follow attentional flows, and both carry features from attention scores and activations. Our approach, CHARM, casts hallucination detection as a graph learning task and tackles it by applying GNNs the above attributed graphs. We show that CHARM provably subsumes prior attention- heuristics and, experimentally, it consistently outperforms other leading approaches across diverse benchmarks. Our results shed light on the relevant role played by the graph structure and the benefits of combining computational traces, whilst showing CHARM exhibits promising zero-shot performance on cross-dataset transfer.
INTRODUCTION
Despite their impressive capabilities, LLMs frequently produce outputs that are factually inaccurate, logically inconsistent, or unsupported by the input context, broadly referred to as hallucinations (Pagnoni al., 202T, Cao al. 2022] Qiu al.) 2023). As LLMs are increasingly applied in diverse domains, detecting ballucinations becomes crucial for ensuring their safe and reliable use. This phenomenon inherently complex and multi-faceted, and methods for automated hallucination detection (HD) have recently received significant attention (Yin et al.) [224 Bar-Shalom et al. 2026).
straightforward approach for HD query LLMs multiple times, either by asking them to judge their own outputs (Kadavath et al., 2022) or by sampling alternative generations to measure semantic variability (Kuhn et al.] 2023). While effective some cases, this strategy requires repeated rollouts, making it both slow and computationally expensive, and thus unsuitable for real-time large-scale use. A more scalable line of work leverages the internal signals produced by LLMs during decoding, which refer to computational traces. In particular, most works focus on linearly probing residual stream activations on selected layers and token positions (Orgad et al., 2025 Azaria & Mitchell] 2023 Belinkov 2022). More recently, attention maps have shown provide additional
perspective on model behaviour, e.g., by leveraging prompt-response attention ratios (Chuang et al.
2024). Although providing meaningful, alternative cues on hallucinations existing attention-based Techniques rely on simple models or handerafted heuristics (Sriramanan et al. 2024; Binkowski al. 2025). Furthermore, all the above methods treat computational traces isolation, despite capturing complementary aspects of hallucinations. To date, systematic exploration of the interplay of computational traces is still lacking. More broadly, the field currently lacks a framework applying modern deep learning techniques to structured, holistic representations of computational traces, leaving the community to rely on heuristic, single-signal approaches.
In this paper, propose unified framework that represents LLM computational traces as attributed graphs, a natural, yet under-explored perspective the HD literature. Similarly to recent works
"Equal contribution.

---

Published as a conference paper at ICLR 2026
LIM
Polluctration prods
GNN
comp trace graph
Figure/ 1: Overview of CHARM We extract attention and activation matrices from LLM computations and build an attributed graph from them: edges and their features are derived from off-diagonal attention scores; node features are based on activations, and diagonal attention values. The resulting graph is processed by GNN-based architecture, which outputs either token-level hallucination scores (as illustrated) or a global hallucination score for the entire sentence.
dealing with the analysis of learnt attention computational flows (Barbero al. 2024: El al. 2025), this formulation considers tokens as nodes and draws connections between them based on the structure of attention maps calculated during text generation. Crucially, both nodes and edges can be endowed with features derived from the values of computational traces across layers (and heads): node features capture token-wise signals such as activations and self-scores (the attention token assigns to itself), while edge features encode pairwise interactions, most prominently the attention between distinct tokens. This perspective casts HD as a graph learning problem, which has recently obtained successes in broad-ranging domains (Monti et al. 2019 Gonzalez et al. 2021 Liu et al.] 2023) and, we argue, is well suited to this task. First, representing computational traces as attributed graphs allows to naturally integrate heterogeneous signals, which may hold varying predictive value across generation tasks. Second, the framework accommodates different levels of detection granularity, with the standard setup of graph classification corresponding to response- detection, and that of node classification to the token-level one. Finally, this formulation directly leverages the rich body of work on Graph Neural Networks (GNNs) and their lide-ribraries (Fey Lenssen 2019), providing principled and well-studied tools for tailored HD models.
Motivated by these advantages, we introduce CHARM (see Figure TDp an HD approach based on a Graph Neural Network operating on computational trace graphs (Gilmer et al., (017). Our framework can jointly process different computational traces, and subsumes known detection heuristics: we prove that it can express recent attention-basee -based methods (Chuang et al. 2024 Sriramanan et al. 2024) either at the token or response granularity levels. Experimentallyy CHARM consistently outperforms these heuristics, Our analyses further reveal that incorporating activations into computational trace graphs alongside attention-features may improve detection of non-contextual hallucinations. Beyond state-of-the-are comparisons, our ablation studies demonstrate the importance of the graph structure, the main princi- ple driving CHARM. Finally, we report promising zero-shot cross-dataset transfer results and observe robustness to graph sparsifications, indicating viable trade-off's between accuracy and efficiency.
well as other leading methods across benchmarks and detection resolutions.
Contributions are summarised as follows. (1) We introduce a unified view of LLM computational traces as attributed graphs, where tokens are nodes connected by attention-induced edges, and both nodes and edges are enriched with features such as activations and attention scores. (2) We introduce CHARM, which casts HD graph learning computational trace graphs. It uses GNN that provably subsumes attention-bant based heuristics, opening new application fronticrs for machine learning graphs. (3) We show that CHARM consistently outperforms leading HD methods across diverse benchmarks and granularities, while exhibiting promising zero-shot transfer capabilities.
2 RELATED WORK
Hallucinations and their detection in LLMs. The term "hallucinations" in LLMs broadly refers to errors in text generation where outputs unfaithful to the input or external facts (Orgad et al.] 2025). These include knowledge inaccuracies, flawed reasoning, biases, and references to non-existing (Liu
et al. 2022
Huang al. 2023: Ji et al. 2023 Rawte al. 2023 Hallucinations involve
2

---

Published as a conference paper at ICLR 2026
complex failures and manifest in subtle ways, including at the granularity of single tokens (Orgdd
(et al. 2025). Early detection approaches leverage uncertainty measures in next-token prediction
or semantic consistency of responses (Kadavath et al.
2022 Varshney et al.)
2023 Kuhn al.
2023 Manakul et al. 2023). Alternatively, recent work
propose detectors on
L.I. computational
Traces; prominent examples include (hidden) activations (Kadavath et al.
2022
Snyder al.
2024
Yuksekgonul et al. 2024, Zou et al., [023] Yin et al.
2024:Chee
Chen et al.
2024
Simhi et al. 2024
Li Liet al. 2024] Marks & Tegmark] |0224] Burns al.
2023 Rateike
[0233]
Bar-Shalom et al.
2025) and attention matrices (Sriramanan et al. 2024 Chuang et al. [024] Binkowski et al. 2025
Bazarova et al., 2025| Zhang et al., [202)). Different traces may be more or less informative for different types of hallucinations: e.g., attention- based heuristics have been evidenced to be predictive in contextual hallucination settings [Chuang et al., 2024). Existing methods mostly rely on heuristics or simple classifiers on specific traces; some are also constrained to coarse detection levels (e.g..
whole responses) (Sirraaanan et al. 2024 Binkowski et al. 2025 Kuhn et al. 2023
Attention-based HD. Irregular or skewed attention behaviours have been observed to often signal
pathological text generation (Xu et al. 2023 Chuang et al. 2024 Binkowski et al.. 2025 Bazarova
et al. 2025 E.g.. hallucinated trans lations may exhibit localised scores narrow context win- dows (uu et al. 2023); ; hallucinations in contextual question answering may correlate with excessive
focus response tokens w.r.f. context ones (Chuang al. 2024). The recent Lookback Lens proposes a detection feature based on this intuition. Other works extract spectral or structural features from attention matrices, c.g., via graph Laplacians combined with logistic regression (Binkowski et al., 2025). These approaches, however, remain limited by fixed heuristics and shallow classifiers. Our work generalises this line by employing attention matrices to construct attributed graphs, a formula- tion that supports predictions at multiple levels of granularity, integrates additional computational signals and unlocks the application of modern (graph-based) deep learning techniques.
Recent works have applied graph- theoretic perspectives to neural computations ( (Vitvitskyi et al. 2025). often graphs induced by attention matrices (Barbero et al. 2024 El et al. 2025). Notably, Barbero al. (2024) analyse signal propagation on attention graphs, uncovering phenomena such as representational collapse and oversquashing (Alon & Yahav 2021 Topping et al., 2022). These studies highlight the value of attention graphs, but are limited to descriptive and structural analyses. In contrast, we extend attention graphs to more general attributed graphs to integrate other computational traces and, importantly, we propose to directly learn on these graphs for the task of HD. To this end, we leverage Graph Neural Networks (Kipf & Welling 2017] Gilmer al. 2017: Battaglia et al. 2018). family of architectures which have recently achieved remarkable results in relevant structured domains (Qasim et al. 2019 Monti et al. 2019 [tokes et al. 20200 Gonzalez et al. 2021 Liu et al. 2023).
Graphs of LLM computation and Graph Neural Networks.
3 LLM COMPUTATIONAL TRACES AS ATTRIBUTED GRAPHS
Preliminaries. Throughout this paper, we focus on attention-basel decoder-only LLMs. Ab- stracting away architectural specifics, we treat them as sharing a common backbone: a stack of transformer-decoder blocks. Let L. denote a reference L.M consisting of T. decoder-block layers of H heads e p refer to a prompt in input, and F' to the response L generates. We consider p7, to be sequences of tokens of size, resp.. ny: n2 (n := nip + n2,). We use T, to refer to token at position i in the concatenation 1 7. Within each transformer block, multi-head attention produces scores, which we collect in attention matrices A [0, 1|mxn for layer 1 and head h. Due to the causal structure of decoder-only transformers, these matrices are lower-triangulas For convenience, we define os E [0, 1]2-H as the vector of attention scores between T; and T3 across all layers and heads. In addition to multi-head attention values, residual stream activations constitute another key source of information about the computation performed by L. For each token T,. we denote by af Rd its d- d-dimensional activation vector at layer 1; this captures the computational state of the model at such token position and processing stage. Together, attention values and activations form the primary signals we use to describe the computational traces of LLMs.. While our framework focuses on these two, it can also naturally accommodate additional sources of information, such as logits.
OOe can also consider, without loss of generality, a different number of heads for each layer.
3

---

Published as a conference paper at ICLR 2026
From computational traces to attributed graphs. The attention values calculated along the way are, in fact, pairwise scores that induce a mion-symmetric) binary relation between tokens. In fact, they define a directed graph G = (V, E) on any sequence of tokens s = p 1 where:
V, the node (vertex) set, is the set of all tokens in s, namely {T:)a:)
E, the edge set, is the set of ordered pairs (T, TT), j. signifying Ti attends to Tý in the generation of next tokens: o > 0 for some of L's layers and corresponding heads.
features representing We consider these graphs as attributed, in the sense that nodes and edges L's computational traces. Edge features are given by the set of attention scores between distinct tokens, i.e., (.(.)))) or, s, with i + j. Node features by token to itself, i.c, TV, = as, Node features can also host other token-wise computational traces; we consider residual stream activations a& at any layer I, so that node / token 7; is endowed with feature vector 2V,; = (as.a a²). Formally, we gather node and edge features in matrices Xv E (Rx((-H+dd) (or -x=x(L-H) should activations be neglected), and Xg E TR The resulting graph is G = L, xxxx)x)) This representation captures both token interactions and per-token computational states, and can be extended to incorporate other traces, e.g., activations from multiple layers or output logits. We leave investigating these aspects future research endeavours.
given by the attention scores "paid"
Sparsifying computational trace graphs. Very small attention scores convey noisy and weak contribution to updating the representation a token. To reduce computational overhead, we threshold attention scores at zeroing values below it and dropping edges unsupported by any head or layer after this process. In formulae, new graph is defined as = (V, E, Xv, XF)), with:
(XE) = f0 Lb o
Lh if a
T. otherwise.
E = ((T,T)) | jンj and 3E s.t. (XF) > 0}. (1)
As experimentally show in Section 5.3 sparsifying the graph in this way may significantly improve the efficiency of our yet-to-be-described model, while retaining information about the most relevant token interactions. In the next section we illustrate how, starting from the above formalism, we can instantiate problems such as automated HD as graph learning tasks.
4
NEURAL MESSAGE PASSING FOR HALLUCINATION DETECTION
4.1 HALLUCINATION DETECTION IS A GRAPHI MACHINE LEARNING TASK
Problem formulation. Computational trace graphs can be naturally associated with labels one seeks to predict for the underlying text generation process. In our specific use-case of HD, these can indeed reflect hallucination annotations at the level of response tokens or the overall response. Coneretely, our reference graph G encoding the computation of L on prompt and response F as per the above Section can be annotated as (G, y), where:
1, if r. contains hallucinating passages
i) y E (0, 13,1
0, otherwise
1, token T > np is part of a hallucinating passage within F.
(ii) y E {0. 1)"r, m/
otherwise
Here, (i) stands for graph-wise label, while (ii) represents labels at the granularity of single (response) tokens. With these premises, we formalise HD as learning parametrie function f(G) mapping a computational trace graph G to predictions of the corresponding labels Depending on the task, ÿ [0, 1]* with k for graph-wise
n, for coken-wise detection.
Our CHARM architecture. We parameterise f in the family of message-passing Graph Neural Network (GNN) (Kipf & Welling 2017; Gilmer et al., 2017). These networks are, to date, the de-facto standard for learning attributed graphs, while possessing an architectural pattern which, as we show next, well aligns them to generalise known approaches. Message paagggggpaiigggggg networks, in particular, implement local computations reflecting the structure of the input graph, hence benefitting
4

---

Published as a conference paper at ICLR 2026
response-lv
mp
pool
preds.
prompt
response
Figure 2: HD with CHARM. The input is attributed graph (shown in the bottom left). First Smp obtains refined node token representations via msg mssgpassing. Next, fpoot aggregates these response level predictions are required. Finally, a projection head, fpedd outputs the detection score.
from the aforementioned sparsification and offering compelling advantage in terms of computational complexity. In particular, we structure as: f fpredd fpool fmp (see Figure 2 where:
fmp stacks learnable message-passing layers to compute updated token representations from the input graph G;
Spool aggregates token representations into a single graph representation (e.g., via averaging summation), or as the identity for token-wise detection;
fpeed applies dense layers to compute graph- or token-wise hallucination predictions.
Starting from the original node (token) features Xy, each layer in Smp calculates and updates hidden node representations by aggregating them (possibly, non-linearly) along the connectivity defined by the attention scores as per Equation (I). That is, the t-th layer updates token i's embedding as:
A (++ up;
(4) JI (F)E
msg, (h.)) .ng . 2E,.))))P..)
where: is is permutation invariant aggregator such sum, average, or max; 2 (...) corresponds
to the features of edge (i,j) in X h IV, are the initial node features; Pt.j is one-hot vector indicating whether edge (i.j) connects prompt to response or response to response tokens. Functions up,, msg, are parameterised as Multi-Layer Perceptrons (MLPs) running on the concatenation of their arguments. We refer to overall approach CHARM, mnemonic formula for "Catching HAllucinated Responses via (learnable) Message-passing""
This formulation offers two main advantages. First, it provides unified framework capable of handling both token-level and response-level HD, in contrast to prior approaches that target a single granularity. Second, by leveraging message-passing over computational trace graphs, CHARM can flexibly integrate multiple signals and naturally subsume heuristic-bare based detectors. As we exemplify next, rather than discarding prior meaningful intuitions, our approach can generalise them.
4.2 EXPRESSIVENESS
Here, we demonstrate how hand-crafted heuristics emerge as special cases of our CHARM. To illustrate this, we focus on two representative methods: (i) Lookback Lens (Chuang et al. (024). which produces token-wing wise hallucination scores, and (ii) LLM-Check (Sriramanan al. 22244) which outputs global sentence-level (graphwise) scores. We show that both heuristics can be approximated, to arbitrary precision and under mild assumptions, by CHARM, highlighting its expressiveness.
Perruutation invariance ensures that calculated message does not depend the ordering of nodes in the neighbourhood.

---

Published as a conference paper at ICLR 2026
Lookback Lens extracts, for each response token i "p:..... ..,,,.. ?p 1, layer I and head h, heuristic feature t quantifying the average proportion of attention paid to prompt w.r.t. previously generated response tokens (Chuang et al. 2024). Precisely, Lockback Lens outputs scores:
p!u
P
R 1
g.h
(3)
7
Rop
pl. + R
where P.A. R correspond to the average attention paid by response tokens to, resp.. the prompt and the (previously) generated response. We argue that this heuristic can, in fact, be interpreted in the form of message-passingg on the (non-thresholded) attention graph described in Section 3 and can
thus be captured by our approach:
message-passing Proposition (informal) 1. Equipped with single-layer message passing stack fmp. CHARM running on a non-onostteeee non-tresholded computational trace graph (T = 0) can arbitrarily well approximate coken-wise Lookback Lens features for bounded prompt and response lengths.
Proof idea. The ability to perform such approximation relies on the following: (1) aggregation over previous tokens, as required by pl is and R. ss is naturally captured by propagating (and aggre- gating) attention features on the neighbourhoods of the directed attention graph as per Equation (2): (2) conditioned on mark input Pt.j. MLP msg can differently route attention features from prompt vs. response tokens in separate subspaces of the internal representations: (3) message summation can separately accumulate attention to prompt versus response tokens (non-normalised Pf R. (4) MLP up can normalise and combine these aggregated scores required, calculating the ratio in Equation 3 A formal statement of informal proposition T is, along with its proof, in Appendix A
LLM-Check proposes detect hallucinations at the level of entire responses: for chosen LLM layer I, LLM-Chk- obtains an "Attention Score" c' by averaging the log-determinants of attention matrices across the set of corresponding heads (Sriramanan al. 2024). Given the peculiar lower- triangular structure of these matrices, such scores can be calculated as:
H-ln-1
d' =
ΣΣ h=0
log(a)))) (.)).
ie., by summing the log- transformed attentions paid by each token to itself, averaged across heads. In practice, the inner summation is replaced with averaging, more robust to prompt and response lengths. We note, in CHARM, these "self-scores" are gathered and processed as node features, which can be transformed and then later aggregated by our architecture to reproduce scores c''s.
Proposition (informal) With a single-layer message saassagg--sssing stack fmp. CHARM can arbitrarily well approximate global SLM-Chk-l f c', provided attentions are clipped away from zero
Proof idea. Intuitively: (1) up MLPs can calculate the initial log- trg-transform on the features of the receiving node/token that is, on each token's ss "self-scores" while discarding information from neighbours; (2) fpoot. set to summation - or averaging, if required - can aggregate these values across tokens; (3) last, fpeed can average these values across heads and selectively for the desired layer to implement the outer averaging in Equation (4). Again, formal statement of informal proposition 2 is reported and proved in Appendix A
Both the two above propositions guarantee that, while general and learnable, approach can also provably default to known, hand-crafted heuristics (under mild, reasonable assumptions). This showcases the expressiveness of the CHARM framework, further evidencing how graph representations and message- paassagg networks can offer a valid and compelling perspective into the task of HD.
5 EXPERIMENTS
We evaluate different aspects of learning with CHARM through the following research questions: (QI) Is our GNN-based formulation effective in practice? (Q2) Is CHARM effective at detecting different
Thiis assumption ensures the log is continuous on appropriate compact set, rendering its approximation amenable (see Appendix A practice we also did observe it necessary ensure numerical stability.

---

Published as a conference paper at ICLR 2026
types of hallucinations and across different granularities (e.g., token-level and full-response)? (Q3) Is the underlying graph structure crucially contributing for CHARM's performance? (Q4) Can CHARM handle large/dense graphs? (Q5) Can the combination of attention and activations be effective in CHARM? (Q6) Does CHARM exhibit any zero-shot transferability across datasets?
We initialise training of CHARM with 3 different random seeds, and, in the following, report the mean test performance along with std. All results are obtained by models optimising validation performance (AUPR, see below). Additional information, including dataset details. hyperparameter searches, implementation notes, and extended results, are available in Appendices B to D Last, we
refer readers to Appendix E for sample visualisations of the constructed computational trace graphs in input to CHARM along with token-wie wise hallucination predictions.
5.1 CONTEXTUAL. TOKEN-LEVEL HALLUCINATION DETECTION
Datasets, We first evaluate our approach at a token-level granularity on the NQ (kwiatkowski et al., 2019) and CNN (See et al. 2017) datasets. These consists of prompt-response pairs with hallucination annotations available at the level of single response tokens (see Section 4.1). These pairs obtained by prompting target LLM to perform either document-based question answering (NQ) or text summarisation (CNN). These datasets contain instances of contextual hallucinations: although the relevant and correct facts are provided in the input context, the target LLM still observed to generate incorrect responses (Chuang et al. 2024). Original generations and annotations for this dataset are derived from (Chuang et al., 2024 coherently with the setup in the same work, we take LLaMa 2- 7B-chat the reference LLM on both datasets. More details are in Appendix B.
Method comparisons. We com- pare against a set of representative baselines. Probability-based detec- tors (Probas) (Guerreiro et al. 2023 Kadavath et al. 2022; Varsh ney et al. 2023 Huang et al. 2025) leverage the next-token prob- abilities to estimate L.MM uncer- tainty and predict hallucinations:
Table I: Test AUROC and AUPR (%) for NQ and CNN (token- wise, higher is better). Bold: best, Underlined: runner-up. 1; we also tune the regularisation strength, differently than the original work.
NQ
CNN
Method
AUROC
AUPR
AUROC
AUPR
Probas
49.8
16.2
$4.4
8.2
Activation probes (Orgad et al.
Act-24
73.0
36.2
71.3
20.3
2025; Azaria & Mitchell, [2023]
Act-28
71.6
34.6
70.1
18.4
Belinkov] (022) (Act-*) train a
Act-32
67.4
28.6
67.7
15.4
logistic classifier on activations at specific layers; the attention- bused, ck Lens heuris-
Lookback Lens
70.8
31.0
71.9
17.4
Lookback Lens
71.9
34.3
74.4
19.7
Neigh-Avg (N)
66.0
24.5
70.1
14.9
tic (Chuang et al.J 2024) fits the same model hand-crafted token- wise features calculated over all lay-
Neigh-Avg (E)
66.8
30.4
70.5
18.6
CHARM (att) CHARM (at)+act-22)
74
40.3
75.4_
22.7
72.2
35.5=16
70.9=02
19.8
and heads (see Equation 3
We run Act-*'s on the common choice of LLM layers 24, 28, 32, motivated by the findings
in (Chuang al. 2024 Azaria & Mitchell 2023). As for CHARM, we instantiate it in configura- tions: one only employing attention features (att), another also utilising activations from a specific layer, which we set to 24 (att+act-24) due its consistently superior performance in *ct We run CHARM on graphs sparsified with a fixed r = 0.05 (see Equation ( and related ablations in Sec- tion 5.3). The HD task is at the level single nodes/tokens, so Speat is set to identity. We additionally consider two ablated versions of CHARM (att): Ne Avv (N). Neigh-Ave (E). These extract token-wise features through single, non-learnable msg-pas bosg-passing step, aggregating, resp., either node or edge features across neighbourhoods in the same computational trace graphs considered in CHARM (details are in Appendix D.I.T). Comparing with these allows us to evaluate the relevance of our multi-layer, learnable procedure. All methods in comparison have their hyperparameters tuned on the Val. set. Performance is measured in terms of Test AUROC and AUPR.
A "Code available at attttps:: //ga::///iiii /tgphhubbb///o///////////o com/ No red red/charm charm
Differently than (Chuang et al. 2024). however, we construct and experiment with a different split ensuring full textual disjointness between train, test, and validation, more in Appendix B.
7

---

Published as a conference paper at ICLR 2026
Results, reported in Table 1 show approach consistently outperforms all baselines across both datasets and metrics in the att.-only configuration. The significant margins over Lookback Lens, Neigh-Ave (N) and Neigh-Avg (E) underscore the benefits of an expressive, learnable graph-based method over attention-based heuristics. This pure attention configuration also surpasses all activation-basee Act-* probes, this contributing to answer positively Q1. We interestingly report that including activations from an intermediate layer into CHARM reveals detrimental. We hypothesise that, on these contextual benchmarks, attention features alone carry most the relevant predictive signal an expressive enough model like ours can leverage it at best and struggle to find additional complementary signals on activations, which could, instead lead fit spurious correlations.
5.2 RESPONSE-LEVEH HALLUCINATION DETECTION
Datasets. We next evaluate CHARM the coarser response-level HD three datasets: Movies OOr gad et al., 2025) WinoBias (Zhao et al. 2018), and Math (Sun et al., 2024). Unlike NQ and CNN, these benchmarks address failure modes different than contextual grounding, namely: factual knowl- edge recall (Movies), intrinsic bias in coreference resolution (WinoBias), and arithmetic reasoning (Math). This allows to assess generalisation across fundamentally different hallucination types. The relative role of attention thereon is not obviously clear, but we hypothesise they can provide infor- mative signals, e.g. by capturing systematic biases in attention demographic cues or by reflecting unusual patterns in intermediate calculations steps. Exploring their interplay with other computational traces thus a insightful direction. For these experiments we derive text generations and hallucination annotations following the procedure in (Orgad et al. (025) and, consistently with this work, target a different LLM, Mistral- 7B-inst-iii instruct.. More dataset details are in Appendix B.2
Method comparisons. As for CHARM and its non-learnable counterparts (iv), set component fpool to average. We compare our method to Act- probing activations in notoriously relevant token positions, e.g., the last token of the prompt, the last of the response (Orgad et al. 2025
(see Appendix C.4). 4). Here, we compare against the response-level attention based LLM-CChed (Sri ramanan al. 2024) (LLM-Chk-") and the spectral-method proposed in (Binkowski et al. 2025)
(LapFig) We also run an enhanced counterpart of (LLM-Chk++-> whereby per-head scores are considered inputs logistic regression, rather than being averaged (Equation (4).
Results are re- ported in Table 2 Overall. CHARM attains the best perfor- mance across these benchmarks, with particularly notable gaps Math. To- gether with the above, these results answer positively to Q1 and Q2. Interestingly, we observe markedly different behaviour than in the previously considered contex-
Table 2: Test AUROC and AUPR (%) for Movies, Winobias, Math (response- lv, higher is better). Bold: best, Underlined: runner-up.
Movies
Winobias
Math
Method
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
Probas
58.6
81.6
64.5
20.0
54.5
57.4
Act-24
77.0
90.4
76.6
37,8
77.7
77.5
Act-28
77.0
90.4
73.9
34.3
78.1
77.8
Act-32
76.3
72.7
35.3
76.6
77.9
L.M-Chk-24
47.5
74.6
38.9
10.9
64.5
68.2
LIM-Chk-28
51.1
76.7
41.6
65.5
69.2
LLM-Chk-32
61.5
82.1
41.6
11.3
64.0
67.6
LIM-Chk++-24
66.3
84.5
64.6
20.5
67.3
69.1
LLM-ChK++-28
67.8
86.3
64.8
21.0
67.0
70.6
LLM-ChX+++32
73.0
67.2
24.1
68.6
72.5
LapEig
72.9
88.4
74.1
33.3
73.6
76.3
Neigh-Avg
78.6
63.8
23.0
77.4
79.2
tual HD datasets. Other than Movies where both our configurations work equally well - on both Winobias and Math, activation-based features work in synergy with attention-based ones. Instead of leading to fit spurious correlations as hypothesised for NQ and CNN [Section[511 they contribute strongly boost performance act.-based methods. This is particularly pronounced on Winobias there, CHARM (att.) is surpassed by Ast but the inclusion of activations leads it (att. & act.-24) to significantly outperform them
Neigh-Avg (E)
54.9
78.5
65.8
21.9
76.7
78.3
outs
CHARM (all)
80.3102
92.0
70.44007
29.1
76.5
79.7208
CHARM (att+act-24))
79.7.19
91.8
77.8194
39.8113
80.8
83.1
the att.-only CHARM, and over all considered att.- and
8

---

Published as a conference paper at ICLR 2026
both. Overall these datasets provide cases leading to positive answer to Q5. We last note that Math the only dataset where Act 24 outperformed by other variants, namely Act-32. We thus also ran CHARM (att+act-32), which scored Test AUROC of 81.7 = 0.2, and AUPR of 83.8 + 0.3.
5.3 ADDITIONAL ANALYSES
We report here a series of additional analyses investigating: (i) The contribution of the graph structure CHARM'S performance; (ii) The impact of the sparsification threshold 7 to the computational efficiency and performance of CHARM; (iii) The zero-shot cross-dataset transfer capabilities of CHARM and comparable approaches. Additional experimental studies are reported in Appendix C including analyses on the impact of the activation layer choice (Appendix (C22 and on the generalisation performance CHARM on sequences of unseen length (Appendix
The contribution of the graph structure.
Table 3: Results from ablating the graph structure.
To what extent does message- message-passing on the attention-induced graph contribute the performance of CHARM? To answer this, we ablate the connectivity in our input sam- ples and train CHARM on two representa- tive datasets: CNN and Math. In this setup, CHARM defaults to set model which, instead of messagg--assingg. applies stack of dense layers over node features. We train and extensively tune this baseline, denoted "CHARM (no g.)", considering both att.-only and att. & act. configurations. We report its best results in Table compared to the best corresponding CHARM, Results clearly show the positive contribution of message-passing on the constructed topology, answering positively Q3.
Method
CNN
Math
AUROC
AUPR
AUROC
AUPR
CHARM (no g.)
70.8 10.5
19.2 105
80.6 107
82.7 10.1 83.8 203
CHARM
75.4 10.2
22.7
204
81.7 102
The impact of efficiency and perfor- mance. We experiment with different values of the attention threshold (Equa- tion ([) studying how graph sparsity and (inference) memory consumption vary relation to performance. We run this study
Table 4: Avg. graph stats. NQ different thresholds, along with GPU memory footprint and Test AUPR.
Num. Edges
Sparsity
Mem. (MB)
AUPR
0.5
1,1 181.8.80
0,993
22.99+3.71
38.420.4
0.1
7.458.67
0,952
60.44
41.0=1.2
on NQ. with results in Table 4 Test AUPR
0.05
14,884.44
0,906
104.15=23.16
40.3+1.7
is reported along with the number of edges,
58,998.88 88
0,645
363.02__98.0
40.3
sparsity and inference memory footprint averaged over test graphs. We observe
0.001
19,7784.82
0.026
1177.20#523.81
40.1
CHARM's performance is robust to various levels of sparsifications, whilst this can provide dra- matic reduction in resource consumption. Performance drops more notably only for r 0.5, which, we note, still outperforms the best competitor, Act Act-24 (see [Tb Overall our default = 0.05 attains good trade-off, whilst note it maximises val. AUPR. These results answers positively to Q4. We finally measure a distinctly contained inference latency of ~ le-3 secs. Refer to Ap- pendix C-I for run-time and performance comparisons with other popular HD methods relying on multiple prompting, which incur significantly higher latency.
Transferring zero-shot to unseen datasets. CHARM is learnable, expres- sive multi-layer approach - this raises natural question: To what extent can it generalise cross-datasets zero-shot? To investigate this, we follow the setup in (Chuang et al. 2024): we train on NQ and evaluate on CNN, and vice-versa.
Table 5: Cross-dataset zero-shot transf. NQ + CNN.
NQ + CNN
CNN + NQ
Method
AUROC
AUPR
AUROC
AUPR
LkbLens Act-24
68.6 63.4
14.9
62.0
26.5
11.3
63.8
29.7
CHARM
64.1 #1.1
12.0
10.98
65.5
10.14
31.6
20.10
Results are in in Table Overall no single method consistently outperforms the others in this challenging setup. In fact, despite its larger expressiveness, CHARM demonstrates promising generalisation: it outperforms activation- probes, ranks best in CNN + NQ, and places second in NQ - CNN (behind Lookback Lens, which conversely performs the worst in CNN NQ). These results suggest that zero-shot transfer

---

Published as a conference paper at ICLR 2026
remains open challenge, but araph-based formulation is competitive and can capture generalisable signals, answering positively to Q6.
CONCLUSIONS
In this work, we proposed attributed graphs as principled formulation of LLM computational traces, showing how diverse signals can be unified this framework and how neural message passing can be applied thereon for diverse HD tasks. We showed our approach, CHARM, can provably generalise prior methods and that it achieves strong empirical performance, consistently outperforming existing methods. Additional analyses underscored the importance of graph structure and demonstrated promising zero-shot generalization across datasets.
Future endeavours will consider integrating other computational traces (e.g., logits), as well as extensions to new tasks such as detecting data contamination, identifying LJ.M-generater text, or flagging jailbreak attempts. Future work may also explore alternative pessage-passing architectures, including positional and structural encodings tailored to these attributed graphs.
ACKNOWLEDGMENTS
F.F. conducted this work supported by an Aly Kaufmmn Post-Doctoral Fellowship. G.B. is supported by the Jacobs Qualcomm PhD Fellowship. HM is supported by the Israel Science Foundation through a personal grant (ISF 264/23) and an equipment grant (ISF 532/23), and by the Career Advancement Chairs in Artificial Intelligence Schmidt Futures. F.F. is extremely grateful the members of the "Eva Project", whose support he immensely appreciates.
REPRODUCIBILITY STATEMENT
Our code is available at https: www /oom/NNola red. cwd//oored//carm along with training and evaluation scripts. Section [5] along with Appendices B to provide the required implementation details to reproduce our results.
REFERENCES
Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. International Conference on Learning Representations, 2021.
Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 967 976,977,220 2023.
Guy Bar-Shalom, Fabrizio Frasca, Yaniv Galron, Yftah Ziser, and Haggai Maron. Beyond token probes: Hallucination detection via activation tensors with ACT-ViT. Advances in Neural Information Processing Systems, volume 38, 2025.
Guy Bar-Shalom, Fabrizio Frasca, Derek Lim, Yoav Gelberg, Yftah Ziser, Ran El- Yaniv, Gal Chechik, and Haggai Maron. Beyond next token probabilities: Learnable, fast detection of hallucinations and data contamination on LLM output distributions. Proceedings of the AAAI Conference on Artificial Intelligence, 2026.
Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João G.M. Araújo, Alex Vitvitskyi, Razvan Pascanu, and Petar Veličković. Transformers need glasses! information over- squashing language tasks. In Advances Neural Information Processing Systems, volume 37, pp. 9811 -91142, 2024.
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinieius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks, arxiv:1806.01261, 2018.
Alexandra Bazarova, Aleksandr Yugay, Andrey Shulga, Alina Ermilova, Andrei Volodichev, Kon- stantin Poley, Julia Belikova, Rauf Parchiev, Dmitry Simakov, Maxim Savchenko, Andrey
10

---

Published as a conference paper at ICLR 2026
Savchenko, Serguei Barannikov, and Alexey Zaytsev. Hallucination detection in LLMs with topological divergence on attention graphs. arXiv:2504.10063, 2025.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguisticss, 48(1):207-219, 2022.
Lukas Biewald. Experiment tracking with weights and biases, 2020. Software available from wandb.com.
Jakub Binkowski, Denis Janiak, Albert Sawezyn, Bogdan Gabrys, and Tomasz Jan Kajdanowicz. Hallucination detection in LLMs using spectral features of attention maps. In The 2025 Conference on Empirical Methods in Natural Language Processing, 2025.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. models without supervision. In International Conference Learning Representations, 2023.
Discovering latent knowledge in language
Meng Cao, Yue Dong, and Jackie Cheung. Hallucinated but factual! Inspecting the factuality of hallucinations in abstractive summarization. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 3340-3354, Dublin, Ireland, May 2022. Association for Computational Linguisties.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs' internal states retain the power of hallucination detection. In International Conference Learning Representations, 2024.
Yung- Chuang, Linlu Qiu, Cheng- Yu Hsich, Ranjay Krishna, Yoon Kim, and James R. Glass. Lookback Lens: Detecting and mitigating contextual hallucinations large language models using only attention maps. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1419-1436. Association for Computational Linguistics, 2024.
Batu El, Deepro Choudhury, Pietro Liò, and Chaitanya K. Joshi. Towards mechanistic interpretability of graph transformers via attention graphs. In ICLR 2025 Workshop on Explainable Al for Science (XAI4Science). 2025.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with PyTorch Geometric, In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1263-1272. PMLR, 2017.
Guadalupe Gonzalez, Shunwang Gong. Ivan Laponogov, Michael M. Bronstein, and Kirill Veselkov. Predicting anticancer hyperfoods with graph convolutional networks. Human Genomics, 15(33), 2021.
Nuno M. Guerreiro, Elena Voita, and André F. T. Martins. Looking for needle in haystack: A comprehensive study of hallucinations in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1059-1075, 2023.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng. Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions Information Systems, 2023.
Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. Look before you leap: An exploratory study of uncertainty analysis for large language models. IEEE Transactions on Software Engineering, 51 2025.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
11

---

Published as a conference paper at ICLR 2026
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Anee Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7b. arXiv:23 arXiv:23/0.0682, 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. arXiv:2207.05221, 2022.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In International Conference on Learning Representations, 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 16622-66,,2 2019.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024.
Gary Liu, Denise B. Catacutan, Khushi Rathod, Kyle Swanson, Wengong Jin, Jody C. Mohammed, Anush Chiappino-Peyy Saad A. Syed, Meghan Fragis, Kenneth Rachwalski, Jakob Magolan, Michael G. Surette, Brian K. Coombes, Tommi Jaakkola, Regina Barzilay, James J. Collins, and Jonathan M. Stokes. Deep learning-guided discovery of an antibiotic targeting acinetobacter baumannii. Nature Chemical Biology, 19(1 (Q11)):1422-333 1350, 2023.
Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui. Weizhu Chen, and Bill Dolan. token-level reference-fre free hallucination detection benchmark for free-form text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 6723-6733 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Potsawee Manakul, Adian Liusie, and Mark F. Gales. SelICheckGPT: Zero-resource black- box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004-9017, 2023.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In Conference on Language Modeling, 2024.
Federico Monti, Fabrizio Frasca, Davide Eynard, Damon Mannion, and Michael M. Bronstein. Fake news detection on social media using geometric deep learning. In ICLR 2019 Workshop on Representation Learning on Graphs and Manifolds, 2019.
Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. LLMS know more than they show: On the intrinsic representation of LLM hallucinations. In International Conference on Learning Representations, 2025.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lnguage Technologies, pp. 4812-4829 Association for Computational Linguistics, 2021.
12

---

Published as a conference paper at ICLR 2026
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta Numerica, 8:143-195, 1999.
Shah Rukh Qasim, Jan Kieseler, Yutaro liyama, and Maurizio Pierini. Learning representations of irregular particle-detector geometry with distance-weighted graph networks. The European Physical Journal C, 79(7), 2019.
Yifu Qiu, Yftah Ziser, Anna Korhonen, Edoardo Ponti, and Shay Cohen. Detecting and mitigating hallucinations in multilingual summarisation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8914-8932, Singapore, December 2023. Association for Computational Linguistics.
Miriam Rateike, Celia Cintas, John Wamburu, Tanya Akumu, and Skyler Speakman. Weakly supervised detection of hallucinations in LLM activations. In NeurIPS Workshop on Socially Responsible Language Modelling Research (SoLaR), 2023.
Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S. M. Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, and Amitava Das. The troubling emergence of hallucination in large language models- an extensive definition, quantification, and prescriptive remediations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. Association for Computational Linguisties, 2023.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp. 1073-1083, Vancouver, Canada, 2017.
Adi Simhi, Jonathan Herzig, Idan Szpektor, and Yonatan Belinkov. Constructing benchmarks and interventions for combating hallucinations in LLMs. arXiv:2404. 09977, 2024.
Ben Snyder. Marius Moisescu, and Muhammad Bilal Zafar. On early detection of hallucinations in factual question answering. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2721-2732 2024.
Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. I.L.-check: Investigating detection of hallucinations in large language models. In Advances Neural Information Processing Systems, volume 37, pp. 34188-3444 34216, 2024.
Jonathan M. Stokes, Kevin Yang, Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz Nina M. Donghia, Craig R. MacNair, Shawn French, Lindsey A. Carfrae, Zohar Aloom-Ackermamm Victoria M. Tran, Anush Chiappino-Pepe, Ahmed H. Badran, Ian W. Andrews, Emma J. Chory, George M. Church, Eric D. Brown, Tommi S. Jaakkola, Regina Barzilay, and James J. Collins. A deep learning approach to antibiotic discovery. Cell, 180(4):688-700 2020.
YuHong Sun, Zhangyue Yin, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Hui Zhao. Benchmarking hal- lucination in large language models based on unanswerable math word problem. arXiv:2403.00 035558, 2024.
Jake Topping, Francesco Di Giovanni, Benjamin P. Chamberlain, Xiaowen Dong, and Michael M. Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. In International Conference on Learning Representations, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu. Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann,
13

---

Published as a conference paper at ICLR 2026
Artem Korenev, Punit Koura, Marie- Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307. 0000009088..... 2023,
Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. A stitch in time saves nine: Detecting and mitigating hallucinations of LLMs by validating low-confidence generation. arXiv:2307.03987, 2023.
Alex Vitvitskyi, João G. M. Araújo, Marc Lackenby, and Petar Veličković. What makes a good feedforward computational graph? In International Conference Machine Learning, Proceedings of Machine Learning Research. PMLR, 2025.
Weijia Xu, Sweta Agrawal, Eleftheria Briakou, Marianna J. Martindale, and Marine Carpuat. Un- derstanding and detecting hallucinations in neural machine translation via model introspection. Transactions of the Association for Computational Linguistics, 1 11:546-564, 2023.
Fan Yin, Jayanth Srinivasa, and Kai-Wei Chang. Characterizing truthfulness in large language model generations with local intrinsic dimension. In International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp.
PMLR, 2024.
Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Eee Kamar, and Besmira Nushi. Attention satisfies: A constraint-sattifactionn lens on factual errors language models. In International Conference on Learning Representations, 2024.
Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang. and Luoyi Fu. Enhancing uncertainty-based hallucination detection with stronger focus. Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 915-932. Association for Computational Linguistics, 2023.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Short Pupers), pp. 15-20, New Orleans, Louisiana, 2018. Association for Computational Linguisties.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency. arXiv:2310.0146 2023.
14

---

Published as a conference paper at ICLR 2026
A EXPRESSIVENESS: CLAIMS AND PROOFS
Proposition (informal) 1. Equipped with single-layer message passiing stack fmp, CHARM running on a non-tresholded computational truce graph (T = 0) can arbitrarily well approximate token-wise Lookback Lens features & for bounded prompt and response lengths.
Proposition 1. Let L L) denote the Lookback Lens features calculated, for token i on string == for LLM E(Equation (.. Also, let K (Gz) denote the t-layer representation for the same token in output from t CHARM mag--assiig layers (Equation (2)) on the corresponding computational trace graph Gz. For any precision > O, there exists a I-layer stack of CHARM's layers which approximate Lookback Lens features up to precision for maximum allowed prompt-length Rpp and response length ny. That is. for any prompt p. response of the aforementioned maximum lengths, and for any response token i, ||0
(2) (G - (p F: lo <.c.
Proof. To prove the above show that, by setting hyperparameters (no thresholding), there exist a single-layer stack fmp which compute the desired approximation. In the following we will consider the attention scores across the L layers and I! heads to be arranged in our node and edge features in flattened manner. We will conveniently denote with (l, h)) the function which evaluates the index of layer and head h in this flattened representation.
Σ (summation) and
(0) Setup and inputs. As our stack is made up of one single message-passinng passing layer, our specific interest is thus in showing the existence of appropriate MLPs msgo, up, enabling fmp to realise the target approximation. Being these components of the firs 1ay-and and only message-passing layer in fmp its input node and edge representations effectively correspond to the original node and edge features Xv, Xg. g. We are thus focussing on the following update:
(SVS))
UPo
msgo (IV,a,
(5)
jUi
(2)
(1)
where the neighbourhood aggregation sums across all previous token positions (j < i) since no thresholding is enforced (r = 0) and due to the fact that attention values cannot exactly evaluate to because of to the application of softmax normalisation.
(1) Message function. Let us first describe what we desire msg, to calculate. We would like it to map the concatenation of its arguments, with dimensionality 3d + 2, d = L H, to vector of dimensionality 2d + 2, where:
The mark feature Pt, replicated in the first two dimensions (channels 0, 1);
Edge features # F. E.(j) are replicated either in channels 2 through + d - 1 if pr.j = [1.0] (message from prompt token) in channels + d through 2d otherwise (message from response token);
Node features æV,,, are discarded.
Now, an MLP exactly implementing the above message function does exist; in fact, it can be explicitly constructed.
First layer. Weight matrix W, is in (2442) We will describe it in terms of columns slices.
A first slice gathers the first two columns (0, 1); these are all zero except for the bottom x block, identity 12. This slice copies the pij mark features in the first channels of the hidden representation.
A second slice gathers columns 2 through 2 d 1; these are all zero except for rows 2d through 3d 1, set to identity Ia- and row 3d, set to Ia one-only vector. This slice calculates the sum between edge features (E.().)) and the first channel of the mark pij+ indicating whether the message comes from prompt token.
15

---

Published as a conference paper at ICLR 2026
A third and last slice gathers columns! - d through 2 2d 1; these are all zero except for rows 2d through 3d 1, set to identity La. This slice copies edge features FF,(i.)) in the last d channels of the hidden representation.
Bias vector bi in I2d+2 is all zero except for channels 2 through 2 + d 1, set to vector - -Ia. Recapping, the hidden representation is a vector in 12 with the following structure:
Pas ... (.EEE66)))) + (pis)
(hr)
(ha)
Second layer: Weight matrix W2 is in ((242))((222) We will describe it in terms of columns slices again.
A first slice gathers the first two columns (0, 1); these all zero except for the top 2 x block, set as identity 12- This slice replicates, again, the ReLU(pp,,) = Pi.j mark features in the first two channels of the output.
A second slice gathers columns 2 through rows 2 through 2 + d - 1 are set to identity I2 and the last d rows are set to Ia. This slice calculates, channel-by-channe
these are zero in the first rows,
(eL, ReL.U(h1) h2 ReLU(hh).
A third - and last - slice gathers columns - d through + 2d - 1; these are all zero except for rows 2 through 2 + d 1, set to identity Ia. This slice copies ReLU(h;) in the last d channels of the output.
Bias vector by is set to zero.
As result, the output a vector in IR 2d22 with the following structure:
Pis (ZE,6S)))an ReLU((EE + (prs)o - -1
1).... ... .... RCLU + (pho)) - -1) .....
(m)
Now, note that, as desired:
* if (pi.)) = (message from response token);
- (o1)au = (KE,Gs))bam ReLU( + (FR
(IE, VLL) = 0
-(02) ReLU +1 +1 (EE,()))
. if (pt.)) = (message from prompt token);
- (01)s(MM) = -FEE - ReLU s(, +0-1) = (FE,0)))) -
(IRL s(.))
((02)aaM)) Rel.U -0-1)=1 1)
Now, when aggregating these calculated messages through summation, it becomes clear that the aggregated message vector m, will eventually hosts:
The number of response tokens preceding token i.e., np in channel
The length of the prompt, i.c., Np, channel 1;
Summation ~np1 0i in channel b(1, h) + 2, denoted p..
Summation Σ Ej=n, aL
i in channel (1,, h) + d, denoted (fR
16

---

Published as a conference paper at ICLR 2026
(2) Update function. Next, desire UPg to implement function up" mapping the concatenation pvy mi to vector of dimension d corresponding to the Lookback Lens output scores in Equation proof.
(3). Showing that up, can approximate up" up to desired precision will complete the
We now describe up". We build it as a composition two functions fA o fB:
* ʃA is such that fa m,) = y: C; + mi, where has the same dimensionality as (2d + 2), and c; [0 0 sv.&] note that vector y/ is an "updated" version of m;
whereby channel now equals i mip and channels b(1, h)) + 2 + d's are now such that:
(R + .V..
L = R
(6)
j
fa is such that fu(v:) 2, where is of dimensionality d and:
(vs) s0A14
(v:),
(3))((.)) = (1)) (0))
(7)
(vi ((())+24
(16)
(1:)
Note that, importantly, (zi) p/ah/ng /np /0-np)
pl pl tef
ph.
First, fA can be realised by a single affine transformation. This has weight matrix WA in R (3d42)) (2d+2), described in terms of row slices as follows.
A first slice gathers the first d rows; it is zero except for the last d-column block, set as identity Ia-
A second slice gathers rows through 3d + 2 - and set as an identity T(2+24)-
The above linear transformation has the effect of copying m, in the output and of summing æV; in its last entries - where the aggregated message from response tokens is stored. Now, bias vector ba is in I2 2+22d and is zero everywhere except for its first element which is set as 1. Adding ba has the effect of simply increasing the second entry by one, thus "updating" the count of response messages stored there.
Second, we note that fa computes the same exact scalar-valued function f3 on each output compo- nent; also, this fi continuous on (the non-compact) domain note that fū be trivially, continuously extended to the compact {1, ................................ Rp f1,1..... n. } x (0, 1J2: it suffices to see that its limits exist and are finite on the boundary of [0, 1j2 This is indeed the case; we note that denominators in each individual normalisation ratios are always greater or equal than one; and that the two addenda in the denominator of the main ratio are always non-negative, can never evaluate simultaneously to zero and their sum is bounded away from zero (given the maximum allowed length for prompt and response).
x [1, .., n, } x (0, 1)² We
Given this premise, term f <s,ext this continuous extension; we can invoke the MLP Universal Approxi- mation Theorem (Pinkus 1999) to claim the existence of an MLP M with one hidden layer which approximates the continuous friest on the compact domain {1,..., mp} (1...., n, } [0. 1/2 up to precision c. This implies the original fī is also c-approximated in its original domain. Now, possible to (easily.) appropriately replicate the weights of M distribute its same exact computation for each of the output coordinates, thus obtaining an MLP M approximating the overall fB: Vy, Vi (M) (fr(v)), < e, that is. Vy, Vi (MM())), (fr(y)), e. But, then:
Vy. Vi ((MB(v)) - -(Jll)), Vy max (I(MB(v)), (IB(v)), i= 0, o,..., ( <e Ay [|Mb()) - fo(u)lbo <c
17

---

Published as a conference paper at ICLR 2026
Denote (W), B: b/)) (W3. b2z) the weight and biases of, respectively, the first and second layers of Mg. Now, the above affine transformation exactly implementing fA can be "absorbed" into the first layer of Mn, by replacing the weight and bias as W WB Wa, (WB 6Λ + bg). The
resulting MLP composed by ((W" ,'), (W bb)) now e-approximates the overall up", concluding the proof.
Proposition (informal) 2. With a single-layer message-passing stack fmp, CHARM can arbitrarily well approximate global L-M-CCh-I featuurs c', provided
attentions are clipped away from zerd
Proposition 2. Let d' (s; L) denote the LLM- LLM-hheck Attention Score (Equation (4) calculated on string 3= 5 for LLM L(Equation 4 and its layer I, Also, let y(Gz) denote the prediction in outpul from an CHARM stacking components Smag: ppotss Sprdd: run on corresponding graph Gg. For any precision e >> 0, when Spoot E and attention values clipped from below to value Ommin O, there exists a 1-layered fmsg, and an MLP fpred such that CHARM approximates LLM-Check Attention Scores up to precision c. That is, for any prompt- response pair s, (y(Gs) -
e (5t L) <c
Proof. To prove the above we show that, in the setting described in the proposition, there exist a single-layer stack fup, well an MLP fpee which compute the desired approximation. Here consider the LLM-Chk-I variant which averages across heads instead of performing a summation, but the proof below easily extended this alternative configuration.
(0) Setup, inputs and proof strategy. Our stack is made up, again, of one single message-passing layer in Smp- followed by sum-based pooling and MIP fpred- Our specific interest is in showing the existence of appropriate MLPs msgo, upo- fpee enabling the full stack to realise the target approximation. Again, msg., UPg are the components of the first and only message- passing layer in Smp, so its input node and edge representations effectively correspond to the original node and edge features Xv, Xg. The whole computation then takes the form:
y fpred
UPo
TV,
Σ (E
msga DVy.
(8)
(3)
(1)
(1) Message function. The LLM-Chk-l method does not perform any aggregation on the attention graph for our purposes it suffices for MLP msg, to simply implement a function which outputs any constant non-negative vector v. W.l.o.g., set v = o; the MLP implementing msg, is trivially obtained by setting both its weights and biases to zero.
(2) Update function. Note that sv, = ar as it will be clearer next, it suffices for upp to approximate the log function applied thereon component-wise Now, log is, in fact, operating on domain [amin 1) in view of the applied clipping: there, the function is continuous. Consider the compact set [amin 1] obtained the closure of the above domain. The considered log is trivially continuously extended to this new domain, since its limit exists finite as the argument approaches from the left (it evaluates to 0). We therefore invoke the Universal Approximation Theorem (Pinkus 1999), which guarantees the existence of an MLP Mogg approximating the component-wiss log function on the extended domain and, in turn, on the original input domain [Omin 1) up to arbitrary precision c. We construct upo by appropriately replicating the weights of Moog to output the approximated log-transform on each of the first input channels in parallel, whilst discarding the last d remaining channels in output from the message function (see above).
(3) Prediction function. At this point, the input of fpeed corresponds to: 2- 1 2i=0 Dt. where Ú an e-approximation of the log function applied element- to ar,, 's. If pred implemented the final averaging over I's H heads (channels), then its output would be overall approximation of the desired quantity e' (of a certain precision yet to be quantified). We note that it easy to explicitly construct such MLP exactly implementing the outer averaging over the selected heads. We describe its two layers in the following.
TThis assumption ensures the log is continuous appropriate compact rendering its approximation amenable (see Appendix A practice we also did observe it was necessary ensure numerical stability.
18

---

Published as a conference paper at ICLR 2026
First layer: it features a weight matrix of the form [La -Ia] and zero- zero-valued bias; this layer expands the d-dimensional input to a 2d representation, where the the first channels host input 2, the last d channels the negated input -2.
Second layer: it features weight matrix in R The upper d x block hosts vector where entries bl, h) equal to 1/H if = I, 0 otherwise. The lower d x block has the same exact structure, but non-zero entries are, instead, set to 1/H.
It is easy to see this construction exactly implements the required averaging, making the ReLU activaction act neutrally.
Now, we ask to what precision does the final output approximate overall target e¹. Note that the only source of approximation is in the previously introduced upo- we are thus only required quantify how it "propagates" to the rest of the following computation. We have, by the triangular inequality:
H
log log(a 1.
b(LM)
H
h=0
1=0
(0:) (0)))
1.4 log(
a
5
7
(0:) (ii))am k(,M) - log(a
2.10
h=0
1 H-1
nc
Hnc
=
H
Hn
h=0
which concludes the proof.
B DATASET DETAILS
B.1 NQ AND CNN
Dataset construction. These datasets are constructed precisely following the implementation deseribed in (Chuang et al. 2024) and provided part supplementary codebase https //github. com/ voi fooidism sm/ wtt..v//noooooooocoeeeaoaeaee Lookkack-bebb....e From this repository we derive both prompts and pre-computed, annotated generations, which we re-use via teacher-foreing to hook out the required computational traces, namely attention and activation matrices. We tested the fidelity of these generated scores in early experiments: recalculated original Lockback Lens features using the generated data and managed to reproduce the original results in (Chuang et al. 2024).
Dataset details iincluding description of the text generation and annotation process, are found in (Chuang al.) 20244 Appendix A) and (Chuang al. 2024 Appendix C.2), to which we refer the interested reader.
Splitting. Chhang et al. 2024) originally split the data randomly and in way that, potentially,
passages from the same response could appear across training and evaluation splits. We argue this is an undesired side-effect and, in effort to cast the HD in more challenging setup, instead split the data at the level whole prompt-response pairs (graphs according to our framework). Specifically, we fix the seed 42 and randomly obtain a prompt-response level split in the proportion 60% 20% / 20% (train / val test).
B.2 MOVIES, WINOBIAS, MATH
Dataset construction. These datasets are constructed following the process described in OOgad et al.] 2025), and by leveraging the authors' code open-sourced at https:: hhtbbb....btt.tg////gg com/ Eechnion cs halppLLLMKnoo (MIT License). The prompts and ground truth labels of all the three considered dataset, in partieular, provided by the authors themselves in the above codebase.
19

---

Published as a conference paper at ICLR 2026
As for hallucination labels, we the annotation process whose code is provided therein. These annotation routines are mostly based on string matching procedures.
Dataset details, including a description of the datasets and how prompts have been derived are provided in (Orgad et al. 2025 Appendix A.3), to which refer the interested reader.
Splitting. We use the same train/test splits provided by Orgad et al. (2025), and additionally carve out a random sample of 20% of training data points, treated our validation set. We perform this sampling by setting random seed to 42.
C EXTENDED EXPERIMENTAL SECTION
C.1 COMPARISON WITH SELF- CHEFCC AND MULTIPLE- MULTIPLE-PROMPTIEE PROMPTING- BASED METHODS
In this section, we compare against baseline methods that rely additional prompting, specifically P(True) Kadavath et al. (202)) and Semantic Entropy (SE) Kuhn et al. (2023). Both approaches operate over multiple LLM generations prompts, which introduces non- negligiiibb computational overhead and may hinder their applicability in real-time settings. Table reports results on the Movies dataset using Mistras a 1 7 18-instruct... For SE, we follow the original evaluation setup (Kuhn)
et al. 2023). employing the DeBERTa entailment model as described in the referenced work.
Table 6: Comparison with methods relying on multiple prompting.
Method
Mis-7B Movies (AUC)
P(True)
62.00
Semantic Entropy
70.06
CHARM (att) (ours)
80.3
CHARM (at++act-24) (ours)
79.7
We observe that in all cases, CHARM variants substantially outperform the competing approaches. To quantify the computational burden of these baselines, we measured the average runtime of SE for producing prediction. This process involves generating 10 additional responses and clustering them by computing mutual entailments with an auxiliary DeBERTa model. On average, SE required 5.9 1.7 seconds per evaluation. We minimised the overhead of auxiliary generations by running them in parallel through batching. Nevertheless, the clustering step alone accounts for about 1.35 seconds of runtime, which not negligible. These findings highlight the advantage of our method, which not only achieves higher accuracy but also operates orders of magnitude faster, with detection runtimes on this dataset in the range of 10 seconds.
C.2 IMPACT OF THE CHOICE OF ACTIVATION LAYER(S)
Here we report results on the Math dataset in an effort to assess the sensitivity of CHARM across different choices of the activation layer(s). We train (and tune) CHARM with features from the layers considered for activation probes, namely 24, 28, 32. We also experiment with considering all these layers jointly, with activations concatenated together across layers. Results in Table 7
CHARM remains robust w.r.t. the chosen activation layer and we observe that concatenating multiple layers together may further improve performance.
C.3 PERFORMANCE BY LENGTH
We additional experiments the CNN dataset better study how model performance, inference run-time and memory footprint vary as a function of the processed sequence lengths.
Runtime and memory consumption by length have their trends reported Figure 3 These results show extremely contained nun-times and memory consumptions even for lengths in the order
20

---

Published as a conference paper at ICLR 2026
Table Performance of CHARM with different choices of activation layers.
Method
Math
AUROC
AUPR
Act-24
77.7
77.5
Act-28
78.1
77.8
Act-32
76.6
77.9
CHARM (att)
76.5 1111
79.7 10.5
CHARM (att+act-224)
80.8 40.7
83.
10.7
CHARM (att+aat--28)
81.2 11.0
83.4 1.3
CHARM (att+actt332)
81.7 #02
83.8 20.3
CHARM (24,-actt(24228332))
82.7 #0.1
84.0 #0.5
Memory Overnead vs sequence length (binned medion)
Runtime lergth (binned
140
Burdine 60
0.0012
Unage (M85
120
100
0.0011
1005
2000
7500
500
1000
1500
2000
25:0
Secuance langth
Secuznce longth
Figure 3: Inference runtime (left) and memory consumption (right) by sequence length on the CNN dataset. Values are grouped in 10 bins; x-ticks report median length per bin; the y-axis reports median measurements per bin, well as the corresponding inter-quartile range.
of thousands of tokens. Overall, runtime and memory consumption scale very favourably in the considered range, underscoring the crucial computational advantage of running neural message- passing on sparsified attention graphs
AUROC vs sequence length
AUPR vs sequence length
0.85
0.35
0.30
0.25
MIPA
0.70
0.85
0.60
0.10
0.55
0.05
0.50
0.00
591.C
1042.5
1531.0
1978.0
2313.0
581.0
1042.5
1532.0
1973.0 par tmni
2533.0
Sequumic length (mMcZiA pae be7)
Suquarse Ne gith tmmedd
Figure 4: Test performance by sequence length on the CNN dataset: AUROC (left), AUPR (right). Values are grouped bins; x-ticks report median length per bin; the y-axis reports the performance within each bin.
Performance by length is reported in Figure both terms of AUROC and AUPR. We observe that CHARM's performance does not degrade with increasing sequence length; in fact, the highest AUROC achieved in the longest-sequenee bin. Overall, these results suggest that CHARM is not particularly sensitive sequence length.
21

---

Published as a conference paper at ICLR 2026
Generalising to longer sequences at test time. We also experimented with training shorter sequences and testing on longer ones, again on the CNN dataset. We produced a new non-uniform split whereby test samples taken as the top 20% longest sequences, the rest considered for training and validation sets. This way, training and model selection is run on sequences only up to 1.2k tokens, while the model tested sequences of length ranging from this value to 2.7k tokens. Results are reported in Table 8 comparing CHARM with other Act-# and Lookback Lens baselines.
Table 8: Performance of CHARM on CNN, split by size (test set contains unseen longer sequences).
Method
CNN (longer seqs.) AUROC
AUPR
Act-24
71.6
16.3
Act-28
70.3
16.0
Act-32
68.4
14.1
Lookback Lens
73.9
18.9
CHARM (att)
74.5 =0.7
21.6 +1.2
We observe that CHARM still outperforms methods comparisons on both metries, and that its performance unseen sequence lengths remains stable, only marginally lower than what obtained on uniform splits (Table D..
C.4 HYPERPARAMETER GRIDS
We employed the same hyperparameter grid search across all datasets considered for CHARM, as summarized in T When incorporating activations into CHARM, we additionally searched separate weight decay parameter, applied only to the encoder of the activations, with candidate values (0.0, 0.05, 0.13.
Table 9: Hyperparameter search space for CHARM.
Hyperparameter
Values
Learning Rate
{0.001, 0.0005}
Learning Rate Sched.
(Reduce On Plateau, Cosine w/ Warmup}
Batch Size
32
Dropout
{0.25, 0.5)
Hidden Dimension
(32, 64, 128]
Number of Layers
{1, 2,3)
Weight Decay
{0.0, 0.001 {yes, no)
BatchNorm
Residual Connections
(yes, no)
C.4.1 BASELINE HYPERPARAMETER SEARCHES
All hyperparameters were selected based on validation performance and, in particular, in order to maximise the AUPR metric. The details provided below.
Probas. We evaluated different readout functions - mean, max, and sum - applied to the next-token probabilities.
Act-# We experimented with the following regularisation parameters for logistic regression:
C. (10-5 10 7 10 5,00.4,,,110 10 1002, 10 1, 10, 100, 10²}. In addition, we probed token positions in: (-3,-2.-1,,,1,,))
22

---

Published as a conference paper at ICLR 2026
LLM-Chk-*, We required to clamp the attention scores from below using 10 6 to avoid numerical errors. For LLM-Chk+++* we experimented with C' E {10-8,10 10 10007,10 10 100 10 10-40040000 10030 1,000 7,10-,,20 2,002100- 10, 100, 10%).
LapEig. We experimented with the following values of k: For datasets where the minimum number of tokens in the test split was less than k. restricted experiments to values of below this threshold. As for logistic regression, we used C' 1. class balancing, and maximum of 2, 000 iterations, consistent with what prescribed in the original paper (Sriramanan et al. 2024).
10, 11, 12, 15, 20).
Neigh-Avg (N) and Neigh-Avg (E) We used mean readout exactly in our model and tuned the logistic regression regularisation parameter over C E (10-8.10 10 1010,000000000
101 10 10 1004. 10 10021.200000 20021001_ 10, 100, 10%).
Lookback Lens. implemented Lookback Lens exactly as described in the original paper, using logistic regression with regularization parameter of C = 1. For Lookback Lens we per- 2,10- 10, 100, 10").
formed a grid search over E {10 10 7,100, 7,10- 10 10 4, 10 2,00,
CHARM (no g.). We use the same exact grid as for CHARM (see Table 99, with the following exceptions: (1) no msg-passing layers; (2) a readout / prediction head that either linear or implemented an MLP.
D IMPLEMENTATION DETAILS AND COMPUTATIONAL RESOURCES
D.1 DETAILED ARCHITECTURA FORMS AND TRAINING PARAMETERS
D.1.1
NEIGHROURHOOF AVERAGING BASELINES
Our baselines Neigh-Ave (N) and Neigh-Ayy (E) calculate features in non-learnable way as described below.
Neigh-Avg (N).
1120
(150))
Σ (E((.))
+ degin (i)
j: (UJ)E
Σ (E(())
I.V. +
1 + degin (1)
J: (IJ)EE
1
(ai,t +
Σ
a3s
1 degin
(i)
J: (iJ)EE
Neigh-Avg (E).
(n
Σ
M 100)
+
+ deg.
j: (iDEE
IV.++ +
Σ
deg.
(i)
J: (UJEE
Σ
(10)
1 deg..
(1)
4: (E,)EE
Outputs h (1) are then fed in input to a logistic regression model, regularised as illustrated Ap-
pendix C.4.1 Before that, they are averaged-pooles in the of response-wiss predictions tasks.
23

---

Published as a conference paper at ICLR 2026
D.1.2 EXPERIMENTAL CHARM FORM
Throughout all our experiments, CHARM implements the following msg- pagsppsingg equation:
h ((+1) up, ([ degin
Σ (iJ)EE
msg. ([1 (E,00 (Pss))]
(11)
Initial node features always include "reflexive attention", i.e., that token pays to itself. When additionally including activations (CHARM (att+act-*)) we employed an additionally encoder to preprocess these. The output of this module is concatenated to the original attention features before message passing takes place. Note that, in all our experiments, for computational reasons and in alignment with the computational flow of Lookback Lens, we remove all connections from prompt to prompt tokens.
D.1.3 OPTIMIZER AND SCHEDULERS
For all datasets and tasks, we use the AdamW optimizer Loshchilov Hutter (2019). We experi- mented with two learning rate schedulers (see Table "Reduce On Plateau" and "Cosine Annealing with Warmup", where warmup spanned 10% of the total training steps. The scheduler yielding the best validation performance was selected.
D.2
CODE IMPLEMENTATION
Thee implementation CHARM was realised by means PyTorch (Paszke et al. 2019) and PyTorch Geometric (Fey & Lenssen, 2019) (available respectively under the BSD and MIT license). We performed hyperparameter tuning using the Weight and Biases framework (Biewald, 2020). For baselines and models running logistic regression, we resorted the implementation exposed by the Sci-Kit Learn library (BSD license). LapEig required also running the PCA dimensionality reduction technique; invoked the python implementation from the same library.
D.3 EXPERIMENTAL RESOURCES AND ARTEFACTS
We ran our all our experiments NVIDIA L40 GPUs. The two employed LLMs were both accessed via Hugging Face python API, in particular:
LLaMa 2- Touvron et al. 2023) (License: LLaMa 2 Community License). Ac-
cessed http:tttttttttgggggggfffc co htt.:mmtaallllll ama- 7al--b--httttte
Mistraal-7b-instzuus (Jiang et al. 2023) (License: Apache-2.0). Accessed at
E VISUALISATIONS
We provide here two sample visualisations of the constructed computational graphs in input to our CHARM architecture. These illustrate test data points in the NQ dataset and are reported in Figures 5 and [] where we show the full computational graph (left) along with zoom-in on the response tokens (right).
These visualisations arrange prompt tokens on the left, and response tokens on the right. Edges are drawn way that their thickness and transparency proportional to the corresponding attention scores, averaged, for illustrative purposes, across layers and heads. Response tokens have their border coloured according to the ground truth - red: the token an hallucinated passage; blue: otherwise. Their interior is filled, instead, with colour that conveys the model prediction "more red": the token is more likely to be in an hallucinated passage; "more blue": more likely to be in a correct, non-hallucinated passage. Output scores for model are matched to this colour map linearly after max normalisation. Note the absence of coompt-to-prompy edges, as they are neglected in our experiments explained at the end of Appendix D.I.2
24

---

Published as a conference paper at ICLR 2026
Figure 5: Visualisation of test sample 1590, along with token wise labels and predictions.
25

---

Published as a conference paper at ICLR 2026
Figure 6: Visualisation of test sample 2354, along with token wise labels and predictions.
26

---

Published as a conference paper at ICLR 2026
F LARGE LANGUAGE MODEL (LLM) USAGE
We employed large language models (LLMs) to support the writing process, specifically for improving clarity in technical explanations, refining grammar and style, and enhancing overall readability. LLMs were also used to a limited extent to aid the process of finding related works. All research contributions, including the design of experiments, data analysis, and conclusions, are entirely our own. The LLMs were used strictly as writing aids to improve presentation quality, not for generating research content or shaping the substance of our work.
27


Reviews:
[{"review_id": "la5WJFgG3i", "reviewer": "Reviewer_9j71", "summary": "This paper introduces CHARM, a framework that recasts LLM hallucination detection as a graph learning problem. It models an LLM's internal computational traces, specifically attention scores and activations, as a single attributed graph where tokens serve as nodes and attention maps define the edges. This formulation elevates the task from relying on isolated heuristics to a principled problem effectively tackled by a Graph Neural Network (GNN).", "questions": "See the Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes a novel approach to hallucination detection by modeling LLM computational traces (attention and activations) as attributed graphs. This formulation provides a unified view of different internal signals and allows for the application of established graph neural network architectures to the problem.\n2. The proposed method is supported by both theoretical analysis and empirical evaluation. The authors show that their framework is expressive enough to subsume existing attention-based heuristics. The experiments, conducted across different benchmarks and detection granularities, demonstrate performance gains over several baselines.\n3. The paper is well-organized and clearly written.", "weaknesses": "1. The choice of LLM layer corresponding to the activation values is not clearly justified. For example, in Tables 1 and 2, some baseline methods report results from multiple layers, while CHARM only uses the layer 24. It would be better to include results from different layers to provide a more comprehensive analysis. Moreover, as mentioned at the end of Section 5.2, CHARM uses the layer 32 only for the Math dataset. Does this imply that performance is highly sensitive to layer selection?\n2. The paper adopts a relatively simple propagation mechanism, whereas in previous studies on GNNs, more sophisticated approaches such as GCN or GAT are widely used. The authors should clarify the rationale for not employing these advanced graph propagation methods.\n3. The paper lacks visualizations. It would be helpful to include concrete examples illustrating the constructed graph, such as visualizing nodes of varying sizes and edges with different thicknesses, or employing heatmaps and other effective visualization methods. Such visualizations could greatly enhance the interpretability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "52sJJ8ehOa", "reviewer": "Reviewer_Ku87", "summary": "In this paper, the authors analyze hallucination detection using LLM-internals by employing message passing over a unified attributed graph, incorporating its activations and attention scores. In detail, the graph is set up by assuming tokens as nodes, edges representing the direction of tokens attending to future tokens in a causal manner, the node attributes represented by on-diagonal attention scores and activations, and edge attributes represented by off-diagonal attention scores. The paper proposes Charm, by using a Graph Neural Network over this graph, with token-level hallucination detection cast as a node classification task, and response level detection cast as a graph classification task.", "questions": "Kindly refer to the questions mentioned in the weaknesses section above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is well-written and introduces the core components of the proposed method in a clear and concise manner. The paper presents a structured and principled framework to detect hallucinations, by using a GNN to learn from an attribution graph that combines attention scores alongside activation states.\n\n2) With the abstracted framework introduced, the authors show that prior techniques such as LookbackLens and LLM-Check can be represented by specific design choices and instantiations of the GNN used over the general attribute graph. \n\n3) Through empirical evaluations on both token-level hallucination detection and response-level detection, Charm is shown to achieve good empirical performance, and achieves significant improvements over existing baseline. Furthermore, the method is shown to require fairly low memory overheads with sparsification, without inheriting a large drop in performance simultaneously.", "weaknesses": "1) The proposed method is considerably more complex to deploy and utilize when compared to baselines: it involves building a potentially large graph for each generation with the number vertices being the same as the maximum token length, and then extracting all attention scores and activations for all layers. This is of course followed by supervised training the GNN on a set of truthful and hallucinated samples, whose distribution is close to the test samples seen at inference time. On the other hand, several baseline methods such as SelfCheck, INSIDE, LLMCheck do not utilize any trainable network components. Furthermore, given that CHARM explicitly trains the GNN, other training based methods such as ITI [1] form a strong baseline of comparison, since the degree of discriminability can be adaptively set for detection.\n\n\n2) The attribution graph is dense prior to sparsification, and quadratic in the sequence length of  N tokens: $L * H * N * (N+1)/2$ attention scores are computed for an LLM of L layers and H attention heads per layer. While Table-4 does indicate that good performance is achieved even at high-sparsity levels, it is unclear how this would scale for longer sequence lengths. Could the authors kindly report the computation run-time and memory overheads over some fixed sequence lengths such as 1K, 10K and 100K? Furthermore, what is the impact on the generalization of the trained GNN when the context size potentially changes by a large manner at test time? \n\n\n3) Evaluation Metrics: The empirical evaluation relies exclusively on AUROC and AUPR. While these metrics give a good overall view of performance, in practice it is crucial to achieve high TPR at low FPR rates. Particularly with hallucination detection, it is quite important to report standard detection metrics such as TPR at 5% or 10% FPR and F1 score, to help assess and compare the efficacy of the proposed method?\n\n\n4) Furthermore, the main experimental evaluations are reported only on LLaMa-2-7B and Mistral-7B, which are slightly dated. The evaluations could be significantly improved by including newer LLMs such as Llama3/4, Gemma and QwQ models. This would also help establish the generality of the method proposed.\n\n\n5) The specific Response level detection datasets used in the paper (Movies, Winobias, Math) are not commonly used for benchmarking hallucination detection. More standard and established datasets such as HaluEval, TruthfulQA, FAVABench or RagTruth could have been utilized instead. \n\n\n[1] Inference-Time Intervention: Eliciting Truthful Answers from a Language Model, Li et al., NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EsSf28TAQ2", "reviewer": "Reviewer_A2Ck", "summary": "The paper describes using a message passing network to use the attention of a transformer to hallucination detection. The method description is straightforward but the implementation and needed input and engineering work is complicated. Experiments across diverse benchmarks (contextual, factual, reasoning) and LLMs (LLaMa-2, Mistral) show that CHARM consistently outperforms existing methods. They evaluation on Natural Qeustions and reasoning datasets.", "questions": "How does the performance (both accuracy and latency) of CHARM degrade as sequence length increases into the tens of thousands of tokens?\n\nThe paper uses activations from a single, pre-selected layer (was it layer 24?). Why choosing a single layer? Could performance be improved by integrating activations from multiple layers ? did you try?\n\nThe conclusion mentions integrating logits as a future direction. How do the you envision this?\n\ndo the you believe the \"signature\" of a hallucination in the attention graph is universal?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "the paper uses internal signal with a well motivated method for attribution. I find this elegat and novel contribution to the best of my knowledge. \n\nThe method via GANs moves from fixed, hand-crafted heuristics  to a learnable, expressive model. This allows CHARM to capture more complex and none obvious patterns in the computation that may signal a hallucination.\n\nvery good evaluation, very much appriciated, looks good. \n\nThe finding that optimal features (attention-only vs. attention+activations) depend on the type of hallucination is a valuable insight.\n\nopen sourcing the code very much appriciated.", "weaknesses": "Like all methods that rely on internals of methods so CHARM too requires \"white-box\" access to the model's internal activations and attention matrices.\n\nWhile GNN inference is reported as fast, it has quite some overhead and implmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PscJ0UednP", "reviewer": "Reviewer_ffYC", "summary": "This paper introduces CHARM (Catching Hallucinated Responses via Message-passing), a unified framework for hallucination detection in LLMs. CHARM represents attention maps and residual activations jointly as attributed graphs, where tokens are nodes and attentional flows define edges. Each node and edge is enriched with features derived from activations and attention scores. Hallucination detection is then formulated as a graph learning problem, addressed via a Graph Neural Network (GNN). The paper shows that CHARM subsumes existing heuristics such as Lookback Lens and LLM-Check and empirical superiority across multiple benchmarks. Experiments cover token-level and response-level hallucination detection across diverse datasets, with comprehensive ablations on graph structure, sparsity thresholds, and zero-shot transfer. CHARM consistently outperforms baselines and shows that incorporating structured, graph-based computation leads to measurable gains over attention- or activation-based heuristics.", "questions": "- How consistent are the results across different activation layers beyond layers 24 and 32? Could the model be extended to dynamically identify and select the most informative layers during training?\n\n- What mitigation strategies could prevent the degradation in hallucination detection accuracy observed when incorporating additional features, such as activations, into the attributed graph?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors presents a novel and unified framework that models LLM computational traces as attributed graphs, effectively integrating both attention scores and activation signals into a single structured representation.\n\n- The proposed approach contributes a new methodological direction by connecting graph neural networks and hallucination detection, opening possibilities for broader applications in explainable AI and reliability of LLMs.\n\n- It provides strong theoretical grounding by formally proving that CHARM subsumes well-known heuristic detectors such as Lookback Lens and LLM-Check, demonstrating clear expressive generality.\n\n- The experimental evaluation is comprehensive, covering both token-level and response-level hallucination detection tasks across five datasets with diverse characteristics.\n\n- CHARM consistently outperforms all baseline methods, including heuristic and activation-based detectors, showing clear empirical advantages in predictive performance.\n\n- The ablation studies and analyses are thorough, investigating the role of graph structure, sparsity thresholds, message passing, and zero-shot transfer in a systematic and transparent manner.\n\n- The paper is well written and organized, with clear mathematical formulations, logical flow, and illustrative figures that make the method easy to understand.\n\n- The results are reported with honesty and nuance, including cases where CHARM’s performance varies in zero-shot transfer scenario, which enhances the paper’s credibility.", "weaknesses": "- The performance drop observed when combining activations with attention maps (Table 1) indicates that CHARM may be sensitive to feature selection and layer choice. A deeper discussion on this sensitivity, along with possible mitigation strategies, would strengthen the paper.\n\n- The current framework still relies on heuristic choices to determine which layer of activations to include in the attributed graphs. Developing a more systematic or data-driven approach to layer selection would make the framework more complete and principled.\n\n- There are a few minor writing issues; for instance, inline citations in the first paragraph of the Related Work section should be placed in parentheses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)