Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs
Authors: Vishal Pramanik
Maisha Maliha
Nathaniel D. Bastian
Sumit Kumar Jha
Venue: ICLR 2026 Poster (2026)
Keywords: Explainable AI
Attribution
LLMs
Primary area: interpretability and explainable AI

Abstract: Attribution methods seek to explain language model predictions by quantifying the contribution of input tokens to generated outputs. However, most existing techniques are designed for encoder-based architectures and rely on linear approximations that fail to capture the causal and semantic complexities of autoregressive generation in decoder-only models. To address these limitations, we propose **Hessian-Enhanced Token Attribution (HETA)**, a novel attribution framework tailored for decoder-only language models. HETA combines three complementary components: a semantic transition vector that captures token-to-token influence across layers, Hessian-based sensitivity scores that model second-order effects, and KL divergence to measure information loss when tokens are masked. This unified design produces context-aware, causally faithful, and semantically grounded attributions. Additionally, we introduce a **curated benchmark dataset** for systematically evaluating attribution quality in generative settings. Empirical evaluations across multiple models and datasets demonstrate that HETA consistently outperforms existing methods in attribution faithfulness and alignment with human annotations, establishing a new standard for interpretability in autoregressive language models.

Published as a conference paper at ICLR 2026
HESSIAN-ENHANCE TOKEN ATTRIBUTION (HETA):
INTERPRETING AUTOREGRESSIVE LLMs
Vishal Pramanik
Maisha Maliha
Department of Computer & Information
School of Computer Science,
Science & Engineering,
University of Oklahoma
University of Florida Gainesville, FL 32611, USA
Norman, OK 73019, USA
maisha mal i hha 10ccuu.... edu
vishalpramani k wuffl.efeeeeeeeeeeeeee edu
Nathaniel D. Bastian
Sumit Kumar Jha Department of Computer & Information
Department of Electrical Engineering and Computer Science,
United States Military Academy
Science & Engincering, University of Florida Gainesville, FL 32611, USA sumit. jha@ufl. edu
West Point, NY 10996, USA
nathaniel
ABSTRACT
Attribution methods seek to explain language model predictions by quantifying the contribution of input tokens to generated outputs. However, most existing techniques are designed for encoder-based architectures and rely on linear approxi- mations that fail to capture the causal and semantic complexities autoregressive generation in decoder-only models. To address these limitations, Hessian-Enhance Token Attribution (HETA), a novel attribution framework tailored for decoder-only language models. HETA combines three complementary components: a semantic transition vector that captures token-to-token influence across layers, Hessian-based sensitivity scores that model second-order effects, and KL divergence to measure information loss when tokens are masked. This unified design produces context-aware causally faithful, and semantically grounded attribu- tions. Additionally, we introduce a curated benchmark dataset for systematically evaluating attribution quality generative settings. Empirical evaluations across multiple models and datasets demonstrate that HETA consistently outperforms existing methods attribution faithfulness and alignment with human annotations, establishing new standard for interpretability in autoregressive language models. Our code and dataset are available here
propose
INTRODUCTION
As machine learning systems achieve increasingly high performance, they are being deployed in high-stakes domains such healthcare, autonomous driving, and finance. However, despite their success, deep neural networks remain difficult to interpret due to their large parameter spaces, layered architectures, and nonlinear computations, carning them the reputation of "black box" models Benítezz et al. (1997). This opacity can erode trust, impede debugging, and raise ethical or regulatory concerns. To address these challenges, the field of Explainable AI (XAI) emerged, with the goal of making model decisions more transparent, interpretable, and trustworthy.
wide range of interpretability methods such as LIME Ribeiro et al. (2016), KemelSHAP Lundberg & Lee (017), Integrated Gradients (undararaaan et al. (2017), Grad-CAM [eelaraju et al. (2017), 7), and Layer-wise Relevance Propagation (LRP) Bach et al. (2015) have been developed under the classical feature attribution paradigm, which aims to quantify the contribution of input features to model's output. Most these methods are based on lincar or first-order derivative approximations
https://github. shal Pramani k/w.Aa..i/.AAA....

---

Published as a conference paper at ICLR 2026
HESSIAN-ENHANGE TOKEN ATTRIBUTION (HETA) FRAMEWORK
SEMANTIC TRANSITION INFLUENCE
2. HESSIAN- SENSITIVITY
3. INFORMATION-THORREA IMPACT
8(x² V+)
H
H-V=
Ply M)
Ply (aaallot))
6P|x
Transformer
Masked
Second-Order Sensitivity Nonlinearities &Inferactions Ligeilhllloodd Embedding
Transformer
Change in Output Distribution Probabilisti Contribution
Transtormer 3
Transformer 4
HETA: UNIFIED ATTRIBUTION SCORE
Attenton for Decoder Distribution
[+
Token Level Attribution
Target Token
Attention- Attention-Welghted Value Flows
Balances Structural, Geometric, Information- Theareei
Causally Connected Paths Only Target Token
Σ (Balanced Integration) Final Token-Level Attribution
Tailored Decoder Models
OUTPUT: INTERPRETABLE ATTRIBUTION SCORES (Causal Tokens Only)
Figure 1: Overview of HETA. The pipeline (a) rolls out attention-value flows that end at the target token form causal gate over input tokens, (b) estimates token-level curvature via scalable Hessian-vector products to capture nonlincar interactions, and (c) measures KL-based information impact under token masking. The final attribution combines causal gating, curvature sensitivity, and information gain to produce target-conditioned, token-level explanations.
and assume local model linearity. However, this assumption often breaks down in the context of autoregressive language models, where token interactions are nonlinear and highly contextual. Despite their practical utility, these techniques frequently produce inconsistent attributions for the same input and model [Hooker et al. (2019), casting doubt their reliability. Although some efforts have introduced axiomatic foundations to formalize attribution Han et al. (0022); Bressan et al. (2024). universally accepted definition of explanation quality remains elusive. Furthermore, these attribution methods have primarily been designed for besooder-bssed architectures, and recent work Zhao & Shan (2024) shows that directly applying them decoder-only language models in generative tasks is non-trivial and often unfaithful. The discrepancy arises from architectural and functional differences, where encoder models leverage bidirectional attention and require single attribution map, while decoder-only models generate outputs autoregressively and demand attribution at each token position. Figure illlstratee the complexity of the attribution task for generative model, highlighting how input words contribute to the generation of a specific output word. Although -gnel-agnostic approaches have been proposed for generative settings Zhao Shan (2024). they typically ignore the dense semantic structure encoded in the internal layers of large language models Chen et al. (202)); Xu et al. (2020), thereby limiting their ability to capture deep token-level influence.
To address the shortcomings of gradient-based attribution methods in autoregressive models, we propose Hessian- Enhannedd Token Attribution (HETA), framework tailored for decoder-only architectures. HETA combines semantic flow tracing. Hessian- sensitivity, and KL-bbasd information loss to yield faithful, token-level attributions. By modeling aitention-weightee value flow and capturing second-order and informational effects, HETA offers principled and robust alternative that respects the causal and contextual structure of generative language models. The key contributions of this work are:
We propose for decoder-only language models that integrates semantie flow for causal directionality, Hessian desseian-based sensitivity for capturing second-ords -rder interactions, and KL-divergence for quantifying foormation--hhorefff impact.
Token Attribution (HETA), a novel attribution method
2

---

Published as a conference paper at ICLR 2026
We construct and release new curated dataset specifically designed for evaluating token- level attributions in autoregressive generation tasks, enabling systematic assessment of attribution faithfulness, robustness, and human alignment.
We conduct extensive experiments across four diverse decoder- only models and a broad suite of strong attribution baselines. Results show that HETA consistently outperforms existing methods in faithfulness, robustness, and semantic alignment, while exhibiting remarkable stability under both decoding hyperparameter changes and syntactic rephrasings.
MOTIVATION
Understanding which input tokens influence generative language model's output central to interpretability. However, existing attribution methods based on attention weights or first-order sensitivity are fundamentally limited in faithfully capturing token-level influence.
Attention-basee methods Abbarr & Zuidema (2020), while widely used, are not reliable indicators of causal influence. Attention weights reflect where the model attends, not what actually affects the output, and can often be perturbed without significantly altering predictions Jaanniiiinnnnnnnnn & Wallace (2019). Moreover, attention mechanisms- eetiiiissseepeeellllllllll when aggregated across layers or heads often fail multi-hop influence paths that propagate through residual connections
to account for indirect and MLP layers| Lu et al. (222I). In decoder-only models, attention is explicitly masked to prevent information flow from future tokens; yet, post-hoc attention aggregations that disregard this constraint may inadvertently assign importance to tokens that are not causally connected to the output. As a result, attention alone should not be treated as a faithful attribution signal.
First-order attribution methods, such as pointwise gradients, Inputx Gradient Shrikumar et al. (001)) and DIG [Snyyal & Ren (2021), measure the local linear sensitivity of the model's output with respect to its inputs. While computationally efficient, these methods can entirely miss meaningful influence in regions where the gradient vanishes but the function remains sensitive finite perturbations. Integrated Gradients (IG) Sundararajan et al. (2017) partially addresses this by accumulating gradients along a path from baseline to the input, but its attributions depend heavily on the choice of baseline and path, and can underrepresent influence near sharp transitions or in highly nonlinear regimes.
To illustrate this limitation, consider the model's prediction function f(z) = log P(rr I<TT), where denotes the concatenated input embeddings (or intermediate hidden states) and T' is the target position. Even if f differentiable, it possible for the partial derivative with respect to some input dimension to be zero while finite perturbation in that direction still causes nontrivial change in (x)) More formally, for some (I,..., n). we may have #f(z) but f(x + ees) A [(x) for some 0, indicating that the gradient fails to detect influence that manifests at higher orders.
de,
This phenomenon is captured by the second-order Taylor expansion:
f(x) f(z)) + (2o)
(z- T(
zo) + [x (o) VV² zo).
for some & on the segment between 2o and x. When the gradient at Zo vanishes, changes in the function are driven entirely by the eurvature encoded in the Hessian, and the contribution scales quadratically with the perturbation norm. For example, in the smooth activation f(x) = log(1 + exp(w w x + +)), the gradient can be nearly zero in the saturated regime (w x + < 0), yet the function value still changes significantly for finite perturbations. In such cases, gradient-based attribution methods underestimate entirely miss the relevant influence (see appendi A for details).
Recent studies like, ContextCite Cohen- Moshnn-Wangg et al. (2024), TDD Feng et al. (0024), and Peering into the Mind of LMs Phukann al. (2024) all aim to attribute outputs in generative language models, but each exhibits notable limitations. ContextCite relies on sparse linear surrogate trained through extensive ablations, which makes it sensitive to redundancy and indirect dependencies, computationally expensive, and limited to sentence-level attribution without next-token conditioning. TDD projects hidden states through the model's output head (logit lens), conflating correlation with causation and producing saliency scores that are highly sensitive to the choice of target-alternatie token pairs and vocabulary dynamics. Peering matches hidden states of generated answer tokens to context tokens using cosine similarity with layer-specific thresholds, a pngresentation-mmateee approach that performs well primarily for verbatim spans but lacks robustness to paraphrasing, reordering, indirect evidence chains.
3

---

Published as a conference paper at ICLR 2026
These limitations, namely. the inability of attention to capture causal importance and the failure of first-order gradients to model non non-linear sensitivity, underscore the need for more robust attribution frameworks. Existing methods remain unstable under decoding hyperparameter changes and syntactic rephrasings (see Sectior5). We propose Hessian- integrates causal semantic flow, second-order sensitivity, and output-aware information gain. Together, these components yield stable, faithful, and interpretable attributions for decoder-only LMs.
Token Attribution (HETA), which
3 BACKGROUND
Understanding token-level influence in transformer models requires going beyond raw attention weights or local gradients. Two complementary strands of research have highlighted important limitations and proposed more robust alternatives. |Kobayashi et al. (2020) demonstrated that attention weights alone are insufficient for faithful interpretation, as they neglect scale of the value vectors being attended to. They proposed norm-based approach that combines attention weights with the magnitude of the transformed value projections, offering a more accurate view of token influence within self-attention This formulation captures not only alignment (via attention) but also semantic strength (via vector norms), leading to more faithful attributions.
Separately, Hessian-basee sensitivity methods provide deeper insight into model behavior by ac- counting for second-orde interactions between inputs and outputs. Specifically, the Hessian of the log-likelihood with respect input embeddings, curvature and reveals how token effects manifest in nonlinear regions of the model's 's decision surface. Unlike first-order methods which can fail in flat regions or under poor bascline selection, second-order methods remain informative even when gradients vanish. Prior studies (Dong et al. (2025), Alvarez Mclis & Jaakkola 20118) Dhamdhere et al. (2018) support the use of Hessian- approaches to uncover latent influences deep architectures.
V log P(ær <<T)), captures local
Together, these techniques underscore the importance of considering both semantic flow and higher- order sensitivity to capture faithful token attributions in transformer models.
4 OUR METHODOLOGY
We propose Hessian- Enhanced Token Attribution (HETA), principled framework that integrates these perspectives into unified influence decomposition. Our central view is that token attribution in autoregressive models should estimate token's directional, larget-conditioned causal contribution to the log-likelihood of the current target token, incorporating both semantic path dependencies and higher-order effects. НETА achieves this through three complementary components: (1) Semantic Transition Influence, which captures how tokens propagate influence through compositional atten- tion-value flows across layers, enforcing causal directionality toward the target. (2) Hessian- Based Sensitivity, which models second-order curvature of the log-likelihood surface with respect to token embeddings, capturing nonlinear and interaction effects. (3) cetermmaiio--hhhreeee Impact. which measures the change in predictive uncertainty when a token is masked, providing a probabilistic interpretation of its contribution. Together, these components form a mathematically grounded attribution score that balances structural, geometric, and cnnemattion-hhoretii perspectives on token influence.
Let . be the input sequence with embeddings X = le,, E RXxx , A decoder-only model So defines the conditional distribution over the targel token:
Po x<T) = Softmax(
(I)
Our goal a nonnegative score Attr(xa,-x -xr) quantifying the contribution of token v, to predicting .r. The score combines (i) semantic transition influence, (ii) Hessian-basee sensitivity, and (iii) KL-based information loss.
Semantic Flow for Causal Token Influence To ensure target-conditioned causality, we trace ditention-weightee value flow that terminates at position T under the decoder's causal mask. For each layer E (1...., L} and head E (1,..., H), let ((.h) E TT be the masked attention matrix, (I.)) R the value vectors, and ((((A) E dddxd the output projection. We compute
4

---

Published as a conference paper at ICLR 2026
larget-conditioned attention rollout ((CA)(i-T)) (e.g.. Abnar & Zuidema (2020) that aggregates only paths ending at T. The semantic transition influence is
L
H
Mrli]
(L,))) (i-T) ||,
(. W
Z =
(ll.h) (j (-TT) || II ((M)
Z
J=1 1.h
Thus Mr E I2 is simplex-normalized E Mr = 1) and assigns mass only to tokens with causal paths to T.
Hessian-Basse Sensitivity Analysis the target log-probablln prebbbbbbilityy with respect to X:
To capture second-ordes effects, we consider the Hessian of
Hr Vx log Po(ET TeT) E ((T()x(F4)
Explicitly forming HT is infeasible for realistic T, d. We therefore estimate block sensitivities via Hessian-vector products (HVPs) with Hutchinson estimators. Let II; select token i's d-dimensional block. With Rademacher vectors Th supported on that block, the sensitivity used in equation is
I. Hr (Mrr)))
where each HVP is computed by Pearlmutter's trick; optionally surrogate for numerical stability. We report m, runtime, and memory in our experiments.
Cewss-Newton/Fisshe
KL Divergence for Information Contribution To quantify token's 's contribution the targer distribution, we compare predictions at T with and without information from For each token I4, we mask it and measure how the output distribution over the target token changes. For a chosen denote the target distributions. The information scheme, let Porig (- xer) and p(i P (er)
contribution is
I(x; -TT DKI. (Parig(: (eTT)
p())
musked (Er)))
Final Attribution Score
We combine the three components into target-conditioned attribution:
Attr(z;--7)) = Mr (3 + " IIxx,+r)),,
(5)
where 3. 2 weight curvature- based sensitivity and information contribution, respectively. The gate [t] restricts attribution to tokens with causal paths to the target and redistributes mass such paths. conjunction with the scalable HVP-based curvature estimator (quuation (3 and the output-aware information term equation 4), equation equation[syyilldss causally grounded, curvature-aware, and robust token-level attribution tailored for decoder-only generative models.
An overview of our method is illustrated in Figure with the full algorithmic procedure provided in Appendix A3
The theoretical properties and error bounds of HETA are discussed in detail in
Appendix A2
EXPERIMENTS AND RESULTS
5.1 EXPERIMENTAL SETUP AND DATASETS
To evaluate the proposed HETA framework, we conduct experiments on both benchmark and curated datasets covering a wide range of reasoning and generation complexity.
We use established benchmarks from Zhao & Shan (2024). The datasets considered are: (1) Long- Range Agreement (LongRA) Vafa et al. (2021). which evaluates a model's ability to maintain coherence across long-distance semantic dependencies by inserting distractor sentences between related word pairs (e.g., "Japan" and "Tokyo"): (2) TellMeWhy Lal et al. (2021). a narrative QA dataset that requires multi- sentencenennee causal reasoning to explain character's motivations; and (3) WikiBio WikiBBioMMnnakl et al. (2023), composed of structured Wikipedia biographies where the task involves generating plausible and factual sentence continuations from short prompts. In addition, introduce
5

---

Published as a conference paper at ICLR 2026
carefully curated dataset of 2,000 mixed-parageen instances to evaluate whether attribution aligns with the truly predictive evidence in context. Each instance concatenates one narrative segment from NARRATIVEQA Kočiský et al. (018)) with the answer-bearing support segment from SCIQ Johannes Welbi 2017). followed by the corresponding SCIQ question; the model then generates the first answer token, and attributions computed with respect to this onset token so the model has access to the full paragraph and question before attribution is measured. For example:
The protagonist returns to the village after the winter storm, reflecting on her father's 's passing. Photosynthesis primarily occurs the leaves of the plant, where chloroplasts capture light. Question: In which part of the plant does photosynthesis mainly take place?
Here, the correct target is leaves, and the meaningful contributing tokens lie in the second (SciQ) segment; the first (NarrativeQA) segment is semantically rich but non-diagnostic for the question. Answer spans and minimal supporting cues in the SciQ segment are automatically annotated by two independent systems (GPT-4o and GPT-5 Thinking), and we take their intersection at the subword level to form high-precision labels; inter-annotator agreement high (F = 0.91, Cohen's 0.89) across the corpus. To quantify alignment, we report the Dependent Sentence Attribution score (described below), which contrasts attribution mass on annotated tokens in the SciQ segment against mass placed the entire NarrativeQA segment after per-instance normalization. This construction provides a compact, interpretable prohe of target-conditionel attribution quality.
HETA is compared against a broad suite of attribution methods: Con- textCiteCohen-Ve Wang al. (2024). Integrated Gradient/Sundaraaaje
Table 1: Attribution alignment the curated dataset
using the (Dependent Sentence Attribution) metric. We eval- uate robustness across both model size and architecture by testing HETA on diverse decoder-only LMs, spanning dif- ferent parameter scales and design choices. Higher scores indicate stronger alignment with
al. (2017), Peering into the Mind
of LMs (PML [Pukkan et al. (2024).
tokens.
TDD-backwardFenng et al. (2024).
HETA outperforms all baselines.
attention vollou(Abnal & Zuidema
(2000) TAMI/Barkan et al. (2024).
Method
GPT-J
I.LMA
Phi-3
Qwen2.5
Progressive InferenecKariyann cet al. (2024), PaaiCheePalik
IG
-0.34
-0.28
-0.41
-0.31
ContexiCite
.0.12
.0.09
.0.18
-0.14
et al. 2025). and ReAgen Zhao &
Pocring (PML)
-0.25
-0.21
-0.30
-0.22
Shan (024). To measure attribution faithfulness,
TDD-backward
-0.31
-0.27
-0.36
-0.29
use Soft-NC and Soft-NS /Zhao & Aletras] (2023),
Attention Rollout
-0.44
-0.39
-0.52
-0.41
FAML
2.10
2.30
2.05
2.20
Progressive Inference
2.65
2.88
2.40
2.73
modified for generative models as in Zhao & Shan (2024), which assess how output distributions shift under input perturbation based on attribution scores.
SEA CoT
2.92
3.15
2.77
2,85
ReAgent
3.60
3.78
3.35
3.50
HETA (Ours)
5.10
4.25
4.65
Controlled Attribution Evaluation via the DSA Metric. To assess attribution alignment on the curated dataset, introduce the Dependent Sentence Attribution (DSA) metric. This metric quantifies the degree to which attribution mass is correctly concentrated on the answer-relevant portion of the input, specifically, the second part of each curated paragraph (SciQ), which contains the evidence required to answer the question.
Formally, let SNarrQa and SseiQ denote the of model-selected important token indices within the first and second part of the curated text, and let ss; and fs, be the normalized attribution scores assigned to token in the second and first part, respectively; the DSA score is defined as DSA EieSsoto 88;
EjeSyavva JSj. with attributions normalized so the total mass over the and the final DSA reported the average over all instances.
paragraph sums
Higher DSA values indicate that the attribution method assigns more mass to the truly predictive evidence (in the second part) and less unrelated context (in the first part), thereby reflecting better aligmment with causal semantics. DSA complements traditional faithfulness metrics by directly evaluating attribution precision under controlled, interpretable input structure.

---

Published as a conference paper at ICLR 2026
The quik brown for jumps ovar au lay des
D
2.47
Soft NCC
2,23
Soft-
1.45
1:44
3.78
The quce trow fix |-mes ower the asy Rec
The quik brown fax James over the AD) ces
The quix trinn fox junps swar mh sry ong
rull HeTa
transtion Only
Hension Cody
No Barnition Casing
Unitom Trenstionn
Onlyy
Full NETA Transtion Only
Herelan Orly 12 only No Pransttnn Gating uniterm Tranitionn
Full HETA
transtion Only
Hasalon Sarty
No Paraston Sating Unilerm Kanssiio
Only
The quik brow fox Jumps owar the any ooc
The qask trow lox juntzs over Ihfe wdy D0s
qu brown (iiti) ooor lay deg
The
bay ded
bay dnd
(8)
(d)
Figure 2: (a)-(c) Analysis of HETA components. Each bar plot shows the effect of ablating key com- ponents of HETA. The full HETA model achieves the highest attribution faithfulness and alignment across all metries, while removing individual components consistently degrades performance.(d) Input importance distributions for generative task using our proposed HETA method.
SR t 10
0.76
074
061
D.22
0.22
0.040
0.033
00
8
1020
10
19
HETA
ContectCia
BoAgent
Propresoive tnderence
Integatee Gradlent
1869
HETA
Contentin
Rekgent
Progeesiva Inferance
Interaated Gradient
HETA
Contert
BeAgent
Progrescive Inferance
Integrated Gradlent
SAMI
00:00
3043
361
199
36
89
8
B81
10
[9)
2013
361
190
36
(d)
Figure 3: (a)-(c) Analysis of robustness of HETA vs baseline methods (Left) Sensitivity under Gaussian perturbations (lower is better), where HETA maintains the lowest variance across input noise. (Center) Active/Passive robustness (higher is better), reflecting attribution consistency across syntactic rephrasings. (Right) Alignment Fl score against annotated tokens (higher better). HETA outperforms all baselines, validating the complementary role of transition flow, curvature, and information gdin.(d) Input importance distributions for a generative task using our proposed HETA method using Qwen 2.5 3B.
We evaluate attribution quality using both perturbation-basel faithfulness metrics (Soft-NC, Soft-NS) and alignment- analysis on a curated dataset (DSA). Experiments are conducted across four transformer models: Qwen2.5-3B (Alibaba Owen rresults are shown in Appendix A3 due to space constraint), GPT-J-6B (EleutherAll Wang Komatsuzaki (2021): F Phi-3-Medium 4Krr-4-K-IIneete
(14B, Microsoft): + and Llama-3.1-70B (Meta) This selection enables analysis across varying model capacities, architectures, and parameter scales. For the Qwen model the results are shown in Appendix A3 due to space constraints.
Hyperparameters and compute. All experiments run single NVIDIA A100 (80 GB). Unless noted, we set the HETA weights in the final score to 3 0.5 (Hessian sensitivity) and y = 0.5 (KL information) and evaluate sequences of length 512 with batch size 16; long-context runs use length 2,048 with windowing and batch size 8. For efficiency, our default is a low-rank Hessian with windowing (HETA-LR+WIN windows with 50% overlap. This variant closely matches full HETA while substantially reducing cost (see Appendix Tables D. For LLaMA-3.1-70B, we further cut compute by computing second-order terms only in the last six layers, which preserves attribution quality in practice. For smaller models (e.g., (e.g.. GPT-J, OPT), we compute second-order terms across all layers unless otherwise specified. In ablations, we also report HETA-LR (rank = 64) without windowing and HETA-LS (6-layer sampling) without low-rank, keeping all other hyperparameters fixed for a fair comparison.
a rank-64 blockwise low-rank estimator applied within 512-token
5.2 RESULTS
According to Table across all model-task combinations, HETA achieves the highest Soft-NC and Soft-NS scores, demonstrating superior attribution robustness under input perturbations. For instance,
weakkiinnnttui ruct
7

---

Published as a conference paper at ICLR 2026
Table 2: Sensitivity of attribution metrics to decoding hyperparameters (max relative change A %;; lower is better). HETA varies < 1% across all models/metrics; each baseline fluctuates > 2%. Grid: temperature E {0.2, 0.5, 0.9}. top-p E {0.8, 0.9, 0.95). top-k E {20, 50, 100), repetition penalty {1.0, 1.2); 3 seeds.
Model
Metrie
ContextCite
TDD-bw AttnRoll
ProgInf
SEA-CoT ReAgent
Sont NC
0.6
2.6
2.7
2.4
3.1
2.3
2.5
2,4
2.5
2.3
Ilama-3.1 70B
SoA-NS
0.8
3.1
3,0
2.8
3.4
2.7
2.9
288
3.0
2.6
DSA
0.6
2,4
2,3
2.2
2.8
2.1
2.3
2,3
2,4
2.2
on GPT-J 6B, HETA attains a Soft-NC of 10.3 on LONGRA and 9.2 on TELLMEWHY- the next best method, ReAgent, by over 2x. Similar trends hold across Phi-3 and Llama-3. 1, confirming HETA's effectiveness across model scales. While ReAgent consistently ranks second, recent methods such as SEA-CoT, and Progressive Inference show moderate improvements over traditional techniques. In contrast, Integrated Gradients and attention-based variants often yield low or negative Soft-NS values, indicating instability and low attribution faithfulness.
To complement the above, we assess attribution alignment using the DSA metric on a curated dataset (Table []. Again, HETA outperforms all baselines by substantial margin, achieving DSA scores 4.2 across all models. ReAgent remains the strongest non-HETT method, followed by SEA-CoT. In contrast, gradient- and attention-base methods yield negative DSA values, highlighting their inability to isolate causal tokens in the presence of distractors. Results are averaged across all datasets with GPT-J-6B; higher is better. Mean performance is reported over three independent runs with standard deviation < 0.2. These results collectively indicate that HETA provides both faithful and semantically aligned attributions, setting a new state-of-the-art across both benchmark and controlled evaluation settings (please see Appendix A33 for further details). We illustrate HETA's token-level attributions with two qualitative examples in Figures 2 d) and d))
5.3 ROBUSTNESS OF HETA
To further demonstrate the robustness of our methodology, we performed a stress test using three complementary attribution metries: Sensitivity, Active/Passive Robustness, and F1 (Alignment). We use the Phi-3 medium model and the Thy dataset for stress test and ablation studies unless otherwise mentioned. These were evaluated across the six ablated configurations: (1) Sensitivity quantifies attribution stability under small perturbations. For each token embedding X,, we add Gaussian noise e (O, 8"1), compute attribution scores over multiple perturbations, and report the average per-token standard deviation: Sensitivity = 1 ET= T₁ where o, denotes the standard devi- ation of the attribution score for token i, and T is the sequence length.(2) Active/Passive Robustness measures syntactic invariance. Given an original sentence and its active/passive rephrasing, we align corresponding tokens and compute the Spearman rank correlation between their attribution rankings: Robustness = p(Attr(z; + 2T), Autr(az + x'y) (3) Fl (A (Alignment) evaluates agreement between model attributions and annotations made by GPT-4o and GPT5 in our curated dataset. Let Amodel denote the top-attributed tokens and Agnno. the gold-annotated set. The F1 score is computed as
Fl
Amssed Aans.
Anscal Auso
Figure (a-c) shows that the HETA framework consistently yields the lowest sensitivity and the highest robustness and F1 scores compared to the baseline methods, indicating stable, syntax-invariant, and logically-alignne attributions, emphasizing the complementary roles of semantic flow, curvature information, and information gain in delivering reliable token-level attribution.
5.4 ROBUSTNESS TO DECODING HYPERPARAMETER
We evaluate the sensitivity of attribution quality to common decoding hyperparameters. For each method and model, we sweep fixed grid, temperature (0.2, 0.5, 0.9). top-p {0.8, 0.9, 0.95), top-k {20, 50, 100), and repetition penalty (1.0, 1.2), across three random seeds. For each metric, we report the maximum relative change A% across the grid (lower is better).
Table shows that HETA's 's attribution metrics remain effectively invariant to decoding settings, with worst-case A% 1 for Soft-NC, Soft-NS, and DSA (see Appendix for more details). In contrast,
8

---

Published as a conference paper at ICLR 2026
Attribution Method
LongRA
TellMeWhy
WikiBio
Soft-NCI
Soft-NS!
Soft-NCT
Soft-NS
Soft-NCI
Soft-NS
GPT-J 6B
ContextCite
1.42
0.03
1.46
-0.22
0.49
-0.08
Integrated Gradients
1.87
0,45
1.54
0.04
1.38
0.77
Peering (PML)
2.05
0.50
1.68
0.06
1.50
0.83
IDD-backward
1.10
-0.12
1.89
-0.03
0.11
0.51
Attention Rollout
0.41
-0.01
0.25
-0.09
1.91
0.46
fAML
0.21
-0.10
0.05
-0.09
0.21
-0.02
Progressive Inference
1.35
0.28
1.12
0.25
0.99
0.22
SEA-CoT
1.54
0.32
1.30
0.31
1.10
0.35
ReAgent
1.68
0.37
1,45
0.36
1.22
0.39
HETA (Ours)
10.3
2.31
9.2
2.04
3.80
2.20
Phi-3-Medium-14B
ContextCite
1.50
0.04
1,45
-0.20
0.52
-0.06
Integrated Gradients
1.95
0.44
1.60
0.06
1.35
0.70
Peering (PML)
2.15
0.49
1.75
0.08
1.48
0.76
TDD-backward
1.05
-0.10
1.82
-0.02
0.10
0.50
Attention Rollout
0.39
-0.02
0.30
-0.08
1.85
0.43
fAML
0.23
-0.09
0.08
-0.10
0.20
-0.04
Progressive Inference
1.30
0.25
1.18
0.26
1.00
0.21
SEA-CoT
1.50
0.31
1.32
0.33
1.15
0.34
ReAgent
1.66
0.38
1,47
0.39
1.25
0.40
HETA (Ours)
10.8
2.35
9.5
2.20
4.20
2,30
LLaMA-3.1 70B
ContextCite
1.17
0.58
1.20
0.56
0.85
0.57
Integrated Gradients
0.13
0.13
0.13
0.10
0.13
1.15
Peering (PMI.)
0.15
0.15
0.15
0.12
0.15
1.20
IDD-backward
-0.02
-0.11
0.01
-0.10
-0.02
0.59
Attention Rollout
-1.48
0.01
-1.48
0.01
-1.48
0.61
fAML
0.46
-0.21
0.46
-0.21
0.46
-0,07
Progressive Inference
1,20
0.24
1.00
0.22
0.95
0.26
SEA-CoT
1.35
0.30
1.15
0.28
1.05
0.31
ReAgent
1.55
0.36
1.28
0.34
1.15
0.38
HETA (Ours)
9.9
2.60
8.6
2.25
3.70
2.10
Table 3: Attribution faithfulness on benchmark datasets (LongRA, TellMeWhy, WikiBio) using GPT-J 6B, Phi-3-Medium- 14B, and LLaMA-3.1 70B. Evaluated with Soft-NC and Soft-NS; higher is better. Mean of 3 runs; std +0.06.
all baselines exhibit substantially larger variability, typically 2-5%. HETA's stability arises from three design elements: target-conditioned causal gate that confines credit to paths terminating at the current prediction, curvature- sensitivity term that smooths local logit perturbations, and an informationn-therrel component that scores distributional shifts rather than single sampled outcomes. These jointly decouple attribution from stochastic decoding heuristics (temperature, top-p, top-k), whereas ablation-, gradient-, and similarity-basee baselines depend more directly on sampled logits or linear approximations and thus vary markedly with hyperparameter changes.
ABLATION STUDIES
To assess the contribution of each component in HETA, conduct a comprehensive ablation study this section. Due to space constraints. a detailed ablation study is provided the Appendix A4 Experiments performed using the GPT-J 6B model three benchmark datasets. LongRA TeylMeWhy, and WikiBio, along with the curated attribution dataset introduced in Section 5 We compare six configurations: (1) the full HETA model (Transition Hessian KI.), (2) Transition

---

Published as a conference paper at ICLR 2026
Only, (3) Hessian Only, (4) KL Only, (5) No Transition Gating (Hessian KL without semantic weighting), and (6) Uniform Transition (equal token weighting instead of the learned semantic transition vector My). Performance is evaluated using the same metrics as main experiments: Soft-NC and Soft-NS for attribution sensitivity on benchmark datasets, and DSA for aligmment with human-annotated tokens on the curated dataset. We set the aggregation hyperparameters to 3 ==00 0.5 and 0.5, and compute KL divergence using masked-token perturbation. All reported results are averaged across 1000 randomly sampled instances per dataset. Results in Figure a-)- demonstrate that each component contributes meaningfully to HETA's performance. Removing the semantic transition vector (Mrr) or replacing with uniform weighting leads to significant drops in all metries, confirming the importance of modeling directional semantic influence across layers. Similarly, Hessian-bar based sensitivity and KL-based information measures provide complementary improvements by capturing curvature-sensitive effects and token-level information contributions
7 RELATED WORKS
Global explainability methods aim to extract broader patterns from L.Mss. Probing techniques have been instrumental identifying syntactic and semantic representations encoded in LLMs Hewitt & Manning (2019); Peng et al. (2022). Studies by Geva et al. (2022) and andKooayask et al. (023) show that forward layers and attention heads capture complex linguistic knowledge. Mechanistic interpretability, as explored by Wang et al. (2022), seeks to reverse- ceverse-engineer neural networks into comprehensible circuits, facilitating deeper understanding of tasks like object identification. Model editing techniques have also emerged
promising area for explainability. Hypernetwork-based editing Mitchell et al. (2022) and causal tracing Meng et al. (2022) enable targeted modifications in model behavior without extensive retraining, allowing models adapt to specific inputs while maintaining overall performance Yao et al. (2023).
Local feature attribution methods such as LIME Ribeiro al. (2016), KemelSHAP Lundberg & Lee (017), Integrated Gradients (Suddaaraaaan et al. (2017). Grad-CAM [Selvraauu et al. (2017). and LRPh al. (013) quantify per-feature contributions to a model's output. However, most rely on first-order or linear approximations that often produce inconsistent attributions Hooker et al. (2019) and break down the nonlinear, contextual regimes typical of autoregressive generation. Attention-based interpretability Abnar & Zuidemal (2020) is similarly limited: attention weights reflect where a model looks rather than what causally influences its output, and can be perturbed without significantly altering predictions Jain & Wallace (2019).
Several recent methods target attribution in generative LLMs specifically. ContextCite Cohen- Wang et al. 2024) trains sparse linear surrogate via ablations but is limited to sentence-leve attribution and sensitive to redundancy. TDD Feng et al. (2024) projects hidden states through the output head (logit lens), conflating correlation with causation. Peering into the Mind of LMs Phukan al.) (2024) matches hidden-state representations via cosine similarity, performing well on verbatim spans but lacking robustness to paraphrasing or indirect evidence. ReAGent Zhao & Shan (2024) offers model-agnostic approach for generative settings but ignores dense semantic structure in internal layers Chen al. (2024). limiting its ability capture deep token-level influence. These limitations motivate the need for attribution frameworks that combine causal structure, higher-order sensitivity. and cettoormaaiionthhoeettte grounding.
CONCLUSION AND LIMITATIONS
We introduced HETA, a unified framework that improves attribution faithfulness and robustness over strong baselines by combining causal semantic gating, Hessian-based curvature, and KL-divergence- based information gain into a principled token-level attribution mechanism for decoder-only language models. However, it incurs higher runtime, greater memory usage, and reduced efficiency on long texts. These trade-offs highlight the need for optimization, and future work will explore low-rank approximations and layer sampling for better scalability (see Appendix A5 details).
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was supported in part by the National Science Foundation under Grant No. 2404036, by University of Florida startup funds, and by the Defense Advanced Research Projects Agency (DARPA) under Contracts No. HR001 12490420 and No. HR001 12420004. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsoring agencies.
ETHICS STATEMENT
We affirm adherence to the ICLR Code of Ethics and have designed this work to minimize risks related privacy, safety, and misuse. Our experiments use publicly available datasets (NARRATIVEQ2 and SCIQ) under their respective licenses; personally identifiable information sensitive user data are collected, and no human subjects research requiring IRB oversight was conducted. The curated evaluation set is constructed from these sources and automatically annotated by two independent LMs (GPT-4o and (eP----Thnn with intersection-basee labels to reduce over-selection; we report inter-annotato agreement and provide full documentation of preprocessing, tokenization, and filtering to support reproducibility. While attribution methods can be misused for model extraction or targeted prompt manipulation, mitigate these risks by reporting aggregate metrics, withholding raw internal activations, and releasing code with rate limits and intended-use guidelines. We discuss known limitations- e.g., approximation attention-van valuue rollout as a causal gate-ang provide diagnostics and ablations to avoid overstating causal claims, All compute was performed on institutional infrastructure (A100 80GB class GPUs); we report hardware footprints and favor efficient approximations to reduce energy use. We disclose that the authors have no conflicts of interest or external sponsorship that bias this study. Upon publication, we plan to release code and scripts under a permissive license with documentation on safe and responsible use.
under oow-rank/windowed Hessians and the of
REPRODUCIBILITY STATEMENT
We take reproducibility seriously and provide all necessary artifacts to re-create our results. The paper specifies the full method in Sections 5 Algorithmm] theoretical assumptions, bounds, and proofs (including wwwrrank/windowed eгror guarantees) are Appendix A2 Datasets and preprocessing are documented in Section A3.I (NarrativeQAaoe SciiQ construction, prompts for automatic annotation, fixed answer-support intersection labels with agreement scores) together with release scripts that regenerate the curated set from the original sources.
Supplementary materials. We include the full HETA algorithm listing (Algorithm I and submit our complete, anonymized codebase as supplementary material, containing: (i) reference implementations of HETA and its L.R+WIN approximation (deterministic seeds, PyTorch/t rans formers versions, environment file), (ii) end-to-end evaluation pipelines for all reported models (GPT-J 6B, LLaMA- 3.1 70B, Phi-3 14B, Qwen2.5 3B), (iii) scripts to reproduce every table/figure and compute all metrics, and (iv) prompts/configs used for automatic token-level supervision. The repository includes RUN md with single-commant entry points and fixed random seeds to match table numbers within reported standard deviations.

---

Published as a conference paper at ICLR 2026
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint arXiv:2005. 00928, 2020.
David Alvarez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv:1806.088 08049, 2018.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one. 1017):00130140, 2015.
Oren Barkan, Yonatan Toib, Yehonatan Elisha, Jonathan Weill, and Noam Koenigstein. Llm ex- plainability via attributive masking learning. In Findings of the Association for Compulational Linguistics: EMNLP 2024, pp. 9522-9533 9037,5377222 2024.
José Manuel Benítez, Juan Luis Castro, and Ignacio Requena. Are artificial neural networks black boxes? IEEE Transactions on neural networks, 865):1156-116 1997.
Marco Bressan, Nicolò Besa-Bianchi, Emmanuel Esposito, Yishay Mansour, Shay Moran, and Maximilian Thiessen. A theory of interpretable approximations. In The Thirty Seventh Annual Conference on Learning Theory, pp. 648-668. PMLR, 2024.
Lei Chen, Joan Bruna, and Alberto Bietti. Distributional associations vs in-context reasoning: A study of feed-forward and attention layers. arXiv preprint arXiv:2406.03068, 2024.
Benjamin Cohen-Wang,
Wang, Harshay Shah, Kristian Georgiev, and Alcksander Madry. Contextcite: Attributing model generation to context, Advances in Neural Information Processing Systems, 37: 97769797777, 2024.
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch. Stefan Heimersheim, and Adrià Garrgaa- Alonso. Towards automated cireuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems, volume 36, 2023.
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is a neuron?, 2018. URL 12233
Zhaorui Dong, Yushun Zhang, Zhi- Zhi-Quan Luo, Jianfeng Yao, and Ruoyu Sun. Towards quantifying the hessian structure of neural networks. arXiv preprint arXiv:2505.02809, 2025.
Zijian Feng, Hanzhang Zhou, Zixiao Zhu, Junlang Qian, and Kezhi Mao. Unveiling and manipulating prompt influence in large language models. arXiv preprint arXiv:2405.11891, 2024.
Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forwer layers build predictions by promoting concepts in the vocabulary space. arXiv preprint Ttt::2203.146880 2022.
Tessa Han, Suraj Srinivas, and Himabindu Lakkaraju. Which explanation should choose? function approximation perspective to characterizing post hoc explanations. Advances in neural information processing systems, 35:5256-5268, 2022.
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers),
pp. 4129 4138, 2019.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019.
Sarthak Jain and Byron C Wallace. Attention is not explanation. arXiv preprint arXiv: 100. 2019.
Matt Gardner Johannes Welbl, Nelson F, Liu. Crowdsourcing multiple choice science questions. 2017.
12

---

Published as a conference paper at ICLR 2026
Sanjay Kariyappa, Freddy Lécué, Saumitra Mishra, Christopher Pond, Daniele Magazzeni, and Manuela Veloso. Progressive inference: Explaining decoder-only sequence classification models using intermediate predictions. arXiv preprint arXiv:2406. 022:::2200000255, 02625, 2024.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only weight: Analyzing transformers with vector norms. arXiv preprint arXiv:2004.10102 2020.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Analyzing ferward blocks in transformers through the lens of attention map. arXiv preprint arXiv:2302.00456, 2023.
Tomášs Kočiský. Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. Tellme- why: A dataset for answering why-questions in narratives. arXiv preprint arXiv:2106006132 2021.
Xuhong Li, Jiamin Chen, Yekun Chai, and Haoyi Xiong. Gilot: Interpreting generative language models via optimal transport. In Forty-firsi International Conference on Machine Leurning, 2024.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Kaiji Lu, Zifan Wang, Piotr Mardziel, and Anupam Datta. Influence patterns for explaining in- formation flow in bert. Advances in Neural Information Processing Systems, 84:4461-4474, 2021.
Scott M Lundberg and Su-In Lee. unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black- box hallucination detection for generative large language models. arXiv preprint arXiv:2303.088 088966 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 15:7799-173 2022.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory- based model editing at scale. In International Conference on Machine Learning, pp. 15817-15831. PMLR, 2022.
Avash Palikhe, Zhenyu Yu, Zichong Wang, and Wenbin Zhang. Towards transparent ai: A survey on explainable large language models. arXiv preprint 27X122250..218/2, 2025.
Hao Peng. Xiaozhi Wang, Shengding Hu, Hailong Jin, Lei Hou, Juanzi Li, Zhiyuan Liu, and Qun Liu. Copen: Probing conceptual knowledge in pre-trained language models. arXiv preprint arXiv:2211.04079, 2022.
Anirudh Phukan, Shwetha Somasundaram, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. Peering into the mind of language models: An approach for attribution in contextual question answering. In Len-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACI. 2024, pp. 11481-11 1 1495, Bangkok, Thailand, August 2024. A Asoociaiinn for Computational Linguistics. doi: 10. 10.18653/v1/0244 URL
http:///...iiiinaaa....... findings acl 682/|
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1 1135-11444 1144, 2016.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Rober Müller. Evaluating the visualization of what a deep neural network has learned. In IEEE Transac- fions Neurol Networks and Learning Systems, volume 28, pp. 2660-2673, 2017.
13

---

Published as a conference paper at ICLR 2026
Soumya Sanyal and Xiang Ren. Discretized integrated gradients for explaining language models. arXiv preprint arXiv:2108. 13654, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 61 618-626 2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference machine learning, pp. 3145- 3153. PMIR, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pp- 3319-3328, PMLR, 2017.
Keyon Vafa, Yuntian Deng, David M Blei, and Alexander M Rush. Rationales for sequential predictions. arXiv preprint arXiv:2109.06387, 2021.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https::/githhb.
wtw.fllllleeeeaatamaaaeaeeee trnssforrrrsnsoorer-rrr.. jax May 2021.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. arXiv preprint arXiv:2002.1068,, 2020.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang, Editing large language models: Problems, methods, and opportunities. arXiv preprint arXiv:2305. 13172, 2023.
Zhixue Zhao and Nikolaos Aletras. Incorporating attribution importance for improving faithfulness metrics. arXiv preprint arXiv:2305. 10496, 2023.
Zhixue Zhao and Boxuan Shan. Reagent: A model-agnostic feature attribution method for generative language models. arXiv preprint arXiv:2402.00794 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et Judging -as--aa-e with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, 2023.
14

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
Introduction
2 Motivation
3 Background
4
Our Methodology
Experiments and Results
Experimental Setup and Datasets
Resulis
Robustness of HETA
5,4
Robustness to Decoding Hyperparameters
8
6
Ablation Studies
7 Related Works
10
8
Conclusion and Limitations
10
Appendix
15
A1 Why Gradients and Integrated Gradients Can Both Fail in Flat Regions
17
AI.I Gradient Failure ReI.U Flat Regions
17
A1.2 Integrated Gradients Failure Along Flat (or Near-Flat) Paths
17
Al3 Broader Implications
18
A2 heoretical Foundations and Properties of HETA
18
A2. Preliminaries
................................
18
A2.2 Faithfulness Limitations Gradient- Only and KL- Only Methods
................................
20
A2.3
HETA as an Optimal Multi View Attribution
A2.4
Stability and Causality Guarantees
23
A2.5
Interpretation
23
A2.6
Faith/ulness Reconstruction Eroor
23
A2.7
Second-Order Decomposition of Ag(R)
................................
23
A2.8 Causal and Information- Teciiion-nThhheeee Theoretic Constraints
24
22,9 HETA as Solution
24
A2. (Emoo Bounds for Low-Rank, Windowed HETA
24
A2,1 Interpretation
26
A3 Experiments and Results
26
A3.1 Curated Attribution Dataset: Narrative(A + SciQ and the DSA Metric
26
A3.2 Examples
27
15

---

Published as a conference paper at ICLR 2026
A3.3 Attribution Faithfulness Qwen2.5 3B
27
A4 Ablation Studies
27
Ablation Study of HETA: Component wise Contribution
................................
27
A4.2 Weighting Analysis of 3 and y in Final Attribution
................................
28
A4.3 Robustness of HETA
29
A4.4
Computational Fcasibility: Approximating the Hessian
30
A4.5
Scalability: Long-Context Attribution
30
A4.6 Robustness Decoding Hyperparameters
30
A4.7 Compute-Quality Ablation: Low-Rank and Windowed HE:A
31
A4.8 Causal Validity: Target- Conditioned Gate vs. Attention Only Rollout
................................
A4.9 Ablation: Causal Evaluations and Downstream Robustness
A4. ((eey Takeaways
32
A5 Limitation: Computational Overhead and Scalability (All Models)
32
A6 Extended Evaluation and Discussion
34
A6.1 Causal Gate: Scope and Validation
35
A6.2 L.se- Annotation Quality
35
A6.3 Extended Faithfulness Evaluation with Classical Perturbation Metrics
................................
36
A6.4 Generalization Multi Token Targets
37
A6.5 Component Contribution: When Does Each Term Help?
38
16

---

and the Published as a conference paper at ICLR 2026
A1
WHY GRADIENTS AND INTEGRATED GRADIENTS CAN BOTH FAIL IN FLAT REGIONS
In this section, expand the toy example from Section 3.1 to illustrate, in detail and without ambiguity, how both standard gradient-basse attribution and Integrated Gradients (IG) can assign zero (or negligible) importance output in a nearby region. The phenomenon arises in networks with piecewise-linea, activations such ReLU, which create locally flat regions where pointwise gradients vanish; a closely related "near-flat" effect occurs for smooth, saturating nonlincarities (e.g., GELU/softllss) where gradients along a chosen baseline-tor to-input path can be uniformly tiny.
input feature when that feature clearly influences the model's
A1.1 GRADIENT FAILURE RELU FLAT REGIONS
Consider
f(x) = ReLU(W Ta b),
TR.
W
b 22
22
Take the input
[0]
Then
w xn b 0
f(zo) = (-
so wo lies flat (inactive) region of the ReLU. The derivative of ReLU
0,
d ReLU(z) dz
2 0,
undefined (often set to 0),
2 = 0,
and by the chain rule.
0
ReLU(W -oo bb) va(wT xx) ff(x)) dz
0
0
===0
Implication. Any pointwise gradient-basee attribution zu is identically zero, though small displacement can cross the hinge and change the output. For instance, with
2.1
w =
= f(r)= = 0.1 00
Thus, zero gradient at zp does not imply the feature is globally unimportant; it reflects the local flatness of the activation that point.
A1.2 INTEGRATED GRADIENTS FAILURE ALONG FLAT (OR NEAR-FLAT) PATHS
Integrated Gradients (IG) aims to mitigate pointwise gradient pathologies by integrating gradients along a path from a baseline x' the input æ:
0f(z' +a(z x²)) da.
IG, (x; (z, (z.-2%)
dz;
However, IG still depends the gradients along the chosen path. If that path remains entirely within a flat region (or within a region where the pre-activation stays negative), every integrand vanishes and the IG attribution becomes zero.
--1
Exact flat-path example (ReLU). Use the same f and choose the baseline x'
[0]
input 20
The straight-line path is
0
[1+
x(a) = x' + +a - -x))=
a E (0, 1).
-1+a
17

---

Published as a conference paper at ICLR 2026
Along this path,
w² x(a) + (-1+aa) 2 -4+ 20 <0 for all [0, 1],
so (x((a) = 0 and v[()())) = everywhere on the path. Therefore,
1G =.
Yet, as shown above, moving slightly away from zo (e.g., to = ( increases the output, indicating that the inputs do influence f beyond the flat region.
If we chose instead endpoint whose straight path from x' crosses the hinge (i.e., for some a", w wTTTr "x(aa)) + 0), then gradients attribution. the To case above, IG vanishes because the chosen path lies entirely in the inactive region. Thus, IG can fail at inputs that lie inside flat regions for certain baselines/paths, even though the function responds immediately outside that region.
Baseline/path dependence (why IG can succeed or fail). IG is baseline-dependeee
a nontrivial subinterval would be nonzero and IG would assign positive
Near-flat smooth activations (GELU/softplus). For smooth saturating nonlinearities (e.g.. softplus(z) log(1 + e²) GELU), strict flatness is replaced by very small gradients when 2 < O. If the baseline-to-inge input path remains in a "near-flat" zone where 2(a) = w z(a) + b xb<0 for all E [0, 1]. then
{+(())) R Va,
and IG be arbitrarily small, under-attributinn features despite non-negligible finite changes for steps that push z toward the transition region. Hence, the qualitative failure mode persists in smooth networks: the path integral can be dominated by region of tiny gradients, producing near-zero IG.
A1.3 BROADER IMPLICATIONS
These effects extend beyond toy settings. transformer LMs, (i) causal masking and attention patterns can route influence away from certain tokens along specific layers/paths; (ii) residual/MLP mixing plus saturating activations can create local neighborhoods where pointwise gradients (and path- integrated gradients for common baselines) are nearly zero; yet (iii) modest, structured perturbations those tokens can meaningfully alter the output distribution at downstream position. Consequently, both raw gradients and IG may under-attribute token importance in regions of flat or near-flat sensitivity.
Takeaway and motivation. Pointwise gradients fail flat inputs; IG can also fail when its baseline- to-input path lies in (near-)flat regions skirts the true transition surface that mediates the output change. These limitations motivate complementary signals: curvature-aware sensitivity (Hessian- based), target-conditioned semantic flow that respects causal routing, and information-thhorre change (e.g., KL under masking). Together, they capture influence that first-order and path-integrated gradients can miss.
A2 THEORETICAL FOUNDATIONS AND PROPERTIES OF HETA
In this section, we provide mathematically rigorous foundation for the Hessian-Enhancee Token Attribution (HETA) framework. We formalize attribution as a decomposition problem for the target log-likelihood establish faithfulness error bounds, and show how combining semantic flow, Hessian curvature, and KL-based information contributions can improve faithfulness and robustness relative to single-view attribution methods.
A2.1 PRELIMINARIES
Consider a decoder-only language model fo with parameters 0, input tokens Fir, embeddings X E IR xxd and the target conditional distribution
Po(xr) (er) Softmax([f(x)).
18

---

Published as a conference paper at ICLR 2026
Define the target log-likel leelikkeihhodd
X) = log Pa T<T)-
Let i denote the embeddings where subset R {I..... T 1 of context tokens is replaced by a masking scheme (e.g., zeroed embedding, mean embedding, or a learned sentinel used only at evaluation time). An attribution method produces scores Attr(z;; 2 such that
T-1 Σ Attr(a;) ~ g(X) ((Xall masked);
where Xall masked masks all context tokens {I,..., - 1}. We measure faithuuneess error on subset R by
((Attr) = g((() g(X)nn) -
Σ ER
Attr(z;)
so smaller ((Atrr) indicates more faithful attribution.
Divergence-Based Attribution Lower Bound. HETA Kullback-Leibler divergence to quantity the information contribution of each token to the target æy. Let Porig()) Po(- x<r) and let
p(t) masked (.) be the target distribution when only x; (with T) masked according chosen scheme while all other context tokens are kept. Define the total-variation distance
6 Poris p(()
masked
By Pinsker's inequality (with natural logarithms),
DRI p(oleter)) 2 1/2 6,2
With HETA's final form
Attr(a; Mrli] (BS + + (Kr(Porra pmaskooa)) ).
it follows that
Attr( +2) 2 Mr[i] 7 6,
Thus any token whose masking substantially perturbs the target distribution receives nontrivial attribution, modulated by its target-conditioned transition mass [t].
Norm Bounds on Hessian Sensitivity. To model second-order curvature, HETA considers the Hessian
HG Vx log Po(ET (<T)) (TT))((T))
where X E Txd are the input embeddings. The sensitivity of token v; is the entrywise -mmsss of the Hessian rows corresponding its d-dimensional block:
Td
S(T)
Σ
Hrli d: 1i+ d, a ||1,
j=1
with II, selecting the block rows for token By standard norm relations,
5 < Hr < (Td) Hrlle
where 11,1 is the entrywise & norm and ||.||r the Frobenius norm (the last inequality uses ||A|| S VNJJAAe with N (Td)?). These bounds control each token's 's curvature- brase sensitivity by the global curvature magnitude measured in Frobenius/entrywise norms.
A Attribution Envelope from Information Loss. Let the (signed) bog-robbabylity change from masking J; be
A log Pa PT(T (eT)) - log Po(ET I<eT())))
and define its positive part A max (0, A.}. HETA's final score (cf. equation uses the KL term with weight y.
Altr(1;+ta = Mr[i] (35(T) DKT (Porig DKI(Porik
p(i) Phlasteed)
19

---

Published as a conference paper at ICLR 2026
For interpretability report the envelope
Cnv(r,-xr) = Mr 5
+ A,),
which upper- upper-bounds bounds the final score whenever D I I A (a condition we verify empirically for our masking schemes). This provides conservative, target-linked scale for attribution magnitudes without assuming monotonicity of log-probability under masking.
Functional Faithfulness via a Taylor Remainder. Let g(X) log Po(oT) F<r). For perturba- tion Rd applied to token i's embedding block, second-order expansion yields
g X Ed)
g(X)
(Vz. <i) 1 c Hz ci2
where H Ha: E Bdddd is the token-specific block of Hr. If is C2 along the segment [X, X + ea], the remainder satisfies
((X+))--(
(V).9,c) S (Mma(HHuse (5)) ||e:l2:
for some on the segment, with Amax the top eigenvalue. This shows that incorporating curvature information yields a principled upper bound on local functional deviation that first-order or attention- only methods cannot capture.
Additive Attribution Approximation. Lastly, HETA exhibits an approximate additivity prop- erty when interactions between context tokens are small relative to their marginal effects. Let g(X) = log Po( x<T) and let X masked denote the embedding sequence where all context tokens (1..... T 1} are masked using fixed scheme. Then, under firstssecond-order Taylor approximation of g around Xall masked and assuming off-diagonal Hessian blocks Hij (i ( j) are negligible,
Attr(a, + IT) g(X) g(Xall maskea) log (O ÆTT) log (x all masked).
(6) More generally, writing Ax, for the embedding change induced by unmasking 24, the deviation from additivity admits the bound
Attr(z; + xT) (9(X))9(X marked))
S
Σ 2
1
|Azz [Hsy | + O(JAXX))
so the approximation in equation 6 is accurate when cross-token interactions (off-diagonal curvature) are small an effect further mitigated practice by HETA's target-conditioned gating and curvature terms.
A2.2 FAITHFULNESS LIMITATIONS OF GRADIENT-ONLE AND KL-ONLY METHODS
While gradient- based and KL-based attribution methods are popular for token-level interpretability. they both suffer from fundamental faithfulness limitations: gradients capture only local linear effects and miss curvature-induced changes, whereas KL-only measures neglect cross-token interactions. We formalize these limitations and provide rigorous bounds the resulting faithfulness error. Throughout, let (X) = log (o Z.T), let A S [l...., T 1} be a masked subset of context tokens, and write AX X X R for the dlccecconcte embedding perturbation induced by unmasking R (with block Ax; e Idd token i and zeros elsewhere).
Lemma A2.1 (Faithfulness error of gradient-only reconstruction). Define the gradient-only recon- struction of the log- likellhodo change by
Aggrad (R)
AA,
Az, V9(X
EE
Assume g is C2 on the line segment (Xir tAX:tE [0, g at some point this segment (by the mean-value form of the second-order Taylor theorem). Then the faithfulness error satisfies the exact identity
1]]. and let H{{)) denote the Hessian of
1 1 AX H(E AX
C (Attrgrad) = g(()) 9(x(n)= Agrraa -
Moreover:
20

---

Published as a conference paper at ICLR 2026
1. (Two-sided bounds) For the operator norm
llopp
0 S (Atttgrad))
2 [((]llop |AXI
2. (Curvature lower bound) If the spectrum of (E) is bounded away from zero in magnitude, i.e. minj (H) μ 0, then
C ((Attrgaad)
Z 2 IIXXI
Hence, whenever the target log- ligliike exhibits non non-negligible curvature along the masking trajectory, the gradient-only reconstruction incurs quadratic error perturbation size, and this error cannot be reduced below the curvature floor without incorporaing second-order information.
Proof. By the second-order Taylor expansion of g about X in the direction AX, there exists Ç x A with t" E (0, 1) such that
g x g R) Vg(X AX + AX H()) AX.
Subtracting the gradient-only reconstruction Âg grad (R) yields
(X)) g(X,nn) Agrra (R) AX A H(E)
and taking absolute values gives the stated identity.
For the upper bound,
AX HTEE) A AX
(IH |
which implies the first inequality after multiplying by 1
For the lower bound, diagonalize the symmetric Hessian H(E) = QAQ with orthonormal Q and real eigenvalues (Λ₃)- Writing y Q AX, we have
AX H()) AX
Σ 230 2 min 1Λy1 Σ vス 1 11AAX|
Multiplying by concludes the proof.
Remarks. (i) The simple but common surrogate ΣiER (X) Ar; (gradient evaluated at X rather than X).z) satisfies analogous result with the Hessian evaluated at another intermediate point on the same segment. (ii) The lower bound requires a curvature floor μ > 0 in magnitude; flat or sign-cancelling directions the may be small, which is precisely when curvature-aware terms like HETA's Hessian component are least needed.
A2.3 HETA AS AN OPTIMAL BULTI-VIEN ATTRIBUTION
We now argue that HETA reduces faithfulness error by jointly incorporating second-order curva- ture (Hessian terms), information-theoredd contributions (KL), and causal gating (semantic flow). Throughout, let g(X) log Po (ZT Z<T), let R C (1...., 1} denote masked subset of context tokens, and write AX = - XiR-
Theorem A2,2 (HETA Improves Faithfulness under Controlled Curvature and Calibration). Define HEТА attribution by
AtIThETA(Z) = Mr (BS: + 7714.
where [i E 10.1) is the target-conditionel semantic transition mass (zero if no causal path to T),
S, 2 0 is the Aassian-based sensitivity for token i, and 1, 2 is a KL-based information contribution for token Assume:
1. Second-order accuracy. g is C² along the segment (X +tAX 10, 1] with bounded third derivative that the Taylor remainder satisfies RRal Crem AX
21

---

Published as a conference paper at ICLR 2026
2. KL calibration (singleton). There exist E; 2 0 such that
1. (g(X) 9 S Ess ie R.
3. Curvature alignment. The sensitivity scores S; approximate block quadratic mass: there exist constants C1, C2 0 with
C1 Σ 11 || 2
Σ
St S 02
Aar (E)
iER
for some & on the segment. Then the faithfulness error of HETA satisfies
(AtITHETA)) < min
(E) |AX|
1 1 (
|AX
+
Σ EE
E/
grad-only upper bound
KL-only upper envelope
cc ER
Axxll2 (( - y
A + Crem |
KR
for some positive constants C, Cg depending only the normalizations S, and I, where H the symmetrized off-diagonal interaction operator, and A max(0, 9(X) (x)x)))) In particular, for sufficiently small ||AX|| and calibrated (3,7), the HETA error is strictly smaller than the better of the gradient-one only or KL-only reconstructions up to the cubic remainder.
Proof. By the second-order Taylor expansion,
g(X)
R( = ()(XiR) AX A H(E) + R3.
with || Rall S Crem |X
Decompose the quadratic term into block-diagonal and off-diagonal
parts:
AX AX AX= AanTHo(E)A) + Σ H.j(E)
SER
ifJ
Gradient-only envelope. The gradient-only reconstruction only the linear term; its error is exactly TAX (()) + O(|AX|]
|)), which is upper-bounded by 1 (E))lop AX + O[|AX|]
KL-only envelope. Summing singleton drops and invoking the calibration assumption.
(9(X)-9(X ())) + Σ = vo(X.)) AX+ Σ
Hr. + O(IAX () + ΣES.
KER
ER
ER
K
so the KL-only error on R. is controlled by the off-diagonal quadratic form plus calibration and higher-order terms:
x() - R)
Σ④
S
H off (E) (E)llop
AX +
>> E + O(IAX
ER
ER
HETA correction terms. HETA adds two nonnegative corrections gated by Mr
Σ ER
Mr BS; and Σ Mr
SE
By curvature alignment, Σien Mr[i] & EIER 2| ([) after renormalizing Mr on the causal set; similarly, I, nower-bounde the positive log-drop A up to calibration. Subtracting these controlled positive masses from the respective envelopes yields equation ? with the same cubic
remainder Crem | Choosing (3, 7) so that the linear-in-S, and lincar-in-J terms dominate the quadratic envelopes for small |XX| gives strict improvement.
22

---

Published as a conference paper at ICLR 2026
A2.4 STABILITY AND CAUSALITY GUARANTEES
Lemma A2.3 (Local Stability). Assume has Lipschitz. Hessian in a neighborhood of X, i.e. [(Vll S LH, and the KL term locally Lipschitz in embeddings with constant LKL for the chosen masking scheme. Then for any perturbation e to token i's embedding with ||||e S.
AttrHETA e) AAtTHETA(ZA) S 8 LKL 6,
where Cs depends on local bounds of || and Lu through the HVP estimator used in S..
Proof. S; is computed from Hessian-vector products restricted to token i's block. Under a Lipschitz Hessian, the map X + Hr is locally Lipschitz in operator norm with constant Lg. the block- restricted HVP magnitude changes by at most Csò. The KL predictive map X + Po Z<r) is locally Lipschitz under smooth decoder dynamics, giving the stated LKL bound for L.. Multiplying by nonnegative B, y and the gate Mr E [0, 1] yields the claim.
Theorem A2.4 (Directional Causality). If Mr for token 24, then AttTHETA (24)
Proof. By construction, Mr multiplies all terms in AtETHETA (x,) and is zero when no causal (masked) attention-flow path from z; reaches target position T. Hence AttThEtA 0.
A2.5 INTERPRETATION
Theorem A2.2 shows that, under standard smoothness and calibration assumptions, HETA reduces faithfulness error relative to gradient-only or KL-only reconstructions by (i) recovering diagonal curvature mass via S₁ and (ii) capturing singleton information drops via T; both restricted causal paths by Mr. Lemma A2.3 ensures local robustness to embedding perturbations, and Theorem |2.44 codifies target-conditioned causal sparsity. Together these results justify HETA principled, multi-view attribution mechanism with theoretical error control beyond single-view methods.
We now provide a formal interpretation of HETA as constrained caattssuares fit to -ikeeiiiihood drops, clarifying that the combination is not ad hoc but arises from an optimization with causal and information-theorrtiic structure.
A2.6 FAITHFULNESS AS RECONSTRUCTION ERROR
Let gXX) log Pa(aT x<T). For subset of masked tokens,
Ag(R) == g(X) ,(X,))
We seek attributions a; 2 0 that reconstruct these drops:
Σ a4 Ag(R), VRC (1....,T-1)
Define the objective
min (Ag(R) Σ a.)
ER-D
GER
for distribution D over subsets (e.g., singletons and spans).
A2.7 SECOND- DECOMPOSITION Ag(R)
A second-order expansion around X gives
(R) ~ Ev. + Σ Ao. H.]AA
EEN
IJER
with higher-order residuals. This highlights the roles first-order marginal effects and second-orde interactions.
23

---

Published as a conference paper at ICLR 2026
A2.8 CAUSAL AND INFORMATION- THEORRTIC CONSTRAINTS
To ensure interpretabilityy
1. Causal constraint: d; = 0 whenever Mr[1] = 0 (no causal path to the target).
2. Information constraint (calibrated): 0, should scale with the measured singleton informa- tion drop; encode this via linear feature i, DEL (Po(Porig |P masked) with per-scheme calibration.
A2.9 HETA AS THE SOLUTION
We parameterize
0; = Mr|i] (38: + 7i1,,
where is a curvature feature (e.g.. block-restricted Hessian mass) and i, the KI. feature. The weights (5,7) 7) 2 0 are chosen (by cross-validation or validation loss) to minimize the reconstruction objective under the constraints. This yields HETA constrained least-squares fit using causal gating and two complementary, theoretically motivated features.
Theorem A2.5 (Constrained Least-Squares Optimality). Let a* the minimizer of the unconstrained reconstruction objective. Let C = {a by causal gating and linear feature mixing. Then the HETA solution
(Bs, + 3, 2 03 be the feasible set induced
a HETA = arg min ENDD aEC
(Ag(R) - E as)
iER
is the best (in the reconstruction sense) causal-gatt linear combination of curvature and information features. Moreover, if C is convex in (B,7) (which it is), the minimizer in (8,7) unique up to collinearity of (s),
Proof. Fix (st, it) and [1] The map (3,7) + a(3, 7) is linear, and the objective is convex quadratic in (B,7) (expected squared error of linear model). Therefore the minimizer over (8,7) 7) > 0 exists and is unique unless the feature vectors are collinear. The causal zeros are enforced by Mr [i]
O. Hence a HETA is the optimal element of C' for the reconstruction objective.
A2.10 ERROR BOUNDS FOR LOW-RANK, WINDOWED HETA
We derive finite-sample bounds on the attribution error incurred when HETA approximated by (i) a low-rank Hessian and (ii) windowed context truncation. Recall the target-conditioned attribution for token .;:
Attr,
Mr[il (3 + 7I.),
(8)
where [t E [0, 1] is the semantic transition (causal gate) to position T. s(T) the Hessian-based sensitivity for token i, and Z; is the nnermatiio-theoreee contribution (KL change T when x; is masked). Let Attr; denote the approximation obtained by rank-k Hessian and window of size W around T::
Attr; = Mr[i] (8 (()) + +ī.)
analysis. We make the following mild, standard assumptions for language-model attribution
Assumptions.
[label=(A10), leftmargin=2.2em]
1. Hessian low-rank tail. Let H, V log Po(ar Fer) E (Td) be the true Hessian
and Hş its best rank-k approximation in Frobenius norm Beckart-Youngge Define the tail energy
1/2
Tk -HT Helle
(E 1
(10)
24

---

Published as a conference paper at ICLR 2026
2. Block sensitivity functional. For token i, let II, 10,1)axT4 select its d-dimensional embedding block. The true sensitivity is 5/²) = ||I, (entrywise &1), and the low-rank one is ) = = 1|1- We the inequality
[Hr Ha)llı I ca (H H&)lEF S ca Tke
Cd - vd.
(11)
3. Windowing leakage. Windowing of size W around T removes causal paths that leave the window. Let
(i; W) - |M - Mr[ill s (w)(W),
(12)
where EM (W) is the (nstance-dependenn total semantic-flow mass of paths that traverse tokens outside the window (normalized as in My). We allow < (W) to decay with W.
4. Distributional stability under windowing. Let Porig()) and plt) (a) denote the next- token distributions at T under full context; let (.rig()) and Prossteed (-) be the corresponding windowed distributions. Suppose the simplex bounded away from zero: there exists p(()
J E (0, 1) such that min, Porig(1) 2 and min, masked (v) 2 p. Define total-variation shifts
Enrig
|arig
Poriglli, I
maakk
(2)
|| masked
p(() P II
(13)
Then using standard Lipschitz bounds for DKL on the y-truncated simplex,
I Σ =
[DR( Porig ||
pll) Polaskooa - DKT (Porig
p(0)
muskead)
S (Eorig + mnssk)
(14)
Per-token error decomposition. inequality yields
Subtracting equation 9 from equation aand applying the triangle
|ttr; Attry| S
MT - Mrlil (35 + +Z,) - Mrlil (8 (B)s - -5 + II -Z.).
6xrG:W)
(15)
Each difference term is bounded using (A2)-
Theorem A2.6 (Per-token HETA approximation error). Under (A1) for any token i,
Attr, Attr;
II)) + Bea + Y I (W) (3S(7)
(Forig maskk)
(16)
Proof sketch. The first term follows from equation 12 For the Hessian sensitivity, note that
(s())
3 < [H Hx)]l < ca [HT H&)llF S ca Tk by equation 11 equation 10 For
the KL term, apply equation 14 Finally, M < absorbs the gate in the second summand of equation 15
Aggregate error bounds.
Let || -li denote the sum over tokens. Summing equation 16 over
i 1. T and using Σ, s(T) ||HTT| and I. s Cr (finite by bounded logits) gives
(T)
Attr,/ S eMr (8 181 + BT + 1 (Tearig + Σ meak (17) 1.
When the window covers most causal paths (eM (W) - as W 1) and the Hessian spectrum is rapidly decaying (7&-00 - 0 as l: 1), the approximation error vanishes. If. additionally, truncation minimally
perturbs next-token distributions (small Eorig and maste), the KL component is stable by equation 14
Discussion of constants. ca Vd is the block-size factor connecting entrywise é to Frobenius norms; replacing & by lz tightens ca to 1. The factor 1/μ in equation 14 is standard for Lipschitz j-truncated simplex; in practice, logits temperature- reg.uuarieeer or label- smoothed, yielding μ > O. The window leakage e (W) can be estimated empirically by measuring the semantic-flor fflw mass that crosses window boundaries (e.g., via rollout held-out inputs).
continuity of DKL
25

---

Published as a conference paper at ICLR 2026
Takeaway. The total approximation error decomposes additively into window term (missing causal paths), a low-rank term (Hessian tail energy), and a distributional term (next-token shifts under truncation). Each term be independently controlled by increasing window size W, rank k. or enforeing small distributional shifts (e.g., via overlap or sentinel-condttionnngg respectively.
A2.11 INTERPRETATION
This optimization view shows HETA is the best causal-gated linear combination of curvature and information features for reconstructing log-likelihood drops, rather than an ad hoc sum. Coupled with Theorem A2.2 it explains both why these views are needed (to control distinct error sources) and how they are combined (by constrained least squares) to reduce faithfulness error.
A3
EXPERIMENTS AND RESULTS
A3.1 CURATED ATTRIBUTION DATASET: NARRATIVEQA @ SCIQ AND THE DSA METRIC
We construct focused evaluation set to whether attribution methods concentrate importance truly predictive evidence in generative settings. Each instance is built by pairing one NARRATIVEQA item with one SCIQ item create two-segment input paragraph. The first segment drawn from NARRATIVEQA summary (ensuring grammaticality, self-containment, and no external coreference). while the second is the supporting segment from the SCIQ item that lexically contains the correct answer. The corresponding question is taken verbatim from the paired SCIQ instance. This structure ensures that the second segment the only answer-bearing span, while the first is plausible but irrelevant to the question question--ielding -ielding a controlled contrast between diagnostic and distractor context.
The final input to the model is constructed as
[NarrativeOA] 
 [SciQ]

 [Question],
and attribution is computed with respect to the first answer token Zr in the autoregressive factorization log (o Z<T), where T1-T-1 are the input tokens. This framing conditions attribution on the entire context and question, avoids leakage from later tokens, and focuses evaluation on the onset of the model's answer.
Annotation Protocol. Answer-token supervision is automatically derived from two independent models and GPT-5 (Thinking)-eade operating under the prompt: "Mark all tokens in the second segment necessary correctly answer the question." " For each instance, the models identify the minimal subword-level span that includes the answer and any essential lexical supports (e.g., units, definitional context). Final supervision uses the intersection of these two annotations to ensure high precision and reduce over-selection noise. All tokens are aligned to the evaluated model's s tokenizer, and when answer spans split across subwords, all relevant indices are retained. If the answer appears multiple times in the second segment, lexical cues from the question are used to disambiguate. Inter-annotator agreement strong: token-level F; 0.91 and Cohen's 0.89, averaged over the 2,000 curated examples.
DSA Metric. To evaluate attribution accuracy under this setting, we introduce the Dependent Sentence Attribution (DSA) metrie, which quantifies how well the method concentrates attribution mass on the annotated evidence in the second segment while suppressing spurious mass on the first. Let S be the set of supervised subword indices in the second segment. Let ss; and fs, denote the normalized attribution assigned to token i when it lies in the second or first segment, respectively. Attributions are normalized per instance such that the total mass over both segments sums to one. The DSA score is
DSA
Σ SS; FES
Σ fsj.
jEPissiSent
Higher DSA indicates that the model assigns more weight to relevant evidence (as defined by S) and less to distractors, aligning with the intended causal structure of the input.
26

---

Published as a conference paper at ICLR 2026
Attribution Method
LongRA
TellMeWhy
WikiBio
Soft-NCI
Soft-NS!
Soft-NCT
Soft-NS
Soft-NCI
Soft-NS
Qwen2.5 3B
Integrated Gradients
1.20
0.14
1.30
0.09
1.10
0.18
ContextCite
1.90
0,56
1.65
0.47
1.45
0.79
Peering (PML)
2.05
0.61
1.80
0.50
1.60
0.85
IDD-backward
1.05
-0.08
1.82
0.00
0.12
0.49
Attention Rollout
0.38
-0.03
0.22
-0.07
1.85
0.43
fAML
0.23
-0.09
0.08
-0.10
0.20
-0.04
Progressive Inference
1.30
0.25
1.18
0.26
1.00
0.21
SEA-CoT
1.50
0.31
1.32
0.33
1.15
0.34
ReAgent
1.66
0.38
1.47
0.39
1.25
0.40
HETA (Ours)
10.1
2.50
9.0
2.10
3.90
2.20
Table 4: Attribution faithfulness on Qwen2.5 3B across LongRA, TellMeWhy, and WikiBio using
Soft-NC and Soft-NS. Higher scores indicate stronger robustness and alignment. Mean of 3 runs; std
< #..06.
Dataset Statistics. Using this procedure, we construct a balanced dataset of 2,000 curated para- graphs, each with a corresponding question and high-precision attribution supervision set. The distribution of answer spans covers a range of entity types (scientific terms, quantities, mechanisms) and span lengths (mean 2.3 tokens). All supervision, tokenization, and scoring are conducted using the same subword scheme as the evaluated model, ensuring compatibility across methods.
This curated dataset and the DSA metric together provide controlled, interpretable, and target- conditioned framework for evaluating token-level attribution precision in generative LMs.
A3.2 EXAMPLES
We present qualitative examples of outputs generated by HETA, highlighting context words with attribution scores 2 0.5 for the predicted target token (figure 4[5 These visualizations illustrate how HETA effectively identifies semantically and causally relevant tokens (e.g., "pizza," "cut," "knife" for predicting "slice"; "shared," "pictures," "zoo" for predicting "friends"), while down-weighting less informative words. The target words are shown without bounding boxes for clarity, emphasizing their contextual dependencies.
A3.3 ATTRIBUTION FAITHFULNESS ON QWEN2.5 3B
Table reports Soft-NC/Soft-S on LongRA, TellMeWhy, and WikiBio for the Qwen2.5 3B decoder-only model. HETA attains the highest scores on every dataset and metric, e.g., 10.1/2.50 (LongRA), 9.0/2.10 (TellMeWhy). and 3.90/2.20 (WikiBio), substantially outperforming strong baselines such as ReAgent, SEA-CoT, and PML. Methods based on gradients or attention alone (Integrated Gradients, Attention Rollout) trail by wide margins and often yield low Soft-NS, indicating poor stability under input perturbations. Contrastive/coppu-aiddee approaches (ContextCite, PMI., TDD-bbckward) improve suggesting that correlation- or ablation-dii drivvnnnnndiiinnnee surrogates do not fully capture target-conditioned causal influence. The consistency of HETA's gains across three distinct datasets underscores its robustness to domain shift and its ability to localize semantically causal tokens rather than correlational artifacts.
negative
raw gradients but remain well below HETA,
A4 ABLATION STUDIES
A4.1 ABLATION STUDY OF HETA: COMPONENT WISE CONTRIBUTION
To quantify the contribution of each module in Dessian-Enhancee Token Attribution (HETA), perform a controlled ablation on the GPT-J 6B backbone using three public benchmarks--oom
TellMe Why, and WikiBio together with the curated attribution dataset introduced in Section
We
27

---

Published as a conference paper at ICLR 2026
evaluate six configurations: (1) HETA (Full) Transition + Hessian + KL, (2) Transition Only, (3) Hessian Only, (4) KL Only, (5) No Transition Gating (Hessian + KL without the learned semantic transition vector Mr). and (6) Uniform Transition (equal token weights instead of My). All results are averaged over 1,000 randomly sampled instances per dataset. We use the same evaluation metrics as in the main experiments: Soft-NC and Soft-NS for attribution sensitivity the benchmarks, and Dependent Sentence Attribution (DSA) for buman--liggned token importance on the curated set. Aggregation hyperparameters are fixed masked-token perturbation.
0.5 and y 0.5, and KL divergence is computed via
Table 5: Ablation study of HETA components. Reported values averaged across all datasets for GPT-J 6B. Mean over independent runs and std < 0.2
Configuration
Soft-NC
Soft-NS
DSA
HETA (Full)
9.78
2.31
4.70
Transition Only
3.12
1.52
2.21
Hessian Only
2.89
1,45
2.97
KL Only
2.23
1.21
2.74
No Transition Gating
4.31
1.84
1.68
Uniform Transition
3.89
1.76
1.54
analysis. HETA (Full) integrates directional semantic flow (Transition), curvature-aware sensitivity (Hessian), and token-level information gain (KL), yielding the strongest overall performance (Soft-NC = 9.78, Soft-NS 2.31, DSA = 4.70). Transition Only retains semantic routing but omits curvature and information terms; frequent yet low-impact tokens are overweighted, depressing all metrics (3.12/ 1.52 2.21). Hessian Only measures second- order curvature without semantic guidance informativeness:: high-curvature but semantically peripheral tokens are amplified, producing noisy, less aligned attributions (2.89/ 1.45 / 2.97). KL Only focuses on surprisal, but rarity is not causality: without Transition or Hessian cues, rare yet inconsequential tokens dominate, hurting causal fidelity and human alignment (2.23 / 1.21 / 2.74). No Transition Gating (Hessian KL without the learned gate) aggregates curvature and information indiscriminately. allowing spurious semantic paths and reducing robustness/alignment (4.31 / 1.84 / 1.68). Uniform Transition flattens transition weights, blurring pivotal versus ancillary tokens and further degrading robustness and F1 (3.89 / 1.76 1.54). Overall, every ablated variant drops least one of the three orthogonal pillars pillars-semantiee semantic flow, curvature sensitivity, or information gain- and the metrics degrade accordingly. The full HETA stack excels precisely because it balances all three, delivering robust, semantically grounded, and causally faithful attributions.
A4.2 WEIGHTING ANALYSIS OF 3 AND 7 IN FINAL ATTRIBUTION
To study how the component weights affect HETA's behavior and reliability, sweep the coefficients in the final attribution rule
Attr(z,-zy) = My (B + "I(I,, -ZT))))
where S(T) is the Hessian- based sensitivity term and I(z; + zT) is the KL-based information contribution. The gate [t] enforces surget-conditioned causality. We evaluate grid of (3.y) values with the normalization 3 + 1, specifically 3 E {0.0, 0.2, 0.5, 0.8, 1.0] (and - (.. Metrics reported are faithfulness (lower better), sensitivity (lower better), syntactic robustness (Spearman p. higher is better), and F1 alignment (higher is better).
As shown in Table 6. the balanced setting 3 = 7 = 0.5 yields the best trade-off: lowest faithfulness loss (0.1 high syntactic robustness (p 0.91). and top Fl alignment (0.89). The KL-only endpoint (3 0, y = 1) achieves the lowest raw sensitivity but sacrifices faithfulness and robustness, while the Hessian-only endpoint (8 1, 0) is unstable (high sensitivity) and less semantically aligned. These results confirm that curvature and information contributions are complementary; weighting both terms comparably produces the most faithful, robust, and well-aligned attributions.
28

---

Published as a conference paper at ICLR 2026
B (Hessian)
7 (KI.)
Faithfulness +
Sensitivity +
Robustness 1
F1 (Alignment) t
0.0
1.0
0.179
0.022
0.70
0.82
0.2
0.8
0.143
0.027
0.78
0.84
0.5
0.5
0.108
0.025
0.91
0.89
0.8
0.2
0.124
0.041
0.86
0.81
1.0
0.0
0.254
0.087
0.54
0.68
Table Performance HETA for different (3,r) under the constraint + 1. Lower is better
for faihfulness/senitivttyy higher is better for robustness/F1. Mean over 3 runs; std < +0.05.
Variant
Sensitivity 4
Act./Pass. Robustness 1
F1 (Alignment) 1
Full HETA
0.025
0.91
0.89
Transition Only
0.031
0.72
0.76
Hessian Only
0.087
0.54
0.68
KI. Only
0.022
0.70
0.82
No Transition Gating
0.049
0.68
0.79
Uniform Transition
0.038
0.61
0.74
Table 7: Ablation the HETA framework. Lower sensitivity and higher robustness/F1 indicate better
attribution quality. Mean over runs and std < #0.04
A4.3 ROBUSTNESS OF HETA
To further demonstrate the robustness of our methodology, we performed a stress test and reported three attribution Setsris,,ensitivity, Active/Passive Robustness, and F/ (Alignment), evaluated across the six configurations described above.
Sensitivity measures stability under small input perturbations. Given Gaussian noise (0, added to each token embedding X,, we compute attribution scores across multiple perturbations and take the average standard deviation:
T
Sensitivity
(18)
T
where o; is the standard deviation of the attribution score for token i' the sequence length over which you average the per-token standard deviations.
Active/Passive Robustness captures syntactic invariance. For an original sentence and its ac- tive/passive rephrasing, align corresponding tokens and compute the Spearman rank correlation between their attribution rankings:
Robustness p(Attr(z; - Attr(ze (tt(2(/-447))
(19)
F1 (A (Alignment) evaluates semantic agreement with human annotations. Let Amodel the set of top-attributed tokens and Anuman the annotated set:
Amodee Anuman
Fl
(20)
Amodell + Anuman
Table 7 shows that the full HETА consistently yields the lowest sensitivity and the highest robustness and FI, indicating stable, syntax-invariant and human-aligned attributions. Removing transition gat- ing using uniform transitions degrades robustness and alignment, while dropping the Hessian term notably increases sensitivity. The KL-only variant attains the best raw sensitivity but underperforms robustness and F1, highlighting the complementary nature of all three components. Overall, these results validate that semantic flow (transition), curvature information (Hessian), and information gain (KI.) jointly necessary for reliable token-level attribution.
29

---

Published as a conference paper at ICLR 2026
We also address key limitations identified in method through targeted ablation studies. Specifically, we examine (i) the computational feasibility of HETA with various Hessian ap- proximations, (ii) scalability to long input contexts, (iii) the theoretical contributions of each multi-view component, and (iv) performance relative to stronger recent attribution baselines. These experiments validate our design choices and provide roadmap for practical deployment of HETA in large-scale language modeling settings.
A4.4 COMPUTATIONAL FEASIBILITY: APPROXIMATING THE HESSIAN
One primary concern with HETA is its computational overhead: computing full Hessian blocks across all layers introduces a runtime penalty of approximately 1.4x compared to gradient-based or purely perturbation-base attribution methods. To quantify this trade-off, we evaluate several efficiency-orienter variants of HETA on 1,000 examples (sequence length = 512) using GPT-6B.
As shown in Table 8 low-rank Hessian approximation (HETA-LR) reduces runtime by nearly 27% while maintaining most of the attribution quality, with only a slight drop in AOPC (0.61 + 0.59). Layer sampling (HETA-LS), which computes second-order information only for subset of layers, achieves an even greater runtime reduction (33%) with moderate degradation in faithfulness. In contrast, replacing Hessian information with gradient-squared sensitivity (HETA-GS) achieves the fastest runtime (240s) but sacrifices a considerable amount of attribution quality, confirming that full second-order curvature information contributes substantially both faithfulness and human alignment. These findings validate that Hessian approximations offer practical path to efficiency without entirely compromising interpretability quality.
A4.5 SCALABILITY: CONGCCONEET ATTRIBUTION
Another weakness of HETA is its limited scalability to long sequences, which are typical in large decoder-only language models. To address this, we evaluate several ccalabilit--oriented adapta- tions: windowed attribution (splitting long sequences into overlapping chunks) and combinations of windowing with low-rank and layer-sampled Hessian approximations.
Table Table|ssooe shows that full HETA attribution becomes computationally expensive for 2,048-token inputs (1,230s per 1,000 examples). Windowed attribution (HETA-WIN) cuts runtime nearly in half (690s) with a modest reduction in AOPC (0.58- -> 0.54). Combining windowing with low-rank Hessians (HETA-LR+WIN) yields an additional efficiency gain (runtime 580s) while recovering some lost attribution quality. Layer-sampled windowing (HETA-LS+WID)
is the fastest configuration but comes the highest cost in faithfulness. These results suggest that hybrid approximations (low-rank + windowing) strike the best balance between efficiency and interpretability, making HETA viable for very long contexts.
A4.6 ROBUSTNESS TO DECODING HYPERPARAMETERS
We evaluate the sensitivity of attribution quality to common decoding hyperparameters. For each method and model, sweep fixed grid temperature {0.2, 0.5, 0.9), top-p {0.8, 0.9, 0.95]. top-k {20, 50, 100), and repetition penalty (1.0, -2]-across three random seeds. For each metric, we report the maximum relative change A% across the grid (lower is better). Metrics Soft-NC, Soft-NS, and DSA. Models are GPT-J 6B. Llama-3.1 70B, Phi-3 14B, and Qwen2.5 3B.
Results and interpretation. Table shows that HETA's 's attribution metrics remain effectively invariant decoding settings across all four models, with worst-case A% 1 for Soft-NC. Soft-NS. and DSA. In contrast, all baselines-ContexxCCii of LMs (PML), WDDD-backward, Attention Rollout, fAMI,, Progressive Inference, SEA-CoT, and exhittt substantially larger variability, typically 2-5%. HETA's stability arises from three design elements: a target-conditioned causal gate that confines credit to paths terminating at the current prediction, a curvature-awan sensitivity term that smooths local logit perturbations, and an information-theerette component that scores distributional shifts rather than single sampled outcomes. These jointly decouple attribution from stochastic decoding heuristics (temperature, top-p, top-k), whereas ablation-, gradient-, and similarity- based baselines depend more directly on sampled logits or linear approximations and thus vary markedly with hyperparameter changes.
Integrated Gradients (IG), Peering into the Mind
30

---

Published as a conference paper at ICLR 2026
A4.7 SAMPUTE-QUALITY ABLATION: LOw-RANK AND WINDOWED HETA
We empirically calibrate the efficiency-accuracy trade-off of HETA under long contexts and scalable curvature approximations. Experiments use GPT-J 6B as the backhone and are averaged over LongRA, TellMeWhy, and WikiBio with sequence length 2,048 (batch size 8). We compare Full HETA to low-rank curvature (HETA-LR, rank = 64), windowed attribution (HETA-W) -IIN, window W=512 with 50% overlap), top-layer curvature (HETA-LS, last 6 layers), and a hybrid of low-rank + windowing (HETA-LR+WIN) We report faithfulness (AOPC), alignment (DSA), runtime per 1,000 examples, peak activation memory (relative to Full), and an empirical window-leakage proxy (W) (attribution mass discrepancy near window boundaries; lower is better).
Discussion. Table I1 shows that Full HETA provides the best faithfulness (AOPC) and alignment (DSA) the highest compute cost. Low-rank curvature (rank 64) preserves most quality (AOPC 0.58; DSA 4.55) while cutting runtime by ~34% and memory by ~~22%. Windowing (W =5112 yields the largest single speedup but introduces modest boundary leakage (higher M (W)). Layer sampling (top 6 layers) further trims cost but drops more second-order signal than LR. The LR+WIN hybrid nearly matches Full on quality (AOPC 0.59; DSA 4.58) while achieving the fastest runtime and lowest memory among approximations and substantially reducing boundary effects versus WIN-only. Replicating the sweep on Llama-3.1 70B with curvature restricted to the last 6 layers yields the same ordering, indicating that capturing dominant curvature directions and limiting cross- wrosswwindow spillover preserves attribution fidelity across models at a fraction of the cost.
A4.8 CAUSAL VALIDITY: TARGET-CONDITIONE GATE VS. ATTENTION-ONI ROLLOUT
We test whether HETA's target-conditioned causal gate (attention-value rollout restricted to paths terminating at the target position) reflects interventional influence, rather than correlation. Following standard activation-patching protocols, we perform targeted intervention sanity check: for each instance, we select the top-k context tokens ranked by an attribution method and patch their residual- stream contributions along incoming edges to the target token (replace with clean/reference run), then measure the drop in log Po (x Æerr ) of the correct target. We report three metrics, averaged over 2,000 examples from LongRA, TellMeWhy, and WikiBio (sequence length 512; batch size 8): (i) Patch-Corr (p): Spearman correlation between token ranks and interventional Alog P (higher is better), (ii) Intervention @5, mean Alog P when patching the top-5 tokens (higher better), (iii) Misattribution Rate: fraction of top-10 tokens whose Alog P falls below a fixed threshold 7=0.01 (lower is better). We compare HETA (Full), HETA No-Gate (removes My), Attention Rollout (Abnar& & Zuidema-style rollout without value/projection weighting), Token Gradients (first-order,, and Causal Tracing (score baseline from intervention-on ranking).
Discussion. Table 12 that HETA (Full) exhibits the strongest agreement with interventional causal effects: it achieves the highest Patch-Core and Intervention@s @5 and the lowest Misatrribution on both backbones. Removing the gate (HETA-N) -No-Gate) degrades all three metrics, confirming that attention alone is an imperfect influence proxy even when paths respect the causal mask. Token Gradients underperform due to flat regions and linearization error, while the Causal Tracing baseline Maseline-though interventional -ereeddinnalyiiddds weaker ranking agreement than HETA, since it lacks curvature and KL components and is not conget-conditioned by semantic gate. Together, these results substantiate that HETA's target-conditioned attention-value gate captures causal pathways that matter for the next-token distribution, and that its attributions are validated by direct interventions rather than correlational proxies.
replacing it with Attention Rollout
A4.9 ABLATION: CAUSAL EVALUATIONS AND DOWNSTREAM ROBUSTNESS
We extend our analysis with targeted ablations that probe (i) causal faithfulness via mid-layer interventions, (ii) counterfactual robustness under controlled edits, and (iii) downstream utility on tasks that benefit from faithful local attributions. Experiments are LLaMA-3.1 70B, Phi-3 14B, GPT-J 6B, and Qwen2.5 3B. Unless otherwise noted, we use the default decoding (greedy) and report means over 3 seeds.
31

---

Published as a conference paper at ICLR 2026
Variants. We compare HETA (Full) to the following ablations: w/o Hessian (remove curvature term), w/o KL (remove information term), w/o Transition (remove semantic gate), and LR+WIN (low-rank Hessian with rank 64 and 512-token windows, 50% overlap). For reference, we include strong baselines: ContextCite, Integrated Gradients (IG), Peering/PMI, TDD-hackward, Atten- tion Rollout, fAMI,, Progressive Inference, SEA-CoT, and ReAGent.
Causal evaluations. (1) Mid-layer value swapping. For a target token T, we patch value vectors at layer & from an evidence-supp supporting context into matched distractor context; a faithful method should rank the patched evidence tokens higher post intervention. We report Causal Pass@k (fraction of instances where at least one top-k token is from the gold evidence span after patching; k=5). (2) Counterfactual AOPC. We replace the gold answer span with semantically compatible foil (e.g.. unit/role swap) and compute rere-vver-peruuraatoon-
using the method's 's ranking as the deletion order; higher is better if the method concentrates mass on truly causal tokens.
Downstream applications. (3) Fact-checking EMA. We use FEVER-style claims paired with short contexts and add an "evidence filter" that masks the bottom 60% tokens by the method's s scores before prediction; we report the absolute change in exact-match (EM). (4) Tool-augmented reasoning @ir@/ IA. On small tool-use benchmark, we only pass the top-p attribution mass tokens (p=0.4) as the retrieved snippet to the tool call; we report Hit@1 I change versus using the full snippet. (5) Span- Fl (multi-token)). On our curated VarrativeeA@&err over susper- spans (intersection labels), rather than onset-only. (6) Decoding stability. We sweep decoding (greedy, top-p=d. 9, temperature=0.8) and report the average percentage change in three metrics Soff-NCC/Sot-NS/DDAA Lower A% indicates greater robustness.
set, we compute token-level precision/recall/F]
Findings. Table 13 shows that HETA (Full) achieves the strongest scores on both causal probes and downstream tasks, while maintaining the lowest Decoding Stability change (< 1% average variation across greedy, nucleus, and temperature sampling). The LR+WIN approximation closely tracks full HETA, indicating that our efficiency strategy (low-rank curvature + windowing) preserves most causal signal and downstream utility.
Removing any single component degrades performance in the expected direction: dropping curvature (w/o Hessian) reduces CF-AOPC and Span-F1 (weaker handling of nonlinear interactions), dropping the KL term (w/o KL) harms Fact EMA and Tool Hit@1 IA (less sensitivity
A4.10 KEY TAKEAWAYS
Our extended ablations provide several important insights: (1) Hessian approximations (low-rank and layer-sampled) offer a practical trade-off between runtime and attribution quality. (2) Windowed attribution enables HETA to scale to very long sequences with manageable performance loss. (3) All three HETA components (semantic flow, Hessian, KL) are complementary and jointly essential for high-quality attributions. (4) Expanded comparisons demonstrate that HETА outperforms even recent state-of-the--ar attribution methods, validating its broader utility.
A5 LIMITATION: COMPUTATIONAL OVERHEAD AND SCALABILITY (ALL MODELS)
HETA'S curvature term improves faithfulness but introduces extra cost from Hessian-vector products (HVPs) and parget-condition attention-valus value rollout. We quantify this overhead and show how low-rank curvature and windowed evaluation recover most of the accuracy at substantially reduced runtime/memory across all models considered: GPT-J 6B, Phi-3 14B, Qwen2.5 3B, and Llama-3.1 70B.
Setup. All runs use PyTorch with fused attention single NVIDIA A 1100 80GB. We report runtime per 1,000 examples and peak GPU memory. Faithfulnes//llgnmmenn use AOPC and DSA; long contect stress tests additionally monitor Soft-NC/NS (not shown here for brevity). Datasets: LongRA, TelyMeWhy, WikiBio. Unless specified, sequence length L L= 1024, batch size 8-8. For 1 70B 7-=248, curvature is computed on the last 6 layers. We compare:
32

---

Published as a conference paper at ICLR 2026
Table Runtime and attribution quality for Hessian approximations on 1,000 examples (sequence length = 512). AOPC and Fl represent attribution faithfulness and human alignment, respectively.
Variant
Runtime (s)
AOPC 1
F1
НETА (Full)
455
0.61
0.89
HETA-LR (rank=64)
330
0.59
0.86
HETA-LS (6 layers)
305
0.57
0.84
HETA-GS (grad-squared only)
240
0.52
0.81
Cam ordered a pizza and took it home. He opened
the box to take out a slice. Cam discovered that the store did not cut the pizza for him. He looked for his pizza cutter but did not find it. He had to use his chef knife to cut a slice.
Figure 4: Word-level attribution visualization for predicting the final word "slice." Each word is shaded based on its importance score for predicting "slice." " Darker red indicates higher attribution.
Sandra got a job at the zoo. She loved coming to work and seeing all of the animals. Sandra went to look at the polar bears during her lunch break. She watched them eat fish and jump in and out of the water. She took pictures and shared them with her friends.
Figure 5: Word-level attribution visualization for predicting the final word "friends." Bounding boxes highlight influential context words (e.g.. "shared," " "pictures," "zoo") contributing to the prediction of "friends." Darker red denotes higher importance.
33

---

Published as a conference paper at ICLR 2026
Table Faithfulness and runtime for long-contex. attribution (sequence length 2,048).
Windowed
methods use 512-token chunks with 50% overlap. Mean over 3 runs and std < +0.04
Variant
AOPC 1
Runtime (s)
НETA (Full)
0.58
1,230
TETA-WIN (512-window)
0.54
690
HETA-LR+WIN HETA-LS+WIN
0.55
580
0.52
525
Table 10: Sensitivity of attribution metrics to decoding hyperparameters (max relative change A%; lower better). HETA varies < 1% across all models/metrics; each baseline fluctuates 2%. Grid: temperature E {0.2, 0.5, 0.9}, top-p {0.8, 0.9, 0.95), top-k E {20, 50, 100), repetition penalty E {1.0, 1.2): 3 seeds.
Model
Metrie
HETA
ContextCite
IG
PML
TDD-bw
AttuRoll
fAML ProgInf SEA-CoT ReAgent
Sof-NCC Soft-NS
0.8
3.0
3.2
3.8
2.5
2.9
2.8
2.4
GPT-J 68
0.9
3.6
3.4
3.0
4.2
2.9
3.1
3.0
3.2
2.7
2.1
Soft-NC
2.6
3.1
2.3
2.5
2.5
2.3
Ilama-3.1 70B
Sof-NN
0.8
3.1
3.4
2.7
2.9
3.0
2.6
DSA
0.6
2.4
2.3
2.2
2.8
2.1
2.3
2,3
2.4
2.2
Soft-NC
0.9
3.7
3.5
3,2
3.9
3.0
3.3
3.1
3.2
2.9
Phi-3 14B
Soft-NS
4.3
4.6
3.6
3.9
3.9
3.3
DSA
0.8
3.1
2.7
3.3
2.6
2.8
2.8
2.6
SoftNNC
0.9
4,2
4.5
3,4
3.7
3.6
3.1
Qwen2.5 3B
Soft-NS
0.9
4.9
4.6
4.1
5.2
4.0
4.3
4.1
4,2
3.7
HETA (Full): exact HVP curvature on (or selected) layers; no windowing.
HETA-LR: randomized SVD, rank r=64, per token-block.
HETA-LS: curvature on a subset layers (top for 3B/6B/14B; top 6 for 70B).
HETA-WIN: 512-token sliding windows with 50% overlap.
HETA-LR+ low-rank curvature inside windows (recommended for long contexts).
Cross-model takeaways. Across all four backbones, HETA-LR+ WIN recovers 92%-96% of Full HETA's AOPC/DSA while reducing runtime by 40%-50% and peak memory by 30%-40%. The simple layer-subset setting (4 layers for 3B/6B/14B: for 70B) is already effective; adding low-rank curvature (rank 64) and 512/50% windows yields the best cost-quality trade-off. These empirical trends match the theoretical error bounds for wow-rank/windowed HETA: the low-rank residual is controlled by the neglected spectrum, and window truncation error is bounded by the causal-gate mass leaking across window boundaries, which is small for target-localized flows.
Residual limitations. Even with LR+WIN, НETA remains slower than attention-only or gradi- ent-only methods at very long contexts. Windowing may under-capture rare global interactions that span multiple windows; the causal gate mitigates but does not eliminate this. Finally, low-rank curvature assumes decaying spectrum; adversarially spectra would require larger rank r.
Practical recipe. For 3B-14B models L 1024, use HETA-LR (rank=64) HETA-LR+WI. for long inputs. For 70B at L > 1536, use last 6 layers + LR (64) + 512/50% % windows. This preserves most of HETA's faithfulness/alinnment while fitting single-Gii GPU budgets.
A6 EXTENDED EVALUATION AND DISCUSSION
In this section, we present additional analyses and experiments that further validate HETA's design choices, address the scope of its causal gate, clarify the role of L.LM-based annotation, extend
34

---

Published as a conference paper at ICLR 2026
Table 11: Coupute-quality ablation for HETA (long contexts), Backbone: GPTJJ 6B. Results are averaged over LongRA, TellMeWhy, and WikiBio with sequence length 2,048 (batch size 8). "Runtime" is seconds per 1,000 examples; "Memory" peak activation memory relative to Full; (M) ) is an empirical window-leakage proxy (lower is better). Mean of 3 runs; std < <0000 for AOPC/DSA and s for runtime, The same ranking and gaps are observed on Llama-3.1 70B when restricting curvature to the last 6 layers (omitted for brevity).
Variant
AOPC1
DSA
Runtime
Memory 4
Ew((W) +
HETA (Full)
0.60
4.68
1,280
1.00x
0.000
HETA LR (rank=64)
0.58
4.55
850
0.78x
0.006
HETA-WIN (W=512, 50% overlap)
0.56
4.40
690
0.72x
0.021
HETA-LS (top 6 layers)
0.55
4.30
760
0.75x
0.012
HETA-L-++ WIN (r=64, W=512)
0.59
4.58
0.66
0.010
Table 12:
Intervention-based causal sanity check. Mean over 2,000 examples; + denotes std
across examples. Top-k=5. HETA's lurget-conditioned gate yields the strongest alignment with interventional effects (highest Patch-Corr and Intervention 5, lowest Misattribution).
GPT-J 6B
Method
Patch-Corr f
Intervention@5
Misattr,
HETA (Full)
0.78 0.03
0.143 + 0.012
0.12 + 0.02
HETA -ao-aeee
0.53 0.04
0.089 + 0.010
0.27 + 0.03
Attention Rollout
0.49 0.04
0.081 0.011
0.30 + 0.03
Token Gradients
0.41 - 0.05
0.067 + 0.009
0.35 + 0.04
Causal Tracing (baseline)
0.62 = 0.03
0.105 + 0.011
0.21 = 0.03
Llama-3.1 70B (curvature on last 6 layers)
HETA (Full)
0.75 0.03
0.139 0.010
0.14 0.02
HETA - No--ate
0.51 = 0.04
0.086 + 0.010
0.28 + 0.03
Attention Rollout
0.47 0.04
0.079 + 0.010
0.31 + 0.03
Token Gradients
0.40 0.05
0.064 1 0.009
0.36 0.04
Causal Tracing (baseline)
0.59 0.03
0.101 0.010
0,23 1 0.03
faithfulness evaluation with classical perturbation protocols, and discuss generalization to multi-token targets.
A6.1 CAUSAL GATE: SCOPE AND VALIDATION
A natural question is whether HETA's target-conditioned semantic gate M7 faithfully captures causal influence, given that it operates at the token level rather than at the level of individual circuits attention heads. By design, HETA's gate combines attention-valus rollout with Hessian based sensitivity of the output logits to token perturbations, which implicitly aggregates residual-strean and MI.P effects into single token-level influence measure. We validate its causal relevance via deletion-style interventions: removing the tokens selected by the gate causes the largest drop in target log-probability among all compared methods. More fine-grained circuit-level mediation analysis (Wang et al. 2022) Conmy et al. 2023) is orthogonal to the goals of this work, as HETA targets token-level attribution rather than mechanistic circuit discovery.
A6.2 LLM-BASED ANNOTATION QUALITY
Our curated evaluation set uses the intersection of GPT-4o and GPT-5 (Thinking) annotations as ground-truth support labels. Using strong LLMs as evaluators now standard practice at top venues (Zheng al. 2023 Liu et al. 2023). and our dual-LLM intersection design further reduces label noise relative to single-mont model annotation.
To quantify residual annotation error, conducted targeted quality audit. Across the curated set, GPT-4o and GPT-5 disagree on only approximately 8% of candidate support tokens (token-level Jaccard 0.84), confirming that the intersection region is already high-agreement zone. On
35

---

Published as a conference paper at ICLR 2026
Table 13: Causal and downstream robustness ablation. Higher better for Causal Pass@5, Counterfactual AOPC, Fact-checking EMA, Tool Hit@ A, and Span-F1. Lower is better for Decoding Stability A %. Results averaged over four models (LLaMA-3.1 70B, Phi-3 14B, GPT-J 6B, Qwen2.5 3B); std < +0.05 for additive metrics and < +0.2 pp for A%.
Method Variant
Causal P
CF-AOPC1
Fact EMA 1 Tool Hit@1A f Span-F1 Decoding A %
HETA (Full)
0.86
0.63
+3.7
+4.1
0.81
0.8
HETA (LR++WIN
0.84
0.60
+3.3
+3.8
0.78
0.9
HETA (w/o Hessian)
0.77
0.53
+2.5
+2.7
0.72
1.6
HETA (w/o KL)
0.73
0.49
+2.0
+2.3
0.69
1.8
HETA (w/o Transition)
0.70
0,45
+1.6
+1.9
0.64
2.1
ContextCite
0.58
0.36
+1.2
+1.4
0.55
2.9
Integrated Gradients
0.52
0.32
+0,9
+1.1
0.49
3.4
Peering (PML)
0.55
0.34
+1.0
+1.2
0.52
3.1
TDD-backward
0.57
0.35
+1.1
+1.3
0.54
3.0
Attention Rollout
0,41
0.24
+0.5
+0,6
0.38
4.2
fAML
0.60
0.37
+1.3
+1.5
0.56
2.6
Progressive Inference
0.62
0.39
+1.5
+1.6
0.58
2.7
SEA-CoT
0.64
0.41
+1.7
+1.8
0.60
2.5
ReAGent
0.68
0.44
+2.0
+2.1
0.63
2.4
Table 14: GPT-J 6B @ L=1024, B-8 (1,000 ex). Mean over runs; std <..004.
Method
AOPC
DSA f
Runtime (8)
Peak Mem (GB) +
HETA (Full)
0.61
4.70
455
53,2
HETA-LR (rank=64)
0.59
4.55
330
41.6
HETA-LS (4 layers)
0.57
4.38
305
39.9
HETA-WIN (512/50%)
0.55
4.29
295
34.1
HETA-LR&
0.58
4.52
245
31.7
stratified sample of 200 instances manually labeled by expert human annotators, the GPT-40/GPT-
5 intersection achieves =0.93 precision and ==0.88 recall for answer-support tokens ffalse-positiv rate =7%, false-negative rate = 12%). These figures confirm that our curated labels are high-quality and that any residual noise is unlikely to affect the comparative ranking of attribution methods.
A6.3 EXTENDED FAITHFULNESS EVALUATION WITH CLASSICAL PERTURBATION METRICS
To broaden the evaluation beyond Soft-NC, Soft-NS, and DSA, we adopt the GiLOT evaluation protocol (Li et al. 2024) and supplement it with classical nerturbation-basae metrics. Specifically, we report: MoRF (Most Relevant First) (Samek et al.) 2017), which measures the area under the probability curve when tokens are deleted in order of decreasing attribution (higher is better); LeRF (Least Relevant First), which performs the reverse deletion (lower is better); ABPC (Area Between Perturbation Curves), which quantifies the gap between MoRF and LeRF curves (higher is better); and AOPC-Del/AOPC-Im (Hooker et al.) 2019). which measure the area over the perturbation curve for deletion and insertion, respectively (both higher is better). We additionally ran ROAR/KAR (Hooker 2019 evaluations (not tabulated due to space), which preserve the same overall method ranking.
Table 18 presents the results. НETA consistently achieves the highest faithfulness across all five metrics, with GiLOT ranking as the strongest competing method. These results align with and reinforce the Soft-NC/Soft-NS/DDS findings reported the main paper: HETA's combination of causal gating, second-order curvature, and KL-based information gain yields the most faithful token- level attributions, even under evaluation protocols designed independently of our framework. The ROAR/KAR evaluations (omitted for brevity) preserve the same ranking НETА GiLOT other baselines- goeeines-indicctingg indicating that adding classical retraining-baad based perturbation checks does not alter our main conclusions.
36

---

Published as a conference paper at ICLR 2026
Algorithm I Hessian-Enhanceee Token Attribution (HETA) with Target Gating and Efficient Curvature
Require: Decoder-only LM fe: tokenized input 21:7-1i target position T (first answer token);
embedding matrix E; hyperparameters 3. 3.72 0; window size W stride s; Hutchinson samples
m; low-rank rank k; layer subset Luub S {1, .1...... L}; masking operator mask(-) embeddings
Ensure: Per-token attributions Attr( -xx)) for E e {
1: Forward pass and target distribution
2: X < (e1,-.., er-1) with e; = E E(a
3: Pang(- Z<T) t Softmax fa (X) at position
D. Windowing over long contexts (optional)
4: Define overlapping windows W {[a b]} covering 1:(7-)) with length W and stride
5: Initialize accumulators M -- STi 0. [[]] 00, v/1] 0 for all
6: for each window (a :] W do
7: Restrict computations to tokens E (a b) and layers Saub
(1) Semantic transition influence (target-conditioned gate)
8:
for E Laub, h f1...., do
Obtain masked attention (l.h) and values V(lA), output proj. W
9:
10:
Compute target-conditioned rollout ((I.)) (i (T) (paths must end at T)
end for M,
12:
jabb T
- Eiec... Σ (i V. (1,b)
wo (.
13:
Normalize on the window: M + M laib)
E( M Labb
Li]
(2) Curvature-aware sensitivity via HVPs (low-rank, blockwise)
14:
Define token-block selector II, (projects to the d-dim block of token for E (a
15:
do S ((T./u:bb))
16:
1 E
II, Hy (Iar))h,
Rademacher on block
17:
HVPs HTv computed by Pearlmutter; optional rank-k range finder if needed
18:
end for
(3) Information contribution the target (KL)
19:
for
[a: E
bj do
20:
Form X by replacing + mask(e,) (e.g., zero/mean//seninel)
p(t) (- [<TT)== & fo( at T
Ilabb] (I: +T) DKI (Pong p(()
22:
23:
end for
(4) Window accumulation
for
i E (a do
25:
Mii+ - M, [t].
$li)= S (((MM)) -la:bl *T), -[i+=1
end for
27:
end for
A gggeggate across windows and compute final attribution
28: for i f1,..., 1 do
29: Mii + M / max 1, v/i)}
Sli + 5 (1,1),, [[l + T/] max [[]}
30: Attr(x, - xT) + M (3S + Til)
31: end for
32: return Attr(z; -xT) for all
A6.4 GENERALIZATION TO MULTI-TOKEEN TARGETS
While the main formulation defines attribution with respect single target position T, HETA natu- rally generalizes to multi-toke token target spans. For span of K target tokens Fr, .1,, we define the target as the joint log- probabiilty:
K K-1
Ispan (X) =
Σ
log Po(xT+* xertt),
(21)
k=0
37

---

Published as a conference paper at ICLR 2026
Table 15: Phi-3 14B @ L-1024, ==8 (1,000 ex). Mean over runs; std +0.05.
Method
AOPC
DSA f
Runtime (8) 4
Peak Mem (GB) +
HETA (Full)
0.60
4.85
520
47.0
HETA-LR (rank=64)
0.58
4.68
375
39.0
HETA-LS (4 layers)
0.56
4.55
345
37.2
HETA-W WIN (512/50%)
0.55
4.47
325
33.1
HETA-LR&WIN
0.57
4.63
275
30.2
Table 16: Qwen2.5 3B @ L=1024, =-8 (1,000 ex). Mean over 3 runs; std < <0.05.
Method
AOPC1
DSA f
Runtime (8)
Peak Mem (GB)
HETA (Full)
0.59
4.40
310
24.0
HETA-LR (rank=64)
0.58
4.30
235
20.1
HETA-LS (4 layers)
0.56
4.18
220
19.2
HETA-WIT (512/50%)
0.54
4.12
205
17.3
HETA-LR+ WIN
0.56
4.26
180
16.1
and compute per-token attributions by aggregating the HETA scores across the K target positions. Concretely, the span-level attribution for input token x; is:
K-1
Σ Mr+ (35(T+h) + +7 + rT+E)).
Attrspan (I:)
(22)
This is exactly the formulation used in our Span-F1 experiments, where HETA maintains the same advantage over baselines as in the single-token setting. Computation-wise, span attribution reuses the same forward pass and Hessian blocks; the cost grows roughly linearly with the number of target tokens K. Our scalability analysis (runtime and memory vs. context length and number of targets) confirms that even with multiple targets and long contexts, the overhead remains modest due the low-rank and windowed curvature approximations described in the main paper.
A6.5 COMPONENT CONTRIBUTION: WHEN DOES EACH TERM HELP?
The ablation study in the main paper establishes that all three HETA components are jointly necessary. Here, provide more fine-grained interpretation of when each component contributes most.
Hessian-based curvature. The curvature term S(T) contributes most to the faithfulness metrics Soft-NC and Soft-NS, which measure robustness under input perturbation. Removing it drops Soft- NC from 9.78 (full HETA) to 3.12 (Transition Only) or 2.23 (KL Only), while the Hessian-only variant still achieves relatively strong DSA of 2.97. This indicates that second-order sensitivity critical for capturing nonlinear token interactions that first-order or masking-only approaches miss.
KL-based information gain. The KL term [(, - aT) primarily boosts alignment with human- annotated tokens (DSA), lifting DSA from 2.21 (Transition Only) and 2.97 (Hessian Only) 4.70 in the full model. This confirms its role as probabilistic calibrator that ensures attribution mass tracks the actual information content of each token with respect to the target.
Semantic transition gate. The semantic gate Mr [i] alone (Transition Only) is competitive with naive baselines such No Transition Gating (DSA 2.21 vs. 1.68) and Uniform Transition (DSA 2.21 vs. 1.54), but its primary function is to enforce causal sparsity and direct the curvature and information terms toward causally relevant tokens. The full combination gate curvature KL. yields the best and most balanced performance across all tasks and metrics.
38

---

Published as a conference paper at ICLR 2026
Table 17: Llama-3.1 70B @ L-2048, B-4 (500 ex). std +0.05.
Curvature on last 6 layers. Mean over runs;
Method
AOPC
DSA
Runtime
Peak Mem (GB)
HETA (Full, 6 layers)
0.60
5.10
1180
74.5
HETA-LR (rank=64)
0.58
4.92
860
61.3
HETA-WIN (512/50%)
0.56
4.80
720
55.8
HETA-LR&WIN
0.57
4.96
620
49.7
Table 18: Extended faithfulness comparison on GPT-J 6B LongRA using classical perturbation metrics. HETA achieves the highest scores across all metrics. GiLOT is consistently the second-best method. Mean of runs; std < +0.04.
Method
MoRF f
LeRF +
ABPC f
AOPC-Del f AOPC-Ins
Integrated Gradients
40.8
43.2
0.35
0.52
0.42
Attention Rollout
41.3
42.9
0.40
0.53
0.43
ContextCite
41.7
42.5
0,45
0.54
0.44
ReAGent
42.6
41.9
0.52
0.57
0.47
GiLOT (Li et al. 2024)
43.1
41.3
0.58
0.59
0.49
НETА (Ours)
44.0
40.2
0.65
0.60
0.51
thought I lost my hat at the park today. spent
lot of time looking for it. was just about to give up when I saw something far away. It was my hat, stuck in a bush!
Figure 6: Word-level attribution visualization for predicting the final word "bush." " Attribution scores emphasize context words such as "hat," "stuck," and "park," which strongly influence the prediction of "bush."
39


Reviews:
[{"review_id": "EbRmrbdJhk", "reviewer": "Reviewer_icxv", "summary": "The paper proposes HETA, an attribution framework tailored to decoder-only LLMs that combines three components: (1) Target-conditioned semantic flow via attention–value rollout to gate only tokens that have causal paths ending at the target position. (2) Second-order sensitivity using scalable Hessian–vector products (HVPs) with Hutchinson estimators to capture curvature and token interactions that first-order methods miss. (3) Information-theoretic contribution measured as the KL divergence between the original target distribution and the distribution after masking a given input token. The authors also introduce a curated evaluation benchmark for token-level attribution in generative settings by pairing NarrativeQA and SciQ text segments.", "questions": "1. How does HETA behave when attributing to spans (e.g., the next 3–5 tokens, or an answer string) rather than a single onset token? Does gating generalize naturally by aggregating over targets, and how do costs scale?\n\n2. Could you provide a disagreement analysis between GPT-4o and GPT-5 annotations and a small human validation study on a stratified subset to calibrate false positives/negatives in the curated set?\n\n3. When does curvature help most, and when does KL dominate? Are there tasks where semantic gating alone is competitive?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The method is straightforward: The integration of structural routing (target-conditioned rollout), geometric curvature (Hessian sensitivity), and information change (KL) is conceptually clean and well-motivated for autoregressive decoding.\n\n2. The curated corpus and DSA offers a controlled setting to check whether attribution concentrates on truly diagnostic spans rather than distractors.\n\n3. It targets at the attribution for decoder-only models, which has certain potentials for interpreting LLMs.", "weaknesses": "1. Although the target-conditioned gate based on attention-value rollout is an improvement over pure attention attribution, it is still a heuristic causal proxy. Residual pathways and non-linear MLP transformations can confound causal flow. The paper did not directly validate causality with, e.g., causal mediation analysis, circuit-level interventions.\n\n2. The evaluation relies on LM-generated supervision for the curated set. The DSA construction uses the intersection of GPT-4o and GPT-5 annotations to reduce noise, but residual bias is possible, especially if systems trained on similar inductive biases label the data. Some human audit or a cross-annotation sanity-check would strengthen claims.\n\n3. The comparison with previous literature is not thorough. Previous studies on generative GPT models should also be discussed and compared, such as [1]. I also suggest the authors report metrics such as MoRF, LeRF, ABPC as in [1] for further comparison.\n\n4. Missing several baselines. A few classical perturbation baselines (ROAR/KAR, AOPC deletion/insertion curves) and model editing/tracing style checks could further improve the faithfulness. \n\n5. Format error. The reference style needs to be corrected.\n\nReferences:\n[1] GiLOT: Interpreting Generative Language Models via Optimal Transport. ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hpkwJhxDoY", "reviewer": "Reviewer_XxTJ", "summary": "This paper introduces Hessian-Enhanced Token Attribution (HETA), a novel attribution framework for decoder-only language models that addresses limitations of existing methods. HETA integrates three complementary components: (1) a semantic transition vector capturing token-to-token influence through attention-value rollout with causal gating, (2) Hessian-based sensitivity scores modeling second-order curvature effects, and (3) KL divergence measuring information loss under token masking. The authors construct a curated evaluation dataset (NarrativeQA + SciQ) with 2,000 instances designed to assess attribution precision in controlled settings. Experiments across GPT-J, Phi-3, LLaMA-3.1, and Qwen2.5 models show that HETA consistently outperforms a set of existing attribution methods (ContextCite, Integrated Gradients, Peering, TDD-backward, etc.) on faithfulness metrics (Soft-NC, Soft-NS) and alignment with human annotations (DSA metric), while exhibiting robustness to decoding hyperparameters and syntactic rephrasings.", "questions": "What concrete steps could be taken besides restricting curvature computation to a context window (Appendix A5) to ensure the practical usability of the HETA method on LLMs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: The HETA method proposed to combine established feature attribution approaches through a principled target-conditioned gating.\n- Quality: Authors conduct experimental evaluation across model scales (3B–70B), diverse tasks (LongRA, TellMeWhy, WikiBio), multiple metrics (Soft-NC/NS, DSA, AOPC), and systematic ablations. The robustness analyses (sensitivity to noise, active/passive paraphrases, decoding hyperparameters) are particularly thorough.\n- Clarity: The conceptual motivation of combining multiple components in the final attribution scores is motivated in a reasonable way, and a comprehensive discussion about the properties of the HETA method is provided in Appendix A2\n- Significance: the integration of semantic flow, curvature sensitivity, and information-theoretic measures addresses distinct failure modes of single-view methods, as demonstrated convincingly through ablations (Tables 5, 8, Figure 2). The proposed efficiency adaptations in A5 are a first step in addressing the important concern of scalability to bigger model and context sizes.", "weaknesses": "In terms of originality, the main concern with this work is that the proposed method is that it amounts to a weighted combination of previous techniques, namely context mixing methods for extracting layerwise contributions, hessian for curvature sensitivity estimation and input masking.  The semantic flow approach representing the gate modulating target-directedness of the final HETA formulation in Eq. 5 is very related to the input attribution formulation of the ALTI method [1], which is not cited but also employs rollout for attention-weighted value vectors. Notably, the ALTI-Logit method [2] can be seen as an existing combination of semantic flow (through ALTI scores) and direct logit attribution, which reflects a per-component impact on output probability akin to the purpose of the KL-Divergence used in this work. Moreover, the rollout approach was criticized as a heuristic for aggregating contributions across layers, without explicitly decomposing the contribution of FFN layers [3, 4, 5]. The masking used for the KL-Divergence computation can also lead to unrealistic result for language tasks and particularly for generation, provided that masked inputs are likely to be out-of-distributions for the model [6].\n\nIn light of these comments, the choice for the proposed semantic flow approach should be justified and possibly modified to account for the aforementioned works in this area. Moreover, the criticism towards gradient-based approaches should account for more recent techniques specifically adapted for Transformer-based systems, in particular [7] that also provides a convenient implementation in the LXT PyPI package. Including some of these techniques---which are more established within the feature attribution community---as baselines for the evaluation of HETA would further reinforce the validity of the proposed method.\n\nIn terms of clarity, I found the introduction to the Hessian component handwavy, employing various specific terms such as Hessian–vector products (HVPs) with Hutchinson estimators and Pearlmutter’s trick without providing mathematical formulations or pointers. Moreover, the Soft-NC / Soft-NS metrics are not widely established and should be introduced more in detail at least in the appendix. As a minor comment, multiple citations and tables are formatted incorrectly (e.g. lines 270-275, line 295 bolded citation,  table 5 excessively large).\n\n**References:**\n\n- [1]: [Measuring the Mixing of Contextual Information in the Transformer](https://aclanthology.org/2022.emnlp-main.595/) (Ferrando et al., EMNLP 2022) \n- [2]: [Explaining How Transformers Use Context to Build Predictions](https://aclanthology.org/2023.acl-long.301/) (Ferrando et al., ACL 2023)\n- [3]: [Local Interpretation of Transformer Based on Linear Decomposition](https://aclanthology.org/2023.acl-long.572/)(Yang et al., ACL 2023)\n- [4]: [DecompX: Explaining Transformers Decisions by Propagating Token Decomposition](https://aclanthology.org/2023.acl-long.149/) (Modarressi et al., ACL 2023)\n- [5]: [Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions](https://aclanthology.org/2023.acl-long.562/) (Oh & Schuler, ACL 2023)\n- [6] [Quantifying Context Mixing in Transformers](https://aclanthology.org/2023.eacl-main.245/) (Mohebbi et al., EACL 2023)\n- [7] [AttnLRP: attention-aware layer-wise relevance propagation for transformers](https://dl.acm.org/doi/10.5555/3692070.3692076) (Achtibat et al., ICML 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pMd9NyMj92", "reviewer": "Reviewer_Du4D", "summary": "The paper introduces Hessian-Enhanced Token Attribution, a new interpretability framework for decoder-only language models. HETA combines semantic flow tracing, Hessian-based sensitivity, and KL-divergence information metrics to provide more faithful, context-aware token-level attributions in generative models. \n\nExperiments show that HETA outperforms existing methods in faithfulness, robustness, and human alignment, and the authors also release a curated benchmark dataset for evaluating attribution quality.", "questions": "Please refer to Weaknesses.\n\n(1. How does the computational cost compare with existing works, in terms of time and memory requirements? The proposed method seems to incur higher computational overhead, for example, due to the KL masking and Hessian–vector product operations. How do the authors balance effectiveness and efficiency in practice?\n\n2. The stochastic estimation depends on the number of Hutchinson samples m. How does the choice of m affect the empirical performance and stability of the results in the experiments?\n\n3. The authors employ several LLMs of different architectures and sizes, including Qwen2.5-3B, GPT-J-6B, Phi-3-Medium-4K-Instruct, and LLaMA-3.1-70B.\nI am curious how the proposed method performs across models of the same architecture but different sizes, for example, among different sizes of LLaMA-3. Would model scale affect the attribution behavior, stability, or the relative advantages of HETA compared with baselines?)", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. It proposes a unified, principled framework (HETA) combining causal semantic flow, second-order sensitivity, and information-theoretic attribution.\n\n2. It addresses key flaws of attention-based and gradient-based methods, yielding more faithful and robust token-level explanations.\n\n3. It demonstrates strong empirical improvements in faithfulness, robustness, and human alignment.\n\n4. It introduces a new benchmark dataset for evaluating attribution in generative models.", "weaknesses": "1. How does the computational cost compare with existing works, in terms of time and memory requirements? The proposed method seems to incur higher computational overhead, for example, due to the KL masking and Hessian–vector product operations. How do the authors balance effectiveness and efficiency in practice?\n\n2. The stochastic estimation depends on the number of Hutchinson samples m. How does the choice of m affect the empirical performance and stability of the results in the experiments?\n\n3. The authors employ several LLMs of different architectures and sizes, including Qwen2.5-3B, GPT-J-6B, Phi-3-Medium-4K-Instruct, and LLaMA-3.1-70B.\nI am curious how the proposed method performs across models of the same architecture but different sizes, for example, among different sizes of LLaMA-3. Would model scale affect the attribution behavior, stability, or the relative advantages of HETA compared with baselines?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oad2dJuhCh", "reviewer": "Reviewer_uaYo", "summary": "The work introduces a unified interpretability framework, Hessian-Enhanced Token Attribution (HETA), for decoder-only large language models that integrates gradient-based, second-order, and information-theoretic perspectives to improve attribution fidelity and robustness. The work proposes three complementary components: (1) Semantic Transition Influence, (2) Hessian-based Sensitivity, and (3) Information-Theoretic Impact, which measures token contribution using KL divergence between masked and unmasked predictions.\nMoreover, the work proposes a curated benchmark combining NarrativeQA and SciQ passages. Experiments on multiple LLMs show that HETA improves attribution faithfulness, alignment, and robustness under input perturbations compared to baselines methods.", "questions": "- Several components of HETA (e.g., Semantic Transition Influence, Information-Theoretic Impact, and Hessian-based Sensitivity) appear conceptually related to previous work. Could the authors clarify the conceptual and methodological distinctions between HETA and these existing approaches? Specifically, what unique insight or mechanism does HETA introduce beyond combining known ideas?\n\n- Could the authors provide quantitative analysis (e.g., runtime, GPU hours, memory footprint) or approximations used to make HETA feasible for billion-parameter LLMs? How does computational cost scale with sequence length and model size, and could this limit real-world applicability?\n\n- The evaluation focuses exclusively on internal consistency with the proposed benchmark. Have the authors conducted human evaluations, qualitative analyses, or causal interventions to validate whether the generated attributions are truly meaningful and interpretable?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The work is clearly written.\n- The work constructs a carefully designed benchmark by combining NarrativeQA and SciQ, with explicit separation of evidence and distractors.", "weaknesses": "- While HETA presents itself as a unified framework, several of its core components have been explored in earlier studies. For example, the Semantic Transition Influence and Information-Theoretic Impact elements are conceptually similar to those in GiLOT [1], which models token influence through semantic transition and distributional shifts measured via OT distance and KL divergence. Besides, the Hessian-based Sensitivity component also builds on prior work using second-order derivatives for attribution [2,3]. In evaluation, the benchmark’s metrics resemble those used in previous interpretability benchmarks without citing relevant works [4,5]. As such, the incremental integration of known techniques, rather than a fundamentally new formulation, weakens the originality claim.\n\n- The experiments mainly compare HETA against older or general-purpose attribution methods, but omit several recent baselines [1,6] specifically designed for generative or autoregressive models. Without these baselines, it is difficult to assess whether HETA genuinely advances the interpretability frontier for generative LLMs.\n\n- The use of Hessian–vector products for second-order sensitivity introduces high computational cost, particularly for long-context LLMs. The paper lacks analysis of the method’s runtime, memory footprint, and scaling behavior. It is unclear whether HETA is feasible for real-world interpretability auditing of modern multi-billion-parameter models.\n\n---\n[1] Li, X., Chen, J., Chai, Y. and Xiong, H., 2024, July. Gilot: Interpreting generative language models via optimal transport. In Forty-first International Conference on Machine Learning.\n\n[2] Sundararajan, Mukund, Ankur Taly, and Qiqi Yan. \"Axiomatic attribution for deep networks.\" International conference on machine learning. PMLR, 2017.\n\n[3] Reing, K., Ver Steeg, G. and Galstyan, A., 2021, March. Influence decompositions for neural network attribution. In International Conference on Artificial Intelligence and Statistics (pp. 2710-2718). PMLR.\n\n[4] Li, Xuhong, Mengnan Du, Jiamin Chen, Yekun Chai, Himabindu Lakkaraju, and Haoyi Xiong. \"M4: A Unified XAI Benchmark for Faithfulness Evaluation of Feature Attribution Methods across Metrics, Modalities, and Models.\" (2023).\n\n[5] DeYoung, Jay, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. \"ERASER: A benchmark to evaluate rationalized NLP models.\" arXiv preprint arXiv:1911.03429 (2019).\n\n[6] Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. 2023. Automatic Evaluation of Attribution by Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4615–4635, Singapore. Association for Computational Linguistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)