DePass: Unified Feature Attributing by Simple Decomposed Forward Pass
Authors: Xiangyu Hong
Che Jiang
Kai Tian
Biqing Qi
Youbang Sun
Ning Ding
Bowen Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Attribution
Mechanistic Interpretability
Generative Models
Large Language Models
Primary area: deep_learning

Abstract: Attributing the behavior of Transformer models to internal computations is a central challenge in mechanistic interpretability. We introduce DePass, a unified framework for feature attribution based on a single decomposed forward pass. DePass decomposes hidden states into customized additive components, then propagates them with attention scores and MLP's activations fixed. It achieves faithful, fine-grained attribution without requiring auxiliary training. We validate DePass across token-level, model component-level, and subspace-level attribution tasks, demonstrating its effectiveness and fidelity. Our experiments highlight its potential to attribute information flow between arbitrary components of a Transformer model. We hope DePass serves as a foundational tool for broader applications in interpretability.

DePass: Unified Feature Attributing by Simple Decomposed Forward Pass
Xiangyu Hong' Che Jiang" Kai Tian', Biqing Qi', Youbang Sun', Ning Ding! Bowen Zhou 1.2
Department of Electronic Engineering, Tsinghua University
Shanghai Al Laboratory
hong Long-xy22dmails.. tsingnuu........ edu. en jc23@mails ls wwwnsghu..edu...o
zhouboren@tsinghua. edu. cn
Abstract
Attributing the behavior of Transformer models to internal computations is central challenge in mechanistic interpretabiliiy. We introduce DePass, a uni- fied framework for feature attribution based on single decomposed forward pass. DePass decomposes hidden states into customized additive components, then propagates them with attention scores and MLP's activations fixed. It achieves faithful, fine- ane-ggrined attribution without requiring auxiliary training. We validate DePass across token-level, model component-level. and subspace- level attribution tasks, demonstrating its effectiveness and fidelity. Our exper- iments highlight its potential to attribute information flow between arbitrary components of Transformer model. We hope DePass serves as a founda- tional tool for broader applications in interpretabilityy Code available https: /ttt:////ttub.... http:///wssssiaaaa/////asaeseeo. Powp://aacooposd--oroaaaraaaaa Pors
1 Introduction
Mechanistic interpretability is the foundation monitor, modify, and predict Transformer- models' behavior. The first step of such reverse engineering is to decompose the neural network and analyze what contributes the model's behavior [1] []. Researchers continue develop methods for decomposing this highly complex system. Without modifying abstracting the network, activation patching [57 all modules is computationally expensive and provides limited insight into intermediate information flow X Gradient baadie attribution methods also face theoretical challenges (.. In contrast, approximating or abstracting the model can partially align with human cognition [0033) but often fails to reach fine-grained components such as neurons or attention heads. Moreover, non-conservative approximations may compromise the faithfulness of attribution. In this paper, propose DePass (Decomposed Forward Pass), a direct and unified feature attribution framework that addresses these shortcomings through a single decomposed forward pass.
directly applying noise ablations 3
The main concept of DePass is simple: we break down every hidden state into additive components, propagate these components through the remaining layers, and then obtain each component's 's exact contribution to the target representation. In the decomposed forward pass, attention scores and MLP activations are fixed, and weighted contributions are assigned based on the decomposed components. This method guarantees several advantages:
Faithfulness and Completeness: By freezing attention scores and MLP activations, DePass elimi- nates second-order effects when propagating forward [8], As result, per-component summation reconstructs exactly the hidden state of the original model.
'Equal contribution. Cooresppnding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Unification Across Components: DePass provides a unified attribution framework for multiple model cettuulariiie-inncuueee input tokens, attention heads, neurons, and subspaces of the residual stream. This enables consistent, fine-grained interpretability without modifying the model or using task-specific approximations.
Representation-LLewe Attribution: Unlike conventional attribution scores M1 DePass tracks how decomposed representations evolve through the forward pass. This facilitates more fine-grained attributions and a natural interface to align with human reasoning and sparse dictionary learning (SDL) such as SAE.
We conduct attribution experiments at the token level, model component level, and subspace level to evaluate the effectiveness and generality of DePass. Our results show that DePass enables lossless, additive decomposition of hidden states throughout the forward pass of Transformer models, allowing faithful tracking of information flow according attribution needs. This unified framework offers new analytical tool for mechanistic interpretability.
Architecture of Transformer-Based Autoregressive Language Models
Transformer-based autoregressive language models use the residual stream as the backbone for information processing. In DePass, the decomposed components follow the same pathway as the original hidden states, passing through each decoder layer's multi-head self-attention (MHSA) sublayer, feedforward network (MLP) sublayer, and layer normalization (LayerNorm) operations. In this section, establish the notations used in the standard forward pass, laying the groundwork for the formal definition of the DePass method.
2.1 Transformer Decoder Layers
Let X E RN denote the token representations at layer i, where N is the sequence length and D the dimension of model's hidden state.
Multi-Heat Self-Attention (MHSA) The multi-head self-attention (MHSA) mechanism enables each token to attend to others in the sequence, capturing contextual dependencies, At layer L, the input x NDD is first normalized by layer normalization, X LN(X*-1).
For each head 1, ... H, queries, keys, and values are computed by linear projections of X and attention scores are obtained via scaled dot-product attention with a causal mask M
(KEST
A Softmax
M
(1)
VD/H
Letting who = wh'we DxD the output of all attention heads is:
H
Output aiin
Σ Oes, where O WWW
(2)
5=1
Finally, the residual connection applied:
x X + Output
Feedforward Network (MLP) After the attention sublayer, hidden states are passed through position-wise feedforward network to enhance per-token representation capacity. In particular, LayerNorm is applied to the attention output, and then the MLP operates independently at each position
Output Wh a (WD LN(X
where W E DgdmxD, w.f Dxdm, and a(-) is nonlinear activation function. MLPs can be regarded as soft key-value retrieval mechanism [4] Let
W
R NXD
WH.
MV
v DxN
(5)

---

where f E RD and v E R represent the k-th subkey and subvalue. The output at position is:
N
Output Σ
m v
m
(Th Xmmns)
(6)
where aim
LN att..) acts as query vector over the subkeys. Finally, the residual connection yields the output for the next layer:
X x++
X + Outputin
2.2 Language Modeling Head
At position i, the final hidden state X E RRP normalized and projected by language modeling head Wim E RVXD, where V the vocabulary size. The resulting logits yield the next-token distribution:
P(w++1 (gge) Softmax (Wmm LN(X,)))
(8)
where w/+1 is the predicted token at the next position..
3 Decomposed Forward Pass
We introduce DePass (Decomposed Forward Pass), a method for isolating and tracing representational components through Transformer layers. This section outlines: (1) the initialization of decomposed hidden states; (2) the reformulated forward pass for propagating these components; and (3) the use of decomposed states for decoding and attribution. Throughout this paper, we use the notation X X ... to index into multidimensional tensors following the NumPy-style indexing convention.
3.1 Initialization of Decomposed Hidden States
Given hidden state (() E NND at layer L, with N tokens and hidden dimension D, we define its decomposed form (() der. E MNMM where M is the number of decomposition components. Each token representation the sum of its components:
M
x d L,,
The index m is shared across positions to ensure semantic alignment, and its interpretation depends on spec-ic decomposition strategies.
Propagating Decomposed Hidden States Through Transformer Blocks
We describe how decomposed hidden states propagate through LayerNorm, Multi-Head Self-Attention (MHSA), and MLP modules in a Transformer block.
LayerNorm Since most layer normalization performs feature-win scaling and shifting, we illustrate here how the more efficient RMSNorm-usse used in our experimental models opeeraess on decomposed M
hidden states. Given token i's hidden state x(( Im=1
x(0 [i, m, :]. RMSNorm computes:
(
where RM)( v |
(MSNorm(X!"))
(10)
RMS(X!"))'
The scaling operation in RMSNorm is additive over decomposed components, allowing the scaling factor to applied independently to each component, so distributes over components as:
x() li m. = x (2) m. where A, = diag
RMS(X.(0)
(11)

---

Multi-Head Self-Attention (MHSA) Decomposed Hidden States Let x((-1) E R represent the decomposed hidden states from the previous layer. To apply multi-head self-attention in component-awaid manner, extend the standard attention formulation (see Eq. 2 to operate on each component individually. For each attention head j. the attention output for token from component m is:
((,)
(1-d) ( (il--)
dec (i, mi, W
(12)
2,m
where (C)) is the original attention scores. The MHSA output is aggregated across all heads and components, combined with residuals:
x(O) attn deo (i, m, = x [2, m, :
(4,)
(13)
m.
/=1
MLP on Decomposed Hidden States The MLP module processes decomposed representations x() det RNNMXD by distributing each neuron's output across components. For each token i, component m, and neuron k. compute relevance score:
X (8) dor 1ú, m.:
exp(ai,m..)
Oi,mm
(14)
2m'=1 exp (xp(as.m'.))
where Oi,m..k determines how neuron k's output is apportioned to component m. Alternative normal- ization methods to softmax are compared in Appendix A
The updated decomposed hidden state is then:
Date
x x((+1)
m,
x-()
aoen 11 m. :]
Σ
ai,mk
m -
(15)
where mf, is the neuron activation and v E RDD is the output projection. The original hidden state recovered by summing over components:
M
x (++1)
Σ
(x++)) dec
(i, m.
(16)
Given the decomposed hidden states from the previous layer, each component can be further prop- agated through the next layer independently. This maintains disentangled contributions across components and allows exact reconstruction of the original hidden state. All computations are fully parallelizable, incurring minimal overhead.
3.3
LM Head and Attribution Score
Let x! Σ m=1 x(L) [i, m, | be the final hidden state token and let w, = WiMly, :] be the LM head vector for target y. The output logit and component-wise attribution
logits, = w,,,
Alogits w.J x (i, m.
(17)
with ΣM Alogits y.m logits,. This yields tne-grained attribution across components. To
attribute with respect to given subspace at particular layer, we project each x(0 [i, m, :] onto its direction and use the projection values as scores.
4 DePass Attribution across multi-granular Levels
4.1 Token- Wise DePass
Token-Wire Decomposed Hidden States Initialization To analyze how each input token con- tributes to the model's hidden states, we initialize a token-wise decomposition of the hidden states: each token's hidden state is split into N additive components, one per input token. At the embedding layer, the decomposition is defined as:
[r. if otherwise
x-1) [m;
(18)
0,

---

where X E IR- is the input embedding and X E RRNxNx is its token-level decomposition.
This structure is propagated layer by layer according to the method in Section 3.2 At any layer, the m-th component of the decomposed hidden state represents the contribution of the m-th input token the overall hidden states.
4.1.1 Token-Level Output Attribution via DePass
Given a model M, input = attribution assigns each token z; a score s; E ER indicating its influence on g with higher scores denoting greater impact.
Problem Definition.
=I [TI,..., Fnl, and output ý = .Mrr,, token-wise
DePass-BBsse Output Attribution. Starting from coken-wise decomposition initialization (Eq. (8) and applying the forward pass decomposition (Section 3.2). we use the language modeling head (Section 3.3) on the final decomposed hidden states. This yields token-level attribution scores, quantifying how each input token contributes to the model's prediction of output y-
Experiment Setup. Baselines. We compare against standard attribution methods on fixed pretrained model. Gradient-based: Input Gradient (I5), Integrated Gradients [16]. Gradient SHAP [77: Attention-based: Mean Attention, Last-layer Attention [88. and Attention Rollout [6]
Tasks. We evaluate on two benchmarks targeting different reasoning types: Known_1000 1000 faccuall QA, e.g.. "Audible.com is owned by") and IOI [99 (indirect object identification, e.g.. "Eleanor and Deanna were thinking about going to the mountain. Eleanor wanted to give a watermelon to").
first compute attribution scores for the correct answer via various methods. Based on these scores, we apply token-level interventions: patch top-mant the top K% tokens with highest attribution; recover top mask the bottom (100-K)6 tokens (lowest attribution) and then restore the top K%.
Evaluation Protocol. For each input
The remaining tokens are reassembled into new prompt and fed to the model. Faithfulness is then evaluated by measuring the change in the predicted probability of the correct answer: Comprehen- siveness: Drop in probability under patch top (higher is better); Sufficiency: Retained probability under recover top (lower is better).
We compute the relative change in predicted probability as:
p(
x)) p(ū (())
Ap! [K)
P(ù
where ý denotes the target token and ((()) denotes the perturbed input under each intervention. The figures report the average Ap( across all data points per dataset at each masking level.
To ensure fair comparison with other token-level baselines, in the experiments corresponding to Figure M. we perform ranking and ablation at the subword token level. DePass also supports word-level attribution, with examples illustrated in Appendix
Results. As shown in Figure I our method yields substantially higher comprehensiveness scores 1000 and IOI benchmarks using 1aama-2-13b-cha-hhe indicating that the tokens identified by DePass are more critical to the model's 's prediction.
on both Known 1000
In terms of sufficiency, all methods comparable. This is partly due to evaluation limits: when keeping only top tokens, most methods recover those key for prediction. Attention- based methods, which preserve broad semantic structure, perform slightly better. representative of output attribution is shown in the top of Figure 2
Additional results other model variants and experiment details are provided in Appendix B
4.1.2 Token-level Subspace Attribution via DePass
While methods such as probing or sparse autoencoders (SAEs) are able to uncover meaningful subspaces, they fall short of directly identifying which input tokens are responsible for activating
Dataset can be found at: httpps:/romee bwulab.
5

---

Disrupt Taken
Recover
Lamn---1.-b-chee Sbbih------------ii on trenn 1200
lemwdd-1f--hhie knowan 1000
Sh-ch-hl--bchhhttl on
1ema-3-1 2twph-------hhhhiii- on
Aectagg relcive changeipe
100
100
92
50
60
60
70
40
Megrited
leguites
100g
Megrieed
821
sgrcc
saysol
Sigeol
sgrce
nam
50
norm
rorm
0.4
0.6
1.0
1.0
0.2
0,6
088
0.4
0.0
1.0
Disrup Percent
Recover Percent
Dinnutt Percent
Recover Percent
Figure 1: Faithfulness evaluation Known 1000 and IOI using yields better comprehensiveness and competitive sufficiency.
Our method
those subspaces. Existing approaches that attempt to link tokens with feature activations 20 often depend on intricate graph construction and pruning, making them computationally expensive and difficult to scale. To address this limitation, we introduce DePass as framework that attributes subspace activations to specific input tokens in a more direct and efficient manner.
DePass-Basse Baass Subspace Attribution. Given semantic direction v E IR and decomposed hidden states X E NxNNX at layer f, the subspace activation at position i is:
= v Σ
dec m,,
(19)
where each term quantifies the contribution token zm the subspace activation at position i. This enables fine-grained attribution to arbitrary linear subspaces.
Experiment: Probing Subspace Attribution via DePass We evaluate our method in the context of factuality, where prior work has shown that truthfulness can be linearly separated in hidden states (21). A linear probe f fzx) w + is trained to detect untruthful activations, with defining factuality subspace. Some methods leverage such probes to mask misleading input tokens based on whether their intermediate hidden states align with untruthful directions [22]. In contrast, our method attributes subspace activations directly to specific input tokens, enabling more fine-grained interventions such selectively masking misleading content.
Evaluation Setup. We evaluate on two factuality benchmarks: CounterFact 4 (modified for generation, prompting completions to factual statements) and TruthfulQA (23) (converted to multiple- choice with misleading options).
We compare the effectiveness of DePass and TACS in identifying misleading tokens that con- tribute model errors. Each example is tested under four settings: (1) No Information: given only the question. (2) Misinformation: given wrong information along with the question, (3) Mis- information + TACS Masking (22): tokens aligned with untruthful directions are masked, and (4) Misinformation + Ours (DePass Masking): tokens contributing most to untruthful activation are masked. For each prompt, both methods mask the same number of tokens based on classifier confidence.
Results. As shown in Table 1 DePass-based masking consistently improves factual accuracy across models and datasets. Compared to direct probing-bart based masking, our method yields stronger gains. For example, Llama-2-7b-chat--h. accuracy rises from 10.16% (misinformation) to 43. 13% (ours), demonstrating DePass's 's ability to isolate and suppress harmful inputs. A representative case of subspace attribution is shown in the bottom of Figure
See Appendix for for details on dataset formatting, probe training, prompt examples, and our strategy for selecting masked tokens based on classifier outputs from different layers. We also provide examples of applying DePass for SAE feature attribution illustrate the potential of combining the two methods (Appendix C.5

---

Table 1: Accuracy (%) across different input settings on CounterFact and TruthfulQA. Masking is applied to 30% of misleading tokens identified using either prior methods or our DePass based attribution.
Model
Dataset
No Info
Misinformation
TACS Masking
(DePass Masking)
CounterFact (Gen)
57.52
10.16
25.68
43.13
Ilama 2-7h-chat-hf
TruthuuQA (MC)
66.10
33.05
43.57
46.51
CounterFact (Gen)
Llama-2-13bchhate 2-13b-chatthf
TruihfulQA (MC)
73.56
38.92
49.33
53,00
Llama 3.1 1-1-8b-Insttuct
CounterFact (Gen)
60.63
3.30
16.03
59.16
TruhfuuQA (MC)
83.60
70.26
71.48
76.62
Qwen2 1.5B Instruct
CounterFact (Gen)
44.04
3,51
17.75
43.32
TruthfulQA (MC)
90
Qwen2-7h In--trrctttt
CounterFact (Gen)
39.43
6,54
23.91
29.68
TruthfulQA (MC)
77.23
47.25
68.42
64.87
Meta-I 3.-a--mmmmmm---m Insteet
CounterFact (Gen)
72.20
33.29
55.62
TruthfulQA (MC)
89.96
65.61
74,54
76.87
Attribution Scores for Model Output: "Germany"
The
artist
lives
in
Berlin
the capital
of
Average Autribution Scores Across Layers for Truthful Classifier
<S> 129
The 2.00
_artist
lives
.in
_Berlin
_the
_capital 8.25
of 0.38
France 3.13
0.79
..899
0.64
-202
0.29
.0.55
Figure 2: Two input-level attribution examples. The top case illustrates contributions the model prediction "Germany". with higher scores indicating greater influence. The bottom case shows token-wise contributions to the classifier's prediction the "truthful" label, where more negative scores support the untruthful classification (label 0).
4.2
Model Component- Wise DePass
Model Component- Wise Decomposed Hidden States Initialization To attribute model behavior to specific architectural components, we perform component-level decomposition of hidden states, targeting attention heads and MLP neurons.
Attention Head: For Transformer layer with H attention heads, assign decomposition component to each head and one to the residual connection, resulting in M = + 1. For token i, define:
decl h. = o(.h))
(e,h)
h h= H;
x H 11 ( (e.
(20)
where o, (2,h) is the output of the h-th attention head at layer l.
MIP Neurons: Similarly, for an MLP block with NMLP hidden neurons, we decompose the output and residual connection into NMLP + components:
x dec [i, n m v.. dec [i, NMLP + 1.:] = x (
where mf D is the contribution of the n-th MLP neuron to the hidden states of token i.
This setup aligns with our decomposition-basee forward pass and enables localized interpretability. Component-win wise importance scores for given output are computed described in $33
Experiment: Evaluating Component Importance via Masking. We validate the functional significance of components using masking-based ablations guided by various importance metrics.
& norm of each component's is activation; Coef: Absolute activations after MLP up projectionn and nonlinearity; used gradient-based importance measure using first-order Taylor approximation: DePass (Ours): Attribution scores derived from decomposed hidden states; DePass-Abs: Absolute values of DePass scores, capturing both supportive and suppressive contributions.
Importance Scoring Methods. We compare several scoring strategies: Norm:
only for ML. neurons. AtP (24 (Activation x Gradient):

---

Experiments are conducted on: 101 (Indirect Object Identification): Synthetic reasoning benchmark; CounterFact (QA): Factual knowledge recall task. Only correctly answered examples are used for meaningful attribution.
Tasks and Setup.
Evaluation Protocol. We perform two complementary masking interventions: Top-k Masking (Comprehensiveness): Mask the top-k components, with a sharp accuracy drop indicating critical components, Bottom-k Masking (Sufficiencyy) Mask only the bottom-k components; high accuracy retention suggests sufficiency for prediction.
Masking is applied structurally; attention heads are ablated by zeroing their output projections W:(ch) and MLP neurons by zeroing their activations before projection. We report average accuracy across examples and for a robust attribution quality assessment.
vo
additional results for other models are provided in the Appendix() Our method consistently outperforms baseline attribution techniques across different masking strategies. In particular, achieves a more pronounced accuracy drop in Top-k Masking and better accuracy retention in Bottom-k Masking, indicating its superior ability to identify and attribute critical components.
Results Figure 3 presents the results on
Madet lana 2 chel-a
Musil Sanio Iestar
Macel (et 12 cran
Defict
10-1
12-1 Hast Wivsiigg Houser
Focentage
Mesting hears
Dclicin Pradhie Fosts
A00. 700
Figure 3: Performance of our method compared to baseline attribution techniques on Llama-2-77 2-7-hh-- hf under various masking strategies. Our approach more accurately identifies critical components, as reflected by the sharper drop in Top-k Masking and stronger performance in Bottom-k Masking.
4.3 Subspace-Level Decomposition
Subspace- Wise Decomposition of Hidden States To analyze how specific representational di- rections affect model behavior, we decompose hidden states at layer E into components within and orthogonal to target subspace. Given projection matrix P. E DDD, each token's 's hidden state (0, [ :] e IRd split as:
x 0. (,X((),)) [1,), x [i, 1, =(1 (I.),)
(21)
where I is the identity matrix. These components are then independently propagated using DePass (Section 3.2), allowing attribution of model behavior to the subspace. Construction details of P, are provided in Appendix E.2
Experiment: Interpreting Language Subspace Effeett We examine whether the decomposition framework can reveal functionally distinct subspaces in multilingual settings. Specifically, hypothesize that model hidden states can be separated into a language subspace capturing language signals, and semantic subspace encoding aanguage-invarian meaning [5]
Subspace Construction. We apply DePass Llama-3. 3.1-BBBBBBBBBBB-. Instructt, a multilingual model trained on English, French, German, and more. We train a token-level language classifier on CounterFact [. translated into multiple languages, to detect the language of each input token. The classifier weights define lunguage-diagnosee subspace. Token hidden states are projected into these two subspaces as described in Section 4.3 and decomposition is propagated forward through the model.
subspuce, and its orthogonal complement serves as the semantic
8

---

Evaluation Setup. To evaluate whether DePass faithfully attributes model behavior to distinct subspaces, we begin by projecting hidden states at an intermediate layer (layer 15) into the language and semantic subspaces. These decomposed components are then independently propagated through the remaining layers of the model, yielding two final representations the last layer: Xang
dec [ :] and X E
X dec i 1,:]-
We decode these representations separately by applying the language modeling head to each of them. Table 2 reports the top-5 tokens generated from Xing and X for multilingual prompts (e.g.. "What Thomas Joannes Stieltjes's 's native language? It is"..
Results. Tokens generated from the semantic subspace (X scm) consistently reflect factual content (e.g., "Dutch", "Holland"), largely invariant to the input language. In contrast. the language subspace (Xlang produces lexical or stylistic tokens (c.g.. "né", "nicht", "de") aligned with the prompt's language. These results show that DePass faithfully propagates and preserves the functional roles of each subspace, enabling clear attribution of linguistic and semantic behavior and highlighting its potential for subspace-level analysis.
To further verify that DePass faithfully attributes language-related behavior, we apply t-SNE to X from multilingual inputs. As shown in Figure the resulting clusters align with language identity, confirming that DePass effectively preserves and propagates language-specifie signals. These results highlight its effectiveness for subspace-level attribution. Additional examples are in Appendix E
TSNE visualization of language embeddings
Language
Language Tokens
TSNE dimension 2
40-
English
a, the, an, not, N né, consid, de, conn, ét nicht, keine, eine, die, das
20-
French
German
french German
Language
Semantic Tokens
Engi sh
English
Dutch, dut, reported, said, Afrika Dutch, holland, Holland, of, N Dutch, Holl, Hol, Holland, also
-80
-60 -00
-50 0
20
60
80
French
TSNE cimension
German
Figure 4: t-SNE visualization of token- wise projections onto the language subspace. Distinct clusters indicate strong language- specific structure.
Table 2: Top-5 tokens decoded from the language and semantic c subspaces for different multilingual prompts.
5 Related Work
Decomposition--assed Attribution. Without modifying or approximating Transformer modules, decomposing hidden states can partially explain model behavior [27. Decomposition of MLPs and attention blocks at the logit level enables attribution of their residual contributions to model's output Additionally, attention scores have been used to construct information flow graphs that trace Combining decomposition with gradient information can further DecompX K applies similar decomposition ideas in restricted settings; our approach generalizes this to arbitrary modules in generative Transformers with finer mnernerained decomposition of MLP activations. Compared to these methods, DePass performs a more direct additive decomposition of the computation, avoiding early mapping to scalar saliency scores and thereby preserving both faithfulness and interpretability.
BT
input tokens" influence [1 32)
highlight salient features 15
17 16
33
Ablation-based Attribution. Another line of work attributes importance by ablating specific compo- nents such adding noise or zeroing out input tokens or internal activations and measuring the resulting change in the output distribution 4 B 37]. Since exhaustively ablating all components incurs high computational cost, approximate methods or surrogate models have been proposed to 39] In contrast, DePass provides more faithful attribution
accelerate activation patching [38 12
through direct decomposition, avoiding the indirectness and potential artifacts of ablation-basee approaches.

---

Sparse-Dictionary-Leennee based Attribution. Sparse Dictionary Learning (SDL) is currently the most popular decomposition methods in interpretability. Based on the superposition hypothesis
40 A1, SDL aims to recover more interpretable components than the original feature dimensionality.
SAE-based methods supervise the reconstruction of current activations [42 45], while Transcoder
20 targets the reconstruction of next-layer activations, and CrossCoder[4A [6 extends this to jointly
reconstruct activations across multiple layers. However, the high training cost [. annotation effort, and reconstruction errors [47] limit their scalability. Some approaches aim track feature evolution during forward propagation (4355), but they often rely on proxies like inter-feature correlation or similarity scores rather than precisely tracing the actual transformations of features. Nevertheless, SAE remains valuable for improving downstream task performance (such as model steering [51]) and can be used effectively in conjunction with DePass to bridge model representations and human- interpretable concepts.
Conclusions
In this paper, we present DePass, a simple yet efficient framework for interpreting Transformer models via decomposed forward pass. By freezing and allocate attention scores and MLP activations, DePass enables lossless additive decomposition, and can be applied to any Transformer-basee architecture. DePass achieves more faithful attribution across different levels of granularity comparing to other methods. We hope DePass serves as a general-purpos tool for mechanistic interpretability and inspires broader adoption and diverse applications across the community.
7 Acknowledgements
This work is supported by the National Science and Technology Major Project (2023ZD0121403) Young Elite Scientists Sponsorship Program by CAST (2023QNRC001), National Natural Science Foundation of China (No.6240616) 15), Shanghai Municipal Science and Technology Major Project, and the Beijing Natural Science Foundation Undergraduate "Qiyan Research Program" (No. QY24259). We extend our gratitude to the anonymous reviewers for their insightful feedback, which has greatly contributed to the improvement of this paper.
References
[1] L. Sharkey, B. Chughtai, J. Batson, Lindsey, J. Wu, L. Bushnaq, N. Goldowsky-Dill, Heimer- sheim, A. Ortega, J. Bloom et al., "Open problems in mechanistic interpretability." arXiv preprint arXiv:2501. 16496, 2025.
[2] J. Ferrando, G. Sarti, A. Bisazza, and M. R. Costa-Jussà, "A primer on the inner workings of transformer- based language models," arXiv preprint arXiv:2405. 00208, 2024.
[3] M. R. G. Madani, A. P. Gema, G. Sarti, Y. Zhao Minervini, and A. Passerini, "Noiser: Bounded input perturbations for attributing large language models," arXiv preprint arXiv:2504.0291 2025.
[4] K. Meng, D. Bau, A. Andonian, and Y. Belinkov, "Locating and editing factual associations in GPT," Advances Neural Information Processing Systems, vol. 35, 2022.
[5] H. Chen, C. Vondrick, and C. Mao, "Selfie: Self-interpretation of large language model embeddings." International Conference on Machine Learning. PMLR, 2024, pp. 7373- 7388.
[6] S. Abnar and W. Zuidema, "Quantifying attention flow in transformers," in Proceedings the 58th Anmual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Online: Association for Computational Linguistics, Jul.
2020, pp. 4190-4197. [Online]. Available:
[7] Y. Zhang. S. Li, Liu, P. Yu, Y. R. Fung, J. Li, M. Li, and H. Ji, "Knowledge overshadowing causes amalgamated hallucination in large language models," arXiv preprint arXiv:2407.08033, 2024.
10

---

[8] A. Makelov, G. Lange. A. Geiger, and N. Nanda, "Is this the subspace you are looking for? an interpretability illusion for subspace activation patching," " in The Twelfth International Conference Learning Representations, 2024.
[9] B. Bilodeau, N. Jaques, P. W. Koh, and B. Kim, "Impossibility theorems for feature attribution," Proceedings of the National Academy of Sciences, vol. 121, no. 2. p. e23044061 20, 2024.
[10] S. Yang. S. Huang, W. Zou, J. Zhang, X. Dai, and J. Chen, "Local interpretation of transformer based on linear decomposition." " in Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2023, pp. 20270-10 287.
[11] J. Ferrando, G. I. Gállego, and M. R. Costa-jussä, "Measuring the mixing of contextual information in the transformer," in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y. Goldberg, Z. Kozareva, and Y. Zhang, Eds. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 8698 87 14. [Online]. Available: httpss//acbaatooooogggg/2222...
main.593/
[12] B. Cohen- Cohen-Wang, H. Shah, K. Georgiev, and A. Madry, "Contextcite: Attributing model genera- tion to context," Advances in Neural Information Processing Systems, vol. 37, pp. 95 764-95 807, 2024.
[13] E. Ameisen, J. Lindsey, A. Pearce, W. Gurnee, N. L. Turner, B. Chen, C. Citro, D. Abrahams, S. Carter, B. Hosmer, J. Marcus, M. Sklar, A. Templeton, T. Bricken, C. McDougall, H. Cunningham, T. Henighan, A. Jermyn, Jones, A. Persic, Qi, T. Ben Thompson, S. Zimmerman, K. Rivoire, T. Conerly, C. Olah, and J. Batson, "Circuit tracing: Revealing computational graphs in language models," " Transformer Circuits Thread, 2025. [Online]. Available:
html
[14] M. Geva, R. Schuster, J. Berant, and O. Levy, "Transformer serd-forward layers are key-value memories," " in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 5484-5495.
[15] M. Denil, A. Demiraj. and N. De Freitas, "Extraction of salient sentences from labelled documents," arXiv preprint arXiv: 14Xx::1122.681.. 2014.
[16] M. Sundararajan, A. Taly, and Q. Yan, "Axiomatic attribution for deep networks." in Proceedings of the 34th International Conference on Machine Learning- Volume 70, 2017, pp. 3319-3328.
[17] S. M. Lundberg and S.-L. Lee, "A unified approach to interpreting model predictions," in Neural Information Processing Systems, 2017.
[18] S. Jain, S. Wiegreffe, Y. Pinter, and B. C. Wallace, "Learning to faithfully rationalize by construction," in Annual Meeting of the Association for Compulational Linguistics, 2020.
[19] K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt, "Interpretability in the wild: circuit for indirect object identification in GPT-2 small," in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: http::///eweeoooee.ec//oooo//o//
[20] J. Dunefsky, P. Chlenski, and N. Nanda, "Transcoders find interpretable Ilm feature circuits," arXiv preprint arXiv:2406. 11444, 2024.
[21] K. Li, O. Patel, Viégas. H. Pfister, and M. Wattenberg, "Inference-tine intervention: Eliciting truthful answers from language model," Advances in Neural Information Processing Systems, vol. 36, pp. 41 451 41 530, 2023.
[22] T. Yu, S. Zhang, and Y. Feng. "Truth-aware context selection: large language models being misled by untruthful contexts," in Findings of the Association for Computational Linguistics ACL.2024 2024, pp. 10 808--008884
Mitigating hallucinations of
[23] S. Lin, J. Hilton, and O. Evans, "TruthfulQA: Measuring how models mimic human falsehoods." " in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), S. Muresan, P. Nakov, and A. Villavicencio, Eds. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 321 4-3222. [Online]. Available: http://aalaattoooyyyyg2222222.22

---

[24] A. Syed, C. Rager, and A. Conmy, "Attribution patching outperforms automated circuit discovery," in Proceedings of the 7th BlackboxVLP Workshop: Analyzing and Interpreting Neural Networks for NLP: Y. Belinkov, N. Kim, J. Jumelet, H. Mohehbi, A. Mueller, and H. Chen, Eds. Miami, Florida, US: Association for Computational Linguistics, Nov. 2024, pp- 407-416. [Online]. Available:
[25] A. Conneau, G. Kruszewski, G. Lample, L. Barrault, and M. Baroni, "What you can cram into single $&!#* vector: Probing sentence embeddings for linguistic properties," in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), I. Gurevych and Y. Miyao, Eds. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 1126-2136 [Online]. Available: https///clanthoogg..r/////8
[26] G. Alain and Y. Bengio, "Understanding intermediate layers using linear classifier probes." arXiv preprint 1011:1161.001444,2 20116.
[27] L. Arras, B. Puri, P. Kahardipraja, S. Lapuschkin, and W. Samek, "A close look at decomposition based xai- wai-methods for transformer language models," arXiv preprint arXiv:2502. 15886, 2025.
[28] J. Ferrando, G. I. Gállego, I. Tsiamas, and M. R. Costa-jussà, "Explaining how transformers use context to build predictions," in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1; Long Pupers), 2023, pp. 5486-5513.
[29] M. Geva, A. Caciularu, K. Wang, and Y. Goldberg, "Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space," in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y. Goldberg, Z. Kozareva, and Y. Zhang, Eds. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 30-45. [Online]. Available: wwwp:org//2222mmmemeee main.3/
[30] Ferrando and E. Voita, "Information flow routes: Automatically interpreting language models at scale," in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024, pp. 17 77742717444 445.
[31] R. Achtibat, S. M. V. Hatefi, M. Dreyer, A. Jain, T. Wiegand, S. Lapuschkin, and W. Samek, "Attnlrp: Attention-aware layer-wise relevance propagation for transformers," in Forty-first International Conference on Machine Learning, 2024.
[32] G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui, "Analyzing feed-forward blocks in transformers through the lens of attention maps," in 12th International Conference on Learning Representations, ICLR 2024, 2024.
[33] A. Ali, T. Schnake, O. Eberie, G. Montavon, K.-R. Müller, and L. Wolf, "Xai for transformers: Better explanations through conservative propagation," in International conference on machine learning. PMLR, 2022, pp. 435-451.
[34] A. Modarressi, M. Fayyaz, F. Aghazadeh, Y. Yaghoobzadeh, and M. T. Pilehvar, "Decompx: Explaining transformers decisions by propagating token decomposition." " Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2023, pp. 2649-2666
[35] H. Mohebbi, W. Zuidema, G. Chrupata, and A. Alishahi, "Quantifying context mixing in transformers." in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, A. Vlachos and I. Augenstein, Eds. Dubrovnik, Croatia: Association for Computational Linguistics, May 2023, pp. -4078-3000--0 [Online]. Available: http:://ccanthoolyy.o////// wtt.:/ooon200...cllmmm....55.5.
[36] A. Syed, C. Rager, and A. Commy, "Attribution patching outperforms automated circuit discov- ery," in Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, 2024, pp. 407-416.
[37] B.-D. Oh and W. Schuler, "Token-wise decomposition of autoregressive language model hidden states for analyzing model predictions." in Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 10 105-10 117.
12

---

[38] J. Kramár, T. Lieberum, R. Shah, and N. Nanda, "Atp": An efficient and scalable method for localizing Ilm behaviour to components." CoRR. 2024.
[39] Y.-S. Chuang, B. Cohen-Wang, S. Shen, Z. Wu, H. Xu, X. V. Lin, J. Glass, S.-W. Li, and W.t. Yih, "Selfcite: Self-supervised alignment for context attribution in large language models," arXiv preprint Mi2i::2522009000 2025.
[40] W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas, "Finding neurons in haystack: Case studies with sparse probing," arXiv preprint arXiv:2305.01611 2023.
[41] N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, Kravec, Z. Hattield-Dodds:
R. Lasenby, D. Drain, C. Chen et al., "Toy models of superposition." arXiv preprint vrXiv2229910652, 2022.
[42] H. Cunningham, A. Ewart, Riggs, R. Huben, and L. Sharkey, "Sparse autoencoders find highly interpretable features in language models," arXiv preprint arXiv:2309.08600) 2023.
[43] L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and
J. Wu, "Scaling and evaluating sparse autoencoders," arXiv preprint arXiv:2406.0409) 2024.
[44] S. Rajamanoharan, A. Conmy, L. Smith, T. Lieberum, V. Varma, J. Kramar, R. Shah, and N. Nanda, "Improving sparse decomposition of language model activations with gated sparse autoencoders," " in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[45] Z. He, W. Shu, X. Ge, Chen, J. Wang, Y. Zhou, F. Liu, Q. Guo, X. Huang, Z. Wu al., "Llama scope: Extracting millions of features from llama-3. 3.1m--3118bb with sparse autoencoders," arXiv preprint arXiv:2410.20526, 2024.
[46] J. Lindsey, A. Templeton, Marcus, T. Conerly, J. Batson, and C. Olah, "Sparse crosscoders for cross-layer features and model diffing," Transformer Circuits Thread, 2024.
[47] J. Engels, L. Riggs, and M. Tegmark. "Decomposing the dark matter of sparse autoencoders," arXiv preprint arXiv:2410. J4670, 2024.
[48] N. Balagansky, I. Maksimov, and D. Gavrilov, "Mechanistic permutability: Match features across layers," arXiv preprint 2xXx:2410.076566 2024.
[49] D. Laptev, N. Balagansky, Y. Aksenov, and D. Gavrilov, "Analyze feature flow to enhance interpretation and steering in language models," arXiv preprint arXiv:2502.03032 2025.
[50] D. Balcells, B. Lerner, M. Oesterle, E. Ucar, and S. Heimersheim, "Evolution of sae features across layers in Ilms," arXiv preprint arXiv:2410.0888 2024.
[51] S. Chalnev, M. Siu, and A. Conmy, "Improving stecring vectors by targeting sparse autoencoder features," arXiv preprint 2xXiv:2411.02/92 2024.
[52] N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen,
T. Conerly et al., "A mathematical framework for transformer circuits," Transformer Circuits Thread, vol. 1, no. 1, p. 12, 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our "DePass Attribution across multi-granular Levels" Section accurately reflect the advantages presented in our introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See "Limitations of DePass" section in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
We do not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See both "DePass Attribution across multi-granular Levels" section and the Appendix.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
See Supplementary Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
sttpiiisssooooiocyyy. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See both "DePass Attribution across multi-granular Levels" section and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We use greedy decoding strategy and our experiments do not include sampling strategy. Our findings account for statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix "Implementation Details"
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in this paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See "Conclusion" and "Limitations" section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the datasets and pre-trained models used in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See Appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Empirical Comparison of Softmax and Alternative Functions for MLP Attribution
To assess the design choice of using softmax for MLP attribution (Eq. compare it against two alternative normalization methods: (1) Linear-norm normalization: subtracting the minimum and dividing by the sum; (2) Linear-weighted decomposition: linearly decomposing the contribution to the original activation value for each element. This comparison assesses whether the softmax rule empirically justified.
report results on the Known_ dataset using the 1lama-2-7b-cia 2-7b-chat-ht model both patch-top and recover-top token removal strategies.
Patch- Top (%)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Softmax
80.8
87.1
94.0
96.6
98.2
98.8
98.8
99.1
99.2
99.1
Linear-norm
64.8
73.6
85.7
93.2
96.6
97.5
98.2
98.8
99.0
99.1
weighe-
59.4
66.8
77.5
86.2
92.4
95.0
96.6
98.4
98.6
99.1
Table 3:
Comprehensiveness (patch-top) results for different MLP attribution methodsn.
Recover-Top (%)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Softmax
97.8
96.9
93.9
89.9
81.6
75.5
68.9
56.8
43.5
0.0
Linear-norm
98.3
97.1
94.8
92.3
87.7
83.55
78.2
68.8
54.9
0.0
weighte
98.3
97.7
96.6
95.3
92.3
90.4
86.7
80.7
71.6
0.0
Table 4: Sufficiency (recover-top) results for different MLP attribution methods.
Overall, the softmax- based attribution consistently outperforms the alternatives across both token removal strategies, supporting its empirical effectiveness despite its heuristic origin.
B More Results and Experiment Details on Token-Level Output Attribution
In this appendix section, we provide additional experimental details and results for token-level output attribution using DePass across multiple model variants. Our goal is to assess the faithfulness of token attributions, measured in terms of comprehensiveness and sufficiency, following established evaluation metrics.
B.1 Tasks and Dataset Details
We evaluate our method on two widely used benchmarks that require distinct types of reasoning:
The Known_ 1000 dataset consists factual question-answering prompts, where each prompt targets known fact (c.g., "Audible.cov is owned by Amazon"). This dataset is designed to probe how factual information is stored and retrieved in the model.
@nown_10000
I0I (Indirect Object Identification) The IOI task [19 involves syntactic reasoning and coreference resolution. Each instance includes sentence involving two named entities and pronoun (e.g.. "Eleanor and Deanna were thinking about going to the mountain. Eleanor wanted to give a watermelon to +"). The model must correctly resolve the pronoun to the indirect object. This task is well-suited for analyzing compositional reasoming capabilities and token interactions.
B.2 Faithfulness Evaluation on More Models
We assess our method by comparing its comprehensiveness and sufficiency scores against standard basclines. Comprehensiveness measures how much the model's confidence drops when top-attributed
21

---

tokens are removed, while sufficiency assesses how much confidence is retained when only the top-attributed tokens are kept.
Denupt Thp Percent Thtan 2-1-2--cab--rr 1000
Thp Percent: Token 20ma-2-7--dhab--- 1000
Denusi Percent Takan famaa--7b-chhhh
Recouer Percent Totan Itmm-2-7--chr-hh
100
90
00
45
20
FoO
Figure 5: Faithfulness evaluation using Jlama---7b-chat-hf. DePass achieves sharper drop in comprehensiveness. and higher sufficiency retention, indicating better identification of faithful tokens.
Dirrupt Top Porcent Taken
Recover Top Percent Taken
Dbrupt Top Percent Taken
Recover Top Percent Tiken
rnlast
Imog Wid
Inspaped
20-
DeNKS5
DePoos
D4P955
3.0
22
3.5
0.5
11.2
3.0
0.6
Figure 6: Faithfulness evaluation on Llama-3.2 I.2-IBBIsttt Despite its small scale, DePass effectively captures key tokens contributing the output, validating its robustness across model sizes.
Dbrust Purcart Token Qwarc--BB instrrdd knowm, 2000
Recover TAp Parcert, Takan Quard-110 wostuct known,
Danget Pertant Token Cown2-778-tetmut on in
Recover Parcett Takkn Daen2-7B--beemure on
21
lnst
alogrvedd
Iniegnatal
Iringatadd
signod
ap1od
sigred
30
sgred
rar
rem
600
0.8
100
02
0.4
0.5
2.0
02
04
Figure 7: Evaluation 1iren-2-B-Inntruuue across both tasks. DePass consistently outperforms other attribution techniques in both comprehensiveness and sufficiency, demonstrating generalizability across architectures.
Dbruct Porcort Kilen Oward 1. Iratnod kersen 1000
Recower Top Parcent Taken Ceens------ttauuu on known,1000 1000
Dirrpt ko Percenc Biken Gwan7-1 30-imatruct en
Recover To Percer:: Taken Ower7-1 38-tratruc
60.5
00.0
65.0
kot
63.5
10
Idtgrated sgrer
Integittee
Véegrites
stred
Sgret
sgeed
norm
term
17.5
rerm
9.4
D/a
US8
15
0.2
0%6
11.8
2.6
300
122
04
U5
Figure 8: Evaluation on 2..en-221.BBBInstrree across both tasks.
22

---

Distuct Peccet Xoen
Recover Percent Taken
Dirrpt Percant Taken
Recover Percent
Irmra-322 known_1000
1amm--322
3.2-3
05.2
00.2
00
Inograted
02.1
rofout (taggand
sclett Fiagreod
72.5
DePass
Cofasss
75.0
DePacl
DePaas
Uhtrat Hoccett
Nicever
Figure 9: Evaluation on Llama-3.2-3B-Inse across both tasks.
These results reinforce the effectiveness of DePass in identifying faithfully contributing input tokens across a diverse set models and tasks.
B.3 More Cases on Output Attribution
We present additional examples illustrating how DePass assigns output-dependent importance scores to input tokens. These cases highlight DePass's ability to differentiate which parts of the input are most responsible for different model outputs, even under sampling variability.
Attribution Scores
Model Output:
"Amy"
Tom 4.40
Arm
went
store 0.51
bag 0:41
0.25
3.93
0.46
0.44
3:67
0:36
1.37
0:34
0:36
Attribution Scores
Model Output: "a"
<>>>
Tom
and
Amy
went
to
the
store
Tom
handed
a
bag
to
0.49
0.80
0.70
0.71
0.0
0:94
0.00
2:10
6:72
2.00
0.74
Figure 10: Token-wise output attribution scores from DePass for the prompt "Tom and Amy went the store. Tom handed bug to". For two sampled sutputs-"Amyy and "D"ass produces distinct attribution patterns, correctly identifying which input tokens support each specific continuation.
Attribution Scores for Model Output: "Paris"


The
capital of
France 5.10
-0.30
1.19
8.94
1.02
1.19
Attribution Scores for Model Output:
"known"


The 1.91
capital 5.98
France
0.09
1.42
181
1.41
Figure 11: Token-wise output attribution scores for the prompt "The capital of France is". When the model produces "Paris" " versus "known" as outputs, DePass assigns higher relevance to different parts of the prompt accordingly. This shows DePass's 's sensitivity to output semantics in attributing input importance.
Illustration for Multi-Token Words' Attribution We provide two examples to illustrate how subword tokenization affects DePass scores. For word-level scores, corresponding subword tokens treated single component from the start of the DePass process, rather than simple of individual scores.
Example 1:
23

---

Token


Catal
onia
belongs
to
the
continent
of
Score
0.28
6.16
1.27
1.75
1.16
1.15
13.06
1.36
Word
<S>
Catalonia
belongs
to
the continent
of
Score
0.28
6.38
1.92
1.37
1.38
13.24
1.59
Example
Token


The
mother
tongue
of
Daniel
Dar
ric
UX
is
Score
0.30
0.87
2.09
9.69
1.23
-0.01
1.31
2.00
2.69
1.36
0.73
Word
<S>
The
mother
tongue
of
Danielle
Darrieus
is
Score
0.30
1.12
2.29
9.85
1.52
1.85
4.12
1.18
Both token-level and word-level attributions with DePass are effective. While token-level scores may disperse across subword tokens, they remain highly informative for distinguishing important tokens. Higher scores on specific subwords can even indicate triggers for the model's 's memory of the full word. DePass naturally supports aggregating subword tokens for desired word-level attribution.
Token-Level Attribution Examples for Multiple Attribution Methods The following tables present token-level attribution scores computed by DePass and several baseline methods for illustrative prompts. Each table corresponds to different target token for the same input, highlighting how DePass distributes relevance across input tokens in a context-sensitive manner. Raw DePass scores provided alongside normalized values to facilitate comparison with other attribution methods.
Token DePass
Normalized DePass
All
Last
Rollout
Integrated Gradients
Signed
Norm


-0.30
-0.02
0.30
0.28
0.35
0.00
0.17
0.40
The
1.20
0.07
0.14
0.14
0.13
0.01
0.17
0.14
capital
8.94
0.52
0.14
0.14
0.13
0.84
0.16
0.17
of 1.02
0.06
0.14
0.14
0.13
0.14
0.17
0.14
France
5.06
0.30
0.14
0.15
0.13
0,00
0.16
0.07
is 1.20
0.07
0.14
0.15
0.13
0.01
0.17
0.09
Table Token-level attribution scores for the prompt "The capital of France is" with target "Paris".
Token
DePass
Normalized DePass
All
Last
Rollout
Integrated Gradients
Signed
Norm


0.09
0.01
0.30
0.28
0.35
0.26
0.18
1.00
The
1.91
0.15
0.14
0.14
0.13
0.00
0.08
0.00
capital
6.00
0.48
0.14
0.14
0.13
0,65
0.20
0.00
of
1.41
0.11
0.14
0.14
0.13
0,00
0.16
0.00
France
1.80
0.14
0.14
0.15
0.13
0.00
0.22
0.00
1.41
0.11
0.14
0.15
0.13
0.09
0.17
0.00
Table 6: Token-level attribution scores for the prompt "The capital of France is" with target "known"..

---

Token
DePass
Normalized DePass
All
Last
Rollout
Integrated Gradients
Signed
Norm


0.32
0.02
0.14
0.12
0.17
0.02
0.07
0.88
gra
1.52
0.11
0.07
0.07
0.06
0.00
0.06
0.00
pe
0.08
0.01
0.07
0.07
0.06
0,00
0.10
0.00
0.20
0.01
0.07
0.07
0.06
0.00
0.06
0.11
pur
5.25
0.36
0.07
0.07
0.06
0.00
0.07
0.00
ple
0.14
0,01
0.07
0.07
0.06
0,00
0.08
0.00
0.10
0.01
0.07
0.07
0.06
0.00
0.06
0.00
ban
0.48
0.03
0.07
0.07
0.06
0.00
0.08
0.00
ana
0.12
0.01
0.07
0.07
0.06
0.00
0.05
0.00
0.15
0.01
0.07
0.07
0.06
0.98
0.09
0.00
yellow
1.53
0.11
0.07
0.07
0.06
0.00
0.05
0.00
0.19
0.01
0.07
0.07
0.06
0.00
0.08
0.00
apple
4.16
0.29
0.07
0.07
0.06
0.00
0.08
0.00
0.17
0.01
0.07
0.07
0.06
0.00
0.06
0.00
Table 7: Token-level attribution scores for fruit-color completion task with target "red".
DePass reports both raw and normalized attribution scores, with raw values summing exactly to the output logit. This allows direct interpretation of each score's impact on the prediction. Attribution distributions vary across different targets for the same input, reflecting the model's context-dependent reasoning. DePass consistently assigns attribution to semantically meaningful input tokens, whereas baseline and gradient-based methods often fail to highlight relevant tokens or exhibit clear semantic alignment.
C
More Results and Experiment Details on Token- Wise Subspace Attribution
C.1
Prompt Construction
C.1.1 CounterFact Prompt Construction
The CounterFact dataset provides:
is designed evaluate factual recall in language models. Each instance
a subject entity (e.g., Go Hyeon-jeong),
target (the correct answer, e.g., Korean),
target_new (an incorrect, but plausible alternative, e.g., French),
multiple natural language prompts querying the relation (c.g., "The mother tongue of Go Hyeon-jeong is'").
We construct multiple prompt variants per example simulate different reasoning settings. Specifi- cally, we create three types of prompts:
Initialization Prompt (No Information): A question-only prompt constructed by sampling one of the available templates, without providing supporting information.
Untruthful Prompt: A misleading information context is constructed by inserting the subject along with the incorrect fact (target_nen into a randomly selected prompt template from the provided list. This prompt is then used as external information preceding the same question template, simulating factual error in the input.
Each final prompt is formatted as follows:
According the given information your knowledge, answer the question.
Information: <Inserted Factual or Counterfactual Sentence> (optional)
Question: <Sampled Question Template>
The answer is:
25

---

During construction, we ensure diversity by randomly sampling prompt templates for both the question and the information. For each data point, one question prompt is selected and paired with either the correct or incorrect fact from the candidate list.
C.1.2 TruthfulQA Prompt Construction
The TruthfulQA dataset
contains questions designed probe language models' susceptibility to producing factually incorrect or misleading answers. For each question, it provides a set of factually correct answers sacc__target and mc2_ targets) and pool of plausible but incorrect ones.
23
We construct two types prompts per example:
Initialization Prompt (No Information): question-only multiple-choiee prompt where the model must choose the best answer from randomized mix of one correct and one incorrect option, without access to any supporting context.
Untruthful Prompt: A misleading information-beartin bearing prompt, where randomly sam- pled incorrect statement from the provided _el_aareets or mc2_ ttargets inserted as Information. The model must then answer the same question with the same randomized options, now influenced by the incorrect prior.
Each prompt is formatted as follows:
According to the given information and your knowledge, choose the best choice from the following options. Information: <Random Incorrect Statement> (optional)
Question: <Original Question>
Options: A: <Incorrect Option>
B: <Correct Option> (random
order)
The answer is:
To ensure variation and avoid position bias, we shuffle the order of the correct and incorrect choices. In all cases, the model is required to pick between exactly two answer candidates, enabling us to isolate the effect of misleading information model predictions.
C.2 Truthful Probe Training and Evaluation
Probe Training Following 22,, we train linear probes on hidden states from transformer- language models detect factual consistency. We use the CounterFace and TruthfulQA datasets, each labeled with truthful and untruthful prompts. The datasets split into training and testing sets with balanced labels to ensure fair evaluation.
For each prompt, we extract the hidden states of the final token across all transformer layers. These per-layer activations serve as features for training logistic regression classifiers (one per layer) to distinguish between truthful and untruthful inputs. We use the saga solver in scikit-learm. with learning rate of 0.01 and maximum iteration count of 1000.
During training, we shuffle and concatenate the hidden states of truthful and untruthful samples and fit each layer-specific classifier independently. After training, evaluate the classifiers held-out test set to assess the layer-wise linear separability of factual information.
Probe Evaluation Figure and Figure 14 show the classification accuracy for different Llama and Qwen model variants. We observe a consistent trend across models: classifier accuracy improves with depth, often exceeding 90% in the middle-to-le layers. This suggests that factual signals become increasingly linearly separable in deeper representations.
26

---

Classifier Accuracy Truthful Probing Hima-2-7b-chat-ti
Classifier Accuracy for Truthful Probing
llama-3 3.1-8biinstruct
0.95
Mean Accuracy: 0.8812
Test Accuracy
Test Accuracy
0.85
0.80
0.70
Accuracy:
15
20
25
Classifier Index
Classifier Index
Figure 12: Truthful classifier accuracy on L. and Llama-3. I-ntttttutt.
Classifier Accuracy for Truthful Probing llama-2-13b-chtt-f
Classifier Accuracy for Truthful Probing www.--aaaaaaaa---mmeeee
0.95
Test Accuracy
0.85
Test Accuracy
0.90
0.65
0.80
0.80
0.75
Mean Accuracy: 0.0743
Mean Accuracy: 0.9118
0
15
20
0
10
20
30
40
50
60
Classifier Index
Classifier Index
Figure 13: Truthful classifier accuracy on Llama-2- 1-13-cchattthff and Meta-Llama-3.1-7000 -ntstt- Instruct
Classifier Accuracy for Truthful Probing
Classifier Accuracy for Truthful Probing
Qwen2-1.. 1ann2-1.50--. 1.5---nnsttt
0.90
Mean Accuracy: 0.8478
0.90
0.85
Test Accuracy
Tist Accuracy
0.80
0.00
0.75
0.75
Mean Accuracy:
Classifier Index
Classifier Index
Figure 14: Truthful classifier accuracy on Qwen-2-1.5B -Inttrcct and Awwen22-B-Inntruuct
C.3
Masking Strategy
Token selection for masking. For given prompt, we extract the hidden states of all tokens across all layers. Based on classifier accuracy trends, we consider only classifiers from layer 10 onward, where truthfulness signals are most separable. For each token, we compute its average predicted probability of being untruthful across these layers.
Tokens with a mean probability S. 0.5 are retained. For those with probability > 0.5, we apply two masking strategies:
Direct masking (TACS) 22]: Remove all tokens predicted as untruthful (p 0.5) from the input and re-run the model.
2. DePass-basee masking: For each token identified as untruthful, we decompose its hidden state to attribute contributions from other input tokens. We average the untruthful tokens'
27

---

contribution vectors to compute a global untruthfulness attribution score. Then, we identify the top k contributing tokens in the input (matching the number of untruthful tokens in direct masking) and remove them.
These complementary methods help evaluate whether selectively removing suspected untruthful information can steer model predictions towards higher factuality, while controlling for deletion
magnitude.
C.4 More Cases on Subspace Attribution
We present additional untruthful examples to further illustrate how DePass attributes subspace-level information to input tokens.
Average Attribution Scores Across Layers for the Truthful Classifier


Hum
ans -0.42
have -0.26
_six -1.02
fingers -5.42
0.38
-0.36
Average Attribution Scores Across Layers for the Truthful Classifier


The
Lou
vre
js
located
in
Berlin
-1.25
Figure 15: Additional examples demonstrating DePass's ability to attribute information from specific subspaces back input tokens. Attribution scores are computed with respect the output label (untruthful); more negative scores indicate stronger contributions toward untruthful signal.
C,5
Case Studies: Potential of Combining DePass with SAE
Combining DePass with SAE. While SAEs can uncover meaningful features, they not inherently indicate which input tokens activate those features. DePass provides a complementary capability: by decomposing activations into additive components, flexibly connects different elements of mondll-tokent components, and subspaces with SAE features. This enables fine-grained analysis of how interpretable features are triggered by specific tokens and how they propagate through the model. Below, we present several case studies demonstrating the potential of combining DePass with SAE for token-level attribution. The feature annotations used in these examples are drawn from Neuronpedia and all cases are based -LaMA-3.1-81 with the corresponding SAE (45
Case 1: Climate Change. Prompt: "In exploring the relationship between climate change and urban planning, the research identifies key strategies for sustainable development. underscores the urgency of integrating environmental considerations into city planning."
Feature: Layer 31, Feature 5226 "climate change and its associated impacts"
We analyze three tokens that activate this feature. DePass identifies that the activation of the SAE feature is primarily driven by the semantic contribution of climate (6), which then propagates to other relevant tokens such as change (7) and environmental (27) (Table
htpp:///www.nuurupedieaa
28

---

Token analyzed (index)
Activation value
Source token (index)
Contribution
climate (6)
35,2500
climate (6)
26.8750
In (1)
3.0781
exploring (2)
1.8750
the (3)
1.4844
relationship (4)
1.4844
between (5)
1.4844
change (7)
15.6875
climate (6)
16,5000
relationship (4)
0.8594
between (5)
0.3984
change (7)
0.5586
In (1)
-1.9141
environmental (27)
7.4688
climate (6)
9,4375
exploring (2)
1.0234
environmental (27)
-0,6562
underscores (22)
-0.1670
urgency (24)
-0.2432
integrating (26)
-0.3516
Table 8: DePass token-level attribution of SAE feature 5226 ("climate change and its associated impacts").
Case 2: Simplicity in Context. Prompt: "Avocado toast, a simple yet trendy breakfast option, gains delightful twist with sprinkle of chili flakes and drizzle of honey for that perfect balance of spicy and sweet."
Feature: Layer 31, Feature 26874 - "the context of simplicity in various contexts"
DePass shows that the activation of this feature largely driven by the semantic link from simple (6) to tokens such as yet (7) and punctuation, highlighting how contextual modifiers contribute to SAE features 9
Token analyzed (index)
Activation value
Source token (index)
Contribution
yet (7)
7.4062
simple (6)
5.8438
toast (3)
1.9219
yet (7)
1.1953
(4)
-0.5312
(5)
-0.5898
<bos> (0)
-1.3750
,(11)
3.5312
simple (6)
1.7031
toast (3)
0.8125
yet (7)
0.7344
<bos> (0)
-1.8906
Table 9: DePass token-level attribution of SAE feature 26874 ("simplicity context").
Case 3: Economic Concepts. Prompt: "In economics, supply and demand is model for under- standing how prices and quantities are determined in market system. This concept foundational in economic theory and affects various market structures."
Feature: Layer 25, Feature 9618 "references to economic topics and concepts"
Here, DePass reveals that the activation of the "economic concepts" feature stems from semantic
contributions of economics (2) that propagate to economic (28) and and (30) (Table I0).
Across these examples, combining DePass with SAE enables precise token-level attribution of interpretable features. Whereas SAE identifies semantically coherent features, DePass traces their origins and propagation across tokens, yielding more complete mechanistic account of model representations.
29

---

Token analyzed (index) economic (28)
Activation value
Source token (index)
Contribution
15.0625
economics (2)
5.9375
economic (28)
4.9062
<bos> (0)
0.9805
(22)
0.1348
and (30)
7.3438
economics (2)
4.7500
economic (28)
1.9844
<bos> (0)
1.1719
concept (24)
-0.1025
In (1)
-0.6680
Table 10: DePass token-level attribution of SAE feature 9618 ("economic topics and concepts").
D
More Results and Experiment Details on Component-Wise DePass
D.1
Additional Model Component-wise Attribution Results Different Models
To further evaluate the generality and robustness of DePass across architectures and scales, present additional results on component-we identify important attention heads and MLP neurons across various model families, including LLaMA-3.2 and Qwen2 at both small and medium scales.
attribution. Specifically, we assess the ability of DePass to
We apply Top-k and Bottom-k: masking strategies: masking the top-k most attributed components should lead notable performance drop, while masking the bottom-k should minimally impact output quality. Across all settings, DePass demonstrates strong alignment between attribution scores and functional importance.
PON IEVUT
Mosk
Dafane
Figure 16: Top-k and Bottom-k masking results for attention head attribution on Llama-3.2-1 B- Instruct and Llama-3.22-33 3-B--BB-nstttcct DePass leads to larger accuracy drop with Top-k masking and better retention with Bottom-k masking, highlighting its effectiveness in identifying key attention heads across different model scales.
30

---

Ter
Cofee
Ricoet
zum
Figure 17: Top-k and Bottom-k masking evaluation on Dwen2-115B-Instrree and Qwen2-7B- for attention head attribution. DePass consistently achieves greater accuracy drop under Top-k masking and stronger performance retention under Bottom-k masking, demonstrating its ability to pinpoint influential attention heads across architectures.
Pabii
Hoick Dupper
Pintlay
Darras
Dofes
God
Mcai 18-1 41
Nac
Figure 18: Top-k and Bottom-k masking evaluation on Llama-3, 1.-.-3--------Bttt- B-Instruct and 1.ama-3.2-3B Instruct for MLP neuron attribution. DePass outperforms baselines by inducing sharper performance degradation when top neurons are masked and maintaining better accuracy when only low-ranked neurons are ablated.
D.2 Efficiency of Neuron-level Attribution
To ensure the practicality of fine-grained attribution, compare the runtime of DePass with the standard ablation approach when attributing neurons in intermediate MLP layers of different LLaMA models. As shown in Table [I] DePass provides a significant speedup, redueing computation time by up to two orders of magnitude, while maintaining attribution fidelity. This efficiency makes DePass especially suitable for scaling larger models and longer prompts.
Table 11:
Runtime comparison for neuron-leved attributions
intermediate MLP layer. DePass
achieves substantial acceleration over the ablation baseline.
Model
Intermediate Size
Method
Time (s)
LLaMA-2-7B-Chen
10008
Ablation
321.04
DePass
7.22
LLaMA-3.2-3B-Instn
8192
Ablation
234.77
DePass
2.91
LLaMA-3.2-IB-Imm In-struct
8192
Ablation
134.76
DePass
2.23
D.3 Important Head and Neuron Distributions
We provide a detailed analysis of the internal model components by visualizing the distributions of top-attributed attention heads and MLP neurons, both at the dataset level and for individual prompts.
31

---

These analyses reveal how different models and attribution methods identify key substructures that contribute model predictions.
D.3.1 Average Distribution over the Dataset
To better understand which internal components contribute most to model predictions, we visualize the average importance scores assigned by DePass to attention heads and MLP neurons across two representative datasets: IOI and CounterFact. These aggregated views highlight consistent attribution patterns across layers and architectures.
For attention heads, each heatmap aggregates importance scores across all prompts, with rows corresponding to transformer layers and columns representing head indices. For MLP neurons, group neurons into bins of 100 for ease of visualization, plotting the average importance per bin across datasets.
These visualizations provide structural insights into how different model components are utilized in practice, and where important predictive capacity is concentrated across architectures and tasks.
Haeme- chabet
www.a-----cchaeee on
Figure 19: Average attention head importance identified by DePass on 2-7-chh-hh-- aggregated across datasets. Rows indicate transformer layers; columns indicate head indices.
18m8-2 EGroups of Heurse)
lema
dcroue of Nearora)
Seaaee Criu
Meart TND holo 1100 muri 37.00
Figure 20: Average MLP neuron importance identified by DePass on 2-7-chh-7 aggregated across datasets. Neurons grouped in bins of 100 for visualization.
Scares Lame- 2 33-tittu on coumarfect
Loyerwbe sanoe (Abeplut Velues
Liire 3 2-10 Do--eei courtenees
Figure 21: Average MLP neuron importance identified by DePass Llama-3.2-3B. Instructt aggre- gated across datasets. Neurons are grouped in bins of 100 for visualization.
32

---

D.3.2 Per-Prompt Case Analysis
We further present case-by-case visualizations to analyze the distribution of important attention heads and MLP neurons for individual prompts. These visualizations compare DePass with baseline attribution methods (e.g.. AtP, Norm) and aim to identify the components most responsible for producing the correct answer under each input. In contrast to dataset-level views, these examples highlight how attribution patterns vary across prompts and methods.
Figure 22: Attention head attribution for the prompt "The mother tongue of Danielle Darrieux is", "French" using Llama-2-7b-chall DePass more effectively isolates attention heads that contribute to generating the correct answer compared to baselines.
Scree
Upvtrebee DePPasao
frem
Figure 23: Attention head attribution for the prompt "After the lunch, Mildred and Eleanor went the market. Eleanor gave watermelon to", "Mildred" using Llama-2-7h chat-hf. DePass reveals a more focused and task-relevant distribution of important attention heads.
Figure 24: MLP neuron attribution for the prompt "The mother tongue of Danielle Darrieux is", "French" using Llama-2-7b-chal Neurons are grouped into bins of 100 for visualization. DePass effectively identifies neurons critical for predicting the correct answer.
33

---

Figure 25: MLP neuron attribution for the prompt "Tarvaris Jackson plays in the position of", "quarterback" using Llama-2-7b-chat-ht. Neurons are grouped into bins of 100 for visualization. DePass more accurately isolates neurons essential for producing the correct answer compared to baselines.
More Results and Experiment Details on Subspace- Wise DePass
E.1
Classifier Training and Evaluation
Dataset for Probing Classifier Training construct a multilingual dataset with identical se- mantics across different languages, we translate subset of the CounterFact 4 dataset into multiple languages while ensuring that the meaning remains strictly unchanged. This allows us to probe whether the identified important subspaces generalize across linguistic variations that share the same semantic content. An example entry is shown below:
Original (English):
"prompt": "The mother tongue of Danielle Darrieux is"
"answer": "French"
French:
"prompt": "La langue maternelle de Danielle Darrieux est 1e",
"answer":
"français"
German:
"prompt":
"Die Muttersprache von Danielle Darrieux ist",
"answer":
"Französisch"
Italian:
"prompt": "La lingua madre di Danielle Darrieux il", "francese"
"answer":
Classifier Training Details We then this multilingual, semantically aligned dataset to train a language classifier. This setup allows us to evaluate whether the subspace directions identified by DePass stable and predictive across languages, thus offering rigorous test of attribution generalization.
For each input prompt, extract the hidden state of the final token and use it as the input feature to the classifier. The corresponding language of the prompt serves as the class label. We train separate multi-class classifier at each transformer layer using the translated subset of the CounterFact dataset, which is evenly split into training and testing sets with balanced label distributions across languages.
We use logistic regression implemented via the scikit- -learn library, with the saga solver, learning rate of 0.01, and a maximum iteration count 1000. During training, we concatenate hidden states across all prompts and shuffle the dataset to ensure robustness and reduce bias. This approach enables us to analyze which layers encode language-ditinguushing information and how DePass-sclected subspaces contribute to that encoding.
34

---

The language classifier achieves high accuracy across both models, indicating that language-speeee directions are reliably encoded in the hidden states.
Classifier Evaluation
Multi-Language Classification Accuracy Qwen2-7B-lnstruct
Multi-Language Classificattin Accuracy lama-3.1-8b-instt
0.90
Test Accuracy
0.94
0.92
0.88
10 Classifier Index
Figure 26: Accuracy of the language classifier across different layers Llama-3. Instruct and Qwen2-7B-Instruet, Results show strong separability of language-specific information in hidden representations.
E.2 Subspace Projection
We describe how to construct the projection matrix P. E Rdxd used to decompose hidden states into components within and orthogonal target subspace. This subspace be defined by any set of directions of interest, such as the row space of a linear classifier, neuron activations, or gradient- based attribution vectors.
Given a matrix W E Raxe whose column space spans the target subspace (e.g., the weight matrix of linear classifier), compute the singular value decomposition (SVD) of WT as:
W =UV
and retain the top-r left singular vectors U, where r rank(W). These vectors form orthonor- mal basis U, E Raxrr for the subspace of interest.
The projection matrix onto this subspace given by:
P. = U,U,T
and its orthogonal complement is I - Pt, where I is the identity matrix.
For any hidden state 2; E R.,, obtain the decomposition:
P Pe
I -I-P,)
where z!! lies in the target subspace and z; is orthogonal to it. This decomposition enables precise attribution and intervention by isolating the contribution of specific representational directions to model behavior.
E.3
More Results on 1oon2-7BBInntrucce
We present further analysis demonstrating the subspace attribution capability of DePass on Qwen2- 7B-Instruct. To validate the separation of representational content, we apply t-SNE to the language subspace projections X-de obtained from multilingual inputs. As shown in Figure 27 the resulting embeddings form distinct clusters based language identity. This indicates that DePass successfully isolates language-specifet patterns within a dedicated subspace, supporting its ability to disentangle and attribute form-related features.
We further examine subspace semantics by decoding from both the language and semantic sub- spaces for shared English input: "In which city did Charles Alfred Pillsbury's life end?" As shown Table [2] tokens from the language subspace are dominated by structural and frequent function words characteristic of each language, while tokens from the semantic subspace correspond to factual entities such as names and cities. This further supports that DePass routes distinct representational fortrr--form and meaning-ing appropriate subspaces.
35

---

TSNE visualization of language embeddings
75
TSNE dimension 2
50
25
French
-25
German
-50
English
Spanish
-75
-60
-20
20
40
TSNE dimension 1
Figure t-SNE visualization of token-wise representations in the language subspace (X from multilingual prompts Qwen2- Brenn--B-Instruct The clear clustering by language confirms that DePass accurately attributes language-spa speeccffi signals to this subspace.
Language
Decoded Tokens from Lan- guage Subspace
Decoded Tokens from Se- mantic Subspace
English
San, "._ New.
order. aug, which, Charles, Charles
French
quoi, chaque, cette, son,
Minneapolis, vec, cause, par- tir, uc
German
und, der, die, San, das
Minneapolis, ámb, Stockholm, Cambridge, St Minneapolis, Bloom, Saint, ven, och
Spanish
las, el, i los,
Table 12: Tokens decoded from the language and semantic subspaces of Frre2-7B-Instruue The language subspace captures structural and high-frequency tokens, while the semantic subspace reflects content-specific entities relevant to the input query.
Implementation Details
We run all experiments on a cluster A6000 GPUs. Since DePass operates entirely via forward decomposition without requiring model fine-tuning or backpropagation, method is highly efficient and incurs minimal memory overhead. All experiments are conducted in PyTorch, using pre-trained models with lightweight modifications enable decomposition--wware forward passes.
G Limitations of DePass
To precisely trace information flow within the model, DePass freezes the attention scores during decomposed forward pass. a result, under the Transformer circuits framework [52] DePass captures only the flow through the output-value (OV) circuit, without explaining query-key (QK) circuit interactions. Decomposing the QK circuit remains a challenging open problem: first, QK and VO circuits entangled yet structurally distinct, making it difficult construct human-interpretable representations their interactions; second, following DePass's 's additive decomposition approach, attribution of QK circuit would involve quadratic complexity explosion. We believe that unwrapping the QK circuit is a key direction for future work.
DePass is compatible with most Transformer architectures. In this paper, we validate it on two widely used families-I.LaMA. and Qwen-acree across different model sizes. Extending DePass to other architectures primarily requires adapting to differences in LayerNorm implementation and updating module names at the code level. We plan to release general- purppse toolkit supporting broad range of Transformer models.
36

---

While DePass supports one-pass attribution across all model components and, in theory, allows hidden states to be decomposed into infinite number of parts, practical memory constraints require grouping components and computing their forward passes sequentially. This can lead to increased time costs when finer-grained analysis is desired. Future work includes optimizing computational efficiency and improving hardware deployment strategies to better support attribution with higher resolution.
37


Reviews:
[{"review_id": "vfFTrq0xeL", "reviewer": "Reviewer_uE26", "summary": "The paper presents a novel method for interpreting how tokens, modules, or subspace components influence the output, which can be used for output attribution and understanding processes inside model. The method decomposes the hidden state into m components, while the original hidden state can be recovered by summing up the components.", "questions": "- Is it possible to apply these method for the model steering?\n- Can we combine DePass with SAE by decomposing input into desired feature subspace and other components? Or apply it token-wise, like in Eq. 19, for vectors from the SAE decoder?\n\nAlso see weaknesses.", "limitations": "See W3. From my perspective, the main limitation is the computational resources required to process m components through the model, which results in approximately m times the original FLOPs count.", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- S1: The method can be applied to pretrained language models without requiring any additional training.\n- S2: The method achieves superior performance across a variety of tasks.\n- S3: The paper is well-written.\n\nWeaknesses\n\n- W1: The method incurs significant overhead during inference, particularly when applied to token-level attribution tasks.\n- W2: The authors argue that the features identified by SAE do not specify which token activates a particular feature. However, there are several works that explore computational graphs [1] and later studies [2], based on transcoders or crosscoders. While these methods require computationally intensive training, the authors could provide justification for these approaches or include a comparison in the context of token-level attribution. This is particularly relevant given the availability of open-source transcoders for large models such as LLaMA-3.1-8B [3].\n- W3: No limitations of the method discussed in paper.\n\nOther comments:\n\nThe authors state that SAE-based methods \"do not capture how specific features evolve throughout the forward pass.\" However, there are methods such as [4], [5], and [6] that discover the evolution of SAE features during the forward pass.\n\n[1] [Transcoders Find Interpretable LLM Feature Circuits](https://proceedings.neurips.cc/paper_files/paper/2024/file/2b8f4db0464cc5b6e9d5e6bea4b9f308-Paper-Conference.pdf)\n\n[2] Ameisen, et al., \"Circuit Tracing: Revealing Computational Graphs in Language Models\", Transformer Circuits, 2025.\n\n[3] [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with  Sparse Autoencoders](https://arxiv.org/pdf/2410.20526)\n\n[4] [Mechanistic Permutability: Match Features Across Layers](https://openreview.net/forum?id=MDvecs7EvO)\n\n[5] [Analyze Feature Flow to Enhance Interpretation and Steering in Language Models](https://arxiv.org/pdf/2502.03032)\n\n[6] [Evolution of SAE Features Across Layers in LLMs](https://arxiv.org/pdf/2410.08869)", "was_revised": true, "final_justification": "After carefully reading the authors’ response, I have decided to maintain my score and remain slightly inclined toward rejection. I would like to acknowledge the experiments conducted during the rebuttal and discussion phases, which demonstrate the potential of integrating DePass into MI research. However, these experiments are largely qualitative in nature and lack quantitative validation. Regarding the paper as it stood before the rebuttal, DePass appeared somewhat isolated, with limited connection to existing methods. I believe this work could significantly benefit from further refinements aligned with the promising direction outlined during the discussion phase.", "rebuttal": "Thanks for the detailed comments from the reviewer, we would like to make several clarifications below.\n\n### DePass's Efficiency & Fidelity Trade-off\n\n- We acknowledge that DePass introduces additional computational cost compared to a single standard forward pass. However, this reflects a deliberate trade-off between efficiency and **attribution accuracy**. Prior methods often prioritize low computational cost but sacrifice attribution **fidelity**, especially in deep, nonlinear models like LLMs. In contrast, DePass maintains the integrity of the original forward process, producing higher-fidelity, more reliable attributions within an acceptable runtime budget.\n\n- The table below demonstrates DePass's runtime advantage for neuron-level attributions over an intermediate MLP layer in different LLaMA models compared to ablation method. The current implementations already show strong performance, with considerable potential for further acceleration.\n\n|Model|Intermediate Size (number of neurons)|Method|Time (s)|\n|:-|:-|:-|:-|\n|LLaMA-2-7B-Chat|11008|Ablation|321.04|\n|||DePass|**7.22**|\n|LLaMA-3.2-3B-Instruct|8192|Ablation|234.77|\n|||DePass|**2.91**|\n|LLaMA-3.2-1B-Instruct|8192|Ablation|134.76|\n|||DePass|**2.23**|\n\n### Further discussion about SAE-based methods\nWe thank the reviewer for pointing out the lack of clarity in our discussion of SAE. We will revise the paper to improve clarity and **include additional citations [2][4][6][7]**.\n\n##### On which token activates particular feature\n- We appreciate the reviewer for highlighting prior work that leverages computational graphs and transcoders to trace which token activates a particular feature. We acknowledge the validity of these approaches and will revise the wording in our paper to reflect this more accurately.\n\n- While these methods analyze token-feature activation, they often involve **complex and hard-to-scale graph computation and pruning**. To our knowledge, their token-level attribution **fidelity hasn't been systematically evaluated across datasets**.\n\n- In contrast, DePass offers a **simple, scalable, and direct attribution framework**. Instead of proxy measures like token-feature correlations, DePass preserves and propagates attribution information throughout the **actual forward process**, enabling accurate token-level attribution without extra training or complex graph tracing. This practicality and fidelity make DePass more suitable for widespread use in large-scale model analysis.\n\n##### On Feature Evolution Tracking\n- We appreciate you raising this important point. We'll leverage the terminology from [1] for further discussion. We acknowledge that existing work on transcoders and crosscoders can track abstract features across layers within their trained **'Isomorphic Models'**. However, when DePass refers to \"feature evolution,\" it denotes **precise tracking within the 'Literal Model.'** We’ll refine our phrasing and provide a more in-depth discussion in the revised version, along with additional relevant references.\n- DePass's \"feature evolution\" or \"tracking\" precisely traces the information flow of a specific vector or subspace throughout the **literal model's computation graph**. It can accurately isolate the subspace that evolved from a target feature at any point in the residual stream.\n- Transcoder-based feature tracking, in contrast, relies on importance scores derived from correlations between sparsely trained encoders and decoders at different layers [2][3][4][5]. While these methods can identify \"circuits\" through careful filtering and pruning, reconstructing a feature's influence at a given point based on activation relationships is inherently lossy. Moreover, this tracking is in the context of **Isomorphic Models**, not the literal model.\n- Crosscoder-based feature tracking integrates sparsely trained encoders and decoders from different layers to identify features that remain relatively stable across layers in the residual stream. While this simplifies tracking shared features across layers within **Isomorphic Models**, it remains lossy in the context of the literal model.\n- Furthermore, work [6] indicates that semantic similarity between features isn't accurately reflected by directional similarity. Methods that derive causal relationships between cross-layer features from activation correlations across large corpora cannot perform feature evolution tracking in either sense.\n- Nevertheless, we believe DePass can serve as an effective bridge connecting analyses in **Isomorphic Models** with the **literal model**, which we'll discuss in the following section.\n\n##### Capacity of combining DePass with SAE\n- DePass can be applied at the subspace level by treating the SAE features of interest as DePass components, which enables direct and convenient analysis of the interactions among SAE features across different layers.\n- Alternatively, DePass can be used at the token level, similar to Eq. 19, to investigate which tokens’ semantics jointly activate a particular SAE feature.\n- We are interested in conducting such experiments. However, due to the substantial annotation effort required to reliably label SAE features in datasets, we currently lack access to sufficiently annotated feature representations. Moreover, mapping SAE features to human-interpretable semantic spaces remains challenging, making it difficult to obtain ground-truth data for rigorous experimental validation at this time.\n\n##### Capacity of combining DePass with model steering\n\n- DePass can be naturally applied to model steering analysis and offers a novel perspective for evaluating the effectiveness of steering vectors. By treating the steering vector as a DePass component during the forward pass, we can accurately quantify its influence on the model’s final output. Furthermore, when combined with tools like SAE or probing methods that analyze hidden states, DePass enables a comprehensive examination of how steering vectors affect the model’s internal dynamics across layers.\n- We believe there is substantial potential for further integrating DePass with model steering techniques to deepen interpretability and control in large language models.\n\n### On Method Limitations\n\n- We appreciate the reviewer's feedback regarding method limitations. We do discuss the limitations of our method in the appendix F section of the paper. We will clarify this in the main text in the revision to ensure readers can easily find and understand the method’s limitations.\n\n\n[1] Sparse Crosscoders for Cross-Layer Features and Model Diffing\n\n[2] Ameisen, et al., \"Circuit Tracing: Revealing Computational Graphs in Language Models\", Transformer Circuits, 2025.\n\n[3] Transcoders Find Interpretable LLM Feature Circuits\n\n[4] Mechanistic Permutability: Match Features Across Layers\n\n[5] Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders\n\n[6] Analyze Feature Flow to Enhance Interpretation and Steering in Language Models\n\n[7] Evolution of SAE Features Across Layers in LLMs"}, {"review_id": "c0bT5naXZv", "reviewer": "Reviewer_D8ba", "summary": "Forward pass an example through a transformer, then hold any nonlinearities (attention scores and MLP activations) fixed. With these static, the state of the residual stream at any layer can be broken down into components that sum to the hidden state, such that these components can be propagated forward through the network and still yield components at a later layer that correctly sum to the later hidden state. In essence, this allows attribution (e.g. breaking down the input by token, then attributing to each token the degree of influence it had on the output). Empirically, this seems to identify reasonable attributions on various tasks.", "questions": "1. In lines 63-66, what is \"position $i$\"? Is there any reference for the MLP layers' purpose being \"to enhance per-token representation capacity\"?\n2. Why perform the softmax in Eq. 14?\n3. What are the settings under which Fig. 1 is collected?\n4. Why is DePass-abs introduced in Sec. 4.2 with no commentary? Intuitively, DePass-abs might make better sense as the \"default\" version of the technique.\n5. In lines 245-249, why is layer 15 chosen for the decomposition projection?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The method appears to work empirically. Compared to other attribution methods, DePass appears able to more effectively identify key input tokens that change the outputs.\n2. The authors highlight useful practical applications, e.g. measuring the truthfulness of a model.\n\n## Weaknesses\n1. Fixing nonlinearities intuitively might take away much of the representational power of a network (e.g. if a particular token's contribution is not on its own, but through allowing a different token to attend to it, like in the IOI task). Should we expect this, theoretically, to limit the method's effectiveness?\n2. Certain parts of the explanation were weak (e.g. definition of neuron was hard to follow).\n3. The paper would benefit from having more examples. Table 2 was helpful, but insufficient.\n4. The key metric seems to be the change from masking, but additional detail would help here -- for instance, the distribution of attribution between different inputs.\n5. Certain aspects of notation and labeling are imprecise (e.g. the various O variables, percents vs. fractions)\n6. The method's value-add could use more clarification. \n  - For instance, compared to ablation-based attribution, how much does DePass help computationally? As DePass still seems to require a forward pass for each \"hypothesis\" tested, it may not help much -- though without needing to compute attention weights, perhaps the computational complexity gain is immense.\n  - Another example, why *can't* SDL capture evolution of a representation through a forward pass, via multilayer methods like the crosscoder.\n  - (More generally, the claim that DePass tracks evolution through the forward pass could use more examination -- presumably all the intermediates along the residual stream are illegible sequences of numbers until the final decoding stage)", "was_revised": true, "final_justification": "I raised my score from 3 to 4.\n\nAs other reviewers pointed out, there are a number of comparisons to other interpretability techniques that are omitted from the work, which is a very significant limitation. However, the experiments are sufficient to demonstrate that the method is picking up on *some* important signals, and the authors' provided computational-efficiency suggested to me that this work at least occupies a place on the Pareto frontier of the expense-information density tradeoff of interpretability techniques. This is the type of work that I would be eager to see follow-ups of, so it seems suitable for the conference despite its limitations.", "rebuttal": "Thanks for the elaborate comments, we will address them according to their proposed order.\n\n### On the effectiveness of fixing nonlinearities\n\n- DePass is not concerned with the model's representational ability. Instead, its goal is to perform attribution analysis on a given forward pass. Freezing the nonlinearities allows us to precisely reconstruct the original forward pass. DePass ensures the target model's representational power is unaffected by using attention scores and MLP activations directly from its forward pass. This aligns with DePass's core principle: **efficient attribution with full fidelity to the original computation**. Our method specifically attributes **direct paths of information flow to the final logits**.\n\n- While some tasks, like Indirect Object Identification (IOI), involve indirect influences via **QK circuits**, capturing these complex interactions (a challenging open problem in mechanistic interpretability [1] ) is beyond our current scope. We believe attributing with QK circuits is a crucial future direction.\n\n### Clarifying the definition of \"neuron\"\n\n- Following prior work [2], each MLP neuron is defined as a subvalue $v_k^\\ell$, as introduced in Section 2 (Feedforward Network). For token $i$, the coefficient of neuron $k$ in layer $\\ell$ is computed by corresponding subkey:\n\n    $$ m_{i,k}^\\ell = \\sigma(f_k^\\ell \\cdot {X}_{{attn},i}^\\ell) $$\n    and output of the neuron is: \n\n    $$ Neuron_{i,k}^\\ell = m_{i,k}^\\ell \\cdot v_k^\\ell $$\n    The full MLP output is the sum over neurons' outputs: \n\n    $$ MLP_i^\\ell = \\sum_k Neuron_{i,k}^\\ell$$\n    We will explicitly include a clarification in the future revisions.\n\n### Additional examples and detail of our results\n\n- We agree that more examples are crucial for demonstrating DePass's capabilities and will expand relative section in the revision.\n- We provide a concise example illustrating the attribution distribution across inputs, including DePass (our method based on decomposed forward pass attribution), Normalized DePass (DePass scores normalized to sum to 1), All (Mean Attention), Last (Last-layer Attention), Rollout (Attention Rollout), Integrated Gradients (Integrated Gradients), Signed (Input × Gradient), and Norm (Gradient SHAP).\n\n**Prompt: \"The capital of France is\" Target: \"Paris\"**\n|Token|DePass|Normalized DePass|All|Last|Rollout|Integrated Gradients|Signed|Norm|\n|:-|:-|:-|:-|:-|:-|:-|:-|:-|\n|`
`|-0.30|-0.02|0.30|0.28|0.35|0.00|0.17|0.40|\n|`The`|1.20|0.07|0.14|0.14|0.13|0.01|0.17|0.14|\n|`capital`|8.94|0.52|0.14|0.14|0.13|0.84|0.16|0.17|\n|`of`|1.02|0.06|0.14|0.14|0.13|0.14|0.17|0.14|\n|`France`|5.06|0.30|0.14|0.15|0.13|0.00|0.16|0.07|\n|`is`|1.20|0.07|0.14|0.15|0.13|0.01|0.17|0.09|\n\n**Prompt: \"The capital of France is\" Target: \"known\"**\n|Token|DePass|Normalized DePass|All|Last|Rollout|Integrated Gradients|Signed|Norm|\n|:-|:-|:-|:-|:-|:-|:-|:-|:-|\n|`
`|0.09|0.01|0.30|0.28|0.35|0.26|0.18|1.00|\n|`The`|1.91|0.15|0.14|0.14|0.13|0.00|0.08|0.00|\n|`capital`|6.00|0.48|0.14|0.14|0.13|0.65|0.20|0.00|\n|`of`|1.41|0.11|0.14|0.14|0.13|0.00|0.16|0.00|\n|`France`|1.80|0.14|0.14|0.15|0.13|0.00|0.22|0.00|\n|`is`|1.41|0.11|0.14|0.15|0.13|0.09|0.17|0.00|\n\n**Prompt: \"grape: purple, banana: yellow, apple:\" Target: \"red\"**\n|Token|DePass|Normalized DePass|All|Last|Rollout|Integrated Gradients|Signed|Norm|\n|:-|:-|:-|:-|:-|:-|:-|:-|:-|\n|`
`|0.32|0.02|0.14|0.12|0.17|0.02|0.07|0.88|\n|`gra`|1.52|0.11|0.07|0.07|0.06|0.00|0.06|0.00|\n|`pe`|0.08|0.01|0.07|0.07|0.06|0.00|0.10|0.00|\n|`:`|0.20|0.01|0.07|0.07|0.06|0.00|0.06|0.11|\n|`pur`|5.25|0.36|0.07|0.07|0.06|0.00|0.07|0.00|\n|`ple`|0.14|0.01|0.07|0.07|0.06|0.00|0.08|0.00|\n|`,`|0.10|0.01|0.07|0.07|0.06|0.00|0.06|0.00|\n|`ban`|0.48|0.03|0.07|0.07|0.06|0.00|0.08|0.00|\n|`ana`|0.12|0.01|0.07|0.07|0.06|0.00|0.05|0.00|\n|`:`|0.15|0.01|0.07|0.07|0.06|0.98|0.09|0.00|\n|`yellow`|1.53|0.11|0.07|0.07|0.06|0.00|0.05|0.00|\n|`,`|0.19|0.01|0.07|0.07|0.06|0.00|0.08|0.00|\n|`apple`|4.16|0.29|0.07|0.07|0.06|0.00|0.08|0.00|\n|`:`|0.17|0.01|0.07|0.07|0.06|0.00|0.06|0.00|\n\n- DePass reports both raw and normalized attribution scores, with raw values summing exactly to the output logit. This allows direct interpretation of each score’s impact on the prediction. For example, when comparing targets like *“Paris”* and *“known”* for the same input, their total attribution scores differ.\n- DePass attributes the output to semantically meaningful parts of the input, while baseline and gradient-based methods often fail to emphasize relevant tokens or align with clear semantic patterns.\n\n### Clarification for certain notation and labeling\n- We appreciate your feedback regarding notation precision, especially concerning the \"O\" variables and the clarity of \"percents vs. fractions\" in our figures.\n\n##### Clarification of \"O\" Notation\n- Regarding the \"O\" symbols, their usage is consistent but we understand the need for clearer definitions. We'll refine these in the revision:\n    - **$O^{l,j}$** (Eq. 2): output of the $j$-th attention head in layer $l$, representing overall attention without token-wise decomposition.  \n    - **$o_{i,m}^{(l,j)}$** (Eqs. 12,13): output of the $j$-th attention head at layer $l$ for the $m$-th decomposed component of token $i$ in DePass, showing propagation through attention.  \n    - **$o_{i}^{(\\ell,h)}$** (Eq. 20): output of the $h$-th attention head at layer $l$ for token $i$ in attention decomposition, used to initialize decomposed states for attribution.\n\n##### Units in Figures\n\n- We acknowledge that the axes in Figures 1 and 3 require more explicit labeling. We will update the labels and captions to clearly indicate units:\n\n    - **Figure 1**: x-axis is masking *proportion* (0 to 1); y-axis is average relative probability change in percentage (0 to 100).  \n    - **Figure 3**: both x-axis (mask percentage) and y-axis (accuracy) range from 0 to 100.\n\n##### \"Position\" in lines 63-66\n- We'll be more precise with new terminology to avoid confusion. \"Position\" specifically refers to each token's location within the sequence.\n\n### Clarification for DePass's value-add\n\n##### Efficiency and Parallel Computation\n\n- DePass's **parallel computation** greatly enhances efficiency. It enables the simultaneous attribution of many hypotheses of the same type, such as all MLP neurons in a layer, in just **one forward decomposition pass**. This contrasts sharply with ablation, which demands individual masking and re-evaluation for each neuron, leading to considerably higher computational overhead.\n- The table below demonstrates DePass's runtime advantage for neuron-level attributions over an intermediate MLP layer in different LLaMA models. The current implementations already show strong performance, with considerable potential for further acceleration.\n\n|Model|Intermediate Size (number of neurons)|Method|Time (s)|\n|:-|:-|:-|:-|\n|LLaMA-2-7B-Chat|11008|Ablation|321.04|\n|||DePass|**7.22**|\n|LLaMA-3.2-3B-Instruct|8192|Ablation|234.77|\n|||DePass|**2.91**|\n|LLaMA-3.2-1B-Instruct|8192|Ablation|134.76|\n|||DePass|**2.23**|\n\n- Moreover, for studying the role of subspaces, ablation is inherently limited. Naively ablating a subspace during execution can be misleading, as subsequent layers may compensate for the missing information [3]. In contrast, DePass allows us to directly trace and quantify the contribution of a specific subspace without perturbing the network's forward dynamics.\n\n##### Further discussion about SDL-based methods\n\n- Given space constraints, we direct you to our response to reviewer uE26 for further details.\n\n### References for MLP layers' purpose\n- Key references on MLPs enhancing per-token representation capacity include [4-6]\n\n### Empirical comparison of softmax (Eq. 14) and alternative functions\n- Given space constraints, we direct you to our response to reviewer w5Wf for further details.\n\n### Settings of results in Fig.1\n- For each data point in the `Known_1000` and `IOI` datasets using the `Llama-2-13b-chat-hf` model, we:\n    * Obtain attribution scores for the correct answer via various methods.\n    * Remove tokens from the prompt based on these scores, following either a \"patch top\" (highest attribution) or \"recover top\" (lowest attribution) strategy.\n    * Reassemble the remaining tokens into a new prompt and rerun the model.\n    * Measure the relative probability change of the correct answer.\n- The figures show the average relative probability change across all data points per dataset at each masking level.\n\n### Clarification for introducing DePass-abs\n- In lines 201-211, we introduced DePass-Abs as the absolute values of DePass scores, capturing both supportive and suppressive contributions.\n- We chose not to use DePass-Abs as the default to preserve the **direction** of contributions. Raw DePass scores reflect signed contributions, distinguishing positive support from negative suppression—critical for interpreting token-level and subspace-level effects, such as identifying inputs driving truthful versus untruthful outputs.\nDePass-Abs, which ignores direction, is used at the component level where negative contributions can still be functionally important.\n\n### Clarification for layer selection in Section 4.3\n- We chose Layer 15 for its strong probing performance and effective separation of language-specific signals, as demonstrated empirically. While other intermediate layers (13-25) exhibit similar effects, Layer 15 serves as a representative example.  We'll include results from additional layers in the revised version.\n\n[1] Circuit Tracing: Revealing Computational Graphs in Language Models, Transformer Circuits.\n\n[2] Neuron-Level Knowledge Attribution in Large Language Models\n\n[3] Is this the subspace you are looking for? An interpretability illusion for subspace activation patching\n\n[4] Dissecting Recall of Factual Associations in Auto-Regressive Language Models \n\n[5] Transformer Feed-Forward Layers Are Key-Value Memories\n\n[6] Locating and Editing Factual Associations in GPT"}, {"review_id": "wfFq91aark", "reviewer": "Reviewer_w5Wf", "summary": "The authors propose DePass, a mechanistic interpretability framework for Transformer decoders, which decomposes the forward pass. This method partitions the hidden states into additive components that are propagated through the model while keeping the attention scores and MLP activations fixed. This decomposition allows for the precise attribution of model outputs to interpretable components at various levels of granularity. DePass does not require gradients or retraining. The paper demonstrates the flexibility and fidelity of the method through experiments on factual question answering and indirect object identification tasks.", "questions": "Please refer to weaknesses above.", "limitations": "The trade-off introduced by freezing attention and MLP activations is only briefly mentioned in the appendix and should be discussed in the main text. The computational overhead is not analyzed empirically. Additionally, a brief note on the potential dual-use risks of interpretability tools would be beneficial.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength:\nThe idea of decomposing the forward pass into additive components is an elegant and clearly described contribution. Although conceptually related approaches exist, this work generalizes them to a broader setting.\nA major strength is the versatility of DePass across various attribution granularities, showcasing its extensibility which is interesting for the community as a tool.\nThe authors demonstrate the method across multiple model families, with evaluations on standard attribution benchmarks, and compelling qualitative examples (e.g., language-vs-semantic subspace). The comparisons to gradient- and attention-based attribution baselines are appropriate and favorable.\nThe paper is well-organized, clearly written, and generally accessible to readers familiar with mechanistic interpretability.\n\nWeaknesses:\nThe method freezes attention scores and MLP activations across all components, which simplifies decomposition but could break causal faithfulness. This tradeoff is briefly noted in the appendix but should be discussed more thorough discussion in the main paper.\nThe MLP attribution rule is based on a softmax over pre-activation dot products. While intuitive, this approach is heuristic and lacks theoretical justification or empirical comparison to alternatives. The effect of this design choice on attribution accuracy is not evaluated.\nWhile the method is compared to gradient-based and attention-based baselines, it omits evaluation against more mechanistic or causal interpretability methods such as activation patching, ablation-based attribution, or residual stream tracing. This weakens the claim of offering a foundational framework.\nThe discussion of computational efficiency is limited. Although the authors argue for parallelizability, the method involves propagating a large number of components, which may be costly in practice. Empirical analysis of runtime would be valuable.\n\nMinor weaknesses:\nThe legend in Figure 1 lacks clarity\nThe LLMs work at the subword token level, which can lead to fragmented attributions for multi-token words. Do the authors aggregate subword tokens and how does this affect the results?", "was_revised": true, "final_justification": "The authors addressed the questions I had, and I think it is a valuable idea above acceptance threshold. For this reason I mantain my initial score.", "rebuttal": "Thanks for the valuable comments from the reviewer, we want to deliver further explanations accordingly.\n\n### On the effectiveness of fixing nonlinearities\n\n- Freezing attention scores and MLP activations is **central to DePass's design**, enabling **lossless additive decomposition**. This allows us to precisely decompose and track contributions of hidden state components, ensuring their sum exactly reconstructs the original hidden state. This approach **eliminates second-order effects** during forward propagation. Without this freezing, dynamic internal computations would lead to non-linear, interactive attributions, undermining DePass's core advantages of additive decomposition and exact reconstruction.\n\n-  DePass's limitations don't stem from freezing these components, but from **not explaining attention scores** and **not modeling QK-OV circuit interactions**. As noted in Appendix F, freezing attention primarily captures information flow via the output-value (OV) circuit, **excluding query-key (QK) circuit interactions**. Decomposing QK circuits is a complex open problem due to their entangled nature and the quadratic complexity it would introduce. Similarly, dynamic MLP activations would compromise DePass's additive properties and exact reconstruction guarantees.\n\n- We'll clarify this rationale in the main paper, emphasizing it as a necessary trade-off for achieving lossless additive decomposition and faithfulness. We'll also expand on this limitation, discussing how future work might address QK circuit decomposition and dynamic MLP interactions while retaining DePass's core benefits.\n\n### Empirical comparison of softmax (Eq. 14) and alternative functions\n\n- To evaluate the design choice of using softmax for MLP attribution, we compare it against two alternatives: a Linear-norm normalization (subtracting the minimum and dividing by the sum) and a Linear-weighted decomposition (linearly decomposing the contribution to the original activation value for each element). This comparison assesses whether the softmax rule is empirically justified.\n\n- Using the `known_1000` dataset with the `llama-2-7b-chat-hf` model, we report the results for both patch-top and recover-top token removal strategies:\n\n| Patch-Top   | 0.1       | 0.2       | 0.3       | 0.4       | 0.5       | 0.6       | 0.7       | 0.8       | 0.9       | 1.0       |\n| ----------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| **Softmax** | **80.79** | **87.08** | **94.02** | **96.64** | **98.20** | **98.80** | **98.79** | **99.11** | **99.20** | **99.09** |\n| **Linear-norm**  | 64.80     | 73.64     | 85.73     | 93.17     | 96.55     | 97.46     | 98.17     | 98.77     | 99.03     | 99.09     |\n| **Linear-weighted decomposition**  | 59.37     | 66.84     | 77.53     | 86.20     | 92.42     | 94.98     | 96.62     | 98.38     | 98.60     | 99.09     |\n\n| Recover-Top | 0.1       | 0.2       | 0.3       | 0.4       | 0.5       | 0.6       | 0.7       | 0.8       | 0.9       | 1.0      |\n| ----------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | -------- |\n| **Softmax** | **97.81** | **96.87** | **93.92** | **89.89** | **81.57** | **75.50** | **68.94** | **56.84** | **43.50** | **0.00** |\n| **Linear-norm**  | 98.26     | 97.11     | 94.77     | 92.25     | 87.67     | 83.46     | 78.18     | 68.84     | 54.90     | 0.00     |\n| **Linear-weighted decomposition**  | 98.31     | 97.73     | 96.60     | 95.30     | 92.30     | 90.42     | 86.65     | 80.73     | 71.63     | 0.00     |\n\n- Results show that the softmax-based attribution consistently outperforms the alternatives across both token removal strategies, supporting its empirical effectiveness despite its heuristic origin.\n\n### Clarification for selection of baselines\n\n- Our decision not to directly compare DePass against activation patching and general ablation-based attribution methods as performance baselines stems from two key reasons:\n\n- **Ablation as a Fidelity Metric (Ground Truth):** \n    - In mechanistic interpretability, ablation is often regarded as the **gold standard for evaluating attribution faithfulness** [1][2][3]. We therefore use ablation as our **ground truth for validation**, not as a competitive baseline. \n    - Our paper extensively demonstrates that DePass faithfully identifies critical components whose disruption or preservation, based on our scores, indeed impacts model behavior. Comparing DePass directly with ablation as if they perform identical tasks would be conceptually misaligned; **DePass provides attributions, and ablation validates them.**\n\n- **Computational Efficiency and Applicability, with Consideration for Activation Patching:** \n    - DePass is a direct and unified feature attribution framework based on a **single decomposed forward pass**, offering **significant computational advantages** over exhaustive, \"computationally expensive\" ablation methods which provide limited insight into intermediate information flow. \n    - Regarding activation patching methods, they typically require constructing specific contrastive datasets for activation replacement, which introduces additional complexity and computational overhead. Since ablation can be viewed as a specialized form of activation patching—specifically, patching with zero values—we did not assess it separately as a baseline. \n\n### Illustration for multi-token word's attribution\n- We provide two examples of `llama-2-7b-chat-hf` below to illustrate how subword tokenization affects DePass scores. For word-level scores, we treat corresponding subword tokens as a **single component from the start of the DePass process**, not a simple summation of individual scores.\n\n**Example 1:**\n\n| Token     | `
` | `Catal` | `onia` | `belongs` | `to` | `the` | `continent` | `of` |\n| --------- | ----- | ------- | ------ | --------- | ---- | ----- | ----------- | ---- |\n| **Score** | 0.28  | 6.16    | 1.27   | 1.75      | 1.16 | 1.15  | 13.06       | 1.36 |\n\n| Word      | `
` | `Catalonia` | `belongs` | `to` | `the` | `continent` | `of` |\n| --------- | ----- | ----------- | --------- | ---- | ----- | ----------- | ---- |\n| **Score** | 0.28  | 6.38        | 1.92      | 1.37 | 1.38  | 13.24       | 1.59 |\n\n**Example 2:**\n\n| Token     | `
` | `The` | `mother` | `tongue` | `of` | `Daniel` | `le` | `Dar` | `rie` | `ux` | `is` |\n| --------- | ----- | ----- | -------- | -------- | ---- | -------- | ---- | ----- | ----- | ---- | ---- |\n| **Score** | 0.30  | 0.87  | 2.09     | 9.69     | 1.23 | -0.01    | 1.31 | 2.00  | 2.69  | 1.36 | 0.73 |\n\n| Word      | `
` | `The` | `mother` | `tongue` | `of` | `Danielle` | `Darrieux` | `is` |\n| --------- | ----- | ----- | -------- | -------- | ---- | ---------- | ---------- | ---- |\n| **Score** | 0.30  | 1.12  | 2.29     | 9.85     | 1.52 | 1.85       | 4.12       | 1.18 |\n\n- Both token-level and word-level attributions with DePass are effective. While token-level scores might disperse across subword tokens, they remain highly informative for distinguishing important tokens. Higher scores on specific subwords can even indicate triggers for the model's memory of the full word. DePass naturally supports aggregating subword tokens for desired word-level attribution.\n\n- To ensure a fair comparison with other token-level baselines, in the experiments corresponding to Figure 1, we perform ranking and ablation at the subword token level. We will clarify this in the revised text and further elaborate on the impact of subword tokenization on DePass.\n\n[1] Ameisen, et al., \"Circuit Tracing: Revealing Computational Graphs in Language Models\", Transformer Circuits, 2025.\n\n[2] Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models\n\n[3] Finding Neurons in a Haystack: Case Studies with Sparse Probing"}]

Author rebuttal:

Meta-review: **Summary:**

The paper proposes a new mechanistic interpretability method enabling feature attribution with a single forward pass.

**Strengths:**

- Reviewers agree the method seems innovative and could add value to the community.
- The results are strong / promising
- “elegant and clearly described contribution” (w5Wf)
- “Compared to other attribution methods, DePass appears able to more effectively identify key input tokens that change the outputs.” (D8ba)
- “versatility of DePass across various attribution granularities” (w5Wf)
- “Although conceptually related approaches exist, this work generalizes them to a broader setting” (w5Wf)

**Weaknesses:**

- Reviewers are dissatisfied with comparison to existing methods, 
   - Reviewer D8ba would like to see more evaluation with sparse autoencoders (SAEs)
   - “omits evaluation against more mechanistic or causal interpretability methods such as activation patching, ablation-based attribution, or residual stream tracing” (w5Wf)
- Make notation more precise and add more examples (D8ba)

**Rebuttal:**

- The reviewers and authors engaged in significant back-and-forth discussion. 
- Reviewer D8ba raised their score from 3->4 as a result, particularly noting the computational efficiency numbers provided in the rebuttal. “The authors' provided computational-efficiency suggested to me that this work at least occupies a place on the Pareto frontier of the expense-information density tradeoff of interpretability techniques”. They ended with, “I think the work is meaningful and worth accepting, particularly with the improved language and example coverage for clarity.”
- Despite significant discussion with the authors, Reviewer uE26 was unwilling to raise their score from 3. Going through the discussion, Reviewer uE26 continued to emphasize the advantages of SAEs as a mechanistic interpretability method, stating: “methods with some information loss but better downstream performance are more likely to be adopted in practice. Therefore, combining DePass with standard MI methods, including SAE, is important for this work.” They also mentioned a primary concern of theirs was computational efficiency. In response, the authors provide a detailed comment showing an example how their method can provide token-level attributions that align with SAE features.  Reviewer uE26 raised the concern this could be cherry-picked. The authors responded with more examples. 
- Reviewer w5Wf maintained their original score of 4 after the rebuttal.

**Discussion:**

- Reviewer D8ba advocated for the paper, arguing that the method “seems cheap enough to run in parallel with other more expensive techniques, and that seems practically valuable”, and “Since the method does seem to pick up on relevant signal with reasonable efficiency, it seems to be on-margin useful.” With the caveat that the reviewer has uncertainty and that the paper as submitted was insufficient, so would have to be updated based on the rebuttal results. 
- Reviewer uE26 reiterated their concern that the method fails to compare with existing techniques. They said, “the direction of the experiments is undoubtedly correct”, but that the rebuttal experiments are too preliminary to justify acceptance on the basis of what we’re able to review. 

**Decision:**

Reviewers agree the method is novel, practical, and showing promising results. Specifically, that it adds value to the literature on mechanistic interpretability by providing a computationally less expensive method that still provides promising attribution results. 

Because of Reviewer uE26’s dissenting opinion, the raw numerical scores would normally be below the threshold for acceptance. However, I disagree with Reviewer uE26’s insistence on the need to include results with SAEs, and rather think that the fact that this method is proposing an alternative to SAEs makes it more promising, given the number of works focusing on SAEs at the moment. 

Therefore, my decision is to recommend acceptance. I strongly urge the authors to revise the paper based on the reviews, and in particular to include the new rebuttal results.
Decision: Accept (poster)