Title: DePass: Unified Feature Attributing by Simple Decomposed Forward Pass
Venue: NeurIPS 2025
Accepted: yes
Keywords: Attribution

Mechanistic Interpretability

Generative Models

Large Language Models
Abstract: Attributing the behavior of Transformer models to internal computations is a central challenge in mechanistic interpretability. We introduce DePass, a unified framework for feature attribution based on a single decomposed forward pass. DePass decomposes hidden states into customized additive components, then propagates them with attention scores and MLP's activations fixed. It achieves faithful, fine-grained attribution without requiring auxiliary training. We validate DePass across token-level, model component-level, and subspace-level attribution tasks, demonstrating its effectiveness and fidelity. Our experiments highlight its potential to attribute information flow between arbitrary components of a Transformer model. We hope DePass serves as a foundational tool for broader applications in interpretability.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose DePass, a mechanistic interpretability framework for Transformer decoders, which decomposes the forward pass. This method partitions the hidden states into additive components that are propagated through the model while keeping the attention scores and MLP activations fixed. This decomposition allows for the precise attribution of model outputs to interpretable components at various levels of granularity. DePass does not require gradients or retraining. The paper demonstrates the flexibility and fidelity of the method through experiments on factual question answering and indirect object identification tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\nThe idea of decomposing the forward pass into additive components is an elegant and clearly described contribution. Although conceptually related approaches exist, this work generalizes them to a broader setting.\nA major strength is the versatility of DePass across various attribution granularities, showcasing its extensibility which is interesting for the community as a tool.\nThe authors demonstrate the method across multiple model families, with evaluations on standard attribution benchmarks, and compelling qualitative examples (e.g., language-vs-semantic subspace). The comparisons to gradient- and attention-based attribution baselines are appropriate and favorable.\nThe paper is well-organized, clearly written, and generally accessible to readers familiar with mechanistic interpretability.\n\nWeaknesses:\nThe method freezes attention scores and MLP activations across all components, which simplifies decomposition but could break causal faithfulness. This tradeoff is briefly noted in the appendix but should be discussed more thorough discussion in the main paper.\nThe MLP attribution rule is based on a softmax over pre-activation dot products. While intuitive, this approach is heuristic and lacks theoretical justification or empirical comparison to alternatives. The effect of this design choice on attribution accuracy is not evaluated.\nWhile the method is compared to gradient-based and attention-based baselines, it omits evaluation against more mechanistic or causal interpretability methods such as activation patching, ablation-based attribution, or residual stream tracing. This weakens the claim of offering a foundational framework.\nThe discussion of computational efficiency is limited. Although the authors argue for parallelizability, the method involves propagating a large number of components, which may be costly in practice. Empirical analysis of runtime would be valuable.\n\nMinor weaknesses:\nThe legend in Figure 1 lacks clarity\nThe LLMs work at the subword token level, which can lead to fragmented attributions for multi-token words. Do the authors aggregate subword tokens and how does this affect the results?\n\nQUESTIONS:\nPlease refer to weaknesses above.\n\nLIMITATIONS:\nThe trade-off introduced by freezing attention and MLP activations is only briefly mentioned in the appendix and should be discussed in the main text. The computational overhead is not analyzed empirically. Additionally, a brief note on the potential dual-use risks of interpretability tools would be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nForward pass an example through a transformer, then hold any nonlinearities (attention scores and MLP activations) fixed. With these static, the state of the residual stream at any layer can be broken down into components that sum to the hidden state, such that these components can be propagated forward through the network and still yield components at a later layer that correctly sum to the later hidden state. In essence, this allows attribution (e.g. breaking down the input by token, then attributing to each token the degree of influence it had on the output). Empirically, this seems to identify reasonable attributions on various tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The method appears to work empirically. Compared to other attribution methods, DePass appears able to more effectively identify key input tokens that change the outputs.\n2. The authors highlight useful practical applications, e.g. measuring the truthfulness of a model.\n\n## Weaknesses\n1. Fixing nonlinearities intuitively might take away much of the representational power of a network (e.g. if a particular token's contribution is not on its own, but through allowing a different token to attend to it, like in the IOI task). Should we expect this, theoretically, to limit the method's effectiveness?\n2. Certain parts of the explanation were weak (e.g. definition of neuron was hard to follow).\n3. The paper would benefit from having more examples. Table 2 was helpful, but insufficient.\n4. The key metric seems to be the change from masking, but additional detail would help here -- for instance, the distribution of attribution between different inputs.\n5. Certain aspects of notation and labeling are imprecise (e.g. the various O variables, percents vs. fractions)\n6. The method's value-add could use more clarification. \n  - For instance, compared to ablation-based attribution, how much does DePass help computationally? As DePass still seems to require a forward pass for each \"hypothesis\" tested, it may not help much -- though without needing to compute attention weights, perhaps the computational complexity gain is immense.\n  - Another example, why *can't* SDL capture evolution of a representation through a forward pass, via multilayer methods like the crosscoder.\n  - (More generally, the claim that DePass tracks evolution through the forward pass could use more examination -- presumably all the intermediates along the residual stream are illegible sequences of numbers until the final decoding stage)\n\nQUESTIONS:\n1. In lines 63-66, what is \"position $i$\"? Is there any reference for the MLP layers' purpose being \"to enhance per-token representation capacity\"?\n2. Why perform the softmax in Eq. 14?\n3. What are the settings under which Fig. 1 is collected?\n4. Why is DePass-abs introduced in Sec. 4.2 with no commentary? Intuitively, DePass-abs might make better sense as the \"default\" version of the technique.\n5. In lines 245-249, why is layer 15 chosen for the decomposition projection?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel method for interpreting how tokens, modules, or subspace components influence the output, which can be used for output attribution and understanding processes inside model. The method decomposes the hidden state into m components, while the original hidden state can be recovered by summing up the components.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- S1: The method can be applied to pretrained language models without requiring any additional training.\n- S2: The method achieves superior performance across a variety of tasks.\n- S3: The paper is well-written.\n\nWeaknesses\n\n- W1: The method incurs significant overhead during inference, particularly when applied to token-level attribution tasks.\n- W2: The authors argue that the features identified by SAE do not specify which token activates a particular feature. However, there are several works that explore computational graphs [1] and later studies [2], based on transcoders or crosscoders. While these methods require computationally intensive training, the authors could provide justification for these approaches or include a comparison in the context of token-level attribution. This is particularly relevant given the availability of open-source transcoders for large models such as LLaMA-3.1-8B [3].\n- W3: No limitations of the method discussed in paper.\n\nOther comments:\n\nThe authors state that SAE-based methods \"do not capture how specific features evolve throughout the forward pass.\" However, there are methods such as [4], [5], and [6] that discover the evolution of SAE features during the forward pass.\n\n[1] [Transcoders Find Interpretable LLM Feature Circuits](https://proceedings.neurips.cc/paper_files/paper/2024/file/2b8f4db0464cc5b6e9d5e6bea4b9f308-Paper-Conference.pdf)\n\n[2] Ameisen, et al., \"Circuit Tracing: Revealing Computational Graphs in Language Models\", Transformer Circuits, 2025.\n\n[3] [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with  Sparse Autoencoders](https://arxiv.org/pdf/2410.20526)\n\n[4] [Mechanistic Permutability: Match Features Across Layers](https://openreview.net/forum?id=MDvecs7EvO)\n\n[5] [Analyze Feature Flow to Enhance Interpretation and Steering in Language Models](https://arxiv.org/pdf/2502.03032)\n\n[6] [Evolution of SAE Features Across Layers in LLMs](https://arxiv.org/pdf/2410.08869)\n\nQUESTIONS:\n- Is it possible to apply these method for the model steering?\n- Can we combine DePass with SAE by decomposing input into desired feature subspace and other components? Or apply it token-wise, like in Eq. 19, for vectors from the SAE decoder?\n\nAlso see weaknesses.\n\nLIMITATIONS:\nSee W3. From my perspective, the main limitation is the computational resources required to process m components through the model, which results in approximately m times the original FLOPs count.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}