Title: Pinpointing Attention-Causal Communication in Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: mechanistic interpretability

interpretability

causal analysis

circuit tracing
Abstract: The attention mechanism plays a central role in the computations performed by transformer-based models, and understanding the reasons why heads attend to specific tokens can aid in interpretability of language models. Although considerable work has shown that models construct low-dimensional feature representations, little work has explicitly tied low-dimensional features to the attention mechanism itself. In this paper we work to bridge this gap by presenting methods for identifying *attention-causal communication*, meaning low-dimensional features that are written into and read from tokens, and that have a provable causal relationship to attention patterns. The starting point for our method is prior work [1-3] showing that model components make use of low dimensional communication channels that can be exposed by the singular vectors of QK matrices. Our contribution is to provide a rigorous and principled approach to finding those channels and isolating the attention-causal signals they contain. We show that by identifying those signals, we can perform prompt-specific circuit discovery in a single forward pass. Further, we show that signals can uncover unexplored mechanisms at work in the model, including a surprising degree of global coordination across attention heads.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work studies LLMs attention mechanism, by identifying low-dimensional features that enable the activation of individual attention heads, by identifying subspaces that the QK (query-key) matrices read from. Naming this features as \"signals\", the paper show how it can by applied to circuit discovery. In an experimental setup of GPT2 and Pythia, the paper show how this method can achieve similar circuit discovery performances to methods such as ACDC and EAP. One of the model's promises is the ability to mechanistically interpret models' attention-heads' behavior, such as attention sink, by identifying the features that enable them.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well written and easy to follow for researcher from the field of mechanistic interpretability.\n2. The method is highly based on model mechanistic and mathematical approximation - making the method highly justify.\n3. The motivation, as well as the method itself, are very well justified. The results of section 5 also highlights the method power.\n\nWeaknesses:\n1. Unclear figures: Figure 4 is hard to read as most lines are over each others. Figure 11-17 are also hard to follow but I assume they are proposed to be view via a separate pdf file.\n2. Regarding Figure 6 and its corresponding discussion: Since LLMs hidden state suffer from rouge dimensions [1][2] it is possible that the clustering be improved via another method than naive cosine-similarity, such as Spearman correlation. \n\n[1] All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality\n\n[2] Outlier Dimensions that Disrupt Transformers are Driven by Frequency\n\nQUESTIONS:\nNotes:\n1. some works are cited twice in the References, like [49] and [58].\n2. rearrange figures order: since figure 4 is presented in the text before figure 3, the index of the two should be replaced.\n3. Regarding Figure 6 and its corresponding discussion: I would really appreciate more conversation on why should the model use global signals patterns. Although the results are mostly solid, it is counter intuitive to believe that only a small number of subspaces (features) dominates most of attention head.\n4. Can you draw a connection between some signals and intermediate layer prediction (such as early exit projection [3][4])? I wonder if the signals has a meaning only when read by attention heads, or, that they hold some interpretable semantic.\n[3] interpreting GPT: the logit lens\n\n[4] A Mathematical Framework for Transformer Circuits\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an analysis of model weights and representations to decompose attention computations into low-rank subspaces. They show that low-rank QK signals causally influence when a token is attended to, and use their decomposition to perform circuit discovery and analyze attention sink behavior in LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper is well written and fairly straightforward to follow. Several times I found myself having a question while reading, which was then almost immediately answered or had a reference to an appendix section with more detail.\n- The authors are aware of relevant related work, and the paper’s contributions are contextualized nicely. \n- The notation being consistent was very helpful when trying to understand the definitions in the presentation of the method.\nUnderstanding attention-causal behavior is an interesting alternative to other circuit-finding work, and has promise if it does not rely on counterfactual pairs.\n\n___\n**Weaknesses:**\n- The paper is pretty dense, and because of this many details of results are left to the appendix. This made some results a little unclear when presented in the main paper. For example, in Figure 2 (Section 3), the distribution is taken over an unspecified set of data (e.g. Is it just attention heads for specific token pairs, or all token pairs, all attention heads across many prompts?).\n- In the paper you say your method does not depend on counterfactual inputs (e.g. lines 9-10, line 61), but you often describe parts of the method being reliant on what seem to be counterfactual token pairs (e.g. lines 249-251). I think I may have misunderstood something about this part of the definition - can you clarify this, and whether you’re using counterfactual token pairs here? (See also the question below)\n- There were two terms I was unfamiliar with that it might be nice to have a citation or clarification for. These were the “Courant-Fischer theorem” (line 49, 188), and an “ersatz function” (line 149, 369).\n- It's unclear whether the method feasibly scales to larger parameter models, but this is a minor limitation. They have shown it works fairly well for up to 2B models. \n\n___\n**Other:**\nMinor Typos:\n- Line 184: Duplicated citation for Pan et al. (uses both [49] & [58])\n- Line 295: “postive” -> positive?\n\nQUESTIONS:\n- The results in Section 3 were a little unclear to me. Can you explain how you get distributions for the $\\mathcal{S}^{\\ell ads}$ plots in Figure 2? Is it just because it’s over many attention heads? Or does this also take into account many prompts, and within prompts many (all?) token pairs? Or only for specific token pairs in a given prompt?\n- In Figure 18 (appendix J), the “boosting” vs. “removing” signals for gemma-2 seem to follow an opposite trend for the GP task. Any idea what’s going on there?\n- Can you clarify the use of counterfactual inputs /counterfactual token pairs in your circuit-finding adaptation of your method? (e.g., line 765 describes using “he” vs “she” in the GP setting, and line 770 seems to describe taking a difference between embeddings of counterfactual tokens in the IOI setup)\n- What are your thoughts on related work studying \"binding\" in LLMs? (e.g., [1-4]). These works also identify low-rank subspaces that seem to control different attributes of or attention to entities in a context. It seems related to the idea you're studying of how attention-causal mechanisms work, but did not see a discussion and am curious if you think there's any overlap between those findings and your  own.\n\n___\n[1] Feng and Steinhardt. [How do Language Models Bind Entities in Context?](https://openreview.net/pdf?id=zb3b6oKO77)\n\n[2] Dai et al. [Representational Analysis of Binding in Language Models](https://aclanthology.org/2024.emnlp-main.967.pdf)\n\n[3] Prakash et al. [Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking](https://openreview.net/pdf?id=8sKcAWOf2D)\n\n[4] Prakash et al. [Language Models use Lookbacks to Track Beliefs](https://arxiv.org/pdf/2505.14685)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the internal mechanisms of transformer-based language models by focusing on how low-dimensional features influence attention behavior. In particular, this paper examines how such representations function as attention-causal communication channels written into and read from token positions that have a provable causal influence on attention patterns. The authors then introduce a method that jointly analyzes model weights and representations (without relying on counterfactual interventions) to isolate these attention-causal signals. This enables prompt-specific circuit discovery and sheds light on latent mechanisms in attention.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1.  The paper studies how and where task-relevant signals reside in LLMs. The idea of sparse decomposition to uncover key subspaces is conceptually compelling.\n\n2. The proposed attention-based tracing mechanism offers a new lens to analyze the flow of information in transformer blocks without counterfactuals, complementing existing interpretability tools.\n\n3. The paper evaluates the method across a variety of known benchmarks such as IOI and GP, demonstrating the potential generality of the idea.\n\n\n**Weaknesses**: \n\n1. The notational system, particularly in the sparse decomposition section, is dense and in places undefined or ambiguous. For example, $\\mathcal{S}^{\\text{lads}}$ (line 212) is introduced in natural language without a formal definition, which impedes understanding.\n\n2. Missing notation for key variables: $f^i$, $g^i$ (line 201) are used without prior introduction or definition, making the formulation difficult to follow.\n\n3. Lack of theoretical foundation: While the empirical motivation is strong, the paper lacks a rigorous theoretical analysis of the proposed decomposition. Key questions remain unanswered: for example, why should attention show sparse decomposition? (as proposed by the authors). Although lines 185–205 attempt to provide some intuition, the discussion is informal and primarily heuristic. Lemma 2 is a standard result in linear algebra and does not offer novel insight specific to the proposed decomposition. As a result, the interpretability claims remain largely empirical and would benefit from stronger theoretical support.\n\n4. The probing tasks (IOI, GT, GP) are referenced but not described. A brief summary of what these tasks entail would be helpful for readers unfamiliar with them.\n\n5. While the paper includes performance comparisons with other circuit identification methods, it lacks a clear conceptual and methodological comparison with widely used interpretability approaches such as attention rollout. This makes it difficult to assess the unique value and practical advantages of the proposed framework. Additionally, the method itself is quite complex (the calculation in Section 4). A more streamlined or intuitive explanation (especially from a practitioner’s perspective) would enhance accessibility and broaden the potential impact of the work.\n\nQUESTIONS:\n1. Line 201: feature set ${f^i}$, ${g^i}$: what are $f$, $g$ and $i$?  Please define these formally.\n\n2. What is the x-axis of Figure 2? Could you help parse the plot? \n\n3. Similarly can you help parse Figure 6 first row? What is x-axis for the dendrogram?\n\n3. How does attention-causal communication relate to or improve upon prior tools? Are there cases where prior methods fail but ACC succeeds?\n \n\n\n**Typos and Wording Suggestions**\n\nLine 48:  \"Our attack on this question\" & Line 138: \"We first attack the problem\": \"approach to\"& \"address\" or \"tackle\" is more appropriate.\n\nLine 179: “than” should be “then.”\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks at the the low-rank structure present in the query-key computation of language models. The authors find that singular vectors encode signals across the layers that communicate and can be found automatically (see prior work)\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n* The proposed research direction is interesting and important. The authors do an adequate job of formalizing and exploring the ideas of attention-causal structure.\n* The proposed solution for materializing QK with models that use relative positional embeddings (RoPE) is a nice contribution and could be widely applicable to other interpretability work\n* The paper is thorough and provides a lot of detail in the appendices\n* The extension of the singular vector method analysis is to the GT and GP tasks are somewhat interesting, but are the expectation given prior work.\n* The authors choose good baselines for circuit comparisons\n\n# Weaknesses\n* Lots of jargon and formalisms mask that the paper essentially is a rehash of several prior works. While I think there are some interesting extensions on prior work [50, 51, 58], this paper provides some clarifying formalisms only to replicate **known findings**.\n\n## Known Findings:\n* Singular vectors of attention weight matrices encode distinct meaningful signals across layers [50, 51, 58]\n* Specifically, the IOI circuit is solved by communication across singular vectors in attention heads [50, 51]\n* [51] expands on the ideas of prior work by providing a data dependent measure for selecting singular vectors. This work provides their own version of this, but does not demonstrate why we should prefer there's and offers no novel insight \n* The \"signals in the stream\" analogy is just a rewording of findings from [18] and by extension, the analogy of low rank communication channels in [50]\n* Section 4: \"Tracing circuits\" is not a novel contribution and the language around circuit tracing is very similar to the \"circuit tracing\" and results described in [51] and does not expand on it meaningfully\n* Distinct semantic feature matching performed by singular vectors of [58 <- vision models]. A lot of the analysis resembles [58]\n\n\n**Overall, I can not recommend this paper for acceptance because of the striking similarity to multiple prior works**\n\n\n## other weaknesses\n* Figure 1 is basically incomprehensible. I don't think it is valuable as an illustration for the method. The colored bars are unclear and there is too much jargon frontloaded. I believe this figure should be remade to better use the space\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}