Pinpointing Attention-Causal Communication in Language Models
Authors: Gabriel Franco
Mark Crovella
Venue: NeurIPS 2025 poster ()
Keywords: mechanistic interpretability
interpretability
causal analysis
circuit tracing
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: This paper introduces a method to identify the specific low-dimensional signals that are causal to the attention, enabling efficient, single-pass circuit discovery and revealing novel, model-wide control mechanisms.

Abstract: The attention mechanism plays a central role in the computations performed by transformer-based models, and understanding the reasons why heads attend to specific tokens can aid in interpretability of language models. Although considerable work has shown that models construct low-dimensional feature representations, little work has explicitly tied low-dimensional features to the attention mechanism itself. In this paper we work to bridge this gap by presenting methods for identifying *attention-causal communication*, meaning low-dimensional features that are written into and read from tokens, and that have a provable causal relationship to attention patterns. The starting point for our method is prior work [1-3] showing that model components make use of low dimensional communication channels that can be exposed by the singular vectors of QK matrices. Our contribution is to provide a rigorous and principled approach to finding those channels and isolating the attention-causal signals they contain. We show that by identifying those signals, we can perform prompt-specific circuit discovery in a single forward pass. Further, we show that signals can uncover unexplored mechanisms at work in the model, including a surprising degree of global coordination across attention heads.

Pinpointing Attention-Causal Communication in Language Models
Gabriel Franco
Mark Crovella
Department of Computer Science
Department of Computer Science and Faculty of Computing Data Sciences Boston University crovella@bu edu
Boston University gyfranco@bu.edh
Abstract
The attention mechanism plays central role in the computations performed by transformer-based models, and understanding the reasons why heads attend to spe- cific tokens can aid in interpretability of language models. Although considerable work has shown that models construct low-dimensional feature representations, little work has explicitly tied dimeeninsional features the attention mechanism itself. In this paper we work to bridge this gap by presenting methods for identify- ing attention causal communication, meaning -iwdimmnsional features that are written into and read from tokens, and that have provable causal relationship attention patterns. The starting point for our method prior work [1-3] showing that model components make use of low dimensional communication channels that can be exposed by the singular vectors of QK matrices. Our contribution to provide rigorous and principled approach to finding those channels and isolating the attention-causa signals they contain. show that by identifying those sig- nals, we perform prompt-specific circuit discovery in single forward pass. Further, we show that signals can uncover unexplored mechanisms at work the model, including a surprising degree of global coordination across attention heads.
Introduction
Transformer based language models exhibit remarkable abilities 64 and consequently have been very widely deployed. However, it is quite challenging explain how language models are able to accomplish such sophisticated tasks [7]. Nonetheless, progress on interpretability for language models is critical, e.g., build foundation for improving model safety and alignment [8].
meaningful explanation for model behavior should capture causal, rather than merely correlative, relationships. Elucidating causal relationships within language models has been largely approached using interventions during model execution [9-15], method termed causal mediation analysis [16].
However, in this paper treat the model itself structural causal model [13, 17] and thereby exactly identify the impact of a given counterfactual on the model's computation.
We focus on causal analysis of key model computation: attention. In the attention mechanism head uses its QK matrix compute an attention weight for each token pair, and when the weight large, say the head aftends to the token pair, resulting in significant movement of information within the model. The attention mechanism has been celebrated for its ability to respond to precise features in activations and enable sophisticated behaviors [18-20]. From the standpoint of causal analysis, the attention mechanism raises basic question: when a head attends to a token pair, what is the most useful counterfactual? That is, what are the most informative features present in the tokens that explain the head's attention to the token pair [16]? Answering this question has potential
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

to make progress on critical challenges in interpretability. As shown in [21], model components work together to implement complex behaviors by reading and writing to residuals, so precisely identifying attention-causal features uncovers key communication taking place within the model.
Leverage in addressing our question comes from the large amount of work showing that many impor- tant features in models are encoded in one-dimensional subspaces [15, 22-28] or in low-dimensional subspaces [29-32]. Hence we seek explain head's attention in terms of the presence or absence of features encoded in low-dimensional subspaces of the residuals.
This suggests focus parsimony in selecting counterfactuals. We frame the question as follows: Given an attention head attending to a pair of tokens, for each token we seek to identify a low- dimensional subspace and small number of upstream components, such that if the components had not written into the token in that subspace, the head would not have attended to the token pair. We show that this strategy identifies low-dimensional signals that are causal for attention, directly provide efficient and useful circuit traces, and can uncover model-wide control mechanisms at work.
Our approach to this question starts from prior work [1-3] showing that the singular vectors of QK matrices will tend to be aligned with important features in residuals. Hence we start by decomposing residuals in the spectral bases of QK matrices. However, placing activations in new basis is not enough. Model heads use the Softmax function to compute attention; this nonlinearity complicates the question of deciding which components of a residual are causal when a head attends to token pair. In the body of the paper we show how we overcome these challenges to identify the signals that are causal for attention.
Once signals are identified, they can be used to address a number of challenges in model inter- pretability. First, they enable precise and simple circuit discovery (emphasized as an open problem in [8, 33)). We show a general method for using signals to identify the model components that are causal for model outputs. Across the models and tasks we study, the circuits we identify are all consistent with those found via previous methods. Each circuit obtained is specific to a given model input, and requires only a single model execution (without need for counterfactual inputs). Together, these aspects constitute an advance over the state of the art in circuit tracing.
Further, identifying signals uncovers and explains fundamental model mechanisms. Because the signals we identify are jointly determined by both model weights and runtime activations, they are not limited to data-dependent features. We broadly identify two classes: data signals and control signals. Control signals are present across many or all tokens, and are often used in the control of large groups of attention heads. This novel phenomenon occurs across models in our study.
Related Work. Our work relates to mechanistic interpretability [34, 35], which seeks to under- stand neural network computations in human-understanddble terms. Significant bodies of work in the area concern (a) how features are represented [22] and (b) how to trace circuits, ie, to identify model components that are responsible for solving specific tasks [23]. Our work contributes both of these areas.
With respect to feature representation, much prior work has established the cowddimensionality of many feature representations [15, 22-32]. Feature representations have been extensively explored using Sparse Autoencoders (SAEs) [36, 371, but those methods have been shown to have significant drawbacks [38-42]; one reason for this can be that SAEs are built only from model activations, and do not take into account model parameters [43]. In contrast, and model parameters (QK matrices) to identify features that are provably causal for attention. On the other hand, our work shares some ideas with [44], but that work is not based on causal analysis.
work jointly analyzes activations
With respect to circuit tracing, much recent work has used methods based on counterfactual inputs (' ('activation patching") " for establishing causality of model components [10, 11, 14, 45-47]. Patching is time-consuming, requiring many forward passes, generally requires the creation of a counterfac- tual data set to provide task- activation patches, and exhibits a number of other weaknesses [48-51]. In this work, we trace circuits using only a single forward pass, eliminating the need for counterfactual inputs and avoiding the problems associated with patching.
Our method has similarities to but also important differences from [3]. Unlike that paper, we start with formal problem definition of attention-causal communication and we present an algorithm that provably identifies attention-causal communication. Further, the method in [3] needs modification to work in general. This is because [3] assumed that moving the attention score of token pair

---

towards zero will decrease the attention on the pair. However, this is not true in general (eg, when attention scores are negative), which is one motivation for our introduction of relative attention.
As noted above, our approach to signal identification builds on prior work showing that model components often communicate in low dimensional subspaces defined by the spectral decomposition of QK matrices [1-3]. Furthermore we show that the control signals we identify are proximal causes for attention sinks [52, 53] and generalize previous attention-sink mechanisms [54].
Background. In the model, token embeddings are D-dimensional, there are H attention heads in each layer, and there are L layers. We defne R = и.. which is the dimension of the spaces used for keys and queries in the attention mechanism. We use N denote the number of tokens in a given prompt. Superscript indices will denote (layer, head, destination token, source token); reduced sets of indices will be used where there is no confusion, and subscripts will generally denote matrix components.
To simplify exposition in the body of the paper, we consider models in which the attention mecha- nism does not apply bias term to computation of keys and queries.' In Appendix B we describe how we handle models with bias terms the QK circuit. Further, the body of the paper only dis- cusses models with global positional encoding; in Appendix C we describe how we handle models that use rotary positional encoding [57] (RoPE). We emphasize that all the methods in the paper extend to models having attention bias terms and using RoPE, and we provide code 2 implementing our methods for those models.
The attention mechanism operates on set of N tokens in D-dimensional embeddings: X E RN Each token RD is passed through linear transforms given by x WK. x WQ. using weight matrices WK, Wo RDxR, Then the inner product is taken for all pairs of transformed tokens to yield attention scores:
in which Ω W. computation. We also refer to Ω as the head's QK matrix.
K- x" is the destination token, and x* is the source token of the attention
To enforce masked self-attention, A'ds is set to -oo for d < 8. Attention scores are then normalized for each destination d, yielding attention weights Aa Softmax(A') (A',/ (A///R)) /R). The resulting attention weight A., is the amount of attention that destination d is placing on source We denote the portion of the output of attention head (E, a) that comes from x and is written into x via the OV-circuit as lads Further details of model computations are given in Appendix A.
2 Signals
Problem Statement. As described in $1, our goal is to identify low-dimensional features that are causal for attention patterns. We define the problem of attention-causal communication as the following. For given model and input, and given a head (L,a) that attends to a token pair (x",, x*), consider one of x" or x", denoted x. Identify a c-dimensional subspace C, and a set M consisting of m model components upstream of (e, a), with e and m as small as possible, such that if the components in M had not written into x in subspace C, head (e, a) would not have attended to (xd x²).
To make our problem statement precise, we need to define when head "attends to" token pair. We adopt a very conservative definition: we say that a head (f, a) attends to the token pair (d, s) when Alc 2 1/n, where n is the number of tokens considered in the Softmax calculation. That is, the head attends to the token pair when the attention it places on this pair is greater than what it would be for uniform distribution over tokens. This could be considered too low a value in many cases- when the context window is large, the value of 1/n will be quite small. However, we emphasize that our methods can be used for any setting of attention threshold greater than 1/n, and a simple matter to use larger threshold in practice. The other phrase need make precise is for "c and
Models in this category include the Gemma [55] and Llama [56] families. "Code available pinpoint wtw.ttnetittnonnno-eeeeee causalll-ccaaalle commuuacaiooommiiiiiiiioooo..
at
https:// /tt//gtttttt www./o////aacceecooooooee htt.g/aabbreerraroooo
3

---

m to be as small as possible." In practice we optimize these separately, first finding a subspace C having small dimension, and then finding the smallest set of upstream components (M) given C.
We do not attempt to solve the attention-causal communication problem exactly, but instead present heuristic method that gives very good results in practice. In the following sections describe the method in detail.
Relative Attention. We first attack the problem of identifying the subspace C. We isolate this problem as follows: consider a head (f, a) and a token pair (x" x²), such that A& 2 1/n. Working for example with x" (methods for x" are analogous), we seek a low-dimensional subspace such that in a counterfactual setting where the component of x in C were removed, we would have A < 1/n. That is, under the intervention do(x = Poax"), the head (l,a) would not attend to the token pair (x", x*).
An additional constraint on C is that upstream components must write into x in C. By virtue of the model's s residual connections, we can consider x to be an initial encoding plus a sum of upstream additions [21]. Hence it is attractive to consider linear decompositions of x in determining C. How- ever, this strategy faces a key challenge: head's attention Ads is nonlinear function of f2, x", and x through the Softmax function.
To address this problem, we introduce an ersatz function that usefully stands for Softmax in causal analysis. Recall that Aa = Softmax(A (A)//V) as described in $1. We define the relative attention at head (l, a) and position (d, s) as:
tade A'e
Σ
N's
(2)
d
JSd
where A/es = x= Relative attention has two important properties: first, it linear in each of x", 2 and x². The linearity dads in each of these will be important for our methods below.
The second important property of relative attention is that it is useful for causal analysis. In particu- lar, we have the following Lemma:
Lemma 1. Let n S N be equal to the number of tokens considered in self-altention at head (e, a) for destination token d, and Lads be defined as in (2). If A 2 1/n, then Lads >> O.
We prove Lemma in Appendix D. We can interpret Lemma as saying that if an attention head puts more weight on a source than would occur under uniform distribution, the relative attention (2) will be positive; likewise, when the relative attention is negative, the attention weight is less than what given by the uniform distribution.
Using relative attention we can reframe the search for causal mediators of attention. Instead of looking for low-dimensional components of that are causal for (nonlinear) attention greater than 1 1/n, we can search for low-dimensional components that are causal for (linear) relative attention greater than zero
Attention Decomposition. Relative attention is useful tool in searching for attention-causal fea- tures, because it allows to consider linear decompositions of x candidate features. We first note that relative attention (2) can be expanded in the singular vectors of Q:
Lads
Σ IEKER
LdT u
1
Σ 1Sd
IdT u o x5
(3)
d d-
where EiskeR A is the singular value decomposition of Qt Then the following hypothesis (adapted from [3]) drives our approach:
Hypothesis (Sparse Attention Decomposition [3] When an attention head performs a task that requires detecting components in pair of dineenenniooal subspaces in its inputs x" and x", and its inputs have significant components in those subspaces, the terms in (3) will show large values for distinct subset of values of k.
In $3 we demonstrate that sparse attention decomposition occurs widely across the models, tasks, and attention heads we examine. Typically the number of significant terms in (3) is 20 or less, and

---

is often as small as 5. This gives us leverage on determining C. When considering an attention head (e, a) that is attending to tokens (x" (x"xx") we will have that cauds will be positive. We then can identify C' defined by the smallest set of terms responsible for the positive value of cfads and by the sparse attention decomposition hypothesis, we expect that C will typically be a low-dimensional subspace. This strategy builds on prior work [l-3] showing that the singular vectors of QK matrices provide useful decompositions of attention and inter-layer communication.
Why Should Attention Show Sparse Decomposition? Here we pause to ask: why should atten- tion show sparse decomposition? To simplify matters, we consider just the bilinear form xxx Ωx* and ask why it should show sparse decomposition in its singular vectors.
We start by noting that for square matrix A, a consequence of the Courant- theorem is that, over all unit vectors x, the maximum value of the quadratic form x Ax is obtained when x is the principal eigenvector of A. This is readily extended to bilinear forms: over all unit vectors x and y, the maximum value of x Ay is obtained when x is the principal left singular vector, and the principal right singular vector, of A. From the standpoint of model training, another view is useful:
Given vectors x and y, among all rank-1 matrices having unit Frobenius norm, the matrix D that maximizes x' T Dy is D [xll lyll
Lemma 2.
x (Proof is provided in Appendix E.)
In a model training setting, we can hypothesize (following [15, 22-32J) that x" and x" contain corresponding feature sets {Ø'}, { that are encoded in one-dimensional or in low-dimensional subspaces. For example, feature Φ² E RR" could correspond to a geographic location encoded in x", as in [27]. We further note that the authors in [22] argue that models will tend to represent correlated feature sets in a manner such that, considered in isolation, the sets are nearly orthogonal. They term this the use of "local, almost orthoogoal bases." In our case, if an attention head with QK matrix Q is tuned to detect feature sets {Ø'} and {r'} that are important when performing a specific task, then we may hypothesize that training will construct the sets to be "nearly- -otroo--rthogoona,""" meaning that cosine similarities among the features in each set would typically be small. In this case, Lemma suggests that the learned l2 will have sets of singular vectors {uk}, ). [v&] that are likely to sparsely encode the (Ø*) and (T*).
Isolating Signals. We can now describe how to decompose residuals to separate causal signals they contain from background. The goal is to identify which terms in (3) form the sparse represen- tation of cf which is akin to denoising in signal processing. Starting from head (e, attending
to token pair (xd,, x*), we have that A 2 1/n, and so by Lemma fade > O. We separate the terms in (3) into two sets, one of which is the smallest set whose sum exceeds Ladds and the other whose sum is close to (but less than) zero
Specifically, we define gtads as (the indices of) the smallest set of terms in (3) that in sum exceed Lods The terms captured in gtode are strictly positive, and are the largest positive terms in (3). Given géads we can decompose model residuals into 'signal' and "noise" in terms of their impact on attention scores as measured by cadds Define subspaces Ut = Span k E Stads] } and V Span [v k E Stada] } and associated projectors Pu and pv. The denoising step separates the residuals x and Xe into:
ddd -uxxl,
-Pux'' s² ='xx''J,,
(4)
where Pipl= = I - Pu and Py1 = I- Pv. Then we have x = sld +z ld x = 5t + Z', and
s NT te St d
Σ
s Ad Sat si so cnds and z LTT 20 z
Σ
x MTT TT z S 0,
d
jSd.j$
jdd
(5)
where [Stada] = dim = dim as small as possible Note that by definition, under the coun- terfactual in which signal sdd is not present in xd (i.e., do(x'd x then clads 0, and so (E, would not have attended to token pair (x"', x*). The corresponding conclusion holds well for Se and X
5

---

Dondtty
genma 2:26
9612 small
110mm
9012 small
sylhe-1100m
Densks
16.0
sp12 small
per perrma-222
pythre-1sdm
2.5
0.0
611
0.4
0.5
0.3
0.5
0.0
0.1
0.2
0.4
Fraction of singuaar vectors glm
Fraction of singuler vectors in ghm
Fraction singiter vectors 5100
Figure 1: Dimension of signal subspace compared to R in (left to right): 10I task, GT task, GP task.
Sparse Attention Decomposition
Next we confirm prior work showing that the phenomenon of sparse attention decomposition is ubiquitous [1-3]. We utilize models spanning various architectures and scales: GPT-2 small [19], Pythia-160M [58]. and Gemma-2 2B [55]. These models are evaluated on multiple tasks: Indirect Object Identification (IOI) [45], the Greater Than (GT) task' [59], and Gender Pronoun (GP) [60].
These models and tasks cover wide range of studies in mechanistic interpretability, specifically in circuit discovery [14, 61-63].
We first determine Studs for significant A > values using the strategy in $2. We then count the number of singular vectors utilized from Equation (3) for each relevant head and token pair. Figure illustrates the resulting distributions of the fraction of singular vectors used to construct glade ie, the distribution of | I/R. The figure illustrates the sparsity of attention decomposition across different attention heads, prompts, models, and tasks. Typically, models use only 5-10% of the available R dimensions (e.g.. 3-6 for GPT-2/Pythia, 13-25 for Gemma-2)-an almost always less than half-to compute their attention scores. Additional plots demonstrating this sparse attention decomposition for individual attention heads in various tasks are Appendix G. These findings confirm that sparse attention decomposition is widespread, indicating that signals generally inhabit low-dimensional subspaces.
4 Application: Tracing Communication and Circuits
The results of $3 suggest an application to tracing communication. In this section we describe how may signals trace communication within the model.
Causal Structure. To start, we note that the use of residual connections in the model makes possible to describe residual x" at the input to layer e in the model as the following sum:
AH outpats
FFN outputs
biases
input at layer
ldd ERD ΣE ISKEEEOSHISES ISENN
o lads
Σ rld
Σ bb
15KE
151KE
where o lads is defined in $1, pid is the output of the FFN at layer for resdual d, b' is the bias term for layer 1, and xd is the input embedding of token d. Equation (6) provides a complete and exact decomposition of residual at any layer in the model.
To characterize the causal structure of the model with respect to relative attention, we define two functions d(-) and s(-) parameterized by head (L,a) and token pair (x" x*). These functions measure the extent to which particular model component affects the relative attention fode The function dadd (w) measures what the relative attention Lads would be if the source tokens were fixed as X while w had been the destination token; and sadds if the of source tokens had been W but the destination token were fixed as x" (formal definitions are in Appendix H). Each function is linear in its argument. Note that
(7) To characterize causality in the model with respect relative attention, distribute (7) over (6) as described in Appendix H. This gives, for each upstream component, its contribution to the relative attention at head (e, a) and token pair (x" x*).
Eads Ladd (x = fads (xx)
Gemma-22 2 2B was not analyzed for the GT task, as its distinct number tokenization (requiring two sequen- tial tokens for two-digit numbers) complicates performance comparisons; details are in the Appendix F.

---

Tracing Attention-Caue Communication. nication within the model. Our goal is, given A 2 1/n, find the upstream components that are responsible for head (e, a) attending to token pair (x*, x*).
The pieces are now in place to trace causal commu-
We describe the tracing process starting from a destination token x The process starting from source token analogous, and we provide full details Appendix H.
Using the methods of $2, we obtain the subspace U in which the signals reside. This allows us to isolate signals as in (4), obtaining gdd By (5), (7) and the properties of the SVD, we have
(8) To identify the upstream components writing these signals into the residuals, distribute (8) over (6). Then we can determine the upstream contributions to gaads by making use of the linearity of
C Lads Re dlade (s!d) d (Prxx"4)
d(-) and the projection operations. Specifically, distributing (8) over (6) results in the upstream contributions:
sads ddads d (Putt didd = dads (Piet dfads = ddads (Pubb).. dgdds daads (Pux
dhat
(where subscripts here denote upstream indices), allowing to decompose the relative attention in terms of each upstream component's 's contribution:
Lads
Lads
Σ glads
Edfads dg plads
I<E
I<E
We provide more details, and an equivalent decomposition for the source token, in Appendix H. Equation (9) shows how measure the contribution of each upstream component to the downstream relative attention. Each term in Equation (9) represents an edge in the communication graph of the model's 's computation.
To build a communication graph for a given model and prompt, we start by finding the heads that contribute the largest signal in the direction of the model output. Then, for any of those heads that are attending to a token pair and writing in that direction, we find the upstream components (atten- tion heads, MLPs, bias terms, input tokens) causal for their attention. We then work backward, recursively tracing signals from downstream to upstream. The recursion terminates on any head in the first layer, that not attending token pair, or on an MLP, bias term, input token. Our algorithm is described in detail in Appendix H, and the code for all our methods is available.
Communication Graphs. We construct communication graphs for GPT-2 small, 1ythia-60MM, and Gemma-2 2B across the IOI, GP, and GT tasks. Appendix H provides details of the process, run- ning times, and examples of resulting communication graphs. All experiments were conducted on CPU-only machines.
0.5
(F(E, (F-EE
Communication graphs (Appendix H) are highly detailed and con- tain extensive information about information flow in a model's 's com- putation. To illustrate, we present aggregated versions for the IOI task in Figure 3, in which each edge's thickness reflects the num- ber of edges between tokens the full communication graph. Even aggregated, these graphs illustrate differences in the task solution strategies used by the three models. For example, they confirm that GPT and Pythia rely on identifying duplication of the subject token ('Simon"), ) and later tokens receive information from both occur- rences (as reported in [45, 63]). In contrast, Gemma uses the token 'Andrew' as an information-aggregatingg anchor, with subsequent tokens receiving information from this anchor. Complete commu- nication graphs for these prompts are in the Appendix.
-0.5
Sinhibitionn >> Name Mover Bever
Prev Taken > Induction
S-inhibitio
Figure 2: Intervention effect, GPT-2/IOI. Green: signal ab- lation; Red: signal boosting; Blue: random ablation; Or- ange: random boosting.
Interventions. To validate our communication graphs, we demonstrate that intervening in the signals we identify has causal impact on model performance. For any signal s, we boost (do(x = x + s)) or ablate it (do(x = x s)) for the relevant token x at input to the relevant attention head (L.a) We then measure the performance of the model on the given task. Perfor- mance is task-specific, but is measured in all cases via relative logit differences between correct and incorrect outputs, with positive values indicating improved model performance.
"Specifically, intervene only the input of either the K transformation.
7

---

Then
S Simon
and Andrem working
the restaurant
Simon decided to
basketbell to
Then
Simon and Andrew were working at the restaurant
Simon decided to give
basketball to
<bos>
Then
Simon
and Andrew were working at the restaurant
Simon decided
to
give
basketball to
Figure 3: Aggregated communication graphs, IOI task. Top bottom: GPT-2, Pythia, Gemma-2.
pythia-160m, 101
gpt2-small, 10)
gpt2-small, GP
gpt2-small, GT
1.0
0.8
+
Metric Value
0.6
0.4
0.2
0.0
Ours
ACDC
EAP
Ours
EP
Ours
EAP ACDC EP Ours
Precision
Recall
F1-Score
Figure 4: Circuit performance. First-reported circuit used for comparison for every task/model.
Figure shows typical results; full results are Appendix We find that ablating any of the test signals leads to performance decreases, and boosting the signal leads performance improvements, across the three models and three tasks. Note also that these interventions have an extremely small impact residuals themselves, with average relative change in vector norm less than 1% and cosine similarity before and after typically greater than 0.99 (see Appendix J).
Circuits. To find circuits (maximal collections of model components causal for performance), aggregate communication graphs over multiple prompts, threshold edges based on their causal impact on model output, and remove components not causal for the output. For comparative anal- ysis. focus methods that report full circuit components, such Edge Pruning (EP) [61], ACDC [14], and EAP [62]. We note that reported circuits for each task vary considerably and lack definitive ground-truth, reported by others. To choose a baseline for comparison, we use the first-reported circuit for each task and model combination (Pythia/IOI: [63], GPT-2/IOI: [45], GPT-2/GP: [60], and GPT-2/GT: [59]), against which we compute precision, recall, and Fi-scores for all other methods. Figure shows that our approach reports circuits that consistent with previously reported circuits. Details and more comparisons are in Appendix K.
seek to know whether our circuits broadly consistent with those
5 Application: Control Signals
As second application of our methods, we examine signals used in a single forward pass (no longer limiting to those in the communication graph as in $4). We find that many signals are data- independent- they are used by heads in many layers, and they are present in many tokens. These signals play role in organizing model computation; call them control signals.

---

0.0
similanity
Cosine similentty
0.2
Loayer
10
1011
1011
Index
Index
0.0
0.0
Cosine simitarity
Cosine similaniy
0.2
0.6
0.6
0.8
0.8
1.0
1.0
Layer
Layer
Attention Head Indek
Altention Head Index
0.2
Cosine similarity
0.4
Cosine
0.6
1.0
1.0
3
9
ayer
15
15
21
21
Attention Head Index
Attention Head Index
Figure 5: Control signals and associated heads: top: GPT-2, middle: Pythia, bottom: Gemma-2.
Control signals are easily identified; each of the models we study, two clear clusters of signals are present, with the larger cluster corresponding to control signals. The functional distinction between control signals and data signals concerns the source token that the head is attending to. When source token is the first token (in GPT-2), the first token punctuation token (in Pythia and Gemma), the signal that causal for attention is from the control cluster. When the source token not one of those, the causal signal is data signal.

---

Hence, control signals are the mechanism implementing attention sink behavior [52-54] at the signal level. The attention sink phenomenon is the tendency of models attend to an initial token when the head does not need to move information into the destination token. The Softmax operation imposes an attention distribution over sources, so the head uses a semantically unimportant token as source in these situations. We observe that typically more than 80% of all token pairs attended to in our experiments are due to attention sink.
Control signals come in pairs: one that resides in the relevant source (eg, start) token, and one that resides in every token as a potential destination. We find that control signals are the causal mechanism for attention sinking; they are strongly predictive of attention-sink behavior, with F- scores of 0.965 (GPT-2), 0.952 (Pythia), and 0.994 (Gemma-2). In intervention experiments akin to those in $4, we observe that boosting the control signal in either source or target tokens causes the head to revert to attention sinking, essentially "shutting it down," regardless of the tokens presented.
Next, we show that most control signals belong to one of a small number of clusters and that attention heads are functionally organized according those clusters. In Figure 5 the top row shows signal clusters (source left, destination right) for the three models. Height corresponds to cosine similarity; note that similarity greater than 0.2 quite significant, and most clusters have internal similarity greater than 0.4. Hence, there are only a few really distinct control signals used in these models - less than dozen. More surprisingly. these distinct control signals generally are used in different parts of the model. The second row shows where each control signal from the top row is used in the model. We see that models are hierarchically organized into groups of heads that tend share a common control signal. The functional significance of this organization is an intriguing direction for further study.
Finally, we ask how models manage the control signals in tokens. We observe that control signals are added in layer O, either by the attention head layer or the MLP. Over the layers of the model, signal strength tends to rise towards the layers where the signal is used (as shown in lower row in Figure 5). As expected, source control signals have a strongly negative cosine similarity with non- start tokens, and vice versa; this is to avoid confusion in the attention sink process. Figures 24 and 25 in Appendix M show how control signals vary over the layers of the models.
6 Conclusions
In this paper we make a number of contributions. We define the attention-causa communication problem, and we develop a theoretically grounded, heuristic approach to address it. Our approach is based on two key ideas: first, singular vectors of the QK matrix should in some sense 'match' relevant features in residuals; and second, relative attention (which we define) provides a causally- useful linear ersatz for the nonlinear attention computation. We build on previous observations that attention generally sparsely decomposable in the QK basis, which leads to an efficient method for identifying oow-dimensional subspaces in which signals reside. These signals enable new, precise methods to expose important communication with the model, and expose mechanisms of global model coordination.
Limitations. Our approach, while promising, shares the quadratic complexity in the number of to- kens inherent in standard attention mechanisms. Additionally, the proposed solution to the attention- causal communication problem is a heuristic. While empirically effective, future work could explore more formally guaranteed methods as well as causality involving MLPs.
Acknowledgments and Disclosure of Funding
This research was funded by a grant from Open Philanthropy and by NSF award CNS- 2312711. Code for our method is available at https: /gtttub......gggg com/ wtt. http//aabbreeeeaeeeeeegae franco/ pinpointing-atterte
coan-causl-- communi catt in: its implementation was enabled by the Transformerl.ent library [64]. We thank Evimaria Terzi, Aaron Mueller, and members of our research group at Boston University for helpful discussions, and we thank the anonymous Ne referees for their feedback which improved this paper.
10

---

References
[1] Xu Pan, Aaron Philip, Ziqian Xie, and Odelia Schwartz. Dissecting query-key interaction in vision transformers. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages $4595-54631. Curran Associates, Inc., 2024.
[2] Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Talking heads: Understanding inter-layer communication in transformer language models, 2024.
[3] Gabriel Franco and Mark Crovella. Sparse attention decomposition applied to circuit tracing, 2024.
[4] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
[5] Trieu H. Trinh, Yuhuai Wu, Quoe V. Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):4764455 2024.
[6] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Bryn- jolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Frmon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. On the opportunities and risks of foundation models. CoRR, abs/2108 07258. 2021.
[7] Zachary C. Lipton. The mythos of model interpretability, 2017.
[8] Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Zhang, Ruiqi Zhong, Seán Ó hffigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Aleksandar Petrov, Christian Schroeder de Witt, Sumeet Ramesh Motwan, Yoshua Bengio, Danqi Chen, Philip H. S. Torr, Samuel Albanie, Tegan Maharaj, Jakob Foerster, Flo- rian Tramer, He Hc, Atoosa Kasirzadeh, Yejin Choi, and David Krueger. Foundational chal- lenges in assuring alignment and safety of large language models, 2024.
[9] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation anal- ysis. In Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12388-12401. Curran Associates, Inc., 2020.
[10] Nicholas Goldowsky-Din Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching, 2023.
[11] Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods, 2024.
[12] Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: a method for rigorously testing interpretability hypotheses. Redwood Research.
[13] Atticus Geiger. Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. arXiv, 2021.
[14] Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 16318-16352. Curran Associates, Inc., 2023.
11

---

[15] Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language mod- els. arXiv, 2024.
[16) Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The Quest for the Right Mediator: A History, Survey, and Theoretical Grounding of Causal Interpretability. arXiv, 2024.
[17] Judea Pearl. Causality. Cambridge University Press, second edition, 2009.
[18] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg,
S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[19] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. OpenAlblog, 2019.
[20]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[21] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread, 2021.
[22] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds. Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy Models of Superposition. Transformer Circuits Thread, 2022.
[23] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024-000 001, 2020.
[24] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff, editors, Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746-751, Atlanta, Georgia, June 2013. Association for Computational Linguisties.
[25] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018.
[26] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Causal Representation Learning Workshop at NeurIPS 2023, 2023.
[27] Wes Gurnee and Max Tegmark. Language models represent space and time. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7- 11, 2024. OpenReview.net, 2024.
[28] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv, 2023.
[29] Amit Amold Levy and Mor Geva, Language models encode numbers using digit representa- tions in base 10, 2025.
[30] Joshua Engels, Eric Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensionally linear, 2025.
12

---

[31] Subhash Kantamneni and Max Tegmark. Language models use trigonometry to do addition, 2025.
[32] Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models, 2024.
[33] Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper, Max Tegmark, William Saun- ders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom McGrath. Open problems in mechanistic interpretability, 2025.
[34] Chris Olah. Mechanistic interpretability, variables, and the importance of interpretable bases. https: index. html.
circuits http:/ppp//eeeemmmeeeeeeeeeeeeee http://ecchhhitteee essay/
[35] Neel Nanda. A comprehensive mechanistic interpretability explainer & glossary. https: /www. nww.nendannnaaammmmmmmm wwwwecoaniiiii.cc..mcccc.......a htt.://antttttiiiieeeeeeeeeatata
[36] Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024.
[37] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nicholas I. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards monosemanticity; Decomposing language models with dictionary learn- ing. https::////. htt:///aanffommmeeeeeeeee cic circuitss....m... pub /223//222//// monosemantic features.
[38] Patrick Leask, Bart Bussmann, Michael Pearce, Joseph Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. Sparse autoencoders do not find canonical units of analysis, 2025.
[39] Lucius Bushnaq.
Lesswrong post. https:/www.eessrrogggcco cowp:///sssoooooooooooooosss www.aaaooooooooooooootttii. ootpssss-oettteee----ooooooooooo
cCgxp333g4a5925xxd// paociussbushnaq-se
wETE2ebypKZAdH8De
[40] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024.
[41] Bart Bussmann, Michael Pearce, Patrick Leask, Joseph Bloom, Lee Sharkey, and Neel Nanda. https::/www.eessvet
Showing SAE latents are not atomic using Meta- wwtp://ts/c///tccctttotootsttttt / 4nC wwc--uusingg-ee
SAEs. showing-saes leree-laeent---ee -ate---a---ttttt---------------- un melig--met-aee saes.
[42] David Chanin, James Wilken-Smith, Tomás Dulka, Hardik Bhatnagar, and Joseph Bloom. A for absorption: Studying feature splitting and absorption in sparse autoencoders, 2024.
[43]
bilalchughtai and Lucius Bushnaq.
Activation space interpretability may be
doomed. activation space- http:/acco.npppeeeeerrreeeeeaaoo -aalllyyyyyyy-yytimmee be doomed.
attps://wwer wttp/wwwwwww.w.....//ooo
[44] Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale, 2024.
[45] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[46] Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in pre-trained language model. arXiv, 2023.
[47] Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv, 2023.
13

---

[48] Aleksandar Makelov, Georg Lange, and Neel Nanda. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching. arXiv, 2023.
[49] Aaron Mueller. Missed causes and ambiguous effects: Counterfactuals pose challenges for interpreting neural networks. arXiv, 2024.
[50] Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. arXiv, 2023.
[51] Cody Rushing and Neel Nanda. Explorations of self-repait in language models. arXiv, 2024.
[52] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024.
[53] Nicola Cancedda. Spectral filters, dark signals, and attention sinks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pages 4792 4808, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[54] Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large lan- guage models, 2024.
[55] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.c 08829,,,, 2024.
[56] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2/ 2783,, 2024.
[57] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104
2021.
[58] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony. Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit. USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling, In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[59] Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. Advances in Neural Infor- mation Processing Systems, 36:76033-7606/ 2023.
[60] Chris Athwin, Guillaume Corlouer, Esben Kran, Fazl Barez, and Neel Nanda. Identifying preliminary circuit for predicting gendered pronouns in gpt-2 small, 2023, URL https:/cmmatww itch. io/identifying-ee prepeeiinnaayy-cirrttttttaa www.fffrr-rriiiiig--ee gen dered-pronouns. inn -rrpoouss--ingt------- 2pt-2-smal.
[61] Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Dangi Chen. Finding transformer circuits with edge pruning. Advances in Neural Information Processing Systems, 37:18506- 18534, 2025.
[62] Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. In The 7th BlackboxNLP Workshop, 2024.
[63] Curt Tigges, Michael Hanna, Qinan Yu, and Stella Biderman. LLM circuit analyses consis- tent across training and scale. In The Thirth-eighth Annual Conference on Neural Information Processing Systems, 2024.
[64] Neel Nanda and Joseph Bloom.
Transformerlens.
com/
Trans www.eeeeeeeeeerrrrrrareeeteeeeee
2022.
[65] Philipp Schmid, Omar Sanseviero, Louis Castricato, and Pedro Cuenca. Welcome gemma 2 google's 's new open llm. https: hww.g/gggggggc..eeeee/o// co /ttp//lg//////////// gemma,, 6 2024. Accessed
on: 2025-05-14.
14

---

A Further Background
Here we provide further background on model computations and associated notation.
To compute the output of the attention head each input token is first passed through an affine trans- formation:
XW + 1by with Wy E IR by E R. Attention weights are then used to combine rows of V construct the attention head's ' outputs: Z AV, with Z E NXR Next each head's output is passed through a linear transformation yielding the per-head output O = ZWo which transforms it back into the D-dimensional embedding space, with O RNXD, For given layer the final output of the attention block is then the sum over heads plus a per-layer bias:
o = Σ 0 -1b
(10)
150SH
For given attention head, we will at times need to decompose the per-head output O = AVWo into the portions contributed by each source token s. Let fad E R denote the output of head (C. a) for token d. This corresponds to row d of O Let Ad E RV be the d-th row of the (,, a) attention matrix, V E N be the value computation in head (f, a), and Wo E R.Rx be the output matrix of head (f, a). Then the per-source output is:
Lods V.Wo
(11)
where V, is row V. Note that we have fad EISISN tods with o Eads 0 when > d due to masked self-attention.
SVD. Our methods make of the SVD of Ω. The matrix l has size D D, but due to its construction it has maximum rank R. We therefore work with the SVD of Ω = UEVT T in which U E RR PxR, V DxR and Σ E _Rx U and V are orthonormal matrices with U I and VV = I, and Σ diag(Ø), ,2....,0R with 2 02 2 ... 2 2 0. Important to our work is that the SVD of Q2 can equivalently be written
R Ω
Eu'o o'
(12)
A=1
in which {u²} and {v²} are orthonormal sets and each term in the sum a rank-1 matrix having Frobenius norm dh-
B Bias in Attention
Here we discuss how to handle models which the attention head's key-query computation includes bias terms.
In such model, each token RD is passed through two affine transforms given by x WK bk- WQ + bj. using weight matrices WK. Wq e R and offsets bx. bo R Then the inner product is taken for all pairs of transformed tokens to yield attention scores. Specifically:
A'ts
Tw + (xx Wx + +T))T
x WoWWW + x w + bJwje + bJbk
(13)
in which xd is the destination token and x" is the source token of the attention computation.
As a result, for models with bias we will use two versions of (13). We define Ωa =
WW and we use x denote bo Wx
[W Wobkl.
Then:
1
A/e 2 + bJ + bobk:
(14a)
Ales = xdd O,xx + x Wobk + boox.
(14b)
15

---

Given these new definitions, when expanding (2) using (14b) we still have that:
lads
x Ed'T
0
TT 0xx00
(15)
jSd J
However, when expanding (2) using (14a) we have that
gods x TTT 88
x RTT
R 28 +
(16)
d
378
where 6 = W - Ejjs by W To understand the significance of this term, note that o₁ is only a function of source tokens; arises due to the inner products of source keys with the constant query bias.
When determining the subspaces corresponding to signals as described in $2, we use the SVDs of Qa S2 appropriate to the setting. Specifically, when determining Py we use
gods
C1 =
Σ SSARR
PTT U
x
URORV
(17)
d
d-1
35
where the SVD of S2g used. Note that can ignore the ei term because it only arises due to source tokens, and so adds a constant that can be ignored when defining Pv for decomposing the destination token. The terms from (17) chosen using same rule as in $2, namely the smallest that equals or exceeds cads C1; these terms are used to define Pu E RDxD in terms of the left singular vectors {ux} of Qag When determining Py we use
Lads
Σ IEKER
x tdT
UNORV
xdd Td
ub
(18)
OSd
where the SVD of Q2 is used. Here again we select the smallest set of terms that equals or exceeds Eods and use those determine Py in terms of the right singular vectors of Ω.
To perform communication tracing as described in $4 requires the following adjustment to (7);
geade (w) w S
w
(19)
d
jSd
778
and
s lads (W) x Edd $250 W
1
Σ
x
edd S
(20)
d
Sdd
With these new definitions, we still have
Lads lads (Xt)
(21)
however, also have
Lads glods Rd) + C1.
(22)
Hence to perform communication tracing on the destination token, we look for the smallest set of upstream components whose portions of the contribution sum to Lnds the 01 term because it arises due source tokens, and we are tracing inputs to the destination token.
Here again we ignore
Finally, one complication that arises is that it can happen that even though 4 > 1/n for some token pair (d, s), the quantity d'ads (xfd) = fads ca is not positive. If this occurs, it is because the bias term e₁ responsible for the positive value of lads leading to A 1/n. In this case, we simply ignore the fact that A 2 1/n - ie, we not compute signals, nor trace upstream from this (E, d, s) tuple.
16

---

C RoPE
In models using Rotary Position Encoding (RoPE) [57], before computing the attention score, each token has a set of rotations performed that depend on the token's 's position in the prompt. The result that for a given pair of tokens (d, s), can capture the effect of RoPE using rotation matrix R(d-s) IR RRR where (d s) is used to denote the relative positions of the and tokens in the prompt. As result, we must define ! with respect to (d -). Specifically, we define
s).
() s) Wo Wo
Then rewrite the attention score computation
A/as = x (
(23)
(All the work in this section will be in the context of given head (L, and will drop the indices for the head so for example Lade will be written as just cda.)
Expanding (2) in terms (23) we have that:
a dd 1)d-8)x d
dd D(d=5)x
(24)
JSd
8
in which we have dropped a constant term that is handled similarly to ci in Appendix B.
It's longer possible to write (3) in its given form because de involves multiple, different Ω matrices. However, our goal is still decompose the contribution de terms of the singular vectors of 1(d=s)) Hence we proceed by defining the projection matrices associated with each singular (d=
vector of ((d=s); Pa. u (u and Po.. k - (v ( Beeause {u (d(-)))) and
{v (v(d)))3 are orthonormal bases for RD, we have that x Σn Pop X Pu, for any x. We can now formulate (3) in two equivalent ways:
T Σ Xu
do
Σ ISKSR
Prpxe
o)
P(d-11)
d
35d
ISkSR
= Σ ISkSR
Th Pure s)
Pug D
(25a)
jcd
378
de TT ((d=s))
Σ Poix"
Σ
xT (((-))
Σ Pox
d-1
ISKER
isd
ISKER
348
x ol) P Po,
Σ jSd
x ((d-)) P. X
(25b)
ISKER
When determining the subspaces corresponding to signals, we proceed with separate analyses for source and destination (similar to the process in Appendix B). We choose the smallest set of terms of (25a) summing to dd define Pus, and the smallest of terms of (25b) summing to eda define Pv.
To perform communication tracing as described in $4 requires the corresponding adjustments to (29) and (30);
(d5(w) = w [)(d-6)
Σ w T (2id=1)
(26)
d
15. d 378
17

---

and
de (W) = x slddddo
Σ xd g(
(27)
d
15d
after which the tracing procedure is unchanged.
D Proof of Lemma 1
Here we prove generalization of Lemma 1.
Lemma (generalized), Let n < N be equal to the number of tokens considered in self- attention at head (l, a) for destination token d, and gads be defined as in (2). For 0 < a < 1, if Ads 2 then Lads vR In
+ Im(nn 1)).
Proof. Assume d > 1. (If 1, the destination is the first token or the BOS token, which always places full attention only itself.) By self-attention, we always have d. Accordingly is equal to (so by assumption, n > 1.)
Expanding the computation of attention weights from scores:
exp(A'10/ /V))
exp( /v Ejj, exp(A' /R) + (
Aas
So if Ade 2 a, then
exp(AAas/ /alla//Vi VR)
Σ
exp(A's) / /R).
a
By Jensen's inequality
Σ
Σ
Ala/V Al//VR
exp
11
8$8
148
SO
1 EA.
exp(A'r,/ Av////RR) R)
(n 1) exp
jf8
N/a/ // In 1-cc + In(n 1)+
Σ N
n
Eads
/R In
(1-a) + In(n-1)
So we see that sufficient condition for fads to be positive is that In(n - 1) 2 In (12) With positive n and o, this condition is met when a: 2 1/n.
An important corollary is the contrapositive:
18

---

In (1-a) + In(n 1)) Ads <
Luds
R
Thus place upper bound the attention weight function of cods.
exple /VR)/mn <a(1-a)) = Ads <a
or
exp(cfode /VR)
<a Ads <a
(28)
71 - 1 exp(clad /VR)
In $4, consider the effect of upstream components downstream gads Here note that (28) shows the precise nature of the nonlinear impact of upstream contributions on attention weight. The relationship (through the logistic) suggests that more sophisticated approach (e.g., Shapley values) may be beneficial in determining the impact of each upstream component. We leave this extension for future work.
E Proof of Lemma 2
Lemma 2. Given vectors x and among all rank-1 matrices having unit Frobenius norm, the matrix D that maximizes x Dy is ||x] [
Proof. Given vectors x and y, among all rank-1 matrices having unit Frobenius norm, the matrix D that maximizes x Dy is D [ y
be expressed the outer product two unit-norm vectors. Consider a rank-1 I matrix having unit Frobenius norm. Since X is rank-1,
First show that any rank-1 matrix having unit Frobenius
write X -xy_. Now construct X ||*|| lly! By construction X is both rank-1 I and unit norm. Matrices X and X differ by constant factor ||x|||y| However, since they have the same norm, we must have ||x|||>||| = 1, and so can be expressed as the outer product of two unit vectors.
Next consider a unit-norm, rank-1 matrix G uv for unit vectors u and By way of contradic- tion, suppose x Gy > x Dy. Then x xuuv y > x [[x liy y- The right hand side is the positive quantity |||||||||| The left hand side is the product of the projections of onto u, and y onto v. The product is maximized when x/||x||, [ y/lly]], or x x/||x| v= -y/llyyl In either case, x Gy = xx1 Dy.
F Details of the Experimental Setup
Indirect Object Identification (IOI). The Indirect Object Identification (IOI) task [45] uses prompts that are structured as in this example: "When Mary and John went to the store, John gave the drink to".. In this scenario, the model's goal to predict Mary, which the indirect object, rather than John. We used the authors' code to generate 256 prompts for this task, using a mix of the ABBA and BABA templates.
Gender Pronoun (GP), In the Gender Pronoun (GP) task [60], the model is given prompts of the form: "So John really great friend, isn't". The objective for the model to predict the correct pronoun, which would be either he or she. We used the authors' code with the 100 provided examples.
Greater Than (GT). The Greater Than (GT) task, as described by [59]. involves prompts of the form: "The attack lasted from the year 1920 to the year 19". In this task, the model's 's objective is to predict any two-digit number greater than 20. We utilized the authors' provided code to generate 256 prompts for this specific task. However, Gemma-2 2B was not analyzed for the GT task. The primary reason is that Gemma-2 2B's tokenizer predominantly uses single-digit tokens for numerical values,
19

---

AH11 7)
0.4
Density
0.2
AH13 09
AHE7
0.2
AriB. 211
AH19. 3)
10
20
30
Number singuar vectors (s
singuar vectors (5/00)
Figure 6: Dimension of signal subspace for key heads the GT Task: (a) GPT-2 (b) Pythia.
AHO0,
0.-
AHG.
1.5
AH12. 7)
1.3
AMH
APS3.
AML2
Dandky
AMX
Dero
0.4
ARE7
Demsty
1.0
AN21
1.0
10)
AHZ2
0.5
0.5
0.0
0.0
0.0
0.0
2.5
7.5
10.0
10
1.5
20
70
40
80
Number of singutor vectors (5
Number of Singubor voctors in (5
Number Singular vectors in (5
Figure 7: Dimension of signal subspace for key heads in the GP task: (a) GPT-2 (b) Pythia (c)
Gemma-2.
which leads to two-digit numbers being a sequence of two separate tokens. Such a fundamental difference in number tokenization makes direct performance comparisons on this task challenging and potentially misleading.
We use the TransformerLeens library [64] for our experiments. To prop- erly attribute contributions of upstream heads to downstream relative attention, we need to take into account the effect of the (downstream) layer norm. For GPT-2 and Pythia models, the layer norm operation can be decomposed into four steps: centering, normalizing, scaling, and translation. Gemma-2 models apply the same steps except for centering. Centering, scaling, and translation are affine maps, which means that they can be folded into different parts of the model with mathematical cquivalence. The TransformerLens library handles the centering step by setting each weight matrix that writes into the residual stream to have zero mean. Moreover, it folds the scaling and translation operations into the weights of the next downstream layer.² The result is that centering, scaling, and translation make changes to the matrices used compute L as shown in (1). The remaining step is the normalizing step. This step does not change the direction of the residual; it only affects the mag- nitude of the contribution the relative attention (33). Since for any relative attention calculation, we are considering specific addition to the residual o² we can simply scale its contribution by the same scaling factor used for the corresponding token x" when it input to the downstream layer.
Handling Layer Norm
G Sparse Attention Decomposition
Here we provide additional examples showing that sparse attention decomposition holds across all tasks, models, and heads we examine. Figure 6 shows typical results for key heads in each model on the GT task, and Figure shows typical results for key heads in each model on the GP task.
Figure presents typical results for key attention heads in each model, specifically for the IOI task. For models like GPT-2 and Pythia, which have an R value of 64, attention scores for various attention heads with distinct roles are typically computed using fewer than 20 dimensions, and sometimes even fewer than 5. A similar trend is observed in Gemma-2, which has a larger R value of 256.
H Details of Tracing
Here we provide more detail and formal definitions to supplement $4. The functions dluds (w) and lads (W) are defined as:
SSee https / ott Lt
further commentemmettteemeeeee and for more details.
20

---

0.20
AH18. 61
0.15
AH12. 6)
AH112:
Denstty
0.35
A054 13)
AHE3 01
0.10
AINA, 11]
Denalty
0.04
ANELO. 6)
AM2,
AM53.
0.05
0.05
0.07
0.00
0.00
0.00
Number singulter leggler vecroos (5%)ng
Number sing wectors in (5%
Number of singuaa vectors in [5hme)
Figure 8: Dimension of signal subspace for heads in (a) GPT-2 (b) Pythia (c) Gemma-2, IOI task.
deads
(w) (lad((w)= w T
Σ w
(29)
d
where w E R.,, and
8 lads (W) = Edd T S2
Σ
x TdT
(30)
d
JSd
where W& E NXD
The computation of upstream contributions to relative attention relies the linearity of deuds (w) and 5 fadds (Wad to distribute these functions over (6). Specifically, for any given contribution (char- acterizing an attention score) between tokens and at head (l, a), compute the portion of that contribution due to each upstream model component's writing into the destination token d as
All outpuis
FFN outputs
biases
imput layer
Lads
ddeado (x (d)
Σ
E
E
Geads (hdte
Σ Isk<
drado (pld)
Σ iske
glade (bb)) +
dlads (x)
(31) and we write the portion of that contribution due to each upstream model component's writing into the source token s as
AH outputs
FFN oulputs
atin, biases
input at layer 0
gads sadd )xx)
Σ
Σ
gads ( (hatt +
Σ
bads F
(F
fads (1b') +
Bads (Xx)
ISICE EEHHH SASN
isice
ISICE
(32)
where Ubeee E RDN is the matrix for which row d is lhdt and F E RND is the matrix for which d is fid.
By (5), (7) and the properties of the SVD, we have eads dlads = fads (Pux and Lads z slads(St) Eads (X+ (PPP)
(33)
The process of distributing (33) over (31) and (32) results in, for the destination token (repeating (9))
Lads
Σ
Lads Ohht
Σ
fadd
Edfado + dgads
(34)
let
let
And for the source token is:
Lads
Σ KELSSHINN
Σ
Sihtt
Eads
s/s
Lads
fads + si
Lads
(35)
Ise
let
Where the terms in (34) and (35) are defined as: that foods (Pyoo ) and
stads cads Plypld)
dtade glads (Pubbo):
Hads llado gladds(Pur (d)
Eads
lads (Othet PT)), Lads
s lads (F PT); Lads tads (1bb PJT), 80 lads = ladds x2P5))
8that
where Uure is the matrix for which row d is o lhdt and F is the matrix for which row d is f'd.
21

---

Constructing Communication Graphs Communication graphs are constructed as follows. The objective is to identify the causal communication pathways within the model that lead to a particular output. We formalize this by first defining an "output of interest" based on the model's final logit predictions, y. This output task-specific metric, calculated as a linear combination of logits, which can be represented as a This formulation captures common task-specific success metrics.
For the Indirect Object Identification (IOI) task, a y represents the logit difference between the indirect object (IO) and subject (S) tokens.
For the Gender Pronoun (GP) task, it is the logit difference between the correct and incorrect gender pronouns (c.g., "he" versus "she").
For the Greater Than (GT) task, it corresponds to the average logit difference between token predictions for two-digit numbers greater than a base number YY and those for num- bers smaller than or equal to YY.
Using the vector a and the model's 's unembedding matrix Wy D (where [Σ| is the vocabulary size), we derive a "success direction" vector g Wya a E IP in the residual stream space. For the IOI task, this would be g 1/ W U where Wt is the column of Wo for token t. We then analyze the model's 's final residual stream output for the token position being predicted, which denote xour The projection of this vector onto the success direction, x g. quantifies the model's performance on the task for that specific prompt. We trace this value back by measuring the contribution of each component from the residual stream decomposition (Equation (6)) this projection. "Seed" components are those with the largest contributions, specifically the smallest set of components whose contributions sum to the total projection value, x="i g. We also include any single component whose contribution is at least half of the total. If no attention heads are present among these seeds, tracing not performed. The token at the output position serves as the initial destination token d for tracing, while the source token for seed attention head is identified from its OV circuit decomposition.
Starting from the identified seed components, we recursively trace their upstream contributors. Typ- ically, the terms in equations (9) and (35) will show few large values and many small values. As result, it is important to filter these terms to isolate the important communication taking place in the model. To do so, we again rely on the property ensured by Lemma namely, that ofdd is a positive quantity. For each of (9) and (35), we select the smallest set of terms that sum to fadd for some < S 1. The parameter 3 is set based the degree which the less important edges should be filtered from the communication graph. In practice, we use 3 = 0.7 in all our results, which filters most of the low- low-weight edges while preserving the largest- edges. Algorithm outlines the pseudocode for the complete process of construction communication graphs
I Communication Graphs
In this section we provide details on how construct communication graph, and we show commu- nication graphs for example prompts from the tasks and models we study.
We generated communication graphs and circuits for specific prompts that met two key criteria:
1. The model successfully predicted the correct answer for the prompt.
2. The prompt has at least one attention head in the "seeds" components set.
The total number of prompts that satisfied these conditions and were traced for each model and task is detailed in Table 1.
On CPU hardware (machines with 28 cores), tracing 22-token prompt (the largest prompt size across all tasks that we used) takes approximately one minute for GPT-2 and for Pythia, and about one hour for Gemma-2. Our code not highly optimized, and significant improvements are possible.
The time complexity for tracing a single prompt is O(a I. T.-C) C.. In this expression, where T, is the number of layers, and C denotes the number of components in the model, ie, attention heads, attention biases, MLPs, and embeddings (see Equation 6). The factor a represents the number of significant attention values (i.e., those with Ata n. it upper-bounded by the number of attention
values in auto-regressive model N + N, but is in practice much less because only heads
22

---

that make causal contributions downstream are included in the trace. Thus, model size differences explain the increased tracing time for Gemma-2 compared GPT-2 and Pythia.
The following figures illustrate the communication graphs used by various models to solve example prompts from three distinct tasks:
Indirect Object Identification (IOI): For the prompt "Then, Simon and Andrew were working at the restaurant. Simon decided to give a basketball to"., communication graphs are shown for GPT-2 small (Figure 9), Pythia-160M, (Figure 10), and Gemma-2 2B (Fig- ure 11).
Gender Pronoun (GP): The communication graphs for GPT-2 small (Figure 12), Pythia- 160M (Figure 13), and Gemma-2 2B (Figure 14) correspond to the prompt: "So John is a really great friend, isn't".
Greater Than (GT); For the prompt "The consultation lasted from the year 1673 to the year 16", communication graphs are presented for GPT-2 small (Figure 15) and Pythia- 160M (Figure 16).
J Interventions
intervention targets an edge connecting an upstream (signal-writing) node and a downstream (signal-consuming) node. The intervention removes the signal from the downstream node's com- putation. Specifically, for destination token edge, as characterized by Equation (9), the signal is removed from the query computation; for source token edge, as characterized by Equation (35), the signal is removed from the key computation.
We implement two types of interventions: boosting, which involves adding the signal to the down- stream attention head, and suppressing, which involves removing To evaluate these, we compare interventions using signals derived from Stads (i.e., the singular vectors identified by our method) against a random baseline, which done by selecting [gtads] random singular vectors that are not in gtads
The effectiveness of an intervention is measured by the error -(E.N)-F, where F(E, h) denotes the F logit difference metric after the intervention, and F represents this value before the intervention. A negative error value signifies that the model's performance has gotten worse, while a positive value indicates improvement.
Figure 17 illustrates these intervention results across three models and all three tasks. We observe that interventions using our signals are more causal than the random baseline. We further illustrate in Figure 18 that interventions have exactly the effect predicted by Lemma - namely that signal ablation reduces attention and signal boosting increases attention. Finally, in Figure 19 we show the very small changes in vector norm and vector direction that result from these interventions.
In terms of time efficiency, performing an intervention requires approximately the same duration as a single forward pass through the model. To illustrate with our largest conssisinn of 256 prompts, each 22 tokens long executing an intervention on 28-core CPU machine takes a few seconds for GPT-2 and Pythia, and around minutes for Gemma-2.
K Circuits
Leveraging the observation that interventions can causally affect model performance, we use this in- sight as basis for identifying circuits. Our method for finding circuits from communication graphs begins by aggregating multiple such graphs into a single graph, G. From this aggregated graph, we first prune nodes that appear in less than 1% of the instances. Subsequently, edges are removed based on their impact on downstream task performance. Specifically, an edge E' that occurred in
prompts is removed from its average intervention impact-calculatee calculated as 1 Σ. -(E.A))F F. falls below threshold T. this formula, F; and F.(E,,, h) represent the logit difference metrics before and after the intervention for prompt i, respectively, and 7 is a threshold determined empir- ically for each task and model. Figures 20, 21, and 22 shows the precision, recall, and F -soress, respectively, considering every possible method as a baseline for comparison.
23

---

L Handling logit soft-capping in Gemma-2 models
Gemma-2 models have soft-capping for attention scores and logits. The soft-capping given by
f(x) tanh
where e is the soft-capping constant in Gemma-2 [55]) and are either the scores of the logits. To handle non-linearity in our tracing, we used the first non-zero term of the Taylor expansion for the logit soft-capping function, yielding the approximation f(x) R I (valid when is small). The Gemma 2 team observed very minor differences when soft-capping is removed during inference [65]. Moreover, we saw empirically that fx) ss x is a good enough approximation for the soft-capping. Figure 23 shows the attention scores and their respective values after soft-capping for an IOI prompt.
50 for attention scores and 30 for the final logits
M Control Signals
Figures 24 and 25 illustrate how control signals vary across the layers of the models study. In these figures, the colors used for signals correspond to those in Figure 5. These control signals were initially identified using the IOI task with a subset of the prompts. To verify their broader applicability, Figures 24 and 25 were specifically generated using a prompt not part of this initial set. Other prompts that not in this initial set also have very similar behavior. The consistent appearance of these signals on prompts that were not used to find the control signals corroborates their dnda--iddeeedeent nature.
We also show in Figure 26 the distribution of inner products between the V signals and zero and non-zero token signals for GPT-2, Pythia- 160M, and Gemma-2 2B. We observe that the V signals have considerably higher inner products with zero token signals than non-zero token signals across all the models.
24

---

Algorithm 1: Communication Graph Construction
Main function to find seeds and initiate tracing
ConstructGraph (M, a, B)
Input : M:: The transformer model.
P: The input prompt.
A vector defining the task metric over the logit vocabulary.
3: The contribution threshold for filtering (0 < 3 < 1).
Output: G: The final communication graph.
G t InitializeEmptGrraphhi
Activations, < ForwardPass(M P);
Wo +- GetUnenbeddingMatrik.c
g - Wya > Calculate the "success direction" vector
xou GetFinalResidual/ P, Activations);
PerformanceScooee +- ouugg;
// Decompose the final residual and project contributions onto the success direction
Contributions DecomposeResiduall ( Using Eq. (6)
ProjectedContributions t {c. g for each Contributions):
// Identify the smallest set of components whose scores sum to the total score
< FindSmallessSSuseeSSun CarreuustssuuuupppeeeeeCCCCoooo ProjectedContribution.. PerformanceScore):
// Also include any single component that contributes more than half the total score Shigh + {c, 1 0.5 x PerformanceScore):: Seeds + UUUSSSS Shight
11
12
13
foreach seed component (l, a, d, 8) E Seeds do if (f, a) is atlention head then Recurs i veerace (G, & a, d, 8, 3);
14
15
16
return G;
// Recursive helper function to trace upstream contributors RecursiveTren (G, E, o, 3)
Input : G: The graph (modified in-place).
(E, a. d, s): The current component (layer, head, destination token, source token).
3: The contribution threshold. We use 0.7 the experiments.
Output: Modifies the graph G.
b Base Cases: Stop recursion for all these cases or for when attention weight less than 1/n (see Lemma 1)
l or d or d or Afa 1/n then
20
return;
21
Lads + RecallateRelaiieAAttteeee
a, s);
/1 Find upstream contributors to the destination token's signal
22
(d) + FindUpstreanConnribttrsss
clads ,B) & Using Eq. (9)
foreach upstream component (E', h' d, t) (d) do Add edge (f,,'',,t) (t, a, d, 5) to
24
25
RecursiveTrace (G, h', t.
// Find upstream contributors to the source token's 's signal
26
U(s)+ FrindlpstrrammContibbuue
8, eduds (3) Using Eq. (35)
foreach upstream component (E', ht 8, E (s)
27
28
Add edge (f','' s, (E, d, to G: RecursiveTrace (G, E B) & Source token destination upstream.
29
25

---

Table I: Number of Traced Prompts per Model and Task
Model
101
GT
GP
GPT-2 Small
230
166
100
Pythia- 160M
159
39
99
Gemma-2 2B
206
94
26

---

EAEE
Figure 9: Communication graph used for GPT-2 small to solve IOI prompt, with 247 nodes and 683 edges. The prompt used is: "Then, Simon and Andrew were working at the restaurant. Simon decided give basketball to".
27

---

Figure 10: Communication graph used for Pythia- to solve an IOI prompt, with 158 nodes and 344 edges. The prompt used is: "Then, Simon and Andrew were working at the restaurant. Simon decided to give a basketball to".
28

---

Figure II: Communication graph used for Gemma-2 2B solve an IOI prompt, with 352 nodes and 1358 edges. The prompt used is: "Then, Simon and Andrew were working at the restaurant. Simon decided give basketball to"..
29

---

Figure 12: Communication graph used for GPT-2 small to solve GP prompt, with 87 nodes and 152 edges. The prompt used is: "So John is really great friend, isn "t
30

---

Figure 13:
Communication graph used for P6thia-160M to solve GP prompt, with 86 nodes and 172 edges. The prompt used is: "oo John really great friend, isn't".
31

---

Figure 14: Communication graph used for Gemma-2 2B to solve a GP prompt, with 38 nodes and 31 edges. The prompt used is: "So John is really great friend, isn on't".
32

---

Figure 15: Communication graph used for GPT-2 small to solve GT prompt, with 150 nodes and 412 edges. The prompt used is: "The consultation lasted from the year 1673 to the year 16".
33

---

Figure 16: Communication graph used for Pythia 160M to solve GT prompt, with 94 nodes and 189 edges. The prompt used is: "The consultation lasted from the year 1673 to the year 16".
34

---

gpt2-small
pythia-160m
gemma-2-2b
S-Inhibition Head Name Mover Head
Induction Head S-Inhibition Head
Previous Token Head Induction Head
0.5
0.0
0.5
1
0
-0.01
0.00 0.01
('end', 'end") ('end', 'is') (1)
('end', 'end') -> ('end', 'is') (2)
G
(fis', 'is)) -> ('end', 'is')
0.1
0.0
0.1
-0.5
0.0
0.5
-0.02
0.00
0.02
('end', 'end") -> ('end', YYY)
Removing (Random)
t ('Y,, 'XXI') =>
Boosting (Random)
('end', (YY) (1)
Removing (SVs)
Boosting (SVs)
(YY,, (XX') ->
('end', "YY") (2)
0.0 0.5 (F(E, h) - F)
-2
(F(E. h) - F)
Figure 17: Intervention effect on GPT-2, Pythia, and Gemma-2 in the IOI, GP, and GT tasks. Green: signal ablation; Red: signal boosting; Blue: random ablation; Orange: random boosting.
gpt2-small
pythia-160m
gemma-22 2-mba
Ads
0.5
IO1
0.0
Interv
-0.5
Ads
0.5
GP
0.0
interv
-0.5
Ads
0.5
Removing (Random)
GT
0.0
Boosting (Random)
interv
Removing (5Vs)
Boosting (SVs)
-0.5
Edges
Edges
Figure 18: Interventions effect on the attention weight. Error bars are the standard deviation.
35

---

6030
1020
1000 400
900
220
1.005 3.200 1.075
Figure 19: Distribution of cosine similarities and norm ratios between the intervened input residual and the original input residual (a) SVs (b) Random (c) SVs (d) Random.
Taskk: gp: Model: gpt2-small
Task: gt: Model: gpt2-smal
Task: lol; Model: gpt2-small
th-0 Fruning ACDC
ACICC
0:53
0:31
0.23
0.41
0.9
EAP
0.89
0.6
0.83
Mastod
Ou'S W
EAP
0:09
0.76
0.6
0.8
Ours er a
Edge
0.68
0:52
0.8
0.6
Patching
Ours
th-
Edge
Ryth
0.50
0.78
0.54
Potching
0.4
ACDC
ACDC
Edge Fruning Method
Ours wr t1w0.2
01w0.2 Pasching
Mcthod
Mochod
Figure 20: Heatmap with Precision scores considering Method as a baseline for comparison for (a) IOI task (b) GP task (c) GT task.
gp: gpt2-small
Task: gl: Model: gpt2-smal
Task: Model: aptllsmall
thi Fruning MCDC
ACDC
0.84
0.79
0:84
0.58
EAP
0.7
0:75
E4P
0.31
1
0.60
3,47
Mathod
Edge
0.55
0.8
Pruning
0:74
0.01
Ours wi
0.76
2.6
Ours wl
0.6
Ours
thwo.22
Path
0.53
0.6
6.53
0.1
Patching
0.15
0.28
0.73
2.4
0.5
ACDC
EAP
ACDC
Edge Pruning
Ours wt Ihm0.2
Our
01-0.2
Posching
Method
Mathod
Maches B
Figure 21: Heatmap with Recall scores considering Method A as baseline for comparison for (a) IOI task (b) GP task (c) GT task.
Task: gp: Model: small
gi: Model: gpt2-smail
Model: opt2-smal
Curs wi M th-0 truning ACDC
1.0
ACDC
0:76
0:79
0:65
0.71
ACDC
0:47
0.63
0:53
Hethod
EAP
0.47
0.72
2.61
Edge
0.63
0.39
Praning
0.79
0.81
0:65
0.55
0.7
0.7
Ours
0.25
0.68
ih-0.2
Potching
0.51
0:72
0.E4
0.6
Patching
ACOC
E3gc
Pacchnn
ACDCC
EXP
ACDC
Edge
Ours wi
Patching
Figure 22: Heatmap with F-scores considering Method A baseline for comparison (a) IOI task (b) GP task (c) GT task.
tanh
-50
-200
-100
100
200
A'g
Figure 23: The attention scores and their respective values after soft-capping for an 1OI prompt.
36

---

0.1
00
Dayer
Figure 24: Cosine similarity between destination control signals and the residuals of the last token. (a) GPT-2 (b) Pythia (c) Gemma-2.
Codlee centaree
Leye
LAFE
Laye
Dapt
Figure 25: Cosine similarity between source control signals and the residuals of the first token. (a) GPT-2 (b) Pythia (c) Gemma-2.
zero Intens
nonooweo tobers
Jen Toleens
noteoren tobers
Ioleers
Doniky
Dendfty
Deenalty
-0.5
-0.50-0225
0.00
0.25
0.50
1.00
-0.25
0.00
0.25
0.50
3.00
Inner product
Inner product
inner procuct
Figure 26: Distribution of inner products between the V signals and zero and non-zero token signals in (a) GPT-2 (b) Pythia (c) Gemma-2.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the claims made in the abstract and introduction are supported by Sec- tions 2, 3, 4, and 5, besides the Appendices.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our method in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificattin, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [es]]
38

---

Justification:
Sections 1-4 of the paper, alongside Appendices D. E, and H provide all the theory assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Besides describing in the main body of the paper our derivations and tracing strategy (Sections 1-4), we provide more details and a pseudocode in Appendix H. All the LLMs used in this paper are open source.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g. in the case of a large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
39

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code reproduce all the experiments and results of htt.ggitth..oooooeeeee htom//aabrieefiie geabrielfranco/
the paper available at pinpoint www.atttniinncssall-e-eee-- cnttmnuiiaiion. vide implementation details and a pseudocode in Appendix H.
Moreover, pro-
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (ht.t::: /ttpss//niii /ntps/ipp..... ccc public/ gu des CodeSu'om i ss onPol icy) for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips cc/ htpppulliio i c/ c/gu i des / CCodeSbmt s onPol icy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification:
These details are described in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
For the intervention results, we report the full distribution across different prompts (c.g.. Figure 2). When we use aggregated measures, we report standard deviations, such as in Figure 18.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
40

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors verified.
For asymmetric distributions, the authors should careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: These are shown in Appendices and J.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttt:::://eeurreee httpp///bll////hccccccooeeeeeee.
Answer: [Yes]
Justification: Our research exclusively uses publicly available open-source datasets and models, adhering to their licenses, and does not involve new data collection from human subjects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
41

---

Justification: As discussed in Section 1, progress on the interpretability of language models is critical for improving model safety and alignment. We do not anticipate negative direct impacts from an increased understanding of the model's operations.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release any data or model. All data and models used in this paper are open source.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The datasets and models that we use are open source, and we cite accordingly all of them.
Guidelines:
The answer NA means that the paper does not use existing assets.
42

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw raperswi wwwithoan thoode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- e-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
The code has README to show how to reproduce all the results of the paper.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
43

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if LLM used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did not LLMs for methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https : /tt///uriiii///////////// pps ps cc//Coneeenneces////////o 202. 2025/LLM) for what should or should not be described.
44


Reviews:
[{"review_id": "32aDHRFJEf", "reviewer": "Reviewer_Zss5", "summary": "This paper looks at the the low-rank structure present in the query-key computation of language models. The authors find that singular vectors encode signals across the layers that communicate and can be found automatically (see prior work)", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n* The proposed research direction is interesting and important. The authors do an adequate job of formalizing and exploring the ideas of attention-causal structure.\n* The proposed solution for materializing QK with models that use relative positional embeddings (RoPE) is a nice contribution and could be widely applicable to other interpretability work\n* The paper is thorough and provides a lot of detail in the appendices\n* The extension of the singular vector method analysis is to the GT and GP tasks are somewhat interesting, but are the expectation given prior work.\n* The authors choose good baselines for circuit comparisons\n\n# Weaknesses\n* Lots of jargon and formalisms mask that the paper essentially is a rehash of several prior works. While I think there are some interesting extensions on prior work [50, 51, 58], this paper provides some clarifying formalisms only to replicate **known findings**.\n\n## Known Findings:\n* Singular vectors of attention weight matrices encode distinct meaningful signals across layers [50, 51, 58]\n* Specifically, the IOI circuit is solved by communication across singular vectors in attention heads [50, 51]\n* [51] expands on the ideas of prior work by providing a data dependent measure for selecting singular vectors. This work provides their own version of this, but does not demonstrate why we should prefer there's and offers no novel insight \n* The \"signals in the stream\" analogy is just a rewording of findings from [18] and by extension, the analogy of low rank communication channels in [50]\n* Section 4: \"Tracing circuits\" is not a novel contribution and the language around circuit tracing is very similar to the \"circuit tracing\" and results described in [51] and does not expand on it meaningfully\n* Distinct semantic feature matching performed by singular vectors of [58 <- vision models]. A lot of the analysis resembles [58]\n\n\n**Overall, I can not recommend this paper for acceptance because of the striking similarity to multiple prior works**\n\n\n## other weaknesses\n* Figure 1 is basically incomprehensible. I don't think it is valuable as an illustration for the method. The colored bars are unclear and there is too much jargon frontloaded. I believe this figure should be remade to better use the space", "was_revised": true, "final_justification": "Thank you to the authors for engaging so much in the discussion and considering my points. After discussing with the authors, I'm deciding to raise my score. The main reasons are that the authors helped me understand the significance of their work more clearly, and I believe we have consensus on reframing these contributions more fairly to prior work. I do think the required revisions to the writing are significant, but are also doable in time before the camera ready deadline, and I trust that the revisions can be made. Thank you for the hard work in the short discussion period", "rebuttal": "We thank the reviewer for their careful read of our paper and the helpful comments. We appreciate the opportunity to clarify the points raised.  \n\nFirst we’ll address the main thrust of the reviewer’s concerns and then we’ll address the detailed points individually.  \nThe reviewer is concerned that our paper does not break new ground over [50, 51, and 58]. The closest paper to ours is [51], so we’ll start there.  \n\nAs the reviewer notes, [51] provides a data dependent measure for selecting singular vectors, which then exposes meaningful communication. The strategy taken in [51] is to select a set of singular vectors that contribute an arbitrary amount (70% of total) to **attention scores** (ie, pre-softmax scores – not attention itself).  \n\nFirst of all, the signals found in [51] have no provable effect on attention itself. Nothing can be said about how much of a head’s attention is actually caused by these signals. That is, there is no formal basis for, nor is there any provable result from the signal identification method used in [51]. As a result, there is no assurance about the nature of the set of signals found in [51] – there may be both false negatives and false positives with respect to attention causality, for example.  \n\nIn contrast, our paper starts with a formal problem definition of *attention-causal communication* (a concept that is a novel contribution of our paper) and then presents an algorithm that **provably** identifies attention-causal communication. This conceptual advance is important, because it provides a formal definition for what researchers seek in understanding why an attention head attends to a particular token pair. See, for example, the recent blog post “Progress on Attention”  in which prominent researchers point to this question as a key open problem in LLM interpretability (“Our biggest remaining question is how to understand *why models attend where they do*.”).  \n\nSecond, and related to this flaw, the method of [51] **does not work in general**. For example, it cannot be used on Pythia. We have worked with the method of [51] from the authors’ code and tried to use it on Pythia. We find that the key idea of [51] is based on assumptions that do not hold in general, and do not hold in Pythia specifically. This is perhaps not surprising for an approach with no formal justification. To explain, the key problem is that many attention scores in Pythia are actually *negative*. There is no way to apply the approach of [51] to situations where the attention scores are negative. The method of [51] fundamentally assumes that moving the attention score of a token pair towards zero will decrease the attention on the pair – but this is *not true* for negative attention scores.  \n\nOur development of relative attention as the proper measure for selecting singular vectors is therefore a critical advance over [51], and part of the reason it was possible for us to apply our method to all three of GPT-2, Pythia, and Gemma-2 (whereas [51] only worked with GPT-2). Furthermore, the decision to define and use relative attention is not obvious; it’s important that relative attention (a linear function) has a provable relationship with attention (a nonlinear function) as shown in Lemma 1. This fact is key to being able to attribute attention causality for a particular head to components upstream of that head, which relies on the linearity of relative attention.  \n\nThe reviewer states that our paper “does not demonstrate why we should prefer” our paper over [51]. This is a fair point. Thanks to the reviewer’s comments we realize we should have made these points clearly in the manuscript and we will definitely make these points clearly in the revision.  \n\nTurning to paper [50], indeed that paper shows the existence of low-rank communication channels between attention heads. It is important to observe that [50] only provides one concrete example for the IOI task. This example is in GPT-2 and is a particular communication from four inhibition heads to one name mover head. Importantly, as the paper admits, the authors were looking for this channel because they knew from prior work that the inhibition heads communicate with the name mover. This example underscores the fact that the method used by the authors in [50] is entirely **manual and heuristic**, based on examining plots involving pre-selected pairs of heads. The point is that paper [50] **does not provide**, or attempt to provide, **a general or automatic method** for identifying all the important communication occurring in the IOI task, nor does it attempt to provide a general method for identifying which attention heads are communicating. In contrast, our paper presents a systematic, general method, with provable results, for finding all the attention-causal communication present, and identifying all the heads that are contributing, and it does so without assuming that circuits have been traced in advance (unlike [50]). We mention here again that the analysis of the IOI task in GPT-2 does not generalize to other models without making use of significant new ideas (such as relative attention) that we contribute.  \n\nNext, regarding paper [58], indeed that paper focuses on semantic feature matching in vision transformers, performed using singular vectors. The main goal in [58] is to show that vision transformers do not just attend to similar objects, but also (in later layers) perform contextual analysis. As such, we believe that [58] is **complementary** to our work. The authors in [58] are focused on exploring the meaning of visual features that correspond to singular vectors. Our work does not dig into the semantics of features but instead focuses on systematically extracting all the relevant features (signals) that are causal for attention. On the other hand, the fact that [58] shows that so many features found using singular vectors of QK matrices are interpretable **adds motivation** to our method, since we show how to extract those (attention-causal) features systematically in a manner that is robust across models and tasks.  \n\nRegarding the bullet points:\n- *Lots of jargon and formalisms mask that the paper essentially is a rehash of several prior works. While I think there are some interesting extensions on prior work [50, 51, 58], this paper provides some clarifying formalisms only to replicate **known findings**.*  \nWe appreciate the opportunity to clarify the ways in which our work advances the field. We don’t claim that we are the first to observe the importance of singular vectors of QK matrices as tools for interpretability. Indeed, we cite [50, 51, 58] as important prior work showing that attention heads communicate in low-dimensional channels identifiable using singular vectors of QK matrices. The focus of our contribution in Section 4 is on showing how to turn this observation into a formalized tracing strategy that is more efficient and easier to use than prior methods (eg, more efficient than using counterfactuals and patching) and that has provable properties and is robust across model architectures (unlike [51]).\n- *Singular vectors of attention weight matrices encode distinct meaningful signals across layers [50, 51, 58]*   \nIndeed, as discussed above, this fact is an important background for our paper; we don’t claim this as a contribution of our paper.  \n- *Specifically, the IOI circuit is solved by communication across singular vectors in attention heads [50, 51]*  \nWhile both [50] and [51] look at the IOI task via singular vectors, neither offer a robust and systematic solution, as explained above. The other tasks we study (GP, GT) have not been addressed before, and showing that the singular vector method works well for those as well is important evidence for the robustness of the method.\n- *[51] expands on the ideas of prior work by providing a data dependent measure for selecting singular vectors. This work provides their own version of this, but does not demonstrate why we should prefer there's and offers no novel insight*  \nAs discussed above, we understand now that we did not explain in our submission why our method is a distinct advance over [51]. We explain that above, and plan to clearly articulate our advance over [51] in the revision.\n- *The \"signals in the stream\" analogy is just a rewording of findings from [18] and by extension, the analogy of low rank communication channels in [50]*  \nUnderstood. Removing it from the title would focus the title better on our paper’s contribution, which we will do in the revision.\n- *Section 4: \"Tracing circuits\" is not a novel contribution and the language around circuit tracing is very similar to the \"circuit tracing\" and results described in [51] and does not expand on it meaningfully*   \nWhile much prior work has focused on tracing circuits, the dominant strategies are still extremely time consuming and require the careful and subtle construction of counterfactuals to expose circuits. Tracing without counterfactuals, and in a single forward pass, offers the field a much more efficient tool.  \nAs described above, our work goes beyond [51] by providing a formal problem definition based on the new concept of attention-causal communication, presenting a provable and efficient solution to that problem, and developing new concepts (relative attention) needed for working with models beyond GPT-2.\n- *Distinct semantic feature matching performed by singular vectors of [58 <- vision models]. A lot of the analysis resembles [58]*  \nAs described above, as [58] focuses on interpreting the features found via singular vector analysis of QK matrices, we see it as complementary to our work, and providing motivation for the approach we take.  \n- Regarding Figure 1, we appreciate the reviewer’s frank appraisal that this figure doesn’t work. This is helpful criticism, and we can see in that light that Figure 1 should be removed and the space used more profitably."}, {"review_id": "iFhQrNTPRY", "reviewer": "Reviewer_5oqm", "summary": "This paper investigates the internal mechanisms of transformer-based language models by focusing on how low-dimensional features influence attention behavior. In particular, this paper examines how such representations function as attention-causal communication channels written into and read from token positions that have a provable causal influence on attention patterns. The authors then introduce a method that jointly analyzes model weights and representations (without relying on counterfactual interventions) to isolate these attention-causal signals. This enables prompt-specific circuit discovery and sheds light on latent mechanisms in attention.", "questions": "1. Line 201: feature set ${f^i}$, ${g^i}$: what are $f$, $g$ and $i$?  Please define these formally.\n\n2. What is the x-axis of Figure 2? Could you help parse the plot? \n\n3. Similarly can you help parse Figure 6 first row? What is x-axis for the dendrogram?\n\n3. How does attention-causal communication relate to or improve upon prior tools? Are there cases where prior methods fail but ACC succeeds?\n \n\n\n**Typos and Wording Suggestions**\n\nLine 48:  \"Our attack on this question\" & Line 138: \"We first attack the problem\": \"approach to\"& \"address\" or \"tackle\" is more appropriate.\n\nLine 179: “than” should be “then.”", "limitations": "Yes", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1.  The paper studies how and where task-relevant signals reside in LLMs. The idea of sparse decomposition to uncover key subspaces is conceptually compelling.\n\n2. The proposed attention-based tracing mechanism offers a new lens to analyze the flow of information in transformer blocks without counterfactuals, complementing existing interpretability tools.\n\n3. The paper evaluates the method across a variety of known benchmarks such as IOI and GP, demonstrating the potential generality of the idea.\n\n\n**Weaknesses**: \n\n1. The notational system, particularly in the sparse decomposition section, is dense and in places undefined or ambiguous. For example, $\\mathcal{S}^{\\text{lads}}$ (line 212) is introduced in natural language without a formal definition, which impedes understanding.\n\n2. Missing notation for key variables: $f^i$, $g^i$ (line 201) are used without prior introduction or definition, making the formulation difficult to follow.\n\n3. Lack of theoretical foundation: While the empirical motivation is strong, the paper lacks a rigorous theoretical analysis of the proposed decomposition. Key questions remain unanswered: for example, why should attention show sparse decomposition? (as proposed by the authors). Although lines 185–205 attempt to provide some intuition, the discussion is informal and primarily heuristic. Lemma 2 is a standard result in linear algebra and does not offer novel insight specific to the proposed decomposition. As a result, the interpretability claims remain largely empirical and would benefit from stronger theoretical support.\n\n4. The probing tasks (IOI, GT, GP) are referenced but not described. A brief summary of what these tasks entail would be helpful for readers unfamiliar with them.\n\n5. While the paper includes performance comparisons with other circuit identification methods, it lacks a clear conceptual and methodological comparison with widely used interpretability approaches such as attention rollout. This makes it difficult to assess the unique value and practical advantages of the proposed framework. Additionally, the method itself is quite complex (the calculation in Section 4). A more streamlined or intuitive explanation (especially from a practitioner’s perspective) would enhance accessibility and broaden the potential impact of the work.", "was_revised": true, "final_justification": "The authors clarified some of my questions regarding notations, which helped address some of my initial confusion. However, key concerns remain unresolved. Most importantly, the current form of the paper is not ready for publication: the notation system is messy, and the figures are difficult to interpret. While some of the results are conceptually interesting, the authors themselves acknowledge a lack of intuitive/theoretically grounded explanation, which limits the accessibility/rigor of the work. In addition, the absence of comparisons with relevant methods reduces the empirical soundness. Overall, while I see potential in the direction, the clarity, rigor, and completeness of the presentation need substantial improvement. Thus I am keeping my score and recommend rejection.", "rebuttal": "We thank the reviewer for their helpful review of our paper.\n\n### Weaknesses:\n1. Thanks for pointing this out.   We’ll provide a clearer definition of $S^{\\ell ads}$ in the revision.   Specifically, each term in (3) is the amount that one singular vector adds to the relative attention $c^{\\ell ads}$.  We take the terms from (3) and sort them in decreasing order, and find the shortest initial sequence of this list that sums to $c^{\\ell ads}$.  Because most terms in (3) are quite small due to sparse decomposition, our method gives a small number of singular vectors.   The indices of these singular vectors are $S^{\\ell ads}$. \n2. Thank you for pointing out the lack of explanation of notation.   We will add an explanation of this notation to the paper in the location you pointed out.   Briefly, the sets {$f^i$}, {$g^i$} are intended to represent sets of features present in a token.   Each feature, eg, $f^0$, is assumed to be a single direction in activation space, following the linear representation hypothesis.   We go into more detail in our next answer immediately below.  \n3. We appreciate the reviewer’s interest in theoretical justification for sparse decomposition of attention scores, which we share.   We recognize that, due to space limitations, we were not able to provide a more detailed explanation of why attention scores should show sparse decomposition.  \n\nWe start from the linear representation hypothesis, which holds that important features are often encoded in one-dimensional or low-dimensional subspaces. We provide a number of cites in the introduction to papers that demonstrate this phenomenon for an enormous variety of different kinds of features [12, 19-29].   Consider a specific feature, such as the fact that a token corresponds to a person’s name.   This feature may be represented by a 1-D direction in activation space.  We denote such a direction $f^0$.   Hence we can think of the model activation carrying this feature as $x = b + f^0$.  \n\nConsider an attention head, one of whose roles is to attend to the personal-name feature.  It calculates its attention score as $x^\\top \\Omega y$. Then the process of model training will tune $\\Omega$ and $f^0$ so that a left singular vector of $\\Omega$ will tend to be in the same direction as $f^0$.   This is the significance of Lemma 2.   Note that the other singular vectors cannot lie in the same direction as $f^0$, because singular vectors form an orthogonal set.   Hence, if the personal-name feature were the only feature that the head attended to, then decomposing the attention score $x^\\top \\Omega y$ in the singular vectors of $\\Omega$ would yield a sparse decomposition.  That is, the one singular vector that is in the direction of $f^0$ would yield a large contribution to the attention score, and the other directions would not contribute significantly.  \n\nTo understand how this extends to the situation where there are multiple features that the head attends to, we note that previous work [19] has shown that features that co-occur will naturally tend to be represented by orthogonal or nearly-orthogonal directions.   That is, features such as “personal name”, “place name”, “organization name” will tend to be represented by vectors $f^0, f^1, f^2$ which are mutually orthogonal or nearly so.   Then by the same argument, model training will tend to align different left singular vectors of $\\Omega$ with each of $f^0, f^1, f^2$, again leading to a sparse decomposition of attention when *one* of those features is present in the input $x$.  \n\nFinally, in the general case where more features are important to the attention head than there are available singular vectors (dimensions $R$), the tendency will be for features to be approximately orthogonal as explained in [19].   In that case, each feature will tend to align with a small number of singular vectors of $\\Omega$, still leading to a sparse decomposition of attention when one of those features is detected.  \n\nThe above is not a formal proof, which would involve defining a distribution over features and a model of the training process, and would be outside the scope of our paper.  Rather it is an explanation of why sparse attention decomposition can arise.  A formal proof, along with experiments that demonstrate the emergence of sparse decomposition during the training process, is a subject of our current ongoing work.  \n\nMoreover, we note that the authors in [58] make assumptions that essentially follow the reasoning we present above, although they do not lay out the reasoning in detail as we do here.\n\n4. Indeed, these descriptions are missing from the body of the paper. The tasks were described in Appendix J due to the lack of space, but we will add short descriptions of the tasks to the body of the paper to aid the reader.\n\n5. Thank you for pointing this out. In attention rollout, the identities of input tokens are linearly combined through the layers based on the attention weights. Thus attention rollout is a heuristic method to approximately track information flow through the model.  It does not attempt to identify causal communication. Rather, being a heuristic, it is typically used as a tool to make general observations about what tokens a model is attending to. In contrast, our method identifies the specific low-dimensional components of a token that are responsible for the fact that a head attends to the token. Accordingly, our method provides a more precise and reliable understanding of why an attention head attends to a particular token pair. \n\nWe agree that a more complete methodological comparison with attention rollout and other internal tracing methods would clarify our contributions. We also agree that having a more intuitive explanation of the entire methodology would make the paper more accessible and easier to understand. As discussed in our response to Reviewer 1, we believe we can use space currently allocated to Figure 1 to address these needs.\n\n### Questions:\n\n1. We address this question in answer #2 (and #3) above.\n2. Using the definition of $S^{\\ell ads}$ that we gave in answer #1 above, the x-axis of Figure 2 is $\\frac{|S^{\\ell ads}|}{R}$, where $R$ is the dimensionality of the attention heads (line 93) and consequently the rank of the $\\Omega^{\\ell a}$ matrix. Therefore, the point of Figure 2 is that, for multiple models and tasks, the attention weights are sparsely decomposable in the singular vectors, almost always using a very small number of singular vectors.\n3. The x-axis of the dendrogram represents all the control signals (D-dimensional vectors) used in a forward pass that were identified by our selection process (lines 332-333). The colors are used to represent the cluster that the control signal falls in, where clustering is based on cosine similarity.\n4. We appreciate the opportunity to clarify how using attention-causal communication improves over previous methods.  \nWith respect to circuit tracing, the dominant strategies used for circuit tracing are activation patching or path-patching [7, 8, 11, 42-44].   These methods are still extremely time consuming and require the careful and subtle construction of counterfactuals to expose circuits.   Each time one seeks a circuit for a new task, one must carefully design counterfactual task instances and then run many forward passes of the model, moving activations from the “corrupted” execution to the “clean” execution. Importantly, there are cases where activation patching can fail - instances where downstream behaviors by the model “undo” the effects of the patching.  \n\nOur method does not use activation patching, and so avoids problems in which downstream components “undo” the effects of the patching.   Further, it allows for tracing without counterfactuals; the only prompt needed is the one that is to be traced.   Finally, our method performs the trace in a single forward pass – which is much more efficient.  \n\nAs we discuss in our response to Reviewer Zss5 above, our paper also improves on the tracing method in [51].    Briefly, the method in [51] does not work for models in general – it only worked in [51] for the single case of GPT-2.   In fact, we have found that the method in [51] does not work for Pythia. Our method is based on a formal problem that we solve in a provable way, and thus works on any model (we show its success on GPT-2, Pythia, and Gemma-2). \n\n### Typos and Wording Suggestions:\n- Thank you for pointing these out. We will correct them in the revision."}, {"review_id": "WWPFlOCq8q", "reviewer": "Reviewer_kEJw", "summary": "This paper proposes an analysis of model weights and representations to decompose attention computations into low-rank subspaces. They show that low-rank QK signals causally influence when a token is attended to, and use their decomposition to perform circuit discovery and analyze attention sink behavior in LLMs.", "questions": "- The results in Section 3 were a little unclear to me. Can you explain how you get distributions for the $\\mathcal{S}^{\\ell ads}$ plots in Figure 2? Is it just because it’s over many attention heads? Or does this also take into account many prompts, and within prompts many (all?) token pairs? Or only for specific token pairs in a given prompt?\n- In Figure 18 (appendix J), the “boosting” vs. “removing” signals for gemma-2 seem to follow an opposite trend for the GP task. Any idea what’s going on there?\n- Can you clarify the use of counterfactual inputs /counterfactual token pairs in your circuit-finding adaptation of your method? (e.g., line 765 describes using “he” vs “she” in the GP setting, and line 770 seems to describe taking a difference between embeddings of counterfactual tokens in the IOI setup)\n- What are your thoughts on related work studying \"binding\" in LLMs? (e.g., [1-4]). These works also identify low-rank subspaces that seem to control different attributes of or attention to entities in a context. It seems related to the idea you're studying of how attention-causal mechanisms work, but did not see a discussion and am curious if you think there's any overlap between those findings and your  own.\n\n___\n[1] Feng and Steinhardt. [How do Language Models Bind Entities in Context?](https://openreview.net/pdf?id=zb3b6oKO77)\n\n[2] Dai et al. [Representational Analysis of Binding in Language Models](https://aclanthology.org/2024.emnlp-main.967.pdf)\n\n[3] Prakash et al. [Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking](https://openreview.net/pdf?id=8sKcAWOf2D)\n\n[4] Prakash et al. [Language Models use Lookbacks to Track Beliefs](https://arxiv.org/pdf/2505.14685)", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The paper is well written and fairly straightforward to follow. Several times I found myself having a question while reading, which was then almost immediately answered or had a reference to an appendix section with more detail.\n- The authors are aware of relevant related work, and the paper’s contributions are contextualized nicely. \n- The notation being consistent was very helpful when trying to understand the definitions in the presentation of the method.\nUnderstanding attention-causal behavior is an interesting alternative to other circuit-finding work, and has promise if it does not rely on counterfactual pairs.\n\n___\n**Weaknesses:**\n- The paper is pretty dense, and because of this many details of results are left to the appendix. This made some results a little unclear when presented in the main paper. For example, in Figure 2 (Section 3), the distribution is taken over an unspecified set of data (e.g. Is it just attention heads for specific token pairs, or all token pairs, all attention heads across many prompts?).\n- In the paper you say your method does not depend on counterfactual inputs (e.g. lines 9-10, line 61), but you often describe parts of the method being reliant on what seem to be counterfactual token pairs (e.g. lines 249-251). I think I may have misunderstood something about this part of the definition - can you clarify this, and whether you’re using counterfactual token pairs here? (See also the question below)\n- There were two terms I was unfamiliar with that it might be nice to have a citation or clarification for. These were the “Courant-Fischer theorem” (line 49, 188), and an “ersatz function” (line 149, 369).\n- It's unclear whether the method feasibly scales to larger parameter models, but this is a minor limitation. They have shown it works fairly well for up to 2B models. \n\n___\n**Other:**\nMinor Typos:\n- Line 184: Duplicated citation for Pan et al. (uses both [49] & [58])\n- Line 295: “postive” -> positive?", "was_revised": true, "final_justification": "The rebuttal has clarified my concerns related to the use of the phrase \"counterfactual\", and clarified minor inconsistencies or details that were not included in the initial submission.\n\nI have also read through the other reviewers' responses and rebuttals and I think the proposed changes will be helpful for increasing the clarity of the paper. With the rebuttal response addressing my concerns, I am inclined to Accept the paper.", "rebuttal": "We thank the reviewer for their helpful review of our paper.\n\n### Weaknesses:\n\n- Thank you for pointing out this lack of clarity. These plots are over all the prompts, and over all the attention heads and respective token pairs (d, s) that appeared in a trace from our tracing algorithm. We see that our use of the term “relevant” didn’t clearly specify what they are. We will change the manuscript to make this part clearer.\n- We can see that there are multiple kinds of counterfactuals that can get confused here.  We discuss this bullet point and how we will clarify the issues as part of our response to the third “Questions” bullet below.\n- Thank you for pointing this out. We will add a citation for the Courant-Fischer theorem. Regarding the “ersatz function”, another term would be “replacement function”. We can see that that term would be a bit clearer so we’ll make that change in the revision.\n- Yes, the method scales well. All SVDs for a model can be computed just once and reused for every prompt. The bottleneck of this method is the same as the attention mechanism, which is quadratic in the number of tokens. We have a small discussion about this in the Appendix (lines 802-808). \n\n### Questions:\n- We address the first question in the first bullet under Weaknesses above.\n- Yes, something odd is going on in that case. What is definite is that our method is causal in the attention weights – this is a provable fact. In other words, if we intervene in the position (d, s), the boosting and removing interventions increase and decrease the attention weight in the (d, s) position, respectively.  \nSo what we conclude for this case is that increasing the attention weight on the token pair in the downstream head actually decreases the performance of the model. We did not investigate further, but one possible explanation is that what this attention head in the (d, s) position is adding to the residual stream is information about suppressing the correct prediction (something similar to the Negative Name Mover heads in the IOI paper). Therefore, boosting this signal will make the performance worse, while ablating it will make the performance better.\n- Thanks for pointing out the need for clarification regarding “counterfactual inputs” along with the comparisons being made between alternatives at lines 249-251 and lines 759-771. \n \nIn the style of circuit tracing that is most common in the literature (causal mediation analysis) a **counterfactual input** is a prompt that is identical to an original input, except that it differs in the single feature under study. For example, the paper [42] – which was the first to trace the IOI circuit – uses counterfactual inputs that vary a name across three different versions of a prompt. Inputs, therefore, come in pairs—the original and the counterfactual—and these pairs form the basis of their causal mediation analysis method. For instance, for the IOI task, the original and counterfactual inputs could be:\n- **Original input:** “When Mary and John went to the store, John gave a drink to”. \n- **Counterfactual input:** “When Mary and John went to the store, Ryan gave a drink to”.  \n\nWhen we state that our method does not require “counterfactual inputs” (e.g., at lines 9-10), we are referring to this specific type of paired, minimally-different prompt As we note, constructing counterfactuals to test hypotheses about circuits can in many cases be a subtle and challenging task (eg, see [46]).  \n\nWe also use the term “counterfactual” in a different context, in which we are discussing a hypothetical situation in which a particular token representation could have been different.   This is the sense on lines 26, 33, and 42 and particularly on line 141.  The intent in those passages is to highlight that the signals we identify are causal for a head’s attention.   For example, at line 141 we are establishing causality by showing that, had a signal not been present, the head would not have attended to the token pair.  These passages are not considering actual counterfactual token pairs, but rather imagined modifications of a single token.  \n\nLines 249-251 are expanding on this hypothetical situation.  These lines are providing a mathematical tool for assessing the impact on attention if a token had been different.  So here, we are again not using actual counterfactual token pairs.  Instead in these lines and those that follow, we are calculating the effects of an imagined modification to a token, in which an upstream component’s contribution to the token had been removed.  \n\nTurning to lines 759 - 767, here we are describing a method for measuring model performance.  Again, for the IOI task, one can evaluate a model's performance by measuring the logit difference between the correct name (the IO name, \"Mary\") and the incorrect name (the S name, \"John\"). These logits are available after a single forward pass, and comparing them serves as a success metric. In all of our tasks, we use a success metric based on the logit difference between “success tokens” and “failure tokens.” For the GP task, the success token is either “he” or “she,” while the corresponding failure token is the other pronoun. Our use of these metrics is consistent with the extensive literature on these tasks.  \n\nLines 768 - 771 are connecting the success metric with a “success direction” in activation space at the model output.   For any success metric there is a corresponding direction, akin to an initial “signal”, that can be used to start our tracing method.  \n\nThat said, we note that our method does not fundamentally rely on this evaluation scheme. We can trace any linear combination of the unembedding vectors, that is, any direction in the logit space, by defining a corresponding “success direction” (line 769). For instance, we could trace the direction corresponding to the top-k tokens for any given prompt.  \n\nWe appreciate the opportunity to tease out these different threads, and these questions have been helpful in exposing the need for clearer language around counterfactuals, which we will add to the paper.\n- Thank you for making this important connection. The concept of \"binding\" is indeed relevant to our work. We view the challenge of correctly associating attributes with entities as a crucial application of the more general attention-causal mechanisms we study. As the reviewer correctly notes, the four works cited identify specific low-rank subspaces that control these binding functions. We hypothesize that these \"binding subspaces\" are the concrete, circuit-level implementation of the causal mechanism our work aims to identify. Therefore, we expect our method not only to find similar low-dimensional directions but also to provide a mechanistic explanation for how attention heads manipulate information within these subspaces to achieve binding. We agree this connection deserves a more detailed discussion, which we will add to the related work section of the paper.\n\n### Other: Minor Typos\n- Thank you for pointing these out. We will correct them in the revision."}, {"review_id": "RVliOlvvl1", "reviewer": "Reviewer_tbcy", "summary": "The work studies LLMs attention mechanism, by identifying low-dimensional features that enable the activation of individual attention heads, by identifying subspaces that the QK (query-key) matrices read from. Naming this features as \"signals\", the paper show how it can by applied to circuit discovery. In an experimental setup of GPT2 and Pythia, the paper show how this method can achieve similar circuit discovery performances to methods such as ACDC and EAP. One of the model's promises is the ability to mechanistically interpret models' attention-heads' behavior, such as attention sink, by identifying the features that enable them.", "questions": "Notes:\n1. some works are cited twice in the References, like [49] and [58].\n2. rearrange figures order: since figure 4 is presented in the text before figure 3, the index of the two should be replaced.\n3. Regarding Figure 6 and its corresponding discussion: I would really appreciate more conversation on why should the model use global signals patterns. Although the results are mostly solid, it is counter intuitive to believe that only a small number of subspaces (features) dominates most of attention head.\n4. Can you draw a connection between some signals and intermediate layer prediction (such as early exit projection [3][4])? I wonder if the signals has a meaning only when read by attention heads, or, that they hold some interpretable semantic.\n[3] interpreting GPT: the logit lens\n\n[4] A Mathematical Framework for Transformer Circuits", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written and easy to follow for researcher from the field of mechanistic interpretability.\n2. The method is highly based on model mechanistic and mathematical approximation - making the method highly justify.\n3. The motivation, as well as the method itself, are very well justified. The results of section 5 also highlights the method power.\n\nWeaknesses:\n1. Unclear figures: Figure 4 is hard to read as most lines are over each others. Figure 11-17 are also hard to follow but I assume they are proposed to be view via a separate pdf file.\n2. Regarding Figure 6 and its corresponding discussion: Since LLMs hidden state suffer from rouge dimensions [1][2] it is possible that the clustering be improved via another method than naive cosine-similarity, such as Spearman correlation. \n\n[1] All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality\n\n[2] Outlier Dimensions that Disrupt Transformers are Driven by Frequency", "was_revised": true, "final_justification": "My main concerns were with the presentation and additional discussions which I think can make the work more comprehensive. The authors address these concerns in their official response. Overall, I found the work to be solid and innovative.", "rebuttal": "We thank the reviewer for their helpful review of our paper.\n\n### Weaknesses:\n1. Thank you for your suggestion about Figure 4. There is some overplotting in these figures, and we will improve them. Regarding Figures 11-17, indeed, they are proposed to be viewed in a separate file. We plan to release a Cytoscape file with Figures 11-17, which will enable them to be visualized interactively.\n2. This is a very good point, thank you for pointing it out. We did not think about it, and indeed, using a rank correlation such as Spearman correlation, could improve the clustering. We will definitely investigate that in the revision.\n\n### Questions:\n1. Good catch; we will fix those in the revision.\n2. Also helpful and we will fix the figure order in the revision.\n3. Thank you for the question. Indeed, it is quite surprising that a few signals are responsible for so many attention weights.  \nTo explain, we start by noting that what is happening in all these cases is attention sink – meaning that the head is attending to the first token (or punctuation token) because it does not have any important function to perform.  In fact, for most of the attention heads and token pairs,  the model only performs the attention sink operation.  In other words, only a small subset of heads and token pairs are actually important for computing the model output in these tasks.  This is perhaps the underlying cause of the effect.  \nWe connect this fact with the observation (from [19]) that when models represent features in superposition, they will tend to allocate a dedicated dimension for features that are used frequently. Because attention sink is so common, the model seems to allocate a dedicated, global feature direction to control this behavior. The simplicity of this function, which is merely to direct attention to the first token, could also explain why a single direction suffices. \n4. We have preliminary evidence that some of the signals our method identifies carry semantics that are shared across different attention heads. For example, we have found that in the IOI task [42], different Duplicate Token Heads and Induction Heads write the *exact same signal* (same one-dimensional subspace) to be read by S-inhibition heads. Another example is the control signals we identified, which appear to represent consistent, meaningful instructions that are shared across multiple heads and layers. However, the extent to which signals in general are shared remains an open question. A more in-depth study of their semantics is an exciting direction for future work that we are eager to pursue."}]

Author rebuttal:

Meta-review: The authors decompose (query-value modules in) attention mechanism components into low-rank structures to identify features that are causally related to attention patterns. All reviewers agreed that this paper offers several nice contributions which build naturally on prior work in interpretability. 

I do think the authors should take care to address the issue of contextualizing the current contribution with respect to prior work (per the discussion with reviewer Zss5), and to ensure the notation is clear and consistent throughout (5oqm).
Decision: Accept (poster)