Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models
Authors: Jack Merullo
Carsten Eickhoff
Ellie Pavlick
Venue: NeurIPS 2024 poster (2024)
Keywords: mechanistic interpretability
interpretability
attention
subspaces
circuit
LLM
LM
language model
transformer
Primary area: interpretability_and_explainability
TL;DR: Interpretable structure in subspaces of language model weights allows us to control attention patterns and has applications in model editing and automated circuit discovery

Abstract: Although it is known that transformer language models (LMs) pass features from early layers to later layers, it is not well understood how this information is represented and routed by the model. We analyze a mechanism used in two LMs to selectively inhibit items in a context in one task, and find that it underlies a commonly used abstraction across many context-retrieval behaviors. Specifically, we find that models write into low-rank subspaces of the residual stream to represent features which are then read out by later layers, forming low-rank *communication channels* (Elhage et al., 2021) between layers. A particular 3D subspace in model activations in GPT-2 can be traversed to positionally index items in lists, and we show that this mechanism can explain an otherwise arbitrary-seeming sensitivity of the model to the order of items in the prompt. That is, the model has trouble copying the correct information from context when many items ``crowd" this limited space. By decomposing attention heads with the Singular Value Decomposition (SVD), we find that previously described interactions between heads separated by one or more layers can be predicted via analysis of their weight matrices alone. We show that it is possible to manipulate the internal model representations as well as edit model weights based on the mechanism we discover in order to significantly improve performance on our synthetic Laundry List task, which requires recall from a list, often improving task accuracy by over 20\%. Our analysis reveals a surprisingly intricate interpretable structure learned from language model pretraining, and helps us understand why sophisticated LMs sometimes fail in simple domains, facilitating future analysis of more complex behaviors.

Understanding Inter-layer Communication in Transformer Language Models
Talking Heads:
Jack Merullo
Carsten Eickhoff School of Medicine
Department of Computer Science
Brown University
University of Tübingen
jack_ bacck_erulllobromm.ee
carsten. eecieen.eiiikee eick @uni coaniittubinngen.d
email
Ellie Pavlick
Department of Computer Science
Brown University
el @llie_pavlick0brom.. edu
Abstract
Although it known that transformer language models (LMs) pass features from early layers to later layers, it is not well understood how this information is represented and routed by the model. We analyze mechanism used in LMs to selectively inhibit items in context in one task, and find that it underlies a commonly used abstraction across many context-retrieval behaviors. Specifically, we find that models write into low-rank subspaces of the residual stream to represent features which are then read out by later layers, forming low-rank communication channels [Elhage et al., 2021 between layers. A particular 3D subspace in model activations in GPT-2 can traversed to positionally index items lists, and we show that this mechanism can explain an otherwise arbitrayy-seeming sensitivity of the model to the order of items in the prompt. That is, the model has trouble copying the correct information from context when many items "crowd" this limited space. By decomposing attention heads with the Singular Value Decomposition (SVD), we find that previously described interactions between heads separated by one more layers can be predicted via analysis of their weight matrices alone. We show that it is possible manipulate the internal model representations as well as edit model weights based on the mechanism we discover in order to significantly improve performance
synthetic Laundry List task, which requires recall from list, often improving task accuracy by over 20%. Our analysis reveals a surprisingly intricate interpretable structure learned from language model pretraining, and helps us understand why sophisticated LMs sometimes fail in simple domains, facilitating future analysis of more complex behaviors
Introduction
Despite the impressive capabilities of LMs, they often suffer from seemingly arbitrary sensitivities
to prompts. These failure cases are particularly troubling because they are not systematic; it is very difficult predict when, for example, the order of information seemingly randomly causes
model to fail (Pezeshkpour and Hruschka] [023] Liu et al.) [024 Li and Gao 2024 /Zheng et al.
2024 Zhou al. 2023) the format of prompt hurts performance [Liu
2023 Scclar et al.
git
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

2023, Zhao et al., 2021 Lu et al., 2022 Webson and Pavlick 2022). As LLMs become increasingly ubiquitous. we will require more principled ways of anticipating and remedying unstable or unwanted behaviors Yu et al. 2024 Yong et al. 2023]. Understanding the mechanisms in play within L.L.s, and connecting those mechanisms to behavior, could enable such principled approaches.
One aim of interpretability research is to explain model behaviors, so is it possible to explain why some particular failure exists? In this paper, we consider a simple laundry list task that exhibits one such undesirable instability. Specifically: Transformer language models (LMs) struggle to reliably recall items from a list as the length of the list increases, and performance can vary wildly depending on the position of the item in the list that is being recalled (Figure D. This instability is not obvious from the model architecture itself-i.e., unlike their predecessors Elman [1991], Transformers (Vaswani et al. (017)) can use attention to recall freely from anywhere in context. Thus, we use this task as case study in order to connect the low-level emergent mechanisms which are encoded during LM pretraining to observable behavior, and illustrate
proof of concept that a precise mechanistic understanding of LMs can be used to explicate and, perhaps, remedy model performance in practice.
Specifically, by building on recent work in circuit analysis [Ehhaee et al. 202 Wang et al. 2022
Goldowsky-Dill et al. 2023 Quirke and Barez 2024 Merullo al. [024] Hanna al.) [025]] we demonstrate how Transtormer LM (GPT2 small, Pythia 160m) passes information from early layers to later ones using low-rank subspaces. These communication channels are proposed in Elhage et al.) [2021]. but understanding their implementation in weights is an open challenge [Makelov et al. 2024]. This conflicts with circuit analysis, which supports the interpretation of specialization and communication between specific transformer components. [Elhage et al. [202T] introduce score for measuring how much weight matrices in a transformer communicate, but outside of toy models, has been difficult to interpret the results (Singh et al.) 2024). We build on this work by proposing method using the Singular Value Decomposition to find these channels by decomposing one matrix into all of its component signals, and find it is much more interpretable than the non-decomposed variant. We focus on two examples of such channels (inhibition and duplicate detection, $2). and find that they are are very low-rank (1 or 2 dimensions), easily interpretable, and causally important for specific model behaviors. We also show this method can be used to perform model editing at training time (Appendix HH.2,, and provide some encouraging early evidence we can perform circuit discovery without running models (Appendix[I) Specifically, our contributions are as follows:
We explore a fundamental question in interpretability on how information passed from layer to layer in an LM is represented internally. We find "communication channels" (Elhage
et al. 2021 encoded in the weights that connect attention heads separated by several layers.
We propose a simple extension to previous weight-based methods that more effectively isolates low-rank signals passed through such channels.
We show that low-rank communication mechanism plays a role in prompt sensitivity on an item recall task that otherwise seems idiosyneratic, and that intervention in the mechanism can be used to affect downstream behavior to improve performance.
Our findings indicate more broadly that LMs are capable of learning intricately structured representa- tions from self-supervised pretraining without inductive biases. This may have important implications for the emergence of abstract and content- inteentinddeendeett operations, and for developing methods for steering and understanding these models (see Discussion, 7)
2 Background
Throughout our work, we consider decoder-only transformer language models and primarily use GPT2-Small. The modern attention mechanism used by these models Vaswani et al. [2017] uses multiheaded attention, where Query and Keys control which tokens from earlier in context are attended to and Value and Output matrices control what information is moved from these tokens. An important abstraction
in our work relies on rewriting these matrices as the products QK (Query* Key)) and OV (Output*Vabll to the embedding dimension of the network (due down-projecting the input e.g., they have size
For an individual head, these matrices are themselves low-rank compared
FFollowing convention, we refer to this matrix as OV, but the Transformer Lens library implements right-hand hand matrix multiplication so actually use VO. This does not effect results

---

Accuracy on Laundry List Data per # of Objects
Context:
Today, when go to the store, will buy a plate, a pear, pen, and textbook.
01
Continuation 1:
Accuracy
First, will get the plate, the pen, the pear, and then the
textbook
Continuation 2:
First, I will get the textbook, the pen, the pear, and then the textbook
Number of Objects
Figure I: Language models are often sensitive to arbitrary changes a prompt, for example the order in which objects are listed (right). This problem is more pronounced as the number of objects increases (left) even though is not obvious where the issue stems from in the model. We broadly explore how information is routed through model and focus mechanism that is in part responsible for this (in)ability.
768x64 dimensions in GPT-2). This is useful property that are written into/read from by these matrices and this reduces the search space.
motivated by looking for subspaces
In order to ground our findings about inter-layer communication to real model behaviors, we focus on attention head interactions which we already understand and work backwards to determine how they communicate. Recent works in circuit analysis provide detailed explanations of how different model components interact on controlled datasets. particular, we make use of the Indirect Object Identification (IOI) circuit discovered in Wang et al. [2022). We use GPT2-Small, study three specific types of interactions between heads: cases where heads write information that is used by the keys, queries, or values of later heads.
When we refer an attention head 3.0 or 7.9, this means layer 3, head or layer head 9.
Three Types of Composition In attention heads, there three ways that earlier heads contribute to the processing done downstream. all cases, information written into the residual stream by the OV matrix of earlier head, and read back out by either the Query, Key, Value matrices of later head. These concepts are introduced in Elhage et al. [2021]. We also provide an example of each composition type that we examine further in Section 3 We look for communication channels in one of each type of composition. These are previous token to induction head composition (key) OOsson et al. 2022, Singh et al., 2024 Reddy 2023), duplicate token to inhibition head composition (value), and inhibition to mover head composition (query) [Wang et al., 2022). The variation the way these heads communicate only changes how we calculate the composition score [Elhage et al. [ and individual implementation of the communication, but we do not make claims about how these types of composition differ from each other in more meaningful ways.
The Inhibition-Moverr Subeireuit We build on work from 10I [Wang et 2022] which documents a circuit that appears in multiple tasks (Merullo et al. 2024]. This circuit includes mover heads, which copy tokens from context the output, and inhibition heads which (optionally) block the mover heads" attention to certain tokens and thus prevent certain tokens from being copied. Inhibition heads are known receive signals from duplicate token head value vectors which help inform which tokens to inhibit. In GPT2-Small, the known inhibition heads are 7.3, 7.9, 8,6, and 8. 10 and we consider their communication to the mover head 9.05 This the example we use for query composition experiments. In Section 5 we explore this circuit's role in problems of prompt sensitivity and learned structure to control the indexing of tokens the context window.
for simplicity only consider this mover head, but do find the choice matters much.

---

Inhibition and Composition Scores for Different Inhibition Heads to Mover Head 9.9
7.3
7.9
8.10
0.7
Avg. Comp, Score
Inhibition
Compositio w/
Score
Unmodifeed Model
0.65
Remove Component
0.1
Composition
Soore
0.06
Component Index
Figure 2: Showing the relationship between the composition score (weight-based, bottom) and inhibition score (data-based, top) between various inhibition head components and mover head 9.9 for the IOI task. The inhibition of each inhibition head is generally highly concentrated in one or two components of matrix, removing it causes large drop the later mover head's ability to downweight one of the names. We therefore show that we can use the composition score when considering decomposed matrices.
3 Identifying Communication Channels in Low-Rank Subspaces
In this section, we test the hypothesis that model components like attention heads communicate through signals in low-rank subspaces of the residual stream and that weights themselves. We investigate one case of each type of composition outlined in Section 2 and find positive evidence for this hypothesis with query and value composition in 1 and 2 dimensional subspaces, but not key composition (see Appendix (C. Because we are able to localize the query and value signals such small representation spaces, we find that we are able interpret and control these features with intervention experiments in Section
find these signals in the
3.1 Composition Score
The Composition Score (CS) introduced in Elhage et al. [021] is a weight based metric of how much two weight matrices 'talk' to each other when they separated by layers. That is, W, might write information into a subspace in the residual stream that read out by W2- In Query composition for example, W, is the OV matrix of some head, and W₂ is the QK matrix.
| W2llF
CS(W1, W2) || || [[a]|F
We take advantage of the fact that circuit analysis in works like Wang al. 2022 tells that, for example, head 3.0 (duplicate head) interacts with head 7.9 (inhibition head) through value composition and 7.9 with 9.9 (mover head) in query composition.
3.2 Composition with Decomposed Component Matrices
We initially use the composition score in Equation I to try predict these interactions in the weights, but find these results largely noisy and uninterpretable. This briefly demonstrated in Appendix I We find that despite empirically knowing interactions exist between heads, we do not find they reliably have higher composition scores than any random head. Although the composition score has been shown to be useful on small toy models [Elhage et al. [222]). previous work has also shown that on larger models, the signal the composition score conveys is extremely noisy [Singh al. 2024]. Therefore, we turn to the Singular Value Decomposition (SVD), defined W = USVT on the QK and OV matrices to decompose the attention heads into orthogonal components which determine the input and output spaces of the matrix. This allows to individually view subspaces

---

read from/written into ordered by the amount of variance of the transformation of the matrix they account for. This helps us answer our original hypothesis that model components communicate across layers in low-rank subspaces of the residual stream.
If d is the dimension of the residual stream (768d in GPT2) and h is the dimension of attention head (64d in GPT2), OV, QK IR dzd and are both rank-h. This is because attention heads project down from the residual stream to h (e.g., the job of the V matrix) and then back up to d (e.g., the job of the O matrix). Therefore, there are only h non-zero singular values for each matrix.
Equation 2 shows useful identity of the SVD: we can rewrite some weight matrix W as the sum of the outer products of the left and right singular vectors, scaled by the corresponding singular value.
h W
5. * &
(2)
1=0
Rewriting the original matrix in this way is useful because we can now use the sum of any subset of component matrices in the composition score (Equation[] [ Let the zero-th component of head 3,0 be written as 3.0.0. We can write the composition score between the 3.0.0 OV matrix and the 7.9 OV matrix as CS(OV3.0.0 OVT.9). Since each component matrix is an outer product of two vectors, each matrix has rank-1. 1. This gives us a way to disentangle the full signal of a head into the sum of its component rank-1 matrices, or the subspaces that the head is able to read from/write to.
We find that decomposing weight matrices this way is very effective at cleaning up the composition score signal. We find attention heads that have very high relative composition scores with one component matrix of another head. For example, the second component of head 8.6 (referred as 8.6.2) composes far more with mover head 9.9 than any of the other 63 components in 8.6 (5 standard deviations higher than the average) or when considering the full matrix as in CSVSV Q The bottom graphs in Figure 2 show these results for the inhibition heads. All of the inhibition head exhibit a similar phenomenon of single component dominance. The duplicate token head 3.0 also value-composes with inhibition heads in similar way, using two components (3.0.1 and 3.0.2) far more than any other. Results are shown in Appendix D
We can also use this decomposition to find specific pairs of heads that talk more than others. With the knowledge that two heads talk through a specific component of one head, we can find the other heads that communicate through this pathway. Doing lets find the signal encoding almost the full IOI circuit in GPT2-Sman -mall directly from the weights, without running the model. We outline these results in Appendix [
We interpret these as communication channels between heads, but we would still like to establish these channels as directly affecting downstream component's 's behavior. We verify this is the case through a weight editing in Scction 3.3 and through activation interventions in Section 4
3.3 Model Editing
In the previous section, we found that within given head, individual component matrices encode a much stronger composition signal than that encoded by the global matrix. This makes the composition score much more useful tool than when only considering full-rank matrices. In this section, we verify that these identified components are indeed communication channels that carry causally important signals for model behaviors. We first look at the inhibition head channels.
3.3.1 The 101 Dataset and Inhibition Score
Because the behavior of the inhibition heads was initially described on the Indirect Object Identifica- tion (IOI) dataset in Wang et al. [2022). we explore the inhibition communication channels on that domain first. An example of the dataset is as follows: "Then, John[S1] and Mary[IO] went to the store. John/S2] gave drink to".. Here, the two name options are possible, but generating "John" does not make sense. The role of the inhibition heads are to tell the mover head (9.9) to attend less to the first John token (and as a result copy the remaining Mary token). We thus define the Inhibition Score as the degree to which the mover head prefers attending to the IO token (Mary) over the $1 token (the first John). This is simply the attention score to the 10 minus the attention score to the S1. Intuitively, full attention to the IO token would give a score of 1.0, -1.0 would be full attention to $1 (inverse inhibition), and 0.0 would be equal attention to both (no inhibition).
5

---

Inhibition Component Activations Projected into PCA of Mover Head 9.9 Queries
The entire output space of an inhibition component 1D, so all activations are some scalar a times the singular vector corresponding this output space
a *
Queries
H
7.9.8
Replace output the head with this scaled vector, addingg it to the residual stream
8.10.1
Figure 3: Because component matrices rank-1, 1, their output spaces are ID and interpreting them becomes easier. On the left, inhibition component activations go to either side of the origin, and selectively inhiibt the name in either position vector lying on this line by some scalar alpha and observe how this changes behavior when we add to the residual stream, replace the output of attention head with it (right), which show in Figure 4
position two in the IOI task. We can scale
3.3.2 Results
Our editing technique is simply to zero out one component at time test how this affects copying behavior across dataset of IOI examples. One way to think about this is zeroing out one singular value of c.g., the OV or matrix, or subtracting one of the component matrices from the sum in Equation We must make the edit to the decomposition and then split the matrices back out so that we can run the model. Given OV USV", after zeroing out some singular value in S (forming S') we can set the Output and Value matrices to be / and /S'V", respectively. In the top graphs of Figure 2 we show that removing the speculated communication channel from the inhibition heads almost always results in a significant decrease in the model's 's ability to pass the inhibition signal, with the exception that changing 7.3 does not have strong effect on its own. In general removing the single component with the highest composition score reduces the Inhibition score by 7-14%, and it is important to consider that this is only when changing single component in one head a time. We perform additional experiments with removing/modins modiiyinddiiigggggoo multiple of these components in Section and Appendix Thus, we have both behavioral and weight based evidence converging on the interpretation of these subspaces mcaningful communication channels.
4 Communication Channels Carry Interpretable Content-Independee Signals
We present further evidence that the communication channels we identify in the model weights carry causally important signals for affecting model behaviors, but also that they carry content-independent signals which are easily controllable and interpretable.
Because the component matrices rank-1 (Equation[], their outputs lie entirely ID subspaces. This subspace (in our implementation) is the right singular vector corresponding to the index of the component matrix multiplied by some scalar. Since have shown that these communication channels have significant impact on downstream performance, a natural question how information (such inhibition) is represented along such a simple feature.
4.1 Interventions Communication Channels
In order to better understand the representations passed through communication channels, design interventions that add to or replace information from certain heads with vectors that lie on the output space of communication channels. Figure provides outline of the approach. Since single

---

Inhibition Component
Control
Inhibition Head 8.10.1
first
Control Head 8.0.0
S1 first
Inhibition Score
45
Singular Vector Scale
Singular Vector Scale
Duplicate Token Components
Control
Adding the Duplicate Token Head 3.0.[1, 2] Signal
Adding the Duplicate Token Head 3,0.(0, 3] Signal to 10 or S1 Tokens
Tokens
Inhibition Score
Duplicate Signall to O Duplicats Adde S1
-20 -228
Singular Vector Scale
Singular Vector Scale
Figure 4: We find that the 1D inhibition components and 2D duplicate token components finely control which name is avoided by the mover head. On the top, we can selectively inhibit either the first second name depending on how we scale vector lying on the 8. 10.1 output space. This is strictly controlling relative position. On the bottom, we find that adding or removing duplicate token information from the duplicate channel at the I0 or SI tokens also effectively modulates which name is inhibited. Neither random heads, nor non- communiiiniaii channel components exhibit these same effects (right). See Appendix for results on other heads.
component is rank-1, 1, we the output of some head be a point the output space line and see how information changes along it.
Our dataset contains 200 examples from the IOI task. We have 100 examples where the 10 token the first name ("Mary and John...John gave drink to") and 100 where the S1 token comes first ("John and Mary... John gave drink to").
On inhibition heads, we find that scaling single component at time is enough to switch the attention of the mover head to the other name. The inhibition score is highest when the SI token is inhibited, but Figure 4 shows, downscaling 10.1 only increases inhibition when S1 is first, and decreases when 10 is first. The opposite is true for upscaling the component. This tells us that the component is passing a positional signal: either inhibit name one or name two. This is consistent with what Wang et al. [022] found, but our intervention shows that we can completely divorce the context from this ability. Since we are setting the head output to a scaled singular vector from the weights, we are bypassing the attention mechanism entirely, from the value vectors of the 10 or S tokens. This tells us that the model is capable of representing pointers, or storing bindings in the keys of earlier tokens that represent indexed lists, similar to the finding in Feng and Steinhardt 22023]. We explore this further in Section S The bottom of Figure 4 shows that scaling diagonally a 2D subspace of the duplicate token head and adding the resulting vector to the residual stream of the 10 or SI token right before the inhibition heads also allows for modulating the selected name.
what to inhibit is coming non of the information
Although we get fine control over the attention of the mover head, we have not answered whether this has a real effect on the output behavior. Additionally, Makelov et al. [2024] argue that interventions on subspaces, such as the case here, are prone to subspace illusion in which the effect is not what it seems. To address this, we measure the Fractional Logit Difference Decrease (FL.D) and Interchange Accuracy of patching the subspaces in the inhibition channel between minimally different IOI examples. GPT2-Small achieves an FLDD of 97.5% and and interchange accuracy of 35%
7

---

Makelov et al. [2024 report a baseline that achieves -8% FLDD and 0.0% Interchange. By taking the gradient to directly optimize single vector for this task they achieve 11 111.5% FLDD and 45. 45.1% Interchange. Our results in comparison, support the view that these 1D subspaces are primary mechanism controlling name selection. In Appendix G we also find that these inhibition signals are active broadly during next work prediction. On Open WebText [Gookaslan and Cohen 2019), we find that the inhibition heads are primarily active in lists and settings where repetitions should be avoided; for example, in comma separated lists (attending from commas to previously seen items). A natural followup given the IOI results and these observations is to explore their role as an indexing function, which we perform with the Laundry List task.
The Inhibition Channel is Crucial in Context Retrieval Tasks
Now that we have established the existence of communication channels and their causal role in model behavior, we can revisit our motivating example on prompt sensitivity in the I Laundry List task. Figure Shoow an example of the task and an example of an arbitrary seeming failure to minor variation the order of presented items. In this section, we explore the hypothesis that the inhibition communication channel presented far plays critical role in how the model selects the right context token to generate next, and reveals a capacity limit that causes the model to fail as the number of objects increases.
5.1 Laundry List Task
We propose synthetic task that is designed to activate the inhibition heads, but allows us to test their effect on an arbitrary number of candidate tokens. An example is given in Figure and more details on how we generated the data are in Appendix [ Each example first mentions N objects, then N-1, and the model must predict the missing item. We create a dataset of 250 prompts for each value of N from 3 to 20.
To complete this prompt, the model needs to recognize the item missing from the second list and retrieve it from the first list. By shuffling the second list and using different sentence format, simple mechanisms (like pattern matching with induction heads) can not be relied on solely to solve the task. With this setup, we can very naturally increase the number of objects being considered. This lets us test not only if the inhibition heads are used for indexing candidates, but whether this mechanism's 's behavior changes it is strained by the number of comparisons that need to be made. The model has a strong bias to predict the last item, regardless of whether it is correct or not, which causes performance to drop, we also curious if the mechanism connects to that as well.
We find that the inhibition heads are highly active (large attention on non EOS tokens) on this task when predicting the last object, like the analysis in Section would predict. The heads attend to and inhibit the occurrences of all of the repeated object tokens.
5.2 Intervention Experiments
We repeat the inhibition component intervention from Section 4 where scale the components in the inhibition channel. In the multi-object (>2) case, we find that scaling only one component at a time does not give enough expressive power change the mover head's attention to reach all of the objects (Figure [] left). Instead it prefers to attend to either the first or last object, seeming to chunk the remaining objects together single point along the line. The bias to ignore information in the middle has also been observed by Liu et al. [2024].
We traverse the 3D space spanned by the top three inhibition head components (7.9.6, 8.6.2, and 8.6. 10) and measure how this changes where the mover head attends, and what the model's 's final prediction is We leave out 7.3 because found that it changes inhibition performance the least (Figure 2 and visualizing with only 3 dimensions is much simpler. In Figure (middle), we visualize this traversal with one dot representing point in the space that we query. We run the entire dataset with the inhibition components set at this point, and the color represents the index of the object that the mover head attends most We find that structure emerges
add items, where areas of this
We test in increments of 10 from [-100, 100] along each axis, including every combination. it's possible this is not the optimal range
8

---

Otjects
Objects
Accuracy on Laundry List Data per Objects
be
Senal rnterriniiiiinnno
158
Accuray
Oojects
Otject
Ctject
Otjects
Otjert
Otpst
Otjerta
Number of Objects
Figure 5: Scaling the inhibition component for a single head (here 8.10, left) is not expressive enough to get the mover head to index between the various objects. Scaling the top three inhibition components (middle) gives us enough expressive power to selectively attend to one of the objects. Here, one dot represents run on the corresponding dataset and the color represents the index of the object the mover head pays the most attention to on average. A surprising structure emerges that partitions the space according to the index of the objects. However, the neat structure begins to break down as the number of objects grows around 10 or higher, and affects the mover head's s ability to attend to the right object, which impacts accuracy. Right: Accuracy improvements as a result of sampling from inhibition space. The model becomes much more capable of handling a bigger number of objects in that the accuracy for N objects after the intervention is about as high as the unaltered model when it sees N/2 objects. However, the representational power of the inhibition channel reaches capacity as the number of objects increases, and performance can not improve as much.
space represent the first and last object, and wedges fill in the space for each item that gets added. Eventually (around or 10 items) these wedges get small and start to fracture (Figure [ middle bottom). We connect these two phenomena to the performance of the model: the bias to predict later objects, and the inability to handle longer lists. We believe the model is not capable of traversing this space well enough on its own, even though it learned to represent it, and longer lists cause worse performance because the space gets fragmented into smaller and smaller areas that repersent each item.
We design an intervention where we set the model components in a certain area of the 3D space. depending on the index of the correct answer and test how much this improves performance. In Figure (right), we show this causes sharp increase in the accuracy of the model: 3 object accuracy goes up from 64% to 86%, and 8 objects goes up to about 51 %, about the level of 4 objects in the unmodified case. Therefore the inhibition channel we identified seems to form part of a more generic, content- independendd subcircuit for indexing items in the previous context.
6 Related Work
There has been significant focus on disentangling features from the representations of language
models and vision models [Olah et al. 2020 Features have been analyzed at the neuron level
[Gurnee et al. 2023 Mu and Andreas 2000 Dai et al.] [022 Tang et al. [0244] Sparse Autoencoders
and Dictionary Learning [Bricken et al. 2023] Mallat and Zhang 1993 Cunningham et al., 2023]
attempt to deconstruct activations into more primitive features (Rajamanoharan et al.) 2024], which is similar in spirit to our decomposition. Park et al. [2033] propose a unification of several perspectives on the lincarity of featuers. The Superposition Hypothesis Elhage et al. [0222 posits that linear features are encoded in interfering ways. Our method is similar in flavor of disentangling tangled features to make them easier to read off of the weights.
The SVD has been used for interpretability and weight based analysis in the past (Sule et al. 2023 Praggastis et al. 2022 beren and Black 2022]. Martin et al. 2222]. for example, use the SVD on weight matrices to measure weiht quality to predict generalization performance and whether or not a model is well-trained. The SVD has been used in the training of LoRA modules (Hu et al. 2022

---

as well as in other finetuning methods [Batazy et al. 2024 Feng et al., 2024 Wang et al. 2024
(Sun et al. 2024 Guo et al. Karimi Mahabadi et al., 2021 Our work analyzes model properties that may facilitate these methods to work. ? is perhaps the first to report that pretrained models have a very low intrinsic dimensionality, which helps explain and support our claim that we see so many low rank communication channels in large models. We are therefore excited about the connections between this line of work and work on the linear representations in language models [Elhage al. 2022, Park et J.. which argues that features in LMs are represented as directions in space. There has been recent work in studying interactions between components in finer-grained
feature spaces in the past [Kim et al. 2018 Marks et al.) 2024 Geiger al. 2023 Lepori et al., 2023 Zhang et al. 2024), whereas our approach begins by first analyzing the weights to find substantial connections subspaces without requiring any data. Future work could connect these two, to find
circuits, subnetworks, and/or directions in space for certain behaviors informed by connections in the weights, which alleviates the concern of finding these things from scratch. We outline preliminary results for such an approach for novel circuit discovery in Appendix []]
7 Discussion & Conclusion
Due to the recent and rapid success of language models, there is growing interest in understanding how they are able to use language so flexibly and solve difficult tasks. Our results contribute positive evidence that intricate condent-independent structure emerges as a result of self-supervised pretraining. Although similar types of structure were previously thought to be impossible or unlikely to emerge in LMs [Lake al., 2017], there is emerging evidence that LM pretraining encourages models to organize mechanisms into neat subprocesses [Feng and Steinhardi 2203). We use weight decomposition to uncover such structure and contribute to a fundamental understanding on how models route information between layers, a core part of understanding feature representation in models. We also show that low-level mechanisms such as those studied here can make real predictions about prompt sensitivity, problem that has long plagued the robustness of LMs.. The method we employ for weight analysis also holds promise for inference time steering, model editing, and automatic circuit discovery. We hope our work promotes future research on the interpretability of neural networks as well as their responsible deployment, and practical capabilities.
References
Klaudia Balazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405. 2024.
beren former
and Sid
Black.
The Singular Value Decompositions
of Trans- November
are Highly Interpretable. URL https://wwzz www.wrong.. com cotm///pstt///thh0o2222288882.2 the mhe-singular-vas value-vaeee--eee dwwoopositoooonsssoooooo of transffrmer-weereee.rrrr weight
Weight
Matrices
2022.
Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbic Bradley. Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//transforme--iiioi http://aaaooooocicciiiiiiiie///e pub/22023/mooosemaniic features/index.hmm
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models, 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493-8502, 2022. 2022.
10

---

Nelson Elhage, Neel Nanda. Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac HatfieldDoodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. https:/transformer- diccuitspubb/2022/ty__mmmo www./m//ddde///h//lll
Jeffrey L. Elman. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 1952255 September 1991. ISSN 1573-0565, doi: 10.1007/BF001 14844.
URL wwwphhtpps/oooot org/ 10. 1007 1007//F0001404
Chengcheng Feng, Mu He, Qiuyu Tian, Haojie Yin, Xiaofang Zhao, Hongwei Tang, and Xingqiang Wei. Trilora: Integrating svd for advanced style personalization in sot-to-image generation. arXiv preprint arXiv:2405. 1236, 2024.
Jiahai Feng and Jacob Steinhardt. How do Language Models Bind Entities in Context?, October 2023. http:ttt:///ratt www. 2rgg/abs/2310... 2310. 17191
Atticus Geiger, Chris Potts, and Thomas Icard. Causal abstraction for faithful model interpretation. arXiv preprint arXiv:2301.047700 2023.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. OpenWebTextCCrpuss 2019.
http:///Skylion007ggiic
Nicholas Goldowsky-Dii Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05669 2023.
Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. The Twelfth International Conference Learning Representations.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding Neurons in Haystack: Case Studies with Sparse Probing, June 2023. URL http:
org/ abs 2305 01610
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in pre-trained language model. Thirty-sevnnth Conference Neural Information Processing Systems, 2023. httpppp://ppeenevee..eeeee/////e p4PckNQRKK
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International
Conference on Learning Representations, 2022. URL httpttts//opeeeeieer... net. www.:nee/f.o/iiii..mmommo nZZVKeeFYf9
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022-1035, 2021.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668-2677, PMLR, 2018.
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017. doi: 40.1017//0140025X 40525X160018888 16001837.
11

---

Michael Lepori, Thomas Serre, and Ellie Pavlick. Break it down: Evidence for structural composition- ality in neural networks. Advances in Neural Information Processing Systems, 04442623-46666 2023.
Ruizhe Li and Yanjun Gao. Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple- 03205
Choice Questions, May 2024.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Trunsactions of the Association for Computational Linguistics, 12:157-173, 2024
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086-8098, 2022.
Aleksandar Makelov, Georg Lange, Atticus Geiger, and Neel Nanda. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching. In The Twelfth International Conference on Learning Representations, 2024. URL httpp::/ net/
Stéphane G Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions on signal processing, 41(12):397-34155 1993.
Samuel Marks, Can Rager, Eric Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403. 19647, 2024.
Charles H. Martin, Tongsu (Serena) Peng, and Michael W. Mahoney. Predicting trends in the quality of state-of-the-art neural networks without access to training testing data. Nature Communicationss 12(1) (4:12, July 2021. ISSN 2041-1723, doi: 10. 1038///4444--002-224000 URL httpp:://ww/ http://wwwwww.r.eeeee/e naturaataee.... com/ awtaaceeeieeeeae11 0201760--2222222----2
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models. In ICLR, 2024.
Jesse Mu and Jacob Andreas. Compositional Explanations of Neurons, June 2020. URL https 7/arxiv. . org/ abs/abbs.....//////// 14032v2
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Zoom in: An introduction circuits. Distill, 2020. doi: 122211/iiillll.0002...000
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 11895, 2022.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.00 1.03658, 2023.
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity the order of options in multiple-choice questions. arXiv preprint arXiv:2308. /483, 2023.
Brenda Praggastis, Davis Brown, Carlos Ortiz Marrero, Emilie Purvine, Madelyn Shapiro, and Bei Wang. The SVD of Convolutional Weights: A CNN Interpretability Framework, August 2022.
URL http: arxiv. org/ abs//2b8....////////////// 2208 22522228888889444
12

---

Philip Quirke and Fazl Barez. Understanding addition in transformers. In The Twelfth International Conference on Learning Representations, 2024. UKL_attpee//pperroier.nt////oo/p wtw.:/rreiiii...ttotomuuiiiaa.
FIx1YXWWZb
Senthooran Rajamanoharan, Arthur Commy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders, 2024.
Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in in-context classification task. In The Twelfth International Conference on Learning Representations, 2023.
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design How learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, 2023.
Aaditya K Singh, Ted Moskovitz, Felix Hill, Stephanie CY Chan, and Andrew M Saxe. What needs to go right for an induction head? mechanistic study of in-context learning circuits and their formation. arXiv preprint arXiv:2404.07129; 2024.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. 2023.
Shashank Sule, Richard G. Spencer, and Wojciech Czaja. Emergence of the SVD as an interpretable factorization in deep learning for inverse problems, August 2023. URL http:://tx/ixii org/ abs/2301.0788
Chengwei Sun, Jiwei Wei, Yujia Wu, Yiming Shi, Shiyuan He, Zeyu Ma, Ning Xie, and Yang Yang. Svfit: Parameter-efficient fine-tuning of large pre- mreitriined models using singular values. arXiv preprint arXivv2409.054 05926, 2024.
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang. Xin Zhao Furu Wei, and Ji-Rong Wen. Language-spectio neurons: The key to multilingual capabilities in large language models. arXiv preprint arXiv:2402. 16438, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Kevin Ro Wang. Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations, 2022.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07337 2024.
Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300-2344, 2022.
Zheng Xin Yong, Cristina Menghini, and Stephen Bach. Low-resource languages jailbreak gpt-4, In Socially Responsible Language Modelling Research, 2023.
Zhiyuan Yu, Xiaogeng Liu, Shunning Liang, Zach Cameron, Chaowei Xiao, and Ning Zhang. Don't listen to me: Understanding and exploring jailbreak prompts of large language models. arXiv preprint arXiv:2403.173:: 2024.
Enyan Zhang, Michael A. Lepori, and Ellie Pavlick. Instilling inductive biases with subnetworks. 2024. https://///arx org/ abs /ab///231...mm 10899
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Out, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL https: //arxiv.... www..orgggbo... abs/ 2205. 01068
13

---

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance language models. In International conference on machine learning, pages 12697-12706. PMLR, 2021.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large Language Models Are Not Robust Multiple Choice Selectors, February 2024. URL http://arxiv. www.aaiwwi..o...oo.a.ooooo www./bs/g///eoooo/e /309
03882
Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Joshua Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? study in length generalization. In The 3rd Workshop on Mathematical Reasoning and Al at NeurlPS 2023.
A Limitations
A limitation of our approach is that we are relying heavily on previous knowledge of the language model that we are using (GPT2-Small), which has been extensively studied. However, the insights that we are able to glean by building on this foundation of knowledge we view more reason to approach interpretability work as building directly on model-specific knowledge. Additionally, our findings may be able to feed back into automating interpretability of new models. Another limitation of our approach is the inability to calculate query and key composition scores with models that implement relative positional embeddings like RoPE (Su et al. [023] because of the non-linearities between the Query and Key products preventing QK to be calculated cleanly. may be possible to simply take the composition between the Q and K matrices individually, but we do not experiment with that extension here.
B Larger Models' Performance When Increasing the # of Objects in the Laundry List Task
In the main text, we consider models that are quite small by today's ' standards, and it reasonable to wonder whether the inability to handle more objects in context goes away with model scale. In Figure []] we show that even up to 6.9B parameters, OPT [Zhang et al., 2022) and Pythia models also struggle with increasingly many objects, although degrade more slowly with scale. Regardless, we are not claiming that we have identified problem that can not be solved through scaling, nor is that of particular interest in this work. Increased capacity is likely yield mechanisms with higher capacity to handle more objects. What is of greater interest for future work would be to identify whether larger models use fundamentally different mechanisms than the one identified here to solve the task.
C The Composition Score with and without Weight Decomposition
We include some examples showing outlier components value and query composition but not with induction head key composition in Figure
D Duplicate Token Heads
We focus on the inhibition communication channel the main paper and not show the where the duplicate token channel comes from. In Figure g we show that two component matrices in duplicate token head 3.0 1 and 3.0.2) compose strongly with inhibition heads (7.9.6 shown here). On long sequences of random tokens, we show that these direction encode whether or not token has been duplicated.
E
More IOI Interventions
Additional interventions testing the efficacy of inhibition heads to change behavior of downstream mover head are provided Figures 9 and TO. We include the remaining inhibition heads and additional control heads for the component scaling experiment from Section 4 in Figure II
14

---

Laundry List Accuracy by Number of Objects
GPT2-XL
OPT-2.78
OPT-6.7B
Pythia-6.99
0.8
0.8
0.8
0.8
Accuracy
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
17 20
14 17
20
17 20
14
17
20
Figure 6: Larger models also degrade performance rapidly we increase the number of objects in the Laundry List task, although more slowly than smaller models. Pythia 6.9B retains strong performance up to around 10 objects.
Figure 7: Examples of composition scores of individual components with other heads. 4.11 is previous token head, 5.5 is induction head, 3.0 is duplicate token head, 8.10 is an inhibition head, and 9.9 is a mover head. We find that there are large outlier components value and query composition, but not in the induction head, thus motivating our focus those heads in the main text.
Projection of Duplicate and Non- Duplicate Tokens along
Composition Scores for 3.0 Componants to 7.9.8 Values
3.0.1 and 3.0.2 directions
Duplicate Non-Duplizate
Compooden 500e
8.65
Compover
Figure 8: Left: Composition scores between each component of duplicate token head 3.0 and inhibition component 7.9.6. Components 1 and clearly outliers. Right: long contexts of random tokens with inserted duplicates, we find that these directions separate duplicates from non-duplicates quite well. This leads us believe that these two components form duplicate communication channel. Our results in Section support this interpretation.
15

---

Effect on Inhibition Score when Forcing Inhibition on Certain Tokens
Full Model
Inhibit S2
Inhilbi
Inhibit S1
Inhibition Score
0.5
-0.5
Full Model
Inhibit S2
Inhibit IO
Inhibit S1
Figure 9: Effect of applying the top component interventions at the same time to some token. We can control the inhibition by selecting which token these components attend Higher score means more inhibition on S2, and lower score means more inhibition on IO.
Laundry List Data Generation
The Laundry List task is leave-one-out task where the model must identify the object that was not mentioned. Each input is two sentences (see Figure for an example). The first sentence lists objects that need be purchased and the second describes the order that they are be bought in, with the next token prediction being the item from the first list that is to be bought last. This setup allows us to freely shuffle the order of the information provided the model as well as vary the number of objects presented in each example. There are 22 objects that can be sampled, given below: "pencil", "notebook", "pee",, "cup", "plate", "jug", "mug", "puzzle", "textbook", "leash", "necklace", "bracelet", "bottle", 'ball", "envelope", "lighter", "bowl", "apple", "pear", "banana", ''orange", "steak"
The first sentence can start a few ways, chosen randomly: Today,', Tonight,', Tomorrow,', ' (, And the second sentence start be chosen randomly: First,', " When I go,', think'
G Inhibition Heads Behavior on Open-domain Text
We find that inhibition heads are consistently active on tokens about to predict the continuation of sequence ("serotonin, dopamine, and...") and attend to previous items in that sequence, consistent with their role in IOI and Laundry List. We therefore argue that the role the inhibition mechanism plays in both IOI and the Laundry List task performs this same operation in generic language modeling
H Inhibition Mechanism in Pythia and Training Progression of Inhibition
We verify both that communication channels appear in other models, and that inhibition a more general mechanism that that just appearing in GPT2. To show this we analyze Pythia- 160m [Biderman

---

Effect on Inhibition Score when Keeping
Only One Component per Inhibition Head
1
Full Model
Components Only
0.8
Zeroth Components Only
0.6
Inhibition Score
0.4
0.2
0
-02
-0.4
-066
Full Model
Top Components Only
Zeroth Components Only
Figure 10: Effect on the inhibition score when removing components from the inhibition heads. only take the top-1 composing heads that affect the inhibition close to have of the average inhibition score (0.7 to 0.3). If we only use the component matrices that correspond the Oth singular value of the inhibition heads, which represents the subspace most strongly written to by the head, the average inhibition score is only 0.04, Recall that a negative inhibition score means placing more attention on the subject rather than IO token.
(circled in Figure 2 retain
Effect of Overwriting Attention Head Output as Scaled Component Vector (V) on the Inhibition Score
1ot fift
B0bntB13 and MaryLTO) wast stoce."
Cettriinee330
Costrol RAJ10
"Soba|
and Hary [21] went
Output of testttionneed
oveneitan
The popprtrr shrcch
We intervene on the forward pass of the model by replacing the output of some attention head as the vector obtained by scaling component vector by some scalar a. By doing so, the actual attention head pattern has effect on the downstream performance. We show the inhibition component vectors have the unique effect of controlling the position the name being attended to by the downstream mover head (9.9). Random control and head components with other functions (like 8.3.0) do not have this effect.
Figure 11:

---

GPT2 8.6
made the impressian herr mestiooed
yn Seijin 1006, the toom Pork
Joo Hyun Jung and Nur Ok
Fowel,,
National
Teleommunicac
former chaiman,,
lasrmakers, fodd so while the concar
respectable bil and froble, and that should continue
his member componics es, which nccue most of major cobls and
broodtond providers, would fooo changes cristigg
tranofisiors spreed let the trandusion enthunieds with nevion-cmppee ponder fact that cecaelonetreral sioe de herm, dhowaur
fact
sawlorcomppen comples ponder the that many pocastons tranhistons harm, spread disesse, frequeettyy tht cause whichs, which, couureee are pubiiieeeee (9L) decared: Nind transtusion anthusiacts savior domppios ponder oo ond many poccsions transfusions doham, spread discase.
treguetentty deaths, which, publiczed" (9A))n deciored: Nid tranefution enthundeds with savioncomple ponder the that many occasions tranef usons do form, spread disone,, and frequamtyy coure
ponder occasions tranefasions
tramafuaions harm, spread disess, frequantia cause deaths, which, course, are publiceed (Pa] in dedaaedd translisionn enthuninats canpor-complec transtusions disose, Brequentyy
cause deaths, which course, nod publicibed (91) Vn deciared Wed the tanduuioo anthus with swior-comppee porder the firt thar on mony acassionsttootiiinn horm, spread diseaes, frequetty caune deatts, courm, and publicioed Watchnoww sucted tracllin ourgeon
Figure 12: Examples of high attention in examples in OpenWebText-10K for head 8.6 GPT-2
al. 2023 Because Pythia provides training checkpoints, we are also able to analyze the formation of the inhibition component we find to some extent.
H.1 Path Patching on IOI
We perform path patching (Wang al. 2022 Goldowsky-Dinn al. 2023
Pythia-160m the
1OI task to see if the model also implements mover and inhibition heads. We find evidence for one inhibition head (6.6) in the model talking to mover head (9.5). We find that like GPT2, the inhibition head communicates primarily through a single component, as is shown in Figure [6]
Additionally, induction head (4.11) strongly value composes with the inhibition component 6.6.2 as shown in Figure
H.2 Training Progression of Inhibition Components
Because Pythia releases 144 intermediate checkpoints (per 1000 steps), we can track the emergence of the inhibition head during training. We saw that inhibition component vector clusters Namel on one side and Name2 on the other side, representing which name is being inhibited. Everything else ends up around the origin. Since we have minimal pairs of examples that differ only in the position of the name that should be inhibited, we can measure the Separability of the inhibition component
18

---

GPT2 8.10
need recreats and bing. Furf your
You mightt wart keep th
collaction, though, bocause when rkiingo
want to seep stom-pricad out in
collection, Bhough
Dccause when Hikingg are together in one place
city inkopools
21, 2013 Hammar Huseum Courtyard yinhi Free
hove enough frontal cortes, you hove excossive
disonbobuuzted inffiiinn, poofly
4omiiiinn
parsonal
damaged company)He0
Nay 2016. Aport for the Nintenco wwitchhee wad released
January 2019 and was ported by British studio Red Phantors Comes,
theaatttonn in simply
simply
that's as huge trake
Iuport", landed cousted
the
Sundanoesci4 dramedy
the Summoniill LCBO, é was deliciuss and cold reall wcl,yycl,ee Tomney sas
Figure 13: Examples of high attention in examples in OpenWebText-100 for head 8.10 GPT-2
vector by making sure that if one name is in one cluster, the minimal pair example is in the other cluster. This a measure of how well the inhibition head is structuring according to this idealized separation of the two names.
The component that we is the 6.6.2 matrix from the fully trained inhibition head. We test parity by projecting the model's activations onto this matrix.
In addition we can measure how well the fully trained component matrix activates (or removes) the inhibition signal by adding or subtracting it from earlier checkpoints. These results are in
I Circuit Discovery with Static Weight Analysis
One of the core claims of this work is that we can find meaningful connections between attention heads by reading them off of the weights. In this section we extend this idea beyond the IOI circuit and use our method to show that we can find novel meaningful connections without running the model. First, we show that we can rediscover the IOI circuit directly from the weights using the decomposed composition score (without knowing anything about its function). Then, we show that we can identify new connections the 9.9 mover head that part of different circuit. We finally run the model on open-domain data and perform causal experiments to begin to identify the functionality of this new circuit, which we implicate retrieving relevant information from context. While the experiments we perform to identify functionality are not replacement for a full circuit analysis, we at least show how our method can used to pinpoint circuits network for further analysis. Future work can extend the analysis of the context-retrieval circuit find here.
19

---

GPT2 7.9
(and fun thee ware rituulllryyyyl dominnnt in 2015, the bearn oark Jo0 Myrrt Jung
deps topurish prrpetiston
Notioe met Nodice mei 6 in
Custonn UI dements are
roquired to sur they are added.
belt thee functioralty in the process
who
cores, reslows will iocusion how good
funciionality
reviows
Two lise functionalitynn in process who cares
reviews will
loces on goodood we toykary Again
Jusnst Rlgp hip sec panal tnt Perthpa
mond
/elt adcitionl 2200 of
panel show Pichaps Mr Galmone lelt
additional F300
necegntton of GeGtrtithooogggte
s01 pansil show In in Pwihaps Ni Salmord felt the addiiional 1200 was
Incogration of de0tstrtsshoogggee owaightng
momeeent
tolog rignette
requeeedinkk lawgerss
Figure 14: Examples of high attention in examples in OpenWebText--106 for head 7.9 GPT-2
L1 Finding the IOI Circuit in GPT-2 Weights
To show the effectiveness of the decomposition finding heads that communicate significant degree, we use the composition score only to find large chunk of the IOI circuit from Wang et al. (2022) encoded directly in the weights. Figure shows these results. The composition score after decomposing the inhibition head 7.9 more clearly reveals the communication between the in-cireuit heads (circled in red) than if the composition score is used without decomposition. It is possible this approach can be built upon to find circuits in models without requiring the model to be run. We leave this for future work.
1.2 Discovering New Connections
We now take this further to show that we can use the composition score facilitate the discovery of new circuits directly from the weights. Although we can not yet identify function directly from the weights, we can identify components that communicate strongly (indicated by their composition scores) to know where to look for circuits that may explain other behaviors in the model.
We focus on one specific extension the inhibition-mover subeireuit by looking for new connections into the 9.9 mover head QK circuit. We search for connections into this head because it seems likely that other heads besides inhibition heads control what the mover head attends to, but has been investigated in prior work. We look for component matrices that have a high composition score with 9.9 QK matrix, and plot heatmap of the highest composing component matrix per head in Figure We find that there are several heads that have components that highly compose with 9.9 that have not been previously identified as affecting copying behavior. These are heads 6.1.1, 6.2.5, 6.7.3, and 8.3.0. We show the per component composition scores for these heads in Figure 24 and find
23
20

---

GPT2 7.3
you losow, the simed
duro would
a resigned myseltto Toaee
have
calmed
insewveww your Korez
wil, and then calmed down,,
Park Salss In
an imtoisow yhut the Korez Times
higgeet imposssion
Thong,
her
meeilooed
2hong and 1ald I hope meetagain
creating
.metadara othancement and clean creating now online cobbio
ond the DACe-eys acoess por tabs
preted rewed superttrr irioin fishian) cromaally
normally
reeerved for concerts swarrdshove,..
irfoina fishioni normally
ureted the NBk's superta bioina fashion pormaly
fresevvdd
rock concerts and shows, Jmmes
Figure 15: Examples of high attention in examples in OpenWebTextt for head 7.3 in GPT-2
that they also contain individual components with very high relative composition compared to other components (with 6.2 components being distributed slightly more smoothly than the others).
1.2.1 Attention Pattern Analysis of 6.1, 6.2, 6.7, and 8.3
We perform similar attention pattern analysis as shown in Section on these heads. We run the model on documents from WehnWee 10K and look at contexts in which these heads attend strongly (2 of probability mass) to some token. We find that 6.2 is not interpretable in this way, as its attention patterns more diffuse, but examples for the other heads are shown Figures 21 and 20 We find qualitatively similar patterns in these heads to inhibition heads, though with less selection for attending previous items in lists. Head 8.3 is of particular interest because it has the strongest composition with 9.9, and we will focus on it for the remainder of this section. This head also has some particularly salient motifs in its attention patterns: much of the time appears to attend to token (or the last token a multitoken phrase) in context that could be plausible continuation of the current text. In the following section explore what the possible function of this head is, and its connection to the 9.9 mover head.
22
I.2.2 Attention Head 8.3 as a Relevant Context Head
The attention patterns in Figure 20 suggest this head plays some role in attending to relevant
9.9 mover head QK circuit (Figure 23 also suggests it is modulating what the copying head attends to. But what is the actual function of this head? First, we establish that this head not a mover or inhibition head, then provide evidence that it is head involved in signaling relevant context that can be copied to the next token by the
continuations the current context. The strong composition
21

---

Remove Component from 6.6 to 9.5
0.9
Vanilla
Remove Component
0.85
Inhibition Score
0.8
0.75
0.7
0.65
0.6
40
60
Component Index
Figure 16: Pythia-160m also has single component (6.6.2) in an inhibition head that dominates the inhibition signal.
mover head (9.9). We begin outline how exactly the 8.3.0 component matrix interacts with the attention pattern of 9.9 with some success, but can not make strong conclusions because we can not control for every variable in the data we have available. We hope this work encourages future study on this head's interactions with mover heads as well as future work in static weight analysis more generally.
8.3 not a mover head It's possible that 8.3 acts like a mover head, copying the tokens it attends into the residual stream, which incidentally causes 9.9 to do the same. We perform the IOI copying head test from Wang et al. (2022) and find that this is likely not the case. In this test, the output of an attention head is decoded into the vocab space. If the top token from this decoding is the same the token it attends to, then it is "copying" that token. We find that 8.3 gets 0% on this test, while 9,9 gets 100%. Other inhibition heads also get 0%, so next we explore whether 8.3 could be interpreted type of inhibition head.
8.3 is not inhibition head We include 8.3.0 (the zeroth component in 8.3, which strongly composes with 9.9) in our analysis in Figure [T] and find it does not act like an inhibition head on the 10I task (although, as we could expect from the composition score, does modulate the inhibition scores). Additionally, removing the zeroth component reduces the inhibition score on IOI by less than .01 (same experiment as performed in Figure 2 From these experiments, we can not rule out that 8.3 never acts like an inhibition, i.e., it could do so distribution of text we do not test, but evidence seems point against this. Regardless, we can provide evidence for this head being involved with identifying relevant continuations (rather than preventing repetitions) in the next sections
8.3 attends to relevant continuations of the current context Based on the observational evidence from the attention patterns on excerpts from Open WebText- 10K, we hypothesize that 8.3 will selectively attend tokens that are logical continuations of the current text. To test this, we create small synthetic counterfactual dataset of verbs and nouns to test 8.3's attentional preferences. Our intuition that: given some mention of a noun n context ("pork"), and a verb v later context ("eat"), 8.3 will only attend to n if it is appropriate direct object of e. Therefore, in the contexts "I see the pork. am eating" and "...I am drinking". ". 8.3 will attend from eating to "pork" strongly, but not from "drinking" to "pork", since this is not typical object of the verb "drink". We generate 84
22

---

read watch open write Composition Scores for 4.11 to Component Matrices of 6.6
0.14
0.12
Comoposition Score
0.1
0.08
0.06
0.04
10
20
30
40
50
60
Component Index
Figure 17: An example of an induction head (4.11) value composing strongly with the single inhibition component 6.6.2 in Pythia-160m, suggesting a circuit for controlling attention through mover head 9.5. We leave analysis of this for future work.
Verbs cook eat drink
Likely Objects
[carrot, cabbage, steak, pork, chicken [carrot, cabbage, steak, pork, chicken] jvater,jjice, milk, soda, coffee]
[book, newspaper, magazine, comic, blog] [movie, TV, video, show, cartoon] [door, window, box, jar, can]
[letter, note, script, report] Igame, piano, guitar, violin, drum, song] [wall, fence, door, floor, ceiling]
play
paint
Table 1: Verbs and appropriate objects used to test 8.3's attention patterns to relevant objects
counterfactual pairs of verb and likely/unlikely object using the verbs and nouns in Table An unlikely object is any noun that does not appear in the verb's 's list of likely objects. We find that 8.3 attends much more strongly to nouns that likely follow the given verb. Results are shown in Figure
Causal Interventions on 8.3.0 Next, we use the output space of the 8.3.0 component as a steering vector (since it outputs onto line) to change the attention of 9.9, as predicted by the composition score to 9.9's QK circuit. We use excerpts from documents in Open WebText-10K that highly activate 8.3 (Figure 20). We hypothesize that 8.3 plays some role in telling 9.9 to attend and copy to relevant context through zeroth component matrix. Specifically, we test whether adding some constant times the 8.3.0 output vector to the residual stream will cause 9.9 to attend more less to the token
23

---

Training Progression of Inhibition Component Subspace and Mover Head Inhibition Score
Inhib, Comp. Separability
Original Inhibition Score
Separability or Inhibition Score
inbibition Comp 6.6.2 Removed
0.8
inhibition Score with Comp 6.6.2 Added
Inhibition with Comp 6.6.2 Added 3x
0.6
Inhibition Score with Comp 6.6.2 Added 10x
0.4
0.2
0
50k
100k
150k
Training Step Checkpoint
Figure 18: Pythia training progression of inhibition component (6.6.2) and effect of model editing. Adding the component matrix to the inhbition head strengthens the inhibition channel and improves the ability to use inhibition in earlier checkpoints, subtracting it makes inhibition weaker. Separability is simply the extent which activations for IOI minimal pairs split into clusters based the order names (IO, S1 or S1, 1O).
8.3 attends to (following a similar methodology as in Figure 4). For instance, given the input "would be better negotiate with David, give [him] some money and [allow]", 8.3 attends from "allow" to "him". We find that this does tend to be the case, although it is difficult to quantitatively measure. Our results in Figure 26 show per-example attention scores after adding 8.3.0 multiplied by -1000 to the residual stream. We see a slight increase on average, but the results are inconsistent across examples. One reason because of multitoken phrases; e.g., for phrases like "the corridor". 8.3 will attend consistently to the last token in the phrase, here "corridor". But if 9.9 is to copy this phrase, it should attend "the", not corridor. Our test does not cover this, and it isn 't clear how fairly handle each edge case. We suggest that some interaction like the one we' ve described here is taking place, but leave more rigorous analysis on the exact interactions between 8.3 and 9.9 for further study.
More Information on Composition
J.1 Value Composition: Duplicate Token Heads
Value composition dictates that the value vectors of earlier head write information that affect the values of later heads. Duplicate token heads are a well established type of attention head that specialize in attending duplicates in the previously seen context. That is, given the text "A B A" the duplicate token head will attend from the second "A" token to the first. The IOI circuit finds value composition between heads 3.0, a duplicate token head, and 7.9, inhibition head.
J.2 Query Composition: Inhibition Heads
Value vectors of earlier heads affect the query vectors of later heads, thus changing what they attend to. A canonical example originating in the IOI paper is with inhibition heads. These are key part in a token copying circuit in which the value vectors of such heads prevent later query vectors from attending to the duplicated name the IOI task. Mover heads (such as 9.9) We study these heads in greater detail in Section Whatever a mover head attends to will be promoted as the next token prediction. An inhibition head tells a mover head to avoid attending to certain tokens, which helpful when there are multiple options to generate.
24

---

Composition Scores to 7.9
Composition Scores to 7.9.6
-32
Composition Scores to 9.9
"
Composition Scores to 9.9.0
1084
Figure 19: Decomposing weight matrices cleans up the composition score enough that we start to read off components that belong to the IOI circuit without running model. By starting with known inhibition head component (7.9.6) we find the heads that compose into that component and the heads for which the inhibition component composes into that belong to the IOI circuit from Wang et al. [2022]. Left graphs show the composition score without any decomposition, which is noisy. On the right, we find in-circuit heads (circled) qualitatively to stand out more. See Wang et al. 2022
for more details.
J.3
Key Composition: Induction Heads
An induction head is pattern completing attention head. For example, seeing the pattern "A B A B A" will cause the model attend from the last A the last since pattern is present where B must follow A. The mechanism that typically implements induction heads requires a previous token head (which will always attend from the current token to the one right before it) affect the key of later induction head. At later timestep the query of the induction head will notice the signal left in the earlier key, and choose attend it. We consider the key composition between the previous token head 4.11 to induction head 5.5 from the IOI circuit.
K
Extra Laundry List Interventions
The results for scaling individual components across Laundry List datasets (varying number of objects) are in Figures 28
29 30
35 and 36
We also include results from traversing the 3D inhibition subspace used in Figure for greater range of settings for the number of objects in Laundry List dataset examples. We test datasets set to have 10 objects and one dataset to have 20 objects. The results are shown in Figure
25

---

GPT2 8.3
Heter Levis San francisco Chronice
sslaff amber. E-ma bihlo
Wchroiicl.ccom Twitter ghan
High HonryKd Loe
Misedeerookiiccom Twittor beryyllofissheeuuuueeo
han umiib with with spl, atoon amiioo serie Inkking squid fresh fnnhaabrrate colors, these amribos all encling Marl in your mom. You might ware thee
purchas ryg uning standing @uaciasicrt
d ation bosths,
saking effect, wome schools scnambling way sround
continue stusints Solitrry sttick disorcion.
Would be betterto to negotione with David, to give. Nin 1. some comemoneyynn and tor allow 10. Nin wit
Termiinneur Besdliees ip
Septersber soseee
13399 UTC) forriiiiieeeeeeeeeeee in
Newember 201 Witteling +
tocay Paicy
thicspanch.
Hobody can bringost rolly after
after
Trally Americans who lost loved beesiteeeeeeeeee
rnlyy Arw.rramamannno..o who inved oree becxuse these immigortan lewe
weee enforred
American lnd joins because
dippavementt frllow through
of brpsppicrringing their own smartin ones and tabless the, office
(ind.gol going back even further, the iself was oflen brought) into
ccopariis ona deportmental lovel) Asimlar
Figure 20: Examples of high attention in examples in Open OppenWeTexx--lle 10K for head 8.3 GPT-2
26

---

GPT2 6.1
Jung and Yun Okr were particular dominaatt, It, breaking Lwo world iccorda and
desooyying France by points seniiiral
way needed just
victory perfect
was a0 in Threegold medals
fourhh bockaned Seeded first
dis and the Book of Normon 1+) His dains
very people, Nephibes, according
revelation
Thet the truhh wourd from the wh and theywoold @-Oeiwindle urbel
huge issu for Goddf-fe Inaders the Book 1
Mormon ln ln contract contrcaz we find
apontsry
those men like
Smafle Her wel, Eroo with
appliaaiitty Netther wildbew
View the whole Intermatiiinnn ona mtymuuityy is dheillocoooorn about Instbution and
bondes should be taienina fully sicw
Figure 21: Examples of high attention in examples in OpenWeeText- for head 6.1 GPT-2
27

---

GPT2 6.7
grandfuther, Gab
Once you complete the laves, put back into this freesse for
lesst un hour to firm upand
begen
to
Mast of my income housing
duddem loars, payment, food and other
festielss in'n The point of living with 3 forme Toot rbout NOT businghhings (
on had!
libless Oresmall
2008rocce ynin For momimitrryytttn voters, the cconomyy was for nt importattsuue with percent respondents saying it was most Important
presers "We everybody,
whit walted
hos Aal
since Sept.5, 5, the lekchander struch out
Figure 22: Examples of high attention in examples in OpenWeeText-100 for head 6.7 GPT-2
28

---

Composition Scores Between Max Component of Every Head and 9.9 Query (Mover Head)
0.1
2-
0.08
Layer
4-
0.06
0.04
10
Head
Figure 23: We look at the top scoring component matrices in every head before layer 9 to attention head 9.9, We find several heads not previously identified in circuit analysis that have strong com- munication to it. These are attention heads 6.1.1. 6.2.5, 6.7.3, and 8.3.0. 8.3.0 in particular, has the strongest composition with 9.9 out of any head.
Composition Scores for 6.1 Components 9.9
Composition Scores Components 9.9
X=1
X
y=.103
y=, 100
Composition Soare
6.69
Compositinn Score
0.22
0.07
Component
Component
Composition Components
Compositio Components
5=5
015
x=
y=0096
y=,117 117
Composition Score
Composition Score
Component
Component
Figure 24: Per component composition scores for the identified heads communicating with 9.9, Each has an outlier component that communicates most strongly with 9.9's QK circuit.
29

---

Mean Attention for Likely and Unlikely Nouns 8.3
0.5
0.4
Mean Attention
0.3
0.2
0.1
0
Likely Nouns
Unlikely Nouns
Figure 25: When predicting the continuation of sentences like 'I saw the book. I am reading the', head 8.3 attends much more strongly the given noun (book) when the verb appropriate vs. inappropriate (e.g., eating). The average attention score is around 35% for likely nouns and around 5% for unlikely nouns.
Attention Scores for 9.9 with and without Steering 8.3.0
Original
Scaled B.3.0
0.8
Attention from 9.9
0.4
10
20
30
Example
Figure 26: Interventions on 8.3.0 cause an increase in the attention of 9.9 onto the token 8.3 attends, although inconsistently. The x-axis shows results for individual examples. This is average increase from 9% to 12%.
30

---

regez
dipect
Cbedt
o0gict
Cteez
Otjecz
Objecd
Objecd
Otject
Object
Obee
Obeec
Obtecc
Object
138
Oieec
Oijec
Digec
Oipect
Oipec
Vasla
Otje11
Otject
Otpect
1085
755
Figure 27: How the 3D inhibition subspace responds to a different number objects in laundry list prompts. As we add objects, new 'slice" of the space is allocated (not always visible) for attention to that object until the middle set of objects is squeezed into small neighborhood of the space. The space is very well structured, except for two cases where artifacts form in the 8 and 10 object settings.
31

---

Effect Singular Vector 7.3.1
Effect Singular Vector 7.9.6 Scale on Mover Heed 9.9 Attention
Scale on Mover Head 9.9 Attention
Score Ratenn
0.5
0.8
0.3
0.5
2000
022
0.4
0.3
Ttem Positin
Them Positton
-100
100
-50
-75
-50
75
Singular Vector Scale
25
Singular Vector Scale
5g
50
75
75
100
Effect of Singular Vector 8.6.2 Scale Mover Head 9.9 Altention
Effect of Singular Vector 8.10.1 Scale Mover Head 9.9 Altention
Score Adenne
0.8
0.4
0.3
0.0
0.0
Item Rosition
Htm Positon
-2
Singudar Vector
25
50
Singulari Vector
25
100
Figure 28: 2 Objects
32

---

Singular Vector 7.3.1
Singular 7.9.6
Scale on Mover Head 9.9 Altention
Scale on Mover Head 9.9 Attention
0.30
0.25
0.3
0.20
0.15
Ntm Position
Rosittom
25
-50
-75
25
-50
Singular Wector Scale
Singular Wector Scale
100
100
Effect of Singular Vector 8.6.2 Scale on Mover Head 9.9 Attention
Effect of Singular Vector 8.10.1 Scale on Mover Head 9.9 Attention
20t0 Score
0.4
0.4
0.3
0.1
Item Rositon
Rositon
-50
-75
-300
-50
-75
200
Singular Mector Scale
Singular Scale
50
50
73
100
Figure 29: 3 Objects
33

---

Effect of Singular Vector 7.3.1
Effect of Singular Vector 7.9.6
Scale Mover Head Attention
Scale Mover Head Attention
20.40
0.30
0.335
0.25
0.000
3 0.25
0.15
000.00
0.15
0.10
0.10
Item Positon
Item Positod
-75
100
-50
-75
-100
Singular Vector Scale
25
Singuiar Vector Scale
50
100
75
5g
Effect Singular Vector 8.5.2
Effect Singular Vector 8.10.1 on Heed Altention
Mover Heed Allention
0.400
00000
0.35
0.35
0.300
0.30
0.15
2 0.15
0.10
0.10
3.05
0.05
itmm Postiio
item Resilon
-100
-100
25 Singdar Vedor Scale
50
Singuior Vector
25
Figure 30: 4 Objects
34

---

Effect of Singular Vector 7.3.1 Scale on Mover Head 9.9 Attention
Effect of Singular Vector 7.9.6
Scale Mover Head 9.9 Attention
0.25
0000000000
0.20
3
0.25
50
0.20
0.15
0.10
0.10
0.05
0.05
0.00
0.00
Inem Positim
Rositto
2
100
-100
-50
-75
50
75
Singular Vector Scale
Singular Vector Scale
30
50
75
100
75
Effect of Singular Vector 8.6.2 on Mover Head Attention
Effect Singular Vector 8.10.1 Scale Attention
E 20.35
0.30
0.25
8 0.25
0.20
0.20
Pos
-100
1000
-25
Singular Vector Scale
25
4
Singular Vector Scale
100
100
Figure 31: Objects
35

---

Effect Singular Scale on Mover Head 9.9 Atcention
Effect of Singular Scale on Mover Head Attention
0.30
0.20
0.15
0.20
0.10
0.15
0.05
0.00
0.00
Rem Positon
2
3
25
25
Singular Vector
25
5
Singular Vector
100
100
Effect Singular Vector 8.6.2 Scale on Mover Head 9.9 Attention
Effect Singular Vector 8.10.1 Scale on Mover Head 9.9 Attention
3408 Uecuass
0.30
00.35
0.25
0.30
0000 0.25
0,20
0.15
0.10
0.05
0.00
0.00
Item Postbio
2
3
-25
-50
-2
30
Singular Vector Scale
Singular Vector Scale
100
100
Figure 32: 6 Objects
36

---

Effect Singular Scale on Mover Head 9.9 Attention
Effect of Singular Scale on Mover Head 9.9 Attention
0.30
0.20
0.15
0.10
0.05
0.00
0.00
Posttaan
Posttin
-00
-0
-25
25
Singular Vector Scale
Singuaar Vector Scale
100
100
Effect Singular Vector 8.6.2 Scale on Mover Head 9.9 Attention
Effect Singular Vector B.10.1 Scale on Mover Head 9.9 Attention
0.25
0.2
0.25
0.15
0.20
0.05
0.05
0.00
0.00
Pestton
Rosiion
-00
-25
-0
-75
-25
-30
Singular Vector Scale
Singular Vector Scale
100
75
5g
100
75
Figure 33: 7 Objects
37

---

Effect of Singular Vector 7.3.1
Effect of Singular Vector 7.9.6 Scale on Mover Head 9.9 Attention
Scale on Mover Head 9.9 Attention
0.25
0.30
0.20
0.25
500
0.15
0.20
0.10
0.05
0.00
Iten Ponitiis
100
100
-75
75
50 25
Singular Vector Scale
50 25
Singular Vector Scale
Effect Singular Vector Scale on Mover Head 9.9 Attention
Effect of singular Vector on Mover Head 9.9 Attention
0.5
0.30
0.20
0.25
0.15
0.20
0.15
0.10
0.10
0.05
0.00
0.00
1
Hten Positton
-75
755
23
-00
25
-30
Singular Victor Scale
Singular Victoo Scale
100
100
Figure 34: 8 Objects
38

---

Effect Singular Vector 7.3.1
Effect of Singular Vector 7.9.6 Scale on Mover Head 9.9 Attention
Scale on Mover Head 9.9 Attention
0.25
0.30
0.0
0.25
30
0.15
0.10
0.05
0.00
0.00
BoOm Rostton
Positon
-75
-50
-50
-2
-25
Singslar Vector Scale
Singdar Vector Scale
100
75
100
75
Effect of Singular Vector 8.6.2 Scale on Mover Head 9.9 Attention
Effect of Singular Vector 8.10 Scale Mover Head 9.9 Attention
0.25
2.330
0.25
0005
0.15
0.10
0.00
0.00
Postton
-75
110
75
200
-50
-50
Singular Vector Smale
Singular Vector Scale
50
100
75
100
75
Figure 35: 9 Objects
39

---

Effect of Singular Vector 7.3.1 Scale Mover Head 9.9 Attention
Effect of Singular Vector 7.9.6
Scale Mover Head 9.9 Attention
0.25
0.30
0.25
0.15
0.15
0.10
0.05
0.05
Rostion
100
-100
25
25
Singular Vector Scale
Singular Vector Scale
9
9
100
100
of Singular Vectar Scale on Mover Head 9.9 Attention
Effect Singular Scale on Mover Head 9.9 Attention
20.30
0.15
0.20
0.30
0.10
0.05
0.00
0.00
1100
100
-75
-75
8
-50 Singular Vector Scale
25
-50 Singular Vector Scale
25
100
75
100 75
Figure 36: 10 Objects
L Compute
The models we use in this paper are small, in the range of 100M parameter tranfsormer models. The compute required to reproduce the results therefore relatively small, but does require access to modern GPUs. We primarily used Nvidia 3090 GPUs for this work. Running the linear combinations of inhibition components in Section was the most expensive experiment, Each dataset took about 12 hours on either a RTX 3090 or Quadro RTX gpu.
40

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We provide evidence for our papers abstract and intro claims in Sections and Additional supplemental experiments are in the appendix
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
41

---

Justification:
We have a limitations section in Section A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Besides the main paper details, we have additional details on replicating experiments, like how to generate the laundry list task in Section
Guidelines:
42

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release the scripts to generate the data and reproduce the experiments in the main paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
cablic//uides/Codeee..
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppulic//guue http://icc/giiie////eeeuubiiibii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
43

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All hyperparameters are included in text or in the figures where applicable. For example, the values scale singular vectors by are included in figures like Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We include error bars in out inhibition measurements and include controls for experiments on specific heads by including random heads ($4)
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We answer these questions to the best of our abilities in Appendix L
Guidelines:
44

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: https://meee http:::////eu...pp/// httplpp..cpppubup wttppeblic/...iiccccieee.......e
Answer: [Yes]
Justification:
We have reviewed the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work does not have immediate application but do discuss the downstream impact of our work on safety and controllability of LMs throughout the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
45

---

Answer:
[NA]
Justification:
No such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Previous work is cited properly.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com ddtasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The Laundry List task is very simple and documented in Section F and code generate it is provided.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their
submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
46

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdworkerss
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No IRB required.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
47


Reviews:
[{"review_id": "AXeRljKrd1", "reviewer": "Reviewer_WKbg", "summary": "This paper primarily employs low-rank communication channels to elucidate how internal layers within the Transformer model transmit information. Initially, the study utilizes existing research to identify duplicate heads, inhibition heads, and mover heads. The objective is then to explore the interactions among these heads. The findings indicate that directly incorporating these elements into the computation of the Composition Score does not yield significant results due to the complexity of the signals involved. Consequently, the paper proposes the application of Singular Value Decomposition (SVD) on the weight matrix, with subsequent sorting by variance, to identify the principal read/write subspace. This approach identifies such communication channels solely through the model's weight file. Furthermore, in certain recall tasks, such as the Indirect Object Identification (IOI) and Laundry List Task, this paper leverages the low-rank characteristics of communication channels to enhance task accuracy.", "questions": "1. The primary experiments in this paper were conducted on the IOI and Laundry List Task. Notably, the Laundry List Task has significant limitations in excluding repeated words. Inhibiting repeated words can only be considered a part of the recall task. Is it possible that recall tasks are the ones where repeated words appear most frequently? Some tasks should enhance the inhibitory effect on repeated words, while others should reduce it. Therefore, the experimental tasks in this paper are insufficient.\n2. In Figure 2, the Inhibition Score is presented, but it is not introduced until Section 3.3.1, which may confuse readers. The use of \\( V \\) in Equation 2 may confuse readers with the \\( V \\) matrix in the Transformer model. There is a spelling error in the caption of Figure 3: \"inhiibt\" should be \"inhibit.\" \n3. The model-editing method proposed in this article involves performing SVD decomposition on a specific head (the inhibition head) and retaining the main subspace components. This approach improves performance on the Laundry List Task, indicating an enhanced inhibitory effect on repeated words. The intervention method in this article primarily demonstrates that other signals unrelated to inhibitory ability exist within the inhibition head, and that dimension reduction can enhance inhibitory capacity.  It should be noted that identifying the inhibition head is not a contribution of this article; rather, it relies on the conclusions of previous work. This article merely reinforces that the inhibition head functions as an inhibitory mechanism and that its combination with the duplicate head improves the inhibition of repeated words, a conclusion already established in prior research. Additionally, this article does not propose a new mechanism for inter-layer communication in the Transformer model.", "limitations": "The author has not fully addressed the limitations of their research. While the proposed method improves the model's inhibitory effect on repeated words, it primarily relies on existing findings and does not introduce a new mechanism for inter-layer communication in the Transformer model. Additionally, the experimental tasks used in the study, such as the Laundry List Task, have notable limitations, enhance the inhibitory effect on repeated words, while others maybe should reduce it.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This article provides a detailed background to facilitate readers' understanding of prior research.\n2. The article presents a hypothesis, verifies it through experiments, and conducts further in-depth research, thereby demonstrating a complete scientific research process.\n3. To validate the proposed hypothesis, the article also develops a Laundry List Task for specific experimental verification.", "weaknesses": "1. As part of its contributions, this article presents a method for identifying nearly complete IOI circuit signals in GPT-2 Small directly through the weight file. However, the instructions provided in lines [148-152] are too brief, making the specific method difficult to understand.\n2. It is well known that the matrix of the attention head is sparse, which naturally suggests using Singular Value Decomposition (SVD) to compress the original matrix. Although the primary aim of this article is to identify subspaces, there is no fundamental difference between the two approaches. Numerous related works, such as LoRA, have applied SVD to the original model weights. Moreover, the conclusion that inter-layer communication is low-rank seems evident due to the inherent low rank of the attention head weight matrix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> “It is well known that the matrix of the attention head is sparse, which naturally suggests using Singular Value Decomposition (SVD) to compress the original matrix… the conclusion that inter-layer communication is low-rank seems evident due to the inherent low rank of the attention head weight matrix”\n\nAttention heads are inherently low rank because of the size of the matrices (e.g., 64x768, projecting down to 64 dims). This is much different from the low rank subspaces we identify here, which are 1d or 2d. We disagree with the claim that this is somehow self-evident, because we find that not all compositions between communicating matrices have this property. For example, see FIgure 6, left. We know that previous token head 4.11 composes with induction head 5.5 from prior circuit analysis (Wang et al., 2023), but they seem to do so with nearly the full-rank space of the attention head.\n\n> “Numerous related works, such as LoRA, have applied SVD to the original model weights.” \n\nDoes the above point address the concern about the connection to LoRA? If not, could the reviewer please expand on this? We are familiar with some of the work on LoRA that is relevant but we are not aware of any work that implies the results we find in this paper.\n\n> “the instructions provided in lines [148-152] are too brief, making the specific method difficult to understand.”\n\nThanks for raising this. We take the distribution of composition scores between component matrices and another matrix (such as mover head 9.9) and take the highest z score components (the outliers). We do not determine an optimal threshold in this work, but we use >4 (upon visual inspection the outliers are extremely obvious: see Figure 6 middle and right).\n\n> “In Figure 2, the Inhibition Score is presented, but it is not introduced until Section 3.3.1, which may confuse readers. The use of ( V ) in Equation 2 may confuse readers with the ( V ) matrix in the Transformer model. “\n\nThanks for pointing this out, we will address this in the camera ready by introducing the term earlier.\n\n>  “This article merely reinforces that the inhibition head functions as an inhibitory mechanism”\nNo other work has been able to find a known circuit within a model solely from the weights without running the model. We think this is a significant contribution to the field. We focus on a well known circuit to establish credibility without too much overhead of verifying a new circuit.\n\nWe localize the inhibition mechanism to a few dimensions in the attention heads. We establish that traversing this space (by intervening on the outputs of these heads) controls the position of the token inhibited completely independently of the content of the token. That is, the edit we make is ‘ignorant’ to the preceding context, yet the downstream effect is predictable. This has not been established by previous literature\n\n> “this article does not propose a new mechanism for inter-layer communication in the Transformer model.”\n\nThe new mechanism is the way that this is implemented in the model, with extremely low rank signals. ‘Bandwidth’ in the residual stream has been hypothesized as being in high demand (Elhage et al., 2021), but it has not been established how these signals are passed. We thoroughly justify the claim that this is one such method native to the transformer (see point 1).\n\nWe believe we have clarified the points raised by the reviewer. This paper makes heavy use of the appendix for supporting figures, which we know reviewers are not necessarily responsible for, so we have provided some pointers here. We hope the reviewer considers these in their determination of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P4Gev7T5sb", "reviewer": "Reviewer_Veos", "summary": "Building upon prior work in Transformer circuits (Elhage et al., 2021; Wang et al., 2022), the paper identifies novel, low-rank communication channels between attention heads across layers via the Composition Score (a generalized cosine between the read-out weights of a lower-level layer and the read-in weights of an upper-level layer) between _low-rank_ factorizations of the weight matrices (obtained by SVD). The low-rank decomposition is critical, resolving the known issue of the vanilla Composition Score being very noisy.\n\nGenerally focusing on the role of inhibition heads, which are known to prevent the copying of certain tokens in the prompt, the paper identifies  a low-rank communication channel for inhibition and validates its role on a synthetic \"Laundry List\" task. The task requires recalling an item from a list of items specified in the prompt. It is shown that, when the identified (rank-1) inhibition channel is zeroed out (intervention), the model effectively stops passing the \"inhibition signal.\" The role of the channel is validated for indirect object identification (IOI) and token indexing tasks in the GPT2-small model.", "questions": "- Are there other baseline methods to be considered beyond the vanilla and SVD-based Composition Scores for identifying communication channels?\n- Page 6, lines 217--218: How exactly is the 2D duplicate token head found?\n- Page 7, line 227: In what sense are these results “strong”? Is there a baseline?\n- Page 7, lines 228--232: any quantitative results for this claim?\n- Page 8, line 266: why is it top-3 this time? how should someone replicating your experiments choose this number?\n- I think the talking heads analogy makes sense, but it’s one that was already used before by [Shazeer et al. (2020)](https://arxiv.org/abs/2003.02436) in a not-so-unrelated context (the paper is fully referenced in the circuits paper, for example). I would suggest changing the title or at least adding a footnote of disambiguation from that paper.", "limitations": "The paper states that it does not make claims about the different types of compositions across attention heads in different layers. Aside from that, the limitations are not well addressed; see weaknesses above.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- Using the SVD as a remedy for fixing the noisy Composition Score is an intuitive and seemingly effective strategy.\n- The paper’s main contributions, including the method and the new synthetic dataset, are reasonably well-motivated.\n- The empirical results make sense (for the most part) and support the claim of low-rank communication channels for inhibition heads. It is interesting to see that the inhibition channels are content-independent (just pointing at names in the IOI task) or that it affects the accuracy on the laundry list task.", "weaknesses": "- Overall, while the paper is well-motivated and includes potentially interesting results, its presentation significantly hinders the reader from understanding the main takeaways of the paper.\n    - First, a key piece of related work is the composition analysis of two-layer, attention-only models by Elhage et al. (2021), but the review of the relevant background work is simply too terse and unorganized. Terms like inhibition heads, inhibitor-mover subcircuit, value composition, and the QK/OV circuits (e.g., which weight matrices are we talking about exactly? why are they “low-rank”, especially when it’s not as low-rank as the subspaces we’re discussing here?), are not formally defined. Even things like why the composition score looks the way it is (why is it a matrix multiplication?) or the basic shapes of the weight matrices ($d$ by $d_h$ or $d_h$ by $d$?) involved are not written out in the “background” section. Without these in place, I think the paper is borderline incomprehensible to anyone outside the mech interp community.\n    - Second, all the overclaims in the paper really hurt the overall message. The paper’s main thesis could simply have been something like “using the SVD in the Composition Score allows us to identify low-rank communication channels, as showcased by the identification of inhibition head channels,” which would be clear and interesting. But instead, the paper keeps trying to convey a broader, unsubstantiated message that their methods and experiments somehow are more novel and general than they should be.\n        - “Understanding Inter-Layer Communication”: Exactly what is the scope of this inter-layer communication? Is it primarily/always from layer $l-1$ to $l$, or something more?\n        - ”… using subspaces we call communication channels” (Intro): this terminology already existed in Elhage et al. (2021), but this phrasing makes it sound like the current paper came up with it.\n        - “model components like attention heads” (what other components does this paper cover?)\n        - “Three types of composition”: It appears that most of the focus in the paper, including the dataset itself, is on inhibition heads.\n        - “Composition Score”: It should be clear to the reader that the score was first developed by Elhage et al. (2021), especially in Section 3.1. More generally, the intro should do a better job of clearly disambiguating what was introduced already in the literature and what this paper is adding to it.\n    - Lastly, the writing is simply not well polished, and there are many typos and unnatural sentence structures that make it even more difficult to understand the overall paper. Just to name some of them:\n        - Abstract, line 3 & line 11\n        - Intro, line 34 (\\citep)\n        - Page 4, line 114\n        - Page 4, line 128 (d \\times d)\n        - Figure 2 caption, second sentence\n        - Page 5, lines 170—172 (broken parallelism)\n        - Figure 3 caption, second sentence\n        - Page 9, line 276\n        \n        There’s also a related issue of overusing informal analogies unnecessarily (attention heads “talking” to each other). \n        \n- Aside from the presentation, the biggest confusion for me is: generally, what implications do we have about inter-layer communication in Transformers from these findings? Besides inhibitions, what compositions do we expect to be represented as low-rank subspaces according to the paper’s approach? How far across layers can the channels be? Do we have any sense of what the rank tells us about the feature? The discussion section should be expanded substantially to at least mention or address some of these points.\n- In terms of the proposed Laundry List task, a possible caveat is that models more recent (and larger) than GPT2-Small may actually do a good enough job of solving them. I am sympathetic to the fact that it is challenging to redo many of the analyses on larger models, but I think it is worth, at least, to test the accuracy (Fig. 1, left) on more recent, open-weight models, to see how much the motivation in the introduction is still relevant.\n- The related work on SVD seems terse, knowing that SVD is arguably one of the most basic linear algebra tools in ML (any method based on PCA would also be relevant, for example).\n- In the intervention experiments, the choices of the exact subspace (the number of components) and the intervention method (e.g., scaling diagonally for 2D) appear a bit heuristic and need better justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We are glad the reviewer found the methods interesting and well-motivated but disappointed the paper was difficult to read. This is a complicated topic that needs to fit in 9 pages, so it is difficult to catch everyone up and make a significant contribution in that time. Below, we answer questions, outline some proposed changes, and would also like to use this space to push back on the notion that we are overclaiming. After considering our points we would like to hear if the reviewer still feels this way/what they specifically feel is being overclaimed.\n\n>“Understanding Inter-Layer Communication”: Exactly what is the scope of this inter-layer communication? Is it primarily/always from layer l−1 to l, or something more?\n\nWe do not find a preference for more recent layers. Please see Figure 2: heads in layer 7 communicate with the mover head in layer 9. The duplicate token head (3.0, see Figure 4) communicates with inhibition head 7.9 (a difference of four layers).\n\n> “using subspaces we call communication channels” (Intro): this terminology already existed in Elhage et al. (2021)\n\nWe use this term to ground the paper to previously established literature but also provide a more specific definition rather than a notion of communication. Elhage et al., is extensively cited in this work, but this point is well taken because it was not our intention to appear to coin this term. This will be reframed in the camera ready to refer to this paper when introducing “communication channels”.\n\n> “QK/OV circuits (e.g., which weight matrices are we talking about exactly? why are they “low-rank””\n\nThese are low-rank because of their shapes. For example, in one head, the value matrix projects from the embedding dimension (768) to the head dimension (64). E.g., the value matrix is 64x768 dims. This is therefore lower rank than the embedding dim. This has been consistent in the transformer architecture since Vaswani et al,. 2017 so we do expect some familiarity from a general reader.\nIf the reviewer agrees it would be helpful, we would be more than happy to define these terms in a glossary in the appendix.\n\nRelatedly, we’d like to point out this isn’t a typo:\n>Page 4, line 128 (d \\times d)\n\nWe will add something like “V \\in d x d_h and O \\in d_h x d, so OV \\in (d \\times d)” to make this very clear\n\n\n> There’s also a related issue of overusing informal analogies unnecessarily (attention heads “talking” to each other).\n\nWe tried to balance formalisms with intuitive analogies due to the complexity of the work. If the reviewer could, we would be interested in hearing which specific places more rigorous definitions would be helpful.\n\n> “all the overclaims in the paper really hurt the overall message”\n\nWe are generally a bit confused about what the reviewer finds to be overclaimed. Perhaps the above points help clarify our work more. If not we would appreciate some specifics if the reviewer still feels this point is relevant after the rebuttal.\n\n> In terms of the proposed Laundry List task, a possible caveat is that models more recent (and larger) than GPT2-Small may actually do a good enough job of solving them.\n\nThis is a great suggestion, please see our rebuttal doc. We tested multiple n-billion parameter models (up to 6.9B) from different model families and found that the larger/more recent models tend to get better but still struggle. We’d also like to point out some relevant related work:\n[Liu et al., 2023](https://arxiv.org/abs/2307.03172): highlights the broader inability of LMs to use information when it’s in the middle of a long context. This negatively affects retrieval augmented generation (RAG) systems, which are highly relevant to current SotA systems. We don’t claim this is a failure of the exact same mechanism, but it definitely connects our work to a substantial current problem.\n\nThis point may also help answer the following: \n\n>”[a] confusion for me is: generally, what implications do we have about inter-layer communication in Transformers from these findings?”\n\nOur work points to a concrete capacity limit in inter-layer communication that leads to performance degradation on certain recall tasks. This work provides a promising avenue for understanding and improving “lost in the middle (Liu at al) type failures.\n\n> “Page 7, line 227: In what sense are these results “strong”? Is there a baseline?”\n\nYes there is a baseline. In Makelov et al., 2023, they examine the IOI dataset. As a reminder, we report 97.5% and 35% with components directly taken from the weights. Their baseline achieves -8% FLDD 0.0% Interchange. By taking the gradient to directly optimize a single vector for this task they achieve 111.5% FLDD and 45.1% Interchange.\n\n> “In the intervention experiments, the choices of the exact subspace (the number of components) and the intervention method (e.g., scaling diagonally for 2D) appear a bit heuristic and need better justification.”\n\nWe do have simple explanations for these:\n\nIntervention: We wanted to be able to plot the results in 3D without dimensionality reduction (Figure 5) because we thought it was very interesting to be able to do that. We chose to leave out 7.3 specifically, per Line 182: “changing 7.3 does not have a strong effect on its own” (see Figure 2). We entirely agree that this point is just too covert in the current draft, so we will raise this decision when introducing the interv. experiment\n\nChoice of scaling: We use z score thresholding on the distribution of composition scores (>4) to choose which components to scale. On all components this led to choosing 1D except for the duplicate token head which was 2D. This is why we used diagonal scaling. The graphs in Figures 6 and 7 show these components compared to others, and we believe it’s agreeable that these are also visually very clear outliers and the choice is justifiable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WEnw5J2vTc", "reviewer": "Reviewer_erzS", "summary": "This paper explores the routes and mechanisms of information transfer between heads of transformer large language model. They hypothesize the presence of low-ranking communication channels between attention heads from different layers within residual connections. The authors propose a method based on Singular Value Decomposition of weight matrices to detect those channels in a pretrained model; this method doesn’t require any additional data. They show that this method can uncover intricate and interpretable intrinsic structures of a transformer, which in turn can be manipulated to targetly adjust the model’s performance. The authors illustrate it by significantly improving the performance of GPT-2 on the list item recall task.", "questions": "1) You identify several types of attention heads: mover, duplicator, and inhibitor, but could a single head exhibit traits of multiple types? Also, is this classification not exhaustive (i.e., there are heads that do not fall into any of the aforementioned categories)?\n\n2) If there are two interventions and each of them is beneficial (i.e., increases the model's performance) on its own, how often is their composition (i.e., applying them simultaneously) beneficial? Is it possible to somehow predict such pairs of interventions without directly computing the quality of each pair?\n\n3) How many heads were affected by the interventions in your experiments in Section 5? Do you have information on how the increase in performance depends on the number of modified heads?", "limitations": "Authors clearly outline the limitations of their work in the dedicated section (in Appendix B).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "•) Very interesting and novel idea.\n\n•) Good analysis; promising results that can facilitate further progress in interpretability and understanding of the innerworkings of large language models.\n\n•) The article is well-written, and the text is easy to follow. The figures are nice and illustrative.", "weaknesses": "1) The proposed method of Communication Channel Interventions improves model performance on a synthetic task (Laundry List); however, it is not clarified how such modification will affect model's performance on other tasks.\n\n2) The contextualization in the research field (Section 6, Related Work) is short and covers rather little information on previous works about information passage within transformer models and attention mechanism.\n\n3) The limitations of the method do not allow for a straightforward implementation on newer models with relative positional embeddings.\n\n4) [Minor] The possible practical applications of the achieved results are outlined vaguely.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We are very happy to hear that the reviewer found the results interesting, promising for facilitating future work, and the paper easy to follow. The reviewer had some concrete questions and concerns about the generalizability of the method to new models and results to new tasks. We believe we have satisfactory answers to these below:\n\n>  The limitations of the method do not allow for a straightforward implementation on newer models with relative positional embeddings.\n\nWe would like to point out that this is only for composition with queries and keys in models with RoPE embeddings, in case it wasn't clear. Value-Output composition is entirely unaffected, and we have results with Pythia (a RoPE model) in Figure 12. This is certainly still a limitation, but we believe in the future it will be surmountable. In addition, composition with mlps (mlp to attn, attn to mlp, and mlp to mlp) is still a valid path for future work that we didn’t explore here, so we don’t think that our method will be limited to only certain kinds of models. Given the complexity of the mechanisms involved already, we figured it would be too much to find a satisfactory workaround to the query/key problem, work with MLPs, and provide our current analysis all in one paper. We'd also like to point the reviewer to the rebuttal which addresses some related points.\n\n> The proposed method of Communication Channel Interventions improves model performance on a synthetic task (Laundry List); however, it is not clarified how such modification will affect model's performance on other tasks.\n\nWe would argue that the mechanism we uncover is an extremely general component in broader language modeling capabilities and possibly affects most tasks that involve some form of recall. Our evidence for this is the activity on both the IOI and Laundry list tasks (which would seem to coincidental otherwise), the content independence of the mechanism (see Line 281), and the consistent general activity on open domain text (see rebuttal document). In our rebuttal document we outline some evidence for this idea\n\nWe will use the rest of the space to answer other questions:\n\n> How many heads were affected by the interventions in your experiments in Section 5? Do you have information on how the increase in performance depends on the number of modified heads?\n\nWe use three of the four inhibition heads (7.9, 8.6, 8.10). We chose to leave out the fourth (7.3) because we wanted to be able to plot the results in 3D without dimensionality reduction (Figure 5) since that often can’t be done. As for why we chose 7.3: per Line 182: “changing 7.3 does not have a strong effect on its own” (see Figure 2).\nWe saw some marginal increase by also including 7.3 but it was computationally expensive to run the full sweep. Based on the evidence we do have, the scores would probably be at baseline if we performed an analogous intervention on three random heads, though.\n\n> You identify several types of attention heads: mover, duplicator, and inhibitor, but could a single head exhibit traits of multiple types? Also, is this classification not exhaustive (i.e., there are heads that do not fall into any of the aforementioned categories)?\n\nWe did not find, but don’t rule out that a single head could exhibit traits of multiple types. This classification is not exhaustive, and we do briefly study induction heads in the appendix. Still, we are excited by recent and evolving work  in characterizing other types of observed specializations in heads (e.g., [successor heads](https://arxiv.org/abs/2312.09230)).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xzUTPfsVfG", "reviewer": "Reviewer_2UBx", "summary": "The paper investigates the communication between attention heads across different layers in Transformer-based language models. First, it establishes that a previous composition metric, which has been shown to be useful in toy settings, is noisy in larger models when tested on the Indirect Object Identification (IOI) task with a known circuit. The authors then propose a modification using Singular Value Decomposition (SVD), rewriting the original matrix as the sum of outer products of the left and right singular vectors, scaled by the corresponding singular values. This new metrics reveals inter-layer communication more clearly and provides a less noisy signal. This is causally verified by zeroing out a single singular value on the IOI task which reduces the inhibition score notably. \n\nThe authors then study these specific communication channels in detail, showing how interventions in these channels can affect the model's downstream behaviour. Interestingly, they demonstrate that this mechanism is independent of the specific context, functioning as a general pointer over lists. Finally, the authors use these insights to study the seemingly arbitrary sensitivity of language models to the order of items in lists using an artificial laundry list task. They show that as the list length grows, the internal structure in the communication partly collapses and makes it hard for the model to select the correct item. This can in part be addresses using a custom intervention, again demonstrating the causality of this phenomenon on the model behaviour.", "questions": "1. Could you elaborate on the intervention used to improve model performance on the laundry list task? Specifically, what operations do you perform to \"set the model components in a certain area of the 3D space\"?\n2. Can the proposed composition score be used to identify compositions of attention heads without prior hypotheses?", "limitations": "I believe the limitations are properly addressed, although the most important limitation - that the composition score is not directly applicable to models with relative positional embeddings - is somewhat hidden in the appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper addresses a fundamental question in interpretability of transformer-based language models, namely how attention heads selectively communicate with each other. Although it was known from prior work that attention heads compose with heads in other layers, to the best of my knowledge, this is the first in-depth study of inter-layer communication in large language models. \n- The authors use a variety of techniques to systematically test and causally verify their hypotheses; I also find the use weight-based decompositions instead of activation-based techniques intriguing, as it enables static analysis of neural networks without the need for input data. \n- The findings are connected to previous observations on language model robustness, explaining a seemingly arbitrary sensitivity of language models to the order of items in lists.", "weaknesses": "- The proposed composition score does not work for models using relative positional embeddings like RoPE, which are standard in state-of-the-art language models such as Gemma [1]. This makes the composition score not directly transferable.\n- The results could be significantly strengthened if the technique was used to conduct an unsupervised search for head compositions within the model, rather than focusing solely on the IOI task with a known circuit.\n\nMinor issues: \n- Typo in line 11: “via analysis *of* their weight matrices”\n- Typo in line 176: “the OV [or] matrix”\n- Typo in line 187: “communicatoin”\n- Overlapping image in Fig. 5, covering part of the “10\"\n\n[1] G. Team et al., ‘Gemma: Open Models Based on Gemini Research and Technology’, arXiv [cs.CL]. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We are happy that the reviewer enjoyed the paper and we appreciate the encouraging comments regarding static weight analysis, we think there is a lot of interesting work to be done in this area.\n\n\n> The proposed composition score does not work for models using relative positional embeddings like RoPE,\n\nWe'd like to point out that this is only for query/key composition and does not occur for composition between values and outputs. Pythia uses RoPE and we cover positive results on this model using composition in the appendix. We use the composition score to do static circuit analysis on part of the Pythia IOI circuit, which was not previously known before (which we later validate with path patching). See the below point. See also the rebuttal for more info.\n\n> Can the proposed composition score be used to identify compositions of attention heads without prior hypotheses?\n\nYes absolutely, the compositions can be taken between pairs of heads, and outliers can be searched for. From there, new connections can be found and later analyzed on data. We had to do IOI circuit analysis on pythia to find inhibition heads, and the composition score was used for part of this. After starting the circuit analysis and finding the inhibition heads, the value composition score revealed the induction heads that connect the circuit, which path patching later confirmed\n\n> The results could be significantly strengthened if the technique was used to conduct an unsupervised search for head compositions within the model, rather than focusing solely on the IOI task with a known circuit.\n\nThank you for raising this. We definitely hear the reviewer on this, but we're not sure how we could fit this in without it disappearing in the appendix. We hope that the above point about our analysis on pythia without a known circuit is satisfactory for the scope of this paper.\n\n> Could you elaborate on the intervention used to improve model performance on the laundry list task? Specifically, what operations do you perform to \"set the model components in a certain area of the 3D space\"?\n\nYes, in plain english, each inhibition head is responsible for one direction (we focus on three of them so that we can visualize the space without dimensionality reduction). We overwrite the output of each head to be some coordinate in its corresponding direction. That is, some scalar times the unit vector corresponding to the inhibition component we previously identified. A very important part of this intervention is that it makes it impossible for the head to attend to any of the previous context (the Q*K operation becomes unused), which is why we are able to say it is content independent: we control which item in the context is being attended to/ignored without letting those heads attend to them", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NZirOVn22M", "reviewer": "Reviewer_7Dor", "summary": "This paper investigates the interaction between attention heads at different layers in a transformer. They primarily study the “inhibition-mover subcircuit”, a previously identified attention head interaction from circuit analysis work [1,2]. They show the interaction between heads can be characterized by a low-rank subspace, and show it is helpful in reducing prompt sensitivity on a simple item-recall task.\n\n[1] Wang, et al. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. 2023. (https://openreview.net/forum?id=NpsVSN6o4ul)\n\n[2] Merullo, et al. Circuit Component Reuse Across Tasks in Transformer Language Models. 2024 (https://openreview.net/forum?id=fpoAYV6Wsk)", "questions": "- When you say a head is “highly active” on a prompt or task [e.g. Line 229, 255], what is meant by that? Is this measured by its attention pattern or something else? When would an attention head be \"non-active\"?\n\n- When using a subspace to steer the inhibition score, the singular values needed to flip attention from IO to S (or vice versa) seem rather large. What are the typical ranges of the singular values you see in your decomposition of the weight matrices you’re investigating?\n\n- As one of the other well-studied/well-known subcircuits, have you done any analysis to understand whether the induction subcircuit is also dominated by a low-rank subspace? Or is this specific to the inhibition-mover circuit you study?\n\n- In section 5, how does trying to learn optimal singular vector weightings either via gradient descent or regression compare to doing a grid search over 3-d points?", "limitations": "The authors have addressed limitations of their work, which are reasonable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The work is well-aware of existing related literature, and the questions posed are interesting.\n- The work is well-motivated - current LLMs are often not robust to prompt variations, and understanding and mitigating the reasons for this is of value.\n- The experiments demonstrate that the identified low-rank subspace can be used to effectively improve model performance on the item-recall task they introduce.", "weaknesses": "- The study is limited to fairly small transformers (GPT2-small, and Pythia-160m in appendix), and it’s hard to know whether the results will generalize to more complicated tasks or methods will scale to larger models. Additionally, It would also be helpful to know whether larger models also struggle with the item-recall task (and whether they have similar inhibition-mover subcircuits). \n- The composition score presented in Equation 1 should be presented in context (i.e. seems to have come from Elhage, et al. [3]? Is there earlier use of this score elsewhere?). Besides substituting individual SVD components in for QK or OV matrices, are there any other changes compared to how [3] measure the composition score?\n- It is unclear to me whether the failure mode of GPT2 small on the item-recall task is due to (a) not correctly identifying duplicated objects, or (b) not suppressing correctly identified duplicated objects when going to copy. This is because Figure 4 shows the attention pattern of mover heads can be influenced by either duplicate token head channels or inhibition head channels. Is this because the duplicate token channel influences the inhibition signal? (i.e. its effect on the mover head is mediated by the inhibition head?)\n- There are some portions of the paper that present results without experimental details. One example is on Lines 228-231: \"On OpenWebText ... we find inhibition heads are primarily active in lists and settings where repetition...\". Providing details about the experiments run to validate this claim would help strengthen the argument made. In addition, what does it mean for an attention head to be \"active\" on text or not? Is this measured by its attention pattern or something else? \n\n\n[3] Elhage, et al. A Mathematical Framework for Transformer Circuits. 2021 (https://transformer-circuits.pub/2021/framework/index.html)\n___\nThere are a few places with incomplete sentences or minor typos. I've listed a few I noticed below:\n\n- Line 174-175 - “... test how this affects.”, (incomplete sentence)\n- Line 187 - “communicatoin” channels\n- Line 214 - “non”\n- Line 276 - “repersent”\n- Line 290 - “featuers”\n- Line 506 - \"These results are in\", (incomplete sentence)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The study is limited to fairly small transformers (GPT2-small, and Pythia-160m in appendix), and it’s hard to know whether the results will generalize to more complicated tasks or methods will scale to larger models. Additionally, It would also be helpful to know whether larger models also struggle with the item-recall task (and whether they have similar inhibition-mover subcircuits).\n\nThank you for raising this. Please see our rebuttal which primarily focuses on this point. In summary, yes larger models struggle with this task despite the simplicity and we argue this is an isolated mechanism that is part of a very basic and general language modeling mechanism learned by the model.\n\n> The composition score presented in Equation 1 should be presented in context (i.e. seems to have come from Elhage, et al. [3]? Is there earlier use of this score elsewhere?). Besides substituting individual SVD components in for QK or OV matrices, are there any other changes compared to how [3] measure the composition score?\n\nAs far as we know, the composition score does not appear anywhere before Elhage et al. And that is correct, this iso our only modification to get it to work\n\n> It is unclear to me whether the failure mode of GPT2 small on the item-recall task is due to (a) not correctly identifying duplicated objects, or (b) not suppressing correctly identified duplicated objects when going to copy. This is because Figure 4 shows the attention pattern of mover heads can be influenced by either duplicate token head channels or inhibition head channels. Is this because the duplicate token channel influences the inhibition signal? (i.e. its effect on the mover head is mediated by the inhibition head?)\n\nThe reviewer is correct in their last statement, it is because the duplicate token head affects the inhibition head, which affects the mover head. This is established in the circuit identified by [Wang et al., 2023](https://arxiv.org/abs/2211.00593). If this is still confusing to the reviewer we can go into more detail. Also to be clear, when we do the edit here, we are only editing the duplicate token subspace at the point the inhibition head sees it, so it is following the logic of duplicate token head-->inhibition head-->mover and **not** duplicate token head-->mover head (where --> is read as 'affects'/'changes'). We expect a lot of familiarity with previous work, which is not entirely fair to all readers, so to remedy this we added a section to the appendix outlining the entire IOI circuit in GPT2 from that paper (as well as the circuit we find for IOI on pythia).\n\n> When you say a head is “highly active” on a prompt or task [e.g. Line 229, 255], what is meant by that? Is this measured by its attention pattern or something else? When would an attention head be \"non-active\"?\n\nYes this is measured by the attention pattern. We apologize for the lack of details, please see the rebuttal for an operationalization of these terms and how this analysis was carried out. They will be included in the final draft.\n\n> As one of the other well-studied/well-known subcircuits, have you done any analysis to understand whether the induction subcircuit is also dominated by a low-rank subspace? Or is this specific to the inhibition-mover circuit you study?\n\nWe also looked at the induction head in Figure 6 (left) and Appendix H and found that the one that we looked at was not low rank, so not every head composition is like this. However this is not specific to the inhibition-mover/duplicate-inhibition subcircuits because we found plenty of other compositions between random heads that were similarly sparse. Unfortunately, we didn't have space to also analyze those and assign function to them because they are not part of any known circuits (like the IOI components). This is a very interesting direction for future work. We'd be happy to discuss more with the reviewer about what information we could/should include regarding this, though.\n\n> In section 5, how does trying to learn optimal singular vector weightings either via gradient descent or regression compare to doing a grid search over 3-d points?\n\nThis is a very cool idea that we did not consider when writing/have time for in the rebuttal. We kept the 3-d points analysis extremely basic. Would this involve optimizing directly in that 3d space? Please re-raise this point during the discussion if the reviewer finds this an interesting addition to the paper.\n\nWe hope we have resolved all of the reviewer's concerns which we believe we properly addressed in the rebuttal document or here, by clarifying details from the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their thorough and thoughtful comments and suggestions. We are glad the reviewers shared our interest in the results on static weight analysis, as we’re hopeful for continued progress in this area, and we’re pleased that there was mostly consensus that are findings supported our hypotheses. After going over the reviews, the main concerns appear related to whether the main mechanism we discuss (related to inhibition) really matters outside of GPT2-small and pythia. We’d like to dedicate the rebuttal to addressing those concerns.

# Other Models
First, we’d like to briefly discuss and expand on the implications of our findings beyond the small models tested here. In Figure 1 of the rebuttal, we wanted to highlight that larger/more modern models do not solve the object selection problem we use to motivate the paper in Figure 1. This is to make the point that we did not pick a mechanism too specific to make broader contributions. We’d also like to draw attention to [Liu et al., 2023](https://arxiv.org/abs/2307.03172) which shows that even very powerful LMs can struggle with recall from context. More generally, we believe our modified composition score can beyond the models tested here. The concept of composition score (and its decomposed variant introduced here) generalizes to mlp-mlp and attn-mlp interactions as well, which we don’t test here. We also believe that the issue with positional embeddings and query/key composition can be solved in the future. Note that we do have positive results with pythia (which uses RoPE) and value composition in this paper. This paper is simply too full already to include such extensions to the work and are hoping to explore these directions in follow ups.

# Other Tasks:
There was some concern from a few reviewers that the tasks studied here (and therefore the mechanisms involved) are too narrow. We argue that the mechanism we identify is part of a more basic language modeling mechanism that underlies most tasks that involve recall from context. In Figure 2 of the rebuttal we show examples of the inhibition heads activity . Some reviewers pointed out that we were vague about our methods that led to claims in Lines 228-231, so we outline them below.
Combined with the observations about larger models above, we believe our results indicate that although we are targeting an extremely specific mechanism, this is actually part of a more general language model capability present in LMs. Thus, we see our findings and our methods of analysis useful and productive progress towards a greater understanding of the inner workings of LMs.

## Details on Identifying passages that highly activate inhibition heads
Some reviewers pointed out that we were not specific about how we arrived at the claims in Lines 228-231. We would like to provide our methodology which was quite rigorous. First, we split passages from OpenWebText-10K into token sequences of length 256 and run them through the model. We cache the attention patterns of the inhibition heads and manually examine tokens for which the attention score is >=0.3 for any token besides the very first. We use this to define “highly active”. We examined about 200 of these passages manually and found the patterns to be extremely consistent. The heads would almost always attend from some token like “and” to some other token that would most likely be a repetitive continuation, e.g., in “Crime and”, the “and” would attend to “Crime” presumably to inhibit “Crime and Crime” from being generated. Figure 2 left shows a screenshot of the interface we built to examine these examples when preparing this manuscript. Note that these were not cherry picked, they are the first examples from the dataset that arise from the .3 threshold. We'd like to add a section in the appendix for further analysis of inhibition heads on open data



# Other edits
We’d lastly like to thank the reviewers for their attention and specific feedback on the presentation of the results. This paper has many small details, and we appreciate the patience that was needed to be able to provide such low-level feedback. After implementing reviewer edits, we have some space left on page 9, which we will dedicate to discussion and expanding related work, which was suggested.

Meta-review: This paper identifies particular mechanisms by which information is communicated between transformer layers. They relate their identified "head type" mechanisms to a particular simple task, and use this to conduct experiments demonstrating the (interventional) effects of targeted modifications following their decomposition.

Reviewers generally agree that the results are interesting and worth publishing. Concerns focused on the degree to which the paper over claims, the clarity of presentation, and whether and to what extent the model and experiments here really have general predictive power for LLMs. (See particularly the discussion with Reviewer Veos).
Decision: Accept (poster)