Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits
Authors: Areeb Ahmad
Abhinav Joshi
Ashutosh Modi
Venue: NeurIPS 2025 poster ()
Keywords: LLMs
Tranformer Circuits
Mechanistic Interpretability
Primary area: deep_learning
TL;DR: We propose a new method for interpretating transformer circuit by performing SVD on query-value and value-output matrices

Abstract: Transformer-based language models exhibit complex and distributed behavior, yet their internal computations remain poorly understood. Existing mechanistic interpretability methods typically treat attention heads and multilayer perceptron layers (MLPs) (the building blocks of a transformer architecture) as indivisible units, overlooking possibilities of functional substructure learned within them. In this work, we introduce a more fine-grained perspective that decomposes these components into orthogonal singular directions, revealing superposed and independent computations within a single head or MLP. We validate our perspective on widely used standard tasks like Indirect Object Identification (IOI), Gender Pronoun (GP), and Greater Than (GT), showing that previously identified canonical functional heads, such as the “name mover,” encode multiple overlapping subfunctions aligned with distinct singular directions. Nodes in a computational graph, that are previously identified as circuit elements show strong activation along specific low-rank directions, suggesting that meaningful computations reside in compact subspaces. While some directions remain challenging to interpret fully, our results highlight that transformer computations are more distributed, structured, and compositional than previously assumed. This perspective opens new avenues for fine-grained mechanistic interpretability and a deeper understanding of model internals.

Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits
Areeb Ahmad"
Abhinav Joshi"
Ashutosh Modi
Indian Institute of Technology Kanpur (ITT Kanpur) {ajoshi, areeb, areeb, ashutoshm)@cse iitk.ac.
Abstract
Transformer-basee language models exhibit complex and distributed behavior, yet their internal computations remain poorly understood. Existing mechanistic interpretability methods typically treat attention heads and multilayer perceptron layers (MLPs) (the building blocks of transformer architecture) as indivisible units, overlooking possibilities of functional substructure learned within them. this work, we introduce more fine-grained perspective that decomposes these components into orthogonal singular directions, revealing superposed and inde- pendent computations within single head MLP. We validate our perspective on widely used standard tasks like Indirect Object Identification (IOI), Gender Pronoun (GP), and Greater Than (GT), showing that previously identified canon- ical functional heads, such as the "name mover," encode multiple overlapping subfunctions aligned with distinct singular directions. Nodes in computational graph, that are previously identified as circuit elements show strong activation along specific low-rank directions, suggesting that meaningful computations reside in compact subspaces. While some directions remain challenging to interpret fully, our results highlight that transformer computations are more distributed, structured, and compositional than previously assumed. This perspective opens new avenues for fine- ame-grained mechanistic interpretability and deeper understanding of model internals.
Introduction
Language models (LMs) exhibit complex and often surprising capabilities across tasks [Radford et al., 2019, Brown et al., 2020, Li et al., 2023, Javaheripi et al., 2023, Touvron et al., 2023, Jiang al., 2023, Grattafiori al., 2024, Joshi al., 2024], yet their internal computations remain poorly understood. Mechanistic interpretability seeks to bridge this gap by identifying the circuits, i.e., networks of interacting components, that realize/show specific functions [Olah et al., 2020, Wang et al., 2022]. Prior work has shown that these circuits can often be decomposed into submodules with distinct roles such as copying, inhibition, or referencing [Wang et al., 2022]. Despite this progress, the current state of discurty methods [Conmy et al., 2023, Syed et al., 2024, Bhaskar et al., 2024] still view model components, such as attention heads and MIPs, as atomic units of computation. Methods such as causal tracing [Meng et al., 2022], activation patching [Wang et al., 2022), and attribution-ban based analyses [Heimersheim and Nanda, 2024, Joshi et al., 2025b,a] typically probe/patch or ablate entire components assess their functional contribution. While these techniques have produced valuable insights, they inherently assume that functionality aligns cleanly with component boundaries. In practice, however, transformer layers may multiplex multiple subfunctions within a single head or MLP, meaning that treating components as monolithic units risks overlooking the fine-grat fine-grained structure of computation within these modules.
Equaa Contributions
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

While most existing methods analyze/study transformer components as monolithic units, recent work has begun to question this assumption by investigating the internal structure of these components. Merullo et al. [2024b], for instance, introduced low-rank perspective showing that attention heads communicate through specific singular directions in residual space, defined by the singular vectors of their value matrices. However, this analysis primarily captures inter-component communication, how heads "talk" to one another via low-rank channels, while leaving intra-component decomposition largely unexplored, i.e., how single head might multiplex multiple independent functions within its internal subspace.
In this work, we extend this perspective, which goes beyond the attention heads with additional inclusion of MLP layers, leading to a comprehensive directional view of transformer blocks. This reveals that low-rank, distributed computations are a general feature of transformer architectures. Moreover, components identified as part of known circuits [aagg et al., 2022, Conmy et al., 2023, Syed et al., 2024, Bhaskar et al., 2024] exhibit strong engagement along specific singular directions, suggesting that meaningful computations are embedded within compact subspaces. We further demonstrate/validdtt this perspective on the widely popular canonical tasks like Indirect Object Identification (IOI) [Wang et al., 2022], Gender Pronoun (GP) [Mathwin et al., 2023], and Greater Than (GT) [Hanna et al., 2023]. In IOI, for instance, our analysis identifies dominant singular directions within the same heads previously characterized as "name movers," [Wang et al., 2022], showing that only sparse subset of these directions meaningfully contributes to task performance. Using our proposed optimization scheme, we learn direction-level masks that remain highly sparse while closely replicating the model's original behavior, indicating that transformer computations can be effectively captured by a compact set of low-rank subfunctions. Moreover, the directions corresponding established IOI heads exhibit notably higher activation and mask weights compared to other heads, supporting the view that known circuit components operate through a small number of active, interpretable directions. In a nutshell, we make the following contributions:
We introduce a directional interpretability perspective that models transformer components (atten- tion and MLP) as superpositions of orthogonal subfunctions rather than atomic units.
We demonstrate this via including an optimization babeeze masking scheme that identifies functionally important singular directions within attention and MLP layers. enabling direction-leve attribution. We provide empirical evidence that multiple low-rank, interpretable computations coexist within single attention heads and MLPs, which diverge from the standard assumptions about circuit modularity.
Our findings suggest that transformer computations are not strictly modular but rather distributed, compact, and compositional, with overlapping subfunctions embedded within shared subspaces. This perspective reframes transformer interpretability through the lens of functional directions. opening new avenue for analyzing, editing, and understanding model behavior at the subcomponent level.
Beyond component-level decomposition, our study/investigations reveal an interesting uncovering that transformer layers naturally form stable, controllable directions in logit space, each aligned with a specific set of tokens (which we also term as logit receptors, more details in Appendix B.2). These directions can be thought of intrinsic mechanisms that the model selectively activates depending on the input context. For example, in gender pronoun resolution, certain directions consistently influence the logits toward tokens he" or she," with their activation strengths varying systematically in response to context. Importantly, we find that just scalar interventions along these directions can reliably control/modify the model's predictions (also see Figure 2), demonstrating that these low-rank, interpretable subspaces may form interpretable building blocks of model computation. This insight provides a natural, mechanistic basis for studying how distributed computations within single heads and MLPs implement distinct functional behaviors, and motivates the ine-grained directional analysis presented in this work.
Overall, we believe that this perspective reframes how we think about transformer computations, i..., rather than being confined to monolithic components, meaningful behaviors are often embedded in low-rank subspaces that can be independently manipulated and interpreted. This opens the door to more precise mechanistic studies, targeted model editing, and fine-grained attribution methods, and suggests that future interpretability work should consider the functional decomposition of components along intrinsic directions as a fundamental lens for understanding model behavior for abstract interpretable concepts learned by these models. We release our codebase for the experiments and additional results at https: //gi thub. com httpa///pppprrraooooooo/o/eooooa

---

2 A Unified Linear View of Transformer Components
To operationalize our directional interpretability perspective, we begin by expressing transformer computations in a unified linear form. We focus on decoder-only architectures and take the transformer circuit formulation from Elhage al. [2021] our foundation. By representing both attention and MLP transformations through augmented matrices that jointly include the learned weights and biases, we obtain consistent linear representation across all components. This "folding in" of biases (i.e.. appending constant dimension to the input and incorporating bias terms into the matrix) allows us to apply Singular Value Decomposition (SVD) uniformly both attention and MLP layers. The resulting formulation enables us to analyze orthogonal directions across different components within shared framework, laying the groundwork for the fine-grained decomposition.
Attention Mechanism (Query Key (QK) Interaction) In a standard decoder-only transformer architecture [Radford al., 2019], each attention head computes attention scores via the dot product between query and key row vectors
(b)
(b)T
9:
Softmax,
aj
diheau
where q (k)
x, w (b) Q b2 and lndh x, w (h)
+ (k Expanding the dot product gives
q: k x, W Q W x, b + bo x + bobk.
To express this interaction compactly as single linear operation, we introduce an augmented matrix formulation that incorporates both weights and biases
x W
(QK) aug
+ [x]
kj.
where the augmented weight matrix defined as
((KK) ang
( bobk Wobk
bo W WoW K
Attention Mechanism (Output Value (OV) Projection) Following the computation of attention
weights, each head aggregates contextual information by taking a attention weighted sum of value vectors, and projecting it into the residual stream using Wo. This operation determines how information written back into the residual stream:
2; = Σ aijy
where V, = x, Wv + by.
Each head's contribution then projected through output matrix:
(h) = z, (h)
(h)
bo-
H
where, |.I is the number of heads. Substituting vi and rearranging terms, the OV transformation becomes
(1) yi
Σ ai x, W + by w!"" + 1 bo -11 -11_ رررب
by w (h) +
H bo
H
W W
(b)
We thus define the augmented output matrix for each head as
by w-(h) (h) O
+ im bo
(OV) aug
E (I+ddaa)))) x deacdall
Wy w (k)
MIPP Layer Reformulation Beyond attention, transformer MLP blocks also consist of two affine transformations separated by nonlinearity. To maintain a consistent linear treatment across all components, explicitly separate these projections and represent each using augmented matrices that include both weights and biases:
y1 xW + Din-
Your f(y1) out + bout
3

---

Each decomposition expresses the component as a sum of rank-1 mappings, where each term rep- resents an orthogonal direction that can, in principle, support distinct subfunction (also used as singular directions in the paper). To study the importance of these subfunctions, we introduce a learn- The augmented representations are defined as:
W auy
bin W
W aug
Dout
W out
Thus, both pre-activation and post-activation projections can be expressed as linear maps over augmented input vectors [1,4% yielding unified affine-to-linear transformation consistent with treatment of attention layers.
This unified formulation enables all major transformer subcomponents to be represented as linear operators over augmented spaces, summarized as:
w(R [R (1+de)))) x(1 (1++daase)
for attention score computation
aug
(OV) aug
R (1 dmade)))) ( Xdmad
for weighted input-to-outply transformation
W(in) nug
(1+ (htdd)) xchitp
for MLP input projection
W(out aug
R (1+da))) dandd
for MLP output projection
By expressing all these transformations in common linear framework, can perform low-rank analyses such as SVD across both attention and MLP components consistent manner. This perspective lays the groundwork for the directional interpretability approach introduced in the next section, where we analyze how specific singular directions correspond to distinct, functionally meaningful computations.
3 Directional Masking via Low-Rank Decomposition
Having established a unified linear formulation of transformer components ($2), we now turn to identifying the specific functional directions that drive model behavior. Our goal is decompose each augmented attention or MLP matrix into a set of orthogonal directions, each representing an independent computational axis, and selectively intervene on these axes to understand their roles.
Rather than treating attention heads or MLP layers as monolithic units, we perform decomposition their augmented matrices, ensuring faithfulness the model's native computation flow while enabling fine-grained directional attribution and masking. This formulation makes it possible to characterize component's 's behavior not in terms of entire weight matrices, but terms of small number of low-rank directions.
Singular Value Decomposition (SVD): Any real matrix M E gmxn admits singular value decomposition M = U VVT, where U E Rmxm and V E Raxn are orthogonal matrices, and E Rmx* is a diagonal matrix of non-negative singular values 01 2 02 2 ... 0. This makes singular vectors a natural coordinate system for direction-lewe interpretability. as they provide orthonormal bases that isolate independent computational units embedded within component.
Masking Directions: To isolate and investigate the contributions individual computational directions, we apply SVD to the augmented matrices of each attention head and MLP layer:
W(QK) UQK ΣOK VoK. aug
W(OV) Uov Σov Vov, aug
W W(in) (in)
Un En ViT
W (out) aug
= Uout Dour VT
aug
able diagonal mask matrix M diag(m1, m2, ... my), m, [0, 1), that scales the contribution of each singular direction. The masked transformation is then defined as:
W aug = U.MV
where higher entries of M retain emphasize specific directions and lower entries suppress them. This allows continuous, direction-level control over component behavior while maintaining differen- tiability for optimization.

---

To ensure interpretability and stability, we preserve the full representational span of the component via:
UVV = -EMM - (
This decomposition allows faithful interventions, i.e., the model's downstream layers continue to receive inputs within their expected distribution (seen while training), since the overall covariance structure of activations remains intact. Thus, both masked and complementary subspaces are retained, enabling controlled investigation without inducing distributional drift.
W(QK) = UQx EQK MOK Vik
E R (rimsaa))))) x (1 (I-danse)
aug
W
(OV) [Uov Σov Mov Vov,
Uov Eov (I - -Mvv) Vor]"
E R d2maa))))))) (dwaa))
Aug
W
In Eim Min Vin
Um Em (I Mm) ]
R (2(1+dmaa)) x)x (dhis)
aug
W(mr)) aug [Vour Your Mout ViTT Uour Your (I Mour) ]
E (2(+ (damsa)
This procedure provides a principled mechanism for identifying the causally relevant subspaces within each transformer component, while keeping the model's computations structurally and statistically consistent with its pre-trained dynamics.
Note that for the QK matrices, we retain only the masked component UQx EQK Mox Vix and omit the complementary term UQK EQK (I - -Mox) VOK. This asymmetry arises from the distinct functional role of the QK block, whereas the OV and MLP matrices operate on feature representations, the QK matrices parameterize the attention kemel, the quadratic form that defines pairwise token similarities. Introducing a complementary (I - M) subspace here would correspond defining additional, independent similarity maps within the same attention head. Because attention scores are normalized via a single softmax, this would implicitly produce multiple incompatible superimposed kernels, leading to interference and spurious correlations. In other words, if the true token represen- tations x = { } become correlated with irrelevant or spurious features zoor = {xcoor, az30), }. (which is often the case due to their similar Traming/syntacticssyy
of æ:for the mixed terms that arise from partially masked QK components can yield misleading similarity/attention scores: Siy m x; z. + (1 m) "Qr"Wa
For instance, if "eorrWi VV term produces similarity score close to 2, z,,oox q R.J the training objective will incorrectly suppress the mask value (m + 0) even if the component is genuinely useful for the task. Thus, we restrict QK masking to the primary MΦk. which preserves a single, coherent attention kernel while still permitting direction-level ablations over its singular basis.
Optimization Objective. To identify the singular directions most responsible for model's behavior on given task, we optimize the learnable masks . to balance faithfulness (preserving model behavior) and sparsity (selecting only minimal subset of directions).
Let ply x) denote the original model's predictive distribution, and PM x) the distribution obtained after applying the masked matrices. We define the optimization objective as:
Lm KL[p(y æ)) p æ)] + > [|diag(MM]l.
where the KI. divergence term encourages the masked model to reproduce the original model's behavior, and the érrreuularization term promotes sparsity in the mask, selecting only minimal subset of singular directions. The trade-off coefficient À governs the balance between behavioral consistency (faithfulness) and sparsity.
This procedure (summarized in Algorithm 1) yields a low-rank, sparse decomposition, where each retained singular direction captures distinct, functionally meaningful axis of computation within the model's layers. While we use ε ba-de sparsity here, fo-reuularization could provide an alternative sparse selection mechanism [Bhaskar et al., 2024, Sung et al., 2021], and is left for future work.
To improve robustness and better isolate relevaelevant directions, we feed the model concatenated activations from both clean and corrupted inputs, where denotes the clean activations from the original input, and Ecorrupt denotes corresponding corrupted or perturbed activations (e.g., with noise or prompt that should consist of small change to x that would result in a different label in the task). The combination x, Teorrupt forms a joint input, leading to the next token prediction (Ex(m + (1 mg ceorrupt uu (details in Algorithm 1), enabling the mask optimization to identify singular directions that robustly capture task-relevan computations.
(M x) = [ (m+
5

---

Algorithm Directional Mask Optimization via Singular Value Decomposition
Require: Pretrained model fe. dataset D {(z y)}, sparsity coefficient X,
1: Initialize learnable diagonal masks MMr.. Mon MMnttMuutt E [0, 1 1|ann
2: for all components (attention heads and MLP layers) do
3: Construct augmented weight matrix W ang
4: Compute SVD: W aug U EV
5: end for
6: Freeze model parameters 0: keep masks M learnable
7: for each batch (I, y) E 'D do
8:
Define helper function for joint representation:
h(r, Tcomupt, W) [[1,a], [1, ccorrup]] W
9: Define component-speciffe reconstruction functions, for each layer I:
Iheadsl
(1 Balln
Σ (|E,
Softmax, [1.a;) [Daa;) Wor (o..ttt) (1aas) ang Viland
wov((.)) aug
Feorrupt;
(1) 8mlp
h GeLLU(b(I
2n corrupt:
Wang)) )),
nout corrupt
Woul(() aug
Run forward pass through masked model using both contributions:
PM x) Softmax LayerNorm
embedding + Σ
Batt
(1) + Bmipp)
We + bu
Compute objective:
CM KL[p(y æ)) (M x)] + [[diag(M)
12:
Update M via gradient descent, and and reconstruct weights W... Wang
13: end for
14:
return Learned masks M and reconstructed weights W ang
Note: æ (row vector) denotes the running forward pass activations while optimization, and Teorrupt (row vector) denotes the appropriate corresponding corrupted or perturbed activations, that are fixed and obtained from corrupted run.
4 Experiments
Having introduced directional masking, which decomposes each transformer component into or- thogonal singular directions and enables selective interventions, we now ask: do these fine-grained directions correspond to meaningful, task-relevant subfinctions within the model?
Prior interpretability studies have associated specific attention heads MLP layers with distinet behaviors, such as syntactic role tracking or inhibition [Wang et al., 2022, Conmy et al., 2023]. However, these analyses treat components atomic units. In contrast, approach allows to examine computation at the level of individual singular directions, revealing whether components multiplex multiple independent computations along distinct low-rank
We apply our method to a pretrained GPT-2 Small model [Radford et al., 2019], tractable benchmark widely used in mechanistic interpretability [Wang et al., 2022, Hanna et al., 2023]. To inspect the generality of directional subfunctions, we evaluate the model on three representative tasks: Indirect Object Identification (IOI) [Wang al., 2022], which tests syntactic reasoning by examining coreference resolution sentences. Greater-Thar (GT) [Hanna et al., 2023], where the model predicts numerical tokens following number in context, assessing its quantitative reasoning. Gender Pronoun Resolution (GP) [Mathwin et al., 2023], which measures semantic reasoning by testing pronoun-to-anteecddet resolution in natural text. Full dataset details provided in App. A.
organize our experiments around four core research questions. R1) Can small number of learned singular directions faithfully preserve model behavior? R2) Do these directions align with

---

Deplicat Tocan Hoads
S-anesttonn
Nami Meag Heats
Figure I: Learned singular value masks for Query-Key (WOK) aug matrices across all attention heads in the model. High mask activations correspond circuit components previously identified for the IOI task [Wang et al., 2022]. Each head exhibits sparsity along its singular directions, revealing the fine-grained subspaces driving task behavior.
components identified in prior circuit-level analyses? R3) Can method decompose known functional heads into interpretable sub-functions? R4) Can we discover new functional axes not identifiable through standard component-level analysis?
For each attention and MLP component in the GPT-2 Small model, we extract its augmented matrices (QK) wing w-(in) and, W (as described in $2) and perform a singular value decomposition. We retain all non-zero singular directions. corresponding to the actual matrix rank rather than a fixed truncation. For the Query-Key (W
auug matrices, the effective rank is 64, hence 64 reflects the full set of functional directions. Similarly, the Output- Value (WOV) matrices exhibit rank of 65 after augmentation, while MLP layers typically retain their full non-zero spectrum. Empirically, in reconstruction faithfulness (KID), confirming that the effective ranks of the augmented matrices capture all functionally relevant subspaces.
found these truncations yield minuseule drops (in the range of (le 6))
Over these singular directions, we introduce learnable diagonal mask that scales the singular values, enabling direction-level modulation of each component's contribution. The masks optimized using the objective defined in $3. The optimization in Algorithm proceeds on mini-batches drawn from each dataset, and optimization is stopped early once the held-out reconstruction loss stabilizes. For corruption (Ecorrupt). we follow the datasets provided by Bhaskar et al. [2024], and create an additional set of datapoints for tasks GT and GP.
Our first experiment evaluates whether the model's behavior can be faithfully reconstructed us- ing only small subset of learned singular directions (Table 1). Remarkably, across all datasets, high-fidelity reconstruction is possible with far fewer directions than the full component. For instance, the IOI task retains only 9% of directions relative to the full component while achieving a KL divergence of 0.21 and an exact match of 0.77, demonstrating that a small frac- tion of singular directions suffices to reproduce the model's behavior. Similar patterns hold for GT and GP, where over 95% of directions can pruned with minimal impact on perfor- mance. The learned masks are distinctly sparse, activating only handful of directions per layer.
7

---

Table 1: Comparison of sparsity, reconstruction fidelity, and task performance across datasets. Sparsity measured relative to the number of non-zero singular directions in each component ("Relative") and as a fraction of the full matrix size ("Full"). KL divergence quantifies reconstruction loss, while accuracy (where applicable) and exact match show downstream task performance. Despite extreme sparsity, the learned directions retain high behavioral fidelity. (see App. B.6 for sparsity computations).
Dataset
Sparsity (Rel/ Full)
KLD
Acc. (Pruned Full)
Exact Match
I0I
91.32/ 98.66
0.21
0.70+1000 / 0.79:+0.05
0.77=00.06
GT
95.21 99.26
0.23:00.03
N/A
0.33
GP
96.81 99.51
0.13++0.01
0.75+ 0.77
0.86++0.07
These selected directions achieve substantially lower KL divergence than top-k magnitude or
So Mary is a very good athloto, isn't
random SVD baselines, suggesting that task-
relevant computation is concentrated along spe-
cific, semantically meaningful directions rather than the largest singular modes (quantitative de- tails in App. B).
Transformer
Next, we investigate how our learned singular directions correspond previously identified mechanistic circuits, using the IOI task as ref- erence and comparing to ACDC [Conmy et al., 2023] and Wang et al. [2022]. While prior anal- yses treat entire attention heads as circuit partici- pants, our direction-level decomposition reveals that most heads exhibit strong activation along only a few singular directions. In other words, the coarse, component-level circuits reported in previous work arise from finer, low-rank struc- tures embedded within each head.
w wrlkk) (3 (aar
WoS0) R Ho
wrimn/ t toe
wread 30 lod
Seales
Tranfforvve Black
(Ai y (4) arast dre
Figures 1, 5, and 6 visualize the learned singular- value masks for Query-Key, OV, and MLP ma- trices, respectively, across the full model. Heads known to play key roles in 1OI, such as Name Mover, Backup Name Mover, and S-Iniibition Wang et al. [2022], show consistently high acti- vations across multiple singular directions, pin- pointing the precise subspaces driving their be- havior. Conversely, components not associated with known circuits exhibit near-zero activations across all directions (App. Figure 7), demonstrating that our optimization selectively isolates task- relevant subspaces.
prtaven ligh
P(u: ss)M)))
Figure 2:
The Figure shows intervention in the logit receptor for the Gender Pronoun task. Con- trolling the logit receptor using a scalar interven- tion modifies the predicted logits.
Overall, these results demonstrate that transformer components are not monolithic, and their internal computation is distributed along a small number of interpretable, low-rank axes/bases. The presented perspective thus bridges component-level circuit discovery and fine-grained mechanistic analysis, providing a new lens to study how distinct computational roles coexist within the same architectural unit.
Extending this perspective, we find that transformer layers contain inherent fixed directions in logit space corresponding to stable token- -eee--preferenc axes. Task behavior emerges from dynamically steering these directions via input-dependent scalar activations; in the Gender Pronoun task, for instance, distinct "he" and "she" directions exist, and the model selectively activates them depending on context. Scalar-based interventions confirm that these directions are causally relevant by flipping gender pronoun predictions with perfect accuracy, demonstrating that low-rank subspaces serve as modular, functional building blocks bridging representation and output behavior. All these results provide unified mechanistic view, i.c., computation is concentrated along sparse, interpretable
8

---

singular directions within components, which in turn map to stable, task-relevant axes in logit space (full details in App. B.2).
Functional Decomposition of Attention Head 9.6 via Singular Directions
Building our discovery that transformer components distribute computation along low-rank, interpretable directions, we next ask: what specific operations does a single head implement through these directions? We focus Head 9.6, previously identified as key contributor to the IOI task [Wang et al., 2022, Conmy et al., 2023, Bhaskar et al., 2024], and examine its singular directions with high learned mask values. Our analysis reveals individual singular directions encode distinct, separable operations, ranging from syntactic cues to semantic entity tracking, that collectively support task performance. Table 9 summarizes the primary roles and their corresponding mask values, highlighting how these directions act as interpretable computational primitives within a single attention head.
Semantic Separation of Entities and Actions Some singular directions carve out conceptually meaningful subspaces, revealing that attention heads implement abstract computations rather than just capturing statistical variance. A striking example is the 7th singular direction (S²) in Head 9.6, which consistently separates named entities (e.g., "Mary", "Kevin") from action-related tokens (e.g.. "went", "gave"). As shown in Table 7, entity tokens exhibit strong positive activations (+33.52 1.42), while action tokens are suppressed (-4.44 + 0.68). For instance, in the sentence "Jerry and Mary went to the school. Mary gave a raspberry to", S7 assigns high activations to "Jerry" (+2.87) and "Mary" (+2.78), but negative values to "went" (-4.33) and "gave" (-4.17). This direction acts like a semantic filter, splitting "who" from "what is being done," creating a foundation upon which downstream heads can operate.
Entity Salience and Detection Another key computation is performed by the 28th singular direction (S2s), which detects and amplifies the salience of named entities. In IOI prompts, tracking participants across multiple mentions is critical, and S28 highlights entities regardless their position or specific lexical form. Table 8 shows that named entities receive high activations (e.g., "Susan": 4.05, "Kevin", 5.22), whereas function words like "the" and "of" remain low. Interestingly, first mentions often get stronger activations than subsequent mentions (e.g.. "Kevin": 5.22, "Kevin_2": 2.07), suggesting this direction is sensitive to positional salience. We interpret S28 as an entity salience signal, priming tokens for further grammatical and referential reasoning.
Sequence Initialization Detection The top singular direction (S1) implements a structural, positional primitive, i.c., it assigns extremely high positive values to the first token in a sequence while giving negligible or negative values to all others. Across prompts, the first token receives activations 20-25 larger than subsequent tokens. This behavior is not unique Head 9.6; similar patterns appear in other heads and layers, independent of task, e.g., S1 of Head 0 in Layer 10, suggesting that sequence initialization is reusable primitive distributed across the network.
Summary and Implications All these examples show that Head 9.6 multiplexes multiple independent computations through orthogonal singular directions: semantic discrimination (S7), entity salience (S2x), and sequence initialization (S1). This supports the hypothesis that transformer components integration points of overlapping, low-rank subfunctions, rather than monolithic units. The directional perspective provides a line-grained lens for mechanistic interpretability, isolating the entangled computations within attention heads that would otherwise appear inseparable. Figure 8 summarizes the functional roles identified. Additional findings in App. B reinforce the idea that circuits should be understood not the level of whole heads, but collection of directions, each contributing in distinct proportions to task-relevant behavior.
6
Related Work
Mechanistic interpretability seeks to break down neural networks into human-undeeee understandable com- ponents. Prior work has uncovered specialized attention heads and MLP circuits supporting tasks like indirect object identification and fact recall, typically treating heads or layers as atomic units [Elhage et al., 2021, Wang et al., 2022, Meng et al., 2022]. Our work complements this by going inside each head using singular value decomposition (SVD) on augmented weight matrices. This reveals independent computational directions, each implementing a distinct function, such as semantic

---

separation, entity salience, sequence initialization in head 9.6. While past SVD-basen analyses focused on isolated components [Gao et al., 2024, Cunningham et al., 2023], we generalize the approach across all core circuits, query-key, output-value, and MLP transformations, showing that individual directions, rather than entire heads, can carry task-relevant computations. This perspective also complements studies showing interdependencies across heads [Merullo et al., 2024a]. Instead of modeling communication between heads, we decompose each head internally, highlighting low-rank primitives that drive behavior. In doing so, we introduce a direction-level granularity, which provides a finer, mechanistically meaningful lens on model computation and opens a path toward understanding how overlapping subfunctions combine within single attention head.
Discussion and Limitations
Rethinking Transformer Components. Our work emphasizes simple but powerful idea: trans- former components are not monolithic units, but rather collections of independent functional direc- tions. Each direction encode distinct computational primitive, such as sequence initialization, entity salience, or semantic separation, allowing a single head or MLP block to multiplex multiple sub-functions. By zooming in these low-rank directions, is invisible when treating heads or layers as atomic, providing a finer- graiiedi understanding of how transformers compute. This perspective has broad implications. It suggests that mechanistic interpretability should move beyond unit-level analysis, toward frameworks that capture both the individual roles of directions and their interactions, potentially defining new "micro-circuits" within components. Future work could explore direction-leve communication patterns, drawing inspiration from recent studies on head-to-heal interaction graphs [Merullo al., 2024b], understand how these sub- fub-functions combine into higher-level behaviors.
can reveal hidden structure that
The directional decomposition perspective we present provides fine-grained view into transformer computations; however, several limitations remain for further exploration. First, our analysis considers each augmented matrix independently, focusing on individual layers and heads. This isolation facilitates precise attribution of function to specific singular directions but may obscure emergent behaviors that arise from interactions across components, which are principal to many transformer computations. Second, the approach assumes that singular directions correspond to interpretable and causally relevant subroutines. Although supported by observed semantic boundaries and activation patterns, this assumption lacks formal justification, and task-relevant computation may be distributed across weaker directions or higher-order interactions. Third, our method applies learnable diagonal masks to fixed singular vectors, thereby restricting optimization to axis- axis-aligned subspaces, which may limit expressiveness. Allowing controlled perturbations of singular directions could capture finer task-dependent variations. Fourth, our evaluation is limited to the standard benchmark tasks and GPT-2 Small. Whether the method scales to more complex reasoning tasks, larger models, or instruction-tunet systems remains an open question. Finally, while our method reveals functionally interpretable directions, does not yet fully elucidate the causal role of each direction in the end-to-end model behavior. Our masking objective encourages output faithfulness, but disentangling true causal mediation from representational correlation remains an open challenge. Future work could integrate causal tracing or activation patching more directly establish mechanistic influence.
Limitations:
Despite these limitations, directional decomposition provides new lens on transformer internals. By treating model components as collections of functional directions, we can uncover the computational primitives that underlie complex behaviors, laying the foundation for more systematic and fine-grained circuit discovery in modern language models.
8 Conclusion
Transformers are often seen as black boxes, complex computational graphs of neurons, heads, and layers whose inner workings are difficult to disentangle. In this work, we demonstrate that a simple shift in perspective, focusing on individual singular directions in parameter space, can reveal surprisingly clean and interpretable sub-computations. This perspective also suggests a broader narrative. Might we someday construct compact, modular explanations of entire model behaviors by stitching together just a few interpretable directions? If so, we may one day piece together modular explanations of entire model behaviors from a small number of interpretable directions.
10

---

Acknowledgments
We would like thank the anonymous reviewers and the meta-reviewer for their insightful comments and suggestions. This research work was partially supported by the Research-1 Foundation of the Department of CSE at IIT Kanpur.
References
Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. In The Thirtt-eighth Annual Conference on Neural Information Processing [Cited pages 1,
Systems, 2024. URL httpss://openreiiew..rtt http://www.e.eeooooom/ddd//////0 5, 7, and 9.]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, 145/2005511415, 2020. URL https:: //wxvv////vv/vva////////eeaaaava htttp::/aaxxv.orr///// wttpooo//bbb///e.....5 14165. [Cited on page 1.]
Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL forum?1d=39ia777728. [Cited on pages 2, 6, 8, and 9.]
netee
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models, 2023. URL /ttpxs///aaxv..ooo abs/2309. 08600. [Cited on page 10.]
Nelson Elhhgge Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfiel-Dodds, Danny Hemandezz, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. hutps:/transformee-cccciiss wttprriuuusssuu//22222e2 pub/2021 wwta///////aammooo///neee html. [Cited on pages 3 and 9.]
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024. URL https:
//arxiv.org/ bboggg/bbs//n 2406 04093. [Cited on page 10.]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aicsha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arricta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas
11

---

Landzaat, Luke de Oliveira. Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike I Lewis. Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang. Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishman, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen. Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamaa Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao. Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen. Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo. Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury. Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape. Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence. Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu,
Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins. David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi. Zhang, Guna Lakshminarayanan. Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam. Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein. Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang. Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark. Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal. Nick Egebo. Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay,
12

---

Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang. Shuqiang Zhang. Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li. Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary De Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https: / http///wxvvvvi.///////eee/e..avt org/ abs /20..//177.2 21783. [Cited page 1.]
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in pre-trained language model. Advances in Neural Information Processing Systems, 36:76033-76066 2023. [Cited on pages 2, 6, 19, and 29.]
Michael Hanna. Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In First Conference on Language Modeling, 2024. URL htt.h::p:://// //ppnreeee///peeeeeeeeeeee wtw.:/eee//oooominddtcccooo
[Cited on page 33.]
Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching, 2024. URL hitps:///ariis https://arxiv.org/abb/ 2404. 15255. [Cited page 1.]
Aliyah R. Hsu, Georgia Zhou, Yeshwanth Cherapanamjeri, Yaxuan Huang, Anobel Odisho, Peter R. Carrol,, and Bin Yu. Efficient automated circuit discovery in transformers using contextual decomposition. In The Thirteenth International Conference on Learning Representations, 2025. [Cited on page 33.]
URL https:/ http://eeet/ooorm///d/1111111///
Mojan Javaheripi. Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, Suriya Gunasekar, Piero Kauffmann, Yin Tat Lee, Yuanzhi Li, Anh Nguyen, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Michael Santacroce, Harkirat Singh Behl, Adam Taumann Kalai, Xin Wang, Rachel Ward, Philipp Witte, Cyril Zhang, and Yi Zhang. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023. [Cited page 1.]
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL httpp:://arxiv abs/2310. 06825.. [Cited on page 1.]
Abhinav Joshi, Areeb Ahmad, and Ashutosh Modi. COLD: Causal reasoning in closed daily activities. In The Tiihth Annual Conference on Neural Information Processing Systems, 2024. URL [Cited on page 1.]
https://openrevieww net met//forumiid=77ooNo0s
Abhinav Joshi, Areeb Ahmad, and Ashutosh Modi. Calibration across layers: Understanding calibration evolution in LLMs. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 14697-14723 Suzhou, China, November 2025a. Association for Computational Linguistics ISBN 979 8 8-88918--333--88888 doi: 10. 18653/v 1/2025. emnip-main.742 URL https::/aclanttholpg org/ 2025. emnlp-main, 742/. [Cited on page 1.]
Abhinav Joshi, Areeb Ahmad, Divyaksh Shukla, and Ashutosh Modi. Towards quantifying com- monsense reasoning with mechanistic insights. In Luis Chiruzzo, Alan Ritter, and Lu Wang. editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages
Albuquerque, New Mexico, April 2025b. Association for Compu- tational Linguistics. ISBN 979-8-89176- 189-6.... doi: 10. 20.11866/11222 1/00l/. 1/2025.naacl-long....... nnaccl-long.48) URL. https: / http://aclanntooog.orrrrr/////// org. naacl- -lng. 487/. [Cited on page 1.]
13

---

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023. URL https://arxiv. org/abs/ 2309. 05463. [Cited on page 1.]
Chris Mathwin, Guillaume Corlouer, Esben Kran, Fazl Barez, and Neel Nanda. Identifying preliminary circuit for predicting gendered pronouns in gpt-2 small. URL: https://icch. sufjan//ncchin//aae//88899 2023. [Cited on pages 2, 6, and 19.]
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35, 2022. [Cited on pages and 9.]
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models, 2024a. URL //arpp http//axiv....g/abboa htrg/bbs/3111 08744. [Cited on page 10.]
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Talking heads: Understanding inter-layer communication in transformer language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL httpss://ppenreiiee..ttfffrooore
LUsx0chTsL. [Cited on pages and [0.]
Neel Nanda and Joseph Bloom. Transformerlens. TransformerLens, 2022. [Cited on page 20.]
Neel Nanda, Chris Olah, Catherine Olsson, Nelson Elhage, and Hume Tristan. Attribution patching: Activation patching at industrial scale, 2023, URL https://ww/ neelnanda. io/ [Cited on page 33.]
mechanistic
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction circuits. Distill, 2020, doi: 10.2391 //1ist/illl.000024..0000
[Cited on page 1.]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Estelle Tompson, Laurens Desmaison, Alban Kopf. Edward Yang, Zachary Viti,o, Martin Raison, Alykhan Tejani, Sasank Chilimbi, and Benoit Prabhakaran. Pytorch: An imperative style, -ergh-eeffrmmece deep learning library. In Advances in neural information processing systems, pages 8024-8033 8035, 2019. [Cited on page 19.]
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. URL https://app www.seescchcoollrr..oooo CorpusID: 60025533. [Cited on pages 1, 3. and 6.]
Yi-Lin Sung. Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks, 2021. URL https: //arsi////xxvvv///// http:/oorg//b////////// 198999. [Cited on page 5.]
Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated cir- cuit discovery. In Yonatan Belinkov, Najoung Kim. Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen, editors, Proceedings the 7th BlackboxNI.P Workshop: Analyzing and Interpreting Neural Networks for NL.P, pages 407-416 416, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10. 1663/////22bbbcccbxxoo blackboxnlp- 1.25. URL https://aclantholl/ org/ blackboxnlp-1 25/. [Cited on pages and 2.J
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Ilama 2: Open foundation and fine-tuner chat models, 2023. URL https: / //twxaaaivv//aa////a//eaa/aaaav/ wwtppxvw..o///.oooooeeeee.oo abs/ 09288. [Cited on page 1.]
14

---

Kevin Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability the wild: circuit for indirect object identification in gpt-2 small, 2022. URL https:/ / 00593. [Cited on pages 1, 2, 6, 7, 8, 9, 17, 18, 19, 25, 26, 27, and 29.]
15

---

Appendix
Table of Contents
A Details of Datasets
19
A.1 Indirect Object Identification (IOI)
19
A.2 Gender Pronoun Task
19
A.3 Greater-Than Task
19
B Additional Results, Discussion and Future Directions
19
B.1
Compute Resources
19
B.2 Inherently formed Logit Directions
20
From Singular Vectors to Logit Receptors.
20
Interpretable Logit Receptors.
21
Scalar-based Causal Ablation of OV Directions.
23
Findings.
23
................................
B.3 Extended Analysis: 10I Task
25
Additional Mask Visualizations.
25
Head 9.9: Entity Prioritization and Inhibition Dynamics
26
Head 10.0: Semantic Role Differentiation and IOI Bias Encoding
26
Overall Summary:
27
B.4 Analysis for Greater Than task
29
Head 9.1: Distributed Temporal Anchoring
30
Head 6.9: Sharply Localized Numerical Discrimination
31
Head 5.5: Redundant High-Magnitude in Reinforcing the Endpoint
Cross-Head Trends and Shared Functionalities
31
Overall Summary:
B.5
Universal Composite Functionalities Discovered
31
B.6
Sparsity Computation
32
Relative Sparsity.
32
Full Sparsity.
B.7 Discussion and Future Directions
33
List of Tables
2 Dataset splits for the three tasks used our experiments, 1OI (Indirect Object Identification), GT (Gender Type), and GP (Gender Pronouns), indicating the number of examples allocated to the training, validation, and test sets.
20
3 Hyperparameters used for training the linear probes and other learned components in our experiments. We report batch size, number of epochs, optimization parameters (learning rate and weight decay), and the coefficient of the L. regularization term.
20
16

---

4 Selected gender- and number-related OV singular directions. Columns show the learned mask, singular value a, an excerpt top tokens from Ww. mean activation v' (mean + std) conditioned on the pronoun based on prompt context(p.), and the activation difference (he minus she).
22
5 Empirical results of intervention(scaling + swapping). Baseline ALogit shows original logit difference between correct and opposite pronoun prediction based on the prompt context. Intervention ALogit shows the same after replacing activations of ' with respective opposite gender means, and amplifying singular value by multiplying with scaale- Flip rates show the baseline pronoun predictions that were switched to the opposite pronoun by the intervention. All ALogit(Logit Difference) values shown mean + std.
25
6 Scalar-based ablation of sender-related OV singular directions. Each intervention replaces the scalar activation of each gender direction with its empirical opposite- gender mean and amplifying the singular value(x20 in below table). n. is the number of data points
25
7 The table shows results for the analysis of S7 attention scores( [1, za 17770 x,, for head 9.6, highlighting the entity-action distinction role the I0I task.
26
8 The Table shows results for analysis of S2s activations, denoting the role of entity salience in the IOI task.
26
9 Learned mask values and associated functional roles for selected singular directions in Layer 9, Head 6. Higher mask values indicate stronger contribution of that singular direction to the corresponding functional role, illustrating how specific QK directions encode distinet computational subfunctions.
27
10 SVD Component Analysis of WQK Attention for the Greater Than Task, Focus aug
on Head 9.1. Multiple low-rank components exhibit high attention to the target year token (YY), which is crucial for accurate prediction. The "Highest Attention % indicates how often YY received the highest attention score.
30
11 SVD Component Analysis of Wok Attention for the Greater Than Task, Focus on Head 6.9. Multiple low-rank components exhibit high attention to the target year token (YY), which is crucial for accurate prediction. The "Highest Attention %" indicates how often YY received the highest attention score.
List of Figures
3 Mean activation of gender-related directions conditioned on Masculine versus Feminine prompt context. The x-axis plots the mean activation E/y Tu prompt context=he] and the y-axis plots mean Ely u prompt context=she). Er- ror bars show one standard deviation. The dashed diagonal line represents x, where activations for both pronouns would be equal.
21
4 Causal interventions (scaling + swapping) show that singular directions control gender pronoun prediction. The plot displays logit differences (Correct Pronoun Opposite Pronoun) and flipping rates after intervention. Singular values a are scaled by an integer factor (oate) (á v u.)u!! leading near-complete prediction reversal at higher scales. This provides causal evidence that these directions are key computational units underlying gender pronoun resolution.
24
5 Learned singular value masks for OV (WOV) aug) matrices across all attention heads in the model. The masks show heads with high activation across multiple singular dimensions correspond to circuit components previously identified by Wang et al. [2022] for the IOI task.
27
6 Learned singular value masks for MI.P (W ang (left) and W (ouut) ( (right)) matrices across all layers in the model for the I0I task.
28
17

---

7 The figure shows the average mask values for W across attention heads, cat- egorized by functional type. Heads identified circuits by Wang et al. [2022]; particularly Name Mover, Backup Name Mover, and Negative Name Mover heads, consistently exhibit higher average mask values than other head types. This suggests correlation between circuit membership and mask activation strength, providing quantitative validation previously identified functional circuits.
ang
29
8 Analysis of head 9.6, previously identified by Wang et al. [2022] as "Name Mover head" that attends to previous names in a sentence and copies them. Our SVD analysis reveals multiple distinct functionalities within the QK interaction, each serving specific roles consistently across the dataset. This decomposition provides more nuanced understanding of the head's 's behavior.
29
9 The above figure shows the attention of head 9.1 for the Greater Than task. S attends highly to the first token, and others, such as S3 and S31, predominantly focus on the year token.
30
10 The figure above shows an instance of attention scores of the final token for com- ponent S1 of head 9.6. It demonstrates that S1 consistently functions start-of- sequence detector across tasks, independent of context.
32
11 The figure above shows an instance of attention scores of the final token for com- ponent S7 of head 9.6. demonstrates that S7 consistently act as an Entity action separator by giving the highest attention score to Name and object entities, and the least attention score actions.
32
18

---

A Details of Datasets
This section provides details about the datasets used in our experiments. While the Indirect Object Identification (IOI) task is our primary benchmark due to its strong alignment with our goals in mechanistic interpretability, we also include two additional datasets, Greater-Than [Hanna al., 2023] and Gender Pronoun [Mathwin et al., 2023], to test the generalizability of our method.
A.1 Indirect Object Identification (IOI)
The IOI task [Wang et al., 2022] is synthetic benchmark designed study language model's ability to resolve coreference between proper names in complex syntactic constructions. Each prompt contains two names in an introductory clause, followed by clause in which one name as the subject. The model must complete the sentence with the indirect object:
"When Mary and John went to the store, John gave drink to" Mary
The task follows a known and interpretable algorithm, i.c., predict the name that is not the subject of the last clause. Its controlled structure and clear ground truth make ideal for circuit-level analysis and interpreting internal representations of models like GPT-2.
A.2 Gender Pronoun Task
This task evaluates pronoun resolution in socially relevant contexts. Each example is declarative sentence about a named person followed by a tag question. The goal is to complete the sentence with the correct gendered pronoun:
"So David is really great friend, isn't" - he "So Mary a very good athlete, isn + she
This dataset complements IOI by testing the model's ability to resolve pronouns and capture gender semantics. It also provides insight into whether our method can disentangle social biases embedded in model representations.
A.3 Greater-Than Task
The Freeter-Thar task [Hanna et al., 2023] targets numerical reasoning by prompting the model with sentence involving two years. The goal is to predict a valid end year that greater than the start year, using a textual format:
"The treaty lasted from the year 1314 to the year 13" + 28
The dataset is automatically generated using 120 nouns representing temporal events (sourced from FrameNet), with years sampled from the 1thh to 17th centuries. Special care is taken to avoid multi-token year completions and boundary cases, ensuring all completions are single-token and meaningful under GPT-2's tokenizing scheme.
B Additional Results, Discussion and Future Directions
In this appendix section, we provide extended results, analysis, and exploratory observations that complement the findings in the main paper. We provide further support for the interpretability of low-rank directions across multiple tasks, highlight composite functional patterns, and outline promising directions for future work.
B.1 Compute Resources
All experiments were conducted on a single NVIDIA A40 GPU with 48 GB of VRAM. Our imple- mentation is based on PyTorch 2.1 [Paszke et al., 2019] and the HuggingFace Transformers library (v4.35), leveraging its integration with pre-trained models and tokenizer utilities. We utilize the GPT-2 small model (124M parameters), with all weights frozen during experiments to ensure
19

---

Table 2: Dataset splits for the three tasks used in our experiments, IOI (Indirect Object Identification), GT (Gender Type), and GP (Gender Pronouns), indicating the number of examples allocated the training, validation, and test sets.
Task
Train
Validation
Test
I01
lk
200
1k
GT
2k
500
2k
GP
1k
155
307
Table 3: Hyperparameters used for training the linear probes and other learned components in our experiments. We report batch size, number of epochs, optimization parameters (learning rate and weight decay), and the coefficient of the LI regularization term.
Hyperparameter
Value
Batch Size
64
Number of Epochs
Learning Rate
1.0 10-2
Weight Decay
1.0 x 10
L1 Regularization Weight
1.5 x
the integrity of the underlying representations. For systematic access to internal model activations and component-wise analysis, we employ the TransformerLens library [Nanda and Bloom, 2022], which provides fine-grained control over the transformer's intermediate computations, enabling singular value decomposition and mask-based interventions at the component level. All training for mask optimization and probing modules conducted using batches of IOI Greater-Thhe task prompts, with early stopping based on validation loss to ensure efficiency and generalizability. The details to the Dataset splits and Hyperparameters are provided in Table 2 and Table 3, respectively, Table outlines the dataset splits used across different tasks, Indirect Object Identification (IOI), Greater- Than (GT), and Generalized Preposition (GP). We relatively small training sets to textita- size the interpretability and efficiency of our method, while validation and test sets are significantly larger ensure robust generalization. The training procedure for low-rank component masks and probing modules was consistent across tasks, with hyperparameters detailed in Table 3. We optimize using AdamW W with moderate weight decay and L.1 regularization encourage sparsity in learned masks. Training is conducted for a maximum of 150 epochs with early stopping based on validation performance avoid overfitting. The relatively small batch size and learning rate were empirically found to balance convergence speed with stability.
B.2 Inherently formed Logit Directions
Beyond component-level decomposition, our analysis reveals that in transformer layers there exist a set of inherent, fixed directions logit space, which we term as "Logil Receptors" (so named because they act as fixed receptive directions that deterministically modulate specific token logits). These are directions that correspond to stable vocabulary preferences learned during training and that operate as modulators for specific tokens. These directions emerge naturally from the singular value decomposition of the augmented OV projection, w!OV), which writes the attention output into the residual stream.
From Singular Vectors to Logit Receptors. Let (I++daaa)) denote the attention antention-weighhed (and one-augmented, to compensate bias) value vector for layer I, head h. where augment the context with constant so that VIj, = =11, E, Osjzyy in the notation above(a: is attention distribution and zy is post-layernorm residual stream representation). Decomposing the corresponding OV projection by singular value decomposition gives
W(OV) Unn Ein. VIT.
20

---

111 01.40.50
LSC.IR.50 SVD
Mean Activation (she)
0.2
0.0
112.00
-02
-0.4
Equal Activation (ymx)
Masculine
Feminine
Gender Neutral
Mean Activation (he)
Figure 3: Mean activation of gender-related directions conditioned on Masculine versus Feminine prompt context, The x-axis plots the mean activation Efy prompt context=he] and the y-axis plots mean EfuJu u prompt context=she). Error bars show one standard deviation. The dashed diagonal line represents x, where activations for both pronouns would be equal.
The output, in equation (I), written by head It in layer in the residual stream can then be expressed
yia = Vih
W ( sug
Σ
Tink (VTh wink)
Vink-
k
where each term corresponds one singular direction with singular value dtkk left singular vector Wikk, and right singular vector Vikk- The scalar term v/J, unk represents an input-dependens coefficient that steers a fixed direction in residual space.
When each right-singular vector viaa is projected through the unembedding matrix Wr,, we obtain a scaled vocabulary-length vector
Vink = (aInk (1TT (unk)) UTAR Wu,
which corresponds to a fixed direction (pointing towards a specific set of tokens) in token-logit space. Importantly, the term wie Wy depends only the learned weights; in other words, it is constant Logit Receptor independent of any specific input. The only input-dependent term is its scalar activation, PI Whkk which modulates how strongly the model calls upon that logit receptor a given step.
Interpretable Logit Receptors. Empirically, found that some of these fixed logit receptors have clear linguistic meaning. In the Gender Pronoun Resolution (GP) task, for instance, certain ViT Wu vectors are more inclined towards gendered pronouns. Their corresponding activations viJ. unna are systematically higher for examples of one gender than the other, forming mechanistie pattern, i.e. the model maintains distinct "she" and "he" receptors in logit space and routes activation selectively between them depending on context.
This perspective reveals fresh way of understanding concepts present inherently in the internal structure of the model, even before considering the mask-learning step; the network already contains
21

---

Table 4: Selected gender- and number-related OV singular directions. Columns show the learned mask, singular value o, an excerpt of top tokens from v Wo, mean activation v u (mean + std) conditioned the pronoun based on prompt context(pe), and the activation difference (he minus she).
Direction
Mask
Top tokens (excerpt)
BluT u pe-he]
Ely u pc=she]]
Diff
L9.
1.00
8.87
His, his, He, he, himself
+0.115 0.027
-0.453 1 0.032
+ +0.568
L11. SV6
1.00
6.52
his, His, him, Hc, he
+0.134 = 0.030
-0.412 = 0.029
+0.546
L10.H9.SV
1.00
9.15
her, she, She, herself, hers
0.273 0.041
+0.652 + 0.038
0..25
111.H7.SV
1.3e-5
22.07
herself, Her, her, hers, She
+0.755 + 0.020
+0.774 + 0.018
00.09
L10.H9. SV1
0.82
7.87
her, she, herself, She, she
-0.241 = 0.035
+0.512 + 0.036
00 757
L9. Sve
1.00
9.23
their, Their, they, their
00.334 + 0.028
-0.174 + 0.031
0. 160
L630
1.00
4.64
they, They, Them, Their
0.196 + 0.022
-0.114 + 0.025
-0.82
L11.H8.SV22
0.87
4.32
their. Their, THEY, THEIR, They
-0.278 + 0.024
-0.143 + 0.027
-0.135
a compact basis of hard-coded logit receptors that modulate the vocabulary space. These receptors define semantic axes that downstream layers can selectively activate express context-sensitive meaning. Learning directional masks (via Algorithm 1) further clarifies which of these fixed logit receptors are actually used by the task. Directions with high mask weights may correspond those carrying discriminative information for the task, while others are effectively pruned. Moreover, it also reflects that some of the inherent structures are not used by a specific dataset, and the discovery of components is highly dependent on the used dataset, which may not contain all the universal inherent directions present in a model.
notable example is L1 1..7...V where despite having the largest singular value (a = 22.07) and a vocabulary projection strongly associated with feminine tokens, this direction receives almost- zero mask. Inspection reveals that its activation 12h umna (a scalar modulating the logit receptor) is nearly identical for male and female examples, essentially encoding variance but discriminative signal (also see Figure 3). In contrast, directions such as L10. H9. SVe show pronounced activation differences across gender contexts and are thus retained. Table 4 summarizes the salient discovered directions, including their mask weights, singular values, top-vocabulary tokens, and conditional activation statistics. Together, these results suggest that the model's 's internal computation operates through set of inherently learned logit receptors, fixed basis vectors whose selective activation underlies task behavior and provides a grained, mechanistic bridge between representation and prediction.
This decomposition helps provide an understanding of how the model performs token-level decisions. Each unembedding projection of right-singular direction i.e. Vite We acts as a fixed logit receptor controlling the prediction vocabulary space, essentially representing stable axis associated with specific token outcome. During inference, the model dynamically steers these receptors through the input-dependent activation coefficients 11, uAk- The masks we learn over these singular directions identify which of these receptors the model actually employs, i.c., directions that combine both capacity (large singular value) and discriminative alignment (systematic activation shifts across labels) receive high mask weights, while those that contribute variance but no discriminative signal are suppressed during optimization. As highlighted in Figure 3, the activations 2j,, uth form linearly separable clusters for masculine and feminine contexts; directions lacking decisive separation receive negligible mask weights, indicating that the masking learns to ignore non-discriminative directions. For example, direction L11.H7. sve, despite its large singular value (o = 22.07) and clearly feminine token projection, receives near-7ero mask because its activation distribution nearly identical across male and female examples.
22

---

We further ask what happens if we intervene in these directions. While the above analysis reveals interpretable correlations between singular directions and task behavior, it does not establish whether these directions are causally responsible for model predictions. To support our central hypothesis, that transformers distribute computation along distinct, low-rank subfunctions, we require causal evidence that manipulating these subspaces directly alters model output in predictable ways. If small number of identified singular directions are genuinely responsible for behavior, then systematically altering them should induce controlled, interpretable changes in the model's output logits, while leaving other computations intact.
Scalar-based Causal Ablation of OV Directions. We therefore perform series of Scalar-based counterfactual ablations designed to test whether the discovered singular directions in the OV projection causally steer gender pronoun prediction. In each ablation, we swap the activations of gender- sensitive singular directions (those with high learned mask values) with the empirically observed mean activations from the opposite gender distribution. For instance, the masculine-sensitive direction L9.H7 HV1 typically exhibits mean activations of +0. for "he" prompts and -0.453 for "she" prompts; the intervention exchanges these values, effectively inserting counterfactual gender evidence while keeping all other activations fixed.
Overall, we conduct four controlled experiments: 1) swapping all gender directions for masculine prompts, 2) swapping all gender directions for feminine prompts, 3) swapping only masculine direc- tions for masculine prompts, and 4) swapping only feminine directions for feminine prompts. Each condition measures how average pronoun-loge differences respond to these targeted perturbations, thereby quantifying the causal influence of individual low-rank subspaces.
Formally, for given singular direction i, the intervention replaces the natural activation (n T uz) with the opposite- -eneder mean activation a/:
A (d- ( 0.
adding this re- resscaled vector to the final residual stream vector directly intervenes in the contribution of specific interpretable low-rank features, allowing precise causal editing of gender-specific behavior. Algorithm 2 illustrates this process in more detail. This precisely modifies the residual-stream contribution of selected interpretable features while preserving the model's 's overall structure, enabling rigorous, scalar-controlled causal testing.
For empirical validation, we consider instances from the GP (Gender Pronoun Resolution) task and segregate prompts into distinct masculine (Pmale) and feminine (ptemale) contexts and pass them separately to the model and record the scalar mean corresponding to specific tokens (Hg = d =
PPl Exep, v U4i 9 {male, female)). This helps provide mean values for each considered context, and for all the pronoun directions E (L9.H7. . SV1, L11. .......V ........ which are later
used in Algorithm 2 to perform intervention and observe the effect of modifying a logit receptor. (also see Figure 2, that illustrates the intervention performed in Algorithm 2)
Findings. The resulting interventions, highlighted in Table 5, Table 6, and Figure 4, yield clear causal signal. In Table 5, the Flip->she% is defined as the (# predictions flipped from hee' "she') / (# baseline he' predictions) x x100, and Flip is defined analogously using the baseline she' predictions, i.e., (# predictions flipped from she' + he / (# baseline she predictions) x 1000. Both of these metrics help capture the true flip rate of recoverable predictions (excluding cases where the model initially predicted an "other" token). Across amplification scales shown in both Table 5 and Figure with their empirically opposite-gender means, and scaling the corresponding singular values by a positive amplification factor, reliably reverses the model's pronoun-logit polarity. For masculine prompts, the mean logit difference shifts from approximately +2.5 (favoring "_he') to around -42, and for feminine prompts from +2.8 roughly 41. Moreover, as also reflected in the high flip rates in Table 5, the model almost entirely reverses its prediction given the same baseline context, i.e., prompts containing masculine cues (where the correct baseline prediction yield she' after the mean-swap plus amplification intervention, and symmetrically for feminine- context prompts. The scalar-level ablations in Table further confirm that both full and partial swaps induce substantial, systematic shifts in logit differences, demonstrating that these singular directions contribute additively and consistently to gender-pronour resolution. All these results provide causal evidence that the identified OV singular directions form modular, low-rank mechanisms that directly
find that swapping all gender-associated singular directions
he')
23

---

Algorithm 2 Intervention for OV Singular Directions
Require: Model M, SVD circuit cache C, data loader D, directions S (each with (e, h, & Mhes Mohe)), and target gender g {he, she}
Ensure: Summary statistics: mean logit difference, variance, etc.
1: Initialize results fcorrect, logit_diff, diff, togi_logit)
for each batch (z. y) do
3: Run model with cache: (logits, cache) +- M.run with cache(æ)
4: Extract final-token index 1*
5: Initialize AR +-0
for each (e, h, Maey Mae)
do Retrieve SVD components S, + Cle h]
7:
8:
Select (u,, 0.) + (U[: f), il. S(i))
9:
Compute attention-weighted context [1, Σ ax)
10:
Current activation a, (v u+)
Target activation di t Motee he
11:
Mhe: she
Aa; (a'- as) 12:
13:
Jacuie oi AR += Da, os Us
15:
end for
16:
Add AR to final residual stream at t"
17:
Compute new logits Z' LaysrNorm(residual AR)We bu
18:
end for
100%
0
Opposite Pronoun Flip Rate (%)
Intervention Logit Diff
-10-
80%
20
60%
-30
40%
Dashed lines: Flip
Dotted lines: Baseline values
20%
Swap Experiments
Swap Masc.,
E.2: Swap ALL,
E.4: Swap Fem.,
0%
1.0
2.0 Amplification Scale Factor saale)
5.0
10.0
15.0
20.0
Figure 4: Causal interventions (scaling + swapping) show that singular directions control gender pronoun prediction. The plot displays logit differences (Correct Pronoun - Opposite Pronoun) and flipping rates after intervention. Singular values are scaled by an integer factor (a in (1,4 u; Ju leading to near-complete prediction reversal higher scales. This provides causal evidence that these directions are key computational units underlying gender pronoun resolution.
control gender pronoun prediction, demonstrating that interpretable subfunctions are not merely correlational artifacts but causal building blocks of transformer computation, which leads to perspective on looking at/understanding predictions made by the model.
We believe that as we move forward, this direeciion/perspectivee of decomposing/undeesaaddngg model decisions will make them more interpretable, and a wider range of transparent conceptualization can be explored/understood in future works.

---

Table 5: Empirical results of intervention(ssaling swapping). Baseline ALogit shows original logit difference between correct and opposite pronoun prediction based on the prompt context. Intervention ALogit shows the same after replacing activations of v with respective opposite gender means, and amplifying singular value by multiplying with scalee Flip rates show the baseline pronoun predictions that were switched to the opposite pronoun by the intervention. All A Logit Difference) values are shown as mean std.
Baseline ALggit
Interv. ALggit
Flip she%
Flip- he %
Experiment
Oscale
Prompt Type
E.I: Swap ALL
1.0
"he"
+2.53 1.48
-1.10 +: 0.83
83.5%
0.0%
E.I: Swap ALL
2.0
"he"
+2.53 1 1.48
-3.82 1: 0.63
100.0%
0.0%
E.1: Swap ALL
5.0
"he"
+2.53 + 1.48
-11.80 2.50
100.0%
0.0%
E.1: Swap ALL
10.0
"hc"
+2.53
-23.96 1 5.32
100.0%
0.0%
E.1: Swap ALL
15.0
"he"
+2.53 1
-34.16 7.50
100.0%
0.0%
E.1: Swap ALL
20.0
"he"
+2.53 1.48
-42.31 8.57
100.0%
0.0%
E.2: Swap ALL
1.0
"she"
+2.84
2.12
+1.01 1: 0.96
0.0%
4.7%
E.2: Swap ALL
2.0
"she"
+2.84
-1733 0.61
0.0%
67.0%
E.2: Swap AL..
5.0
"she" +2.84 2.122
-9.87 4.08
33.3%
100.0%
E.2: Swap AII.
10.0
"she"
+
-23377
33.3%
100.0%
E.2: Swap ALL
15.0
"she" +.18
33.3%
100.0%
E.2: Swap ALL
20.0
"she" +2.8
40.82
15.44
33.3%
100.0%
E.3: Swap Masc.
1.0
"he" +.45
+0.60 1.22
8.9%
0.0%
E.3: Swap Mase.
2.0
"he"
1.45
0.94
43.0%
0.0%
E.3: Swap Masc.
5.0
"he" +2.53
3.53
100.0%
0.0%
E.3: Swap Masc.
10.0
"he" +2.33
8.60 1.69
100.0%
0.0%
E.3: Swap Mase.
15.0
"he" 1.48
13.50 2.98
100.0%
0.0%
E.3: Swap Masc.
20.0
"he" +.48
-18.15 4.15
100.0%
0.0%
E.4: Swap Fem.
1.0
"she" 2.18
+2.05 = 1.39
0.0%
2.8%
E.4: Swap Fem.
2.0
"she" 2.11
+0.35 = 0.73
0.73
0.0%
12.3%
E.4: Swap Fem.
5.0
"she"
1.11
-4.75 1.92
16.7%
99.1%
E.4: Swap Fem.
10.0
"she"
+2.84 2.12
13.04
50.0%
100.0%
E.4: Swap Fem.
15.0
"shc" +2.84 2.12
-20.73
50.0%
100.0%
E.4: Swap Fem.
20.0
"she"
+.12
-27.59
50.0%
100.0%
Table 6: Scalar-based ablation of gender-related OV singular directions. Each intervention replaces the scalar activation of each gender direction with empirical opposite-gender mean and amplifying the singular value(x20 20 in below table). is the number of data points Experiment
Prompt Context
n
Baseline Aogii
Interv. LLogi
A
E.1: Swap ALL dirs
"he"
150
+2.53 +:2 1.48
-42.31 + 8.57
-44884
E.2: Swap ALL dirs
"she"
156
+2.84 + 2.12
-40.82 15.44
-43.66
E.3: Swap Masc. only
"he"
150
+2.53
1.48
-18.15 4.15
-21.38
E.4: Swap Fem. only
"she"
156
+2.84 2.12
-27.59 11.44
-19.93
B.3 Extended Analysis: 101 Task
Additional Mask Visualizations. Figures 5, 6, and 7 provide detailed visualizations of the learned singular value masks for OV, MLP, and QK matrices, respectively. These figures complement the results discussed in the main text by showing head- and layer-level activation patterns across the model. In particular, Figure highlights OV heads with high activation across multiple singular directions, Figure 6 shows layer-wise MLP mask patterns, and Figure 7 quantitatively correlates QK mask values with functional head types identified in prior work [Wang et al., 2022]. These visualizations help provide further evidence that the learned masks capture meaningful functional subcomponents of transformer computation. Figure 8 shows a conceptualization of the SVD-based analysis of Head 9.6, previously identified by Wang et al. [2022] a "Name Mover" head. The figure
25

---

Table 7: The table shows results for the analysis of S7 attention scores( 1,111, xy,
for Example values
head 9.6, highlighting the entity-action distinction role in the IOI task.
Category
Value (Mean + SD)
Example tokens
Entities
+3.52 = 1.42
"Jerry", "Kevin", "Susan" "went", " "gave", "decided"
+2.87, +2.29, +2.92
Actions
-4.44 0.68
-4.33, -.17. 55.9
Table 8: The Table shows results for analysis of S2s activations, denoting the role of entity salience in the IOI task.
Token type
Example tokens
SVD 28 values
Named entities (first mention) Named entities (sccond mention)
"Jerry", "Susan", "Kevin" "Mary", "Kevin_2" "Marilyn_2" "the", "to", "of"
5.68, 4.05, 5.22 3.69, 2.07, 2.07 0.50, 1.93, 0.15 2.92-3.85
Function words
Discourse connectives
"and"
illustrates the combination of multiple distinct functionalities within the QK interaction, including sequence initialization (S1), semantic discrimination (S7), and entity salience (S28), leading to a specific utility in the IOI task.
To complement the in-depth analysis of attention head 9.6 in the main paper, we extend our inves- tigation to two additional heads, 9.9 and 10.0, both previously implicated in the Indirect Object Identification (IOI) task by Wang et al. [2022]. Our aim is to determine whether their internal struc- ture. when decomposed through our singular vector masking framework, reveals similarly modular and interpretable subfunctions. The results provide compelling support for the view that heads encode multiple, independent functional primitives, each realized along distinct low-rank directions.
Head 9.9: Entity Prioritization and Inhibition Dynamics Attention head 9.9 has been previously labeled "Name Mover" head due its role in copying names across syntactic spans. However, singular vector analysis reveals a richer and more nuanced structure. The first and seventh singular directions, S and S7, consistently exhibit the characteristic pattern of start-of-sequence detectors. These directions assign disproportionately high attention to the first token of each prompt, while suppressing all subsequent tokens. This behavior mirrors similar mechanisms found in other heads and suggests general strategy for anchoring temporal computations.
More interestingly, the 51th direction, S51. exhibits selective affinity for named entities, especially those appearing early in a sequence. It systematically favors tokens corresponding to person names while suppressing their repeated mentions. This results in a subtle form of repetition avoidance that biases the model toward novel or contextually salient entities, behavior consistent with entity tracking and attention modulation observed in prior mechanistic studies.
Crucially, Sie implements a strong inhibitory signal targeting second mentions of entities. On average, the unnormalized attention score assigned to repeated entities is 2.31 points lower than that assigned to their initial mention, a pattern aligned with the role of S-Inhibition heads described by Wang et al. [2022], which helps suppress the attention. This directional suppression creates structural preference against re-attending to previously mentioned entities, thereby enhancing the likelihood of selecting appropriate indirect object. Collectively, these observations demonstrate that head 9.9 multiplexes multiple functions, entity recognition, novelty bias, and inhibition, across orthogonal subspaces.
Head 10.0: Semantic Role Differentiation and 101 Bias Encoding Head 10.0 presents another compelling case of functional modularity. As with head 9.9, the directions S₁ and S3 operate as strong -fart-ef- indicators, highlighting the recurrence of this structural primitive across layers. However, the sixth singular direction, Sg. shows an especially focused pattern: it consistently assigns high attention scores to named entities and location nouns, while strongly de-emphasizing function words and verbs. For instance, tokens such as "Melissa." "Kelly," and "zoo" receive the highest activations in our sampled prompts, whereas verbs like "gave" and conjunctions like "and" are suppressed. This direction functions as precise semantic filter, elevating entities and key nouns that anchor the core referential structure of the IOI task.
26

---

Duplicate Tocan Hoadt
S-anestton
Namoo Mengr Heads
Figure 5: Learned singular value masks for OV (Way))
aug) matrices across all attention heads in the
model. The masks show heads with high activation across multiple singular dimensions correspond to circuit components previously identified by Wang et al. [2022] for the IOI task.
Table 9: Learned mask values and associated functional roles for selected singular directions in Layer 9, Head 6. Higher mask values indicate a stronger contribution of that singular direction to the corresponding functional role, illustrating how specific QK directions encode distinct computational subfunctions.
Singular Direction
Functional Role
Learned Mask Value
S
Sequence Initialization Detection
0.53
S7
Semantic Separation of Entities and Actions Entity Salience and Detection
0.64
S28
0.97
In contrast, the third direction, S3. displays a consistent aversion function words. Words such as "a," "the," and "and" receive markedly negative attention scores, suggesting broader mechanism of syntactic sparsification. Meanwhile, the seventh direction, S7. echoes the behavior observed in head 9.6, acting as semantic separator between entities and verbs. These orthogonal semantic dimensions help disentangle the "who" from the "what," facilitating downstream resolution of coreference.
Finally, the twentieth singular direction, S20, exhibits statistical preference for indirect objects. In approximately 54-59% of cases, this direction assigns higher attention the token corresponding to the indirect object, closely mirroring the empirical success rates of the model on the 1OI task. This suggests that S20 encodes a partial bias that systematically favors the correct grammatical resolution in ambiguous syntactic constructions.
Overall Summary:
The analyses of heads 9,9 and 10.0 reinforce our core hypothesis, i.e., trans- former components, rather than operating as indivisible functional units, exhibit finely grained internal specialization along interpretable low-rank directions. Each direction performs targeted role, whether syntactic anchoring, semantic discrimination, or statistical preference modulation, that contributes to the overall behavior of the head. Our results not only validate prior circuit-leve findings
27

---

MLP Singular Value Hanks
Laverd Projectie
Layer Haad Propccis
Laye Oegurt Popectan
15
Layor Uead Pnggecte
Sape Super Pogortan
Layer -iasd Propeciue
Laye D- Priectinn
LayerT Projectte
layr7
Sayurr Newee
Sepatr
Figure 6: Learned singular value masks for MLP (W (Wiing and W (right)) matrices across all layers in the model for the IOI task.
28

---

220
Average
Previo us Taken Heads
induction Heads Sarribition
Negative Name Mover Heads
Ba dkup Name Nover Heads
Duplicate
Name
Figure 7: The figure shows the average mask values for WQK across attention heads, categorized by aug
functional type. Heads identified in circuits by Wang et al. [2022]; particularly Name Mover, Backup Name Mover, and Negative Name Mover heads, consistently exhibit higher average mask values than other head types. This suggests a correlation between circuit membership and mask activation strength, providing quantitative validation of previously identified functional circuits.
U
T 28
W QK aug
81
S7
S28 * 1228
Start- SoorttOf-Soe OS-Sequence Detector
Entity- Action Sepurator
Entity Detector
Figure 8: Analysis of head 9.6, previously identified by Wang al. [2022] as "Name Mover head" that attends to previous names sentence and copies them. Our SVD analysis reveals multiple distinct functionalities within the QK interaction, each serving specific roles consistently across the dataset. This decomposition provides a more nuanced understanding of the head's 's behavior.
but also refine them by isolating the precise mechanisms responsible for observed model behavior. This decomposition presents a powerful perspective for interpretability, enabling a move from coarse component-level attributions to direction-level mechanistic understanding.
B.4 Analysis for Greater Than task
To evaluate the generality of our method beyond the IOI task, we extend our analysis the Greater Than benchmark introduced by Hanna et al. [2023]. which investigates a model's capacity for numerical comparison. Each input prompt presents two years within templated sentence, e.g., "The treaty lasted from the year 1314 to the year 13", and the model must complete the final token(s) such that the resulting year is strictly greater than the first. Crucially, the completion must yield valid multi-token year (e.g., 28 completing "1328"), with careful curation avoid boundary conditions and single-token years that could confound interpretability.
We analyze this task by focusing attention heads previously identified as critical to the model's s numerical comparison behavior, specifically heads 6.9, 9.1, and 5.5 in GPT-2. Using our method, we dissect the W-Qk matrices of these heads into their dominant singular directions and interpret their respective contributions.
29

---

aug Table 10: SVD Component Analysis of W Attention for the Greater Than Task, Focus on Head 9.1. Multiple low-rank components exhibit high attention the target year token (YY), which is crucial for accurate prediction. The "Highest Attention % indicates how often YY received the highest attention score.
SVD Component
Avg. Attention (# Std)
Highest Attention (%)
Mask Value
S31
7.09 1.68
100
1.00
S3
4.79 + 1.08
99,8
1.00
S26
1.86 + 0.87
90,9
3.29 x 10
S59
2.94 1.86
81,8
1.00
S37
4.71 2.18
81.5
1.00
S32
3.90 + 1.65
72.7
1.00
-20
$_61
$_57
$_53
20
S 49
SVD Components
$_45
$_41
10
$_37
$ 33
$_ 29
$_25
21
S 17
-10
13
9.9
$_5
__1
The
work
lasted
from
17
66
10
17
Token Position in Sequence
Figure The above figure shows the attention score of head 9.1 for the Greater Than task. S attends highly to the first token, and others, such as S3 and S31, predominantly focus on the end-of-year token.
Head 9.1: Distributed Temporal Anchoring Attention head 9.1 (Figure 9) reveals a distributed mechanism for endpoint detection. The first singular direction (S1) consistently assigns high positive attention to the sfeuucef token, while giving negative scores to subsequent ones. This positional prior, commonly reused across tasks.
behavior functions
More importantly, multiple directions contribute to highlighting the terminal year token (YY). For instance, S31 and S3 achieve near-perfect precision in identifying YY as the attention focus, doing so 100% and 99.8% of prompts, respectively (see Table 10). Other components such as Sa2 and S39 also consistently amplify YY, although with lower magnitude or frequency. This suggests that head 9.1 does not rely on a single dominant axis, but instead employs compositional mechanism where multiple orthogonal components redundantly encode attention to temporal anchors.
30

---

Table 11: SVD Component Analysis of W Attention for the Greater Than Task, Focus on Head 6.9. Multiple low-rank components exhibit high attention the target year token (YY), which is crucial for accurate prediction. The "Highest Attention % indicates how often YY received the highest attention score.
aug
SVD Component
Avg. Attention + Std
Highest Attention (%)
Mask Value
2
10.747 2.858
100.0
1.00
S7
4.324 + 2.482
83.3
6.81 x 10-6
S18
4.680 - 2.327
83.3
0.15
S20
4.897 + 4.967
66.7
0.99
S29
3.249 = 3.587
50.0
1.14 x 10
Head 6.9: Sharply Localized Numerical Discrimination In contrast, head 6.9 demonstrates highly concentrated behavior. The second singular direction, S,, exhibits extremely sharp selectivity, consistently assigning the highest attention to the final year token with 100% accuracy (u 10.747 = 2.858) (see Table 11 for reference). This direction appears to isolate the second numeric entity in the input, crucial for the Greater Than judgment. Supporting directions such as S7, S18, and S20 reinforce this emphasis with selection rates above 66%, albeit at lower attention magnitudes. This pattern reflects a localized, narrowly targeted strategy for operand comparison.
Head 5.5: Redundant High-Magnitude Reinforcing the Endpoint Head 5.5 features richer distribution of strong aitentioo-inducing components. The leading singular direction (S1) again exhibits the sfart-of-sequence pattern, while several others like S17, S28. S52, robustly identify the YY token. Notably, S1s stands out with the highest attention magnitude across all heads (19.14 + 7.15), suggesting an exceptionally focused mechanism for endpoint amplification.
While some of these components exhibit redundancy in their selection behavior (e.g., S22 and S52 both achieving 87.5% attention on YY), they differ significantly in their mean scores and variance. This dispersion implies ensemble encoding, where multiple vectors provide convergent evidence toward the correct numeric endpoint, increasing the model's reliability across input variations.
Cross-Heat Trends and Shared Functionalities Across all heads, we observe the emergence of two recurring functional patterns: (1) -equeeeee structure detection via S1, and (2) endpoint amplification via a sparse set of highly specialized directions. These shared functionalities suggest that certain subspaces, such as those capturing positional priors or token-type distinctions (e.g.. numbers nouns), may be reused across attention contexts. Moreover, the existence of multiple low-rank directions targeting the same token suggests that GPT-2 distributes the implementation of numerical reasoning across a sparse ensemble of interpretable basis vectors.
Overall Summary: This decomposition of the Greater Than task reveals a consistent, structured strategy whereby numerical comparison is encoded not in monolithic head-level behavior, but in sparse, orthogonal subspaces within each attention matrix. Each direction contributes a distinct yet complementary role, which includes establishing context, identifying operand positions, and assigning salience to temporally relevant entities. These findings provide compelling evidence that transformers perform symbolic reasoning via emergent low-dimensional structures, and that these structures are modular, reusable, and interpretable via singular decomposition.
B.5 Universal Composite Functionalities Discovered
Our analysis reveals that certain singular value components exhibit consistent, reusable function- alities across different attention heads and tasks, pointing the presence of universal composite functionalities within GPT-2's internal representations. For example, component S₁ in attention head 9.6 consistently acts as a to the initial token regardless of the task context, as illustrated in Figure 10. This persistent role suggests that some attention subspaces are dedicated to foundational structural signals critical for sequence processing. Similarly, component S7 of the same head functions reliably an entity-action separator, selectively attending to name and object entities while suppressing attention to action or functional tokens, as shown in Figure 11. The consistency of these components across diverse
detector, robustly assigning high attention scores
31

---

Greater Than
offect
-12
SVD Value
Gender Pronoun
Jacbb
phrnp
101
Token Index
Figure 10: The figure above shows an instance of attention scores of the final token for component Si of head 9.6. It demonstrates that S consistently functions start-of-sequence detector across tasks, independent of context.
SVD
101
Kewin
Index
Figure 11: The figure above shows an instance of attention scores of the final token for component S7 of head 9.6. It demonstrates that S7 consistently highest attention score to Name and object entities, and the least attention score to actions.
Entity action separator by giving the
contexts implies that transformer models implement set of primitive, composable mechanisms that are repurposed modularly to support a variety of reasoning and linguistic tasks. Uncovering and characterizing such universal functionalities not only enriches understanding of transformer interpretability but also paves the way for targeted interventions, modular editing, and transfer of learned behaviors across models and domains. These findings motivate further investigation into other universal components and their interplay. which may reveal a hierarchical structure of model reasoning primitives embedded in singular value decompositions of attention matrices.
B.6 Sparsity Computation
We define sparsity terms of the number of singular directions that are effectively suppressed during the mask learning process. Prior to optimization, all directions with negligible singular values are discarded, leading to a negligible penalty of approximately 10-5 in KL divergence. The remaining directions constitute the learnable subspace over which mask optimization is performed.
Since the model undergoes two kinds of compression, (i) Zeroing out directions with near-zero singular values, and (ii) Pruning directions with high singular values through learned masks, we report two complementary sparsity measures:
Relative Sparsity. Let Nastive denote the number of singular directions having greater mask value then the threshold(1 x 10-2) after mask optimization, and Nicamable the total number of directions subject to training. Then the relative sparsity is
Naetive
Sui =
Nicamable
This measures sparsity within the Iearnable subset only.
32

---

Full Sparsity. Considering the complete model, let Nota be the total number of singular directions
available across all OV projections. The full sparsity is defined
Ractive
Still
Nuaa
This reflects overall model compression after both truncation and pruning.
B.7
Discussion and Future Directions
Our proposed decomposition- based framework presents scalable, model-agnostic approach for discovering interpretable subspaces within pretrained transformers. By applying singular value decomposition to attention weight matrices, identify low-rank directions that robustly encode functional roles, such as temporal endpoint detection or entity repetition suppression. This enables fine-grate fine-grained dissection of emergent behaviors, such as the "Name Mover" or "Greater Than" circuits, and reveals that these are implemented through sparse combinations of reusable subspaces rather than monolithic structures. Notably, we observe recurring components across tasks, such as start- of-sequence detectors and temporal amplifiers, suggesting that transformer models like GPT-2 leverage a compact of primitive functions in compositionally rich ways. Our results align with and extend prior mechanistic interpretability research, complementing methods such as contextual decomposition [Hsu al., 2025], attribution patching [Nanda al., 2023], and circuit overlap metries [Hanna et al., 2024], while providing new axis of interpretability rooted in linear decomposability. The approach computationally efficient, all analyses were performed on a single NVIDIA A40 GPU, and it reveals that many key behaviors are localized to small number of interpretable singular directions. This raises intriguing possibilities for circuit editing, model compression, or steering via low-rank intervention. However, our results are not exhaustive; rather, they provide first step toward uncovering modular internal mechanisms through decomposition. We emphasize that this work identifies functionalities that are interpretable under this perspective, but there is much more to uncover. Future directions include conducting targeted intervention experiments to validate causal contributions of discovered components, examining whether these mechanisms generalize to larger or more capable models (c.g., Phi, LLaMA), and exploring their activation in real-worke settings. Additionally, the search for composite patterns, i.e. higher-order patterns composed of recurring singular directions, may yield deeper insight into how transformers orchestrate symbolic reasoning.
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide evidence for claims in Section 2, Section 3, Section 4, and Section
5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations sub-section included in Section 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
34

---

Justification:
We provide the derivation in Section 2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide details in Section 4, Section 5, and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
35

---

Answer: [Yes]
Justification:
We will release model and experiment code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm wweiiisssoooooiioio
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. pttps.cc//ulle cc/publ
Seuuuddes/Cooeuuiiiii wwriiisssooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are provided Section 4, Section 5, and Appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Details are provided in Section 4 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
36

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are provided in Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httts://heerrrs..cc///li/ittttt
Answer: [Yes]
Justification:
We have reviewed code of ethics and have abided by them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: In this work, we do not develop any new model or technology but perform analysis of existing pre-trained models. To the best of our knowledge there no negative societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
37

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
We are not creating any new datasets or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We attribute the creators of various models in various sections of the paper and in the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper thcode. com/ com/dataser datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
38

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We creating any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not perform any human experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not perform any human experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
39

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: This paper is about analysis of inner- inner- workings of LLMs and we have described their usage in detail in several sections of the paper.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ htt.rrpp....c/nffeeeeee///////// for what should or should not be described.
40


Reviews:
[{"review_id": "ud8pAaco3l", "reviewer": "Reviewer_vMFZ", "summary": "Interpreting model components that support particular behaviours is an interesting direction of research, and this work adds to this effort by hypothesizing that distinct singular directions in model's weight matrices correspond to distinct functions. To this end, they present a framework that optimizes a mask over the singular values of augmented weight matrices to identify the components that support model behaviour in a specific task. This optimization framework, that results from optimizing an L1-penalized KL loss, yields sparse masks that are used to assign functional relevance to model's components, e.g. individual heads. Together, this study demonstrates promising and exciting results that could help shape the field of mechanistic interpretability by offering an alternate perspective of singular vector directions (or eigendirections) being atomic units of computation in a model.", "questions": "1. Can the optimization objective be seen as a form of model pruning (for a specific task)?\n2. What is the effect of the optimization process on the model's task performance? What is the effect of $\\lambda$ on the task performance? I assume it is a hyperparameter, and its choice affects the final results?\n3. Is the optimization performed jointly for all layers and all matrices in the model, or is it one mask at a time? This was not clear from the text.\n4. Can Fig 1 be made more concise? It seems more like a supplementary figure now (with Fig 2 conveying the same key message). Also, can you order the mask values to be in increasing or decreasing order, because the matrices are permutation invariant and the singular vectors can be permuted without affecting the overall matrix, right?\n5. Same question as above about reordering in Fig 2, but for the heads within a single layer. I believe this reordering might help put the core point across to the reader more easily.", "limitations": "Yes, added in checklist.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is generally well-written and the core idea is easy to follow. The hypothesis is clearly described and the experimental setup is well designed.\n2. The authors do a good job in connecting their idea and results to the literature, which helps situate the reader well in the field.\n3. The results are interesting and definitely present a strong case for the framework being widely shared with researchers in the field of mechanistic interpretability and the larger ML community. I specifically like the results presented in Section 5, because it helps argue against the prevalent view of monolithic functionality of each head. \n\nWeaknesses:\n1. While the current results are interesting, they do not completely support the authors' initial hypothesis. To truly support their claim, the results need to be augmented with a loss-of-function study (ablation) and/or gain-of-function study. Specifically, it is not clear from the current study what the effect of applying the mask is on the model's task performance. Is it roughly preserved? What happens if the mask is inverted, i.e. the identified components are removed/ablated? Does it lead to the model's performance being similar to a random baseline? The current results do not answer these questions and therefore, the study falls short of supporting the original claim that the model's function is compositional and be decomposed into individual singular vector directions.\n2. While the authors present results in Fig 1 and 2 demonstrating that previously-identified functional heads are generally associated with higher mask values, it is unclear whether the distribution of mask values for these heads is significantly different from the \"other heads\". A more quantitative justification might help make the point to be clearer. Moreover, I am not sure if this analysis is sufficient to demonstrate the correctness of the proposed method.\n3. While the authors present and mention that the proposed method can be used for the augmented output-value and the MLP matrices, no results are presented from these experiments in the main text. It would be great if the authors could comment on the reason, or defer the corresponding segment from the methods section to the appendix.\n4. The functional description of specific singular vector directions within a head, albeit impressive and interesting, is quite qualitative at the moment and would benefit from a more quantitative exploration. Perhaps, the authors could consider adding an analysis across examples and compare the distribution of activations for \"functionally-relevant\" vs irrelevant tokens, and add statistical tests to demonstrate differences?", "was_revised": true, "final_justification": "The authors presented new evidence that keeping only the learned singular directions enabled preserving the task performance, along with other controls to demonstrate the validity of their approach in identifying functionally relevant computational components. Overall, I am convinced that the authors' proposal argues for a fundamental shift in the mechanistic interpretability literature, something that will be of significance and interest to the NeurIPS community.", "rebuttal": "Thank you for engaging with our work and understanding the key aspects of this work. We are pleased that your summary captures the spirit of what we aim to present, i.e., shift the focus in mechanistic interpretability from components like attention heads to the directions within them, represented by singular vectors, as the real atomic units of function.\n\nWe believe this perspective opens up a different way of seeing what’s happening inside transformers. Rather than assuming that heads or layers are monolithic, our work shows that they are better understood as overlapping bundles of functional subroutines/subfunctions, each of which can be observed as embedded in a low-rank direction. This reframing is the core contribution, and we’re encouraged that you found this idea both clear and interesting.\n\nWe respond to your suggestions and questions point by point as below: \n\n* We agree with your insights regarding providing an ablation study to provide more empirical justification, for understanding the behaviour of the learned masks, that include the behaviour preservation, inversion (i.e., removing these directions that lead to performance that is near random), and comparison that includes observing the learned directions outperforming, both top-magnitude and random SVD baselines [also see table below]. We will try to include these comparisons in the updated version of the paper. However, we would also like to highlight that in the current version, we followed the prior art and took the circuits/components by Wang et al. as ground truth for model components playing a role, and it was interesting to see that similar components show a high mask value when compared to the rest of the model. We believe that since the mask optimization is done using the model behavior preserving metric, it is easier to get the learned directions that retain 96% of exact prediction match and 0.07-0.12 KLD with the full model and 82% accuracy on the IOI task, confirming they’re sufficient for behavior (due to the optimization objective of the task). In this work, the primary goal was to see if the independent directions also play a role in this behaviour preservation, i.e., the directions with high singular values are not the only ones that are important, and there are some task-specific directions that make the subroutine/behaviour possible for a task. \n\n| Mask Type    | Exact Prediction Match(%) | Accuracy(%) | KL Divergence |\n|------------------|-------------------------------------|-------------------|---------------------|\n| random 25   |           3.11                           |    0.00            |       7.81           |\n| random 10   |           0.00                           |    0.00            |       7.56           |\n| top 25          |           7.10                           |   0.00             |       4.6             |\n| top 50          |            38.14                        |  32.37            |       1.99           |\n| learned        |            96.56                        |   82.12           |       0.12           |\n| inverted       |            6.01                          |    0.00            |       4.01           |\n\n\n\n* Figures 1 and 2 demonstrate that previously identified functional heads are generally associated with higher mask values, we agree that the stastical distribution may be matching with some of the other heads (non role playing heads), as also shown in the figure 2, however the primary focus is that there are independent directions (not always with a highest singular values), that play a curtial role in this model behaviour, to validate it in more detail, we provide the entire section 5 and the additional investigations in the appendix about different component's different directions playing a specific role for a specific task. In this work, the primary goal was to look at circuit discovery from a different lens that goes away from taking a module/component as an atomic unit and discover the abstract spaces learned/used by the model to perform a task. \n\n* We are happy for your detailed suggestion regarding comparing the distributions of activations for \"functionally-relevant\" vs irrelevant tokens, and adding statistical tests to demonstrate differences. We believe this would add an additional set of verifications regarding the independent directions. In the current version of the paper, we considered investigating the behaviour of the found directions in a qualitative sense, as done in the prior arts on the mechanistic interpretability literature. We believe a lack of suitable metrics to make this comparison between activations is the primary bottleneck that limits the ability to consider/take a statistical comparison as a strong source of verification. Activation comparison in general is a vast field that is filled with some limitations of comparing the model comparison, and the behaviour preservation, like the faithfulness score (comparison with the original model), is the only metric widely used in the community. We believe the qualitative investigation was more necessary, and it took too much time to help us frame a justification for the current approach of finding abstract components and studying patterns that came with some of the directions. This was helpful for suitable validation that some of the directions are more important than others, and a new way to think about model components is to take these directions as atomic units instead. If you have any other specific metric for the activation comparison that is used in the literature and would make this study more general, we would be happy to try out the same. \n\n\nResponse to Questions/Clarifications:\n\n* Q: Is this like model pruning?\nYes, we are glad that you noticed this in detail, the current way of masking can indeed be seen as a task-specific, low-rank pruning strategy, but one that operates in the directional basis of parameter matrices rather than weights or neurons or any other component. The primary difference is that we’re not just simplifying, we’re studying/exploring the functional geometry (with discovered directions) of how models think/make decisions.\n\n* Q: What is the effect of lambda (L1 sparsity)?\nThe lambda controls the trade-off between sparsity and performance. It is a hyperparameter that can be tuned to control the sparsity. In the current setup, we used annealing lambda with linear increase [0,5e-4] to lead to interpretable masks with minimal task degradation. Intuitively, we can say that too high a lambda can lead to losing coverage; on the other hand, too low values will make the masks lose meaning.\n\n* Q: Is optimization done jointly or separately?\nWe optimize all the masks jointly to get a more coordinated structure. \n\n* Q: “Can Fig. 1 be made more concise?”\nYes, we agree with the suggestion to make Figure 1 more concise. We specifically used Figure 1 (with ordering of singular values, rather than that of mask values), to highlight that the peak mask are different from the singular values, in the SVD decomposition it is natural to assume that the more singular values will play a more important role in matrix reconstruction, however we believe the peaks coming in the middle highlight that some of the directions are more task relevant than the ones with more singular values, showcasing the non monolithic nature of these modules with more detail. As per the suggestion, we can keep another image with sorting heads and singular directions by activation strength. Thanks for this practical and helpful suggestion.\n\n\nWe started this work from a simple question, i.e., what if we’re looking for functional units in the wrong place? Rather than treating entire components as atomic, we asked whether a function lives/occurs in directions, and whether those directions can be disentangled, isolated, and understood. In our initial study, we found that the early evidence suggests yes, and that a lot of what we consider “component function” may be an emergent property of overlapping directional subroutines, which we aim to disentangle in this work. We believe there is a lot to explore when taking/studying model components, and this work may lead to one such foundational way of thinking about model behaviour."}, {"review_id": "TU4kh6wIVX", "reviewer": "Reviewer_76ZH", "summary": "This is a really interesting paper that puts together ideas from singular value decomposition, linearization through augmented inputs, and learnable diagonal masks to present a novel approach to interpretability of transformer circuits.", "questions": "None", "limitations": "Yes, the authors have addressed limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The idea of using singular value decomposition by linearizing attention, MLP and output computation using input augmentation is very interesting.\n2. The use of learnable diagonal masks coupled with freezing of the other model weights is a natural solution to the problem of learning the relevant componends of the singular value decomposition. The insight is well appreciated.\n3. The experimental results on GPT-2 are clear and interesting.\n\nWeaknesses:\n1. There would be great value in looking at these results on larger models like Llama. However, even the current results in the main paper look promising to me.\n2. More examples like a circuit for addition or multiplication would have been valuable too.", "was_revised": true, "final_justification": "1. The rebuttal has been very helpful in understanding why the work focused primarily on GPT2 and I am happy with the author's responses on studying more complex algorithmic/compositional circuits.\n\nI have studied the other reviews and find them all to be insightful. I have assigned more weight to the high-level idea and the core theme of the paper and relatively less weight to the details of the implementation. However, I respect and appreciate the reviews that have correctly focused on the details that will ensure the paper's value to its readers. On the whole, I am inclined to keep my positive review in light of the paper's interesting focus on identifying functional subspaces in the paper despite the correctly-identified shortcomings in the other reviews.", "rebuttal": "Thank you for your thoughtful and encouraging review. We’re very happy that the primary ideas of the paper came through clearly, especially the integration of input augmentation, SVD-based decomposition, and sparse mask learning to isolate functional subspaces inside transformer models.\n\nWe are also pleased with your recognition of the simplicity and coherence of this approach, i.e., freezing weights and learning diagonal masks to identify task-relevant directions. This line of thinking grew out of an inclination to shift the conversation in mechanistic interpretability away from fixed components (like heads and layers) and toward functional directions embedded across the model.\n\n\n* On Scaling to Larger Models (e.g., LLaMA): We agree this is a natural and exciting next step. The framework was intentionally designed to be scalable, learning masks over SVD directions in frozen models is memory-efficient and parallelizable. We used the GPT2 architecture following the current literature in mechanistic interpretability. Moreover, GPT2 is been widely studied and help make the experimentation faster, with an added advantage of ground truth components already discovered by studies like Wang et al. that help validate the working of the presented approach, in the future, it would be interesting to see a combination of such directions playing role in bigger/larger models, making them more interpretable and alignable/steerable for a specific behaviour. More specifically, we chose GPT-2 for this study to keep our focus on interpretability and avoid confounding factors introduced by very large-scale models. Our aim was to first establish the methodological foundation, validate it with the existing literature, and later go towards scaling it for bigger models.\n\n* On Additional Circuits (e.g., arithmetic tasks), this is a great suggestion. We agree that the arithmetic tasks present unique opportunities/ways for discovering compositional or algorithmic circuits; however, they also require careful setup to ensure alignment between directionality and symbolic function. We believe this would be an interesting direction to explore in future works.\n\nWe are happy for your support of the work and for identifying its originality and potential impact. The positive signals from the current results, clear task-relevant directions, behavior preservation under masking, and alignment with known heads are encouraging and strong signs that this direction-based lens could become a useful method in interpretability, presenting a new direction to look at model behaviour."}, {"review_id": "9fZu16r5MC", "reviewer": "Reviewer_BMMX", "summary": "The paper interprets singular vectors of weight matrices in GPT 2, which are responsible for predicting the answer to popular tasks like \"Indirect Object Identification\" and \"Greater than\". They propose a novel masking algorithm to classify the most important directions for completing the task in the SVD bases. This algorithm maximises the KL divergence at the answer token while learning a sparse mask over the top-k singular values for the weight matrices. They found that each single attention head performs many computations in parallel, denoted by each singular value. This could not be observed previously at larger granularities used in popular circuit discovery algorithms like ACDC, thus providing valuable information to further progress our understanding of LLMs.", "questions": "I've listed down most questions and suggestions in the strengths and weaknesses. I'll classify the suggestions in terms of importance and detail them here.\n\nHigh importance:\n- Reporting the performance before and after applying the learned mask on a test set\n- Mentioning the early stopping criteria used on the validation set and the number of epochs after which each run used was stopped at. \n- Reporting sensitivity to hyperparameters for learning the mask. Most importantly, the value of \"r\" used while learning the mask. \n\nDecent importance: \n- Clarifying the sparsity and mean of mask scores for attention heads shown in Figures 1, 2 and 5. Things that could help:\n    - A direct correlation between the causal effect observed through previous circuit discovery work and these metrics could be useful\n    - Interpreting singular values of some of these heads as well, to see if they are doing anything task-specific or do things that are, for example, universally useful. \n    - Looking at the counterfactual importance of singular values, especially the ones labelled as \"other head types\" might help. (Note: This would be ideal, but understandable if not done, as the authors mentioned the causal effects of the directions obtained to be a part of future work.) \n- Adding in the requested related work and reframing the novelty of the paper to be along the lines of extending SVD-based interp to task-specific settings/circuits.\n\nGood to have:\n- Difference between logit difference and KL divergence as the loss function\n- Do different random seeds affect the mask learnt a lot?\n\nI would be happy to change my score if the important parts of my concerns are addressed.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This work extends circuit discovery to a finer level of granularity, without using expensive SAEs. \n- Provides valuable information about what goes on in various components of language models, which were previously not well known. \n\n---\n\nWeaknesses:\n\n- Little to no comparison between other circuit discovery algorithms: \n    - Although the authors claim that since their analysis is at a different level of abstraction, metrics like KL divergence recovered, and the number of steps/forward passes required to obtain a mask/circuit are still relevant. I also expect there to be some similarities with other mask-based approaches that work on neuron level (like subnetwork probing variants), which weren't mentioned in the paper. \n- Unclear claims:\n    - The paper claims to have studied \"Can a small number of learned singular directions faithfully preserve model behavior?\", but I can't seem to find the performance lost/recovered after applying the mask reported anywhere. The authors write, \"Learned masks (...) achieve significantly lower KL divergence than top-k or random SVD baselines\". However, I couldn't find this result reported in the appendix. In case this was missed, I request that this result be provided. \n\n    - What's up with the mask values of other head types being high/not sparse? \n\n       The authors claim that the heads found by Wang et al. have masks with low sparsity and high mean values (high activation across multiple singular dimensions). However, there seem to be quite a few other attention heads that have many important singular vectors according to the learned mask. They have a higher importance (wrt the mask) than some of the heads in Wang et al.'s circuit. Are they important when resample ablated? If so, why do they not show up in Wang et al.? If not, what are they up to?\n \n- Little to no ablation studies and some unclear details:\n    - What is the effect of choosing \"r\" for top-r singular values v/s performance and interpretability of the mask found? \n    - The paper mentioned using early stopping on the validation dataset, but not the exact criteria for it. (i.e., what was the accuracy on the validation set that was assumed to be good enough to stop?)\n    - Is there any difference between using KL divergence v/s logit difference on the answer token?\n\n---\n\nOther points: \n\nRelated Work:\n- Although no one has used the SVD bases to study task-specific circuits, there have been some works studying singular values of weight matrices agnostic to any task (see here: https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight). This does not reduce the novelty of this work. However, the previous work is significantly relevant to be included in the literature review. \n- There has also been some work to decompose attention heads into components doing independent computations using sparse autoencoders (see here: https://arxiv.org/abs/2406.17759)\n\nWriting clarity:\n- line 107 should have \"interpretability\" instead of \"interoperability\"?\n- Worth defining x as the residual stream in section 2 (and other similar variables properly)\n\n---\n\nOverall: Although this work provides valuable and novel insights into the workings of LLMs, there is still some work needed for it to establish this methodology.", "was_revised": true, "final_justification": "I decided to keep my scores the same as the preliminary review. I have noted down exactly what points were not addressed by the rebuttal in my official comments. I add a one-line description for them:\n\n- Clarifying their claims around sparsity and the mean of mask scores for attention heads that seemed unclear to me. \n- Incomplete comparison with previous work.\n\nEdit: Hyperparameter search was clarified, and I have removed this from my justification.", "rebuttal": "Thank you for your thoughtful and constructive feedback. We're glad that the core contribution, bringing interpretability to the level of singular vector directions within weight matrices, stood out clearly. Your comment that this work extends circuit discovery to a finer granularity without using SAEs captured exactly what we hoped to demonstrate. The primary focus is to think in a different direction (level of granularity) for finding/abstracting out functional components playing a specific role in a specific task. \n\nWe are particularly pleased that you resonated with the main idea of reframing interpretability from a component-level view to one grounded in singular vector directions. Our motivation was not to simply replicate known circuits through another lens, but to challenge how we define atomic units of function inside LLMs. Rather than focusing exclusively on causal faithfulness or reconstructing known pathways, we explored towards understanding what abstract predictive directions exist in model weights, even if not strictly necessary in a minimal circuit sense.\n\nWe are happy for your careful reading and your detailed points. We respond to the major points/clarifications below. \n\n\n* Comparison with Other Circuit Discovery Algorithms\nWe agree that the paper could more clearly position/fit itself in relation to prior circuit discovery methods. However, we would like to emphasize that [also discussed in Section 5.1 (“Comparison with Existing Circuit Discovery Methods”)], our work proposes a fundamentally different level of abstraction, i.e., instead of treating entire components (e.g., heads, MLPs) as atomic units, we decompose these into granular orthogonal directions via SVD. This perspective does not completely align with previous work, such as ACDC or SAE-based methods, which focus on edge-level causal paths between predefined components. The proposed method instead focuses on uncovering predictive structure within components, revealing overlapping and distributed subfunctions that standard tracing cannot detect. We also agree that neuron-level mask-based approaches (e.g., subnetwork probing) may share similar optimization objectives. However, those typically treat neurons as the smallest unit, while we show that singular directions inside each weight matrix provide a finer and often more meaningful granularity (as reported roleplaying directions in Section 5 of the paper). We thank you for this insight, and we will revise the Related Work section to better reflect this distinction and reference relevant efforts (e.g., sparse subnetworks and Merullo et al. [2024b]). If you have any other works, we would be happy to include those as well.\n\n* Faithfulness of Masked Directions\nThank you for highlighting this. We agree that explicitly reporting performance retention after masking would be essential to support our claim about preserving model behavior. In our experiments, we did learn the masks with early stopping based on KL divergence convergence to the unmasked model (on held-out validation set as done in prior arts like ACDC, EP, and EAP), so functional preservation was expected. However, to make this more straightforward in the updated version, we will report the KL divergence and task performance (e.g., top-1 accuracy) after applying the learned masks. These results show that learned masks consistently outperform top-k and random SVD baselines in preserving behavior, supporting the idea that our method selects functionally meaningful subspaces. Section 5 is intentionally qualitative. Rather than recovering known causal circuits (that can be obtained via any faithfulness-preserving optimization objective), we explored whether individual singular directions encode interpretable roles, suggesting that attention heads are not monolithic but contain overlapping subroutines. The examples were chosen to support this hypothesis through detailed case studies, not an exhaustive search (so we may have missed a few roles). We agree that a more systematic faithfulness is needed, but that is still an open research problem in the interpretability domain, which remains future work.\n\n* High Mask Scores in Non-Wang Heads\nWe thank you for this insightful observation, and we agree it deserves more discussion. In general, we found that some heads not part of the Wang et al. circuit still receive high mask scores, and our current interpretation/speculation is that these heads may encode redundant or auxiliary predictive routines not captured by causal tracing (since Wang et al do not specifically use KLD). They could reflect task-agnostic computations (e.g., syntax, position handling) or serve as fallbacks that are predictively useful but not causally necessary. We plan to include resample ablation experiments in the revision to understand their role better. Preliminary results suggest that masking these directions does lead to measurable degradation (better if we can show that using some experiments), supporting their functional relevance, even if they were not selected by prior tracing work.\n\n* Effect of Choosing \"r\" for Top-r Singular Values\nWe selected r = 64 for QK and r = 65 for OV based on the effective rank after augmentation (i.e., number of non-zero singular values). This was a pragmatic/natural choice to balance computational efficiency and expressiveness. While it worked well in our experiments, we agree that sweeping r or optimizing over the full orthogonal basis (specifically for MLP layers) could yield richer insights, possibly uncovering weak but interpretable directions. We will clarify this design decision and suggest full-rank optimization as an avenue for future work.\n\n* Early Stopping Details\nOur early stopping strategy was based on KL divergence convergence on a held-out validation set (like done in works like ACDC, EAP, and EP, we specifically use EP's held-out validation set), the training was stopped when L1 norm of the mask and kl divergence were saturated (KL value was in range 0.07-0.12 at the point of saturation, depending on the penalty on L1). We agree that a hybrid strategy combining KLD and sparsity metrics could be more principled and propose it as future work. This also balances out the reward hacking (extra amount of pruning to preserve task-specific behaviour), and keeps the optimization joint.\n\n* KL Divergence vs. Logit Difference\nWe chose KL divergence as our loss function because it compares entire output distributions, forcing the masked model to match the original model’s behavior globally, not just on the answer token. In general, we observed that the KLD is compromised when logit difference is used as a loss objective; hence, to maintain the model behavior intact/close as possible, we prefer using KLD as a loss function. We thank you for pointing this out. We will add a small discussion with some preliminary results (if possible) to add an extra layer of transparency for future work. \n\n* Additional Related Works\nWe thank you for pointing these out. We agree that the mentioned previous works are significantly relevant to be included in the literature review.\n\n\nResponse to the Questions:\n\n* Reporting the performance before and after applying the learned mask on a test set. \nWe agree that this is essential for evaluating the effectiveness of the learned mask. We have now added results (showing the KL divergence and task accuracy on the test set, i.e., KL divergence was 0.07-0.12 (IOI) (masked vs. unmasked model),and the learned masks recover 96%(exact prediction match for IOI) of the original behavior (better than top-k or random masks). Also, for the Task accuracy, the performance drops are minimal (masked model has 82% accuracy, full model has 84% accuracy on IOI task), supporting the faithfulness of the learned subspace. These results confirm that the selected singular directions preserve core model behavior and are not only interpretable but also functional.\n\n* Early stopping criteria and number of epochs\nWe used early stopping based on KL divergence saturation on a held-out validation set (similar to the one used in EP), i.e., the mask optimization terminated when KLD stopped decreasing and plateaued near the baseline model. \n\n* Sensitivity to hyperparameters, especially the value of \"r\", we selected r = 64 for QK and r = 65 for OV, based on the effective rank after augmentation. Although we did not perform a wide hyperparameter sweep (specifically for MLP). We will include a discussion on this and propose full-rank optimization as promising future work.\n\n\n* We would like to higlight that the L1 values that we obtain is around 700, which is around 2k for the wang et al circuit components, that would be one way of comparison. We believe that the current perspective is quite different from what is done in the past, the recent work with MIB benchmark [Mueller et al 2025], also lays down a generic measure of sparcity going to the dimension level (also referred as neuron level measure, refer to section 3.1 \"Measuring circuit size.\" subsection)\n\n* Regarding the universally useful directions, we agree that some of the roles that we discover play a similar role in different tasks when investigated manually, some of the ones that are reported in the paper/appendix include Figures 6 and 7 [also see description for these roles]. \n\n* We agree that it would be interesting, looking at the counterfactual importance of singular values, especially the ones labelled as \"other head types,\". We are currently running these experiments and will report some of the initial findings in the paper if possible. \n\n* We would be happy to add the requested related work and reframing the novelty of the paper to be along the lines of extending SVD-based interp to task-specific settings/circuits.\n* The random masks were initialized using a normal distribution, and multiple experiments with different seeds led to similar mask distributions learned during the optimization process."}, {"review_id": "BJ8wtCx48c", "reviewer": "Reviewer_CfUp", "summary": "In this paper, the authors propose a framework for analyzing a large language model by applying SVD to its weight matrices, including attention heads (QK and OV matrices) and MLPs.\n\nThe central idea is a masking procedure they developed. This technique allows them to amplify or diminish the impact of specific singular vectors. By doing this, they can identify which SVD directions are most critical for the model's overall performance. This masking is learned using a loss that tries to preserve the model's final probability distribution while encouraging the sparsity of the mask.\n\nThey focus their analysis on the QK matrix of attention head 9.6. Their results suggest that certain singular vector directions in this matrix correlate with high-level semantic features that the attention head may be performing.", "questions": "# Question 1\n\nI did not understand exactly how the scores are given. What is \"mean activation\" in this context? In my understanding, using Section 2 of the paper, we write an attention score of a token $i$ (destination token) attending to another token $j$ (source token) as:\n\n$$\nA_{ij} = \\begin{bmatrix} 1 & \\mathbf{x}_i^{\\top} \\end{bmatrix}  \\\\mathbf{W}\\_{\\mathrm{aug}}^{(QK)} \\begin{bmatrix} 1 \\\\\\ \\mathbf{x}_j \\end{bmatrix}\n$$\n\nSo, the $S_7$ contribution to the attention score (let's call it $A_{ij, 7}$) can be written as:\n\n$$\nA_{ij, 7} = \\begin{bmatrix} 1 & \\mathbf{x}_i^{\\top} \\end{bmatrix} \\left( \\mathbf{u}_7 \\sigma_7 \\mathbf{v}_7^{\\top} \\right)  \\begin{bmatrix} 1 \\\\\\ \\mathbf{x}_j \\end{bmatrix}\n$$\n\nWhen you say that $S_7$ direction assigns high values to \"Jerry\", does it mean a high $A_{ij, 7}$? If this is the case, do you evaluate \"Jerry\" as a source token only, with the end token as a destination?\n\n# Question 2\nAre the masks binary or values between 0 and 1? Using $[0, 1]^r$ is a bit ambiguous. By looking at Figure 1, there are multiple values between 0 and 1, but I was thinking that the mask is binary. \n\n# Question 3\nThe paper says the following: \"We retain the top-$r$ singular directions (typically $r$ = 64)\". The rank of the QK matrix is 64, which makes sense. But is this $r=64$ used for the other matrices as well? Or do you use $r$ as the rank of the matrix that you are taking the SVD?\n\n# Suggestions\nMy suggestions to improve the work are described in the Weaknesses section.", "limitations": "Yes.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\n- Using a new approach based on masks to learn the singular vector directions that are important for the model's performance.\n- Showing that there is a correlation between some of the singular vector directions and features that a name mover attention head may be using. \n\n# Weaknesses\n\n- The paper does not mention previous work in this area:\n    - [1]: This paper shows that the SVD basis can be very useful in interpreting the QK interaction in vision transformers. They perform an in-depth analysis of the singular vectors of multiple vision transformer models, showing numerous examples of the semantics of the singular modes (the rank-1 matrices given by the SVD).\n    - [2]: This paper proposed the sparse attention decomposition, which states that the attention scores can be low-rank reconstructed using the SVD of the QK matrix. They show multiple examples of sparsity in the attention scores reconstruction. By using this decomposition, they identify the components that are responsible for the attention head to attend to a token pair. Then, they trace an IOI circuit using this approach and validate it by performing multiple causal interventions.\n    - [3]: This is a blog post that shows that the singular vectors of the OV and MLP matrices have highly interpretable semantic directions, and uses the singular vectors to change the model behavior. \n- There is no comparison with baselines. [2] shows how to find a sparse set of singular vectors to reconstruct the attention score. Comparing the mask sparsity with this method would give an idea about how good the level of sparsity of the masks is. Even though they are not solving the exact same problem, they are very similar.\n- The evaluation part could be more systematic rather than observational. Many of the conclusions are simply interpretations without justification (Section 5). It is very unclear whether the conclusions made in Section 5 could be extended, and how, to other tasks and models. For instance, in Figure 1, attention head 9.6 has around 10 singular vectors with very high mask values (close to 1), but only 3 were analyzed in the paper. One way that this could be done is by performing a causal analysis of the directions analyzed, similarly to [2, 4].\n- A fair amount of Sections 2 and 3 are about the OV and MLP matrices. However, there is no analysis of the SVDs of these components. Is there any specific reason for that? What is the application of learning masks for these matrices?\n- There is no mention of how good the masked model is in comparison to the original model: no mention of the loss value (I also did not find it in the Appendix). So, how faithful are these masks to the model performance?\n- This approach does not cover more modern models that use RoPE, which makes it harder to analyze the QK interactions, especially using a learning-based approach. Do you have any ideas about how your methodology could be applied in RoPE models?\n\nReferences: \n- [1]: Pan, Xu, et al. \"Dissecting query-key interaction in vision transformers.\" arXiv preprint arXiv:2405.14880 (2024).\n- [2]: Franco, Gabriel, and Mark Crovella. \"Sparse Attention Decomposition Applied to Circuit Tracing.\" arXiv preprint arXiv:2410.00340 (2024).\n- [3]: Millidge, Beren, and Sid Black. \"The singular value decompositions of transformer weight matrices are highly interpretable.\" AI Alignment Forum. 2022. (https://www.alignmentforum.org/posts/mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight).\n- [4]: Merullo, Jack, Carsten Eickhoff, and Ellie Pavlick. \"Talking heads: Understanding inter-layer communication in transformer language models.\" Advances in Neural Information Processing Systems 37 (2024): 61372-61418.", "was_revised": true, "final_justification": "I believe this paper is built on a strong and interesting idea. Using SVD to interpret model behavior at a sub-component level has clear potential advantages over other methods in circuits research: it is more granular, does not require a replacement model, and is more faithful by design.\n\nHowever, as it stands, the experimental validation does not yet match the strength of the core idea. My primary concerns, which are also shared by Reviewer vMFZ (on quantitative analysis) and Reviewer BMMX (on comparisons to related work), remain. Addressing these two points would require significant new experiments that go beyond the scope of a rebuttal period and would likely lead to substantial changes throughout the paper.\n\nFor these reasons, I will be maintaining my current score. I hope this feedback is helpful for strengthening the paper in the future. With more robust experimental support, I believe this line of work could be a valuable contribution.", "rebuttal": "Thank you for taking the time to write a detailed and broad review. We are pleased with your broader perspective, mentioning additional references (going beyond circuit discovery literature) and suggestions to clarify and better articulate what this work is trying to do, and what it is not.\n\nAt its core, the goal of this paper is not to rediscover existing circuits, nor to benchmark a new variant of an existing circuit discovery pipeline. Instead, we’re trying to take a step back and ask, What if the atomic units we analyze, like neurons or heads, are too coarse to fully describe the computations these models perform? Singular vector directions in weight matrices provide a finer-grained, orthogonal basis that isn't constrained by component boundaries. This paper tries to make a case for interpreting behavior in that subspace.\n\n\nWe address the clarifications/suggestions below.\n\n* Suggested Related Works\nWe agree with your suggestions; we should have cited the works you mentioned. We thank you for providing a broader scope to this work by including works in vision, feature reconstruction, and visualization. We hope you understand that the omission of these works was an oversight, not an attempt to overstate novelty. The novelty lies in shifting how SVD directions are used, i.e., not just for reconstruction or feature visualization, but as functional units optimized to preserve model behavior. The novelty lies in shifting how SVD directions are used, not just for reconstruction or feature visualization, but as functional units optimized to preserve model behavior, which was missing from the prior art. We will be including all of them in the updated version of the paper. \n\n* We agree that Sparse Attention Decomposition (Franco & Crovella, 2024) is somewhat related to our work, and in hindsight, we should have framed the relationship more clearly. That said, the goal in [2] is to identify a sparse subset of singular directions that reconstruct the attention scores themselves. Our objective is quite different; we’re optimizing for preservation of end-to-end model behavior (as done in circuit discovery methods), not intermediate reconstruction (done in SAE-type methods). This distinction matters. The directions they find reconstruct attention scores well, but aren’t necessarily predictive in the downstream sense. In contrast, the directions we find preserve the full output distribution (measured via KL divergence) and often correlate with interpretable, task-relevant roles. It is important to note that this work's primary goal is to look at things differently and see if abstract circuits exist, rather than using components as atomic units; we use these directions that were found to play specific roles in the tasks. \n\n* We would like to emphasize that Section 5 is intentionally qualitative. Rather than recovering known causal circuits (that can be obtained via any faithfulness-preserving optimization objective), we explored whether individual singular directions encode interpretable roles, suggesting that attention heads are not monolithic but contain overlapping subroutines. The examples were chosen to support this hypothesis through detailed case studies, not an exhaustive search (so we may have missed a few roles). We agree that a more systematic faithfulness is needed, but that is still an open research problem in the interpretability domain, which remains future work. In our analysis, we included the patterns that were explainable; we explored the other directions as well, but could not generalize the findings. Also, it takes much more time to analyze each direction separately.  Overall, what we’ve consistently found is that directions tend to be overlapping and distributed, multiple directions in the same head contribute to distinct, sometimes competing subfunctions. This is precisely the reason we moved away from component-level analysis; the head as a whole doesn’t do “one thing”; it houses several subroutines. The SVD basis lets us isolate these roles. The key idea we want to put here is to understand not just \"which head is active?\", but \"what kinds of structure live inside a single head?\", formulating a different level of abstraction/atomic units of role-playing components. \nSections 2 and 3 set up the decomposition for all major matrices (QK, OV, MLP), but we focused our case studies (that required some sort of manual inspection, which is time-consuming) and visualizations on QK matrices, since the QK matrices showed the highest sparsity in learned masks, making them easier to interpret direction-wise in manual case study/inspection. Moreover, the semantics of QK directions (via dot products) lend themselves to clearer qualitative interpretation as reported in the paper. On the other hand, OV and MLP components tend to exhibit denser, more distributed masks, complicating direction-wise analysis. This could be another direction to explore in future work. Regarding the faithfulness, the masks learned over these matrices are still predictive and preserve model behavior (as confirmed via KLD), but are less sparse and harder to dissect cleanly. This observation aligns with the idea that different components in transformers express different forms of computation, some more modular and interpretable than others.\n\n* Thank you for pointing this out; it’s an oversight that we didn’t report this clearly in the main paper, since the objective that we minimize in the algorithm is expected to learn a mask that preserves model behaviour. Hence, the main focus was to check if these directions that came out are actually interpretable, and whether this could be a new direction/atomic units to be used in the circuit discovery literature.  We’ve now added these results, including KL divergence between masked and unmasked model outputs typically falls within 0.07-0.12 of the original, depending on the component, along with 96% of exact prediction match and 82% accuracy (full model has 84% accuracy) on the IOI task.  Also, the learned masks outperform both top-k singular direction and random masks, which degrade performance more significantly. All of these metrics validate that the learned subspaces not only align with interpretable roles but also retain the functional essence of the model. We'll report these results more prominently in the revised version.\n \n\n\n\n| Mask Type    | Exact Prediction Match(%) | Accuracy(%) | KL Divergence |\n|------------------|-------------------------------------|-------------------|---------------------|\n| random 25   |           3.11                           |    0.00            |       7.81           |\n| random 10   |           0.00                           |    0.00            |       7.56           |\n| top 25          |           7.10                           |   0.00             |       4.6             |\n| top 50          |            38.14                        |  32.37            |       1.99           |\n| learned        |            96.56                        |   82.12           |       0.12           |\n| inverted       |            6.01                          |    0.00            |       4.01           |\n\n\n* On RoPE Models and Generalization\nWe agree that applying this framework to RoPE-based models presents challenges. Unlike fixed positional embeddings, RoPE introduces position-dependent rotations, meaning the dot product between Q and K is not directly interpretable as a simple inner product on a static SVD basis. This introduces two issues: 1) Each token pair effectively uses a different QK interaction subspace, making a single SVD-based decomposition less meaningful. 2) Interpreting SVD directions in this dynamic context requires either reversing or compensating for the positional rotation or projecting into a unified subspace post-attention. That said, this challenge is also an opportunity. The fact that multiple position-dependent QK subspaces exist could enable richer decompositions, and perhaps even multiple, composable attention circuits within the same head. We believe adapting SVD decomposition to the RoPE regime is a natural next step, and could be explored in future work. At this point, we chose to focus on models without RoPE to first test the viability of the idea that singular directions themselves can act as interpretable subroutines. We’ll now add this limitation explicitly and outline how the method might extend to RoPE-based models.\n\n\n\nResponse to the Questions:\n\n* When we refer to mean activation, we mean the average activation of a given singular direction over the entire dataset. This averaging helps identify directions that are consistently active and not just responding to spurious examples or rare token contexts. Regarding the formulation, you're absolutely right in how you wrote the formulation. \n\n* The learned masks are continuous values between 0 and 1, not binary. We use a sparsity-inducing regularization (via an L1 penalty), which encourages many values to go to zero, but doesn’t hard-threshold them. We intentionally chose this softer form of masking to 1 avoid hard zeroing that may disrupt gradients during learning. 2) Allow finer control over relative importance between directions. \n\n* Yes, we use the full-rank decomposition for all components, including QK, OV, and MLP layers. For QK matrices, the effective rank is often 64, so r = 64 is both natural and complete. For OV and MLP matrices, the rank can differ (observed to be 65 for OV after the augmentation matrix formulation), and we retain all non-zero singular directions based on the decomposition (i.e., no truncation). So even though we say “top-r”, in practice we don’t discard any singular directions unless we explicitly test lower-rank baselines (e.g., top-k SVD vs. learned mask). We thank you for noticing this, as it may come out in the other way to the readers, we will make it clearer in the updated version of the paper."}]

Author rebuttal:

Meta-review: This paper proposes a direction-level view of transformer computation. Instead of treating heads or MLPs as monolithic units, the authors linearize QK/OV/MLP via augmented matrices, take SVDs, and learn sparse, continuous masks over singular directions to preserve the model's output distribution on task prompts. The central empirical claim is that a small set of learned directions can retain behavior nearly as well as the full model, while also revealing that single heads implement multiple subfunctions in parallel (illustrated most deeply for the IOI name-mover head).

On my read, and reflected in the discussion, the strengths are (i) a clear, well-motivated reframing that moves beyond component-level circuits, (ii) a simple, transparent procedure (freeze weights and optimize diagonal masks against KL to the unmasked model with an L1 penalty) that others can reproduce, and (iii) evidence that the learned subspaces are not just cosmetic.  The rebuttal added concrete numbers showing that learned masks preserve IOI behavior closely outperforming random or top-k SVD baselines by a wide margin, and that inverting the mask collapses performance. Two reviewers highlighted that this lens surfaces multi-functionality within heads and may help shift how we phrase questions in mechanistic interpretability.

The weaknesses are also clear. As some of the reviewers point out, the scope is narrow (GPT-2, IOI plus a small set of tasks), much of the semantic analysis is qualitative, and several comparisons that would strengthen the case remain absent, for instance  head-to-head against SVD-based sparse attention reconstruction, against DAS / subnetwork-masking approaches, or quantitative significance tests showing that heads deemed functional by prior circuit work have distinctly higher mask values than others. Even after the rebuttal and discussions, questions remain about sensitivity (such as mask sparsity trade-off), treatment of OV/MLP beyond QK, and generalization to RoPE models. Finally, while the method is behavior-faithful by construction, causal validation is left for future work.

However, despite these gaps, the rebuttal and discussion added much needed evidence to the authors' claims. Specifically, the table added in rebuttal directly addresses the most material concern raised across reviews, which is "do the learned directions actually preserve behavior better than simple SVD heuristics?". The answer appears to be yes by a large margin. The authors also clarified the confusing top-r terminology and committed to including OV/MLP visualizations and statistics, and acknowledged the need to add significance tests contrasting functional vs other heads. However, two reviewers maintained their negative stance even after the rebuttal primarily because comparisons to closely related SVD and rotation-based baselines and more systematic quantification are still thin. I agree those additions would elevate the work, but in my judgment the paper, especially after the rebuttal, provides a novel, useful, simple, and reproducible viewpoint with sufficient quantitative evidence of behavioral faithfulness.
Decision: Accept (poster)