Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
Authors: Zhengfu He
Junxuan Wang
Rui Lin
Xuyang Ge
Wentao Shu
Qiong Tang
Junping Zhang
Xipeng Qiu
Venue: ICLR 2026 Poster (2026)
Keywords: Mechanistic Interpretability
Attention Superposition
Sparse Dictionary Learning
Circuit Analysis
Primary area: interpretability and explainable AI
TL;DR: We propose a method to interpret transformer attention blocks by decomposing them into interpretable units.

Abstract: We propose Low-Rank Sparse Attention (Lorsa), a sparse replacement model
of Transformer attention layers to disentangle original Multi Head Self Attention (MHSA) into individually comprehensible components.
Lorsa is designed to address the challenge of \textit{attention superposition} to understand attention-mediated interaction between
features in different token positions. Lorsa helps find cleaner and finer-grained versions of previously discovered MHSA behaviors like induction
heads, successor heads, attention sink, and a comprehensive family of arithmetic-specific Lorsa heads. Interestingly, we identify a novel
head type called \emph{subtoken induction heads} that function at character level rather than token level.
Automated interpretability analysis indicates
that Lorsa achieves parity with SAE in interpretability while Lorsa exhibits superior circuit discovery properties.
We also conduct extensive experiments on architectural design ablation, correlation to original MHSA heads and error analysis.
Our early attempt to fully sparsify a toy Transformer succeeds to reveal clean global circuits. Eventually, we hope Lorsa would help
us greatly understand attention computation and enable full sparsification of model computation along with its MLP counterparts.
Lorsa is open-sourced at https://anonymous.4open.science/r/Lorsa-5686/.

Published as a conference paper at ICLR 2026
TOWARDS UNDERSTANDING THE NATURE OF ATTEN- TION WITH LOW-RANK SPARSE DECOMPOSITION
Zhengfu Hel
Junxuan Wang! Rui Lin
Xuyang Ge Xipeng Q
Wentao Shu 1.2
Qiong Tang* Junping Zhang²
Shanghai Innovation Institute *OpenMOSS Team, School of Computer Science, Fudan University
z 2 1 90 fudan. edu. cn
Multi Head Self Attention (MHSA)
Rank Sparse Attention (Lorsa)
Superposition
MHSA Heads
Sparsify
leartive
Emror
808
To
Lorsa Head 1
Lorsa
MLP
Transcader
1 0
Sparsify
MHSA
Lorsa
sparse replacement model for MHSA designed to
Lorsa
Attack
Attention
Figure 1: (A) Low-Rank Sparse Attention (Lorsa) comprises thousands of sparsely activated attention heads with ID outputs, designed to extract interpretable attention units from the original Multi Head Self Attention (MHSA). (B) Lorsa serves replacement model for Transformer attention, substituting sparse interpretable components for attention modules. (C) Each Lorsa head explains an atomic feature-feature interaction across token positions, which was originally part of an MHSA head or spread across multiple heads, i.e. put in attention superposition.
ABSTRACT
We propose Low-Rank Sparse Attention (Lorsa), sparse replacement model of Transformer attention layers to disentangle original Multi Head Self Attention (MHSA) into individually comprehensible components. Lorsa designed to ad- dress the challenge of attention superposition to understand attention- attention-mediated mediated interaction between features different token positions. Lorsa helps find cleaner and finer-grained versions of previously discovered MHSA behaviors like induction heads, successor heads, attention sink, and a comprehensive family of arithmetic- specific Lorsa heads. Interestingly, we identify a novel head type called subtoken induction heads that function at character level rather than token level. Auto- mated interpretability analysis indicates that Lorsa achieves parity with SAE in interpretability while Lorsa exhibits superior circuit discovery properties. We also conduct extensive experiments original MHSA heads and error analysis. Our early attempt to fully sparsify a toy Transformer succeeds to reveal clean global circuits. Eventually, we hope Lorsa
architectural design ablation, correlation
Equual Contribution. Corresponding Author.

---

Published as a conference paper at ICLR 2026
would help greatly understand attention computation and enable full sparsifica- tion of model computation along with its MLP counterparts. Lorsa is open-sourced at hhttps:///gthh/ com/ Owtp/OseeKK/////// Language-
INTRODUCTION
When examining the function of individual attention heads in a Transformer model, one might identify some of these heads implementing specific behavior. A canonical example is induction heads which predicts 'Potter' following the token 'Harry" when 'Harry Potter' is present in the context (Olsson et al. 2022). Ablating these heads substantially prevents the model from correctly performing corresponding tasks, which indicates causal relation of these heads and the model's s macroscopic behaviors. These interpretable attention units constitute the basic building blocks of the model's 's inter-token information mixing algorithm.
Not all attention heads, however, exhibit clear functionality. Most heads distribute attention across diverse contexts. Although some heads exhibit identifiable patterns, there might be inter-head collaboration that explains the whole story. These challenges in attention head interpretation is analogous to feature superposition in understanding individual neurons, which suggests the existence of attention superposition (Jermyn al., 2024) in Multi Head Self Attention (MHSA), which we will further discuss in Section
Inspired by the recent success of Sparse Autoencoders (SAEs) to extract monosemantic features from Transformers' hidden space (Templeton et al. 2024b) or approximate part of the network's computation as sparse computation (Templeton et al.j 2024a Ge et al.] 2024; Dunefsky al.,
2024), we propose Low-Rank S Sparse Attention (Lorsa) to disentangle the atomic attention units from attention superposition (Section B). Lorsa serves as a replacement module of the original MHSA with
an overcomplete set of attention heads featuring single-dimensional OV circuit (Elhage al. 2021
and sparsity constraints.
We evaluate the reconstruction fidelity and sparsity trade-off of Lorsa in Section 4 along with scalability analysis. In Section [ we introduce our exploration interface following Bricken et al. (2023), providing multifaceted information on each Lorsa head. We also quantitatively assess Lorsa head interpretability using top activations and their attribution patterns (= pattern) with automated interpretability (Bills al. 2023). The results indicate that Lorsa's monosemanticity comparable to SAE features.
Section 6 presents findings with Lorsa on Pythia-160M (Biderman et al. 2023 and Llama-3.1- 8B (Dubey et al. 2024). For validation, we first identify the Lorsa instantiations of known attention mechanisms: induction heads, name mover heads (Wang et al.) (023). successor heads (Gould et al.) 2024). and attention sinks (Xiao et al. 2024). Furthermore, we characterize a family of arithmetic- specific Lorsa heads in Llama-3. 1-8B. We also identify a subset Lorsa heads in 1lama-3.1-88 that function as theme anchors by exhibiting long-range, topic-specific attention patterns.
To the best of our knowledge, Lorsa the first attempt to extract sparse interpretable attentional computation. We hope these findings facilitate future research (see Section 9 for limitations).
Note on Terminology: While prior work refers to the atomic computational units we aim to independently understand as attentional features (Jermyn et al. 2024, Ameisen et al.) 2025). we adopt attention wnits to avoid conflating with activation-space features (which denote ID linear features in representation spaces (Elhage et al. 2022)). The term head denotes MHSA or Lorsa heads as context dictates. We recommend viewing Lorsa as an interpretability tool rather than drop-in replacement, since overparameteriizatoon may compound reconstruction errors.
ATTENTION SUPERPOSITION
Analogous how ReLU neurons in Transformer MLPs learn to represent more features than they have dimensions Elhage et al., 2022). a similar phenomenon may occur in Multi-HE Muad Self Attention (MHSA). We hypothesize MHSA may comprise multiple attention units in attention superposition, each attending between certain token pairs with interpretable read/write operations on the residual stream. Under this hypothesis, would expect (1) atomic attention unit is spread
2

---

Published as a conference paper at ICLR 2026
across multiple MHSA heads. (2) One MHSA head includes multiple units. We list three points of evidence of attention superposition in Transformer language models.
1. A Few Neurons (Heads) Are Polysemantic. Gurnee et al. (2023) discovered compound word neurons activating across diverse unrelated n-grams, while Bricken et al. (2023) reported neurons responding mixed stimuli including academic citations and Korean text. (link). Similarly, successor heads (Gould et al.) 2024) which increment 'Monday' into "Tuesday' and 'I' into '2' simultaneously exhibit Acronym behavior, Copying behavior and Greater-than behavior.
2. Most Neurons (Heads) Exhibit Uninterpretable Activating (Attention) Patterns. Multiple studies report the predominance of MLP neurons lacking clear activation patterns (Arora et al.) 2018
Bricken et al. 2023). Likewise, Krzyzanowski et al. (2024) reports failed interpretation attempts for more than 90% heads in GPT-2.
3. Attention Superposition in the Wild. He et al. (2024a) and Kissane et al. (0024) both found attention output SAE features collectively contributed by multiple attention heads. If we consider SAE features to represent monosemantic directions, such distribution provides evidence for attention superposition. Furthermore, Jermyn et al. (2024) directly demonstrate this through a toy model where 5 ground-truth attention units are put in superposition over 2 attention heads. We also show that about 25% of our learned attention units are spread across multiple MHSA heads (Appendix E.2).
Why Does Attention Superposition Matter? Practically, attribution- based circuit tracing Ge et al. 2024 Ameisen et al., (2025) becomes challenging when features are computed collectively: individual QK patterns do not explain the full mechanism and may be misleading due to interference from other features' computations within the same heads. The structure of attention superposition may relect intriguing motifs of model biology. For example, what makes some privileged attention
units like induction heads mostly implemented by single MHSA head (Olsson et al. (0222 while
others are put in superposition? This parallels privileged bases in MLP neurons (Elhage et al. 2023).
3 LOW-RANK SPARSE ATTENTION
3.1 LORSA ARCHITECTURE
Algorithm 1: Low-Rank Sparse Attention (MHSA Lorsa)
Input: X E Raa Input sequence (n tokens, dimensions)
Wh. W E Rdxda; Query/Key weights for head h. We adopt a QK sharing strategy so QK weights are not independent. See details below.
Wh E & Rdxda w E [Rdx1, 1-Dim Value weights
wht Rdxxd w E IRlxd, 1-Dim Output weights
HUSSA HoOsa E Z Number of Lorsa heads
K E ==: Max number of activated Lorsa Heads Output: Ý Ruxd; Output sequence
for f- 1 to Hiona do
Qh XWh E Ruxdx;
// Query projection for head h
K h XW E Ruxds
// Key projection
h = Xw E R
1/ d,-Dim 1-Dim Value projection A softmax (2*(k(1") E RRnx; // Attention patterns (Causal Mask)
A = Ahp" Rux!; // d 1-Dimensional Weighted sum of values // Output of a single Lorsa head S TopKIndices( (=* h 1, K.) // Select top heads by
h zhww gaxd.
8 Exes ph;
// Add up all selected heads
10 return Y
We detail Lorsa's 's architectural designs in this section, with Algorithm Tiihlighhting how Lorsa architecture differs from a standard MHSA layer. Lorsa takes in the same inputs of MHSA and is
3

---

Published as a conference paper at ICLR 2026
trained to predict MHSA outputs. The training objective is simply minimizing the mean square
(MSE): L = ExED ||Loosa(xx -MHSA(X)
Rank-1 1 Output- Value Circuits. Each MHSA head reads from and writes to a residual stream subspace via its OV circuit (Elhage al. 2021), whose rank is decided by its head dimension dn. Under the linear representation hypothesis that unidimensional features are encoded in the residual stream, we design Lorsa heads with rank-1 OV circuits. This offers the advantage of restricting read/write operations to one or few residual stream features (directions). Although ideal implementations would use rank-1 QK and OV circuits, we restrict dimensionality reduction to OV circuits for practical reasons.
Query and Key Weights with Parameter Sharing. We observe significant performance drop as rank of QK circuits Qx decreases, which severer when QK < QK
DMSSA This may suggest
QK circuits for attention units multidimensional. In result, we choose D D and implement parameter sharing for QK weights across every heads. Unless otherwise specified, we set G Da so that each head maintains parameter count of mostll in average equivalent to setting Di to 1 without parameter sharing, which crucial for Lorsa scalability.
Our parameter binding strategy renders Lorsa QK circuit strikingly similar to MHSA a QK-sharing group of Lorsa heads is almost identical to an original MHSA head except the sparsity constraints applied on each OV dimension. We describe Lorsa heads as individual heads with shared QK circuits rather than sparse dimension MHSA architecture because they often exhibit correlated yet distinct interpretable functionalities, as we will show in Section 6 And there are cases where QK-sharing group of Lorsa heads show no clear semantic correlation (Appendix Cp.
We also show in Appendix B.3 that Lorsa QK circuits are not solely learning to copy the original QK circuits. This distinguishes Lorsa from only applying sparse dictionary learning or Independent Component Analysis on OV circuits (Ameisen al. 2024).
Orders of Magnitudes More Heads and Sparsity. To capture numerous underlying attention units, Lorsa employs an overcomplete architecture with Hoos > HMHSA heads per layer, activating only K < Lorsa heads per token. This parallels learning more features than the input dimension while enforcing sparsity in SAES.
For given token position, Lorsa's output aggregates the Top-K heads with largest z's, where is the scalar activation value of a Lorsa head") The active head subset dynamically varies across token positions. This sparsity mechanism resembles TopK-SAEs (Gao et al. 2024), as both select the K most salient linear components.
Connection to Sparse Autoencoders. Lorsa shows notable resemblance to attention SAEs (Kissane)
al. 2024) for its rank-1 OV circuits. Lorsa learns an overcomplete linear basis of the attention output space we h h= ...... ... HLona) with sparsely activated scalar components {=/* h = 1,..., F
at the i-th position, which is analogous to SAE decoder and sparse feature activations.
However, whereas SAE features are computed via single linear encoders with ReLU, Lorsa head activation at a given position z derives from attention patterns A and x² of previous tokens. Moreover, SABs take in and predict the same activations while Lorsa, like Transcoders (Ge al. 2024; Dunefsky et al.] 2024) learns to predict downstream activations. It is more similar to Gated (Rajamanoharan et al. 2024) Transcoder taking in activations from multiple positions, where the QK circuit resembles the gate with non-linearit and w. is simply a linear encoder.
3.2 LORSA TRAINING
The Low-Rank Sparse Attention modules we are studying throughout this work are trained on all layers of Pythia- 160M and Llama-3.1 3-BB. The training data is sampled from 800 million tokens for
CConepptully, Lorsa head's activation sequence should be a a/1lw rather than 2² For analytic simplicity and clarity, we construct a model with identical predictions but set wi 6- w Ilo0 1|2 66 +- 68) walla and w w/ ///w/// This operation isolates activation z" from output direction w.!
4

---

Published as a conference paper at ICLR 2026
each model. The prompts are collected from SlimPajama (Soboleva al. 2023) truncated to 256 tokens for Pythia and 1024 tokens for Llama.
Best practices for Lorsa training (c.g. Adam optimizer. warm-stable- schedule, optimal Ir scaling law, etc.) largely complies with ones adopted in Templeton et al. (2024b). Training one Lorsa module with settings described in Table T takes Nvidia A100 GPU hours for Pythia (batch size 4,096 tokens) and 24 hours for Llama (batch size = 16,384 tokens).
Active Heads
Params Layer
# Heads
Head Dimension
Target Model
per Token
MHSA
Independent
Lorsa
Lorsa
MHSA
Lorsa
Lorsa
MHSA
Lorsa
MHSA
Lorsa
Lorsa QK
QK
OV
OV
Pythia- 160M Llama-3.1-8B
6K
6K
64
2.25M
18M
256
32K
32K
128
128
32
128
64M
512M
Table 1: Architectural setups for both target models. We primarily focus Lorsa modules with 500-1,000 times more heads than the original MHSA. For instance, we have 6K Lorsa heads for an MHSA layer in Pythia-160M) 160M, with every DLarsa DMHSA 64 heads sharing QK weights. This gives us 96 independent QK weights.
Both models adopt Rotary Embedding (RoPE) (Su et al. 2021 and Llama uses Grouped Query Attention (GQA) (Ainslic al. 2023). We show how Lorsa fits these modifications in Appendix A
EVALUATING LORSA FIDELITY--SPRSITY PERFORMANCE
4.1 L(N, K) SCALING LAWS
We explore Lorsa scaling laws with respect both number of learnable parameters N and their sparsity
256
Fraction Variance Unexplained
K (i.e. number of active Lorsa heads per token)
shown in Figure 2 compared to Top-K K SAEs (Gao
2024). Despite similar scaling trends, there is notable gap between Lorsa and SAE under the same parameter budget and sparsity, especially when is large. Such comparison in terms of reconstruction fidelity and sparsity is in favor of SAEs since Lorsa learns QK and OV circuits to predict attention output with hundreds of activations, while SAE adopts a standard dictionary learning setting with the same input and output.
10-
Sparsity (LO)
FVU (%)
+ Lorsa
* -AE
10M
200M
Number Parameters
4.2 PER LAYER EVALUATION
Figure shows Lorsa's ' per-layer reconstruction er- Pythia 160M and 1.1ma-3.1-8B in terms of fraction variance unexplained (FVU).
Figure 2: Scaling laws FVU against num- ber of parameters and fixed LO for SAEs and Lorsas trained on layer 3 in Pythia- 160M.
We would like to highlight the notable correlation between trends of FVU across layers yielded by Lorsa and SAE in both models. We also observe strong correlation between these two sparse dictionary learning methods terms of per-token error norm and direction (Appendix G
5
ASSESSING LORSA INTERPRETABILITY
5.1 INTERPRETING INDIVIDUAL LORSA HEADS
Top Activations.
With Lorsa heads' output restricted to single direction, their activation strength at given position i be described with scalar 2t (Section 3.1). Similar SAE interpretation methods (Bricken et al. 2023) Templeton et al. 2024b). we iterate over 100M activations from held-out dataset to identify the 16 aighest-activatingg tokens for each Lorsa head.
5

---

Published as a conference paper at ICLR 2026
25
33
20-
15-
10-
44
Layer
Layer
(a) Pethia-160M (12 Layers)
(b) 1lama-3.1-88 (32 layers)
Figure 3: Per-layer reconstruction FVU for Top-K SAES and Lorsas. All Pythia modules (left)
comprises 18M learnable parameters and K = 64. Llama modules (right) have 512M parameters and K 128. We evaluate the mean and standard deviation (shown as shaded areas) with 64K tokens.
Pattern. According Algorithm | the top activations 2f decompose linearly into token-wise contributions from preceding positions: =/ Alt Ej= Al, where A denotes attention weight from token to token j and ot x. Conceptually this tells from which previous tokens the activation 2/ is computed. Thus we call the x pattern. This is analogous to direct feature attribution (DFA) analysis for attention SAEs (Kissane et al., 2024 He et al. 2024a). An SAE feature's s activation at the i-th token fü can be decomposed along heads and sequence position, i.c., fa EEÿzi ΣnER Wymoof, where of is linear component of MHSA output token j from head h. The DFA from token j is then defined as ΣNEH W df. In comparison, Lorsa's 's attribution includes only one rank-1 OV circuit and single, though shared, QK circuit without multi-head aggregation. This enables QK circuit attribution for attention units distributed across multiple MHSA heads.
5.2 VISUALIZATION INTERFACE
SAE features strongly contributing to Lorsa QK pattorn
Top activating samples and patterns
Interpretability
explanation
Head ID In Layer
Lorsa.5 5.4055
Corvelated (Q)
"with")
Conrelated Features
Pattern (Each now
induction Lorea Head for "you"
Rurelenury (Mediiiedd
"mak"
be
SAE features this Lorsa head reading
Predicts "veu" given the context
you...
Cervelated Fastsnus
Cormetated Features [O)
Prodite "yor"
562
SAE features this Lorsa head is activating
Activation Frequency
Previcsa taken
Estimated sparsity
0.4% 250 soiendl
2/075
Logit DIT
space)
with
Correlation to MHSA
Correlated induction
you
-V941
you
oblique projection
heads
5.10
+3.755
+2.732
Jin
.0.04
0.480
Direct effects model output
Figure 4: Visualization dashboard for "you"" induction Lorsa head. We provide an example interpretation of each item below.
Our visualization interface provides multifaceted information on Lorsa head interpretation. We illustrate our dashboards with the example in Figure 4 which visualizes induction Lorsa head specifically firing for the token "you". The methods used to identify correlated MHSA heads and SAE features are described in Appendix B and F
Correlation to SAE features / Logits via OV: mainly reads from current token is "you" your" features via its wh: It strongly activates say "you" feature (i.e., a feature amplifying the logit of "you" via the logit lens (nostalgebraist 2020 It amplifies the logits of a variety of "you" tokens.
Correlation to SAE features via QK: Its QK attention pattern is mainly computed by current token is "X" features the query position and previous token is "X" & current token is "you" features on the key side, where "X" can be a number of tokens that often precedes "you", such as "with", "thank" or "do".

---

Published as a conference paper at ICLR 2026
Correlation to MHSA heads: This Lorsa head is almost equally distributed in MHSA.5.0 and M.SA.5.7. Both MHSA heads exhibit induction functionality, as shown in Appendix E
5.3
QUANTITATIVE EVALUATION WITH AUTOMATED INTERPRETABILITY
Autointerp Scores
Scores
Top Activating Tokens
SAE Scores
0.8
0.4
Distriutton
0.2
0.6
0.4
2
6
9
10
11
Layer (0-11)
Figure 5: Automated interpretability scores Lorsa heads and SAE features. Each distribution estimated with 100 heads features. The average score of each group is represented by horizontal dash line. We highlight distributions with larger mean value suggested by t-tests with a = 0.05.
To quantify the interpretability of Lorsa heads in terms of its top activations and pattern, we perform automated interpretability (autointerp) (Bills et al. 2023) with GPT-4o to estimate how comprehensible each Lorsa head is. We apply standard autointerp extend to Lorsa a-patterns and direct feature attribution attention output SAEs (Kissane et al. 2024). Prompt design, scoring method and choice of few-shot examples detailed in Appendix All results are obtained with Pythia- 160M Lorsa and SAEs the same size.
max activating samples and
As shown in Figure Lorsa achieves higher
in 6 cases, with 3 losses and 15 ties a = 0.05 significance across 24 layer-wise comparisons, suggesting comparable interpretability to SAE features. Both methods exhibit descending scores in deeper layers. Potential explanations include: (1) increased polysemanticity in later layers, or (2) limited capacity of current autointerp pipelines capture long-range dependencies.
6 SEARCHING FOR SPECIFIC LORSA HEADS
We use path patching Wang et al. 2023 Conmy et al. 2023) to find the Lorsa heads involved in specialized tasks. For given Lorsa head path patching ablates its output and allows the influence to propagate only through residual connections and MLPs (but not through other attention heads). This measures the head's counterfactual influence on the model's behavior.
6.1 LORSA RE-DISCOVERS PREVIOUSLY REPORTED HEADS
Previous works have documented attention heads with specific functionalities in well-characterized contexts (Section 7.1 We demonstrate that Lorsa rediscovers more specialized units of these attention behaviors due to its rank-1 OV circuit. Lorsa also isolates an important phenomenon called attention sink (Xiao al. 2024) from other semantically meaningful heads. Figure showcases four such heads, with their visualization dashboards provided in Appendix D.2 A representative selection of interpretable Lorsa heads is presented Table 2
We want to highlight interesting variant of induction heads we call subtoken induction heads where the prediction operates at the subtoken level. When the sequence contains "I Marion] ... [.]],, the head predicts "[arion]". despite involving three distinct tokens ([A] [B] [CC. This occurs because the leading space in "[ Marion]" causes tokenization misalignment, splitting what would otherwise be single token into subcomponents.

---

Published as a conference paper at ICLR 2026
Lemo (Subtoken) Induction Head
Lero
Loms.10.0066
(Arabic Numerall Successor Head
(poly/ mix]
Attention Sink Head
Copy Suppression Head
PP
BOS
ANY
Mar on
arfon
2
poly E PPppelyls T3
and
mylon
Sentingo
amisago
polyy or
might
Pitmiiia
Hauser
RUSeR
mix
and
match
Figure 6: Examples of Lorsa heads re-discovering finer-grained or cleaner versions of previously reported heads. 1orsa.5.1025: A subtoken induction head for names, see details below. Lorsa.6.2814. A successor head attending to the previous arabic numeral token (almost exclusively 1, 2, and 3) and predicts its successor. Lorsa.8.5963: A copy suppression head attending to the previous token (almost exclusively "poly' and 'mix') and suppresses its copy. Lorsa.10.4066: An attention sink head almost exclusively attending to the "<lbeginoftextl>> token.
Lorsa Head ID
Manual Interpretation Induction for "vve" Induction last names Induction for abbreviations Induction after "and"""wiit Induction Italian "ddfiie"J""nclln"" in PHP
Lorsa.5.3955
Lorsa.5.4010
ap1 (27-43)
Lorsa. 7,403
mod 10 (4.04... 5.5
1.15a.9.132
med 10 (R 7.01
Lorsa.9.1622
Lorsa.4.32
Lorsa.4.0013
"public static" in Java
Lorsa 16 20791
Lorsa.5.4035
Say "Four" FFive"""
Lorsa.8.142
Apple Inc. and products (iPhone etc.)
16, 20931
Lorsa...5167
Previous token "can" ""ould""
op2 m0
20947
Lorsa.1
Previous token "make"
Lorsa.4.487
Abbreviations (parenthseesquoots))
S02 (89-33)
Lorsa.6.491
Abbreviations in pareatheses
Lorsa.6.1787
Abbreviations parentheses
Lorsa.6.5499
Abbreviations in parentheses
Lorss.15 19.3615 56
Lorsa.4.1420
Russian contexts
Lorsa15.3311 15.3813
Lorsa.9.1622
Induction Italian
Lorsa 10AA11..0 4001
Lorsa.4.3388
Attention sinks
0 000
Lorsa.7.862
Attention sinks
36 [op1] [op2]
Lorsa.6.2592
"the other" ""another"
10.2a.10.1232
Year of hirth and death
Figure 7: For the prompt "36 + 62 =", Lorsa moves two operands to the last position with heads each. The first operand (36) is attended in terms of 2 pattern by an "op1 E 27- 43". "op1 % 10 [4, 5, 6)" and an "op1 % 10 [6, 7, 7,8 head, which uniquely determines "op1 = =36". The same applies op2.
Table 2: A non-exhaustive collection of inter- pretable Lorsa heads we have found, which are grouped by color from top bottom: induc- tion heads, specific token heads, previous token heads, acronym heads, language-specific heads, attention sink heads, and miscellancous heads.
6.2 FAMILY OF ARITHMETIC LORSA HEADS IN LLAMA-3.1-8B
We identify a group of arithmetic-specific Lorsa heads in Llama-3.1 1-BB that activate during simple arithmetic operations following the template [op1] [operator] [op2] [=]. One observation is that each head fetches certain operands with a number of unrelated heuristics, consistent to
prior findings at neuron level on arithmetic mechanisms (Nikankin et al.) 2024), despite Lorsa's architectural differences.
Figure 7 demonstrates an example of the prompt "36 + 62 =". Similar to Ameisen et al. (2025), we visualize the function of each Lorsa head with operand plot, displaying its activity on the 100 x 100 grid of potential inputs of the template "ap1+ "op1+op2-".
These six Lorsa heads exhibit consistent interpretations in terms of their operand plots and patterns sampled from natural language prompts like "The price went up by 27% from $100 to". We exemplify this in Appendix [33] along with more examples of tettheeti--speiine Lorsa heads. We also conduct very preliminary pertubation experiments in arithmetic tasks to validate Lorsa's 's causal influence on the model's behavior, as described in Appendix D.4
8

---

Published as a conference paper at ICLR 2026
6.3
LORSA HEADS AS THEME ANCHORS
While exploring through Lorsa heads in Llama-3.1-8B, we notice a distinctive subset of Lorsa heads attending keywords with remarkable theme consistency from all subsequent tokens in sentence. Figure 12 in Appendix D.5 illustrates two representative cases which exhibit relatively selective, long-range attention to tokens related to presidency and dynamical systems as evidenced by pattern. Through manual inspection we also find Lorsa heads activating on topics like alcohol addiction, dynamic system, medication instructions and terms of service.
These heads may serve as theme anchors maintaining topic representations to bias predictions. They may relate to SAE features "smeared" across token positions, as mentioned Lindsey et al. (2025) (link) (example).
RELATED WORK
7.1
EXPLAINING INDIVIDUAL ATTENTION HEADS
With the help of activation patching (Meng et al. 2022 Zhang & Nanda 2024) or path patching (Wang al. 2023 Commy al. 2023). the literature has discovered a number of heads that exhibit certain functionality in pre-defined contexts. This line of research starts from a composition of previous token heads and induction heads (Olsson et al.) 2022) which closely related to in context learning. More work on this line includes name mover heads (Wang et al. 2023), number comparison heads (Hanna al. 2023), copy suppression heads (McDougall et al.) 2023). successor heads (Gould al. 2024)
and long context retrieval heads (Wu et al. 2024).
7.2 SUPERPOSITION HYPOTHESIS AND SPARSE AUTOENCODERS
The superposition hypothesis (Arora et al. 2018 Olah et al. 2020 Elhage et al. 2022) assumes that neurons are related to multiple non-orthognal underlying features. Sparse Autoencoders (Cunningham et al. 2023 Bricken et al. 2023) are proposed to extract an overcomplete set of the sparse and linear comprehensible features. importantly, the success of the technique also sheds light on universality of
superposition across model size (Templeton et al. 2024b; [iiebrum et al. 2024 He et al. 2024b)).
model architectures (Wang et al. 2024) and modality (Abdulaal al. 2024).
7.3
SPARSE AUTOENCODER VARIANTS
SAFs have developed multiple variants. Some improve initialization (Conerly al. 2024), loss function (Conerly 2024; Bussmann et al.] (024) or sparsity constraints (Gao et al. 2024) to solve specific issues such as shrinkage (Wright & Sharkey 2024 and massive inactive features (Bricken
et al.
2023).
Another direction of improvement is the SAE architecture. For instance, Gated SAEs (Rajamanoharan et al. 2024) are proved effective in mitigating shrinkage. Transcoders (Ge et al. Duneisky
ct al. 2024) aims to simplify sparse circuit analysis by replacing MLPs, whose non-linear nature makes causal attribution intractable.
DISCUSSION AND LIMITATIONS
We report intriguing findings and limitations. Despite promising results, key challenges remain for future work in the following aspects.
Unbinding QK circuits. A key limitation is shared QK circuits: heads may not be fully independent despite positive pattern findings, since z mixes Q K and V. Circuit tracing risks mis-attributing QK to components other heads sharing the same circuit.
Dynamically Reducing QK Rank. One solution to unbind QK circuits is to reduce QK rank for each Lorsa head. If we could overcome the performance degradation low-dimensional QK circuits, it is possible scale up Lorsa with more independent QK circuits and fewer residual stream features

---

Published as a conference paper at ICLR 2026
interacting via QK This is also crucial for circuit tracing methods to have clearer attribution of QK circuits with fewer features involved.
Moreover, current design of Lorsa QK circuits assumes that all attention units have the same rank (i.e., deedd) ). In Appendix C we show that Lorsa QK rank can be varied across heads by visualizing the singular values of We and WK. A mechanism to dynamically determine the rank QK circuits for each Lorsa head would be a promising direction for future work.
Dark Matters. We find non-trivial correlation between Lorsa error and SAE errors trained on the same attention layer in terms of (1) average loss per layer (2) loss per token on the same context and (3) error direction, as shown in Appendix G This may suggest the existence of universal dark matters (Olah & Jermyn 2024 Engels et al., 2024) for sparse dictionary learning methods like SAE and Lorsa. Any progress along this direction to reduce or understand SAE / Lorsa dark matters should reveal many interesting behaviors of neural networks.
Inactive Attention SAE Features and Lorsa Heads. Despite efforts on hyperparameter search, we find that attention SAE and Lorsa both contains majority of inactive feature heads (i.e. not activated once in le6 tokens). This phenomenon renders most computation wasted and raises question about the difference between structure of attention output space and MLP output space or residual streams, where SAEs of the same size only have few dead features if configured properly.
Cross Layer Attention Superposition. If certain inter-token feature interaction is performed in more than one layer, our current method which decomposes only one MHSA layer does not suffice to find such relation. This parallels the problem of cross-layer superposition (Templeton et al. 2024b) for residual stream features. cross-layer variant of Lorsa (Lindsey et al.] 2024 might be tractable.
Global Weights and Systematic Q/K/V Composition. To better understand the global attention behavior of Transformers, one important research direction is to identify systematic Q/K/V compo- sition like induction heads and previous token heads. Since Lorsa reveals finer-graines versions of MHSA heads, we can expect to find more of such cross-layer collaboration behavior. However, we failed in our early attempts to find Lorsa heads with Q/K composition.
CONCLUSION
In this work, we introduced Low-Rank Sparse Attention (Lorsa) to disentangle atomic attention units from attention superposition in Transformer models. Our experiments validated that Lorsa can recover known attention mechanisms and uncover novel interpretable behaviors. The scalability and quantitative autointerp results suggest the potential of Lorsa to adapt to real-world applications, especially unveiling the nature of attention computation in systematic end-to-end circuit tracing.
We hope Lorsa will eventually enable full sparsification of model computation alongside MLP counterparts. Our initial attempt on a two-layer Transformer unveils a clean induction circuit (Appendix H)
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (No. 62525602).
PI might also be the case that attention units must he described in multidimensional QK circuits, like induction heads requiring attending multiple "the previous token is X" features.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Ahmed Abdulaal, Hugo Fry, Nina Montaña Brown, Ayodeji Ijishakin, Jack Gao, Stephanie L., Hyland, Daniel C. Alexander, and Daniel C. Castro. An x-ray worth 15 features: Sparse autoencoders for interpretable radiology report generation. CoRR, abs/2410.03333 2024. doi: 10.48550/ARXIV. 2410.03334. URL httphh:t//ddeet org/ 48550/ (at8550/ariii/ 2410.....441...m 03334
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10 60, 2023, pp. 4895-4901. Association for Computational Linguistics, 2023. doi: 298. URL https: //doi org/10 15. /653/v112022
emn palllppmain..m 298
Emmanuel Ameisen, Joshua Batson, and Jack Lindsey, Investigating successor heads. Trans- former Circuits Thread, 2024. URL ltttp:://rans www.aasssoooeeerrrttttttt cwrccitstt....ee (tt.uitss.uu///2211 pub/2024/
September-update//iee simpddate/indexx. inw/x.nddxxx.//// html
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. BRLhttps. //transformez ctte--iicuutttt wttpc//ww...ppp../ee/e....tt htt.a//ppp.ae22aaaaaaaa.o.....0. graphs hosess/metthoo hods html
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Trans. Assoc. Comput. Linguistics, 6:483-495, 2018. doi: 10. 10.1162/TACIA A000004 URI. http:
/ooi www.oor...... 10.
Joshua Batson, Brian Chen, and Andy Jones. Circuits updates march 2024. Transformer Circuits Thread, 2024. URL httpss//rrrassfooorrrrrocaaa pttp:sss..pppp22//e./ httbb/2222//acc-uuuttteee index. h
Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 2397-2430. PMLR, 2023. URL http:::://pcceeeeee mlr. wttp://eww./.2..........aaa
html
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models explain neurons in language mod- els.
i ppupliccc.... b.bl.....b... core. weerre.window windows. www.:/eeoooneeeeeeeeeeee n
paper pepeer/inde..oo html 2023.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravee, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023, URL https: http///taasffooo citerrrccrcuttss
pub/ 2ttp/223//aa 30023//moosemmnticc-omm wtw.rrannnttteeteeeeeeeee findxx.hmm
Bart
Bussmann, Patrick level features with
Leask, and
Neel
Nanda. LessWrong
Learning
multi- URL
matryoshka saes.
2024.
https: http:://www..esssoooooooooooooop

---

Published as a conference paper at ICLR 2026
Tom Conerly. Circuits updates february 2024. Transformer Circuits Thread, 2024. inddx.idxxx... html#
URL
dict Dict-learnng Licc-Leeaminn-rreaam..in
Tom Conerly, Adly Templeton, Trenton Bricken, Jonathan Marcus, and Tom Henighan.
Circuits updates april 2024. Transformer Circuits Thread, 2024. URL
https://transformm ciruiits........... pttpatts...///2//aprrr// http://///apppppaaa/ta/aa//a/a/a -update/index html# training-sass
Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adria Barriga-Alonss Towards automated circuit discovery for mechanistic interpretability. In Al- ice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023, URL http: ingrrsssiiiie nips. cc/ wwwp:ppppppppppeeeeeepeeeeeeee.e files/ bal.
html
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey, Sparse autoen- coders find highly interpretable features language models, CoRR, abs/2309.086000 2023. doi: 10. (S550/ARXIV.2309.08668
BRLhhttp:: / /tw. org/10 (1888/5500001/// 2309. 08600
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson. Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia Olano, Diego Perino. Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CORR, abs/2440.21178, 2024. doi: 1048550/ARXXV2220.2222222 URL https
//doi.org/10. 48540 48550/ 2407 21783
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. CoRR, abs/2406. 1 11944, 2024. doi: 10.46550ARXXX.2200.11999 11944. URL https
//doi. 100 walpg//...855////0 48550/ 2406. 11944.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/transformme-
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Haffield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposi- tion. Transformer Circuits Thread, 2022. URL httpss://transseeeeeee citmmmerrrrruuee ts pub/
(002//ooymodee//neeeet model/indee.... html
Nelson Elhage, Robert Lasenby, and Christopher Olah. Privileged bases in the transformer residual stream. Transformer Circuits Thread, 2023. URL html
pub/2023/privileeee
12

---

Published as a conference paper at ICLR 2026
Joshua Engels, Logan Riggs, and Max Tegmark. Decomposing the dark matter of sparse autoencoders. CoRR, abs/2410. 2024. doi: 10.48550/ARXIV22100.01 URL http::
10. 25.48550/arxiv..oo 2ww.:xiwii....1...mmmm 14670
Iee Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, llya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoderrs. CoRR, abs/2406.0400 040933 2024. doi: 10.48550ARXXV.24466000 040.04.3.. URL https:///oi.ozg/110 1406.04090
Xuyang Ge, Fukang Zhu, Wentao Shu, Junxuan Wang. Zhengfu He, and Xipeng Qiu. Automatically identifying local and global circuits with linear computation graphs. CoRR, abs/2405. 13868, 2024. doi: 10.48550/ARXXV.2405.. 13868. URL http:://doo..oggg/// 21t1//0...4500/xxvv..ooo 13868
Rhys Gould, Euan Ong. George Ogden, and Arthur Conmy. Successor heads: Recurring, in- terpretable attention heads in the wild. The Twelfth International Conference on Learning Representations,, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL htttps://ppeneevier net www.oooidddddwceeee-
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in haystack: Case studies with sparse probing. Trans. Mach. Learn. Res., 2023, 2023.
URL bttts:::: http:///opeeeeeeeeeeeeee httpe//wwwweeeeeeeeeeeeeeeeeeeee wateet/fooummiddd/ 1t
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater- than?: Interpreting mathematical abilities in a pre-trained language model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, IA, USA, December 10 16, 2023, 2023. URL htRptttt:// nips. wwcppppprrilllleceeeeeeeeeeeappp files/ /tlles/paparr/2233/aasss e http:://555577722222222222222243 -tw.eeeeeeeee--ttttttt---------- Cesstract-Cofferenece Conf-rooooeeeeeeeeeeeeee-eeeee-e httnferrnncehtml]
Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Dictionary
learning improves patch- free circuit discovery in mechanistic interpretability: A case study on othello-gpt. CoRR, abs/2402. 122011 2024a. doi: 10.488550/ARXIV22000..222 URL https
0ttp///1...50//01111111111 wtw.aaawwi........ 12201
Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. Llama scope: Extracting millions of features from llama-3,1-8 with sparse autoencoders. CoRR, abs/2410.20526. 2024b. doi: 10. 40550/AAXXI.2440000555 URL https: wwt.:oo..orrro/// 48550/0004855////////// 241 14100.20 20526
Adam Jermyn, Chris Olah, and Tom Conerly. Circuits updates january 2024. Transformer Circuits Thread, 2024. URL https::/tranafoomeee ciruutsssrruuu
index. html #attn- superpositeonnnnnn
Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Interpreting attention layer outputs with sparse autoencoders. CoRR, abs/2406. 17759, 2024. doi: 1044850/ARKXI..446..11 17759
URL https:/t /05t:///////....0.//i/.......
Robert Krzyzanowski, Connor Kissane, Arthur Conmy, and Neel Nanda. We inspected every head in gpt-2 small using saes so you don't have to. Alignment Forum, 2024. URL https: www. alignment forum. org/ posts wttp///eeeeeeeeeaaeeeeeeee
we inspected every head in gpt 2 sma 1 using saes so you-do-oooooooooooo
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca D. Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. CoRR, abs/2408 010.4088.554444 047, 2024, doi: 10.48550/ ARXIV.2408.05147. URL https: / org 48550/ 2408. 05147
13

---

https//diiiilllpb//222/ccctttiis Published as a conference paper at ICLR 2026
Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for cross-layes features and model diffing. Transformer Circuits Thread, 2024. h wirtteerrrrrrrtsi..eem httpp/////////c//soeer/.... ht.m
Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025. URL halb/2225/tttibian weareiionngraass/iboooooo wwt.s/iioooo.....o.o....ooo..o.. html
Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding an attention head. CoRR, abs/2310.04625,, 2023. doi: 10448550ARXXV.231..4046 URL httph: /tttss://doi.ooo wrg/ 485504855553.. 2310 04625
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 De- cember 2022, 2022. URL http: http::::/ppppp....ee nips ww/paperpppppppp.pppppp fwwesfillsssaee filess/ppper/2222 -tw.:/aaattaaaaaaat------------e
Abstract Abstract-Confeeencee.te Conference.htm
Neel Nanda and
Joseph
Bloom.
Transformeriens. 2022.
https://github.co
Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without algorithms:
Language models solve math with a bag of heuristics. CoRR, abs/2410.21272, 2024. doi: 10.
48550/ARXIV.241022222...
nostalgebraist. interpretintnnnn gpt: the logit lens. lesswrong 2020. URL http:://wwwweessrrr..oo
gpt Sche-Logit-1ee
Chris Olah and Adam Jermyn. Circuits updates july 2024. Transformer Circuits Thread,
2024. URL
Zoom in: An introduction to circuits. Distill, 2020. doi: hotplly-pddae/iindeee
html #hurdles)
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Haffeld-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Ka- plan, Sam McCandlish, and Chris Olah. In-comtext learning and induction heads. Trans- former Circuits Thread, 2022. URL fttps::/Eranffoomee cwapitiiccutt... pub/ 2022/ In-context-learningg and- htt.raaaeiiiicccctcoeeeeeaaaaa/e htg.:e/eeeerrnnnnnnccnncnnoooaar
html
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse au- toencoders. CoRR, abs/2404. 16014, 2024. doi: 1044850/ARXXI22444.6000 URL https
Daria Soboleva, Faisal Al-Khateeb. Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and
deduplicated version of RedPajama.
2023. URL /ttp
627B
14

---

Published as a conference paper at ICLR 2026
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09864, 2021. URL /tt.ttata////iiv..oor////
0986.
Adly Templeton, Joshua Batson, Adam Jermyn, and Chris Olah. Circuits updates - january ci siruiits
2024. Transformer Circuits Thread, 2024a. URL
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Trans- former Circuits Thread, 2024b. URL htt.:/tttttttttaaassottttoo rrcp
rwciiirrciice ircuitts..... pub/20247
scal Teaaling--mae hngg-momssenattt..ttt/////aa/ae
Junxuan Wang, Xuyang Ge, Wentao Shu, Qiong Tang, Yunhua Zhou, Zhengfu He, and Xipeng Qiu. Towards universality: Studying mechanistic similarity across language model architectures. CoRR, abs/2410.00672. 2024. doi: (044850/ARXI.2411..006 URL htt.tttts//....... org/
10. 48550/arxixx 241.0.2410667
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:22 1rXi::2211000993 2022.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: /sttps://ppezevvale net forum?id
NpsVSN6o4ul,
Benjamin Wright and Lee Sharkey. Addressing feature suppression in saes. Less- Wrong, 2024. URL httt::::/www.eesswrorre 1 addressing- fewture- sww.reesssssuppr.soo.oso..o..oe. -t
www.coo//ooso.e http://///.ttt///tt///////aeee.. saes
Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically
explains long-context factuality. CoRR, abs/2404 15574, 2024. doi: 10,48550ARXXV..404.111 155774.
4ww50//a..i....... URL /tRpssss://oooooorg http://............/.....a..x../
15574
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference Learning Representutions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
https://cpenrevie
net/
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL htt.h/ttss/////m /ttp:pps/////eeeeeeeeeeeeee net FoorumiiddHHa.
15

---

Published as a conference paper at ICLR 2026
A APPLYING LORSA TO MHSA VARIANTS
Modern transformer-basee models commonly employ variants of multi-head self-attention (MHSA), such those incorporating rotary position embeddings (RoPE) (Su et al. 202T) and grouped-query attention (GQA) (Ainslie et al. (2023). Lorsa demonstrates compatibility with these MHSA variants through straightforward adaptations.
For RoPE-based MHSA layers, we apply the same rotary transformations to Lorsa's 's computed queries and keys before computing attention scores, maintaining the positional information encod- ing.
GQA implementations, Lorsa operates without modification cnddfication-spccfffilllll we intentionally avoid introducing grouped queries within the Lorsa framework.
Empirical results on both Pythia- 160M and Llama-3. demonstrate that this design choice does not adversely affect performance. We apply these architectural variants based on the TransformerLeens
library (Nanda Bloom 2022).
B ABLATION STUDY ON CRUCIAL ARCHITECTURAL DESIGNS
We conduct ablation studies on two crucial architectural designs: (1) the query and key dimension and (2) the binding ratio. Our experiments validate the necessity of maintaining both the QK dimension and the binding mechanism in our proposed architecture. Additional ablation tests on other implementation details further validate our decisions.
Furthermore, we derive two hard constraints for parameter selection (violating these constraints leads to significant performance degradation):
The QK dimension must not be smaller than the head dimension in MHSA
The number of QK pairs not be fewer than the number of attention heads in MHSA
B.
ABLATION STUDY QK DIMENSION
Fraction Variance Unexplained (FVU) (FVU)
Fraction Varlance Unexplained (FVU)
Original MHSA OK Dimension
Original MHSA OK Dimension
70%
Layer
Layer
60%
Layer
40%
Layer
Layer
Layer
50%
40%
30%
20%
20%
10%
10%
8
16
32
64
128
256
16
32
128
256
QK Dimensions
QK Dimensions
(a) Context Length: 256
(b) Context Length: 1024
Figure 8: Ablation study on the QK dimension using Pythia- 160M under different context lengths (K 64). We fix the parameter budget across all settings and observe that reducing the QK dimension below the original MHSA head dimension (dhena = 64) results in significant performance degradation, highlighting the importance of maintaining a high QK dimension.
We conduct ablation studies on the QK dimension using Pythia- 160M, evaluating performance under different context lengths (256 and 1024 tokens). To ensure fair comparison, we fix the parameter budget at 4Dmestet per attention head and maintaining total parameter count equivalent to 4x the original MHSA configuration throughout all experiments. As shown in Figure & reducing the QK dimension below the original MHSA's head dimension (dhest = 64) leads to severe performance degradation. This empirical evidence supports our design choice maintain high QK dimension.
16

---

Published as a conference paper at ICLR 2026
B.2 ABLATION STUDY ON BINDING RATIO
Fractton Werteron Unsppimmmpooo
17%
Prection Vartance Unaplemete
Ilewdx-261
Lore Ilewdk-361
16%6
Lome 11e6
25%
Lorsa 11eess-1516
30%
Lorea 1lesdx-1576
Lome 20s2
Lorsa 1osdss-0022
Traction lartnne
1474-
20%
Lorsa Hoa
256
Meads-5144
12%
19%
115
Independent Lorsa QK Heads
Independent Lorsa QK Heads
Independent Lorsa QK Heads (c) Layer
(a) Layer
(b) Layer 6
Figure 9: Ablation study on the binding ratio. We vary the number of independent Lorsa QK heads and evaluate model performance under different settings. Appropriate binding maintains performance while reducing QK circuit cost, whereas overly aggressive binding (below the number of original MHSA heads) leads substantial degradation.
We conduct a systematic study on the impact of the number of independent Lorsa QK heads (i.e., the number of Lorsa heads divided by the binding ratio) across range of configurations, illustrated in Figure 9 Our experimental results highlight two key observations:
Appropriate binding effectively preserves model performance while substantially reducing both the parameter count and the computational cost the QK circuit (scaling proportionally with the binding ratio).
Model performance deteriorates significantly when the number of independent QK heads falls below the original MHSA head count, establishing this threshold as critical lower bound for binding ratio selection.
B.3 ABLATION STUDY ON QK INITIALIZATION
Given that QK matrices maintain high dimensionality and adopt binding strategy, natural question arises: can we directly reuse the original MHSA QK parameters in Lorsa? To investigate this, evaluate three settings: (1) randomly initializing the QK parameters of Lorsa, (2) initializing the QK parameters of Lorsa with the original MHSA QK parameters and allowing them to be updated during training, and (3) fixing the QK parameters to the original MHSA QK parameters throughout training. The results, summarized in Table 3, show that directly fixing the QK parameters to those of MHSA leads to worse performance compared to the other two setups. This suggests that during optimization, Lorsa learns QK parameters that capture information not present in the original MHSA parameters.
Initialization Strategy
Fraction Variance Unexplained (FVU)
Random Initialization Initialization with Original QK (Trainable) Initialization with Original QK (Fixed)
11 11.3%
1.22%
12.4%
Table 3: Comparison of different QK initialization strategies for Lorsa.
B.4 DOES (ToP-K) LORSA NEED RELU NON-LINEARITY TO GUARANTEE NON-NEGATIVE OUTPUTS?
To align with the superposition hypothesis and the architectural design of the SAE, we apply a ReLU to ensure that the activations non-negative However, observe that this modification has negligible impact on training dynamics, as the top-k activations are almost always positive for reasonable choices of k. This is consistent with findings reported in Gao al. 2024
17

---

Published as a conference paper at ICLR 2026
Q
K
Relative Singular Value
1
1
Circuit
Circuit
Circuit
Circuit
0.5
0.5
0
32
64
96
32
64
96
Component
Figure 10: Sorted relative singular values of We and W k for each QK circuit at pythia- layer 5. Each circuit shows strong alignment between the spectra of Wq and WK. suggesting similar structural properties. Circuits 4 and 15 have relatively high effective rank, while Circuits 16 and 17 exhibit significantly lower rank.
C DOES QK RANK VARY ACROSS ATTENTION UNITS?
We analyze the structure of 24 independent QK projections trained at layer 5 of Pythia-160M, Specifically, we estimate the effective rank of each pair of Wo and Wx by sorting their relative singular values in descending order, shown Figure TO, Among these QK circuits, Circuit exhibits subtoken induction, previous-token, and successor attention patterns; Circuit 15 also shows clear induction behavior. These circuits tend have relatively high ranks. In contrast, Circuit 16 attends to itself on certain special tokens, and Circuit 17 functions as an attention sink while also attending itself on specific inputs. Both of these circuits exhibit lower effective ranks.
D ADDITIONAL CASE STUDIES
D.1 ATTRIBUTION ALGORITHM FOR !DENTIFYING LORSA HEADS WITH SPECIFIC FUNCTIONALITIES
In addition to the path patching method discussed in Section 6.1 we employ attribution algorithm, inspired by the approach for detecting important features with attribution in Batson al. (2024), to identify Lorsa heads associated with specific functionalities.
The attribution score for given Lorsa head h, is defined as:
attr == On V
Here, V,C the gradient of the logit on the prediction the target token with respect to the attention output O₂ of the Lorsa head. For different prompt, also try logit difference probability difference to calculate C.
quantifies the contribution of Lorsa head to the prediction of the correct token.
D.2 EXAMPLES OF LORSA'S REDISCOVERY OF REPORTED FUNCTIONAL HEADS
The detailed information the Lorsa heads discussed in Section 6.1 is provided in Figure II where we visually demonstrate the logit differences induced by the Lorsa head ,along with the most strongly correlated MSHA heads and SAE features.
18

---

Published as a conference paper at ICLR 2026
Lor38
Gervelazed Features
*
constant
Featuees (R)
Lorsa.6.2814
Previuus tnken
Featuees
Previuua tofear la
Correlated
Auteintery (Maditiee)
"whats
Autolsterp (Modified)
conetont bias:
constant briss
'rom"
Suczeeeet Head (Foassee idartifving the number sequanoe)
"kme" "arr""yarr"" -4
(Focuasa identifing
Previnus token '1:
(0)
Correlated Festuree (V)
Correlated Features (0)
& sibredatoons 0.220
taken after number 0:526
Activation Frequancy
sttredattions
0.201
Thateer
175
Activaation Frequency
"ty" in "Petty"
0.300
0.25%
10 & sttrecatioon 0.303
0.74%
Previcus token is
Corrlared MHSA Heads
starederians 77 0.302
Corrolated MHSA Heads
*I"
0.223
legit DIlT
6.
1.212
Logit Dif
+
+0.22
-3.576
en
0.095
6
10 104455
ent
-2121
0.364
+0.317
confirmed
-3.524
63
0.080
6
10.39
_wand
-2.77
+3002
stored
-3.536
6
-0.329
_dcenward
-20074
Parterr (Each row cerrespends a Query
-0.005
strong
-3.408
Pattern row
+0 10.320
bioceciee
-2.058
-0.23
-3445
cornaponnd Quary
+0 256 Auluaa
-2.317
BB 00:000
Lor93.8.5963
Correlated Featunee (Q)
Correlated Features (K)
Corsa.10.4066
Corrolitted Featurre (Q)
tolkon *15
Previous tolen
Correlated Features (8)
Autoisterp (Modified)
first boken aftee *y ofter
Autointerp (Modiffed)
Presicux Inloen
Predicut Irloen
Hhai
ghaine
***
Prevous token
Comelated Fastuaees (V)
Cormlated Festures (0)
"baginataani-1
Correlated Feanures
Correlated Features (0)
Activation Frequency
Corvelated MHSA
1.26%
"21"
taken
0.100
0.186
Previous
Thar Literr
D.17
0.55
2.109
3152%
Corelated MANS Hlaads
Activatton Frequencc
caphal lrtten
Pervical takie "eor"
0123
0,577
0098
cad Nature
WMert
6337
0311
8.
0.987
Logit
10.12
DIf
8
0.0P0
cel
-2.057
-0378
-2.243
<baginahecti>
B.
0.023
inde
-2.041
-0.290
Doniets
-1.94 11948
-0.282
Pasrern (Each cornerponda Quary tolenn)
Bert
-1.93
-0.264
2 Pattern (Each roW
.wwaass
-1998
-0.208
correspends a Query tokees
Prpely's
Figure 11: Detailed information Lorsa's rediscovery of reported functional heads.
D.3 ARITHMETIC LORSA HEADS
We present the SAE features related to the reported arithmetic Lorsa heads in Table which shows consistent interpretation in terms of operand plot and pattern. Additionally, Table 5 provides a broader set of examples for these arithmetic Lorsa heads, including functional descriptions and the z- patterss of their top activations.
D.4 PRELIMINARY PERTUBATION RESULTS
We feed Llama-3.1-88 "75 3 =" as the clean prompt and it succeeds to predict the answer 25 (p 0.73). With attribution from the correct answer logit we identify "op2 3" Lorsa head in layer 15 (Lorsa. 15. 15.2668) with notable contribution. We then set the activation strength 2 of this head 0 at the last token position ("=") and copy its original value to a an "op2 = 5" head s.orsa.15.309. and rerun the forward pass from layer 15 attention. This gives an answer of 15 (p- 0.66).
Since 2 of a Lorsa head indicates its output norm along the w, direction, this pertubation experiment greatly resembles steering SAE vectors (Templeton et al. 2024b). There is also an alternative
19

---

Published as a conference paper at ICLR 2026
Lorsa head ID
Manual Interpretation with Operand Plot
Manual Interpretation with Pattern
Lorsa. 16.20791
op1 E 27 - 43
near 30
Lorsa. 1
opl % 10 e [4 5,
ending with 4 or
Lorsa. 16.20947
op1 % 10 E [6, 7,8
ending with 7, sometimes 6
Lo.sa.15.366
op2 % 10
ending with from - 9
Lorsa. 15.3813 Lorsa.15.4000
op2 E 55 - 99
op2 E 38 - 63
near 50
Table 4: Supplementary information of Lorsa Head in Figure 7
We observe alignment between
interpretations obtained from operand plots and top activating = patterns sampled from natural language text corpus.
ID
Operator
Operand
Top Activation Z Pattern
Addition
op2 ends with 2 min(opl, op2) ends with 2
Subtraction
15.36415
Multiplication
op2 = or 12
drop
From
23788
Division
op2=2 op2 ends with 4
Addition
Subtraction
min(opl, op2) ends with 4 op2 24, or 40
Multiplication
regimented
formation
Division
op2 =4
Addition
Inactive
Subtraction
Inactive
15.26.65.2668
Multiplication
op2 = 3, 6, 30, or 60
Volume
PPA
Division
op2 around 3 or 30 Inactive
Addition
Subtraction
Inactive
15.2770
Multiplication
op2 around 62 and its multiples op2 around 62 and its multiples
Alvares
landed
202
punches
percent
Division
Addition
Inactive Inactive
Subtraction
Lorsa. 15.2945
Multiplication
op2 11 and their multiples op2 11 and their multiples
grewe
From
04B
$57,388
3.80%
Increase
Division
Table 5: Additional cases of arithmetic heads
interpretation that we are intervening attention computation OV circuits this result can be precisely achieved by swapping the w, of these Lorsa heads. In consequence, the pertubed Lorsa head recieves "op2 = 3" but tell subsequent computation that "op2 = 5". Such pertubation independent from QK circuits both Lorsa heads share the same QK weights. This serves as evidence in the wild that Lorsa heads with shared QK circuits often show similar functionalities.
D.5 THEME ANCHOR HEADS
E
ASSESSING CORRELATION WITH MHSA
How to understand the correlation between Lorsa heads and original MHSA heads? We try to answer this by computing the attribution of each Lorsa head to the original attention heads using an oblique projection method (Appendix H.I). Analyzing all Lorsa heads trained Pythia-160M, (Appendix E.2 we find that roughly half of the Lorsa heads originate from single original head, while the other half are superpositions across multiple original heads.
E.1
OBLIQUE PROJECTION METHOD FOR ATTRIBUTION
Given the output an original attention head, we project it obliquely onto the (generally non- orthogonal) basis formed by the outputs of all Lorsa heads at the same layer. The resulting coefficients
20

---

Published as a conference paper at ICLR 2026
Presidency-Relatedd Theme Anchor Head
Dynamical System Theme Anchor Head
(Sear) "attine
President
Self
-sffline
iiltage
hove
Kay - Pessteett
4ysaniccs
Under
nomilne
the
herssonorppiinn
compast
(Sel)) "sartne"
Kay puminssion
_pmoddenta
(a) z pattern presidency-related theme anchor (b) a pattern a theme anchor Lorsa head related Lorsa head.
dynamical systems.
Figure 12: Two examples of theme anchor Lorsa heads.
represent the contribution of the original head each Lorsa head. Since the summed outputs of original heads and Lorsa heads closely match, the contribution coefficients for given Lorsa head approximately sum one. Conversely, we similarly compute the fraction of each Lorsa head's output that can be attributed to each original attention head by projecting the Lorsa head's 's output onto the basis formed by the original heads' outputs. reported results are averaged over more than 1M tokens.
E.2 HOw MANY ATTENTION UNITS ARE DISTRIBUTED ACROSS MHSA HEADS?
n
60
1
Percentage (%)
50
3
40
>3
30
20
10
0
2
6
10
Layer
Figure 13: Distribution of Lorsa heads based on the number of original attention heads they are superposed over. No clear trend is observed across different layers. Approximately 50% Lorsa heads are primarily associated with a single original head, about 25% are superposed over two different original heads, around 10% are superposed over three different original heads, and others superposed over more than three original heads.
We compute the attribution statistics for all Lorsa heads trained on Pythia- Pythia-160M. head, we define n as the minimum number of original heads whose cumulative contributions exceed 90%. We interpret as the effective number of original heads Lorsa head superposes over. As
160M. For given Lorsa
21

---

Published as a conference paper at ICLR 2026
shown in Figure L approximately half of the Lorsa heads primarily derived from single original head, about a quarter involve two original heads, and the remaining quarter involve three or more original heads.
E.3 INDUCTION MHSA HEADS IN PYTHIA- 160M
Table 6: Contribution of each MHSA head to induction behavior in Pythia- 190M,a--660M, measured via path patching. Notable induction heads (L5 O, 14. 6, L5. 7, L9. O, L5. 6) are bold.
Layer' Head
0
1
2
3
4
8
9
10
11
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.07
-0.15
-0.10
0.03
0.09
-0.08
-0.07
0.06
-0.01
0.11
0.34
-0.05
2
-0.14
0.07
0.10
0.14
0.14
-0.13
0.60
-0.03
-0.14
0.10
0.04
0.03
-0.24
-0.14
-0.96
-1.20
-0.49
-0.14
0.20
-0,38
40.10
0.06
40.11
-0.07
4
0.13
-0.26
0.09
-0.6
-0.10
-0.02
0.89
0.13
0.09
-0.28
-0.14
0.30
5
4.00
-0.20
0.05
0.06
-0.53
-0.04
0.48
0.62
0.06
0.08
0.05
-0.23
6
-0.04
-0.23
-0.04
-0.22
0.02
0.09
0.04
-0.33
0.02
-0.04
-0.38
0.04
7
-0.28
0.17
0.03
0.06
-0.28
-0.07
0.01
-0.18
-0.23
-0.03
-0.02
0.18
-0.07
0.03
0.50
0.00
0.15
-0.02
0.01
-0.22
0.02
-0.02
-0.08
0.38
0.54
-0.03
0.07
-0.09
-1.10
-0.04
0.04
0,00
0.04
0.10
-0.01
0,02
10
-0.01
0.03
0.00
0.00
-0.03
-0.10
0.01
-0,01
0.00
-0.04
0.03
0,01
-0.14
-0.13
-0.05
-0.04
0.00
-0.02
-0.11
-0.02
0.01
-0.07
-0.02
0.06
We use path patching to measure the contribution of each MHSA head in Pythia- to induction behavior. The results are shown in Table We find that heads L5. O, L4 6, L5. L9. L5. exhibit the most prominent induction signals.
F INTERACTION BETWEEN LORSA HEADS AND SAE FEATURES
We trained Sparse Autoencoders (SAE) on both the inputs and outputs Lorsa to facilitate the understanding of its functionality. Since Lorsa's Q. K. and V computed from the input, with the output derived from O contributing to the final result, interactions between SAE features and these components exist across all four aspects: Q K, O, and V. To evaluate the influence of SAE features on Q and K, we employ an ablation method (Appendix F.I. The correlation between the OV and SAE features assessed using cosine similarity (Appendix F.2) For each Lorsa head, identify the SAE features most strongly correlated with different aspects. The results are visualized in the Lorsa head dashboard.
F.1 QUANTIFYING FEATURE IMPACTS ON Q AND
For given Lorsa head, the impact of specific feature on Q is calculated as follows: First, compute the attention pattern at the activation locations of the Lorsa head. Then, the feature is ablated from the input, and Q' and the new attention pattern are computed (with K remaining unaffected). The Kullback-Leibe to quantify the effect of the feature on Q. After iterating over million tokens, the maximum KI. divergence observed across all activations of the Lorsa head is taken the measure of the feature's influence on Q for this head. A similar approach is used to calculate the impact a feature on K, with the difference being that when recalculating the attention pattern, all instances of K are recomputed using the modified input, while Q remains unchanged.
(KL) divergence between the original and modified attention patterns is used
F.2 QUANTIFYING DIRECT FEATURE ATTRIBUTION VIA O AND V
For given Lorsa head, both the weight vectors Wo and Wy are one-dimensional vectors of size Dmodet Therefore, for each SAE feature trained on the Lorsa input, the contribution to V is linear, meaning that the contribution of each feature to scales proportionally with the feature's activation value. Similarly, for each activation of the head, the contribution of SAE features trained on the Lorsa output to the activation value also linear. We compute the cosine similarity between the decoder of each SAE feature trained on the Lorsa input and Wy. which quantifies its correlation
22

---

Published as a conference paper at ICLR 2026
with V for the given Lorsa head. Similarly, the cosine similarity between the encoder of each SAE feature trained on the Lorsa output and Wo is computed to measure its correlation with for the given Lorsa head.
G LORSA DARK MATTER
Figure 14 illustrates the per-token error norms of Lorsa and SAE across layers 2, 6, and 10 of 1ythia-T60M. on a set of 64 tokens. Figure Lorsa and SAE's per-token error [T quantifies the distribution of cosine similarity between the same layers, measured approximately 10,000 tokens. These results indicate that the loss pattern between pre token between Lorsa and SAE has nontrivial correlation.
is interesting that both Lorsa and SAE exhibit a positive correlation in their magnitudes and trends for FVU and per-token eπrоr norms.
We propose that this is not a coincidence, and hypothesize that it stems from a shared gap between sparse dictionary learning and the representation structure of data within the model. Alternatively, this correlation may arise from the challenge that sparse dictionary learning faces in capturing super-rare data features or certain nonlinear or dense components within the features.
This supports the hypothesis of universal dark matters (Olah et al. 2020 Engels et al. 2024) that a certain fraction of error results from the superposition hypothesis itself that cannot be addressed simply with larger Lorsas (SAEs).
Per Token Error L2 Norm
SAE Orer Nore Meer
200 More
Figure 14: Per-token norms of Lorsa and SAE layer 2, 6, and 10 of Pythia- 160M for randomly sampled sequence with 64 tokens.
Frequency
Cosine Similarity
Figure 15: Cosine similarity distribution of per-token error between Lorsa and SAE on layer 2, 6, and 10 Pythia- 160M, measured with approximately 10,000 tokens.
23

---

Published as a conference paper at ICLR 2026
H TOWARDS FULL SPARSIFICATION OF A 2-LAYER TRANSFORMER
Since our final goal is to understand Transformers' inner working by breaking down MHSA and MLPs into atomic units (Fi we train Lorsa and Transcoder (Dunefsky et al., 2024 on a 2-layer Transfomer (link). We follow the method introduced in Ge et al. (2024) where they multiply features via QK circuit to find the most salient feature pairs contributing to QK scores. Alternatively applying attribution through Transcoder features / Lorsa heads and QK ablation gives us the clear attribution graph for induction behavior (Figure[To, Due to the capability constraint of this model, we failed to observe more interesting behaviors attribution graphs involving Transcoder features. Nonetheless, we believe applying Lorsa and Cross-Layer Transcoders (Ameisen et al. (0255 to a larger model may reveal lot of surprising behaviors, following the spirit of Lindsey et al. 2025)
odes
Token Embed.
Pos. Embed.
Lorsa Output
borsa.1.4058
Say "odes"
Prev="An
Lorsa.0.426
K Q
An
odes
An
Figure 16: An induction circuit found in our fully sparsified replacement model.
I
AUTOMATED INTERPRETABILITY DETAILS
Evaluation Protocol. Our automated interpretability assessment employs a two-phase explanation- simulation paradigm adapted from Bills al. (2023):
Explanation Phase: GPT-4o generates mechanistic explanations using:
For activation patterns: 8 top-activating token contexts
For z-patterms/DFAs: Contribution graphs to max-activating tokens
2. Simulation Phase: GPT-4o predicts activations/patteems for:
4 top-activating contexts (testing pattern recognition)
4 randomly sampled contexts (testing generalization)
Top Activation Explanation Phase Prompt.
Prompt
We are analyzing the activation levels of features in a neural network, where each feature activates certain tokens in text. Each tokená activation value indicates its relevance to the feature, with higher values showing stronger association. Your task is to infer the common characteristic that these tokens collectively suggest based on their activation values.
Consider the following activations for feature in the neural network. Activation values are non-negative, with higher values indicating a stronger connection between the token and the feature. Summarize in single sentence what characteristic the feature is identifying in the text.
24

---

Published as a conference paper at ICLR 2026
Doní list examples of words. Do not start with "This feature is identifying... Go straight to the explanation.
Sentence 1:
<START>
<lendoftext>><tab>>000
<<ab>>0.0
*/ttab>02
...(iieteed)
<END>
Sentence 2:
(omitted)
Top Activation Simulation Phase Prompt.
Prompt
We're studying neurons in a neural network. Each neuron looks for certain things in a short document. Your task is to read the explanation of what the neuron does, and predict the neuron's activations for each token the document.
For each document, you will see the full text of the document, then the tokens in the document with the activation left blank. You will print the exact same tokens verbatim, but with the activation values filled in according the explanation. Pay special attention to the explanation's 's description of the context and order tokens or words.
Fill out the activation values with integer values from 0 to 10. Don't use negative numbers. Please think carefully. No need to include rationales. Directly start with the first token and do code blocks, i.e.,
Neuron explanation: This feature is indentifying vowels.
Sequence 1: Tokens without Activations:
a<tab>
b<tab>
<<tab>
d<tab>
e<tab>
f<tab>
Sequence Tokens with Activations:
a<tab>>10
b<tab>>0
e<tab>0
d<tab>>
e<tab>>10
f<ta>>0
Neuron 2 explanation: <Autointerp explanations generated in the previous phase>
<Few shot examples>
Pattern DFA Explanation Phase Prompt.
Prompt
We are analyzing the attention map of attention heads in a neural network, where each head attends between tokens in text. Given head and query token, we provide each previous tokens contribution value, with higher values showing stronger association. Your task is to infer the common characteristic of this head that these sequences collectively suggest based on their attention map.
Consider the following attention maps for an attention head. Each line in the format of <token><tab><value>> Query tokens are additionally highlighted with <to-
25

---

Published as a conference paper at ICLR 2026
kenp<tabb<caluee>>ttt****** token** Note that query tokens also attend to themselves. Higher values indicates stronger contribution from this token to the query token.
Summarize in a single sentence what characteristic the head is attending from and to the text. It might be helpful to summarize both the commonality of query tokens and source tokens (if any). It is also recommended to mention this head is often attending to itself.
Doní list examples of words. Do not start with "This head is ... Directly start with the explanation.
Sentence
<START>
<lendofrett><tta>>0000
<tab>0.0
(omitted)
1<taa>>0.0<taa>>=*teeee token**
Pattern DFA Simulation Phase Prompt.
Prompt
We're studying attention heads in a neural network. Each head follows certain attention pattern short document. Your task is to read the explanation of what the head does, and predict the head's attention pattern for each previous token in the document, given specific query token. For each document, you will see the full text of the document, then the tokens in the document with the activation left blank. You will print the exact bution values filled in according to the explanation. Pay special attention to the explanation's description of the context and order of tokens words.
tokens verbatim, but with the contri-
Each line is in the format of <token><tab>. mookn>><ta>***uuee> token >xken*<<ab>>>
Query tokens are additionally highlighted with
Fill out the contribution values with integer values from 0 to 10. Don't use negative numbers.
Please think carefully. No need include rationales. Directly start with the first token and do not use code blocks, i.e.,
Head explanation: This head is attending from one vowel to previous vowels and itself.
Sequence Tokens without Activations:
a<tab>
b<tab>
e<tab>>
d<tab>>
e<tab>**Quee token**
Sequence Tokens with Activations:
a<tab>10
b<tab>>
e<tab>>0
d<tab>>
ectab>**Query t<ken**<tab>10
Head 2 explanation: <Autointerp explanations generated the previous phase>
<Few shot examples>
J THE PATCHING ATTRIBUTION APPROXIMATION BOUND BETWEEN MHSA AND LORSA
Definition (MHSA). attention module, the calculation by MHSA can be formalized as
AMHSA(X) = hx(xx) + ........ + (x),
(1)
where is the number of atlention heads, Red is the input, and hy is the i-th attention head.
26

---

Published as a conference paper at ICLR 2026
Definition 2 (Lorsa). For attention module, the calculation by Lorsa can be formalized as Lorsa (x) Σ
where >> n the number of Lorsa heads, hy is the j-th Lorsa head defined in previous Section .. Pj x TR represent the activation of hi, and Topk is the Topk activation. Specifically, the Topk
activation finction can be expressed by
Py (x(x) is in the top-k activations, 0, Py (x) is not in the top-k activations,
(x),
TopK(p;
From the Linear Representation Hypothesis, assume that the attention head in MHSA can be approximated by the the linear combination of Lorsa Heads, and the approximation error is bounded: Assumption (Linear Representation Hypothesis of Attention). For each MHSA attention head ha there exists a Lorsa head set S; satisfying
AMHSA(x))
Px,(())((x)) (((x,,
jES:
where 6x(x) > O is the approximation error from Lorsa. The approximation between MHSA and Lorsa is bounded, ie., there exists satisfying
|(0MHSA(X) - -L S e, Vx D,
(5)
where D is the dataset.
Previous studies have also referred to this estimation error as dark matter, which inevitable.
Moreover, from the superposition hypothesis, the activation of Lorsa heads is sparse for each input. And, since we initialize Lorsa's s QK module by MHSA, it natural to assume that the Lorsa head will align with the head of specific MHSA. Therefore, we have the below assumption.
Assumption (Superposition Hypothesis). For Lorsa, the activation is sparse, i.e., for any Lorsa head set S, we have
(oxTopK(,((x))h/feee jES
Σ
(6)
where nofTopk is defined similar to Topk in eq.
3
For the MHSA attention head, we have the Lorsa heads sets {S.} eq. partition of the all Lorsa heads, i.e.,
for each MHSA head is
St S, =0, for i for j.
= (1,
(1.2. ...
Therefore, we can prove that, from the perspective of patching, the behavior of the i-th MHSA attention head is approximately equivalent to that of the Lorsa head in S,, i.e., this sparsification does not alter the model's 's underlying behavior in feature-level. First, following the direct logit attribution (DLA) (Wang et al. 2022), we define the influence of the heads MHSA and Lorsa
Definition 3 (Variation for DLA in MHSA and Lorsa). The variation for DLA (VDLA) i-th MHSA heads for the input pair (x,, X.) (x, the reference input, and the X is the counterfactual input transformed from xx) can be defined as
VDLAMHSA(Xx, Xc, x), = f(h((x,)) - f(h((x)));
(8)
where f d + R the composite map for DLA. And we assume that the I is Lipschitz continuous, i.e., there exists Lipschitz bound C > O such that
[f(x) f(y)] S C(x-xx) yl--
(9)
And the VDLA of Lorsa head sets S for the imput pair (Xx, xx)) can be defined as
V DLA Lonsa (X,, X,, S): f
Σ
(x,K(
-
Σ
TopK (xp)(
JES
JES
(10)
27

---

Published as a conference paper at ICLR 2026
The VDLA metric reflects the strength of influence exerted by certain heads in MHSA and Lorsa on model behavior. And we can prove that the influences in MHSA and Lorsa are approximately equivalent by the theorem below.
Theorem 1 (The VDLA Approximation Bound between MHSA and Lorsa), From the Assumption and we have
VDLAMHSAAX,, VVDLA Lorea (X, Xc, S S 20e,
(11)
where e is the error bound defined in Assumption 1 and C is the lipschitz bound of defined in Definition 3
Proof. For the VDLA error, we have
V VDLAMHSA(X,, Xc, - V VVDLAA Lx, Xc. St)l
< [ - ((1x(xx)) f
Σ
Top (x, ( (1x)) +
Σ
(pp (x (x.)) (xo)
jES
jES
- [()x(x,))) -f Σ
(xx + [(0x(x))) -
Σ
Top (Xx)))
jES
jES
00 h1(x;) - - (pp
+C hx(xx) Σ (xx)))
JES
JES
(12)
From the Assumption| T for the first term, we have
hx(x,))
Σ
(x,)(
jES
(xxx)) +
(x,,)))
(13)
jES
{lc((,)]1 +
Σ
jES
From the Assumption 2 for all x E D, have
{ Loors (x)ll = 1
- ((x) Sc.
(14)
j=1
j=1
where the second equality is from (x)) > 0. Therefore, we have
cj(x) Se
(15)
From the eq. and eq. 15 we have
h - (xx)))
jES
(16)
Sct
Σ (xx
jES
Then, from the Assumption
we have
h hx(x,))
TopK(p, (x.)))))))
2 e.
(17)
jES
28

---

Published as a conference paper at ICLR 2026
Similarly, for the second term in eq.
12 have
(x,))
TopK (hx
E c.
(18)
jES
Substituting eq. and eq.
18
into eq.
12
we have
(x VDDA Lorsa (xr Xzy Sill 2Cc.
(19)
The proof is completed.
From the Theorem D we obtain the following corollary.
Corollary 1. For the dataset Dr, where (X,,Xa) ~ Dr, X, ~ D is the reference input, and X, is counterfactual input transformed from x,, D the original input dataset, we have
E VVL (x,, xx, V (x,, X,, S.) 2Cc
(20)
29


Reviews:
[{"review_id": "nDAdgeTlT3", "reviewer": "Reviewer_um4H", "summary": "The paper proposes a sparse replacement model of the attention layer,  Lorsa, to understand the attention   mechanisms. The authors demonstrate that Lorsa can rediscover previously reported mechanisms such as induction heads and attention sinks, and even identify new patterns such as subtoken induction heads.", "questions": "- Is it possible to use existing methods (e.g. SAE) to identify the same newly discovered attention mechanisms reported in this paper?\n- Could the authors provide more details on how well autointerp performs, and clarify whether it can be trusted as a quantitative metric for comparing interpretability across different methods in Section 5.3?\n- How are gradients propagated through the Top-K operation during training?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed Lorsa method successfully rediscovers known attention mechanisms such as induction heads and attention sinks, and also discovers new interpretable phenomena (e.g. subtoken heads and arithmetic Lorsa-heads. I'm not very familiar with the literature, but the proposed approach appears to be novel. \n- The paper is well organized and is easy to follow.", "weaknesses": "- The work may be better if including motivation and ablation study of several of the design choices, especially the difference with SAE, e.g. predicting the downstream activations, the top-K operation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AeDyOCgfZL", "reviewer": "Reviewer_ju5a", "summary": "This paper introduces Low-Rank Sparse Attention (Lorsa), a method to decompose transformer attention into sparser, more interpretable components. It effectively addresses \"attention superposition,\" rediscovering known mechanisms like induction heads and identifying novel ones (e.g., arithmetic heads). The design is scalable and evaluation is thorough.", "questions": "What might the unreconstructible \"dark matter\" in the attention output represent?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper clearly articulates the problem of \"attention superposition\" as a key obstacle to interpretability, drawing a compelling analogy to superposition in MLP layers. This provides a strong theoretical motivation for the work.\n\n2. The design (rank-1 OV circuits, top-K sparsity, QK weight sharing) is effective and scalable.\n\n3. Comprehensive experiments, including scaling laws, automated interpretability, and compelling case studies on models up to Llama-8B.", "weaknesses": "1. The primary limitation is that QK circuits are shared across heads, potentially hindering full disentanglement and causal attribution.\n\n2. Evidence for novel head functionalities is mostly correlational; more causal intervention experiments would strengthen the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QvYtWRNPJG", "reviewer": "Reviewer_g9Wx", "summary": "This paper introduces Low-Rank Sparse Attention (Lorsa), a sparse replacement model for MHSA designed to tackle the problem of attention superposition. To do that, the authors decompose the OV circuit into a large overcomplete set of sparse rank-1 heads. The work demonstrates that this method successfully rediscovers known mechanisms (like induction heads and successor heads) and uncovers novel, finer-grained units (such as subtoken induction heads and specific arithmetic heads). The paper provides quantitative analysis showing that the interpretability of Lorsa heads is comparable to that of Sparse Autoencoder (SAE) features, positioning Lorsa as a promising new tool for the mechanistic interpretability of attention layers.", "questions": "**Questions:**\n\n1. Could the authors please clarify the difference between Algorithm 1 and the text description? \n2. Could you please clarify the notation issues mentioned in the Weakness section?\n3. The FVU values, while comparable to SAEs, are non-trivial. Appendix H introduces the Lorsa Dark Matter hypothesis, suggesting this error is structured and shared with SAEs. This is a critical point for the entire replacement model paradigm. Could the authors elaborate on the implications of this? If 10-30% of the variance is unexplained and potentially uninterpretable by replacement models based methods, how does this limit the functional completeness of the discovered units and the conclusions we can draw from them?\n\n\n**Minor comments:**  \nThere is a typo in line 126: “privleged”", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses attention superposition, a critical but less-studied counterpart to feature superposition in MLPs. The application of sparse dictionary learning to decompose attention head outputs (specifically via rank-1 OV circuits) is a novel and significant methodological contribution, providing a much-needed alternative to SAEs for interpreting attention.\n- The method is well-validated. The authors successfully recover a wide range of previously identified attention mechanisms, which serve as a strong sanity check. More importantly, the discovery of novel, fine-grained units like \"subtoken induction heads\" and families of \"arithmetic-specific Lorsa heads\" demonstrates the utility and resolving power of the method.\n- The authors provide a strong quantitative benchmark by comparing Lorsa's interpretability directly against SAEs using automated interpretability methods. The paper also includes valuable ablation studies (in the appendix) that explore the architectural design choices, such as the impact of QK dimension and initialization.", "weaknesses": "- The notation is confusing:\n    - The total number of Lorsa heads is denoted as $H$ in Algorithm 1 (line 140) but as $N_{Lorsa}$ in the main text (e.g., line 185). This is compounded in Section 4.1, where $N$ is used for the number of parameters.\n    - The term $D_{QK}^{Lorsa}$ is used in Section 3.1 (line 169) without any prior definition. Its usage is also conflated: it appears to mean a dimension in line 169 (\"...as $D_{QK}^{Lorsa}$ decreases\") but a count of heads in line 171 and the Table 1 caption (\"...across every $D_{QK}^{Lorsa}$ heads\"). This makes the QK sharing mechanism difficult to parse.\n- There is a direct contradiction between the pseudocode and the main text describing the model's architecture:\n    - Algorithm 1 explicitly shows per-head query and key weights ($W_q^h, W_k^h$), implying no parameter sharing.\n    - Section 3.1 (lines 168-183) explicitly describes the opposite: a parameter-sharing mechanism where groups of Lorsa heads share a single QK circuit. This made the paper unclear while reading, as the QK sharing mechanism is a core part of the Lorsa design.\n- Some important limitations and conceptual discussions are in the Appendix. I understand that this could be because of spacing issues, so I would definitely recommend having these discussions in the camera-ready version of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LgziCuoWXk", "reviewer": "Reviewer_8qJU", "summary": "This paper studies attention superposition by introducing **Low-Rank Sparse Attention (Lorsa)**—a per-layer module trained to **reconstruct** an MHSA layer’s output as a top-$K$ sum of many rank-1 OV “attention units,” with shared QK parameters to control cost. Training minimizes an MSE objective against the layer’s MHSA output (Algorithm 1), yielding a sparse, overcomplete basis over the attention-output space that can be analyzed with standard interpretability tooling. The authors apply Lorsa to Pythia-160M and Llama-3.1-8B, report fidelity–sparsity scaling and per-layer FVU, and present qualitative/automated evidence that Lorsa rediscovers known head families (induction, successor, sinks) and surfaces new phenomena (subtoken-induction and arithmetic-specific heads), while broadly tracking SAE trends in reconstruction and interpretability metrics. Lorsa is compatible with RoPE and GQA through straightforward adaptations.", "questions": "- On theoretical grounding, can you provide a result formalizing when Lorsa is a faithful surrogate for studying MHSA? For instance, assume a layerwise error bound $||\\text{Lorsa}(x)-\\text{MHSA}(x)||\\le \\varepsilon$ and conditions on QK similarity; then prove stability of **linear readouts** and **attribution functionals** (e.g., path-patching scores, projection-based attributions) to errors of size $\\varepsilon$. A theorem of the form\n   $$\n   \\mathbb{E}_{x\\sim \\mathcal{D}}|| \\mathcal{A}(\\text{Lorsa};x) - \\mathcal{A}(\\text{MHSA};x) ||_p \\le C\\cdot \\varepsilon,\n   $$for a relevant class $\\mathcal{A}$ would directly justify that we can study MHSA via Lorsa. Clarifying how shared-QK and Top-K selection affect such stability (e.g., via Lipschitz or margin assumptions on head activations $z$) would be especially valuable.\n- I am not an expert in this field, so Id like to refer to other reviewers for the final recommendation.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "First, the methodological design is original yet well-justified: the rank-1 OV constraint sharpens semantics; shared-QK binding preserves QK dimensionality where needed and avoids the degradation observed when QK is reduced below the MHSA head dimension. The top-$K$ sparsity yields monosemantic components that are easy to inspect. The paper provides clear algorithms and ablations supporting these choices.\n\nSecond, the presentation empirical analysis is substantive and yields new insights. Lorsa tracks SAE scaling and per-layer patterns, recovers canonical heads, and identifies novel mechanisms (e.g., arithmetic-specific and subtoken-induction heads) in Llama-3.1-8B, with preliminary causal probes that connect heads to behavior.", "weaknesses": "- My main concern is with the paper’s positioning: the manuscript repeatedly states that “Lorsa serves as a replacement model for Transformer attention”. However, the scope of evaluation is strictly __per-layer reconstruction__ and interpretability; there is no end-to-end evaluation or deployment-oriented analysis. As written, the “replacement” phrasing risks misleading readers about the paper’s primary contribution, which is pattern understanding of MHSA via decomposition.\n    - A related, practical limitation is the scale of the Llama-3.1-8B setting: ~32K Lorsa heads per layer with (K=128) and ~512M parameters per layer (vs. 64M for MHSA) make the module expensive, and training (more like reconstructing for approximation) each Llama layer’s Lorsa takes ~24 A100-hours under the stated batch size. This underscores that the artifact is currently best viewed as an interpretability tool rather than a deployable substitute .\n- A second concern is _behavioral equivalence vs. output approximation_. I am not totally convinced that closely approximating MHSA outputs implies similar underlying behavior. For instance, two architectures (A) and (B) can be trained to produce near-identical outputs while differing substantially in inductive biases and internal mechanisms. Although Algorithm 1 shows Lorsa as a simplified Top-K variant of MHSA, architectural changes e.g. rank-1 OV, shared-QK, and Top-(K) gating may alter how features are represented and composed. Thus, even if $||\\mathrm{Lorsa}(x)-\\mathrm{MHSA}(x)||$ is small, approximation alone does not justify studying them “as the same”; a theoretical link (e.g., bounds ensuring stability of feature-level attributions or circuit-level predicates) is needed, particularly given superposition and shared-QK.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)