Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures
Authors: Junxuan Wang
Xuyang Ge
Wentao Shu
Qiong Tang
Yunhua Zhou
Zhengfu He
Xipeng Qiu
Venue: ICLR 2025 Poster (2025)
Keywords: Mechanistic Interpretability
Sparse Autoencoders
Universality
State Space Models
Primary area: interpretability and explainable AI
TL;DR: With sparse autoencoders, we find Transformers and Mambas are basically similar in their internal representation.

Abstract: The hypothesis of \textit{Universality} in interpretability suggests that different neural networks may converge to
implement similar algorithms on similar tasks. In this work, we investigate two mainstream architectures
for language modeling, namely Transformers and Mambas, to explore the extent of their mechanistic similarity.
We propose to use Sparse Autoencoders (SAEs) to isolate interpretable features from these models and show
that most features are similar in these two models. We also validate the correlation between feature similarity
and~\univ. We then delve into the circuit-level analysis of Mamba models
and find that the induction circuits in Mamba are structurally analogous to those in Transformers. We also identify a nuanced difference we call \emph{Off-by-One motif}: The information of one token is written into the 
SSM state in its next position. Whilst interaction between tokens in Transformers does not exhibit such trend.

Published as a conference paper at ICLR 2025
TOWARDS UNIVERSALITY: STUDYING MECHANISTIC SIMILARITY ACROSS LANGUAGE MODEL ARCHITEC- TURES
Junxuan Wang Xuyang Ge
Wentao Shu!
Qiong Tang
Yunhua Zhou
Zhengfu He Xipeng Qiu
'OpenMOSS Team, School of Computer Science, Fudan University Shanghai Innovation Institute
junxuanwa @m. fudan. aadan.edd... (z izeee19 9, xpqiu) u) fudan. edu, on
ABSTRACT
The hypothesis of Universality in interpretability suggests that different neural networks may converge to implement similar algorithms on similar tasks. In this work, we investigate two mainstream architectures for language modeling, namely Transformers and Mambas, to explore the extent of their mechanistic similarity. We propose to Sparse Autoencoders (SAEs) to isolate interpretable features from these models and show that most features are similar in these two models. We also validate the correlation between feature similarity and Universality. We then delve into the circuit-level analysis of Mamba models and find that the in- duction circuits in Mamba structurally analogous to those in Transformers. We also identify nuanced difference we call Off-by-One motif: The information of one token is written into the SSM state in its position. Whilst interaction between tokens in Transformers does not exhibit such trend.
INTRODUCTION
The field of mechanistic interpretability seeks reverse engineer real-world neural networks into interpretable features and circuits. Due to the massive societal importance of Transformer language
models (Vaswani et al. 2017 Brown al. 2020 Ouyang al. 2022 Sun al. 2024), existing
literature in this field mostly focuses on Transformers (Elhage et al. 2021 Olsson et al. 2022 Wang
al. 2023). This poses a natural research question:
Will mechanistic findings in one model transfer to others? The answer to this question may tell interpretability researchers how much of their current understanding will be of use the future. The conjectural transferability of mechanistic understanding of specific model is called the Uni- versality Hypothesis (Olah et al. 2020, Chughtai al., (023). Concretely, there are several aspects of Universality that may be of interest: (1) Training: Will models with different training settings converge to similar representational structures? (2) Model Size: Are features and circuits in smaller models present in larger models? If so, what do larger models build further these structures? (3) Model Architecture: How similar is the mechanism in models of different architectures? Does the universal part teach more general motif of neural networks and language modeling? Does the divergence indicate the pros and cons of each architecture? (4) Multilingualism Cross-Modality; Are neural networks able to obtain similar knowledge from different languages modalities?
This work mainly focuses on Architectural Universality. We believe the answer to this, if exists, lies somewhere in the middle of the spectrum. It may not be the case that all models are isomorphic, nor that they implement totally distinct algorithms. If the commonality of features and circuits across models of different architectures is revealed, we may be more confident to generalize some of our current mechanistic findings in pretrained Transformer language models to perception of the
"Corresponding author.

---

Published as a conference paper at ICLR 2025
task of language modeling. Pragmatically, such analysis also sheds light whether one family of models fail to represent some certain type of features or implement some circuits present in another. This may help provide a microscopic comparison among model architectures Vaswani et al. 2017 Peng et al. 2023 Sun et al. 2023 Gu Dao 2023).
To this end, mainly investigate two of the most popular types of models for language modeling, namely (decoder-only) Transformers and Mambas (Gu & Dao 2023). family of models based on State Space Models (SSMs).
We manage to bridge these two models with features extracted by Sparse Autoencoders (SAEs) in an unsupervised manner. Qualitative and quantitative experiments show that these family models learn lot of similar features. We also utilize our SAEs to investigate induction circuits (Olsson al. 2022) in Mamba, which are also highly analogous to ones in Transformers.
However, the mechanism in these two models is nuanced by what we call Off-by-One Preference in Mamba. We find that Mamba often writes the information of a specific token into the SSM state in its next token. We show that this is done by the local convolution layer.
Our main contribution can be summarized follows. (1) To the best of our knowledge, this the first work to look for linear, interpretable features with Sparse Autoencodersand quantitatively eval- uate the similarity between Transformer and Mamba features (Section 4 (2) We propose novel metric to isolate and quantify feature universality with respect to architectural changes (Section and 5). (3) We perform circuit analysis Mamba and qualitatively reveal structural analogy and some nuances compared with Transformer circuits (Section 6)
2 RELATED WORK
Representational Universality. introduced by Olah al. 2020) and has been extensively studied representational universal- ity (Klabunde et al. 2023 Paulo et al., 2024). Raghu et al. (2017) and Kornblith et al. (2019) measured representation similarity using canonical correlation analysis, while other methods (Ding et al. 2021 Khrulkov & Oseledets) 2018 BBox-Adserà et al. (002)) quantify similarity across neu- ral networks. Efforts also include examining neuron correlations identical models with different seeds (Li et al. 2016 Wang et al. (2018; Gurnee et al. 2024), with recent work suggesting conver- gence across modalities (Huh al. (024).
The concept of Universality in mechanistic interpretability was
Circuit Universality. Circuit-based mechanistic interpretability was pioneered by Cammarata (ct al. (2020); Olah et al. (2020); [Ehhaee et al. (202I), investigating whether features and circuits analogous across models. Olah et (2020) showed early layer similarity in vision models, with further studies extending to MLPs and Transformers on algorithmic tasks (Chughtai et al.) (002)).
Sparse Autoencoders. Neuron-based analysis often hampered by polysemantic neurons (El- hage et al. 2023 2022b). SAES (Bricken et al.) 2023 Cunningham et al. 2023 Templeton et al. 2024) extract monosemantic sparse codes from activations, providing interpretable features. SAEs have been applied to language models (Rajamanoharan et al. 2024 Ge et al. 2024 and tasks like Othello move prediction (He al. 2024).
Mamba Interpretability. Mamba (Gu & Dao 2023 Dao & Gu 2024), family of models based on State Space Models (SSMs), is relatively new in interpretability studies. Ali et al. (2024) found that Mamba implicitly attends previous tokens via its B. C matrices. Studies have explored acti- vation patching on Mamba (Meng et al. 2022 Sharma et al. 2024), with a unified view of Trans- formers and Mambas emerging (Zimerman et al. 2024
Dao & Gu 2024).
2

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES
3.1 MODELS
We choose to study Pythia-160M, (Biderman et al. 2023) and an open-source version of Mamba- 130M" Pythia-160M uses an untied word embedding and unembedding, leading to about 30 million (i.e. vocabulary size by hidden dimension) more parameters. They adopt the both are trained on the Pile dataset (Gao et al. 2021). Unless otherwise specified throughout this paper, we refer to these two models Pythia and Mamba. Since Mamba adopts GAU-type archi- tecture (Hua et al. 2022) which implements both token and channel mixing in one Mamba block, has twice the depth (i.e. 24 layers) as Pythia (i.e. 12 layers). We also include experimental results for the 2.8B models of both families in Appendix D.5 to demonstrate the generalizability of our findings.
tokenizer and
3.2
A BRIEF MATHEMATICAL FRAMEWORK OF MAMBA CIRCUITS
As shown in Figure Mamba architecture incorporates a number of Mamba Blocks. This architecture adopts a layer-stacking approach, akin to the stacking of Transformer blocks in Transformer models with residual connections employed between layers. Each block takes in copy of the residual stream r (l-1), which is used to independently com- pute the gate output g," and SSM output s!") which are merged with a Hadamard product and written into the residual stream with a linear transformation :
SSM
Conv
r r (1-1) + (s!) ())
(I)
The gate output 9 is simply linear projection followed by non- linear activation:
SiLU(W(",.""))
(2)
1-1
State space model outputs stand as the core of Mamba computation, which performs inter-token information mixing. Specifically, a local convolution layer with a window size of deom is applied to linear pro- jection of r
Figure 1: A simplified Mamba block.
((-1).
(1) 2 w-(!),.! (1-1),
SiLU(Conv (1) 22--1 (i-dmn+1))
(1)
(4)
After this local information aggregation, subsequent SSM layer performs global token mixing, which combines its current state h and updates (n") with e Since all variables and compu- tation are restricted at layer I, we omit the superscripts in the following equations.
l: = Fa(c) h1-1 + Fo(c),
(5)
8; = h,,(WW Wa G.
(6)
Concretely, Fa and F3 implement the data-dependent Selection Mechanism for Mamba, which to some extent similar to forget gate and input gate in LSTMs, respectively. W. and Wa decide the output of this SSM layer, much like output gate. Such analogy provides analytic simplicity but also vagues a number of details of Mamba architecture. We refer readers to Gu & Dao (2023)
and Dao & Gu (0044) for further details.
https://hugginggacccc state- spaces/nanba htt..///ttaattssppceeeaaaaaaaaaa
1330m

---

Published as a conference paper at ICLR 2025
3.3 SPARSE AUTOENCODERS
Our SAEs have only one hidden dimension larger than the input dimension (i.e. > D), with the training objective of reconstructing any given model activation and an Ll penalty on its hidden layer to incentivize sparsity. An SAE can be formulated as follows:
f((x) (R.. b
(7)
Wide
(8)
where x RD is the hidden activation decomposed with F features. Wene Px and W DxF are the encoder and decoder of an SAE. is linearly mapped onto the feature space by the encoder and bias dene, followed by a ReLU to ensure that f(x) (i.e., feature activations) is nonnegative.
SAEs are trained to reconstruct the original activation with a linear combination of the decoder columns (i.e. features), determined by [xx). We also set an I.1 sparsity constraint on f(x) to obtain a sparse coding of each activation. Our training technique is known as vanilla SAEs. We include results of more performant variants, i.e., TopK-SAEs (Gao et al. 2024).
CROSS-ARCHITECTURE UNIVERSALITY WITH FEATURE SIMILARITY
4.1 IN SEARCH OF INTERPRETABLE PRIMITIVES
We seek address this Universality problem in fine-grained manner by comparing models with an independent decomposition of the activation space. We see this kind of microscopic comparison first pursued by analyzing correlation between individual MLP neurons in models with the identical architecture (Li et al. 2016 Gurnee et al. 2024).
However, there is hardly counterpart of MLP neurons in Mamba. This suggests the necessity to utilize tools invariant to architectural changes to study architecture-agnostic similarity. To this end, leverage trending tool in interpretability called Sparse Autoencoders to extract more monose- mantic units (i.e., features) from the models' activation.
4.2 MATCHING FEATURE PAIRS ACROSS ARCHITECTURES
4.2.1
TRAINING SPARSE AUTOENCODERS
We trained SAEs in the residual stream after each Transformer block in both models, getting 12 and 24 SAES in Pythia and Mamba, respectively (See Section 3.1). Both models have a hidden dimension D 768. We the hidden dimension of SAEs F 32D 24576 for all Sparse Autoencoders (SAEs). We refer readers to Appendix A for more details of our SAES.
4.2.2 MAIN EXPERIMENTS
SAE features are analogous to MLP neurons in that they both linearly correspond to a direction and has (almost) nonnegative activations. Thus, we can follow Li et al.) (2016); [Bau al.) (2019) and Gurnee et al. (2024) to evaluate feature similarity by identifying feature pairs activated on the same inputs.
We iterate over million tokens sampled from SlimPajama and save the activation of each SAE feature in both models. For each SAE feature Pythia, we computed the maximum Pearson cor- relation among all features for all SAEs across all layers in Mamba, which we refer to the Max Pairwise Pearson Correlation (MPPC).
B[(v 1 (V m
max
(9)
where vP, vi E N are the activation pattern of the i-th feature in Pythia and the j-th feature in Mamba. We compute the mean and variance for each pattern, denoted by ni 137 of and ofy.
4

---

Published as a conference paper at ICLR 2025
It is worth noting that the superscript in p implies that looking for the Mamba feature most correlated to the i-th Pythia SAE feature. However, this operator is not commutative. We use this metric throughout this paper our main experiment. To show that this unilateral analysis does not affect our conclusion, we include all reverse results, i.e., pm in Appendix C
4.3
BASELINES AND SKYLINES
4.3.1 RANDOM AND NEURON BASELINE
When computing MPPC, its value reflects the maximum from numerous samples, introducing a positive bias. Additionally, SAE properties like sparsity constraints can further shift the distribution towards a higher value. To mitigate this, we established baseline by calculating MPPC for each Pythia feature against random SAE with matching feature count and sparsity level (by masking all but the Top K activating features) to the Mamba SAE.
As noted in Section 4.1 it does make intuitive sense to compare architectures in neuron basis. We empirically validate this by computing the correlation introduced in Equation 9 between all Pythia MLP neurons and all Mamba post-SiL.U activations.
4.3.2 HIERARCHY DIMENSIONS OF VARIATION
Studying universality across neural network architectures with SAEs implicitly discusses a number of dimensions of variation. We believe there are three levels of variables with some relation of inclusion.
SAE Variables: (Innermost area in Figure 2 It is important ensure that SAEs actually learn its latent features in a replicable manner. That is, we have sweep hyperparameters for training SAEs the same activation space and make sure they mostly learn similar features. SAE features can serve our purpose only if SAE itself do not introduce much randomness in its learned features. Such invariance property needs to be validated with changes to SAE initialization, training batch size, learning rate / schedules, number of training tokens and order of training data, etc.
Architectural Variables
Language Model Training Variables
SAE Training Variables
Figure 2: Hierarchy of invariance.
Language Model Variables: (Middle area) Similarly, language model training a stochastic progress and may further introduce uncertainty
the model's internal. Model variables here include data distribution, training tokens elapsed and training hyperparameters similar to SAE variables. Training SAEs on different models must give different results since at least their training data (i.e. model activations) are different.
(Outermost area) Most aforementioned model variables, especially ones related to initialization and training, are implicitly included in architectural changes. For ex- ample, certain that we cannot find an identical initialization or training trajectory for Mamba models and Pythia models.
Architectural Variables:
To this end, we propose two skylines to indirectly estimate the marginal effect of each area by ablation.
Skyline 1: Different Models of Identical Architecture (Model Seed Variant). For each feature in Pythia, we look for similar features in another Pythia model trained with a different random seed. This changes model initialization and training data order. The intuitive motivation of this is that Mamba model has all differences the seed variant has plus the architectural change.
Skyline 2: SAEs Trained on the Same Language Model (SAE Seed Variant). We also trained another set of SAEs on the same Pythia model as higher skyline. This setting further constrains the language model initialization and training factors. The only difference lies in SAE training. The discrepancy between SAEs trained the same model tells how convergent SAE features are.
5

---

Published as a conference paper at ICLR 2025
Distribution of Max Pairwise Pearson Correlation For SAE Features
Experiment
00
Averege 0.881
Pertentage (54)
Skyline1:
Average 0.725
SkylineZ:
Average 0.806
19
Random Baseline
P(Averege 0.196) 1%
Comelation
Distribution of Max Pairwise Pearson Correlation For Neurons
Experiment
Average 0.291
Random Baseline
Average 0.078
Correlation Value
(b)
MPPC Difference Distribution
Depth Distribution of Matched Features
2.23
0.07
Progortin
Pythia Layer
Porcertage
20
MPPC Difference
Mamba Layer
Figure 3: (a) MPPC distribution for SAE features: main experiment, model seed variant skyline, SAE seed variant skyline, and random baseline. (b) MPPC distribution for neurons: main exper- iment vs. random baseline. (c) Distribution of MPPC differences for SAE features, computed as main experiment minus model seed variant skyline. (d) Layer-wise frequency of best-matching fea- ture pairs, mostly concentrated near the diagonal (similar depth). The anomaly in Pythia's 's final layer may result from significant residual stream differences, reducing SAE quality.
4.4 RESULTS
Figure 3a
presents the results of all MPPC experiments SAE features, namely random base- line, Pyth SAE similarity (main experiment), model seed skyline and SÁE seed skyline. Figure
3b presents the results of random bascline and main experiment on neurons.
Significant Cross-Architecture Similarity of SAE Features Neurons show much lower simi- larity across architectures, with fewer than 15% achieving MPPC 0.4, compared of SAE features exceeding 0.7. This supports the hypothesis that comparing privileged directions across models is uninformative (Elhage et al., 2023). In contrast, SAE features in the main ex- periment significantly outperform the random baseline, with the mean MPPC: value exceeding the 99th percentile of the mean MPPC distribution for the random baseline, demonstrating statistical significance. These results highlight the strong cross-architecture similarity of SAE features.
55%
Similarity Slightly Falls Short of Skylines. Cross-arch SAE MPPC has an average of 0.681, compared 0.725 for model seed variant and 0.806 for SAE seed variant. More than 25% Pythia features find their matched feature in Mamba with an MPPC greater than 0.95.
We repeated the baseline 16 times and calculated the mean MPPC for each repetition to fit this distribution (assuming the MPPC follows normal distribution).

---

Published as a conference paper at ICLR 2025
We illustrate the distribution the difference between our Cross-arch SAE MPPC and the model seed variant skyline in Figure 3c i.e. pf. P "p where indexes a feature in Pythia, ranging from to 12 & 2 One can interpret this as the marginal effect of architectural changes. 90% of Pythia features fall in the interval of 20. to 0.07 under this metric. More than 60% features have near-zero MPPC difference(absoluce value < 0.05), which means in our case, they are consistently present after architectural changes.
Our findings provide strong evidence that SAE features exhibit universality across architectures. We further validated our method and conclusions in additional settings, including alternative model architectures, larger models, varied SAE hyperparameters, advanced SAR designs, and diverse datasets (see Appendix D for details). However, using SABs for feature decomposition is rela- tively new approach, and these conclusions rely on specific assumptions. For a detailed discussion of limitations, refer the limitation section(Appendix
4.5 DEPTH SPECIALIZATION
Another interesting phenomenon is that SAE features form similar hierarchy in different architec- tures. Our findings extend the scope of neuronal depth specialization (Gurnee et al. (024) where the authors find universal neurons to be present near layers. As illustrated in Figure 3d if feature in Pythia is located in the 1-th layer out of 12 layers, its matched feature in Mamba is most likely to appear around layer 2/ in Mamba (out of 24 layers). This provides evidence for more macroscopic conjecture of language model, i.e. the general structure of a specific language model internals is a stretched or sealed version of a universal motif (Elhage et al., 2022a Huh al. 2024).
5 A COMPLEXITY-BASED INTERPRETATION OF SIMILARITY DIFFERENCE
We demonstrate in Section 4 quantitatively that great number of Pythia features can be paired with one Mamba feature in terms of MPPC. It is further required to map such statistics onto our hypo- thetical Universality. Anecdotally, obeserve that higher MPPC feature pair does not necessarily mean more semantic similarity than lower ones due to the influence of feature complexity. This offers us a new perspective of the aforemen- tioned MPPC difference between our Cross-Arch SAE MPPC and the seed variant skyline (Section 4.3): such difference can be deemed as cancelling out the effect of feature complexity.
pythia
mamba
L6. 2982
phrose less".
L12. 609
phrose "nore or less".
2'm Seas Indi/ffrent. conseieece fortee commess con weee hoo conatttt gronth. their indapenderx (pure leso,
Nes wore less takle Face wilue less this
0.830
equst
the Indiecting Toration plocement.
L1. 16964
"ia" os en inficoner lecatian.
L1 18557
682
Chas University
driveeby svoeting Necoce
Nia hoos Rassia tuen norther Jandon
vonkiing En nadio St Bouatoe
1952
Rarward University Contridge
L7 12760
specific intellectuel parts there.
L19. 2275
rultisyllobic
0.355
Pritiing Myctegtes
Eattce Rhegress, +UBnSSe Cojus
Syntogee de Coey Gode")
callaal "orbbooaani hotic Defander
concept beting, being.
"was" along srsoctooeed preceding
L10. 880
L19 4876
0.295
ahe auganssing soople tine-vorasd cetoes aece giing in cost ue seaTE sapoosed to
doan't ratter
manager
880 -hohhess and 1o
gree"
followking
We manually inspected a number of matched feature pairs across archi- tectures and seed variants. An in- triguing trend is summarized as fol- lows and also illustrated in Figure
Figure 4: We present four cases of feature pairs with dif- ferent Pearson correlation values beside. To the right of each feature index is the auto-interpretation result gener- ated by GPT-4o, with activation examples shown below.
"This the number of layers multiplied by number of features in each SAF
7

---

Published as a conference paper at ICLR 2025
5.1 CASE STUDY
Higher MPPC pairs are mostly simple features. Features with a maximum pairwise Pearson correlation close to are typically very simple, activating exclusively on individual tokens. These include wide range of linguistic units such as all parts of speech, word prefixes, suffixes, letters, punctuation marks, and digits (the first row in Figure
Lower MPPC pairs tend be more complex. As the complexity of features increases, the MPPC scores of most matched pairs fall in the interval 0.3 < / < 0.8. Intuitively, both features fire at the same high-level structure, it more likely that the activation magnitude of one feature is larger for some specific examples. While another feature may instead prefer some other instances, leading to inconsistency firing strength but agreement in theme. We provide further discussion in Section 5.2
Ultra-Low MPPC pairs are mostly uninterpretable. Features with extremely low maximum pairwise Pearson correlation typically uninterpretable and considered meaningless, possibly aris- ing from noise introduced during SAE training or model training. We provide further discussion in Section 5.2
5.2
QUANTIFYING FEATURE COMPLEXITY AND MONOSEMANTICITY
MPPC Statistics Under Different Levels Variation and Complexity
Statistics Different Levels of Moncsemanticity
0.8
MPRC
0.6
0.4
0.2
2
3
Auta Labeled Monosemanticlty Score
Auto Labeled Complexity Score
(a)
(b)
Figure 5: (a) Distribution of MPPC auto-labeled complexity scores ranging from 1 (simple) to 4 (complex). Both Model Seed Variant and Cross-Arch SAE MPPC exhibit correlation, while in SAE Seed Variant weaker since the mean MPPC score is smaller than that score 2. (b) Distribution MPPC over monosemanticity scores ranging from (Polysemantic) to (Monosemantic). Monosemanticity scores almost divide MPPC into two interval categories about 0.2.
We are aware that complexity and monosemanticity is somehow abstract property of a feature. To effectively validate our conjecture, we utilize GPT-4o to automatically evaluate the complexity and activation consistency of each feature's ' activation pattern.
We mainly follow Bills et al.) (2023) and [Cunningham & Conerly (2024) prompt GPT-4o to score the degree of complexity and activations consistency of each feature. The activation pattern on the first 10 top activating samples of each feature are fed into GPT-4o in a structured format (Appendix E It then returns integer score ranging from to 4. For complexity, score of indicates the simplest features, while a score of 4 represents the most diverse contexts with a coherent theme. We validated the effectiveness of GPT-4''s scoring by comparing it against human expert evaluations (Appendix!! !.
We observe clear correlation between automatically labeled complexity scores and seed variant sky- line MPPC, i.c. p as well as Cross-Arch SAE MPPC, as shown in Figure 5a Statistically, the simpler Pythia feature is, the more likely it matches another feature in model seed variant or Mamba with a high Pearson Correlation score.
This offers another interpretation of isolating architectural variables with MPPC difference proposed in Section 4.3.2 Cross-Are SAE MPPC reflects a mixture of several variables. One of them the semantic complexity of features, which is present in both Cross-Arch SAE MPPC and seed variant
8

---

Published as a conference paper at ICLR 2025
skyline MPPC. Subtracting the latter from the former can be deemed cancelling out the effect of feature complexity.
The trend for the Monosemanticity Score (Figure 5b is ary around 0.2. Features with MPPC values below this threshold are highly likely to be uninter- pretable. We refer readers to Appendix F for some cases.
pronounced, with clear bound-
5.3 IMPLICATIONS TO MODEL DIFFING METHODS
The complexity issue of MPPC confounds the true different features and those matched but with an abstract theme. Thus, fail to identify any architecture-specffi feature. Our complexity view- point of MPPC raises a broad concern for cross-architecture model diffing. For example, a recent promising model diffing method crosscoder (Lindsey et al., 2024) is also prone false positive of divergent features, where features activating on the same concept might be identified as differ- ent ones due to their respective preference for specific instances. This suggests the need for sanity checks of subsequent model diffing methods.
ZoOMING OUT TO INDUCTION CIRCUIT UNIVERSALITY
We also want to look into larger scale of universality by studying the circuits in Mamba and Transformers. We choose to start from a relatively simple and well-studied circuit named induction circuit (Olsson et al. 2022).
The in-context learning behavior is when sequences with pattern [A][B]..[[A] appear, the model is very likely to output [B] in the subsequent step to complete the 2-gram with knowledge from the past. has been shown that this is closely related to a kind an algorithm mainly implemented by two attention heads in Transformers and reoccurs broadly in many Transformer language models (Olsson et al. 2022
Todd et al., 2024).
It has been already demonstrated by Fu et al. (2023) Gu & Dao 2023). inter alia, that Mamba is also capable of in-context learning. An important curiosity is how Mamba implements this algo- rithm since it does have attention heads. We find that Mamba employs similar mechanism Transformers, but with some slight differences in its implementation.
6.1 IDENTIFYING INDUCTION CIRCUITS IN MAMBA
We employ a circuit discovery method called path patching (Wang et al. 2023) to search for which components in Mamba directly affecting the output logits, shown in (F)
Clean Run
Corrupted Run
Path Patching
O+O-O+
clean path
corrupted path
O-0-0-
Path patching replaces the activations along specified paths and allows this influence to propagate along the al- lowed path then observe the difference of interest. This measures the counter- factual influence of the specified paths only through allowed paths. The data results for all experiments in this sec- tion are provided in Appendix G
recomputed path
A O-0---
Figure 6: Path patching includes three times of forward pass. We patch a specified path (red edges) with a cor- rupted path and uses the recomputed activations only through the paths of interest (yellow edges). Activations through other paths (green edges) are fixed the same as the clean run.
Layer 17 SSM States Are Key in Induction. We label key positions in [A][B]..[[A] as A1. B1, and Ag, and start with a counterfactual input [A]]B'...[A], Path patching is applied to Ag for I layers (0-24), allowing the influence propagate except through other SSM states. Patching layer 17 results in the largest logit drop, with layer 20 having a weaker effect and other layers mostly irrelevant.

---

Published as a conference paper at ICLR 2025
Key Information Flows into SSM States at Token B+1. We next patch components of h (17)
A
(17) which depends on (17) AA2-1 (17) Results (Figure 7 show patching CB,+ significantly reduces the logit for B, with no substantial changes at other positions.
Local Convolution Aggregates A and B into B+1. Finally, patch components of (17)
CB.+1
linear combination of "B,-1 FB. 2.A (17) and x "'A--
(17) [17)
(17) (Equation 4 Patching #B, I'A, with
(17) (17)
corrupted inputs, [A][B"]...14 [A.] or [A.]B]...[A]] significantly affects the final logits, indicating that
the short convolution layer aggregates information from both [A] and [B] into the SSM states.
6.2
MAMBA AND TRANSFORMER IMPLEMENT THE SAME INDUCTION ALGORITHM
SSN stote
Erpuut
plock Lnout
Logits
Resid
Upper
Middle
Lower
togits
Induction Head
State Spaces
Previous Token Head
Convolution
A
B+1
B
(a)
(b)
Figure 7: Induction Circuit in (a) Mambas. (b) Transformers.
We have abstracted the induction circuit in the Mamba model (Figure 7a). while in Transformers, this implemented through a previous token head and an induction head (Figure(7b) In both models, the induction algorithm mixes information from the previous token and stores to predict the next token. The model retrieves this stored information to perform induction.
key-value format
In Transformers, the previous token head attends to the preceding token, boosting the induction head's attention for next-token prediction through the OV circuit. In contrast, Mamba uses con- volution specific layers merge neighboring tokens' information into the SSM state, which subsequent tokens query for induction (Figure
6.3 LOCAL CONVOLUTION OF MAMBA PREFERS OFF-BY-ONE
curious nuance is that Mamba merge information into the SSM input at token B, whilst Transformers directly performs token mixing between token B₁ and A2. We term this the 'off by one' token mixing approach adopted by Mamba. is theoretically possible that Mamba do not apply this strategy and instead write the information of B, into the states without delay. We also observe the same preference to 'off by one' in an early investigation of the IOI task (Wang et al. 2003)) (Appendix HD. We are still unclear why such mechanism forms what it is for, which suggests further study in Mamba circuit analysis.
7 CONCLUSION
In this paper, we propose measuring cross-architecture universality with sparse autoencoder feature similarity. novel metric introduced to estimate the marginal effect of architectural difference to representational universality and find that most features are quite similar in both models. We also provided a intuitive interpretation of this metric: Pearson Correlation is affected by the semantic complexity of features. Thus cancel out such effect by ablation. We also observe circuit-level analogy in induction mechanism and notice some interesting nuances called the Off- Cif-by-One motif between Mamba and Transformer.
10

---

Published as a conference paper at ICLR 2025
LIMITATIONS
SAE Features as Universal Computational Units. We are aware that SAEs, no matter how well they perform, yield only a subset of the true features (if exist) in the model. Though SAEs reveal levels of magnitude of matching features across models, we still lack a totally reliable way to lo- cate those features uncaptured by SAEs. It is possible that these nuanced and rare features behave differently across models. It is also possible that SAES capture a combination of the true features instead of the features themselves. Overall, Sparse Autoencoders and their implicit superposition hypothesis is still an immature research area. We think there are still a lot of room for improvement till we can measure the true universality of features.
Qualitativeness of Circuit Universality. Though we have shown that Mambas and Transformers implement the same induction algorithm, we have not yet quantitatively measured the similarity with some automated tools. It would be very interesting if there circuit discovery and comparison can be automated like some existing works in circuit discovery on Transformers (Conmy et al., 2023 Ge et al. 2024).
REFERENCES
Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models. CoRR, abs/2403.01550. 2024. doi: I048550ARXXV.2400001500 URL httpps://oii org/10. 48550/arXiv. 2403. 01590
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James R. Glass. Identifying and controlling important neurons in neural machine translation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. www.:neetformmme d= 3=H12-PsR5KK
OpenReview.net, 2019. /
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 2397-2430. PMLR, 2023. URL https:// proceedings mlr. prwss/ w21///2222///2
Setres//2022/idermmaat http///idermaa2222 3a.htm.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models explain neurons in language models. /ttpps:/opeeaiiin public coublic.blob.com blob. core. www........wwndwww.............. windows net/ neuron-expla ner wwweer/aaper//eee hitpeerr/iidxx.eeeee index. html, 2023.
Enric Boix-Adserà, Hannah Lawrence, George Stepaniants, and Philippe Rigollet. GULP: a prediction-based metric between representations.
Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, pttppppe.iiis.cccpppeeeeeee
2022, 2022. URL. 1E0336cffer11100eeccccCC Aastaeeeeeeeeettttttttttttoeoooe Cot.tratttttttoteeeeeee....e
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jemmyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Haffield-oddss, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/trrnssfrmmer- pubb20023/mooosemattieeeeeee
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh,

---

Published as a conference paper at ICLR 2025
Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL http::://pocceedigg...ooo
1457c0d6bf cb 96 741 www.aac114ff44200.. neurips.......pp................ hcc/paper/2C2//haae
html
Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 2020. doi: 10.2391 /5/distil..0004.
Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineer- ing how networks learn group operations. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference
Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 0267. PMLR, 2023. URL https:
//proceedingg iggs. ml prwss v202 chught.. 23a. html
Arthur Commy, Augustine N. Mavor-Parker, Aengus Lynch. Stefan Heimersheim, and Adrià Garriga-Alons Towards automated circuit discovery for mechanistic interpretability. In Al- ice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 wt/pppppppppppppppppppppp..p..ep http::/p.......a.aaaaaaaaaaa////
16, 2023, 2023. URL
html
Hoagy Cunningham and Tom Conerly. Circuits updates june 2024. Transforee Circuits Thread, 2024. htts://taannffmmer-crrruuus wwtp:/p////jjjeuuuuaaeaeee.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse toencoders find highly interpretable features in language models. CORR, abs/2309.08600, 2023. doi: 10.48 10448500/AXXXV22300006666 URL https: sattps://do// http::::://ooi.ogg///// http:////111.1100010101100 2309.
08600
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, arXiv preprint arXiv:2405.21060, 2024.
Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. Grounding representation similarity with statistical testing. CoRR, abs/2108.01661 2021.
URL [ttps://arrdv.orr//az
2108.016611
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodee Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/transformer-iicce wttpsspp//.l/////oooooooooot
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer EIShowk, Nicholas Joseph, Nova DasSarma, Ben Mann. Danny Hermandez, Amanda Askell, Ka- mal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hotfield-Dodds, Jackson Kermion, Tom Conerly, Shauna Kravec, Stanislav Fort, Sauray Ka- davath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCan- dlish, Dario Amodei, and Christopher Olah, Softmax linear units. Transformer Circuits Thread, http:/mmreeiccuuusssp//////////0
2022a.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei. Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022b. https:/transformee- Strcuits.pub200220 model/index.tl html.
12

---

Published as a conference paper at ICLR 2025
Nelson Elhage, Robert Lasenby, and Christopher Olah. Privileged bases in the transformer resid- ual stream. Transformer Circuits Thread, 2023. https//transformer-iinn pttrcuutsppb//222//piiii. basis/index.htm html.
Daniel Y. Fu, Tri Dao, Khaled Kamal Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. In The
Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, net forum?id-
May 1-5, 2023. OpenReview.net, 2023. URL
COZDydWYGg
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021. URL
https://aaxxa
org/ abs 200020002000
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. CoRR, abs/2406.04093,, 2024. doi: 10.46550ARXVV224666000
1406.04099
Xuyang Ge, Fukang Zhu, Wentao Shu, Junxuan Wang, Zhengfu He, and Xipeng Qiu. Auto- matically identifying local and global circuits with linear computation graphs. arXiv preprint arXiv:2405.13888, 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, 0bs/2312.00055, 2023. doi: 10-8550AARXX.2312.00555 URL htttps::/doi.c:g/1C..
http://axiiv..........2 00752
Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons GPT2 language models. CoRR, abs/2401.12181, 2024. doi: 10.4550/ARXX..201.21188 URL https::/doi....: otg//...oogg.....
48550/arXiv. 2401. 12181
Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on Ahte. CoRR, abs/2402. 122011 2024. doi: 1048550/ARXIV.2002.222 2402. 12201]
URL https:
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V. Le. Transformer quality in linear time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.). International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9099-9117. PMLR, 2022. URL (ttpss: /ttpc//eoiiiioppoccooooocoooo.co / peoooeedding...... ml www.....r.s///........... 62 62/h/aua htt./uuaaaa... html
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024.
Valentin Khrulkov and Ivan V. Oseledets. Geometry score: A method for comparing generative adversarial networks. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th In- ternational Conference on Machine Learning, ICMI. 2018, Stockholmsmüsssnn Stockholm, Swe- den, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2626-22222222 2634. PMLR, 2018. URL
mir 2tteess//80//hrukoor11.... htt.kol111....m11.1....
Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. CoRR. abs/2305.06322, 2023. doi: 1044550/ARXIV.2305.666
URL httts:://o// http:::///ooo.o///////
48550/arXiv 2305.06329
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9- 15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 9519-3529, PMLR, 2019. URL htt.: httt:://proceecingg.. mir. htt.oeeccdinnns......pees/////// kornbl 1 ht1 9a. html
13

---

Published as a conference paper at ICLR 2025
Yixuan Li, Jason Yosinski. Jeff Clune, Hod Lipson, and John E. Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL 1511 075443
Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread, 2024. http:///ttttomteeeimiiiiiiii//p/
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural In- formation Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 De- cember 9, 2022, 2022. URL ht http:::pp/ppprpseee ni httpc///pppppeeeeeeeppppaaaeeeee htm
/2022/
2ash/6f1143d55533 7 233
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi:
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodee
Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. hutp::/transforner-ccccuius wtt.:/iiiiissssb222222oooooeetee
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feed- back. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference Neural Informa- tion Processing Systems 2022, NeuriPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http: http:/aaapeeseee nips. wttp://s.nnn......ppeeeeeeeeee lorr__illll__ files/illeeeeee.e wateeesppaprr22222 222p://222/2///////// htmml
Gonçalo Paulo, Thomas Marshall, and Nora Belrose. Does transformer interpretability transfer to mns? CoRR, abs/2404.05971, 2024. doi: 10.4855/ARXX.2244405000 URL https:
/ 24w.://xa.i............ 05971
// //doi. addii.org/1... 00w.4445550///.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kran- thi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong. Bart- lomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri lpsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan S. Wind, Stanislaw Wozniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: reinventing rmss for the transformer Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 44047-14077. Association for Computational Linguistics, 2023. doi: findings-ennnnn
In Houda
URL
httpps://db/ org. 111////...115///2223333 fi
936
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio. Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Confer- Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, pp.
2017. URL ott.o//ooooooolllltaataaooooooor
wtt.:ediiiinnnnnnninnee neuriiippp cC/ wwtppppppppp...../////pp/ html
14

---

Published as a conference paper at ICLR 2025
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders. arXiv preprint 4rXlv:2404.1601, 2024.
Arnab Sen Sharma, David Atkinson, and David Bau. Locating and editing factual associations in mamba. arXiv preprini UrXiv:2404.0360 2024.
Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhe- jian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang. Lingling Wu, Zhangyue Yin, Xuanjing Huang, Yu-Gang Jiang, and Xipeng Qiu. Moss: An open conversa- tional large language model. Machine Intelligence Research, 2024. ISSN 2731 doi: 10. 1007/s11633-022 1502-8.
httthtts: org/
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor transformer for large language models. CoRR, abs/2307.08621, 2023. doi: 10-88500ARXII22370006.... URL hattps://doi.cz/
(8550/arXive 2307 08621
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman. Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Trans- former Circuits Thread, 2024. URL https::/ttret scall Seealing-aee inn mnngmmonosemante intmmminniiiiiii cityy wndii.idxi...em wwnnde..ttte html
Former-ci www.tsiccitssoomom pub//22244
Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL htttps://openrevier net forum? fieeedddwwwweeeee yMwaG
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 -008, 2017. URL htttps://proceeddnggs 3f5ee2435 2eteee233477dee 9 1tteeeeee0000000101144443aaa.a0. Abatssaa--bbbtsaaaaa html
wwcppppe///0/ 2anper/2017/hae 0170/ahh/////
Kevin Ro Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. In- terpretability in the wild: a circuit for indirect object identification GPT-2 small. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:://penrewwe wwwpeerrviiee.ee//ddee net/pdf?id=
NpsVSN6o4ul.
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John E. Hopcroft. Towards understanding learning representations: To what extent do different neu- ral networks learn the same representation. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman. Nicolò Cesa-Biane Bianchi, and Roman Garnet (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp- 4607-9616 2018. URL htt.hhttpps:/prrceeeine neurips cct/pappr//21//hast// abt
Abstract.ing html
Itamar Zimerman, Ameen Ali, and Lior Wolf. A unified implicit attention formulation for gated- linear recurrent sequence models. arXiv preprint arXiv:2405. 16504, 2024.
15

---

Published as a conference paper at ICLR 2025
A SPARSE AUTOENCODERS TRAINING DETAILS
A.1 COLLECTING ACTIVATIONS
We truncate each document to 1024 tokens and prepend a <bos>> token to the beginning of each document. During training, we do not use the activation of <bos>> and <eos>> tokens.
It has been shown that the activations of different sequence positions of the same document are highly correlated and may lack diversity. To address this issue, it is common introduce random- ness into the training data. Our shuffling strategy is to maintain a buffer and shuffle it every time the buffer is refilled.
A.2 INITIALIZATION AND OPTIMIZATION
The decoder columns Widee ;f are initialized uniformly and normalized to have 2-norm of 2
V expansion factor" We initialize the encoder weights Wene with the transpose of the decoder weights Wdee. The encoder bias b and decoder bias gdee are initialized to zero.
This ensures that the initialized SAE almost always achieve a near-zero reconstruction loss. It has been widely observed that this kind of initialization is beneficial for training SAEs.
We train SAEs with Adam optimizer with B1 0.9, 32 0.999 of and e 10- The learning rate is set to to 8e-4 for all SAES.
B DISTRIBUTION OF MAX PAIRWISE PEARSON CORRELATION ACROSS LAYER
We plotted the layer-wise distribution of MPPC calculated from Pythia to Mamba. For clarity, divided the layers into four groups: 0-2, 3-5, 8,, and 9-11, and presented the MPPC distribution histograms for each group (Figure 8
As the layer number increases, the distribution gradually shifts to the left. In conjunction with the conclusion in Section 5.2 MPPC decreases, feature complexity tends to increase, infer that lower-layer features are simpler, while higher-layer features become increasingly complex.
Pearson Correlation Distribution
D-2
9-31
Probability Density
32
Correlation Value
Figure 8: Histogram of the MPPC distribution for Pythia features, grouped by layers 0-2, 3-5, 6-8, and 9-11.
C BIDIRECTIONAL ANALYSIS OF MAX PAIRWISE PEARSON CORRELATION
The MPPC used in the main text is a one-way, irreversible metric, lacking symmetry. The experi- ments presented in the main text only report the results of finding the maximum Pearson correlation from Pythia to Mamba and Pythia (model seed variant). Here, we provide the reverse results, i.c., the distributions of p d and P; m- well as p and p. +p (Figure Additionally, we show
16

---

Published as a conference paper at ICLR 2025
Comparison of Max Painwise Pearson Correlation Distributions Between Pythia and Mamba
A
Average
B Average 0.74
Perceriage 0%)
15
(a) A: Pythia, B: Mamba
Comparison of Max Pairwise Pearson Correlation Distributions Between Two Pythia Model
A --
Average
25
B Average D.71
Percentage (N)
15
6.
Value
(b) A: Pythia, B: Pythia(model seed variant)
Figure 9: (a) MPPC distribution for Pythia and Mamba in both forward and reverse directions. (b)
MPPC distribution for Pythia and Pythia (model seed variant) in both forward and reverse directions.
Depth Distribution of Matched Features
Proportion
04
Pythia
Mamba Layer
Figure 10: Frequency of the best matching feature pairs falling on each layer.
the Depth Distribution of Matched Features when calculating the MPPC from Mamba to Pythia (Figure I0..
The forward and reverse results for Pythia and Pythia (model seed variant) are very close. The re- verse results for Pythia and Mamba outperform the forward results, primarily because Mamba has higher proportion of lower-level features, which tend to have higher MPPC values (Section 5.2) This phenomenon confirmed in the reverse Depth Distribution of Matched Features. The first five layers of Mamba tend to match with the first two layers of Pythia, especially the first layer, while the first seven layers of Mamba tend to match with the first three layers of Pythia-on one more layer than generally simpler (Appendix Com-
proportional scaling would suggest. Lower-level features pared to Pythia, Mamba has a larger proportion of layers dominated by simpler features. However, depth specialization phenomenon observed in both the forward and reverse matching processes.
17

---

Published as a conference paper at ICLR 2025
Distribution of Max Pairwise Pearson Correlation (I1 1e-04)
Main Experiment
Percentage (1)
Average
20
Skylinee: Model Training Variant
Average = 0.717
Skylinc2:
Average 0.800
Random Baseline
02
0.4
3.6
(a) = 1 x 10-4
Distribution of Max Pairwise Pearson Correlation (I1 2e-04)
Main Experiment
Percemage (%)
Average 0.681
Training Variant
Average 0.725
Skyline2: SAE Seed Variant
Average
Random Baseline
02
0.4
3A
0.
(b) L1 2 x 10-4 (baseline)
Distribution Max Pairwise Pearson Correlation (Il 4e-04)
Main Experiment
Percentoge (96)
Average 0.720
Skylinel: Training Variant
Average 0.766
Skyline2: SAE Seed Variant
20
Average 0.833
Random Baseling
02
0.4
36
08
Correlation Value
(c) LI 4 x 10"4
Figure 11: MPPC distributions under different LI regularization strengths.
D COMPREHENSIVE ROBUSTNESS ANALYSIS OF MPPC ACROSS MULTIPLEE DIMENSIONS
To thoroughly validate the robustness and generalizability of our method, we conducted exten- sive experiments across five key dimensions: hyperparameter sensitivity, dictionary size variation, dataset diversity, model architecture differences, and model scale. This comprehensive analysis ensures the reliability of our conclusions and demonstrates the method's stability under diverse con- ditions.
D.1 HYPERPARAMETER SENSITIVITY ANALYSIS
L1 Regularization Strength We systematically evaluated the impact of sparsity constraints by training SAES with three distinct LI coefficients: 1 x 10-4, 2 x 10-4 (baseline used main ex- periments), and 4 10-4 Figure 11 presents the resulting MPPC distributions, revealing consistent patterns across different regularization strengths.
D.2 DICTIONARY SIZE VARIATION
To assess the impact of feature space dimensionality, trained SAEs with expansion factors of 32 (baseline) and 64. The comparative MPPC distributions Figure 12 demonstrate the method's stability across different dictionary sizes.
18

---

Published as a conference paper at ICLR 2025
Distribution of Max Pairwise Pearson Correlation (Expansion Factor 32)
Main Experiment
Percentage (1)
Average
20
Skylinee: Model Training Variant
Average = 0.725
Skylinc2:
Average 0.806
Random Baseline
0.4
(a) Expansion factor 32 (baseline)
Distribution of Max Pairwise Pearson Correlation (Expansion Factor 64)
Main Experiment
Percernage (%)
Average
30
Skylliel: Model Training Variant
Average 0.733
Skyline2:
Average 0.795
Random Baseline
6
Correlation
(b) Expansion factor = 64
Figure 12: MPPC distributions with different dictionary sizes.
D.3 DATASET ROBUSTNESS EVALUATION
We validated our approach on three distinct datasets:
SlimPajama (baseline)
RedPajama-Githul (specialized code corpus)
OeenWeeText (general web text)
The consistent MPPC patterns across these datasets, as shown in Figure 13 highlight the method's robustness to data distribution variations.
D.4 ARCHITECTURAL VARIANTS ANALYSIS
We extended our validation different SAE architectures:
Vanilla SAE (baseline)
Top-k SAE (architecture variant)
Figure illlustrate the comparable performance across these architectural variants.
D.5 MODEL. SCALE ANALYSIS
To investigate scalability, we evaluated our method on larger models:
Pythia-2.80 -.hhi
Pythia-2.8B-Ba
Mamba-2.8B
The results in Figure 15
demonstrate consistent behavior across different model scales.
19

---

Published as a conference paper at ICLR 2025
Distribution of Max Pairwise Pearson Correlation
Main Experiment
Percentage (1)
Average
20
Skyliiee: Model Training Variant
Average = 0.725
Skylinc2:
Average 0.806
Random Baseline
0.4
(a) SlimPajama (baseline)
Distribution of Max Pairwise Pearson Correlation
Main Experiment
Perceraag (%)
Average
Skylline: Model Training Variant
20
Average 0.782
Skyline2:
Average 0.806
Random Baseline
0
Correlation
(b) RedPajama- Githhub
Distribution of Max Pairwise Pearson Correlation
Neuron Baseline
Main Experiment
Percentage (N)
Aucrage 0.74
Skylinel:
Average 0.76
Skyline2: SAE Seed Variant
Average
Correlation
(c) Open WebText
Figure 13: MPPC distributions across different training datasets.
D.6
CROSS-ARCHITECTURE VALIDATION
Finally, we conducted comprehensive cross-architecture analysis using three distinct model fami- lies:
Pythia-160m (Transformer-based)
Mamba- 130m (State Space Model)
RWKV-169m (RNN-Transformer hybrid)
The complete set of cross-architecture results in Figure [ releell consistent inter-model correlation patterns, further validating the generalizability our approach.
AUTOMATIC FEATURE SCORING PROMPTS
E.
USE FOR COMPLEXITY
We used the following prompt to have the LLM evaluate our Complexity:
We are analyzing the activation levels of features in a neural network, where each feature activates certain tokens in text. Each token's 's activation value indicates
20

---

Published as a conference paper at ICLR 2025
Distribution of Max Pairwise Pearson Correlation
Main Experiment
Percentage (1)
Average
20
Skyliiee: Model Training Variant
Average = 0.725
Skylinc2:
Average 0.806
Random Baseline
0.4
(a) Vanilla SAE (baseline)
Distribution of Max Pairwise Pearson Correlation
Main Experiment
Percentage (%)
Average
Skylliel: Model Training Variant
Average 0.680
10
Skyline2:
Average 0.726
Random Baseline
6
Correlation
(b) Top-k SAE
Figure 14: MPPC distributions for different SAE architectures.
Distribution Max Pairwise Pearson Correlation
Main Experiment
Percentage 1961
Average 0.681
Skyline1: Training
Average 0.725
20
Skyline2: SAE Seed Variant
Average 0.806
Random Beseline
02
0.4
3.9
Correiztion Value
(a) Small Model (baseline)
Distribution Max Pairwise Pearson Correlation
Main Experiment
Percentage (N)
Average 0.792
40
Skyline1: Training
Average 0.847
Skyline2: SAE Send Variant
Average
Random Baseline
67
0.4
3.6
08
Correlation Value
(b) Large Model (2.8B)
Figure MPPC distributions across different model scales.
its relevance to the feature, with higher values showing stronger association. Your task is to infer the common characteristic that these tokens collectively suggest based on their activation values and give this feature a complexity score based on the following scoring criteria: Complexity
5: Rich feature firing on diverse contexts with an interesting unifying theme, e.g. "feelings of togetherness"
21

---

Published as a conference paper at ICLR 2025
Comparison of Max Pairwise Pearson Correlation Distributions Between Pythia and Mamba
Average 0.68
Percenage (56
Average 0.74
52
0.4
59
Correlation
Pythia vs. Mamba
Comparison of Max Pairwise Pearson Correlation Distributions Between Pythia and RWKV
Average 0.61
Percetagee (5)
Average 0.49
Conrelation
Pythia vs, RWKV
Comparison Max Pairwise Pearson Correlation Distributions Between RWKV and Mamba
A-2-
Average 0.55
B->
Percentage 196)
Average 0.71
Correlation Value
(c) RWKV Mamba
Figure 16: Cross-architecture MPPC distributions.
4: Feature relating to high-level semantic structure, e.g. "return statements
in code"
3: Moderate complexity, such phrase, category, tracking sentence
structure e.g. "website URLs"
2: Single word or token feature but including multiple languages or spelling,
e.g. "mentions of dog"
1: Single token feature, e.g. "the token '(""
Consider the following activations for feature in the neural network. Activation values are non-negative, with higher values indicating stronger connection be- tween the token and the feature. Don't 't list examples of words. You only need to give me number! Just number! It represents your score for feature complexity.
Following this prompt, we provided ten sentences containing top activations, each consisting of 17 tokens with the activation value for each token.
E.2 USE FOR ACTIVATION CONSISTENCY
We used the following prompt have the LLM evaluate Activation Consistency:
22

---

Published as a conference paper at ICLR 2025
We are analyzing the activation levels of features in neural network, where each feature activates certain tokens in a text. Each token's 's activation value indicates its relevance to the feature, with higher values showing stronger association. Your task is to give this feature a monosemanticity score based on the following scoring criteria:
Activation Consistency
5: Clear pattern with deviating examples
4: Clear pattern with one or two deviating examples
3: Clear overall pattern but quite a few examples not fitting that pattern
2: Broad consistent theme but lacking structure
1: No discernible pattern
Consider the following activations for feature in the neural network. Activa- tion values are non-negative, with higher values indicating a stronger connection between the token and the feature. You only need give me number! Just number! It represents your score for feature monosemanticity.
Following this prompt, we provided ten sentences containing top activations, each consisting of 17 tokens with the activation value for each token.
CASE OF UNINTERPRETABLE FEATURES
We manually inspected substantial number of features from the Pythia model with cross- architecture MPPC values below 0.2. We observed that these features are predominantly polyse- mantic, showing activations on text inputs with no discernible pattern or lacking clear structure. Figure 17 illustrates four randomly selected examples of uninterpretable feature pairs with different Pearson correlation values.
G DATA RESULT OF PATH PATCHING
For each condition, we selected 128 randomly generated induction data points for path patching.
Result1
The results of path patching for h 1 shown in Table
Use [A][B']...IAA
rupted input, [A][B]..[[A] as clean input. Distance is the distance between two [A].
distance
layer
0
2
4
5
6
7
0.01
-0.05
0.33
-0.09
0.05
0.1
-0.04
0.12
16
0.02
-0.04
-0.46
-0.03
-0.07
0.0
0.06
0.0
32
-0.01
0.15
0.88
0.35
-0.0
0.09
0.08
0.07
64
-0.0
-0.03
-0.12
0.31
-0.06
0.07
0.04
0.02
128
0.01
-0.03
-0.11
-0.03
0.13
0.03
0.04
-0.01
distance |layer
8
9
10
11
12
13
14
15
0.04
0.13
0.02
0.9
0.63
-0.11
0.51
-0.13
16
-0.02
0.01
0.06
-0.02
-0.54
0.14
-0.14
0.24
32
0.57
-0.05
0.03
0.65
0.79
-0.03
0.09
1.32
64
0.02
0.01
0.01
-0.26
-2.18
0.1
0.09
0.8
128
-0.03
0.0
-0.01
0.08
-1.94
0.02
-0.07
0.0
distance layer
16
17
18
19
20
21
22
23
0.16
-9.65
0.04
-0.19
-2.02
-0.0
-0.03
-0.0
16
-0.04
-8.09
-0.04
-0.04
-1.81
-0.02
-0.02
-0.01
32
0.08
-8.73
0.09
-0.2
-1.48
-0.03
-0.01
-0.01
64
0.49
-10.16
-0.04
-0.12
-0.99
-0.01
-0.01
-0.0
128
-0.1
-8.44
-0.04
0.07
-0.66
-0.01
-0.03
-0.0
Table 1: The mean logit diff of token B after patching each state
23

---

Published as a conference paper at ICLR 2025
pythia
mamba
L354
L11.16406
mоге troubling thein
Over to
the neet and greet the
0 126
ArnomenbelelSee edit notivation of the Trolka connot
Beyond Android
4.3 4.3
The
Final
race
American
polots
Shown
L5. 7996
L17. 6406
0.087 Despitee the kn aathack, Lukoszuk said Beparture From Republican arthiodexy adds onother Layer of financial savings
on the rise thanks kn na snall part get wrong: got
We have proved Eliné and kine again
32008
L19.9650
0 0.6
Alfred features
good the bad
Keeps updated copy Bi the ledger.
Rohde advised to snont the Anber Roon While the harder material works better
if convictedd offeenses chorged
L10 8800
L19.4876
0.159
questions they can silly to bothered with your criticisn Spear iishing attempts have been
ffmpegZtheora Wddebbitrtte to surprise modern-day pal agontologists
Chronosterepps appears
Figure 17: We present four uninterpretable cases of feature pairs with different Pearson correlation values. Each feature index is followed by activation examples shown below.
Result2 The results of path patching for c
(17)
shown in Table 2 Use [A][B']..[A] as corrupted
input, [A][B]..[[]] clean input. Distance is the distance between two [A].
distance \positioon
A
B1
B₁ +
B1
B +
8
0.0
-0.0
-9.69
0.11
-0.03
16
0.0
-0.01
-11.25
0.01
0.11
32
0.0
-0.01
-16.84
-0.01
-0.05
64
0.0
-0.0
-10.64
-0.33
0.4
128
0.0
-0.01
-9.13
0.52
0.16
Table 2: The mean logit diff token after patching SSM input at each position
The results of path patching for (2(17) are shown in Table Use [A][B']..[[] as corrupted input, [A][B]..[[]] as clean input. Distance is the distance between two [A].
Result3
(17)
Result4
The results of path patching for 2; (17) are shown in Table Use [A'][B]..[[A] as corrupted
input, [A][B]...[AA [..[]] clean input. Distance is the distance between two [A].
24

---

Published as a conference paper at ICLR 2025
distance sposition
A1
B1
B1 +1
0.0
-9.11
0.26
16
0.0
-15.17
0.24
0.0
-13.95
-0.06
64
0.0
-13.71
0.27
128
0.0
-9.5
-0.11
Table 3: The mean logit diff of token after patching x corrupted imput
(17)
at each position with [A][B'] [..[A] as
distance posstiin
A
B1
B1 +
8
-8.27
-0.88
0.04
16
-9.44
-0.52
0.17
32
-11.3
0.72
-0.02
64
-9.07
-1.5
0.2
128
-8.96
-0.54
0.13
Table 4: The mean logit diff of token B after patching z; corrupted input
(17)
at each position with [A.][] as
H OFF BY ONE IN IOI TASK
H.1 TASK DESCRIPTION
The indirect object identification (IOI) task involves sentences with clauses: an initial clause (e.g., "When Mary and John went to the store") and a main clause (e.g., "John gave Mary bottle of milk"). The initial clause introduces the indirect object (IO) and the subject (S), while the main clause mentions the subject again, indicating the action of the subject giving the object to the IO. The task is to predict the final token corresponding to IO. We use "S1" and "S2" to represent first and second occurrences of the subject, respectively.
H.2 OCCURRENCE OF THE OFF-BY-ONE PHENOMENON
We use [IO][S)...(S.... as clean input and [I''['"'...'S''..... as corrupted input. Path patching is applied to c/ allowing its influence the logits through all paths except those involving other layer SSMs. We compute the logits difference between IO and S, as shown in Figure [8] It is observed that the causing the logit change always appears at the token immediately following IO or S. This suggests that the information from IO and S enters the SSM state at the next token, which similar the induction task phenomenon and represents another instance of the off-by-one effect.
I HUMAN EXPERT EVALUATIONS AND CONSISTENCY WITH GPT-4 SCORING
In this section, present comprehensive human evaluation conducted to assess the complexity of 64 features and the monosemy of 32 features. The human experts were tasked with providing scores for each feature based on predefined criteria, ensuring a consistent and thorough assessment process. To gauge the alignment between human judgments and automated scoring, we compared the human scores with those generated by GPT-4. The comparison of the two sets of evaluations is illustrated in Figure 19 This analysis highlights the level of agreement between the expert and machine ratings.
25

---

Published as a conference paper at ICLR 2025
Logit Difference in 1OI Task
Logit diff.
lay prr
-5
</endoftextjs
Then
GBrandon
Gand
GMark
Gwent
Gto
Gthe
Ghospital
GMark
Ggave
Ga
Gcomputer
Figure 18: Logit changes after patching of for each layer and each position. For each cell, the left side represents IO, and the right side represents S. Token positions are labeled using the tokens from the example sentence: "Then, Brandon and Mark went to the hospital. Mark gave a computer to Brandon."
26

---

Published as a conference paper at ICLR 2025
Al Alignment Between Auto and Manual Labeled Complexity Scores
Manual ly Labeled Comp eexity Score
y 0.45x 1.21 R 0.29
25
05
015
3.5
25
3.5
45
Auto Labeled Complexity Score
(a) Complexity
Al Aiggneet Between Auto and Manual Labeled Monosemanticity Scores
Manually Labeled Nomosemanticitty Score
y 0.38x R² 0.17
0.97
1.5
Auto Labeled Monosemanticity Score
(b) Monosemanticity
Figure 19
27


Reviews:
[{"review_id": "WDVqjcl7JG", "reviewer": "Reviewer_4GEc", "summary": "The paper investigates the \"universality hypothesis\" in mechanistic interpretability, which suggests that different neural network architectures may converge to implement similar algorithms when tasked with analogous objectives. The authors focus on two mainstream architectures for language modeling: Transformers and Mambas. They propose using Sparse Autoencoders (SAEs) to extract interpretable features from these models and demonstrate that a significant portion of features are shared between the two architectures. The paper validates the correlation between feature similarity and universality and delves into the circuit-level analysis of Mamba models, finding structural analogies with Transformers, particularly in induction circuits. \n\nThe paper's contributions include:\n- Introduction of a novel metric to isolate and quantify feature universality in the context of architectural variations.\n- Empirical evidence shows that Transformer and Mamba models learn similar features through the application of SAEs.\n- Circuit analysis of Mamba models reveals structural analogies and nuanced differences compared to Transformer circuits.\n- Support for the universality hypothesis by demonstrating cross-architecture feature similarity and identifying the \"Off-by-One motif\" in Mamba models.", "questions": "1. Why did you choose OpenWebText as your primary dataset for analysis? How might the choice of OpenWebText as the dataset influence your results? Have you tested if the feature similarities hold across different domains (e.g., code, mathematics, or structured data)? Would analyzing domain-specific text reveal different patterns of architectural universality?\n\n2. Have you performed any statistical significance tests to support your claims of feature similarity and universality?\n\n3. How generalizable are your findings to other tasks beyond language modeling?\n\n4. Can you provide more details on why the \"Off-by-One\" motif exists in Mamba models?\n\n5. Is there a risk that the Sparse Autoencoder pre-processing itself may impose a degree of alignment between features in Transformers and Mambas? Could the sparsity constraint inadvertently enhance apparent similarity?\n\n6. How do you expect your findings to scale to larger models? Did you observe whether model size impacts universality between architectures? Could smaller or larger versions of Transformers and Mambas exhibit different degrees of feature similarity?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The identification of the \"Off-by-One motif\" in Mamba models is a unique contribution that highlights nuanced differences between architectures.\n\n- The introduction of a complexity-based interpretation for understanding feature similarity differences is innovative.\n\n- The circuit-level analysis of Mamba models, revealing structural analogies with Transformers, is good and adds depth to the study. The validation of feature similarity and its correlation with universality further strengthens the study.\n\n\n- The findings of this paper have implications for the field of neural network interpretability. By demonstrating that different architectures can converge to similar algorithms and features, the study provides valuable insights into the generalizability of mechanistic findings across models.", "weaknesses": "- While the paper focuses on Transformers and Mambas, it would benefit from a broader examination of additional architectures. Including a more diverse set of models, such as recurrent neural networks (RNNs) or convolutional neural networks (CNNs), could strengthen the universality hypothesis by offering a more comprehensive understanding of feature similarity across a wider range of neural networks. This would enhance the generalizability of the findings.\n\t\n- The paper utilizes OpenWebText for correlation analysis but does not discuss how the choice of dataset might affect the results. A more detailed examination of the potential biases and limitations introduced by the dataset choice would provide a clearer context for the findings and ensure that the results are not overly dependent on a specific dataset.\n\n- The claims of feature similarity and universality would be more robust if supported by statistical significance tests. Including such tests would provide stronger evidence for the observed correlations and enhance the credibility of the conclusions.\n\n- SAE-related technical gaps:\n    - The paper does not include ablation studies on SAE hyperparameters (dictionary/code size, training duration, etc.). Conducting these studies would help to understand the sensitivity of the results to different hyperparameter settings and ensure the robustness of the findings.\n    - There is no discussion of how SAE reconstruction quality relates to feature similarity. Addressing this relationship would provide insights into the effectiveness of SAEs in isolating interpretable features and validate the methodology used.\n\n- The use of GPT-4 for complexity scoring lacks rigorous validation. The paper does not provide inter-rater reliability metrics or comparisons with human annotations, nor does it discuss potential biases in the automated scoring. \n\n- The paper provides a limited exploration of why the \"Off-by-One\" motif exists in Mamba models. A deeper investigation into the underlying reasons for this motif would enhance the understanding of the structural differences between the Mamba and Transformer models and provide more insights into the universality hypothesis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G5LCkBXzH3", "reviewer": "Reviewer_nuP5", "summary": "This paper studies the mechanistic similarity between language model structures (Mamba, RWKV, Transformer). The authors focus on their Universality, a hypothesized property that suggests different neural architectures implementing similar algorithms on similar tasks.\n\nIn the first part of the experiment section, they use Sparse Autoencoder (SAE) as the major tool for their analysis. The representations from two LM architectures are taken to train SAEs. The latent vectors in the SAEs, in which each dimension corresponds to various syntactic and semantic phenomena, exhibit mechanical similarities, and it is possible to find a matching between the vectors from different LM architectures.\n\nIn the second part, the authors study the induction behavior of two LM architectures. They found that the 17th layer of Mamba is the most important for the LM’s inductive ability.\n\nI think this paper studies an important problem and is well executed. I found the experiments in this paper to be well implemented. My only concern is with the role of circuit analysis experiments. They are indeed very interesting but I’m not sure how they contribute to building a mechanistic analogy between SSMs and Transformers. Do Transformers have such layer-specific behavior when it comes to inductive ability? Is there a way to empirically verify the claims in Sec 6.2?\n\nMinor:\n\nAppendix D: How does the feature mapping between RWKV and Mamba look like?\nSec 6.1: Is the layer-17 phenomenon robust to random initializations? I.e., if one retrains the SSM with another seed, would layer 17 still be the key in induction?\nLine 179: missing section reference.\nLine 861: missing space between ‘Universality’ and ‘is’", "questions": "* Do Transformers have such layer-specific behavior when it comes to inductive ability?\n* Is there a way to empirically verify the claims in Sec 6.2?\nAppendix D: How does the feature mapping between RWKV and Mamba look like?\n* Sec 6.1: Is the layer-17 phenomenon robust to random initializations? I.e., if one retrains the SSM with another seed, would layer 17 still be the key in induction?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper studies an important problem.\n* It makes good use of sparse autoencoders for analysis.\n* The experiments in this paper are well implemented.", "weaknesses": "* The role of circuit analysis experiments is unclear.\n* The claims made in Section 6.2 need to be empirically supported.\n* (Minor) The size of LMs is limited. Only ~100m models are used in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d3Sh4NQ0iB", "reviewer": "Reviewer_eqTo", "summary": "This paper presents an exploration of the Transformer and Mamba models through mechanistic interpretability methodology. Despite the architectures being very different, the features and circuits of these models turn out to be very similar.", "questions": "- While the heatmap matrix in Figure 3c is mainly diagonal, I can see that there is a cluster of features located in the last layer of the Pythia and distributed fairly uniformly in the middle layers of Mamba. Can the authors clarify the meanings of these features?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written. \n- I appreciate the idea of using skylines, as it helps support the authors' claims. \n- The results are interesting and useful for further research.", "weaknesses": "I couldn't identify any specific weaknesses. However, below are some suggestions that could enhance this work from my perspective:\n\n\n- It would be interesting to explore more circuits through SAE, as suggested in [1] (see Section 4.3). However, it is unclear where SAE should be placed within the Mamba architecture to achieve similar features.\n- While the Pearson correlation appears to be a natural choice for measuring feature similarity, it assumes that the feature space has linear properties. It might be worthwhile to explore other correlation measures, such as distance correlation, which could potentially yield better results.\n- A clear statement clarifying that MPPC refers to the maximum Pearson correlation between models' features is needed to improve understanding.\n\n[1] Interpreting Attention Layer Outputs with Sparse Autoencoders (Kissane et al.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tLUiqEbEIY", "reviewer": "Reviewer_jsst", "summary": "This work investigates the similarity of sparse autoencoders (SAE) features and induction circuits between a Transformer-based and a comparable Mamba-based language model trained on the same dataset. Results show that many features from Transformers SAEs show a high max pairwise Pearson correlation with Mamba SAE features, with their depth along model layers roughly matching across the two models. The correlations found for cross-architecture comparison are compared to a neuron baseline and two skylines obtained from different models and SAE training seeds, showing that the cross-architecture comparison falls only slightly short of skylines. Authors further examine the correlation between cross-architectural matching features and their complexity, finding that features with the most overlap are generally simpler and more monosemantic. Finally, the authors briefly investigate the similarity between induction circuits in the same architectures using path patching, finding a similar mechanism mediated by the convolutional operation of the Mamba architecture. Notably, the information mixing necessary for the induction operation is performed earlier in Mamba (\"off-by-one\" mixing).", "questions": "What justified the choice of plain SAEs over more performant variants such as [Gated](https://arxiv.org/abs/2404.16014) or [Top-K SAEs](https://openai.com/index/extracting-concepts-from-gpt-4/)? It is currently hard to gauge the impact of this choice on the obtained results, and whether findings could have been different if improved SAE variants were tested.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The universality evaluation pursued in this paper is timely and relevant, given recent advances in non-Transformer architectures for language modeling. The baseline and skylines employed in this work are well-motivated and provide helpful reference points for the analysis. The analysis of feature correlation based on complexity is also interesting, showing convincing proof that most commonalities across architectures are found for simpler SAE features. Overall, the figures are designed clearly and compellingly to support the findings detailed in the main body of the paper.", "weaknesses": "**Novelty of the findings** While to my knowledge, this is the first work evaluating the similarity of SAE features between Transformers and Mamba models, other works such as [Paulo et al. 2024](https://arxiv.org/pdf/2404.05971) and [Sharma et al. 2024](https://arxiv.org/abs/2404.03646) already showed that many interpretability approaches such as logit lenses, steering vectors, and probes produce similar results across a variety of LLM architectures (Mamba, Transformer and RWKV). It is, hence, not particularly surprising that such findings extend to SAE decompositions of model activations.\n\n**Generality of findings for larger models**  Authors experiment only with 2 tiny models with ~100M parameters each. This can be reasonable in light of the requirements (same training data, effort of training SAEs on each layer), but these systems are significantly smaller than those used by [Paulo et al. 2024](https://arxiv.org/pdf/2404.05971) for comparable cross-architecture interpretability experiments. Notably, larger checkpoints for both models used by the authors are publicly available, including the same training data for control, and could have been used to further prove the generality of the reported results. Importantly, without further experiments, it cannot be excluded that the limited capacity of tiny models might be the main motivation behind the high similarity features and circuits across the two architectures, and this could not be the case for more capable models with e.g. 1B or 7B parameters.\n\n**Multiple Comparison and Correlation Analysis without Hypothesis Testing** The maximal correlation of feature activation patterns with other (24576 x # of layers) features is bound to be quite high due to the enormous amounts of comparisons. In Section 4.4, no hypothesis is formulated regarding the expected similarity of features found across the four tested variants, and consequently, no significance measure for the correlation coefficients is reported. As a result, conclusions regarding the similarity of Mamba and Pythia SAE features are ambiguous (e.g. the statement \"[...] our neuron baseline almost exhibits zero sign of similarity between Mamba and Pythia\" at line 268 does not agree with Figure 3a, where at least 15% of neurons exhibit a correlation > 0.4). To make the analysis more convincing, a clear hypothesis regarding the degree of similarity in resulting SAE features should have been formulated and tested for baseline, experiment, and skylines, each including a correction procedure such as the Bonferroni method to account for multiple comparisons.\n\n**Minor formatting/clarification points:**\n\nLine 135: The mention \"$F_a$ and $F_b$ are some kinds of operation function.\" is too generic in this context. The purpose of these functions should be specified, and at least one example of functions used for this purpose should be provided.\n\nLine 179: Broken reference.\n\nFigure 1 is too tight with the inline text, making the distinction between caption and main body text unclear.\n\nSection 4.1: title is ungrammatical. I imagine you meant something like \"Searching for / In search of Interpretable Primitives\".\n\nLine 202: Clarify that you mean all features for all SAEs across all model layers (it becomes clear only from Figure 3c later in the paper)\n\nLine 263: The acronym MPPC is never introduced alongside its meaning.\n\nFigure 5: The mention \"both Model Seed Variant and Cross-Arch SAE MPPC exhibit correlation, while one in SAE Seed Variant is weaker\" in the caption is not very meaningful, since the trends for all three variants are pretty similar. For part (b), the mention \"scores ranging from 1 (No) to 2 (Yes)\" is confusing: it would be better to say \"Distribution of MPCC for polysemantic (1) and monosemantic (2) auto-generated feature labels.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)