Mechanistic Permutability: Match Features Across Layers
Authors: Nikita Balagansky
Ian Maksimov
Daniil Gavrilov
Venue: ICLR 2025 Poster (2025)
Keywords: Interpretability
LLM features
SAE
Primary area: interpretability and explainable AI

Abstract: Understanding how features evolve across layers in deep neural networks is a fundamental challenge in mechanistic interpretability, particularly due to polysemanticity and feature superposition. While Sparse Autoencoders (SAEs) have been used to extract interpretable features from individual layers, aligning these features across layers has remained an open problem. In this paper, we introduce SAE Match, a novel, data-free method for aligning SAE features across different layers of a neural network. Our approach involves matching features by minimizing the mean squared error between the folded parameters of SAEs, a technique that incorporates activation thresholds into the encoder and decoder weights to account for differences in feature scales. Through extensive experiments on the Gemma 2 language model, we demonstrate that our method effectively captures feature evolution across layers, improving feature matching quality. We also show that features persist over several layers and that our approach can approximate hidden states across layers. Our work advances the understanding of feature dynamics in neural networks and provides a new tool for mechanistic interpretability studies.

Published as a conference paper at ICLR 2025
MECHANISTIC PERMUTABILITY: MATCH FEATURES ACROSS LAYERS
Nikita Balagansky Ian Maksimov*1, Daniil Gavrilov!
T-Tech, "Moscow Institute of Physics and Technologies, 3 HSE University
ABSTRACT
Understanding how features evolve across layers in deep neural networks is a fundamental challenge in mechanistic interpretability, particularly due to polyse- manticity and feature superposition. While Sparse Autoencoders (SAFs) have been used to extract interpretable features from individual layers, aligning these features across layers has remained an open problem. In this paper, we introduce SAE Match, novel, data-free method for aligning SAE features across different layers of neural network. Our approach involves matching features by minimizing the mean squared error between the folded parameters of SAEs, a technique that incorporates activation thresholds into the encoder and decoder weights to account for differences in feature scales. Through extensive experiments on the Gemma 2 language model, we demonstrate that our method effectively captures feature evolution across layers, improving feature matching quality. We also show that features persist over several layers and that our approach can approximate hidden states across layers. Our work advances the understanding of feature dynamics in neural networks and provides new tool for mechanistic interpretability studies.
INTRODUCTION
In recent years, foundation models have become pivotal in natural language processing research (Llama Team 2024 Team et al. 2024). As these models are applied a growing range of tasks, the need to interpret their predictions in humaan-nddersanddabl terms has intensified. However, this interpretability challenged by the presence of polysemantic
that correspond to multiple, often unrelated concepts- prevalent in large language models (LLMs).
significant advancement in addressing polysemanticity the superposition hypothesis, which posits that models represent more features than their hidden layers can uniquely encode. This leads to features being entangled in a oothoorhooon basis within the hidden space (Bricken et al. 2023 Arora et al. 2018 Elhage al. 2022), complicating interpretation because individual neurons or features do not correspond neatly to singular, human- uumaan-underttndabble concepts.
To unravel this complexity, Sparse Autoencoders (SAEs) trained on the hidden states of LLMs were employed, using feature sizes significantly larger than the models' hidden dimensions (Yun et al. 2021 Bricken al., 2023). SAES aim extract monosemantic Seatures-Sparse activations that occur only when processing specific, interpretable functions. For instance, Templeton et al. (2024) Munstance,Teepletton
used this approach to interpret the Claude Sonnet 3 model, while Lieberum al. 20244) open-sourced an SAE for the Gemma 2 model, enabling researchers to delve deeper into LLM interpretability.
While SAEs have advanced the interpretability of individual model layers, crucial question remains:
How do these interpretable features evolve throughout the layers of a model during evaluation? Understanding this evolution is essential for a comprehensive interpretation of the model's internal dynamics and decision deciiion-mmainn processes.
In this paper, we introduce SAE Match, data-free method for aligning SAE features across different layers of neural network. Our approach enables the analysis of feature evolution throughout the model, providing deeper insights into the internal representations and transformations that occur as data propagate through the network. By addressing the challenge of feature alignment across layers,
bal aaaaaaaiiyleee leasskiyttbank..

---

Published as a conference paper at ICLR 2025
contribute to more comprehensive understanding of neural network behavior and advance the field of mechanistic interpretability.
Our main contributions are follows:
We propose SAE Match, novel method for aligning Sparse Autoencoder features across layers without the need for input data, enabling the study of feature dynamics throughout the network.
We introduce parameter folding, a technique that incorporates activation thresholds into the encoder and decoder weights, improving feature matching by accounting for differences in feature scales.
We validate our method through extensive experiments on the Gemma 2 language model, demonstrating improved feature matching quality and providing insights into feature persis- tence and transformation across layers.
By advancing methods for feature alignment and interpretability. work contributes to the broader goal of making neural networks more transparent and understandable, facilitating their responsible and effective deployment in various applications.
BACKGROUND
The goal of the mechanistic interpretability field is to develop tools for understanding the behavior of neural networks. The main challenge with naive model interpretation approaches stems from the polysemanticity of features in trained models, which states that each feature is linked combination of unrelated inputs (Bricken et al. 2023). One explanation for polysemanticity is superposition hypothesis, which suggests that the number of features trained by a model exceeds the number of
neurons in that model (i.e., the size of its hidden layer) (Arora et al. 2018 Elhage et al. (022). In this scenario, the features might be represented using not-oortooannl basis the hidden space, making them challenging to interpret with simple methods.
correct,.Bricken Assuming the superposition hypothesis correct, Bricken al. (023) used Sparse Autoencoders (SAEs) Yunn al., 2021) to uncover features in language models that are comprehensible to humans. Given the assumption that models capture more features than their hidden dimensionality allows, unveiling these features can be achieved by training with a large representation size (Templeton et al. (2024) used representation sizes orders of magnitude larger than model's hidden size to interpret the Claude 3 Sonnet model):
autoencoder
the hidden states of model
f(x) (W en + bene),
((f) = Wder f + bives
(I)
f(x) E R x BEERRF
Here, is an activation function (e.g., ReLU). Importantly, f(x) is made be sparse help the SAE capture monosemantic features. The final loss is defined as L(x) = MSF(æ, ( (x))) + Xrreg- The sparsity of the model can be increased by adjusting the À hyperparameter, which balances the trade-off between the reconstruction term and the regularization term. However, determining the optimal value for is not straightforward.
Building on this approach, Lieberum et al. (2024) introduced a family of SAE models trained on the hidden states of the Gemma model for each layer. These models, along with detailed feature descriptions, are available on Neuronpedi:!" Notably, this work departs from previous methods by employing JumpReL.U (Rajamanoharan et al. 2024) as the activation function:
JumpReLU(z)+ = z @ H(= 0), R.:
(2)
where H(-) is the Heaviside step function, and represents learnable thresholds for each component of z.
2

---

Published as a conference paper at ICLR 2025
200
[h
Dato mabia
Ho.bn 2000 7077
Hitton soae Prt.enottoooooooooooo
0115
100
500
0.5
Figure 1: Left: Hidden state norms and the mean 0 value in trained JumpReLU activations within SAE modules. Right: Dynamics hidden state norm changes and the differences in norms of matched decoder columns Wdec after the folding operation. These results suggest that O captures the growth of hidden state norms. After folding 0 into the weights, the decoder weights Wee become dependent the dynamics of hidden state norms, leading lower overall MSE during matching. For more details on the folding operation and method description, see Section
MATCHING SPARSE AUTOENCODER FEATURES
While SAEs can extract human-intergn interpreatbeee features from specific layers of a language model, understanding how these features evolve across layers remains an open question. To address this, we introduce SAE Match, a data-free method align SAE features across different layers. This approach enables to analyze the evolution of features throughout the model's depth, providing deeper insights into the model's internal representations.
Our central idea is that features from different layers might be similar but permuted differently that is, the i-th feature in layer A might correspond the j-th feature layer B. Therefore, aligning features across layers involves finding the correct permutation that matches semantically similar features.
Hypothesis 1: Features f(A) from layer A can be matched with features f(B) from layer B by calculating the mean squared error (MSE) between the relevant SAE weights. These weights might be either the rows of the encoder weights WiA and (( or the columns of the decoder weights w(A) and W(B) (or both). Rows columns that have a low MSE between them suggest semantic similarity between the features.
This matching problem resembles the task of aligning weights from two neural networks that perform the same function but have permuted weights (Ainsworth et al.) 2022). In our case, we aim to find permutation matrix (A) E Pp (the set of permutation matrices) that aligns the features of layer A with those of layer B. Formally, seek a permutation that minimizes the MSE between the decoder weights:
p(A-B) = arg min
(A) (A) dec..: dec,, 112 arg max
PW (B)
P, (wild) w((B)
T
(3)
doc
PEP
PEP,
where B) = Ei, Aij is the Frobenius inner product. To solve this, one can utilize Linear Assignment Problem (LAP) solver (Ainsworth et al. 2022). The same logic applics to the encoder weights, provided they transposed.
3

---

Published as a conference paper at ICLR 2025
Lowest MSE
Distinct reconstructions
W Inc.
*
Matching without folded parameters
s/4
W/e
Features with different
Weights with similar scales
Matched features recoustruction are distinet
Lowest MSE
Close reconstructions
wier
*
Matching with folder parameterrs
Win
Features with similar scalles
Folded weights with different.
Matched features recousteuction are close
Figure 2: A schematic illustration of the differences in SAE matching with and without folded parameters. When no folding is performed (top), 0 encapsulates differences in hidden state norms, causing features f(A) and f(B) to have different scales, while the columns of decoder weights w(A) and W have similar norms. Matching similar columns leads to differences in the actual dees,
dees.,
reconstructions of the imput æ. which we hypothesize detrimental for matching SAE features. With folding (bottom), we transfer the differences in input (and thus feature) norms to the decoder weights w and w dec..." thereby matching features while accounting for differences in input norms. As result, reconstructions matched features are closer each other than in the unfolded variant of the algorithm. See Section 3.1 for more details.
doc.,:
3.1 FOLDING SPARSE AUTOENCODER WEIGHTS
While the above method can be directly applied, it does not utilize the information stored in the learnable thresholds 0 of the JumpReLU activation function. Moreover, differences in feature scales can lead to suboptimal matching when minimizing MSE between vectors with varying norms. To address this, we propose technique called parameter folding, where we incorporate the activation thresholds into the encoder and decoder weights:
( 0
1 W/eec Witee (Oag 0' 1,
W Wone one diag
b'nc
Done
where diag(O) (0) creates diagonal matrix from 0, and O denotes element- dlement-wise multiplication. This transformation does not alter the SAE's output but adjusts the weights to account for differences in feature scales.
Our hypothesis that parameter folding helps align decoder weights in a way that reflects the actual dynamics of hidden state norms during model evaluation (see Figure I.. We observed that 0 values, fact, encapsulate the growth of hidden state norms. By folding into the weights, we normalize the decoder weights align with these dynamics. This process allows us match decoder weights based on their MSE in the space of actual hidden state norms, By folding parameters, we can match features across layers data-free manner, without needing input data to capture input scales. For schematic illustration of behavior, refer to Figure 2
Hypothesis 2: Folding the activation thresholds into the weights improves the quality of matching.
By incorporating the folding process, we ensure that the comparison decoder weights is sensitive to the scale of the hidden states, rather than relying solely on angular proximity. This allows for a more accurate reflection of the underlying feature dynamics and enhances the model's 's ability to effectively match features across layers. Consequently, by aligning decoder weights to hidden state norms, we achieve a more robust evaluation feature similarity, improving the overall performance of the Sparse Autoencoder model.
4

---

Published as a conference paper at ICLR 2025
3.2 COMPOSING PERMUTATIONS
When a permutation is obtained from layer A to layer B, and another from layer B to layer C, it allows approximate permutation from layer to layer C by composing these two per- mutations p(A-c) x p(B-C) p(A+B) ((A-B) instead of evaluating actual permutation between layers and C. Extending this to a model with T layers, we define all possible permutation paths as: P= (p i E [0:j)].
Hypothesis 3: We can approximate the matching between any two layers, A and C, by composing permutations. However, the quality of this approximation decreases as the relative distance between these layers increases.
By examining these composed permutations, we aim to uncover deeper insights into feature dynam- ics ics-how how certain features retain transform their semantic meanings as they pass through multiple layers of the network.
EXPERIMENTAL SETUP
We conducted experiments using Sparse Autoencoders (SAEs) trained on the Gemma model, with feature descriptions from Neuronpedia generated by GPT-4o. We focused on SAES with descriptions provided by Neuronpedia (for complete list of SAEs used, refer to Appendix Table For matching we used MSE from both decoder and encoder layers. During our initial experiments, we observed that the decoder-only option performs similarly to our scheme, while the encoder-only suffers from poor quality of matching (see Appendix Figure [2 for comparison). A more detailed analysis of the differences in matching using various sets weights deferred to future work.
To measure the quality of feature matching, evaluated the quality of the match using two key metries:
1. Mean Squared Error (MSE): The error between the permuted parameters of the matched features.
2. An external large language model (LLM) compared Neuronpedia descriptions of matched features, categorizing them as "SAME" (identical meanings), "MAYBE" (possibly similar meanings), or "DIFFERENT" (distinct meanings). Each experiment involved approximately 1,600 LLM evaluations over 100 feature paths spanning 16 layers (details in Appendix Section
When assessing how approximating hidden states with matched features affects language modeling performance, used:
Change in Cross-Entropp Loss (AL): The difference in loss (in nats) between the model operation and the original model.
using the cercode-permut-ddee
Explained Variance: Calculated as - Var(x((+))). This metric assesses how well the approximated hidden state (t+1) estimates the true hidden state x providing insight into the accuracy of the approximation.
Matching Score: The probability paired feature activation between two matched layers.
We tested our methods subsets of WebTeeteext (Gokaslan al. 2019). Code and WikiText (Meriyy et al. 2016).. From each dataset, we randomly sampled 100 examples, truncated them to 1,024 tokens, and excluded the gegginingg-o-seqqenneg (BOS) token when calculating metrics.
5
EXPERIMENTS
5.1
UNDERSTANDING PARAMETER FOLDING
We start by exploring parameter folding. understand it, we used our method to create permutation matrices p()-1-4) t [1:26] based on the MSE values for folded and unfolded parameters.
https::/hhugiinggface http:///wgggiaaaa...cooaaaaaaa/a
thub- smal 1--nar--eddin dedup
5

---

Published as a conference paper at ICLR 2025
0.33
W. W.meMMSE
bLons NSE
VV Vo.derte NOF
10
Lager
Laghr
Layer
Figure 3: Results of the MSE objective for different layer matching methods. "Vanilla matching" refers matching without any permutations. The "Matched" and "Folded+Matched" variants correspond to unfolded and folded matching, respectively. In all cases, MSE is evaluated with folded parameters (i.e., for unfolded matching, parameters are first matched, then folded, and finally MSE evaluated). When considering input scales differences (see Section 3.T). this can be interpreted as the MSE in the scale of actual input reconstructions in the relevant layers. The unfolded matching consistently showed higher MSE in this scale, supporting Hypothesis 2
Note that baec is omitted as
does not affect the order of features in the SAE layer. For further details, refer to Section 5.1
Manched
Folded Matched
10:
50
loyee liyee
CorFt
CAART
30
OFFERONT
OFFERONT
Figure 4: External LLM evaluation split by layers. As before- folding thresholds results in an optimistic labeling, it also affects deeper layers making them also more optimistic.
As Hypothesis relies on the observation that 0 incapsulates differences in hidden state norms, we evaluated MSE of obtained permutations. More concretely, we evaluated MSE for folded matching, and for unfolded and naive (without any permutations) matching strategies, for which we first permuted SAE weights based on unfolded MSE values, and then evaluated MSE of appropriate folded weights. See Figure3 for the results. Initially, naive matching displayed high MSE across all SAE parameters, indicating the lack of assurance in maintaining feature order between layers. Unfolded matching showed higher MSE in the scale of hidden state representations. Also note that since MSE is plotted with folded parameters, the MSE of the Encoder layer decreases, while the MSE of the Decoder layer increases (this behavior is explained by Equation
We compared the performance of folded and unfolded matching based semantic similarity with an external LLM. The results can be seen in Figure for LLM evaluation across different layers. Folded matching exhibited improved feature matching quality compared the unfolded variant. This result supports Hypothesis 2 regarding the behavior of matching with folded parameters.
Also, we observed performance drop the 10-th layer, which we study in Section 5.3

---

Published as a conference paper at ICLR 2025
3402200 FOERE SuPicubiity Herrer and itfennnct AMoCtAAE with individues
240
13000 ueturrsstt to spott plaaers purticuuuly thai stas and achevverana
1.00
182405 tomore andt touth and salety rouus
800
2622998 keperdortiooal cancer, frad mediaal
ENIRRL edttac relone to sigitiatiin and eoaarh Wecte and
400
2011 less: fedt posmetee infonss Saleddoce (ats maragee
28P18 reforence selstaal exends and time anc
3312573 seckiiia Piatinn purendaus sollerr atrotunn
33:5354 clmman rnlecad lo data aor uctare durrabann and SQerRr018 prggunniig progtammiing
0.02 0:86 3.50
1.200
5:025 0.6 0.04
Figure 5: Features matched with folded parameters from the 19th to the 20th layer using the proposed method are sorted by their MSE values across the relevant SAE decoder weights. Features with small MSE values (on the left) indicate semantic similarity, while those with large MSE values (on the right) indicate that no similar features were found. For further details, refer to Section 5.2
Figure 6: The distribution of MSE values with folded matching, grouped by three labels provided by an external LLM, is presented. Lower MSE values between pairs of layers indicate better matching. Note that the MSE value distribution varies across different layer pairs due to differences in weight scales For further analysis, refer to Section 5.2.
5.2 FEATURES MATCHING
Once we estabilished that folded parameters provide more accurate feature matching, we dive into exploring feature matching itself. We tested our hypothesis that features from different layers can be matched by calculating the mean squared error (MSE) between the parameters of Sparse Autoencoders (SAE). In this experiment, we focused on matching the 19-th and 20-th layers using parameter folding. The results are shown in Figure 5 Low MSE values indicated semantic similarity between features, while high MSE values suggested differences.
We also evaluated the distribution of MSE values for three pairs of layers matched with folded parameters, grouped according to external LLM evaluation. The results, shown in Figurelo, similarly revealed that lower MSE values were associated with the semantic similarity of matched features. It is important to note that the distribution of MSE values across different layer pairs varied due to differences in weight scales (as described in Section 3.T. See Appendix Section B for experiments with MSE thresholds.
Together, these results supported Hypothesis D confirming that MSE reflects feature similarity and enabling us to further investigate feature matching.
5.3 ON THE PERFORMANCE OF MATCHING ACROSS LAYERS
In previous sections, we observed a decline in the quality of feature matching evaluated by an external LLM in the lower layers of the model, particularly up to the 10th layer. To investigate this phe-
7

---

Published as a conference paper at ICLR 2025
FMACE Nakhed
Figure 7: Left: External LLM evaluation of feature matching across layers with folded matching. Right: Matching Score across layers and across different matching methods. Folding parameters has a positive effect on Matching Score. The LLM evaluation shows near-zero "SAME" features in the initial layers, increasing after the 10th layer. The Matching Score shows similar pattern but does not drop to zero in the initial layers, indicating some predictive ability. These results suggest that initial layers may have more polysemantic features, making them harder evaluate using Neuronpedia descriptions. See Section 5.3 for more details.
nomenon, we extended our evaluation of folded matching across all layers of the model-excluding the final layer responsible for mapping to the vocabulary. We also examined the Matching Score metric (as defined in Section to gain additional insights into the matching performance.
The results are presented in Figure We observe that the number of "SAME" features those with identical or closely related meanings- cenrsi-rmmins near zero in the initial layers and begins to increase after the 10th layer. similar trend is seen the Matching Score, although does not drop to zero in the early layers, suggesting some ability predict features from previous matched features.
Based on these results, SAE Match fails to align similar features the initial layers. Our main explanation for this phenomenon lies in the differences in 4nnors at these early layers (see Figure 15 for more details). We observe that when utilizing canonical SAEs, modules in the initial layers exhibit large variance in lo values. If we utilize SAEs with similar norms each layer, the Explained Variance improves significantly. While we continue our experiments with canonical SAFs for consistency, this result suggests that these layers should be selected more precisely in future studies.
76
Figure 8: Results the external LLM evaluation. We compare features from 10th, 12th and 20th layers with matched features in subsequent layers. It can be observed that feature similarity gradually declines, but remains significant for approximately five layers. We use for DIFFERENT features, 2 for MAYBE features, and 3 for SAME features, and then average this score across layers. See Section 5.4 for more details.
8

---

Published as a conference paper at ICLR 2025
5.4 FEATURE PERSISTENCE
In purpose to study dynamics of the feature from layer to layer we decided to conduct additional experiments with feature semantics persistence along the feature path.
We used three starting layers 10, 12, and 20 to evaluate permutations with all subsequent layers. We matched features between layers by both evaluating full permutation matrix (Exact) and by approximating it with composition (Composition) of composite permutation matrices (see Section
3.2
We then evaluated these permutations with an external LLM.
Encodor
Pormutation
Decoder
Encoder,
Pormutation
Docodor,
Attention,
MLP.
Attenttonn
MLP
Figure 9: Schematic illustration of layer pruming with matched features. Instead of the standard evaluation (left), we compute features at layer t, apply the permutation to match them to layer 1 + 1, and then reconstruct the hidden state for layer + 1, effectively skipping layer t. See Section 5.3 for more details.
See Figure & for the results. We observed that composition of permutations showed adequate results for nearby SAEs especially the later layers, then it starts to diverge, which supports Hypothesis Also, features were well matched when evaluating full permutation matrices, supposing that in later model layers features between layers are similar, which aligns with observations from Section 5.3
5.5 SAE MATCH AS A LAYER PRUNING
To further evaluate the effectiveness of our method, conducted experiments where pruned layers between matched Sparse Autoencoders (SAEs). The key assumption is that features remain consistent during one-layer forward pass. If the permutation mapping between SAEs is accurate, we can approximate the output of a pruned layer using an encode mroode-permute-decoee operation:
(x) = (w ((t) eue
f((1)= = =P(+11
(5)
((++1 = w(t+ f((+1) + ((+1)
Figure 9 provides schematic illustration this layer pruning method. See Figure 10 for the results.
We observed the smallest performance drop the Open WebText dataset, while decrease in quality was noted at the last layer of the Code dataset. These findings indicate that the matched features share similar semantics, enabling to approximate the next hidden states for all layers beyond the 10th layer. Also see Figure TT for the Explained Variance and the change in loss across layers when using matching strategies. as well as with plain encoding and decoding of hidden states. We observed that at lower layers, matching yields lower explained variance compared to encoding and decoding the previous hidden state. This can be explained by errors introduced by the matching algorithm at these layers.
6 LIMITATIONS AND CONCLUSION
We introduced SAE Match, novel data-free method for aligning Sparse Autoencoder (SAE) features across neural network layers, tackling the challenge of understanding feature evolution amid polysemanticity and feature superposition. By minimizing the mean squared error between folded
9

---

Published as a conference paper at ICLR 2025
7
Figure 10: Left:
Change in cross-entropy loss (AL) after pruning each layer using matched features. Right: Explained variance of the approximated hidden states compared to the true hidden states. Starting from the 10th layer, pruning results in minimal performance loss, indicating that matched features effectively approximate the skipped layers. See Section |5.5 for more details.
$4C240
PnSEee FOOD
The change in cross-entropy loss (AL) with each layer of the SAE. Right: The explained variance of the approximated hidden states for each layer of the SAE. We present results for plain encoding and decoding of the current and previous hidden states, as well for matching in both folded and unfolded variants. While folded matching eventually matches the results of encoding and decoding the previous hidden state, at lower layers introduces errors that reduce performance.
Figure 11: Left:
SAE parameters- which integrate activation thresholds into the weights we accounted for feature scale differences and improved matching accuracy. Experiments the Gemma 2 language model validated our approach, showing effective feature matching across layers and that certain features persist over multiple layers. Additionally, we demonstrated that our method enables approximating hidden states across layers. This advances mechanistic interpretability by providing a new tool for analyzing feature dynamics and deepening our understanding of neural network behavior. We also showed that this approach can be applied not only to SAEs trained with the Gemma 2 model (see Figures
In the current version of the paper, we utilized only scheme where we matched features based on all weights of the SAE (encoder, decoder, and biases). Future studies may include a more precise understanding of the behavior of different weights within the module. As we pointed out in Section 5.3 selecting appropriate SAEs based on their lo value also a topic worth discussing. This value can be seen not only as trade-off between sparsity and reconstruction quality, but also factor that affects matching features across layers. While in this work we focused on bijective matching to perform step-by-step understanding of SAE features, is evident that some features not matched in this way. Therefore, one may want to explore more sophisticated methods to understand feature dynamics.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Samuel K. Ainsworth, J. Hayase, and S. Srinivasa. Git re-basin: Merging models modulo permutation
symmetries. International Conference on Learning Representations, 2022. doi: 10.48550/arXiv. 2209.04836.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational doi: 10.1162/tacl a 00044. URL
Linguistics, 1:483-495, 2018.
org/018-1034
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. features/index.htm.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Satfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposi- tion. Transformer Circuits Thread, 2022. URL
pub/
2022/toy wttttooooooeeeieieeeeeeeeee html
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus, http 7/Skylion007 git.ub........... io httpoo/oooopppeeeeeeeettrrrrrpp 2019.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024. URL http:ta///aaxwwvoor////bb
0408.05144
AI @ Meta Llama Team. The llama 3 herd of models, 2024. URL htt.ttt://aaxii.oogg//// 2407. 21783
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, arXiv preprint arXiv: 2407.14435, 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk. Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgeni Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezkk-Plucinsst Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre. Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow,
Harleen Batra,
11

---

Published as a conference paper at ICLR 2025
Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samanch Saadat, Sara Mc Carthy, Sarah Perrin, Sébastien M. R. Amold, Scbastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alck Andreev. Gemma 2: Improving open language models at a practical size, 2024. URL https: /tt:://axxin org/ abs/ 2408. 00118
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Trans- former Circuits Thread, 2024. URL Bealiingmmoomeeennniictyty/yyy/m
.acisssppbbb/// pubp 2024/
html
Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding as linear superposition of transformer factors. In Eneko Agirre, Marianna Apidianaki, and Ivan Vulić (eds.), Proceedings of Deep Learning Inside Our (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 1-10, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021..6
URL https: // www.thooog...rooo
dee 1eelio-1.1
Layers 19 20
bxccke
0.5 Decoder
0.4
02
0.1
SAE Modza
Figure 12: Matching Score evaluation of feature matching with different modes: encoder-only. decoder-only. and encoder-decoder matching; evaluated between 19-th and 20-th layers. Encoder- only suffers from poor performance, while decoder-only performs on par with the encoder-decoder scheme.
A ON THE MATCHING WITH DIFFERENT SPARSITY
In this section, we investigate how varying the sparsity level of Sparse Autoencoders (SAEs) affects the quality of feature matching across layers. The Gemma Scope release (Lieberum et al. 2024) provides SAES trained with different levels of sparsity, measured by the mean lo-norm of their
12

---

Published as a conference paper at ICLR 2025
Figure 13: Left:
The change in cross-entropy loss (AL) with each layer of the SAE. Right: The explained variance of the approximated hidden states for each layer of the SAE. Results are for LLAMA-3. 3.B-BB model
Bytt drip
anoode deorde cureet hablen
decode
58
Explained Varance
layer diop
enoodo docode cursrnt raddon
decode
07
Figure 14: Left: Change in cross cross-entropy loss (AL) with respect to the sparsity level (mean ly lo-norm) of the SAE. Right: Explained variance of the approximated hidden states different sparsity levels. Interestingly, explained variance has peak on value at lo as 70. See Section A for discussion.
activations. Understanding the impact of sparsity is crucial because it influences the number of active features and, consequently, the effectiveness of our matching method.
To evaluate approach under different sparsity conditions, selected five SAEs trained on the 20th and 2 layers of the Gemma model, each with a different mean no-mm.. We applied our feature matching method to these SAEs and measured performance using the metrics described in Section 4 Specifically, we assessed how well the matched features could approximate the hidden states when a layer is omitted.
layer drop: We skip the t-th layer entirely, using the hidden state (f as input for the (t + 1)-th layer.
encode-decode current hidden: use SAE reconstruction (+) hidden state at t-th layer instead of x
use SAE reconstruction from t-th layer 401 as a hidden state x (+++1) and omit evaluation of (-th layer.
encode-decode previous hidden:
First, when analyzing the change in cross-entropy loss (AL), observe that higher mean 4 no-m values correspond to a lower difference in loss across all methods. This expected because SAEs with higher sparsity (i.e., lower normm)) may not reconstruct the hidden states effectively, leading
13

---

Published as a conference paper at ICLR 2025
to higher AL. Conversely, SAEs with lower sparsity (higher (e-norm) capture information, resulting in better approximations and lower AL.
Notably, using features from lower layers approximate subsequent layers leads to higher AL compared to both the encode-decode current and encode-decode previous hidden schemas. This indicates that directly reconstructing the hidden state of layer using its own SAE provides better performance than attempting to predict from earlier layers.
Regarding the explained variance-whirn measures how well the approximated hidden state a ((+11) captures the true hidden state (r((+1)__ we observe that it peaks when the mean ly-norm is approx- imately 70. This suggests that there is an optimal sparsity level where feature matching is most effective. At lower fg-norms (higher sparsity), SAES may fail to reconstruct hidden states adequately due to insufficient active features. On the other hand, when the number of active features exceeds 100, accurately matching all features between layers becomes more challenging. This can lead to the inclusion of noisy features, which negatively impacts the estimation of the hidden state, pushing it in the wrong direction. As other schemas do not utilize matching, we do not observe peak as for matching plot.
This experiment underscores a key insight of our work: the sparsity level of SAEs plays a crucial role in the success of data-free feature matching across layers. It emphasizes that not only does method effectively align features across layers, but it also allows for the identification of optimal model configurations that balance sparsity and performance, further advancing the interpretability and efficiency of language models.
140
Lager
Lager
Figure 15: Left: The lo norms of the SAEs used in Neuronpedia and the most similar SAEs in GemmaScope (Lieberum et al. 2024). Right: Explained variance of the permutations. To analyze the behavior of the matching in the early layers, we conducted an additional experiment where we used a different SAE compared to Neuronpedia. We hypothesize that if two SAEs have different lo norms, then our approximation of the dynamics through permutations is less effective. To test whether method performs better under these conditions, selected SAEs with the most similar lo norms (see the red line in the left figure) and obtained permutations for this set. As shown in the right figure, reducing the "bumps" in between layers leads to better explained variance metric. See Section for more details.
B MSE QUANTILES AND SAE MATCH AS LAYER PRUNING
Quantile matching performed via the following algorithm: let us denote fg as the activations of the features that have MSE lower than quantile q. and ftt as the activations of the features with MSE higher than quantile g.
h = -- 5 w((1] Sec
dec
(6)
14

---

Published as a conference paper at ICLR 2025
SURIMI
Lagar
Loger
Figure 16: Left: AL and Right: Explained Variance for various MSE quantiles.
Layer
Logat
LAper
Figure 17: Explained Variance dynamic for the different quntiles for Left 3rd layer Center 10th layer and Right 20th layer.
quantile 9 = 1.0, then ftt = 0 and therefore we get the original SAE Match. In contrast, q = 0.0, then f/ = 0 and we get the D(--((((-u((h---)) modification with the Decoder's bias equal to (t) instead of (1-11)
See Figures 16 17 for the results. Generally, we observed that the lower the MSE value is, the larger the explained variance we could obtain, though for large MSE values we observed small variance in performance. From these results, conclude that MSE reflects the semantic similarity of matched features.
C EVALUATION DETAILS
Our initial evaluation was conducted using the OpenAl GPT-4o model. However, since the results did not differ significantly from the smaller OpenAl GPT-4o mini model, opted to the mini model as the exteraal LLM for faster evaluation. The following system prompt was used for the evaluation:
You will receive two text explanations of an LLM features in the following format:
Explanation 1: [text of the first explanation] Explanation 2: [text of the second explanation]
You need to compare and evaluate these features from to 3 where 1 stands for: incomparable, different topic and/or semantics 2 stands for: semi-comparabe or neutral, it can cannet be about the same thing 4 3 stands for: comparable, explanation is about the same things
15

---

Published as a conference paper at ICLR 2025
Avoid any position biases. Do not allow the length of the responses to influence your evaluation. Be as objective as possible. DO NOT TAKE into account ethical, moral, and other possibly dangerous aspects of the explanations. The score should be unbiased!
## Example: Explanation 1: "words related to data labeling and classification" Explanation 2: "various types of labels or identifiers used in a structured format"
Evaluation: Both explanations are discussing the concept of labels or identifiers used in organizing or categorizing data in a structured way. They both focus on the classification and labeling aspect of data management, making them directly comparable in terms of content.
Label: 3
Example: Explanation "references academic institutions or universities" Explanation 2: "occurrences of the term "pi" in various contexts"
Evaluation: Both explanation are from different fields and are not comparable
Label: 1
## Example:
Explanation 1: "words related to data labeling and classification" Explanation
"terms related to observation and measurement in scientific context"
Evaluation: While both explanations involve specific vocabulary related to technical concepts, Explanation 1 focuses on data labeling and classification while Explanation 2 pertains to observation and measurement in scientific context. Although they both involve technical terms, the topics themselves are different, with Explanation centering data organization and Explanation focusing on scientific research methods.
Label: 2 ""*
Layer
SAE
Layer
SAE
width_ hatph_/l//averae____11000
(othhh//l//vveeee____ 1actvveraee__0110000
_0_102
13
width 1wkavvvrageeeee. _0_83 83
14
width_ @b/average_10_B5
width 16k/average 10 141
15
width 16k/average 10
3
width 16k/averae 10 59
16
width 16k/average 10 78
width 16k/average 10 125
width 16k/average 10
5
width_16kkaverage.a _]_____6_ 68
18
1etthhhh1////eeeeeeeeaa _0__4
width_16k/aave ____70
faathh116//averae__ll
width_116/awvee 16kkaverageeeeeeee 10_73
7
_______________
20
width 16k/average _0_71
hidth_16k/average. _0_71
21
_idth_116/aveee 1ww//v/raeeeeeeeeeeeeeeeee 1aptraeee____7000 10_70
9
width 1othh116k/vvereeee 10_73
22
width 16k/averageeeeeeeeeeeeereera 10 |0_72
10
width_ 1athh__kk//veere_____1 _0_77
23
width 16w/averaeeeveeeeee 10_75
11
Midth_16k/averagg _aaeeeerree____888888 10_80
24
width_16kkawerran _0_73 73
12
width_ Sndth_16k/average__00 _0__2
25
width_ width_16kkaverrae__0000 10_
Table 1: Full list of SAEs we used for our experiments.
See https://huggiigface.coo
gooq le/ http://aaaopppp--tt----e---- for more details.
16

---

Published as a conference paper at ICLR 2025
D PATH EXAMPLES
Bellow are the path examples obtained via permutation composition.
anghasidnng progresaion
alemends rebited to intarr neverieeee and iesteringggggg aciion processns
2-35 Je-pomsorr- PO
words Irdicainng Caucadion or conclations
prease related videc recerting and interdoess ressarch
Al wo-tomsaaaruuoooo 2007 Je1o
PIOTIVE
BOSITYY Aet
Pastoutti
corqanctona irtorationa presees indicate sandusio
priases procestes dicernntttion oinervation
2012
PIOTIVE
HAMTINL LEIT'S
PASTIR LNITS
VCT EUnOTr com o
varitioom indinting craphado
HEMPTON LEST'S
FACTUR LOOTS
SUBITY L.cen
HEOTTEE LEST'S
PARTIO LISTES
SENETTY
reforencee collectioo
MISTOME LIATT
PACTURR LICTT
SMETT LLDO
Jellan an illated sagireent process Tow
UNIT'S
PROTONE
LIEENO
MELATIVE
PESTILE LOETE
BIAETT L.L.
varbo indicating actiona stateo
duratione 1NR9
foree verb Too les and phrenes indiosiine states being
intated time durution schetsllg
Aspreedions ARVOEG ASAEAS axparinicn
mendionn of Vina durationa and Intervale
enstiional express lors of pride and grethaue
guanidative resuunmeent relited to time and Enstacce
coriugatel fermes the work
mamerica relsedd reforansne, pantoutarly years ransthc
HEATINC LSETS
PESPINE UNETS
KOOSTY KUOO
Figure 18: Examples of paths first, second, third and fourth features from 20th layer to 25th layer for Gemma-2-2B model.
17

---

Published as a conference paper at ICLR 2025
Aetarco FIDAY mode to
Welnecer cleaay nappoese andh rikene Lboies
(149))
wewacce pockss de sssvciates ardidik manii mork
straes watr cebets castarca
Milrece conkall avet Bice
rnternccro cauure meree acriom
onder
Wifermee cergal ahile pu
LSMB coveeett oiiire cli0
amt wardd coeegg and accoerscc gevemance
werencer fhe seatr prodictiss simouty
2011
ieteecca oytemic amnatian caemmoee
Ihe Ch-Lumati
inferseet beakeney vn and profecttonn
This feature lnown acthurdans.
Figure 19: Examples of paths first, second, third and fourth features from 10th layer to 28th layer for ALLAA-3311-8 model.
18


Reviews:
[{"review_id": "2DTvUV80O1", "reviewer": "Reviewer_3W8E", "summary": "The paper introduces SAE Match, a technique that produces a bijection from [sparse autoencoder (SAE) features in layer n] and [SAE features in layer n+1], with the objective of reducing the average squared distance (i.e., mean squared error (MSE)) between a feature and its target. They also introduce parameter folding, which seeks to augment SAE Match to work on features with different scales. The paper shows via experimentation on Gemma Scope SAEs  that features have more similar LLM-derived interpretations if they have lower MSE, and that SAE Match with parameter folding results in more similar interpretations.", "questions": "Questions for the authors:\n\n1. Parameter folding serves as a form of normalization for encoder/decoder weights (as is mentioned in lines 156-157). What happens if one instead matches feature with encoder/decoder weights normalized to be unit vectors? This would be equivalent to maximizing cosine similarity instead of minimizing MSE, as ||x-y||^2=||x||^2+||y||^2 -2 ||x||*||y||*cossim(x,y).\n\n2. One might expect the set of features to change across layers as the model processes information. In that case, the \"correct\" form for a matching might not be a bijection. Could one instead define a non-bijective \"matching\" via P(f)=argmin_i ||f-g_i||_2 where f is a feature in layer n, and {g_i}_i=1^k is the set of features in layer n+1? How does this compare to Feature Matching?\n\n3. In Section 5.4, Figure 8, the y-axis is labelled \"GPT Score\". What is that metric and how was it calculated? Previous metrics have been a \"GPT Score\" on a scale of Different/Maybe/Same, or Matching Score in a range of 0-1, but this does not appear to be either of those.\n\n4. In Section 5.5, it seems Layer Pruning may introduce error in two ways: 1) feature activations in layer N do not result in perfect reconstruction of the residual stream even for layer N, and 2) features in layer N do not perfectly match features in layer N+1. Did any experiment disentangle those effects? For instance, what is the \\Delta L from replacing x with \\hat x? This would provide useful context for the quantities shown in Figure 10.\n\n5. Many SAEs have \"dead\" features, which presumably contribute little to the matching process. Could SAE Match be modified to exclude \"dead\" features, and if so how?\n\n\n\nIf it is permitted for authors to make revisions before the final submission, there are several small changes that could improve the quality of the paper:\n\n- Line 93: The loss function as written is incorrect. The L2 term is squared, and what is written as L0 should be L1. See e.g. Equation 4 in (Cunningham et al, 2023) (https://arxiv.org/pdf/2309.08600). \n\n- Line 132-133 (equation 3): The L2 norm in the argmin needs to be squared to get the mean *squared* distance.\n\n- Lines 147-149 (equation 4): It appears that b_dec should be b_enc.\n\nReferences:\n\nWes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Trans. Mach. Learn. Res., 2023, 2023. URL https://openreview.net/forum?id=JYs1R9IMJr.\n\nHoagy Cunningham, Logan Riggs Smith, Aidan Ewart, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=F76bwRSLeK.\n\nSamuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, Aaron Mueller. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. 2024. https://arxiv.org/abs/2403.19647", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper's SAE Match technique appears to work well at finding corresponding SAE features between layers, which can contribute to the goal of mapping a \"feature circuit\" (as in Marks et al. (2024) https://arxiv.org/pdf/2403.19647). This technique is data-free, meaning only the SAE weights are required, and not any model or SAE activations.\n\n- The paper also provides useful empirical evidence that SAEs find features between layers that are simultaneously 1) close in the space of parameters, and 2) interpreted to have similar meaning.", "weaknesses": "- The authors get far worse results on layers 0-9 of the model than on layers 10-25, indicating that the technique may not fully generalize. The authors claim that this is to be expected, saying \"This phenomenon aligns with findings from previous research. Gurnee et al. (2023) also reported increased polysemanticity in the early layers of neural networks.\" This explanation is unsatisfactory because Gurnee et al. (2023) were working with LLM neurons, not SAE features. Additionally, Cunningham et al. (2023) found that earlier layers were more interpretable (see Figure 2 in https://arxiv.org/pdf/2309.08600).\n\n- The proposed technique of parameter folding is only defined for SAEs using the JumpReLU activation function, and it is not clear how could be adapted to over activation functions like ReLU or TopK.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QBwVnVfBDi", "reviewer": "Reviewer_mxq3", "summary": "The authors define a similarity metric between two learned encodings, based on permutations. They find that, specifically for the JumpReLU architecture, θ tracks the growing residual stream norm. Accounting for this observation by normalizing weight vectors with θ improves permutation matching. They find that W_dec MSE decreases substantially for high layers, while not making differences for early layers. They further employ an LLM to judge the semantic similarity of matched features. Interestingly, they find that feature matching works well in early layers (<10) as per MSE but show that feature descriptions do not match. Additionally, they compare matching features across multiple layers by exact match and layer-wise composition, and investigate SAE match as layer pruning technique.", "questions": "### Questions\n- I'm curious about the takeaways of the experimental results from Section 5. How do I interpret a scale of ∆L about 1? Do the authors believe their results indicate that layers between 10 and 20 can be collectively pruned with the SAE matching method?\n- I'd be interested in a discussion of why encoder matching performs worse than decoder matching.\n\n### Further notes\n- Line 154: Repetition of meaning in two following sentences.\n- Why is Hypothesis 2 formatted as a hypothesis? It seems like a definition of a method to me.\n- Line 234: Calling \"average ℓ0\" a regularization coefficient is misleading; calling it (average) sparsity is clearer.\n- Line 235: Do the authors refer to Equation 3 when mentioning MSE? A reference of that equation or a different naming would be useful, since MSE is often used for the reconstruction loss in the context of SAEs.\nI- 'm very curious how these results compare to findings with crosscoders (https://transformer-circuits.pub/2024/crosscoders/index.html). Matched features might share decoder vectors of a single crosscoder feature.\n- Section 5.3: Gurnee operates on MLP neurons, which are not incentivized to be sparse. I expect SAE features to be (more) monosemantic, so Gurnee's results ideally shouldn't apply to SAE features.\n- I'd be curious about a deeper investigation of the sharp increase in semantic feature similarity in Figure 4 left.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Main finding: Cosine similarity alone is not a great proxy for late layers, as residual stream norms increase. The authors propose parameter folding, which effectively addresses this problem for JumpReLU SAEs. Current work relies on cosine similarity, and I am convinced the field should adopt this proposed technique.", "weaknesses": "### Critiques that can be addressed in this paper\n- I am unsure whether the original hypothesis of permutation is answered. The term \"matching\" implies a binary measure of whether a feature mapping is true or false. This might require the introduction of a cutoff threshold, or applying a clustering technique. Otherwise, the framing of similarity measures might be clearer than permutation. I'm curious about the authors' opinion on whether there is a binary criterion for whether features do/don't match.\n- To quantify the fraction of \"true matches,\" the MSE over all features is too coarse; only the matching score distribution provides a clear picture of which fraction of features are \"well enough\" matches.\n- Line 315: \"...unfolded matching showed higher MSE in the scale of hidden state representations, supporting behaviour described with Hypothesis 2.\" Comment: This is not true for benc MSE.\n- I'm unsure about the statistical significance of the LLM evaluation. I understand that 100 out of 16k (<0.1%) SAE latents (aka features) were chosen. Did the authors choose these features at random? Increasing the number of SAE features would increase the significance of their findings. Using an LLM to judge the coherence of max activating examples of features would be an insightful additional metric that would track the suspected polysemanticity in early layers.\n- I do not agree that Figure 7 right reflects the findings in Figure 7 left, where the matching score suddenly increases at layer 10.\n- The quantification of \"too far apart\" in Hypothesis 3 would be a useful improvement.\n\n### Further suggestions (that can be addressed in future work)\nHow does the mechanistic permutability transfer to other SAE architectures? The only difference will be adapting scaling factors defined in Equation 4. It would be great to provide a generalized definition of parameter folding for all SAE architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e7YQADlQ9l", "reviewer": "Reviewer_hmcz", "summary": "This paper proposes a new method for comparing and matching features of SAEs of nearby layers in transformers. They propose scaling features according to the activation threshold $\\theta$ of a JumpReLU activation function in order to match features in a more natural norm for the underlying activations.", "questions": "Could you provide more direct evidence for Hypothesis 2, or provide a clearer argument for why your current result provide evidence for it? Could you provide more discussion of what the takeaways for Section 5.5 were intended to be?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "- Proposes a novel and interesting strategy for pairing features between layers.\n    - Studies some of the shortcomings (e.g. long tail of pairing 'failures') of this strategy.\n- The presentation is very clear and understandable.", "weaknesses": "- It would be good to spend more time justifying the hypothesies of Section 3. I do not think that the results in Figure 3 constitute much evidence for Hypothesis 2, since the reasoning here seems slightly circular - you propose parameter folding based off the observation that $\\theta$ tracks the activation norms, but then evaluate feature similarity using the same objective that you are explicitly trying to minimize. Therefore, it is trivially true that 'folding+matching' outperforms 'matching'.\n    - The evidence for Hypothesis 2 could be strengthened by including an analysis of how the scale of $\\theta_i$ tracks with the scale of feature $i$ *on the same layer* (and similarly measuring MSE by scaling the features according to data-dependent statistics instead of $\\theta$, e.g. by comparing MSE in the norm induced by some whitening transformation?); the observation in Figure 1 shows that *mean* $\\theta$ tracks with *mean* activation norm, but your hypothesis rests on the assumption that $\\theta_i$ is predictive of the scale of feature $i$ for features of the same layer.\n    - It would be more convincing if you backed up your claims in Figure 2 that \"reconstructions of matched features are closer to each other than in the unfolded variant of the algorithm\" by comparing the reconstruction loss (perhaps again under a norm induced by a whitening transformation?) of 'folded+matched' and 'matched' permutations when 'skipping' a layer, as in Section 5.5.\n- There was little analysis of the results of the experiments in Section 5.5. The loss differences resulting from using feature-matched activation reconstructions were not compared to any nontrivial baseline (e.g. perhaps something like a linear approximation of the layer?) and it was hard to see what the desired conclusion/hypothesis or suggestion at a direction for future study was here, and the paper might benifit from expanding on this a bit.\n    - It might also be useful to baseline the figures in Figure 10 using the reconstructions of an SAE at layer $t+1$, i.e. substituting $x^{(t+1)}$ with $W_\\mathrm{dec}^{(t+1)} \\sigma ( W_\\mathrm{enc}^{(t+1)} x^{(t+1)} + b_\\mathrm{enc}^{(t+1)} ) + b_\\mathrm{dec}^{(t+1)}$; it is known that SAEs achieve nonzero reconstruction loss and that this incurs a performance penalty when using the reconstructions in place of the original activations. Intuitively the penalty incured from using the matched activations is 'mixed' with this base reconstruction penalty.\n    - As stated before, it would also be good to include the 'matched' baseline in this section as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "srWl0uD66V", "reviewer": "Reviewer_UKCQ", "summary": "The paper presents a technique to match SAE latents from two SAEs across different layers called SAE Match. The technique finds permutations of latents which minimize MSE between the encoder and/or decoder representation of the feature in both SAEs. SAE Match addresses different layer norms by folding in the jumprelu threshold parameter into encoder and decoder matrices. The paper evaluates the technique on the Gemma Scope SAEs for Gemma 2.", "questions": "- In Figure 3, there are different results for folder and unfolded b_enc, but b_enc is not affected by folding according to equation 4. How is b_enc different due to folding?\n- In Figure 3, vanilla matching performs not much worse than the actual matching techniques. Is vanilla matching essentially equivalent to randomly pairing decoder rows? If so, why is the MSE loss so low?\n- In Section 4, it says SAEs with regularization coefficient near 70 are used. Is this referring to the L0 of the SAE in Gemma Scope? Neuronpedia only supports Gemma Scope SAEs with L0 closest to 100 - if Neuronpedia is used for all SAEs, does this mean that the L0 near 100 Gemma Scope SAEs are used as well?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Using MSE loss between SAE decoders is simple and fast. The idea of folding the jumprelu threshold parameter into weights to account for differences in layer norm is a really nice idea as well. Aligning SAE latents across layers is important for understanding how features progress throughout the foward pass of the model, so this work is useful to the field.", "weaknesses": "The paper only focused on a single model (Gemma) and a single type of SAE (JumpReLU). The paper evaluates on Gemma Scope, but uses only a single SAE per layer despite Gemma Scope containing multiple SAEs per layer with different sparsities. It would be a good check to see how the method performs comparing different SAEs trained on the same layer as well. The method seems like it does not address feature splitting, where a single latent in one SAE becomes multiple latents in a different SAE - the method seems like it will only pair latents 1-to-1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)