Interpreting CLIP with Hierarchical Sparse Autoencoders
Authors: Vladimir Zaigrajew
Hubert Baniecki
Przemyslaw Biecek
Venue: ICML 2025 poster (2025)
Keywords: SAE
CLIP
Interpretability
Matryoshka Representation Learning
Primary area: deep_learning->other_representation_learning
TL;DR: Matryoshka sparse autoencoder outperforms ReLU and TopK SAE, enhancing the interpretability of vision-language transformers through hierarchical concept learning.

Abstract: Sparse autoencoders (SAEs) are useful for detecting and steering interpretable features in neural networks, with particular potential for understanding complex multimodal representations. Given their ability to uncover interpretable features, SAEs are particularly valuable for analyzing vision-language models (e.g., CLIP and SigLIP), which are fundamental building blocks in modern large-scale systems yet remain challenging to interpret and control. However, current SAE methods are limited by optimizing both reconstruction quality and sparsity simultaneously, as they rely on either activation suppression or rigid sparsity constraints. To this end, we introduce Matryoshka SAE (MSAE), a new architecture that learns hierarchical representations at multiple granularities simultaneously, enabling a direct optimization of both metrics without compromise. MSAE establishes a state-of-the-art Pareto frontier between reconstruction quality and sparsity for CLIP, achieving 0.99 cosine similarity and less than 0.1 fraction of variance unexplained while maintaining 80\% sparsity. Finally, we demonstrate the utility of MSAE as a tool for interpreting and controlling CLIP by extracting over 120 semantic concepts from its representation to perform concept-based similarity search and bias analysis in downstream tasks like CelebA. We make the codebase available at https://github.com/WolodjaZ/MSAE.

Interpreting CLIP with Hierarchical Sparse Autoencoders
Vladimir Zaigrajew' Hubert Baniecki 12
Przemyslaw Biecek 1.2
Abstract
(A) Architecture
(B) Training
x A
Sparse autoencoders (SAEs) are useful for de- tecting and steering interpretable features in neu- ral networks, with particular potential for un- derstanding complex multimodal representations. Given their ability uncover interpretable fea- tures, SAEs are particularly valuable for ana- lyzing vision-language models (e.g., CLIP and SigLIP), which are fundamental building blocks in modern large-scale systems yet remain chal- lenging to interpret and control. However, current SAE methods are limited by optimizing both construction quality and sparsity simultaneously, as they rely on either activation suppression rigid sparsity constraints. To this end, we in- troduce Matryoshka SAE (MSAE), a new archi- tecture that learns hierarchical representations multiple granularities simultaneously, enabling direct optimization of both metries without compromise. MSAE establishes state-of-the. art Pareto frontier between reconstruction qual- ity and sparsity for CLIP, achieving 0.99 cosine similarity and less than 0.1 fraction of variance unexplained while maintaining 80% sparsity. Fi- nally, we demonstrate the utility of MSAE as tool for interpreting and controlling CLIP by extracting over 120 semantic concepts from its representation to perform concept- bastd similar- ity search and bias analysis in downstream tasks like CelebA. We make the codebase available at https:: /tt:://githbb com/ www/wolo//oooooooooooooooooooooo jolo /az/MSAE. /SSAE.
MSAE
21 Rd
Dec(z))
2(x Dec(s))
Eno(e
22 Rd42
E R44
Dec(z,)
Dec(s)))
Dec
E R
Dec(g,)
dz Dec(s))
Dec(z.)
a Fer, Dec(za))
(C) Reconstruction
(D) Manipulation
Enc(x)
Enc(x)
R
Edited
Dec(e)
Dec(:*)
Figure 1. Matryushka Sparse Autoencoder (MSAE) enables learning hierarchical concept representations from coarse to fine- grained features while avoiding rigid sparsity constraints in TopK and activation shrinkage problem ReLU SAE. (B) train- ing, MSAE uses multiple top-k values up to dimension d instead of a single & like in TopK SAE, combining losses across differ- ent granularities. (C) At inference, our method uses the whole d-dimensional representation. (D) MSAE allows for more precise editing and manipulation in the concept space.
learning robust representations that bridge visual and tex- tual information. Through contrastive learning on massive datasets, CLIP and its less adopted successor SigLIP (Zhai et al., 2023) demonstrate remarkable capabilities that ex- tend far beyond their primary objective of cross-modal sim- ilarity search. Its representation is foundational compo- nent in text-to-image generation models like Stable Diffu- sion (Podell et al., 2024) and serves as powerful feature extractor for numerous downstream vision and language tasks (Shen et al., 2022), establishing CLIP as a crucial building block in modern VLMs (Liu et al., 2023; Wang et al., 2024).
1. Introduction
Vision-language models, particularly contrastive language- image pre-training (CLIP, Radford et al., 2021; Cherti et al., 2023), revolutionize multimodal understanding by
Despite CLIP's widespread adoption, understanding how it processes and represents information remains challenge. The distributed nature of its learned representations and the complexity of the optimized loss function make it par- ticularly difficult to interpret. Traditional explainability approaches have limited success in addressing this chal- lenge: gradient- based feature attributions (Simonyan, 2013;
WWrssww University of Technology. Warsaw, Poland 2 UUniversity Warsaw, Warsaw, Poland. Correspondence to: Vladimir Zaigrajew <vladimir zaiidimmi zaigrajew. pw.edu.pl>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Matryoshka SAF
Shrikumar et al., 2017; Selvaraju al., 2017; Sundararajan al., 2017; Abnar & Zuidema, 2020) struggle to provide human-interpretable explanations, perturbation-baaed ap- proaches (Zeiler& & Fergus, 2014; Ribeiro et al., 2016; Lund- berg & Lee, 2017; Adebayo et al., 2018; Baniecki et al., 2025) yield inconsistent results, and concept-based meth- ods (Ramaswamy et al., 2023; Oikarinen et al., 2023) are constrained by their reliance on manually curated concept datasets. This interpretability gap hinders our ability to iden- tify and mitigate potential biases or failure modes of CLIP in downstream applications (Biecek & Samek, 2024). Re- cent advances in mechanical interpretability (Conmy et al., 2023; Bereska & Gavves, 2024) use sparse autoencoders (SAEs) as a tool for disentangling interpretable features in neural networks (Cunningham et al.. 2024). When applied to CLIP's representation space, SAEs offer the potential to decompose complex, distributed representations into human- interpretable components through self-supervised learning. It eliminates the need for concept datasets and limits prede- fined concept sets in favor of natural concept emergence.
Contributions. We introduce a hierarchical SAE archi- tecture that establishes new leading Pareto frontier be- tween reconstruction quality (0.99 cosine similarity and < 0.1 FVU) and sparsity (~ 80%). while maintaining com- putational efficiency comparable to standard SAEs at infer- ence time. We develop a robust methodology for validat- ing discovered concepts in CLIP's multimodal embedding space, successfully identifying and verifying over 120 in- terpretable concepts across both image and text domains. Through extensive empirical evaluation on CC3M and Ima- geNet datasets, we demonstrate progressive recovery capa- bilities and the effectiveness of hierarchical sparsity thresh- olds compared to existing approaches. We showcase the practical utility of MSAE in two key applications: concept- based similarity search with controllable concept strength and systematic analysis of gender biases in downstream clas- sification models through SAE activations and concept-level interventions on the CelebA dataset.
2. Related Work
However, training effective SAEs poses unique challenges. The richness of data distribution and high dimensionality of CLIP's multimodal embedding space require tuning the pparsityrecoonstructio trade-off (Bricken et al., 2023; Gao et al., 2025). Furthermore, evaluating SAE effectiveness extends beyond traditional metrics, requiring the discov- cry of interpretable features that maintain their semantic meaning across both visual and textual modalities. Current approaches for enforcing sparsity in autoencoders use either L, (Bricken et al., 2023) or Topk (Gao al., 2025) proxy functions, each with significant drawbacks. L1 regulariza- tion results in activation shrinkage, systematically underesti- mating feature activations and potentially missing subtle but important concepts. TopK enforces a fixed number of active neurons, imposing rigid constraints that may not align with the natural concept density in different regions of CLIP's embedding space (Gao et al., 2025; Bussmann et al., 2024).
Interpreting CLIP models. CLIP interpretability research follows two main directions: direct interpretation of CLIP's behavior and using CLIP explain other mod- els. Direct interpretation studies focus on understanding CLIP's components through feature attributions (Joukovsky et al., 2023; Sammani et al., 2024; Zhao et al., 2024), resid- ual transformations (Balasubramanian et al., 2024), atten- tion heads (Gandelsman et al., 2024), and individual neu- rons (Goh et al., 2021; Li et al., 2022). Li et al. (2022) discovered CLIP's tendency focus image backgrounds through saliency analysis, while Goh et al. (2021) identi- fied CLIP's s multimodal neurons responding consistently to concepts across modalities. For model explanation, CLIP is used to analyze challenging examples (Jain et al., 2023), ro- bustness to distribution shifts (Crabbé et al., 2024), and label individual neurons (Oikarinen & Weng, 2023). In this work, we explore both directions in Section via the detection of semantic concepts learned by CLIP using MSAE (Sec- tion 5.1) and the analysis of biases in downstream models built on MSAE-explained CLIP embeddings (Section 5.3).
To this end, we propose hierarchical approach to sparse autoencoders, new architecture inspired by Matryoshka representation learning & Kusupati et al., 2022), as illus- trated in Figure 1. While Matryoshka SAE (MSAE) can be applied to interpret any neural network representation, we demonstrate its utility in the CLIP's complex multimodal embedding space. At its core, MSAE applies TopK opera- tions h-times with progressively increasing numbers of k neurons, learning representations at h granularities simulta- neously - from coarse concepts to fine-grained features, By combining reconstruction losses across all granularity levels, MSAE achieves a more flexible and adaptive sparsity pat- tern. We remove the rigid constraints of simple TopK while avoiding the activation shrinkage problems associated with L1 regularization, resulting in the state-of the-art Pareto frontier between the reconstruction quality and sparsity.
Mechanistic interpretability. Mechanistic interpretabil- ity seeks to reverse engineer neural networks analogously to decompiling computer programs (Conmy et al., 2023; Bereska & Gavves, 2024). While early approaches focus generating natural language descriptions of individual neurons (Hernandez et al., 2021; Bills et al., 2023), the polysemantic nature of neural representations makes this challenging. A breakthrough comes with sparse autoen- coders (SAEs) (Bricken et al., 2023; Cunningham et al., 2024), which demonstrate the ability to recover monose- mantic features. Recent architectural advancements like Gated (Rajamanoharan et al., 2024a) and TopK SAE vari-
2

---

Matryoshka SAE
ants (Gao et al., 2025) improve the spansity-recoonstuuiiion trade-off, enabling successful application L.LMs (Temple- ton et al., 2024), diffusion models (Surkov et al., 2024), and medical imaging (Abdulaal et al., 2024). Recent work on SAE-based interpretation of CLIP embeddings (Rao et al., 2024) shows promise in extracting interpretable features.
single, flattened vector before applying TopK. This allows for a flexible number of active features per sample, with the total number of active features across the batch averaging to k x batch size. Although BatchTopp relaxes the fixed spar- sity of traditional TopK, it still relies on a predetermined l: parameter that requires careful tuning and continues to suf- fer from the potential for certain features to become 'dead' or rarely activated if they consistently fall outside the TopK selection.
Concept-based explainability. Concept-basee explana- tions provide interpretability by identifying human-coherent concepts within neural networks' latent spaces. While early approaches relied on manually curated concept datasets (Kim et al., 2018; Zhou et al., 2018; Bykov et al., 2023), recent work has explored automated concept extrac- tion (Ghorbani et al., 2019; Kopf et al., 2024) and explicit concept learning (L.iu et al., 2020; Koh et al., 2020; Es- pinosa Zarlenga al., 2022), with successful applications in out-of-distribution detection (Madeira et al., 2023), image generation (Misino et al., 2022), and medicine (Lucieri et al., 2020). However, existing methods often struggle to scale to modern transformer architectures with hundreds of millions of parameters. Our approach addresses this limitation by first training SAE without supervision on concept learning, then efficiently mapping unit-norm decoder columns de- fined vocabulary concepts using cosine similarity with CLIP embeddings.
3.2. Matryoshka SAE Architecture
Following Matryoshka representation learning ( Kusu- pati al., 2022), we propose SAE architecture that learns representations at multiple granularities simultaneously. In- stead of enforcing a single sparsity threshold l or using L1 regularization, our approach applies multiple TopK op- erations with increasing values, optimizing across all granularity levels. We set & values as powers of 2, i.e. k; = 2 up dimension d, which provides effective cov- erage of the representation space while maintaining rea- sonable computational costs. For given input x, MSAE computes h latent representations during training using sequence of increasing k values {k1, k2,,,,.... .2,...,kk with h < k2 < <... < kn S d:
2, (TopK,,WW bpre) + beue)),
3. Matryoshka Sparse Autoencoder
i; Wdec + Upren
3.1. Preliminaries
(2)
4
((x) = Σ a a - 2:112
Sparse autoencoders (SAEs) decompose model activations E R into sparse linear combinations of learned direc- tions, aiming for interpretability and monosemanticity. The standard SAE architecture consists of:
where a are weighting coefficients for each granularity level. At inference time, we can either apply TopK with any desired granularity discard it entirely, leaving only ReLU, which allows the model to utilize all neurons it deems essential for reconstruction.
z = ReLUU(once (nnne (x (pro)) + benn);
f- Wdecz + bpre:
where encoder matrix Wene E RRaxd encoder bias bene E Rd, decoder matrix Wdee E [Rdxn, and preprocessing bias bpre E R" are the learnable parameters, with d being the dimension of the latent space. The basic reconstruction objective is [(x) = I| $1...
Hierarchical learning. The key insight of our approach is that different samples require different levels of sparsity (numbers of concepts) for an optimal representation. By simultaneously optimizing across multiple k values, MSAE learns natural hierarchy of features. Our TopK opera- tions maintain nested structure where features selected at each level form a subset of those selected at higher k values, i.e. TopK, S TopK, _... C TopK, Such a hierarchical structure ensures coherence between granular- ity levels, where low & values capture coarse, high-level concepts while higher k values progressively enable fine- grained feature represennation.
Existing approaches established two primary sparsity mech- anisms. ReLU SAE (Bricken et al., 2023) uses L1 regu- larization with the objective L(x)= == | 1 + >||=||1, while TopK SAE (Gao et al., 2025) enforces fixed spar- sity through z ReLU (TopK (Wenc (a- bpre) + bene)). However, each approach faces distinct limitations: L1 regu- larization causes activation shrinkage (Rajamanoharan et al., 2024a), while TopK imposes rigid sparsity constraints. A recent effort to address the rigidity of TopK is BatchTopK (Bussmann et al., 2024), which replaces the standard TopK function with Batch TopK within the TopK SAE method. The Batch TopK function treats all batch activations as a
Sparsity coefficient weighting. We propose and evalu- ate two strategies for setting the weighting coefficients aq The uniform weighting (UW) approach sets a; = 1 for all i, while the reverse weighting (RW) strategy uses ax = h - i+ 1, giving higher weights to lower values. By
3

---

Matryoshka SAE
weighting the loss more heavily for sparser reconstructions, RW encourages the model to learn features that maintain reconstruction quality at lower k values. As shown in Ta- ble 1, this results in improved sparsity without significant performance degradation as the model learns that sparse representations achieve better loss even with slightly worse reconstruction quality, compared to UW which focuses pri- marily on reconstruction quality.
(6144, 12288, 24576] neurons for ViT-L/14, and from 512 to (4096, 8192, 16384) neurons for ViT-B/16. We provide further details on the implementation and hyperparameter settings in Appendix B.
4.1. Evaluation Metries
Here, we briefly define each metric used to evaluate SAE.
Lo denotes the mean proportion of zero elements in SAE activations.
3.3. Training and Inference
CLIP embeddings exhibit misalignment across modalities, which can impact SAE training convergence and cross- modal transferability. Following (Bhalla et al., 2024), we normalize embeddings to ensure consistent behavior across modalities. We first center embeddings by subtracting the per-modality mean estimated from the training dataset. Next, we scale the centered embeddings by a dataset-computed scaling factor following (Conerly et al., 2024) to obtain Exex ||||||| vn. This scaling ensures that X has consis- tent effects across different CLIP architectures and modali- ties. For training, compute the mean vector and scaling factor from the image modality. During inference on text embeddings, we apply the text-specific mean and scaling factor. Additionally, at inference, we remove Topk con- straints from TopK- models, allowing the model to adaptively select the number of active features based only on ReLU activation.
Fraction of variance unexplained (FVU), also known as Normalized MSE (Gao et al., 2025), measures reconstruc- tion fidelity by normalizing the mean squared reconstruction error [(z) by the mean squared value of the (mean-centered) imput. Explained variance ratio (EVR) is FVU's comple- ment metric, defined as 1 - FVU.
Linear probing (LP) assesses how well SAE preserves semantic information in the reconstructed embeddings on the downstream task. To evaluate this, we train a linear probe model ImageNet-Ik 1k using CLIP embeddings as backbone, with AdamW optimizer (Ir = le-3), ReduceL- ROnPlateau scheduler, and batch size=256. We measure performance by comparing predictions from original versus reconstructed embeddings using two metrics: Kullback- Leibler divergence (KI.) between predicted class distribu- tions and classification accuracy (Ace), where accuracy uses argmax predictions from original embeddings as targets.
Centered kernel nearest neighbor alignment (CKNNA) (Huh et al., 2024) measures kernel alignment based on mu- tual nearest neighbors, providing a quantitative assessment of aligmment between SAE activations and input embed- dings. A detailed explanation provided in Appendix E.
4. Evaluating MSAE
In this section, we conduct extensive experiments to evalu- ate MSAE against ReLU and TopK SAEs. We compare the sparsity-fideliy trade-off (Section 4.2), at multiple granu- larity levels (Section 4.3). We follow with evaluating the se- mantic quality of learned representations beyond traditional distance metries (Section 4.4), analyzing decoder orthogo- nality (Section 4.5), and examining the statistical properties of SAE activation magnitudes (Section 4.6). To verify that MSAE successfully learns hierarchical features, we conduct experiments on the progressive recovery task (Section 4.7). We conclude with an ablation study comparing the influence of different training modalities in Section 4.8. In response to reviewer feedback, we've 'e've incorporated an analysis of Topc models within Section 4.2. However, given their performance characteristics were not as competitive, we didn't 't extend their evaluation to subsequent sections.
Decoder orthogonality (DO) calculates the mean cosine similarity of the lower triangular portion of the SAE decoder, where 0 indicates perfect orthogonality. This metric assesses how orthogonal the monosemantic feature directions are in the decoder.
Number of dead neurons (NDN) is a metric that measures how many neurons remain consistently inactive (zero the SAE activations layer) across all inputs during training or evaluation, indicating the network's inability to fully utilize its capacity for learning semantic features.
4.2. Sparsity-Fidelity Trade-off
Setup. All SAE models are trained on the CC3M (Sharma al., 2018) training set with features (post-pooled) from the CLIP ViT-L/14 or ViT-B/16 model. Image modality is evalu- ated on ImageNet-1k Ik training set (Russakovsky et al., 2015), while text modality is evaluated on the CC3M validation set. Each SAE is trained with expansion rates of 8x, 16x and x2x, effectively scaling the latent layer from 768 to
We assess SAE performance using sparsity trade- off, measuring sparsity with Ly and reconstruction quality with EVR, following previous work. Figure 2 reveals that ReL.U SAE shows difficulty balancing performance, achiev- ing either high fidelity with low Lo or the opposite, with expansion rate primarily improving sparsity. TopK SAE with higher k values achieves better but not ReI.Ullevl
4

---

Matryoshka SAE
Table 1. Quantitative comparison of SAE models on ImageNNttII: We compare the following SAEs with expansion rate 8: ReLU with varying sparsity regularization (A), TopK and BatchTopK with 64 or 256 active neurons, and Matryoshka using uniform (UW) or reverse weighting (RW) a coefficients. Arrows (t/L) indicate the preferred direction of metrics. NDN values parentheses show the dead neuron count the training set. LP (KL) values are scaled by 10 for readability. Extended results for higher expansion rates and the text modality are reported in Appendix F.5.
Model
Lo f
FVU 4
CS f
LP (KL)
LP (Ace)
CKNNA t
DO
NDN +
ReLU (2- 0.03)
9200.000
.185..03
.928=.009
50.5277.1
.977..149
.712= 005
.002
0(0)
ReLU (A- 0.003)
6419..007
.0012.000
.998
0.66=1.03
1984
.781 .004
.003
0(0)
TopK (k= = 641)
250
.1722.026
.9122.013
60.1 29.1
.930..255
.762= 004
.002
0(335)
TopK (A: 256)
900+.004
.011+.003
.994..002
2.71=5.40 40
.987+.114
.874 .003
.003
((296)
BatchTopK (A: = 64)
8774.012
.1624.022
.917=.011
56.9 56.9285.8
.931+.28
.769 .044
.002
0(1477)
BatchTopK (A: 256)
8824.005
1010g.00s
.995..002
2. 2.42_5.12
.9884.108
.8602.00x
.002
3(919)
Matryoshka (RW)
.007
.997
3.13 os
.9874.115
.809 .022
.002
2(4)
Matryoshka (UW)
.7484.006
.0022.001
..09
0.35_n.82
.9952.070
.8482.003
.002
0(22)
1.000
As an ablation, we evaluate Cosine Similarity as an al- ternative reconstruction metrie, motivated by observations that SAEs primarily struggle with embedding magnitude reconstruction and CLIP embeddings are commonly Ly- normalized. Results in Appendix F. show consistent find- ings, with MSAE showing even clearer advantages through stable, low-variance performance across both modalities.
EVR (- better reconstructin)
0.975
Models
ReW [A = 0.03)
0.950
ReLU IA 0.01)
ReLU I 0.003)
0.925
Topk 64)
0.900
Topk ik 128)
Tapk (k- = 256)
0.875
BatchTapk Ik- 64)
BatchTopK ik- 128)
0.850
BatchTopk ik 256)
4.3. Ablation:
Matryoshka at Lower Granularity Levels
Matryoshka (RW)
0.825
Matryoshka (UW)
We train both MSAE variants (RW and UW) on two gran- ularities [128, 256] and compare them against TopK with k == 128 and h = 256 to analyze MSAE behavior at lower granularity levels. Figure shows that Matryoshka achieves similar sparsity to at least the lower TopK variant while maintaining CKNNA and EVR performance comparable to the best TopK variant, and even better with MSAE RW. This demonstrates that even at small granularity, MSAE maintains or improves the Pareto frontier over TopK across various metrics, with RW achieving better trade-offs. As ob- served also in Section 4.2, MSAE's performance advantages over TopK increase at higher expansion rates.
0.65
0.70
0.80
0.85
0.95
LO (+ sparser)
Figure 2. Comparison of sparsity-fidelity trade-off's across SAE architectures on ImageNet-Ik. 1k. Each model presents re- sults from all expansion rates, comparing ReLUU SAE (A- = (0.03, 0.01, 0.003)), TopK SAE (k = 64, 128, 256)). Batch- TopK SAF (k- 64, 128, 256)) and MSAE (RW, UW). The opti- mal SAE would occupy the upper right corner, achieving both high sparsity and reconstruction fidelity. For extended results across both modalities, refer to Figure 11.
4.4. Semantic Preservation Analysis
fidelity while offering improved sparsity yet consistently suffering from at least 5% of dead neurons (Table 1). While the Botc variant performs similarly to TopK, it ex- hibits higher fidelity with lesser sparsity when trained on the same k. Both variants of MSAE achieve better sparsity than ReLU and better fidelity than TopK or BotchTopK, es- tablishing a superior Pareto frontier while maintaining less than 1% of dead neurons. The RW variant further improves sparsity as expected, with only minor fidelity degradation. Notably, only Matryoshka consistently improves on both metrics with higher expansion rates, while TopK struggles with reconstruction, BatchTopK with sparsity, and ReLU shows improvements only in the highest at increased ex- pansion rates.
In Section 4.2, we only evaluated SAES using Lo for activa- tion sparsity and EVR for reconstruction fidelity, however these metrics have limitations. Lo only counts active neu- rons without assessing how well SAE representations align with original embeddings, and EVR focuses solely on dis- tance reconstruction rather than semantic preservation. To address these limitations, we introduce additional metrics. Following (Yu et al., 2025), we adopt the CKNNA metric to assess how well SAE activations preserve the neighborhood structure of CLIP embeddings. We also evaluate seman- tic preservation through linear probing metries (Gao et al., 2025; Lieberum et al., 2024), we use LP (KL) to measure prediction distribution alignment and LP (Acc) to compare
5

---

Matryoshka SAE
EVR (-) better reconsrruction
1.00
CKNNA (- better alignment)
0.875
Flattened Activation Frequency
Models
0.98
0.850
to
ReLUU (A 0,003)
Count
0.825
TopK (k - 32)
0.98
10*
Matryoshka (RW)
0.800
0.94
0.776
0.92
0.750
Count
16
0.726
0.90
10"
0.98
0.32
L0 (-+ sparser)
LO (-> sparser)
Modials
Topk - 128)
Matryoshka (RM) [128. 254]
Topk (k+ - 258)
Matryoshka (UN) (129,256)
Count
10"
10
Figure 3. Low Granularity Level Matroshka vs. TopK SAE on ImageNet-Ik. We report FVU (left) and CKNNA (right) metrics for two TopK variants (k= = 128, 256), and Matryoshka trained on these granularities in RW and UW variants at expansion rates 8 and 16. Even at this small granularity, MSAE improves the Pareto frontier relative to both TopK variants, pushing it the expansion rate grows from 8 to 16. For extended results across other metrics, refer to Figure 13.
0
5 10
15
20
Figure 4. Distribution of non-zero SAE activations on ImageNet-Ik validation set. Frequency histograms for ReLU (A = 0.003), TopK (k- = 32), and Matryoshka (RW) models at expansion rate 8. Matryoshka models exhibit a double-curvature distribution similar to ReLU models but without activation shrinkage, while TopK shows this pattern only higher k values, as can be seen in an extended Figure 14. Extended results for higher expansion rates are reported in Figure 15.
classification accuracy. All metrics are defined in Section 4. and presented in Table 1. Our analysis reveals that while cosine similarity and FVU correlate well with linear probing metrics, the alignment metric demonstrates Matryoshka's strength in preserving semantic structures.
sponding to rare, nearly singular concept images (Figure 10). As the sparsity parameter in Topk methods increases (Fig- ure 14), the transition from one to double-curvature behavior suggests that stronger sparsity constraints create compos- ite features, supported by Appendix C showing that high- activation features (> 15) in TopK methods have lower ratio of valid named features compared to Matryoshka.
4.5. Orthogonality of SAE Features
SAEs disentangle polysemantic representations into monosemantic features, as shown and explained by (Bricken et al. 2023). To evaluate feature monosemanticity, we mea- sure decoder orthogonality using the DO metric, with re- sults reported in Table 1. While all methods achieve high orthogonality as indicated by low DO values, none reach perfect orthogonality. This might stem from multiple fac- tors, including feature absorption as noted in (Chanin al., 2025), or just learning similar concepts (such as different numbers). We argue that understanding these sources of non-orthogonality is crucial for advancing the development of more effective monosemantic feature learning in SAES.
4.7. Progressive Recovery
To verify that our method learns hierarchical structure, we perform a progressive reconstruction task by using an in- creasing number of SAE activations, ordered by magnitude, to recover the original vector. Figure 5 shows that recon- struction quality improves with decreasing sparsity thresh- olds (increasing k) during inference. TopK variants exhibit performance plateaus shortly after their training thresholds (k: (32, 64)), while ReI.U- models show contin- ued improvement but with inferior performance at higher sparsity. MSAE demonstrates a better hierarchical structure that combines TopK's s efficient high-sparsity performance with ReLU's scaling capabilities. While our method per- forms slightly below TopK (k = 32) at the highest spar- sity, it quickly surpasses TopK's plateau at lower sparsity, achieving performance levels above ReLU models. We observe similar patterns in the CKNNA aligmment metric, with MSAE outperforming both TopK (k = 32) and ReLU models beyond A = 10 while performing only slightly be- low TopK (k = 256) at the lowest sparsity. Evidence of improved hierarchical feature learning across metrics and modalities is presented in Appendix F.4.
4.6. Activations Magnitudes Analysis
To analyze the impact of sparsity proxies on SAE, we ex- amine non-zero activation distributions across ViT-L with expansion rate Figure 4. Matryoshka models display a distinctive double-curvature distribution similar to ReLU- based models, with values between 10 appearing al- most linear in logio space. Following (Templeton et al., 2024), we attribute low activations to reconstruction pur- poses rather than semantic meaning. The second curvature reflects natural images' complexity, which requires multiple concept reconstructions rather than single dominant features, as evidenced by the small number of very high values corre-
6

---

Matryoshka SAE
Table Training modality influence on MSAE performance. We train MSAE on the text version of the CC3M train set and compare to models trained on its original image version, evaluating across both domains using the CC3M validation text set and ImageNet- 1k. While models perform best on their training modality, text- text-trained variants show better cross-domain generalization. Bold values indicate the best performance per metric, with NDN showing dead neuron count from the final checkpoint.
Matryoshka
Language metrics on CC3M
Vision metrics on ImageNet-lk
NDN +
SAE variant
La 1
FVU L
CS
CKNNA f
Lo f
FVU 4
C
CKNNA
Image (RW)
-214.029 020
.060+.022
4.026
2755
..29
.007
.997+.002
.8094.002
Image (UW)
.555..22
.066..oz 0227
.9884.012
.790
.748 D00
.002-.001
.999_ 000
.848 003
22
Text (RW)
8814.
.008
.9964.002
.782 008
.841
.008+.003
.9964.002
.782+.008
Text (UW)
.791 d. 010
.001 .001
.999, 2.00 .000
.784 0077
2.12
.015...013
.993..000
.877 $877..005
Reconstruction Fidelity k
CKNNA
5. Interpreting CLIP with MSAE
FVU (- better reconsscciiiinnn
CKNNA (- better alignment)
10'
In this section, we demonstrate how MSAE can enhance interpretability and control interpretable features in CLIP- based applications. We first establish neuron-concepp map- pings in the activation layer through an automated technique descrihed in Section 5. Then, we show its effectiveness in concept- based similarity search across the ImageNet valida- tion set, enabling retrieval images with varying degrees of explicit concept presence. Moreover, we leverage MSAE to study potential conceptual biases in gender classification model trained on the CelebA dataset (Liu et al., 2015).
10
0.8
10
0.4
10
10
10
10"
10"
10
10
10
10
10'
LD (+- sparser)
Models
LO
(- sparser)
(A 0.000)
(x 64)
Matryoshka (RWW
RoLU 0.0000
Took (X 256)
Matryoshka (UW)
Figure S. Progressive recovery performance on ImageNet- 1k. We report FVU (left) and CKNNA (right) metrics for different SAE architectures with expansion rate as functions of increas- ing top k values by magnitudes of SAE activations during infer- ence. SAE trained with TopK variants (k 32, 64) show per- formance plateaus beyond their training thresholds, while ReLU- based models (A = 0.001, 0.003) and Matryoshka variants (UW and RW) demonstrate continuous improvement. Extended results for higher expansion rates and across other metrics are reported in Figures 16& & 17.
5.1. Concept Naming
While self-supervised training of SAE enables learning up to d monosemantic concepts, mapping these concepts to spe- cific neurons remains non-trivial. Previous work used LLMs for identifying neuron-encodee concepts (Bills et al., 2023), but we adopt the more efficient method for CliPetrained SAE proposed in (Rao et al., 2024), which leverages CLIP's representation space. Our concept detection and validation methodology is detailed in Appendix A, with comprehen- sive results valid concept counts across SAE models presented in Table 3. Figure 8 demonstrates the highest- activating text and image examples for the best- matched feature concept 'face' across ReLU, TopK, and MSAE. The consistent concept presence across diverse inputs observed primarily in MSAE variants suggests that only Matryoshka- based methods were capable of learning this monosemantic feature. Supplementary analysis of highly activated con- cept examples in Appendix G.1 showcases SAE's ability to learn a wide range of concepts, from simple textures and colors to more complex ones like light (lights in darkness), countable concepts like trio (groups of three), and even nationality-related concepts like ireland or germany.
4.8. Training Modality: Language and Vision
We evaluate how training modality affects MSAE perfor- mance by comparing models trained on text with the origi- nal image-trained models, validating both modality models across text and image domains in Table 2. While both vari- ants perform best in their training domains, text-trained models achieve superior cross- moosl performance, demon- strating stronger generalization capabilities. Moreover, text- trained models achieve higher sparsity on both modalities with no dead neurons, showing better utilization of learned features. These findings position text training as a preferred approach for multi-modal applications where balanced per- formance is desired. Future research could explore training SAES on varying ratios of text and image data to optimize cross-modal performance or try to train crosscoders (Lind- sey et al., 2024) on both modalities simultaneously. We defer extended MSAE evaluations, to Appendix F.
Naming SAE features enables using SAE to conduct diverse interpretability analyses related to CLIP. We present two use cases where we apply the MSAE RW variant with an expansion rate of 8.
7

---

Matryoshka SAE
Class: Female; p=0.86
Effect of Concept bearded
Effect of Concept glasses
Effect of Concept blondes
0.975
0.8
Class probability
Class Probability
Original value
0.950
0.8
Threshold
0.925
IIVAL
0.900
0.4
0.875
0.2
Neuron Magnitude
Neuron Magnitude
Neuron Magnitude
Figure Impact concept manipulation on gender classification. By increasing concept magnitudes (bearded, glasses, blonde) SAE space and mapping back to CLIP space, observe changes in gender classification probabilitics. Results reveal the model's learned gender associations through platcauing effects: bearded and glasses bias toward male classification, while blonde bias toward female.
Concept maagitude
Distance 0.18
Top 6 Active Concepts
semantic concept patterns and explaining why both NNs match the query image of an Irish police vehicle, with the first NN (Irish police vehicle) being closer than the second (British police vehicle). Second, compare similarity search in CLIP embedding space against SAE activation space using Manhattan distance (detailed and visualized in Appendix G.2). While the first NN remains consistent across both spaces, the second NN in the SAE space shows the same vehicle type from a different angle, demonstrating that similarity searches can be done in both spaces while SAE enables additional concept-based interpretability. Finally, we demonstrate controlled similarity search by manip- ulating concept magnitudes. In Figure 7, increasing the germany concept strength preserves the original image as the top match but shifts the second NN from an Irish to a German police vehicle, while preserving the overall input image structure. The increasing distances from the original image embedding show how larger magnitude adjustments affect embedding coherence.
ireland
poliic foadtruck
britain automotive
Distance 0.37
police germany
automotive
afterno
crowdfunding
Concapt Magnitude 30
Distance 0.31
ireland
police
foodtruck
britain
Distance 0.37
germany
police
automotive
afterno
street
5.3. Bias Validation on a Downstream Task
CLIP models are commonly used as feature extractors for downstream tasks, enabling efficient fine-tuning with lim- ited data. With MSAE, stream models learn to associate specific concepts with classes. To demonstrate this, we train a single-layer classi- fier on CLIP embeddings from the CelebA dataset to per- form binary gender classification (1 for female, for male), achieving an F1 score of approximately 0.99. Through statistical analysis in Appendix G.3, we uncover several concept-gender associations: bearded biases toward male classification, blonde toward female, and glasses showing modest male bias. To validate these findings, Figure 6 demonstrates an example of how increasing these concepts' magnitudes affects classification scores for a female exam- ple (see Figure 23 for a male example). The results confirm our statistical analysis, and the plateaus in classification probabilities concept magnitudes increase help quantify the strength of concept-gender associations in the model.
Figure 7. Nearest neighbor analysis with enhanced germany concept. By increasing the magnitude of the germany concept in SAE space (from 0.3 to 20, then 30) and mapping back to CLIP space, we observe shifts in nearest neighbors. While the input image remains the top match (with increasing distance), the sccond-ncarest neighbor changes from British police vehicle (shown in Figure 21) German one.
investigate whether down-
5.2. Similarity Search
CLIP embeddings are widely used for cross-modal similar- ity search between images and text through cosine similarity metric, primarily for retrieval engines. We extend this capa- bility using SAE in three ways.
First, SAE provides interpretable insights into nearest neigh- bor (NN) image retrievals. Figure 21 shows the top 8 con- cepts for the two closest retrieved images, revealing shared
8

---

Matryoshka SAE
ReLU
Topk
MSAE Valid feature 4289 with similarity 0.47
Invalid feature 5367 with similarity 0.20
Invalid feature 0977 with similarity 0.60
more sparse
1. how we wish these faces were not covered with huge
1. head of woman 1963.67)
1. face of female mannequin close-
beards
2. face drawn
trunk
2. face young woman looking of woman 2963.879
welcomes guests to grant '
up depicting headache
consisting with heavy
camera
3. person a lovely smile
4. person and film character by man
1. abstract digital art image of woman face
5. portrait of girl with sunlight on her face
5. person lnoks like cage !
close up on colorful taxtured background.
6. close up the head and beak of animal
6. you are looking al person
5. face of
woman
7. cluse up of a wild bull showing details of it face
7. video game series y figure of soldier
6. face of temale mannequin close
and his mustache
7. this face mask looks lot like film character
8. close up young man looking sideways
8. person looks cat
8, close up carved into sumpkin.
better reconstruction
Invalid feature 5687 with similarity 0.34
Invalid feature 1098 with similarity 0.28
Valid feature 1098 with similarity 0.43
1. medium hairstyles for rounc faces are pretty fun and intoresting if you know a thing about the ,
1. look buff bright color underneath the eye and into inner rim of waterline
bright colar underneath eye and inner watertine-
2. wedding hairslyles with garment over the face
2. beauty and makeup concept with half face of beautiful young woman. if
2. face a female mannequin
up
3. look 2, buff bright color underneath the eye
beauty and makeup concept with half face beautiful young woman
and into inner rim of waterline
3, actor, pertect.
makeup, this would be
4. highlight eyelashes beauty
4. actor. eye wore makeup, would be perfect.
beauty and mekeup concept: beautiful young woman
face
5. highlight the eyelashes beauty salon.
bridal makeup final touches before wedding
someuiess eye makeup, portrait girl sunlight on her
best
6. bridal makeup- final touches before the wedding
6. somellimes with eye makeup less best
8. how a young eyehrows with pencil
7. steps for fresh face
7. face of a female mannecuin close- up
8. how to drew on eyebrows with a pencil
8. todays 1950s hair & make up inspiration from actor
Figure& 8. Comparison of top-activating examples for the 'face' concept across SAE methods. Through automated interpretability process, we identified the best matching SAE neuron for the face concept, quantified by a similarity score and a validation status (Valid or Invalid). For this neuron, we then identified the highest activated examples in both and image modalities across TopK, ReLU, and MSAF, cach presented in variants optimized for either sparsity or reconstruction. Both text and image examples with the highest activation values strongly confirm. confirm the concept's presence and demonstrate that only MSAE variants learned the concept of the face. We show additional examples of valid concepts with their top-activating examples in Figure 19.
6. Conclusion
we demonstrated MSAE's efffcctiveness using CLIP embed-- dings, it has great potential to explain hierarchical represen- tations in other embedding spaces, such SigLIP (Zhai et al., 2023) or modality-specific representations. Finally, since not all neurons correspond to simple concepts in our vocabulary, investigating complex semantic features through LLM-baseed interpretability methods could provide deeper insights into the learned hierarchical representations. In con- current independent work, Bussmann et al. (2025) propose an MSAE approach to interpreting language models.
We propose Matryoshka SAE to advance our understand- ing of CLIP embeddings through hierarchical sparse toencoders. MSAE improves upon both TopK and ReLU approaches, achieving superior sparsity-fidelity trade-off while providing flexible sparsity control via the a coeffi- cient. Our experiments demonstrate MSAE's effectiveness through near-optimal metries, progressive feature recovery, and extraction of over 120 validated concepts, enabling new applications in concept-based similarity search and bias detection in downstream tasks.
Impact Statement
Limitations and future work. MSAE faces three limita- tions with clear paths for future improvement. The current implementation's use of multiple decoder passes with dif- ferent TopK activations introduces computational overhead, which could be addressed through optimized CUDA kernels enabling parallel processing of multiple granularities. While
This paper presents work whose goal is to advance the field of Machine Learning Interpretability. There are some poten- tial societal consequences of our work, none which we feel must be specifically highlighted here.

---

Matryoshka SAE
Acknowledgement
Bussmann, B., Leask, P., and Nanda, N. BatchTopk sparse autoencoders. arXiv preprint arXiv:2412.064 2024.
Work on this project was financially supported from the SONATA BIS grant 2019/3//E/ST600002 funded by Polish National Science Centre (NCN). We also thank the anony- mous reviewers for their useful comments.
Bussmann, B., Nabeshima, N., Karvonen, A., and Nanda, N. Learning multi-level features with matryoshka sparse autoencoders. arXiv preprint arXiv:2503. 17547, 2025.
Bykov, K., Kopf, L., Nakajima, S., Kloft, M., and Höhne, M. M. Labeling neural representations with inverse recog- nition. In NeurlPS, 2023.
References
Abdulaal, A., Fry, H., Montañia-rrown, N., Ijishakin, A., Gao, J., Hyland, S., Alexander, D. C., and Castro, D. C. An x-ray is worth 15 features: Sparse autoencoders for interpretable radiology report generation. arXiv preprint arXiv:2410.03334, 2024.
Chanin, D., Wilken-Smith, J., Dulka, T., Bhatnagar, H., and Bloom, J. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. In ICLR, 2025.
Abnar, and Zuidema, W. Quantifying attention flow in transformers. In ACL, 2020.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, Reproducible scaling laws for contrastive language-iman image learning. In CVPR, 2023.
Adebayo, J., Gilmer, Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. Sanity checks for saliency maps. In NeurlPS, 2018.
Conerly, T., Templeton, A., Bricken, T., Marcus, J., and Henighan, T. Update how train saes. https://transformer- cirooitttrrrirrtttt..om pub/ 2024 / ht44/aprll-uppae index.nnxx. html, April 2024.
Balasubramanian, S., Basu, S., and Feizi, S. Decomposing and interpreting image representations via text in ViTs beyond CLIP. In NeurlPS, 2024.
Transformer Circuits Thread.
Baniecki, H., Casalicchio, G., Bischl, B., and Biecek, Efficient and accurate explanation estimation with distri- bution compression. In ICLR, 2025.
Conmy, A., Mavor-Parker, A., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. Towards automated circuit discovery for mechanistic interpretability. In NeurIPS, 2023.
Bereska, L. and Gavves, E. Mechanistic interpretability for AI safety-A review. Transactions Machine Learning Research, 2024.
Crabbé, J., Rodriguez, P., Shankar, V., Zappella, L., and Blaas, A. Interpreting CLIP: Insights on the robustness to imagenet distribution shifts. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Bhalla, U., Oesterling, A., Srinivas, S., Calmon, F., and Lakkaraju, H. Interpreting CLIP with sparse linear con- cept embeddings (SpLiCE), In NeurlPS, 2024.
Biecek, P. and Samek, W. Position: Explain to question not to justify. In ICML, 2024.
Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. In ICLR, 2024.
Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J. and Saunders, W. Language models can explain neurons in language models. https: //openaipubll blo.. core wi windows.net. wwtprconeexppaaner/paeea 2023.
Espinosa Zarlenga, M., Barbiero, P., Ciravegna, G., Marra, G., Giannini, F., Diligenti, M., Shams, 7., Precioso, F,, Melacci, S., Weller, A., et al. Concept embedding models: Beyond the accuracy-explaiaablity trade-off. In NeurIPS, 2022.
html, May
Gandelsman, Y., Efros, A. A., and Steinhardt, Interpreting CLIP's image representation via text-based decomposi- tion. In ICLR, 2024.
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, L., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y.. Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing language models with dictionary learn- pubb
Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders. In ICLR, 2025.
ing. https: http:://raaaaffoeeeieeeee
Gemma, T. and DeepMind, G. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.00/ 18, 2024.
2023 / wewttiieeceeueeeeeeeeeeeeeee preettureeindexx htt.:reeeeeddddddmm., October 2023. Transformer Circuits Thread.
10

---

Matryoshka SAE
Ghorbani, A., Wexler, J., Zou, Y., and Kim, B. Towards automatic concept-based explanations. In NeurIPS, 2019.
2024 / www.scoodesssssecccooccooos index html, October 2024. Transformer Circuits Thread.
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., and Olah, C. Multimodal neurons in artificial neural networks. Distill, 6(3):e30, 2021.
Liu, H., Li, C., Wu, Q. and Lee, Y. J. Visual instruction tuning. NeurIPS, 2023.
Liu, Y., Zhang, X., Zhang. S., and He, X. Part-aware pro- totype network for few-shot semantic segmentation. In ECCV, 2020.
Hernandez, E., Schwettmann. S., Bau, D., Bagashvili, T., Torralba, A., and Andreas, Natural language descrip- tions of deep visual features. In ICLR, 2021.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In ICCV, 2015.
Huh, M., Cheung, B., Wang. T., and Isola, The platonic representation hypothesis. In ICML, 2024.
Lucieri, A.. Bajwa, M. N., Braun,
Malik, M. I., Den- gel, A., and Ahmed, S. On interpretability of deep learn- ing based skin lesion classifiers using concept activation vectors. In IJCN,, 2020.
Jain, S., Lawrence, H., Moitra, A., and Madry, A. Distilling model failures as directions in latent space. In ICLR, 2023.
Lundberg. S. and Lee, S.-I. unified approach to interpret- ing model predictions. In NeurlPS, 2017.
Joukovsky, B., Sammani, F,, and Deligiannis, N. Model- agnostic visual explanations via approximate bilinear models. In ICIP, 2023.
Madeira, P., Carreiro, A., Gaudio, A., Rosado, L., Soares, F., and Smailagic, A. ZEBRA: Explaining rare cases through outlying interpretable concepts. In CVPR, 2023.
Kim, B., Wattenberg, M., Gilmer, J., Cai. C., Wexler, J., Viegas, F., et al. Interpretability beyond feature attribu- tion: Quantitative testing with concept activation vectors (TCAV). In ICML., 2018.
Misino, E., Marra, G., and Sansone, E. VAEL: Bridging variational autoencoders and probabilistic logic program- ming. In NeurlPS, 2022.
Koh, W. Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E.,, Kim, B., and Liang, P. Concept bottleneck models. In ICML, 2020.
Oikarinen, T. and Weng, T.-W. CLIP-Dissect: Automatic description of neuron representations in deep vision net- works. In ICLR, 2023.
Kopf, I.., Bommer, P. L., Hedström, A., Lapuschkin, S., Höhne, M. M., and Bykov, K. Cosy: Evaluating textual explanations of neurons. In NeurlPS, 2024.
Oikarinen, T., Das, S., Nguyen, L. M., and Weng, T.-W. Label-free concept bottleneck models. In ICLR, 2023.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Similar- ity of neural network representations revisited. In ICML, 2019.
Paulo, G. and Belrose, N. Sparse autoencoders trained on the same data learn different features. arXiv preprint arXiv:2501. 16615, 2025.
Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., et al. Matryoshka representation learning. NeurlPS, 2022.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. SDXL: Im- proving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
Li, Y., Wang, H., Duan, Y., Xu, H., and Li, X. Exploring visual interpretability for contrastive engguage-image pre- training, arXiv preprint ar www.i:2009.00444.. 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. JCML, 2021.
Licberum, T., Rajamanoharan, S., Conmy, A., Smith. L., Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., and Nanda, N. Gemma Scope: Open sparse autoencoders everywhere all at once on Gemma 2. arXiv preprint arXiv:2408.00 05147,,00511,...1 2024.
Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T. Varma, V., Kramár, J., Shah, R., and Nanda, N. Improving dictionary learning with gated sparse autocncoders. arXiv preprint arXiv:2404.1600. /60144 2024a.
Lindsey, J. Templeton, A., Marcus, J., Conerly, T., Batson, J., and Olah, C. Sparse cross- coders for cross-layer features and model diff- ing. https://transfot
Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., and Nanda, N. Jumping ahead: Improving reconstruction fidelity with JumpReLU sparse autoencoders. arXiv preprint arXiv:2407. 14435, 2024b.
rewits..pb).

---

Matryoshka SAE
Ramaswamy, V. V., Kim, S. S. Y., Fong, R., and Rus- sakovsky, O. Overlooked factors in concept-based expla- nations: Dataset choice, concept learnability, and human capability, In CVPR, 2023.
Templeton, A., Conerly, T., Mareus, J., Lindsey, J., Bricken, T. Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C.. MacDiarmid, M., Tamkin, A., Durmus, E., Hume, T., Mosconi, F,, Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. Scaling monosemanticity: Extracting inter- pretable features from Claude 3 Sonnet. https: / //trrnnffrmerr-creulleee http::ccrerutttt.pppu/2222424 pub/2024/ how.iiinnnooooomttiiiiitttt//// /index/y///iii//e/eeee html, May 2024. Transformer Circuits Thread.
Rao, S., Mahajan, S., Böhle, M., and Schiele, Discover- then-name: aano-tgi concept bottlenecks via auto- mated concept discovery. In ECCV, 2024.
Ribeiro, M. T,, Singh, S., and Guestrin, C. "Why should trust you?": Explaining the predictions of any classifier. In KDD, 2016.
Wang. W., Lv, Q.. Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang. Z., Zhao, I.., Song, X., Xu, J., Xu, B., L., Dong, Y., Ding, M., and Tang, J. CogVLM: Visual expert for pretrained language models. In Neur/PS, 2024.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. ImageNet large scale visual recognition chal- lenge. International Journal of Computer Vision, 115: 211 211-252, 2015.
Yu, S., Kwak, S., Jang, H., Jeong. J., Huang. J., Shin, J., and Xie, S. Representation alignment for generation: Training diffusion transformers is easier than you think. In ICLR, 2025.
Sammani, F., Joukovsky, B., and Deligiannis, N. Visualizing and understanding contrastive learning. IEEE Transac- tions on Image Processing, 33:541-555, 2024.
Zeiler, M. D. and Fergus, R. Visualizing and understanding convolutional networks. In ECCV, 2014.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. LAION-400M: Open dataset of CLIP- filtered 400 million image-text pairs. arXiv preprint (tti::2/11.011111 2021.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In ICCV, 2023.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
Zhao, C., Wang, K., Zeng, X., Zhao, R., and Chan, A. B. Gradient-basee visual explanation for transformer- CLIP. In ICML, 2024.
Zhou, B., Sun, Y., Bau, D., and Torralba, A. Interpretable basis decomposition for visual explanation. In ECCV, 2018
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Con- ceptual captions: cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACI., 2018.
Shen, S., Li, L. H., Tan, H., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., and Keutzer, K. How much can CLIP benefit vision-and-led ann-language tasks? ICLR, 2022.
Shrikumar, A., Greenside, P., Shcherbina, A., and Kundaje, A. Not just black box: Learning important features through propagating activation differences. In ICML., 2017.
Simonyan, K. Deep inside convolutional networks: Visu- alising image classification models and saliency maps. arXiv preprint arXiv: 2013.
Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribu- tion for deep networks. In ICML, 2017.
Surkov, V., Wendler, C., Terekhov, M., Deschenaux, J., West, R., and Gulcehre, C. Unpacking sdxl turbo: Inter- preting -ot--eg--iaeee models with sparse autoencoders. arXiv preprint arXiv:2410.22366 2024.
12

---

Matryoshka SAE
Appendix for "Interpreting CLIP with Hierarchical Sparse Autoencoders"
Concept Discovery and Validation
14
Implementation Details
16
B.1 Hyperparameters
16
B.2 Optimal Parameters
16
Highest Neuron Magnitudes
17
Activation Soft-capping
19
D.I Definition of Soft-capping.
19
D.2 Results.
19
E
CKNNA Alignment Metric
20
F
Evaluating MSAE: Additional Results
21
El
Sparsity-Fidelity Trade-off
21
F2
Ablation: Matryoshka at Lower Granularity Levels
................................
23
F3
Activation Magnitudes Analysis
24
F.4
Progressive Recovery
26
F5
Comprehensive Evaluation with CLIP ViT-L/14 and ViT-B/16 Architectures
28
G
Interpreting CLIP with MSAE: Additional Results
34
G.1
Concept Visualization Analysis
34
G.2 SAE-Enhanced Similarity Search
37
G.3 Gender Bias Analysis in CelebA
37
H Stability Evaluations
39
13

---

Matryoshka SAE
A. Concept Discovery and Validation
Here, we describe our approach for detection, which concepts SAE learned, and how we validated the mappings of these concepts to specific neurons. While LLMs are commonly used identify cauron-encoded concepts (Bereska & Gavves, 2024; Conmy al., 2023), we follow Rao al. (2024) in implementing a more computationally efficient approach, which more tailored to the CLIP-based SAE.
CLIP-Based concept matching. The method uses predefined vocabulary of concepts (e.g.. "hair", 'pink") to compute cosine similarity between CLIP embeddings and SAE decoder columns. After mapping concepts to CLIP's embedding space and applying the same preprocessing as during SAE training, we remove bpre from the preprocessed CLIP embeddings for comparison with decoder. For the feature columns in the SAE decoder, which are unit-magnitude by definition, the best matching concept to the neuron is determined by maximizing cosine similarity, where value of indicates perfect alignment. Thus, the optimal concept 8, for neuron Pe is defined as:
Pr: C [Pe (PeI(CLIP(v)]
5c = arg max [cos(pe, CLIP()))] = arg max eEV
eVV
Pre-activation bias in similarity calculations. While the method above suggests removing byre from CLIP embeddings. our empirical analysis revealed this significantly masks neuron-concept relationships. Without byre. in Figure 9, we show that similarities cluster around 0.1 (mean) with maxima around 0.15 0.2, whereas retaining bpee yields higher similarity scores (>0.42) that correspond to correct concepts. Importantly, both approaches preserve neuron rankings, with over 95% of concepts sharing identical highest matching neurons, so not removing bias doesn't 't destroy ranking. Manual evaluation confirmed that neurons with bias-removed similarities (~~0.2) are under-estimated compared their bias-inclusive counterparts (~0.5). Based on these findings, retain 6pre in our caleulations.
Max Similarity Distribution with no bias for MSAE (RW) Max Similarity Distribution with no bias for MSAE (UW)
6144
2500
12288
24576
2000
Count
1000
500
-0.15
-0.10
-0.05
0.00
0.05
0.10
0.15
0.20
0.25
0.30
-0.15
-0.10
-0.05
0.00
0.05
0.10
0.15
0.20
0.25
0.30
(a)
Max Similarity Distribution with bias for MSAE (RW)
Max Similarity Distribution with bias for MSAE (UW)
3000
12288
24576
Count
1000
0.1
0.2
0.3
0.4
0.5
0.6
0.1
0.2
0.3
0.4
0.5
0.6
Similarity
Similarity
(b)
Figure 9. Impact of pre-activation bias on concept similarities. We take the highest neuron similarities concept across expansion rates for (RW) and (UW) MSAE variants, (a) with and (b) without bpre. Not removing bprc yields better distribution with higher similaritics that better reflect neuron intorpretability.
14

---

Matryoshka SAE
Limitations of the current approach. We identify several limitations in the current concept mapping approach. First, the method assigns concepts neurons based on the highest similarity regardless of the absolute matching quality, potentially leading to poor concept assignments when no good matches exist. Second, hierarchical concepts pose challenge when matching with more specific neuronal features. For example, high-level concept like 'mammal' may show strong similarity to both 'cat' and 'dog' neurons, resulting in imprecise assignments. This issue stems from either semantic feature vectors that aren't I perfectly orthogonal or incomplete vocabulary coverage.
prreshold-based validation. To address the challenges identified above, we propose three validations to remove weak assignments. Before applying the validations, we switch the mapping from concepts neurons, mapping of neurons concepts to reduce spurious assignments. Based on this, threshold results by either:
1. Cosine similarity > 0.42, which ensures that the neurons exhibit strong alignment with their assigned concepts, preventing weak or ambiguous concept mappings
2. Concept similarity ratio Second highest similarity least twice as strong as the second-bes best concept, avoiding distributed representations
Top similarity
> 2.0, confirms concept uniqueness by requiring the best match to be at
3. One concept per neuron (with the highest similarity) enforces monosemanticity by assigning only the most strongly aligned concept to each neuron, which is needed due to the vocabulary structure containing multiple variations of the same concept (e.g., 'bird' and 'birdie')
Vocab data. Following (Ramaswamy et al., 2023) principle that vocabulary concepts should be simple, adopted the vocabulary from (Bhalla et al., 2024). This vocabulary comprises the most frequent unigrams from LAION-400m captions dataset (Schuhmann et al., 2021). To account for semantic relationships between concepts, we perform manual validation of top concepts for each discovered neuron.
Semantic consistency. Manual evaluation of top concepts per neuron verifies concept consistency and identifies hierarchical relationships, where top vocab similarities (such as dog breeds) can indicate broader categorical concepts (such as 'dog').
Results across SAE architectures. We evaluate concept neurons across architectures in Table From 37,445 neurons at expansion rate 8, only small fraction passed similarity validation: ~10% for TopK and 1-3% for ReLU and Matryoshka architectures. While higher expansion rates typically reduce valid neurons, both TopK variants and ReLU (A = 0.001) exhibit increased valid mappings under best vector validation. Although these results suggest limited concept learning or concept distribution across neurons, the vocabulary structure prevents definitive conclusions, due to the dominance of non-semantic unigrams, and many semantically similar concepts appear across vocabulary (c.g., 'bluc', "blau' and 'bleu'). The validation results from the table demonstrate that sparser architectures (TopK) yield 3-8 times more interpretable concept neurons compared more dense ones (ReLU), with Matryoshka being between the two, supporting the hypothesis that sparsity promotes concept specialization.
Table 3. Comparison of valid concept neurons detected across different SAES and validation methods. The validation methods include a cosine similarity threshold above 0.42, selecting the best matching neuron, combining both criteria, applying the concept similarity ratio threshold between the first and second hest vocab concept for the neuron, and enforcing all conditions simultaneously.
Measurements were made for each model at three expansion rates (x8) x16 x32).
Model
Similarity above 0.42
Best vector
Above and best
Ratio threshold
All conditions
ReLU (A- 0.03)
3308
3765
4181
2740
3608
5129
874
1046
1304
380
175|
45
97 31
16
ReLU 0.003)
896
781
799
2372
3305
5129
217
196
188
395
251
194
29
19
ReLU (A- = 0.001)
351
247
128
4116
6793
11417
77
63|
|
322
169
47
|
8
|
2
TopK (k 32)
4081
4719
5027
2755
3415
3827
1021
1259
|141
999
857
858
216
197
203
TopK (k 64)
3797
4504
4915
2557
3272
167
873
1080
1238
1322
1151
1167
238
232
238
TopK (K= 128)
2141
2590
3059
2167
2670
3306
455
565
745
1508
1383
1379
211
226
231
TopK (K- 256)
943
888
962
1883
2191
2631
168
167
171
1579
1523
1554
134
126
127
Matryoshka (RW)
1136
1109
1038
1628
2213
25-11
237
257
259
1429
1135
1059
140
132
121
Matryoshka (UW)
907
894
748
1517
1908
2396
195
191
167
1254
1169
1069
125
128
198
15

---

Matryoshka SAE
B. Implementation Details
We conducted experiments using CLIP VIT-L/I4 (and ViT-B/16, reported later in the appendix) pre-trained on the CC3M dataset image training subset. Following (Bricken et al., 2023; Gao al., 2025) and this blog (Conerly et al., 2024), our SAE implementation uses unit-norm constraint on the decoder columns with untied encoder and decoder. We initialized 6pre and bene with zeros, decoder with uniform Kaiming initialization (scaled L2 norm 0.1), and encoder the decoder's transpose. Gradient clipping was set to 1. For data preprocessing. we centralized embeddings per modality (Bhalla et al., 2024) and scaled by a constant to achieve Ezex [|]]||]] = vn. All models were trained for 30 epochs on a single NVIDIA A100 GPU with batch size 4096, except for the model with an expansion rate of 32, which was trained for 20 epochs. While MSAE and TopK showed dead neurons, we omitted revival strategies as only in TopK the number of dead neurons exceeded 1%.
B.1. Hyperparameters
We first conducted experiments on CLIP RN50 using hyperparameters from (Rao et al., 2024), later validating them on ViT-L/14. For ViT-L/14, we explored parameters near RN50-optimal values to ensure cross-architecture consistency. With expansion factor (768 - 6144), we explore:
Learning rates per method: 1 11-05, 5 10-5, 1 10 10-1,111111 10-3
ReLU L1 coefficients (A): 1- 10-4, 4,0 4. 10 10-3, 1 10 3, 3.3 10-2
TopK values: E {32, 64, 128, 256}. up to 256 as (Gao et al., 2025) suggests higher values do not learn interpretable features
Matryoshka K-lists: [32... 6144) and {64.. (64...6144) 6144), for higher expansion rates we adjust the upper limit
a coefficients: uniform weighting (UW) [1.I.I,I.IIII1} and reverse weighting (RW) (7.6,5,4.3,2.1)
The optimal parameters from these experiments were applied to larger expansion factors 16 (768 + 12288), 32 (768 - 24576), and all expansion rates VIT-B/16.
Following reviewer suggestions, we extended our evaluations to include TopK 64, 128, 256). The BatchTopp models were trained using the same hyperparameters as TopK models. These additional results are presented in the Appendix. We also attempted to integrate JumpReLU (Rajamanoharan al., 2024b) into evaluations, but did not achieve meaningful results, which may be attributed to an implementation error in our code.
512) and BatchTopK for {16, 32,
B.2. Optimal Parameters
Based on RN50 experiments and subsequent adjustment to VIT-L/14 with expansion factor 8, we selected the following optimal configurations: ReLU with learning rate 5 10-5 and À values of 1 10-3, 3 10-3, 3,-3,33- 3- 10-2; TopK with learning rate 5 10-4 and k values of 32, 64, 128, and 256; MSAE with learning rate 1 10-4, K-list (64... 6144), ), for both uniform (UW) and reverse weighting (RW) a strategies.
16

---

Matryoshka SAE
C. Highest Neuron Magnitudes
Based on results from Figure 4, we analyze images from ImageNet-Ik validation set that produced the highest neuron magnitudes for TopK and MSAE architectures. In Table we show that more constrained SAEs (TopK (k 128)) produce higher number of samples with neurons above 15; however, the percentage of valid neurons is lower than in MSAE and TopK (k 256) which have significantly less high magnitude samples. This indicates that high-magnitude neurons in highly constrained TopK may presumably learn complex features. Figure 10 presents the top valid highest neuron magnitude images per model, demonstrating that very high magnitudes often correspond to images with almost singular concepts.
Table Analysis of nigh-magnitude neurons across architectures. We analyze samples with magnitude > 15 in the ImageNet- Ik validation set, showing the number of total occurrences, the proportion of valid concepts among high-magnitude concepts, and the rate of high-magnitude valid concepts relative to all valid concepts in the model from Table 3.
Model
High-Magnitude Samples
Valid Concept Rate
High-Magnitude Concept Rate
TopK (A: 32)
113
6 (5%)
216 (3%)
TopK (A: 64)
18
0 (0%)
238 (0%)
TopK (A: 128)
3
0 (0%)
211 (0%)
TopK (A: -256
12
8 (67%)
134 (6%)
MSAE (RW)
21
(38%)
140 (6%)
MSAE (UW)
22
(32%)
125 (6%)
17

---

Matryoshka SAE
Concept graduate Magnitude 16
Concept cabbage Magnitude 16
Concept merry Magnitude 15
Concept food Magnitude 15
Concept horse Magnitude 16
Concept medical Magnitude 16
Concept bell Magnitude 15
Concept corn Magnitude
Concept golf Magnitude 15
Concept alcoholic Magnitude 15
Concept interiordesign Magnitude 16
Concept military Magnitude 16
(a)
(b)
Concept bread Magnitude 17
Concept birthday Magnitude 16
Concept military Magnitude 15
Concept sheep Magnitude 15
Concept birthday Magnitude 15
Concept clock Magnitude
HAPPY RTHDA
HAPPY BIRTHDAY
12
10
2
Concept clock Magnitude 16
Concept alcoñolic Magnitude 15
Concept candles Magnitude 15
Concept korse Magnitude 16
Concept aireraft Magnitude 17
Concept money Magnitude 17
11
10
8
5
(c)
(d)
Figure 10. Images with the highest valid concept neuron magnitudes. We took 6 images from ImageNet- Ik validation set per model based on results from Table with (a) TopK it 32, (b) TopK 256, (c) Matryoshka RW, and (d) Matryoshka UW.
18

---

Matryoshka SAE
D. Activation Soft-capping
Analysis of MSAE in Figure reveals that despite effective handling multi-granular sparsity, the model learns to encode concepts using extremely large activation values (>15). This can lead more composite rather than atomic features, was in the case of TopK 2 128) revealed in Appendix C.
D.I. Definition of Soft-capping.
To address this, we introduce activation soft-capping (SC), adapting the logit soft-capping concept from language mod- els (Gemma & DeepMind, 2024). This technique prevents too high activation magnitude and circumvention of sparsity constraints via activation magnitude manipulation:
2 softcap tanh(z/softcapp, a
2 + Upre:
(4)
where softcap hyperparameter controls maximum activation magnitude. Combined with ReLU, this bounds SAE activations (0, softcap).
D.2. Results.
In Table 5, we show soft-capping's 's impact on MSAE performance across key metrics using the ImageNet-Ik training set. Our analysis reveals two key benefits of applying soft-capping MSAE. First, it consistently improves Lo sparsity, with MSAE RW (SC) achieving values of 0.830 and 0.889 for 6144 and 12288 sizes, respectively. Second, while basc MSAE UW maintains better FVU and CS scores, the soft-capped MSAE RW significantly reduces the number of dead neurons. With a latent size of 1 12288, MSAE RW exhibits only 66 dead neurons, compared to 491 in the model with a latent size of 6144. These findings show that soft-capping is particularly beneficial for large-scale SAEs with wider sparse layers, where neuron utilization becomes more challenging. The technique provides practical approach to reducing dead neurons while maintaining high Ly sparsity, with only minimal impact on reconstruction fidelity.
Table Impact of soft-capping (SC) MSAE performance. We evaluate soft-capping across different expansion rates (8 and 16) on the ImageNet-Ik Ik validation set, comparing UW and RW variants. While base MSAE maintains better FVU and CS scores, soft-capped variants show improved Lo sparsity and reduced number of dead neurons, particularly larger sizes. Bold values indicate the best performance per metric and size, with NDN in parentheses showing dead neuron counts from the final checkpoint.
Size
Model
Lo t
FVU 4
CS f
CKNNA f
NDN 4
Matryoshka (RW)
0.829 4.00s
0.007 + 0003
0.997 4.002
0.809
2(4)
Matryoshka (UW)
0.7484.000
0.001 4.002
0.9994.000
0.8484.003 0003
0(22)
6144
Matryoshka (RW, SC)
0.830=.007 007
0.010 003
0.995 002
0.8391.004
1(2)
Matryoshka (UW, SC)
0.7744.006
0.0044.001
0.998
0.856 .003
1(3
Matryoshka (RW)
0.8844.o06
0.005
0.998
0.801
32(124)
Matryoshka (UW)
0.830
0.000
1.000 .000
0,853 .022
22(191)
12288
Matryoshka (RW, SC)
0.889 :: 005
0.007 = 0033
0.997 1-
0.8331.002
11(66)
Matryoshka (UW, SC)
0.8422..005
0.001 =: 0000
0.999 1. 0000
0.8491.002
87(172)
19

---

Matryoshka SAE
E. CKNNA Alignment Metric
Introduced in Section 4.4, CKNNA (Centered Kernel Nearest-Neighbor Alignment) measures representation similarity between networks while focusing local neighborhood structures. Unlike its predecessor CKA (Kornblith al., 2019), CKNNA refines the alignment computation by considering only k-nearest neighbors, making more sensitive to local geometric relationships. The alignment score between two networks' representations in case CLIP embeddings and SAE activation is computed as:
Align(K, L)
CKNNA(K,L)
/HSIC(K, L)
HSIC(K, L)
1
ΣΣ (1)) Billbi (())) tby) Ba ob))))
(n 1)2
(5)
Align K, (n 1)²
1
ΣΣ a(i.))((o,,))) - Eil(o,, (1))) vy) Ed(( 21111)
a(i,j:k) = 1[i + j and oj knn k) and +b knn( k)),
where HSIC measures the global similarity between kernel matrices, and Align introduces the neighborhood constraint through jii, k). The indicator function (i,, j: k) ensures that only pairs of points that are k-nearest neighbors both representation spaces contribute the alignment score. Here, 6,. oj represent CLIP embeddings and th 1) represent SAE activations for corresponding input data points and Following (Yu al., 2025), set k 10 it provides better alignment sensitivity and calculate CKNNA over randomly sampled (batch size) 10,000 representations when evaluating. Higher CKNNA scores indicate stronger similarity between the CLIP and SAE learned representations.
20

---

Matryoshka SAE
F. Evaluating MSAE: Additional Results
extend the results from Section by analyzing multiple expansion rates, SAE variants, input modalities, and CLIP architectures. Unless otherwise specified, experiments use CLIP ViT-L/14 with an expansion rate of on image modality. For text modality evaluations, use the CC3M validation subset, while image modality evaluations are performed on the ImageNet- Ik training subset.
ImageNet- 1K
CC3m Text
1.00
EVR ( better reconstruction)
1.0
0.95
0.90
0.5
0.85
0.0
0.80
0.75
0.5
0.70
-1.0
0.65
-1.5
0.6
0.7
0.8
0.9
1.0
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
1.000
CS ( better reconstruction)
1.0
0.975
0.9
0.950
0.8
0.925
0.7
0.900
0.6
0.5
0.875
0.4
0.850
0.3
0.825
0.6
0.7
0.8
0.9
1.0
0.4
0.5
0.6
0.7
0.8
0.9
1.0
LO
sparser)
Models
LO
sparser)
ReLU (A 0.03)
Top-K (k 32)
Batch Top-K (k = 32)
Batch Top-K (k 256)
ReLU 0.01)
Top-K 64)
Batch Top-K 64)
Matryoshka (RW)
ReLU (A 0.003)
Top-K (k 128)
Batch Top-K (k 128)
Matryoshka (UW)
ReLU (A 0.001)
Top-K (k = 256)
Figure 11. Extended -parsity-fidelity trade-off analysis across modalities. Expanding Figure 2, compare ReLU SAE (A 0.03, 0.01, 0.003, 0.001). TopK SAE 32, 64, 128, 256), BatchTopK SAE (k: 32, 64, 128, 256), and MSAE (RW. UW) using two reconstruction metrics: mean EVR fidelity (top) and mean cosine similarity (bottom). Results shown for both image (left) and text (right) modalities, with standard deviation also reported for each metric, demonstrating MSAE's consistent performance across modalities and metrics.
F.1. Sparsity-Fidelity Trade-off
Figure presents an extended analysis of sparsity-fidelity trade-offs, including standard deviations and an alternative reconstruction metric tailored for CLIP embeddings (cosine similarity). The results demonstrate MSAE's superior stability
21

---

Matryoshka SAE
both modalities, partieularly in text representations where only MSAE models shows stable and elevated results. Furthermore, observe that models learned with lower & higher sparsity regularization show high variance the trained (image) modality, and this instability becomes even more pronounced on the text modality, Due to MSAE's inherently low variance in the image modality, this similar low variance is preserved in the other modality, leading to consistently high and stable performance across both modalities. Figure 12 further strengthens our findings by showing MSAE's superiority on a different CLIP architecture (ViT-B/16).
ImageNet-1K
CC3m Text
1.00
EVR ( better reconstruction!
1.5
0.95
1.0
0.90
0.5
0.85
0.0
0.80
0.75
0.5
0.70
-1.0
-1.5
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.4
0.6
0.8
1.0
1.00
CS ( better reconstructiony
1.0
0.98
0.9
0.96
0.8
0.94
0.7
0.6
0.90
0.5
0.88
0.4
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.4
0.6
0.8
1.0
LO (
sparser)
Models
L0
sparser)
ReLU (A 0.03)
Top-K (k 32)
Batch Top-K (k = 32)
Batch Top-K (k 256)
ReLU 0.01)
Top-K (k 64)
Batch Top-K (k 64)
Matryoshka (RW)
ReLU 0.003)
- Top-K (k 128)
Batch Top-K (k 128)
Matryoshka (UW)
ReLU (A 0.001)
Top-K (k 256)
Figure 12. VIT-B/16 sparsity-fidelity trade-off analysis across modalities. Parallel analysis ViT-L/14 (Figure I1), demonstrating that MSAE's superior performance and stability generalizes across CLIP architectures.
22

---

Matryoshka SAE
F.2. Ablation: Matryoshka at Lower Granularity Levels
Figure 13 extends the analysis from Figure 3 by evaluating four key metries: reconstruction fidelity (EVR), reconstruction error (CS), alignment (CKNNA), and neuron utilization (NDN). Our expanded comparison reinforces MSAE's competitive performance against TopK SAE, demonstrating that MSAE (RW) achieves similar better results across most metrics except for NDN, where (UW) version of MSAE performs better.
EVR LO
CKNNA vs
Cosine similarity LO
Number dead neurons LO
EVR (- -bbbtee reconstruction)
1.00
CKNNA (1 better alignment)
0.875
C8 [ better reconstruction)
1.00
70
NDN (- better utilization))
0.98
0.99
0.96
0.825
0.98
50
Imagenet1k
40
0.94
0.800
0.97
30
0.775
0.92
0.96
0.750
0.90
0.95
10
0.88
0.94
0.700
0.90
0.95
0.90
0.95
0.90
0.95
0.900
0.925
0.950
LO (-+ sparser)
LO (- sparser)
L0 (- sparser)
L0 (- sparser)
Models
TopK (k = 128)
TopK (k = 256)
Matryoshka (RW) [128,256]
Matryoshka (UW) [128,256]
Figure 13. Comprehensive comparison of Matryoshka and TopK SAE on ImageNet-Ik. Extension of Figure 3 comparing model performance on reconstruction fidelity (EVR), reconstruction error (CS), concept alignment (CKNNA), and neuron utilization (NDN) against sparsity (LO). MSAE (RW) demonstrates competitive performance across most metries, while MSAE (UW) achieves better results in neuron utilization.
23

---

Matryoshka SAE
F.3. Activation Magnitudes Analysis
We extend the activation magnitude analysis from Figure 4 by including varying versions of model sparsity from each evaluated SAE architecture. Figure 14 shows non-zero and maximum SAE activations for expansion rate 8, revealing that less constrained TopK models exhibit double-curvature distributions similar MSAE and ReLU. Maximum activation analysis highlights ReLU's shrinkage effect, while TopK and MSAE maintain distributions closer to normal. These patterns persist at higher expansion rates (16 and 32) as shown in Figure 15.
Flattened Activation
Max Activation Per Sample
10²
600
ReLU (A 0.000)
Count
10
400
10
10
10²
600
ReLU =A=
Count
10
400
10
200
10'
10²
600
Topk (K = 64)
Count
10
400
10
200
10
10"
600
Topk (k= 256)
Count
10
400
10"
200
10.
10²
600
Matryoshka (RW)
Count
10
400
10
200
10'
Matryoshka (UW)
10"
600
Count
10
400
10a
200
10
10
20
10
15
Figure 14. Activation distributions at expansion rate 8. Extended analysis of Figure 4 showing: (left) non-zero activation distributions, revealing TopK's convergence to double-curvature patterns at lower constraints (higher k), (right) maximum activation distributions,
demonstrating ReLU shrinkage problem compared to TopK and MSAE behavior which resembles normal distribution.
24

---

Matryoshka SAE
Flattened Activation
Max Activation Per Sample
Flattened Activation
Max Activation Per Sample
10'
ReLU (A = 0.003)
10
RoLU (A 0,003)
600
600
Count
10
Count
10
10
200
10
200
10
10
ReLU (A = 0.001)
10"
ReLU (A 0.001)
600
Count
10
Count
10
400
400
10
200
10
200
10
10
10"
600
TopK (k 64)
600
Topk 64)
Count
10
Count
10
400
400
10
200
10
200
10
10'
TopK (k= 256)
10
TopK (k 256)
600
600
Count
10
Count
10
400
10
200
10
200
10
10
Matryoshka (RW)
10
Matryoshka (RW)
600
600
Count
10
Count
10
400
400
10
200
10
200
10
10
Matryoshka (UW)
10
Matryoshka (UW)
600
Count
10
Count
1
10
400
400
200
102
200
10
5
10
15
20
20
5
10
15
(a) Expansion rate 16
Expansion rate 32
Figure 15. Activation distributions at higher expansion rates. Extended analysis of Figure 4 for expansion rates 16 (a) and 32 (b).
showing consistency distribution patterns across scales.
25

---

Matryoshka SAE
F.4. Progressive Recovery
We extend the analysis from Figure 2 by examining progressive reconstruction performance across additional metrics and modalities. Figure 16 demonstrates performance expansion rate for reconstruction quality (EVR, CS) and neuron utilization (NDN) across both modalities, while Figure 17 extends this analysis to expansion rates 16 and 32.
Reconstruction Fidelity vs
CKNNA
Cosine similarity
Number Dead neurons vs
FVU better reconstruction FVU better reconstruction)
CKNNA (- better alignment)
CS (- better bateer reconstruction
1.0
NDN (- better utliiatioon)
0.8
5000
10
4000
Imagenet-1k
10
0.6
0.6
10
0.4
2000
0.4
10
0.2
1000
10
10"
CKNNA (- better alignment)
0.8
C5 ( better reconstruction.
NDN (- better utilization)
6000
10
5000
10
0.6
0.8
4000
CC3M Text
10
0.4
3000
10
0.4
2000
10
1000
0.2
10
10"
10'
10'
10'
10"
10"
LO (- sparser)
LO (- sparser)
L0 (- sparser)
LO
sparser)
Models
ReLU (A 0.003)
ReLU (A 0.001)
TopK (k 64)
Topk (k = 256)
Matryoshka (RW)
Matryoshka (UW)
Figure Progressive recovery analysis at expansion rate 8. Extension of Figure showing reconstruction (EVR, CS), alignment (CKNNA), and neuron utilization (NDN) metries against increasing number of utilized top magnitude SAE neurons for image and text modalitics, MSAE demonstrates comparable performance to TopK (k: 256) on image modality and superior performance on text.
26

---

Matryoshka SAE
Reconstruction Fidelity vs
CKNNA vs k
Cosine similarity vs
Number of Dead neurons vs k
FVU {- better reconstruction FVU (- better reconstruction)
10
CKNNA (- better alignment)
12000
better reconstruction
NDN (-- better utllization)
0.8
10
10000
0.8
10
0.6
Imagenet-1k 1k
10
0.6
6000
0.4
10
4000
10
0.2
2000
G9
0.2
10
10
CKNNA (- better alignment)
0.8
CS better reconstruction
1.0
12000
NDN (- better utilization)
10
10000
0.8
0.8
CC3M Text
10
0.4
6000
10
4000
0.2
2000
0.2
10
10
10
10
10
10"
10'
10"
L0 (- sparser)
L0 (- sparser)
LO (- sparser)
LO (- sparser)
Models
ReLU =A 0.003)
ReLU (A 0.001)
Topk (k 64)
TopK (k 256)
Matryoshka (RW)
Matryoshka (UW)
(a) Expansion rate 16
Reconstruction Fidelity vs k
CKNNA vs k
Cosine similarity vs
Number of Dead neurons vs k
FVU (- better reconstruction FVU (- better reconstruction)
10
CKNNA (- better alignment)
0.8
CS (- better reconstruction
NDN (- better utiliization)
25000
10
20000
0.8
10
Imagenet-1k
15000
0.6
10
0.4
10000
1
0.4
10
0.2
5000
0.2
10
0.0
10²
CKNNA (- better alignment)
0.8
L better reconstruction
1.0
NDN (- better utilizztion)
25000
10
0.8
20000
0.6
10
0.6
15000
CC3M Text
10
0.4
10
0.4
10000
-
0.2
0.2
5000
10
3
0.0
10"
10
10"
10
10'
10
10"
10"
LO (+ sparser)
L0 (- sparser)
L0 sparser)
L0 sparser)
Models
ReLU 0.003)
ReLU 0.001)
TopK (k= 64)
TopK (k 256)
Matryoshka (RW)
Matryoshka (UW)
(b) Expansion rate 32
Figure 17. Progressive recovery analysis at higher expansion rates. Analysis parallel to Figure expansion rates (a) 32 (b), demonstrating the stahility of the results over higher expansion rates.
27

---

Matryoshka SAE
E5. Comprehensive Evaluation with CLIP VIT-L/14 and ViT-B/16 Architectures
We present an extensive quantitative comparison of SAE variants across CLIP ViT-L/14 and ViT-B/16 architectures. Our evaluation encompasses ReLU (A 0.03, 0.01, 0.003, 0.001), TopK (k 32, 64, 128, 256, 512), BatchTopK (k 16, 32, 64, 128, 256), and MSAE (RW, UW) models, tested across three expansion rates (8, 16. 32) for both image and text modalities. For text modality and ViT-B/16 architecture, we omit LP (Acc) and LP (KL) metrics based on our findings in Section 4.4 that CS and FVU correlate strongly with linear probing metrics.
F.5.1, RESULTS FOR IMAGE MODALITY
For image modality, Tables 6-8 present detailed results for ViT-L/14 across expansion rates 8, 16, and 32, while Tables 9-11 show parallel performance metrics for ViT-B/16. These tables extend the analysis from Table 1, providing comprehensive measurements across different metrics and model configurations.
Table 6. CLIP VIT-L/14 SAE comparison at expansion rate 8. Extended evaluation from Table I with additional TopK, ReLU
and BatchTopK variants on ImageNet- lk. Arrows indicate preferred metric direction, NDN values show training set dead neurons in parentheses.
Model
Lo f
FVU +
CS f
LP (KL)
LP (Acc)
CKNNA f
DO
NDN
ReLU (A 0.03)
.200..os
.18
.928+.099
50.5
.364..24
.727 004
.003
0(0)
ReLU (A = 0.01)
.762+.010
.033+.00s
.985..002
7.16
197
.742+
.002
0(0)
ReLU (A= 0.003)
66194.007
.0012.000
.9982.000
0.66=1.03
.9941.083
.781 .044
.003
0(0)
ReLU 0.001)
.004..06
.0022.001
2.09
0.36
.9551.073
.822 .004
.002
0(0)
TopK (k: = 32)
9604.010
.245..043
.874..021
:61
.903
.711 005
.002
0(1235)
TopK (k: 64)
950+
.172#.025
.912#.013
60. 00.1gon.s
.930+.256
.762
.002
0
TopK (k 128)
.28
.098
.951=.007
2.71
.987+.144
.811 0044
.003
0(117)
TopK (k 256)
.004..044
.011#.om
.9944.002
2.71=5.40
.9874.114
.8744.00
.003
(
TopK (A: 512)
+.22+.015 .018
3464.442
+923
56. 13.1
.950
.006..003
.002
0(1)
BatchTopK (k: 16)
4.81
+.60
4.07
281 4:326
8364.372
1988.002 .33
.002
0(4278)
BatchTopK (k: = 32)
.7764.019
2242+.084
.873..02
113
.901
.735.0004
.002
((3880)
BatchTopK (k= 64)
8774.012
.1622.222
.917..011
56.9285.8
.911
.669 004
.002
0(1477)
BatchTopK 128)
.884..00
.0822.012
.959
23.3236.5
.9594.197
.805..044 .044
.003
0(539)
BatchTopK 256)
.882..005
.0102.005
.9952.002
2.42=5.12
.988..108
.860 003
.002
3(919)
Matryoshka (RW)
8294.008
0.03
.977..002
3.13
.987..115
.809 .022
.002
2(1)
Matryoshka (UW)
.7484.006
.0022.001
.909
0.35
.9954.070
.848 003
.001
0(22)
Table CLIP ViT-L/14 SAF comparison expansion rate 16. Results parallel to Table 6 showing performance scaling higher
expansion rate on ImageNct-1k.
Model
To
FVU 4
CS
(KL)
LP (Ace) f
CKNNA
DO 4
NDN
ReLU 0.03)
.95...006
.17 #.03
.939 008
41.1
4945
.714..00 004
0(0)
ReLU (A = 0.01)
4.88.
0036..00
.983 002
8.41 8.41 113.7
.975
.738..00 .005
.002
0(0)
ReLU (A = 0.003)
.716
.0064.001
.997
1.08
4..9
.695
.002
0(0)
ReLU (A 0.001)
664 0077
.001
.999 000
0.1420.22
.977..056
.7892.04
.002
0(0)
TopK (k: 32)
.972. .099
2449
.873_ 022
112+168
.8994.301
.692-.005
.002
0(4727)
TopK (A= 64)
.93 006
.174=.028
.911 014
61.729968
.927
7455
.002
0(2079)
TopK (A: 128)
.600..06
1004+.017
.948 ott
30.7+449.
.911..115
.001 801+.000
.002
1(997)
TopK (k 256)
.377.006
.019+.004
.991
3.76+6.85
.9844.17
.871
.004
15(1383)
Topk (k 512)
.644..00 0008
3364.413
.9262
72.9
.944..20
.007
.002
0(29)
BatchTopK (k 16)
.6694.021
404_.055
.786_
310+:353
.8994.77
000
0(9859)
Bopc (k: = 32)
.712..021
271
.866..20 020
122
.897
736
.002
0(8016)
BatchTopK (k: 64)
.800
.167
.9162
55.9466.0
.32
.759
.002
0(5113)
BatchTopp (l: 128)
8894.01
.0894.012
.957
24.6#38.5
.9564.204
.86-
.002
1 (2967)
BatchTopK (k 256)
880 .009
.023
.990
4.12+7.11
.9834.111
.854..0s .003
.002
12(5558)
Matryoshka (RW)
8844.006
.005...03
.988
2.08
.999
.801 2.03
.002
32(124)
Matryoshka (UW)
.8304.008
.000=..00
.999_
0.12
.9884.000
.853..02
.002
22(491)
28

---

Matryoshka SAE
Table 8. CLIP VAT- comparison at expansion rate 32, Analysis at maximum tested expansion rate ImageNet-Ik, completing the scaling study from Tables 6 and
Model
Lo f
FVU 4
CS f
LP (KL)
LP (Acc) t
CKNNA
DO 4
NDN +
ReLU (A 0.03)
964 004
120 029
9484.007
36.1260.2
.99 221
.707..00s
.003
0(0)
ReLU 0.01)
.833 006
.032 005
.854.0002
7.65
150
.7522.008
.002
0(0)
ReLU = 0.003)
.781 .781 + 007
0111..00 002
1905
2.0623.40
:98
.619..007
.002
0(0)
ReLU (A = 0.001)
653 005
.004+.001
.984.000
0.77=1.25
.93+.085
.93+.007
.002
0(0)
TopK (k 32)
.38
.246..047
.872..02
102.37
.964..2
.697..012
.002
64(14535)
TopK (k 64)
.73
.174..08
.911..014
61.7=96.8
.977.
.454..00
.002
0(9347)
TopK (k 128)
.64
110
4.774.009
31.4=51.2
..94
10(5604)
Topk (k 256)
942
+.08.
.866..00
5.39
..9
$8644.003
.003
(1(90)))
TopK (k = 512)
9664.004
+08
.964..002
1.30
.922
.422..018
.002
7822(22446)
Topc (k 16)
.638
4622.057
.7754.038
344.41377.0
.255 380
.7354.005
.001
0(21631)
BatchTopK (k =32)
.756
.399..02
.8554.024
139. 139.44186.6
8944..388
.7472.004
.001
0(18965)
BatchTopK 64)
880
.190+.024
.911+.018
55.4284.7
.9342.245 249
.7644.004
.002
0(15035)
BatchTopK (k 128)
$69
.129+.016
.917..007
24.7
.9564.206
.7944.00s
.002
1(11019)
BatchTopK (k 256)
8877..014
.069
.994.004
8,22
.964.154
.851
.002
12(11802)
Matryoshka (RW)
927
..03
.999..00
1.00
.9921.000
.8104.002
.002
79(112)
Matryoshka (UW)
.98
..00+.000
.994..00
0.09
.998+.
.850+.003
.002
297(162)
Table CLIP VIT-B/16 SAE comparison expansion rate 8. Parallel analysis ViT-L/14 (Table 6) using smaller CLIP architecture on ImageNet-Ik.
Model
To f
FVU 4
CS
CKNNA
DO
NDN 4
ReLU 0.03)
.908
.144.036
.9344.010
.671+.004
.004
0(0)
ReLU 0.01)
..12
.0304.000 000
.9864.002
682
.003
0(0)
ReLU (A = 0.003)
..29
.0034.000
.9994.000
.737 +003
.003
0(0)
ReLU (A = 0.001)
3664.012
.009
.964..011
6954.003
.002
0(0)
TopK (k: 32)
.46
.200..037
.8971.019
6844.000
.004
0(672)
TopK (k 64)
.35
.1384.024
.930+.012
.7304.005
.003
0(196)
TopK (k 128)
.833.020
.1164.026 026
.918..010
.735
.003
0(95)
TopK (k 256)
.859..08
.0244.007
.9884.003
.787+.002
.004
0(8)
TopK (k = 512)
882+.008
.058+.032
.92+.026
.005
.003
0(0)
BatchTopK
16)
.7064.023
.2994.050
.8454.000
.662+.006
.002
((2666)
Batc 32)
.800.019
.1964.031
.900+.017
.695
.003
0(1763)
Batch BatchTopK = 64)
.877..013
1.000
.9364.010
.750+.004
.003
0(830)
BatchTopt (k = 128)
882
1.48
.966
806
.003
0(387)
BatchTopK (k: 256)
8764.000
.003
.9994.001
.843
.003
35(1766)
Matryoshka (RW)
.783
.003 0.03..01
.999..00
.9224.003
.003
0(0)
Matryoshka (UW)
.7114.004
.000
.9994.000
.8144.002
.003
0(1)
29

---

Matryoshka SAE
Table 10. CLIP VIT-B/16 SAE comparison at expansion rate 16. Extension of Table to expansion rate 16 on ImageNet- Ik.
Model
Lo f
FVU 4
CS
CKNNAT
DO J
NDN 4
ReLU 0.03)
..40
.155..00
.945..008
.6644.006
0(0)
ReLU (A = 0.01)
..03
.844..002
6699
.003
0(0)
ReLU (A = 0.003)
695
.0044.008
.984..00
.355..00 005
.003
0(0)
ReLU (A 0.001)
6464.008
+.01+.000
4999
.742+.00
.003
0(0)
TopK (A 32)
.514..15
20644.042
.8964.021
0(2819)
TopK (k: 64)
.624
4.2
.930+.012
.710
.003
0(1268)
TopK (k 128)
950
.0721.016
.65...007
.7822.000
.004
0(597)
TopK (& 256)
935 003
.0031.002
.988.000
.839 002
.003
2(1686)
TopK (k 512)
.431.015
201
.9551.047
.006
.003
1(1)
BatchTopk Topk = 16)
.658+.022
.39
.837+.033
.670+.006
.002
0(6178)
BatchTopK (k 32)
.757..022
230
.8644.019
.7044.000
.002
0(5114)
BatchTopK (k = 64)
853+.016
.1354.019
.988.0099
..41+.005
.003
0(3145)
Botc 128)
8871.010
10611.011
1..0
.7994.004 0.00
.003
0(1817)
BatchTopK 256)
.8644000
.003
49994.001
4.03
.002
40(4947)
Matryoshka (RW)
8611.005
.0024.001
.9994.001
.7784.003
.003
8(63)
Matryoshka (UW)
.005..044
.000
.999+.000
.813+0003
.003
44(275)
Table 11. CLIP ViT-B/16 SAE comparison at expansion rate 32. Completion of ViT-B/16 scaling analysis ImageNet- 1k maximum
tested expansion rate.
Model
La t
FVU 4
CS
CKNNA t
DO 4
NDN
ReLU 0.03)
.56 005
.104
953+.007
.656.004
.003
0(0)
ReLU 0.01)
879
031+.000 000
.864..002
6884.000
.003
0(0)
ReLU (A 0.003)
.757 009
.010+.0P 002
.96+..01
5684.005
.003
0(0)
ReLU (A 0.001)
6655
.004+.001
.984+.000
.164..05
.003
0(0)
TopK (k 32)
.33 025
.230
.81
.6741.005
.003
0(9223)
TopK (k 64)
967 1.7.0012
1.22..02
.977..014
4.98
.003
0(5643)
TopK (k 128)
.600
.085..023
.611
.7724.002
.003
2(3321)
TopK (k 256)
.22--014
.015+.006
.995+.002
.8224.002
.003
1(10480)
TopK (k 512)
.967
.001+.001
1.0004.001
.0164.007
.002
4902(14866
BopK (k 16)
6115-.088
.50+.060
.822+.037
66554.005
.002
0(14345)
TopK 32)
u21
.322.043
.890+.099
.7194.u0b
.002
0(12492)
BatchTopK = 64)
835+.017
.184..28
.933+.010
.7314.005
.002
2(9645)
BatchTopK (k 128)
856+.018
.0964.015
.9664.005
.7984.003
.003
0(6855)
BatchTopK (K 256)
844.011
.0401.007
.9111.002
.8271.003
.002
0(11638)
Matryoshka (RW)
.955
.001+.001
.9994.000
7944.003
.003
6(23)
Matryoshka (UW)
880+
.000+.000
.9994.000
.8044.002
.002
15(26)
30

---

Matryoshka SAE
F.5.2. RESULTS FOR TEXT MODALITY
For text modality, Tables 12-14 present results for ViT-L/14, while Tables 15-17 show ViT-B/16 performance on CC3M validation text data, enabling cross-man cross-modal and cross-architecture comparisons.
Table 12. CLIP VIT-L/14 SAE text analysis at expansion rate 8. Evaluation CC3M text validation set parallel to image results in Table highlighting cross-moud modal performance diffcrences.
Model
Lo 1
FVU
CS
CKNNA f
DO
NDN
ReLU (A 0.03)
.011..10
427+.114
.802.009
.622..ux
.003
0(0)
ReLU 0.01)
5222.041
.036 058
.9441.014
7766..006
.002
0(0)
ReLU 0.003)
2009
.041
9814.011
..14.002
.003
0(0)
ReLU (A = 0.001)
5222.045
.035 038
.814.0014
7706
.002
0(0)
TopK (k 32)
.724+3118
1.095+.830
.511..006
.037+.081
.002
0(1235)
TopK (A: 64)
.755.299 295
.760
.854.209
.025+.020
.002
0(335)
TopK (k 128)
.781
2.77
.708
.022..011
.003
0(117)
TopK (k 256)
..83
3661..36
742
.0881.007
.003
0(296)
TopK (k = 512)
4000..06
4.86
8654..101
.088
.002
0(1)
BatchTopK (k 16)
18851.111
1.222
.416..213
.031 1.026
.002
0(4278)
BatchTopK
32)
.608..221
848
4.222
.022+.012
.002
0(3080)
Batc = 64)
.712
229
.581
.0194.018
.002
0(1477)
Batc (k 128)
2888..08
A15
.7874.102
.122.009
.003
0(539)
Botc (k 256)
.899..22
.1594.175
.9184.108
.7164.004
.002
17(919)
Matryoshka (RW)
8244.029
.0601.052
.911.006
.775..001
.002
0(4)
Matryoshka (UW)
.7552.024
.0261.027
.9881.012
.7904.002
.001
0(22)
Table 13. CLIP VIT-I/14 SAE text analysis at expansion rate 16. Extended CC3M text evaluation showing scaling effects at expansion
16, complementing image results from Table 7.
Model
Lo
FVU 4
CS
CKNNA t
DO 4
NDN 1
ReLU (A 0.03)
9304.027
5100..00
.812 052
.581 1
.003
0(0)
ReLU (A 0.01)
8094.052
2.11.225
.926 043
.643
.002
0(0)
ReLU (A 0.003)
6674..06
.0704.060
.933 0233
.654..009
.002
0(0)
ReLU (A 0.001)
.5994.08
.021+.019
.9904.000
.781
.002
0(0)
TopK (k 32)
782
1.237
.944
.038+.028
.002
0(4727)
TopK (A: 64)
.7744.280
.7904.258
5834.203
.023+.008
.002
0(2079)
TopK (k 128)
8134.212
.6044.255
.600..10
.029+.u10
.002
0(897)
TopK 256)
848
4.07
.740..20
.093+.004
.004
0(1883)
TopK (k- = 512)
.950.02s
4354.455
8554.107
.073+.u14
.002
0(29)
Topc (k 16)
6600..115
.977.3326
.501..02
.0321.018
.001
0(9859)
Totc (A: 32)
.6194.181
..58 209
.5811.32
.031
.002
0(8016)
BatchTopK (k: 64)
.7064.242
..874.236
.5591..200
.0291..20
.002
0(5113)
BatchTopK 128)
.8664.034
.654.230
.7964.081
.210
.002
(4
BatchTopK (k: 256)
.889
.322..71
.7414.334
.036+.004
.002
7(3558)
Matryoshka (RW)
.800..021
.0431.038
.980..019
.783
.002
4(124)
Matryoshka (UW)
832+.028
.0174.017
.9922.008
.7882.001
.002
0(491)
31

---

Matryoshka SAE
Table 14. CLIP VIT-L/14 SAE text analysis at expansion rate 32. Maximum expansion rate analysis on CC3M text.
Model
Lo f
FVU
CS f
CKNNA
DO 4
NDN 4
ReLU (A 0.03)
.951
.540..667
.822+.053
2557
.003
1(0)
ReLU (A = 0.01)
8677.050
3394.546
.977..047
.519
.002
1(0)
ReLU (A 0.003)
.7494.09m
.1724.200
.966..027
.336 008
.002
0(0)
ReLU (A 0.001)
6314.004
.052+.045
.983+.014
.36+..07
.002
0(0)
TopK (k 32)
8.44.144
1.149
1.99
.058+.u18
.002
0(14535)
TopK (k: 64)
8884.156
.7954.
.630+.157
U226
.002
0(2079)
TopK (k: 128)
156
.6121.252
.717:.161
.053
.002
0(5601)
TopK (k 256)
164
1354.342
.740..240
.130
.003
0(6590)
Topk (k 512)
.9771.062
.1154.131
.944.069
.070+.034
.002
240(22116)
Topk (k: = 16)
.601+.079
.188.209
1964.191
.031+.024
.002
0(21631)
Batc (I: 32)
6881..18
.33
.592+.190
.034+.000
.002
0(21631)
BatchTopK (k: 64)
.7844.178
.6364.225
.657+.181
.042+.066
.002
0(18965)
BatchTopK (k: 128)
8611.016
1.7
880
.526#.012
.002
94(11019)
BatchTopk (k 256)
..84
3324.333
.778 288
.060
.002
0(11802)
Matryoshka (RW)
9254.014
.0304.026
.9862.013
.774
.002
32(142)
Matryoshka (UW)
.901..26
.013+.013
.9944.006
.784+.000
.002
126(162)
Table 15. CLIP VTT-B/16 SAE text analysis at expansion rate CC3M text evaluation using smaller CLIP architecture, enabling cross-modal and cross-anchitecture comparisons.
Model
Lu
FVU
CS
CKNNA
DO 1
NDN 1
ReLU (A 0.03)
870+.02
.42
.661+.06
.661
.004
0(0)
ReLU (A 0.01)
4.44
.41
.920
.721+008 003
0(0)
ReLU (A 0.003)
580+.028
.030+.o28
.864.013
.764+ 000
.003
0(0)
ReLU (A 0.001)
392
.1184.177
.975.022
.129
.002
0(0)
TopK (A: =32)
757
1.095
.533
.035
.004
0(672)
TopK (k 64)
.664..22
4.33
6644
.033
.003
0(196)
TopK (A: 128)
1.14
.5154.343
1.06
.275
.003
0(95)
TopK (A: 256)
.7834.095
2229..152
888
.759-.000
.004
0(8)
TopK (A 512)
.7944.174
.2831.282
.800..144
.079
.003
0(0)
BatchTopK (k: 16)
1562..16
1.172+1.110
.614.178
.018+.010
.002
0(2666)
BatchTopK (k: 32)
.6624.113
.8984.110
.547
.016+.0m 00S
.003
0(1763)
BatchTopk (A: 64)
.7164.204
.6564.212
6637+.166
.029 OUS
.003
0(830)
BatchTopK (k: 128)
.7154.170
.6544.a22
.662+.266
.033
.003
0(387)
BatchTTpp (A: 256)
.7741.177
3177..358
.7761.301
.103 004
.003
0(1766)
Matryoshka (RW)
7624.063
.044
.979
.799
.003
0(0)
Matryoshka (UW)
.7094.048
.021+.026
.990+.012
.81++ 008
.003
0(1)
32

---

Matryoshka SAE
Table 16. CLIP VIT-B/16 SAE text analysis at expansion rate 16, Results for expansion rate 16 with ViT-B/16 on CC3M text data.
Model
Lo
FVU +
CS
CKNNA
DO 4
NDN 1
ReLU (A 0.03)
.910+.02
457+.178
.767
.614+.008
.004
0(0)
ReLU (A 0.01)
.7771.065
2.48
9.1
600 003
.003
0(0)
ReLU (A 0.003)
.6414.055
.0521.034
.94
.767
.003
0(0)
ReLU (A 0.001)
1.777
.0131.012
.999..06
.678 002
.003
0(0)
TopK (A =32)
1.12.232
1.163
.518 1.85
.043 .20
.003
0(2819)
TopK 64)
2.11.
404
.18
.028 000
.003
0(1268)
TopK (A: 128)
.7814.217
.601
696
.045 002
.004
0(597)
TopK (A 256)
.7874.224
430+.401
6888..26
.098 007
.003
0(4686)
TopK (k 512)
.918..011
4474.804
891
.070
.003
0(1)
BatchTopk (k 16)
.592+.109
9404.610
.880..18
.018..077
.002
0(6478)
BatchTopk (A: 32)
.650+.174
8434.442
.604.182
.01+.007
.002
0(5114)
BatchTopK (A: 64)
.7344.174
.6491.212
6554..153
.022 000
.003
0(3145)
BatchTopK (k: 128)
.7644.218
.6431.310
.685..216
.032 004
.003
0(1817)
BatchTopk (k: 256)
.9991.194
.337
.755+.321
.061
.002
0(19-17)
Matryoshka (RW)
8471.040
.0331.036
.84
800 002
.003
0
Matryoshka (UW)
8011.003
.017:.021
.922.010
8033 0001
.003
(
Table 17. CLIP ViT-B/16 SAE text analysis at expansion rate 32, Final expansion rate evaluation for ViT-B/16 on CC3M text.
Model
Lo f
FVU +
CS
CKNNA f
DO 4
NDN 1
ReLU 0.03)
.344.019
472 4.35
.790+.088
.610 008
.003
0(0)
ReLU (A 0.01)
428
1.3941.62
.844..77
.145+.01. 015
.003
0(0)
ReLU (A = 0.003)
.713
.183
.9634.028 028
.304 004
.003
0(0)
ReLU (A 0.001)
.600+.551
.041+.033
.9874.009
.422 .002
.003
0(0)
TopK (k: 32)
1854..18
1.091+1.50
.4171.163
.076 008
.003
099223)
TopK (k: 64)
879
832+.622
.645..18
0755-002
.003
0(5643)
TopK (k: 128)
8634.146
.590+.247
.740+.12
.150 029
.003
0(3321)
TopK (k 256)
.793
.354..50
.776+.253
.197
.003
0(10480)
TopK (l 512)
.171.109
.0971.17
.9551.087
.027=
.002
73(14864)
BatchTopK
16)
.581 1.066
.769
.5361.157
.027=.012
.002
0(14345)
BatchTopK = 32)
.657
.726 1.244
.628
.032 020
.002
0(12492)
BatchTopp (k: = 64)
.7662.126
.631 2:22
.7044.113
.052= 024
.002
0(9645)
BatchTopK (k 128)
.7834.168
.6284.335
.727+-168
.046 .16
.003
0(6855)
BatchTopK 256)
.7734.172
.311
.7994.270
.117
.002
0(11638)
Matryoshka (RW)
8977..05
.022+..26
.900+.013
.807 0002
.003
1(23)
Matryoshka (UW)
.834..00
.010+.014
.9954.007
.8062.003
.002
0
33

---

Matryoshka SAE
G. Interpreting CLIP with MSAE: Additional Results
this appendix section, we provide additional analysis supporting Section 5. Section G.1 presents high-magnitude activation samples across modalities from MSAE (RW) with an expansion rate of Section G.2 demonstrates how SAE enhances similarity search with interpretable results. Section G.3 presents statistical gender bias analysis on CelebA dataset, supported by concept manipulation visualizations that reinforce the statistical findings. These analyses strengthen our findings from Section 5 while providing deeper insights into MSAE's interpretability capabilities.
G.1. Concept Visualization Analysis
Figures 18 and 19 showcase six valid concepts through their highest-activating images and texts, confirming concept validity, Conversely, Figure 20 demonstrates two invalid concepts, highlighting the importance of validation methods from Section A.
Neuron 4169 (blue) with similarity 0.71
Neuron 3730 (germanyy) with similarity 0.48
Neuron 3121 (light) with similarity 0.62
Neuron 3728 (incradibleindia) with similarity 0.54 Neuron 310 (nighttime) with similarity 0.64
Neuron 2242 (ireland) with similarity 0.53
Figure High-magnitude image activations for valid concepts. gather top activating ImageNet-Ik -kk images for valid MSAE (RW) concept neurons.
34

---

Matryoshka SAE
Neuron 3091 (alooholic) with similarity 0.52
Neuron 3552 (smale) with similarity 0.59
1. because this smile can cure the blues
beverage type the original beer
2. pouring bear into glass, over green background
2. with lovely smile
portrait smiling bay
customers drink
4. portrait of a senior man smillig
set bear bottles with no labels isolated on
5. detail view of pretty middle aged woman smillng
alcohol empty bottles in a window after a party
even
walking ... amiles!
young with of beer
7. man smillng to the camera
(a)
(b)
Neuron 5071 (trzo) with similarity 0.61
Neuron 2539 (heart) with similarity 0.50
person... person and myself before the race
a symbolic electronic circuit in heart shape
2. myself and the newlyweds at wedding that person private residence
ill ilssrration of the flag shaped like heart
heart shape single wine halder capable of holding wine glasses
putting brave face shortly before they announced their
couple person
4. love in the air heart waiting us climbed
4. family three on summer vacation.
5. backstage beauties and artist after the main event
happily posed with
6. flag with pole and on heart shape
(c)
(d)
Neuron 2495 (questions) with similarity 0.53
Neuron 220 (runnin) with similarity 0.55
1. sisters running to the beach
asks questions monday of people arriving the emergency room
2. person running at the beach
which players could leave football team january
3. friends running towards sea beach
3 animal wants know if you would like hold next
couple running the sea
what year the bridge undergo award winning
5, teenage soccer players running down the field
6, person, is seen running the streets..
name the team to face city
7. person runs through the wildemess..
(c)
Figure 19. Cross-modal highest valid concept activation samples. Extending Figure show the highest-activating ImageNet- images and CC3M texts from valid MSAE (RW) concepts: smile, alcoholic, trio, heart, running, and questions.
35

---

Matryoshka SAE
SIX
H
Midee from the
B.RGHESSSOON LTD.
must of
Neuron 17006 most active for: (0.45) hri (0.44) hig (0.43)
Concept 6 with two highest neuron similarities 0.4 and 0.17
Texts:
Texts:
1. hard rock artist the heavy band performs a live concert
1. where sediment comes from
the hawk first entered service both as an advanced flying training aircraft and weapons
2. blues artist performs on stage during the fifth day
3. 60th anniversary: a sneak peek at new food and drinks
habitat the type enviranment which organism lives where organism commonly
smile during photo
film 58th
students participate in activities for the 75th anniversary celebrations
holiday : has heritage that stems
the harbour summer day
6. reserve at sixty three 63 has wide variety of spacious floor plans
hand drawn head of hen label white background
50th anniversary of 8. the numbers and two painted on dilapidated brick wall
concert poster folk rock
7. the chocolate bar that was launched and is based on hill
8. some different wheels the hawk
(a)
(b)
Figure 20. Analysis of invalid concept neurons in MSAE (RW). In (a), showcase the invalid concept '6' with low similarity (< 0.42), which shows inconsistent presence of the number six in the top active samples. In (b), we present how a low ratio threshold (0.45/00.44 indicate broader 'h' concept rather than specific 'hl'/"hri" from the vocabulary.
36

---

Matryoshka SAE
G.2. SAE-Enhanced Similarity Search
Building upon Section demonstrate how SAE enhances nearest neighbor (NN) search by revealing shared semantic concepts between query and retrieved images. Figure 21 illustrates how SAE uncovers interpretable features that drive CLIP's similarity assessments. Furthermore, show that conducting similarity search directly in the SAE activation space produces comparable results to CLIP-based search while providing more semantically meaningful matches.
Input Image
CLIP Distance 0.28
CLIP Distance 0.39
SAE Distance 384.08
SAE Distance 401.58
ireland police
ireland
britain police foodtruck
ireland police foodtruck
britain
police
police
foodtruck
foodtruck
manchester
buri
britain
automotive
britain
foodtruck
britain
officer
officer
automotive
ouring
bor
ouring
automotive
chatt
automotive
alumnus
automotive
crowdfunding
crows
prohib
durham
Figure 21. SAE-enhanced similarity search. Examples demonstrating how SAE reveals shared semantic concepts (hottom row) hetween query images and their CIJIP nearest neighbors (top row), providing interpretable explanations for similarity matches. Additionally. the two rightmost examples show nearest neighbors retrieved based on SAE activation similarity, demonstrating searching SAE space yields similar results to CLIP-basee based search while making the retrieval process more semantically interpretable.
G.3. Gender Bias Analysis in CelebA
We analyze gender biases in CLIP- based classification model using the CelebA dataset, which forms the foundation for our analysis in Section 5.3. Through statistical analysis of concept magnitude distribution against the model gender predictions Figure 24, we identify significant gender associations for concepts bearded, blondes, and glasses in classification model. To verify that these concepts align with the true features in the CelebA dataset, we visualize sighest-activatiin images for each concept in Figure 22. Further concept manipulation experiments both female (Figure 7) and male (Figure 23) examples confirm and strengthen these statistical findings, providing greater insight into the relationship between gender classification and the chosen concepts.
Neuron 1877 (bcarded)) with similarity 0.38
Neuron 4890 (glasses) with similarity 0.59
Neuron 1899 (Mondes)) with similarity 0.49
Figure Highest-activating CelebA images gender-associated concepts. visualize images from CelebA test set that produce the highest activations for the concepts bearded, blondes, and glasses, validating their alignment with the concept.
37

---

Matryoshka SAE
Class: Male; p=0.08
Effect Concept bearded
Effect of Concept glasses
Effect Concept blondes
Class Probability
0.10
0.8
0.08
0.6
0.2
0.06
Class probability
0.1
Original value
0.2
0.04
Threshold
0
100
200
300
100
200
300
50
75
Neuron Magnitude
Neuron Magnitude
Neuron Magnitude
Figure 23. Impact of concept manipulation for the male example. Complementing Figure 7, we further strengthen our findings of male association for bearded, moderate for glasses, femalc bias for blondes concept.
bearded Density Plot
bearded Plot
black
Density Plot
black Plot
Concept Magnitude
Concept Magnitude
Concopt Magnitude
Concept Magnitude
2.0
1.5
1.0
2
2
0.5
0
0.0
ginger
Density Plot
ginger
Box Plot
hair
Density Plot
hoir Box Plot
Concept Magnitude
Concept Magnitude
4
Concept Magnitude
Concept Magnitude
1.5
3
1.0
2
5
2.5
0.5
0
0.0
0.0
blondes
Density Plot
blondes Plot
glasses
Density Plot
glasses Plot
Concept Magnitude
Concept Magnitude
Concept Magnitude
10.0
Concept Magnitude
7.5
5.0
1
2.5
0.0
0
Gender Prediction
Gender Prediction
Gender Prediction
Gender Magnitude
Figure 24. Statistical analysis of concept-gender associations. We analyze six concepts: bearded, blondes, black, hair, glasses, and ginger. For each concept, we show its density distribution of concept magnitude against gender prediction alongside corresponding boxplots. Results reveal that bearded, blondes, glasses exhibit significant cender-specific associations.
38

---

Matryoshka SAE
H. Stability Evaluations
To gain a deeper understanding of the stability of learned feature directions for the decoder and encoder across various training seeds, we calculated the stability metric proposed by Paulo & Belrose (2025). Table 18 shows the results for all tested SAEs at expansion rate of 8. We observe that the stability metric is highly correlated with sparsity. Furthermore, Matryoshka SAE demonstrates comparable stability trade-off to alternative architectures.
Table 18. Sattis--SSpassi----eetttrtttiee trade-off (Parcto front) for CLIP (VIT-L/14) on ImageNet-Ik 1k. Rows are sorted by sparsity. We observe that (I) stability is highly correlated with sparsity, and (2) Matryoshka SAE exhibits an on-par stability trade-off as compared to other architectures.
Model
Sparsity (LO 1)
Reconstruction (FVU !)
Stability (Decoder/Encoderr f)
TopK (A: = 32)
.960
.245
649//245
TopK (A- 64)
.950
.172
.688 .240
Topk (A 128)
.928
.098
.625/ 6187
Topk (k 512)
.922
.366
.28/ 1187
ReLU (A 0.03)
.920
.15
522 .124
TopK (k: = 256)
.900
.011
6624/ .235
BatchTopK (k = 128)
.898
.082
.622/ .228
Botc (I: 256)
.882
.010
573/ .231
BatchTopK (k 64)
.877
.162
.86/ .228
Matryoshka (RW)
.829
.007
.437.102
BatchTopp 32)
.776
.242
.667/168
ReLU (A 0.01)
.762
.033
401//068
Matryoshka (UW)
.748
.002
.366/6600 3666/065
BatchTopK (k- 16)
.698
.371
.352/ ,108
ReLU (A 0.003)
.649
.004
.334/ .042
ReLU (A 0.001)
.553
.002
.200 .041
39


Reviews:
[{"review_id": "OB21XYslE4", "reviewer": "Reviewer_Zxzv", "summary": "This article explains the CLIP model from the perspective of model parameters. The author uses sparse autoencoders to sparse the content learned by the neurons of the CLIP model. Specifically, the author proposes Matryoshka Sparse Autoencoder, which is a hierarchical encoder used to hierarchicalize conceptual information when training SAE. The author has done a lot of applications to prove the practicality of this method.\n\n## update after rebuttal", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We gratefully thank the reviewer for their engagement with our work and appreciation of our contribution.**\n\n> The authors seem to lack quantitative comparisons with some existing SAE-based neuron interpretability methods, only some simple ReLu and TopK-based methods.\n\nExisting SAE-based methods have been primarily compared in interpreting language models, **while our work goes beyond to specifically interpret vision–language models instead**. Therefore, we choose ReLU and TopK as the potential baseline methods. As suggested by Reviewer #T2kP, we now attempted to compare with JumpReLU SAE. Unfortunately, the JumpReLU preprint has no official code implementation available. We now used the unofficial implementation shared in the SAELens software to compare with MSAE (ImageNet-1k, CLIP ViT-L/14). In our preliminary experiments, the results of JumpReLU are virtually the same as those of ReLU. Note that a similar issue with reproducibility has been publicly pointed out in “Interpretability evals case study” (Transformer Circuits Thread, August 2024). We now share our result in the updated Tables 6 & 7 in anonymous cloud storage at https://drive.google.com/drive/folders/11OOSpexmU5ul8nBhJqpaf65Dh_eqBlZY (file `a_updated_tables_6_and_7.pdf`).\n\nWe initially also considered comparing with Gated SAE, but acknowledged that TopK outperformed it based on the results from (Gao et al., 2024).\n\n> Similarly, although the method is practical in downstream tasks, it is difficult to intuitively feel the degree to which it surpasses advanced methods. I suggest that the author consider supplementing the latest comparison method, and then try to explain the same parameters of the same model, and then conduct some applications to help us intuitively feel the difference in the ability of different SAE-based methods to explain parameters.\n\nThank you for highlighting the practical utility of our method. Kindly, it is unclear to us which “advanced methods” were meant by the reviewer. We agree it is challenging to provide an intuitive feeling regarding how different SAE architectures explain the model. Thus, **we rely on 5 quantitative metrics established in the literature, and introduce 2 novel metrics (CKNNA and DO) to assess MSAE against baselines**, an evaluation previously unexplored for CLIP (Section 4). We then demonstrate the best-performing SAE's utility in three downstream tasks (Section 5), providing visual explanations to further illustrate its effectiveness (Appendix Figures 18–23). \n\nTo address the reviewer’s suggestion, we supplement our work with additional visual examples akin to Figure 10. **We now share figures with explanations of 502 concepts across 6 different SAEs (appearing in Table 2) in anonymous cloud storage at** https://drive.google.com/drive/folders/11OOSpexmU5ul8nBhJqpaf65Dh_eqBlZY (files `concept_visualization_*.pdf`) to facilitate building up this intuitive feeling. Each figure presents inputs the most activating a given concept across both modalities (vision and language). For a broad overview, we visualize all validated concepts (see a discussion in Appendix A) appearing in any of these 6 models (see the last column in Table 3). \n\n> The author explains CLIP throughout the paper, but the method in this paper does not seem to be a method for explaining the CLIP model specifically, and it can still be applied to other single-modal models. The author needs to clarify the relationship between the method in this paper and CLIP. Is it a specific design, or can it be universal? (If it is universal, it is recommended that the author can add some experiments for interpreting other model parameters).\n\nThis is a good point. While we demonstrate our method on the multi-modal CLIP model for broader appeal (encompassing both vision and language), our approach is indeed applicable to single-modal models as well. We chose CLIP due to its widespread interest and the potential for insightful interpretations across modalities. Unlike much of the previous work, which focused on language models, we apply SAE to interpret vision–language models often used as feature extractors. Application of SAE to the final representation of the feature extractor like CLIP avoids the issue of concept re-emergence after steering, which remains a challenge in LLM applications. Note that our steering presented in Sections 5.2 & 5.3 uses the CLIP’s final representations, circumventing this problem.\n\nWe appreciate the reviewer’s feedback and will incorporate the discussion in the next version of the paper.", "claims_and_evidence": "The author mentioned in the introduction that the previous SAE method was limited by Top-K and L1 losses, and the hierarchical learning proposed in this paper can overcome these limitations. Although the key issues are mentioned, it seems unclear how the method in this paper solves these problems.", "methods_and_evaluation_criteria": "The authors adopted the evaluation metrics commonly used in previous work to evaluate SAE.", "theoretical_claims": "No theoretical analysis in the article, if I miss it, please remind me.", "experimental_designs_or_analyses": "The authors designed many downstream applications to demonstrate the practicality of the proposed method.", "relation_to_broader_scientific_literature": "The research in this article helps promote the development of explainable AI, especially parameter-level explanations.", "essential_references_not_discussed": "Not sure", "other_strengths_and_weaknesses": "**Strengths:** It is very meaningful to study the use of sparse autoencoders to explain model parameters. The author proposed hierarchical learning to distribute the concept learning of sparse autoencoders from coarse-grained to fine-grained, and demonstrated the practicability of the method in multiple downstream tasks.\n\n**Weaknesses:**\n\n- The authors seem to lack quantitative comparisons with some existing SAE-based neuron interpretability methods, only some simple ReLu and TopK-based methods.\n\n- Similarly, although the method is practical in downstream tasks, it is difficult to intuitively feel the degree to which it surpasses advanced methods. I suggest that the author consider supplementing the latest comparison method, and then try to explain the same parameters of the same model, and then conduct some applications to help us intuitively feel the difference in the ability of different SAE-based methods to explain parameters.\n\n- The author explains CLIP throughout the paper, but the method in this paper does not seem to be a method for explaining the CLIP model specifically, and it can still be applied to other single-modal models. The author needs to clarify the relationship between the method in this paper and CLIP. Is it a specific design, or can it be universal? (If it is universal, it is recommended that the author can add some experiments for interpreting other model parameters).", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review."}}, {"review_id": "xkVNfr71JG", "reviewer": "Reviewer_26gq", "summary": "Sparse Autoencoders (SAEs) have been adopted to interpret CLIP’s feature representations, but the trade-off between reconstruction quality and sparsity has made it difficult to strike an ideal balance for interpretation. This paper proposes Matryoshka Sparse Autoencoder (MSAE), a hierarchical extension of SAEs, to analyze CLIP’s representation space in multiple granularities. By doing so, MSAE can extract more than 120 semantic concepts from CLIP’s embedding space and demonstrate its applicability in concept-based similarity matching (with controllable concept strength) and gender bias analysis for downstream tasks.\n\n## Update after rebuttal\nI agree with the concern raised by other reviewers regarding the lack of quantitative comparisons. The authors' response does not seem to adequately address this issue. Therefore, I have lowered my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank the reviewer for acknowledging the quality and significance of our work.**\n\n> While Section 4 demonstrates the advantages of MSAE over standard SAEs, Section 5 primarily focuses on MSAE’s applications—showing concept extraction and bias analysis with CLIP. It would have been even stronger if the paper made explicit how previous SAEs would fail or underperform in these same tasks. [...] It might help to clarify in Section 5 exactly where and why alternative SAEs (e.g., single-threshold TopK) fall short for concept-based similarity or bias detection. This would further underscore MSAE’s value for real interpretability applications.\n\nThank you for raising this point. We agree with the reviewer that a measurement of interpretability would further strengthen our paper and view it as a natural future work direction. To the best of our knowledge, **our work is the first to evaluate well-established SAE architectures using multiple quantitative metrics beyond LLM applications**. Given this scope, there is a limit to how many new contributions a single paper can explore exhaustively (new architecture, bi-modal setting, state-of-the-art performance, potential applications, and also new evaluation protocol). \n\nTo facilitate research into this direction, we supplement our work with additional visual examples akin to Figure 10. **We now share figures with explanations of 502 concepts across 6 different SAEs (appearing in Table 2) in anonymous cloud storage at** https://drive.google.com/drive/folders/11OOSpexmU5ul8nBhJqpaf65Dh_eqBlZY  (files `concept_visualization_*.pdf`). Each figure presents inputs the most activating a given concept across both modalities (vision and language). For a broad overview, we visualize all validated concepts (see a discussion in Appendix A) appearing in any of these 6 models (see the last column in Table 3). \n \n> In Section 5.3, the authors train a single-layer classifier on CLIP embeddings to identify bias for the CelebA experiment. Couldn’t the text encoder in CLIP itself be used for classification without additional training? Why did you decide to train a new classifier for this bias detection step?\n\nYes, the CLIP's text encoder can generally be used for zero-shot classification. However, we chose to fine-tune a classifier on top of CLIP to demonstrate the broader applicability of MSAE to interpret other models. For example, another classifier trained on the CLIP’s representation, or even another feature extractor (backbone) model that lacks zero-shot classification capabilities, unlike CLIP. We use MSAE to generate counterfactual explanations, which are particularly valuable in bias-sensitive predictive tasks like CelebA.", "claims_and_evidence": "**Claim 1**: Existing SAE-based methods face limitations when interpreting CLIP’s multimodal representations.\n- Evidence: Prior studies indicate that simple L1 regularization can undervalue important features, while a strict TopK approach enforces overly rigid sparsity, imposing interpretability constraints.\n\n-> The authors cite some earlier works convincingly, making a clear and logical argument.\n\n**Claim 2**: MSAE achieves a superior balance between sparsity and reconstruction quality compared to existing SAE baselines.\n- Evidence: As the shown in Figure 2, MSAE consistently outperforms standard ReLU and TopK SAEs on EVR vs. L0.\n\n-> Experimental results show clear and convincing evidence.\n\n**Claim 3**: MSAE facilitates more robust interpretation of CLIP’s multimodal representation.\n-  Evidence: By applying MSAE to CLIP’s embedding space, the authors discover more than 120 interpretable concepts (Table 3).\n\n-> Experimental results show clear and convincing evidence.", "methods_and_evaluation_criteria": "- The proposed multi-TopK approach is well aligned with the goal of preserving hierarchical structure in CLIP embeddings.\n- The evaluation metrics effectively measure reconstruction fidelity.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "The authors highlight the original issue—finding a workable sparsity-fidelity trade-off—and thoroughly address it using multiple metrics. Their design effectively analyzes how MSAE balances reconstruction precision against high sparsity. Moreover, ablation studies illustrate the hierarchical advantage of MSAE over single-threshold SAEs.", "relation_to_broader_scientific_literature": "This work stands at the intersection of mechanistic interpretability and concept-based explanations for large vision-language models. It also addresses limitations in prior SAEs, bridging techniques like L1-based ReLU and fixed TopK with a flexible hierarchical approach.", "essential_references_not_discussed": "There appear to be no critical references missing from the paper's discussion.", "other_strengths_and_weaknesses": "**Strengths**:\n- The paper systematically evaluates MSAE under multiple metrics, convincingly showing that MSAE attains a better sparsity-fidelity trade-off than existing approaches.\n- Through experiments in Section 5, the authors illustrate how MSAE enables meaningful interpretation of CLIP, including concept-based similarity matching and downstream bias analysis, demonstrating its real-world applicability.\n\n**Weaknesses**:\n- While Section 4 demonstrates the advantages of MSAE over standard SAEs, Section 5 primarily focuses on MSAE’s applications—showing concept extraction and bias analysis with CLIP. It would have been even stronger if the paper made explicit how previous SAEs would fail or underperform in these same tasks.", "other_comments_or_suggestions": "It might help to clarify in Section 5 exactly where and why alternative SAEs (e.g., single-threshold TopK) fall short for concept-based similarity or bias detection. This would further underscore MSAE’s value for real interpretability applications.", "questions_for_authors": "* In Section 5.3, the authors train a single-layer classifier on CLIP embeddings to identify bias for the CelebA experiment. Couldn’t the text encoder in CLIP itself be used for classification without additional training? Why did you decide to train a new classifier for this bias detection step?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NgWraHFHGa", "reviewer": "Reviewer_T2kP", "summary": "The authors introduce Matryoshka SAE (MSAE), a novel Sparse Autoencoder (SAE) architecture that simultaneously learns hierarchical representations at multiple granularities. This is achieved by applying the topK operation multiple times while incrementally increasing the number of considered neurons. The proposed architecture exhibits an improved balance between reconstruction accuracy and sparsity compared to traditional SAEs (topK and ReLU SAEs). The authors apply MSAE to interpret CLIP embeddings and extract semantic concepts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We gratefully thank the reviewer for a thorough and very insightful review of our work.** Due to space limits (5000 characters), we concisely respond to each of the major points raised. We share tables and figures with results from the requested analyses in anonymous cloud storage at https://drive.google.com/drive/folders/11OOSpexmU5ul8nBhJqpaf65Dh_eqBlZY \n\nA. **Reconstruction/sparsity tradeoff comparison.** (Mentioned in: Methods & Eval. 1, Exp. Designs 1)\n> The authors are comparing the different trade-off for topK and the ReLU SAE with the MSAE, by sampling only 3 points (i.e. different lambda of K). [...] I would urge the authors to rerun their comparison by systematically considering more lambda or K.\n\nOur initial experiments indeed span 4 different lambda and K hyperparameter values, as listed in Appendix B.1. We show the majority of results for 3 hyperparameter values due to observing low performance in the fourth one (lambda=0.01 & K=32, respectively). Following related work (Gao et al., 2024), we excluded the more extreme values of hyperparameters (e.g. K=512) since these lead to subpar results. As the reviewer suggested, **we now added the comparison for lambda=0.01 and K={32,512} to Tables 6 & 7** (see cloud link). We observe that lambda=0.01 gives worse results than MSAE, while K=512 collapses during training (FVU=0.3, CKNNA=0.07).\n\nB. **Stability of SAE-derived concepts across training runs.** (Methods & Eval. 2, Exp. Designs 4, Weakness 2, Other Comments 1, Question 1) \n> [...] Such a property strongly affects the reproducibility of the SAE and impairs their use as a widespread interpretability tool (see [1]). [...] But instability is just one example, there are plenty of interesting other metrics that would give a more complete comparison of the SAEs (see table 1, page 8 of [2], to have a non-exhaustive list of interesting metrics to evaluate SAEs). [1] \"Sparse autoencoders trained on the same data learn different features.\" preprint arXiv:2501.16615 (2025). [2] \"Archetypal SAE [...]\" preprint arXiv:2502.12892 (2025). Could you include metrics or analyses to quantify the stability of the concepts learned by MSAE across different random initialization?\n\nThank you for highlighting this work. **We politely note that neither of the mentioned papers was publicly available at the time of our paper’s submission.** The [2] preprint appeared over two weeks (18 Feb 2025) after the ICML submission deadline; similarly, the [1] preprint appeared a day before (29 Jan 2025). Not relating to literature that was unavailable at the time of submission should, in our opinion, neither be considered a weakness of our work nor affect the review score. Following the reviewer’s recommendation, **we now included an analysis regarding the stability of SAEs [1] for CLIP in the new Table 18** (see cloud link). We envision that applying the archetypal framework [2] to our proposed architecture would improve the stability of MSAE.\n\nC. **Comparison with JumpReLU.** (Exp. Designs 2, Weakness 1)\n> It would have been interesting to compare the MSAE to the jump ReLU SAE [1], as it is known to have a good reconstruction/sparsity tradeoff. [1] \"Jumping ahead: Improving reconstruction fidelity with JumpReLU sparse autoencoders.\" preprint arXiv:2407.14435 (2024).\n\n**Note that the JumpReLU preprint has no official code implementation available.** During our initial work, we were unable to reproduce its results. We now used the unofficial implementation shared in the SAELens software to compare with MSAE (ImageNet-1k, CLIP ViT-L/14). In our preliminary experiments, the results of JumpReLU are the same as those of ReLU. Note that a similar issue with reproducibility has been publicly pointed out in “Interpretability evals case study” (Transformer Circuits Thread, Aug 2024). **Our result is visible in the updated Tables 6 & 7** (see cloud link).\n\nD. **Can you elaborate on why UW outperforms RW on all tested semantic preservation metrics, and discuss whether sparsity/reconstruction is indeed the best criterion for concept extraction performance?** (Comment 2, Question 2)\n\nIn brief, RW is trained with an alpha parameter that values sparser representations more highly during training than UW, where alpha does not affect the loss. RW models learn to benefit from a sparser reconstruction, sacrificing optimal reconstruction in favor of sparsity, as indicated by lower FVU in UW. In our experiments, CKNNA and LP semantic metrics are correlated with reconstruction, while the quantity of valid concept neurons (Table 3) is correlated with sparsity.\n\nE. **Why were alternative concept extraction methods (e.g., Convex-NMF) not included in your comparative analysis?** (Exp. Designs 3, Q. 3)\n\nThank you for bringing this work to our attention; we view such a comparison as a natural future work direction. We primarily focused on evaluating various SAE architectures for interpreting CLIP using multiple quantitative metrics; beyond LLM applications.", "claims_and_evidence": "I have identified two main claims in this article : \n1) MSAE achieves superior trade-offs between reconstruction and sparsity compared to traditional SAEs. This claim is backed by empirical results showing a better trade-off for MSAE against topK and ReLU SAEs.\n\n2) MSAE effectively captures semantic concepts from CLIP embeddings. The authors propose various quantitative and qualitative analyses (concept naming, similarity search, bias validation).", "methods_and_evaluation_criteria": "1 ) Concerning the reconstruction/sparsity tradeoff comparison. The authors are comparing the different trade-off for topK and the ReLU SAE with the MSAE, by sampling only 3 points (i.e. different lambda of K). So few points seem to be not enough to make strong evidence for the first claim. I would urge the authors to rerun their comparison by systematically considering more lambda or K\n\n2) Using only the reconstruction/sparsity tradeoff to compare various SAEs does not provide a complete view of the SAE performance. For example, articles have shown that SAE suffers from unstable concepts (i.e. 2 SAE trained with a different seed lead to different concepts). Such a property strongly affects the reproducibility of the SAE and impairs their use as a widespread interpretability tool (see [1]). So metrics quantifying the instability would also help to better evaluate the different SAE. But instability is just one example, there are plenty of interesting other metrics that would give a more complete comparison of the SAEs (see table 1, page 8 of [2], to have a non-exhaustive list of interesting metrics to evaluate SAEs).\n\n[1] Paulo, Gonçalo, and Nora Belrose. \"Sparse Autoencoders Trained on the Same Data Learn Different Features.\" arXiv preprint arXiv:2501.16615 (2025).\n[2] Fel, Thomas, et al. \"Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models.\" arXiv preprint arXiv:2502.12892 (2025).", "theoretical_claims": "There is no theoretical claims (this is an experimental article).", "experimental_designs_or_analyses": "Here are the weaknesses I found in the experiments :\n1) Not enough data points (on topK and ReLU SAE) to strongly conclude that the MSAE exhibits a better reconstruction/sparsity than other SAEs (already mentioned before)\n2) It would have been interesting to compare the MSAE to the jump ReLU SAE [1], as it is known to have a good reconstruction/sparsity tradeoff\n3) Comparison with other dictionary learning methods (i.e. not SAE) that are known to work well for extracting meaningful concepts (NMF, Sparse NMF...) would strengthen the comparison. This paper is oriented toward a specific task (i.e. concept extraction), so it would be great to compare the proposed algorithms with more concept extraction methods (and SAE is far from being the only good method to extract concepts).\n4) As already mentioned before, quantifying the performance of SAEs based only on the reconstruction/sparsity tradeoff is not enough. It would be interesting to include additional metrics.\n\n\n[1] Rajamanoharan, Senthooran, et al. \"Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders.\" arXiv preprint arXiv:2407.14435 (2024).", "relation_to_broader_scientific_literature": "SAE approaches are well situated within existing literature; however, alternative concept extraction methods such as Convex-NMF, semi-NMF, Sparse PCA, ICA, SVD, and KMeans, known to perform well on similar tasks, are notably absent from comparisons.", "essential_references_not_discussed": "I did not find any important reference missing", "other_strengths_and_weaknesses": "Strengths:\n* Clear and well-justified approach.\n\nWeaknesses:\n\n* Limited comparative scope, omitting modern SAEs (e.g., JumpReLU) and alternative concept extraction methods.\n* Lack of consideration for stability of SAE-derived concepts across training runs, a critical aspect for interpretability.", "other_comments_or_suggestions": "* It would be valuable to discuss or quantify the stability of concepts learned by MSAE, as stability significantly affects interpretability and practical utility in concept extraction tasks.\n* Discuss more explicitly why the UW variant consistently outperforms RW on semantic metrics despite having lower sparsity, questioning the suitability of sparsity/reconstruction as the primary evaluation criterion.", "questions_for_authors": "* Could you include metrics or analyses to quantify the stability of the concepts learned by MSAE across different random initializations?\n\n* Can you elaborate on why UW outperforms RW on all tested semantic preservation metrics, and discuss whether sparsity/reconstruction is indeed the best criterion for concept extraction performance?\n\n* Why were alternative concept extraction methods (e.g., Convex-NMF, Sparse PCA, ICA) not included in your comparative analysis?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Matryoshka Sparse Autoencoder (MSAE), a novel architecture that addresses the fundamental trade-off between reconstruction quality and sparsity in sparse autoencoders when interpreting vision-language models like CLIP. By learning hierarchical representations at multiple levels of granularity simultaneously, MSAE achieves superior performance compared to traditional approaches such as ReLU and TopK sparse autoencoders. The reviewers acknowledged several strengths of the paper, including the clear methodology and strong empirical results showing MSAE's ability to establish a new state-of-the-art Pareto frontier between reconstruction quality and sparsity for CLIP. The practical applications demonstrating MSAE's utility as an interpretability tool for extracting semantic concepts and performing concept-based similarity search and bias analysis were also highlighted positively. Some concerns were raised regarding limited comparative analysis with more recent SAE variants and alternative concept extraction methods, as well as insufficient evaluation metrics beyond reconstruction/sparsity trade-offs. The authors addressed these concerns in their rebuttal by conducting additional experiments comparing with JumpReLU SAE, adding stability analysis, and clarifying their contribution of two new evaluation metrics. Given the novel approach to interpreting multimodal representations, the comprehensive experiments, and the demonstrated practical applications for understanding CLIP, I recommend accepting this paper for publication at ICML 2025.
Decision: Accept (poster)