Learning multimodal dictionary decompositions with group-sparse autoencoders
Authors: Chiraag Kaushik
Davis Barch
Andrea Fanelli
Venue: ICLR 2026 Poster (2026)
Keywords: sparse autoencoders
dictionary learning
multimodal representation learning
group sparsity
interpretability
Primary area: interpretability and explainable AI

Abstract: The Linear Representation Hypothesis asserts that the embeddings learned by neural networks can be understood as linear combinations of features corresponding to high-level concepts. Based on this ansatz, sparse autoencoders (SAEs) have recently become a popular method for decomposing embeddings into a sparse combination of linear directions, which have been shown empirically to often correspond to human-interpretable semantics. However, recent attempts to apply SAEs to multimodal embedding spaces (such as the popular CLIP embeddings for image/text data) have found that SAEs often learn ``split dictionaries,'' where most of the learned sparse features are essentially unimodal, active only for data of a single modality. In this work, we study how to effectively adapt SAEs for the setting of multimodal embeddings while ensuring multimodal alignment. We first argue that the existence of a split dictionary decomposition on an aligned embedding space implies the existence of a non-split dictionary with improved modality alignment. Then, we propose a new SAE-based approach to multimodal embedding decomposition using cross-modal random masking and group-sparse regularization. We apply our method to popular embeddings for image/text (CLIP) and audio/text (CLAP) data and show that, compared to standard SAEs, our approach learns a more multimodal dictionary while reducing the number of dead neurons and improving feature semanticity. We finally demonstrate how this improvement in alignment of concepts between modalities can enable improvements in the interpretability and control of cross-modal tasks.

Published as a conference paper at ICLR 2026
DECOMPOSING MULTIMODAL EMBEDDING SPACES WITH GROUP-SPARSE AUTOENCODERS
Chiraag Kaushik
Davis Barch
School Electrical and Computer Engineering Georgia Institute of Technology
Dolby Laboratories
San Francisco, CA 94103
Atlanta, GA 30308, USA
davis is.is... www.barrrhobbbbb..o.oo.o.ob .com com
ckaushik7@gatess
Andrea Fanelli
Dolby Laboratories San Francisco, CA 94103
andrea fane 11 bydobyy..o....oo com
ABSTRACT
The Linear Representation Hypothesis asserts that the embeddings learned by neu- ral networks can be understood as linear combinations of features corresponding high-level concepts. Based this ansatz, sparse autoencoders (SAEs) have recently become a popular method for decomposing embeddings into a sparse combination of linear directions, which have been shown empirically often correspond to human-interpretable semantics. However, recent attempts to apply SAEs to multimodal embedding spaces (such the popular CLIP embeddings for image/text data) have found that SAES often learn "split dictionaries." where most of the learned sparse features are essentially unimodal, active only for data of single modality. In this work, we study how to effectively adapt SAEs for the setting of multimodal embeddings while ensuring multimodal alignment. We first argue that the existence of split dictionary decomposition embedding space implies the existence of a non-split dictionary with improved SAE-based approach to multimodal
aligned
modality alignment. Then, we propose embedding decomposition using cross-modal random masking and group-spars regularization. We apply our method popular embeddings for image/text (CLIP) and audio/text (CLAP) data and show that, compared to standard SAES, our ap- proach learns more multimodal dictionary while reducing the number of dead neurons and improving feature semanticity. We finally demonstrate how this im- provement alignment of concepts between modalities can enable improvements in the interpretability and control of cross-modal tasks.
INTRODUCTION
Multimodal representation learning models like CLIP (Radford et al. 2021). SigLIP ZZhii et al., 2023). and CLAP (Elizalde al. 2023 Wu et al. 2023) aim to learn shared latent representa- tion for various types of input data, such as images, text, and audio. The goal of these multimodal encoders is learn embeddings which are in some sense (e.g., cosine similarity) "similar" one another when the input data contains shared semantic information, regardless of the data's 's modality. The ability of such models to learn expressive multimodal data representations has been extensively demonstrated by their excellent performance cross-modal tasks like zero-shot classification (Rad-) ford et al. 202I), retrieval (Elizalde et al., 2023) Wu et al. 2023), and generation (Ramesh et al, 2022). However, despite the promising empirical performance of these models, the precise way in which they encode semantic information not well-understood.
Motivated primarily by the task of interpreting the behavior of large language models, sparse autoen- coders (SAEs) have recently become popular approach for understanding how human-interpreeabll concepts encoded in the embeddings of modern neural networks. Building on classical ideas

---

Published as a conference paper at ICLR 2026
Modality Split Dictionary
Multimodal Dictionary
Image only concept
only concept
Shared/mmutimodii modil
"A close-up of red panda"
No shared concepts
Overlapping support
Figure 1: Split versus multimodal dictionaries: (Left) Standard SAEs trained on aligned em- beddings from different modalities (like CLIP) tend learn dictionary vectors ("concepts") which activate only for input embeddings of one modality. (Right) We develop an SAE-based approach for learning dictionaries where multimodal embeddings containing the same semantic information have similar sparse dictionary decompositions.
from dictionary learning and sparse coding, SAEs are unsupervised models that aim to decompose neural network embeddings sparse linear combinations of large (overcomplete) set of dictio- nary vectors. It has been empirically observed a variety domains that these dictionary elements tend correspond more uunrasstanndtt concepts, allowing for improvements in inter- pretability, steering, and control of models through linear algebraic manipulation of embeddings (Cunningham et al. [2023] Gao al. 2024 Pach al., (025). The so-called "Lincar Representation Hypothesis" proposes the existence of such a linear semantic decomposition (Elhage et al.) 2022 Park et al. 2023), and this notion has been empirically supported for various concepts such as query refusal in language models (Arditi et al. 2024) and the presence of certain facial characteristics in
images (Voynov Babenko 2020).
Recent attempts to apply sparse autoencoders to the aligned embedding space of multimodal models like CLIP have obtained mixed results. On the one hand, the neurons learned by SAFs trained on CLIP do demonstrate improved monosemanticity and often correspond to human-interprettalle concepts (Yan et al., 2025 Pach et al.) 2025).. However, many works identify a "split-dictionary" phenomenon, where many of the sparse features only ever active for inputs coming from single modality (Papadimitriou et al. 2025) Costa et al. 2025). We depict this pictorially in Figure [ In other words, SAEs learn map embeddings which are aligned between modalities sparse features which may be more semantic, but which have very poor modality alignment. This phenomenon limits the potential of SAEs in vrossmmdaa tasks like retrieval generation, where, for example, manipulation of text emheddings based on a certain "concept" should ideally be able to influence a visual or audio output.
In this paper, we focus on answering the question of how to mitigate the tradeoff between monose- manticity and modality alignment in SAEs trained on multimodal embeddings. We investigate whether it is possible to learn multimodal linear concept dictionaries and we design a model which mitigates the implicit bias towards dictiooniisss observed in regular SAEs. We make the follow- ing contributions:
1. New metrics for multimodal SAEs. Motivated by the image-specific score proposed in (Pach et al., 2025), we formally define paired-modality monosemanticity score which quantifies the semanticity of neuron with respect to any two modalities, taking value of 0 the neuron never co-activates for inputs of both modalities and with higher value if the co-activating inputs contain similar semantic information. Our metrics generalize existing methods for measuring semanticity at the neuron-specific level to the setting where SAE neurons expected to co-activate for data from diverse modalities.
2

---

Published as a conference paper at ICLR 2026
2. Argument for the existence of multimodal dictionaries. Under realistic technical as- sumptions, we prove that the existence of a split dictionary on an aligned embedding space implies the existence of non-split dictionary with strictly improved modality alignment. This suggests that the poor modality alignment in SAES is not exclusively due to limitations of the LRH, but rather is also product of the implicit bias of SAEs trained exclusively on reconstruction loss.
3. Group-sparse autoencoder architecture with improved multimodality. We propose the use of group-sparse loss function acting on paired samples during the training of SAEs, along with form of cross-modal random masking, which together encourage an SAE to learn multimodal dictionary vectors. We train the proposed model on image/text (CLIP) and audio/text (CLAP) data, and evaluate our approach using a variety of metrics. Our re- sults show improvements in the number of multimodal concepts, reduction in the number of dead neurons, improvements in zero-shot cross- moosl tasks, and improved capabilities for interpreting multimodal models (compared regular SAEs). We also note that, to our knowled leowe,ege, joint embedding spaces for audio/text data have not been analyzed using SAFs in prior literature.
2 RELATED WORK
Dictionary learning and sparse coding The decomposition of a data sample I E Rd as a sparse
linear combination of learned dictionary elements W E Rd .p >> d. is a well-studied problem in many applications, such as image processing (Olshausen et al. 2009). medical imaging (Mailhé (ct al., 2009), and audio processing (Plumbley et al. 2009). Several popular algorithms like the Method of Optimal Directions (MOD) (Engan et al. 1999), the K-SVD (Abaron et al. (006), and unrolled ISTA (Gregor & LeCun 2010) have been proposed for the joint learning of sparse codes and the dictionary. A few works have considered modifications of these algorithms with additional structural constraints on the sparse codes, typically by adding a parrity-promottng norm penalty such as the group norm 12211 norm (Bengio al. 2009; Li et al. 2012). Compared to these works, we also apply a group-sparsity penalty to the loss function of a sparse dictionary learning method (the SAE), but our setting, the groups are given by pairs of data from different modalities containing the same concept. We additionally incorporate a type of per-group random masking to promote shared sparsity structure within each group, and we show that the combination of these two regularization techniques is effective in the context of multimodal embedding spaces.
SAES and neural network interpretability Sparse decompositions of neural network embed- dings have recently been studied in many works in the field of neural network interpretability (EI- hage et al. 2022, Cunningham et al., 2023 Rajamanoharan et al., 2024). Many of these works leverage (variants of) sparse autoencoders (SAEs), which are simple neural network-based models for sparse dictionary learning. In several of these works, the dictionary elements learned by sparse autoencoders (SAEs) trained embeddings from language models have been empirically shown to often correspond to interpretable high-level concepts (Bussmann et al. 2024 Gao et al.) [0024). Inspired by the efficacy of SAES in language models, recent works have also proposed the of SAES in other domains, such as for vision models (Thasarathan et al. 2025 Fel et al. 2025) and neural data (Freeman et al. 2025).
Decomposition of multimodal embeddings While most recent works on SABS have studied in- terpretability of language or vision models, few recent works also have applied SABS to modality- aligned representations, such as those from models like CLIP, SigL II, and DeCLIP (Papadimitriou al. 2025 Pach al. 2025 Yan et al. 2025 Zaigrajew et al. 2025). Most of these works train SAES to reconstruct embeddings originating from visual and text data, using the same architectures as are prevalent the language model setting. Several works that train in this way observe that many SAE neurons activate only for inputs of one modality (Costa et al. 2025 Bhalla et al. 2024b) Papadimitriou et al., 2025) Yan et al. 2025). Previous works have addressed this by explicitly learn- ing transformations from one modality to the other or by identifying co-activating pairs of dictionary elements (Papadimitriou et al. 2025). Unlike these approaches, we seek to identify a first princcpris approach to learning multimodal dictionary using SAEs, overcoming the bias of vanilla SAEs towards modality -split dictionary elements. The authors of (Costa et al. 2025) also cite this discrep- ancy and find that an SAE variant based on the classical matched- pursuit algorithm can help increase
3

---

Published as a conference paper at ICLR 2026
the number of multimodal concepts, but their focus is primarily on learning hierarchical concepts, rather than explicitly mitigating the split-dictionary phenomenon.
MODALITY ALIGNMENT AND SEMANTICITY IN SAES
In this section, we provide overview of the sparse autoencoder (SAE) model, formally define modality-split dictionaries, and describe a proposed metric for quantifying the monosemanticity of multimodal SAEs.
3.1 SPARSE AUTOENCODERS AND SPLIT DICTIONARY PHENOMENON
We consider the general setting where a multimodal encoder maps data from multiple modalities into a joint representation space X' C IRd Motivated by the hypothesis that neural network representa- tions consist of linear superpositions of interpretable "concepts" (Elhage et al. 2022). dictionary learning aims to learn an overcomplete dictionary W E RdxP such that representations x E X can be decomposed R W z, for some sparse vector E R2.
Sparse autoencoders (SAEs) are one popular method for learning this type dictionary decom- position neural network embeddings (Cunningham et al. 2023). Given an input æ E R SAEs take the following form:
z [I(Weno bo) + b)
Φ W + bp-
Here, bn E Idd is a bias applied prior to encoding/decodiig, Wene E Rexd is the encoder matrix, b is the encoder bias parameter, and W E RRdx is the decoder matrix. The projection function II: R² - Iº is chosen to promote sparsity in the latent vector z. In this paper. we focus our attention the TopK function (Gao et al., 2024). which keeps only the K largest elements of the input and sets the remaining elements to O. However, we note that our proposed methodology can in principle be applied to other common SAE variants. Other common choices of IT in recent literature include the ReLU (Cunningham et al. 2023), BatchTopK (Bussmann et al. 2024), and JumpReLU (Rajamanoharan et al. (0224) functions, and these variants are often trained with additional 4 reg- ularization to promote sparsity of z. In the following, we refer to the decoder matrix Wdec as the dictionary matrix and its columns as the dictionary vectors, or concept vectors. We will refer to as the SAE latent vector or sparse code, and its coordinates neurons of the SAE. The SAE parameters are learned by optimizing the reconstruction error || - = over all training samples x in the training set D.
Recent attempts to train classical SAEs on multimodal embeddings from models like CLIP and SigLIP have found that the learned dictionary elements tend to correspond primarily to only one of the input modalities (Papadimitriou et al. 2025; Costa et al. (025). other words, many of the features in the SAE's latent space tend to only be active for inputs corresponding to single modality. We define this property of a dictionary formally below:
Definition 1 (Modality-split dictionary). Let W Rdxp with p d be dictionary matrix that admits a K- K-sparse decomposition of any x E X. Then, W is said to be modality-spli with respect X if. for any two embeddings (1)), .2(2) E X corresponding data from different modalities, su)p(z n supp(z(?))
where (1) = W (1) and (2) (2 are any K K-sparse decompositions of (1 and a! in the dictionary.
An important consequence of this definition is that paired data samples of different modalities (such as an image and a textual deseription of the same image's 's content) have sparse codes with no over- lapping support, even though the original embedding space is aligned between modalities. In other words, classical SAEs trained on popular aligned multimodal embedding spaces have poor modal- ity alignment in the SAE's latent space. A crucial application of SAE embedding decompositions is the steering or manipulation of SAE latents for interpretability and control of downstream tasks (Bhalla et al., 2024b). In cross cross-modal applications. therefore, it is important for data from different modalities that contain similar semantic information to have aligned sparse codes. The remainder of the paper is dedicated understanding when and how this can be achieved in SAE-based models.
4

---

Published as a conference paper at ICLR 2026
3.2 MEASURING SEMANTICITY ACROSS MODALITIES
A key reason for recent interest in SAES is the observation that the learned dictionary elements (or equivalently, the neurons of the latent space) often seem correspond to interpretable concepts or ideas. Before proceeding to our main results, we first define an evaluation metric which we will use to jointly evaluate the semanticity and multimddliity of each coordinate of the SAE's latent space. Our metric generalizes the main idea behind the monosemanticity score proposed in (Pach et al.) 2025) to measure semanticity with respect to a pair of modalities. For each neuron and any pair of modalities (m, 11) (e.g., image and n text), we define the multimodal monosemanticity score (MMS) with respect to (m, n) as follows:
On an unseen validation dataset, find all samples from each modality with non-zero activa- tions for the given neuron. Store these activations as vectors (mm) M and a!n) R..
2. Let S IMMx be the matrix of cosine similarities of these samples under a separate multimodal encoder (i.e., a different encoder than the one used in learning the dictionary).
3. Compute the co-activation matrix A E RMXN, where Aij = :
(m) aj [(1 - 6mn), where
6mn = if m = n, else 0.
4. Normalize co-activations to sum to 1: A.j = Eig Ary Aij-
5. Compute M (M,,m, n) as the weighted sum of cosine similarities, with weights given by normalized co-activations:
MMS(m, n) = S.yS
1.
The high-level intuition for this metric is that neuron should be considered more monosemantic if it is active for inputs that are semantically similar to each other. In addition to capturing the semanticity of each SAE neuron, the case m n captures the multimodality of a concept. In particular, if a dictionary is fully modality-spiit (cf. Definition T. then there are no co-activating inputs and M MMS,m, n) trivially equals A positive M MMS n) scores indicates that there are test samples of different modalities that co-activate for given neuron and that concept can be considered more multimodal. In Section [ we will see that our proposed methodologies for multimodal SAEs increase the M S score significantly compared to classical SAEs.
Comparison to metrics proposed in the literature: Our metric most similar to the MS score in (Pach et al. 2025). but it also allows monosemanticity to be measured between two different modalities. Additionally, we normalize the co-activations to sum to 1, rather than using min-max normalization the activations themselves. This allows to interpret the metric more naturally as a weighted average of cosine similarities. We also note that our MMS metric measures a funda- mentally different quantity than the BridgeScore in (Papadimitriou assigns a value to each pair of neurons/concepts (answering the question of how often neuron i and neuron j are jointly active across paired samples), while our MMS score assigns value single neuron/concept (answering the question of how often a single neuron co-activates for seman- tically similar inputs of different modalities). Additionally, we measure semanticity by using cosine similarity as proxy, which obviates the need for additional paired data to compute the metric.
al. 2025). The BridgeScore
LEARNING MULTIMODAL CONCEPTS WITH GROUP-SPARSE AUTOENCODERS
In this section, we first argue that modality-split dictionary on an aligned embedding space can be improved to multimodal dictionary with similar reconstruction loss and strictly better multimodal alignment.
This result is detailed below in Theorem I and we defer the proof to Appendix A.I
Theorem 1. Consider a set of paired unit-vector embeddings corresponding data from different modalities (xx ( Suppose the following conditions are met:
1. Paired embeddings are aligned: There is a >> O such that (x,(), (0))) cfrr all E [n].
5

---

Published as a conference paper at ICLR 2026
2. Sparse decomposition in split dictionary: There exists modality split dictionary W R satisfying the following: for all v E {I1, V1.............................. Enn Vn there is K K-sparse vector By with non-negative entries such that W 2,,-
Then, there exists dictionary W of size p + n admitting (K + 1)-sparse decomposition of all 2m. embeddings, and where sparse codes of all pairs have strictly positive inner product.
Here, the first assumption follows naturally from the fact that we are considering dictionaries learned from aligned embedding spaces, where embeddings of paired samples containing similar semantic information should be positively correlated. We also note that the second condition can be relaxed in a straightforward way to the case where sparse decompositions under W hold only approximately, up to some small error adding O(e) error terms to the emor of the decompositions with respect to W. The theorem states that modality-split dictionary can always be improved to dictionary where sparse codes are more aligned between modalities. Given this result, we turn to the question of how induce more favorable implicit bias in SAEs training on multimodal embeddings.
To this end, we propose a new training paradigm that can be applied to a training set of paired multimodal samples. We first define the group-sparse loss that will use learn multimodal dictionaries. The group-sparse loss corresponding to two sparse codes z and u is given by
(z w) =
=
+w..
w
2,1
The use of the convex 12,1 norm to enforce structured sparsity in parameters has been studied extensively in the context problems like the group LASSO Yuaan Lin 2006) and multi-task learning (Argyriou et al. 2008). Intuitively, this loss function encourages coordinates of z and w to be jointly sparse, i.e., have the same support. Although we focus here on the case of two modalities, we note that the overall formulation naturally extends to an arbitrary number of modalities.
Given paired embeddings æ, y R (corresponding semantically similar data from different modalities), our model takes the form:
2, = Topp (ReLU(Weerr (wee(( bo) + b)),
Zy ÿ deccZy + bj.
enc (y- - b1) + b)),
Zx + bo:
Here, by and b, are learnable pre-coding bias terms and the remaining parameters are shared across modalities. Similar the standard SAE formulation, our approach trains SAE to reconstruct embeddings from different data modalities, but we also add the group-sparse loss term specified above with regularization hyperparameter À > 0, to encourage zy and zy to have shared sparsity structure:
== - z
قl (Z, zy).
Additionally, to mitigate the occurrence of dead neurons (dictionary elements which never activate for any imputs) and to further encourage multimodality of concepts, also introduce moosl random masking during training. Specifically, we apply a shared random mask with probability during the computation of 2, and my prior to the TopK operation. This forces the TopK operation to choose from the same subset of coordinates at each step prior to decoding. The full training approach is summarized in Figure
RESULTS
To evaluate our approach, in this section we compare three main types of SAE training schemes:
SAE: Standard TopK SAE (Gao et al. 2024) trained on reconstruction loss of multimodal embeddings.
Group-sparse autoencoder trained on both reconstruction and group-sparsity losses, but without random masking.
GSAE:
MGSAE:
Masked, croupespprse autoencoder trained using our complete training pipeline from Figure

---

Published as a conference paper at ICLR 2026
SAE
Pretrained ercoddar
SAF encoder
TopK
decode (linear]
Random mack
Pretrained encoder
SAE encoder
TopK
SAE decoder (liner))
training samples
Nigned embeddings
Reconstructed embeddings
Ereeonetruation Egrrappsperitty
Figure 2: Masked group-sparse autoencoder for multimodal concept extraction: Our proposed approach is trained on paired data. Embeddings are encoded higher-dimensional vector, masked (with the same mask for each modality), sparsified using TopK, and decoded with a linear layer.
We train each these variants on different multimodal embedding spaces: (1) CLIP ViT-B/16 (Radford et al.) (20TT) embeddings obtained from the CC3M dataset of image/text caption pairs (Sharma al. 2018) and (2) LAION CLAP embeddings (Wu al.) 2023) obtained from the mendoMaxCapp dataset of music/text pairs (Roy et al. 2025). In the latter setting, we use CLAP checkpoint specifically finetuned on music data. To our knowledge, ours the first work to apply SAEs to the joint embedding space of music/text samples and measure semanticity of the learned dictionary. each case we fix the sparsity level = 32 and the dictionary size as p = 16d, where d 512 is the dimension of the original CLIP/CLAP embedding. We train for 25,000 steps in the image/text setting and 10,000 steps in the music/text setting (chosen to ensure convergence in the fraction of explained variance for both modalities). All embeddings unit norm prior to training. For further experimental details, see Appendix A.2 In Section 5.2 we also provide additional comparisons the BatchTopK (Bussmann al., 2024 and Matryoshka (Zaigrajew et al. 2025) SAE variants. which have both been applied to CLIP embeddings in pre- vious works (Papadimitriou et al., 2025 Pach et al. 2025). While we focus on CLIP and CLAP embeddings in the main paper for clarity of exposition, we also provide additional experiments on the SIGLIP2 (Tschannen et al. 2025)and AIMv2 (Fini al. (024) text/image encoders in Appendix A.3 to demonstrate that our methods generalize other popular multimodal embedding spaces.
normalized to have
5.1 DEAD NEURONS AND MONOSEMANTICITY
We first evaluate our approach by considering the prevalence of dead neurons in modality- specific way. In other words, given unseen validation set of paired compute how many neurons are active for least one sample from a given modality. For the im- age/text case, we use 10,000 pairs from the validation set of CC3M, and for the music/text case, 10,000 pairs from the MusicBench dataset (Melechovsky al.
Middd
9000
Model SAC
6000
9000
9000
samples,
2000
1000
Inagenityy
Text only
Rith
Mask onby
Toclenl Ankistnginii
Brzh
Notre
Figure 3: Number of neurons activating for each individual modality, both modalities, and neither modality. Left: models trained CLIP embeddings, validation on CC3M val. set. Right: models trained on CLAP embeddings, validation on MusicBench.
2024
We visualize the number of dead neurons in Figure
We

---

Published as a conference paper at ICLR 2026
that, compared to the standard (TopK) SAE, the GSAE and MGSAE both have significant increase in the number of neurons that activate for both modalities and a reduction in the number of neurons that activate for neither modality. Overall, the MGSAE consistently obtains the highest number of multimodal activations and smallest number of dead neurons.
On the same validation sets, we also compute the multimodal monosemanticity score we de- fined in Section 3.2 between all pairs of modalities: Mimage image), M MM text), M MMS(imaage teat) (and likewise for audio/text). We recall here that high MMS score for given neuron indicates that inputs which activate that neuron are similar to each other (and hence that neuron encodes a coherent concept). To compute the similarity matrix between validation sam- ples, we use the CLIP RN-50 encoder for CC3M and Microsoft CLAP (Elizalde et al.) (202)) for MusicBench, Our results, plotted Figure 4 confirm the notion that standard SAES have a large proportion of neurons that do not activate across different modalities (and large number of dead neurons more generally). By contrast, we see that the GSAE and, to an even greater extent, the MGSAE, both have much larger proportion of neurons with high monosemanticity scores relative to the baseline score achieved by the original dense CLIP/CLAP embeddings. Our results indicate that the group-sparse variants are more semantic and more multimodal than the standard SAE in these settings.
mage Imagei-CUP CUP
MW SWStotteeett-co CLIP
MWSimageteet www.geeteettoooo CUP
mi
0119
Nor Comermaaatilyy Spore
344
07
Normallzed Nouron Indox
Normal Nouron Inddx
Normakaed Nouron Indiox
MWSmmusi music
MMS[jnntaan)
MM6jjmsicteen) -CAP
HIGHL
HGGAL
05
MISAL
Nonoommanccty
Normakized
Normallzed
Figure 4: Multimodal monosemanticity (MMS) scores for each feature, arranged in descending order (higher is better). Neuron index (on the horizontal axis) is normalized due the difference in dimensionality between the "No SAE" case (d 512) and the SAE variants (d 16 512). Top row: MMS scores for models trained on CLIP embeddings for image/text data. Bottom row: MMS scores for models trained on CLAP embeddings for music/text data.
5.2 ZERO- SHOT PERFORMANCE ON CROSS-MODAL TASKS
Another way to evaluate the ability of SAES to learn multimodal dictionaries is to measure zero-shot performance of sparse codes on cross-modal tasks. We note that the purpose of these evaluations is not to claim that sparse codes themselves constitute a good embedding which should be used zero- shot tasks; instead, we use these metries as proxy to measure how much modality alignment preserved when extracting sparse features using SAE-based models. Indeed, for any dictionary with a small number of multimodal concepts, the sparse codes corresponding to paired embeddings from different modalities have low cosine similarity, so we expect zero-shot tasks to perform quite poorly (in the extreme case of a modality-spli dictionary, the cosine similarity is always zero and zero-shot tasks are impossible). By contrast, if embeddings are decomposed into concepts which depend on semantic information in a modality-agnostic way, then we would expect zero-shot performance to be retained to larger extent.
8

---

Published as a conference paper at ICLR 2026
Table I: Performance comparison on zero-shot image/text cross-modal tasks between SAE (3 types), GSAE, and MGSAE. Original performance on the dense CLIP embeddings is provided for ref- erence. Reported numbers are classification accuracy. Numbers with an asterisk indicate results reported from the original authors of the model.
Model
ZS image/Text Tasks CIFAR-100
CIFAR-10
ImageNet
SAE Topp (Gao al. 2024)
0.657
0.418
0.303
BatchTopK SAE (Bussmann et al.
2024
0.657
0.277
0.178
Matryoshka SAE Zaigrajew al.
2025
0.587
0.166
0.185
GSAE (ours)
0.808
0.526
0.354
MGSAE (ours)
0.842
0.554
0.373
CLIP ViT B/16
0.916*
0.687*
0.686*
Table 2: Performance comparison zero-shot audio/text cross-modal tasks between SAE, GSAE, and MGSAF. Original performance on the dense CLAP embeddings is provided for reference. Re- ported numbers are classification accuracy mean reciprocal rank (for FMACaps). Numbers with an asterisk indicate results reported from the original authors of the model.
Model
7S Audio/Text Tasks
GTZAN Genres
NSynth Instruments
FMACaps retrieval
SAE TopK (Gao et al. 2024)
0.376
0.265
0.023
GSAE (ours)
0.705
0.303
0.050
MGSAE (ours)
0.672
0.354
0.061
LAION CLAP
0.710*
0.339
0.075
In Tables Tande 2] we state results for zero-shot performance variety of tasks for each multimodal setting. For the models trained on the CLIP embedding space. we consider zero-shot classification on the CIFAR- 10 (Krizhevsky et al.] 2009), CIFAR-100 (Krizhevsky al., 2209). and ImageNet (Deng et al. 2009) datasets. In all cases, the standard TopK SAE (together with recently proposed variants like the BatchTopK and Matryoshka SAF) has a drastic performance reduction on zero-shot tasks compared to the original CLIP embeddings, while the group-sparse variants demonstrate marked improvement over the SAE of almost 20 percent for CIFAR-10, 10, 15 percent for CIFAR-100, and 7 percent for ImageNet.
For models trained on the CLAP embedding space, we consider zero-shot classification on the GTZAN genres (Tzanetakis & Cook 2002) and NSynth instruments (Engel et al. 2017) datasets and zero-shot text-to-music retrieval on the FMACaps dataset (Melechovsky et al. 2024) (with performance measured as reciprocal rank). For these tasks, the group-sparse variants also sig- nificantly outperform standard SAFS, in some cases performing close to twice as well. Interestingly, find that in the music/text setting, the sparse codes are actually quite competitive with (and on one task even better than) the original dense CLAP embeddings, despite being 16 times more sparse and more semantic, as measured by the metrics in the previous section.
5.3 CASE STUDY: CONCEPT NAMING AND INTERPRETING LINEAR PROBES
In the previous sections, we showed that group- sparse autoencoders can improve multimodal align- ment and semanticity the dictionaries learned by sparse autoencoders. Next, we show a simple application to illustrate the limitations of classical SAFs for interpretability studies in multimodal spaces and the advantages of the group-sparsity approach we propose. We consider binary classifi- cation on the CelebA dataset (Liu et al. 2015). We first train a simple linear classifier on the CLIP ViT-B/16 embeddings of the training set to predict the presence of blonde hair. Given this trained linear probe, denoted è, a natural application of SAEs is to identify which semantic properties of photo contribute most to it being classified as "blonde" (and to detect the presence of potential

---

Published as a conference paper at ICLR 2026
spurious features). To do this, the following procedure, where W is the learned dictionary matrix (with normalized columns):
1. Concept naming: We follow the approach used in multiple recent works for interpreting CLIP embeddings (Rao et al. 2024; Zaigrajew et al.) (025). In words, we use a large vocabulary corpus V of potential concept names and assign each dictionary vector the word or phrase in V with highest CLIIP cosine similarity (See Appendix A.2 for more details).
2. Compute O's coeflicients in the concept dictionary: The entries of a W 0 give the contribution of each concept to the prediction of the classifier. The concept names corresponding to the largest entries of a can be interpreted as the concepts with highest contribution to positive classification (in our example, "blonde").
As shown in Figure [ the above procedure applied with the MGSAE dictionary yields dominating cepts which are highly correlated with the true label being predicted ("blonde") and offers suggestion of spurious correlations with gender ("girl", "woman") "). This consistent with known results on that gender is spurious correlation for the blonde hair attribute in CelebA (Liu al., 2015). The SAE trained with- out group-sparsity, by contrast, is much less success- ful at identifying which concepts are important for this task, which we attribute to the fact that the success of the concept naming step relies crucially on having a multimodal dictionary. In Appendix A.5 provide an additional example of how multimodal dictionaries can be useful for the task of steering retrieval systems.
SAE:
MGSAE:
"beautiful blonde" "golden retriever" "cactus"
"beautiful blonde" "blond girl"
"golden retriever" brrtney spears" "blonde woman"
"polar bear"
"bleach"
"heron"
"beautiful blonde"
"Australian shepherd
"young blonde"
Figure 5: Leading concepts contributing to a classification of "blonde" on the CelebA dataset, extracted using standard SAE (left) and MGSAE (right).
CONCLUSION
Based on the observation that standard SAEs tend to learn many unimodal dictionary elements, we propose a new method for learning sparse dictionary decompositions of multimodal embedding spaces. Our method erucially makes connection between multimodality of the dictionary elements and group-sparsity of the sparse codes corresponding to paired data of different modalities. We en- courage this group-sparse property during training both through explicit regularization (inspired by classical techniques in sparse recovery) and cross-modal random masking. We show through a vari- cty of metrics that our results learn more semantic and aligned embeddings than standard SAEs, and we demonstrate that our method can be more effective for interpretability analysis and identifying spurious correlations, since improved modality alignment leads more accurate concept naming- Our results promote improved interpretability and control of multimodal representation spaces and lay the groundwork for further research on improving alignment in multimodal SABs. In partieu- employ have straightforward
lar, note that the group-sparse loss and the masking technique extensions to settings with more than two modalities. Additionally, our proposed methodology can also be adapted to settings where large amounts of unpaired data are also available. In such settings, the group-sparse loss can be applied only for paired samples, and reconstruction loss alone can be used for the remaining data samples. While we leave detailed investigation of this for future work, we believe that following this approach with even a small amount of paired data would provide improvements over the baseline SAE.
ACKNOWLEDGMENTS
Support for this project generously provided by Dolby Laboratories. We also thank Jacob Abernethy for compute assistance.
REFERENCES
Michal Aharon. Michael Elad, and Alfred Bruckstein. K-SVD: An algorithm for designing over- complete dictionaries for sparse representation. IEFE Transactions signal processing, 54(1
10

---

Published as a conference paper at ICLR 2026
4311-4322, 2006.
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. Advances in Neural Infor- mation Processing Systems, 37:1 37:136037-136088 2024.
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Convex multi-task feature learn- ing. Machine learning. 73(3):243-273 2008
Samy Bengio, Fernando Pereira, Yoram Singer, and Dennis Strelow. Group sparse coding. Advances in neural information processing systems, 22, 2009.
Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio Calmon, and Himabindu Lakkaraju. Inter- preting CLIP with sparse linear concept embeddings (SpLiCE), Advances in Neural Information Processing Systems, 37:84298-84328, 2024a.
Usha Bhalla, Suraj Srinivas, Asma Ghandeharioun, and Himabindu Lakkaraju. Towards unifying interpretability and control: Evaluation via intervention. arXiv preprint arXiv:2411.04430, 2024b.
Bart Bussmann, Patrick Leask, and Neel Nanda. BatchTopK sparse autoencoders. arXiv preprint arXiv:2412.06410. 2024.
Valérie Costa, Thomas Fel, Ekdeep Singh Lubana, Bahareh Tolooshams, and Demba Ba. From flat to hierarchical: Extracting sparse representations with matching pursuit. arXiv preprint arXiv:25006.03093, 2025.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.0800 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Icee, 2009.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dod Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposi- tion. arXiv preprint arXiv:2209. 200.::2200.0055..... 2022.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP: Learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Kjersti Engan, Sven Ole Aase, and Hakon Husoy, Method of optimal directions for frame design. In 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceed- ings. ICASSP99 (Cat. No. 99CH36258), volume 5, pp. 2443-2446. IEEE, 1999.
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi. Neural audio synthesis of musical notes with wavenet autoencoders, 2017.
Thomas Fel, Ekdeep Singh Lubana, Jacob S Prince, Matthew Kowal, Victor Boutin, Isabel Papadim- itriou, Binxu Wang, Martin Wattenberg, Demba Ba, and Talia Konkle. Archetypal SAE: Adap- tive and stable dictionary learning for concept extraction in large vision models. arXiv preprint arXiv:2502. 12892, 2025.
Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju. Victor Guilherme Turrisi da Costa, Louis Béthune, Zhe Gan, Alexander T Toshev, Marcin Eichner, Moin Nabi, Yinfei Yang, Joshua M. Susskind, and Alaaeldin El-Nouby. Mul- timodal autoregressive pre-training of large vision encoders, 2024. URL https://arxiv: bog/abs//00 /2.//241... 14402
Laurence Freeman, Philip Shamash, Vinam Arora, Caswell Barry. Tiago Branco, and Eva Dyer. Beyond black boxes: Enhancing interpretability of transformers trained on neural data. arXiv preprint arXiv:2506. 140/4, 2025.
11

---

Published as a conference paper at ICLR 2026
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04009 04093, 2024.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference on international conference on machine learning, pp. 2010.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Shutao Li, Haitao Yin, and Leyuan Fang. Group-sparse representation with dictionary learning for medical image denoising and fusion. IEEE Transactions on biomedical engineering, 59(1)) 2012.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll'a a and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, hts//405.0312, 2014. URL htLp:tt::/// httthttp///xiiii wwtpa.ogg....///////ooo
1405. 0312
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Boris Mailhé, Rémi Gribonval, Frédéric Bimbot, Mathieu Lemay, Pierre Vandergheynst, and J-M Vesin. Dictionary learning for the sparse modelling of atrial fibrillation in eog signals. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 465-468.8 IEEE, 2009.
Samuel Marks, Adam Karvonen, and Aaron Mueller. lletion.ry.leamiing https:////thub. 2024.
com/ com/ /ttp:/aaaaa//iiiiiinaa....a
Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, and Sou- janya Poria. Mustango: Toward controllable text-to-music generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (Volume 1: Long Papers), pp. 8286-8309, 2024.
Bruno Olshausen, Charles F Cadieu, and David Warland. Learning real and complex overcom- plcte representations from the statistics of natural images. In Wavelets XIII, volume 7446, pp. 236-246. SPIE, 2009.
Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata. Sparse autoencoders learn monosemantic features in vision-language models. arXiv preprint arXiv:2504.02821, 2025.
Isabel Papadimitriou, Huangyuan Su. Thomas Fel, Sham Kakade, and Stephanie Gil. In- terpreting the linear structure of vision-language model embedding spaces. arXiv preprint arXiv:2504.1 1695, 2025.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint vrXiv:2311.03655 2023.
Mark D Plumbley, Thomas Blumensath, Laurent Daudet, Rémi Gribonval, and Mike E Davies. Sparse representations in audio and music: from coding to source separation. Proceedings of the IEEE, 98(6):995-1055 2009.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 9748-8763. PmLR, 2021.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Commy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with JumpReLU sparse autoencoders. arXiv preprint arXiv: 14435, 2024.
12

---

Published as a conference paper at ICLR 2026
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06/25, 1(2):3, 2022.
Sukrut Rao. Sweta Mahajan, Moritz Böhle, and Bernt Schicle. Discover-then-Name: Task-agnostic concept bottlenecks via automated concept discovery. In European Conference Computer Vision, pp. 444 461. Springer, 2024.
Abhinaba Roy, Renhang Liu, Tongyu Lu, and Dorien Herremans. JamendoMaxCaps: A large scale music-caption dataset with imputed metadata. arXiv:2502.0746. 2025.
Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Sorieut. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
Harrish Thasarathan, Julian Forsyth. Thomas Fel, Matthew Kowal, and Konstantinos G Derpanis. Universal sparse autoencoders: Interpretable cross-model concept alignment. In Forty-second International Conference on Machine Learning, 2025.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features, 2025. URL httts:://axxii wttp:/gg////oooo attp//////2222222. 14786
George Tzanetakis and Perry Cook. Musical genre classification of audio signals. IEEE Transactions on speech and audio processing, 10(5):293-302, 2002.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the GAN latent space. In International conference on machine learning, pp. 9786-9796. PMLR, 2020.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Hanqi Yan, Xiangxiang Cui, Lu Yin, Paul Pu Liang, Yulan He, and Yifei Wang. Multi-faceted multimodal monosemanticity. arXiv preprint arXiv:2502. 14888, 2025.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(1):49-67, 2006.
Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek. Interpreting CLIP with hierarchical sparse autoencoders. arXiv preprint arXiv:2502.20578, 2025.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986, 2023.
A APPENDIX
A. PROOF OF THEOREM
Proof. We can assume that the columns of W are unit-norm, by appropriately scaling the sparse codes of each embedding in the set. Take any pair of embeddings; for simplicity of notation, call these embeddings & and y. By the two conditions in the theorem statement, we know that (a,, y) > and there are K sparse vectors 2, and zy with non-negative entries such that Wz, Since modality-split, 2n and 2, have disjoint support, so their inner product is and
Wz. and
Lgs (Zz, 2n) = ||z|||1 + ||
13

---

Published as a conference paper at ICLR 2026
Without loss of generality, assume the support of these two sparse vectors is {1,.... K and K + ......,K] Then, we have
K
2k
c << (Zz,
2
W. wj.
i=
K+1
Assume for the sake of contradiction that w. wy summation above. Then, we would have
= for all (i,j) pairs in the double
2k
K
2K
Σ
22 m
w
2 vi
2
j=j==
which a contradiction, Hence, there exists some (i.j) with {I,..., K} and j {K + 1,..., 2K} satisfying w Wj = |
We can assume WLOG that these are w and Wk+1-
The construction of W then follows naturally from the Gram-Schmin orthogonalization process. Define the matrix W W1 Wi:.... WK WK+11 WK+2 wp: Kx]+ where
WK+1 (w] W wk+1)w;
ØK+1
WK+1 ( wk+1 w1
In this new dictionary, all embeddings besides y admit the same sparse code with 0 appended in the p + 1th position. A decomposition of y in the dictionary W can be computed
2K
Σ
y = WK
+
j=K+2
2K
(2KK (w) wx+1)w1l1e (w) wx+1)w]) +
Σ Z..20
j= K42
2K
-y, wK+10 WK+1W1 + Zg.K+1lluu WK+1 (wi wk-1)w/ll20kk
Σ
2y.j Wj-
j=K+2
Letting the sparse codes of x and y under W be denotes as Σn and žy, respectively, we can therefore conclude (2 2ny) = (Z kn) 2n,1 2g. WK+1
Zg. 2 2,,I1
0.
Noting that we can apply the same argument iteratively to increase the alignment of the other pairs by adding single dictionary element each step completes the proof.
A.2 EXPERIMENTAL DETAILS
In this section, we provide additional experimental details for the results contained in Section
A.2.1 SAE TRAINING DETAILS
In the main paper, train and evaluate each SAE variant in settings, listed below:
CLIP VIT-B/16:
We train SAES on the CLIP embeddings of the CC3M (Sharma et al.) 2018) dataset image/text captions, with balanced number of samples per modality. We fix the sparsity level K = 32 and the expansion factor as 16, resulting in dictionary dimensionality of 8192. We train for 25000 iterations with a batch size of 128. We adapt the TopK SAE implementation from Marks et al. 2024) for setup, with additional group-sparye regularization and cross-modal ran- dom masking. We sweep the group-sparsity regularization parameter over E (0.01, 0.05, 0.1, 0.2) and choose the largest value (0.05) that does not push the average sparsity within batch below
14

---

Published as a conference paper at ICLR 2026
= 32. We then choose random mask probability from {0.1, 0.3, 0.4) such that the frac- tion of explained variance is similar to the GSAE with = 0.05, and we decide on = 0.2. The learning rate in each case is chosen based on the scaling law in Figure 3 of (Gao et al. 2024 (which the default learning rate in the library (Marks et al. 2024)), and all models are trained with the Adam optimizer. We train the BatchTopk SAE with the same expansion factor of 16 and choice of K 32 128 for the sparsity level of each batch. For the Matryoshka SAE, we use the checkpoint provided by the authors of Zaigrajew et al. 2025) with an expansion factor 16, uniform weighting. and inference time value of K 32.
LATON CLAP:
Parameters are chosen the same way as the CLIP case, except we train for only 10000 iterations, noting that the loss converges well before this point. Using the same heuristics for hyperparameter selection, we choose a group-sparsity parameter À = 0.05 and random mask parameter p 0.1. The specific checkpoint of CLAP which we use ('mu- sic audioset audioset.epoch. 15 esc 90.14.pt") is finetuned for music data, and we train on the Jamendo- MaxCaps dataset (Roy et al., 2025), which we preprocess into pairs of second music clips and corresponding captions (using the provided annotations file).
A.2.2 EFFECT OF EXPANSION FACTOR AND K
In our main experiments, fix the expansion factor of all SAE variants 16 and the sparsity level as = 32. In this section, we provide additional experimental results to demonstrate that the key insights in our paper stable for wide range of these hyperparameters. Concretely, different choices of expansion factor and K and train 3 models-SAE. GSAE, and MGSAE-for each configuration, with group-sparsity parameter
sweep
0.01 and mask parameter p 0.05. In Table 3, we report the zero-shot performance on ImageNet for the sparse latents from each of these models. As seen the table, the MGSAE consistently achieves the highest zero-shot performance, and both variants with the group-sparse loss outperform the baseline SAE in all cases. Interestingly, we find that the expansion factor has minimal effect on multimodal alignment (as measured by zero-shot performance), while K has significant effect, with larger values leading to better performance.
Table 3: Zero-shot performance ImageNet for different expansion factor and sparsity level con- figurations.
Expansion factor
K
SAE
GSAE
MGSAE
16
0.287
0.294
0.309
32
0.335
0.361
0.381
64
0.357
0.407
0.435
8
16
0.295
0.308
0.321
32
0.322
0.349
0.371
8
64
0.326
0.400
0.436
16
16
0.276
0.313
0.314
16
32
0.295
0.341
0.361
16
64
0.304
0.375
0.399
A.2.3 LINEAR PROBE ON CELEBA
In this section, we provide additional details on how we use concept dictionaries to interpret the behavior of a linear classifier on the CelebA (Liu et al. (015) dataset. Suppose W is a learned dictionary which (approximately) admits sparse decomposition x = Wz z for any embedding E X. And the parameter of a trained linear classifier (we use logistic regression with no bias parameter and 2000 iterations of LBFGS).
We first assign a concept name to each column of W using the Discover-then-Namte approach from (Rao et al.) 2024). In particular, let 1 be large vocabulary of words. Then, for each column w; of W. we assign a concept name using:
Concept (i) = arg max vEY
w; |00,
[w1)
15

---

Published as a conference paper at ICLR 2026
where is the CLIP text encoder. In words, each dictionary element is assigned the label of the word in the vocabulary with the most similar CLIP embedding. We use a vocabulary provided in the implementation of (Bhalla et al. 2024a), where V is the set of 10000 most frequent tokens in the LAION-400m dataset and 5000 of the most frequent bigrams in the LAION-400m dataset.
Given these concept labels for each wo we proceed to the analysis of the linear probe Ô. In particular, note that a positive label ("blonde") is given to test samples æ for which x is large. Decomposing this in the dictionary, we obtain
w O.
Hence we see that when wi T large, concept is weighted more heavily by the classifier. The concepts listed in Figure 5 are the those corresponding to largest seven values of up
A.3 FURTHER EXPERIMENTS ON SIGLIP2 AND AIMv2
To show that our proposed methodologies yield benefits across a wide range of multimodal settings, also report experimental results for SAE, GSAE, and MGSAE models trained the SIGLIP2 TTccannen et al. 2025) and AIMv2 (Fini et al., 2024) text/image encoders. All GSAE models are trained using group-sparsity parameter of 0.01 and all MGSAE models parameter of 0.05. The expansion ratio and choice of k are fixed as in our CLIP experiments. For each encoder and each SAE variant, we report the dead neuron counts for each modality and the performance on three cross-modal tasks. We see that across both encoders, the MGSAE achieves the highest number of multimodal neurons and the smallest number of dead neurons. the zero- shot tasks, the GSAE and MGSAE both consistently outperform the SAE while having similar performance to each other. As in the main paper, dead neuron counts are computed using validation set of CC3M.
trained with mask
Table 4: Dead neuron counts for models trained on SIGLIP2 ViT-B/16 embeddings of the CC3M dataset.
Model SAE
Image only
Text only
Both
Neither
1590
2234
3206
5258
GSAE
1286
2140
6584
2278
MGSAE
1346
2095
6972
1875
Table 5: Performance comparison zero-shot cross-modal tasks between SAE, GSAE, and MGSAF, all trained on SIGLIP2 embeddings. Original performance on the dense SIGLIP2 em- beddings is provided for reference.
Model
ZS Image/Text Tasks
CIFAR-10
CIFAR-100
ImageNet
SAE
0.918
0.608
0.451
GSAE MGSAE
0.922
0.629
0.477
0.906
0.628
0.481
SIGLIP2 ViT B/16
0.939
0.739
0.686
Table 6: Dead neuron counts for models trained on AlMv2 embeddings of the CC3M dataset.
Model
Image only
Text only
Both
Neither
SAE
232
231
2152
9673
GSAE
185
192
3572
8339
MGSAE
185
216
3910
7977
16

---

Published as a conference paper at ICLR 2026
Performance comparison zero-shot cross-modal tasks between SAE, GSAF, and MGSAE, all trained on AIMv2 embeddings. Original performance on the dense AlMv2 embed- dings is provided for reference.
Table 7:
Model
7S Image/Text Tasks
CIFAR-10
CIFAR-100 ImageNet
SAE
0.963
0.739
0,443
GSAE
0,972
0.799
0,493
MGSAE
0,970
0.802
0.522
AIMv2
0.976
0.858
0.745
A.4 ADDITIONAL EXPERIMENTAL RESULTS ON MS COCO
To demonstrate the robustness of our approach, also provide results obtained by training the three model variants we consider Section on a different dataset, MS COCO (Lin et al.) 2014), consisting of 330000 images with captions. In Table 8 provide neuron counts per modality, to show that the group-sparse variants contain more multimodal concepts than the standard SAE implementation. In Figure [) plot the MMS scores for each modality pair. Both of these results are calculated using an unseen collection of 10000 (image,text) pairs from the MS COCO validation Since MS COCO contains multiple captions per image. balance the modalities prior to training to ensure that each model is trained on the same number of image embeddings and text embeddings. These results are consistent with the findings on CC3M, indicating that our models (GSAE and MGSAE) are able to learn larger number of semantic concepts, particularly in the (text,text) and (image,text) settings.
Table Dead neuron counts for models trained on CLIP ViT-B/16 embeddings of the MS COCO dataset.
Model
Image only
Text only
Both
Neither
SAE
2118
652
5364
58
GSAE
625
541
6987
39
MGSAE
603
635
6909
45
Table Zero-shot performance of sparse codes for models trained CLIP ViT-B/16 embeddings of the MS COCO dataset.
Model
CIFAR-100 CIFAR-10
ImageNet
SAE
0.547
0.341
0.189
GSAE
0.805
0.403
0.237
MGSAE
0.820
0.423
0.247
Next, we also report the zero-shot performance of sparse codes for for these models, listed in Table g As in the case of models trained on CC3M, we find that the MGSAE obtains the best performance overall, and the group-sparse variants both significantly outperform standard SAES.
17

---

Published as a conference paper at ICLR 2026
mag SUUP
NWSjol/aol) CUP
MW miggimagoool CUP
0044 544
C54C
C248
Wrioomantelp SOAT
No 544
No 546
0.20
84
8.15
0.10
62
0.08
0.0
60
0.6
04
Normalladd
Normal 3ndox
houreh
Figure 6: Multimodal monosemanticity score (MMS) for models trained on the MS COCO training dataset. Validation is performed using the MS COCO validation set.
A.5 STEERING EXAMPLE ON CLAP EMBEDDINGS
In addition our main contribution of proposing improved dictionary learning techniques for mul- timodal embedding spaces, our work is to our knowledge the first to apply and interpret SAEs in the context of aligned audio/text embedding spaces like CLAP. So, in this section, we provide simple example demonstrating the utility of multimodal dictionary decompositions in this domain. We consider zero-shot text-to-music retrieval problem the MusicBench dataset (Melechovsky eet al. 2024), where a text input is matched to the most similar CLAP embedding of all music files the dataset. We demonstrate in Figure how interventions using the MGSAE be used steer the behavior of this task. We steer using the following process:
1. Concept naming: We follow the method of (Rao al. 2024 to assign concept name to each dictionary element. For this, we use the Google 20k word corpus as the concept vocabulary and LAION-CLAP similarities to assign concepts to dictionary elements. Using this method, we identify a neuron labeled with the concept "violin".
2. Intervene on sparse codes of the input prompt: We take the input prompt "A peaceful song", encode adding values of 0.5 (Low) and 1.5 (High), decode to dense vector, and then perform zero-shot retrieval on MusicBench.
sparse code using the MGSAE, intervene the "violin" neuron by
Text prompt
Intervention strength on "violin" neuron
Caption of retrieaal result
"This easygoing piece features delivering melocious performance. tune ukulele atmospheree...
peaceful song"
Nome
"A gentle and romanti is instnumental piece from seap soundrrack featur the violin guitar play mellow accoparree background.
Low
piano
sorrowful melody by the sound violin be ingng played strong
High
Figure 7: Top retrieval result from MusicBench for the input prompt "A peaceful song" using zero- shot retrieval on CLAP embeddings. We vary the intervention strength on the "violin" neuron and report the caption of the top retrieved music clip.
18

---

Published as a conference paper at ICLR 2026
As shown in the figure, increasing the intervention strength leads to corresponding emphasis violin in the top retrieval result, from no violin (in the original retrieval) to a recording with violin recording featuring a violin as the main instrument. We are similarly able to
in the background, identify several neurons corresponding to genre names ("reggae", "jazz") and instrument types ("or- gan", "fiddle") that effectively steer retrieval results towards certain concepts. In general, hope that our preliminary results in this multimodal setting can lead to more interpretable and controllable systems for audio/text, well further research into the unique requirements and challenges of SAEs in domains aside from vision- vision-language language.
19


Reviews:
[{"review_id": "Wlm3Ce1jmv", "reviewer": "Reviewer_NjK2", "summary": "The authors propose both a metric and a methodology for better understanding and training sparse autoencoders (SAEs) on vision language model spaces. They propose a metric that combines how useful SAE codes are for aligning semantically aligned pairs, combined with how much SAE codes activate for multiple modalities or just one. They propose that instituting a group-sparse (GS) loss for SAEs is the solution for split-modality concepts, and show that their group-sparse SAEs are better along a suite of metrics like fewer dead codes, more semantic alignment, and better cross-modality transfer.", "questions": "Q1 What are the issues that arise from training an SAE on paired data? Might it be a problem that paired data is less broad in domain? Is there a way to integrate these findings about the GS loss with the advantages of using a broader base of data.\n\nQ2 Do you have any idea about why dead neurons are so influenced by the GS loss, and why this seems to be linked also with the multimodality of features?\n\nQ3 It took me some time to understand how the MMS metric differs from some other similar metrics (like the BridgeScore from Papadimitriou et al 2025), so a slightly expanded explanation would be great. After reading through it more carefully, I think I’m convinced that it’s a subtle way of going about measuring split-modality. Could you expand on the metric, and provide some helpful analysis on if there is interference between monosemanticity and multimodality and how this can be quantified?\n\nQ4 It seems that the main methodological contribution of this paper is the group-sparse loss term, but it is not very intuitively explained in the text (eg after line 269). Can you expand slightly on what group-sparsity means and why joint support is encouraged when using this norm? The text mentions this, but as it is the main methodological contribution to the paper I think a slightly more developed theoretical side could be useful. \n\nQ5 (minor) Is alignment targeting the same evaluation as monosemanticity? I’m not sure that aligned pairs is a measure that can be translated trivially to mean monosemanticity, it might warrant a sentence or two in the rewrite if you think so \n\nQ6 (minor) I didn’t find Figure 2 very helpful, it seems to emphasize that there are two encoders/decoders, when in fact the main contribution is the loss term which is in quite small font.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper nicely distills some of the recent research on VLM interpretability into a nuanced metric and and interesting new methodology. \n\nI find the evaluations of the GS-SAEs convincing, with both the intrinsic SAE evaluations of dead codes and MMS, as well as the more extrinsic zero-shot transfer (which comes out within a reasonable distance of the solidskyline of the actual embeddings), being interesting.", "weaknesses": "I think this is a paper that makes a solid contribution, and I don’t consider any of the below weaknesses to be especially strong.\n\nW1 I don’t feel that the paper has analyses or ablations that increase our intuitions in what it is about Sparse Group loss that causes the decrease in dead neurons, and the multimodal concepts (the latter is more clear). This makes the paper weaker, as what we can mostly get from it is that group-sparse SAEs are better, and so for multimodal models we should switch to them. Since as I understand it the paper would be strongest if it is meant to increase our intuitions about what makes good training for linear concepts, I would like to see another analysis helping with that. Possible questions are: How does the group-sparsity play out in the training dynamics? How do the dictionaries differ geometrically when we include or don’t include that", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O4bZAJvYOJ", "reviewer": "Reviewer_GGM9", "summary": "This paper is a well-placed venture in expanding interpretability for multimodal domains by using Sparse AutoEncoders. The authors tackle the issue of modality splitting in the “features” captured by SAEs that are trained on just reconstruction loss for multimodal models. Key contributions are expanding the metric of monosemanticity of a given feature to include the modality of activations and introducing an updated training recipe to improve alignment between the sparse encodings of different modalities.", "questions": "1. In Figure 5, the right side (MGSAE) shows 2 entires of concepts with the name “beautiful blonde” - do the authors understand why this might have happened and how the actual underlying concepts differ ? \n\n2. Table 1: Do the authors any intuition as to why there seem to be cases where the (M)GSAE methods perform so competitively despite seemingly having many dead neurons ? Would a potential No SAE baseline for these tasks be helpful in building understanding? \n\n3. Have the authors considered if the MGSAE method learn stable multimodal features even with random masking ? That is to say, could the concepts learned by the dictionaries in differently seeded runs be aligned ? \n\nNits: \n* Line 176: The ‘W’ is missing the subscript for ‘W_dec`\n* Line 641: Switch up between z and x, y", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. Investigates and quantifies the role of modality alignment in feature discovery using SAEs\n2. Introduces a simple but seemingly effective method to improve the discovery of interpretable monosemantic features \n3. Expands the application of SAEs to audio/text CLAP models alongside image-text CLIP models\n4. Discusses implications on downstream tasks like classification, retrieval, and steering", "weaknesses": "1. Only considers a limited range of test models (one for each set of modalities), which limits the evidence for the applicability of the proposed method. In particular, state of the art models like AiMV2 are shown to have different distribution of feature weights between modalities compared to the studied encoder [1], and the applicability of the proposed MGSAE method in such situations is unclear. \n\n2. Ablation studies aren’t presented, in particular, different expansion ratios for the SAE (number of features), and different sparsity measures (K) aren’t explored in the body of the text. This leaves unanswered questions about the applicability of MGSAE and if the method would continue to improve alignment and reduce dead neurons in different settings. \n\n3. Does not fully expand on the distribution of activations / neurons between different modalities. In particular, he authors do not delve into the comparative strength of activations between the modalities, nor the frequency of the activated concepts. A show of how the weight of the features are split between the modalities would be very useful evidence to further justify the paper’s claims of improved alignment between modalities using the presented method. \n\n4. Does not present baselines / benchmarks comparing the introduced MGSAE method with other known works that present potential improvements in multimodal settings over the TopK SAE method. In particular the methods proposed in [2] which are cited by the work would help position this work in the existing literature. \n\n5. While the MMS metric is motivated in intuition by the paper, no quantitive comparisons are present to further validate the measure compared to the results in [3]. An understanding of how MMS and raw MS compare, especially between the known methods and those introduced in the paper could further cement the metric’s usefulness.\n\n[1] “Interpreting the Linear Structure of Vision-language Model Embedding Spaces”, COLM 2025\n\n[2] “Interpreting CLIP with Hierarchical Sparse Autoencoders”, ICML 2025\n\n[3] “Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models“, NeurIPS 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xYur0sNvtI", "reviewer": "Reviewer_Nk7i", "summary": "This work studies interpretability of multimodal embeddings using sparse auto-encoder. The main objective is to find aligned sparse multimodal latent vector from sparse autoencoder. To this end, a definition of modality-split dictionary is provided and the author propose multimodal monosemanticity score (MMS). Moreover, with masking approach, their proposed SAE approach shows improved multimodal dictionary from experiments.", "questions": "1. Can the author provide what is the impact on choosing $K$ in interpretability and downstream tasks (e.g., classification)?\n2. What is the rationale to use random mask? I saw the random mask gives very marginal gain. Why does this method give such a small gain?\n3. Section 5.3 is interesting. Could the author provide more example or similar studies?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In terms of interpretability, understanding aligned multimodal embedding is important. The paper propose important concept of modality split dictionary, which I think a crucial for interpretability of multimodal embeddings. Proposed approach is simple yet effective as the experimental results verify the authors claim. Overall, although I think the contribution seems a bit incremental, problem and the proposed definition are worth looking at.", "weaknesses": "While I enjoyed reading the paper, I think the paper could be more improved if more experimental analysis is conducted. For example, I do not see ablation study in choosing $K$ in TopK step, and there is no intuitive explanation or implication from it. Moreover, there is no ablation study for $p$ for random mask. It is even difficult to see why the random masking is required. Thus, I think the paper needs more evidence and experiments, which would make it more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dQYYuiEY4F", "reviewer": "Reviewer_G3nW", "summary": "The authors have tackled the problem of “split dictionary” observed while using SAEs on multimodal models. They propose the existence of a unified dictionary with improved alignment and also a modified architecture of SAE to realise this. This is applied to CLIP and CLAP data. The authors are also the first to use audio as one of the modalities with text. They also propose a paired multimodal monosemanticity metric.", "questions": "1.\tThe authors propose a loss term with two group sparse codes – are there ablation studies without this loss?\n2.\tAlso is there any way to prevent the $L_{gs}$ term from causing the codes z and w from collapsing to zero vectors? If this happens, the loss term goes back to being like a standard SAE loss.\n3.\tThe authors propose the use of random masking to encourage multimodality of concepts. Are there ablation studies to show this helps? \n4.\tMy key concern is that the comparisons are only with SAE and GSAE. Can the authors provide empirical experiments comparing their method against the prior approaches described in the literature [1-4]?\n5.\tThe authors mention that they are the first to use audio data in a multimodal setting. Please provide insights on why audio data is hard to include in general. How does their approach compare with audio only methods[5]?\n6.\tIn Figure 3, MGSAE does best consistently, is there any intuition for why it is the best?\n7.\tThe approach has been evaluated on CLIP and CLAP, can SigLIP be added? If not, please explain why.\n\n[1] Isabel Papadimitriou, Huangyuan Su, Thomas Fel, Sham Kakade, and Stephanie Gil. Interpreting the linear structure of vision-language model embedding spaces. arXiv preprint arXiv:2504.11695, 2025.\n\n[2] Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata. Sparse autoencoders learn monosemantic features in vision-language models. arXiv preprint arXiv:2504.02821, 2025.\n\n[3] Hanqi Yan, Xiangxiang Cui, Lu Yin, Paul Pu Liang, Yulan He, and Yifei Wang. Multi-faceted multimodal monosemanticity. arXiv preprint arXiv:2502.14888, 2025.\n\n[4] Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek. Interpreting CLIP with hierarchical sparse autoencoders. arXiv preprint arXiv:2502.20578, 2025.\n\n[5] Pluth, D., Zhou, Y., & Gurbani, V. K. (2025, February). Sparse Autoencoder Insights on Voice Embeddings. In 2025 Conference on Artificial Intelligence x Multimedia (AIxMM) (pp. 1-6). IEEE.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe problem is well motivated\n2.\tThe proposed metric calculation is explained well.\n3.\tThe authors have also considered the problem of dead neurons.", "weaknesses": "My key concern is that the comparisons are only with SAE and GSAE. Can the authors provide empirical experiments comparing their method against the prior approaches described in the literature [1-4]?\n\nOther questions are detailed in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GY84dIl5kO", "reviewer": "Reviewer_xuXW", "summary": "The paper “Learning Multimodal Dictionary Decompositions with Group-Sparse Autoencoders” introduces a new approach to understanding multimodal embeddings (such as CLIP or CLAP) through sparse autoencoders (SAEs). While traditional SAEs have been effective for interpreting single-modality models by decomposing embeddings into sparse, interpretable features, they struggle in multimodal settings—often producing “split dictionaries” where features activate for only one modality. The authors address this limitation by theoretically showing that a non-split, multimodal dictionary with better modality alignment can always exist, motivating their proposed model: the Masked Group-Sparse Autoencoder (MGSAE). This model incorporates a group-sparsity regularizer and cross-modal random masking to encourage paired activations between modalities during training, thereby promoting shared multimodal features.", "questions": "1. Your theorem shows that a non-split dictionary with improved modality alignment always exists. Do you have any insights into how this theoretical result translates into practice—for example, what properties of the optimization landscape or initialization help the model actually discover such dictionaries?\n2. You base your approach on the Linear Representation Hypothesis. Have you observed any systematic deviations from linearity in multimodal embeddings (e.g., nonlinear interactions between visual and textual features) that limit the effectiveness of linear sparse decompositions?\n3. How sensitive is the model’s performance to the choice of the group-sparse regularizer (L₂,₁ norm)? Did you consider other structured penalties, such as mixed ℓ₁/ℓ∞ norms or hierarchical sparsity, to encourage shared but flexible activations?\n4. The random masking step seems crucial for reducing dead neurons. Did you explore deterministic or learned masking strategies, or investigate how the masking probability affects multimodal feature sharing?\n5. You share encoder and decoder weights across modalities, except for biases. Have you tried partially shared architectures (e.g., modality-specific encoders with shared latent layers)? If so, how does this affect multimodal alignment?\n6. How does your method scale with more than two modalities (e.g., image–text–audio)? Would group-sparsity naturally extend to this case, or would you need a modified loss to maintain balanced multimodal alignment?\n7. Did you observe high sensitivity to hyperparameters such as λ (for group sparsity), K (for sparsity level), or the masking probability p? Are there guidelines or heuristics you recommend for stable training?\n8. Could you share specific examples of learned multimodal concepts (e.g., visualizing dictionary elements that align text like “a dog running” with corresponding visual features)? This would help illustrate what kinds of shared semantics are captured.\n9. Does improving multimodal alignment ever come at the cost of losing fine-grained, modality-specific concepts (like visual texture or acoustic timbre)? How do you balance these competing objectives?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper does more than propose an empirical modification—it provides a theoretical result (Theorem 1) demonstrating that a modality-split dictionary can always be improved to a more aligned, multimodal one. This theoretical grounding strengthens the motivation for their method and shows that poor cross-modal alignment in SAEs is not a fundamental limitation, but rather an optimization bias that can be addressed.\n2. The introduction of group-sparse regularization combined with cross-modal random masking is elegant and well-motivated. These additions directly tackle the issue of unimodal activation in SAEs, encouraging shared sparse representations across modalities without requiring extra supervision or hand-engineered alignment losses.\n3. The paper is the first to systematically analyze and apply sparse autoencoders to audio/text embeddings (CLAP). This extends interpretability research into a new multimodal domain beyond the widely studied CLIP image–text models.\n4. The authors introduce new multimodal monosemanticity metrics that quantify neuron-level semantic alignment across modalities. This is an important methodological contribution for future interpretability research, as it offers a way to measure the coherence and cross-modal consistency of learned concepts.\n5. The proposed MGSAE not only improves interpretability—by learning more multimodal, semantically coherent features—but also enhances zero-shot cross-modal task performance, showing practical benefits in alignment-sensitive applications.\n6. The paper successfully theoretical insights about dictionary decompositions and practical model design choices. The theoretical results directly inform the architecture, and the empirical results validate those theoretical predictions.", "weaknesses": "1. Although the paper evaluates both image–text and audio–text embeddings, these experiments are still confined to two-modality settings. The proposed method’s scalability to more complex multimodal spaces (e.g., video–audio–text or vision–language–action models) remains unexplored. Additionally, the experiments are conducted on relatively small or well-curated datasets (CC3M and JamendoMaxCaps), leaving open questions about generalization to noisier or larger-scale data.\n2. The training configurations (e.g., 25k–10k steps with fixed dictionary size and sparsity) seem tuned for feasibility rather than large-scale rigor. There is limited discussion of hyperparameter sensitivity (e.g., λ for group sparsity, masking probability, or K-sparsity), which could significantly affect outcomes. Without broader ablations or scalability studies, it’s difficult to assess robustness or reproducibility across architectures and modalities.\n3. While Theorem 1 provides an elegant existence result, it is relatively abstract and does not characterize the conditions under which optimization will find such multimodal dictionaries in practice. The theory does not account for the stochastic and non-convex nature of training neural autoencoders, so the link between theory and empirical convergence remains somewhat heuristic.\n4. Most evaluations revolve around interpretability metrics (monosemanticity, dead neurons, activation overlap). While important, the work provides limited evidence of improvements in real downstream tasks such as retrieval, captioning, or multimodal reasoning. Demonstrating such task-level benefits would strengthen claims of practical relevance.\n5. The group-sparse and shared masking mechanisms force stronger alignment between modalities, which may inadvertently suppress modality-unique features that are still valuable (e.g., color-specific features in vision but not text). The paper does not deeply analyze whether this trade-off affects the diversity or richness of learned representations.\n6. While the paper reports quantitative metrics, it provides few qualitative examples of what specific multimodal concepts the MGSAE learns (e.g., visualizing dictionary atoms corresponding to text–image pairs). Such examples would have made the interpretability claims more concrete and convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)