Cross-Modal Redundancy and the Geometry of Vision–Language Embeddings
Authors: Grégoire DHIMOÏLA
Thomas Fel
Victor Boutin
Agustin Martin Picard
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal
concepts
sparse autoencoder
modality gap
applications of interpretability
Primary area: interpretability and explainable AI
TL;DR: Understanding the geometry of multimodality through a concept-based approach, leading to applications like semantic vector arithmetic and modality gap free embeddings.

Abstract: Vision–language models (VLMs) align images and text with remarkable success, yet the geometry of their shared embedding space remains poorly understood. 
To probe this geometry, we begin from the Iso-Energy Assumption, which exploits cross-modal redundancy: a concept that is truly shared should exhibit the same average energy across modalities.
We operationalize this assumption with an Aligned Sparse Autoencoder (SAE) that encourages energy consistency during training while preserving reconstruction.
We find that this inductive bias changes the SAE solution without harming reconstruction, giving us a representation that serves as a tool for geometric analysis.
Sanity checks on controlled data with known ground truth confirm that alignment improves when Iso-Energy holds and remains neutral when it does not.
Applied to foundational VLMs, our framework reveals a clear structure with practical consequences: 
**(*i*)** sparse *bimodal* atoms carry the entire *cross-modal* alignment signal; 
**(*ii*)** *unimodal* atoms act as *modality-specific* biases and fully explain the modality gap; 
**(*iii*)** removing unimodal atoms collapses the gap without harming performance; 
**(*iv*)** restricting vector arithmetic to the bimodal subspace yields in-distribution edits and improved retrieval. 
These findings suggest that the right inductive bias can both preserve model fidelity and render the latent geometry interpretable and actionable.

Published as a conference paper at ICLR 2026
CROSS-MODAL REDUNDANCY AND THE GEOMETRY OF VISION-LANGUAGE EMBEDDINGS
Grégoire Dhimoila a.b.c
Thomas Fel
Victor Boutin"
Agustin Picard
Brown University
ENS Paris Saclay
DDEL IRT Saint Exupéry
Kempper Institute, Harvard University
CNRS
httts//ggtthho.ccc//aaaaaaaellss
ABSTRACT
Vision-language models (VLMs) align images and text with remarkable success, yet the geometry of their shared embedding space remains poorly understood. To probe this geometry, begin from the Iso-Energy Assumption, which exploits cross-modal redundancy: a concept that is truly shared should exhibit the same average energy across modalities. We operationalize this assumption with Aligned Sparse Autoencoder (SAE) that encourages energy consistency during training while preserving reconstruction. We find that this inductive bias changes the SAE solution without harming reconstruction, giving us a representation that serves as a tool for geometric analysis. Sanity checks on controlled data with known ground truth confirm that aligmment improves when Iso- Iso-Energy holds and remains neutral when does not. Applied to foundational VLMs, our framework reveals a clear structure with practical consequences: (i) sparse bimodal atoms carry the entire cross-modal alignment signal; (ii) unimodal atoms act as modality- specific biases and fully explain the modality gap: (ii) removing unimodal atoms collapses the gap without harming performance; (iv) restricting vector arithmetic to the bimodal subspace yields in-distribution edits and improved retrieval. These findings suggest that the right inductive bias can both preserve model fidelity and render the latent geometry interpretable and actionable.
INTRODUCTION
Vision- Lainuala models (VLMs) (Radford et al., 2021; Zhai al., 2023; Fini et al., 2025; Tschannen et al., 2025) have become central to applications from visual question answering (Chen et al., 2024) and medical imaging (Singhal et al., 2023) to autonomous driving (Zhou al., 2024) and embodied AI (Shukor et al., 2025), creating shared embedding spaces where visual and textual representations of similar concepts align. Despite their empirical success, we lack a principled understanding of how these models internally organize and align semantic content across modalities. This work investigates the geometric structure of cross-modal embeddings: How do VLMs organize the alignment between visual and textual semantics, and what principles govern this shared representational space? Under- standing these mechanisms is crucial for designing more robust and interpretable vision-language architectures.
From Attributions Concepts. To address this, the interpretability community has developed an array of tools (Gilpin et al., 2018; Bau et al., 2017) aimed at dissecting learned representations. Early efforts centered on attribution methods (Zeiler & Fergus, 2014; Sundararajan et al., 2017; Smilkov et al., 2017; Petsiuk et al., 2018; Fel et al., 2021) that highlight "where" a model focuses its attention, but these approaches often fall short (Nguyen et al., 2021; Kim et al., 2022: Colin et al., 2022: Hase & Bansal, 2020; Sixt et al., 2022) of explaining "what" abstractions and on which basis the model operates. More recent concept-based methods (Ghorbani et al., 2019; Zhang et al., 2021; Fel al., 2023b; Flhage et al., 2022; Fel et al., 2023a) have emerged answer this, seeking to extract meaningful features (concepts) that models implicitly compute over. Concept extraction is typically
Correspondence to: gregoire. dhimai reggmmee.dhimoi ledenssparis-sane

---

Published as a conference paper at ICLR 2026
framed dictionary learning problem (Tošić & Frossard, 2011; Rubinstein et al., 2010; Elad, 2010; Mairal et al., 2014; Dumitrescu & Irofti, 2018): identifying an overcomplete set of basis vectors that explain internal activations via sparse coding (Olshausen & Field, 1996; 1997; Lee et al., 2006; Foldiak & Endres, 2008; Rentzeperis et al., 2023). This approach is a direct response to an underlying phenomenology of the activation space: the Linear Representation Hypothesis (LRH) (Elhage et al., 2022; Wattenberg & Viégas, 2024) which posits that model activations can be viewed as sparse combinations of latent directions drawn from a high-dimensional concept basis. This motivated not only by empirical findings, but also by geometric arguments: in nigh-dimensional spaces, sparse sets of nearly orthogonal directions can represent exponentially many distinct concepts with minimal interference, principle reminiscent of Balnnson-Lindeesttauss-tttt & Nelson, 2017) embeddings. Sparse autoencoders (SAEs) (Makhzani & Frey, 2014; Elhage et al., 2022) directly operationalize this phenomenology by learning approximate inverse mapping from model activations to latent conceptual directions. They recover a sparse code that identifies which overeomplete basis elements (concepts) are active in given representation, and have been effective at uncovering semantically meaningful structure in both vision (Gorton, 2024; Fel et al., 2025; Dreyer et al., 2025; Rao et al., 2024) and language models (Cunningham et al., 2023; Bricken al., 2023; Rajamanoharan et al., 2024; Gao et al., 2025; Surkov et al., 2025).
(Johnson et al., 1984; Larsen
Multimodal Interpretability. However, applying SAEs to VLMs (Bhalla et al., 2024; Pach et al., 2025) reveals a puzzling behavior: concept dictionaries often segregate by modality (Papadimitriou et al., 2025). Although VLMS are trained for cross-modal alignment, the extracted concepts tend to activate exclusively for either image or text inputs. This observation is the concept-based view of a now well-documented phenomenon called the modality gap. Prior work has described this separation geometrically, attributing it to a conical structure in the embedding space (Liang et al., 2022; Ethayarajh, 2019), or through the lens of training dynamies induced by the contrastive loss (Fahim et al., 2024; Shi et al., 2023; Yaras et al., 2024). Yet these accounts do not explain what such separation means at the level of shared concepts.
In this work, we introduce a framework for analyzing multimodal representations grounded in an explicit generative model. Central to this framework is the Iso-Energy Assumption - if concept is truly shared across modalities, it should exhibit invariant energy, defined as the average squared activation, regardless of input domain. This assumption provides a concrete, testable criterion for identifying bimodal concepts. We then operationalize this simple assumption with a natural method: an aligmment- penaaized Matching Pursuit Sparse Autoencoder (Aligned SAE), which encourages energy consistency across modalities during training. This approach allows us diagnose whether extracted concepts genuinely support cross-modal alignment or merely reflect modality-specific patterns. This work makes the following contributions:
We introduce the Iso-Energy Assumption, which exploits cross-modal redundancy by requiring that shared concepts exhibit the same average activation energy in image and text.
We operationalize this assumption with an Aligned Sparse Autoencoder that enforces energy consistency during training while preserving reconstruction, and validate it with ground- sanity checks where classical SAES fail.
Applied to dual-encoder vision-language foundation models, this inductive bias reveals a geometric decomposition invisible to classical SAFs: (i) sparse bimodal atoms carry the entire cross-modal alignment signal, while (ii) unimodal atoms carry prodalitysppccfffe information and fully explain the modality gap. with few high energy atoms acting modality-speciffe biases.
Moreover, our work reveals that cross-modal information is carried by shared atoms, as opposed to
idiosyncratic ones described by Papadimitriou et al. (2025).
This structure enables actionable interventions without loss of performance: (ii) removing unimddal atoms collapses the modality gap, and (iv) restricting vector arithmetic to the bimodal subspace yields in-distribution edits.
At the core lies single theoretical premise: if concepts are genuinely shared across modalities, they must imprint redundant statistical traces in each domain. We formalize this intuition by modeling multimodal representations as partial inverses of a shared generative process, and introducing the Iso- Iso-Energy Assumption as the inductive bias that makes concept recovery feasible.
Nomenclature. We distinguish the following terms when referring to concepts based on their behavior and based on the type of information they carry. (i) Activation patterns: (a) unimodal concepts activate exclusively on single modality, whereas (b) bimodal concepts activate on both. This
2

---

Published as a conference paper at ICLR 2026
distinction is made by thresholding the modality score μ., comparison of domain wise energy (see Appendix E.2.2). (ii) Information carried: (c) shocciity-ppecific concepts carry information specific to given modality, e.g. that an image has cropping artifacts, while (d) cross-modal concepts carry cross-modal information and participate in the contrastive geometric alignment between modalities.
RELATED WORK
Previous work extensively describes salient phenomena in the shared space of multimodal dual- encoders (Schrodi et al., 2025; Levi & Gilboa, 2025; Jiang et al., 2023; Udandarao, 2022). Most notably, Liang et al. (2022) describes what is now commonly known as the modality-gaps that image and text embeddings reside in disjoint cones in the latent space. This modality gap has been attributed to a cone effect (Liang et al., 2022; Ethayarajh, 2019; Schrodi et al., 2025) and training dynamics induced by the contrastive loss and by mismatched pairs of data (Fahim et al., 2024; Shi et al., 2023; Yaras et al., 2024). (Levi & Gilboa, 2025) additionally shows that embeddings are contained near the surface of ellipsoid empty shells centered near the mean of the two distributions μ* and prr.
The cone effect naturally comes with a salient difference in modality wise means A = P1 PT (Levi & Gilboa, 2025; Liang et al., 2022; Fahim al., 2024), or a perfect linear separability between image and text embeddings (Schrodi et al., 2025; Levi & Gilboa, 2025; Fahim et al., 2024; Shi et al., 2023). Furthermore, Schrodi et al. (2025) shows that, surprisingly, small subset of coordinates in the canonical latent basis accounts for most of the norm of A. Previous work tries to get rid of the modality gap by shifting modality-wise means (Liang et al., 2022; Levi & Gilboa, 2025), or by projecting out the few canonical directions mentioned above (Schrodi et al., 2025). In all of these works, the proposed intervention decreases cross-modal performance of embeddings, with the notable exception of Zhang et al. (2023), whose Proposition I precedes Figure 4 shows that this difference in means, while explaining the bulk of the modality gap and linear separability between the two sets of embeddings, is not enough to explain the full distributional mismatch. For that, we need to account for modality-specific information.
more general Proposition 1.
Jiang al. (2023) takes an information-theone in representations is necessary to preserve unimodal capabilities. They use this insight to design new architecture that would explicitely regularise for the representation of such information. We show that (i) unregularised foundational VLM encoders possess such representations, and that (ii) the underlying mechanism is to organise these types of information linearly distinct subspaces- let us call Q and Qr the subspaces containing modality specific information, and I the one containing shared information. The projection by Schrodi et al. (2025) described above, though not introduced for this purpose, is a first attempt at identifying and intervening on Pg &fr, characterised by the span of the canonical basis vector selected, while I' would be the orthogonal complement. However, their results are negative, their intervention is detrimental model performance even on cross-modal focused tasks indicating significant unintended alteration of r while leaving the gap wide open (see Appendix L), thus showing that their identified directions can't 't describe Ωr Ωr.
angle show that modality specific variability
Our novelty lies not in the description of most phenomena discussed above, but rather in the characterisation of these structures in foundational VLM encoders. Through the use of a concept- based approach, we are able to identify (i) high-energy unimodal features to unimodal biases, (ii)
T -= Cone(ã- D) and Q22T Cone(ō1/T D). Here, Cone(ei, ex) 1Σx Xre; A 2 0}. D De Kxd is the set of K dictionary atoms of latent dimension d, and 5 (resp. 5,, 57) E (0, 1] is the binary mask selecting bimodal (resp. image-, text-only) atoms. To claim this characterisation, we carefully analyse four complementary aspects of the dictionary through novel metrics. Once validated, we further test its practical value through targeted interventions on these structures.
EXPLOITING CROSS-MODAL REDUNDANCY FOR CONCEPT RECOVERY
We formalize the setting by modeling each datum (image, caption, ...) as generated from latent concept vector and a domain-specifie generator. From this perspective, encoders serve as partial inverses of a shared multimodal generative process. However, recovering the underlying concepts is in general an ill-posed inverse problem: without additional inductive biases, nonlinear ICA is provably unidentifiable (Hyvarinen & Morioka, 2016; Locatello et al., 2019; Khemakhem et al., infiniinfiittel many mappings can account for the observed data in the absence of cross-domain constraints. Introducing Iso- Iso-Energy provides a selection principle among these solutions and yields a representation that is useful for studying the geometry observed in practice. We begin by formally
3

---

Published as a conference paper at ICLR 2026
2 SAEs Concept pa
Concept space
X
Input space
Activations
Fimg
Overcomplete
Itxt
Learning life the shared embedding toltigee dimendoon spanee oode, exposing concept- thk atoms.
drawing a Rabbit leaping through suniit forest
(10 Nonr finor generators map abatreet concept real instante
encoder (ug, Sqgip.. compressed embedding
image.)
Figure Multimodal data-generating process. latent concept vector с E C (e.g., rabbit, forest, light, running) is sampled sparse combination of abstract concepts and rendered through domain-specific generators g (e.g., image or text). Dual-encoder models (e.g., CLIP, SigLIP) map these observations shared activation space, which sparse autoencoders (or other overcomplete dictionary learning methods) then attempt to lift back to concept-like atoms. However, without additional inductive bias, encoder-decoder pairs (f, Φ) identifiability problem nonlinear ICA. Here we leverage cross-modal redundancy as a useful inductive bias, nudging the solution toward recovering bimodal concepts.
uniquely determined, well-known
introducing our data generative process: each datum arises by (i) sampling a sparse concept vector and (iī) rendering it through a domain-specific generator. Formally, let E the number of latent concepts and C C CRR the concept space. Let D (di,...,dpp) d(=)) index domains with observation spaces (d).
Definition 1 (Multimodal Concept Generative Process). Sample e p -I with (supp(c)] << K. For each de D, a deterministic generator g(d) : CC+x(() yields x = g (c). We admit that each g(d) is C and locally invertible.
VLM encoder f maps each observation x (d) to a shared embedding, and a sparse autoencoder & attempts disentangle this embedding back into concept coordinates, that & f approximates the inverse of Definition 1. However, recovering latent concepts from nonlinear generators is not identifiable in general: without additional structure, many different dictionaries can explain the same data (Hyvarinen& & Morioka, 2016; Khemakhem et al., 2020; Locatello et al., 2019). This ambiguity visible in practice, where SAE dictionaries often vary substantially across random seeds (Fel et al., 2025). Domain membership provides auxiliary information, but by itself insufficient ensure consistent recovery. We therefore introduce a cross-domain simple inductive bias. the Iso-Energy Assumption, which states that genuinely multimodal concepts should maintain consistent average energy across modalities. Because such concepts manifest parallel across domains, their observations contain redundant signals that can be exploited to guide dictionary recovery toward more stable and plausible solutions. Formally,
Definition Geo-Ennrgy Assumption). Let ub USED X(i) + RR denote the learned encoder (VLM composed with SAE Φ)) We say vp satisfies Iso-Energy if the second moment of euch coordinate is
E X
(x)))
E XER
(v(X))), with &s E E11, (d,d)
Iso-Energy provides a testable, domain-agnosti inductive bias: cross-modal features should cor- respond to bimodal concepts and maintain comparable energy across domains, whereas modality- specific factors need not. This constraint narrows the solution set without requiring instance-level matching. It is reminiscent of the rosetta neurons (Dravid et al., 2023; Gresele et al., 2020) and the platonic representation hypothesis (Huh et al., 2024), suggesting that independent models (in our case, the vision and language encoders) converge shared features.
To operationalize this assumption, we adopt as our base the Matching Pursuit (MP) sparse autoencoder, recently introduced by Costa al. (2025) and rooted in the original frame- work of Mallat & Zhang (1993). MP enforces to sparsity through sequential residual updates, which
Operationalization.
for any there exists neighborhood U(c) on which g is injective with continuous inverse.
4

---

Published as a conference paper at ICLR 2026
aligns with the sparse generative model introduced above and has shown strong empirical reconstruc- tion performance in vision compared to ReLU (Bricken et al., 2023), JumpReLU (Rajamanoharan et al., 2024), BatchTopK SAEs (Gao et al., 2025; Bussmann et al., 2024) (see Appendix A for their formal definition). We then incorporate Iso-Energy into the sparse autoencoder recipe as a soft regularizer: small penalty encouraging the activations of the same atom to maintain similar strength across domains. Given Eg-normalized codes Z(d), Z(d') R our training loss becomes:
LSAR-A LSAR + B Lalign with
Latign - Tr Z Z )
(I)
with CSAE being the training loss of standard SAE, typically E norm measuring reconstruction error with a sparsity constraint on the encoder's output. Lalign is the soft inductive bias, where b is the batch size and K the number of latent atoms. With small weight (3 & 10-4, see Appendix B), this regularizer gently biases the dictionary toward bimodal features while preserving reconstruction performance. In Section 4, we demonstrate that this bias does not force the model to create multimodal concepts that don't 't exist in the input data. It is important to note that the minimum of this loss function is consistent with the iso-energy principle for cross-modal concepts formalized in Definition 2: maximizing the cosine similarity of codes coming from the aligned samples from the different modalities leads to codes with the same energy across modalities. From this point onward, we denote the unregularized model SAE, and its dlignment-augmenter counterpart SAE-A.
4 RECOVERING MULTIMODAL STRUCTURE WITH ALIGNED SAES
Sanity check. Before turning to large-scale embeddings, we validate our approach on controlled toy data with known ground truth. We construct two synthetic lata-generating processes that mimic CLIP-like cosine similarity statistics (Fig. 9) while giving us exact control over which atoms are unimodal and which are bimodal (Appendix C.I). Each sample is generated by drawing sparse code z with || = = 20 and producing normalized embeddings
x (+) z D (1)
a (2) = (D")),
|a = ||x: 1|2 1.
Two parameters govern the process. First, Ti sets the cross-modal alignment of each shared (bimodal) atom l B by fixing the cosine between its per-modality components. Second, 72 fixes the average paired image-text similarity at the embedding level:
(d("). ((o))) = T1, and E (o x x011)\
cos <d a("))
||a 1l2
|| (2) 112
T2.
with the Iso-Energy case corresponding to T1 (the bimodal atom is identical across modalities up to scale).
Using this approach, we can generate a distribution that matches the alignment statistics found in CLIP embeddings. We then compare standard Sparse Autoencoder (SAF) to its "aligned" variant trained with the loss function in Eq. 1. This new loss encourages energy consistency without compromising reconstruction quality, indicated by the high R-squared value (1t² > 0.99) we observed in all our experiments. Recovery is evaluated against (D*, Z*) using the Wasserstein distance W between learned and true atoms (lower is better) and mean matching accuracy (mma) of usage patterns after optimal bipartite matching (higher better). When Iso-Energy is violated (T₃ + 1), both SAE and SAE-A recover the dictionary equally well (W 0.19, mma 0.82), confirming that the regularizer does not hallucinate himodal atoms. When Iso-Energy holds (7) = 1), the standard SAE fails (W- 0.396, ıma 0.29) while the aligned SAE-A succeeds (W 0.184, mma = 0.52), showing that the inductive bias is neutral when unnecessary and decisive when appropriate. Full experimental details are provided in Appendix C.2. Having validated the principle in controlled settings, we next study its behavior on embeddings from foundation-senn VLMs.
4.1 EMPIRICAL EVALUATION ON VISION-LANGUAGE FOUNDATIONS
Setup. We train both SAE and its aligned counterpart SAE-A on activations from six representative models: CLIP (VTT-B/S B/32B/32, ViT-L/14) (Radford et al., 2021), OpenCLIP, OpenCLIP-L (Cherti et al., 2023), SigLIP (Zhai et al., 2023), and SigLIP2 (Tschannen et al., 2025) (see Appendix D for more details on these models). All SAE models are trained with identical hyperparameters (expansion ratio
5

---

Published as a conference paper at ICLR 2026
Metric
CLIP
CLIP-L
OpenCLIP
OpenCLIP-L
SigLIP
SigLIP2
MSE (Q)
0.141
0.163
0.207
0.213
0.246
0.257
0.244 0.253
0.212
0.214
0.115
0.115
R (t)
0.859 0.837
0.793
0.787
0.754
0.742
0.755
0.747
0.788
0.784
0.884
0.885
Pace (1)
0.847 0.915
0,843
0.868
0,849
0.880
0.845
0.873
0.897
0.899
0.886
0.903
(())
0.327 4.232
1,566
4.086
4.072
16.02
8.737
16.58
1.370
2.182
0.713
1.475
FDA (1)
2.630
4.559
3.914
4,800
4.369
8,160
9.787
16.49
8.831
34.95
8.246
18.24
6 (1) 0.224 0.125
0.039
0.021
0.037
0.018
0.001
-0.000
0.023
0.006
0.007
-0.006
Table 1: Comparison of unregularized vs. aligned sparse autoencoders across six VLMs. We report reconstruction fidelity (MSE, RR2) and multimodality-sesiiiee metrics: probing accuracy (Pace), functional alignment (p), Functional and Distributional Agreement (FDA), and interventional robustness (6.). Left values correspond to SAE, right values to SAE-A. While reconstruction is nearly identical, the aligned variant consistently improves on all multimodality metrics : increases by more than order of magnitude, FDA doubles (or triples), and 6, remains near zero, demonstrating that bimodal atoms alone sustain cross cross-modal modal alignment while unimodal ones contribute little beyond modality specific bias.
8, target 8o 20) using subset of million LAION embeddings chosen at random, ensuring that observed differences are attributable to the Iso-Energy regularizer rather than training artifacts. Clas- sical reconstruction metries (MSE, R²) reveal little difference between the two methods, confirming that the alignment penalty does not compromise fidelity. Yet reconstruction alone is uninformative: two dictionaries with identical R2 may encode radically different concept structures. To capture these distinctions, we introduce a suite of multimodality-senne -sensiiive metrics, each designed to probe a complementary aspect of the recovered dictionary. For the full definitions of these metrics, we refer the reader to Appendix E.
Tuning 3. In Equation (1), we select 33 via a log-sweep over {10-6, 10-1} and pick the largest value such that the difference in explained variance compared to non regularised SAE is less than 0.05. This rule reproduces our settings without hand-tuning and, under this criterion, we also observe no degenerate (always-on) features in practice; Appendix B for details.
Metries:
(i) Probing accuracy Pace (Appendix E.2.2). This metric tests whether the geometry of dictionary atoms reflects the modality structure of the embedding space. Unimodal atoms should act as strong linear classifiers for domain membership, while bimodal atoms should remain domain-agnostic Pace gathers all these classifiers' performance in a single scalar. High Pace therefore indicates that the dictionary correctly distinguishes modality-speciff and shared information.
(ii) Functional alignment (Appendix E.2.2). Beyond geometry, we ask which features actually drive cross-modal alignment. P measures the ratio of alignment explained by bimodal versus unimodal features through instance-level co-activation patterns. Values of p > 1 indicate that alignment is predominantly supported by bimodal concepts, consistent with the Iso-Energy Assumption.
(iii) Functional and Distributional Agreement (FDA) (Appendix E.2.2). At the population level, FDA measures whether the functional role of features is consistent with their geometric organization. While is local and sestance-based. FDA checks distributional alignment across large batches. Intuitively, high FDA confirms that bimodal atoms globally sustain the match between modalities. Neither p nor FDA considers the contrastive aspect of alignment, which is done by our final metric.
(iv) Interventional robustness & (Appendix E.2.2). To test for causality, we measure how performance in retrieval tasks changes when we remove unimodal features. We use d, to quantify the change in recall after ablating these unimodal atoms. A small 8, suggests that these atoms aren't 't essential, while a large drop indicates they are necessary for the contrastive aspect of the embeddings.
As shown in Table 1, SAE-A matches SAE's MSE and R² scores but consistently out- performs it on all multimodality-sensiiin of magnitude across models, indicating that alignment is almost entirely driven by bimodal atoms. Likewise, FDA improves substantially, confirming consistency at the distributional level. Probing accuracy improves modestly, reflecting a more distinct geometric separation, while ó, remains small, demonstrating that unimodal atoms can be safely ablated without harming retrieval performance and therefore do not contain cross cross-modal modal information. Taken together, these results offer converging evidence that, in SAE-A, unimodal atoms encode modality-specifie information, while bimodal
Results.
metrics. In particular, p increases by more than an order

---

Published as a conference paper at ICLR 2026
10-1
1.0
10-
0.8
10
Modality Score
density
0.6
10
10
0.2
10
iu E (0.1.0.,, 0.5,0.9)
Enorgy
10
10
Fimz
10
10
Figure 2: (Left) Energy distribution across learned atoms. The majority of features are bimodal and medium-energe (inside diagonals defined by constant modality score μーAppendix E.2.2), while only a handful of high-energy unimodal features dominate modality sppciffc variance. These high- energy unimodal atoms behave like modality biases and responsible for much of the observed modality gap. (Right) Geometric organization of concepts. Low-dimensional projections reveal three distinet clusters: image-only, text-only, and bimodal. Unimodal atoms align with the modality cones of the embedding space, while bimodal atoms occupy to these directions, thereby sustaining cross-moda alignment.
subspace orthogonal
atoms, identified via Iso-Energy, constitute the principal basis for cross-modal alignment. In SAE, however, the picture less clear, and some unimodal atoms carry cross-modal information.
These findings indicate that the Iso-Energy Assumption reveals qualitatively distinct structure: compact cross-modal subspace, entirely spanned by bimodal concepts, that retains the contrastive power of the original embeddings while fully supporting cross-modal alignment. As will see in Section 5, making this structure explicit also makes it actionable: it allows us to manipulate represen- tations directly, from closing the modality gap to performing in-distribution semantic arithmetic. But before making this structure actionable, we propose an in-depth analysis of the solution to understand some aspects of the geometry.
4.2 CONCEPT GEOMETRY UNDER ISO-ENERGY
Having established the effectiveness of Iso-Energy on real VLMs, we now turn to a qualitative characterization of the learned concepts. This analysis focuses on how energy distributed, how atoms are geometrically organized, and how interpretable they are in practice.
Energy distribution. A first observation concerns how energy is distributed across modalities. As illustrated in Figure 2, the vast majority of features are himodal and exhibit moderate energy levels, whereas small subset of unimodal features concentrates disproportionately high energy. These high-energy unimodal atoms dominate modality-specific variance and act as biases, as further detailed in Appendix F.
Geometric organization. The high values of Pace show near- pear-perfect alignment between latent structure and concept organization. This can be visualized by projecting the learned atoms into a low-dimensional space, revealing a clear separation into three clusters: image-only, text-only, and bimodal (Fig. 2, right). Unimodal atoms align tightly with the cones spanned by image and text embeddings, reproducing the geometry of the modality gap. In contrast, bimodal atoms occupy modality- -agnostic subspace, orthogonal to the unimodal directions. This geometry explains why bimodal (resp. unimodal) atoms carry cross-modal (resp. modality-specifft information.
Qualitative inspection. Finally, examine the semantic meaning of individual atoms by inspect- ing their most activating examples (Appendix G). Bimodal atoms are semantically stable, consistently capturing the same concept across modalities (e.g., colors, objects, actions). Unimodal atoms, on the other hand, often reflect idiosyneratic modality -specffic signals (such as poor cropping artifacts in images or nnmm patterns"' in text) that contribute little cross-modal semanties. Together, these
7

---

Published as a conference paper at ICLR 2026
of CLIP embeddings
UMAP of D atoms
10a
10a
Modality Gap
Text Dousity
Density
0.72
Image only Rimodal Text-only
Modality Gap
0.04
Latent Space: zD
Concept Space: Z E IR NxK
Zdiag( 6)
AA)
Latent Space:
10
10a
A IR Nxd
Á R.vd
Figure 3: The modality gap arises from multiple unimodal concepts, while bimodal concepts are sufficient sustain cross-modal alignment. Left: CLIP embeddings are re-expressed through learned dictionary. A PCA projection highlights the separation between modalities, and UMAP layout distinguishes two types atoms: unimodal and bimodal. Right: Removing unimodal atoms with a binary mask 8 E {0, 1j closes the gap. The reconstructed embeddings A continue to support retrieval, indicating that bimodal atoms alone capture the structure necessary for alignment.
10th Nearest Neighbor in
Distibation
Recall
0.909
Images
00D Texts
Score
shift
Laan
Recaling 1
Recall@
Removing the modality gap
Normalize
Figure 4: Filtering unimodal atoms closes the modality gap without harming performance. (Left) Synthetic illustration comparing our method with the embedding shift baseline (Liang et al., 2022). Only our approach merges image and text distributions. (Right) Histogram of distances from each image (ID) and caption (OOD) embedding to its 10th nearest image neighbor. The modality gap measured the separation between the ID and OOD histograms. Filtering unimodal atoms aligns the two distributions, whereas shift degrades performance and leaves the gap wide open.
three perspectives converge on the same conclusion: unimodal atoms function as modality-speciff biases, while bimodal atoms encode the shared conceptual backbone that supports cross- rross-modal alignment.
5 ACTIONABLE INTERVENTIONS ON MULTIMODAL EMBEDDINGS
Together, these analyses demonstrate that our dictionaries yield a structured and interpretable de- composition of multimodal embeddings. Building this foundation, now shift from analysis to intervention: once Iso-Energy isolates the cross-modal backbone, it enables direct manipulation of embeddings in ways that were previously inaccessible. In particular, consider the minimal intervention that removes modality information and examine its effect on two structural aspects. In fact, show that such transformations are possible without altering ranking-related capabilities even when the modality information is non-trivial (c.g.. not bias, as proposed by Zhang et al. (2023)'s Proposition and under realistic assumptions of orthogonality.
Proposition 1 (Modality information removal impact ranking.). Consider Rdd with decompo- sition v = w(x) + (x)) where (xx) E Q encodes modality-specitt information, (xx) E I captures cross-modal content, and Rd Ω + Т. If visual and textual information are orthogonal, then ranking preservation guaranteed.
Proof. See Appendix K.
Closing the modality gap. First, we find that by filtering out unimodal atoms with a binary mask (Figure 3), nearly eliminate the modality gap while preserving retrieval and zero-shot performance.
8

---

Published as a conference paper at ICLR 2026
Concepts
Text Bias
Red
Embedding
Stone
Image Bias
Blue
Query
Red + Blue
"Is not red but blue"
Figure 5: Semantic vector arithmetic restricted to cross-modal information. Starting from Ruby (red stone), the target is a Sapphire (blue stone). The classical edit vector A Text + Blue Red is polluted by unimodal directions, producing a query Q Irc + A that drifts out-of-distribuce contrast, restricting to bimodal atoms yields QSAE, which lies on the semantic manifold and reliably retrieves the correct target. This illustrates how unimodal features inject -ppeciit--ppcciiic bias into A, while Iso-Energe Sso-Eneryy isolates the truly shared concepts that support valid semantic arithmetic.
In
As illustrated in Figure 4, this intervention merges the image and text distributions, unlike embedding shift baselines (Liang et al., 2022), which enforce matching means but still leave distributions well approach preserves contrastive capabilities, showing that dictionaries
separated. Crucially, faithfully capture both contrastive and modality-specific information separately through bimodal and unimodal atoms. Measuring the modality gap is typically done by measuring the distance between the mean of text and image distributions, or by measuring their linear separability. However, these fail to account for distributional mismatches remaining post intervention. For this reason, we chose to turn to the out-of- oi-diitributtinn (OOD) literature to measure the modality gap, and borrow method described by Sun al. (2022). This method measures the separation between the blue and orange histograms in Figure 4. Our method consists in the following intervention on the activations, indicated with a tilde: A concept dictionary, and is the binary mask filtering out unimodal features, broadcast to the size Z. Reconstructed activations are indicated with hat: A = ZD. The embedding shift intervention, and variants described in the appendix, consists adding a modality- constant, essentially moving the mean of each distribution /T/T- Liang et al. (2022)'s s shift method transforms the images by
Z o 8)D, where Z R N contains the sparse codes, D is the
Thin I f& H4, similarly for texts Tahin-
Semantic vector arithmetic. Iso-Energy also grounds semantic manipulations. Let Iave be the source image, and A be the textual description of the difference between the source and target image iee, the relative caption. Restricting vector arithmetic to bimodal atoms QSAE := Lare + A (Figure 5) produces queries that remain n-distribution (Figure 6 and Table 2) while preserving retrieval (Ap- pendix H). In contrast, classical arithmetic Q Jare + A incorporates unimodal noise from the text embedding on top of the interesting cross-modal information, yielding degenerate queries that drift outside the embedding distribution. Our intervention consistently produces queries in-distribution without degrading performance on the FashionIQ benchmark Wu et al. (2021), demonstrating that the bimodal backbone revealed by Iso-Energe is practically useful.
Tax
Density
RAA
Target
OOD score (L)
Q
QSAE
CLIP
0.97
0.77
CLIP-L
0.95
0.76
0.7
OpenCLIP
0.86
0.68
Neighbor
OpenCLIP-L
0.87
0.72
Figure 6: Out-of-distribution behavior of semantic queries. Histogram distances between each query and its 10th nearest neighbor in the target image distribution. Classical arithmetic Q Sse + A drifts out-of-distribution while our bimodal- restricted query QSAE remains aligned with the target space. Base- lines using only the source image (Q baseline) or only the caption Tue
SigLIP
0.99
0.70
SigLIP2
0.99
0.61
Table 2: OOD scores for clas- sical queries vs. concept-based queries QSAE- Lower better, best results per line in bold.
difference A confirm the expected extremes: perfectly
in-distribution and fully OOD, respectively.
9

---

Published as a conference paper at ICLR 2026
Consider the following example. Let Iscc be an image of a ruby, A be the prompt "is not red but blue" and the target be an image a saphire. This is the example illustrated in Figure 5. Adding A to the ruby produces a query that contains both textual-only and visual-only concepts and therefore that do not correspond to any realistic embedding. Adding only the cross-modal concepts of A, however, produces query that actually corresponds an image of saphire.
6 CONCLUSION
The Iso-Energy Assumption introduces a simple yet effective inductive bias for analyzing multimodal representations. In synthetic settings, it facilitates the recovery of ground-truth structure; in large-scale vision-language models, consistently reveals a compact bimodal basis that supports cross-modal alignment. This basis makes multimodal concepts accessible, isolates unimodal concepts, closes the modality gap, and enables controlled semantic edits even in foundation-scalle VLMs. In contrast, standard sparse autoencoders tend to learn a diffuse mixture of unimodal and bimodal atoms despite similar reconstruction quality, which obscures the structure that underpins alignment.
Despite its promising results, our approach has several limitations. First, the alignment penalty sensitive to the choice of its weighting coefficient 3: when too small, it becomes ineffective; when too large, it can lead to degenerate feature representations. In this work, select through simple sweeps, but a more principled calibration-e.g. by coupling the penalty to a performance an important direction for future work. Second, our analysis
constraint or stability criterion-remant is conducted on reconstructions produced by the sparse autoencoder, rather than on the original embeddings. This constrains the reported performance to the autoencoder's reconstruction regime. Finally, our experiments are limited to dual-encoder vision-language models. Whether the same structural invariants and alignment properties hold in models with cross-attention mechanisms or generative training objectives remains an open question.
More broadly, our results support a hypothesis-driver approach interpretability:: inductive biases should be selected to reflect the structural properties relevant to downstream tasks, rather than applied indiscriminately. When properly aligned with task-relevant objectives, even simple biases can illuminate underlying mechanisms that might otherwise remain control without compromising core performance.
offer actionable
ACKNOWLEDGEMENT
This work has benefited from fundings from ANITI through the contract n D'ANR-2--ACC--002
REFERENCES
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability deep visual representations. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 6541-6549, 2017.
Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio Calmon, and Himabindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice). Advances in Neural Information Processing Systems (NeurIPS), 37:84298-84433 2024.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.
Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. ArXiv e-print, 2024.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision- language models with spatial reasoning capabilities. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14455-14465, 2024.
10

---

Published as a conference paper at ICLR 2026
Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language- image learning. Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 2818 8-2299,, 2023.
Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What cannot predict, i do not understand: hunan-centered evaluation framework for explainability methods. Advances in Neural Information Processing Systems (NeurIPS), 35:2832-2888 2845, 2022.
Valérie Costa, Thomas Fel, Ekdeep Singh Lubana, Bahareh Tolooshams, and Demba Ba. From flat to hierarchical: Extracting sparse representations with matching pursuit. arXiv preprint arXiv:2506.03093, 2025.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey, Sparse autoen- coders find highly interpretable features in language models. ArXiv e-print, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255 2009.
Amil Dravid, Yossi Gandelsman, Alexei A Efros, and Assaf Shocher. Rosetta neurons: Mining the common units in model 7oo. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1934-1943, 2023.
Maximilian Dreyer. Jim Berend, Tobias Labarta, Johanna Vielhaben, Thomas Wiegand, Sebastian Lapuschkin, and Wojeiech Samek. Mechanistic understanding and validation of large ai models with semanticlens. Nature Machine Intelligence, pp. 1-14, 2025.
Bogdan Dumitrescu and Paul Irofti. Dictionary learning algorithms and applications. Springer, 2018.
Michael Elad. Sparse and redundant representations: from theory applications in signal and image processing. Springer International Publishing, 2010.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, 7ac Hodseeld-Dodss, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCan- dlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the ge- ometry of BERT, ELMo, and GPT-2 embeddings. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 55-65, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/v 10/55///1/D19110000 1006. URL https://aclanthol..og org/ 2rgg/11 1006/.
Abrar Fahim, Alex Murphy, and Alona Fyshe. It's not modality gap: Characterizing and addressing the contrastive gap. arXiv preprint arXiv:2405. 18570, 2024.
Thomas Fel, Remi Cadene, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. Advances in Neural Information Processing Systems (NeurIPS), 34:26005-26014, 2021.
Thomas Fel, Victor Boutin, Mazda Moayeri, Remi Cadene, Louis Bethune, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. Advances in Neural Information Processing Systems (NeurIPS), 36:54805- 54818, 2023a.
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2711-2721, 2023b.
11

---

Published as a conference paper at ICLR 2026
Thomas Fel, Ekdeep Singh Lubana, Jacob S Prince, Matthew Kowal, Victor Boutin, Isabel Papadim- itriou, Binxu Wang. Martin Wattenberg, Demba Ba, and Talia Konkle. Archetypal sae: Adaptive and stable dictionary learning for concept extraction in large vision models. Proceedings of the International Conference on Machine Learning (ICML), 2025.
Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor G Turrisi da Costa, Louis Béthune, Zhe Gan, et al. Multimodal autoregressive pre-training of large vision encoders. Proceedings of the IEEE Conference on Computer Vision 2025.
and Pattern Recognition (CVPR), pp.
Peter Foldiak and Dominik Maria Endres. Sparse coding. ArXiv e-print, 2008.
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, llya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. Proceedings of the International Conference Learning Representations (ICLR), 2025.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Leilani H. Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. Proceedings of the IEEE International Conference on data science and advanced analytics (DSAA), pp. 30-89, 2018.
Liv Gorton. The missing curve detectors of inceptionvl: 1: Applying sparse autoencoders to inceptionvl early vision. ArXiv e-print, 2024.
Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Scholkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. Uncertainty in Artificial Intelligence, 2020.
Peter Hase and Mohit Bansal. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5540-5552, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020. acl-main.449 1.1. URL https: / /cclnthhooy wtwp:thhooogyyoo//222220ee org/2020 acl mell-main.... 491/.
Sai Sumedh R Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba. Projecting assumptions: The duality between sparse autoencoders and concept geometry. ArXiv e-print, 2025.
Minyoung Huh, Brian Cheung, Tongzhou Wang. and Phillip Isola. Position: The platonic represen- tation hypothesis. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 20617-20642. PMLR, 21-27 Jul 2024. URL https: htctps:/pproceee http:///oocceee...eeee////o mlr....m........ wlr. wttpgg..............m. htt...rree//vv//hhhhete html.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 2016.
Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping. Son Dinh Tran, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Understanding and constructing latent modality structures in multi-modal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7661-7671, 2023.
William Johnson, Joram Lindenstrauss, et al. Extensions lipschitz mappings into a hilbert space. Contemporary mathematics, 26(189-206):1, 1984.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. Proceedings of the International Conference on Machine Learning (ICMI.,, 2020.
12

---

Published as a conference paper at ICLR 2026
Sunnie S. Y. Kim, Nicole Meister, Vikram V. Ramaswamy, Ruth Fong, and Olga Russakovsky. Hive: Evaluating the human interpretability visual explanations. In Computer Vision ECCV
2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII, pp. 280-298, Berlin, Heidelberg, 2022. Verlag. ISBN 978-3-031-11774411 doi: URL htt.tttts///i..ooo 100.///0/000//1111//9//////// /97000979778000 1-010799788---1111777--------
Kasper Green Larsen and Jelani Nelson. Optimality of the johnson-lindenstraus lemma. In 2017 IEEE 58th annual symposium on foundations of computer science (FOCS), pp. 633-638. IEEE, 2017.
Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Ng. Efficient sparse coding algorithms. Advances in Neural Information Processing Systems (NeurIPS), 19, 2006.
Meir Yossef Levi and Guy Gilboa. The double-ellipsoid geometry of CLIP. In Forty-second International Conference on Machine Learning, 2025, URL https://openreviiww net/f/wrwne////mr id=QGUju9B682.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap multi-modal contrastive representation learning. Advances in Neural Information Processing Systems (NeurIPS), 35:1 17622-17625, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. Proceedings of the IEEE European Conference on Computer Vision (ECCV), pp. 740-755, 2014.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Scholkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. Proceedings of the International Conference on Machine Learning (ICML), pp. 4114-4124, 2019.
Julien Mairal, Francis Bach, Jean Ponce, et al. Sparse modeling for image and vision processing. Foundations and Trends@ in Computer Graphics and Vision, 8(2-3):85-22) 2233, 2014.
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Stéphane G Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions on signal processing, 41(12):339734115, 1993.
Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances in Neural Information Processing Systems (NeurIPS), 34:26422-2644222 2021.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, (816633))66076609 1996.
Bruno A Olshausen and David Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):331-3322 1997.
Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata. Sparse autoencoders learn monosemantic features in vision-language models. ArXiv e-print, 2025.
Isabel Papadimitriou, Huangyuan Su, Thomas Fel, Sham Kakade, and Stephanie Gil. Interpreting the linear structure of vision-language model embedding spaces. ArXiv e-print, 2025.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. Proceedings of the British Machine Vision Conference (BMVC), pp. 151, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PmLR, 2021.
13

---

Published as a conference paper at ICLR 2026
Senthooran Rajamanobaran, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, Janos Kramar, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. ArXiv e-print, 2024.
Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. Diseover-then-name:: Task-agnostic concept bottlenecks via automated concept discovery. Proceedings of the IEEE European Confer- ence on Computer Vision (ECCV), pp. 444-461, 2024.
Ilias Rentzeperis, Luca Calatroni, Laurent U Perrinet, and Dario Prandi. Beyond sparse coding in v1. PLoS Computational Biology, 19 1459, 2023.
Ron Rubinstein, Alfred M Bruckstein, and Michael Elad. Dictionaries for sparse representation modeling. Proceedings the IEEE, 98(6): 1045:-00455557 2010.
Simon Schrodi, David T. Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevieet ew. net/forum? d=-AFFHCRR.A....
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of fllt-fitered 400 million image-text pairs. ArXiv e-print, 2021.
Peiyang Shi, Michael C Welle, Märten Björkman, and Danica Kragic. Towards understanding the modality gap clip. In ICLR 2023 workshop on multimodal representation learning: perks and pitfalls, 2023.
Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, Simon Alibert, Matthicu Cord, Thomas Wolf, and Remi Cadene. Smolvla: A vision-language-action model for affordable and efficient robotics. ArXiv e-print, 2025.
Karan Singhal, Shekoofch Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina
Blaise Agüiera Arcas, Dale Webster, Greg Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. Nature, 620(7972): 172-1888 2023.
Leon Sixt, Martin Schuessler, Oana-Iulian Popescu, Philipp Weiß, and Tim Landgraf. Do users ben- efit from interpretable vision? study, baseline, and dataset. Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. Proceedings of the International Conference on Machine Learning (ICML), 2017.
Yiyou Sun, Yifei Ming, Xiaojin Zhu and Yixuan Li. Out-of-distributin detection with deep nearest neighbors. Proceedings of the International Conference on Machine Learning (ICML), pp. 20827-20840, 2022.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. Proceed- ings of the International Conference on Machine Learning (ICML), pp. 3319-3328, 2017.
Viacheslav Surkov, Chris Wendler, Mikhail Terekhov, Justin Deschenaux, Robert West, and Caglar Gulcehre. Unpacking sdxl turbo: Interpreting text-to-image models with sparse autoencoders, In Mechanistic Interpretubility for Vision at CVPR 2025 (Non-proceedings Track), 2025.
Ivana Tošic and Pascal Frossard. Dictionary learning. IEEE Signal Processing Magazine, 28(2):
27-38, 2011.
14

---

Published as a conference paper at ICLR 2026
Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Nacem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-languagg encoders with improved semantic understanding, localization, and dense features. ArXiv e-print, 2025.
Vishaal Udandarao. Understanding and fixing the modality gap in vision-language models. Master's thesis, University of Cambridge, 32, 2022.
Martin Wattenberg and Fernanda Viégas. Relational composition in neural networks: A survey and call to action. JCML 2024 Workshop on Mechanistic Interpretability. 2024.
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Roge- rio Feris. Fashion iq: A new dataset towards retrieving images by natural language feedback. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
11307-11317, 2021.
Jingkang Yang. Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, et al. Openood: Benchmarking generalized out- of-distribution detection. Advances in Neural Information Processing Systems (NeurIPS), 35: 2022.
Can Yaras, Siyi Chen, Peng Wang, and Qing Qu. Explaining and mitigating the modality gap in contrastive multimodal learning. ArXiv e-print, 2024.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. Proceed- ings of the IEEE European Conference on Computer Vision (ECCV), pp. 818-833, 2014.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. Proceedings of the IEEE International Conference Computer Vision (ICCV), pp. 11975-11986, 2023.
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible concept- explanations for enn models with non-negative concept activation vectors. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 35(13):1 1682-1 1690, 2021.
Yuhui Zhang, Jeff Z HaoChen, Shih- Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Diagnosing and rectifying vision models using language. In International Conference on Learning Representations (ICIR), 2023. URL https: /ttp:///p//ee///eeeeeeeeeeee evrirveviiwi httpt/pddEP-------TTETBBBBG
Xingcheng Zhou, Mingyu Liu, Ekim Yurtsever, Bare Luka Zagar, Walter Zimmer, Hu Cao, and Alois C Knoll. Vision language models in autonomous driving: A survey and outlook. IEEE Transactions Intelligent Vehicles, 2024.
15


Reviews:
[{"review_id": "P2FGxaMJlL", "reviewer": "Reviewer_U7cA", "summary": "This paper proposes an Iso-Energy prior for learning aligned sparse concept dictionaries on top of VLM embeddings. By mildly enforcing equal second-moment (“energy”) of a concept across image/text domains, the aligned SAE separates bimodal atoms (semantic carriers) from unimodal atoms (modality-specific bias). This yields two actionable interventions: (i) closing the modality gap by masking unimodal atoms without hurting retrieval, and (ii) performing robust semantic vector arithmetic within the bimodal subspace, reducing OOD drift.", "questions": "1. To what extent do the conclusions generalize to more complex tasks and architectures, such as VQA on LLaVA-series models?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Clear and effective framing of a testable modeling intuition.\nThe paper presents a well-motivated and conceptually coherent formulation. It articulates a precise inductive bias: that shared cross-modal concepts should exhibit similar activation statistics across modalities. This idea is not only intuitively appealing but also operationalized in a mathematically minimal way through second-moment constraints. The writing and structural clarity further reinforce this framing, making the contribution accessible and theoretically grounded.\n\n2. Methodologically grounded execution with dual functionality.\nThe proposed method delivers more than conceptual framing. It constructs a sparse, interpretable bimodal subspace that supports both analysis and intervention. The same subspace allows for attribution-style interpretation as well as semantically coherent editing, demonstrating that the learned structure is not only intelligible but also functionally controllable. This dual capacity is rarely achieved in the interpretability literature and gives the method both analytical and practical value.", "weaknesses": "1. Sufficiency versus necessity of the Iso-Energy criterion.\nEqual second moments across modalities can indicate shared concepts, but they are not required. Without invariance to modality-specific anisotropy or rescaling, genuinely shared factors may be labeled unimodal. It would be better to add invariance controls such as per-modality whitening or variance normalization, and to compare with covariance-aware baselines such as CCA or CORAL to verify that the findings are not driven by marginal variance.\n\n2. Sensitivity to pairing noise and frequency imbalance.\nThe alignment term relies on paired image and text data, where long-tail frequencies and noisy matches are common. Energy equality can be confounded by corpus artifacts rather than semantics. It would be better to add two controls: a frequency-matched subsample that balances concept prevalence across modalities, and a shuffled-pairs stress test to quantify robustness to misalignment noise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q6n5sx19o3", "reviewer": "Reviewer_Yfxj", "summary": "The paper studies the geometry of vision–language embeddings through a proposed Iso-Energy assumption, which states that shared cross-modal concepts should have equal activation energy across modalities. To explore this, the authors introduce an aligned sparse autoencoder (SAE-A) that adds a cosine-similarity–based alignment loss to a standard sparse autoencoder. The numerical experiments on CLIP, OpenCLIP, and SigLIP embeddings show that the aligned SAE could improve cross-modal alignment metrics while maintaining reconstruction quality.", "questions": "1. Can the authors clarify the precise theoretical link between the Iso-Energy Assumption in Definition 2 and the cosine-similarity–based alignment loss in Equation (1)? \n\n2. As the alignment loss in (1) reduces to a simple sum of cosine similarities, did the authors experiment with other similar regularizers (e.g., the sum of the squared or absolute value of the inner products in (1)) or other regularizers that can more directly enforce the Iso-Energy property in Definition 2?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper provides an interesting perspective on the geometry of vision–language embeddings by introducing the Iso-Energy assumption.\n\n2. The numerical results are consistent, showing that the aligned SAE can improve cross-modal alignment metrics without damaging reconstruction quality.", "weaknesses": "1. The connection between the Iso-Energy Assumption in Definition 2 and the implemented loss in Equation (1) is not that clear. Definition 2 describes a population-level equality of per-coordinate activation energies across modalities, whereas the alignment loss in (1) simply quantifies the batch-level sum of cosine similarity between sample codes. The paper does not provide a derivation or justification showing that this cosine similarity sum term directly enforces or meaningfully approximates the Iso-Energy property.\n\n2. The alignment loss in Equation (1) effectively reduces to a vanilla sum of cosine similarities between the latent codes from two modalities. This formulation looks too simple and somewhat ad hoc, lacking a clear connection to encourage equalized energy statistics as defined by the Iso-Energy assumption.\n\n3. The paper introduces the aligned sparse autoencoder without providing sufficient background on the baseline SAE formulation, its reconstruction, and sparsity terms. This makes the method less self-contained and more difficult for readers less familiar with the SAE framework to follow.\n\n4. Some of the mathematical definitions, particularly in Definition 2, are not presented rigorously. The conditional expectation is written as if conditioned on the specific sample $X$, which collapses the expectation to the outcome for that given value of $X$ in the conditional expectation of (1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vXdTMs94vx", "reviewer": "Reviewer_BMn2", "summary": "This paper studies the geometry of VLM embedding spaces via the Iso‑Energy Assumption—shared concepts should have domain‑invariant average squared activation. The authors train an Aligned Matching‑Pursuit SAE with a small cross‑modal alignment regularizer, yielding a dictionary that separates bimodal atoms (which carry all cross‑modal alignment) from unimodal atoms (a few high‑energy modality‑specific biases explaining the modality gap). On synthetic data and CLIP/OpenCLIP/SigLIP variants, this preserves reconstruction while markedly improving multimodality metrics, and enables interventions such as removing unimodal atoms to close the modality gap without hurting retrieval and performing in‑distribution semantic arithmetic restricted to the bimodal subspace.", "questions": "**External Validation of Atomic Concepts:** While visualizations are provided, the \"semantic stability\" of the atoms is largely qualitative. Is it possible to introduce quantitative measures for concept purity, namability, or alignment with human annotations to further validate the interpretability and meaningfulness of the identified bi-modal and uni-modal atoms? This would provide stronger evidence that the method is indeed recovering genuine, human-understandable concepts.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  **Clear Problem Formulation and Strong Motivation:** The paper articulates a pertinent and significant problem in VLM interpretability and manipulability. By focusing on the geometric underpinnings of cross-modal alignment and the \"modality gap,\" the work addresses a critical area for improving VLM transparency and control. \n2.  **Novel and Intuitive Hypothesis:** The \"Iso-Energy Hypothesis\" offers an elegant and interpretable statistical prior for identifying shared concepts within a sparse dictionary. This hypothesis provides a concrete, measurable criterion that transforms the abstract notion of \"cross-modal redundancy\" into an actionable constraint for dictionary learning.\n3.  **Demonstrated Practical Interventions:** The ability to close the modality gap by masking uni-modal atoms and to perform \"in-distribution\" semantic arithmetic within the bi-modal subspace represents a significant practical contribution. These interventions offer concrete pathways for improving the robustness and interpretability of VLM applications.", "weaknesses": "1.  **Reliance on Paired Data for Alignment Regularization:** Although lines 158-160 allude to the potential of leveraging \"cross-modal redundancy alone,\" the current formulation of the alignment regularizer explicitly requires instance-level image-text pairs. The robustness of the method to noisy or imperfect pairings, or its applicability in settings with weak or no explicit pairings (e.g., using only domain labels), remains unexplored. This dependency may limit its generality and practical scope.\n2.  **Limited Assessment of Dictionary Stability and Generalizability:** While the paper aims to enhance SAE dictionary stability via the Iso-Energy Assumption and demonstrates improved recovery on synthetic data during \"Sanity check\", it lacks a systematic and multi-faceted analysis of this robustness on large-scale real-world VLM datasets. The reproducibility of the learned dictionary under varying conditions, such as different expansion ratios, sparsity targets, or subsets of training data, remains unexplored. Thus, the evaluation of this crucial aspect in practical scenarios is not yet comprehensive.\n3.  **Scope of Evaluation and Downstream Task Relevance:** While the paper demonstrates strong results on retrieval-oriented metrics and interventions, the generalizability to other VLM tasks (e.g., visual question answering, image generation, localization, counting, spatial reasoning) is not fully explored. The claim that \"masking uni-modal atoms does not hurt performance\" might hold for certain tasks, but could be detrimental for tasks that rely on more modality-specific information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DuOtipv5xc", "reviewer": "Reviewer_XysU", "summary": "The paper investigates the geometry of the embedding space of CLIP-like models using sparse autoencoders. The authors augment SAE training by an iso-energy regularization term that encourages SAE latents to have similar spreads (i.e., second moments) for both modalities (Def. 2). They show that only a small set of features explain the modality gap and the remaining features are sufficient for cross-modal alignment. Removing former features reduces modality gap while retaining performance and the latter allows for vector arithmetic.", "questions": "* Q1: Is the standard SAE (l. 176/177, 185) the MP-SAE or is it truly standard SAE?\n\n* Q2: How are unimodal or bimodal features separated?\n\n* Q3: Do the unimodal features approximate the modality gap vector? Related to that, does it explain why they all have such high cosine similarities (Fig. 16b)?\n\n* Q4: What is $\\mu$ in Fig. 2 left?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* S1: The energy penalty (Def. 2/ Eq. 1) is interesting and a simple addition to the SAE loss.\n\n* S2: Synthetic & real experiments confirm that the proposed aligned SAE better matches the geometry of CLIP-like models. Particularly, if the SAE features can distinguish between shared or modality-gap-specific.\n\n* S3: The paper introduced four metrics to evaluate whether the SAE variants capture the geometrical or functional properties of the VLMs.\n\n* S4: The proposed SAE allows semantic vector arithmetic.", "weaknesses": "* W1: 3 out of the 4 key findings have been reported in previous work (see bullet points below). While the findings are reached using a different, more complex approach, the current paper seems to re-report these findings.\n\t* Few (unimodal) features fully explain the modality gap (Fig. 2 left, 3) ~> see Fig. 4 in [3] or Fig. 3 in [4]\n\t* Bimodal features carry the entire cross-modal alignment signal (Fig. 2 right, Fig. 3) ~> cross-modality transferability experiments in [2], e.g., Tab. 2.\n\t* Removing those modality-gap features reduces the modality gap without loss of performance (Fig. 4) ~> again, see cross-modality experiments in [2], e.g., Tab. 2.\n\n* W2: The paper provides little to no experimental details in the main text, making it hard to understand the results without searching the supplemental.\n\n* W3: It is assumed that bimodal atoms are semantically aligned across modalities (“bimodal atoms encode the shared conceptual backbone” l. 345) and few qualitative examples are provided in Appendix G. However, there is no quantitative evaluation for this claim.\n\n* W4: Only contrastive models are evaluated. For example, the modality gap has been also observed in multimodal LLMs. It’d be important to include such results.\n\n## Comment\n\n* C1: I’d encourage the authors to include discussions on missing relevant literature [1-4].\n\n* C2: This work’s proposition 1 seems closely related to [2]’s proposition A.1. The only difference seems to be that the modality information can be adaptive here.\n\n* C3: The caption of Fig. 4 is partially occluded from Fig. 5.\n\n---\n\n[1] https://www.mlmi.eng.cam.ac.uk/files/2021-2022_dissertations/understanding_and_fixing_the_modality_gap_in_vision-language_models_reduced.pdf \n\n[2] https://openreview.net/forum?id=D-zfUK7BR6c \n\n[3] https://openreview.net/forum?id=uAFHCZRmXk\n\n[4] https://openreview.net/forum?id=QGUju9B68Z", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)