SPARC: Concept-Aligned Sparse Autoencoders for Cross-Model and Cross-Modal Interpretability
Authors: Ali Nasiri-Sarvi
Hassan Rivaz
Mahdi S. Hosseini
Venue: Accepted by TMLR ()

Abstract: Understanding how different AI models encode the same high-level concepts, such as objects or attributes, remains challenging because each model typically produces its own isolated representation. Existing interpretability methods like Sparse Autoencoders (SAEs) produce latent concepts individually for each model, resulting in incompatible concept spaces and limiting cross-model interpretability. To address this, we introduce SPARC (Sparse Autoencoders for Aligned Representation of Concepts), a new framework that learns a single, unified latent space shared across diverse architectures and modalities (e.g., vision models like DINO, and multimodal models like CLIP). SPARC's alignment is enforced through two key innovations: (1) a Global TopK sparsity mechanism, ensuring all input streams activate identical latent dimensions for a given concept; and (2) a Cross-Reconstruction Loss, which explicitly encourages semantic consistency between models. On Open Images, SPARC dramatically improves concept alignment, achieving a Jaccard similarity of 0.80, more than tripling the alignment compared to previous methods. SPARC creates a shared sparse latent space where individual dimensions often correspond to similar high-level concepts across models and modalities, enabling direct comparison of how different architectures represent identical concepts without requiring manual alignment or model-specific analysis. As a consequence of this aligned representation, SPARC also enables practical applications such as text-guided spatial localization in vision-only models and cross-model/cross-modal retrieval.

Published in Transactions on Machine Learning Research (3/2026)
SPARC: Concept-Aligned Sparse Autoencoders for Cross- Model and Cross-Modal Interpretability
Ali Nasiri-Sarvi
ali o...esrraaoiiemmiilccoooooooo...
Department of Computer Science and Software Engineering (CSSE)
Concordia University, Canada
Hassan Rivaz
Department of Electrical and Computer Engineering (ECE)
Concordia University, Canada
Mahdi S. Hosseini
machd.lasseenn@@@oonnndd.cc
Department of Computer Science and Software Engineering (CSSB)
Concordia Unéversity, Canada
Mila Quebec AT Institude
Reviewed on OpenReview: https: // openreview. net/ forum? id- IJfuoc20bZ
Abstract
Understanding how different AI models encode the same high-level concepts, such as objects or attributes, remains challenging because each model typically produces its own isolated representation. Existing interpretability methods like Sparse Autoencoders (SAFs) produce latent concepts individually for each model, resulting in incompatible concept spaces and limiting cross-model interpretability. To address this, introduce SPARC (Sparse Au- toencoders for Aligned Representation of Concepts), new framework that learns single, unified latent space shared across diverse architectures and modalities (e.g., vision models like DINO, and multimodal models like CLIP). SPARC's alignment is enforced through two key innovations: (1) a Global TopK sparsity mechanism, ensuring all input streams activate identical latent dimensions for given concept; and (2)
Reconstrrtttttninn Loss, which explicitly encourages semantic consistency between models. On Open Images, SPARC dra- matically improves concept alignment, achieving Jaccard similarity of 0.80, more than tripling the alignment compared to previous methods. SPARC creates shared sparse la- tent space where individual dimensions often correspond to similar high-level concepts across models and modalities, enabling direct comparison of how different architectures represent identical concepts without requiring manual alignment or model-spocific analysis. As a con- sequence of this aligned representation, SPARC also enables practical applications such as text-guided spatial localization in vision-only models and cross-modee/crros-mmdde retrieval. Code and models are available at httpt///....o/etaaaaaaaaaaaaaaaa
1 Introduction
As AI models rapidly grow in numbers, fundamental question emerges: do different archilectures, trained with different objectives and modalities, independently converge similar ways of representing the world? Answering this requires tools that can compare models directly. However, current interprotability methods, including powerful Sparse Autoencoders (SAEs) Bricken et al. (2023); Huben et al. (2024), are designed to analyze models in isolation. While effective, this approach creates isolated concept. spaces unique to each model, making direct comparison difficult.
Recent work has begun addressing cross-model interpretability more directly. Universal Sparse Autoencoders (USAE) Thasarathan al. 2025) introduced the concept of training single sparse dictionary across

---

Published in Transactions on Machine Learning Research (3/2026)
Original
DIN (SPARC)
CLIP Image (SPARC)
CLPP Text (SPARC)
CLIP Simiarity
10000
Concept Aligned Interpretable
BPA
10,00
10001
CLIP Image
SPARC
CLIP Text
X0-00
Figure 1: SPARC enables consistent concept visualization across models and modalities using shared latent dimensions. The figure shows how individual concept-specific latents (bus, cat, balloons) produce coherent spatial heatmaps across DINO Oquab et al. (0024 and CLIP Radford et al. (0221) vision encoders, while also DINO(Oquab
generating meaningful text attribution scores in CLIP's text encoder when processing full image captions. For comparison, standard CLIP similarity uses only concept names ('a bus", 'a cat", "balloons') rather than full captions, as CLIP produces diffused and scattered heatmaps when given complex captions. SPARC's concept-specific latents can handle full captions while maintaining focused attribution on the target concept.
multiple vision models, using random encoder selection during training to balance computational costs. At each training iteration, USAF randomly selects one model's encoder to produce shared concept activations, then reconstructs all models' features using their respective learned decoders. This approach demonstrated the fcasibility of learning shared representations and revealed interesting patterns in how different models encode visual concepts, representing an important step forward cross mroel interpretability.
However, USAE faces several inherent limitations for cross- cross-model interpretability. Methodologically, USAE's training approach randomly selects one model's encoder per iteration while reconstructing all models, which can lead to training instabilities and uneven concept learning across architectures. The method lacks explicit mechanisms to ensure consistent activation patterns across models, potentially allowing latent dimensions to activate differently across architectures without constraint. Additionally, USAE's focus on vision-only models limits its applicability to modern multimodal AT systems. From an evaluation perspective, while USAE demonstrates statistical co-activation and reconstruction fidelity, the fundamental challenge remains in validating whether these statistically aligned activations represent gemuinely equivalent semantic concepts across different architectures, rather than merely correlated but distinct representations.
Building on these insights, we introduce SPARC (Sparse Autoencoders for Aligned Representation of Concepts), a method designed to achieve concept alignment across heterogencous models and modalities while dramatically reducing the manual analysis burden that limits current approaches. By learning a sin- gle interpretable latent space that works across multiple models simultancously, SPARC enables experts analyze concept represenlations once rather than repeatedly for each architecture, directly addressing the scalability challenges that motivated this work.
SPARC achieves these objectives through two key methodological innovations: (1) Global TopK sparse activation mechanism that enforces identical activation patlerns across all input streams, directly addressing the dead latent problem by ensuring every dimension either activates across all models or remains inactive across all models; and (2) Cross-Reconstruuction Loss objective that promotes semantic consistency by training each model's latent representation to reconstruct inputs from other models, creating optimization pressure toward shared semantic understanding rather than mere statistical correlation.
2

---

Published in Transactions on Machine Learning Research (3/2026)
Our evaluation framework validates these aligned representations through multiple complementary ap- proaches: (1) semantic concept alignment measurement; (2) practical utility demonstration through down- stream applications including cross-modal attribution, semantic segmentation, and retrieval tasks; and (3) systematic dead neuron quantification across models.
SPARC's design is highly effective, achieving a concept alignment. Jaccard similarity of 0.80 on Open Images. In direct comparison using the same evaluation protocol, USAE achieves only 0.22 (Section 1.2p. This gap reflects key architectural difference: USAE relics soft alignment through reconstruction without an explicit constraint on which latent indices activate, allowing different streams to use different subsets of latents for the same input. SPARC's Global TopK, by contrast, enforces identical active indices across all streams, ensuring each latent dimension either activates consistently across models or remains inactive across all of them. This creates unified latent space where a single dimension consistently represents the same concept across models like DINO and CLIP, and even across vision and text modalities, as demonstrated in Figure 1
2 Related Work
Understanding what neural networks learn and how to describe those latent mechanisms in human terms has become central challenge for interpretable and trustworthy Al. Early interpretability efforts uncovered neurons that align with human concepts (e.g., textures, object parts) in vision models (Bau et al. (2017 2019h) and language models (Karpathy et. al. (2015); Radford et al. (2017)), but later work revealed that individual units are often polysemantic, entangling many unrelated features within the same activation Gooh et al. 20021) Elhage et al. (2022)
Representation engineering (reats linear directions in model's activation space as semantic units that can be read edited Zou et al.
Zou
(2023). Reading tools include TCAV, which measures a concept vector's influence
predictions (Kim et al. (0188)) ACE, which discovers such vectors by clustering activations Ghorbani ct al. [2019); and concept-bottlenee models that expose an explicit concept layer for inspection and override Koh et al. (2020). Steering tools shift activations along learned directions, as in Plug-and-Pla, Language Models for controlled text generation Dathathri et al. (2020) and InterFaceGAN for semantic image edits Shen et al. 22020). Unsupervised work has also uncovered latent knowledge vectors that reveal and toggle factual beliefs in language models Burns et al. (2023).
While representation engineering works with model's cxisting geometry, sparse autoencoders learn new, disentangled basis first. Dictionary-learning SAEs with an I penalty were initially applied to transformer language models, revealing tens of thousands of single-concept directions in GPT-2 activations Bricken et al. (023); follow-up work showed that the same technique scales to larger language models and yields higher automated interpretability scores across layers Huben et al. (2024). Parallel studies vision apply sparse autoencoders to convolutional and ViT feature maps, uncovering units aligned with colors, textures, and object parts, and enabling direct feature-level editing in images Lim et al. (2024). A separate line of work abandons the 4 penalty in favor of the hard TopK selection rule introduced by k-Sparse Autoencoders Makhzani & Frey (2013); scaling this constraint to billion-paramete networks further improves feature purity and removes dead units et al. (2025). Building on this, BatchTTopK Sparse Autoencoders propose relaxing the hard sparsity constraint from a per-sample to a per-batch basis, which allows for a variable number of active latents per sample and enables the model to adaptively allocate more features to more complex inputs Busssann al. (2024). Together, these results establish sparsity as a reliable way to manufacture monosemantic features.
Rosetta Neurons identifies neurons in vision backbones that respond to the same visual patterns Dravid et al. 2023 and Rosetta Concepts clusters semantically aligned features in transformer video models Kowal et al. 2024 Kondapaneni et al. (0225) compares models by aligning interpretable concept masks, while Dominici et al. 2023) builds a manifold of concepts across modalities by embedding direction vectors from each stra into a shared space. Most of these techniques operate post-hoc or depend on modality-speciff heuristics.
Universal Sparse Autoencoders (USAE) take a more general approach, learning a single sparse dictionary that reconstruct multiple vision models at once, revealing a common pool of interpretable features
3

---

Published in Transactions on Machine Learning Research (3/2026)
Thasarathan al. (2025). However, USAEs rely random encoder sampling during training and soft alignment through their reconstruction objective, which can lead to inconsistent concept activation across models. SPARC addresses these limitations through two key innovations: (1) Global Topk mechanism that enforces hard structural alignment by ensuring all models activate identical latent dimensions for the same inpuls, and (2) a systematic cross-reconstruction objective that combines self-reconsttuutiion and cross- roconstruction losses rather than random sampling. We demonstrate SPARC's power by learning unified concept representations that operate both cross-model (across different vision models) and cross-modal (bridging vision and language), revealing how the same semantic concepts manifest consistently across diverse model types and modalities.
3
Methodology
This section presents our methodology for SPARC: SParse Autoencoder for Aligned Representation of Concepts. We first define the problem sctup and core objectives: faithfulness, interprctability, and cross- stream concept alignment (Section 3.1 We then detail the SPARC architecture, highlighting the novel Global TopK activation mechanism that enables concept alignment (Section 3.2). Finally, we formulate the training objective that optimizes the model toward these goals (Section 3.3
3.1 Problem Setup and Objectives
We consider multiple (M) distinct streams of information, indexed by the set S = (81,
.M). .2.....888! The core assumption is that we process data samples with multiple related representations of the same underlying entity features from an image extracted using both CLIP-image Radford et al. (2021) and DINO Oquab ett al. (2024) models, or imagc-caption pairs passed DINO and CL.IP-tett ). For each sample, we obtain set of corresponding feature vectors (x"),s, where cach x* E Rdd is produced by stream s's feature extractionn process with dimensionality dg.
These streams represent heterogeneous processing pathways; they may use different architectures, training objectives (like CLIP vs DINO), or process different modalities associated with the input (like image vs lext). Consequently, our framework must. accommodate varying input feature dimensionalities (d, # de for s # is permissible).
We aim develop interpretable representations for each input stream while ensuring consistent cross-stream semantic alignment. Given different feature representations (x*)xes of the same underlying data sample, the SPARC model maps these inputs into concept-aligned sparse latent representations {z"}ees These latent representations z" share common L-dimensional space, z" R&,, maintain sparsity ||z*|lo < L, and crucially, activate the same semantic dimensions across different streams when processing the same underlying content.
The design and learning of the SPARC are guided by three key objectives:
Faithfulness:
The model aims to accurately reconstruct the original features for each stream from its latent representation. Specifically, the reconstruction X should closely approximate the original input x", ensuring the learned interpretable model captures the information of the upstream features.
2. Interpretability: The enforced sparsity ([|2* lo T.) is designed to promote monosemanticity, where individual latent dimensions ideally capture distinct and human-understandable concepts, reducing semantic entanglement (Olah et al. 2020).
3. Concept Alignment: Tdeally, the interpretation, or the underlying concept represented by each latent dimension j E {1...., [1....,],; L}, should remain consistent across all input streams s E S. When dimension j represents a certain concept when activated via stream 8, it should represent the same concept when activated via stream This concept alignment leverages the shared origin of the input fcatures (x* Jes, they derive from the same underlying data sample.
The subsequent sections detail the SPARC architecture (Section 3.2 and training objective (Section 3.3 specifically designed to pursue these three objectives.
4

---

Published in Transactions on Machine Learning Research (3/2026)
3.2 SPARC Architecture
The SPARC architecture processes multiple imput streams through stream-specife encoders and decoders, coupled by shared sparse activation mechanism that promotes concept alignment. An overview of SPARC is presented in Figure
SPARC
imgg batom,
A enccee adviiiee
the opanso lon sane instoss scioss sraons
Self Loss
CL
2
grm vorrsiudi
background, see dloth""
objezt,
Gross Loss
Glotel Topk
Salf Loss
ReLU
23-1
Loss
Drss
Cross Loss
Indions
Rel u
DIVO (Cocoo
Cross Loss
Self Loss
gutter
ReLU
Figure 2: Detailed architecture of the SPARC model as well the Global TopK mechanism.
Stream Encoders. Each input feature vector x* Rd is processed by its corresponding stream-specific encoder F, Rd, + R4. The encoder performs affine transformation to map the input features to the L-dimensional latent space, producing pre-activation logits h". This involves centering the input using a learnable pre-bias bize E 1Rd,, applying a linear transformation with weights W E RLxd, and adding a learnable latent bias bia E R.L:
h'=E,(x' W (x"
by + bjat
(1)
Global TopK Sparse Activation. SPARC employs sparse activation mechanism to jointly satisfy Interpretability and Concept Alignment. Rather than applying sparsity independently to each stream's logits h", we use a Global TopK approach that enforces shared feature activation across streams.
Logits are first aggregated across all streams:
hags Σ h
(2)
The TopK indices are then selected using this aggregated logit haggi
Iglobal = HaapK(aags" k)
(3)
where lopk(;; k) returns the indices of the top k elements.
This shared index set Iglobal is then used to construct sparse latent representations for each stream. For every stream s, we select the corresponding values from h" the indices Iglobal, apply a RcLU activation, and zero out the rest:
z ReL.Ugatheer(h Iglobal))
where gather(h". Iglobal) keeps values from h" at positions in Iglobal and zeros elsewhere. This ensures both sparsity (122 & k) and concept alignment by forcing identical latent features to activate across all streams
5

---

Published in Transactions on Machine Learning Research (3/2026)
for the same underlying data sample, unlkke standard per-stream TopK approaches where each stream selects topk(h", k) independently.
It is important to note that while Global TopK enforces a shared support (identical active indices) across streams, we maintain separate latent vectors z rather than collapsing them into a single shared 2. This design reflects that different feature spaces may require different. activation magnitudes on the same latent dimension for faithful reconstruction and allows direct comparison of how strongly each model activates given concept.
Stream Decoders. Finally, each stream-specific decoder D, R - RRd, maps the sparse latent represen- tation z back to the original feature space of stream The decoder applies a linear transformation using weights W E R4.xL and adds back the stream's pre-bias beee (which was subtracted by the encoder) to produce the reconstruction x*:
x x== Wiz + ber
(5)
We note that all encoder and decoder mappings are affine (linear transformation plus bias). This design follows common practice in sparse autoencoders and linear probes, where late-layer representations of large pretrained models such as CLIP and DINO have been shown to support strong linear readouts. Exploring shallow nonlinear mappings (e.g., two-layer MLPs) a natural extension that we leave for future work.
3.3 Training Objective
The objective combines the two components with a weighting factor X:
Crotal Lself + XCcross
Σ
(x Σ (
(6)
SES
5,ESS sift
Here, Dz(z)) is the self-reconstruction of stream 8, while Dz(z*) is the cross-ecconslruction of stream L's input using the latent code from stream s.
While Global TopK activation structurally enforces shared activation patterns, the rronssrccontttttiion loss provides complementary semantic alignment that ensures the meaning of those activations is transferable between streams. This dual approach to concept alignment combines a hard structural constraint (which set of neurons to activate) with a soft semantic constraint (what information those neurons encode).
4 Evaluation
The goal of SPARC is interpretable neuron alignment where individual latent dimensions represent consistent and monosemantic concepts across different input streams. We measure SPARC quality through concept alignment analysis, reconstruction fidelity assessment, and probe loss evaluation. Throughout this section, we compare SPARC (Global TopK with cross-reconsstuction against two baselines: (1) Local TopK, which applies the TopK operation independently (2) USAE (Thasarathan et al.] 2025), a prior cross-model SAE method that uses random encoder selection during training. We also ablate the effect of the eross-eccontttuction loss by varying À {0, 1).
each stream's logits h" rather than to the aggregated logits, and
4.1 Latent Activation Alignment
Figure 3 shows neuron 6463's lop-activating images across four training configuratious. In Local TopK with x = 1 (top right), CLIP-text stream shows no activations (dead neuron) while DINO and CLIP-image streams activate on different image types. In contrast, Global TopK with X = 1 (bottom right) shows consistent activations across all three streams for the same object type.
We find this pattern to be common where without Global TopK, one or two streams have dead latents while others remain active. We quantified this pattern across all 8,192 latent dimensions (Table I Local TopK with À produces 48.8% mixed activation patterns where only 2/3 streams are active, creating alignment failures. Global TopK with X achieves 84.4% all-alive neurons with consistent. cross-stream

---

Published in Transactions on Machine Learning Research (3/2026)
Lod
Beve CCP
Figure 3: Top-activating samples for the latent dimension 6463 across three streams (DINO, CLIP-img, CLIP-txt) under four SPARC configurations. Each row shows top-10 images that activate the latent. The CLIP-text stream shows no activations under Local TopK with À = 1 (top right) due to dead neuron, demonstrating that cross-reconstruction loss alone is insufficient to ensure consistent. activation patterns across streams, In contrast, Global TopK with Ả = (bottom right) shows consistent activations across all three streams for the same object type.
activation and 0% cases of partial activation. This means with Global TopK with À 1, all neurons are either active across all streams dead across all streams. We also report per-stream percentages of dead neurons, showing that with Local TopK with À 1, almost half of the CLIP-text neurons are dead. With Global TopK with À = 1, dead neurons are distributed equally across all streams.
As an external baseline, USAE shows only 45.3% all-alive neurons, with the majority of latents in mixed 1/3 or 2/3 patterns (17.7% and 33.6%) and markedly uneven dead-neuron rates across streams (31.0% for CI, 39.1% for CT, and 9.3% for D), indicating substantially lower cross-stream activation consistency and highly imbalanced dead-neuron distribution.
Neuron activation patterns and stream-specifie dead neuron rates across 8192 latent dimensions. Mixed patterns indicate partial cross-stream alignment where only 1/3 or 2/3 of the streams are active for the same latent. Cl = CLIP-image, CT = CLIP-text, D = DINO.
Table 1:
Activation Patterns
Dead Neurons
Method
TopK
A
All Dead
1/3
2/3
All Alive
CI
CT
D
SPARC (ours)
Global
1.0
15.6%
0.0%
0.0%
84.4%
15.6%
15.6%
15.6%
Ablation
Global
0.0
1.6%
0.0%
0.2%
98.2%
1.6%
1.8%
1.6%
Ablation
Local
1.0
0.0%
7.6%
48.8%
43.6%
16.5%
45.0%
2.4%
Ablation
Local
0.0
0.0%
0.0%
14.3%
85.7%
0.3%
14.1%
0.0%
USAE
3.5%
17.7%
33.6%
45.3%
31.0%
39.1%
9.3%
Additional latent visualizations are provided in Appendix D
4.2 Concept Alignment
To evaluate concept consistency, we test if single latent dimension represents the same high-level concept across all input streams. Although assigning discrete labels to neurons has inherent limitations as monose- mantic features may not align perfectly with dataset labels this analysis provides a systematic framework for measuring concept alignment.
Our procedure first constructs quantitative proxy for each latent's meaning. For every latent dimension and stream 8, we create "concept profile" by aggregating the ground-truth labels from the 50 test images
7

---

Published in Transactions on Machine Learning Research (3/2026)
that yield the highest activations (zf). This produces label frequency vector for each (latent, stream) pair. We then measure the aligmment. between two streams (s1, for the same latent by computing the generalized Jaccard similarity of their concept profiles, a and b:
Ee miinaae,ue be) Ee max((arbb))
J(;; 8,)
In this formula, ag and be are the frequency counts for a concept within each profile. The resulting score quantifies the conceptual overlap, ranging from 0 (no overlap) to 1 (identical concept profiles).
Mean Jaccard similarity of concept profiles using fine-grained labels at hierarchy depth 5 of Open Images. We report mean ju and standard error (SE) over 8,192 latents.
Table 2:
The full SPARC model (Global TopK, X = 1) shows substantially higher concept alignment within this evaluation framework.
Method
TopK
A
Jaccard (μ + SE)
SPARC (ours)
Global
1.0
0.8018 + 0.0039
Ablation
Global
0.0
0.7344 + 0.0015
Ablation
Local
1.0
0.2599 + 0.0026
Ablation
Local
0.0
0.1651
0.0010
USAE
0.2166 + 0.0024
Table [ reports the mean Jaccard similarity averaged over all latents and stream pairs. The results show a stark contrast: the full SPARC model (Global TopK, A= 1) achieves an alignment score of 0.80, whereas ablations perform poorly. Using Local TopK drops the score to 0.26, and removing the cross-rcconttruction loss (X 0) degrades it further. For comparison, USAE attains Jaccard score of 0.22, closely matching our Local TopK regimes and remaining far below SPARC with Global TopK and cross-reconstruction This provides strong quantitative evidence that both Global TopK and cross-reconstruction are critical for learning a robustly aligned concept space.
4.3 Label Purity: Interpretability Comparison with USAE
Beyond cross-stream alignment, we also ask whether individual latent dimensions concentrate their strongest activations on semantically coherent image sets. Following a simple label-purity protocol, we measure, for each latent and stream, how often the most frequent Open Images label appears among the top-N most- activating examples for that latent.
Coneretely, for each latent and stream s, we: (i) rank test images by activation magnitude 27; (ii) take the top-N examples (we vary N E {10, 25, 50, 100} and also consider using all samples): (iii) compute the fraction of these images that contain the most frequent Open Images class label in that subset. Because Open Images is multi-label, we normalize by the number of images, not the number of label instances. We then average this fraction over all active latents to obtain a mean label-purity score and report mean + standard error (SE).
Table compares SPARC (Global TopK, A = against USAE, using the official USAE implementation adapted to our Open Images feature pipeline. Across all choices of N and all three streams, SPARC consis- tently achieves higher label purity, indicating that its latents not only align across models but also respond more selectively to coherent semantic concepts within each model.
While label purity is a useful sanity check, it is also limited by the quality and granularity of dalaset labels. Open Images annotations are multi-label, incomplete, and not guaranteed to align cleanly with the monosemantic features the SAE discovers, so high purity does not imply a perfect concopt neuron, and low purity does not prove the opposite. We therefore treat Table 3 primarily as a relative comparison between SPARC and USAE under the same labeling noise, rather than absolute measure of interpretability.
8

---

Published in Transactions on Machine Learning Research (3/2026)
Label purity on Open Images for SPARC (Global TopK, = 1) vs. USAF. Entries are mean standard error over active latents.
Table 3:
# Top Images
Stream
SPARC (Global+ Cross)
USAE
CLlP--mage
0.7950 0.0025
0.6014 + 0.0032
10
CLIP-Text
0.7256 + 0.0026
0.5479 + 0.0035
DINO
0.7960 + 0.0024
0.6426 = 0.0029
CLIP-Image
0.7416 + 0.0026
0.5525 + 0.0032
25
CLIP-Text
0.6825 0.0025
0.4941 = 0.0035
DINO
0.7403 + 0.0025
0.5862 = 0.0029
CLIP-Image
0.6950 +: 0.0026
0.5235 + 0.0033
50
CLIP-Text
0.6495 + 0.0025
0.4681 = 0.0035
DINO
0.6892 + 0.0026
0.5427 = 0.0029
CLIP-Image
0.6395 0.0026
0.4947 + 0.0032
100
CLIP--Txt
0.6121 + 0.0025
0.4468 + 0.0034
DINO
0.6298 + 0.0025
0.4960 + 0.0029
CLIP-Image
0.5300 0.0024
0.4203 + 0.0032
All
CLIP- -ext
0.5256 + 0.0024
0.4018 + 0.0032
DINO
0.5275 + 0.0024
0.4201 = 0.0028
4.4
Ra Score for Cross-Model and Cross- reconstruction
We quantify reconstruction quality with the coefficient of determination R² Wright (1921) for each ordered pair of streams (s t). Given SAE codes 2," from stream s and target decoder De. the reconstruction of
sample is 8 D=(z,)) Let x be the mean of the target features {x/ over the evaluation set E, i.., x 1 Eies x/. Then Re is defined as: [8]
Eiee ||x x
R² n²
Excellxf x x112
where R2- measures the fraction of the target stream's variance explained by the reconstruction. A value of R = indicates perfect reconstruction, =0 matchos the mcan prodictor, and R? <0 is worse than predicting the mean.
Table 4 reports RZ for self-reconstruction (diagonal terms) and cross-reconstruetion (off-diagonal terms),
with rows targets and columns sources. Across the three streams (CLIP-imag) CLIP-text, DINO), Global TopK yields consistently positive cross-stream R values (0.407-0.559) and balanced self- reconstruction scores (0.663/0.725) (0.699))) In contrast, Local TopK increases CT.IP self R²,, (0.732/0.874) but fails on cross-stream transfer, especially with DINO as target, where the values are negative (-0.391 from iLieeimage, -4.485 from CLIP-text) and the DINO self term is near the mean baseline (0.069). For comparison, USAE achieves moderate CLIP self- and cross-reconstruuctoon (self R of 0.506/0.616 and cross terms in the 0.389-0.477 range between CLIP-image and CLIP-text), but performs poorly on DINO: DINO self-reconstruction is weak (0.111), and cross-stream R R2 into DINO near-zero or slightly negative (0.017 from CLIP-imane -0.005 from CLIP-text).
Notably, when DINO is the source and CLIP is the target, R remains positive (0.456/0.304), indicat- ing that Pross-reconstrucce asymmetry could reflect differences in encoder/decon ics that preferentially aid the DINO->CLIP pathway; we do not attempt to disentangle these factors here. Empirically, Global TopK improves both directions and yields stable cross-stream and self reconstrcctionn outperforming both Local TopK and USAE on this three-stream setup.
into CLIP retains usable signal, whereas the reverse direction weak. This capacity, feature compatibility, or training dynam-
9

---

Published in Transactions on Machine Learning Research (3/2026)
Table 4: R2 (varianco-weighhedd; higher is better) for three streams. Rows are targets (t), columns are sources (s). Diagonals (self) bold; negative values indicate worse than predicting the mean.
Global TopK (source)
Local TopK (source)
USAE (source)
Original
Target clip_img
clip_txt
dino
clip_img clip_txt
dino
clip_ img
clip_txt
dino
clip_img
0.663
0.513
0.556
0.732
0.234
0.456
0.506
0.389
0.421
clip_txt
0.526
0.725
0.519
0.340
0.874
0.304
0.474
0.616
0.454
dino
0.559
0.407
0.690
-0.391
-4.485
0.069
0.017
-0.005
0.111
mecns
self = 0.693, cross 0.513
self = 0.558, cross
0.590
self = 0.411, cross = 0.292
The three-stream configuration used throughout this section enables detailed analysis of concept alignment and interpretability across vision and language modalities. Appendix C extends this evaluation to ten encoders spanning diverse architectures (including SigLIP, ViT, Swin, E5, GTE, and Qwen), demonstrating that Global Topk maintains stable cross-stream reconstruction across a broader range of models.
4.5 How well can 1d probes recover known concepts?
Following prior work Gao et al. 2202 ; Gurnee et al. (2023), we assess whether individual latent dimensions in SPARC encode recognizable semantic features. Specifically, we use the Open Images test set, which includes 601 binary classification labels. We filter retain only those labels with least 50 positive samples, resulting in 432 binary tasks.
For each task, apply 1D logistic probe over individual latent dimensions using the sparse values. The probe parameters are optimized to minimize cross-entrope loss:
min E [y log o (wz, + b) (1 y) log(1 a oww + b))], b,
where zi is the i-th latent dimension and is the sigmoid function. We report the best-performing dimension (i.e., the one with the lowest loss) per task and average this across all tasks for each configuration and stream, following the method used in Gao et al. 2025))
Table 5: Probe loss (lower is better) across 432 Open Images binary classification tasks, Values are mean + standard error over tasks.
Method
TopK Type
A
CLiP-Image
CLIP-Text
DINO
SPARC
Global
1.0
0.5354 = 0.0049
0.5646 + 0.0042
0.5409 + 0.0049
Ablation
Global
0.0
0.5336
0.0049
0.4942 = 0.0057
0.5194 = 0.0058
Ablation
Local
1.0
0.4990
0.0056
0.5363 = 0.0053
0.5170 = 0.0053
Ablation
Local
0.0
0.5238 + 0.0052
0.4904 + 0.0054
0.5265 + 0.0062
USAE
0.6372 + 0.0028
0.6470 + 0.0025
0.6494 + 0.0023
All rows in Table 5 report average probe losses below 0.69, which is the expected loss for a random binary classifier on balanced labels (i.e., when predictions are always 0.5 and labels are equally likely to be 0 or 1, the expectod binary cross-entropy is - log(0.5) = log(2) 2s 0.693). For both Global and Local TopK variants, probe losses tend to be lower when the cross-loss weight À set to zero. Compared under the same protocol, USAE yields substantially higher probe losses than any SPARC configuration across all three streams, indicating that its individual latents are less informative for recovering these labcled concepts.
This probing approach offers a simple and efficient way to assess whether certain labeled concepts are linearly recoverable from the latent space. However, relies on several assumptions, including that labeled attributes correspond to isolated latent directions. Probe loss differences do not necessarily reflect alignment quality, interpretability, or general usefulness of the representations, and we therefore use these results primarily relative comparison between SPARC, its ablations, and USAE under identical conditions.
10

---

Published in Transactions on Machine Learning Research (3/2026)
Full implementation details, including sampling strategy, candidate filtering, and classifier configuration, are provided in Appendix A.3
4.6 Semantic Segmentation via SPARC's Aligned Latents
This section tests whether SPARC's concept-aligned latents be used for semantic segmentation by serving scalar targets in gradient-based attribution methods. Gradient-based attribution methods like relevancy maps (Chefer et al. 2021) and GradCAM (Selvaraju et al. 2017 require scalar targets to compute VA Pharget for attribution computation.
8A
We find that SPARC's aligned latents can effectively serve as these scalar targets. While standard SAEs could also provide individual latent activations for attribution, SPARC's key advantage is aligument: the latent indices (e.g., latent 279) represent the same concepts across different streams, enabling systematic cross-modal analysis. Additionally, SPARC enables cross-modal dot products like z zdino z ixt that would be meaningless with separate SAEs due to incompatible latent spaces.
We explore two SPARC approaches and compare them against established baselines. First, we use concept- specific latent selections Ljer 2j as scalar targets, where J represents the set of latent indices relevant to the target concept. Second, we compute cross-modal similarities z² z' in the aligned latent space as scalar targets, enabling text-guided attribution in vision-only modcls.
Single Latent Attribution. We examine cases where J contains a single latent dimension, using 2j as scalar targets for attribution computation. Figure demonstrates this using 2279 (where = (279)) as the scalar target. The saliency maps show this same latent dimension responding to cat-related features across all streams: in the image (concentrated regions around the cat) and in the texi (highest relevance for "cat" token). We compare against CLIP similarity baseline, but use simplified prompts (e.g., 'a cat") for CLIP since its cross-modal similarity naturally responds to all concepts mentioned in complex captions, while SPARC's concept-specifie latent focuses solely on the target concept even when processing full captions.
Relevency Map (SPAC)
Relevancy Map
GredCAN
GradCAM
Relevancy Map
GradCAM
Original
CUP (SPARC)
SPARC CLIP Text Token Relevance Score
in 0.04 this 0.11
picture 0.13
0.04
can 0.08
observe 0.24
white 0.21
color 0.24
cat 1.00
0.09
the 0.06
floor 0.43
0.07
in 0.04
the 0.02
bottom 0.16
right 0.20
side 0.13
there 0.03 is 0.02
0.00
water 0.10
mark
0.35 the 0.01
background 0.32
is in white color
Figure 4: Individual latent attribution using SPARC dimension 279 vs. CLIP similarity baseline. (Above) Saliency maps show the same latent responding to cat-related features across image and text modalities. (Below) Text token relevance scores using SPARC and CLIP texl.
Additional qualitative results for single latent attribution can be found in Appendix
Cross-Modal Similarity Attribution. We examine cross-modal dot products z'-z' computed in SPARC's aligned latent space scalar targets. Figure 5 demonstrates this approach with dino ccipp_txt for DINO visualizations and clip__img, _lipp_sxt for SPARC's CLIP implementation, compared against the bascline CLIP similarity eliip img elip txt This enables texl-guided spatial atlention in vision-only models.
Additional visualizations for cross-modal similarity attribution can be found in Appendix F
Quantitative Semantic Segmentation. To quantify the heatmaps generated using SPARC's aligned latents, conduct weakly supervised semantic segmentation the MS COCO validation set (note that
11

---

Published in Transactions on Machine Learning Research (3/2026)
Relevency Map (SAACC
Relevancy Map Si
GradcAM
Original
ISPARC)
SPARC CLIP Text Token Relevance Score
CLIP Sim
Token Relevance Score
this
image
0.12
there
shoop 1.00
the in this image there
shoep
the grass
0.23
grass
0.03
surface
0.41
0.11
surface 0.12
Figure 5: Cross-modal similarity attribution comparing SPARC's aligned latent space against CLIP similarity baseline.
Both methods process the same image and caption, showing different attribution patterns.
the SPARC model was trained on the Open Images training set, not COCO). Since our text-based approach cannot distinguish between multiple instances of the same object class, we convert instance segmentation annotations to semantic segmentation by merging all instances of the same class into unified binary masks.
We follow the evaluation protocol of Chefer et al 22021). using IoU threshold of 0.2 and excluding small objects. All methods use Chefer et al. (021) for relevance map generation but differ in their similarity computation approaches. DETR uses attention maps from its encoder-decoder architecture, representing an upper bound of what Chefer et al. (2021) can achieve with a specialized object detection model. The CLIP baseline computes dot product similarity celip_ img elip txt directly the original feature space.
Table 6: Weakly supervised segmentation results on MS COCO comparing cross-modal similarity approaches. All methods use Chefer et al. (2021) for relevance map generation. SPARC methods compute similarities in the concept-aligned sparse latent space, while basclines and USAE operate in their respective Craturo/latent spaces
Evaluation follows the protocol from Table of Chefer al 2021 with IoU threshold of 0.2.
Method
Variant
TopK
AP
AP (M)
AP (L) AR
AR (M)
AR (L) mloU
Baselines
DETR CLIP Similarity
0.467
0.526 0.370
0.539
0.644
0.762
0.682
0.305
0.248
0.282
0.420
0.422
0.625
0.157
DINO Variants
Moes-Modal-Sim
USAE
0.126
0.118
0.166
0.253
0.139
0.415
0.100
Cross-Modal ModdalSimm
SPARC
Local
0.194
0,221
0.242
0.343
0.265
0.546
0.129
Cross-Modal-Sinn
SPARC
Global
0.222
0.253
0.274
0.373
0.297
0.581
0.143
Senrett-Latent-S55
USAF
0.108
0.089
0.140
0.224
0. 0.105
0.364
0.096
Concept-Tatent-Sun
SPARC
Local
0.212
0.243
0.251
0.349
0.283
0.518
0.136
Concept-Latent-Su
SPARC
Global
0.222
0.244
0.264
0.352
0.284
0.516
0.137
CLIP Veriants
Cross-Modal-Sin
USAE
0.004
0.019
0.004
0.022
0.021
0.029
0.023
Crose-Modal-Sin
SPARC
Local
0.176
0.242
0.204
0.304
0.266
0.472
0.102
Cross-Modal-Sim
SPARC
Global
0.223
0.283
0.262
0.369
0.319
0.569
0.138
Concept-Latent-S
USAE
0.006
0.027
0.007
0.027
0.028
0.036
0.035
Concept-Latent-Sun
SPARC
Local
0.203
0.238
0.234
0.325
0.258
0.474
0.123
Comeept-Latent-Surt
SPARC
Global
0.222
0.276
0.254
0.347
0.310
0.493
0.131
Our SPARC methods implement the Iwo approaches described earlier. Cross-ModallSimm compules cross- modal similarities in the aligned sparse latent space: zelip_txt, _xt dino for SPARC DINO and clip__te txt clip img for SPARC CLIP. Concopt-Latent--5
uses concept-specific latent selcctions Σjer 2 where J contains latents that activate frequently (250 times) for the target class. For each image and target class, we use the class name text. input. to generate the text representations. Table presents the quantitative results.
12

---

Published in Transactions on Machine Learning Research (3/2026)
The results reveal two key observations about SPARC's concept alignment. First, Global TopK consistently outperforms Local TopK across both backbones (DINO: 0.143 vs 0.129 mloU; CLIP: 0.138 vs 0.102 mloU), indicating that shared activation patterns produce more coherent cross-modal representations. Second, SPARC DINO Global achieves 0.143 mIoU compared to the CLIP baseline's 0.157 mIoU, demonstrating that texl-based spatial localization through a vision-only backbone can approach the performance of natively cross-modal similarity computation when operating in SPARC's aligned latent space. Compared under the same evaluation, USAE variants obtain substantially lower AP and mloU than all SPARC configurations (e.g., DINO-based USAE reaches only 0.096 mloU and CLIP-based USAE that their aligned latents are less effective as targets for weakly supervised segmentation in this setting.
mloU), indicating
5 Ablation Study
Unless noted otherwise, we train for 50 epochs with L=4096, k=64, X=1, and 7=10-4 All numbers are mean NMSE over the three streams; dashed lines are Global TopK, solid lines Local TopK.
5.1 Number latents
Figure shows the effect of scaling latent dimension L E {2048, 4096, 8192, 16384) while keeping sparsity level k and TopK type fixed. For oel-rrcconstuuctonn, Local TopK improves in most. cases with additional latents, while Global TopK exhibits higher reconstruction loss at each specific However, for cross-reconstruction, Global TopK demonstrates dramatic improvements when increasing the number of latents, whereas Local TopK shows much higher loss and actually degrades with additional latents. These patterns reveal that Global TopK's structural constraint comes at the cost of selfrecconstructinn but delivers substantial gains in cross-reconstruction.
MOSF
0.44
0.34
6:22
0.32
0.38
0.34
0.2
6.00
0.2
Number
K-16 (global)
<-32 Iglobal
K-64 (giobal)
K-128 (global)
- K-256 (giobal)
K-16 (lcaa))
K-64 (lccal)
K-128 (ocaa))
K-256
Figure 6: Self- and cross-reconstruucion NMSE vs. number of latents.
This tradeoff better quantified in Figure 7 which shows that Global TopK incurs a consistent self- reconstruction cost compared to Local TopK. The left panel reveals penalties of 0.030-0.060 NMSE across different configurations. However, the right panel demonstrates that these costs are more than compensated by substantial rrosssrccontttuiiion gains of 0.044-0.156 NMSE. For Global TopK, the cross-rcconstructiin benefits are consistently 2-3x larger than the sell-reconstruction penalties, indicating that the Global con- straint represents a favorable trade-off. This benefit ratio becomes even more pronounced when increasing the number of latents, particularly at higher sparsity levels where k-256 shows gains reaching 0.156 while costs remain below 0.047.
5.2 Sparsity
Figure g demonstratee the expected trado-off between sparsity and reconstruction quality as le varies from to 256. Note that our goal is higher sparsity (lower k), making this analysis particularly relevant for understanding performance under our target operating conditions.
13

---

Published in Transactions on Machine Learning Research (3/2026)
0.000
C-33
0110
C-224
Aug Aag KMGZ
(ioral NMPOCE
0.17
0,000
0.090
K-16
0035
H8H 8w130
K-122
0011
0352
4093
0352
22334
Namber Latents
Wurbbe of Letents
Figure 7: Global-vs-Local loss gap.
0.24
bormatoed MYC
0.32
0.2
0.22
638
Jutter
012
0.2
UU Sparsity (U6)
Litt
Spors (U01
Lotents=2040 lgloba)
Lotents:40966 (gicbel)
Latents=0119 (globel)
Lstents=15300 (gicbe)
Lo4ents-2048 llocal)
L26ents-4286 (local)
Latents-6132 (ocal)
1538 (lcca)
Figure 8: Effect of TopK sparsity.
For selfrreconstruction, both Global and Local TopK show consistent improvement with higher k, as expected when more features are available. However, cross-reconttruction reveals more complex pattern: Local TopK initially improves with higher & but then degrades, while Global TopK shows consistent improvement across the entire range. This suggests that without the structured constraint of global activation selection, Local Topk struggles to maintain cross-stream alignment when more features are active, potentially due to increased difficulty in coordinating which features to activate across different streams.
We provide further ablation on the cross-loss weight and learning rate in Appendix
6 Conclusion
We introduced SPARC, sparse autoencoder that learns single, shared latent space across heterogeneous representation streams. Its two core mechanisms, Global TopK sparsity and cross-reconstruuciin loss, enforce consistent index selection and semantic alignment across modalities.
Without either mechanism, aligmment is minimal: Local TopK without cross-loss achieves just. 0.16 general Jaccard similarity on Open Images. In contrast, SPARC with both components achieves 0.80 general Jaccard. This shared space climinates dead and mixed neurons across both models and modalities, and enables capabilities such text-driven localization using vision-only models. Overall, this unified approach allows identifying shared concept in cross-model and cross-modal settings.
14

---

Published in Transactions on Machine Learning Research (3/2026)
Broader Impact
SPARC enables direct comparison of how different models represent semantic concepts, which has both beneficial and potentially harmful applications.
On the positive side, a unified concept space across models can support cross-model auditing, helping re- searchers identify what concepts different architectures have learned and whether they encode problematic biases. This capability may also aid model debugging and safety analysis by revealing shared failure modes unintended representations across model families.
However, the same capabilities raise dual-use concerns. A shared concept space could be used to transfer harmful concepts between models or to localize sensitive content (e.g., identifying individuals or private information) across vision and language modalities. The text- gext-guiied localization demonstrated in this work, while useful for interpretability, could potentially be repurposed for surveillance or content targeting.
To mitigale these risks, recommend avoiding training SPARC on sensitive harmful larget concepts, restricting deployment in high-risk domains without appropriate oversight, and coupling use of rross-model interpretability tools with governance frameworks and auditing practices. We hope that the interpretability benefits of SPARC outweigh these risks when deployed responsibly.
Acknowledgments
This project was funded by Fonds de recherche du Québec - Nature et technologies (FRQNT) PBEEE scholarship [A.N]. This work was also partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) RGPIN-2022-05378 [M.S.H], RGPIN-2025-06770 [H. R.J, AWS AI Amazon [H.R.].
Research Awards (ARA) [M...H), and FRQNT 361263
References
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 6541-6549. 2017.
David Bau, Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. Visualizing and understanding generative adversarial networks. In International Con- ference on Learning Representations, 2019. URL https://openreviicn http:://oppaaooooeeeecc/oooouoaa X2C5FX
Trenton Bricken, Adly Templeton, Joshma Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-oodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosc- Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//rraasoonnecciiniiiss,p http:/isi..........oooommmmmaaaa tnnnnnsemmnniic-ffsteeeeeeeeeee html.
manticity:
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh Internotional Conference on Learning Representations, 2023. URL
https:://penreevie net/ wett/forummidee Idrouuund--TTouuyhhoess
Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. ar preprinl arXiv:2,12.066110, 2024.
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-modes explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 397 406, 2021.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. International Conference on Learning Representations, 2020. URL htctpss//opparrvvoo..ee////ooor HiDdEyBKDS
15

---

Published in Transactions on Machine Learning Research (3/2026)
Gabriele Dominici, Pictro Barbiero, Lucie Charlotte Magister, Pictro Lió, and Nikola Simidjievski. Shares: Shared concept space for explainable multimodal learning. arviv preprinl ar arXiv:2377.00316 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkorcit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL
Amil Dravid, Yossi Gandelsman, Alexci A Efros, and Assaf Shocher. Rosctta neurons: Mining the common units in a model zoo. In Proceedings of the IEEE/CVE International Conference on Compuler Vision, pp- 1934-1944 2023.
Nelson Elhage, Tristan Hume, and Catherine et al. Olsson. Toy models of superposition. Transformer Cireuits pdis)ill.uub), 2022. Sept 14.
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference Learning Representations, 2025. URL
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based expla- nations. Advances in newral information processing systems, 32, 2019.
Gabriel Goh, Nick Cammarata, and Chelsea et al. Voss. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.23955/distil..000
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features language models. In The Twelfth International Conference on Learning Representations, 2024. URL
Andrej Karpathy, Justin Johnson, and Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint ar Xiv: 02078, 2015.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). Tn International conference on machine learning, pp. 2668 2677. PMLR, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference machine learning, pp. 5338-5348. PMLR, 2020.
Nechar Kondapaneni, Oisin Mac Aodha, and Pictro Perona. Represcntational similarity via interpretable visual concepts. In The Thirleenth International Conference on Learning Representations, 2025. URL https://openrevierg mett/forummdd=iih33mmm....
Matthew Kowal, Achal Dave, Rares Ambrus, Adrien Gaidon, Konstantinos G Derpanis, and Pavel Tokmakov. Understanding video transformers via universal concept discovery. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 10946-10956, 2024.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint orXio:2308. 03281, 2023.
Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider. Sparse autoencoders reveal selective remapping of visual concepts during adaptation. ar preprint arXiv:2412.05227, 2024.
16

---

Published in Transactions on Machine Learning Research (3/2026)
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CV international conference on computer vision, pp. 10012 10022, 2021.
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. Xr preprint ar arXio:1312.5000 2013.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. /tsttill..0002..0000
ZZoom in: An introduction to circuits. Distill, 2020. doi:
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Woj- ciech Galuba, Russell Howes, Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabricl Synnacve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bo- janowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856. URL Certification.
htt.:///eee.....ooooooooooaaaaar Featured
Alec Radford, Rafal Jozefowicz, and Tlya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint KKic:70040144 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine leurning, pp. 8748 8763. PmLR, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from doep networks via gradient-based localization. In Proceedings of the IEEE international conference on compuler vision, pp. 618 626 2017.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolci Zhou. Interfaccgan: Interpreting the disentangled face representation learned by gans. TPAMI, 2020.
Harrish Thasarathan, Julian Forsyth, Thomas Fel, Matthew Kowal, and Konstantinos Derpanis. Universal sparse autoencoders: Interpretable cross model concept alignment. arXiv preprint wxXi:2502.03714, 2025.
Michael Tschanmen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Tbrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, al. Siglip Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXix:2502. 14786, 2025.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXi preprint arXiv:2212. 03533, 2022.
Sewall Wright. Correlation and cansation. Journal of Agricultural Research, 20(7):557-585, 1921.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.0511 2025.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint Xrx::23100001. 014000 2023.
17

---

Published in Transactions on Machine Learning Research (3/200 (3/2026)
Appendix
A Experimental Details
A.1 Datasets
We use the denso-annotated subset of Open Images V7, containing 1.9M images (1.7M train, 41k val, 125k test), for all training and evaluation of SPARC. This subset includes bounding boxes, segmentation masks, and image-level labels. MS-COCO 2017 (118k train, 5k val) is used only for downstream segmentation and retricval experiments.
A.2 Hyperparameters and Training Configuration
We provide complete hyperparameter settings for all SPARC experiments to ensure reproducibility.
Model Architecture. All experiments use a latent dimension of L = 8,192 with sparsity level k = 64. Input feature dimensions vary by dataset: for Open Images, DINOv2-ViT-I/14 features have dimension 1024 and CLIP-VYT-L-14 features have dimension 768; for MS-COCO, DINOv2-ViT-B/L4 features have dimension 768 and CLIP- CLIP-ViT-B/I features have dimension 512.
Training Hyperparameters. We train all models for 50 epochs using batch size of 256. The optimizer is Adam with learning rate 10-4, 31 = 0.9, 8n = 0.999, and = 10- All experiments use fixed random seed of 42 for reproducibility. Training data comprises 80% of the available samples (train_ratte ratio = 0.8), with the remaining 20% reserved for validation.
Loss Function Configuration. The total loss combines self-reconstruction, cross-rcconstruction, and auxiliary reconstruction terms. For cross-loss experiments, we set the cross-reconstruction coefficient. À 1.0, while no-cross experiments use À = 0.0.
We implement an auxiliary loss (AuxK) identical to Gao et al. 2025), which targets dead neurons to prevent dormant latent dimensions. Given the main reconstruction error e = x X, the auxiliary loss reconstructs this residual using the top-k' dead latents: Laux |le ----eff where è = Zzz The key difference between Local and Global TopK variants the source of the residual: Local TopK computes residuals from stream- specific reconstructions, while Global TopK computes residuals from reconstructions using globally shared indices. We auxiliary sparsity level k' 64 and coefficient. 7 0.03125 (1/32), with neurons considered dead after 1000 inactive steps.
Weight Initialization. Encoder and decoder weights are initialized with tied weights: W = (W;) Decoder weights are unit-normalized column-wise and maintainod through gradient adjustments that project out components parallel to unit vectors. All bias parameters are initialized to zero. Dead neurons are reinitialized using Gaussian noise with a standard deviation of 0.01.
Dead Neuron Management. Latent dimensions with activation frequency below 10-3 ( threshold) for more than 1000 consecutive training steps seaad__teps__tie This prevents the emergence of inactive dimensions during training while the auxiliary loss encourages their reactivation.
are considered dead and reinitialized.
Data Loading Optimization. To address HDF5 loading efficiency, we implement a custom ContiggoussaardomlatccSaapppep that shuffles the batch level rather than sample level, reducing training time from hours minutes per epoch while maintaining training dynamics. DataLoaders use workers with memory pinning enabled.
Feature Model Specifications. For Open Images experiments, extract features using DINOv2-ViT- L/14 (with registers variant) and CLIP-ViT-L-14 trained on DataComp-1B, For MS-COCO experiments, we use DINOv2-VIT-B/L1 (with registers variant) and CLIP-VTT-B/1C trained on DataComp-XT, dataset (datacomp_x1 ____pp________ 1tt_oop__x__1_3__00____0
Experimental Configurations. We evaluate four training configurations across two datasets, resulting in eight total experimental conditions: Global/Tocal TopK activation Cross-loss/No /No Cross-loss x Open
18

---

Published in Transactions on Machine Learning Research (3/2026)
Images/Ms- MS-COCO datasets. The Global TopK variant aggregates logits across streams before index selec- tion, while Local TopK applies independent TopK selection per stream.
A.3 Probe Implementation Details
We evaluate concept recoverability using 1D logistic probes following the methodology of Gao et al (0025)
This section details the complete experimental procodure for the Open Images binary classification evaluation reported in Section
4.5
Dataset and Task Selection. We use the Open Images test set containing 112,699 samples with 601 available binary classification labels. To cnsure statistical reliability, we filter tasks to retain only those with al least 50 positive examples, resulting in 432 binary classification tasks used in our evaluation.
Data Preprocessing and Balancing. For each binary classification task, address class imbalance by randomly sampling negative examples to match the number of positive examples, creating balancod datasets. Latent activations are binarized using threshold of zero: sparse representations z' are converted to binary indicators (ef > 0) for probe training.
Data Splitting Strategy. We employ a stratified three-way split with 70% training, 15% validation, and 15% test samples from the balanced dataset. Stratification ensures both positive and negative classes are represented proportionally across all splits. We task-specific random seeds (1000 + task_id 2000 task_id. id, 3000 + task_id) id) for reproducible splits across experiments.
Candidate Latent Selection. Since our goal is to test whether individual latent dimensions can recover semantic concepts, we focus the evaluation on the most promising candidates rather than all 8,192 latent dimensions. We first exchude dimensions that show zero activation during training, then rank remaining candidates by counting how many positive training examples activate each dimension. For each task, we select the 20 most frequently activated dimensions as candidates, reducing computational cost while focusing latents most likely to encode the target concept.
Probe Training Configuration. For each candidate latent dimension, we train 1D logistic regression probe with the following hyperparameters: L-BFGS solver, maximum 200 iterations, L2 regularization with default strength (C = 1.0), single latent activation value 2 as input, and binary class labels as targets.
Model Selection and Evaluation. We select the best-performing latent dimension for each task based on validation set performance. For each of the 20 candidate dimensions, train individual probes, evaluate on the validation set using binary cross-entropy loss, select the dimension achieving the lowest validation loss, and report final performance on the held-out test set.
Evaluation Metrics. We report mean binary cross-cntropy loss across all 432 tasks for each stream and training configuration. As a baseline reference, random binary classification on balanced data yields an expected loss of 10g(0.5) x 0.693.
Experimental Scope. We evaluate four SPARC training configurations (Global/Local TopK Cross- loss/No Cross-loss) across three feature streams (DINO, CLIP-inage, CLIP-text), resulting in 12 total experimental conditions. Each condition processes all 432 binary classification tasks using the identical probe methodology described above.
A.4 Computational Cost
This section summarizes the computational profile of SPARC and compares it to Universal Sparse Autoen- coders (USAE) Thasarathan et. al. 2025
Asymptotic cost of the SPARC objective. For M streams, SPARC computes stream-specific logits hees, aggregates them into hagg: applies Global TopK once to obtain the shared index set, and then
19

---

Published in Transactions on Machine Learning Research (3/2026)
evaluates both self- and cross-rcconstruction terms:
Laelf Σ (xMMSE("2Dalxx
Leross
Σ a,EES seft
LNMSE (x', Dr(x))).
Each encoder or decoder is a dense linear layer between R& and It,,, so for batch size B, a single forward pass costs O(Bdd,). Assuming comparable feature dimensions d, = and treating B as fixed, a single minibatch therefore performs: .M encoder passes, costing ((M and M² decoder passes across all ordered pairs (s, (,,1), costing (MM
Thus, the dominant term for SPARC training scales OdM- per batch.
By contrast, USAE (Thasarathan et al. 2025 encodes activations from a single randomly selected model at cach step and decodes into all M models: one encoder pass, O(Ld), and - M decoder passes, O(MLd) for an overall cost of Ld) per batch. In other words, SPARC pays an additional factor of roughly M decoder work compared to USAE's random-encode, scheme. For the regimes we consider (M I 10), this is modest constant factor rather than prohibitive blow-up.
Cached backbone features. In all experiments, we follow standard SAE practice and treat rpstream encoders (e.g., CLIP, DINO, SigLIP, ViT) frozen feature extractors. Features from each stream are computed once on the training split and stored; SPARC training then operates entirely on these cached activations, optimizing only the linear /decodr layers in the shared latent space. USAE is also trained on pretrained activations rather than updating the backbone models (Thasarathan et al. 2025), so both approaches incur a comparable one-off cost to run the dataset through the backbone encoders. We therefore focus our quantitative reporting on the SAE training itself and do not attempt to micro- benchmark the feature-extraction pass, which is dominated by the backbone architectures and hardware configuration.
Wall-clock runtimes. With cached features, optimizing SPARC is cheap absolute terms. On a single H100 GPU, main three-stream configuration Open Images (CLIP-image, Baap--amne/CLLPttttttt/ CCLICttxt//DI/ /Dictt,ODO, L-8192, k=64, batch size 256, 50 epochs) takes under 25 minutes end-to-end (i.e., less than 30 seconds per epoch). The ten-stream configuration used in Appendix C (CLIP-image, CLIP-text, SigLIP-image, SigL.P-textt, DINOv2, VIT, Swin, E5, GTE, Qwen) takes about hours for 50 epochs, corresponding to roughly 2.5 minutes per epoch.
For context, USAE (Thssarathan et al. 2025) reports that training their model on ImageNet on a single Nvidia RTX 6000 GPU takes approximately three days (Appendix A.1). The datasets, architectures, and hardware are not strictly comparable, so this should be viewed only coarse reference point rather than head benchmark. Nonetheless, it illustrates that the additional O(M) factor in SPARC's objective does not translate into large practical overhead once backbone fcatures are cached, and that SPARC's wall-clock training cost is small relative to both feature extraction and existing universal SAE selups.
A.5 USAE Baseline Configuration
We provide complete details for the Universal Sparse Autoencoder (USAE) baseline Thasarathan et al. 2025 to ensure reproducibility.
Implementation. We used the authors' official implementation, specifically importing TopKSAE and top_k_aau L__axxiliarr____ loss from the overcomplete package in their GitHub repository. We adapted only the data loading pipeline to process our pre-extracted Open Images features in the three-stream configuration (CLIP-image, CLIP-text, DINO).
To ensure a fair comparison, we matched key hyperparameters to our SPARC sctup: dictionary size L 8, 8,192, sparsity level k 64, batch size 256, and 50 training epochs. The optimizer is Adam with learning rate 7 - B1 0.9, 3g 0.999. USAE was trained from scratch on the same pre-extracted frozen features used for SPARC (see Section A.2 for feature extraction details).
Matched Hyperparameters.
20

---

Published in Transactions on Machine Learning Research (3/2026)
USAE-Specific Parameters. For parameters specific to the USAE architecture, used the authors' defaults: auxiliary loss coefficient 0.1, gradient clipping at 1.0, linear encoder module, and L.2 dictionary normalization.
Training Procedure. Following the original USAE training scheme, at each iteration we randomly select one stream as the encoder, compute latent codes z², and reconstruct all streams using their respective decoders. This contrasts with SPARC, which encodes all streams simultaneously and applies Global TopK to the aggregated logits.
Evaluation Protocol. We evaluated USAE using identical metrics and procedures SPARC: Jaccard- based concept alignment (Section 4.2). R²² reconstruction (Section 4.4 activation consistency (Section
and label purity (Section 4.3
21

---

Published in Transactions on Machine Learning Research (3/2026)
B Extra Ablation
This section details the ablation experiments used to select the optimal cross-loss weight X and learning rate
!/-
B.1 Cross-loss weight À
0.2
local top-k
local top
AVq licross atnumms Swat Swlf-haccnn MASE
Ivg (ecrote strammo Crots Prets-Reecon MMST
0.7
05
.314
0.2
3323
106
5206
010
10
101
2319
10
10-
33
101
10
Figure 9: NMSE as a function of À (log scale).
shows the impact of À parameter. Cross-reconstruction drops quickly up to A A 1. Beyond that point, the alignment gain saturates while selftrecontttuttion keeps rising, so we keep A=1.
Figure 9
B.2
Learning rate ?
0.60
local top
local k global top-k
global top-k
Aeg (ocross stremms Self-tecon NASE
Ang fecross streamsi Cross-itecon NMSE
0.50
0.33
0.11
0.30
0.40
0.2
0,64
0.25
0,20
0.30
6300
0.365
0.15
0257
0.3
107"
10T4
10"1
10"
1011
1274
107
leaming rate
learning rate
Figure 10: Learning-rate sweep.
The impact of learning rate is shown in the Figure 10 smaller rate under-fits; y 2 10-3 destabilizes training and erases the Global ad advantage.
The learning rate 10 minimizes both losses. A
22

---

Published in Transactions on Machine Learning Research (3/2026)
C Full Rg Reconstruction Evaluation
We report Rg matrices for more model combinations and training setups. Rows are targets (4), columns
are sources (s). Negative values indicate worse than predicting the mean.
Table summarizes the ten encoders used across the configurations below. Features are extracted as follows: for ViT we take the [CLS] token from the last hidden state; for Swin we use the model's pooled output; for DINOv2 use the backbone's global representation returned by the hub model; for CLIP (image/ (text/)/ we use OpenCLIP's encode image eacode_texte for SigLIP2 (image/text) get_ features get get_text fext_features (captions lowercased, padding-max, bedding-axx_ennghh,, length, max 1ength=64, truncation=true)) for E5 we encode captions with "query: prefix and normalized embeddings; for GTE and Qwen we use the default SemtenceeTrasformnee embeddings. Unless a model's own processor is used (SigLIP2), images are resized to 256, center-cropped to 224, converted to tensors, and normalized with ImageNet mean/std.
the HF AutoProcessor and
Table 7: Encoders used across configurations.
Stream
Modality
Backbone / Model (brief)
clip_img img
Radford et al.
(0021
image
OpenCLIP CLIP-VIT-L/14 (DataComp)
clip_txt Radford et al. 2021
text
OpenCLII' CLIP-VIT-L/14 (DataComp)
siglip_ing [mg Tschannen al. 2025
image
gongle//sgiip--so/00nnppaaaal (image)
siglip_ txt. Tschannen et al
2025
text
8oogle/siigin2--so000nnaaaaaaaa (text)
dino
Dquab
al.
2024
image
DINOv2 VIT-L/14 (registers)
vit
Dosovitskiy et 2021
image
(CLS)
swin Liu et al.
2021
image
microsoft/swiin-large-pttte wwtpppahhhhwow112-233-3--3 (pooler)
e5
Wang et al. 2022
text
inffloal/e5-large-22
thealper/gte-large TaaileenceTTannnommenr
gtel
Li
2023
text
qwen
Zhang et al.
2025
text
Qwen/ hemm///een--mmmbedinggg.. (SentenceTrane Transformers)
CLIP: This case focuses on CLIP, whose image and text encoders are natively aligned by training. Table reports the results.
Table 8: R² (higher is better), CLIP image/text. Rows are targets (t), columns are sources (s).
Global TopK (source)
Local TopK (source)
USAE (source)
Original Target
clip_img
clip_txt
clip_img
clip_txt
clip_img
clip_txt
clip_img
0.657
0.574
0.737
0.239
0.577 0.495
0.398
clip_txt
0.676
0.789
0.333
0.882
0.708
SigLIP:
This case focuses on SigLIP, whose image and text encoders are also natively aligned by training.
Table reports the results.
Table 9: R2 (higher is better), SigLIP image/text. Rows are targets (f), columns are sources (s).
Global TopK (source)
Local TopK (source)
USAE (source)
Original Target
siglip_img
siglip_txt
siglip_img
siglip_txt
aiglip_img
siglip_tae
siglip_img siglip_txt 0.547
0.490
0.387
0.702
0.195
0.594
0.497
0.704
0.261
0.831
0.516
0.657
Vision-only: This case considers vision encoders trained without multimodal objectives. Table 10 reports the results.
23

---

Published in Transactions on Machine Learning Research (3/2026)
Table 10: R²
(higher is better), Vision-only. Rows are targets (t), columns are sources (s).
Global TopK (source)
Local TopK (source)
USAE (source)
Original Target
dino
swin
vit
dino
swin
vit
dino
swin
vit
dino
0.716
0.579
0.533
0.482
0.335
0.255
0.177
0.061
0.010
swin
-1.630
-1.574
-1.854
-13.691
-9.542
-11.171
0.073
0.153
0.086
vit
0.562
0.589
0.671
0.442
0.519
0.669
0.137
0.171
0.259
Text-only: This considers text. encoders trained without multimodal objectives (E5, GTE, Qwen).
Table 11 reports the results.
Table 11: R2 (higher is better), Text-only. Rows targets (I), columns are sources (s).
Global TopK (source)
Local TopK (source)
USAE (source)
Original Target
e5
gte
quen
e5
gte
qwen
e5
gte
quen
e5
0.805
0.752
0.725
0.854
0.778
0.775
0.834
0.828
0.826
gte
0.771
0.823
0.747
0.811
0.868
0.800
0.853
0.858
0.852
qwen
0.757
0.761
0.832
0.788
0.781
0.898
0.656
0.657
0.670
Image: This case combines image encoders, some trained multimodally (CLIP-ima) -ing SigLIP-img! -img and
others unimodally (DINO, Swin, ViT). Table 12 reports the results.
Table 12: R² (higher is better), Image. Rows are targets (t), columns are sources (s).
Global TopK (source)
Original Target
clip_img
dino
siglip img
swin
vit
clip_img dino
0.698
0.563
0.634
0.568
0.539
0.525
0.678
0.544
0.536
0.497
siglip_img
0.569
0.495
0.653
0.501
0.467
swin
0.477
0.476
0.490
0.580
0.490
vit
0.523
0.525
0.532
0.568
0.651
Local TopK (source)
Original Target
clip_img
dino
siglip_imp
swin
vit
clip_img
0.746
0.474
0.591
0.494
0.429
dino
0.265
0.648
0.263
0.400
0.347
siglip_ing
0.539
0.403
0.693
0.424
0.356
swin
-8,475
-5,250
-5.427
-2.565
-3.824
vit
0.437
0.444
0.429
0.522
0.688
USAE (source)
Original Target
clip_img
dino
ziglip_img
swin
vit
clip_img
0.427
0.372
0.397
0.380
0.371
dino
0.002
0.064
0.006
0.012
0.007
siglip_img
0.492
0.480
0,510
0.484
0.480
swin
0.065
0.066
0.068
0.097
0.070
vit
0.066
0.067
0.069
0.081
0.122
Text: This case combines text encoders, some trained multimodally (CLIP-txt, SigLIP-txt) and others unimodally (E5, GTE, Qwen). Table 13 reports the results.
24

---

Published in Transactions on Machine Learning Research (3/2026)
Table 13: R (higher is better); Text. Rows are targets (t), columns are sources (s).
Global TopK (source)
Original Target
clip_txt txt
e5
gte
qwen
siglip_tar txt
clip_txt
0.813
0.721
0.720
0.717
0.705
e5
0.672
0.784
0.735
0.710
0,641
gte
0.689
0.754
0.800
0.731
0.659
qwen
0.717
0.750
0.753
0.819
0,686
siglip_txt
0.598
0.583
0.585
0.572
0.743
Local TopK (source)
Original Target
clip_txt txt
e5
gte
qwen
siglip_txt
clip_txt
0.903
0.722
0.704
0.763
0.758
e5
0.775
0.830
0.741
0.773
0,714
gte
0.792
0.777
0.837
0.796
0.737
quen
0.815
0.763
0.752
0.898
0.754
siglip_txt.
0.668
0.583
0.569
0.612
0.841
USAE (source)
Original Target
clip_txt
e5
gte
qwen
siglip_txx txt
clip_txt
0.497
0.476
0.473
0.475
0.474
0.809
0.816
0.813
0.811
0.809
gte
0.833
0.836
0.840
0.835
0.833
quen
0.608
0.612
0.614
0.620
0.606
siglip_txt
0.517
0.516
0.513
0.513
0.535
All: This case spans all ten streams, covering both unimodal and multimodally trained encoders across image and text. Table reports the results. We can see that Global TopK is required avoid the large negative numbers.
25

---

Published in Transactions on Machine Learning Research (3/2026)
Table 14: R (higher is better) for all streams. Rows are targets (t), columns are sources (s).
Global TopK (source)
Original Target
clip_img
clip_txt
dino
e5
siglip_ing ing
siglip_txt
swin
vit
clip_ing ing
0.563
0.286
0.407
0.315
0.320
0.302
0.498
0.293
0.433
0.401
clip_txt
0.447
0.786
0.429
0.700
0.696
0.708
0.471
0.701
0.447
0.431
dino
0,241
0,123
0.376
0.135
0,139
0.133
0.246
0.126
0.259
0.239
es
0.356
0.643
0.322
0.715
0.680
0.674
0.395
0.612
0.356
0.327
gte
0.381
0.662
0.343
0.701
0.734
0.699
0.424
0.630
0.381
0.348
quen
0.462
0.716
0.441
0.725
0.726
0.788
0.487
0.684
0.461
0.445
siglip_ Lng
0,437
0.239
0.335
0.271
0.276
0.256
0.511
0.248
0.368
0.332
siglip_tx txt
0.330
0.590
0.308
0.568
0,566
0.566
0.357
0.704
0.329
0.315
swin
0.333
0.204
0.329
0.220
0.223
0.216
0.340
0.208
0.445
0.366
0.357
0.245
0.260
0.256
0.363
0.248
0.413
0.492
Local TopK (source)
Original Target
clip_img
clip_txt txt
gte
quen
siglip iin
siglip_ txt
clip_ing
0.750
0.212
0.397
0.186
0.186
0.207
0.550
0.204
0.441
0.382
clip_txt
0.344
0.880
0.301
0.667
0.658
0.730
0.367
0.719
0.339
0.306
dino
-2.246
-11.318
-0.408
-7. -7.649
-7.961
-8.587
-1,393
-13.086
-1,524
-1.237
e5
0,243
0.706
0.190
0.808
0.706
0.737
0,287
0.654
0.227
0.196
0.277
0.729
0.220
0.741
0.825
0.763
0.327
0.679
0.258
0.225
quen
0,240
0,726
0.180
0.715
0.711
0.875
0.309
0.676
0:211
0.184
siglip_ing
0.497
0,175
0.345
0.153
0.154
0.172
0.684
0.166
0.379
0.320
siglip_txt
0,238
0.621
0.198
0.531
0.527
0.572
0,263
0.817
0.233
0.204
swin
-20.136
-145.388
-27. 07.6418
-6-1633
-69.942
-91.360
-16.134
-222.031
-9.140
-17.421
vit
0.370
-0.015
0.351
0.049
0.040
0.051
0.343
-0.094
0..49
0.684
USAE (source)
Original Target
clip_img
clip_txt
dino
e5
siglip_ ing
siglip_txt
swin
vit
clip_ing
0.335
0.318
0.322
0.319
0.319
0.319
0.328
0.319
0.324
0.322
clip_txt
0.339
0.359
0.334
0.353
0.352
0.353
0.341
0.353
0.339
0.336
dino
-0.002
-0.005
0.011
-0.004
-0.004
-0.004
-0.001
-0.004
0.000
0.000
e5
0.790
U.792
0.790
0.794
0.793
0.792
0.790
0.792
0.790
0.790
gte
0.820
0.822
0.821
0.823
0.824
0.823
0.820
0.822
0.821
0.821
quen
0.548
0.554
0.547
0.556
0.556
0.558
0.549
0.554
0.548
0.548
siglip ing
0.466
0.461
0.464
0.462
0.462
0.461
0.469
0.461
0.464
0.464
siglip_tx txt
0.444
0.451
0.443
0.451
0.451
0.450
0.446
0.455
0.445
0.444
0.055
0.053
0.056
0.053
0.053
0.053
0.056
0.053
0.061
0.056
0.025
0.025
0.025
0.029
0.025
0.031
0.039
26

---

Published in Transactions on Machine Learning Research (3/2026)
D Latent Dimension Visualizations
This section provides examples of latent activation patterns across training configurations (Local/Globar TopK x A-0/1). Each figure shows the 2x2 grid, with each configuration displaying top-10 activating images across three streams (DINO, CLIP-image, CL.IP-text) for latent dimensions.
and demonstrate concept alignment under Global TopK with À = where latent dimen- sions exhibit consistent activation behavior across all streams, either fully active or completely dead across all three modalities. In contrast, Local TopK with À shows mixed activation patterns, where latents may be active in some streams while remaining inactive in others.
Figures
For more results, check md
27

---

Published in Transactions on Machine Learning Research (3/2026)
Local Tep 2r0
!1
Latest
CANa ReC
TOL
Latest
Uecal
Figure 11: Latent activation examples for dimensions 965, 3667, and 4371 showing top-10 activating images across different configurations.
28

---

Published in Transactions on Machine Learning Research (3/2026)
Uncal Typk Berl
Local Spp
Latest w128
Clibal Tee
2n3
10m: 0145
UTS
Oltal TeC
Figure 12: Additional latent activation examples for dimensions 4950, 5720, 6627, and 8186 showing top-10 activating images different configurations.
29

---

Published in Transactions on Machine Learning Research (3/2026)
LAST Tepk 2o12
Local Tep:
Latest
Byc
Land Tgri
cs
Iam: 6033
Oabal Tec
Gloul Beat End
Figure 13: Additional latent activation examples for dimensions 36, 213, 3729, and 6419 showing top-10 activating images different configurations.
30

---

Published in Transactions on Machine Learning Research (3/2026)
E Latent Attribution based on concepts
This section demonstrates attribution analysis using SPARC's concept-aligned latents scalar targets for gradient-based methods. We identify concept-relevant latents by analyzing their activation patterns on labeled data, then use their combined activations as attribution targets for spatial and textual attribution.
To identify which concepts cach latent represents, we examine what types of images most strongly activato each latent dimension. For each latent, we analyze its top-50 activating samples and determine the most frequent concept category among them. We measure semantic consistency by computing the purity score, the fraction of top-activating samples that belong to the dominant category. When this purity exceeds 0.3, indicating sufficient semantic consistency, we assign the corresponding concept name to that latent. Latents without clear semantic patterns remain unlabeled. This process enables identification of multiple latents representing the same concept across different streams.
For attribution, we collect all latents assigned a larget concept across streams, forming the set of concept-relevant indices. We then compute spatial attribution using both relevancy maps (Chefer et all 2021) and GradCAM (Selvaraju et al. 2017) with the summed activations Σjer 2] as scalar targets. For Text allribution, we compute token relevancy scores (Chefer el al. 2021 Each figure displays spatial
attribution heatmaps alongside text token relevance scores, with scores below 0.1 omitted for clarity.
For more results, check http:///wgg.......aaa..ac///a///
E.1 Concept Specific Latents
Figures
16 and
17 show attribution results using concept-specifie latent selections across various
object categories.
Releuency Mae [SPARC]
Selevoncy Hap CUP (SPANC)
GrodCAN
GrodCAM (SPARC)
Raleuancy Map
GradC4M
SPARC CLIP Text Token Relevance Score (Concept: Kite)
in this image can see 0.12 red color kite
1.00 is
flying 0.15 in the sky with the help of white
thread 0.36 and
also the two red color tails 0.19 attached to the kite 0.49 and in the background 0.27 we can see the sky
GradCAN DINO (SPARC)
GrodCAM CUP (SPARC)
GraCAMM CLI Sim
Original
DINO (SPACC)
CUP (SPANC)
CUP Si
SPARC CLIP Text Token Relevance Score (Concept: Balloon)
in this edited image, where we can see hot, 0.14 air 0.39 balloons 1.00 in the sky seems like a branch
in the bottom 0.10 left 0.11 side..
Figure 14: SPARC CLIP text token relevance for kite and balloon concepts. CLIP similarity baseline uses concept names 'a kite" and 'a balloon' rather than full captions.
31

---

in the background we can trees and the sky we Published in Transactions on Machine Learning Research (3/2026)
Relevency Map (SPACC)
Relevancy Map Sim
CLI (SPANC)
ISPARC)
SPARC CLIP Text Token Relevance Score (Concept: Leopard)
in this picture we can see & leopard 1.00 sleeping on the branch 0.16 of a tree in the background we
can see branches 0.12 and trees
Relevency Map DIO (SPARC)
Map CUP (SPARC)
GradCAM DIVO ISPARC)
GredCAM CUP (SPAAC)
Relevancy Map CLP Sm
GradCAM CLP Sim
Originel
SPARC CLIP Text Token Relevance Score (Concept: Tiger)
in this picture i can see
tiger 1.00 in the middle of number of leaves
Relevoncy Map DINO (SPARC)
Selevency Hap CUP (SPANC)
GradCaM DINO (SPARC)
GradCAM CUP (SPARC)
Relcuoncy Map
GradC4M CLI Sim
Original
CLI Sm
SPARC CLIP Text Token Relevance Score (Concept: Rhinoceros)
in this picture we
rhino 0.94
1.00
0.24
the ground 0.30 the right side of the
picture we can see elephants can see plants 0.18 and grass 0.23
Relevancy Map
GradCAM DINO (SPARC)
GradCAM CUP (SPAAC)
GradCAM CUP 5m
Original
DINO (SPARC)
CUP (SPARC)
CLP Sm
SPARC CLIP Text Token Relevance Score (Concept: Red panda)
in this image we can see a rod panda 1.00 on branch 0.11 of a with leaves in the back 0.14 there is wall
Figure 15: SPARC CLIP text token relevance for leopard, tiger, rhinoceros, and red panda concepts. CLIP similarity bascline uses concept names 'a leopard', 'a tiger", 'a rhinoccros", and "a red panda" rather than full captions.
32

---

Published in Transactions on Machine Learning Research (3/2026)
Relevancy Map (SPARC)
Relevancy (SPANC)
GradCAN ISPARC)
GradCAM (SPAIC)
Relevancy Map Sim
GradCAM
Original
SPARC CLIP Text Token Relevance Score (Concept: Banana)
in this pict ure we can see there is a wall
bananas 1.00
there are some plants and in the background 0.18 of the picture
Relevency Map
Selevancy Map CUP (SPARC)
GradCAM DIOO ISPAAC)
GredCAM CUP (SPARC)
Releventy Map CUP Sim
GradCAM CJP Sim
Original
DINO (SMARC
SPARC CLIP Text Token Relevance Score (Concept: Croissant)
in this
image 0.11 there is bowl 0.14
surface 0.21
in that bowl 0.11 there are tissue 0.13
papers 0.26 and croissant 1.00
Relevency Mae
Relevancy Map
GradCAN (SPARC)
GradCAM (SPARC)
Relevency Map Sm
GradCAM
Original
[SPARC]
CUP (SPARC)
SPARC CLIP Text Token Relevance Score (Concept: Cake)
at the bottom of this image there are some cakes 1.00 arranged on table in the middle of this image there is woman in violet 0.12 colorful
shirt 0.28 having a badge 0.20 smiling 0.13 and
standing in the background there is a white color curtain 0.34 a wall 0.10 and other objects 0.21
Relevoncy Map
GradCAN DINO (SPARC)
GradCAM CLP (SPARC)
Relevency Map
GraCCAM CLP 5m
Original
DINO (SPARC)
CUP (SPARC)
CUP Sm
SPARC CLIP Text Token Relevance Score (Concept: Pasta)
in this image we can see noodles 1.00 and vegetables 0.12 in plate 0.12 on the table to the right side of the image there glass
Figure 16: SPARC CLIP text token relevance for banana, croissant, cake, and pasta concepts. CLIP similarity baseline uses concept names 'a banana", "a croissant', 'a cake", and 'pasta" rather than full captions.
33

---

Published in Transactions on Machine Learning Research (3/2026)
Relevency Map SSAACC]
Relevancy ISPANC)
Relevancy Map Si
Original
ISPARC)
SPARC CLIP Text Token Relevance Score (Concept: Bicycle)
in this image we can see two
people 0.16
riding 0.19
bicycles 1.00 in the background of the image
there are trees 0.16
plants 0.15
pole 0.20
grass 0.10 and other objects 0.17 at the bottom of the on the right side bottom 0.11 of the image there is
image there is the grass 0.18 and object 0.13
ground 0.13
Relevency Mae
Relevancy Map
GradCAN
GradCAM
Relevancy Map
GradCAM
Original
(SPARC]
CUP (SPARC)
SPARC CLIP Text Token Relevance Score (Concept: Bicycle)
in this image can see a woman 0.13 holding musical 0.15 instrument 0.18
we can see a bag 0.11
hat 0.21 and some objects 0.12 in the background
the fence 0.18
bicycle 1.00
walls and
frames 0.20
0.12
Relevency
Relevoncy Sm
GradCAM
Original
SPAACI
CLI (SPARCI
(SPARC)
CUP (SPARC)
SPARC CLIP Text Token Relevance Score (Concept: Bow and arrow)
in this image can see a person standing and holding a bow 1.00 and there is an arrow 0.64 in the air
at the bottom of the image i can an object 0.15 the grass 0.14 in the background 0.16 there
trees 0.12 and pole 0.15
Relevency Map DIN (SPARC)
Selevancy Map CUP (SPARC)
GradCAM DINO (SPARC)
GredCAM (SPARC)
Relecanty Map Sim
GradCAM sim
Original
SPARC CLIP Text Token Relevance Score (Concept: Wood-burning stove)
in this picture 0.21
see 0.19 few
sticks 1.00 and few objects 0.35 the ground 0.21 and in the
background 0.20 see 0.15 the fire 0.55 place 0.76 and the wall 0.32
Figure 17: SPARC CLIP text token relevance for bicycle, bow and arrow, and wood-burning stove concepts. CLIP similarity baseline uses concept names 'a bicycle", "bow and arrow", and wwood-burning stove' rather than full captions.
34

---

Published in Transactions on Machine Learning Research (3/2026)
E.2
Same image/caption, different latents
Figure 18 demonstrates how different concept selections produce attribution patterns for identical inputs. By changing the set J while keeping same image and caption, SPARC generates different attribution patterns.
Relevancy Map DIO (SPAA3)
Relevancy Hap CUP (GMAR)
GradCAM (SMARC)
GradEAM CLP (SINRC)
Relevancy Hap
GradCAM
Drighaa
Sim
SPARC CLIP Text Token Relevance Score (Concept: Person)
in this 0.11 image 0.27
0.11
woman 0.43
0.11
standing 0.26 on 0.12 the padide 0.85
hoard 0.65
she 0.10
holding
the raft
she is
wearing
cap
there dog 0.35
the
paddle board
background 0.45 there is water 0.17
0.10
bottom
0.30 the image 0.35 there
0.15 are
plants 1.00
0.22
Relevency Map (SAAC)
Relevency Map CUP (SPARC)
GradCAM DINO (SPARC)
GradCMM CUP (SNACC
Relevancy Hap
GrudCAM
Original
5m
CLIP Si
SPARC CLIP Text Token Relevance Score (Concept: Dog)
in this image woman is standing on the paddle board she is holding the raft she is wearing a cap there is
dog 1.00 the paddle board. background there is water bottom of the image there plants
Relevency
Map
GradCAM
Relevancy Hap
GradCAM Sin
CLP (SPARC)
(SPARC)
SPARC CLIP Text Token Relevance Score (Concept: Surfboard)
this
image 0.15
0.19 standing 0.19 0.12 the paddle 0.30
board
she is holding 0.15 the
raft 0.36
she is wearing 0.17 a cap 0.55 there a dog 0.15 on the paddle 0.25 board 0.61
background 0.43 there
0.11
bottom 0.41 of the image 0.12 there plants 0.40
Relevency Map
Relecancy Map
GradCAM
GredEAM
Relecancy Hep
GradDAM
SPARC CLIP Text Token Relevance Score (Concept: Paddle)
in this Image 0.19 woman 0.24 is standing 0.11 the paddle 1.00 hoard 0.95
she is holding 0.19 the raft 0.61
she is wearing 0.23 cap 0.36 there is a dog 0.27 on the paddle 0.46
board 0.40
background 0.30 there is water.
bottom 0.20 of the image 0.25 there 0.11 plants 0.70
Figure 18: SPARC CLIP text token relevance for the same image/caption using different concept-specific latent sets. Each panel shows attribution for a different target concopt as indicated in the table headers.
35

---

Published in Transactions on Machine Learning Research (3/2026)
E.3
Using latents of concepts that are not present in the image/caption
Figure 19 examines SPARC's behavior when applying concept latents to samples lacking those concepts. For these samples, when using irrelevant concept latent sets J, SPARC latents will be zero, meaning no gradients are produced and all attribution scores remain zcro.
Relevency Map DIO [SMAC)
Selewancy Map CUP (SPARC)
GredCAN DNOO ISPACC)
GredCAM CUP (SPARC)
Relevanty Map CUP Sm
GradCAM CLiP Si
Original
SPARC CLIP Text Token Relevance Score (Concept: Cat)
in this image a woman is standing on the paddle board. she is holding the raft she is wearing a cap there is dog on the paddle board background there is water bottom of the image there are plants
Relevency Mae (SPARC)
Relevancy Map CUP (SPARC)
GradCAN (SPARC)
GrodCAM (SPARC)
Relevancy Map Sm
GradCAM
Original
SPARC CLIP Text Token Relevance Score (Concept: Apple)
at the bottom of this image there are food ilems arranged on table in the middle of this image, there is woman in violet colorful shirt having a badge smiling and standing, in the background there is white color curtain wall and other objects
Relevency
Relevency Map Sm
GradCAM
Original
(SAAC
CLIP (SPARCI
DINO (SPARC)
(SPAAC)
SPARC CLIP Text Token Relevance Score (Concept: Flag)
in this image can woman holding musical instrument objects. in the background we see the fence, bicycle, walls and frames
bag hat and some
Relevancy DINO iSPAac]
GradCaN DINO ISPARC)
GradCAM CLP (SPAAC)
GradCAM
Original
CUP cSnkiC:
CLP Sm
CLI Sim
SPARC CLIP Text Token Relevance Score (Concept: Tiger)
in this picture can bananas there are some plants and in the background of the picture there is wall.
Figure 19: SPARC text token relevance for image/captions with concept that's not present in the sample. Using irrelevant latent dimensions in SPARC causes no gradients. For text scores, all scores are 0.
36

---

Published in Transactions on Machine Learning Research (3/2026)
E.4 Limitations of concept-based latent attribution
For certain common classes such as person and car, many latents receive concept assignments, over 500 latents for 'person' and nearly 300 for "car", This is mainly a limitation of assigning concepts to latents as discussed in Scction
This breaks the selective attribution behavior of SPARC demonstrated in Appendix specific latents for most classes produce no gradients when the concept absent, common concepts with mumerous assigned latents produce spurious activations even when not present. Figure 20 demonstrates this limitation, where 'person' and "car" latents generate non-zero attributions images containing neither concept.
While concept-
Relevancy Map
Relevoncy Map CUI (SPANC)
GradCAN DINO ISPARCY
GradCAM CLP (SPARC)
Relcuanc Map
GradCMM CLI Si
Original
DINO (SPARCC)
CLI Si
SPARC CLIP Text Token Relevance Score (Concept: Person)
in this image
there 0.03 are
food items 0.17 in 0.03 the trays which 0.02
are placed 0.03 on 0.03 the glass 0.12 display with the tags
Relevency Map
GradCAN DIVO (SPARC)
GradCAM CUP (SPAAC)
Relevancy Map
GradGAM CLP 5im
Original
DIO (SPARC)
CUP (SPARC)
CLP Smn
SPARC CLIP Text Token Relevance Score (Concept: Car)
in
this
picture
0.31
can
see
an
eagle in the sky
Relevancy Map
Selevancy Map CUP (SPARC)
GradCAM DINO (SPARC)
GradCAM CUP (SPARC)
Relevancy Map CLP Sm
GradCAM CLI 5im
Original
DINO [SPARC]
SPARC CLIP Text Token Relevance Score (Concept: Person)
in 0.02 the 0.02 foreground 0.18 of 0.02 the 0.01 image 0.04
0.02
0.02
0.11 the 0.01
floor 0.10
and metal rods 0.27
the left 0.10 side
can see table 0.33
on 0.01 which 0.01
books 0.07 are 0.01
there 0.01 blue 0.02
color 0.03 box 0.08
wooden 0.03
object 0.01 and
poster 1.00
like structure 0.08
0.01 there 0.01 0.01 on 0.01 the
right 0.25 side 0.03
we 0.01 can see 0.06
broom 0.06 stick 0.29 and wall 0.07
in
the middle 0.03 we 0.02
can 0.01 see 0.06
stones 0.07
wall 0.11 on 0.01 which 0.02 0.01 white 0.06
color 0.05 object 0.09 is 0.02
Figure 20: SPARC CLIP text token relevance for image/captions with a concept that's not present in the sample. SPARC produces non-zero gradients for some of common concopts even in the absence of the concept.
37

---

Published in Transactions on Machine Learning Research (3/2026)
F Cross-Modal Similarity Attribution
This section demonstrates attribution using cross-modal similaritics x² 22' in SPARC's aligned latent space as scalar targets. We compute spatial attribution using both relevancy maps (Cheler et al. 2021 and GradCAM [Selvaraju et al. 2017). while text attribution uses token relevancy scores (Chefer et al. 2021
F.1 Cross-modal heatmaps with full captions
Figures 21
22 and 23 demonstrate cross-modal attribution using full captions as text input. Each figure
shows spatial heatmaps alongside text token relevance scores, comparing SPARC's aligned latent similarities against CLIP similarity baselines across various object categories. For more results, check huttps://git com
WWA.IUAAAAAATIINS..... md
Relevency Map
Relevancy Map CUP (SPARC)
GradCAM ISRNRC)
GradCAM
Relevency Map
GredCAM
Original
[SPARC]
SPARC CLIP Text Token Relevance Score
CLIP Sim Token Relevance Score
in the center 0.23 of the image 0.21 can 0.22 in the center of the image can see 0.39
cat 1.00 on book 0.44 at the bottom 0.16 of the book 1.00
the bottom of the image we can see the carpet 0.38 the top of the image can the cloth
Image 0.14 we can see 0.14 the carpet 0.62 at the top 0.17
the image 0.13 we can see 0.14 the cloth 0.49
0.12
Relevency Mae (SPARC)
Relevoncy Hop CUP (SPARC)
GrodCAN
GrodCAM (SPARC)
Relevancy Map
GradCAM
Original
(SPARC)
Sm
SPARC CLIP Text Token Relevance Score
CLIP Sim Token Relevance Score
this image we can see two green 0.12
apples 1.00 and an in this image we can see two green apples 1.00 and an object are platform on the right side at the top corner there is an object
object platform 0.15 on the right side the top corner there an object
Relevency Map (SPARC)
Selevancy Map CUP (SPARC)
GradCAM (SPARC)
GradCAM
Relevency Map
GradCAM
Original
(SPARC)
SPARC CLIP Text. Token Relevance Score
CLIP Sim Token Relevance Score
in this image see 0.12 butterlly 1.00 and 0.10 can
in this image
0.10 see 0.12
butterfly 1.00 and
also see flowers 0.14 flower 0.10 buds 0.22 stems 0.13 and
can also see flowers 0.18
flower buds 0.16 sterss 0.29
leaves 0.17 and the background 0.24 is blurry 0.30
and leaves 0.18 and the background is blurry 0.11
Figure 21: Cross-modal similarity attribution for mixed concepts (cat, apple, butterfly) comparing SPARC's aligned latent space against CLIP similarity baseline.
38

---

Published in Transactions on Machine Learning Research (3/2026)
Relevency Map (SPAAC]
Relevancy Map Sm
Original
CUP ISPARC)
SPARC CLIP Text Token Relevance Score
CLIP Sim Token Relovance Score
in this picture
bird
sitting
in this picture
bird 1.00 sitting
wire 0.74 in the background 0.35
can
see
0.18 the
wire 0.40
the background can see the clear
clear 0.17 sky 0.50
sky 0.15
Relevency Map
Selevancy Map
GredCAN (SPARC)
GradCAM (SPARC)
Relevancy Map
GredCAM
Original
[SPARC]
CUP (SPARC)
SPARC CLIP Text Token Relevance Score
CLIP Sim Token Relevance Score
image 0.17
center 0.20 there 0.12 is one
this image center there is one dog 0.14
dog 0.84
and there is ball 1.00 at the bottom 0.13
there is ball 1.00 at the bottom there is carpet
there is carpet 0.43 and background 0.27 there
and in the background there wooden object
wooden object 0.31
Relevency
Relevency Sm
Original
(SPARC)
CLIP (SPARCI
(SPARC)
(SPAAC)
SPARC CLIP Text Token Relevance Score
CLIP Sim Token Relevance Score
this image is taken outdoors
the bottom of the image there grass 0.14 on the ground in the middle
this image is taken outdoors at the bottom of the image there grass the ground 0.11 the middle of the image there is rabbit 1.00 and we can see the soil 0.19
the image there rabbit and see the soil
Relevancy Map
Belevancy Map
GradCAM ISPARC)
GradCAM (SPARC)
Ralevancy Map
GradCAM
DIND [SPARC]
CUP (SAARC)
Si
SPARC CLIP Text Token Relevance Score
CLIP Sim Token Relevance Score
the bottom 0.28
0.11 this image
there is a at the bottom 0.18 this image there a person 0.11 holding red color cloth
person 0.24 holding 0.27 red color 0.25 cloth 1.00 on 0.13
the ground
the ground in front of him there is buffalo
in front him there buffalo 1.00 on ground in the background there persons banner sign boards walls fences 0.56 and other objects.
on the ground 0.13 in the background 0.13
there
persons 0.33
banner 0.25 sign boards 0.23
walls u.27
fences 0.42 and other 0.10 objects 0.24
Figure 22: Cross-modas similarity attribution for animal concepts comparing SPARC's aligned latent space against CLIP similarity baseline.
39

---

Published in Transactions on Machine Learning Research (3/2026)
Relevency Map (SPARC)
Relevancy Map Sim
GradcAM
Original
CUP (SPANC)
ISPARC)
SPARC CLIP Text Token Relevance Score
CLIP Sim
Token Relovance Score
in the
image 0.18
there
three
balls
in the image
there three
balls
wooden
wooden
0.23 surface
0.72
0.26
surface 0.16
Relevency Map
Releveniy Map
GradCAN DINO (SPACC)
GradCAM CUP (SPARC)
Releventy Map CLP Sim
GradCAM CJiP 5m
Original
DINO (SPARC
CUP (SPARC)
SPARC CLIP Text Token Relevance Score
CLIP Sim Token Relevance Score
in this picture can see a bag 0.34 in front and see that and see that it is of black white 0.14 cream
this picture 0.11 can see 0.10 bag 1.00 front 0.24
it is of black 0.58 white, cream and pink color i see that it is white color in the background
and pink 0.22 color 0.14
that, is white color
in the background 0.24
Ielevaacy Map
Relevancy Map
GradCAN DIVO ISPARC)
GradCAM CUP (SPAC)
Relevancy Map
GradCAM CLI Sim
Original
DIND (SPARC)
CUP (SPARC)
CUP Si
SPARC CLIP Text Token Relevance Score
CLIP Sim Token Rolevance Score
in this picture 0.14 we see
0.11
yellow 1.00 color 0..4
in this picture we see a yellow color 0.11 vehicle 0.10 it
vehicle 0.49 it looks like tractor 0.15 loader 0.79 the bottom, we see the snow 0.20 on the right side, we see the wooden 0.12 sticks 0.19 or logs 0.31 there are trees 0.38
looks like a tractor 0.98 loader 1.00 at the bottom 0.22
we see the snow 0.11 on the right side 0.13 we
see the wooden sticks 0.25 or logs 0.34 0.11 there are
in the background 0.37 the top
the sky 0.32
trees 0.32 the background 0.12
0.18 at the top we
0.28
the sky 0.17
GradCAM CLP Si
Original
DIN [SPAAC]
CUP (SPARC)
DINO ISPARC)
CLP (SPARC)
CUP Sm
Teli
Teli
SPARC CLIP Text Token Relevance Score
CLIP Sim Token Relevance Score
in this image there is table
that there
in this image there table
that there that plates there is fork spoon
plates 0.50 on that plates 0.40 there is a fork 1.00
plates
are
0.78 and tag 4.09 0.11 that there is text 0.61 and a tag 1.00 on that there is text 0.16
similarity attribution for object concepts comparing SPARC's aligned latent space against CL.P similarity baseline.
Figure 23:
40

---

Published in Transactions on Machine Learning Research (3/2026)
F.2 Same image, different captions
Figure 24 demonstrates how different lext queries produce distinct spatial attribution pallerns when applied to the same image. Using simple concept names ("Banana', "Apple', "Kiwi', "Cat') as text inputs, we examine how cross-modal similaritics z" z' generate different heatmaps based on text guidance, inchuding cases where the queried concept is absent from the image.
For this sample image, CLIP similarity shows focused attribution on target objects when present in the image. SPARC exhibits more varied attribution patterns, localizing to different image regions with less precision than CLIP similarity. We make claims about performance relevant to what the actual encoders are looking at. We present these examples to demonstrate that SPARC's spatial attribution varies with different text inputs, indicating text-guided behavior rather than text-independent object highlighting.
whether these heatmaps are meaningful
Relevancy Map DIND [SPACC]
Belevancy Hap CUP (SPANC)
GradCaM DING ISPARC)
GradCAM (SPARC)
Relevancy Map Sm
GradcAM
Si
Figure 24: Captions used are 'Banana', "Apple", 'Kiwi", and 'Cat" (non-existent concept).
41

---

Published in Transactions on Machine Learning Research (3/2026)
F.3
Cross-modal attribution limitations
Figure 25 shows case where SPARC's spatial attribution produces less meaningful localization compared
to Figure 24 Using detailed spatial queries ("Cat's Ears', 'Cat's Eyes", "Cat's Nose', 'A Cat"), SPARC
generates similar, poorly localized attribution patterns with minimal variation across different text inputs. Although CLIP doesn't match the captions perfectly, it shows more responsiveness to the text input.
Relevency Map (SPARC)
Relevancy Map CUP (SPARC)
GradCAN
GradCaM (SPARC)
Relevency Map Sm
GradCAM
Original
ISPARC)
Figure 25: Captions used are 'Cat's Ears", "Cat's Eyes", 'Cat's Nosc", and 'A Cat'. We find SPARC fails in the case of detailed heatmaps.
Similarly, Figure 26 further illustrates this limitation using distinct objects within scene. When querying for specific components of the bedroom scene ('Wall", 'Lamps", 'Pillows'), SPARC's attribution maps exhibit minimal variation, consistently focusing on the dominant central object (the bed and pillows) regardless of the text input. In contrast, the CLIP similarity baseline demonstrates effective responsiveness, shifting its focus to the background for 'Wall', the poripheral regions for 'Lamps', and the center for "Pillows".
A similar failure mode observed Figure 27 when distinguishing fine-grained facial features. Using specific queries ("Mouth', "A tie', "Eyes"), SPARC generates a static attribution pattern focused primarily on the lower face, failing to localize the eyes or the tie. Conversely, CLIP similarity accurately localizes the tie and eyes when prompted, showing superior sensitivity to fine- ane-grained part-level text queries.
42

---

Published in Transactions on Machine Learning Research (3/2026)
nelevancy Map
Retewancy MAD
DEadIAM
Relevancy Map CLIP-ii
Televenty Map CIRO ISPARC)
Releve iyy Map CLIP (SPARC)
CradCAM DIO (SPACC)
GredCAM CLP (SPARC)
Relevanty Map CUP Sm
GradCAM CLIP Sim
Original
Original
CIRG (SPARC)
ELIP (SPARC)
DIND (SPARC)
C.P ISPARC)
CUP
CLI Sim
Figure 26: Captions used are 'Wall,, 'Lamps', and 'Pillows'. SPARC produces nearly identical heatmaps for all queries, defaulting to the central object (bed), whereas CLIP correctly shifts focus to the queried objects.
Televancy Map
Retevancy Mao
CEACCAN
CradCAN
Belevancy Map
GradCAM
Original
nelevency Map CIRO (S(ARC)
Relevency Map CLI (SPAC))
GradiAM DIND (SP48C)
GredEWN CLP (SPARO)
Relevancy Map CLP Sm
GradCAM CLP 5i
Original
Onginnl
UIO (SPVIC)
CUP [SPAIC]
(SPAIIC)
[SPAII)
CUPSm
CUP Sim
Figure 27: Captions used are 'Mouth", 'A tie", and Eyess". SPARC fails to disentangle the specific features, while CLIP similarity successfully localizes the tie and eyes.
43

---

Published in Transactions on Machine Learning Research (3/2026)
29 and 30 where single-word queries that refer primarily to background scene elements ( ("Trees", 'Grass", 'Rocks"). Across all three examples, SPARC's cross-modal attribution remains biased toward salient foreground objects and produces diffuse or weakly localized heatmaps that only partially overlap with the queried regions. contrast, the CLIP similarity baseline shifts attribution more clearly loward the corresponding background structures.
Additional failure cases are shown in Figures 28
televancy Map ISPARC)
Relevancy Map CLIP ISPANC)
GradGar (SPARC)
GEedCAM
Selevancy Map
GraBCAM
Originel
Figure 28: Caption used is "Trees'. SPARC produces diffuse, weakly localized attribution, while CLIP similarity better aligns with the tree regions in the scene.
selevancy Map CIRO ISPARC)
Relevancy Map CLIP (SPARC)
GradLAM DID (SPARC)
GradLAN CLP (SPARC)
Relevancy Map CLIP Sm
GradLAM CLIP Sim
Original
Figure 29: Caption used is "Grass". SPARC's attribution primarily attends to the foreground object (dog) rather than the ground, whereas CLIP similarity allocates more emphasis to the grassy regions, better matching the queried concept.
melesarry Map rinn (CRORT)
Palevancy Map CTI (SRART)
DEADCAM DIND (PAR5)
DEAGEAN CP (SRAR))
Ralevancy Map CLiP-im
arddCAM
Original
CLIP Sin
Figure 30: Caption used is 'Rocks". SPARC's attribution remains focused on the foreground animals, while CLIP similarity better aligns with the rocky background queried by the text.
44

---

Published in Transactions on Machine Learning Research (3/2026)
G Downstream Retrieval
To assess the holistic alignment of the latent space, beyond individual neuron analysis, we measure perfor- mance cross-stream vector retrieval task (z* x²). Table 15 summarizes the Recall@1 (R@1) scores across datasets and training configurations. A clear pattern emerges: the combination of Global Topk and cros-reconstruction loss (A > 0) yields a substantial improvement in alignment, especially for vision-to-vision retrieval.
Table 15: Latent alignment R@1 scores across datasets and training regimes.
CI = CLIP _MG, CT
CLIP -AP__XT, D = DINO.
TopK À
CI-CT CI+D
CT~CI CT-D D-II
D-CT
Open Images
0.034
0.352
0.031
0.018
0.284
0.021
Global
0
0.010
0.078
0.007
0.002
0.173
0.010
0.011
0.366
0.010
0.008
0.302
0.009
Local
0
0.000
0.000
0.000
0.000
0.000
0.000
MS- COCO
Global 1
0.421
0.762
0.386
0.325
0.716
0.347
0.181
0.259
0.204
0.102
0.369
0.188
0.365
0.694
0.329
0.261
0.714
0.301
Local
0.000
0.000
0.000
0.000
0.000
0.000
Cross-modal retrieval on Open Images shows weak performance compared to MS-COCO. SPARC's best cross-modal results (Global TopK, = 1.0) achieve 3.4% R@1 for C1-CT and 3.1% R@1 for CT-CI. To contextualize these numbers, raw CLIP features achieve 9.8% and 7.7% R@1, respectively the same tasks, indicating that Open Images presents retrieval challenges representations.
for CLIP's well-aligned cross-modal
retrievals Qualitative examples in Appendices H.l H.2 show semantically retrieal despite low R@1 scores. Cross-model image retrieval for amage-to-image tasks is provided in Appendix HL. Out-of-distributtio tests (Appendices H.4 H.5 use external images and froe-form captions without corresponding pairs in the datasct.
H Retrieval Qualitative Results
This section presents qualitative retrieval examples using SPARC's aligned latent representations. The following subsections show retrieval samples across in-distribution tasks using test data queries, and out-of- distribution tasks using external queries not present in the dataset. In both cases, the reference database consists of test samples.
H.1
Image Caption Retrieval (In-Distribution)
We evaluate cross-modal alignment through image-to-caption retrieval using SPARC latent representations. Tables
16
25
show retrieval results comparing Global Local TopK training configurations.
The four model configurations represent:
Global DINO: Query image's DINO features - Reference database of CLIP-text features (both from Global SPARC)
Local DINO:
Query image's DINO features Reference database of CLIP-text features (both from Local SPARC)
45

---

Published in Transactions on Machine Learning Research (3/2026)
Query image's CLIP-image features Reference database of CLIP-text features (both from Global SPARC)
Global CLIP:
Local CLIP: Query image's CLIP-image features + Reference database of CLIP-text features (both from Local SPARC)
Each model shows top-5 retrieved captions ranked by cosine similarity in the SPARC latent space.
Open Images dataset's Image -> Caption Retrieval
Tables
16
19 and
20 show examples on the Open Images lest set across diverse scene types.
46

---

Published in Transactions on Machine Learning Research (3/2026)
Model Rank Caption
In this image I can see bed with bed sheet and pillows and I can also see table, lamps, something looking like glass and in the background I can see curtains and wall.
Global DINO
2 This is picture of a room, in this image in the center there is a bed, on the bed there are pillows and there are lamps, photo frame, tables. On the tables there telephones and some papers, and on the ri...
3
This picture is clicked inside the room. In this picture, we see the bods and the pillows. We see the blankets in grey color. In between the beds, we see a table on which a telephone and remote are place...
4
In this image there are two wooden beds with mattresses and pillows on them. In between the beds on the table there is a telephone and some other objects. Behind the bed there are switches, lamps and curta...
5
In this image there are pillows, book, paper on the bed. Beside the bed there another bed. On top of the bed there is pillow. There are lamps, landline phone and some other objects on the table. There...
1
In this image I can see bed. can see few pillows, towels and blankets on the bed. can see a lamp on the stool. On the left side can see few curtains.
Local DINO
2
In the center the image we can see beds. On the beds we can see clothes, curtains and some objects. At the bottom we can see shoes on the floor. In the background there is wall.
3
In this image there a bed in the room, behind bed there a wall, beside the bed there are doors. This image is taken the room, In this image there bods and There television placed on the stand. We can see table and there is laptop, lamp and some o... This is inside view of room. In this picture we soe bed. On this bed, we see blankets and pillows. We can walls, window with window doors and window shelf. We see an object on th... This is picture of room, this image in the center there is bod, the bod there pillows and there are lamps, photo frame, tables. On the tables there telephones and some papers, and the This picture is clicked inside the room. In this picture, we two beds and the pillows. In between the bods, In this picture can see two beds with blankets and pillows the beds. can can few items on the table. Looks like another lamp on the table the right side, curtain...
4
see clothes placed the beds.
5
1
Global CLIP
2
table which remote placed. On the left side, see white color object. In...
3
lamp on the table and
4
In this picture can see two beds with pillows and bed sheets it and these beds and a table on the floor and on this table we see two lights table lamp, papers, pen and in the background we can se...
5
In this picture we can see two beds, there are pillows and bed sheets placed on the heds, in the background can a wall, curtain, photo frame and light, there is some text at the right bottom.
1
In this image I can see two beds with pillows and blankets and can also see wooden table with drawers, lamp, telephone, some other items and in the background can picture frame, floor, door and wal...
Local CLII
2
In this picture can see two beds with blankets and pillows on the beds. can see a lamp on the table and can see few items on the table. Looks like another lamp on the table the right side, curtain...
3
In this picture we can see two beds, there are pillows and bed sheets placed on the beds, in the background we can see a wall, a curtain, a photo frame and a light, there is some text at the right bottom.
4
This image is taken indoors. In the middle of the image there are two beds with mattress, bed sheets, blankets and pillows on them. On the right side of the image there is wall. At the left bottom of the...
5 This image is taken from inside. In this image there are two beds with pillows on it, in the middle of them, there is a table. On the table there is a book, paper, telephone, lamp and other object. In the
Original
This is an inside view of a room, we can see two beds and there are some pillows on the beds. On the left side, there is table and we can see the chairs. There is an air conditioner on the wall, we can see the window and curtains.
Table 16:
The query image
is used to retrieve the captions. None of the retrieved text is
the exact caption of the query image, but still highly relevant captions.
47

---

Published in Transactions on Machine Learning Research (3/2026)
Model Rank Caption
In this image there are people sitting on wheelchairs and playing basketball, in the background it is blurred. In this image there are a few people in whoelchairs are playing basketball. Behind them there are few people sitting in chairs, behind them there is a person performing gymnasium on the ropes. In the bac... In this image, can see a group of people sitting on the wheelchairs, which are on the floor. Among them one person is holding a basketball. In the background there is a wall and a railing, In the top lef...
2
Global DINO
3
4
In this image I can see few people playing wheelchair basketball.
5
In imagee see a man sitting a wheel chair while holding basket ball in hands and behind him see group of people sitting wheel chairs. In background see a group people standing...
1
In the picture can see the men wearing the sports jersey and they are playing the basketball. can see the spectators sitting on the chairs.
Local DINO
2
In this image I
few people playing basketball. can see a person sitting on the metal stand, few stadium and few stains.
persons sitting
3
In this image we watching the game.
are playing baskethall. In the background, we can people sitting and
4
In this image we background there people standing.
people playing a game. The person in the center is holding ball. In the
5
In this picture can few people wearing jerseys. can few people sitting. can the ball. I can basketball rings.
1
In this image, I can group of people sitting the wheelchairs, which the floor. Among them one person holding basketball. In the background there is wall and railing, In the top lef...
Global CLIP
2
In this image there people sitting wheelchairs and playing basketball, in the background it is blurred.
3
In this image I can see few people playing wheelchair basketball.
4
In this image there are a few people in wheelchairs are playing basketball. Behind them there are few people sitting in chairs, behind them there is a person performing gymnasium on the ropes. In the bac...
5
In this image, I see a man sitting in a wheel chair while holding basket ball in his hands and behind him see group of people sitting in wheel chairs. In the background I see a group of people standing...
1
In this picture can see two people wearing medals and sitting on the wheelchairs. I can see a few people standing. I can see a person sitting the wheelchair on the left side. I can see the roof at the...
Local CLIP
2
In the image there are few people sitting in the wheelchairs. And those wheelchairs are on the road. In the background there are doors which are looking blur.
3
In this picture we can see a person sitting in wheel chair, holding a bag and talking in phone, back side there is aeother person pushing the wheel chair.
4
In this image we can see person sitting on the racing wheelchair. In the background there are people. At the bottom there is a road.
5
In this image we can see wheelchair. the wheelchair there a dog. Behind the wheelchair there are two people. In the background the image there are trees, a shelter, glasses, table, chairs and ot...
Original
This image is taken indoors. In the middle of the image many people are sitting in the wheelchairs. We can see the floor. On the right side the image a person is standing on the floor, In the background we can see the boards with text. We can see the chairs. We can see the railings. Many people are sitting on the chairs. We can see the stairs. There is banner.
BT
ana
uur''
Table 17:
The query image
is used to retrieve the captions.
15
15
48

---

Published in Transactions on Machine Learning Research (3/2026)
Model Rank Caption
In this image, we can see a person wearing astronaut suit and he wearing a hat and the background is dark.
Global DINO
2
In this image, we can see an astronant suit in front the wall.
3
In front of the image there space suit on the display, behind the suit there's wall. In the center of the image curtain and some drawing on the wall. At the top there is light.
4
astronaut suit. In the background we see wall,
5
In the picture I can see man wearing a spacesuit.
1
In this image there is a person standing and looking at the right side of the image, behind him there are few people.
Local DINO
2
In this image there is a person standing.
In this image we can see a person standing on the floor and a stand.
4
In this image 1 can there are few persons walking.
5
In this image there are people standing, in the background there are clothes.
1
In this image, background is dark.
see person wearing astronaut suit and he is wearing hat and the
Global CLIP
2
In the picture I can see man wearing a spacesuit. In the center of the image we can see astronaut suit. In the background we can see wall, curtain and some drawing on the wall. At the top there is light.
3
In the image can see space suits.
5 In this image there is an astronaut suit which visible.
1
In this picture, we see two girls and the boys are standing. They might be exercising. On the right side,
the legs of two people. At the bottom,
the floor.
Local CLIP
2
In this image can see man and a woman standing. In this image we can see two persons holding each other and behind them, we can see woman standing.
3
4
In front of the image there woman, behind the woman there is a person standing.
In this image we can see there are two women standing with smile, behind them there are few people. The background is dark.
Original
In this picture it looks like the cutouts of space suits holding flag pole with girls standing behind them. In the background, can other toys, trees, lights, games etc.,
Table 18:
The query image
is used to retrieve the captions.
49

---

Published in Transactions on Machine Learning Research (3/2026)
Model Rank Caption
In this image in the center it looks like a toy train and there is toy railway track, at the bottom it might be floor. In this image beneath that there is a tunnel, trees, building made of cardboard and a smow. In the background there wall and the top of the image there is ceiling with lights.
Global DINO
2
scale model,
toy trains the track, which is on the bridge,
3 In this image,
train the track and there people inside the train. In the background, there are trees, railings, stairs, plants, flowers and there is a rock wall and a shed. On the left,
pole.
4 In this image, can see toy train a toy railway track and there are few other toy railway tracks. In the top left side the image, can see an object on the surface. this picture I can see miniature set, where can see railway tracks, a and overhead tank. see that, it is blurred in the background.
5
1
In this image there is person standing.
2
In this image we can see person standing on the floor and stand.
Local DINO
3
In this image there is a person standing and looking at the right side of the image, behind him there few people.
4
This image consists of person. At the bottom, there are clothes the legs. The person is standing on the floor.
5 In the middle the image a person is standing and watching. Behind him we can see locomotive.
this image the we can see colored background. the middle of the image can see the toy houses and buildings. We can see the toy trees and plants. We can see the railings. We can see the toys.
This image consists miniatures.
Global CLIP
2
In this image we can see miniature model. There are railway tracks. Also there a train. And there is a building with windows. Also there is sky.
3
There is model of a train in the foreground area of the image and the background is white. In this image we can see scale model, we can see toy trains on the track, which is on the bridge, beneath that there is tunnel, trees, building made up of cardboard and snow. In the background there is wall and at the top of the image there is ceiling with lights.
4
5
In this image in the center looks like a toy train and there is toy railway track, at the bottom it might be floor.
1
In this image there person standing and looking at the right side of the image, behind him there are few people.
Local CLIP
2
In the middle the image a person is standing and watching. Behind him we can see a locomotive. this image there person standing.
3
4
In this image we can see rocks and a person on the land.
5
In this image
slanding and holding object, before him there is animal and we can see grass. On the left there are rocks.
Original
In this picture we can see a machine on a wooden platform. We can see poles, metal chains, a ladder and few objects. We we can see rock hills and the sky.
rocks and there are stones the ground. In the background
Table 19:
The query image
is used to retrieve the captions.
50

---

Published in Transactions on Machine Learning Research (3/2026)
Model Rank Caption
Here in this picture we can see a group of people standing over a place and we can see all of them are wearing ice skates, gloves and helmet and we can see they are standing on an ice floor and playing ice hockey, as we can see they are holding hockey sticks in their hands.
Global DINO
2 In the center of the image we can see three people are ice skating and they are in different costumes. And we can see they holding sticks and they are wearing helmets. In the background there is wall, transparent glass, banners with some text and barriers. On one of the barriers, we can see an object.
3 Here in this picture we legs and we can see they are wearing gloves, helmet and holding hockey sticks in their hands and we can see they are playing ice hockey over there and on the right top side we can a goal post with net present and we can see a person is standing near it with helmet, gloves, knee pads and ice skates and in the bottom we can see the glass walls present.
group of people skating the ice floor with ice skate under their
4 Here in this picture we can see some people skating on the ice floor present over place and we can see all of them are wearing ice skates, gloves, helmet and holding a ice hockey stick in their hand and behind them we can see hoarding present and we can also see fencing present and we can see number of people standing and sitting in the stands over there and watching the game.
5
In this picture I can see a group of people wearing ice skating shoes and standing on the ice. There are few people holding hockey sticks. I can see sports net with poles.
1
In this picture I can see group of people wearing ice skating shoes and standing on the ice. There are few people holding hockey sticks. T can see sports net with poles.
Local DINO
2
In this image can see two people with ice-skates and one person holding the stick. These people are the ice.
3
In this image can see the people wearing the helmet and holding the hockey sticks and playing the ice hockey. the background we can see some person's legs. In this image, there a man wearing helmet and seems like he is playing ice hockey, At the top, there are legs of person.
4
5
In this image, I can see a person standing on ice with helmet, gloves, ice skates and holding an ice hockey stick. In the background, can a group of people standing and there are walls.
1
Here in this picture we can see some people skating on the ice floor present over a place and we can see all of them are wearing ice skates, gloves, helmet and holding a ice hockey stick in their hand and behind them we can see hoarding present and we can also see fencing present and we can see number of people slanding and sitting in the stands over there and watching the game.
Global CLIP
2 In this picture we can see a man and a woman wearing ice skates visible on the floor. We can see the lights, stairs and other things. We can see a few people and the dark view in the background. 3 Here in this picture we can see group of people standing over a place and we can see all of them are wearing ice skates, gloves and helmet and we can see they are standing on an ice floor and playing ice hockey, as we can see they are holding hockey sticks in their hands.
4
In this image we see children doing ice skating ice. In the back there is wall.
5
In this image we can see two persons ice skating. We can see a person holding hockey stick. In the background, we can see people, boards with text, wall, poles and some objects.
1
In this image there a person standing and looking at the right side of the image, behind him there are few people.
Local CLIP
2
In this image we can see person standing on the floor and stand. In this picture, we see two girls and the boys are standing. They might be exercising. On the right side, we see the legs of two people. At the bottom, we see the floor.
3
4
This image consists of person. At the bottom, there are clothes the legs. The person standing on the floor.
5 In this image there are people standing, in the background there are clothes.
Original
In the picture can see the design floor and there are objects on the floor. can see the lights at the top of the picture. can the logos the wall and there are glass windows the top left side of the picture.
Table 20: The query image
is used to retrieve the captions.
51

---

Published in Transactions on Machine Learning Research (3/2026)
MS COCO dataset's Image
Caption Retrieval
Tables
22
23
24
and
show results on the MS COCO validation set.
Model Rank Caption
A kite being flown in the middle of a beach.
2
People flying kites on a sandy beach while bucket sits in the sand.
Global
3
Kites being used by people on a beach.
DINO
A group of people flying kites at the beach Two people
5
beach flying a kite in the air.
1
A kite being flown in the middle of a beach. People flying kites sandy beach while bucket sits in the sand.
2
Local DINO
3
A person standing
top of beach flying a kite.
4
Kites being used by people on a beach.
5
A shot of the blue water with people flying kite.
1
A kite being flown in the middle of a beach.
2
A person standing on top
beach flying a kite.
Global CLIP
A man flying kite at the beach.
man flying kite at on the shore the beach
5
Kites being used by people
beach.
A person standing
top of beach flying a kite.
2
A kite being flown in the middle of a beach.
Local CLIP
3
People flying kites on a sandy beach while bucket sits in the sand.
4
A man is flying kite at the beach.
5
A man flying a kite on a beach with people standing around.
Original
A man flying a kite at the beach.
Table 21:
The query image
is used to retricve the captions. Green
color is for the original caption from the dataset.
52

---

Published in Transactions on Machine Learning Research (3/2026)
Model Rank Caption
A girafle is walking in some tall grass
2
A giraffe standing on a grass covered field.
Global DINO
3
A single giraffe looks over the green brush. there is very tall girafle standing in the wild
4
5
giraffe in field with trees in the background
1
A giraffe standing
grass covered field.
2
A tall girafle standing lop of grass covered field.
Local DINO
3
there is a very tall giraffe standing in the wild
A giraffe is walking in some tall grass
5
A giraffe standing by a pair of skinny trees.
1
A giraffe stands near a tree in the wilderness.
2
A giraffe is walking some tall grass
Global CLIP
3
A girafle standing on grass covered field.
4
A group of giraffes that are standing in the grass. there is very tall giraffe standing in the wild
5
1
A Iall girafle standing lop of grass covered field.
2
A giraffe standing on grass covered field.
Local CLIP
3
A giraffe is walking in some tall grass
4
A single giraffe looks over the green brush. there a very tall girafle standing in the wild
5
Original
A single giraffe looks over the green brush.
Table 22:
The query image
is used to retrieve the captions. Green
color is for the original caption from the dataset.
53

---

Published in Transactions on Machine Learning Research (3/2026)
Model Rank Caption
A man riding a surfboard on a wave in the ocean.
A surfer in the ocean trying not to wipeout.
Global DINO
3
A man on surfboard riding wave the occan.
4
person riding a skale board on wave
5
A man is surfing
small wave.
1
A group of people swimming in the occan with surfboard.
2
there are many surfers that are in the water
Local DINO
3
Pair of surfers paddling out to open ocean.
A man riding on the back of a surfboard next to kids.
5
a man on a blue surfboard on top of some rough water
1
A man riding a surfboard on a wave in the ocean.
2
A man on surfboard riding a wave the occan.
Global CLIP
3
person riding a skate board on wave
4
person riding surf board on wave
5
A man surfboard, who is riding wave.
1
A person surlboard in the water.
2
A para sailor with his board with sail in the surf.
Local CLIP
3
A group of people swimming in the ocean with a surfboard.
4
A man riding on the back of a surfboard next to kids.
5
Pair of surfers paddling out to open ocean.
Original
person riding a surf board on wave
Table 23:
The query image
used to retrieve the captions. Green
color is for the original caption from the dataset.
54

---

Published in Transactions on Machine Learning Research (3/2026)
Model Rank Caption
A brown bear walking with rocks in the background.
2
A large brown bear standing next to a pile of rocks.
Global
3
A big burly grizzly bear is show with grass in the background.
DINO
4
A majestic bear looks out across grass plain.
5
brown bear is walking away from a river
1
A baby brown bear standing on top of a rock.
2
A majestic bear looks out across grass plain.
Local DINO
3
A brown bear walking with rocks in the background.
4
A statue of large brown bear tcaring off cars door.
5
A brown bear lays down in the woods.
1
A brown bear walking with rocks in the background.
2
A large brown bear standing next to a pile of rocks.
Global CLIP
3
A big burly grizzly bear is show with grass in the background.
4
A baby brown bear standing on top of a rock.
5
A majestic bear looks out across grass plain.
1
A baby brown bear standing on top of a rock.
2
A majestic bear looks out across grass plain.
Local CLIP
3
A brown bear walking with rocks in the background.
4
A big burly grizzly bear is show with grass in the background.
5
A large brown bear standing next to a pile of rocks.
Original
A brown bear walking with rocks in the background.
Table 24:
The query image
is used to retrieve the captions. Green
color is for the original caption from the dataset.
55

---

Published in Transactions on Machine Learning Research (3/2026)
Model Rank Caption
Person cooking an eggs on a black pol on a stove.
2
A man pokes his head front of an oven open to baking cookies.
Global DINO
3
Belgium waffle loaded with bananas topped with powdered sugar with syrup and more fruit garnish.
4
Several breakfast foods
top of refrigerator. A plate has waflle, some fruit and ice cream it.
5
1
A person is holding a spatula near slices of bread on stove.
2
Twp cake pans sitting and cooling on the stove
Local DINO
3
an image of a man slicing a small pizza
4
A woman observing something on a kitchen stove.
5
A man pokes his head front of an oven open baking cookies.
A pastry station, with an assortment of fillings and sauces
2
A young man is working behind a counter.
Global CLIP
3
A group of three chefs preparing food in kitchen.
4
A man preparing food in restaurant kitchen.
5
The donut robot machine is mechanically making donuts.
1
A table with many different objects, including a plate of sandwiches.
2
A pastry station, with an assortment fillings and sauces
Local CLIP
3
A table topped with plates, bowls and containers of food.
A buffet of casserole dishes on a kitchen counter.
5
A bunch of items that are on counter.
Original
A pastry station, with an assortment of fillings and sauces
Table 25:
The query image
is used to retrieve the captions. Green
color is for the original caption from the dataset.
56

---

Published in Transactions on Machine Learning Research (3/2026)
H.2 Caption + Image Retrieval (In- -i-Diisttibbtion
We evaluate cross-modal alignment through caplion-lo-image retrieval using SPARC latent representations. Figures
31
33
and 34 show retrieval results comparing Global vs Local TopK training configurations.
The four model configurations represent:
Query caption's CLIP-text features + Reference database of CLIP-inage features (both from Global SPARC)
Global CLIP:
Local CLIP: Query caption's CLIP-text features Reference database of CLIP-inage features (both from Local SPARC)
Global DINO: Query caption's CLIP-text features + Reference database DINO features (both from Global SPARC)
Local DINO: Query caption's CLIP-text features Reference database of DINO features (both from Local SPARC)
Each model shows top-10 retrieved images ranked by cosine similarity in the SPARC latent space.
Open Images dataset's Image -> Caption Retrieval
Figures and
show results on the Open Images test set using diverse query captions.
Global TopK (DINO)
Local TopK (DINO)
Global TopK (CLIP)
Local TopK (CLIP)
0
01
Figure 31: Images retrieved for captions are (1) "In this picture we can see some food products in the glass jars.", (2) "In this image might be taken in the airplane. In this image we can see the speedometers, knobs and some digital screens." Captions are from Open Images test dataset and images are retrieved from the same split. Green boxes indicate when the corresponding image for a caption is successfully retrieved. The second caption shows such match (Global TopK with CLIP, 3rd rank).
57

---

Published in Transactions on Machine Learning Research (3/2026)
Global TopK (DINO)
Local TapK (DINO)
Global TopK (CLIP)
Local Topk (CLIP)
Figure 32: Images retrieved for captions are (1) 'In this image in the center there is one bird flying, and in the background there is sky.', (2) "In this image we can see a table on which some glasses are there which some food items and straws are there and we can see a pot like structure. On the left side we can see a person hand. Tn the background we can see some posters and bottles.', (3) 'In the picture we can see an ice cream with green and brown color cream.', and (4) 'In this image we can see a wooden basket placed on the ground, we also see the photo frame on the wall.' Captions are from Open Images test dataset and images are retrieved from the same split. Green boxes indicate when the corresponding image for caption is successfully retrieved. The third caption shows such a match (Global TopK with CLIP, 1st rank).
58

---

Published in Transactions on Machine Learning Research (3/2026)
MS COCO dataset's Image -> Caption Retrieval
Figures 33 and 34 show results on the MS COCO validation set. We show of the captions available per
image in MS COCO in our results.
Global Topk (DINO)
Local Topk (DINO)
Global Topk (CLIP)
Local Topk (CLIP)
Figure 33: Images retrieved for captions are (1) "A woman stands in the dining area at the table.", (2) "A shower curtain sits open in an empty and clean bathroom." " (3) "a man a blue shirt and red tic." and (4) 'These people are going to have pizza and wine." " Captions are from Open Images lest dataset and images are retrieved from the same split. Green boxes indicate when the corresponding image for caption is successfully retrieved.
59

---

Published in Transactions on Machine Learning Research (3/2026)
Global Topk (DINO)
Local TapK (DINO)
Global Topk (CLIP)
Local Topk (CLIP)
Figure 34: Images retrieved for captions are (1) WWhte and orange fur lays on a white blanket.", (2) 'A kitchen that has carpeted floors and wooden cabinets.", (3) "A street scene with horse pulling white carriage.", and (4) 'A large dogs comfortably sleeping someones bed". Captions are from Open Images test dataset and images are retrieved from the same split. Green boxes indicate when the corresponding image for a caption is successfully retrieved.
60

---

DINO Local: Query image's DINO features Reference database of CLIP features (both from Published in Transactions on Machine Learning Research (3/2026)
H.3 Image -> Image Retrieval (In- Distribution)
We evaluate cross-model aligument by relrieving images across DINO and CLIP-image encoders using SPARC latent representations. Figures 35 vs Local TopK training configurations.
36
37
and 38 show results in a 4-row layout comparing Global
The lour rows represent:
DINO Global: Query image's DINO features + Reference database of CLIP features (both from Global SPARC)
Local SPARC)
CLIP Global: Query image's CLIP features + Reference database of DINO features (both from Global SPARC)
CLIP Local: Query image's CLIP features + Reference database of DINO features (both from Local SPARC)
Each row shows the query image (left) followed by 10p-10 retrieved images.
Open Images dataset's Image -> Image Retrieval
Figures 35
36
and 37 show cross-model retrieval results on the Open Images test set.
Query
2
3
4
5
6
7
9
10
CLIP Golell
CUP Guael
CUP Lock
Figure 35: Cross-model image retricval results. Each image shows a 4-row layout comparing query stream and reference database combinations. Query image (left) with top-10 retrieved images from reference database (right). All models trained with X = 1 on Open Images training set. The retrieval is done on the test set of Open Images. Green border is used to show the exact match of the query image was found.
61

---

Published in Transactions on Machine Learning Research (3/2026)
Query
2
4
5
6
7
9
10
CSE 9964
Cross-model model image retrieval results. Each image shows a 4-row layout comparing query stream and reference database combinations. Query image (left) with t0p-10 retrieved images from reference databaso (right). All models trained with À = on Open Images training set. The retrieval is done on the Lest set of Open Images. Green border used to show the exact match of the query image was found.
Figure 36: Cross-
62

---

Published in Transactions on Machine Learning Research (3/2026)
Query
2
3
4
5
6
7
9
10
CuP CaPENNe
DRO
Figure 37: Cross-model image retrieval results. Each image shows a 4-row layout comparing query stream and reference database combinations. Query image (left) with top-10 retrieved images from reference databaso (right). All models trained with À = on Open Images training set. The retrieval is done on the lest set of Open Images. Green border used to show the exact match of the query image was found.
63

---

Published in Transactions on Machine Learning Research (3/2026)
MS COCO dataset's Image Image Retrieval Figure 38 results on the MS COCO validation set.
shows cross-model mage-to-immaee retrieval
Query
2
3
4
5
7
8
9
10
MASTERS
TTET ARLO
PA PARBAS
MAST S
NTE ARLO
PA ALYAAS
MASTERS
BAS
TTETARLD
MASTERS
BAS
DNO
Figure 38: Cross-moded image retrieval results. Each image shows a 4-row layout comparing query stream and reference database combinations. Query image (left) with top-10 10 retrieved images from reference database (right). All models trained with = 1 on MS COCO training set. The retricval is done on the validation set of MS COCO. Green border is used to show the exact match the query image was found.
64

---

Published in Transactions on Machine Learning Research (3/2026)
H.4 External Image - Caption (OOD)
evaluate SPARC's out-of-distributtion generalization using external images nol present in the Open Images
dataset, following the same methodology Section H.1 Tables
26
27
28
29
30
31 and 32 show top-5
retricved captions for cach model configuration using external query images with the Open Images test set as the reference database.
ModelRank
Retrieved Caption
In this image we laptop with screen and keys. the bottom the image there is surface. On the surface we can see reflections. On the image there a watermark.
Global DINO
In this image we see black and white picture of laptop, we also see some pictures on the screen. In this image there poster. There is screen, sound speakers, laptop the devices. Bottom of the image grey color. Top of the image there there is some text. Background laptop containing
3
text.
4
In this image
text its
In this picture
keypad of laptop.
In this image we can see laptop.
In this picture I can observe black color joystick on the laptop. There is an apple logo on the laptop. The background is in white color.
Local
DINO
In this image we can see laptop containing some text on its screen.
There a white color laptop present on the cloth as we can see in the middle of this image. There is one device connected to the laptop. Tt. is dark in the background.
the center of the image we can see the text. the background the image we can see a laptop. On the laptop we can see the objects.
This image taken indoors. the background there wall. the middle the image there are two laptops on the table.
Global CLIP
this picture we can see laptop, keys and screen, In the background the image it is blurry. In this image there laptop on the table. There are letters, numbers, symhols the keyboard. There some text at top of the image.
4
this picture we In this picture
laptop with keys and this laptop
text, symbols and buttons. a person, numbers, symbol and few things the screen of this laptop.
5
laptop, We
1
In this picture looks like laptop. We the laptop keyboard which has the alphabet and the number the buttons. At the bottom,
keys. We the text written the laptop and the laptop.
the touchpad
Local CLIP
2
In this picture In the image there is a laptop cloth and in front of the laptop there is teddy bear and it seems like both the things are kept sofa and in the background there is wall.
laptop with keys and this laptop
text, symbols and buttons.
3
4
In this picture can laptop. We can a person, numbers, symbol and few things the screen of this laptop.
5
In this image, I can see laptop. On which can see keys having letters, numbers, symhols and some text.
Table 26: The query image
is used to retrieve the captions.
65

---

Published in Transactions on Machine Learning Research (3/2026)
Rank
Retrieved Caption
In this image we can see laptop with screen and keys. At the bottom of the image there is surface. On the surface can see reflections. On the image there is a watermark.
Global
In this image we can see black and white picture of a laptop, we can also see some pictures on the screen. In this image there a poster. There is screen, sound speakers, laptop on the devices. Bottom of the image there is some text. Background is in grey color. Top of the image there is some text.
DINO
In this image we can see laptop containing some text on its screen.
In picture we can see keypad laptop.
In this image we see laptop.
In this picture I observe black color joystick the laptop. There is apple logo on the laptop. The
Local
In this image background white color.
DINO
laptop containing There is white color laptop present the cloth device connected to the laptop. It is dark in the background.
text its see in the middle of this image. There is one
4
In the center of the image we see the text. In the background of the image can laptop. On the 5
laptop we see the objects.
This image is taken indoors. In the background there is wall. In the middle of the image there are two laptops the table.
Global CLIP
2
In this picture we see laptop, keys and screen. In the background of the image it is blurry. In this image there is a laptop on the table. There are letters, numbers, symbols on the keyboard. There is some text at the top of the image.
In this picture we can see laptop with keys and on this laptop screen we can see text, symbols and buttons. In this picture we can see a laptop. We can see a person, numbers, symbol and a few things on the screen of this laptop-
In this picture looks a laptop. We see the laptop keyboard which has the alphabet keys. We see the text written on the laptop and we see the buttons. At the bottom, we see the touchpad on the laptop.
number
Local CLIP
this picture we can see laptop with keys and on this laptop screen we can see text, symbols and buttons. In the image there a laptop both the things are kept on a sofa and In this picture can see a laptop. We can see a person, numbers, symbol few things on the screen this laptop.
cloth and
the laptop there a teddy bear and seems like background there is wall.
4
5 In this image, see laptop. On which
keys having letters, numbers, symbols and text.
Table 27: The query image
is used to retrieve the captions.
66

---

Published in Transactions on Machine Learning Research (3/2026)
Rank
Retrieved Caption
In this picture we can see few people, snowy mountains and hills. We can see other things and the cloudy
sky in the background.
Global
In the picture we can see the mountains covered with the snow and behind it, we can see the sky with clouds.
DINO
In this picture we can see the man wearing a black jacket and standing. In the front we can see some stones. Behind we can see mountains and some snow. On the top we can see the sky and clouds. In this picture can see few people. There are rocks. can see snowy mountains, and the background there is the sky.
In image we can see mountains with snow. In this picture can see mountains can see snow and blue cloudy sky. can see text the bottom left corner picture and looks like cross symbol on the mountain.
background there is sky.
Local
In the image can the person standing, wearing clothes, gloves, shoes and spectacles, here we the stones, snow, mountains and the sky.
DINO
this picture,
the jacket, standing. He wearing red cap, spectacles and the gloves. the left. side, see snow. In the background,
the hills. These hills covered with the snow.
4 In this image white color cap and gaggle well. We also bag and stick the covered ground. In background can see mountain and clear blue sky.
in bluc color jacket siting
covered ground and he is also wearing
5
In this picture I can see there person and he wcaring coat and bag pack, there fow mountains in the background and they covered with rocks and
In this picture can mountains and
snow and blue cloudy sky. I can text the bottom left symbol the mountain.
corner of the picture and looks like rocks, mountain, soil and people. In the middle of the image there mountains. In the background it is the sky.
Global CLIP
2
In the foreground
3
In this image can see person standing on the mountain, he is wearing bag. There are few rocks on the mountain, many rocks and snow the mountain. In background can mountains and clear blue sky.
more there few mountains in the background and the sky is clear.
4
In this image we
In this picture can see few people. There are rocks. T can see snowy mountains, and in the background there is the sky.
In this image can some people the mountain, and there is snow the mountain and the sky is bluc. In this picture we can see few people, snowy mountains and hills. We can see other things and the cloudy sky in the background.
Local CLIP
In this picture I can see person standing in a foreground of the image and there are few rocks visible, in the background there is a mountain visible and there are few trees and there is snow covered on the mountain and I can see the sky.
In this image in the center there are a group of people who are standing, and they are wearing bags. And at the bottom there is snow and one stick is there, and in the background there are mountains and at the top of the image there is sky.
In the image there are few people sitting on the rocks. And there is a person stauding on the rocks. In the background there is a hill covered with snow. And also there are few hills in the background. At the top of the image there is sky.
Table 28:
The query image
is used to retrieve the captions.
67

---

Published in Transactions on Machine Learning Research (3/2026)
Rank
Retrieved Caption
In this picture can see a seat otter in the water and looks like rock at the top left. corner. In the middle of this image I can see a sea otter in the water. At the bottom, can see the sticks and leaves in the water.
Global
DINO
this picture, we see sea otter is swimming in the water. the background, we see water and this water might be in the swimming pool.
the bottom picture, we see the rocks. front the picture, we see walrus sleeping on the rock.
Behind that, we see water and this water might be the pond. There are rocks and a walrus the background.
this image we can see a sea lion the water.
this image we can see paper. the paper we can see painting of birds on branch. Also we can see this image,
leaves. And there text on the paper.
Local
this image, can see painting of parrols on the branch and some
the bottom. animal picture cream surface. At top and bottom of the image,
DINO
4
red color crab with sticks paper which something written. In this picture can see the bactrian camel in the foreground. It is looking like the green grass, plants in the background.
In this image
looking like the fence the right side. In this image In the image can see water, the water can some rafts. On the rafts we can see some seals.
lion in the water.
2
Global
In this image can see two sea lions in the water.
CLIP
In this image we can see sea lion in the water. this image we can see sea lion this image can see the hippopotamus and the rock in the water. this image can see hippopotamus. Also we can see water. In the back we can see stones.
water.
Local CLIP
In this picture, hippopotamus the right top, we see the railing and wall. We see rods In this image
the water. It might be pool. At the bottom,
wall. In stands in the pool. trees.
4
hippopotamus water and ground. In the background In this image can see an animal in the water, looks like hippopotamus.
Table 29:
The query image
is used to retrieve the captions.
68

---

Published in Transactions on Machine Learning Research (3/2026)
RadklRank
Retrieved Caption
In this image there is ground at the bottom. There are rocks in the foreground. And there is greenery in the background. And there is sky at the top.
Global
In this image there are few rocks, graas, a river, tree and in the background there is the sky. this picture we can see rocks, trees and the background we can see the sky with clouds. this image we can see rocks. On the ground there is grass, In the background there sky. Tn this image we can see rocks. Also there are people. the background there is sky.
DINO
This image consists of stones, rocks, grass, group of people, and the sky.
2
In this image
see grass the ground. There are trees. In the background stones and here
stones, Also there sky with clouds.
Local
DINO
3
In this picture
the grass and trees the ground. In the background the sky.
4
In this image can rocks. On the ground there is grass. In the background there is sky. In this picture, we see the stones, rocks and the grass. In the background, we see the stones and the rocks. In this image we can see stones. Also we can see grass on the ground. There are trees. In the background there is sky with clouds.
Global CLIP
In this image we can see rocks. Also there is water. On the left side we can see grass on the ground. In the
background there is sky with clouds.
In this picture we can see stones and here we can see the grass and trees on the ground. In the background we can see the sky.
In this image we can see in front. there are many rocks, trees, at the back there are hills, mountains, the sky is at the top. There are stones and grassland the foreground area of the image, there are people, trees, grassland and the
sky in the background.
this image we can see rocks. On the ground there is grass. In the background there sky. In this picture, we see the stones, rocks and the grass. In the background, we see the stones and the rocks.
Local CLIP
This image consists of stones, rocks, grass, group people, trees and sky. 4 There stones and grassland in the foreground
of the image, there people, trees, grassland and the sky the background.
5
In this picture can stones and here we can see the grass and trees the ground. In the background the sky.
Table 30:
The query image
is used to retrieve the captions.
69

---

Published in Transactions on Machine Learning Research (3/2026)
Rank
Retrieved Caption
In this image in the foreground there is bree, and at the bottom there is a river and in the background there are hills and trees. And at the top there is sky.
Global
In this image, there are mountains and clouds.
DINO
this picture we can see hills in the background, it. looks like snow at the bottom, we can see the sky
top of the picture, there are clouds in the middle.
this image, there a sewing machine. Under this sewing machine, there is sheet. On the right side of this image, there is gray color sheet. And the background of this image is dark in color.
this image, there are few electronic devices on there a woman sitting on the chair on the floor operating a computer. the background, there are glass walls, few lights to the ceiling, chair, dustbin, few electronic devices on the desk and also there are few drawers on the right.
and also there are drawers the bottom. Beside,
this image there is dog on the rock. Behind the dog there are plants and trees. In the background of the image there are mountains. At the top of the image there are clouds in the sky.
Local
In this image can see person standing on mountain, is wearing bag. There are few rocks on mountain, can see there are few mountains in the background and the sky clear. and there background there is snow on mountains and the sky.
DINO
In this image we can see person sitting on rock, front the person there a water dog.
In image there are cats sitting on stones. At top the image there water. In this picture there dogs and
5
rocks, and water. In the background of the image hills and sky with clouds.
In this image there dog couch which In the image can the picture of dog which is the seat. In the image there is dog couch. And also there is pillow and towel.
the floor. Background is blurry.
2
Global CLIP
3
In this image there are two dogs on a sofa, on that dogs there a blanket, in the background there is a wall. In this picture, we see dog is on sofa or on bed. The dog is in black and white color. The leash of the dog is in red color. In the background, we see a bed sheet or a cloth in brown color.
In the image there is dog walking on the rock surface, behind the dog there are huge rocks. In this image we can see person sitting on a rock, in front the person there a water flow and there is dog. In the background there is snow on the mountains and the sky.
Local
CLIP
this image, see group of dogs standing on a stone field with belts around their neeks and see couple of this picture there are dogs and we can see rocks, trees and water. background of the image we can see hills and sky with clouds.
persons standing beside them.
the bottom image there dog in the water. the background there are rocks, And also there arc
few stones in the water.
Table 31:
The query image
is used to retrieve the captions.
70

---

Published in Transactions on Machine Learning Research (3/2026)
Rankl
Retrieved Caption
In this image can see two umbrellas with lot of colors like pink, blue, yellow, orange, red, green, and the sky is cloudy.
Global
In this image can see different color of umbrellas. In the background I can seo clouds in the sky. This image consist an umbrella. In the middle, we can see a pole. The background is blue color.
DINO
the top of umbrella. This image consists of umbrella along with metal rod.
top, there is sky.
Tn this picture can a partial part colorful umbrella. We In this image we different colors and background can see white color iron and can different paintings in in color.
blue color umbrella and the umbrella
tip the umbrella
Local
DINO
the middle the image In this image, there is black and red umbrella the white surface. Behind it, there is white wall. In this picture there is woman holding the umbrella. At the back there is a wall. At the bottom there is a flour.
umbrella on the dried grass.
4
In this image
holding an umbrella.
In this image we can see blue color umbrella and on the umbrella we can see some different paintings in different colors and in the background we can see white color iron and can see the tip of the umbrella is in red color.
Global CLIP
2
This image consist an umbrella. In the middle, can a pole. The background is bluc in color. In this image, I can see an umbrella with colorful design and there is thread. In the background, can see glass windows. In the bottom left corner of the image, there is an object.
In this image can see two umbrellas with lot of colors like pink, blue, yellow, orange, red, green, and the aky is cloudy.
this image, we can see an umbrella.
this image we can see a man holding an umbrella. this picture there is a woman holding the umbrella. At the back there a wall. At the bottom there a floor.
Local CLIP
In this image, we can see an umbrella.
4
In the middle of the image can an umbrella the dried grass. In this image, in the foreground we see person wearing costume and hold an umbrella. On the right side,
5
person. Background of the image is blurred.
Table 32:
The query image
is used to retrieve the captions.
71

---

Published in Transactions on Machine Learning Research (3/2026)
H.5 Caption Image (OOD)
We evaluate SPARC's capability with form caplions not present in the Open Images dataset, following show retrieval results using simple descriptive
the same methodology as Section H.2 Figures 39
40
and
captions to query the Open Images test databasc.
Global Topk (DINO)
Local TooK (DINO)
Global Topk (CLIP)
Local Tapk (CLIP)
OL
Figure 39: Images retrieved for captions: (1) 'Black cat", (2) 'Orange cat', (3) 'A child flying a red kite on
sunny beach", (4) 'A barista making coffee behind a counter.' The images are retrieved from the test set of the Open Images datasct, but the captions are not part of the datasct.
72

---

Published in Transactions on Machine Learning Research (3/2026)
Global (DING)
Tonk
Global
Local
PM
The priest and the stripper
Figure 40: Images retrieved for captions: (1) 'People crossing a busy city street in the rain', (2) 'A passenger airplane flying clear blue sky", (3) dog running through a grassy field', (4) man reading a newspaper at bus stop.' The images are retrieved from the test of the Open Images dataset, but the captions are not part of the datasct.
73

---

Published in Transactions on Machine Learning Research (3/2026)
Global (DING)
Tonk
Global
Local
0
Sk
Figure 41: Images retrieved for captions: (1) "A person riding bicycle on a country road', (2) 'A bowl of fresh strawberrie on a kitchen counter", (3) "A train arriving at an underground metro station", (4) "Books stacked on wooden desk." The images are retrieved from the test of the Open Images dataset, but the captions not part of the datasct.
74

---

Published in Transactions on Machine Learning Research (3/2026)
H.6 External Image->lmage (OOD)
We evaluate SPARC's oul-of-distribution generalization using external images not present in the Open Images dataset for rross-model image retrieval. Figures 42 using external query images with the Open Images test set as the reference databasc.
43 and 44 show cross-model retrieval results
Query
2
4
7
9
10
Figure 42: Cross-model image retrieval results using OOD qucry images. The references are from the Open Images test sel.
75

---

Published in Transactions on Machine Learning Research (3/2026)
Query
2
4
5
7
9
10
CLP
Figure 43: Cross-model image retrieval results using OOD query images. The references from the Open
Images test set.
76

---

Published in Transactions on Machine Learning Research (3/2026)
Query
2
3
4
5
6
7
9
10
OND MNOSAN
DRO
DIO
Figure 44: Cross-model image retrieval results using OOD query images. The references are from the Open
Images test set.
77


Reviews:
[{"review_id": "cDZ0tjHx2t", "reviewer": "Reviewer_8EwW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduced a method (SPARC) that learns a unified latent space across modalities (CLIP image, CLLP text and DINO). The main innovations over vanilla SAE are: 1. a cross-modality Top K mechanism (as related to Top K in SAEs); 2. a cross-modality re-construction loss. Both innovations were shown to improve cross-modality re-construction loss over standard SAEs and CLIP similarity.\n\n**Strengths**:\n\n1.\tThe method is well-motivated and intuitive.\n\n2.\tThe evaluation covered multiple aspects of desired outcomes.\n\n3.\tAbalation studies clearly demonstrate the merits of the innovations.\n\n**Weaknesses**: \n\n1.\tA direct comparison with one existing method (USAE) seems to be missing.\n2.\tThe improvement in cross-modality alignment comes at the cost of self-reconstruction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, the experiments are nuanced and well explained, supporting the claim of cross modality performance. Please refer to the requested change for one statement where the evidence seems insufficiently presented.", "audience_yn": "Yes", "audience_explanation": "Learning a unified latent space across modalities is interesting and valuable for composite tasks.", "requested_changes": "In the last paragraph of the introduction, it was briefly mentioned that USAE suffers from “inconsistent concept activation across models” and this weakness is addressed by this paper (SPARC). This point seems insufficiently supported by evidence: the experiments showed that SPARC produced consistent concept activation but not how USAE underperformed in this aspect. It would be helpful to add citations or experiments that supports the claim.\n\nTypo: Figure 3 “with \\lambda = 1 (top right)”: should this be “with \\lambda = 0 (top left)”?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FJKfTGXSpN", "reviewer": "Reviewer_w77c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes SPARC, a sparse autoencoder framework that learns a shared, interpretable latent space across heterogeneous model architectures and modalities, overcoming the isolation of concept spaces produced by standard SAEs. SPARC introduces two innovations to achieve aligned representation: a Global Top-K sparsity mechanism that enforces consistent latent activations across streams, and a cross-reconstruction loss that encourages semantic consistency between models. Experiments demonstrate large gains in concept alignment, with Jaccard similarity improving to 0.80 compared to 0.26 for Local Top-K, along with a substantial reduction in dead or partially aligned neurons and improved cross-stream reconstruction fidelity. These aligned sparse latents also enable practical applications, including cross-modal attribution and weakly-supervised segmentation in vision models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, the empirical evaluation supports the central claims regarding cross-model concept alignment. The paper includes comprehensive ablations demonstrating that both core components, namely Global Top-K sparsity and cross-reconstruction loss, are necessary to achieve high alignment performance. This is supported by the substantial increase in Jaccard similarity to 0.80 and the elimination of mixed-activation neurons across streams, as well as improvements in cross-stream reconstruction quality. The experiments also present practical applications such as cross-modal attribution and weakly supervised segmentation, providing additional confirmation that the learned latents retain semantically meaningful information across modalities.\n\nThat said, some interpretability claims would benefit from deeper validation. While the work argues for improved monosemanticity, the evaluation does not include automated interpretability scores or human annotations of latent semantics. In addition, comparisons to related approaches such as USAE remain primarily qualitative, and a more direct quantitative comparison would strengthen the argument for superiority. Despite these limitations, the evidence is largely convincing with respect to the paper’s primary objectives of achieving structurally and semantically aligned latent representations.", "audience_yn": "Yes", "audience_explanation": "The work directly advances cross-model interpretability, a topic of broad interest in both vision and multimodal communities. The method is applicable to many model families and aligns well with current interest in feature-level interpretability, sparse dictionary learning, and unified concept spaces across architectures.", "requested_changes": "1. Add quantitative comparisons against USAE: Provide numerical alignment metrics to strengthen claims of improvement over prior cross-model SAE approaches.\n2. Include additional interpretability evaluation: Incorporate ablation experiment metrics or human-verified labeling to better validate interpretability claims beyond alignment.\n3. Expand the diversity of evaluated streams: Add more modalities or distinct model types in the core experiments to support generalization claims.\n4. Provide computational cost and runtime analysis: Clarify the efficiency trade-offs introduced by Global Top-K selection and cross-reconstruction loss relative to baseline approaches.\n5. Discuss segmentation and attribution limitations: Offer a more nuanced evaluation of failure modes or conditions where SPARC’s downstream performance remains modest.", "broader_impact_concerns": "The work unlocks cross-model interpretability, which is largely positive. However:\n- Shared concept spaces could enable model-to-model transfer of harmful or private representations if misused.\n- Text-guided spatial localization can be misapplied for surveillance or privacy-sensitive content analysis.\n\nI recommend adding a Broader Impact section noting dual-use risks and proposed mitigations. No critical ethical flaws identified.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mIxGYsonWg", "reviewer": "Reviewer_7VUR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors aim to align the representations learned between different ML models and modalities. Previous approaches of (universal) sparse autoencoders tried to achieve this by randomly selecting one encoder to produce a latent vector and decode in all modalities. With the goal of improving latent-, and therefore, concept alignment, the authors propose to simultaneously produce a joint activation vector from all encoder outputs and select the same top-k indices from every latent stream. To further promote latent alignment the authors employ a cross-reconstruction loss to reconstruct expected latent outputs of one stream from those of the others.\n\nThe authors inspect latent/concept alignment by measuring the simultaneous activation/vanishing of activations across streams and Jaccard similarity to concepts. Results indicate that the global top-k loss strongly promotes joint activations. Furthermore, the authors inspect the ability to reconstruct latent across streams and monosemanticity showing moderate reconstruction and class prediction abilities, as well as improved alignments for activation maps and relevance scores. Finally, several variation experiments, e.g. on the number of latents and the top-k sparsity, are presented that seem to indicate a robust training and reinforce the requirement of the global top-k regularization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors propose a well formulated method on improving cross-model and cross-modal representation alignment. While the obtained results show moderate improvements in representation alignment, the authors do not overstate contribution and accurately represent the abilities of their method. Claims in are backed up with data and the interpretation of results in throughout the paper seems to align with the obtained results.", "audience_yn": "Yes", "audience_explanation": "While the presented results show moderate reconstruction abilities, class prediction abilities, and partly noisy heatmaps, the method seem to generally be a step in the right direction on post-training representation alignment. More generally, the proposed method seems easy to plug into different models without costly retraining steps. The experiments are well conducted, highlighting strengths and possible weaknesses of the proposed approach.", "requested_changes": "**Linearity of Latents.** In Sec. 3.2 the authors detail an linear affine transformation from model features into/from the shared latent space. Aligning latent spaces between models via affine transformations requires that features are already linear within the models. While there has been some evidence that larger baseline models tend to learn linearized representations, this assumption should be made explicit and might be tested in the paper. (Alternatively, nonlinear transforms could be considered instead, e.g., two-layer NN, although I'm aware that such linear transforms are commonly accepted in literature and would require a whole new training and evaluation run.)\n\n**Stream separate latents.** In Eq. (4) the authors create separate 'intermediate' latent vectors $z^s$ for every stream.  Given that $\\hat{x}^s$ is expected to be produced once by the first term via $D\\_s(z^s)$ and equally by all other streams via $D\\_s(z^t)$ in the second term, wouldn't this imply $z^s \\approx z^t$ anyway? More generally, I'm wondering why the authors choose this seemingly more complicated setup of creating individual latents per stream, which are only then implicitly enforced to obtain the same value via the second loss term in Eq. (6). Wouldn't using $\\mathbf{h}\\_{agg}$ (or its average version) in Eq. (4) suffice?\n\n\n\n**Standard Errors.** Across all tables the authors only report means. The paper might be improved by reporting standard error where applicable. (E.g., per latent dimension or stream variances in table 2, ...)", "broader_impact_concerns": "None", "additional_comments": "I generally find the paper well written and the method to be clearly described. The proposed architecture and loss terms are well visualized and formalized. A general benefit of the method seems to be its straight forward implementation and general applicability, which might make it easy to extend to models beyond the ones tested. Furthermore, the learning of a joint 'drop-in' latent space leaves the weights of the original models unchanged, therefore avoiding costly retrainings or finetunings of model weights.\n\nMy main concern with the method lies with the point of 'stream separate latents' as described above. The authors seemingly propose a more complicated setup than necessary only to then counteract the induced complexity with additional loss terms. While I can imagine how the proposed multi-latent setup can be beneficial, in terms of performance and disentanglement, in contrast to an aggregated latent representation, I feel like the authors need to justify and compare benefits over the more simple approach.\n\n\n**Further Comments:**\n* Following Eq. (3), \"$\\mathcal{I}$global\" is not properly set in subscript.\n* Local TopK is first mentioned in Sec. 4.1 without any prior definition. While I assume it to simply be selecting the top-k activations from each latent stream individually, the authors might want to add a brief clarification.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)