Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Authors: Harrish Thasarathan
Julian Forsyth
Thomas Fel
Matthew Kowal
Konstantinos G. Derpanis
Venue: ICML 2025 poster (2025)
Keywords: Interpretability
Concepts
Sparse Autoencoders
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We introduce Universal Sparse Autoencoders, a new framework for discovering and aligning interpretable concepts shared across multiple deep neural networks

Abstract: We present Universal Sparse Autoencoders (USAEs), a framework for uncovering and aligning interpretable concepts spanning multiple pretrained deep neural networks. Unlike existing concept-based interpretability methods, which focus on a single model, USAEs jointly learn a universal concept space that can reconstruct and interpret the internal activations of multiple models at once. Our core insight is to train a single, overcomplete sparse autoencoder (SAE) that ingests activations from any model and decodes them to approximate the activations of any other model under consideration. By optimizing a shared objective, the learned dictionary captures common factors of variation—concepts—across different tasks, architectures, and datasets. We show that USAEs discover semantically coherent and important universal concepts across vision models; ranging from low-level features (e.g., colors and textures) to higher-level structures (e.g., parts and objects). Overall, USAEs provide a powerful new method for interpretable cross-model analysis and offers novel applications—such as coordinated activation maximization—that open avenues for deeper insights in multi-model AI systems.

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Harrish Thasarathan
Julian Forsyth
Thomas Fel
Matthew Kowal 1245
Konstantinos G. Derpanis 1627
Universal Sparse Autoencoder (USAE)
Unsupervised Concept Discovery
B) Coordinated Activation Maximization
Dataset
DinoV2
Z
SigLIP
ViT
Dino
SigLIP
VIT
Dino SigLIP VIT
Figure Overview of Universal Sparse Autoencoders. (A) introduce Universal Sparse Autoencoders (USAEs), a method for discovering common concepts across multiple diffferent deep neural networks. USAEs are simultaneously trained on the activations of multiple models and are constrained to share an aligned and interpretable dictionary of discovered concepts. (B) We also demonstrate one immediate application of USAEs, Coordinated Activation Maximization, where optimizing the inputs of multiple models activate the same concepts reveals how different models encode the same concept. Visualization reveals interesting concepts at various levels of abstrac- tion, such 'curves' (top), 'animal haunch' (middle) and 'the faces of crowds' (bottom). Code: github.
PorkUCCIL/Uniienese
Abstract
jective, the learned dictionary captures common factors of variation connationccoccpptssso concepts-across across different tasks, architectures, and datasets. We show that USAEs discover semantically coherent and im- portant universal concepts across vision models; ranging from low-level features (e.g., colors and textures) higher-level structures (e.g.. parts and objects). Overall, USAFs provide powerful new method for interpretable cross-model analysis and offers novel applications such as coordinated activation maximization that open avenues for deeper insights in multi-model Al systems.
We present Universal Sparse Autoencoders (US- AEs), framework for uncovering and aligning in- terpretable concepts spanning multiple pretrained deep neural networks. Unlike existing concept- based interpretability methods, which focus on single model, USAEs jointly learn a universal concept space that can reconstruct and interpret the internal activations of multiple models once. Our core insight is to train single, overcom- plete sparse autoencoder (SAE) that ingests ac- tivations from any model and decodes them to approximate the activations of any other model under considcration. By optimizing a shared ob-
1. Introduction
In this work, we focus on discovering interpretable concepts shared among multiple pretrained deep neural networks (DNNs). The goal is to learn universal concept space joint space of concepts that provides unified lens into the hidden representations of diverse models. We de- fine concepts as the abstractions each network captures that transcend individual data points- spiissnnpnnnnggg low-level fea- tures (e.g., colors and textures) to high-level attributes (e.g..
York University, Toronto, Canada VVector Institute, Toronto, Canada KKmmpner Institute, Harvard University, Boston, USA "FAR.AI TTrajectory Labs. Toronto "University of Toronto, Toronto, Canada 'Samsung Al Centre, Toronto. Correspondence to: Harrish Thasarathan <harryt yorkucaa>
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
emotions like horror and ideas like holidays).
Contributions. Our main contributions are as follows. First, we introduce USAEs: framework that learns a shared, in- terpretable concept space spanning multiple models, with focus on visual tasks. Second, we present detailed anal- ysis contrasting universal concepts against model-specific concepts, offering new insights into how large vision mod- els--trined on diverse tasks and datasets- datasets-comparee and diverge in their internal representations. Finally, we demon- strate a novel USAE application, coordinated activation maximization, showcasing simultaneous visualization of universal concepts across models.
Grasping the underlying representations within DNNs is crucial for mitigating risks during deployment (Buolamwini & Gebru, 2018; Hansson et al., 2021), fostering the de- velopment of innovative model architectures (Kowal et al., 2025; Darcet et al., 2024), and abiding by regulatory frame- works (Commision, 2021; House, 2023). Prior interpretabil- ity efforts often center on dissecting a single model for specific task, leaving risk management unmanageable when each network is analyzed in isolation. With a growing num- ber of capable DNNs, finding a canonical basis for under- standing model internals may yield more tractable strategies for managing potential risks.
2. Related work
Recent work supports this possibility. The core idea behind 'foundation models' (Henderson et al., 2023) presupposes that any DNN trained on a large enough dataset should en- code concepts that generalize to an array of downstream tasks for that modality. Moreover, recent work (Moschella et al., 2022) has shown that regardless of architecture, ini- tialization, and task, differently trained models may yield semantically equivalent latent representations, and recent studies (Dravid et al., 2023; Kowal et al., 2024a) even found shared concepts across vision models. This implies that universality may be more widespread than previously as- sumed. However, current techniques for identifying univer- sal features (Dravid et al., 2023; Huh et al., 2024; Kowal et al., 2024a) typically operate post-hoc, extracting concepts from individual models and then matching them through compute-intense innuute-intensiee filtering or optimization. This approach is limited in scalability, lacks the efficiencies of gradient- based training, and precludes translation between models within a unified concept space. Consequently, tasks that re- quire simultaneous interaction across multiple models, e.g. coordinated activation maximization shown later, become more cumbersome.
Our work introduces a novel concept- interpretability method that adapts SAEs to discover universal concepts. We now review the most relevant works in each of these fields.
Concept-based interpretability (Kim et al., 2018 2018) emerged as a response to the limitations of attribution methods (Si- monyan et al., 2014; Zeiler & Fergus, 2014; Bach et al., 2015; Springenberg et al., 2014; Smilkov et al., 2017; Sun- dararajan et al., 2017: Selvaraju et al., 2017; Fong et al., 2019; Fel et al., 2021; Muzellec et al., 2024), which, de- spite being widely used for explaining model predictions, often fail to provide a structured or human- inntannitteppretable understanding of internal model computations (Hase & Bansal, 2020; Hsieh et al., 2021; Nguyen et al., 2021; Colin et al., 2021; Kim et al., 2022: Sixt et al., 2020). Attri- bution methods highlight input regions responsible for a given prediction, the where, but do not explain what the model has learned at a higher level. In contrast, concept- based approaches aim to decompose internal representa- tions into unnnannndddest concepts (Genone & Lom- brozo, 2012). The main components of concept-based inter- pretability approaches can generally be broken down into two parts (Fel et al., 2023b): (i) concept discovery, which ex- tracts and visualizes the interpretable units of computation and (ii) concept importance estimation, which quantifies the importance of these units to the model output. Early work explored 'closed-world" concept settings in which they evaluated the existence of pre-detined concepts in model neurons (Bau et al., 2017) or layer activations (Kim et al., 2018). When access to an aligned text-image representation space is available, output-level image representations can be decomposed into interpretable components using text representations as a dictionary (Gandelsman al., 2024). Similar to our work, 'open-world' concept discovery meth- ods do not assume the set of concepts is known a priori. These methods pass data through the model and cluster the activations to discover concepts and then apply a concept importance method on these discoveries (Ghorbani et al., 2019; Zhang al., 2021; Fel et al., 2023e; Graziani et al., 2023; Vielhaben et al., 2023; Kowal et al., 2024a;b).
To overcome these challenges, we introduce a universal sparse autoencoder (USAE), Fig. designed to jointly encode and reconstruct activations from multiple DNNs. Through qualitative and quantitative evaluations, we show that the resulting concept space captures interpretable fea- tures shared across all models. Crucially, a USAE imposes concept alignment during its end-to-end training. differing from conventional post-hoc methods. We apply USAES to three diverse vision models and make several interesting findings about shared concepts: (i) We discover broad range of universal concepts, at low and high levels of ab- straction. (ii) We observe a strong correlation between con- cept universality and importance. (iii) We provide quanti- tative and qualitative evidence that DinoV2 (Oquab et al., 2023) admits unique features compared to other considered vision models. (iv) Universal training admits shared repre- sentations not uncovered in model-specific SAE training.
2

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Sparse Autoencoders (SAEs) (Cunningham et al., 2023; Bricken al., 2023; Rajamanoharan et al., 2024; Gao et al., 2024; Menon et al., 2024) are specific instance of dictio- nary learning (Rubinstein et al., 2010; Elad, 2010; Tošié & Frossard, 2011; Mairal et al., 2014; Dumitrescu & Irofti, 2018) that has regained attention (Chen et al., 2021; Tasissa et al., 2023; Baccouche et al., 2012; Tariyal et al., 2016; Papyan et al., 2017: Maldizadehaghdam et al., 2019; Yu et al., 2023) for its ability to uncover interpretable concepts in DNN activations. This resurgence stems from evidence that individual neurons are often polysemantc--iee. they activate for multiple, seemingly unrelated concepts (Nguyen et al., 2019; Elhage et al., 20 suggesting that deep net- works encode information in superpositiio (Elhage et al., 2022). SAEs tackle this by learning a sparse (Hurley & Rickard, 2009; Eamaz et al., 2022) and overcomplete rep- resentation, where the number of concepts exceeds the la- tent dimensions of the activation space, encouraging dis- entanglement and interpretability. While SAES and clus- tering bear mathematical resemblance, SAEs benefit from gradient- bassdd optimization, enabling greater scalability and efficiency in learning structured concepts. Though widely applied in natural language processing (NLP) (Wat- tenberg & Viégas, 2024; Clarke et al., 2024; Chanin al., 2024; Tamkin et al., 2023), SAEs have also been used in vision (Fel et al., 2023b; Surkov et al., 2024; Bhalla et al., 2024a). Early work compared SAFS to clustering and ana- lyzed early layers of Inception v1 (Mordvintsev et al., 2015; Gorton, 2024), revealing hypothesized but hidden features. More recently, SAEs have been leveraged to construct text- based concept bottleneck models (Koh et al., 2020) from CLIP representations (Radford et al., 2021; Rao et al., 2024; Parekh et al., 2024; Bhalla et al., 2024b), showcasing their versatility across modalities. Unlike prior work that apply SAES independently to models, here we consider a joint ap- plication of SAES simultaneously across diverse models.
Universal SAE
Vicar mosder
Vision crassder
Shaaed Datasel
f(i
(
A
D
Z
Figure 2. USAE training process. In each forward pass during training, an encoder of model i is randomly selected to encode a batch of that model's activations, Z = v ( (A()). The concept space, Z, then decoded to reconstruct every model's activations, A using their respective decoders, D
versal concepts rather than learning a shared conceptual space. This reliance on retrospective discovery is compu- tationally prohibitive for many models and prevents direct concept translation between architectures. A concurrent study (Lindsey et al., 2024) explores training SAEs (termed crosscoders) between different states of the same model before and after fine-tuning. In contrast, our work discovers universal concepts shared across distinct model architec- tures for vision tasks.
3. Method
Notations. Let ||- ll2 and ||: denote the l and Frobenius
norms, respectively, and set [n]] = [I..... n). We focus on broad representation learning paradigm, where a DNN, f X + A. maps data from X' into a feature space, A S Rd. Given dataset, X C X' of size n, these activations are collated into a matrix A E RRdd Each row A, (for E [n)) corresponds to the feature vector of the i-th sample.
Feature Universality studies the shared information across different DNNs. One approach, Representational alignment, quantifies the mutual information between different sets of representationss-ieeeee
Background. The main goal of a Sparse Autoencoder (SAE) is to find sparse re-interpretation of the feature representations. Concretely, given a set of n inputs, X (e.g.. images or text) and their encoding, A = f(x) E Raad an SAE learns an encoder ( that maps A codes o (A) E R=xm, forming a sparse representation. This sparse representation must still allow faithful reconstruction of A through a learned dictionary (decoder) D E Rmmxd i.e., ZD must be close to A. If m > d, we say D is overcomplete. In this work, we specifically consider an (overcomplete) TopK SAE (Gao et al., 2024), defined as
across models or between bi- ological and artificial systems (Kriegeskorte et al., 2008; Sucholutsky et al., 2023). Typically. these methods rely on paired data (e.g., text-image pairs) to compare encod- ings across modalities. Recent work suggests that founda- tion models, regardless of their training modality, may be converging toward a shared, Platonic representation of the world (Huh et al., 2024). Another line of research focuses on identifying universal features across models trained on different tasks. Rosetta Neurons (Dravid et al., 2023) iden- tify image regions with correlated activations across models, while Rosetta Concepts (Kowal et al., 2024a) extract con- cept vectors from video transformers by analyzing shared exemplars. These methods perform post-hoc mining of uni-
= = (A) = Topk( Wene (A - bpc))) , A = ZD, (1)
where Wane E Rmmd and 6pre E Rdd are learnable weights. The TopK(.) operator enforces 1Z S K for all i E [m]. The final training loss is given by the Frobenius norm of the
3

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
reconstruction error:
def Frain_usae(VV, D, A, T, Optimizers); M = (V))
CSAE = -I vo ||A 2D-|DD| F. (2)
for t in range (T):
with the -ssarsiy constraint applied to the rows of Z.
random(M)
Z v 0 (A
3.1. Universal Sparse Autoencoders (USAFs)
L = 0.0
for j in range(M):
Contrasting standard SAEs, which reinterpret the internal representations of a single model, universal sparse autoen- coders (USAEs) extend this notion across M different mod- els, each with its own feature dimension, d; (see Fig. 2). Concretely, for model i E [MM] let ( Rnxd, denote the matrix of activations for samples. The key insight of USAEs is to learn a shared sparse code, Z E R=xm, which allows every model to be reconstructed from the same sparse embedding. Specifically, each activation from model i in A is encoded via mpdel-specific encoder
A = Z @ D
C += (A A norm(p===))))
L. backwward())
Optimizers(i).
return vo. D
Figure 3. Training Universal Sparse Autoencoder. During each training iteration, Linisersat is the aggregated error computed from decoding each activation A We then take an optimizer step for
T (t)
randomly selected encoder v (8) and associated dictionary (
Z = ())
(A (1) Topk( W (A
pre
(3)
M || ()) - A
Universsl =
(5)
Crucially, once encoded into Z, each row of any model j E [M] can be reconstructed by model-specinf dictionary,
j=1
DG) E R4 xm as
M
|| (o(A D (
(6)
A = ZD
(4)
By jointly learning all encoder-decoder pairs, {(U)) (1) D the USAE enforces a unified con- cept space, Z, that aligns the internal representations of all M models. This shared code not only promotes consistency and interpretability across model architectures, but also ensures each model's features can be faithfully recovered from a common set of sparse 'concepts'.
Using this universal loss, backpropagation updates the cho- sen encoder v/(") and decoder (6), This method promotes concept alignment, ensures an equal number of updates be- tween encoders and decoders, and strikes a practical balance between training speed and memory usage.
3.3. Application: Coordinated Activation Maximization
3.2. Training USAFS
A common technique for interpreting individual neurons or latent dimensions in deep networks is Activation Maxi- mization (AM) (Olah et al., 2017; Tsipras et al., 2019; San- turkar et al., 2019; Engstrom et al., 2019; Ghiasi et al., 2021; 2022; Fel et al., 2023a; Hamblin et al., 2024). AM involves synthesizing an input that maximally activates specific component of a cept vector (Williams, 1986; Mahendran & Vedaldi, 2015; Kim et al., 2018; Fel et al., 2023c). However, in the case of a USAE, the learned latent space is explicitly structured to capture shared concepts across multiple models. This shared space enables novel extension of AM: Coordinated Activation Maximization, where a common concept index, k, is simultaneously maximized across all aligned models.
Recall that X C X' is our dataset of size n, mapped into (M).
their respective feature space using DNNs A naive approach to train our respective encoder and decoder would simultaneously encode and decode the features of all M models, which quickly grows expensive in memory and computation.
as a neuron, channel, or con-
Conversely, randomly sampling a pair of models to encode and decode results in slow convergence. To balance these concerns, we adopt an intermediate strategy (pseudocode detailed in Figure 3) that updates single encoder-decode pair at each iteration with a reconstruction loss computed through all decoders. Concretely, at each mini-batch it- eration, single model i e [M] is selected at random, and a batch of features, A(i) Raxd, is sampled and
Given M models, our objective is to optimize one input per model, xx ..... xx (M) ensuring that all inputs maximally activate the same concept dimension A: This approach en- ables the visualization of how a single concept manifests across different models. By comparing these optimized inputs, we can identify both consistent and divergent repre- sentations of the same underlying concept. Let ar (3) denote
encoded into the shared code space, Z = { (A()( (A). This code space, Z, is then used to reconstruct the feature rep- resentation A AD) of every model j E [M] via its decoder: A = DD),, where DGI is the model-j decoder. All reconstructions are aggregated to form the total loss:

---

Universal Sparse Autoencoders: Interpretable Crdes-Model Concept Alignment
Concept 3152 Yellow
Concept 4226 Bird Background
Concept 5611 Dog Face
Concect 972 Animal Jaw
Input
Dino
SigLiP
VIT
Concept 4235 Blue
Concept 3859 Thin Objects
Concent 2824 Bolts
Concept 1935 Animal Group Faces
Input
Dino
SiGLIP
VIT
Figure Qualitative results of universal concepts. discover and visualize heatmaps universal concepts across a broad range of visual abstractions. where bright green denotes stronger activation of given concept. We observe colors, basic shapes, foreground- background, parts, objects and their groupings across all considered models.
the input to model i, and let f( (z) Rd. represent its internal activations. Each model is associated with USAE encoder (x) which maps activations to the shared concept
4. Experimental Results
This section split into six parts. We first provide ex- perimental implementation details. Then, we qualitatively analyze universal concepts discovered by USAEs (Sec. 4.1). Next, provide quantitative analysis of USAFS through the validation of activation reconstruction (Sec. 4.2), measur- ing the universality and importance of concepts (Secs. 4.3), and investigating the consistency between concepts in US- AEs and individually trained SAE counterparts (Sec. 4.4). Finally, we provide finer-grained analysis via the appli- cation of USAEs to coordinated activation maximization (Sec. 4.5).
0 space. The activation of concept for model given input (+) defined as
(+) (z))
(7)
where indexes the universal concept dimension in the USAE. The goal is to independently optimize each (i) such that maximizes the activation of the same concept across all M models:
(6)
arg max 2 (a:(0) (1),
Implementation Details. We train a USAE on the fi- nal layer activations of three popular vision models: Di- no (Oquab et al., 2023; Darcet et al., 2024), SigLIP (Zhai et al., 2023), and ViT (Dosovitskiy et al., 2020) (trained on ImageNet (Deng et al., 2009)). These models, sourced from the timm library (Wightman, 2019), were selected due to their diverse training paradigms imaae and patch-level discriminative learning (DinoV2), image-text contrastive learning (SigLIP), and supervised classification (ViT). For all experiments, we train the USAE on the ImageNet train- ing set, while the validation set is reserved for qualitative visualizations and quantitative evaluations. Our USAE
where R(x) is regularizer that promotes natural and in- terpretable inputs (e.g., total variation, l2 penalty, or data priors), and controls its strength. In all experiments, follow the optimization and regularization strategy of Maco (Fel et al., 2023a), which optimizes the input phase while preserving its magnitude. Once the optimized inputs (3) are obtained for each model, they reveal the specific structures features (e.g., model- task task-specific biases) that model associates with this universal concept.
5

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Decoder and Activations used for reconstruction
Distribution Rring Entropy
firing Proportions
SiqLIP
DinoVz
Number of Concepts
1500
[a]
Naximum Entropy
1.0
0.8-
Model Activalions Encodee to
SigUP- 0.61
0.36
0.25
1000
0.
0.4
500
Dinov2
0.31
0.77
0.31
0.0
2A
Nomalzed Finng Entrogy [HA]
0.2 0.4
0.6
1.0
T0p Cencapta by Eraagy iPropor Soeted
200
403
650
1200
Relationship Between Co-firing Frequency and Concept Energy
VIT-
0.30
0.45
0.59 02
Concent Energy Oan scale)
[c)
0.063
0.89
Figure 5. Cross model activation reconstruction. Each entry (i,.)) represents the average It² score when activations ( from model i are encoded into the shared code space, Z. then decoded via ( to reconstruct A Positive off-diagonal R² scores indicate the presence of shared features across models captured by USAFs.
10-1
30
10-2
Stope 0.23
Slope 0.70
Co-fring Frequency
trained on the final layer representations of each vision model, as previous work showed final-layer features facil- itate improved concept extraction and yield accurate esti- mates of feature importance (Fel et al., 2023b). We base our SAE off of the TopK Sparse Autoencoder (SAE) (Gao al., 2024) and for all experiments, use dictionary of size 6144. We train all USAEs on single Nvidia RTX 6000 GPU, with training completing in approximately three days (see Appendix A.I for more implementation details).
Figure 6. Quantitative analysis of universality and importance of USAE concepts via co-firing rates. (a) Histogram of firing en- tropy across all k concepts. We observe bimodal distribution over firing entropy with peaks at H2 = 1 and H2 = 0.6, demonstrating group of concepts that fire uniformly across models and group that preferentially activates for some models. (b) Proportion of con- copt co-fires for the top 1000 energetic concepts per model. The first 200 concepts co-fire between 60 - 80% of the time suggest- ing high universality. (c) Relationship between concept co-firing frequency and concept energy. We show all concepts (left) and only frequently co-firing concepts (2 1000 co-fires) (right). The correlation strengthens (r 0.63 vs 0.89) when focusing on high frequency concepts, suggesting a strong correlation between how energetic concept is and its universality.
4.1. Universal Concept Visualizations
We qualitatively validate the most important universal con- cepts found by USAEs. We determine concept impor- tance by measuring its relative energy towards reconstruc- tion (Gillis, 2020), where the energy of a concept k is de- fined as
USAE reveals fine- compositional concepts such as 'oouth-open animal jaws' and 'accss of animals in a group', which generalize across ImageNet classes and persist even in ViT, despite its lack of explicit structured supervision.
Energy(k)= = E 12
(9)
This measures how much each concept contributes to re- constructing the original features formally, the squared & norm of the average activation of concept multiplied by its dictionary element. Higher energy concepts have greater impact on the reconstruction.
Overall, these findings show that USAES discover robust, generalizable concepts that persist across different archi- tectures, training tasks, and datasets. This highlights their ability to reveal invariant, semantically meaningful repre- sentations that transcend the specifics of any single model.
Figure 4 presents eight representative concepts selected from the 100 most important USAE concepts. These con- cepts span a diverse range of ImageNet categories, demon- strating the ability of USAES to capture meaningful features across multiple levels of abstraction and complexity (Olah et al., 2017; Fel et al., 2024). At lower levels, the USAE extracts fundamental color concepts, such as 'yellow' and "blue', activating over broad spatial regions across multiple classes. Notably, the blue bottle caps example highlights a precisely captured checkerboard pattern, demonstrating spatial precision. At intermediate levels, the USAE uncov- structural relationships consistent across models, such as
4.2. Validation of Cross-Model Reconstruction
A viable universal space of concepts should enable the re- construction of activations from any model. To quantify the reconstruction performance, we use the coefficient of determination, or R22 score (Wright, 1921), which measures the proportion of variance in the original activations that is captured by the reconstructed activations, relative to the mean activation baseline, A. The R score is defined as
R² = 1 A|
/ A A
(10)
contrasts (e.g., birds against the sky) and thin, wiry objects, independent of model archi- tecture. At higher levels, it identifies object-part concepts, like 'dog face', excluding eye regions, and 'bolts', which activate across materials like metal and rubber. Finally, the
where ||A AlP represents the residual sum of squares (the reconstruction error), and | - A is the total sum of squares (the variance of the original activations relative to their mean). A higher R² indicates better reconstruction quality, with score of one for a perfect reconstruction.
6

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Figure shows the R2 scores as confusion matrix across all three models. As expected, self-reconstruction along the
Concept Matches Acress
Fraction or Concepts CS Threstold
Cosine Simliarity Thresholds
1.0-
SigLIP
diagonal achieves the highest explained variance, confirm- ing the USAE's effectiveness when encoding and decoding within the same model. More notably, positive off-diagonal R2 scores indicate successful cross-model reconstruction, suggesting the USAE captures shared, likely universal, fea- tures. V2 exhibits the highest self- reconssoosttttionn per- formance, aligning with individual SAE results where its R2 score averages 0.8, compared to 0.7 for SigLIP and ViT. This suggests V2 features are sparser and more decom- posable, a trend further supported in Secs. 4.3 and 4.5,
0.8
VIT
DineV2
Model
AUC
% Z > 0.5
0.6
Rand, Baseline
SigLIP
0.30
0.23
DinoV2
0.36
0.26
0.4
VIT
0.41
0.38
0.2
Baseline
0.13
0.00
0.0
0.4
1.0
Casina Similarity Threshald
Figure 7. Concept consistency between independent SAEs and Universal SAEs, (left) Our universal training objective discov- ers concepts that have overlap (i.c., cosine similarity) with those discovered with independent training. Specifically, ViT has notice- ably more overlap. suggesting its simpler architecture and training objective may yield activations that naturally encode universal visual concepts. (right) We consider a cosine similarity (CS) > 0.5 as match hetween concepts the SAE and USAE learned dictio- naries. Across each vision model used in training, the Area Under the Curve (AUC) suggests 23 37% of the universal concepts Z discovered by our approach exist in independently trained SAEs.
4.3. Measuring Concept Universality and Importance
Having established the efficacy of cross-model reconstruc- tion, we now assess concept universality using firing en- tropy and co-firing metrics. We further examine the relation- ship between universality and importance in reconstructing ground truth activations.
Let r be threshold value and V be the ImageNet validation set of patches. Given data points æ E V. let 200 (x)
vor two models but not all three. Few concepts fall in the low-entropy range (0.0-0.2), suggesting most are shared rather than model-specific. Appendix A.2 further examines these low-entropy concepts, revealing Dino unique en- coding of geometric features as well as SigLIP's encoding of textual features.
(1)
(f(() (z)) denote the sparse code from model i E [M]-
6
We define a concept firing for dimension & when Z (a) >
T. A co-fire occurs when concept fires simultaneously
across all models for the same input. Formally, for concept dimension k. the set of co-fires is defined as
Ck (ac E V: min 2 (x) > r). (E(ad]
(11)
Co-Fire Proportion (CFP) quantifies how often concepts fire together for the same input. While previous results show many concepts fire uniformly across models, they do not reveal how frequently they co-fire on the same tokens. For each model i and concept k. we compute the proportion of total fires that are also co-fires:
Similarly, let F/( fx E V: set of "fires" for model and concept k. We are now ready to introduce two metrics (i) Firing Entropy (FE) and (ii)
x (6) (x) 7 denote the
Co-Fire Proportion (CFP).
Firing Entropy (FE) measures, for each concept k, the normalized entropy across models, as
CFP k = |c
(14)
High co-fire proportions indicate concepts that are more universal, i.e., when one model detects the concept, others tend to as well.
FE
1
Pk
logrt (4)
(12)
log M
1=1
where
M
Figure (b) shows the CFP for the top 1000 concepts per model. The first 100 concepts exhibit high co-firing (> 0.5), activating together 50-80% of the time, indicating core set of consistently recognized concepts across net- works. The gradual decline in CFP suggests a spectrum of universality, from widely shared to model-specific. For our chosen models, we again notice a pattern distinguish- ing D2n which has a lower co- co- fo-firing proportion (0.266) compared to SigLIP (0.344) and ViT (0.326), suggesting the latter two share more concepts. This may stem from D2's architecture and distillation-base training, which enhance its adaptability to diverse vision tasks (Amir et al., 2022). These findings also hint at a correlation between co-firing and concept importance, raising the question: How important are these highly co-firing features?
(3) = 1 Pk
Σ 1
(13)
The normalization ensures FE, E [0, 1]. with FE = indi-
cating a shared concept with uniform firing across models
and low entropy indicating that a concept has model bias and fires for single architecture or subset.
Figure 6 (a) shows a histogram of firing entropies across all concept dimensions K. Fully universal concepts should have a maximum entropy of one, indicating uniform firing across models. Our results exhibit a bimodal distribution, with over 1000 concepts at peak entropy, confirming the USAE learns a strongly universal concept space. A second group shows moderate entropy, indicating concepts that fa-
7

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Concept 4276 Curves
Concept 1423 Crosses
Concept 6066 Object from above
Concept grass
Concept Round objects
Concept 5160 Keypads
DinoV2
SigLIP
ViT
DinoV2
SigLIP
VIT
DinoV2
SigLIP
ViT
Figure 8. Coordinated Activation Maximization. We show results for the three model USAE along with dataset exemplars, where bright green denotes stronger activation of the concept. We visualize the maximally activating input for broad range of concepts, including basic shape compositions, textures, and various objects.
To answer this, we plot the co-fire frequency of all concepts as well as their energy- based importance in Fig. 6 (c). We see a moderate positive correlation r = 0.63, slope = 0.23: however, zooming into concepts with > 1000 co-fires, shows a much stronger correlation. Indeed, past a certain threshold, co- co-firing frequency becomes highly predictable of concept importance. This suggests that the most impor- tant concept are also highly universal, firing consistently across models.
that universal training discovers concepts that may not emerge in independent training.
When looking at the top 1,000 co-firing concepts (see Sec. 1.5.I) we find an overall increase in consistency be- tween individual and universal concepts: the most universal (highest co-firing) concepts are more likely be found in each model's respective independently trained SAE. Uni- versal training naturally selects for concepts that are well- represented across all models, since these will better mini- mize the total reconstruction loss, biasing towards discover- ing fundamental visual concepts that all models have learned to represent. Independently trained SAFs have no such se- lection pressure, learning represent any concept that helps reconstruction, including architecture objective specific concepts that not universal.
4.4. Concept Consistency Between USAES and SAES
How many concepts discovered under our universal training regime present in independently trained SAE for single model? Further, what percentage of highly universal concepts appear in these same independently trained SAEs? To assess the alignment between mndependently-trinee trained and universal SAEs, we analyze the similarity of their learned conceptual spaces. We quantify concept overlap by com- puting pairwise cosine similarities between decoder vectors and use the Hungarian algorithm (Kuhn, 1955) to optimally align concepts, measuring consistency across models.
4.5. Coordinated Activation Maximization
Figure shows a visual comparison of several universal concepts and their corresponding coordinated activation maximization inputs. Our method produces interpretable vi- sualizations for a given USAE dimension across all models for broad range of visual concepts. We show examples of all models encoding low-level visual primitives, e.g., 'curves' and 'crosses'. Other basic entities are also shown, like 'brown grass' texture and 'round objects', Finally, we visualize higher-level concepts corresponding to 'objects from above" and 'keypads'. In all cases, our coordinated activation maximization method produces plausible visual phenomenon that can be used to identify differences between how each model encodes the same concept.
Figure 7 presents concept consistency distributions across models. For baseline to compare against, we sample con- cept vectors from normal distributions, where the mean and variance are those of each independent model's dictionary. We observe that ViT has the strongest concept overlap with 38% of its concepts having cosine similarity > 0.5 with its independent counterpart. This suggests ViT's conceptual representation under the independent SAF objective is most well preserved under universal training. USAES achieve far better performance than the baseline (Area Under the Curve (AUC)=0.13) across models, suggesting that univer- sal training preserves meaningful concept alignments rather than learn entirely new representations. On the other hand the relatively low proportion of overlap (23% and 26% for LPP and DinoV2 respectively) for concepts indicates
For example, we note an interesting difference between DinoV2 and the other models: low-mid level concepts (i.e., left two columns) appear at a much larger scale than the other models. Further, shown in Fig. 1. Dino exhibits stronger activation for the 'curves' concept, particularly for larger curves, compared to the other models. Additionally,
8

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Concept 3715 Perspective Right
Concept 2562- -ilt Right
Concept 4189 Perspective Left
Concept 3003 Tilt Left
Figure 9. D2 2 low-entropy concepts. We show examples of low-entropy concepts that fire solely for DinoV2. These concepts fire for perspective cues related view invariance such as converging perspective lines (concept 3715 and 4189) and angled scenes (concept 2562 and 3003).
while 'brown grass' activates on grass in our heatmaps, some models' activation maximizations include birds, suggesting animals also influence the concept's 's activation.
no'2's training regime and architecture (Oquab et al., 2023). Specifically, self-distillation across different views and crops at the image level emphasizes geometric con- sistency across viewpoints. This, in combination with the masked image modelling iBOT objective (Zhou et al., 2021) that learns to predict masked tokens in a student-teacher distillation framework, would explain the sensitivity Di- no perspective and geometric properties, as well as niew-invarian features. We further explore unique concepts in SigLIP using this same approach in A.3 finding concepts that fire for both visual and textual elements of the same concept.
4.6. Discovering Unique Concepts with USAFS
Our universal training objective provides the opportu- nity to explore concepts that may arise independently in one model, but not in others. Using metrics for universal- ity, Eqs. 13 and 12, we can search for concepts that fire with a low entropy, thereby isolating firing distributions whose probability mass is allocated to a single model. We explore this direction by isolating unique concepts for Di- no and LigLIP, both of which have been studied for their unique generalization capabilities to different downstream tasks (Amir al., 2022; Zhai et al., 2023).
5. Conclusion
In this work, we introduced Universal Sparse Autoencoders (USAEs), a framework for learning a unified concept space that faithfully reconstructs and interprets activations from multiple deep vision models at once. Our experiments re- vealed several important findings: (i) qualitatively, dis- cover diverse concepts, from low-level primitives like col- ors, shapes and textures, to compositional, semantic, and abstract concepts like groupings, object parts, and faces, (ii) many concepts turn out to be both universal (firing consis- tently across different architectures and training objectives) and highly important (responsible for large proportion of each model's reconstruction), (iii) certain models, such as Dino encode unique features even they share much of their conceptual basis with others, and (iv) while univer- sal training recovers a significant fraction of the concepts learned by independent single-model SAEs, it also uncovers new shared representations that do not appear to emerge in -peel-speeific training. Finally, we demonstrated a novel application of tion that enables simultaneous visualization of universal concept across multiple networks. Altogether, our USAE framework offers practical and powerful tool for multi- model interpretability, shedding light on the commonalities and distinctions that arise when different architectures, tasks, and datasets converge on shared high-level abstractions.
4.6.1. 1. UNIQUE DINOV2 CONCEPTS
V2's unique concepts are presented in Figures 9 and 10. Interestingly, we find concepts that solely fire for DinoV2 related to perspective and depth cues. These fea- tures follow surfaces and edges vanishing points as in concept 3715 and 4189, demonstrating features for converg- ing perspective lines. Further, in Figure 10 we find features for object groupings placed in the scene at varying depths in concept 4756, and background depth cues related to uphill slanted surfaces in concept 1710. We also find features that suggest a representation of view invariance, such as con- cepts related the angle or tilt of an image (Fig. 9) for both left (concept 3003) and right views (concept 2562). Lastly, observe unique geometric features in Fig. 11 that suggest some low-level 3D understanding, such as concept 4191 that fires for the top face of rectangular prisms, concept 3448 for brim lines that belong to dome shaped objects, as well as concept 1530 for corners of objects resembling rectangular prisms.
activation maximiza-
View invariance, depth cues, and low-level geometric con- cepts are all features we expect to observe unique to Di-

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Impact Statement
Bricken, T., Templeton, A., Batson, J.,, Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C.,, Denison, C.,, Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dode K., Mclean, B., Burke, J. E.,, Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity; Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023, https://ranssfrmmr- html.
This work advances interpretability for machine learning systems. More specifically, understanding shared represen- tations across deep neural networks (DNNs) is essential for scalable interpretability, enabling more effective risk miti- gation, robust model design, and compliance with evolving regulations. By moving beyond single-model analysis, we aim to help establish a unified framework for interpreting diverse architectures, fostering transparency and account- ability in Al deployment.
Z., Tamkin, A., Nguyen,
Buolamwini. J. and Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Trans- parency, 2018.
Acknowledgements
This work was completed with support from the Vector In- stitute, and was funded in part by the Canada First Research Excellence Fund (CFREF) for the Vision: Science to Appli- cations (VISTA) program (K. G.D. H.T), the NSERC Discov- ery Grant program G.D. the NSERC Canada Graduate Scholarship Doctoral program (M.K), the Ontario Graduate Scholarship (H.T), and a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University (T.F).
Chanin, D., Wilken-Smith, J., Dulka, T., Bhatnagar, H., and Bloom, A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409. /4507, 2024.
Chen, J., Mao, H., Wang. Z., and Zhang, X. Low-rank representation with adaptive dictionary learning for sub- space clustering, Knowledge-Ban Based Systems, 223:107053, 2021.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S.,. and Vedaldi, A. Describing textures in the wild. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014.
References
Amir, S., Gandelsman, Y., Bagon, S., and Dekel, T. Deep ViT Features as Dense Visual Descriptors. Proceedings of the European Conference on Computer Vision Workshops 2022.
Clarke, M. Bhatnagar, H., and Bloom, J. Compositionality and ambiguity: Latent co- occurrence and interpretable subspaces, 2024.
Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., and Baskurt, A. Spatio-temporal convolutional sparse auto- encoder for sequence classification. In Proceedings of the British Machine Vision Conference, 2012.
conpsstiinallyyaaamamaiiiiiiiiee and.
Colin, J., Fel, T., Cadène, R., and Serre, T. What cannot predict, do not understand: human-centered evalua- tion framework for explainability methods. In Advances in Neural Information Processing Systems. 2021.
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), 2015.
Commision, E. Laying down harmonised rules artificial intelligence (artificial intelligence act) and amending cer- tain union legislative acts European Commision, 2021.
Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. Network dissection: Quantifying interpretability of deep visual representations. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly inter- pretable features in language models. arXiv preprint arXiv:2309.08600 2023.
Bhalla, U., Oesterling, A., Srinivas, S., Calmon, F. P., and Lakkaraju, H. Interpreting CLIP with sparse linear con- cept embeddings (splice). Advances in Neural Informa- tion Processing Systems, 2024a.
Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. Vision transformers need registers. Proceedings of the Interna- tional Conference on Learning Representations, 2024.
Bhalla, U.. Srinivas, S., Ghandeharioun, A., and Lakkaraju, H. Towards unifying interpretability and control: Evalua- tion via intervention. arXiv preprint arX1v:2411.04430, 2024b.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009. 2009.
10

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn,
Fel, T., Bethune, L.,, Lampinen, A. K., Serre, T., and Her- mann, K. Understanding visual feature reliance through the lens of complexity. Advances in Neural Information Processing Systems, 2024.
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Fong, R., Patrick, M., and Vedaldi, A. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEE International Conference on Computer Vision 2019.
Dravid, A., Gandelsman, Y., Efros, A. A., and Shocher, A. Rosetta neurons: Mining the common units in a model Z0o. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, 2023.
Gandelsman, Y., Efros, A. A., and Steinhardt, J. Interpreting CLIP's image representation via text-based decomposi- tion. Proceedings of the International Conference on Learning Representations, 2024.
Dumitrescu, B. and Irofti. P. Dictionary learning algorithms and applications. Springer, 2018.
Famaz, A., Yeganegi, F., and Soltanalian, M. On the build- ing blocks of sparsity measures. IEEE Signal Processing Letters, 29:2667-2671, 2022.
Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scal- ing and evaluating sparse autoencoders. arXiv preprint arXiv:2406.0400 2024.
Elad, M. Sparse and redundant representations: From theory to applications in signal and image processing. Springer Science & Business Media, 2010.
Genone, and Lombrozo, T. Concept possession, experi- mental semantics, and hybrid theories of reference. Philo- sophical Psychology, 25(5)7777774,, 2012.
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodss, Z.,, Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C. Toy models of superposition. arXiv preprint arXiv:2209.1063 2022.
Ghiasi, A., Kazemi, H., Reich, S., Zhu, C., Goldblum, M., and Goldstein, T. Plug-in inversion: Model-agnostic in- version for vision with data augmentations. Proceedings of the International Conference on Machine Learning, 2021.
Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Tran, B., and Madry, A. Adversarial robustness as a prior for learned representations. arXiv preprint arXiv: 19045 2019.
Ghiasi, A., Kazemi, H., Borgnia, E., Reich, S., Shu, M., Goldblum, M., Wilson, A. G., and Goldstein, T. What do vision transformers learn? A visual exploration. arXiv preprint 2rXiv:2212.06727,, 2022.
Fel, T., Cadene, R., Chalvidal, M., Cord, M., Vigouroux, D., and Serre, T. Look at the variance! Efficient black- box explanations with sobol-based sensitivity analysis. In Advances in Neural Information Processing Systems, 2021.
Ghorbani, A., Wexler, J., Zou, J. Y., and Kim, B. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems, 2019.
Gillis, N. Nonnegative matrix factorization. SIAM, 2020.
Fel, T., Boissin, T., Boutin, V., Picard, A., Novello, P., Colin, J., Linsley, D., Rousseau, T., Cadène, R., Goetschalekx, L.. Gardes, L., and Serre, T. Unlocking feature visual- ization for deeper networks with magnitude constrained optimization. In Advances in Neural Information Pro- cessing Systems, 2023a.
Gorton, L.
The missing curve detectors of inceptionv I: Applying sparse autoencoders to inceptionvl early vision. arXiv preprint arXiv:2406.0 036440660036.,... 2024.
Graziani, M., Nguyen, A., O'Mahony, L., Müller, H., and Andrearczyk, V. Concept discovery and dataset explo- ration with singular value decomposition. In Workshop- Proceedings of the International Conference on Learning Representations, 2023.
Fel, T., Boutin, V., Moayeri, M., Cadène, R., Bethune, L.., Chalvidal, M., and Serre, T. A holistic approach to unify- ing automatic concept extraction and concept importance estimation. Advances in Neural Information Processing Systems, 2023b.
Hamblin, C., Fel, T., Saha, S., Konkle, T., and Alvarez, G. Feature Accentuation: Revealing 'What' features respond to in natural images. arXiv preprint 2rXiv:2402.10039, 2024.
Fel, T., Picard, A., Bethune, L., Boissin, T., Vigouroux, D., Colin, J., Cadène, R., and Serre, T. CRAFT: Concept recursive activation factorization for explainability. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023c.
Hansson, S. O., Belin, M.-.., and Lundgren, B. Self-driving vehicles-Al An ethical overview. Philosophy Technology, pp. 1-26, 2021.

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Hase, P. and Bansal, M. Evaluating explainable AI: Which algorithmic explanations help users predict model behav- ior? Proceedings of the Annual Meeting of the Associa- tion.for Computational Linguistics, 2020.
Kowal, M., Wildes, R. P., and Derpanis, K. G. Visual con- cept connectome (VCC): Open world concept discovery and their interlayer connections in deep models. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024b.
Henderson, P., Li, X., Jurafsky, D., Hashimoto, T., Lemley, M. A., and Liang, P. Foundation models and fair use. Journal of Machine Learning Research, 24(400):1-79, 2023.
Kowal, M., Siam, M., Islam, M. A., Bruce, N. D., Wildes, R. P., and Derpanis, K. G. Quantifying and learning static vs. dynamic information in deep spatiotemporal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
House, T. W. President biden issues executive order on safe, secure, and trustworthy artificial intelligence. The White House, 2023.
Kriegeskorte, N., Mur. M., and Bandettini, P. A. Repre- sentational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249, 2008
Hsieh, C.-Y., Yeh, C.-.., Liu, X., Ravikumar, P. Kim, S., Kumar, S., and Hsieh, C.-J. Evaluations and methods for explanation through robustness analysis. In Proceedings of the International Conference Learning Representa- tions, 2021.
Kuhn, H. W. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2:83-97, 1955. doi: 10. 10.1000/..3300002000000
Huh, M., Cheung, B., Wang, T., and Isola, Position:
The Platonic Representation Hypothesis. In Proceedings of the International Conference Machine Learning, 2024.
Lindsey, J., Templeton, A., Marcus, J., Conerly, T., Bat- son, J., and Olah, C. Sparse crosscoders for cross-layer features and model diffing. 2024. https://rrassormmrr html.
Hurley, N. and Rickard, S. Comparing measures of sparsity. IEEE Transactions on Information Theory, 55(10):4723- 4741, 2009.
Liu, Z., Luo, P., Wang. X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, December 2015.
loffe, S. and Szegedy, C. Batch normalization: accelerat- ing deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning, 2015.
Mahdizadehaghdam, S., Panahi, A., Krim, H., and Dai, L. Deep dictionary learning: A parametric network ap- proach. IEEE Transactions on Image Processing, 28(10): 4790-4802, 2019.
Kim, B., Wattenherg, M., Gilmer, J., Cai, C.,, Wexler, J., and Viegas, F. Interpretability beyond feature attribu- tion: Quantitative testing with concept activation vectors (TCAV). In International Conference on Machine Learn- ing, 2018.
Mahendran, A. and Vedaldi, A. Understanding deep image representations by inverting them. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
Kim, S. S. Y., Meister, N., Ramaswamy, V. V., Fong, R., and Russakovsky, O. HIVE: Evaluating the human inter- pretability of visual explanations. In Proceedings of the IEEE European Conference on Computer Vision, 2022.
Mairal, J., Bach, F., and Ponce, J. Sparse modeling for image and vision processing. Foundations and Trends@ in Computer Graphics and Vision, 8(2-3):85-283, 2014.
Menon, A., Shrivastava, M., Krueger, D., and Lubana, E. S. Analyzing (in) abilities of SAEs via formal languages. arXiv preprint arXiv:24 1166,, 2024.
Kingma, D. P. and Ba, J. Adam: A method for stochastic op-
timization. Proceedings of the International Conference on Learning Representations, 2015.
Mordvintsev, A., Olah, C., and Tyka, M. Incep- tionism: Going deeper into neural networks. https:://bog.reeeearrr wwwggeoooo0o00/i/oeooiiiiiiiiiii leenn--deppr-into-neeee himl? m, 2015.
Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. Concept bottleneck models. In International Conference on Machine Learning, 2020.
Kowal, M., Dave A., Ambrus, R., Gaidon, A., Derpanis, K. G., and Tokmakov, P. Understanding video transform- ers via universal concept discovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024a.
Moschella, I.., Maiorca, V., Fumero, M., Norelli, A., Lo- catello, F., and Rodolå, E. Relative representations enable zero-shot latent space communication. Proceedings of the International Conference on Learning Representations, 2022.
12

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Muzellec, S., Andeol, L.., Fel, T., VanRullen, R., and Serre, T. Gradient strikes back: How filtering out high frequen- cies improves explanations. Proceedings of the Interna- tional Conference on Learning Representations, 2024.
Santurkar, S., Hlyss, A., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. Image synthesis with single (robust) classifier. Advances in Neural Information Processing Systems, 32, 2019.
Nguyen, A., Yosinski, J., and Clune, J. Understanding neu- ral networks via feature visualization: survey. Explain- able AI: interpreting, explaining and visualizing deep learning, 2019.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-CAM: Visual explana- tions from deep networks via gradient- based localization. In IEEE International Conference on Computer Vision, 2017.
Nguyen, G., Kim, D., and Nguyen, A. The effectiveness of feature attribution methods and its correlation with auto- matic evaluation scores. Advances in Neural Information Processing Systems, 2021.
Simonyan, K., Vedaldi, A., and Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. Proceedings of the Interna- tional Conference on Learning Representations, 2014.
Olah, C., Mordvintsev, A., and Schubert, L. Feature vi- sualization. Distill, 2017. doi: 1.23391/ddisiill.0000 htts://disilll.pbb2000 t/ffeatuee-
Sixt, L.,, Granz, M., and Landgraf, T. When explanations lie: Why many modified BP attributions fail. In Proceedings of the International Conference on Machine Learning. 2020.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H.,, Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, Alaaeldin Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. Dinov2: Learn ing robust visual features without supervision. Transac- tions on Machine Learning Research, 2023.
Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Watten- berg, M. Smoothgrad: Removing noise by adding noise. arXiv preprint 1706::::700..3880 2017.
Springenberg, J. T., Dosovitskiy, A., Brox, T., and Ried- miller, M. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 2014.
Papyan, V., Romano, Y., and Elad, M. Convolutional dictio- nary learning via local processing. International Confer- ence on Computer Vision, 2017.
Sucholutsky, I., Muttenthaler,, L., Weller, A., Peng, A., Bobu, A., Kim, B., Love, B. C.,, Grant, E., Groen, I., Achterberg, J., et al. Getting aligned on representational alignment. arXiv preprint arXiv:23 10. 13018, 2023.
Parekh, J., Khayatan, P., Shukor, M., Newson, A., and Cord, M. A concept-based explainability framework for large multimodal models. Advances in Neural Information Processing Systems, 2024.
Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribu- tion for deep networks. In Proceedings of the Interna- tional Conference on Machine Learning, 2017.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Surkov, V., Wendler, C., Terekhov, M., Deschenaux, J., West, R., and Gulcehre, C. Unpacking sdxl turbo: Inter- preting sok-too-mmae models with sparse autoencoders. arXiv preprint arXiv:24 10. 22366, 2024.
Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., and Nanda, N. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. 14435, 2024.
Tamkin, A., Taufeeque, M., and Goodman, N. D. Codebook features: Sparse and discrete interpretability for neural networks. arXiv preprint arXiv:2310.172300 2023.
Rao, S., Mahajan, S., Böhle, M., and Schiele, B. Discover- then-names Task agnosiii concept bottlenecks via auto- mated concept discovery. In Proceedings of the IEEE European Conference on Computer Vision, 2024.
Tariyal, S., Majumdar, A., Singh, R., and Vatsa, M. Deep 2016.
dictionary learning. IEEE Access,
Tasissa, A., Tankala, P., Murphy, J. M., and Ba, D. K-deep simplex: Manifold learning via local dictionaries. IEEE Transactions on Signal Processing. 2023.
Rubinstein, R., Bruckksein,, M., and Elad, M. Dictionaries for sparse representation modeling. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2010.
Tošić, I. and Frossard, P. Dictionary learning. IEEE Signal Processing Magazine, 28(2):27-38, 2011.
13

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., and Madry, A. Robustness may be at odds with accuracy. In Proceedings of the International Conference on Learning Representations, 2019.
Vielhaben, Bluecher, S., and Strodthoff, N. Multi- dimensional concept discovery (MCD): A unifying frame- work with completeness guarantees. Transactions Machine Learning Research, 2023.
Wattenberg, M. and Viégas, F. B. Relational composition in neural networks: A survey and call to action. arXiv preprint arXiv:240.11662,, 2024.
Wightman, PyTorch Image Models. https://gt thub. com/ http://wwwhhttta//oooooooo--eeee models, 2019.
Williams, R. Inverting connectionist network mapping by pack-propagation of error. In Proceedings of the Annual Meeting of the Cognitive Science Society, 1986.
Wright, S. Correlation and causation. Journal of Agricul- tural Research, 20(7):557-588, 1921.
Yu, Y. Buchanan, S., Pai, D., Chu, T., Wu, Z., Tong, S., Haeffele, B., and Ma, Y. White-box transformers via sparse rate reduction. Advances in Neural Information Processing Systems, 2023.
Zeiler, M. and Fergus, R. Visualizing and understanding convolutional networks. In Proceedings of the IEEE European Conference on Computer Vision, 2014.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L.. Sig- moid loss for language image pre-training. In IEEE Inter- national Conference Computer Vision, 2023.
Zhang, R., Madumal, P., Miller, T., Ehinger, K. A., and Ru- binstein, B. I. Invertible concept-based explanations for CNN models with non-negative concept activation vec- tors. In Proceedings the AAAI Conference Artificial Intelligence, 2021.
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. iBoT: Image bert pre-training with online tokenizer. Proceedings of the International Conference on Learning Representations, 2021.
14

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
A. Appendix
A.1. SAE Training Implementation details
We modify the TopK Sparse Autoencoder (SAB) (Gao et al., 2024) by replacing the &g loss with an & loss, as we find that this adjustment improves both training dynamics and the interpretability of the learned concepts. The encoder consists of single linear layer followed by batch normalization (loffe & Szegedy, 2015) and ReLU activation function, while the decoder is simple dictionary matrix.
For all experiments, we use dictionary of size 8 x 768 6144 which is expansion factor of 8 multiplied by the largest feature dimension in any of the three models, 768. All SAE encoder-decoder pairs have independent Adam optimizers (Kingma & Ba, 2015), each with an initial learning rate 3e-4, which decays to 1e-6 following cosine schedule with linear warmup. To account for variations in activation scales caused by architectural differences, we standardize each model's 's activations using 1000 random samples from the training set. Specifically, we compute the mean and standard deviation of activations for each model and apply standardization, thereby preserving the relative relationship between activation magnitudes and directions while mitigating scale differences.
Since IPP does not incorporate class token, we remove class tokens from D2 and ViT to ensure consistency across models. Additionally, we interpolate the V2 token count to match patch size of 16 x 16 pixels, aligning with SigLIP and VIT. We train all USAEs on a single NVIDIA RTX 6000 GPU, with training completing in approximately three days.
A.2. Unique Dino V2 Concepts
Vi's 's unique concepts are presented in Figures 10 and 11. Interestingly, we find concepts that solely fire for D2 related to depth. perspective, and geometric cues.
Concept 4756 Object Depth
Concept 1710 Background Depth (Uphill)
Figure 10. Qualitative results of low-entropy concepts that fire for Dino V2. We discover features related to depth cues for foreground objects well background in concept 4756 (above) and 1710 (below).
15

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Concept 4191 Top Face Prism
Concept 3448 Brim of Dome
Concept 1530 Prism Corners
Figure Qualitative results for low-entropy concepts that fire Dino We discover DinoV2 independent features that are not universal suggesting 3D understanding like corners (concepts 1530), top face of rectangular prism (concept 4191), and brim of dome (concept 3448).
A.3. Unique SigLIP Concepts
Similar to DinoV2, we isolate concepts with low tring-entrrppy where probability mass is concentrated for SigLIP. Example concepts presented in Fig. 12. We observe concepts that fire for both visual and textual elements of the same concept. Concept 5718 fires for both the shape of star, as well as regions of images with the word or even just a subset of letters on bottlecap and sign at different scales. Additionally, concept 2898 fires broadly for various musical instruments, well as music notes, while concept 923 fires for the letter 'C', For each of these concepts, the coordinated activation maximization visualization has both the physical semantic representation the concept, as well as printed text. The presence image and textual elements are expected given SigLIP trained as a vision-language model with a contrastive learning objective, where the aim is to align image and latent representations from separate image and language encoders. While we do not train on any activations directly from the language model, we still observe textual concepts in our image-space visualizations.
A.4. Dut-of-Distributiin Generalization
In order to assess the -ut-of-distribution capabilities of our approach, we use DTD (Cimpoi al., 2014) and CelebA (Liu al., 2015) as the validation dataset for our ImageNet trained USAES and show strong evidence of generalization outside of the training distribution seen in Table 1. We find consistent activation reconstruction accuracy (measured by MSE and R2), consistent trends in co-firing metrics in Fig. 13 and visualize some of the most important concepts for these new datasets, along with their associated highest activating images, from ImageNet in Fig. 14 and 15. Despite differences in domain and semantics, USAES trained on ImageNet exhibited robust generalization to both DTD and CelebA. Importantly, many of the concepts identified these datasets also aligned with high-actio activation concepts from ImageNet, suggesting that the USAE dictionary captures generalizable structure beyond training data.
16

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Concept 5718 Star (Shape
Letter)
LONE
YUMA
Concept 923 Capital C
Concept 2898 Music (Instruments + Notes)
Figure 12. Qualitative results low-entropy SigLIP concepts. We consistently find concepts that fire for abstract concepts image space such images or text of 'star' (concept 923), letters (concept 5718), and music notes (concept 2958).
Mean Squared Error (MSE) 4
Coefficient of Determination (R2) 1
Model
ImageNet
DTD
CelebA
ImageNet
DTD
CelebA
SigLIP
0.39
0.38
0.48
0.61
0.54
0.52
D2
0.19
0.26
0.22
0.77
0.69
0.75
ViT
0.41
0.56
0.69
0.59
0.46
0,45
Table 1. Performance comparison of vision models across different datasets. Lower MSE and higher Ra indicate better performance.
A.5. Additional Results
A.5.1. ADDITIONAL QUANTITATIVE RESULTS
Figure 16 presents concept consistency distributions across models for the top 1,000 co-firing concepts. We find overall increase in consistency between individual and universal concepts: the most universal (highest co-firing) concepts are more likely to be found in each model's respective independently trained SAE. Within this thresholded range, we find DinoV2 to exhibit the highest similarity between individual and universal concepts with an average cosine similarity of 0.65, followed by VIT 0.52 and SigLIP 0.40. Dino V2 concepts seem be better represented the universal space, suggesting that some models may have more universal concepts than others.
A.5.2. ADDITIONAL QUALITATIVE RESULTS
We provide additional universal concept visualizations for the top activating images for that concept across each model. Specifically, we showcase low-level concepts in Fig. 17 related to texture like shell and wood for concepts 1716 and 2533, respectively, as well as tiling for concept 5563. We also showcase high-level concepts in Fig. 18 related to environments like auditoriums concept 4691, object interactions like ground contact concept 5346, as well as facial features like snouts in concept 3479.
17

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Firing Metrics OOD Generalization to CelebA
Co-firing Proportions for Top Energy-Rande Concepts
sigl
Correlation Cofiring Importance (log-log log r=0.807)
Co-ltring Proportion
DinoV2
VIT
10P
0.6
Mean
Concapt importanan cance (log scale)
0.4
10-2
400
600
800
1000
1300 Concapts Enengy (Sorted
10"1
Distribution Aring Entropy
000
600
Number Co Fres (log scale)
400
200
brttcpp
Firing Metrics OOD Generalization to DTD
Co-firing Proportions
Top Energy-Rankers -arrgy---ankddd Concepts
0.8
SigLIP
inoV2
Corrclation between Cofiring and Importance (log-log r=0.808)
Co-ttring Proportion
0.6
VT
0.4
0.2
10-1_
Concept Ingortance
0.0
10-2
Top Concepts by Energy (Sorted by Mean Proport
Firing Entropy
10-2
Encropy
Number Conceets
020
10a
10a
10P
10P
10a
169
Number Co scalo)
400
08
Figure 13. Zero-shot generalization quantitative results of universal concepts on out-of-distribution datasets. Top: When applying our ImageNet trained USAE the validation set of CelebA we find consistent trends across each our universality metrics. We find clear correlation between co-firing and concept importance. The distribution over firing entropy also indicates concepts that fire uniquely for single model, two of three models, and universal concepts USAE to the validation of DTD find consistent trends across each of universality metrics. We find clear correlation between co-firing and concept importance. The distribution over firing entropy is tri modal, indicating concepts that fire uniquely for a single model, of three models, and universal concepts that fire for all three.
for all three. Bottom: When applying our ImageNet trained
18

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Concept 486 Sunglasses
CelebA
Imagenet
DinoV2
SigLIP
VIT
Concept 972 Lower Jaw
CelebA
Imagenet
DinoV2
SigLIP
Figure 14. Qualitative examples of zero-shot generalization of ImageNet trained USAE on CelebA. Top: We depict high-level visual concept dimension related sunglasses and the highest activating images for the validation sets of both ImageNet and CelebA. Bottom: We depict the lower-jaw concept's 's highest activating images for the validation set of ImageNet and CelebA. This jaw concept generalizes beyond animal jaw to include human jaws as seen from our CelebA heatmaps.
19

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Cancept 273 Checkerboard
DTD
Imagenet
DinoV2
SigLIP
Concept 4312 Zebra
DTD
Imagenet
DinoV2
SigLIP
Figure 15. Qualitative examples of zero-shot generalization of ImageNet trained USAE on DTD. Top: We depict checkerboard concept's highest activating images for the validation of ImageNet and DTD. This checkerboard concept generalizes from low-leve textures in DTD like tiles to sudoku and crossword puzzles in ImageNet. Bottom: We depict a concept for zebra stripes and its highest activating images for the validation set ImageNet and DTD. This stripe concept generalizes across scales images zoomed on the animal in DTD to across whole zebra in ImageNet.
20

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Concept Matches Across
Cosine Similarity Threshalds
Top 1000 cofiring Concepts
1.0
Fraction of Concepts C5 Thresholo
0.8
0.6
0.4
(AUC=0:40)
0.2
DinoV2 (AUC-0.65)
VIT (AUC=0.52) 52)
0.0
Random Baseline (AUC=0.13)
Cosine Similarity Threshold
Figure 16. Top 1000 co-firing concept consistency between independent SAEs and Universal SAEs. Our universal training objective discovers universal concepts that have overlap (i.c., cosine similarity) with those discovered with independent training. In descending order, Universal SAFs have highest overlap with independently trained DinoV2, ViT, and SigL.I. The smaller overlap observed with SigLIP suggests the aligned image-language embedding space produces unique concepts that are more distinct from those DinoV2 and VIT.
A.6. Limitations
Our universal concept discovery objective successfully discovers fundamental visual concepts encoded between vision models trained under distinct objectives and architectures, and allows us to explore features that fire distinctly for a particular model of interest under our regime. However, we note some limitations that we aim to address in future work. We notice some sensitivity to hyperparameters when increasing the number of models involved in universal training, and use hyperparameter sweeps to find an optimal configuration. We also constrain our problem to discovering features at the last layer of each vision model. We choose do so tractable first step in this novel paradigm of learning discover universal features. We leave exploration of universal features across different layer depths for future work. Lastly, do find qualitatively that small percentage of concepts are uninterpretable. They may be still stored in superposition (Elhage et al., 2022) or they could be useful for the model but simply difficult for humans to make sense of. This is a phenomena that independently trained SAEs suffer from as well. Many of the limitations of our approach are tightly coupled to the limitations training independent SAEs, an active area research.
21

---

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
Concept 1716 Shelll
Concept 2535 Wood
Concept 5663 Tiling
Dino
SigLIP
VIT
Figure 17. Qualitative results of universal concepts. We depict low-level visual features related textures, such shells (concept
1716), wood (concept 2533), and tiling (concept 5563).
Concept 4691 Auditorium
Concept 5346 Ground Contact
Concept 3479 Dark Snout
Dino
SigLIP
VIT
Figure 18. Qualitative results of universal concepts. We depict high-level visual features related to environments, such as auditoriums (concept 1691). ground contact (concept 5346), and animal snouts (concept 3479).
22


Reviews:
[{"review_id": "cRuVtoXaUF", "reviewer": "Reviewer_11mk", "summary": "The paper proposes a recipe to jointly train Sparse AutoEncoders (SAE) across different vision models in a shared (universal) space. The novel idea is to force the SAE to extract features (and, therefore, concepts) that are as shared as possible across models. This shared space enables cross-model and new within-model applications: the former allows studies of how different models encode the same concept, while the latter is used to characterize the distinction between model-specific and shared concept decompositions by comparing SAE and USAE decompositions.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank 11mk for their thorough analysis of our work. We appreciate that the reviewer found our work to be clearly written and well-motivated. We too believe cross-model interpretability is a truly exciting topic! We answer the questions raised below. For plots and figures, please refer to our temporary anonymous website hosted on Netlify: https://sparkling-queijadas-998747.netlify.app/\n\n### **Dictionary stability**\nWe agree that concept stability is an important metric to consider. Note that our cosine-similarity-based analysis (Sec. 4.4 & Sec. A.3.1) in the main paper measures the stability between independent SAEs and USAEs, which found that USAEs do indeed preserve some of the concepts found in independent SAE dictionaries. However, we appreciate the suggestion to explore stability across training runs of hyperparameter-matching USAEs. \n\nFurthermore, concept stability (via cosine similarity) in SAEs has been analysed in recent work (Fel et. al, 2025). We first independently verify this paper’s findings on ‘individual model’ TopK SAEs, closely matching the stability score they observed (~0.5). We further find that (TopK) USAEs exhibit similar stability to individual TopK SAEs (**Fig. F**), with scores of 0.52, 0.41 and 0.55 for SigLIP, DinoV2, and ViT, resp. \n\nFurthermore, we observe a strong positive correlation between concept stability and importance (**Fig. G**). The most important concepts—those contributing most to reconstruction—are also the most stable across runs, suggesting that universality and stability are linked.\n\n\n### **Bias in the learned dictionary**\nFor discussion on bias in the learned dictionary, see response to 51dn: **Dataset Generalization OOD**.\n\n### **Comparison: smaller/larger dictionary sizes**\n\nTo investigate whether smaller dictionaries favor more universal or entangled concepts, we ablated the expansion factor (4/8/12 = 3072/6144/9216 concepts). We observed that while all dictionary sizes recover many of the same high-importance concepts, **the smallest dictionary (×4) tends to emphasize the most universal and high-entropy concepts (Fig. D and Table B)**, likely due to its limited capacity. In contrast, larger dictionaries better capture low-frequency and low-entropy concepts, which tend to be more model-specific.\n\n### **Comparison with a subset of models** \nAs suggested, we train a USAE on a subset of the models (SigLIP & ViT) and, as expected, find it has a significant overlap of concepts with the 3-model USAE. The stability scores between their dictionaries are 0.47 and 0.5 for SigLIP and ViT, resp. In addition, we again observe the trend of stability/importance correlation: the top 1000 Energy concepts from the 3-model USAE exhibit an average stability of 0.65 with the 2-model USAE. We also note that our firing metrics remain consistent with the 3-model USAE in (**Fig E**). \nThese results reinforce our finding that the most **important and universal concepts are also the most stable**, and that universality is recoverable even from subsets.\n\n### **Universality of concepts**\nWe appreciate and agree with your framing: USAEs identify a possible set of shared concepts, not the definitive one. We will revise the paper’s language accordingly.\nStill, we show that the universal concepts we do identify are:\n- **Stable** across training runs,\n- **Important** to activation reconstruction,\n- **Generalizable** across datasets.\n\nThese properties form a compelling working definition of universality, and while we cannot claim full coverage, we believe USAEs capture a meaningful and interpretable subset of the shared conceptual space across models.\n\n### **Failure case analysis**\nWe appreciate the suggestion to include failure modes of USAEs. We did find difficult-to-interpret concepts  (concept 2188 in **Fig. H**) or model-biased concepts; e.g., typically related to positional information (concept 5728 in **Fig. H**) which was clearly represented in DinoV2 but not strongly represented in the other models. \n\nWhile these may appear as failure cases from a universality standpoint, we believe they help illustrate a key strength of USAEs: they surface not only shared concepts, but also **highlight the set of unique concepts of each model**. We will include these findings in the revised version.\n\n### **Other: References and description refinements**\nWe appreciate your feedback regarding Platonic Representations as well as the referral to review and the recommendation to clarify the high-level intuition for Sec. 3.3, and will implement these changes in the revised version. \n\nThank you again for the detailed review and suggestions for improvement! We hope we have addressed your main concerns, especially regarding the robustness of USAEs. If so, we would appreciate the consideration to raise your score or let us know what we can further demonstrate to help your decision!\n\nFel et al. “Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models” ArXiv 2025", "claims_and_evidence": "The claims are backed up by strong empirical evidence. The compatibility of extracted features is backed by quantitative measures (sections 4.2 and 4.3), while the interpretability applications are validated through qualitative experiments (sections 4.1 and 4.5). The combination of both makes for a strong case.", "methods_and_evaluation_criteria": "The methodology is straightforward (a plus!), with extra credit for the joint optimization process. Instead of optimising every pair, using a single space as a pivot, or adding an extra loss term, the paper takes a cleaner route: individually encoding each model’s activations while jointly decoding across all spaces. However, even though it’s noted in the appendix that the hyperparameters (I guess the SAE ones) require some tuning, it’s unclear how sensitive the method is to these choices. Adding some more information about it would certainly strengthen the paper.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments feel carefully designed and informative, with a mix of quantitative and qualitative measures that reinforce the key claims. \n\nMy main curiosity in this section is how stable the concept universality metrics (firing entropy, co-firing proportion, and concept consistency) are across different (U)SAE hyperparameters, such as the dictionary size, the chosen models, or the dataset. This would provide a clearer picture of the method’s robustness and how “universal” the extracted features really are. *For example*, does a smaller dictionary recover a subset of the strongest concepts found with a larger dictionary, or does it lead to a more entangled representation? Does the method extract similar concepts with just a subset (2) of the models?", "relation_to_broader_scientific_literature": "This work has broad implications for the representation learning and interpretability fields. The study of representation compatibility and alignment is an active research topic, and the proposed framework naturally fits within this space by unlocking cross-model studies. One particularly interesting direction would be applying the same method to other modalities (yes, I’m referring specifically to LLMs). A shared concept space is, by definition, abstract. Therefore, it would be interesting to see some cross-domain extensions. A starting point could be applying USAE to the vision branch of CLIP but then applying the learned encoder $\\Psi$ and decoder $D$ on the language branch.", "essential_references_not_discussed": "A few comments on the references:\n- The citations to \"The Platonic Representation Hypothesis\" (PRH) (Huh et al., 2024) aren’t always accurate. The perfect placement is in the related works section under Feature Universality, supporting the discussion. But in the Introduction, it’s cited as a “technique for identifying universal features,” which isn’t quite right. PRH is more of a hypothesis about the existence and the convergence to a shared representation space than a method for discovering them.\n- In the Introduction, I would reference \"Relative Representations enable Zero-Shot Latent Space Communication\" (Moschella et al., 2023). This paper shows that different models, regardless of initialization, pretraining task, or architecture, can exhibit latent spaces that align well enough to be projected into a shared/universal one. That’s very much in line with the idea of universality in USAE-extracted concepts.\n- In the \"Concept-Based Interpretability\" paragraph of the related works section, I would include \"Interpreting CLIP’s Image Representation via Text-Based Decomposition\" (Gandelsman et al., 2024). This paper analyzes CLIP’s image encoder by decomposing (output-level) image representations into interpretable components (e.g., contributions from individual attention heads), using text representations as a dictionary for the decomposition.", "other_strengths_and_weaknesses": "### Extra Strengths\n- The writing is **amazingly clear, well-structured, and well-motivated**. The paper is a pleasure to read. I particularly appreciated that many questions that came up during my first read were addressed immediately (e.g., the USAE vs. SAE analysis). The only section that initially gave me trouble was 3.3 (Coordinated Activation Maximization), which could have been improved by adding a high-level sentence or two explaining the methodology in practical terms, making it more accessible.\n- Personally, I think cross-model interpretability is a **truly exciting topic**!\n\n### Extra Weaknesses\n- **No failure case analysis**. The paper only shows successful examples of extracted concepts, but what happens when the USAE fails? Does it sometimes miss clearly present concepts or identify ones that aren’t? Seeing failure cases where the decomposition disagrees with human intuition would give a better sense of its limitations.\n- **Potential bias in the learned dictionary**. Since the USAE extracts concepts from ImageNet (I found this information in the appendix, I would add a sentence in \"Implementation details\" under section 4 specifying it), it’s likely biased toward the dataset’s structure/composition. But how does the dictionary change if trained on a different dataset? Some analysis of how the discovered dictionary shifts across datasets would be valuable.\n- An analysis of **concept stability across runs** is missing. I would expect different runs to produce slightly different dictionaries. But how much is \"slightly\"? Do the same concepts emerge consistently?", "other_comments_or_suggestions": "I'm adding it here because it can be considered a concurrent paper. I think \"Sparse Autoencoders Reveal Universal Feature Spaces Across Large Language Models\" (Lan et al., 2024) is highly related to this work. In that one, representations of independently learned SAE on LLMs are aligned/compared post-training with similar universality findings.", "questions_for_authors": "I find the paper to be very well-executed, and the core idea is highly interesting. I'm more than open to increasing my rating if the authors could further strengthen it by showing some experiment that addresses the \"universality\" robustness (see weaknesses 2 and 3 and \"Experimental Designs Or Analyses\"). \n\nTo be clear, I don't think this is needed for the paper to be acceptable (that's why I'm already giving a 4), but given how central the “universality” claim is, appearing prominently throughout the text and even in the title, some extra validation would make it more substantiated. With the current validations, I would only refer to it as **a** shared set of concepts to jointly decompose multiple models.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p0RBUhaHDk", "reviewer": "Reviewer_51dn", "summary": "This work introduces Universal Sparse Autoencoders (USAEs), a method to discover concepts shared across different deep learning models. The authors focus on the study of USAEs for the last-layer representations of three popular vision models, showing that the methodology enables the construction of an interpretable feature space shared across different architectures. The authors present a coordinated activation maximization procedure to investigate concept alignment. The paper supports that the proposed procedure produces meaningful universal features through qualitative and quantitative analyses.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 51dn for their thorough review of our work. We appreciate that they found the paper original and well-structured, and that they found the proposed Coordinated Activation Maximization application a promising tool for concept visualization. For plots and figures, please refer to our temporary anonymous website hosted on the Netlify platform: https://sparkling-queijadas-998747.netlify.app/\n\n### **Dataset Generalization (OOD)**\nWe agree that testing for potential dataset bias is critical. To this end, we evaluated OOD generalization using two diverse datasets outside of ImageNet: DTD, a texture dataset  (e.g., stripes, spotted, …),, and CelebA a face dataset with 40 binary attributes (e.g., glasses, wavy hair, …).\n\nUsing DTD and CelebA as the validation dataset for our ImageNet trained USAEs show strong evidence of generalization outside of the training distribution (**Table A**). We find consistent activation reconstruction accuracy (measured by MSE and R2), consistent trends in co-firing metrics (**Fig. C**) and visualize some of the most important concepts for these new datasets, along with their associated highest activating images, from ImageNet (**Fig. A and B**). Despite differences in domain and semantics, USAEs trained on ImageNet exhibited robust generalization to both DTD and CelebA. Importantly, many of the concepts identified in these datasets also aligned with high-activation concepts from ImageNet, suggesting that the USAE dictionary captures generalizable structure beyond its training data.\n\n### **Scalability, automatic strategies beyond visual inspection** \nAs in prior work [Ghorbani et al., 2019; Fel et al., 2023c; Kowal et al., 2024b], we determine concept semantics via qualitative inspection: for each concept, we collect its top-activating image examples and generate corresponding spatial token heatmaps to aid interpretation. While this manual approach remains standard, we believe that emerging vision-language models (e.g., vision-capable LLMs) offer promising tools for automating concept summarization and interpretation. We view this as an exciting direction for future work.\n\n### **USAE vs SAE features**\nWe agree with the reviewer that further exploring the differences between USAE concepts and common SAE concepts is an interesting direction. We assume 51dn is asking for a characterization and comparison between universal features mined between independent SAEs vs universal concepts learned from our joint approach. To the best of our knowledge, all previous work does only pairwise analysis to mine overlapping concepts between independent SAEs [Lan et al. 2024], scaling these approaches beyond pairwise comparisons is currently not possible without substantial modifications to the previous methods. Developing a working baseline that extends these post-hoc mining approaches beyond pairwise is out of the scope of our work. We demonstrate that USAEs and independent SAE have fewer overlapping features (USAE-SAE, Sec 4.4 Fig. 7), yet demonstrate much higher overlap between themselves i.e., USAE-USAE, and SAE-SAE (see 11mk Dictionary Stability), further indicating USAEs do learn unique features that are not captured by independent SAEs.\n\t\nLan et al. “Sparse Autoencoders Reveal Universal Feature Spaces Across Large Language Models.” ArXiv 2024", "claims_and_evidence": "The claims in the paper are well supported by experimental evidence.\nBoth qualitative (Fig 1) and quantitative (Fig 3 and 4) results support the evidence that USAEs learn meaningful features shared among the three models.  \nThe claim that USAEs learn features at different levels of granularity is supported by clear qualitative evidence. \n\nThe work would benefit from further evidence that the method allows the construction of truly universal features. Considering that all three models have been trained on Imagenet, the claim would be further strengthened by showing that this is still true for images not in the training set.", "methods_and_evaluation_criteria": "The method proposed by the authors is suited for the purpose of constructing interpretable features that are shared among different vision models.\n\nThe evaluation criteria are generally well constructed but can be strengthened in some parts. For example, in Section 4.4, the authors could better characterize what type of features the USAE method allows to learn with respect to SAE features, which are shared among the different models. The analysis in this section would benefit from a more clear description.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The cross-model reconstruction, co-firing rate, and energy-based importance metrics are appropriate for measuring concept alignment. Qualitative visualizations effectively illustrate shared concepts.", "relation_to_broader_scientific_literature": "The authors contextualize well their work within existing literature.", "essential_references_not_discussed": "To the reviewer's knowledge, there are not major papers that have been overlooked by the authors.", "other_strengths_and_weaknesses": "The authors propose an original, well-structured way to analyze shared representations across multiple architectures, which leads to interesting observations concerning model alignment.\nThe Coordinated Activation Maximization method proposed in the work is a useful strategy for comparing concept representations across different models.\nThe presentation of the results and the definition of the necessary mathematical concepts are explained with a good level of clarity.\nThe work would benefit from further evidence that the feature identified by the USAE methodology generalize to other dataset and are not restricted to the three models considered in the work.\nA more thorough discussion of the advantages of the proposed strategy in terms of scalability could benefit the work, specially in the light of the fact that the authors do not outline (at least in terms of a proof of concept) automatic strategies to interpret USAEs coordinates beyond visual inspection.", "other_comments_or_suggestions": "I have no further comments or suggestions.", "questions_for_authors": "I have no further questions for the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hCEOC9FN5G", "reviewer": "Reviewer_42Tj", "summary": "The authors introduce USAEs, a framework that jointly learns a universal concept space for the internal activations of multiple vision models. By optimizing a shared objective, they show that USAEs semantially coherent universal concepts at different levels across vision models. Their results showcase the strong correlation between concept universality and importance, and also identify unique features that are learned by individual models. A unique application of USAEs, coordinated activation maximization, is also presented to achieve simultaneous visualization of universal concepts across models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful analysis of our work, and appreciate the positive compliments regarding the presentation quality, novelty, and extensive experimental results. We answer the questions raised below in text form. For plots and figures, please refer to our temporary anonymous website hosted on the Netlify platform: https://sparkling-queijadas-998747.netlify.app/\n\n\n### **USAE vs SAE Features**\nSee our response **USAE vs SAE features** to (51dn) for further discussion.   \n\n### **Firing Entropy vs Co-Firing Proportion**\nSee our response **Firing Entropy vs Co-Fire Proportion** to (SLTN)\n\n\n### **Clarifying Questions** \n\n1. *Is Energy(k) (Equation (9)) computed for just one model? Or is it computing a universal score for the concept k across models?*\n\n    Energy is first computed individually across each model and we take the average to rank concepts. \n\n2. *A baseline would be trying to identify shared concepts in independent SAEs. Do USAEs find more universal concepts than just comparing independent SAEs? Also, there is a lack of visual comparison between USAEs and SAEs.*\n\n    Please see **USAE vs SAE features** in our response to (51dn) \n\n3. *Will USAEs learn more redundant concepts than SAEs?*\n\n    Assuming redundant concepts refers to when multiple concepts that look visually similar and appear repeatedly, we do not observe many of them in the learned dictionary. Designing a formal way of quantifying their frequency and performing a deeper analysis of their frequency is an interesting direction to explore for future work.\n\n4. *Although the authors claim Equation (6) will \"strike a practical balance between training speed and memory usage\", there is no empirical evidence supporting this.*\n\n    The alternative approach for mining universal concepts between individual SAE’s is restricted to being pairwise [Lan et al. 2024]; scaling this approach greatly increases its computational complexity. Our approach, being learning-based, maintains the same complexity at the cost of training time as we increase the number of models used. \n\n5. *It is not clear to me how the concept meanings shown in this paper were generated. Were they manually summarized by humans?*\n\n    Concept meanings were determined by qualitative inspection, as is common practice in previous works  [Ghorbani et al. 2019, Fel et al. 2023c, Kowal et al. 2024b]. However, we believe the rise of more capable vision+language models (e.g., Vision-LLMs) could aid in automated summarization of the results. \n\n6. *How is the threshold determined for results in Figure 5?*\n\n    Assuming the threshold being referred to is Figure 5. (C): We observed a clear phase transition beyond 1000 concepts (e.g., r=0.63 vs. r=0.89). Thus, we aimed to analyze properties of these highest co-firing concepts and set the threshold to >1000 highest co-firing concepts.\n\n7. *More discussion/analysis should be made on the connection between FE and CFP. Will the high value of one metric always lead to the high value of the other?*\n\n    See (SLTN) Firing Entropy vs Co-Fire Proportion\n\n8. *Why do the results in line 357 indicate that SigLIP and ViT share more concepts?*\n\n    Phrased differently, the results in L357 indicate that SigLIP and ViT share a higher fraction of total concepts which co-fire across all three models, this is likely due to DinoV2 possessing a higher fraction of unique concepts due to its training objective promoting 3D scene understanding, the concepts of which we observe in appendix Fig.10 and 11 encoded as low-entropy concepts. \n\n### **Implementation Details**\nSee response to (SLTN) **implementation details**.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "While previous papers focus on learning the concepts in individual models with SAEs, this paper propose a new USAE for learning universal concepts across models.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The introduction of USAEs is novel, exploring a new direction of learning universal SAE concepts across models\n2. Extensive experiments are done to demonstrate the capture of universal concepts using USAEs.\n3. The presentation is clear and easy to follow.\n\nWeaknesses:\n1. Some implementation details are not clear (e.g., what is the threshold used in Figure 5; how are the concept meanings summarized).\n2. The comparison between USAEs and independent SAEs is not sufficient (e.g., do they find more universal concepts than just comparing independent SAEs; will USAEs show more consistent visual results?).\n3. Some experiments need more discussions (e.g., what is the connection between FE and CFP?)", "other_comments_or_suggestions": "1. While $x\\in \\mathcal{X}$ is written in Equation (8), I don't see where $x$ is used in the equation. Is it the same as $x^{(i)}$?\n2. Where are the results for the discussion in Line 317?", "questions_for_authors": "1. Is Energy(k) (Equation (9)) computed for just one model? Or is it computing a universal score for the concept k across models?\n2. A baseline would be trying to identify shared concepts in independent SAEs. Do USAEs find more universal concepts than just comparing independent SAEs? Also, there is a lack of visual comparison between USAEs and SAEs.\n3. Will USAEs learn more redundant concepts than SAEs?\n4. Although the authors claim Equation (6) will \"strike a practical balance between training speed and memory usage\", there is no empirical evidence supporting this.\n5. It is not clear to me how the concept meanings shown in this paper were generated. Were they manually summarized by humans?\n6. How is the threshold determined for results in Figure 5.\n7. More discussion/analysis should be made on the connection between FE and CFP. Will the high value of one metric always lead to the high value of the other?\n8. Why do the results in line 357 indicate that SigLIP and ViT share more concepts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qKGB7sLjQY", "reviewer": "Reviewer_SLTN", "summary": "The submission is based on learning a single concept space that is shared between SAEs trained on multiple vision models.  The aim is to learn a universal set of concepts which can be used to translate between different models and highlight differences in how models represent visual information.\n\nThe experiments center on a universal concept space trained on a set of three models (SigLIP, ViT, and DinoV2).  Various characterizations are performed: the dominant concepts are visualized via activation heatmaps; the reconstruction quality across and within each model; concept firing distributions; comparison to singular SAEs; and universal concept activation maximization.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough analysis of our work, and thank them for finding our paper is nicely written with well-motivated analyses. We aim to answer the questions raised below. For plots and figures, please refer to: https://sparkling-queijadas-998747.netlify.app/ \n\n### **Implementation Details**\n*“Can any evidence at all be presented for L1 improving interpretability over L2?”*\nIn our early experiments, we found that training USAEs with an L1 reconstruction loss tended to yield more interpretable concepts than L2—particularly in the sparsity and visual clarity of activations. While this remains a qualitative observation, it aligns with prior findings in cross-model interpretability settings [Lindsey et al., 2024]. We emphasize that USAEs are not tied to a specific loss; we simply chose L1 based on these initial results and precedent.\n\n*“How many epochs were needed for convergence?”*\nAll USAEs were trained for 30 epochs, after which reconstruction metrics plateaued and we observed diminishing returns. This setting was consistent across all experiments.\n\n*“Was the interpolation bilinear?”*\nWe appreciate the suggestion and agree that the token interpolation choice is an interesting direction. We used bilinear interpolation in our experiments. Its main effect appears as smoothing in the border regions of attribution maps. However, we haven't observed meaningful differences in the types of concepts discovered. We plan to explore alternative schemes (e.g., bicubic, with and without anti-aliasing) in future work.\n\n*“What were the specific versions of the model used?”*\nSigLIP: 'vit_base_patch16_siglip_224'\nDinoV2: ‘facebookresearch/dinov2_vits14’\nViT: ‘'vit_base_patch16_224'\n\n*“In the top-K SAE implementation, what is k and how was it determined?”*\nWe set K = 32 for all experiments, following prior work on sparse autoencoders [Gao et al., 2024]. While this choice worked well empirically, we acknowledge that concept interpretability and stability may vary with K, and we leave a deeper hyperparameter sweep to future work.\n\n### **Firing Threshold $\\tau$**\nWe agree that one could just use the K largest dictionary entries; however, our use of a threshold was to reduce noise in the case for which some of the top K firing dimensions are very close to 0. For example, some concepts may be captured successfully with only a subset of K dimensions, which could be identified with a small threshold to ensure the magnitude of the activation is sufficient.\n\n### **Firing Entropy vs Co-Fire Proportion**\nWe agree with the reviewer that modifying the firing-entropy (FE) metric to measure the distribution of fires per concept between models may yield additional useful insights. The FE metric focuses on firing counts (irrespective of co-fires) to initially probe whether concepts were particularly biased to any of the models. We choose to measure this to avoid degenerate solutions of reserving subsets of concept dimensions for specific models. Once we determined that many concepts fired with equal probability for each model, we used co-firing proportion (CFP) as a more granular token-level analysis to see if these concepts fired for the same tokens. \n\n\t\n### **Training on Different Layers for Frame of Reference**\nWe appreciate the suggestion to train our USAE on the first and last layers of a single network as a baseline. We expect partial concept overlap between layers, as shown in Fig. 4 where low-level features (e.g., color blue, concept 4235) emerge from last-layer training. Our DTD experiments (**Generalization OOD** 51dn) further demonstrate detection of low-level concepts when applying ImageNet-trained USAE to texture data. Given neural networks' hierarchical feature learning, finding low-level features in the last layer suggests our method may identify \"layer-redundant\" features encoded across multiple network depths. This makes cross-layer training an imperfect mismatch but still a valuable experiment.\n\n### **R2 off-diagonal scores**\nIn the confusion matrix (Fig. 5), the maximum possible off-diagonal R2 score is 1, which would imply that activations from one model (e.g., SigLIP) can be perfectly reconstructed by encoding activations from a different model (e.g., DinoV2 or ViT). While the upper bound for such cross-model reconstruction is unclear, our positive off-diagonal R2 scores already provide strong evidence of shared structure across models. These results suggest that USAEs capture meaningful, transferable representations even across architectures and training paradigms.\n\nWe believe that further optimizing the USAE design—including architecture (increasing the depth of the encoder, Matching Pursuit encoder…), loss functions (HSIC, Cosine, …), training objectives (iBot style…), and hyperparameters—could improve cross-model reconstruction and raise this empirical bound. We leave a more detailed exploration of these design factors to future work.", "claims_and_evidence": "The paper convincingly supports the claim that a shared concept space was learned, and I agree that the analysis is of a depth that novel insights are offered about how these different models represent visual information.  The claim of novelty around “coordinated activation maximization” appears overblown -- it’s activation maximization on the learned space, repeatedly applied.", "methods_and_evaluation_criteria": "Largely, yes.  There are a few issues which would strengthen the paper if addressed:\n\n- Why introduce a firing threshold $\\tau$ when using a top-k SAE?  In other words, we know exactly what concepts fire for each input -- it’s the top k.  By introducing a degree of arbitrariness, and one that is decoupled from the functioning of the SAE, subsequent processing is called into question.  \n\n- The firing entropy does not actually use firing patterns across the models; it merely compares counts.  A concept could fire completely disjointly across the models (say, 20 times for each model on a total of 60 inputs) and the paper’s firing entropy metric would be maximized.  Thus the evaluation is problematic, and might be improved by directly assessing the firing distribution per concept.  This would be easy -- each firing pattern is a three bit vector, and the authors could calculate something like the total correlation of the joint distribution from the 8 probability values.", "theoretical_claims": "I saw no theoretical claims.", "experimental_designs_or_analyses": "The paper provides little information about the experimentation that culminated in the final setup, which weakens the presentation of the method.  \n\nCan any evidence at all be presented for L1 improving interpretability over L2 in the topK SAE implementation?\n\nHow many epochs were needed for convergence?\n\nThe interpolation of the size 14 patches used in dinoV2 to the size 16 patches for the other two models is reasonable, though it would be great to have more of a discussion and demonstration of the effect of this interpolation vs any other form.  Was the interpolation bilinear?\n\nWhat were the specific versions of the models used, given that `timm` has multiple for each?\n\nIn the top-K SAE implementation, what is k and how was it determined?", "relation_to_broader_scientific_literature": "The paper leverages SAEs to connect multiple vision models into a shared space -- a nice idea given that SAEs are increasingly used to gain interpretability in individual models.", "essential_references_not_discussed": "No, the references appear quite thorough.", "other_strengths_and_weaknesses": "The paper is nicely written and the analyses are well-motivated, even if there is some room for improvement.", "other_comments_or_suggestions": "Could it provide a useful frame of reference to train a USAE with a known mismatch, such as with an early and a late layer of the same model?  The sense in which concepts are shared or not might be clearer, in order to build a better understanding of the method before application to varied vision models.  I understand this is most likely too much for the rebuttal period, so commenting how the authors see this experiment playing out would be fine.", "questions_for_authors": "- Regarding reconstruction $R^2$ values, the paper states “positive off-diagonal $R^2$ scores indicate successful cross-model reconstruction...” (L313).  Merely having a positive $R^2$ is an extremely low bar for success.  The off-diagonal values are around 0.3-0.4 -- which still does not seem particularly successful.  Can the authors provide more grounding for why this is good enough, and not higher?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Strengths:**

(1) The introduction of USAEs for learning shared concept space across vision models is original and timely.

(2) The paper presents a diverse set of qualitative and quantitative experiments, including cross-model reconstruction, concept alignment, and visualizations.

(3)  The paper is clearly written and well-structured, making complex ideas easily understandable.

**Weaknesses:**

(1) Initial framing of “universality” could be overstated without deeper robustness checks (e.g., stability, OOD generalization, concept overlap).

(2) The distinction between USAEs and post-hoc pairwise SAE comparisons could be better emphasized, especially visually.

(3) Some implementation specifics (e.g., thresholds, interpolation choices) should have been better clarified in the main paper. Additionally, certain metrics (e.g., firing entropy) could benefit from more principled, distribution-aware formulations to strengthen their interpretability and robustness.

**Discussion:**

The paper proposes a sparse autoencoder based framework that opens up new directions in cross-model interpretability. While reviewers 11mk and SLTN reasonably endorsed the work, other reviewers (51dn, 42Tj) expressed mild issues, primarily about the strength of universality claims, implementational clarity. Importantly, the authors responded thoroughly to all criticisms, providing additional experiments on concept stability, OOD generalization, and clarifications around metrics and implementation choices. The rebuttal successfully addressed core concerns of the reviewers and highlighted the authors’ dedication to rigor and openness.

**Recommendation: Accept**

This paper makes a novel and well-supported contribution to the interpretability and representation learning literature. Its insights into shared concept structure across models are timely and practically relevant. While some aspects can be strengthened in future works, the current work is a strong and impactful addition to the field.
Decision: Accept (poster)