Representational Difference Explanations
Authors: Neehar Kondapaneni
Oisin Mac Aodha
Pietro Perona
Venue: NeurIPS 2025 poster ()
Keywords: interpretability
explainability
XAI
graph
comparison
similarity
differences
representational similarity
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: A new XAI for model comparison that isolates representational differences.

Abstract: We propose a method for discovering and visualizing the differences between two learned representations, enabling more direct and interpretable model comparisons. We validate our method, which we call Representational Differences Explanations (RDX), by using it to compare models with known conceptual differences and demonstrate that it recovers meaningful distinctions where existing explainable AI (XAI) techniques fail. Applied to state-of-the-art models on challenging subsets of the ImageNet and iNaturalist datasets, RDX reveals both insightful representational differences and subtle patterns in the data. Although comparison is a cornerstone of scientific analysis, current tools in machine learning, namely post hoc XAI methods, struggle to support model comparison effectively. Our work addresses this gap by introducing an effective and explainable tool for contrasting model representations.

Representational Difference Explanations
Nechar Kondapancni'
Oisin Mac 2odha²
Pictro Perona
Caltech
University of Edinburgh
Abstract
We propose a method for discovering and visualizing the differences be- tween two learned representations, enabling more direct and interpretable model comparisons. We validate our method, which we call Representa- lional Differences Erplanations (RDX), by using it to compare models with known conceptual differences and demonstrrate that it recovers meaningful distinctions where existing explainable Al (XAI) techniques fail. Applied state-of-the-art models on challenging subsets the ImageNet and iNat- uralist datasets, RDX reveals both insightful representational differences and subtle patterns in the data. Although comparison is a cornerstonc of scientific analysis, current tools in machine learning, namely post hoc XAI methods, struggle to support model comparison effectively. Our work addresses this gap by introducing an effective and explainable tool for contrasting model representations. Project Page: RDX
Introduction
In recent years, deep learning researchers and engineers have explored the costs and benefits of using larger datasets and more complex architectures. These changes often lead to distinct models with different representations of the same data. An intuitive approach to understanding the effects of different architectures and training choices to analyze the representational differences between models. Dietionary learning (DL)-based explainable AT (XAT) methods, like sparse autoencoders (SAFs) and non-negative matrix factorization (NMF), are powerful tools for analyzing model representations that surface model concepts, i.e., semantically meaningful sub-components of the input data 15 are formulated as dictionary learning problem [3] such that model representations are decomposod into a lincar combination of learned concept vectors. Concept vectors are then converted into human-frient friendlyy explanations by selecting a subset of input items (c.g., a set of images) that maximally align with the vector. These explanations have been shown to help users better understand models [ 15 26 54. However, when adapting existing DL-based XAT methods for comparing models with known differences, we find that they often generate explanations that are unrelated to the known difference between models.
26 54
These approaches
We identify three issues with existing DL-based XAAI methods that limit their power of analysis, especially when comparing representations. First, when representational differences are relatively small, concepts from different models tend to overlap and is thus difficult to spot differences. Second, we observe that cxisting methods explain concepts by sampling and visualizing items with the largest activations that are casiest to interpret, and miss more nuanced differences, thus offering incomplete explanations. Finally, undersland subtle differences between models, users need to consider the weighted sum of several concepts via their incomplete explanations which can be difficult due to the imprecise nature of the task.
26 12
which tend be the ones
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Repr. A
Repr. B
RDX(A, B)
A
A
All shapes are separable.
Circles and squares are separable.
RDX(A, B) isolates regions that are considered similar in A, but not B.
Figure 1: Intuition behind our method. Representational Difference Explanations (RDX) aim to highlight the substantive differences between two representations (c.g., A and B, which are the embedding matrices produced by two different models for the same set of data). Here A supports discrimination between circles and squares, whereas B does not. Clustcring the two representations independently would not reveal the square/circk sub-structure unique to A. By "subtracting" B from A, RDX reveals which items are considered similar in A, but not in B. RDX isolates differences, and ignores data that may be equally well grouped in both representations, such as the triangles and diamonds.
We propose new XAI method, named Representational Difference Explanations (RDX) for explaining model differences. Rather than focusing on one representation at a time, RDX comparcs representations against cach other to isolate the differences between them (Fig. D. Additionally, unlike Do-methods which generate incomplete explanations for concept vectors, take the perspective that concept and its explanation are one and the same thing. To achieve this, we enforce that each concept is defined by a small set of semantically related samples from dataset.
We make the following key contributions: (1) A new method, RDX, that identifies explainable differences between model representations. (2) A new metric to measure the effectiveness of such representational difference explanations. (3) Experiments comparing RDX against baseline methods for explaining model differences, demonstrating its superiority.
Related Work
Explainability. Explainable AI methods for computer vision atlempt. to generale expla- nations to help users understand model behavior. There are two broad classes of methods: local methods A B7 55 b0 attribute regions of an image to model's decision and often take the form of heatmaps, while global methods [8] 26 72 generate global explanation (e.g., grid of images or image regions) that represent a visual concept that learned by a model. Visual concepts are often represented by a set of images that are considered similar due a sharod visual fcature. For example, the visual concept of 'rod circle' maybe represented by set of images that all contain red circles. Visual concepts can either be defined by a user-selocted set of similar input images [26] or be discovered directly from the model by grouping images the model considers similar [3] These visual concepts are used
to help users achieve more general understanding of model behavior [ 15
20 23
For example, they can reveal that model has learned to use water cue for detecting a certain species of waterbird. Local and global methods can also be combined to provide detailed explanations that describe both the concepts and the image regions used by model 62
when making a decision [
Representational Similarity. Representational similarity methods 21
22
30
35
50
aim quantify the similarity between network representations. These methods operate by passing the same set of items through two models to generate two embedding matrices. These embedding matrices are then compared, resulting single value that quantifies their degree of similarity. While these approaches can provide useful, coarse-grained insights 31
40 42
[0 73 they do not help with understanding fine-grained model differences. Recently, several methods have been proposed which compare networks through interpretable

---

RSVC 28 vision SAEs b8, and NLMCD 6d extract concepts independently for each model and match them in a subsequent step, resulting in partially overlapping concepts that can make interpretation challenging. USAFs G2
concepts 28 58
62
6o
employ "universal" sparse autoencoders that must be trained for each new model and dataset to learn a common representational space across several models. This training step makes generalization to different models or datasets challenging. Additionally, none of these methods are designed to specifically seek out differences, although differences may be detected as byproduct of their approaches. In contrast, our approach uses information from both representations ions simultancously to discover differences between them. It requires training, making it casy to apply to new models and bespoke datasets.
Comparing Graphs. Our approach is related to graph comparison methods. Many methods have been developed for comparing graphs, including methods for matching the largest common subgraphs 5,, detecting anomalies 45 grouping network types 22, and measuring the similarity between graphs via kernels 00 The majority of existing methods are concerned with developing specialized strategies for comparing very large web-scale graphs with mismatched nodes. In addition, these approaches aim to quantify network similarity with score rather than to visualize and understand qualitative differences. In contrast, is to provide fin--grinndd, qualitative understanding of the differences between two "graphs" that have the same nodes, but different edge weights. While some approaches []] AN| have been developed to visualize differences between graphs, these methods focus on highlighting
goal
the addition and removal of nodes. Most relevant to our work is DISC [57], a modification of the spectral clustering algorithm. DiSC addresses a setting in which there are two experimental conditions, where the same types of measurements are taken in both conditions. Given this shared feature space, it seeks out features that cluster together in one condition, but not in the other. This paradigm is relevant for biological experiments, in which genes may co-activate in certain experimental conditions. Our approach differs from DiSC in two key ways: (1) Different neural networks do not have a shared feature space, therefore we focus on discovering differential clusters of inputs, not features. (2) We construct an affinity matrix emphasizing the difference between representations. This makes our approach more flexible than DiSC, since it can be used with any clustering algorithm.
3 Method
We propose a method, RDX, to explain the differences between two models via concepts by identifying inputs that only one of the two models considers to be semantically related. To do so, we construct an affinity matrix that assigns high affinity to pairs of inputs that are similar according to representation A, but dissimilar according to representation B. We cluster this affinity matrix to reveal distinctive similarity structures in A. At a high-level, RDX performs the following steps: (1) compute the pairwise distances between inputs in A and B to build distance matrices, DA and DB. (2) compute the normalized difference between the matrices to form difference matrices GA,B and GB,A, and (3) use the difference matrices to sample difference explanations, i.., explanations that reveal where the two representations disagree. Intuitively, negative edges in GA,B indicate that the corresponding pair of inputs were closer together in A than they were in B.
As input, we have data items from which we compute two embedding matrices obtained from two different models, A E gaxda and B E RRaxan, where d and dg are the embedding dimensions for models A and B respectively. A and B contain embeddings over the same set of inputs, where each row corresponds to the same input item, i.c., each row is an embedding vector. We refer to the 5th embedding vector in A as a, We consider several options for each step of RDX and provide details for the best choices in the following sections. Additional model variants are described in Appendix B
3.1 Computing Normalized Distances
To contrast representations using their distance matrices, the distances must be comparable.
Neighborhood Distances. We compute the pairwise Euclidean distance matrices, DA E Raxa and DB E R=xa, for each embedding matrix separately. For each entry a, in a given embedding matrix, we rank all other entries by their distance to a. This rank is used as the
3

---

scale-invariant nearest neighbor distance between a, and a,. Thus, distances are integers between and n. We refer to the outputs as the normalized distance matrices DA and DB.
3.2 Constructing Difference Matrices
Given DA and DB. we develop a method for comparing the normalized distances that emphasizes instances where either model considers two inputs similar. The method is asymmetric. Here present it in one direction.
Locally Biased Difference Function.
Consider two pairs of embeddings with indices i, and k. Suppose D 500 and D 600
in the first pair, and D and D 101 in the second. Comparing the distances across
the representations (500- - 600, 1 - 101) results in the same amount change (-100, -100),
but a change from distance of 1 to distance of 101 suggests a more important conceptual difference. To address this issue, we propose a locally-biased difference function (Fig. A2
G A,B tanb(y -DD - / D))).
(1)
By dividing by the minimum distance across both representations, this function prioritizes differences in embedding distances in which either representation considers the embeddings to be similar. This ensures that large differences in distant embeddings are ignored, but large differences in nearby embeddings are emphasized. To avoid exponential growth in our difference function when distances small, apply tanh function to normalize the outputs, with y controlling how quickly the function saturates. Given an item indexed by i, this function will output negative values when the distance between i and j is smaller in A than in B. Thus, negative matrix entries denote items that are closer in A than in B.
3.3 Sampling Difference Explanation Grids
The next step is to communicate representational differences to the user. Sets of images, presented an image grid of 9-25 images, have been used to communicate concepts for visual data 15 26 We aim to sample m sets of images (i.e., image grids) from a difference matrix. Each set of images should contain images that are considered similar by only A, i.e. indices that have pairwise negative matrix entries in G 'A."' We refer to this set of images as difference explanation (E) that defines concept. unique to one model and we refer to the collection of difference explanat ions as AA, If we consider the difference matrix as the adjacency matrix of a graph, we are essentially looking for a subgraph of size |E] with large negative values on all edges. There are many options for sampling subgraphs, we consider a direct sampling (see Appendix B.5) and spectral clustering based approach.
Spectral Clustering. We convert GA,B into an affinity matrix: FA,B (xp(-3-GAAB) To ensure the affinity matrix is symmetric, affinity matrix, we sock to sample a set of m difference explanation grids A Given an aflinity matrix, spectral clustering solves relaxed version of the normalized cut problem [7] Normalized cuts (N-Cut) seek out a partition of a graph that minimizes the sum of the cut edges, while balaneing the size of the partition 56 Since, edges in FA,D are large when inputs are closer in A than they are in B, spectral clustering is biased to finding partitions in which inputs are close together in A, but far apart in B. In practice, when both representations have a similar structure, edges in that structure will have an affinity close 1, since the difference near 0. To discard these regions, we generate m clusters and discard the cluster with lowest mean affinity as it contains regions we are uninterested in. Spectral clusters can contain too many inputs to be visualized all at once. To convert each cluster into an explanation grid (E), we define the k-neighborhood affinity (KNA). For each image in the spectral cluster, the KNA the sum of the edges between that image and its k-nearest neighbors (from within the cluster). Recall that larger alfinity edges indicate more disagreement about the similarity between two images, thus we select the image and neighbors corresponding to the max KNA (see pseudo-code in Appendix B.1
average it with its transpose. From this
4

---

3.4 Representational Alignment
When models have significant representational differences, it possible that these differences could be mitigated by aligning the representations. For example, both models may be the same up to some (e.g., linear) transformation. In these settings, it can be useful to first maximize the alignment between models before exploring the representational differences, since this can reveal fundamental differences between the models. To align the representation A E Raxda to B E [Rxxds, we learn transformation matrix MAB RdAxda that minimizes the centered kernel alignment. (CKA) loss
5 between the transformed A and B:
M = arg min 1 CKA(AM, B),
(2)
M
where CKA(..-) represents the linear CKA similarity. We denote A aligned to B as A'. See Appendix for training details.
3.5 Evaluating Explanations
All explanation methods in this work pro- duce set of explanation grids for both rep- resentations, i.c., recall that E is the set of explanation grids for A. The goal of difference explanation grid is to identify sets of items that are closer logether in one rep- resentation than they are in the other. We develop metric, the binary success rate (BSR), to evaluate whether a method has succeeded at this task. D represents the distance between two ilems representation A and D' represents the same for represen- tation B. We measure how frequently the 9: end for
Algorithm 1: Evaluation of Explana-
tions Representation A.
1: Input: Grids EA ......... Ex),
and distances D Da
2: BSR = 0
3: for each grid E E E do
4: for each pair (i.j) E E, i do
5:
if D < D then
6:
BSR
end if 8: end for
7:
distance for a pair of items from an explanation grid is smaller in A than in B. In the pseudo-code above, we provide the algorithm for computing BSR(EA). In addition to the BSR metric, we compute three metrics from SemanticLens [12 Redundancy (modified), Clarity, and Polysemanticity. We present details for computing these metrics in Appendix B.2
4 Results
We conduct three experiments to evaluate the effectiveness of RDX. In Sec. 4.2
we compare two simple representations with subtle differences to show that cxisting XAI methods fail to explain these differences. In Sec. 4.3 show that these trends continue to hold in more realistic settings. Through modifications of various models, we manipulate representation ions to have "known" differences. We then stage comparisons and assess whether existing XAT methods are able to recover these differences. In Sec. 4.4 we use RDX to compare models with unknown differences and find that it can reveal novel insights about models and datasets.
4.1 Implementation Details
We use several models in our evaluation. Unless specified otherwise, for our modified MNIST experiments, we use a 2-layer convolutional network with an output dimension of 64. We also train post-hoc concept bottleneck model (PCBM) 70 with ResNet-18
20 backbone
on the CUB dataset, 6S using a standard training procedure 770 Finally, we conduct
experiments using several models that are available from the timm library 64: DINO 6
vs. DINOv2 A3] and CLIP 49 vs. CLIP-iNat (i.e., CLIP model fine-tuned on data from the iNaturalist platform 24). In these experiments, models are compared on subsets of images from 2-4 commonly confused classes in ImageNet [I0] or iNaturalist 24 More
training dotails are provided Appendix C.1 We compare our approach to several DL for sparse auto-encoders (SAE) 41
XAI basclines: top-k sparse auto-encoders (TKSAE) 17
38 non-negative matrix factorization (NMF) [3], principal component analysis (PCA) [G and KMeanS 30 We use convex non-negalive matrix factorization (CNMF) [[]] if the activations of the last. layer contains negative values. We also compare to method explicitly designed for model comparison, non-linear multi ditti-dimensional concept discovery (NLMCD) [5 We provide
5

---

5))
NMF(M g)
sM)( Explanations
Ms
M
5
3 3 5
8
8
8 8
(MF( concept pt samples.
MMFM concept samples.
555
3 3
3
8
8
555
3
3
3
8
NMF
3) Explanations
5
5
S
3 3 3
8 8
553
3 3 3
8
5
3
3 3
8
(A) NMF explanations are indistinguishable despite significant differences in representation.
RDX(M .,M g)
RDX(M s. M
Explanations
E
5 5 5
5
Image samples RDX(Mg.MM co
5
3 3
3
3
5
5 5
5
6
3
3
3
5
8
5s are spread apart in Mg.
but close in Ms.
A style of 3s and 5s that only
Ms can't distinguish.
A style of 3s, 5s and 8s that Ms can't distinguis,, but can.
Acc: 95%
Acc: 98%
(B) RDX highlights differences between Ms and Mg, by identifying concepts specific to Ms.
Figure 2: Comparing RDX to NMF. We train small CNN on a modified MNIST dataset that only contains images of the digits 3, 5, and 8. We compare a strong model checkpoint representation (Ms, 95% accuracy) with a final 'expert' model representation (Mr, 98% accuracy). The left and middle columns show PCA projections of the Mş and Mg repre- sentations, respectively. The transparent colors indicate classes in the dataset: 3 (light-blue) orange,, and (light-green). The right most columns visualize the images selected by the explanation methods. We extract three concepts for each method. (A) We generate explanations using NMF [5] with maximum sampling [3 15, 28 for Ms and Mg. Bold in the
colored points the PCA plots indicate the location of the sampled images right-most column. We find that NMF is unable to reveal any representational difference between Mg and ME because it produces indistinguishable explanations for both models. (B) In contrast, RDX discovers concepts unique to Ms by identifying images that are more similar in Mg than in MR. The sampled points are overlaid on both models' represeniations and show tight clusters in Ms that contrast with diffuse points in Mg. The right column shows the corresponding explanations, highlighting how model representations differ.
details on basclinc methods in Appendix C.3 and details for RDX are provided in Appendix We conduct ablations in Appendix
4.2 Dictionary Learning Fails to Reveal Differences in Similar Representations
Dictionary learning (DL) approaches for XAT are commonly used to discover and explain vision models 15
72. We hypothesize that explanation grids sampled from DL. concepts are not helpful for describing differences between similar representations, even if the representations contain behaviorally significant differences. To test this, train a 2-layer convolutional network with an output dimension of 8 on a modified MNIST dataset that contains only images for the digits 3, 5, and 8. We compare checkpoint from epoch 1 with strong performance (95% accuracy) to the final, 'expert' checkpoint at epoch 5 (98%). We refer to these checkpoints Ms (strong model) and Mg (expert model). We conduct this experiment to assess if an XAI method can discover subtle differences between two models.
58
62
A good difference explanation should reveal the concepts that explain why Mş under- performs ME by 3%. Tn Fig. we show that NMF with maximum sampling generates effectively the same explanation grid for both representations. This is because NMF has learned

---

MNIST
CCBM
Unaligned
Aligned
1.0
88
0.5
85
0.5
84-
0.4
03
0.3
0.3
0.2
0.2
11
01
20
RDN
75545
0.
RDDN
0.
BOX
BBX
(A) Recovering "Known" Differences
(B) Discovering Unknown Differences
Figure 3: Binary success rate evaluation of XAI methods. For each XAI method, we
compute the binary success rate (BSR) (Sec. 3.5 all difference experiments, where higher is better. We use neighborhood distances to mcasure BSR (Sec. 3.1). Each method (x-axis) is assigned different color, we show BSR(E A) (darker box) and BSR (lighter box). (A) We show results on the MNIST and CUB PCBM experiments (Sec. 4.3 in which we modify
representation and Lest if RDX can help identify the modification. B We show results when comparing large vision models with unknown differences (Sec. 4.4p- We compare recovering differences without (left) and with (right) an initial alignment step (Sec. 3.4. In all cases, our RDX approach consistently outperforms the dictionary learning baselines. A complete set
of results available in Table A3
KMeans (,,)
(
RDX M M M₁
4
5
2 R
8 6 0
C
8
6
333
4
7
3
2
8
6
6
4
4
8
8888
8
3
3
6
6
S
6
6
8
6
&
3
3
3
KMeans (1)
NMF M 1L)
RDX(M
M
NJ M2)
Q
3 2
2
8 8
4
0
4
3 3
3
2
a
O
5
&
1
4
4
4
3
3
3
2
Q
9
%
0
3
3
3
2
Figure 4: Recovering vertical flip modifications on MNIST. We visualize explanations produced by three XAI methods, RDX, KMeans, and NMF, to compare models Mj and M Both models are trained on dataset with vertically flipped and normal images. Mj is trained to associate the original label to flipped digits and Mj is trained to predict a new set of labels for flipped digits. We expect Mj to mix flipped and unflipped digits while My should separate them. We gencrate three explanations for cach method. (Left, Middle) KMeans and NMF generate explanations that are difficult to understand. (Right) RDX captures the expected difference. RDX RDX(M,, M21) reveals that M: represents flipped and unflipped 6s, 7s, and 9s closer together than in Mr 10: shows that Mf. has clean clusters of 3s, flipped 5s, and flipped 2s without any mixing.
highly similar concepts for both representations, and the representational differences are captured in smaller and noisier concept coefficients for images that Ms is less certain about. Maximum sampling selects the images with the largest coefficients, meaning these images are not sampled when visualizing concepts (Fig. A2 A). In Fig. A1 show that SAE and KMeans also fail to explain representational differences. An alternative approach to understanding differences could be inspect individual images of interest and try to understand them through their concepts. In Fig. A2 B, we show the difficulty of reasoning about an image via weighted combination of concept explanations. In contrast, RDX concepts are equivalent to their explanation grid and are sampled from regions of differences. This ensures that RDX explanations select images considered similar by Ms that Mg does not consider similar. In Fig. 2 B, we can see that Ms is confused by certain styles of 3s, 5s, and 8s that look Mg. In Fig. [ICC C we visualize the reverse direction for RDX and
similar when compared find that Mg contains clusters of challenging 3s and 5s, that are confused by Ms. Finally, in Appendix A.1.3 we discuss perfectly monosemantic DL concepts would solve these issucs. We argue that monosemanticity is likely infeasible when trying to compare ropresentational differences and that, even achieved, it cannot solve the challenge of interpreting weighted combinations of explanations.

---

RDX(CA-S: CA)
RDX(CA Ca-s)
CNMF (C
CNMF(CA)
Mix of non-spotted and spotted wings in explanation for Ca-s. In contrast, only spotted wings in Cą explanation.
Indistinguishable explanations for both models that are unrelated to wing spotted-ness.
Figure 5: Recovering the "Spotted Wing" concept in CUB. We train a post-hoc concept bottleneck model on the CUB dataset. For each image, we use the predicted concept scores as the image's embedding vector (i.e., representation). Here we compare & model using the complete concept representation (CA) with model representation withoul the spotted wing concept (CA-s). We visualize one of five generated explanations for each model using RDX and CNMF. We observe that RDX's explanation focuses the spotted wing concept. It shows us that only CA-S mixes images with and without spotted wings. In contrast, the CNMF explanations for each model are both unrelated to the spotted wing concept and indistinguishable from each other, since the represontations are highly similar and CNMF discovers nearly the same concepts in both. See Fig. A4 for all five explanations.
4.3 RDX Recovers "Known" Differences
Here we evaluate different XAI approaches by comparing MNIST trained models that have a modified training procedure. We select modifications such that we can have strong expectations the differences between the learned representations (see Table A8 for full list). For example, we trained two models on MNIST dataset with vertically flipped digits, where M2 was trained with the same labels for both normal and flipped digits and Mr was given new labels for flipped digits. We expect that only M, will mix flipped and unflipped digits. In Fig. 4 we visualize the outputs of three XAI methods for comparing Mj and M- We clearly see that RDX's explanations focus on the actual expected difference, that M2 considers flipped and normal digits as being more similar than M In contrast, KMeans and NMF result in unfocused and seemingly random explanations. In Fig. 3 A (left) we can see that this trend is consistent as all baseline methods have a lower BSR than RDX.
shows
To explore differences between models trained on more complex images, we use post-hoc concept bottleneck model (PCBM) trained on the CUB bird species dataset (Appendix C.1). The CUB PCBM (CA) predicts a score for 112 human-defined concepts, these concepts are then used to make species classification decisions, where we treat the concept predictions as feature vector for an image. In each comparison, we remove single concept from the feature vectors and compare the representations. The list of eliminated concepts used in this experiment can be found in Table A8 In Fig, 3 A (right) method for this experiment. We find that RDX performs better than the baselines, especially for difference explanations showing concepts unique to the complete representation CA- In Fig. 5 we visualize the outputs of RDX and CNMF when comparing a model without the spotted wing concept (CA-s) against CA. As expected, we find that difference explanations show that CA-S mixes images with and without spots, whereas, CA is much better at show that CNMF can result in both
report the BSR score for each
grouping images with spotted wings. In contrast, unrelated and indistinguishable explanations. We show more examples in Appendix A.2 Taken together, these results indicate that RDX is capable of revealing how changes in both training and fino-grained concepts can affect a model's representation.
4.4
RDX Discovers "Unknown" Differences
In our final experiment, we test the effectiveness of RDX for knowledge discovery by applying two models with unknown differences. We compare DINO with DINOv2 on four groups of ImageNet classes. We also compare CLIP against (CT.IP-iNat) model on three groups of different species. We conduct all of the knowledge discovery experiments with and without alignment. We align one model at time, resulting
iNaturalist fine-tuned CLIP
8

---

RDX 02. M) on Primates (ImageNet)
Expl. 2
Expl. 4
M D2
M'
Expl:
Expl:
Expt:
RDX cN, Mc) on Maples (NNaturalist)
Expl. 4
Expl. 5
M CN
M
Pxpl:
Expl.
Expl.
Expl
Expl.
Figure 6: Discovering unknown differences. We use RDX to generate difference expla- nations for representations with unknown differences. We visualize two comparisons with alignment. In both comparisons, we visualize the shared structure (gray), spectral cluster membership (light colors), and selected samples for explanations (dark colors) on PCA pro- jections of the representations. We in the left PCA plot, but are spread apart in the right one. (RDX MD)) on Primates. We discover unique concepts in DINOv2 for commonly confused primates in ImageNet. In the PCA plot, we see that the green (Expl. 2) and purple (Expl. 4) explanations are cleanly separated in MD2, but mixed in M The explanations show that only DINOv2 has unique concepts for tan-colored gibbons and for gibbons with white chin fur. (RDX(McN: M2)) on Maples. We compare the representations of CLIP-iNat and CLIP on four types of maple trees from iNaturalist. We see that CLIP-iNat contains a unique concept for fall-colored Red Maple leaves (Expl. 4) and second concept that mixes fall-colored and green Silver Maple leaves (Expl. 5). Further analysis is provided in Sec. 4.4
that the selected indices grouped compactly
twice the number of comparisons for baseline methods. We can see in Fig. 3 that RDX oulperforms all baseline methods in discovering representational differences. Additionally, we see that aligument makes more challenging discover differences for the baseline methods, but RDX maintains good performance in both settings In Fig. 0 (RDX (MX M)),, we visualize difference explanations by comparing DINOv2 to DINO using images from three primate classes from ImageNet. We find that DINOv2 does better job organizing two types gibbons with different visual characteristics, suggesting that would be more capable than DINO at fine-grained classification. In Fig. t (RDX(McN. Mc)),, visualize fine-grained difference explanations on specics of maple trees. We find that only CLIP-iNat contains well-separated concepts for two different species of maple, despite both clusters sharing a secondary characterislic of leaves with fall-colors. While CLIP does not mix the images from these concepts, we see that it does not group them as lightly, suggesting it may be organizing images using a different characteristic like color. We apply RDX to several more examples in Appendix A.3 and vision language model to assist in the analysis.
4.5 Additional Metrics and Concept Consistency
Finally, we present detailed results for each comparison using the BSR and SemanticLens 12 metrics in Appendix A.4 As expected, we find that RDX is the best performer on the BSR (Table A3) and Redundancy (Table A4p metrics. is comparable
other methods on the Clarity (Table A5p and Polysemanticily (Table A6 metrics. We also conduct an analysis of concept consistency under in-distribution dataset variations Appendix A.5 and find
that RDX generates reasonably consistent concepts Fig. All

---

5 Discussion
Here we discuss some of the limitations of RDX and our analysis.
5.1 Limitations
Compute. Computing and storing the full pairwise distance matrix requires O(n2) memory, which may become impractical for large n. In this work, we are able to apply our method to at least 5000 data points but we have not explored larger values of n.
Concept Definition. While our decision to define concepts via a grid of images is helpful for users, it does not allow us to communicate concepts like "roundness" that may vary linearly in response to objects with increasing roundness. Instead, concepts like "roundness" would be discretized into sub-concepts that can be communicated by an explanation grid. BSR agreement with human-interppetabilie have the same BSR, but have significant differences in what they focus on in their explanations. Thus, propose that BSR should not be dircctly optimized for, but should instcad be proxy metric, and that qualitative results should always be used to support BSR scores. Alignment. Comparing after alignment is more likely to result in detecting fundamental representaaliona ional differences. However, it is possible that there are two different a aligned representations that result. in the same aligmment training loss. This would lead to different, but equally valid explanations and would require users to reason about feature correlations. Dependence on Euclidean distances. RDX relies on Euclidean distances which could cause issues for accurate concept discovery. Specifically, given two datapoints, it is possible that the Euclidean distance between the points small, but the points are quite far apart along the data manifold in the representation. RDX does not explicitly handle this issue, but there are two main reasons it is likely not a major concern. We discuss these reasons in Appendix A.6
In Fig, A15 we find that two methods can
Breadth. Our approach works on any representation, but we focus on vision models in our experiments. Future work should explore if this approach can be useful when comparing text. and multi-modal representations. Utility. We find that RDX explanations are useful for identifying representational differences, but more work needs to be done to link these representational differences to performance differences on specific tasks such classification. In Appendix A.2 (Maples), we see only some RDX explanations align with differences in classification. RDX unsupervised in that it only requires two sets of representations input, but in future work it would be interesting to explore incorporating classifier information into RDX explanations.
5.2 Societal Impact:
We do not anticipate any specific ethical or usage concerns with the method proposed in this work. We propose a method for model comparison which we hope will improve our understanding of model representations. Deeper insight can lead to better detection of bias, better understanding of methods, and discovery new knowledge about our datasets that may be beneficial for society. However, better understanding can also amplify negative usages of AI.
6 Conclusions
As models become larger and more powerful they are encoding more and more distinct concepts. As a result,, describing and understanding these discovered concepts is an important question in XAI. In this work, we posit that comparing representations allows us to filter away common structure and reveal concepts that may be more interesting to the user. To achieve this, we introduced RDX, a new approach for isolating the differences between two representations. RDX requires no training, can be applied to any model that generates an embedding for an input, and is general framework that can casily be modified with different choices for its intermodiate steps. In our experiments we show that RDX is able to recover known model differences, and is also able to surface interesting unknown differences. The resulting observations can provide insight into model differences and the training data used by different models. The next step is testing RDX in real-life applications to see if it can be used to help experts, such as radiologists or ecologists, discover new concepts in their models and datasets.
10

---

Acknowledgements. We thank Atharva Sehgal, Rogério Guimarães, and Michael Hobley for providing feedback on the work. OMA was supported by a Royal Society Research Grant. NK and PP were supported by the Resnick Sustainability Institute.
References
[1] Reduan Achtibat, Maximilian Dreyer, Iloaa Eisenbraun, Sebastian Bosse, Thomas Wiegand, Wojeiech Samek, and Sebastian Lapuschkin. From altribution maps to human-undderslandabbee explanations through concept relevance propagation. Nalure Machine Intelligence, 2023.
[] Edoardo M Airoldi, Xue Bai, and Kathleen M Carley. Network sampling and classifi- cation: An investigation of network model representations. Decision support systems, 2011.
33 Daniel Archambault. Structural differences between two graphs through hierarchies. In Graphics Interface, 2009.
[4] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus- Robert Müller, and Wojciech Samek. On Pixel- Wise Explanations for Non-Linear Classifier Decisions by Layer-Wiss Wise Relevance Propagation. PLOS ONE, 2015.
[5] Horst Bunke. a relation between graph edit distance and maximum common subgraph. Pullern recognilion lellers, 1997.
66] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bo- janowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[7] ChatGPT-40, 2025.
88 Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What cannot predict, i do not understand: A human-centered evaluation framework for explainability methods. NeurIPS, 2022.
99] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. ICLR, 2024.
[10] Jia Dong, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image databasc. In CVPR, 2009.
[1] Chris HQ Ding, Tao Li,, and Michael T Jordan. Convex and semi-nonnegative matrix factorizations. PAMI, 2008.
[12] Maximilian Dreyer, Jim Berend, Tobias Labarta, Johanna Vielhaben, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. Mechanistic underslanding and validation of large ai models with semanticcens. Nalure Machine Intelligence, 2025.
[13] Thomas Fel, Victor Boutin, Louis Béthune, Rémi Cadène, Mazda Moayeri, Lóo Andóol, Mathicu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. NewrIPS, 2023.
[14] Thomas Fel, Ekdeep Singh Lubana, Jacob S Prince, Matthew Kowal, Victor Boutin, Isabel Papadimitriou, Binxu Wang, Martin Wattenberg, Demba Ba, and Talia Konkle. Archetypal sae: Adaptive and stable dictionary learning for concept extraction in large vision models. In ICLR, 2025.
[15] Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. CRAFT: Concept recursive activation factor- ization for explainability. In CVPR, 2023.
[16] Karl Pearson F.R.S. Liii. on lines and planes of closest fit. to systems of points in space. The London, Edinburgh, and Dublin Philosophicol Magazine and Journal of Science, 1901.
11

---

[17] Leo Gao, Tom Dupré la Tour, Henk Tilmman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In ICIRR, 2025.
[18] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. NeurIPS, 2019.
[19] Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez Addressing lcakage in concopt bottleneck models. NewrIPS, 2022.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[21] Harold Hotelling. Relations between two sets of variates. In Biometrika, 1936. The platonic
[22] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola.
representation hypothesis. In ICML, 2024.
[23] Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, et al. Openclip. Zenodo, 2021.
[24] iNaturalist, 2025.
[25] Kempner Institute / overcomplete. Overcomplete: Vision-based sae toolbox. https 2025. Accessed: 2025-10-16.
[26] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TACV). In ICML, 2018.
[27] Dioderik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[28] Neehar Kondapaneni, Oisin Mac Aodha, and Pietro Perona. Representational similarity
via interpretable visual concepts. In ICLR, 2025.
[29] Neehar Kondapaneni, Markus Marks, Oisin Mac Aodha, and Pictro Perona. Less is more:
Discovering concise network explanations. In ICLR Workshop on Representational Aligument, 2024.
[30] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICML, 2019.
[31] Harold W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 1955.
[32] Yann LeCun. http://yann lecun.
The mnist database of handwritten digits. 1998.
com/ lew.n.coom///oce.e..o......o..l.
[33] Daniel Lee and H Sebastian Seung. Algorithms for non- non-negative matrix factorization. NeurIPS, 2000.
[34] Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Out-of- distribution robustness via disagreement. In ICLR, 2023.
[35] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In International Workshop on Feature Extraction: Modern Questions and Challenges at NewrIPS, 2015.
[36] Stuart Lloyd. Least squares quantization in pem. Transactions on information theory, 1982.
[37] Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In NewrIPS, 2017.
12

---

[88] Alireza Makhzani and Brendan Frey, K-sparse autoencoders. In ICLR, 2014.
[39] Leland Mclnnes, John Healy, and Steve Astels. hdbscaa: Hierarchical density based clustering. The Journal of Open Source Software, 2017.
[40] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang, What is being transferred in transfer learning? NewrIPS, 2020.
[41] Andrew Ng et al. Sparse autoencoder. CS294A Lecture notes, 2011.
[42] Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In ICLR, 2021.
[43] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2024.
[44] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford infolab, 1999.
[45] Panagiotis Papadimitrion, Ali Dasdan, and Hector Mallia-Moolin. Web graph similarity for anomaly detection. Journal of Internet Services and Applications, 2010.
[46] Young-Jin Park, Hao Wang, Shervin Ardeshir, and Navid Azizan. Quantifying represen- tation reliability in self-supervised learning models. In UAI, 2024.
[47] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. JMLR, 2011.
[48] Helen C Purchase, Eve Hoggan, and Carsten Görg. How important is the "mental map" an empirical investigation of a dynamic graph layout algorithm. In Graph Drawing: 14th International Symposium, 2007.
[49] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[50] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singu- lar vector canonical correlation analysis for deep learning dynamics and interpretability. NeurIPS, 2017.
[5]] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? NeurIPS, 2021.
(52) Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. or arxiv:2407. 14435. 2024.
53] Aninda Saha, Alina Bialkowski, and Sara Khalifa. Distilling representational similarity using centered kernel aligument (cka). In BMVC, 2022.
[54] Lisa Schut,, Nenad Tomašov, Thomas McGrath, Demis Hassabis, Urrich Paquet, and Been Kim. Bridging the human ai knowledge gap through concept discovery and transfer in alphazero. PNAS, 2025.
(55] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vodantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient- based localization. IJC,, 2020.
[56] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. TPAMI, 2000.
13

---

[57] Ram Dyuthi Sristi, Gal Mishne, and Ariel Jaffe. Disc: Differential spectral clustering of features. NeurIPS, 2022.
[58] Samuel Stevens, Wei-Lun Chao, Tanya Berger Wolf, and Yu Su. Sparse autoencoders for scientifically rigorous interpretation of vision models. arXiv:2502.0677 2025.
[59] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep
networks. In ICML, 2017.
[60] Oleg Sémery.
Pytorchev: Computer vision
models for pytorch.
https:/ https://pypi.oorr http://pppp..ogppooo// http:////otttooreeeeee pytorchev, 2018
[61] Joshua Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 2000.
[62] Harrish Thasarathan, Julian Forsyth, Thomas Fel, Matthew Kowal, and Konstantinos Derpanis. Universal sparse autoencoders: Interpretable cross-model concept alignment. In ICML, 2025.
[63] Christian Thurau. https://pypi.orge
Python matrix factorization module. http://pppppoooooppjctt//t/ project/PyMMi 2011. Version 0.2.
Pymf:
[64] PyTorch Image Models (timm), 2025. https://timm.faase fast.ai.
[65]
Johanna Vielhaben, Dilyara Bareeva, Jim Berend, Wojciech Samek, and Nils Strodthoff. Beyond scalars: Concept-based alignment analysis in vision transformers. In NewrIPS, 2025.
[66] S Vichy N Vishwauathan, Nicol N Schraudolph, Risi Kondor, and Karsten M Borgwardt. Graph kernels. JMLR, 2010.
[67] Ulrike Von Luxburg. tutorial on spectral clustcring. Statistics and computing, 2007.
[68] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[69] Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In CVPR, 2023.
[70] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. ICLR, 2023.
[71] Lihi Zelnik-Manor and Pictro Perona. Self-tuning spectral clustering. NewrIPS, 2004.
[72] Ruíhan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible concept-based explanations for models with non-negative concept activation vectors. In AAAI, 2021.
73 Wentao Zhang, Jiawei Jiang, Yingxia Shao, and Bin Cui. Efficient diversity-driven ensemble for deep neural networks. In ICDE, 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we perform a quantitative evaluation Sec. 3.5
conduct experiments
to show that existing methods fail qualitatively (Appendix A.1.2 and quantita-
tively (Sec. 4.3 and show that our method works (Sec. 4.3 and Sec. 4.4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
No or NA answer this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: Limitations are discussed in Sec. 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations ouly holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting, Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algo- rithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fcar that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assump- tions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not prove any theorems in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the thcorems, formulas, and proofs in the paper should be numbered and cross-reforenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can cither appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: Our method described in Sec. and implementation details for reproducing experiments are provided in Appendix C Code will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in varions ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make possible for others to replicate the model with the same datasct, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference docs require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should doscribe the architecture clcarly and fully.
(c) the contribution is a new model (e.g,, a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e-g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code will be published provided the work is accepted. The data is public and we will provide instructions on how to download We will provide links custom datascts.
Guidelines:
The answer NA mcans that paper does not include experiments roquiring code.
Please see the NeurIPS code and data submission guidelines (https://nips
cc/public/ cc/publicc/uidees/Cooleee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submis- sion guidclines httpps///aas...coooaoolll//////s for more details.
The authors should provide instructions on data access and preparation, in- cluding how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/ /detais
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification:
We provide all details Appendix
C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental sctting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
17

---

Justification:
We provide error bars in our quantitative plots in Fig. 3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yos" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CT, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: These are provided in Appendix C The compute costs for this work are fairly low, we provide times, but not have precisely recorded memory usage. We provide the total RAM available to the machine.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htte:://harrrrp.ccc/opaa///hhhhh
Answer: [Yes]
Justification: This work does not have any ethical issues specific to its methodology. The images downloaded for our datasets have public licenses with attribution.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
18

---

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negalive societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential socictal impacts in Sec. 5.2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g,, disinformation, generating fake profiles, surveillance) fairness consid- crations (o.g., deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from foodback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not any datasets models with high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Relcased models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implemeenting safely filters.
Datascts that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
19

---

Answer: [Yes]
Justification: All code, data, and models are attributed according to the licenses they have provided.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
ios: Our small comparison dataset license descriptions are provided in Ap- pendix C
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code /ooeel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include an anonymized zip file.
14. Crowdsourcing and rescarch with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not conduct crowdsourcing or human experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be inchuded in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
20

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (TRB) approvals (or an equivalent approval/revinn based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No human experiments were conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects rescarch. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between insti- tutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs are not used in any important way in the core methods. We use them lo assist with generating pseudocode from textual descriptions of the algorithm and for suggesting notation for the equations in the text. We assess LLMs on their ability describe qualitative results in Appendix A.3 but this is not main focus of the work.
Guidelines:
The answer NA means that the core method development this research does not involve LLMS as any important, original, non-standard components.
Please refer to our LLM policy (https://neurii) ce for what should or should not be described.
21

---

Appendix
Additional Results
22
A.1
Additional Results MNNIST-3,5.
22
A.2
Additional Results for Recovering "Known" Differences
26
A.3
Additional Results for Discovering "Unknown" Differences
30
A.4
Additional Metrics and Detailed Results for Comparison Experiments
37
A.5
Concept. Consistency Under In- Di-tirtibbttion Dataset. Perturbations
40
A.6
Discussion of Euclidean Distances)
41
B
Additional Methods Description
41
B.1
K-ncighborhook Affinity (KNA) Pscudocode
41
B.2
Computing SemanticLens metrics
B.3
Normalized Distance Variants
43
B.4
Difference Matrix Function Variants
44
B.5
Diflerence Explanation Sampling Variants
44
B.6
Results) ................................
44
C
Implementatonn Details
47
C.1
Model Training
47
C.2
Aligmment Training
C
Basclines
C.4
RDX Details
C.5
Comparison Summary
C.6
Computational Cost
49
A
Additional Results
A.1
Additional Results on MNIIT-(3,,588
We train a small convolutional network on a modified MNIST dataset containing only images for digits 3, 5 and 8. We compare two checkpoints from training at epoch (Ms) and epoch 5 (Mr). These checkpoints differ in representation and overall performance. In Fig, 2 we showed the results from applying NMF this setting. Here, we also evaluate SAE and KMeans.
A.1.1 SAE and KMeans Fail to Explain Representational Differences
In Fig. Al we visualize the explanations generated by SAE and KMeans. We find that both methods fail to generate explanations that can help understand the difference between the two representations. The SAE generates confusing explanations that may even be misleading. Surprisingly, the SAE explanations for Ms are less mixed than Mg, suggesting Mg has better separated representational space, which we know to be incorrect. This is likely a result of random variations in the concepts discovered by the SAE, a phenomena also observed in [4. KMeans, like NMF, generates indistinguishable explanations for both representations This is due the images near the centroids of similar ropresentations being effectively the same, since these are regions in which model confidence is higher.
A.1.2 General Issues with Interpreting Dictionary Learning Methods.
There two critical issues with interpreting explanations from dictionary learning methods. We visualize these issues in Fig. A2 We show that an explanation consisting of images corresponding the maximum coellicients of concept is incomplele explanation. This is because concepts can be polysemantic and encode multiple visual features. Polysemantic concepts will have relatively large coefficients for multiple groups of semantically related images. However, by only visualizing the top-k images for concept vector, we miss important information about other images that shares the same concept voctor. These "other" images, with smaller concept cocfficients, are critical for understanding the task of comparing two representations, as they may underlic the subtle variations that distinguish one model from another (Fig. A2 A).
22

---

SAB(Ms)
SAE(M g)
Ms
Mg
SAE(M
Explanations
8
3
3 3 3 3 3 3
3
3
3
3 3
3 3
3
3
3
3
3 3
SAB
R)
Explanations
S
8 8
6 6
5 5
5
8
8
5
&
5 5
S
8
8
(A) SAE explanations are surprisingly misleading about the representational difference.
(Means(Ms)
KMeans(M
KMeans M, Explanations
Ms
M
5
5
i 8
8
3 3 3 3
5
5
8
3 3
5
5
3 J
K Meanss g) Explanations
S 5
i 8
3
3
S 5
S
8
3
3
5
5 S
3
3
KMeans explanations are indistinguishable despite differences representation
RDX(M g, Ms)
M,
M
RDX(M R. Mg) Explanations
3
3 3
3 3 3
5
3
3
3
3 3 5
5
5
Image samples for RDX(M M2 concept
3
3 3
3 3
3
5
3s are closer together in Mg- A style of 3s that are confused by M, with
A style that Ms confuses with 3s.
Acc: 95%
Acc: 98%
(C) RDX(M M)) highlights inputs considered similar Mg, but not Ms-
Figure A1: Comparing RDX SAE and KMeans. In Fig. 2 we visualized NMF explanations for two model represenlations, from 'strong' Ms and 'expert" model Mr, trained MNIST-(3.5,8) Here we show explanations generated using SAE with maximum sampling and KMeans with centroid sampling. (A) SAE explanations are confusing and potentially misleading. SAE(Ms shows mostly 38 in all explanations, whereas SAE(ME) shows one explanation with mixed 5s and 8s, and two explanations with 5s and 8s respectively. These explanations do not convey which the two representations is weaker and may even suggest that the Mş is the expert representation. (B) KMeans explanations are indistinguishabu Given that these two reprosentations are highly similar, the controids for the clusters in both representations nearly the same. (C) RDX(Mg, Ms) shows explanations that helps us understand that Mg does a better job of grouping 3s and 5s than Ms, matching the known difference between the two models. The lack of an explanation for 8s suggest that Ms is relatively better at distinguishing 8s. We confirm this Appendix A.1.4.
23

---

s) Coeff.
NMF(M g)
SAE(Ms)
SAE(M g:
200
Image Index
400
600
5-
800
(1)
(2)
1100
1200
1400
Class
Label
Concept Index
Concept Index
Concept Index
Concept Index
(A) Maximum sampling for NMF and SAE hides complexities of concept vectors.
SAE(ME)
Explanation for Image 800
F
S
&
8
8
8
5 Image 800
3.68
5
&
+ 0.90
8
8
g
=
f
5
&
8
8
8
(B) Weighted combinations of maximally sampled explanations unintuitive.
Figure A2: Interpreting Dictionary Learning Concepts. We identify issues with dictionary learning methods for XAT that make them challenging to understand. These results are from the same experiment as Sec. 4.2 and Appendix A.T Maximum sampling to explain concept vectors hides important muances in model behavior. In (1) and (2), highlighted in red, we can see that Ms and Mæ both encode roughly the same concept, with maximum activations for images of 5s (indices 500-1000) and weaker activations for images of 3s and 8s (indices 0-500 and 1000-1500), The generated explanations for both concepts that the activations for 3s and 8s relatively lower in (2)
show 5s. However, than the activations for 3s and 8s in (1). These subtle nuances are critical for understanding how the two models behave differently, but are completely lost in the existing approaches for generating explanations. Thus, existing explanations for dictionary learning concepts are incomplete. (B) Analyzing single image through the lens of these concepts is extremely challenging. Users are tasked with using incomplete explanations of a concept in a weighted with un-intuitive coefficients. For example, image 800 is image of "standard" but is encoded by a weighted combination over concept of "strange" 5s and 8s and normal 8s. See Appendix A.1.2 for discussion the impact of monosemanticity and polysemanticity in this context.
24

---

A.1.3 Discussion of the Feasibility and Sufficiency of Monosemanticity
Importantly, the issues with polyscmanticity raises questions about the fcasibility and sufficiency of docomposing models into monosemantic concepts. Monoscmantic concepts are defined as concepts that have single, unambiguous mcaning and extracting them are the goal of sparse auloencoder methods for XAI g Ir 52 Consider a concept vector that encodes the concept of "roundness". This concept is neither monosemantic nor polysemantlic, as is too ambiguous for monosemanticityy, but not disparate enough be polysemantic. On dataset of objects that are interpolations from squares to circles, this concept would react to all round objects, but most strongly to circles. A maximally sampled explanation for this concept could easily mislead the user into believing that the concept reacts only for circles. Trying to convert the "roundness" concept into several discrete monosemantic concepts that only react to well-defined shapes lcads to questions about the boundaries of the discretization and the number of concepts that can be mcaningfully analyzed by a human. When comparing two models that share the "roundness" concept, differences in discretization could lead to partially overlapping concepts, such as those seen in [28].
When comparing two representations it is sometimes necessary to analyze specific images that the two representations disagree upon. When applying dictionary-learning based methods to understand what concepts make up image, users are required to mentally perform a weighted combination over incomplete concept explanations (Fig. A2 B). This task is un-intuitive and imprecise in the context of single model. If the concepts for the two models being compared are even slightly different, this task becomes essentially impossible. Notably, this issuc persists even if concepts are monoscmantic since most images are likely to contain several concepts.
A.1.4 Using RDX to Discover Concepts Specific to Me
RDX is not symmetric method. In Fig. A1 C visualize the second direction RDX(ME, Ms). These explanations reveal images considered similar in Mg, but not in Ms. These explanations show that the expert model ME is able to group challenging images of the same digit that Ms unable to. Additionally, we note that the explanations exclude 8, suggesting that the difference in similarities between images of 8 in Mg and Ms is smaller than the difference in similarities for images of 3 and 5. Indeed, the prediction agreement for linear classifiers trained those two reprosentations is 95% on 3, 95% 5, and 98% confirming our insight from the RDX explanations.
25

---

A.2 Additional Results for Recovering "Known" Differences
We describe the modifications to models in "known" difference comparisons in Table A8 Comparison details are found in Table A11
MNIST
In Fig. A3 visualize the explanations from RDX, KMeans, and NMF for M49 vs. Ma, M3s vs. M49, and Me vs. M
In all comparisons, RDX explanations clearly show the expected difference between the two representations. In contrast, KMeans and NMF generate unfocused explanations that are often indistinguishable from each other. At best, find that the baseline approaches may contain 2/6 explanations focused on the known difference between models.
CUB PCBM
In Fig. A4 and Fig. A5 we visualize five explanations for comparing CA-S vs. Ca and CA-YB vs. CA using RDX and basclinc method. CΛ is the CUB PCBM concopt vector with all concepts retained. CA-s removes the spotted wing concept from the concept vector and CA-YB removes the yellow back concept from the concept vector. We expect that explanations are composed of images that contain these concepts. In both comparisons, the baseline method (CNMF or SAE) generates indistinguishable and unfocused explanations that provide insight about the known differences. This limitation arises because the representational change induced by removing a single component from the concept vector is extremely subtle. As result, CNMF and SAE primarily capture the dominant conceptual directions within the representation spaces of both models and remain largely insensitive to these smaller, yet important, represcntational differences. On the other hand, RDX explanations focus on differences and can reveal intoresting insights about the impact of removing concept. In Fig. AA the RDX explanations help teach about how the PCBM uses the "yellow-back" " concept. On first glance, the model without the yellow-back concept (C^-Yn) appears to do better job of grouping colorful yellow /red birds. When inspected more closely, it becomes clear that there is mixture of birds with black faces and colored backs and birds with red/yellow faces and black backs. This indicates that the yellow-back concept is used as fine-graie -inenera discriminator between these two color patterns. It also indicates that the PCBM model may be suffering from leakage [9] and does not discriminate between bright red and yellow colors. In the other direction, we see that the ycllow-back concept helps organize birds with colorful backs into organized groups (explanations 3-5), but also helps organize "regular" birds into well-separated groups (explanations 1-2).
26

---

KMeans M 49)
NMF(M 49
RDX 49, MD)
3
3
6
6
3
3
6
6
6
4
9
9
4
3
5
9
1
6
6
3 3 3
8
8
6
1
1/1
9
4
9
4
9
4
4
9
3
5
4
/
6
6
6
3
3
6
8
6
4
4
4
4
4
4
KMeans MM) b)
NMF(M b
RDX M
M49
2
3
3
4 1 Y
Q
6 6
4
4
8
8
3
3
3
4
4
4
9
4
5
5
3
1
6
6
6
4
4
8
8
3
3
3
4
7
4
4
4
A
4
4
3
4
7
6
C-
4
4
8
3
3
3
4
4
4
KMeans M 35)
NMF M 35
RDX M
M 49)
35
5
5 3
Q 6
8
3 3
5 5
3
5
5
5
5
5
5
6 6
6
Y
2
4
4
6
8
3
33
5
5
3
3
5
5
3
5
5
S
6
6
6
2
4
4
6
3
3
3
5
3
3 3
3
3
3
KMeans 49)
NMF M
49
RDX M 49,
M
35
2
3 3
f 7 Y
6 6
3 3 3
6 6
6
4
/
3
3
4
9
9
9
4
3
5
2
7
6
6
-
3
3
6
6
1
/
3
3
4
4
4
3
5
4
4
6
6
6
3
3
3
6
8
6
/
/
3
3
3
9
9
4
4
KMeans M ..)
NMF(M
RDX(M
M M
7
6
3
5
3
8
0
S
Z
4
3
6
5
5
3
8
0
4
3 6
P
9
&
C
3
4
6
Q
4
&
8
8
666
3
3
KMeans (M_)
NMF(M )
RDX M ....
5 C
E
0 8
2 2 2
5
43
8
2
8
8
0
2
غ 5
5
7
1
&
9
3
5
2
3
8
0
8
2 2 2
& C
&
5
5
5
Figure A3: Additional explanations for MNIST comparisons. Modifications described in detail in Table A8. (Top) We compare M49 (mixes 4s and 9s) to M9 (no modifications). We observe that RDX generates explanations focused on the known difference. KMeans and NMF have 1/6 explanations related the known difference. (Middle) We compare M3. (mixes 38 and 5s) to M19 (mixes 4s and 9s). RDX conveys the modifications made to both models, specifically M35 mixes 3s and 5s and that M49 mixes 4s and 9s. Also, it shows that M49 organizes 3s better than Mas. KMeans has one explanation related to the known differences, while NMF has none. (Bottom) We compare Me, (mixed flipped and unflipped digits) to M_ (scparated flipped and unflipped digits). RDX reveals mixing between flipped and unflipped 6s, 9s and 3s in M and no mixing for 2s, flipped 3s and 5s in M_. KMeans explanations are confusing, NMF has 2/6 explanations that align with the known difference. See Appendix A.2 for discussion.
27

---

RDX(C Ct)
C CA-S)
CNMF CNMF(CA-8) (C
CNMF(C,)
Figure A4: Explanations for the "Spotted Wing" concept. We selected few explanation grids to show in Fig. Here we visualize all five explanations generated by two XAI methods, best viewed zoomed in. (Top) In row 1, the RDX explanation grids show birds with and without spotted wings mixed together. In row the explanation grids are predominantly made of birds with spotted wings. In explanation five, we see that RDX can generate clusters with too few images to generate a full grid. (Bottom) In both rows, each CNMF explanation grids shows a different kind of bird, unrelated to the known difference. For example, we can see concepts for yellow birds, seabirds, and black birds. The explanations for both representations are indistinguishable. See Appendix A.2 for discussion.
28

---

RD A-YB CA)
R CA-YB)
SAE(C A YYB
SAE(CA)
Figure A5: Explanation for the "Yellow Back" concept. We visualize explanations from comparing CA-YB CA. (Top) In row the RDX explanation grids reveal mixture of red and yellow birds. Closer inspection shows that these grids, especially explanation and 4, contain birds with yellow instance, in explanation 4, the central bird has a black back, and the lower-left bird has yellow back. In contrast, row 2 shows that CA does not mix birds with and without colored backs within the same concept. In explanations 1 2, CA isolates birds with black or gray backs into distinct concepts, while in explanations 3, groups multi-colored birds with tan, yellow, and red backs together. These patterns suggest that the "Yellow Back" concept may serve disambiguate birds with similar color distributions but differing spatial arrangements of those colors. (Bottom) In both rows, each SAE explanation grids shows concepts that correspond different bird types, unrelated the known difference. The explanations for both representations are also indistigguishable. See Appendix A.2 for discussion.
red backs intermixed with those having black backs. For
29

---

A.3
Additional Results for Discovering "Unknown" Differences
Table A1: Linear probe accuracy across datascts described in Table A11
Mittens
Primates
Maples
Corvids
DINO (Mp)
0.933
0.918
DINOv2 (MD2)
0.965
0.921
CLIP (Mc)
0.752
0.790
CLIP-iNat (McN)
0.796
0.787
We visualize complete RDX results for four comparisons using the representational alignment step from Sec. 3.4
1. MD vs. Mp on Mittens (Fig. A7
2. Mn vs. Mp on Primates (Fig. A8
3. Mc vs. McN on Maples (Fig.
A99
4. Mc McN Corvids (Fig. A10
See Table A11 for description of the datasets and Table A9 for details about the models. Although explaining classifier predictions is not the goal RDX, we train a linear classifier on these representations to gain an insight into the quality of their organization and to assist in interpretation (Table A1 Training details are provided in Appendix C.1 We use the classifier accuracies and predictions as supplemental information to understand representational differences.
In all representational difference comparisons, RDX reveals interesting insights about differences in model representations.
In the first two comparisons (Mittens and Primates), classification performance indicates that the representations well-organized and able to easily interpret the discovered concepts without using the dataset labels.
In the next two comparisons (Maples and Corvids), we evaluate how RDX performs when classifier performance is much lower. We expect these representations to be more poorly organized and subsequently more challenging to interpret. In the first comparison (Maples), we select a comparison where there a large performance difference (4%). the second (Corvids), explore setting in which fine-tuning CLIP iNaturalist images did not improve the quality the representation, although may have changed it.
Mittens
We find that DINOv2 does a better job of organizing mittons by their orientation (sce Fig. A77. It also has unique concept for children's mittens that is not present in DINO. On the other hand, DINO has unique concepts for children around Christmas related items and Christmas items on their own. These two concepts appear to be entangled in DINOVv2.
Primates
We notice that DINO contains four unique concepts that appear fixate on secondary characteristics (see Fig. A8). Explanation 1 seems to react to siamangs on grass, explanation 3 picks up on lower quality images, like screenshots from a video or images taken through enclosure glass, explanation 4 contains scenes of branches and greenery where primates are distant, and explanation contains scenes of varicty of primates behind fencing. contrast, DINOv2 explanations tend to be focused on the primate type. For example, explanation 1 also picks up on siamangs, but a variely environments, suggesting that DINOv2 may be less biased by background. Explanation 3 shows gibbons swinging in trees and explanation 5 shows spider monkeys in trees, these two concepts are entangled in DINO, suggesting DINO is more sensitive to the background, activity of the monkeys than the species.
30

---

Table A2: Ground truth label counts for explanations on Maples.
RDX Mc,
M
RDX McN- Mc)
Maple Type
E1
E2
F3
E4
E5
F1
F2
E3
F4
F5
Norway Maple
1
2
0
9
0
0
0
Silver Maple
3
1
1
0
0
7
0
Sugar Maple
3
4
0
0
0
Red Maple
2
0
9
Maples
Fino-grained maple tree classification is a challenging task which is beyond the skill most people. The clear performance gap between model's indicates that CLIP-iNat has learned important feat ures, we explore if RDX is able to help us generate hypotheses on what those might be (see Fig. A9 In Fig- This allowed us to hypothesize that CLIP was biased towards encoding Maple leaves by color/season rather than species. Despite this bias, find that classifiers trained on both representations perform reasonably well on these two image grids. This suggests that this reprosentational difference may be important for classification. However, we notice that there are differences in the classifier predictions for explanations 2 and 3. We use proposc some hypotheses. The dataset labels for explanation indicate
analyzed explanations and from RDX(McM My).
this information that all of the images are sugar maples. CLIP-iNat gets 7/9 correct, while CLIP gets 5/9. With this information, we hypothesize that CLIP-iNat is able to detect sugar maples leaves in images with variely of seasons, backgrounds, and lighting. In explanation 3, we see young, bushy maple trees around rocks and waterways. The classification labels indicate the majority of these images contain silver maples. CLIP-iNat gets 8/9 correct, while CLIP gets only 4/9 correct. This suggests that CLIP-iNat has learned to associate this visual concept with silver maples and that this is effective strategy for classification on this dataset. We also observe some higher-level charactoristics of the explanations when analyzed with their ground truth labels. Explanations for CLIP-iNat tend to have labels that correspond with one of the ground truth labels, while CLIP does not (Table A2). This make sense, as CLIP-iNat was fine-tuned a classification dataset.
Corvids
Neither representation supports good classification on the challenging Corvids dataset (sce Fig. A11)) However, RDX reveals some interesting concepts unique to each model. For example, RDX RDX(Mc, Mc,'''x) explanation 4, shows CLIP specific concept for Corvid footprints. In explanation 5, we see concept. for flocks of crows. In the other direction, CLIP-iNat has a learned a concept for large ravens in natural settings like hillsides or beaches (explanation 1). It has also learned concept for crows in urban settings like schools, fields and pavement (explanation 2). Additionally, CLIP-iNat makes a stronger distinction between perched crows in urban settings (explanation 4) and flying crows (explanation 5).
Effects of Alignment.
In Fig. A6 we visualize the effect of alignment when comparing CLIP to CLIP-iNat on the Maples datasct. We see that alignment can result in significant change in the spectral clusters detected from the affinity matrix. In particular, one discovered concept is only present in the unaligned comparison. This indicates that both representations contain the information to represent the concept, but their initial configurations differ. After alignment, the concepts discovered are more likely to be fundamental differences between the two representations. Although there are some limitations this interpretation (see Sec. 5 we focus aligned comparisons in qualitative plots, as it makes interpretation simpler.
31

---

Maples (iNaturalist)
RDX(M C. (NN)
RDX(Mc. MCN)
RDX(M c. MCCN)
Explanation
Mc
MNN
Mc
MCN
Unaligned
Aligned
Figure A6: Effect of Alignment (Appendix A.3). (Top) We compare CLIP and CLIP-iNat with and without aligning CLIP-iNat (denoted by M' notation). We visualize PCA plots of the representations with cluster membership (light colors) and samples (dark colors). In each direction, the left plot contains the concept "source" representation while the right plot has the selected clusters overlayed on its representation. (Unaligned, Left) We generate five spectral clusters, we can see that they group nicely in the left plot and are spread apart in the right plot. We highlight the region of the red cluster for comparison after alignment. (Aligned, Right) After aligument, we can see that is more difficull to get clusters that have large differences in their distances in the two representations. We find that the region that the red cluster came from the unaligned comparison no longer selected after alignment. (Explanation) The red cluster (unaligned) contains maples in fall foliage. Both networks can represent this concept, although the unaligned CLIP-iNat representation does not prioritize it.
ChatGPT-40 Analysis
Analyzing and annotating several explanations for each model is time consuming and cognitively demanding. We explore if ChatGPT-4o [ is capable of annotating the images for us in the Maples and Corvids comparisons. We use the prompt:
Prompt
am going to ask you to analyze image grids. You will receive strip of five image grids. The images in the grid will be from the categories: <category list>, Your task is to concisely describe the consistent features that appear in each image grid. You do need to use complete sentences. The format of your output should be dictionary like this. E1: descl, E2: desc2, E3: desc3, E4: deso4, E5: desc5.
The outputs are provided in the figure captions of Fig. A9 and Fig. annotations clear, reasonable and helpful.
A10
We find that the
32

---

RDX(M D2, MD) PCA
RDX(M D. M2) PCA
M D2
D
MD
D2
RDX( bb)
Explanations
RDX(M D. M D2) Explanations
Figure A7: Investigating DINOv2 vs. DINO on Mittens (aligned). We visualize RDX difference explanations in both directions on the Mittens dataset (Table A11). This dataset contains images of mittens, socks and Christ stocking from ImageNet 110 For example, RDX D2, M generates explanations for concepts that are in M (DINOv2).
but not MD (aligned DINO). We refer to the explanations as E1 to E5 (left to right). (Top) PCA plots of the representations with cluster membership (light colors) and samples (dark colors). In each direction, the left plot contains the concept "source" representation while the right plot has the selected clusters overlayed generally appear better organized than in the right plot. (RDX(Mp2) Mt))) E1: crocheted socks, E2: horizontal mittens, E3: vertical pairs of mittens, E4: crocheted mittens, E5: children's mittens. (RDX (DD Mb2)) E1: multi-colored wool socks, F2: assorted pairs of mittens, E3: children with Christmas decorations. F4: Christmas paraphernalia. mittens, and E5: woolen clothes being worn. See Appendix for interpretation.
its representation. Clusters on the left plot
33

---

RDX D2. M) PCA
RDX(M D. M2) PCA
M D2
MD
M D
M D2
Expi:
Expl
Expi
Expl
Engi:
Expi
RDX(M D2, MD)) Explanations
RDX(M, D. D2) Explanations
Figure A8: Investigating DINOv2 vs. DINO Primates (aligned). We visualize RDX difference explanations in both directions on the Primates dataset (Table A11). This dataset contains images of gibbons, siamangs and spider monkeys from ImageNet [10 For example, RDX M generates explanations for concepts that are in M (DINOv2), but not MD (aligned DINO). We refer the explanations E1 to E5 (left right). (Top)
PCA plots of the representations with cluster membership (light colors) and samples (dark colors). In each direction, the left plot contains the concept "source" representation while the right plot has the selected clusters overlayed on its representation. Clusters on the left plot generally appear better organized than in the right plot. (M2 Mb)) E1: black siamangs in diverse environments, E2: tan gibbons, E3: orangutans and gibbons playing, E4: whito-chocked gibbons, and E5: spider monkeys. (RD,(M,, Mb2)) E1: siamangs laying in grass, E2: gibbons swinging, E3: lower resolution images of primates, viewed through glass or from videos, E4: tree environment with distant primales, and E5: assorted primates behind fencing. See Appendix A.3 for interpretation.
34

---

RDX CN. Mc)) PCA
RDX(M C₁ McCN) PCA
McN
M'c
Mc
Eark
M McC
Expl.
RDX(M CN, M'²) Explanations
RDX(M C, MCCN) Explanations
Figure A9: Investigating CLIP-iNat vs. CLIP on Maples (aligned). We visualize RDX difference explanations in both directions the Maples dataset (Table All). This dataset contains images red maples, sugar maples, Norway maples, and silver maples from iNaturalist 24. RDX(McN (MXMM),Mec) generates explanations for concepts that are in Mcn (CLIP-iNat), but not M (aligned CLIP). We refer to the explanations as E1 to E5 (left to right). (Top) PCA plots of the representations with cluster membership (light colors) and samples (dark colors). In each direction, the left plot contains the concept "source" representation while the right plot has the selected clusters overlayed its representation. Clusters on the left plot generally appear better organized than the right plot.
These types of maples have subtle differences beyond the expertise of most people so we ChatGPT-40 to generate descriptions. (RDX Mc)) El: "Large, dark green, sharply lobed leaves; smooth surface; some handheld, often against tree bark or forest background", E2: "Varied color (green, red, yellow), symmetric lobes with central point, often single leaves photographed on flat surfaces", E3: "Small clusters of light green to reddish leaves, forest floor or rocky environment, less prominent lobes", E4: "Bright red leaves, often handheld, five lobes with narrow points, smooth margins, clear vein structure", and E5: "Yellow mottled leaves, some black spotting, thick lobes, visible veins, photos taken in autumn light against tree bark". (RDX(Mc, M E1: "Leaves with deep sinuses, bright green, flat edges, consistent lighting, often low ground or with visible bark", E2: "Yellow-gree green foliage, broad flat leaves with few teeth, tree clusters with hanging leaves, slight curl", E3: "Five-lobed leaves, medium green, toothoo edges, spread flat, some variation in lighting and angle", E4: "Red spring buds and samaras, visible, bare branches, sky background, some birds", and E5: "Light green leaves with coarsely toothed edges, translucent lighting, some purplish tinge in parts, lobed leaves". See Appendix A.3 for interpretation.
full leaves
35

---

RDX(M CN. M2 PCA
RDX(M c. MC'N) PCA
M'c
M
MCCN
M
CN
Cepl.
RDX(M CN. Mcc) Explanations
RDX(M c, McCN) Explanations
Figure A10: Investigating CLIP-iNat vs. CLIP on Corvids (aligned). We visualize RDX difference explanations in both directions on the Corvids dataset (Table ATl). This dataset contains images of crows and ravens from iNaturalist 124. For example, RDX(McN, Mcxx) generates explanations for concepts that are in McN (CLIP-iNat), but not My (aligned CLIP). We refer to the explanations as E1 to E5 (left to right). (Top) PCA plots of the representations with cluster membership (light. colors) and samples (dark colors). In each direction, the left plot contains the concept. "source" representation while the right plot has the selected clusters overlayed on its representation. Clusters on the left plot generally appear better organized than in the right plot.
These types of corvids have subtle differences beyond the expertise of most people so we use ChatGPT-40 to generate descriptions. (RDX(McN. M()) El: 'Birds in arid rocky environments; perched flying: often alone in small groups; slimmer builds; modium size; matte black feathers", E2: *Urban and suburban settings; birds near buildings, fences, and pavement; typically foraging: in pairs groups;
compact build", E3: Close-up or detailed views of large, shaggy birds; prominent beaks and throat hackles; perched interacting with environment", and E4: 'Birds flying in sky; high contrast silhouettes; open sky backgrounds; wing shapes and flight patterns emphasized", E5: Biidss with other wildlife (e.g. bear, eagle); perched alone with others; prominent size; thick bills and throat feathers". (RDX Mc. M) El: "Birds in wooded or forested environments; perched branches; medium size; matte black feathers; mostly solitary or in pairs", 152; "Birds on open branches or tall perches; slightly larger size; thick beaks; prominent neck feathers (hackles); more upright posture", E3: 'Birds on ground urban/part environments: sparse trees; usually in small groups; foraging walking", E4: 'Footprints in mud, sand, or snow; distinct three-toed tracks; measurement tools in several images; variable substrate", and E5: "Flocks of birds flying or perched in large groups; sky or treetops visible; misty or open-air environments". See Appendix A.3 for interpretation.
36

---

A.4 Additional Metrics and Detailed Results for Comparison Experiments
In addition to our BSR metric, we compute the Redundancy, Clarity, and Polysemanticity metrics from SemanticLens [2] These metrics use a model, OpenCLIP 23] to evaluate the explanation grids for each concept. Details on how each metric is computed are provided in Appendix B.2 We modify the Redundancy metric from the original work [1], such that it designed to compare the redundancy of concepts across models. Lower redundancy indicates that the concepts discovered for each model are less similar, while higher redundancy indicates that the concepts for each model are more similar. The results are presented in Tables A3 A4 A5 and A6 In Appendix C we provide more details on the models (Table A8) and datasets (Table A used in each comparison. Comparisons are written in a single direction, but results are reported for both directions. The is used to indicate experiments conducted after an aligument step has been performed.
RDX is designed to isolate and visualize concepts unique to each model in a comparison. Thus, we expect and find that it performs the best on the BSR and Redundancy metrics (Tables A3 and [n4 However, isolating model differences may also result in highlighting parts of the representation that are more likely to be complex (higher polysemanticity) and less clear (lower clarity). Therefore, it is not surprising that RDX does not have an advantage on either of these metrics (Tables A5 and A although it performs comparably to other bascline methods.
Additionally, there are some caveats to interpreting the SemantieLens metrics. These metries assume that the generalist model (OpenCLIP) is an unbiased evaluator, but it is likely that OpenCLIP has its own biases that will affect each method's scores. This is especially true when generating explanations for complex, niche datasets that are out of domain for OpenCLIP. We can see this bias in the MNIST dataset comparisons under the redundancy metric (Table A4 first five rows). Despite the RDX concepts for each model being semantically different (Fig. A3 OpenCLIP incapable of distinguishing the nuanced differences that are plainly visible to human observers, resulting in high redundancy scores for all methods. benefit of our BSR metric (Table A3 first five rows) that it does not depend on a separate generalist model and better matches the qualitative performance differences observed in Fig.
Table A3: BSR results across all comparison experiments. The best (highest) mean value per row is bolded (ties bolded). RDX has the best scores on the BSR metric.
Comparison
RDX
KMeans
TKSAE
NMF
SAE
NLMCD
Ms vs. M.
0.83 1 0.03
0.62 1 0.00
0.61 + 0.05
0.58 1 0.06
0.57 1 0.06
0.52 1 0.03
Ma vs. M.
0.86 + 0.02
0.67 : 0.03
0.45 : 0.05
0.57 + 0.03
0.56
+ 0.04
0.52 +\200 0.08
Ma vs. Mw
0.87 + 0.00
0.68 = 0.01
0.65 = 0.02
0.50
- 0.04
0.56
0.05
0.56 = 0.06
M.. M
0.90 0.05
0.79 0.06
0.69 = 0.03
0.59 0.07
0.51
0.04
0.57 = 0.02
M, va. M
0.91 + 0.04
0.72 + 0.07
0.56 + 0.13
0.72 0.02
0.69 + 0.04
0.54 + 0.19
CA Ca
0.71 + 0.12
0.30 = 0.00
0.32 = 0.14
0.38 = 0.11
0.38 4 0.19
0.27 1 0.05
CA-YB vs. CA
0.66 + 0.08
0.21 = 0.05
0.33 + 0.11
0.38 - 0.10
0.37 + 0.08
0.25 - 0.05
CA-YO vs. CiA
0.64 + 0.13
0.30 + 0.00
0.36 + 0.18
0.41 + 0.13
0.38 + 0.09
0.32 + 0.15
CA E vs. Ca
0.69 0.11
0.21 + 0.01
+ 0.16
0.38 + 0.12
0.32 0.06
0.39 + 0.15
CA-D CA
0.68 T 0.12
0.21 0.07
0.35 1 0.06
0.40 0.01
0.32 0.12
0.26 T 0.11
MD vs. MD2 (MT)
0.92 + 0.01
0.88
0.01
0.76 + 0.02
0.68 + 0.05
0.59 + 0.04
0.53 + 0.11
Mo vs. Mp (B)
0.88 + 0.02
0.87 + 0.07
0.66 + 0.07
0.78 + 0.06
0.56 + 0.07
0.50 + 0.04
MD vs. MD2 (D)
0.93 1 0.04
080 1 0.02
0.71 L 0.05
0.66 1 0.01
0.48 1 0.03
0.55 T 0.14
MD vs. Mo (P)
0.92 1 0.03
0.79 1 0.02
0.74 + 0.03
0.76 1 0.05
0.55 0.01
0.42 1 0.14
Mc vs. Mcx (G)
0.97 + 0.01
0.93
0.02
+ 0.08
0.82 0.06
0.71 0.0T
0.51 + 0.03
Mc vs. Mes (C)
0.93 1 0.03
0.86
0.03
0.76 1 0.07
0.75 1 0.05
0.65 1: 0.01
0.53 1 0.09
Mc vs. Mcn (MP)
0.98 1 0.01
0.95 1 0.02
0.87 1 0.03
0.80 1 0.07
0.64 1: 0.05
0.46 T 0.23
MD vs. Mp (MT)
0.90 + 0.01
0.56 + 0.05
0.46 + 0.09
0.49 + 0.06
0.45 + 0.12
0.52 + 0L.7
MD vs. Mp (B)
0.92 1 0.01
0.58 1 0.07
0.46 + 0.05
0.47 1 0.01
0.40 1 0.02
0.45 1 0.11
MD Vs. Mp (D)
0.88 + 0.04
0.52 - 0.01
0.35 + 0.02
0.42 + 0.06
0.43 + 0.02
0.52 - 0.03
MD VS. My (P)
0.90 0.04
0.52 0.09
0.45 + 0.07
0.41 + 0.00
0.44 0.01
0.49 + 0.10
Mc V3. McN (G)
0.94 1 0.03
0.50
0.37 + 0.09
0.39 0.13
0.46 0.02
0.52 1 0.21
Mc v3. M (C)
0.89 + 0.02
0.51 = 0.02
0.28 = 0.12
0.29 - 0.02
0.39 + 0.04
0.53 - 0.07
Mc VS. MON (MP)
0.95 1 0.01
0.58 1 0.09
0.31 1 0.07
0.21 1 0.02
0.30 1 0.00
0.50 1 0.19
Average
0.86 1 0.12
0.61 - 0.23
0.52 - 0.20
0.54 4 0.18
0.49 4 0.14
0.47 1 0.15
37

---

Table A4: Concept Redundancy results across all comparison experiments. We
compute the Redundancy metric from SemanticLens [2 row is bolded (ties bolded). RDX has the best scores the Redundancy metric.
The best (lowest) mean value per
Comparison
RDX
KMeans
TKSAE
NMF
SAE
NLMCD
Mss vs. M.
0.99 + 0.00
0.99 = 0.00
0.99 = 0.00
0.99 + 0.00
0.98 = 0.00
1.00 + 0.00
Ms M.
0.99 0.00
0.98 0.00
0.99 = 0.01
0.95
0.97 + 0.00
+ 0.00
Ms. M
0.96 + 0.02
0.98 + 0.00
0.99 = 0.00
0.96 + 0.00
0.99 + 0.00
0.95
+: 0.00
M.. vs. M-
0.95 + 0.00
0.99 + 0.00
0.98 = 0.00
0.98 = 0.00
0.99 + 0.00
0.99 + 0.00
M₁ vs. M
0.97 + 0.01
0.98 + 0.00
0.99 + 0.00
0.97 0.00
0.98 + 0.00
0.99 + 0.00
Ca-8 vs. CA
0.87 + 0.01
0.99
0.00
0.86 0.00
0.98 + 0.00
0.92 + 0.01
+ 0.00
CA-YB vs. CA
0.87 + 0,02
1,00
0.00
0.89 0.00
0.98 + 0.00
0.90 + 0,01
0.90 + 0,00
CA YC vs. CA
0.81 + 0.02
0.99 + 0.00
0.87 = 0.00
0.98 + 0.00
0.90 + 0.00
0.89 + 0.00
CA-E vs. Ca
0.84 + 0.01
1.00 + 0.00
0.88 - 0.01
0.98 + 0.00
0.92 + 0.00
0.91 E 0.00
CA-D vs. Ca
0.80 + 0.05
1.00 + 0.00
0.85 = 0.00
0,98 + 0.00
0,90 + 0.00
0.91 + 0.00
MD Maa (MT)
0.80 0.02
0.82
0.78 = 0,00
0.73 0.02
0.81 = 0,00
0,80 = 0,02
MD vs. Mp2 (B)
0.81 + 0.02
0.86 + 0.00
0.86 = 0.01
0.84 + 0.02
0.80 0.00
0.91 F 0,00
Mp vs. Mp2 (D)
0.76 + 0.00
0.86 + 0.01
0.80 F 0.01
0.84 + 0.00
0.85 + 0.00
0.89 # 0.01
MD vs. Ma2 (P)
0.76 + 0.05
0.87 = 0.00
0.86 = T,00
0.80 + 0.00
0.85 + 0.00
0.89 # 0.00
Mc Vs. McN (G)
0.84 0.02
0.84
0.81
0.85 = 0.02
0.84 = 0.00
0.88 = 0.00
Mc VS. MON (C)
0.74 + 0.02
0.81 0.02
0.73 0.08
0.85 + 0.01
0.79 + 0,01
0.86 + 0.00
Mc VS. McN (MP)
0.82 + 0.02
0.88 = 0.01
0.82 = 0.08
0.84 + 0.02
0.84 F 0.01
0.89 H 0.00
My vs. Ma (MT)
0.79 + 0.04
0.83 = 0.00
0.80 = 0.01
0.81 + 0.00
0.84 + 0.02
0.84 H 0.00
MD VS. Mya (B)
0.83 + 0.02
0.87 1 0.02
0.83 = 0.00
0.84 + 0.02
0,85 + 0.01
0.85 F 0.00
MD VS. M2 (D)
0.79 + 0.00
0,89
0.82 = 0,02
0.84 = 0.02
0.87 = 0,00
0.89 - 0,00
MD v3. Ma2 (P)
0,80 + 0.03
0.87 = 0.01
0.88 = 0.00
0.85 = 0.01
0.85 = 0,01
0.88+4 + 0.01
Mc vs. M. (G)
0.82 + 0.01
0.84 + 0.01
0.83 = 0.01
0.87 + 0.00
0.86 + 0.01 0.01
0.87 # 0.00
Mc Vs. M (C)
0.80 + 0.00
0,81 + 0.06
0.84 + 0.03
0.81 + 0.02
0,80 + 0.00
0.00
5810 + 0.01
Mc Vs. McN (MP)
0.78 1 0.04
0.88 4 0.06
0.83 = 0.02
0.84 + 0.06
0.81 + 0.01
0.89- F 0.00
Average
0.84 1 0.07
0,90
0.07
0.87 = 0.06
0.89 1 0.07
0.87 1 0.06
0.90 1 0.05
Table A5: Concept Clarity results across all comparison experiments. We compute The best (highest) mean value per row is bolded (ties bolded). No method has clear advantage on the Clarity metric.
the Clarity metric from SemanticLens 12.
Comparison
RDX
KMeans
TKSAE
NMF
SAE
NLMCD
M55 vs. M,
0.94 0.04
U.93 +: 0.06
0.94 0.03
0.94
+ 0.04
0.90 + 0.08
0.99 + 0.01
Me vs. M.
0.96 0.03
0.93
0.05
0.91 = 0.06
0.91
1 0.06
0.94 1 0.09
0.99
: 0.01
Mss M1
0.95 0.02
0.94
0.05
0.95 0.04
0.92 + 0.05
0.92 1 0.02
0.97 0.03
M., vs. M=
0.92 0.06
0.88
+ 0.05
0.87 0.04
0.91 + 0.05
0.91 + 0.04
0.89 + 0.04
M. vs. M11
0.94 0.03
0.80 + 0.05
0.89 0.02
0.88 + 0.04
0.89 1 0.04
0.92 + 0.05
Ca 8 vs. CA
0.43 0.04
0.45
0.04
0.44 0.03
0.11 + 0.03
0.45 1 0.06
0.43 + 0.06
CA-YB vs. CA
0.43 0.06
0.45
0.04
0.42 0.03
0.42 1 0.03
0.44 1 0.06
0.45 0.09
CA-YO vs. CA
0.48 0.06
0.45 +: 0.04
0.43 0.03
0.41 + 0.03
0.43 1 0.05
0.45 + 0.04
CA B vs. CA
0.43 + 0.02
0.44 + 0.04
0.42 0.02
0.41 + 0.02
0.45 + 0.06
0.45 + 0.06
CA-D vs. Ca
0.46 0.08
0.45 0.04
0.44 0.03
0.42 + 0.03
0.44 1 0.05
0.44 + 0.08
MD vs. Mpz (MT)
0.40 0.06
0.42 0.04
0.38
0.05
0,1
0.36 1 0.04
0.39 = 0.06
MD vs. Mp2 (B)
0.49 + 0.05
0.50
+ 0.0G
0.50 + 0.04
0.48 1 0.08
0.14 - 0.05
0.45 + 0.05
MD vs. Mp2 (D)
0.45 + 0.05
0.50
1 0.05
0.45 0.07
0.50 + 0.06
0.42 1 0.06
0.43 F 0.04
Mu vs. Mo2 (P)
0.48 = 0.12
0.47 0.06
0.42 = 0.08
0 + 0.12
0.40 1 0.05
0.45 F 0.09
Mc Vs. McN (G)
0.41 0.06
0.41
0.04
0 0.04
0.42 0.04
0.36 = 0.07
0.39 = 0,04
Mc vs. MoN (C)
0.43 + 0.11
0.46 1 0.11
0.45 0.12
0.46 + 0.08
0.39 1 0.09
0.41 F 0.07
Mc VS. MeN (MP)
0.46 + 0.09
0.50 1 0.08
0.49 0.06
0,7 + 0.07
0.43 1 0.09
0.44 F 0.07
MD VS. M (MT)
0.40 0.04
0.38 0.03
0.42 0.06
0,1 + 0.0G
0.37 1 0.03
0.38 = 0.04
MD vS. M (B)
0.44 0.05
0.47
0.49 0.05
0.48
0.46 =
0.43 = 0.0G
MD vs. Mp2 (D)
0.44 + 0.04
0.48 + 0.06
0.48 0.06
0.50 0.07
0.48 + 0.06
0.42 = 0.03
MD vs. M (P)
0.46 + 0.07
0.43 + 0.05
0.48 + 0.04
0.48 + 0.09
0.42 + 0.05
0.39 + 0.06
Mc Vs. MON (G)
0.40 1 0.06
0.38 4 0.03
0.41 0.02
0.45 + 0.06
0.39 1 0.06
0.36 1 0.03
Mc VS. MèN (C)
0.41 0.05
0.45 1: 0.10
0.42 0.09
0.45
0.11
0,40 1 0.10
0.38 + 0.06
Mc VS. MON (MP)
0.49 0.08
0.52
0.08
0.48 0.06
0.50 1
0.40 2 0.07
0.38 - 0.06
Average
0.55 0.21
0.53 + 0.17
0.52 0.18
0.52 + 0.18
0.49 - 0.19
0.51 - 0.20
38

---

Table A6: Concept Polysemanticity results across all comparison experiments. We compute the Polysemanticity metric from SemanticLens [2]. The best (lowest) mean value per row bolded (ties bolded). No method has clear advantage on the Polysemanticity
metric.
Comparison
RDX
KMeans
TKSAE
NMF
SAE
NLMCD
Mss M,
0.08
0.07
0.08
Mr M.
0.09 0.07
0.08
+: 0.09
0.06 0.04
0.14 + 0.15
0.09 + 0.14
0.00 + 0.00
Mas vs. M
0.07 + 0.04
0.07 + 0.09
0.06
0.04
0.11 + 0.13
0.09
= 0.07
0.04 = 0.07
M vs. M
0.06 0.05
0.11
+ 0.04
0.12
0.05
0.12 + 0.07
0.11
+ 0.07
0.09 + 0.08
M. M11
0.11
0.10
0.10
0.08
CA-S CA
0.38 0.06
0.34
+ 0.05
0.38
0.07
0.42 + 0.06
0.35 = 0.08
0.37 = 0.09
CA-YB vs. CA
0.40 + 0.13
0.34 + 0.05
0.40
0.06
0.40 + 0.07
0.39 + 0.06
0.34 + 0.10
CA-YC vs. Ca
0.33 = 0.10
U.35
0.05
0.40
0.09
0.38 + 0.06
0.36 + 0.06
0.34 + 0.07
CA-E
0.38
CA-D CA
0.27 0.10
0.35
0.04
0.39
0.08
0.39
0.06
0.38
0.37 + 0.12
Mo vs. Mon (MT)
0.41 + 0.06
0.39 + 0.07
0.45 0.09
0.40 + 0.07
0.41 1 0.06
0.38 + 0.10
My vs. Mo2 (B)
0.30 0.09
0.29 + 0.08
0.29 0.05
0.32 + 0.09
0.37 = 0.09
0.33 H 0.07
vs.
(D)
0.36
0.32
+ 0.45
0.36 + 0.07
MD Mon (P)
0.32 0.10
0.30
0.07
0.41
0.34 1
0.39 H 0.12
Mc vs. MoN (G)
0.38 + 0.08
0.41 + 0.06
0.37 + 0.0T
0.37 + 0.09
0.48 - 0.10
0.43 # 0.10
Mc vs. MON (C))
0.41 0.12
0.39 + 0.12
0.38 0.11
0.38 0.72
0.42 + 0.10
0.43 + 0.08
vs. McN (MP)
0.39 0.12
0.34
0.12
0.34 0.09
0.40 1 0.09
E 0.10
MD M (MT)
0.39
0.40
0.38 0.10
0.39
0.41 = 0.07
MD VS. M (B)
0.35 + 0.06
0.30 + 0.0G
0.36 0.07
0.33 + 0.09
0.34 = 0.08
0.36 = 0.06
MD vS. M (D)
0.36 + 0.09
0.32 + 0.08
0.32 = 0.07
0.34 + 0.09
0.36 = 0.10
0.39 = 0.09
MD vS. M (P)
0.33 0.06
0.38
0.08
0.33 0.09
0.34 + 0.09
0.37 + 0.08
0.44 + 0.14
Mc vs. MON (G)
0.43
+ 0.44 0.11
+
Mc MON (C)
0.44 0.09
0.39
0.38 0.11
0.38 0.18
0.51 0.15
Mc VS. MΦN (MP)
0.33 1 0.11
0.31
1 0.13
0.33 0.06
0.30
0.07
0.43 1 0.08
0.43 1 0.12
Average
0.31 0.12
0.31 1 0.11
0.33 1.11
0.33 4 0.11
0.35 - 0.12
0.34 - 0.13
39

---

Cluster Disagreement
0.4
0.3
0.2
0.1
0.0
RDX
KMeans
TKSAE
CNMF
NLMCD
SAE
PCA
Figure A11: Cluster Disagreement. We report how much concepts change under in- distribution datasct perturbations (sce Appendix A.5 On the y-axis plot the cluster disagreement, which measures how much clusters disagrce under in-distribution perturbations of the dataset. Lower cluster disagrecment indicates a higher concept consistency. RDX has fairly low disagreement, indicating it generates reasonably consistent concepts under datasel perturbations.
A.5 Concept Consistency Under In-Distribution Dataset Perturbations
Our aim in this section is to test whether concepts remain the same when there in- distribution changes to the data. While concopt shifts are expected when the distribution changes, a desirable property of concept extraction methods is consistency when the imputs are changed, but the underlying distribution remains the same, e.g., uniformly adding or removing images from a class to if it is has any impact on the identified concept differences.
To test this, we four ImageNet-derrieed base datasets: Primates, Mittens, Buses, and Dogs (details in Table All.. y. For each base dataset, we create two in-distribution variants by (i) uniformly removing 20% of datapoints (-20%) and (ii) uniformly adding 20% more datapoints sampled from the same classes as the base datasct (+20%). This yields 4 x 3 = 12 datasets: (base, -20%, +20%} for each of the four bases. On cach of the 12 datasets, extract concepts for the DINO G vs. DINOv2 43 comparison using seven methods: RDX, KMeans, TKSAE, CNMF, NLMCD, SAE, and PCA. For the present analysis, however, our focus is nol on cross-model differences; instead, we assess how consistent the extracted concepts are within a given base dataset across its in-distribution variants (base, -00% +20%).
To measure consistency, we would like to compare how similar concepts are across the different dataset variants. We represent each concept via a set of images that contain it. Specifically, we convert each method's output into a hard clustering over images, where each cluster corresponds to concept. For RDX, KMeans, and NLMCD we their native hard cluster assignments. For TKSAE, CNMF, SAE and PCA, which yicld soft assignments, we assign each image to the concept with the largest coefficient (argmax
concept activations).
For each base dataset, we compare clusterings across the three in-distribution variants using the pairs: (base, -20%), (base, +20%), and (-20%, +20%). To make the comparisons feasible, we restrict them to the intersection of images that are present in all three variants; after filtering, each dataset contains 1200 images. We align clusters between dataset variants using the Hungarian algorithm B1 and report the cluster disagreement: the fraction of images whose assigned clusters disagree after alignment. Lower values indicate higher concept consistency.
Fig. A1l summarizes the disagreement across all bases and pairs. SAE shows the highest consistency, exhibiting almost no disagreement across dataset variations. RDX performs well, maintaining low disagreement under in-distribution perturbations. In contrast, NLMCD, TKSAE and CNMF display larger disagreement scores, indicating less stability. KMeans and PCA remain relatively consistent as well.
40

---

A.6 Discussion of Euclidean Distances
Euclidean distances can sometimes be misleading in neural network representations (see Sec. 5p. In this section discuss why RDX can still work despite this issue.
Many models have meaningful Euclidean distances in the final layer. Many modern self-supervised models, like DINOv2 [3]] have meaningful Euclidean distances in the latent space as seen by their performance on KNN probing and linear probing (Table 4 in [3). Additionally, the training recipe for most classification models is likely to result in relatively meaningful Euclidean distances. Classification models are commonly trained with final linear layer and a cross-entropy loss. The final layer outputs logits that are converted to probabilities using function like softmax. These logits are not guaranteed to have meaningful Euclidean distances, since scaling and translation do not affect the softmax function, however, in order to minimize the cross-entropy loss they must have meaningful relative Fuclidean distances. In experiments, we the final layer in all experiments. Impact of using the nearest neighbors ranking. It is unlikely that the nearest neighbors for given point have small Euclidean distances, but large distances on the manifold, since we generally expect that representations are locally linear. Neighborhood distances and our locally-biased difference function push RDX to seek out groups of points that locally linear. Notably, the non-linear dimensionality reduction technique Isomap [61 uses top-k nearest neighbors construct graph and estimate distances along the manifold of the data. Our locally-
locally-biased biased difference function is soft version of the same idea.
B Additional Methods Description
y 0.01
y = 0.05
y 0.1
1400
1400
1400
0.75
0.50
1050
1050
1050
0.25
700
(350, 700)
350
350
350)
0050
-075
0
350 700
1050
1400
350
700
1050
1400
0
350 700
1050
1400
-1.00
D
D
Figure A12: Locally Biased Difference Function. Here visualize the locally biased difference function described in Sec. 3.2 On the x-axis we plot D% which is the normalized distance A. On the y-axis we plot Dz Each panel corresponds
different y value. We analyze the = 0.1 sub-panel in more detail. When 1 is smaller than the distance in DJ the difference value is negative, indicating by darker colors in the top left triangle half. However, we can see that the same relative change does not result in the same difference value. We mark two points in red on the plot. Both points represent the same amount of change when comparing the distance matrices, D D 350. However, the difference functions considers the change from to 350 as more important (returns -1), whereas the change from 350 to 750 is less important and (returns around -0.25). Thus, the difference function reacts most strongly when highly similar items are considered less similar in B. The y value adjusts how quickly the function saturates.
B.1 K-neighborhood Affinity (KNA) Pseudocode
Let FA E RRR denote the full affinily matrix computed between all image pairs using representations A and B. For a given spectral cluster C. S (1....,n) ..). we extract the submatrix A.B gexr, where [C), by selecting only the rows and columns of FA.B
41

---

corresponding to the indices in C. This subset of the affinity matrix FA.B captures pairwise affinities within the cluster and serves as input to the KNA-based selection procedure.
Algorithm 1 Selecting Image and Neighbors with Maximum KNA
Require: Cluster == (2; Fr). Subset of affinity matrix FC
A.B RRrxr Number of
neighbors k
Ensurc: Image zmax and its k-nearest neighbors Nmax
1: for cach image in C' do
2: N; f- indices of the & largest values row :[i,
3: KNA(x;) t EJEN, A
4: end for
5: Tmax + arg maxx, KN)(2))
6: Niax f- N Nzmax
7: return Tmaxs Nmax
B.2 Computing SemanticLens metrics
We compute the Redundancy, Clarity, and Polysemanticity metrics from SemanticLens 12 These metries were originally designed for analyzing concept. explanations for a single model. We measure Clarity and Polysemanticity for each concept as described in the original work. However, we modify the Redundancy metric to better analyze concepts in the context of model comparison. Rather than compute similarity between all concepts, similarity between concepts from one model to the concepts of the other model.
measure the
Following SemantieLens, we use generalist model, OpenCLIP 23 to analyze each concept k via its explanation grid E& (see Sec. 3.3). We refer to the OpenCLIP image encoder as F. Let E = { 192 denote the set of images in the explanation grid for concept k. The with minor changes
following equations are the same as those presented in SemanticLens 12 to match the notation used in this work.
For each image Fk E Eg, obtain its OpenCLIP image embedding via
Uhei F(rk,s) Rd
(3)
and collect them into
V2 [Uk,1 Uk
Ukma 1 Rdaxd
(4)
B.2.1 Clarity
The Clarity of concept & is computed using the embeddings (Vx) of the images in explanation (Eg). The more similar the images within explanation are to each other, the clearer the explanation is. Cosine similarity, denoted Scas; is used to measure the similarity between images.
V
Jeiarity (V²) Milev
Sona (VR,i Vkr.))
(5)
1=
IVal |M/-
(1)
Σ
E
M4
[Vxl E M [
(6)
||vaalla
2
B.2.2 Polysemanticity
Polysemantic concepts are concepts that can be decomposed into two or more simpler concepts. In SemanticLens, concepts considered polysemantic if subsets of Fg can be identified that result in diverging semantic directions. The metric is defined as
(V " (()))((h)) = - Leaarity (f (li=.....))))
42

---

where Vi(i C V for i 1; h is subset of the embedded explanation images. Subsets are k
discovered using KMeans with two clusters.
B.2.3 Redundancy
In the context of model comparison, we define redundancy as the similarity between concepts derived from two models. Let A and B denote the models, and let Vi and V1" represent the set of OpenCLIP embeddings corresponding to their respective concepts. In the following equations, use the meon concept embedding, denoted by VgA. to represent the mean embedding of the 4th concept extracted from model A. It is computed as:
Th
where n& denotes the number of images assigned to the explanation of the ttth concept and Vi is the embedding the ith image belonging to that concept. We compute the cosine similarity between mean concept emboddings follows:
VA Vi I l2 |17 E[-1,1]
Tein(Vi V.P) = Scoo (V V.
(9)
We compute this value for each pair of concepts across the two models, resulting in k I similarity matrix. To summarize this information into a single score, we compute the mean of the maximum similarity values per concept, which refer The maximum similarity for concept is low when there is no similar concept from the other model. Consoquently, low redundancy score indicates that the concepts from the two models are less similar, a desirable property when seeking concepts that are unique to each model.
the redundancy score.
VA
Tooa(VA ym
max (T (VA [(') E [-11,
(10)
To obtain a symmetric measure of redundancy, average the scores in both directions:
Fodd (VA, VB)= = (1od BB) + Lr DaA]
(11)
B.3 Normalized Distance Variants
Here we describe two alternatives to neighborhood distances. Both variants start by com- puting the pairwise Euclidean distance for each embedding matrix, resulting in DA paxx and DB E ma
Max- normalized Euclidean Distances. Each distance matrix is divided by the maximum distance in the matrix, such that both DA and DB are normalized between 0 and 1. Referred to as M
Locally Scaled Euclidean Distances. We compute a locally-scaled Euclidean distance that has been shown to have desirable properties for clustering [71.. For each embedding vector a,, this function scales the latent distanccs between a; and all other inputs aj by the distance Dik where ag is the 7th neighbor of a; Referred to RDX,
BSR Variants. We also use these variants to compute the BSR metric. We refer to the variants as BSRMN and BSRLS. BSR with no subscript uses neighborhood distances.
43

---

BSR
BSR MN
BSR
10
09
0.9
0.9
0.8
0.7
07
0.7
06
0.
0.6
0.5
04
0.
0.4
0.3
02
U2
0.2
RDX
Rox (Sabi
BDXare
RDKar (Sabs
boxia (Buds)
0.
RDX
aDx (3ah)
BDX
RDXae (Bab)
BUKis
BoXas (Bab)
0.0
RDX
RDX (Sald)
aax (Sub))
RDXEs
ankes (ilah)
Figure A13: variants for RDX difference function variants. We compute BSR variants for all RDX distance variants using each difference function. By default we use the locally biased difference function, we denote experiments with the subtraction difference function as RDX (Sub). We compute the BSR metric on the MNIST experiments. We see, that BSRLs poor metric, thus we focus on BSR (neighborhood) and BSR to assess the different variants, See Appendix [.6 for longer discussion. Under all distance variants, can see that RDX with the locally biased difference function outperforms subtraction.
B.4 Difference Matrix Function Variants
Subtraction. The simplest approach to comparing the normalized distance matrices is subtraction:
GA. DA DB.
(12)
If the distance between two imputs is small in DA and large in DB, it would result in large negative value in the difference matrix. If the distances are approximately equal in both matrices, then it would result in a value near zero in the difference matrix. Therefore, images considered similar in only one of the two reprosentations would be identified by large negative values in GA,BB Unfortunately, subtraction can be sensitive to imperfect normalization and/or large changes already distant embeddings.
B.5 Difference Explanation Sampling Variants
PageRank. We rank nodes in the graph by their PageRank 401 largest PageRank be i. We select the |E| - nodes corresponding to the |E] 1 largest edges with one endpoint at i. We remove these nodes from the pool and iterate until all m sets of explanation grids (E) sampled.
Let the node with the
B.6 Results
In Fig. A13 and Fig. A14 we evaluate the different variants for RDX. We find that all RDX variants perform better than bascline methods indicating that using both representations to isolate differences is an effective strategy.
First, when comparing difference functions on known MNIST comparisons (Fig. A13), we see consistent. advantage for the locally biased difference function. In all other experiments, use the locally biased difference function. Second, PageRank 44 sampling is slightly worse than our spectral cluster and sample approach (Fig, A14). Third, we notice that BSRLS flawed metric. One comparison direction consistently scores near perfectly while the other is quite poor, indicating that distances are not comparable across the two representations. This indicates that local scaling [7]], is not appropriate when comparing across represcntations. although future work may be able to modify it appropriately. Finally, when comparing RDX to RDXMN we notice that they perform reasonably similarly in the metrics. In Table A7 we show results for a comparison between Mas and M, under BSR and BSR We can see that RDX with neighborhood distances performs well under BSR, but worse on BSR In contrast, My performs well on both metrics. We visualize the explanations for these methods in Fig. A15 While both methods have good explanations for M35 Mo, can
44

---

BSR
MSRMN
BSR
10
1.0
0.9
0.9
0.
0,8
0.
0.7
0.7
0.4
0.4
0.3
0.3
0.3
02
0.2
0.2
0.1
0.1
RDX
antes
Eleann
SaB
CRMP
PCA
RIX
Rokis
KHeana
SAB
my
Pot
RDN
RO3
BAX B
RDt,s
Kleana
1915
pCa
Figure A14: BSR variants for RDX variants and basclines. We compute BSR variants for several methods. We evaluate RDX variants with neighborhood distances (RDX), neighborhood distances and PageRank 44 sampling (RDXpR), max-normallize distances (RDXMN), ) and locally-scaled distances I LRD We compute the BSR metric with all three distance function variants the MNIST, CUB, and ImageNet/ BoageNet/LNaturaliie /Naturallt experiments (without alignment). We see that BSR poor metric as all methods perform perfectly one of the two comparison directions, suggesting that the scaling technique does not make distances across representations comparable. We focus on BSR (noighborhood) and B to assess the different variants. First, broadly that RDX variants outperform all bascline methods. Among them, RDX and RDXMN perform the best, although RDXMN shows slightly greater variance.
see that in the other direction the two methods differ significantly. RDX with neighborhood distances much more focused on the known difference than RDXMN- This likely due to the issue described in Sec. 3.2 Thus, we use RDX and BSR with neighborhood distances for the main experiments.
Table A7: Comparing RDX variants on Ma5 vs. MG under different BSR variants.
RDX (ax MD)
RDX (Ma MD)
RDX M,, Ma5)
RDX [M,, Mas)
BSR
0.80
0.86
0.82
0.88
BSR
0.95
0.63
0.97
0.81
45

---

RDX M 35, M)
RDX MN (M
35, MD)
5
5
5
5 3
5
5
3
3
3
3
5
5
3
5
5
3
5
5
5
3
5
5
5
3
5
5
5
3
5
5
5
3
3 5
3
3
3
3
5
S
3
3
3
5
3
3
5
3
5
5
5
RDX M
M
35 )
RDX MN
(M,
M
35
3
3
3
3
3
3
3
3
3
3
3
3
9
8
8
3
3
3
3
3
3
>
3 2
3 3
3
8
8
0
Ô
3
3
3
3
3
5 3 3
3
3
3
3
a
8
Figure A15: Comparing explanations using max-normalized distances vs. neigh- borhood distances. (Row 1) Both RDX variants generate good difference explanations that capture mixing that is unique to M35- (Row 2) RDX with neighborhood distances focuses much more on the known difference with all three explanations showing cleanly grouped 3s. In contrast, MDXMN shows one group of 3s and two other groups unrelated to the known difference.
46

---

C Implementation Details
Table A8: Expocted effects of "known" differences.
Repr. ID
Modification
Expectation
MNIST dataset only contains 3, 5, and 8. Model checkpoint is from epoch 1, step 184 with 94%
Mistakes on more challenging images. Clusters have slight overlaps.
Ms
MNIST dataset only contains 3, 5, and Final model checkpoint with 98%
Mg
Clusters have little to no overlap.
M,
None
Baseline model with well-separated clusters.
3s and 5s are mixed together in the representation.
M35
Labels for are replaced with 3.
4s and 9s are mixed together in the representation.
M49
Labels for replaced with 4.
Dataset includes horizontally flipped images and uses the original label for flipped images.
M+
Will mix flipped and unflipped digits together.
Dataset includes horizontally flipped images and uses new labels for flipped images.
Horizontally flipped digits are separated into new clusters.
M
Datasel includes vertically flipped images and uses the original label for flipped images.
M2
Will mix flipped and unllipped digits together.
Dataset includes vertically flipped images and uses new labels for flipped images.
Vertically flipped digits are separated into new clusters.
M
Baseline model with organized representational geometry.
CA
None
Remove the spotted wing concept.
Representational changes for images of birds
CA-S
with spotted wings.
Representational changes for images of birds with yellow backs.
CA-YB
Remove the yellow back concept.
Remove the yellow crown concept.
Representational changes for images of birds with yellow crowns.
CA-YC
Remove the eyebrow on head concept.
Representational changes for images of birds Remove the duck-like shape concept.
CA-E
with head eyebrows.
Representational changes for images of birds with duck-like shapes.
CA-D
C.1 Model Training
We train 2-layer convolutional network with an output. dimension eight on a modilied MNIST 32 dataset that only contains the digits 3, 5, and 8 (3,55)T-33558).. The network is trained for five epochs with a batch size of 128. We use the Adam 27 optimizer with the learning rate set to 1e-2 and one-cycle learning rate schedule. The global seed is set to 4834586. For the
Failures of Existing Methods on MNIST-(3,,58) (Sec. 4.2
47

---

comparison experiment, we select checkpoint at epoch 1, step 184 with strong performance (Ms, 95%) and the final checkpoint epoch 5 with expert performance (ME, 98%).
First, we train a 2-layer convolutional network with output dimension of 64 on several modified MNIST datasets. See Table A8 for modification details. The network is trained for five epochs with batch size of 128. We use the Adam 27 optimizer with the learning rate set to 1e-2 and a one-cycle learning rate schedule. The global seed is set to 4834586 for all models. Models are evaluated on the modified dataset that they were trained on. Second, we train a post-hoc concept bottleneck model (PCBM) [70] on the CUB dataset [88] using the original procedure [70] The model backbone a ResNet- [20] pre-trained on CUB from pytorchev [60] The concept classifier is from scikit-learn 47 and is trained with stochastic gradient descent with the elastic-net
Recovering "Known" Differences (Sec. 4.3
penalty. The learning rate is set. le-3 and the model trained for a maximum of 10000 iterations with batch size of 64. For the comparison experiments, we eliminate a concept by deleting the corresponding concept index from the predicted concept vector. The eliminated concepts are provided in Table A8 Models are compared on all images in the CUB train set.
Table A9: Models and their identifiers from the TIMM library.
Repr. ID
Description
Timm Library ID
MD
DINO
vit_base aat_base_patchie 220___2222____ dino
MD2
DINOv2
vit_
Mc
CLIP
openai
hub:tinm/vii Sartt__arge__pee laa.ar__ppacc___liiiii... 336. laion2b ft in12k in1k inat21
McN
CLIP ft. iNat
Discovering "Unknown" Differences (Sec. 4.40. All models in this experiment were downloaded from the timm library 64. Details are available in Table Ag
C.2 Alignment Training
To aligu representation A to representation B, we learn transformation matrix MAR Rdaxds, We randomly sample 70% of the embeddings in our dataset to train the transfor- mation matrix. The other 30% are used as validation set. The matrix is trained for 100 steps, with the Adam optimizer [7 with a learning rate of 0.001. We measure the CKA on the validation set and keep the best transformation matrix.
C.3 Baselines
For the bascline methods use the scikit-learn 1r implementations for PCA, NMF, and KMeans. For CNMF, we use the pymf [3 implementation. For TKSAE (top-k sparse auto- encoder), we use the Overcomplete repository [25]. Top-k SAEs use an overcomplete basis of concepls, but zero all concept coefficients below the top-k coefficient values. In our experiments, we
the k = depending experiment. The encoder for the TKSAE is a linear layer with batch normalization. It is trained for 200 epochs with a batch size of 1024 or the maximum number of images. We use a linear learning rate warmup over the first 10 epochs, after which the learning rate is fixed at 0.0005. The model trained with the Adam 27 optimizer. The sparsity coefficient is set to 0.0004. After training, TKSAE generates 50 concepts per model. In this work, cach mothod is evaluated on k concepts per model. To fairly compare TKSAEs, we only generate explanations for the i concepts with the largest average coefficient. Note that coefficient values for each concept are directly comparable because the magnitude of each concept vectors is n-normalizedd to 1. For SAEs, we use code that is adapled from b8 The SAE has a linear encoder, relu activation and linear decoder. Inputs are z-score normalized. Tt. trained for 500 epochs with batch size of 2000 the maximum number of images. The
basis of 50 concepts and set =
48

---

dimension is set to the number of desired explanations (3 or 5 depending on the experiment). We use linear learning rate warmup over the first 10 epochs, after which the learning rate is fixed at 0.001. The model is trained with the Adam 277 optimizer. The sparsity coefficient is set to 0.0004. FOrNLMCD Gs we use the publicly available official implementation, We modify this implementation to accept pre-computed embedding matrix for each model, allowing us to use the same input embeddings for all methods. We use the default settings from the original paper to generate clusters using HDBSCAN [9] This results in unspecified number of clusters for each model. For a fair evaluation of NLMCD, we keep the clusters from A that had the lowest similarity score to clusters from B, since are interested in finding concepts unique to one of the two models.
For PCA, NMF, CNMF, SAE and TKSAE we generate explanations by sampling the | images with the largest coefficients for each concept. vector. For KMeans, we sample images closest to the centroid of the cluster. For NLMCD, we follow the methodology proposed in the original work and randomly sample |E] images from each cluster.
C.4 RDX Details
We sweep y on one comparison from each experiment group (see breaks in Table A11 and select the value that results in the highest performance on BSR. We find that y of 0.05 or 0.1 works well. We set 3 to 5 in all experiments.
C.5 Comparison Summary
A complete list of comparisons, the data used in the comparison, and the number of images available in Table [AI]] We choose to generate 3 explanations for all MNIST comparisons and 5 explanations for all other comparisons. We choose 3 5 because we prefer small set of explanations for users to analyze. For all experiments, we use 3 image grids. In all comparisons where images are from existing dataset, we use images from the train split because the train split is usually larger. Note that our method is training free and not impacted by the dataset splits. For the iNturalist comparisons, 600 research grade images are downloaded from the iNaturalist website [24] with licenses (cc-by.ce-by-neccom) Images are restricted to be maximum of 500 pixels the longest side.
C.6 Computational Cost
All experiments were conducted using on a machine with an AMD Ryzen 7 3700X 8-Core Processor and single GeForce RTX 4090 GPU with 128GB of RAM. In Table A10 we show the time taken for each method on the CUB dataset (5000 images). RDXPR uses PageRank 4 to rank nodes for sampling and is slower. The time for SAE varies with the model's output dimension and the number of images. In this table, the model backbone ResNet18 20 and has an output dimension of 512.
Table A10: Runtime (in seconds) for selected XAI methods.
RDX
RDX
KMeans
CNMF
SAE
PCA
Classifiers
Time (s)
32.71
187.78
9.65
14.95
629.95
8.49
97.3
49

---

Table A1l: Experimental settings where we report comparison name, dataset, number of images, concepts, and gamma values. We name the comparisons using one direction, but compare in both directions in all experiments.
Num. Ims.
RDX-
Comparison
Comparison Dataset
|8]
?
Ms vs. Mg
MNIIST(33,588
500 x 3
3
0.05
M35 vs. M
MNIST
500 x 10
3
0.1
M49 vs. Ms
MNIST
500 x
0.1
M35 vs. M49
MNIST
500 x 10
3
0.1
Me. M
MNIST w/ hilip
250 x 20
0.1
Mj VS. M
MNIST w/ vflip
250 x 20
0.1
CA-S VS. CA
CUB
5000
5
0.1
CA-YB CA
CUB
5000
0.1
CA-YC VS. CA
CUB
5000
5
0.1
CA-E VS. CA
CUB
5000
5
0.1
CA-D VS. CA
CUB
5000
0.1
(P) Primatcs-lgibbon, siamang, spider monkey] (ImageNet)
Mp vs. Mp2
500x3
0.05
(MT) fiittens-[mitten Christmas stocking, sock] (ImageNet)
MD vs. Mp
500x3
0.05
(B) Buses-[trolley bus, school bus, passenger car] (ImageNet)
MD vs. Mp2
500x3
0.05
(D) Dogs-[whipped Saluki, Italian greyhound] (ImageNet)
Mp vs. Mp
500x3
0.05
(C) Corvids- (Crows, Ravens] (iNaturalist)
Mc McN
500x2
0.05
(G) (Americn Alligator, American Crocodile] (iNaturalist)
Mc McN
500x2
0.05
(MP) Maples [Suger Maple, Red Maple, Norway Maple, Silver Maple] (iNaturalist)
Mc vs. McN
500x4
5
0.05
50


Reviews:
[{"review_id": "glXm38KiDF", "reviewer": "Reviewer_HVHo", "summary": "The paper introduces Representational Difference Explanations (RDX), a post-hoc analytic technique that surfaces where two models disagree in their learned representations. RDX builds pairwise distance matrices for two embedding spaces, forms an asymmetric, locally-biased difference matrix that emphasises disagreements on near-neighbours, converts that matrix into an affinity graph and applies spectral clustering, then turns each high-affinity cluster into a concise image grid that itself is the explanation. Experiments showcase the metholodogies performance on CUB, MNIST and large-scale “unknown difference” settings.", "questions": "Questions:\n- Could the authors explain their choice for conducting experiments on a modified MNIST setting, instead of the original?\n- Is there any comparison with recent explanation methods?\n- Could the authors cite some metrics or numerical evaluation for their method is the main text of the paper?", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper tackles an interesting problem of comparing two models' explanations through post-hoc analysis\n- The paper does not require gradient access or optimization, making it practical for real world applications\n \nWeaknesses:\n- In sections 4.1 and 4.2, the author conduct experiments for dictionary learning a modified MNIST setting, without explaining the rationale behind this choice. MNIST is utilized for comparing digits as some of them can look similar to others.\n-Authors do not compare their method with other XAI methods. The baselines include models that have been largely explored in the literature.\n- The evaluation basis of the method is largely visual. Tables are placed in the appendix, making the main text of the paper difficult to follow, as it's very notation-heavy.", "was_revised": true, "final_justification": "I have read the rebuttal and I am satisfied with the responses to my comments.", "rebuttal": "We thank **HVHo** for their feedback and suggestions. We believe that addressing these comments will make the paper even clearer.  \n\n**(HVHo-1) Why conduct experiments on a modified MNIST?**   \nIn our first experiment, we compare two checkpoints trained on a MNIST dataset modified to include only the digits: 3, 5, and 8 (see L225). Our motivation was to create a simple and clean experimental test bed that could demonstrate an issue we observe with existing explanation methods like NMF, KMeans, PCA, SAE, etc. In Fig. 2, we show that existing methods, that are designed to analyze one representation at a time, are not suitable for comparing representations across models as they result in explanations that are unrelated to the core difference between the representations. In Fig. A2, we discuss issues with explanations that rely on linear decompositions of activation matrices as well. Importantly, we do conduct experiments on more realistic settings using modern models (DINO, DINOv2, CLIP, PCBMs) on real datasets (CUB, ImageNet, and iNaturalist). We will clarify the motivation why we selected this modified version of MNIST in the revised text.  \n\n**(HVHo-2) Comparison with additional XAI methods.**   \nWe compare RDX to a variety of XAI methods (e.g., NMF, KMeans, PCA, and SAEs ) throughout the work (e.g., see Fig. 3). They are popular [A, B] both in practical applications and as the basis of recent research [D, E].  Following the suggestion from **mmBM**, we include comparisons to an additional explanation method (NLMCD) [C], designed for comparing representations specifically, and observe that our method outperforms it on the BSR metric. We test NLMCD on our unaligned representational comparisons for ImageNet and iNaturalist (Fig. 3B, Unaligned). The results are presented below. \n\n|                        | RDX                  | KMeans           | SAE              | NLMCD            | TopKSAE          |\n|------------------------|----------------------|------------------|------------------|------------------|------------------|\n| Unaligned (Q1, Q2, Q3) | **0.90, 0.94, 0.97** | 0.81, 0.88, 0.92 | 0.54, 0.60, 0.64 | 0.45, 0.52, 0.59 | 0.71, 0.76, 0.82 |\n\n[A] Fel, et al. \"Craft: Concept recursive activation factorization for explainability.\" CVPR 2023   \n[B] Cunningham, et al. \"Sparse autoencoders find highly interpretable features in language models.\" arXiv:2309.08600 2023   \n[C] Vielhaben, et al. \"Beyond scalars: Concept-based alignment analysis in vision transformers.\" arXiv:2412.06639 2024   \n[D] Bussmann, Bart, Patrick Leask, and Neel Nanda. \"Batchtopk sparse autoencoders.\" arXiv preprint arXiv:2412.06410 2024)     \n[E] Gao, Leo, et al. \"Scaling and evaluating sparse autoencoders.\" arXiv preprint arXiv:2406.04093 2024  \n\n**(HVHo-3) Analysis is largely visual, is there numerical evaluation?**   \nIn our opinion, the most effective way of explaining the behavior of interpretability methods for vision models to humans is via visual examples. Thus, we have used visual examples throughout the paper to demonstrate our results to the reader. However, we also report comparisons to existing methods using *quantitative metrics*. The primary goal of our work is to develop a method that can compare two models and reveal the concepts unique to each one. We measure this quantitatively using the Binary Success Rate (BSR) metric, described in Sec. 3.5. The results of this metric are presented in Fig. 3, in which we compare several methods on comparing models on multiple datasets. Ablation experiments can be found in Table A3. Additionally, following the suggestion from reviewer **jwKi**, we have included new quantitative metrics from [F] and find that RDX performs well under these metrics as well. The table and analysis is available in the response **jwKi-1**.\n\n[F] Dreyer, et al. \"Mechanistic understanding and validation of large AI models with SemanticLens.\" arXiv:2501.05398 2025  \n\n**(HVHo-4) Tables are placed in the appendix making it hard to follow due to reliance on notation.**   \nThanks for the suggestion. We will polish the notation and clearly point to the additional results in the appendix."}, {"review_id": "PeXR4TxEnS", "reviewer": "Reviewer_jwKi", "summary": "The paper proposes RDX, a novel explanation method that identifies differences in representations between two models and visualizes them as concepts via examples. Explanations are evaluated qualitatively, as well as quantitatively through BSR, a newly proposed metric for evaluating difference explanations.", "questions": "- Please rebut my concerns regarding using only the BSR metric for quantitative comparisons or include a comparison via other evaluation metrics, to offer a more holistic view.\n- Please answer my above question regarding scalability of RDX beyond pair-wise comparisons", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths.** The paper is extremely well-written and structured. The proposed method is innovative and provides a new type of difference-based concept explanations that will be of significant interest to the XAI community. The addressed issue of revealing behavioral differences between models is explained in detail and supported by experimental results. Explanations are evaluated both qualitatively and quantitatively. The proposed method is compared to several state-of-the-art related methods for dictionary learning. Visualizations are informative. The large appendix ensures reproducibility and provides several additional results that answered many of my initial questions. \n\n**Weaknesses** \nBSR, the proposed metric for evaluating explanations (and similarly its variants in the Appendix) simply counts the frequence where two examples from the explanation grid have a lower representational distance in A than in B (i.e., $D_A$ < $D_B$). The sampling of difference explanation grids for RDX is done in a way that only indices with pairwise negative entries in $G_{A,B}$ (line 149), i.e. with $D_A < D_B$ according to Equation (1) are chosen. While I agree that BSR may be a desirable property for difference explanations, its formulation is too close to that of RDX to serve as the (only) quantitative comparison to other explanation methods. I would have liked some comparison via other quantitative metrics as well, e.g., evaluating how/if the found representational differences affect model performance or some of the metrics listed here [1] to offer a more holistic comparison to other explanation methods.\n\nIIUC, since RDX is not symmetric, comparing two models requires inspecting two sets of explanations: RDX(A, B) and RDX(B, A) to discover potential representational differences. I am wondering how and if this scales to comparing more than two models at once, due to this asymmetry. \n\n[1] Dreyer, Maximilian, et al. \"Mechanistic understanding and validation of large AI models with SemanticLens.\" arXiv preprint arXiv:2501.05398 (2025).", "was_revised": true, "final_justification": "I am satisfied with the author's answers to my questions; however I still believe that evaluation is a slight issue here, due to the unavailability of unbiased evaluation metric(s) for difference explanations (owed in part to the fact that, afaik, the concept of difference explanations is novel, and thus lacks evaluation availability).\nTaking this and the other reviews and corresponding rebuttals into account, I believe this paper is a strong 5, and am thus keeping my initial rating.", "rebuttal": "We thank **jwKi** for their careful reading of the paper and insightful comments and suggestions. \n\n\n**(jwKi-1) Concerns about using only the BSR metric.**  \nWe agree that BSR is a desirable property for difference explanations and for this reason our design of RDX is optimized for the BSR target metric. We report performance using this metric as it best matches our methodological goals, i.e., finding groups of inputs that are considered similar by only one of two representations. Additionally, we report results for the BSR metric using alternative distance normalization methods in Fig. A13. There we observe that even when the normalization method used by RDX and the normalization method used in evaluation with BSR are mismatched, RDX still performs the best. This demonstrates RDX’s robustness even when not aligned directly with the evaluation metric. \n\nNonetheless, we agree that additional evaluation metrics would strengthen our results further. To address this we report additional results using the metrics from Dreyer, et al. [A] as suggested. These metrics use an independent, “semantic” model, OpenCLIP, to assess the *clarity*, *redundancy*, and *polysemanticity* of a concept via its explanation. We computed these metrics for the ImageNet and iNaturalist experiments (from Sec. 4.4 and Fig. 6) for both unaligned and aligned representations. Note that for *redundancy*, we adapt the metrics to compare the explanation grids from each model. Essentially, we compute the *similarity* for each concept from Model A to each concept from Model B. We use this similarity matrix to compute the *redundancy* metric (which summarizes the similarity matrix). We present the results below. Q1, Q2, and Q3 indicate first quartile, median, and third quartile, respectively.\n\n**Unaligned**  \n|                      | RDX                  | KMeans               | SAE              | CNMF             | PCA              |\n|----------------------|----------------------|----------------------|------------------|------------------|------------------|\n| Clarity (Q1, Q2, Q3) | 0.38, 0.43, 0.49     | **0.41, 0.45, 0.51** | 0.35, 0.38, 0.42 | 0.39, 0.44, 0.49 | 0.39, 0.43, 0.51 |\n| PolySem              | 0.29, 0.36, 0.44     | **0.27, 0.33, 0.41** | 0.35, 0.41, 0.47 | 0.29, 0.35, 0.42 | 0.31, 0.35, 0.45 |\n| Redundancy           | **0.76, 0.80, 0.82** | 0.83, 0.86, 0.87     | 0.81, 0.84, 0.85 | 0.81, 0.84, 0.85 | 0.76, 0.84, 0.85 |\n\nThe reduction in redundancy is even more apparent after alignment.  \n\n\n**Aligned**  \n\n|                      | RDX                  | KMeans               | SAE              | CNMF                 |\n|----------------------|----------------------|----------------------|------------------|----------------------|\n| Clarity (Q1, Q2, Q3) | 0.44, 0.45, 0.46     | 0.45, 0.45, 0.49     | 0.42, 0.43, 0.49 | **0.48, 0.52, 0.57** |\n| PolySem              | 0.28, 0.32, 0.45     | **0.26, 0.32, 0.37** | 0.40 0.43, 0.44  | 0.34, 0.34, 0.35     |\n| Redundancy           | **0.76, 0.78, 0.79** | 0.85, 0.87, 0.90     | 0.81, 0.81, 0.82 | 0.82, 0.84, 0.86     |\n\nWe observe that RDX generates significantly less redundant explanations as we would expect. We do notice that the RDX explanations are slightly less clear than some other baseline methods according to some of these metrics. However, most methods have similar clarity and polysemanticity scores. We would like to emphasize three caveats to interpreting these results: (1) RDX generates explanations for differences, which naturally highlights parts of the representations that are more likely to be complex and less clear. (2) The metrics here presume that the generalist model (OpenCLIP) is an unbiased evaluator, but it is likely that OpenCLIP has its own biases that influences what it estimates is more or less clear/redundant. (3) Explanations for complex, niche datasets that are out of domain for OpenCLIP would likely have significant issues when evaluated using this metric. A benefit of the BSR metric is that it is insensitive to the complexity of the input images being analyzed.  \n\nAdditionally, RDX is the only method in this work that is able to isolate and explain differences between two representations as seen in Figs. 2, 4, 5, 6, and A4-A10. We believe that a slight increase in concept complexity is natural and acceptable when focusing on understanding nuanced representational differences.  \n\n[A] Dreyer, et al. \"Mechanistic understanding and validation of large AI models with SemanticLens.\" arXiv:2501.05398 2025   \n\n\n**(jwKi-2) Scalability of RDX beyond pairwise comparisons.**   \nThis is an interesting question. There are at least two challenges that would need to be addressed. Suppose we have three representations, A, B, and C. (1) If we want to identify the differences between A vs. “B and C”, we have to decide if this means the union of the differences between (A, B) and (A, C), or the intersection. (2) We would need to develop a method to merge difference graphs from each pairwise comparison so that the output of clustering is the union/intersection of the concept sets. While this would scale combinatorially, RDX is fast enough on datasets of 5k points with a 512 dimensional latent space (32 seconds), that in some settings the pairwise comparisons would be feasible. Our work represents one of the first approaches to automatically identify explainable differences between representations from two different models. Extending our work efficiently beyond pairs of models, is a really interesting question which we leave for future work. We will highlight this opportunity in the revised text."}, {"review_id": "J8LOaxwbC9", "reviewer": "Reviewer_e4CM", "summary": "The paper proposes a new method, Representational Differences Explanations (RDX), to compare representations of different models. The authors evaluate the method both on representations with known and unknown differences and show examples where RDX yields better explanations of the differences.", "questions": "(see the weaknesses for more information and context)\n1) Can you provide any guarantees that changing the dataset would have only a minor effect on the final explanations? Or, at least, show it empirically?\n2) Is there any reason to believe that using Euclidean distances in your method makes sense even when the data lies on curved manifolds?\n3) How did you choose the examples of explanations shown in the paper? Were there any objective criteria used for choosing what to report (and what not to)?\n4) Regarding representational alignment (section 3.4): it is not clear to me when exactly this step happens. Do you use the transformation in all the experiments, or only in some? How does it change the resulting explanations?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Overall, I think the paper proposes an interesting method that can give new insights into the differences between the representations of models. However, there are some problems that need to be addressed or justified.\n\n**Strengths:**\n- The problem, as described in the Introduction and illustrated by Figure 1, is well-motivated and easy to understand. \n- The paper is clear, well-written, and easy to understand.\n- The problem addressed in this paper is timely and important.\n- It seems to me that there are sufficient details needed to replicate the experiments.\n- Based on the figures provided, it seems that the method works and can reveal interesting insights into the differences between the models. It is able to capture differences between two models instead of focusing on the most important explanations for each model separately.\n\n**Weaknesses:**\n- The examples shown might potentially be cherry-picked. For example, in Figure 6, five different explanations were identified in each case, but images are shown only for two of them. How can one make sure that these were not shown solely because they support the claims and make a nice story? \n- Thorough evaluation of the method is missing. The binary success rate metric (BSR) seems to me a bit cyclic, since it evaluates something that the RDX method was explicitely developed to do (identify pairs that are closer in one representation than the other). But the other methods, that RDX are compared to, do not have this as a primary goal, so it is perhaps not so surprising that RDX outperformes the rest. But does it neccessarily imply that it gives better explanations? (I also acknowledge that objectively evaluating explanations is a complicated issue that perhaps goes beyond the scope of this paper.) \n- The method depends on Euclidean distances between points from a chosen dataset. I think this is problematic from two perspectives. First, there is a strong dependency on the dataset choice. Since the distances are defined as a ranking of the nearest neighbors (and not how far the points actually are), adding or removing points from the dataset would heavily influence the distance matrices. For instance, in the experiments where only a subset of ImageNet classes are used, would the explanations be different if all ImageNet classes were used for testing? Second, the representations usually live on curved manifolds in high-dimensional spaces, where Euclidean distances do not make sense. For example, two points can be close in Euclidean geometry, but very distant in manifold geometry. How are these differences taken into account in RDX?", "was_revised": true, "final_justification": "I was satisfied with the responses to my comments, and I also did not find anything disturbing in the other reviewers. Hence, I raised the score.", "rebuttal": "We thank **e4CM** for their comments and suggestions. Please do not hesitate to follow up in the discussion period if there are any remaining questions or points that we can clarify.   \n\n**(e4CM-1) How were the qualitative results in the paper selected?**   \nWe select the clearest and most representative examples for the main paper. We include the full set of explanations for most comparisons in the appendix. We selected two concepts to visualize in Fig. 6 due to space constraints, the remainder are available in Figs. A8 and A10. More results for MNIST are found in Fig. A4 and more results for the CUB experiments are shown in Figs. A5 and A6. Complete results for the Mittens (ImageNet), Primates (ImageNet), Maples (iNat), and Corvids (iNat) are available in Figs. A7-A10. There is no specific objective criteria for choosing which results are presented in the main text vs. presented in the appendix. However, we feel that we have presented a large number of results from our experiments for the reader to inspect. \n\n**(e4CM-2) BSR is cyclic. Is it possible to improve evaluation?**   \nWe designed BSR to measure a property of difference explanations that we believe is important for comparing representations, i.e., finding groups of inputs that are considered similar by only one of two representations. We show that existing methods do not succeed at this task and design a method, RDX, to optimize for this metric (see Figs. 2, 4, and 5). We agree that it is perhaps not surprising that our method performs better than existing methods for this task, but we report BSR as it demonstrates that we have succeeded in our goals. Additionally, we have tested BSR with different distance normalization schemes and found that RDX generally performs the best even when the distance normalization schemes are mismatched (Fig. A13).\n\nWe agree that an in-depth human evaluation is out of the scope of this work. We include many qualitative visualizations in both the main paper and appendix for the reader to scrutinize and assess if the explanations are meaningful. In addition, in Sec. B.3 we report results for some case studies to discuss how explanations may, or may not, link to classification performance. \n\nTo strengthen our quantitative results further, following suggestions from **jwKi** and **mmBM**, we include new metrics from [A] and a new method that was explicitly designed for comparison [B] in our results. The new metrics can be found in the response to **jwKi** (see response **(jwKi-1)**).  The new method’s results can be found in the response to **mmBM** (see response **(mmBM-1)**).  \n\n[A] Dreyer, et al. \"Mechanistic understanding and validation of large AI models with SemanticLens.\" arXiv:2501.05398 2025   \n[B] Vielhaben, et al. \"Beyond scalars: Concept-based alignment analysis in vision transformers.\" arXiv:2412.06639 2024   \n\n\n**(e4CM-3) Points may be nearby using Euclidean distance, but far apart on the data manifold. How is this handled?**   \nThis is a good point, two datapoints may indeed be close together based on Euclidean distance but quite distant if we follow the manifold that the data lies on. We do not explicitly handle this case, but there are two reasons we believe RDX can work despite this issue.  \n\n(1) Many models (e.g., like DINO) are trained to have meaningful Euclidean distances in their latent space and so we assume the distances are meaningful. For classification models, the last layer is a linear classifier, resulting in a linearly separable representation. While this does not guarantee meaningful Euclidean distances, it strongly implies it. We use the last layer for all experiments in this work. \n\n(2) It is unlikely that the *nearest* neighbors for a given point have small Euclidean distances, but large distances on the manifold, since we generally expect that representations are *locally* linear. Neighborhood distances and our locally-biased difference function push RDX to seek out groups of points that are locally linear. Notably, the non-linear dimensionality reduction technique Isomap [C] uses top-k nearest neighbors to construct a graph and estimate distances along the manifold of the data. Our locally-biased difference function is a *soft* version of the same idea.\n\nOur results are reported across a variety of backbones and training objectives (e.g., see Table A5) and demonstrate robustness in the presence of this variation.  Finally, RDX is not restricted to Euclidean distances. In the future, it would be interesting to explore other choices for the various components of the algorithm.  \n\n\n[C] Tenenbaum, et al.. \"A global geometric framework for nonlinear dimensionality reduction.\" Science 2000  \n\n \n**(e4CM-4) How does adding or removing points affect explanations?**   \nRDX explanations are conditioned upon both the model and the input data. To the best of our knowledge, this is true for all concept-based explanation methods. For example, dictionary-learning approaches (NMF, SAE, PCA, etc…) will have different dictionaries depending on the data used to generate model activations. If the input data changes, the representation will change, and, subsequently the explanations will change. \n\nHowever, if the number of points changes, but the data distribution stays the same, the concepts discovered will be very similar. We have conducted a new experiment, in which we add or remove 20% of the data (from the same distribution) and observed that the spectral clusters are highly similar. We are happy to include this result in the revised paper. Thanks for the suggestion!     \n\n\n**(e4CM-5) How does representational alignment work? When does this step happen, how does it change the explanations?**    \nThe step does not happen for all experiments. For the “unknown” difference experiments (ImageNet and iNaturalist), we conduct experiments without and with alignment and report results for both unaligned (Fig. 3B Unaligned) and aligned representations (Fig. 3B Aligned). When aligning representations, we test alignment in both directions. We align representation A to representation B and vice versa. We use the symbol ‘ to indicate aligned representations (see L180). In Fig. 3B Aligned, we can see that results are reported twice for the baseline methods. KMeansA’ indicates that A was aligned to B, and KMeansB’ indicates that B was aligned to A. The qualitative results we show for these unknown experiments are all after alignment (Fig. 6 and Figs. A7-A10).\n\nRepresentations may be misaligned due to a linear transformation of the representations. In these cases, aligning the representations before comparing them can help us isolate more fundamental differences (Sec 3.4). Alignment does change the explanations. In Fig. A3, we visualize the effect of alignment and show how it changes the explanations. We will highlight this result in the revised text and update the description of the alignment so that it is clearer."}, {"review_id": "pU9lrB8XTz", "reviewer": "Reviewer_mmBM", "summary": "This paper proposes a method called Representational Differences Explanations (RDX), aimed at identifying and explaining differences between the learned representation spaces of two models. The method automatically identifies groups of input samples that are close in one model’s representation space but far apart in the other, or vice versa. The authors introduce a quantitative metric to assess how well such groups reflect the divergence between models’ internal representations.\nRDX is compared against classical clustering (k-means) and sparse dictionary learning methods (e.g., NMF, Sparse Autoencoders). The evaluation is carried out on controlled settings such as MNIST and CUB, and the paper includes an ablation study and qualitative examples that demonstrate how RDX can uncover previously unknown model differences.", "questions": "(1) Did/Can dictionary learning methods uncover any different types of representational differences compared to RDX?\n\n(2) line 57: The statement “Visual concepts are usually defined as images that the model considers to be similar” may be unprecise. I was under the impression that visual concepts rather correspond to recurring input patterns or localized semantics, not just described by overall image similarity.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts:\n- The paper tackles the challenge of systematically explaining differences between learned representations across models, a question that has received far less attention than single-model interpretability.\n- The paper is clear and accessible. Many natural questions that arise while reading are addressed in the appendix, making the paper thorough without being dense.\n- RDX is composed of well-established components, such as spectral clustering and local neighborhood distances, combined in a novel way to address the representation difference problem. \n- The authors propose a concrete, intuitive metric to assess the quality of difference explanations. While simple, it aligns well with the goals of the method and is easy to compute.\n- The method is evaluated both quantitatively and qualitatively on controlled datasets like MNIST and CUB.\n- The case studies clearly show that RDX can uncover subtle, meaningful, and non-obvious differences between model representations.\n- The comparison to dictionary learning methods (e.g., NMF, Sparse Autoencoders) illustrates where these classical approaches fall short in identifying contrastive differences between models.\n\nWeaknesses:\n- The baselines used in the paper (e.g., k-means, basic sparse dictionary learning) are relatively simple and unsurprisingly underperform. This weakens the strength of the empirical comparisons. Incorporating more recent and relevant methods, such as those explicitly designed to compare representation spaces [1, 2], would provide a stronger benchmark. In particular, USAE [1] could be applied by training a larger sparse autoencoder over both models and then identifying concepts that are selectively activated in only one model.\n- The method relies on computing pairwise distance matrices, which scale quadratically with the number of samples. While experiments use up to 5,000 samples, this becomes problematic for large-scale datasets (e.g., millions of samples). The paper does not address how RDX would scale to such settings, where sparsity of concepts or subsampling may become essential.\n- The SAE comparison is limited to very low-dimensional hidden spaces (e.g., 3 or 5 units/neurons), which may restrict the richness of the discovered differences. Modern alternatives like top-K sparse autoencoders could offer more expressive and interpretable explanations, and should be considered\n- The method assumes that shared local concepts imply similarity in the representation space. However, this assumption may not hold in more complex settings where the relevant concept is a small part of the input (and background features differ a lot). In such cases, using input-space similarity to assess representational similarity might be misleading or insufficient.\n\nReferences:\n\n[1] Thasarathan, Harrish, et al. \"Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment.\" arXiv preprint arXiv:2502.03714 (2025).\n\n[2] Vielhaben, Johanna, et al. \"Beyond scalars: Concept-based alignment analysis in vision transformers.\" arXiv preprint arXiv:2412.06639 (2024).", "was_revised": true, "final_justification": "The paper introduces RDX, a novel and intuitive framework for identifying and explaining differences between the learned representation spaces of two models. This is a relatively unexplored but important direction in interpretability research.\nStrengths include a clear problem formulation, a principled combination of established methods, and a simple yet effective metric for assessing difference explanations. The qualitative case studies convincingly demonstrate RDX’s ability to uncover subtle and non-obvious representational differences.\nThe rebuttal addressed an earlier concern by providing results with additional baselines, which strengthens the empirical contribution.\nHowever, some issues remain: scalability to large datasets is not addressed (the method is limited by quadratic complexity in the number of samples), and the conditions under which other, more established baselines might perform favorably remain underexplored.\nOverall, the novelty of the problem setting, the soundness of the approach, and the improved empirical evidence after the rebuttal outweigh the remaining limitations. I maintain a positive recommendation, with the expectation that future work will explore scaling strategies and broader baseline comparisons.", "rebuttal": "We thank **mmBM** for their comments and suggestions. Based on the requests, we have added new quantitative comparisons to further demonstrate the effectiveness of our RDX approach.  \n\n**(mmBM-1) Baselines are relatively simple. Can we compare to more recent baselines?**   \nIn the paper we compare to a range of distinct baselines which span different families of methods, e.g., PCA, dictionary learning, and sparse autoencoders (see Fig. 3). They have been used with some success in interpretability tasks (e.g., in [D, E]) and we felt it was important to assess them on the model comparison task in this work.\n\nAs indicated in the review, there have been some recent contemporaneous works comparing model representations, e.g., [A] on arXiv in Feb 2025,  [B] on arXiv in Dec 2024, and [C] on arXiv in Mar 2025. We already discuss [A, C] in our related works section (see L72 and L70). We found several challenges when trying to fairly compare to USAEs [A], so we decided to exclude it.  \n* The official code for USAEs has not yet been published at the time of this rebuttal. The USAE submission on OpenReview has a link to a GitHub page, but this leads to a 404 page. Also, the overcomplete repository (a Vision-based SAE Toolbox on GitHub) appears when searching for the USAE code, but this is because the README states that overcomplete was used by the USAE authors. Despite this, the code for training USAEs was not available in overcomplete.   \n* The methodology proposed by the authors of [A] requires training a universal representation space for every comparison. We had concerns about fairly evaluating their methodology due to the number and variety of comparisons in this work. For example, we conduct comparisons for several model variants trained on modified MNIST datasets, PCBM models trained on CUB, models trained on ImageNet, and models trained on iNaturalist. It is not clear if USAEs will be sensitive to dataset size and/or the number of models used in learning the universal latent space.  \n* Additionally, we work with small focused datasets with similar classes, it has not been established how to correctly apply USAEs to small scale settings like the modified MNIST experiment we present in Fig. 2.  \n* We also would also like to highlight Fig. A2, which shows several issues that are true of any linear decomposition method that make them less ideal for comparison tasks.  \n* Finally, RDX is explicitly designed to find differences, whereas both USAEs [A] and RSVC [C] *may* find differences as a byproduct of their main goals.  \n\nThank you for the suggestion to include a comparison to NLMCD [B]. We were able to run experiments with this approach on our unaligned ImageNet representations (i.e., DINO vs DINOv2) and iNaturalist experiments from Fig. 6 and Figs. A7-A10. Note that this is a total of 7 experiments. We had to adapt NLMCD to make it compatible with our evaluation protocol as it generates an arbitrary number of concepts for each representation and measures the similarity of concepts from each representation. For our experiments, we selected the *k* concepts that had the lowest similarity score as the representative concepts for differences and measured the BSR metric. We find that RDX outperforms NLMCD consistently on all 7 experiments. We provide first, second and third quartiles for the BSR metric.   \n\n|                        | RDX                  | KMeans           | SAE              | NLMCD            | TopKSAE          |\n|------------------------|----------------------|------------------|------------------|------------------|------------------|\n| Unaligned (Q1, Q2, Q3) | **0.90, 0.94, 0.97** | 0.81, 0.88, 0.92 | 0.54, 0.60, 0.64 | 0.45, 0.52, 0.59 | 0.71, 0.76, 0.82 |\n\n[A] Thasarathan, et al. \"Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment.\" arXiv:2502.03714 2025   \n[B] Vielhaben, Johanna, et al. \"Beyond scalars: Concept-based alignment analysis in vision transformers.\" arXiv:2412.06639 2024   \n[C] Kondapaneni et al... \"Representational Similarity via Interpretable Visual Concepts.\" ICLR 2025  \n[D] Fel, et al. \"Craft: Concept recursive activation factorization for explainability.\" CVPR 2023   \n[E] Cunningham, et al. \"Sparse autoencoders find highly interpretable features in language models.\" arXiv:2309.08600 2023  \n\n**(mmBM-2) Scaling due to requirement for pairwise distance matrices.**   \nOur intended setting for this approach is the fine-grained analysis of a smaller dataset. We believe this setting to be quite reasonable for scientists (such as astronomers or biologists) studying specific datasets with vision models. Scaling this approach would require a sub-sampling strategy using a sparse similarity matrix to find groups of images that are near each other in the representational space. Beyond constructing a large similarity matrix, we see no other conceptual issues associated with applying our method to larger datasets. \n\nAdditionally, we would like to highlight that it is not clear that analyzing models at scale results in a meaningful understanding of the underlying models, as seen in recent evaluation studies [F, G]. We believe our approach of focusing on smaller datasets makes it easier to assess the efficacy of a method and will be more fruitful in the long-run. However, we agree that this is an important limitation to highlight and we will discuss it in the revised text. \n\n[F] Kantamneni et al. \"Are sparse autoencoders useful? a case study in sparse probing.\" arXiv:2502.16681 2025  \n[G] Smith et al. \"Negative results for SAEs on downstream tasks and deprioritising SAE research (GDM mech interp team progress update# 2).” 2025   \n\n**(mmBM-3) Issue with low dimensional SAEs. Why not use top-K SAEs with a larger latent space?**   \nWe chose to use low-dimensional SAEs to have a fair, consistent, and straightforward evaluation for all methods. However, it is true that SAEs tend to work better with an overcomplete basis. Based on this  suggestion, we trained a top-K SAE with 50 concepts on unaligned and aligned experiments on the ImageNet subsets and iNaturalist subsets. We selected the 5 concepts with the largest average weight as representative concepts for our evaluation. \n\nWe present the BSR results of the 7 “unknown” experiments here for both aligned and unaligned representations. Q1, Q2, and Q3 indicate the first quartile, median, and third quartile respectively.  We found that top-K SAEs outperformed SAEs, but still underperformed RDX, especially when comparing aligned representations. Aligned representations generally have more subtle differences making it more challenging for methods that are not designed to seek out differences.\n\n|                        | RDX                  | KMeans           | SAE              | NLMCD            | TopKSAE          |\n|------------------------|----------------------|------------------|------------------|------------------|------------------|\n| Unaligned (Q1, Q2, Q3) | **0.90, 0.94, 0.97** | 0.81, 0.88, 0.92 | 0.54, 0.60, 0.64 | 0.45, 0.52, 0.59 | 0.71, 0.76, 0.82 |\n| Aligned                | **0.89, 0.91, 0.94** | 0.51, 0.60, 0.69 | 0.38, 0.44, 0.50 |       ---        | 0.35, 0.41, 0.57 |\n\nAlso, we are happy to include additional NLMCD results on Aligned representations for the final version. We did not have time to run this experiment as it requires some extra modifications of the NLMCD code.  \n\n**(mmBM-4) Assumption that shared local concepts imply similarity in the representation space.**    \nThe assumption our work makes is actually the inverse statement. We assume that similarity in the representational space implies that there is a shared concept in the input space. This is a commonly used assumption in concept discovery, eg., [D, E]. For example, in Craft [D], concepts are visualized via the maximally activating images for that concept. The images that fit this criteria are considered similar in the representational space, specifically along the direction of that concept.  \n\nBy visualizing images that are considered similar *by the model*, we hope to identify a consistent visual feature that the model has picked up on. If the model organizes images by a subtle, small concept, then we assume it will be one of the consistent visual attributes in our explanation grids. We will clarify this assumption in the revised text.  \n\n**(mmBM-5) Can dictionary learning methods uncover any different types of representational differences compared to RDX?**  \nWe found it challenging to interpret the results from the dictionary learning methods due to the issues we highlight in Fig. A2, i.e., even when there was a concept discovered by a method, it was unclear how unique it was and how to interpret it alongside the other concepts extracted by the method. In contrast, RDX identifies concepts that maximize the uniqueness of the concept and does not require reasoning over multiple concepts in a weighted linear combination. In Fig. 2, we show how RDX focuses on the core difference between representations and results in easy to interpret explanations. \n\n**(mmBM-6) Definition of visual concepts on L57.**  \nWe appreciate how this definition could lead to confusion. Generally there are two approaches to concept based XAI methods: (1) concepts are pre-defined via a set of inputs [H] and then the model is analyzed to see how much the concept contributes to decision making or (2) concepts are discovered directly from the model’s activations [D]. Our definition only applies to the second setting, we will adjust the definition to be more general. Thanks for the suggestion. \n\n[H] Kim et al. \"Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).\" ICML 2018"}]

Author rebuttal:

Meta-review: This submission addresses the problem of discovering and explaining differences in the learned representations of two models, proposing a method called Representational Differences Explanation (RDX) as well as a related Binary Success Rate (BSR) evaluation metric.

Strengths identified by the reviewers are as follows:
- The problem is novel and important (Reviewer jwKi calls it a "new type of difference-based concept explanations that will be of significant interest to the XAI community").
- Both qualitative and quantitative evaluation:
    - The qualitative examples show convincingly that RDX can reveal non-obvious differences.
    - The quantitative evaluation uses the proposed BSR metric, which is a contribution in itself.
- The paper is well-written and organized, and includes a long appendix that helps with reproducibility and anticipates reader questions.

Weaknesses:
- Reviewers commented that the BSR metric is closely related to the RDX method and thus other quantitative metrics should be considered. The author rebuttal reported additional metrics from a paper suggested by Reviewer jwKi, and explained why the new results are sensible. Reviewers did acknowledge that the lack of evaluation metrics is understandable given the newness of the problem.
- Reviewers also noted that the baseline methods are not explicitly designed to compare representations. The rebuttal addressed this by reporting on additional comparisons with NLMCD by Vielhaben et al. (2024) and also with top-K SAEs. It also explained why comparison with USAEs is difficult (and USAEs should be treated as contemporaneous). 
- RDX has some shortcomings, which could be clarified as limitations:
    - Quadratic scaling with the number of samples.
    - Dependence on a chosen dataset. This is also true of dictionary learning approaches, and the rebuttal promised to report on a new experiment where 20% of points are removed/added, which the authors should follow through on.
    - Dependence on Euclidean distance, although the rebuttal discussed mitigating factors.

All reviewers were in favor of acceptance, and the rebuttal strengthened the quantitative evaluation by adding additional metrics and baselines, which should be incorporated. I also recommend acceptance and feel that this was a well-executed piece of work.
Decision: Accept (poster)