Representational Similarity via Interpretable Visual Concepts
Authors: Neehar Kondapaneni
Oisin Mac Aodha
Pietro Perona
Venue: ICLR 2025 Poster (2025)
Keywords: representational alignment
representational similarity
explainability
interpretability
comparison
Primary area: interpretability and explainable AI
TL;DR: a method to visually, interpretably compare two neural networks.

Abstract: How do two deep neural networks differ in how they arrive at a decision? Measuring  the similarity of deep networks has been a long-standing open question. Most existing methods provide a single number to measure the similarity of two networks at a given layer, but give no insight into what makes them similar or dissimilar. We introduce an interpretable representational similarity method (RSVC) to compare two networks. We use RSVC to discover shared and unique visual concepts between two models. We show that some aspects of model differences can be attributed to unique concepts discovered by one model that are not well represented in the other. Finally, we conduct extensive evaluation across different vision model architectures and training protocols to demonstrate its effectiveness.

Published as a conference paper at ICLR 2025
REPRESENTATIONAL SIMILARITY VIA INTERPRETABLE VISUAL CONCEPTS
Neehar Kondapaneni' Oisin Mac Aodha²
Pietro Perona
Caltech Univeersity of Edinburgh
ABSTRACT
How do two deep neural networks differ in how they arrive at a decision? Mea- suring the similarity of deep networks has been a long-standing open question. Most existing methods provide a single number to measure the similarity of two networks at given layer, but give no insight into what makes them similar or dis- similar. We introduce an interpretable representational similarity method (RSVC) to compare two networks. We use RSVC to discover shared and unique visual concepts between two models. We show that some aspects of model differences can be attributed to unique concepts discovered by one model that are not well represented in the other. Finally, we conduct extensive evaluation across different vision model architectures and training protocols to demonstrate its effectiveness. Code: github. com hatmm//kodaaa/////
Project. Page: RSVC
INTRODUCTION
The accuracy of deep neural networks has steadily increased over the last few years thanks to im- provements model architectures, dataset size, and pretraining strategies. However, much less understood regarding how the representations of different models have changed to make the mod- els more effective. Thus, there is growing interest in developing methods that allow practitioners to compare different networks. Comparing the activation matrices of two neural networks over the same set of inputs underpins current representational similarity methods, e.g., CCA (Hotelling, 1936). CKA (Kornblihh al. (201)), RSA (Kriegeskorte et al., 2008), and Brain-score (Schrimpf et al. 2018). While these approaches provide a score denoting the similarity between two differ- models, they do not identify the specifics of what makes models' computations similar or dissimilar, and what aspects of representation lead to differences in model decisions.
In parallel, methods for concept-based eXplainable AI (XAI) have improved our ability to under- stand what features individual models use to arrive at decisions. Understanding these features is critical for ensuring model fairness and identifying potential sources of bias (Kaminski & Urban 2021 Kop 2021). In general, XAI methods sacrifice model fidelity to produce explanations that are simple enough for human interpretation (Fel et al. 2023a Cunningham al.) 2024).. Thus, there is a tension between model fidelity and human understanding.
We propose that contrasting two models an effective way to identify and highlight what makes a model unique, so that users can identify critical features that drive differences in model behavior. To investigate this idea we develop a new approach that extends concept-based XAI methods so they can quantitatively measure representational similarity and provide interpretable insights into the differences between models. We name our method Representational Similarity via Interpretable Visual Concepts (RSVC). Our method builds interpretability approaches in which the activations of a given layer are decomposed into a coefficient matrix and a vector basis. These approaches group images that produce similar activation patterns together. Thus, define visual concept to be the equivalence class of images that produce similar activation patterns. After computing visual concepts from each model, our method evaluates whether the visual concepts from one model are also used by the other model. We make three contributions:
RSVC VC, a new approach for providing hluman-interpredde insights into model differences (Fig.
A validation strategy to link representational differences to model decisions.
Experiments to show that RSVC measure similarity both coarse- and fine-grained levels.

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 REPRESENTATIINAL. SIMILARITY
Similarity methods attempt to quantify the similarity'dissimlarryy between pairs of different mod- els (Hotelling. 1936 Korbllihh et al., 2019; Raghu et al.] 2017 Li et al.] 2015 Huh et al.) 2024). Models can be compared based their functional similarity (i.e., how their outputs relate) their representational similarity (i.e., how the activations of intermediate layers relate) (Klabunde et al.) 2023). While functional similarity can tell us about how model outputs vary, two models can achieve the same performance with significantly different representations. These differences matter, e.g.. while two different forms of pretraining can achieve similar performance on certain datasets, they may transfer poorly to others (Xie et al. (023).
Representational similarity metrics have successfully been used to analyze the differences between architectures (Nguyen et al. 2021 Raghu et al., (02T), explore the effects of different kinds of pretraining (Xie et al. 2023 Neyshabur et al. 2020 Park et al., 2024). develop novel strategies for efficient ensembling (Zhang et al., 2020), or perform ensembling that is robust to distribution shifts (Lee et al., 2023). Some methods have leveraged representational similarity to build tools for text-to-image generation (Rombach et al. (020) or model-to-model translation (Dravid et al., 2023). In neuro- and cognitive science, representational similarity is used to measure how well models are able to approximate the neural recordings of the brain and/or the behavior of humans (Schrimpi et al. 2018: Muttenthaler et al., 2023 Fel et al.] 2022b) Ahllet et al. 2024 Kriegeskorte et al, 2008 In the disentanglement literature, representational similarity has been used to evaluate how well the learned representation matches known ground truth latent factors. A popular approach is use regularized linear predictors to map learned factors ground truth latent factors (Eastwood
& Williams, 2018: Eastwood et al. 2023: Locatello et al. 2019, Roth et al. 2023 Duan et al. 2020). Recently, Dravid et al. (2023) proposed a method that uses correlated activation patterns across networks to mine for "Rosetta Neurons". These neurons provide insights about features that re-occur consistently many models. In contrast to our proposed method, Rosetta Neurons do not quantify the overall similarity between networks and do not identify neurons that explain model differences. Most closely related to our work is that of (Schrimpf et al. 2018) and (Li et al. 2015) which use a linear regression model as a similarity metric between the activations of two networks. We describe these methods in more detail in Sec. 3.2
The primary limitation of most existing representational similarity methods is that they can quantify how similar the representations of two models are, but can not tell users what makes the models similar dissimilar. We propose new approach that aims address this question. Our approach leverages concept-based explainability as an intermediate step to measure representational similarity.
2.2 EXPLAINABLE AI (XAI)
XAI methods aim to answer the questions (1) what features did model use to arrive at a decision and (2) where is the relevant information in the input. Local explanation methods focus on pixel- based attribution, in which a heatmap indicates the region of the image that is most relevant the model's 's decision (Selvaraju et al.) 2020; Ribeiro et al. 2016; Lundberg & Lee 2017). While local explanations are able answer the "where" question, it can be challenging to interpret "what" is being highlighted as attributions can be noisy.
To better address the "what" question, global concept-based explanations can be used (Kim et al. 2018 Ghorbani et al. 2019: Zhang et al., [021]] Fel et al.] 2023a Kowal et al. [024 Poeta et al., 2023 Bau et al., 2020). These approaches discover groups of images (or image regions) that share some visual feature that is relevant to the model's decision making. In addition, "glocal" methods have been developed to answer both questions simultaneously (Schrouff et al.] 2021 Fel et al.,
2023b Achtibat et al. 2023 Kondapaneni et al. 2024).
Ilyas et al. (2022) take a different approach to explaining model decisions whereby linear surrogate models are trained to reproduce the output of a deep neural network. Then, each training image is given a score that reflects how much it contributed to the final weights of the learned model. In (Shah) et al. 2023), this approach was extended to analyze the differences between two models by iden- tifying the differences in training images that each model relies on to arrive at different decisions. While work also aims to compare two models in an interpretable manner, our approach uses
2

---

Published as a conference paper at ICLR 2025
Concept Extraction
bluejay tail
u beckground
U₁
W1
A
(M)
Concept Regression
Concept Similarity
Ma.
Mluaja tail
M, Mr
backgrourd
Model 2 (M₂)
A2
W 2-
244 12--1
Pearzon: 0.60
Pearson: 0.83
Gan Model prodict Model concepts?
The sky background (u}) shared, but biuejay tait (uf) is "missing""
Figure Representational Similarity via interpretable Visual Concepts (RSVC). (Concept Extraction): First, activations for a set of image patches, I", are computed for each model (M1 and M2). Second, the activation matrix for M₁ factorized into the concept coefficient matrix U1 and the concept basis Wi, i..,, A = U, W Each entry in column vector of the coefficient matrix U, represents the strength patches that correspond to the top coefficients. Here, we highlight only two concepts, uf and u²- The top four images for these concepts indicate that wf represents bluejay tail and uf represents sky background. (Concept Regression): To measure concept similarity, learn weight matrix W to map A2 to the concept coefficient matrix U2. We denote the predicted coefficient matrix U2 2-+11 (Concept Similarity): Finally, compute the correlation between columns of U2-11 and U, If A2 contains a concept in U,, then the predicted coefficient vector should be highly that the bluejay tail concept is
concept an image. Concepts are visualized by the image
correlated the real coefficient vector. In this example, poorly represented in M2, but both models share the sky background concept.
the activations of the original model itself. This gives RSVC the advantage of being able to link representational changes to functional model behavior.
METHOD
We propose a new approach to compare the representations of two models using concepts. Our approach closely related methods for computing representational similarity, such Brain- Score (Schrimpf et al., 2018) and the metric in (Li et al. 2015), and builds upon prior work in concept extraction (Fel et 20036 Ghorbani et al.) 2019 Fel al. (023a). We bridge approaches from these two fields, resulting in interpretable method to measure representational similarity for deep neural networks (Fig. T].
3.1 CONCEPT EXTRACTION
Fel et al. (202a)) showed that many concept based explainability approaches can be generalized as of n input images I used to compute
dictionary learning methods, In these approaches, activations from a specific layer of a neural network resulting in an activation matrix A E Rn Then, dictionary learning algorithm
where each row is an activation vector of dimensionality can be used to approximate the activations as A z UW. The row vectors of the vector basis k can be interpreted as set of concepts. Similarly, the rows the coefficient matrix U E R represent the importance of particular concept vector in W for given image. By visualizing the images that have the largest concept coefficients for a given concept, we are able to understand what visual concepts the network has identified in the data. The effectiveness of such concept based XAI methods has been demonstrated via user studies in previous work (Fel et al,
2023b Ghorbani al. 2019 Kim et al. 2018)
3

---

Published as a conference paper at ICLR 2025
Our goal is to develop method to compare the representational similarity between two models, not just in terms of single numerical score, but via interpretable concepts. For each model, we use the concept extraction approach proposed in CRAFT (Fel et al., 2023b). We denote the first model as M1, and the second as M2. Importantly, we make no assumption that these two models are from the same model family, e.g-, one could be a CNN and the other Vision Transformer. We outline the concept extraction process for M1, but the same process is applied to M2.
For specific object class c, we select the set of images Ií that Mj predicted to contain class By grouping images according to model predictions, as opposed to only the ground truth labels, we are able to identify concepts used in all images that the model believes are part of class c. This allows the method to provide better insight into both correct and incorrect predictions. Images are usually composed of several visual concepts, so we extract evenly spaced patches from the image to form a set of "concept proposal" images. These patches are more likely to contain single visual concept and are easier to interpret. We re-use Zi to refer to the set of concept proposal images.
Each concept proposal is resized to the model's input resolution and passed through the network. Note that all networks are trained with cropping augmentation, such that patches are in the domain of training images. We denote the activations from specific layer I and class c as A1 E [| where d is the dimension of the activations of the layer. We then use a dictionary learning algorithm x with A components to decompose A into matrix U, E R and W E kg such that A₁ P 1. We refer U1 as the concept coefficient matrix and W1 as the concept basis. We repeat this process for M2, resulting in U2 and W2. Intuitively, each row of a concept coefficient matrix U encodes the contribution each concept vector in W the activation vector of a particular image in I.
U₁ W1.
To measure concept similarity between two models, we need to understand how each concept reacts to the same set of images. For example, if both networks have discovered a concept that reacts to the color red, they would both have an activation pattern that spikes when red objects are presented. Following this logic, we propose that if two concept vectors are encoding the same information, their concept coefficients over the same set of images should be correlated. To obtain a shared set of images, we take the union over the image sets Ii UIZ to form I". The proposals passed through the model Mi to produce A for the shared concept set. Given the concept basis W, (which is specific M1), re-compute Un over the shared set of images. We repeat this process for M2 to compute U2. In practice, we use a non-negative least squares solver, since the original coefficient matrices are non-negative (Appendix B
3.2 CONCEPT SIMILARITY
Here we address the following question: in a specific pair of layers, does Mj encode the same concepts as M2? In the following sections, we make use of Ai, A2, U, and U2 to compute the similarity between concepts encoded in Mj and My. We consider two different approaches that trade-off computational cost and error. In Appendix [.. we describe correlation based metric that has a low computational cost that we use to coarsely compare many layers of two neural networks.
[A.]
accurately measure similarity in single layer, we propose regression based metric similar to the strategy in (Li et al., (015) and the BrainScore (Schrimpf et al., 2018). which was originally introduced to compare artificial neural networks to biological neural networks. In (Li et al. 2015), the outputs of convolutional layer in one model are mapped to the outputs of a convolutional layer in another network using a sparse weight matrix. The prediction error between the predicted outputs and true outputs are used as metric for the similarity between the two layers. In BrainScore, for a set of n stimuli (e.g., images), the activations from a layer of the DNN are stored in matrix A E Baxd, where d is the dimensionality of the layer activations. For the same set of stimuli, neural recordings are measured from an animal and processed forming vector y E T for each target "neuroid". A linear mapping is introduced to predict the neural responses from the DNN activations, y Aw" + b, where w' are the weights of the regressor and b is the bias. The regression model is trained where the predicted outputs of the regressor are compared to the true neural responses using Pearson correlation, giving score between -1 and 1.
In order
set of training images and evaluated held out test images
To compare two neural networks using these methods, each column of A₂ would serve as prediction targets for regression with A₁ resulting in similarity score for each column of Ag. However,
4

---

Published as a conference paper at ICLR 2025
visualizing and interpreting each neuron results in an explanation that is too complex for users. Instead, a more interpretable result can be achieved by setting the coefficient matrix U2 as regression targets for A (Fig. D. Essentially, RSVC encodes similarity by measuring how well M1 predict the concept coefficients of M2 and vice-versa,
A A,Wi-2 = U1-2 and
A2 = U2-11
where we learn W* such that the following regularized (11) mean-squared error is minimized.
min
E (AW" - U)² 1/ W
(2)
I regularization guides the regression model to seek a sparse set of neurons in M1 that can be used predict U2 and reduces over-fitting the regression training data. We also compute baselines for each model from their own activation matrices, learning Wi-1 and W2-2- Finally, we compute the Pearson and Spearman correlation between columns the predicted coefficient matrix and columns of the true coefficient matrix to get similarity score for each concept between -1 and 1. We refer the score as cross-mode mosel concept similarity (CMCS) when computed across two models, and as same-model concept similarity (SMCS) when computed within the same model.
Finally, we investigate whether similar or dissimilar concepts are more important to model decisions by applying concept integrated gradients (Fel et al. 2023a). Concept integrated gradients measure the contribution of each concept to a model's decision (Appendix[B.I)
3.3 REPLACEMENT TEST
To link differences in predicted concept coefficients and real concept coefficients to model be- havior, we conduct "replacement test". For each model comparison, we conduct a replacement operation over each column of the coefficient matrix and keep track of the resultant reconstructed activation matrix, model log- its, and model predictions (as seen in the pseudocode on the left). We perform the replacement operation using the same model predicted coefficients U,++1 (baseline) and also the cross-model predicted coefficients U We measure the impact of replace- ment at three levels during classification task. We denote as the classification head that produces logits (z) for each input image. We compute the mean di-dint between each row of A and A the mean KL- divergence over the logits 2/-,1 and zi-,1 and the match accuracy between v and yi-,1-
1: for each i = 1 to K do
2:
Ü = Copy U1
3:
U -2 U2 12
4:
A/-11 =Ü; - W
5:
2,111 A(A 1
6:
F2-1 arg max 1)
7: end for
3.4 INTERPRETING Low SIMILARITY CONCEPTS
How should we visually compare predicted and real concepts? Concept based XAI methods like CRAFT (Fel el al. 20236) and CRP (Achtibat et al. 2023) visualize the n images with the largest concept coefficient proach can be used to visualize similar concepts, since, by definition, similar concepts have highly correlated activation patterns
representatives of the visual feature encoded by the concept. The same ap-
the shared set of concept proposal images. However, this ap- proach is misleading when low similarity concepts are discovered. When a concept is dissimilar it may share the same top n images, but have entirely uncorrelated coefficients over the remaining im- ages. This possibility is further amplified due to the mean squared error (MSE) loss used to estimate the regression matrix Wi-2- The MSE loss penalizes prediction error on the largest coefficients disproportionately, leading to a higher chance the two models share the same top n images.
To address this, we develop new approach to specifically visualize how two models are dissimilar with respect to a concept. We start by visualizing the target concept by using an image collage corresponding to the top n concept coefficients. The target concept is compared to the predicted concept by visualizing the top over-predicted coefficients and top under-predion This allows users to reason about visual features that one model entangles or disentangles with the target concept, improving their overall understanding of the compared models. To ensure sample diversity, we enforce that we visualize one patch per image. We also exclude the top- 10 real concept images when selecting the over and under predicted points. In Figs. and 5 demonstrate proposed approach for interpreting the dissimilarity of the concepts. We include more visualizations in Appendix A.2
coefficients.
5

---

Published as a conference paper at ICLR 2025
3.5 IMPLEMENTATION DETAILS
We choose a ResNet-18 (RN18), ResNet-50 (RN50) He et al. 2016), ViT-S, ViT-L (Dssovit- Skiy 2021), DINO ViT-B (DINO) Caron et al. 2021 and MAE ViT-B (MAE) (He et al., 2022) from the timm library (Wightman
2019) for our experiments. All models were trained on Ima- geNet (Deng et al. 2009). For our exploration with DINO and MAE, we finetune the models on NABirds (Van Horn et al. 2015). We compare four pairs of models: RN18 vs. RN50, VIT-S vs. ViT- L. RR50 vs. VIT-S, and DINO vs. MAE. Model performance on their respective datasets are reported in Tab. AT The first two pairs have a clear difference in performance, implying that there are sig- nificant representational differences between the models. The second two pairs are roughly equal in overall performance, allowing us to explore how representational differences may result in different behavior when overall performance is the same. For the I penalty, we sweep À on subset of data and find À = 0.1 to be a reasonable choice (Fig. (11). Additionally, we set the number of con- cepts i 10 to balance reconstruction error and computational cost (Fig- AA44). We provide further details on the concept extraction, concept comparison and computational cost in Appendix B
4 RESULTS
4.1 DISCOVERING A "TOY" CONCEPT
While understanding models on real data is closely related to real-world use cases, it results in complex concepts that can be more challenging to interpret. To better understand the properties of RSVC, design an experiment in which we train model on images modified with simple toy visual concept. In this experiment, we are able to control what the model learns and explore how RSVC works in more controlled setting.
We train two ResNet-18 18 models from scratch on all classes from a modified NABirds dataset VVn Horn al. 2015). The first model, Mps- is trained make use of the toy concept in its decisions and the second, Mne is trained to become invariant to the toy concept. The toy con- cept is 20px 20pz pink square that is stochastically placed on the images at a random location. For Mps. the concept appears on images from the Common Eider class with a 70% probability, giving the concept predictive power. For M the concept appears on images from any class with a 50% probability, giving the concept no predictive value. Thus, Mps should learn to attend to the visual concept while Mne should learn to ignore the concept, since fo models are tested on a dataset in which the concept appears on images from the Common Eider class with 1 100% probability. Provided that Mn.. successfully learns to ignore the concept, RSVC should have a low similarity score any concept in Mps that primarily fixates the pink square. Recall that concepts are visualized using image patches. This allows us to break the 100% correlation be- tween the pink square and the image during testing, such that only some patches contain the added concept. Both models achieve ~34% classification accuracy on the test set of NABirds.
is simply noise. Finally, both
In Fig. 2 visualize the similarity from Mnc to Concept 1 in Mps- We find that Mnc has near zero similarity to Concept in Mps. which we visually identify to be a concept that fixates on the pink square. Importantly, the modified training paradigm does not affect the similarity scores between other concepts in the two models. In Fig. [5] we show that Mnc has high similarity to two other concepts in Mps. Thus, we show that RSVC clearly identifies the primary conceptual difference between the two models in this controlled experiment.
4.2 CONCEPT SIMILARITY VS. CONCEPT IMPORTANCE
To analyze real data, we start by exploring the relationship between concept similarity and impor- tance in the penultimate layers of different models trained on ImageNet (Deng et al., (009). We compute the cross-model concept similarity (CMCS) and the concept importance (CI) for every extracted concept. In Fig. 3 we plot the concept similarity from Mi to M2 against the concept importance for My across four pairs of models. In this figure, a point with low similarity and high importance would indicate that in layer I, M1 M2 finds important in decision making. We use color to indicate the density of points in a region (warmer colors indicate more density). In Appendix A.7 we compare same-model concept simi-
not predict the coefficients of concept that

---

Published as a conference paper at ICLR 2025
Common Eider Concept 1 Mac
Mps
Sim: 0.02 Div: 0.07
Top-10 Concept Patches
0.50
0.45
Predicted Coeffs
0.40
0.35
pink squares overlayed and tails
Real Coeffs
Top-10 Overpredicted Concept Patches
Top-10 Underpredicted Concept Patches
over-sensitivity to water without pink square
under-sensitivity to pink squares
Figure 2: Adding and Discovering Toy Concept. Here we train two ResNet-18 models, Mys and Mne- Mas trained to associate a pink square (i.e., Concept 1) with the Common Eider class, while Mnc is trained to be invariant to the pink square concept. We find that the similarity score from Mne + Mps for Concept is 0.0, indicating that Mac unable to predict Concept from Mps. To understand various aspects of the differences between the two models, RSVC inspects three distinct regions of the predicted vs. real coefficient scatter plot (Sec. 3.4). (Green): RSVC visualizes images corresponding to the top-10 Mps target concept coefficients. This allows the user to under- stand what the target concept is encoding. This concept clearly reacts strongly to the pink square visual feature. (Blue): RSVC visualizes the image patches with the largest Mue under-predicted co- efficients. Mne under-reacts to the pink square when compared to Mps- (Orange): RSVC visualizes the image patches corresponding to the top- 10 Mne over- wer-predicted coefficients. The over ver-predicted patches show that Muc cannot distinguish between background and the pink square.
larity (SMCS) to CL. As expected, we find that SMCS values are significantly higher than CMCS values, since the model is predicting its concepts.
In cross-model comparisons, we observe that models tend to have medium/high similarity for most concepts, since dense regions in the plot tend to be above 0.6 similarity. We also notice that the similarity scores from ViT's and ResNets have a different overall structure, with ResNets having a longer tail of low similarity and low importance concepts. Finally, except for DINO vs. MAE find that there are several low/medium similarity concepts that also have a medium/high importance. In Appendix/A.S) we systematically vary the training protocol (seed and data) for ResNet- 18 model and measure the impact of the changes on model similarity. We find that changes in model training lead to intuitive changes the two models differ (Fig. [A)). In summary, we observe two key results: (1) model differences are largely driven by medium similarity, medium importance concepts, jointly contributing to significant changes in model behavior and (2) some models do learn "unique" low similarity, high importance concepts. In the following sections we explore both of these results further.
similarity and use RSVC to reveal some concepts that suggest how
4.3 REPLACEMENT TEST
In order to better understand how variations in similarity impact model behavior conduct placement test (described Sec. 3.3). This test allows us to measure how changes in concept sim- ilarity impacts the lg-distance of the activations, KL-divergence of the logits, and match accuracy of the predictions. We investigate this question because is possible that changes in Pearson corre- lation due changes in predicted coefficients unimportant images for particular concept,

---

Published as a conference paper at ICLR 2025
ResNet-18
Res 50
0.3
05-
12
0.0
0.2
0.4
10
8
ResNet-50 - ResNet-18 Concept Similarity
10.0
VIT-5
ResNet-50 Concept Similarity
VIT.S ResNet-50 CI
75
85
sn
VIT.S
88
0.0
0.8
0.4
Us
0.8
1.0
0.0
02
0.4
0.6
0.8
1.0
ResNet-18 18
ResNet-50 Concept Similarity
ResNet-50 VIT-S Concept Similarity
1.0
C
DINO 5483
0--
10
VIT-L VIT-S Concept Similarity
MMEE s483 + DINO_ 5483 Concept Similarity
-
1.0
MAE 5483 CI
VIT-L CI
0.1
0.5
0.0
0.0
06 0.6
1.0
1.0
VIT-S VIT-L Concept Similarity
DINO 5483 MAE_ $483 Concept Similarity
Figure 3: Concept Similarity vs. Concept Importance. We compare four pairs of models using
CMCS: (A) RN18 vs. RN50, (B) RN50 vs. VIT-S, (C) ViT-S vs. ViT-L, and (D) DINO vs. MAE. The y-axis represents the concept importance (CI) measured using concept integrated gradients. Warmer colors represent the density of points in a region. We highlight several regions in the plots: (1) low similarity and low importance concepts that are unique model but contribute little to its decisions, (2) high importance and high similarity concepts that are shared across both models and also contribute greatly to decision making,
(3) low similarity, high importance concepts that only model has discovered, but are very important to that model's 's decisions.
leading to no change in model behavior. In Fig. 4 we visualize the change in Pearson correlation (APearson) against the change the three aforementioned metrics. We color indicate con- cept importance (warmer colors are more important). As expected, we observe trend showing that ly-distance increases as similarity decreases. For the KL-divergence, when the importance is sufficiently high, the KL-divergence increases similarity decreases and (2) when the importance is low, there no effect on the model's logits. Finally, we observe a trend that shows that model predictions change as function of both similarity and importance. We find
observe two trends: (1)
that these trends roughly hold for models, although the structure of the plots changes for ViTs. See Appendix A.8 for more results.
4.4 Low SIMILARITY CONCEPTS
In Fig. we observe that model comparisons identify low similarity, high importance concepts. These concepts are particularly interesting because they identify visual features that one model has constructed that the other has not. In Fig. 5 we apply our proposed approach for understanding the dissimilarity between predicted and real concepts (see Sec. 3.4). We analyze a RN50 concept used on the barbell class, which primarily reacts to images containing hands/arms lifting a barbell. When regression model is trained the VIT-S activations to predict the coefficients of the RN50 concept, the model becomes over-sensitive to any image with weight plates or gym equipment and remains under-sensitive images of people around gym equipment. This result suggests that the ViT-S does not have a feature encoding "hands lifting weights". In Fig. A3 we show some more examples of important low similarity concepts that visually interpretable.
8

---

Published as a conference paper at ICLR 2025
ResNet-18 ResNet-50
0.300
0.4
0.075
MatchAce
0.00
0.8
0.6
0.050
-0.05
0.2
-0.10
0.0
0.000
0.0
-0.4
-0.2
0.0
A. Pearson
A Pearson
Pearson
ResNet-50 ResNet-18
0.25
0.8
0.20
A MatchAcc
0.00
0.8
0.15
00.05
0.6
0.4
-0.30
0.2
-0.15
02
0.00
0.0
-0.4
-0.2
0.0
-0.4
-0.2
0.0
-0.4
-0.2
Pearson
Pearson
A Pearson
Figure 4: Replacement Test. We determine whether poorly predicted coefficients for concepts tually impact model behavior (Sec. 4.3p. We use color to represent the concept importance (warmer higher importance). When ignoring low importance concepts, we observe expected trends, i.e., decreases in similarity (APearson) result in increases in the 12-distance, increases in KL-divergence on the classifier logits, and decreases in model accuracy. The effect also seems to be scaled by importance, for example, changes to low importance concepts (black) has no impact on AKL.
4.5 LLVM GENERATED CONCEPT DIFFERENCE ANALYSIS
Fig. contains manual analysis of the difference between visual concept for the barbell class from two different models. Here we use an LLVM (ChatGPT-44) Achiam et al. (2023)) to analyze image collages and describe them. This approach has similar goal to Gonndellnnan al. (2024). in which the authors use an LLVM to interpret specific MLP heads in a CLIP ViT model. The input and output structure for this experiment provided in Fig. A4 We do emphasize differences in the system prompt, instead we ask the model to describe both similarities and differences. We provide the top-k image collage (IC1) and the over- verrppeeiccted image collage (IC2) as input to the LLVM. The LLVM outputs for the barbell concept are presented below. We observe that the LLVM description similar the manual description in the caption:
LLVM Analysis of Visual Concept from the Barbell class
ICI: collage of people engaging in strength training exercises, with a focus on lifting dumb-
bells, barbells, and performing weightliting movements.
IC2: A collage showcasing gym equipment and weights, with some emphasis on exercises involving barbells and dumbbells but with fewer people depicted.
Similarity: Both collages center on weightlifting and gym-related activities, featuring equip- and exercises.
Difference: IC1 highlights people actively performing weightlifting exercises, while IC2 pri- marily focuses on the gym equipment and setup with minimal human involvement. Semantically different: [Yes] IC' emphasizes the activity of weightlifting whereas IC2 fo- cuses more on the tools and environment of the gym.
LIMITATIONS
We limit our analysis to computer vision models. In practice, there is nothing restricting the ap- plication of the RSVC approach to large language models, but we leave this for future work. Our work builds on concept-based XAI methods, and thus inherits the trade-off between fidelity and varies for each model, which
interpretability. In particular, note that the reconstruction may explain some properties of concept similarity in our experiments. However, we are agnostic to the precise concept extraction method used and thus approach will benefit from further ad-

---

Published as a conference paper at ICLR 2025
Barbell| VIT-S ResNet50
Sim: 0.64 KL-Div: 0.08
(IC 1)
Top-10 (R50) Concept Patches
Predicted Coeffs
0.3
Tos
0.0
hands and arms lifting barbell
0.2
0.6
Real Coeffs
(IC 2) Top-10 (VIT-S) Overpredicted Concept Patches
Top-10 (VIT-S) Underpredicted Concept Patches
ol
over-sensitivity to weight equipment
under-sensitivity people gym equipment
Figure 5: Interpreting Low Similarity Concepts. In this example, we find a RN50 concept for barbell class that the VIT-S is not able predict. (Green): The RN50 concept reacts to images of hands lifting barbells. Additionally, many images contain vertical supports for squat rack. We train regression model the VIT-S activations to predict the RN50 concept coefficients, (Blue); The VIT-S regression model under-reacts to images containing hands, people, and squat racks. (Orange): It over-reacts to images that have greater focus weight plates. These results suggest that the the specific concept of hands lifting barbells is not represented in the ViT-S, In Sec. [.5) we use an LLVM analyze the image collages (IC1 and IC2) and find that it detects similar differences in the visualizations.
vances in these methods. For example, incorporating a recursive strategy like the one presented in CRAFT (Fel et al. 2023b), may significantly improve the number of interpretable comparisons dis- covered. Outside overly artificial settings whereby two models are trained on completely different datasets, we note that it can be challenging to compare the representations of two models trained on same or similar datasets (e.g., ImageNet) as done this work. However, we believe that this more challenging setting is of most interest and relevance.
CONCLUSION
We introduced a new method for exploring representational similarity via interpretable visual con- cepts (RSVC). In contrast to existing representational similarity methods that simply provide single numerical score to denote similarity, our approach fuses ideas from concept- based explainability and shows what visual concepts make two models similar dissimilar. In particular, we demonstrate that comparing models can be an effective path towards understanding what concepts a model is missing. In future, this may be helpful in identifying sources of model failures. We presented exper- iments on a range different vision models and demonstrated that our approach is general and can be applied across variety of different backbone models, irrespective of the pretraining objective used by the model. Finally, we suggest that explaining the functional differences in two models' behavior could serve valuable testhed for future XAI research. We hope that work opens the door to further investigation into how concepts are represented inside of deep networks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
NK and PP were supported by the Simons Foundation and the Resnick Sustainability Institute. OMA was supported by a Royal Society Research Grant. The authors thank Rogério Guimares, Markus Marks, Atharva Sehgal, and the anonymous reviewers for their valuable feedback.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv:2303.0877 2023.
Reduan Achtibat, Maximilian Dreyer, Ilona Eisenbraun, Sebastian Bosse, Thomas Wiegand, Woj- ciech Samek, and Sebastian Lapuschkin. From attribution maps to human-understandabl expla- nations through concept relevance propagation. Nature Machine Intelligence, 2023.
Jannis Ahlert, Thomas Klein, Felix Wichmann, and Robert Geirhos. How aligned are different alignment metrics? In ICLR Workshop on Representational Alignment (Re-Align), 2024.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role individual units deep neural network. PNAS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. In ICLR, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Chris HQ Ding, Tao Li, and Michael Jordan. Convex and semi-nonnegative matrix factorizations. PAMI, 2008.
Alexey Dosovitskiy, An image is worth 16x16 words: In ICLR, 2021.
Transformers for image recognition at scale.
Amil Dravid, Yossi Gandelsman, Alexei Efros, and Assaf Shocher. Rosetta neurons: common units in model zoo. In ICCV, 2023.
Mining the
Sunny Duan, Loic Matthey, Andre Saraiva, Nicholas Watters, Christopher P Burgess, Alexander Lerchner, and Irina Higgins. Unsupervised model selection for variational disentangled represen- tation learning. In ICLR, 2020.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disen- tangled representations. In ICLR, 2018.
Cian Eastwood, Andrei Liviu Nicolicioiu, Julius Von Kügelgen, Armin Kekić, Frederik Träuble, Andrea Dittadi, and Bernhard Schölkopf. Dei-es: An extended disentanglement framework with connections to identifiability. In ICLR, 2023.
Thomas Fel, Lucas Hervier, David Vigouroux, Antonin Poche, Justin Plakoo, Remi Cadene, Math- ieu Chalvidal, Julien Colin, Thibaut Boissin, Louis Bethune, Agustin Picard, Claire Nicodeme, Laurent Gardes, Gregory Flandin, and Thomas Serre. Xplique: A deep learning explainability toolbox. Workshop on Explainable Artificial Intelligence for Computer Vision at CVPR, 2022a.
Thomas Fel, Ivan F Rodriguez Rodriguez, Drew Linsley, and Thomas Serre. Harmonizing the object recognition strategies of deep neural networks with humans. NeurIPS, 2022b.
Thomas Fel, Victor Boutin, Louis Béthune, Rémi Cadène, Mazda Moayeri, Léo Andéol, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. NewrlPS, 2023a.

---

Published as a conference paper at ICLR 2025
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. CRAFT: Concept recursive activation factorization for explainability. In CVPR, 2023b.
Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting CLIP's image representation via text-based decomposition. In ICLR, 2024.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. NeurIPS, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Harold Hotelling. Relations between two sets of variates. In Biometrika, 1936.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. In ICML, 2024.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Data- models: Predicting predictions from training data. In ICML, 2022.
Margot E Kaminski and Jennifer M Urban. The right to contest ai. Columbia Law Review, 2021.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TACV). In ICML, 2018.
Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. arXiv:2305.06329, 2023.
Neehar Kondapaneni, Markus Marks, Oisin Mac Aodha, and Pietro Perona. Less is more: Discov- ering concise network explanations. In ICLR Workshop on Representational Alignment, 2024.
Mauritz Kop. Eu artificial intelligence act: the european approach to ai. Stanford-Viest Transat- lantic Technology Law Forum, Transatlantic Antitrus and IPR Developments, 2021.
Simon Korbllith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICML, 2019.
Matthew Kowal, Achal Dave, Rares Ambrus, Adrien Gaidon, Konstantinos G Derpanis, and Pavel
Tokmakov. Understanding video transformers via universal concept discovery. In CVPR, 2024.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fine-graine grained categorization. In ICCV Workshops. 2013.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2008
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Out-of-distribttion ro- bustness via disagreement. In ICLR, 2023.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hoperoft. Convergent learning: Do different neural networks learn the same representations? In International Workshop on Feature Extraction: Modern Questions and Challenges at NeurIPS, 2015.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations. In JCML, 2019.
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. NeurlPS, 2017.
12

---

Published as a conference paper at ICLR 2025
Mathurin Massias, Alexandre Gramfort, and Joseph Salmon. Celer: fast solver for the lasso with dual extrapolation. In ICML, 2018.
Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert Vandermeulen, and Simon Kornblith. Human alignment of neural network representations. In ICLR, 2023.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learn- ing? NeurlPS, 2020.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In ICLR, 2021.
Young-Jin Park, Hao Wang, Shervin Ardeshir, and Navid Azizan. Quantifying representation relia- bility in sulf-super learning models. In UAI, 2024.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn Machine learning in Python. JMLR, 2011.
Eleonora Poeta, Gabriele Ciravegna, Eliana Pastor, Tania Cerquitelli, and Elena Baralis. Concept- based explainable artificial intelligence: A survey. WrXiv:2312112936 2023.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamies and interpretability. NeurIPS, 2017.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? NeurlPS, 2021.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should trust you?" Explaining the predictions of any classifier. KDD, 2016.
Robin Rombach, Patrick Esser, and Bjorn Ommer. Network-to-netwon translation with conditional invertible neural networks. NeurlPS, 2020.
Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentangle-
ment of correlated factors via hausdorff factorized support. In ICLR, 2023.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias Issa, Ko- hitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, al. Brain-score: Which artificial neural network for object recognition is most brain-like? bioRxiv, 2018.
Jessica Schrouff, Sebastien Baur, Shaobo Hou, Diana Mincu, Eric Loreaux, Ralph Blanes, James Wexler, Alan Karthikesalingam, and Been Kim. Best of both worlds: local and global explana- tions with understdadde concepts. arXiv:2106.08644 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient-based local- ization. IJCV, 2020.
Harshay Shah, Sung Min Park, Andrew Ilyas, and Aleksander Madry. Modeldiff: A framework for comparing learning algorithms. ICML., 2023.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 1996.
Grant Van Hom, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pictro Perona, and Serge Belongic. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In CVPR, 2015.
Ross Wightman. Pytorch image models.
https://githai com/ rwightman./////
pytorch- mage models 2019.
13

---

Published as a conference paper at ICLR 2025
Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets masked image modeling. In CVPR, 2023.
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. In- vertible concept-based explanations for enn models with non-negative concept activation vectors. In AAAI, 2021.
Wentao Zhang, Jiawei Jiang, Yingxia Shao, and Bin Cui. Efficient diversity-driven ensemble for deep neural networks. ICDE, 2020.
14

---

Appendix
Additional Experiments
15
A.1
Layerwise Concept Similarity
15
A.2
Analyzing Concepts
17
A.3
LLVM Generated Concept Difference Analysis
19
A.4
Specificity RSVC on the Toy Concept Experiment
A.5
Varying ResNet- 18 Training
22
................................
A.6
DINO and MAE Seed Variation Experiments
25
A.7
Same Model Concept Similarity vs. Concept Importance
25
A.8
Additional Replacement Tests
25
Additional Implementation Details
29
B.1
Concept Extraction
29
B.2
Concept Similarity
30
B.3
Computational Cost
A
ADDITIONAL EXPERIMENTS
A.1
LAYERWISE CONCEPT SIMILARITY
A.1.1 CORRELATION
We propose correlation based comparison method that has lower computational cost than the regression method presented in the main text (Sec. 3.2p. This approach allows us to efficiently compare many concepts and layers for two models. Recall that each column of U contains the concept coefficients of a specific concept for each image. If the concepts encode the same informa- tion, they should have highly correlated activation patterns since they would react similarly to same proposal images. Thus, compute the correlation for each vector ui E Columns (U1) and uz Columns(U2) We measure both Pearson and Spearman correlation to form the correlation matrices R E k and R R Since concepts extracted from each network do not have a direct correspondence, MCS measures the concept similarity between a concept from Mj and all of the concepts from M2 and keeps the maximum. We compute the maximum concept similarity (MCS*) over each dimension of the correlation matrix for each concept and class
MCS max R.g and MCS; max Rig. J
MCS is fast to compute and can be done with relatively few image samples (Appendix B.2). Thus, we use to compute layerwise concept similarity which gives us coarse-graine sarse--rained insights into the compute
similarity between each layer of two different models. For layerwise concept similarity, the correlation matrix R between Mj and M2 at each pair of layers for every class c. Then, for each correlation matrix we compute the mean maximum concept similarity (MMCS) over each dimension and then take the average of the two matrices
1
MMCS,,
MCS m
K
MMCS = (MMCS) + MMCS.)/2.
However, correlation based similarity can be affected by confounds from concept extraction. For example, extracted concepts can entangle or disentangle visual features that are encoded by the respective models. Suppose U, encodes features for both ears and snouts of a dog together in a single concept, but these two features are disentangled into two concepts U2, the maximum match between these concepts would be lower even though both networks are encoding the same information. Additionally, extracted concepts do not contain all the information in the network since there is some reconstruction error when learning the decomposition. Thus, correlation matrices tell us when two concepts are highly similar, but do not tell us a concept is missing in layer. However, they can
noisy lower bound for measuring concept similarity (Appendix A.1.3
15

---

Published as a conference paper at ICLR 2025
ResNet-18 vs, ResNet-50 MMCS
ResNett 18 Layers
VIT-S ResNet-50 MMCS
Layor
10
10-
04
2.4
20
20
30
40
ResNet-50 Layers
ResNet-50 50 Layers
VIT.S VTT MMCS
MAE vs. DINO MMCS
0.45
0.50
10-
Layers
0.40
IAAE Layers
0.15
0.35
0.40
0.30
0.35
0.25
0.30
0.70
Layers
DINO Layers
Figure A1: Layerwise Mean-Max Concept Similarity. We compare four pairs of models across many selected layers using Pearson correlation. Each entry in the matrix is the mean maximum concept similarity (MMCS) between M1 and M2 at a particular pair of layers. Brighter colors represent higher MMCS values. We that, in general, concept similarity is highest in carlier layers and decays as networks get deeper. We also notice that there is a slight increase in similarity towards the final layers (Appendix A.1
A.1.2 COMPARING MODEL LAYERS
In Fig. AT we explore how concept similarity arises across many layers of each network. We compute the MMCS each pair of layers and visualize the resulting matrix. For all models, find that concept similarity is highest at the early layers of each model and decays gradually as network depth increases. In all model comparisons, final layers relative to the preceding layers, suggesting that the way networks organize information converges the network get closer to producing a final decision. Interestingly, (Fel al. 2022b) also found the last layer to have better properties for concept extraction.
slight increase in similarity towards the
We also notice several properties unique to each comparison. When comparing different sized mod- els with the same architecture, the similarity is related to the relative depth of the layer. For ResNets, we notice that matrices show pattern increased similarity after residual blocks and lower simi- larity for layers within blocks. For the ViT-S and VIT-L we find that there is broad band of concept similarity in the middle of each network in which the ViT-S layers through 9 have higher similar- ity to ViT-L layers 5 through 20. In addition, there is increase in concept similarity between the last layer of the ViT-S and the last three layers of the ViT-L. When comparing the RN50 to VIT-S find that concept similarity of layers 0 through 25 are most similar layers 0 through 3. This finding matches observations found in previous work, in which relatively earlier layers in the ViT match relatively later layers in the ResNet (Raghu et al. 202T). Finally, when comparing the MAE to DINO, we see high concept similarity hetween the first 3 layers of DINO and the first 8 layers of the MAE. However, this similarity decays significantly as the layer index of DINO increases. This divergence in concept similarity may be due to differences between supervised and self-supervised training, but further research is needed.
A.1.3 COMPARING MCS (PEARSON) TO LASSO REGRESSION (PEARSON)
In Appendix A.I.I we claimed that MCS (Pearson) could serve as noisy lower bound to better measurements of similarity like CMCS (Pearson). This because MCS is computed over concepts extracted by the decomposition method, which introduces its own entanglements, disentanglements, and reconstruction error. Fig. A2 we compare the penultimate layers the four pairs of models and compute both MCS (Pearson) and the CMCS (Pearson). We find that MCS is correlated to CMCS, but, as expected, under peedicts the concept similarity. For this experiment only, due to the fact that computed Pearson correlation over images in the training set of the deep neural network, we compare to the mean similarity score that is computed from the held-out folds of the five lasso regression models. The held-out folds not part of the training set for the regression models, but
16

---

Published as a conference paper at ICLR 2025
part of the training set of the deep neural network. However, this comparison is more fair, since we compare the two methods on the same set of images.
Reshett19 18
ResNict 50
Reshet 50 VIT.5
VIT.S VIT
DINO MAE
1.
Lao0 Regression (Pearsnn)
Laso Regression (Poseton)
Lsop Regression (Penrzant
Leco Regression (Pesrran)
MCS (Pearson)
MCS (Pearson)
MCS (Pearson)
MCS (Pearson)
Reshet-50
VIT-S ResNet-
VIT-L
MAE DINO
Lasgl Regression (Poirson)
Lasso Regression (Porson)
Lasso Regression (Perroon)
Lanso Regression (Peirson)
(Pearson)
(Pearson)
(Pearson)
(Pearson)
Figure A2: MCS (Pearson) vs. Lasso Regression. We see that the most points lie above the red- line. This means that lasso regression (followed by Pearson correlation on the predicted and real coefficients) usually predicts a higher similarity value than the MCS values directly on the columns of the coefficient matrix. Thus, we experimentally validate that the Pearson correlation acts noisy lower bound on concept similarity.
A.2 ANALYZING CONCEPTS
In Fig. A3 we display some more examples of dissimilar concepts found through RSVC. We also provide interpretations of the dissimilarities between the concepts. Note that these dissimilarities do not necessarily indicate worse performance on images from this class because there are many possible correct strategies when trying to make decision about an image.
Rugby Ball RNI8 + RNSO
The RN50 has learned a visual concept that entangles the arms of a rugby player and the rugby ball. We see in the under-predicted samples that the regression model under-predicts samples with the players' limbs and rugby balls together. When visualizing the over-predicted samples we can see that the regression model increases sensitivity to both close-ups of rugby balls and the legs of the players. These results suggest that the RN18 encodes for the legs of the rugby players independently of the rugby ball and the regression model is using these independent features to try and reproduce the RN50's concept. It also suggests that the RN18 is not encoding the arms as an independent feature.
Grey Whale VIT-S + VIT-L
It appears that the VIT-L has learned visual concept for whales surfacing parallel to the surface of the water. We can see that the regression model under-predicts images of the whales back and also images of its eye in horizontal orientation. The regression model seems to have increased sensitivity to breaching whales, either raising their tails or their heads. This suggests that the ViT-S has entangled calm whales floating the surface with active whales breaching the water.
Strawberry RN50 + VIT-S
This concept is one of the lowest similarity concepts that causes a meaningful change in model behavior (as seen by the KL-Divergence). We can that the ViT-S has leamed concept for mixed fruits that include strawberries. The under-predicted and over over-predicted samples show that the RN50 has ability to reproduce this pattern, suggesting that it ignores mixed fruits entirely.
17

---

Published as a conference paper at ICLR 2025
Rugby Ball
ResNat1 ResNet50
Sim 0.85 RL-Bos
M
Presictes
06
2.8
Real Coeffs
Grey Whale VIT-S - ViT-L
Presteted
Resl Coefts
Strawberry ResNet50 - ViT-S
Sim
Fredicses Cootts
Beal Coaffs
Volleyball ResNet18 - ResNet50
Sim 0.65 d--W 0.05
Predicts Coeffe
60
Bluejay ViT-B MAE ViT-B DINO
Sim
Predlcted Coofts
0-4
Coeffs
Top-9 Concept Patches
Overpredicted
Underpredicted
Figure A3: Qualitative Samples. In each row, we show visualizations for selected concepts from different model comparisons. In the first column of each row, we show scatter plots between real and predicted concept coefficients. Colored points mark the top-9 images in different subregions of the scatterplot. Each subregion indicates different aspect of dissimilarity. (Green): Top-9 images the real concept. These images are used to help the user understand what the target concept pays attention to. (Orange): Top-9 images that are over-predicted by the contrasted model. You may need to zoom in to best analyze the image grids. (Bluc): Top-9 images that are undcrpredicted by the contrasted model. We discuss possible interpretations of the concepts in Appendix A.2] See Sec. 3.4 for a detailed breakdown of how to interpret these plots.
18

---

Published as a conference paper at ICLR 2025
Volleyball RN18 - RN50
The RN50 has learned to encode volleyball players in variety of active positions. There seems to be an emphasis on hands and arms near or above the net. The RN18 under-predicts ulose-up images of volleyballs and players at the net and it tends over-predict images containing balls high in the air and close-ups of nets/grids. This suggests that the regression model is trying to reproduce the be- havior of the RN50 concept using variety of related features learned by the RN18. We explore the top-6 neurons that most contribute to the regression model's prediction of this concept in Fig. A6 We compute the neuron contribution using permutation feature importance (Appendix [B) We find that the regression model uses RN18 neurons that are highly sensitive to volleyballs in the air, play- ers near nets, and close-ups of nets qualitatively explaining some of the differences between the predicted and real concept.
Bluejay VTT-B MAE > VIT-B DINO
The DINO model has isolated the tail of the bluejay as an important visual feature. The regression model under predictss images with bluejay tails and over weer-predicts random images, suggesting that this concept is not independently encoded in the feature space of the MAE
A.3 LLVM GENERATED CONCEPT DIFFERENCE ANALYSIS
In Sec. 4.5 provide an example that uses an LLVM for analyzing the image collages produced by RSVC. In Fig. A4 we describe the procedure in detail. We apply this method to the concept comparisonsin Fig. A3 and report the results below. We find that the LLVM annotations fairly similar to the manual annotations.
Rugby Ball RN18 RNS0
ICI: A collage featuring various close-ups of players holding rugby balls during gameplay, empha- sizing hands and ball contact.
IC2: A collage with mixed shots of rugby gameplay, focusing on players' lower bodies, shorts, and action involving the rugby ball from different perspectives.
Similarity: Both collages depict scenes from rugby games and include elements of players and rugby balls in action.
Difference: ICI focuses on hand interactions with the rugby ball, while IC2 emphasizes lower body dynamics and broader gameplay elements.
Semantically different: [Yes] ICI highlights ball handling, whereas IC2 portrays physical move- ment and positioning.
Grey Whale VIT-S + VIT-L
ICI: A collage featuring close-up views of whales, focusing on surface water interaction and de- tailed textures of their bodies.
IC2: A collage showing whales from various perspectives, including body parts like tails, heads, and fins, with more diverse settings and details. Similarity: Both collages depict whales and highlight different parts of their bodies in aquatic envi- ronments.
Difference: ICI emphasizes close-up textures and water interaction, while IC2 provides broader views of whales, showing distinct body parts and their environment. Semantically different: [Yes] IC1 focuses on textures and surface detail, while IC2 captures wider perspective of whale features and behaviors.
Strawberry RN50 + VIT-S
ICI: A collage featuring various fruits, with focus on strawberries along with other fruits like grapes, blueherries, and kiwi, emphasizing vibrant colors and diversity. IC2: A collage entirely centered on strawberries, showing detailed close-ups of their textures and seeds in different lighting and angles.
Similarity: Both collages prominently showcase strawberries and their textures. Difference: ICI includes a mix of other fruits alongside strawberries, while IC2 exclusively focuses on strawberries and their intricate details.
Semantically different: [Yes] ICI presents a broader context of mixed fruits, whereas IC2 isolates strawberries as the sole subject.
19

---

Published as a conference paper at ICLR 2025
going you analyze image collages. You will receive image collages and your task is to: (1) provide one sentence to describe each image collage. (2) Provide one sentence to image collages
similarities between the imaae collages, (3) Make judgement whether the two collages convey different semantic information and summarize it in few words. The format of your output should be organized follows:
sentence
differences
System Prompt
ICT: desc IC2: desc
Similarity: desc Difference: desc
Semantically different (Boolean Answer) desc
201
Input
Analysis
IC1: collage featuring close-ups players holding rugby balls during gamepley, emphasizing hands and ball contact.
IC2: collage with mixed shots of rugby gemeplay, focusing on players' lower bodies, shorts, and action involving the ball from different perspectives.
Output
Similarity: collages depict rugby balls action.
players Difference: IC1 focuses on hand interactions with the rugby bail,, while IC2 emphasizes lower body dynamics and broader gameplay elements.
rugby games
Semantically different:
highlights handling, movement and positioning.
portrays physical
Figure A4: LLVM Concept Difference Analysis. We use ChatGPT-40 (Achiam et al. 2023
analyze concept differencee. We provide a general system prompt asking the LLVM to describe the collages, provide a description of the similarities and differences between the collages, and provide final judgement on whether there example for the rugby ball class. The LLVM receives the image collage (IC) corresponding the top-k concept images (IC1) and the over-predicted concept images (IC2), see Fig. more details on image collages. Here, the LLVM notices that IC1 focuses on hands and ball contact, whereas the IC2 focuses more lower bodies. The described difference matches the manual annotation provided in Appendix A.2
semantic difference between the collages. We provide an
Volleyball RN18 + RN50
ICI: A collage depicting volleyball gameplay, focusing on players' hands, the volleyball, and inter- actions around the net during eetkon-packed moments.
IC2: A collage showing volleyball gameplay well but with wider variety of scenes, including players near the net, different environments, and more casual setups.
Similarity: Both collages revolve around volleyball and the central theme of gameplay near the net. Difference: ICI emphasizes action shots with players and the ball in dynamic moments, while IC2 includes mix of gameplay contexts and less action-oriented scenes.
Semantically different: [Yes] ICI focuses on the intensity and dynamics of volleyball action, whereas IC2 portrays broader variety of volleyball scenarios and contexts.
20

---

Published as a conference paper at ICLR 2025
Bluejay ViT-B MAE ViT-B DINO
IC1: A collage of close-up views of blue jay feathers, primarily showcasing the distinctive blue and black patterns on their wings and tails.
IC2: A collage of blue jays in various settings, including full-body shots, close-ups of their heads, and natural habitats like branches and foliage.
Similarity: Both collages feature blue jays and focus on their unique physical features and natural environment.
Difference: ICI highlights detailed patterns on blue jay feathers, while IC2 captures broader context of the bird's appearance and behavior in its habitat. Semantically different: [Yes] ICI emphasizes feather patterns, whereas IC2 provides a holistic view of blue jays in their environment.
Common Eider Concept 2| Mac Mps
Sim:: 0.94 KL-Div: 0.00
Top-10 Mps Concept Patches
Predicted Coeffs
1.25
3.00
0.50
0.0
0.5
1.0
1.5
black and white color pattern of Common Eider
Real Coeffs
Common Eider Concept 3 Mиn
Mps
Sim: 0.95 KL-Div: 0.00
Top-10 Mp Concept Patches
2.0
Predicted Coeffs
1.5
1.0
0.5
light ocean background
0.5
1.0
2.0
Real Coeffs
Figure A5: Specificity of Toy Concept. In Fig. we showed that Mae able predict the pink square concept from Mps. In this figure, we show that the toy concept does not impact the similarity between other concepts learned by the networks. We visualize the top-10 patches from Concept 2 and Concept 3 of Mps in the same class (Common Eider). These concepts correspond to the white and black color pattern of the bird and water background. Note that these models have been trained from scratch on NABirds resulting in relatively low 34% accuracy. This leads to noisier concepts that are more challenging to interpret. Importantly, we can see that Mne still has very high similarity score for these two concepts, highlighting the specificity of RSVC.
A.4 SPECIFICITY OF RSVC ON THE ToY CONCEPT EXPERIMENT
In Sec. 4.1 we showed how RSVC can be used to distinguish between a model trained to use the pink square concept and model trained to ignore the pink square concept. However, training with the pink square concept could have undesirable effects on other model concepts for the Common Eider class. In Fig. A5 show two shared concepts that unaffected by the pink square. This experiment suggests that our toy concept training procedure does not impact other concepts learned by the networks.
21

---

Published as a conference paper at ICLR 2025
Volleyball ResNet18 - ResNet50
Sim: 0:68 KL-Drw: 0.0466
04-
Predicted
0.3
0.
0.1-
Real Coeffs
ResNet1 8 Neurons
Neuron 465 Imp: 0.10
Neuron 340 Imp: 0.08
Neuron 297 Imp: 0.05
Neuron 33 Imp: 0.02
Neuron 203
Imp: 0.02
Neuron 355 Imp: 0.01
Figure A6: Neuron Analysis For Volleyball Concept Difference Fig. A3 we visualized RN50 concept for the volleyball class that the RNI8 did not contain. In this figure, we explore the top-6 neurons used by the regression model to predict the RN50 concept. We find that Neuron 465 is sensitive to edges between volleyball net and the background. It seems to mistake some grid-like textures for nets as well (image [1, O]. [1, 1], and [2, 0)). In addition, it seems be sensitive to volleyballs high in the air. Neuron 340 seems to activate for athletes in indoor gyms and seems partial to lower bodies. Neuron 297 is sensitive to close-ups of nets with hands or arms in the frame. In summary, these neuron visualizations help to explain some of the images over-predicted by the regression model.
A.5 VARYING RESNET-18 18 TRAINING
Next, we conduct controlled variations model training and measure how effects model concept similarity in the last layer. We train ResNet-18 18 model on variations of the NABirds dataset.
We start by comparing two ResNet-18 models trained on NABirds with different seeds, 4834586 (R18 18 s483) and 87363356 (R18 s873). We find that, despite varying the seed during training, both
22

---

Published as a conference paper at ICLR 2025
NAB R18 (s483)
NAB+SC R18
Sim. C:06 K0-30 0.003T
Sem: 0:33
Roal Cooffe
NAB+SC R18 - NAB R18 ($483)
1.79
0.68
Rcal Cooffs
Ei-iic
1.79
Prddccdd
5
Resl Coeelt
Ser: 0:34 2.0037
2.0
UN
BALEYY
Protico
Roal
Top-9 Concept Patches
Overpredicted
Underpredicted
Figure A7: Impact of Training on Stanford Cars. each row, we show visualizations for selected concepts from comparing R18 NAB+SC to R18 s483. In the first two rows, we visualize two R18 NAB+SC concepts that RI8 s483 cannot reproduce. The first concept associated with the Shelby Mustang. The R18 s483 model appears to sometimes entangle this concept with blue color, irrespective the car model. The second concept appears to be common features associated with Mercedes more abstract and challenging to interpret. We visualize NAB R18 s483 concepts the next three rows. First, show R18 $483 concept that R18 NAB+SC is unable predict. We see that this concept is very abstract without clear pattern, but is generally related to sandy textures. Next, we visualize two car-related concepts from RI8 s483. We find that these concepts are sensitive to the combination of the presence of car and specific color. For the orange car concept, the R18 NAB+SC makes small over-predice predprressssoos with different shades of orange. For the yellow car concept, the ever-predicted group shows different shade of yellow and specific style of car. A discussion of these results available in Appendix A.5
racing stripe that is
For this concept, the difference between the two models is
23

---

Published as a conference paper at ICLR 2025
models discover highly similar concepts (Fig. A8A). Then, we train a ResNet-18 model on a mod- ified version of the NABirds dataset in which waterbirds (169 classes) have been excluded during training (R18 NAB-WB). After training, the backbone is frozen and just the classification head is trained on the full dataset. We compare this model to R18 s483 trained on the full dataset (34%). We find that training without waterbirds results in a significant decrease in performance (25%) and, surprisingly, only slight increase in dissimilar concepts (Fig. A)
We then explore if introducing novel features from an out-of-domain dataset would result in more dissimilar concepts. In this experiment, we train one model (R18 NAB+SC) on a combined dataset of NABirds and Stanford Cars (Krause et al. 2013) achieving 37% accuracy on the combined classi- fication task. To compare to R18 s483, we freeze the backbone and re-train the model head on both NABirds and Stanford Cars, achieving 26% accuracy. We find training the backbone on Stanford Cars significantly increases concept dissimilarity A8C). Interestingly, we find that the increase in dissimilarity is bi-directional, both models are less able to predict the concepts of their contrasted pair. In order to better understand the bi-directional nature of this dissimilarity, we visualize a few concepts from each model in Figure A7 These concepts were selected by (1) filtering concepts above the 75th percentile in delta KL-divergance, (2) visualizing the 15 lowest delta Pearson con- cepts and (3) manually selecting concepts that were easiest to interpret. We find dissimilar concepts from the R18 NAB+SC model that seem be semantic concepts specifie to car models. In contrast, we found no dissimilar car related concepts that met the criteria from R18 $483. Instead, we find that concepts that meet this criteria are from NABirds classes and tend to be challenging to interpret. We then visualize R18 $483 concepts that result in the largest kl-divergence when replaced by the predictions from RI8 NAB+SC. When visualizing these concepts we find two car related concepts that seem to be primarily driven by color, but not car model. These results match the intuition that R18 NAB+SC should contain more complex, car model aligned concepts that can be used to better classify images from Stanford Cars. However, it is not a complete explanation of model behavior and further analysis is needed to make concrete statements about concept dissimilarity.
Finally, we compare a R8sNet-18 trained on ImageNet and fine-tuned on NABirds to a NABirds model (R 18 ImgNet PT). Unsurprisingyy, this results in the most dissimilar concepts (Fig. A8D) In sum, we find that larger changes during training results in more dissimilar concepts.
5873 438 5483
R1B 5403 R15 NAB-W
NM0 SC noO neld)
RS4 trgget 5185483
nua Ieghee
Ki8
Noa 50
4ikingun
15.1
1120 -403
RL1 5073
R12 54123
MLO NA2-5C
Inglett
Figure A8: Effect of Seed and Dataset ResNet-18 Similarity. We compare several pairs of ResNet-18 18 models while varying their training protocols. We use the same base model in all com- parisons, a ResNet-18 model trained with the seed set to 4834586 (R18 s483). (A) We compare the base model to a model trained with seed 87363356 (R18 s873) and find that the two models are highly similar despite the change in seed. (B) We train a ResNet-18 18 modified dataset where we exclude 169 classes that belong to the coarse category of waterbirds (R18 NAB-WBI When compar- ing to the seed variation experiment. we see a slight increase in the number of dissimilar concepts. (C) We train a ResNet-1 8 on combined dataset of NABirds and Stanford Cars (R18 18 NAB+SC). To compare to the base model, we freeze the base model's backbone and re-train the linear classifier on this combined dataset. We find that introducing Stanford Cars results in significant increase in dissimilar concepts. (Right) Finally, we compare to a model pre-trained on ImageNet and fine-tuned on NABirds (R18 ImgNet PT). We find that training on ImageNet introduces many novel concepts that are dissimilar to the features of the base model.
24

---

Published as a conference paper at ICLR 2025
MAE_5483 vs. DINO MMCS
DINO MMCS
MAE 5483 Layers
6.32
DINO 3873 Layars
0.50
0,43
6.88
040
635
6J5
6.32
6.30
0.30
10
DINO ,483 Layers
DINO, 5073 Layers
DINO Layers
DIN 5873
DWO 3183
DINO
M4E_ DINO_5483 5403 Concept Simllrity
M873 DINO_5873 Concest Smiartt
DINO_8873
DINO,4883 Concept Simllaity
Ci
9483
MAV 5073
DING
In Concept Smilrriy
D1NO MAE 5483 Concopt Simlarity
5873
DINO $483
DIND 5873 Concept Similarity
Figure A9: DINO and MAE Seed Variation. We explore the effects varying seed on finetun- ing DINO and MAE model on the NABirds dataset. (Left)): We show layerwise and last layer comparisons of MAE s483 vs. DINO s483. These plots are reproductions from the main text. The black line denotes the average concept similarity. For this comparison, the average similarity in both directions 0.80. (Center): We compare DINO __70__873300 vs. MAE_s873. We see similar layerwise matrix and last layer similarity to DIN0_s483 vs. MAE_s483. The average similarity for both mod- els is, once again, 0.80. (Right): We compare DINO .483 vs. DINO s873 and find that there is a better sayer-to-layer mapping in the layerwise comparison matrix. In addition, the average similarity in both directions is 0.89, higher than comparisons across the different pretraining strategies. Taken together, these results indicate that individual concepts change due different seeds, but the global structure of the relationship between these models is not affected by seed.
A.6 DINO AND MAE SEED VARIATION EXPERIMENTS
In Section we compared a DINO pretrained model and a MAE pretrained model that were fine- tuned on the NABirds dataset. In those experiments, models were finetuned with the seed set to 4834586. In this section, we explore comparisons to models finetuned with different seed, 87363356. The models finetuned with the new seed are denoted as DINO .873 and MAE_s87, We compare two pairs of models, DINO_s87, to MAE_s873 and DINO_s483 s483 to DIN0 In Fig- ure A9 we show that changing the seed does change the concepts learned, but that the general relationship between the different pretraining strategies is preserved. In particular, we find that com- paring DINO models finetuned with different seeds results in a higher average similarity (~-0899) than models with different pretraining strategies (~0.80), indicating that the seed has smaller im- pact on finetuning than the initialization.
A.7 SAME MODEL CONCEPT SIMILARITY VS. CONCEPT IMPORTANCE
In this section, we validate the feasibility of the regression task. Due to the reconstruction error inherent in decomposition methods, it is not possible to perfectly predict the concept coefficients from the activation matrix. However, in Fig. [AO we show that the regression models do well when trying to do same-model concept regression and significantly better than cross-model concept regression.
A.8 ADDITIONAL REPLACEMENT TESTS
In Figs. ATI and A12 we visualize the replacement tests for the three pairs of models not presented in the main text. We see the same effects as before, with a decrease in similarity corresponding to increasing changes
model behavior. Notably, although DINO and MAE models (finetuned on NABirds) have high similarity relative to the other models, they show stronger changes in model behavior for smaller changes similarity. However, is not clear whether these differences are due to changes in dataset or changes in pretraining.
25

---

Published as a conference paper at ICLR 2025
ResNet- 18 CI
ResNet-50 CI
0.5
0.5
3.0
3.0
0.0
0.2
0.8
1.0
0.0
0.2
0.4
1.0
ResNet-18
ResNet-18 Concept Similarity
VIT-S
VIT-S Concept Similarity
ResNet-50 CI
1.0
1.0
VIT-S CI
0.5
0.5
0.0
0.2
0.8
1.0
0.0
0.2
1.0
ResNet-50 Concept Similarity
ResNet-50
ResNet-50 Concept Similarity
1.0
1.0
VIT-S
DINO
VIT-5 VIT-5 Concept Similarity
DINO DINO Concept Similarity
0.5
MAE CI
0.5
VIT-L
3.0
3.0
0.0
0.2
0.8
1.0 0.0
0.2
0.4
1.0
VIT-L VIT-L Concept Similarity
MAE MAE Concept Similarity
Figure A10: SMCS vs CI. We visualize same-model concept similarity (SMCS) against the concept importance. We find that reconstructing more important concepts tends to be easier for ResNets. However, for some ViT models, there be important learned concepts that are hard to predict. Importantly, SMCS significantly higher than CMCS indicating that the regression task is feasible.
26

---

Published as a conference paper at ICLR 2025
DINO MAE
1.0
0.00
0.5
A MatchAcc
0.8
0.4
4 KL
0.2
-0.05
0.6
0.4
0.1
-0.10
0.2
0.2
0.1
0.0
-0.15
0.0
-0.2
-0.1
-0.2
-0.1
A Pearson
Pearson
Pearson
MAE DINO
1.0
0.4
0.00
A MatchAcc
0.8
0.3
L2
KL
-0.05
0.6
0.2
-0.10
0.2
-0.15
0.0
0.0
-0.3
-0.2
-0.1
0.0
-0.3
-0.2
-0.1
0.0
-0.3
-0.2
-0.1
0.0
Pearson
A Pearson
A Pearson
Figure All: Replacement Test for DINO vs. MAE (NABirds). We find that for the DINO vs. MAE comparison. As Pearson correlation decreases the lg-distance increases, KL-divergence increases, and the match accuracy decreases. Notably, the Pearson correlation decreases smaller amount than for the other three pairs models, but the change in the three metries on the same order the other comparisons. This suggests that these two models are more sensitive to changes concept.
27

---

Published as a conference paper at ICLR 2025
ViT-S - VIT-L
0.03
1.0
0.20
1.50
MatchAcc
0.00
0.8
-0.02
L2
1.00
KL
0.6
0.10
-0.05
0.4
0.50
0.05
-0.07
0.2
-0.10
0.00
0.0
-0.4
-0.2
0.0
-0.4
-0.2
0.0
-0.4
-0.2
0.0
Pearson
A Pearson
Pearson
VIT-L-VITS
1.0
3.00
0.20
MatchAcc
0.00
0.8
2.00
A KL
0.6
0.10
0.4
1.00
-0.10
0.2
0.00
0.0
-0.4
-0.2
0.0
-0.4
-0.2
0.0
-0.4
-0.2
0.0
A Pearson
A Pearson
A Pearson
1.0
0.4
A MatchAcc
0.00
0.8
KL
0.3
-0.05
0.6
-0.10
0.2
-0.15
0.2
0.0
VIT-S - ResNet-50
1.0
0.00
2.0
A MatchAcc
0.8
1.5
KL
-0.05
0.6
0.5
-0.15
0.2
0.0
-0.4
-0.2
-0.4
-0.2
-0.4
-0.2
A Pearson
A Pearson
A Pearson
Replacement Test for VIT-S vs. VIT-L and RN50 vs. VIT-S. We find that for these model comparisons, as Pearson correlation decreases the (2-distance increases, KL-divergence in- creases, and the match accuracy decreases.
Figure A12:
28

---

Published as a conference paper at ICLR 2025
ResNet-18 ResNet-50
A: 0.1
ResNett
6.6
3000
2000
0.1
Count
1500
0.2
ResNet 50 ResNet-16
0.4
0.5
100
Simierity
non zero confficinnss
022
ResNet 18 ResNet 50
04
ResNet-50 - ResNet-18
1.0
0.5
Restiet:1
3000
X: 0.01
3: 0.05
Count
0.2
2000
ResNet-50 ResNet-18
14
0.3
1.0
1000
1000
ResNet 50
85
6.
0.0
0.2
0.4
0.5
0.0
1.0
200
400
600
Simiarity
nons-eeer coufficients
ResNet-18 ResNet-50
Figure A13: Impact of Regularization on Regression. Here we vary the À for the I penalty on the regression model. We the first 200 classes of ImageNet for these visualizations. In the left column, we visualize the distribution of similarity values for each value of A. In the center, visualize the number of non-zero coefficients. In the right column, we visualize the similarity vs. importance plots for À= = 0.1 and À = 0.5. We find that, as expected, increasing the 4 penalty reduces similarity by increasing the number of zeroed coefficients. In all experiments the paper, we use an l penalty of 0.1.
B ADDITIONAL IMPLEMENTATION DETAILS
CONCEPT EXTRACTION
For each model considered in this study, we provide information about concept extraction in Tab. A2 First, images are resized to 224x224 and then processed into 16 evenly spaced 64x64 pixel patches. Patches are then resized back the image resolution of the network. We sample 100 images per model for concept extraction. All models taken from the timm library were trained with Inception style random cropping. Custom trained models were trained using random resized cropping with horizontal flipping. This ensures that the resized patches are in-domain for the network. To pro- duce an activation matrix A that can be decomposed, the outputs of the network are processed. The ResNets (He et al.) 2016) produce outputs with batch b, channel height h and width w dimen- sion. To create matrix that can be decomposed, we use global average pooling over the and w dimensions. The ViTs (Dosovitskiy 2021) produce outputs with batch b, sequence s, and feature dimension d. We select the class token from the sequence dimension resulting two-dimensional matrix. We use NNMF for ResNets since they contain ReLU layers and produce positive only activations. NNMF restricts the and W matrix to be positive. For, ViT models, we use Semi- NMF (Ding et al. 2008) which allows for both positive and negative values in the W matrix. but requires positive values in the U matrix. We use a non-negative least squares solver to fit coefficients set of data points:
min subject to
A
U, 3
(5)
U 2
B.1.1 CONCEPT INTEGRATED GRADIENTS
Integrated gradients measures the importance of each pixel by averaging the gradients of the input image, the input image is varied from baseline value its true value (Sundararajan et al. 2017). To compute concept integrated gradients the formulation is modified. Let h; represent the head of M,, i.e., the final layer(s), and A be the output activations from the layer preceding h1. As
29

---

Published as a conference paper at ICLR 2025
ResNet-18 ResNet-50
k
10
ResNet-
6.5
5000
15
15
Count
2000
20
ResNet- 50 ResNet-16
1000
0.4 Similarity
Recon, Err
0.2
ResNet 18
04
ResNet 50
0.
ResNet-50 ResNet-18
Restle:11 18
1.0
k
20
1.0
15
Count
2000
ResNet-50 ResNet-18
10
ResNet 50
3.9
1000
5-
85
0.0
0.0
0.2
0.4
0.5
0.0
1.0
140
160
180
200
220
Similarity
Recon, Err
ResNet-18
ResNet-50
Figure A14: Impact of Number of Concepts on Similarity, Here we vary k; the number con- cepts in the dictionary and explore the impact the similarity distribution. We the first 200 classes of ImageNet for these visualizations. In the left column, we plot the distribution of similarity scores for 5, 10, 15, and 20 concepts. In the center column, we visualize the distribution of recon- struction errors for different number of concepts. As expected, increasing the number of concepts results in lower reconstruction errors. the right column, we visualize similarity vs. importance for 10 and 20 concepts. We observe that increasing the number of concepts disproportionately increases the number of dissimilar concepts. For all results the paper we use 10 concepts.
described earlier, we factorize A ~ U, W,. We denote row vectors of U1 as r E Rows (Ow)(U1). such that r| E IR. 1xd d To link model predictions to learned concepts, we compute model predictions as
hn Wi),
where 2] e R is a row vector of prediction probabilities. Then, to compute concept integrated gradients, average over the gradients linearly step from baseline vector r 0 to ri
((r)) (ri
V ((or' +11 az) - -r)) W)do.
Thus, for each class and concept we have a single value that represents the importance of that concept to model decisions. We implement concept integrated gradients based on the implementation in the xplique library (Fel et al. 2022a). For all experiments we integrate over 30 steps.
B.2 CONCEPT SIMILARITY
Correlation. Pearson and Spearman are computed using scikit-learn (Pedregosa al. 2011 We use 50 images for each class from the training set of the model. Images are resized to 224 x 224. We use a patch size of 64 664 resulting in 16 patches per image. Thus, Pearson and Spearman correlation is computed using 800 total patches per class. The patches are resized and passed through the model generate activations at a given layer.
Regression. We use lasso-regression (Tibshirani 1996) with a 0.1 weight on the 4 penalty. We visualize the effect of this parameter on similarity in Fig. A13 Regression models are trained on the activations from at least 5 images (80 patches) and at most 200 images (3200 patches) sourced from the original training split of the dataset. For each concept and class, we train five lasso-regression models different equally sized folds. The regression model weights are averaged and then the model evaluated on images from the validation/test split of the original dataset. The inputs and targets are standardized to have mean of zero and standard deviation of one. The regression is trained using the Celer library (Massias et al., 2018). Finally, the predicted coefficients are unnor- malized before the Pearson and Spearman correlation computed. To compute feature importance
30

---

Published as a conference paper at ICLR 2025
scores for regression models, the permutation feature importance implementation from scikit- learn (Pedregosa et al. 2011). We use the default parameters with 5 repeats and the random state set to 0.
Layerwise Comparisons. We list all the layers used in the layerwise comparisons in Tab. A4
Spearman Correlation. In all experiments, Pearson correlation, thus we have excluded these results.
found Spearman correlation to be very similar to
Visualizing Dis-similar Concepts. We select one patch per image in order of maximum concept coefficient. The top n patches for the real images are excluded from the pool of images used to visualize the under-predicted and over-predicted coefficients.
B.3 COMPUTATIONAL COST
All experiments were conducted using machine with an AMD Ryzen 3700X 8 8-Core Processor and single GeForce RTX 4090 GPU. In Table A3 we detail the computational cost of each step of our proposed method. For comparison between ResNet-18 18 and ResNet-50 on all 1000 classes of ImageNet, RSVC takes approximately 20 hours.
Table Al: Model performance.
Model
timm Model
ImageNet Accuracy
NaBirds Accuracy
ResNet-18
resnet]8.a2_inl1
70.6%
ResNet-50
resnet50.az a.2..nn1.
79.8%
ViT-S
vit small patch16 224. aug.eu in2 inlk
81.3%
VIT-
vit large p 224.22222 a2g.eg in21k ft in
85.8%
DINO ViT-B
sit_aase_pacch 16_224.dino vit_base Sitase_patccl6.220 12_2266222mmeee
71.2%
MAE ViT-B
71.2%
Table A2: Concept extraction.
Model
Layer Post-
Method
Number of Concepts
Patch Size
Recon.
processing
Error (Last
Layer)
ResNet-18
GAP GAP
NNMF
10
64
176.2
ResNet-50
NNMF
10
64
205.5
ViT-S
Class Token
Semi-NMF
10
64
926.9
ViT-L
Class Token
Semi-NMF
10
64
1650.8
DINO VIT-B
Class Token
Semi-NMF
10
64
191.0
MAE VIT-B
Class Token
Semi-NMF
10
64
656.5
Table A3: Computational cost for ResNet18 vs. ResNet-50 on ImageNet.
Step
sec/it
Total Time
Activation Extraction (RN50)
1.50
25m
Concept Extraction (RN50)
2.00
33m
Concept Comparison (CMCS) Last Layer
9.00
2h30m
Concept Comparison (MCS) All Layers
14.56
4h
Concept Int. Grad
41.56
1 1h 30m
Regression Evaluation
2.30
38m
Total Time
19h36m
31

---

Published as a conference paper at ICLR 2025
Table A4: Selected layers.
Model
Layers
ResNet-18
act1, layer1.0.act/. layer1.0.act2, layerl. fayerI.1.acti llatll, layerI.1.ace latt2, layer2.0.act2, layer2.Iactl, layer2.1.act) layer3.0.act lact,,,Lactl, layer4.1.ac2
layer2.0.act1,
layer3.0.act2.
layer3,Lact2, layer4.0.act1,
layer4.0.act2,
layer4.Lactl,
ResNet-50
actl, layer1.0.actl,
layer1.0.act2.
...act2,
layer1.0.act3
layerI.1.acll,
layer1.
layer1.2.actl layer1.1.act3,
layer1.2.act2.
layer1.2.act3,
layer2.0.act1,
layer2.0.act2,
layer2.0.act3,
layer2.1
1ayer2.1.act2,
layer2. 1.act3, layer2.3.ac22,
layer2.2.actl,
layer2.2.act2,
layer2.2.act3,
layer2.3.act1,
laytr2.3.act3,
layer3.0.actl,
layer3.0.act2,
layer3.0.act3,
layer3.
layer3.1.act3, layer3.1.act,,
layer3.2.act1,
layer3.2.act2,
layer3.2.act3,
layer3.3.actl,
layer3,3.act2,
layer3.3.act3,
layer3.4.act1,
1ayer3,4.act2,
layer3.4.act3,
layer3,5.act1, layer3.5.act2,
layer3.5.act3,
layer4.0.act1,
layer4.0.ac12, layer4.2.actl,
layer4.0.act3, layer4.1.act. layer4.1.ac22,
layt,,,
layer4.2.act2, layer4.2.act3
ViT-S
block0, block1, block2, block3, block4, block5, block6, block7, block8, block9, block10, block11
ViT-L
block0, block1, block2, block3, block4, block5, block6, block7, block8, block9, block10, block11, block12, block13, block14, 14, block15, block16, block17, block18. block19, block20. block21, block22, block23
DINO VIT-B
block0, block1, block2, block3, block4, block5, block6, block7, block8, block9, block10, block11
MAE ViT-B
block0, block1, block2, block3, block4, block5, block6, block7, block8, block9, block10, block11
32


Reviews:
[{"review_id": "V90202KUGq", "reviewer": "Reviewer_8D1Q", "summary": "This paper studies the representational similarity between different vision networks. The main novelty is to provide insight into not just _how_ similar are two networks but in _what_ ways are they similar. This is achieved through a dictionary learning method that approximates the activations at some layer of a network as a nonnegative combination of \"visual concepts\", which are just a set of activation vectors.  Then the similarity between two networks is measured as the similarity of the visual concepts they each represent, either through a correlation-based metric or via regression from one set of concepts onto the other. The paper finds that there is some degree of similarity between the visual concepts in different networks, that there are some important concepts that differ between different networks, and that layers near the input or near the output of the network are most alike, while middle layers differ.", "questions": "1. I didn't fully understand why use image patches rather than whole images. This could be motivated more clearly, or justified with an experiment comparing using patches vs whole images.\n2. I would appreciate if the authors can address the missing reference I mentioned above. Why should a reader prefer your proposed method over these prior methods? What are the advantages and disadvantages of each approach, or are they not comparable?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I like the motivation of this paper. It is convincing that 1) understanding _what_ differs between different neural representations is important,  and 2) prior work has mostly overlooked this question (although a few papers that were not cited do address this question; see weaknesses).\n\nOther strengths include:\n* The writing is mostly clear.\n* The methods appear to be reasonable and I don't see obvious errors.\n* The paper delivers a tool that could increase interpretability of representational differences, because the differences can be visualized. \n* Several of the findings are interesting; I was especially intrigued by 1) the method can identify low similarity, high importance concepts, 2) middle layers show the greatest differences between different networks.", "weaknesses": "The main premise of this paper is to go beyond prior work on representational similarity, which only gave summary numerical scores, and instead show \"_what_ visual concepts make two models similar or dissimilar.\" This is a great question, but the paper falls short of fully delivering on it.\n\nThere is only one figure, Fig 2, in the main paper that directly addresses this central question, comparing two models and showing how they differ in terms of the visual concepts they learn. The appendix has more results along these lines but I'm still not sure what insights to take away from these results. \n\nI take it to be that the contribution is not to *show* what is different between different nets but rather to provide a *tool* that is in principle capable of showing what is different. This is a valid contribution but I think the paper would nonetheless be stronger if the tool were used to provide new insights into the model differences.\n\nThe other results (Figs 3, 4, and 5) are interesting but fall back into the category of summary statistics that don't go deep into what makes two models similar or dissimilar. Instead we only learn that 1) there are important differences (intriguing! but what are they), and 2) the differences live primarily on the middle layers of networks (although from Fig 5 it looks like the increase in similarity at the deepest layers is quite minor indeed).\n\nIf this paper is indeed meant to just introduce a tool, rather than a discovery, then the reader needs to be convinced that the tool is valuable. I'm mostly convinced that the tool is valid and in some ways new. But I'm not sure what to do with it, nor if the community would be able to use this tool to make new discoveries. This isn't an absolutely critical weakness -- time can tell if it is useful -- but I would rather have seen more validation of the usefulness of the tool in this paper. It would be wonderful if the paper could use the tool to tell us something about neural representations we didn't know before, like, say, that resnets focus more on background elements than transformers, or, ideally, something deeper than that. To make this super compelling, I would like a demonstration that this new way of identifying differences has advantages over other ways of identifying representational differences, in a head to head comparison. But I'm not sure how to do that.\n\nThere are also a few overlooked prior works I think are worth discussing (or, ideally, comparing to):\n\n1. \"Rosetta Neurons: Mining the Common Units in a Model Zoo,\" Amil Dravid, Yossi Gandelsman, Alexei A. Efros, Assaf Shocher, ICCV 2023\n\n-- This paper also identifies visual concepts that are shared between different models. In their case, the concepts are coded by neurons, across different models, that are correlated in terms of their responses to the same set of image patches.\n\n2. \"Understanding the Role of Individual Units in a Deep Neural Network,\" David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedrizad, Bolei Zhouf, and Antonio Torralba, PNAS 2020\n\n-- This paper also provides a tool for identifying which visual concepts are encoded by different networks, and compares two networks in this way.\n\n3. \"Exploring Neural Networks with Activation Atlases,\" Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, Chris Olah, Distill, 2019\n\n-- This paper introduces \"activation atlases\", which are yet another way of identifying visual concepts learned by different networks. This work does not use these atlases to compare different models, which differs from the current paper's goals.\n\nThere is also a recent line of work on interpretability via sparse autoencoders that may be worth looking into, e.g., https://transformer-circuits.pub/2024/scaling-monosemanticity/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9MAXrCe0fE", "reviewer": "Reviewer_wky2", "summary": "This paper introduces Representational Similarity via Visual Concepts (RSVC), a method to compare representations between neural networks by analyzing shared and unique visual concepts instead in pairs of pretrained models, with the goal of providing a better comparative breakdown of pairwise model similarities.\nConceptually, RSVC extends work by Fel et al. 2023a, b on a dictionary-based formulation of concept-based XAI, and introduces different mechanisms to study similarities between concept explanations of two different networks: (1) A correlational analysis of concept coefficients, and a (2) regression-based approach, in which a linear model is trained to predict concept assignments between models. These two concept similarity methods are then used in conjunction with a concept importance estimator and a concept-performance-dependence study to broadly understand the existence and prevalence of unique and shared concept pairs between models.\nExperiments are conducted for RN18, RN50, ViT-S, ViT-L, MAE & DINO pretrained on ImageNet to showcase concept similarities between different model architectures and training paradigms.", "questions": "See weaknesses. I'm happy to raise my score if the authors can address my first two points, and during the rebuttal can provide baseline studies and sanity checks; as I believe the paper to be an overall meaningful contribution if these points can be incorporated.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "In general, I find this paper to be sufficiently well written and structured; with the presentation and detailed image visualization and captions to be commendable: Figures and their captions can be read standalone and still convey key contributions; which significantly enhances readability. Moreover, the proposed method is described and visualized in a way that makes it much easier to grasp all relevant key aspects.\n\nAt the same time, the extension of works by Fel et al. seems sensible, and to the best of my knowledge, the use of a dictionary-based concept attribution method for __model similarity__ studies is novel.\n\nMoreover, I very much appreciate the experimental breadth the authors followed to test the different potential usecases of RSVC.", "weaknesses": "Unfortunately, I currently have some more pressing issues and questions which stop me from recommending acceptance at this stage. Ordered by importance, these are:\n\n---\n\n__Lack of sanity checks and proof of concepts__.\nInterpretability methods are strongly reliant on sanity checks (Adebayo et al. 2018, https://arxiv.org/abs/1810.03292, Boehle et al. 2022, https://arxiv.org/abs/2205.10268). The authors unfortunately directly apply their proposed methods on arbitrary model pairings, without first firmly establishing the validity of the proposed approach. To do so, I believe that at the very least, the following steps need to be included:\n\n(1) Establishing a baseline. This means that using two models for which concept similarities are highly likely or even guaranteed, and applying both MMCS and RSVC (regression-based) to understand how visualizations in Fig. 3 changes. As it is now, it is much more difficult to relate computed values and their respective significance. A baseline could e.g include the same model architecture trained on the same data ordering, but with small changes in the initialization; or a fixed initialization but changes in the data ordering.\n(2) Studying variations away from the baseline above. By structurally varying the baseline (e.g. going to different augmentations, different data subsets, ...) understand how the proposed metric changes.\n(3) More clearly establish what extracted concepts actually look like. The example in Fig. 1 with bird tail and sky background seems quite convincing, but then when looking at actual examples in Fig. 2, these become much less convincing: The net appears consistently in all scenarios (alongside arms and hands), and I find it difficult to see any consistency here.\n\nWithout establishing such baselines and sanity checks, it is very difficult to assign actual meanings to extract values.\n\n---\n\n__Conceptual / Architectural Inconsistencies__.\nThere are some elements in the proposed approach that currently do not make much sense to me, particularly:\n\n* L171: \"Each concept proposal is resized to the model’s input resolution and passed through the network.\" >>> Learned features are generally resolution-dependent unless speficially trained not to be. As such, the resulting extract concepts (from the resized image patches) hold much less meaning with respect to the actual network, no? I may not understand the setup sufficiently here, but resizing patches and using thereby derived U_i, W_i for concept comparisons does not make much sense to me. Why not just store the full activation output and break it down corresponding to the patch formulation?\n\n\n* As the authors correctly state for the correlational setup described in 3.2.1, MCS does not accout for any confounding elements. But it is not completely clear to me how RSVC avoid the issue of confounders? If I have a confounding element in e.g. M2 which would produce similar concept coefficient than the actual extracted concept would, this would still be an issue even when using the learned linear map, right?\n\n* For the replacement tests: What are the guarantees that U_2-1 operates in the same basis as U_1, such that a direct replacement is well defined? The learned maps in Eq. 4 are trained without any guarantees that the bases between both models overlap, no?\n\n---\n\n__Some less important notes that are not majorly impacting my decision, but are useful to address:__\n\n* L133-135: \"While our work also aims to compare two models in an interpretable manner, our approach does not make use of surrogate models and instead uses the activations of the original model itself.\" But isn't the projection matrix another learned linear model? I.e. isn't RSVC also using surrogate models?\n\n* The use of a linear predictor (or one with directly assignable importance values) to test matching up to permutation, scale and shift has been studied in-depth in disentangled representation learning, e.g. in Locatello et al. 2018 (https://arxiv.org/abs/1811.12359), Eastwood et al. 2018 (https://openreview.net/forum?id=By-7dz-AZ), Locatello et al. 2019 (https://jmlr.org/papers/volume21/19-976/19-976.pdf), Eastwood et al. 2022 (https://arxiv.org/abs/2210.00364) or Roth et al. 2023 (https://openreview.net/pdf/f1f46edad3b1c36e9c46c00c16e8592cfd1e4df6.pdf). These may provide a useful resource for experimental ablation studies, and should be discussed to augment the short reference to disentangled representation learning in the related works section.\n\n* Section 3.3 is fairly redundant, and should either be excluded entirely, or in my eyes more crucially, described in much more detail, as the use of concept importance is quite prevalent throughout the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I8ifcuAJSz", "reviewer": "Reviewer_RkXK", "summary": "The paper presents an approach for measuring the similarity between representations of two models using \"concepts\". It quantifies the similarity between the internal concepts of the models, extracted with a concept-based xAI method.", "questions": "Why does the approach use per-class concepts? Different classes of images can have shared computation and shared intermediate concepts? Using per-class concepts might result in missing shared concepts, and functional similarity.  \n\nThe current model assumes implicitly that the models $M_1$ and $M_2$ are classifiers (the matching is applied per class). What modifications are needed to make it work on other types of models (e.g. representation learners/segmenters)? \n\nWhat data is used to compute the linear mapping in line 282? Is it part of $I^c$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, easy to follow, and defines well the different stages of the method and the evaluation.\n\n2. The idea of the paper is novel - instead of comparing functional similarity or direct similarity in representations, it uses a decomposition into \"concepts\" and compares the similarity between concepts. Instead of providing one similarity measure, this paper also demonstrates visually what are the concepts that make two models similar or dissimilar. \n\n3. The paper provides a comprehensive study of the relations between concept similarity and concept importance to address the main possible limitation of using concepts - their non-straight-forward connection to the computation in the model.", "weaknesses": "1. This approach might be sensitive to the concept extraction method. The extraction of $U_i$ and $W_i$ provide very different results for different concept dictionary sizes $k$ and if some L1 sparsity regularization is applied. A comparison of different approaches for concept extraction and ablation of their hyper-parameters can provide better insights into the approach sensitivity.\n\n2. Computational cost - this process requires computing representations on many images, extracting representations from each layer, training linear models, and comparing correlations. The computational cost of this process is not discussed, and the trade-offs between the number of images that are needed for each of the steps are also missing.\n\n3. Usability - The current approach for coming up with meaningful observations about the representations requires a human in the loop, who goes over all the concepts and investigates the similar and dissimilar concepts (as shown in Figure 2). Can it be automated somehow? \n\n4. Missing related work:\n\nDravid et al., Rosetta neurons: Mining the common units in a model zoo, ICCV 2023\n\nRombach et al., Network-to-network translation with conditional invertible neural networks, NeurIPS 2020\n\nTypo in 250: We use it *to* compute...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2mvBCvNWjd", "reviewer": "Reviewer_fUV9", "summary": "The paper presents a method for comparing the representations of two neural networks using visual concepts. Concepts are extracted by factorising the matrix of activations corresponding to a set of images. The authors present two methods for the comparison of visual concepts, one based on correlations between the coefficient matrices, and another using regression, using one set of concept coefficients to predict the other and vice-versa. Also proposed is a method for measuring the dissimilarity of representations, where one network can contain a concept deemed important to classification, which is not present in the other network. These methods are evaluated on a variety of networks, investigating the relationship between concept importance and concept similarity.", "questions": "-  How many images are used to construct  $\\mathcal{I}_{1,2}^c$, and is it a requirement for the models to overlap slightly so that $\\mathcal{I}^c$ has enough samples? How consistent are the concepts extracted with less samples? The images used to measure similarity seem very important for the ability to interpret results, as the images need to represent the concepts important for classification.\n- Does the size of the patch effect the concepts found? And what patch size was used in the paper.\n- Along the same lines, what is the split of $\\mathcal{I}^c$ between test and train for the regression task?\n- Do the authors have any intuition for using this method to compare models that do not necessary predict the same number of classes?\n- In Figure 2, could you look at the regression weights learned, and see that the concept for the RN50 would be predicted by two separate concepts in the RN18? This would substantiate the claim in line 465, that possible the RN18 contains separate concepts for hands and net, as these concepts would have large contribution when predicting the 'hands/arms near a volleyball/net' concept in RN50.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The topic is of great interest. Providing interpretable model comparisons that go beyond model performance will be useful to practitioners.\n- The method goes beyond just measuring similarity, but provides interpretation for both similar and dissimilar concepts, something that is often overlooked in model comparison.\n- The example in Figure 2 is very convincing.\n- The proposed method for measuring similarity and dissimilarity is very intuitive.", "weaknesses": "- Some implementation details are missing (clarified in the questions section)\n- Brainscore does link representation similarity scores to functional differences; models are evaluated using similarity in representations but also comparisons in the output. In places it is described as comparing representations between two neural networks, which is not true. It is between the activations in a network and  macaque brain activations.\n- Colour bars in Figure 3 and 4 would make the figure slightly clearer.\n- In Eq. 5, mentioning why L1 regularisation is used would make it clear to the reader.\n- The introduction is quite brief, and fails to motivate the problem addressed. It could benefit from a small discussion on why interpretability is important and why concepts are more useful than say saliency maps or superpixels (LIME).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "StwEbL3Rbp", "reviewer": "Reviewer_73oU", "summary": "### Post rebuttal\n\n---------------\n\nThe rebuttal clarifies most of my concerns. I think this work may provide some direction on how to compare models in terms of model explanations.\n\nFor the final version, I urge the authors to:\n\n1. Point to some of the GPT based model explanations to Fig 2 and Section 3.4.\n2. Clean the plots in Fig 3 and Fig 4 to remove the majority of less important points to make the trends more visible.\n3. Add to the main paper, that they used a separate validation set to tune lambda from Eq. 4\n\n---------\n\nThis work aims to analyze similarity between two trained networks in terms of what visual concepts they encode. It builds on top of prior work (Fel et al 2023) which provides a formulation to extract “concepts” from learned network. Namely, the activations of a network $A  \\in R^{N \\times D} $ are decomposed to form $U \\in R^{N \\times k}$ (concept coefficient matrix)  and $W  \\in R^{k \\times D}$ (concept basis). Given $U_1$, $R_1$, $U_2$ and $R_2$ of two different models, they propose “correlation” and “regression” to measure concept similarity. Correlation (MMCS) (loosely) measures the correlation between $U_1$ and $U_2$, while regression (CMCS) measures how well $U_2$ can be decoded from $U_1$ using a linear model (and vice versa).\n\nThe paper performs three experiments: comparing concept importance vs CMCS, how changes in concept similarity affect final outputs and layer-wise conceptual similarity using MMCS.", "questions": "* The abstract says that this work presents a practical tool to discover the cause of model failures but this discussion is missing in the main text.\n* L167 - L169. Next, as visual features are highly correlated, patches from each image are sampled to form a set X c 1 of “concept proposal” images. More detail is required on how these patches are extracted from the image? Are they just randomly sampled, if yes then how many? And is each patch resized independently to the original image resolution?\n* L202 - L203. It’s unclear why there is a tradeoff between computational cost and error between these two techniques.\n* L201 - 211. $R^{\\rho}$ and $R^{S}$ are mentioned, so which one does R employ?\n* L214: Eq 2) takes the max across all columns for each row and vice-versa. IIUC, this is because there is no direct correspondence between the columns in U1 and the columns in U2. That is the first concept of U1 may represent something completely different from the first column U2. So to measure similarity, one matches each column of U1 to the column that has maximum correlation in U2. Even though this may be obvious to the authors, it will be nice if some discussion is added.\n* L297: Eq1) How is $\\lambda$ tuned?\n* L348 - L349 For our exploration with DINO and MAE, we fine-tune the models on NABirds (Van Horn et al., 2015). Why do this?\n* Fig 2) plots the correlation plots between $U_1 \\in R ^{N \\times K}$ and $U_{1\\rightarrow 2}$. Are both matrices flattened to 1D to plot the correlation plot. What is the value of n and k?\n* Fig 2) In both orange and light blue grids, there are grid-like textures and hand and arms at net. So, I find the descriptions and explanation unconvincing \n* Fig 3). The paper introduces low, medium CI and concept similarities without any precise definitions. One could for example precisely define the top 5% percentile of concept importance as high and so on.\n* Fig 3) How many points are in each subplot of Fig 3. Some amount of detail atleast for one subplot may be required.\n* Fig 3) The paper says that “model differences are largely driven by medium similarity, medium importance concepts”. I found this conclusion hard to believe for the y-axis, the center of each heatmap seems to be close to 0.0 For the x-axis, it is indeed centered at medium importance.\n* Fig 4) The presentation will be clearer if separate plots are presented for high importance and low importance plots.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper explores a new area which is representational similarity using visual concepts. Finding interpretable ways on how different networks are similar may offer valuable insights to the community.", "weaknesses": "On reading the title, I was expecting insights that show what sort of visual concepts are learned by a model, and what is different as scale, data and architecture is varied. But I found these insights not present in the paper. The paper performs three sets of experiments but its a bit unclear why these are interesting to study. For example:\n\n* In Figure 5, the authors investigate their proposed metrics layer-wise across various models. However, what additional insights does this offer over (Raghu et al 2021)?\n* In Figure 4, high importance features show high correlation between L2 and pearson coefficients, But this is also a bit obvious.\n\nI also found the presentation sometimes difficult to follow. See below for detailed feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)