Objective drives the consistency of representational similarity across datasets
Authors: Laure Ciernik
Lorenz Linhardt
Marco Morik
Jonas Dippel
Simon Kornblith
Lukas Muttenthaler
Venue: ICML 2025 poster (2025)
Keywords: representation learning
factors influencing representational similarities
computer vision
vision foundation models
representational similarity across stimuli
Primary area: deep_learning->other_representation_learning
TL;DR: This work seeks to identify the variables that drive the consistency of pairwise representational similarities across different datasets for a wide range of vision foundation models.

Abstract: The Platonic Representation Hypothesis claims that recent foundation models are converging to a shared representation space as a function of their downstream task performance, irrespective of the objectives and data modalities used to train these models (Huh et al., 2024). Representational similarity is generally measured for individual datasets and is not necessarily consistent across datasets. Thus, one may wonder whether this convergence of model representations is confounded by the datasets commonly used in machine learning. Here, we propose a systematic way to measure how representational similarity between models varies with the set of stimuli used to construct the representations. We find that the objective function is a crucial factor in determining the consistency of representational similarities across datasets. Specifically, self-supervised vision models learn representations whose relative pairwise similarities generalize better from one dataset to another compared to those of image classification or image-text models. Moreover, the correspondence between representational similarities and the models' task behavior is dataset-dependent, being most strongly pronounced for single-domain datasets. Our work provides a framework for analyzing similarities of model representations across datasets and linking those similarities to differences in task behavior.

Objective drives the consistency of representational similarity across datasets
Laure Ciernik 123 Lorenz Linhardt 14 Marco Morik 14 Jonas Dippel 145
Simon Kornblith
Lukas Muttenthaler 145
Abstract
1. Introduction
Representation learning has seen remarkable progress in recent years, with ofate-of-the- art models achieving or even surpassing human-level performance a wide range of computer vision tasks (Radford al., 2021; Dehghani et al., 2023; Oquab et al., 2024; Zhai et al., 2023; Muttenthaler et al., 2024). With this progress in task performance, model representation spaces tend to approach each other irrespec- tive of the models' training data or architecture (Huh et al., 2024). Even between models trained on different modalities, better task performances result in more similar representa- tions. Two questions naturally arise from this phenomenon: Do pairwise similarities of model representations transfer from one dataset to another, and are they generally linked to the models' downstream behavior?
The Platonic Representation Hypothesis claims that recent foundation models are converging to a shared representation space as a function of their downstream task performance, irrespective of the objectives and data modalities used to train these models (Huh et al., 2024). Representa- tional similarity is generally measured for indi- vidual datasets and is not necessarily consistent across datasets. Thus, one may wonder whether this convergence of model representations is con- founded by the datasets commonly used in ma- chine learning. Here, we propose a systematic way measure how representational similarity between models varies with the set of stimuli used to construct the representations. We find that the objective function is a crucial factor in determin- ing the consistency of representational similarities across datasets. Specifically, self-supervised vi- sion models learn representations whose relative pairwise similarities generalize better from one dataset to another compared to those of image classification or image-text models. Moreover, the correspondence between representational sim- ilarities and the models' task behavior is dataset- dependent, being most strongly pronounced for single-domain datasets. Our work provides a framework for analyzing similarities of model representations across datasets and linking those similarities to differences in task behavior.
Investigating the correspondence between representation and behavior has a long history in representation learning and adjacent fields (e.g. Hermann & Lampinen, 2020; Su- cholutsky et al., 2023). Two models can have different intermediate layer representations even though they show the same task behavior (Muttenthaler et al., 2023; Lampinen et al., 2024). However, if two models show different behav- ior, one can be certain that the output layer representations are different, since identical output representations map to the same behavior. What remains unclear is how behavior (e.g., downstream task accuracy) affected by the con- vergence of model representations (i.e., the similarity of representations). Does this relationship depend on the simi- larity measure at hand, or may it be due to the nature of a dataset?
The field lacks consensus on how to define (pairwise) repre- sentational similarity (cf. Kriegeskorte et al., 2008; Korn- blith et al., 2019a; Williams et al., 2021; Duong et al., 2023; Sucholutsky al., 2023) and how systematically measure its relationship to behavior (cf. Geirhos et al., 2021; Mut- tenthaler et al., 2023; Sucholutsky et al., 2023; Lampinen et al., 2024). Here, we use similarity measures that are widely used and have been empirically proven to be useful for measuring representational similarity of neural networks, such as Centered Kernel Alignment (CKA; Kornblith et al., 2019a; Raghu al., 2021), and Representational Similarity Analysis (RSA; Kriegeskorte et al., 2008).
Equal contribution Machine Learning Group. Tech- nische Universität Berlin, Berlin, Germany HHector Fellow Academy, Karlsruhe, Germany 'European Laboratory for Lear- ing and Intelligent Systems (ELLIS), Tübingen, Germany "BIFOLD-Berlin Berlin Institute for the Foundations of Learning and Data, Berlin, Germany Aignostics, Berlin, Germany AAthhooic,, California, United States of America. Correspondence to: Laure Cieriik <ciernik beebe, Lukas Muttenthaler clukas.muteenthalere @ tu berlin.de>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Objective drives the consistency of representational similarity across datasets
However, pairwise representational similarity is usually measured for single dataset and not across datasets. It may be that the representations of two vision models are highly similar for dataset of raccoons but dissimilar for a dataset of sunflowers - due to their different domains. Similarly, the representations of two models may exhibit high similarity for satellite data but show low similarity for images of fruit because of the images' different structure.
ing Canonical Correlation Analysis (CCA; Hotelling, 1992; Morcos et al., 2018), Singular Vector Canonical Correla- tion Analysis (SVCCA; Raghu al., 2017), the previously mentioned CKA and RSA, among others (e.g., Williams et al., 2021; Ding et al., 2021; Barannikov et al., 2022; Cui et al., 2022). They can be distinguished from fonctiona sim- ilarity measures, such as performance difference (Klabunde et al., 2025), error consistency (Geirhos et al., 2020b), and true-positive agreement (Hacohen et al., 2020), which are based on model outputs rather than internal representa- tions (Klabunde et al., 2025), or model stitching (Bansal et al., 2021: Csiszárik et al., 2021; Merullo et al., 2023), which measures compatibility, not similarity, of internal representations (Hernandez et al., 2022).
Thus, it appears crucial to scrutinize the factors that de- termine pairwise representational similarity across sets of stimuli. Knowing about those factors will enable us to gen- eralize pairwise representational similarity across datasets and more effectively find model sets that have learned a sim- ilar representation of the world. For large set of diverse vision models, we examine their pairwise representational similarities for both structured and unstructured datasets from different domains. Our contributions and findings are as follows:
Importantly, similarity measures may differ in the invari- ances they evoke and the properties of representational spaces they capture (Klabunde et al., 2025). For example, local structures may be better captured by measures based on nearest neighbors (Huh et al., 2024) or localized kernels (Kornblith et al., 2019a), while capturing global similarities may be better achieved by wide kernels. Here, we make use of kernel-based CKA, as it allows probing both local and global representation structures by varying the kernel, does not require the compared representations to be of the same dimension, and is widely used (e.g., Raghu et al., 2021; Maniparambil et al., 2024; Saha et al., 2022; Zong et al., 2023; Ding et al., 2019; Baratin et al., 2021).
We propose principled way of measuring the consis- tency of pairwise similarity across datasets, by mea- suring whether the (relative) pairwise similarities of model representations from one dataset are transferable to the (relative) pairwise similarities of another dataset.
We find that the objective function is a crucial factor for determining the consistency of pairwise representa- tional similarities across datasets, whereas architecture and model size do not appear to be major factors. In particular, SSL models show more reliable generaliza- tion across stimulus sets compared to image-text and supervised models.
Drivers of similarity in vision models. In contrast to lan- guage processing systems (e.g., Achiam al., 2023; Team et al., 2023), vision models are trained with wide range of different learning objectives. These include (a) supervised learning objectives using classification datasets such as Ima- geNet (Krizhevsky et al., 2012b); (b) SSL objectives with a reconstruction or multi-view matching loss (Chen et al., 2020a; Caron et al., 2021; He et al., 2022); and (c) objectives for jointly learning image and text representations (Radford et al., 2021; Zhai al., 2023). This makes them interesting candidate models for analyzing representational similari- ties. The trend to larger model sizes (e.g. Dehghani et al., 2023) combined with increased dataset sizes (e.g. Schuh- mann et al., 2022; Oquab et al., 2024) and multi-modality appears to be leading to convergence of model represen- tations (Huh et al., 2024). Similarity between otherwise identical networks trained from different random initializa- tions increases with width (Morcos et al., 2018; Kornblith et al., 2019a), as predicted by theoretical studies of proper- ties of neural networks in the infinite width limit (Lee et al., 2018; Matthews et al., 2018; Yang & Hu, 2021). Huh et al. (2024) hypothesize that convergence is further driven by task generality and inductive bias. Other works find that architecture and dataset (Raghu et al., 2021) play a central role in determining the representational structure of a model.
Our findings show the same trend across different sim- ilarity measures, including those that emphasize local versus global representational structure.
Pairwise similarities of model representations strongly correlate with the models' differences in task perfor- mance for single-domain datasets, while multi- domti- datasets show highly variable and specialized datasets consistently low correlations.
2. Related work
Measuring the similarity of deep neural networks. Repre- sentational similarity measures quantify the degree of over- lap between the representations of two models when pro- cessing the same input. They have been used as a tool un- derstand deep neural networks (Li et al., 2015; Nguyen et al., 2021; Mehrer et al., 2020), investigate human-machine alignment (Sucholutsky et al., 2023; Xu & Vaziri-Pashkann 2021), and as an objective for model distillation (Tian et al., 2020; Saha et al., 2022; Zong et al., 2023). Different repre- sentational similarity measures have been proposed, includ-
2

---

Objective drives the consistency of representational similarity across datasets
Consistency of similarity. Beyond the similarity function, representational comparisons depend on both the extraction layer (Kornblith et al., 2019a; Raghu et al., 2021) and the (probe) dataset (Cui et al., 2022). For example, measuring similarity for a subset of a data distribution can yield dif- ferent similarities than measuring it for the whole dataset, albeit maintaining the ranking of similarities between mod- els (Brown et al., 2024). However, recent work demon- strated that representational similarities (e.g., measured via RSA) can be used to effectively select (downstream) task- specific models (Dwivedi & Roig, 2019; Borup et al., 2023). From a behavioral perspective, it has been shown that the ranking of the agreement of ImageNet classifier pairs gen- eralizes to out-o--distriuttion data (Baek et al., 2022). Due to the limited scope of prior work (c.g., only studying a few models (Klabunde al., 2023; Brown al., 2024), the driv- ing factors of the consistency of representational similarities across vision datasets have not yet been identified.
3. Methods
In this section, we propose a novel approach for analyzing the consistency of representational similarities across multiple datasets. Fig. illustraaee this framework.
Extracting representations. We are interested in compar- ing the similarities of different vision models. We perform this comparison by extracting their latent representations and examining them. Let So R2d + IR? be pretrained neural network function parameterized by a fixed set of parameters that maps the d-dimensional images dimensional vector representations. Let X E Raxd be dataset of n stacked images. For each image x E R.d in dataset, we extract its corresponding latent representations So (a:) = == E RP. For supervised models, we typically use the penultimate layer to extract this representation; for self-supervised vision models, the average pooling layer; and for image-text models, the image encoder. We denote by Z "xxp the matrix of stacked vector representations, which is unique for each model and dataset combination.
p-
Oaticnt par Fasture of model 8, Medot pinilanites
Sintivitty condoocnyyy
Computing representational similarities for a dataset. We compute the similarity between two models fo and f for a dataset by applying CKA to their vector representations Z and Z CKA computes the similarity based on the normalized Hilbert-Schmidt Independence Criterion (Gret- ton et al., 2005), applied to the kernel matrices of both representations. Using CKA with linear kernel focuses on global similarity structure, while an RBF kernel with small a measures local similarity structure (Kornblith et al., 2019a; Alvarez, 2023). We compared their behavior (shown in Fig. 3 and in Appx. F) and observed similar trends for both kernels. Therefore, we use CKA with linear kernel for the remainder of this paper if not mentioned otherwise.
ImageNet-1k (A)
fa
CKK
Similanibes ((BB.))) 9)
Flovers
P(Fcua Fiaan)
f
Correlation coofficiont
CK
[ol Suanns
4 6.00
z
all pairs (A v))
Similanites (.8.0.0)
Dstnbbttun RE8, 4)
Computing the similarity requires the full kernel matrix over the dataset, which scales quadratically with the num- ber of data points. Hence, the exact computation becomes intractable for large datasets. However, we demonstrate in Appx. C that for CKA linear a diverse subset of 10,000 samples is sufficient accurately estimate the similarities of two models for an ImageNet-scale dataset.
Figure 1: Pairwise similarity analysis framework. Let A and B be two sets of stimuli and O and + be two sets of models. For each dataset, we use all models within the two sets to extract representations, e.g., Z/A being the features extracted from dataset A using model f. Subsequently, for each pair of models (e), where fo E θ and fa E Φb, we compute the CKA similarity between their representa- tions (i.c., CK A z²x)), yielding a similarity vector (,,0,))) This vector can be displayed as a matrix, where each entry represents the similarity between two models for a single dataset. In the scatter plot, we contrast two such vectors computed on the same model sets but evaluated on different datasets A and B. The Pearson correlation coeffi- cient p between the similarities quantifies the consistency of similarities across the two datasets. The distribution of across all dataset pairs. R (0, Φ). indicates the consistency of (relative) representational similarities across stimuli.
Representational similarities across datasets. Because image representations depend on both a model and a dataset, the representational similarities between two models may vary across different datasets. To quantify how consistent these similarities remain across datasets (i.e., the transfer- ability of similarity relationships), we compute the correla- tion coefficient for pairwise model similarities (measured for large set of model pairs) between datasets. We can then analyze these correlations for specific subsets of models grouped by their training factors.
Specifically, we scrutinize how the pairwise similarities
3

---

Objective drives the consistency of representational similarity across datasets
of models from two model sets, O and Q. correspond between two datasets, A and B. By computing the pairwise representational similarities between all model pairs (fo, fe), separately for each dataset, where fo E @. fo E Φ and So # Sa. we obtain a similarity vec- tor 8 E IR*, where k denotes the number of distinct model combinations. A similarity vector for two sets of models and a dataset X is then defined as ((x,0,0)
4.1. Setup
Models. We evaluated 64 vision models, representing a di- verse range of architectures, training paradigms, and param- eter scales. We focused on general- pprppsse vision models trained on large-scale datasets (minimum ImageNet-Il lk size) with broad semantic diversity, excluding models trained on specialized or synthetic datasets. A complete list of models and their characteristics can be found in Appx. A.
(CKA(Z& x So H, E f0 fo.sot1 + (6) E RR
Model grouping. Instead of focusing on the representa- tional similarities of individual model pairs, analyze aggregated similarities between sets of models. We cate- gorize models based on four attributes: training objective, training data (size), model architecture, and model size (see Tab. 3). This allows us to examine how different model characteristies relate to the behavior of representational sim- ilarities while acknowledging that these attributes are often correlated across models (see Fig. 9).
To quantify the consistency of similarities between two datasets A and B, we use the Pearson correlation coefficient between the similarity vectors ()),,,) and ()) i.c., ())).),))
The Pearson correlation measures the degree to which the similarity trends between models are preserved across datasets, focusing on the relative po- sitioning of model pairs rather than the absolute similarity values. In other words, if two models L and In more similar than fo and f one dataset, a high Pearson cor- relation coefficient indicates that this relationship transfers to the other dataset. Conversely, a low correlation value suggests that the models' relationships shift significantly depending on the dataset, showing variability in processing different data distributions. Thus, the Pearson correlation co- efficient is meaningful indicator of how detaset-dependent or dataset-invariant the model similarities are.
Datasets. We evaluated pairwise model similarities across 20 datasets from the CLIP benchmark (Cherti & Beau- mont, 2022) and 3 datasets from Breeds (Santurkar et al., 2021). This set includes various VTAB datasets as well as ImageNet-Ik: Following the categorization proposed in (Zhai al., 2020), we classified the datasets into three main types: natural (e.g., ImageNet- specialized (e.g.. PCAM), and structured (e.g.. DTD) image datasets. Further- more, we partition the natural image datasets into single- and multi-domain categories. A list of all datasets can be found in Tab. 1.
To assess the consistency of these correlations across dataset pairs, we examine the distribution of cor- relation coefficients for all possible dataset pairs within the set of available datasets D R(O, +) {e ($(1,0,)) (())),))) A, :BE,A+ A B). spread of the correlation coefficients across all dataset pairs indicates the consistency of the relative pairwise model
The
4.2. Do representational similarities transfer across datasets?
The transferability of model similarities across diverse datasets is crucial for model comparison and selection. similarities are consistent across datasets, we can evaluate models single dataset and generalize their relationships to other datasets. We analyzed this by computing pairwise CKA similarities for all model pairs across our datasets.
similarities (similarity consistency).
4. Experiments
Our experiments investigate the consistency of represen- tational similarities between vision models across various datasets using framework. Following the experimental setup 1),.1), we first analyze how model similarities trans- fer across datasets (64.2, 4.4) and find consistent model groups through similarity clustering ($4.3). We then in- vestigate how model and dataset characteristics influence similarity consistency ($4.5 and 4.6), and conclude by exam- ining the relationship between representational similarities and models' performance differences on downstream tasks ($4.7). The code and the data to run our analyses and re- produce the experimental results are publicly available at https: / ht///gtttuu. wtw.ttuub.com cieeriieiiie k/s mimi lari ty ccnnsistency.
Model similarities (e.g., the range of values and the similar- ity trends) appear to vary across datasets (Fig. 2, Appx. D). We observe the highest similarity within image-text mod- els (yellow boxes), while group of self-supervised mod- els (white boxes) shows consistent similarity trends across datasets. Yet, there are also model groups with higher vari- ability (cyan boxes), mainly containing supervised models. The standard deviation matrix (rightmost matrix in Fig. 2) quantifies these differences,, indicating that model similari- ties do not transfer uniformly across datasets.
To further investigate the transferability of model similari- ties across datasets, examine the relationship between the mean and standard deviation of similarity values. The left panel in Fig. 3 shows an inverse U-shape trend (fur-
4

---

Objective drives the consistency of representational similarity across datasets
ImageNet-1k
Flowers
(Natural (multi-domain)) (Natural (single-domainn)
PCAM (Specialized)
CKA
Std CKA
0.80
0.15
0.60
0.10
0.40
0.20
0.00
0.00
Figure 2: Representational similarity using linear CKA. Left to right: natural multi- and single-domain, and specialized datasets, followed by mean and standard deviation across all datasets, Models (rows and columns) are ordered by hierarchical clustering of the mean matrix. Yellow and white boxes highlight regions with more stable similarity patterns across datasets, corresponding to some image-text (yellow) and self-supervised model pairs (white), while cyan boxes show higher variability for mainly supervised model pairs.
ther analyzed in Appx. E): model pairs with extreme mean similarities exhibit low variability, while variability for mid- range similarities higher. To verify that this observed variability is not an artifact of the similarity metric, we com- pare the mean similarity values obtained using linear CKA with two other metries: CKA with an RBF kernel where 0.2 (a local similarity measure, cf., Appx. F for selection) and RSA using Spearman correlation (a global similarity measure). The high correlations among these measures confirm that mid-range similarity values consis- tently vary across metrics, indicating limited stability across datasets.
observed in the t-SNE embedding of the model-model simi- larity matrices, as illustrated in Fig. 4. The training objective appears to be the most distinctive categorization, with all image-text and most SSL models forming distinct clusters across all three datasets. The objective and dataset cate- gories show substantial overlap, potentially confounding further analysis. For instance, only image-text models have been trained on XLarge datasets, while all models trained on ImageNet-211 are supervised models (Appx. A). Addi- tionally, no image-text model has been trained on ImageNer- or ImageNet-211 In contrast, no obvious clustering discernible based on architecture or model size. These ob- servations provide initial evidence that the training objective might play key role in representational similarity.
Given this variability, we investigate persistent trends in model relationships across datasets by addressing two ques- tions: a) Can we identify subgroups of models whose simi- larity to other models (consistently) cluster across different datasets? b) Do the relative representational similarities within between these subgroups remain stable/consistent datasets? Answering these questions will allow make generalizable, albeit potentially weaker, claims about models' relationships from single data source.
Training objective Img-litt Sup
Architecture ONN
Training data
Model size small large medium
INIK
XLarge
xlarge
imagetee 1K ISNE
F
6.00
Flowers TSNE 2
CA
U.S.
CA
34
15 60
34 06
46
Figure 3: Variability of representational similarities across datasets. Left: Mean versus standard deviation of pairwise similarities. Center/Right: Mean similarity values using CKA RBF (o 0.2) or RSA as function of linear CKA. High Pearson correlation coefficients indicate consistency across different similarity metrics.
PCAM TSNE 2
TSNE
TSNE
TSNE
TSNE
Figure 4: t-SNE visualization of model similarity matrices for ImageNet- 1k, Flowers, and PatchCamelyon (PCAM) datasets. Embeddings are color-coded by model attributes: training objective, architecture, training data, and model size.
4.3. Do representational similarities cluster according to model categories?
We qualitatively assess the alignment between our prede- fined model categories ($4.1) and the clustering patterns
5

---

Objective drives the consistency of representational similarity across datasets
4.4. Do relative representational similarities remain consistent across different datasets?
Training objective
Architecture
Correstion coet.
1.0
Given the observation that representational similarities are not directly transferable across datasets but depend on both the models and the dataset, we now examine the consistency of relative representational similarities across dataset pairs. Following the method described in $3, the top row of Fig. 5 reveals positive correlations between pairwise model sim- ilarities across three exemplary dataset pairs. Across all dataset pairs, the mean Pearson correlation coefficient is 0.756 (std=0. 124). This indicates that the ordering of pair- wise model similarities is largely consistent across datasets, suggesting a degree of transferability in the relationships of representational similarities. However, the high standard deviation and the distributions of similarities shown in Fig. 5 suggest the existence of model-pair groups that may exhibit distinct consistencies of similarity values.
0.0-
-0.5
TuE
Sup
TX 1X
CNN CNN
CNN TX
931
Sup
55L
Img-TSr
Sup
Sup
Training
Correctton coort
1.0-
0.0-
-0.54
IN1
IN21k
INLK
INLK XLarge
XLarge Xlarge
IN2IK Xlarge
IN21K
IN3K
IN214
IN21K
Large
XLarge
Large
Large
Model
Conrebatoo coet..
1.0-
0.5-
0.0
mediummedium modiummedium small
Xtange
large
small
Xlarge
Xlarge
small
Atarge
Figure 6: Distribution of correlation coefficients for model similarities across dataset pairs (consistency values), Each subplot represents model category: training objective, architecture, training data, and model size. Within each category, distributions show the similarity consistencies between model set combinations (e.g., R( Txt, @up))) The boxes in each subplot are sorted in decreasing median correlation. The dotted line indicates the overall median, while the gray area spans the 25th 75th percentiles of cor- relations across all model pairs (no category consideration).
The second row in Fig. 5 shows model pairs within the same training objective, highlighting differences in group- specific consistency levels. We find that SSL models show the strongest similarity consistency across datasets, image- text models form a cluster of generally high similarities, and supervised models achieve the weakest consistency. In the following section, will analyze the aggregated measure across all dataset pairs and revisit all categories in detail.
the highest similarity consistency, even when compared to models with different training objectives. This suggests that their relative similarities are most transferable across datasets, likely due to the ability of SSL to capture dataset- independent features. In contrast, supervised models show the weakest correlations, particularly when compared to image-text models (9ing-Txx 2uup), indicating a less reliable transfer of relative pairwise similarities across datasets.
Flowes
RCAM
ROM
coooft: 0.71
r codtt. 3.46
0.57
1.0
Fuxus
0.
Sup: 0.61
Sup: 0.49
r Sup: 0.45
55L: 0.56
e 551: 0.88
i 558: c. 88
IingeTa:: 0.70
Imgg-Tat 0.65
Tatg 0.74
8.0
0.5
0.5
0.0
0.5
10 0.0
0.5 Flowers
Model architecture. While no significant differences in similarity consistency exist between transformers and con- volutional networks, comparisons across architectures show slightly lower consistency, potentially indicating distinct architectural inductive biases.
ImageNee-1k
ImtagNNet-1k
Figure 5: Pairwise representational similarities for model
pairs across three dataset combinations: ImageNet- 1k vs. Flowers, ImageNet-1k k vs. PCAM, and Flowers vs. PCAM. Top row: Pearson correlation coefficients for all model pairs. Bottom row: Same data, with colored points high- lighting model pairs within the same training objective cate- gory (Ping Tal, PSLL. and Supp)
Training data. We find a low median correlation and high variance for the model set pairs (LIN2Ik, Large) and (TINZIS, x ). However, these effects may be con- founded by the training objectives, as both Drange and LLaage are mainly image-text models, while PINZIK con- sists of supervised models. On the other hand, PINIK. which includes models trained on an ImageNet-21ke likke dataset but with diverse training objectives, does not show the same trend. Consequently, training data (size) appears to have a less significant impact on the consistency of similarities than other factors. However, this applies to models trained on large and diverse datasets; as shown in Appx. I, training
4.5. Which model categories influence similarity consistency?
Fig. 6 shows the distribution of Pearson correlation coeffi- cients R(O, )) for each combination of model sets (θ, Φ) across all dataset pairs.
Training objective. Self-supervised models (Psst. show
6

---

Objective drives the consistency of representational similarity across datasets
on less diverse data noticeably affects consistency.
two matrices reveal distinctly different patterns, highlighting how model categories influence similarity consistency.
Model size. Models in the same size category show higher similarity consistency than models of different size cate- gories (e.g., 2 small and & Pxlagg). This indicates model size influences the transferability of relative similarities across datasets, though less significantly than training objective.
For SSL model pairs, we generally observe high consis- tency across most datasets, with FGVC Aireraft being an exception, showing lower correlations with the rest. This confirms that SSL models tend to produce more transferable similarity structures across datasets. In contrast, image- text/supervised model pairs show relatively higher consis- tency for multi-domain than single-domate datasets, sug- gesting better similarity transfer when semantic categories in the dataset match the training data of the models. How- ever, Pascal VOC 2007 appears as an outlier when compared to ImageNet- Ik and its subsets, likely because its bounding- box- hox-ceopped images lack contextual information, causing single-domain-lik behavior.
Our analysis identifies the training objective as a primary factor influencing similarity consistency, with model size playing less pronounced role. Self-supervised models consistently demonstrate above-average correlations across datasets. The effects of model architecture and training data (when mainly considering dataset size) are minimal, suggesting that these factors are less critical determining the transferability of relative similarities across datasets (see Appx. G for an analysis using single-element model sets controlling for model architecture and size).
Netura
Selune
Specte-
Strip- tuned
(xingleccmm)
cIPe 10
1.0
caww 120
4.6. Do dataset categories influence similarity consistency?
Country
Dreby33
Imagelles It
kimp 17
06
Previously, we analyzed similarity consistency from the perspective of models, noting substantial variation in con- sistency across all dataset pairs. In this section, we shift our focus to identifying groups of datasets that exhibit more stable relative similarities, aiming to uncover variables that might explain these variations.
korkiin 20
STL-10
voc 2237
FdVG Arcot
Prow
STSRR
Fex
8WH
Strkkrr Cas
Ratel Relin
FuestAT
FOW
The top panel in Fig. 7 shows the Pearson correlations of all model pairs for each dataset pair individually, i.c., ( (A..))) PA. (aa)) ( for all datasets pairs (A, B). The consistency of representational similarities varies strongly across dataset pairs, showing higher consistency within dataset categories than across them, as exemplified by high consistencies within natural multi-domain datasets but lower ones when paired with specialized datasets. We hypothesize that high-consistency dataset categories largely share visual features. For example, the Breeds datasets, Caltech-101, 101, Coumtry-211, STL-10, and Pets contain (often- times centered) natural objects and scenes. For CIFAR-10 and CIFAR-I00 100, the particularly high consistency may also be supported by their relatively low resolution. Moreover, some datasets inconsistent with almost any other dataset, particularly FGVC Aircraft and medical imaging datasets (Diabetic Retinopathy and PCAM) most likely due to their unique visual structure, which differs across medical domains. Interestingly, ImageNet-Il k exhibits a milder yet significant pattern of inconsistency, in particular, in combi- nation with CIFAR-10 10 and C10.
RE9EC4s
D'D
P3R2212
cane
Ingg Sup
Figure 7: Pearson correlation of representational similarities for all dataset pairs computed on different model sets. Top: all available model pairs; Bottom left: SSL models (ØSsL. Psst.) Bottom right: image-text and supervised models ( (min--Txa sap) Dataset categories are delineated by dotted lines.
Overall, we observe the highest consistency within multi- domain image datasets, suggesting that models are con- sistent when dealing with diverse semantic categories. In contrast, we find weaker consistency for (0 Txis D between multi-domain and single-domain datasets, indicat- ing that even though these models effectively generalize across diverse datasets, their representational structures dif- fer substantially when evaluated on single domains. Self-
To investigate whether specific dataset categories respon- sible for the difference in consistency observed in the last section, we focus on the correlations for the model sets with the highest ($ssL, sssL and lowest (mimg-Tx Psup) consis- tency (see the bottom left and right panels of Fig. 7). The

---

Objective drives the consistency of representational similarity across datasets
supervised models do not exhibit this sensitivity datasets; their similarity relationships appear to be more stimuli- invariant, maintaining consistent similarities regardless of whether the data is multi-domain or single-domain.
successful models in these datasets rely on capturing spe- cific discriminative features for closely related classes. Ob- serving this behavior in both structured and single-domain datasets suggests that models may depend on representing more structural attributes to distinguish between classes However, the limited number of structured datasets restricts broader conclusions.
4.7. Can representational similarity predict performance gaps?
To investigate the relationship between downstream task performance and representational similarity, we contrast two measures for each dataset: (1) the CKA value for each model pair and (2) the absolute difference in classification performance, referred to as performance gap (see Appx. B for details on performance computation). Fig. 8 shows this relationship for one dataset per category (see Fig. 21 & 22 for additional datasets). While prior work suggests well- performing models produce similar representations (Huh et al., 2024), we find this relationship is dataset-dependent.
In summary, our analysis shows different trends depend- ing the dataset. Multi-domain datasets support wider range of successful strategies, while performing well on single-domain and structured datasets appears to require a more limited set of features, resulting in stronger cor- relations between performance and similarity. This phe- nomenon aligns with findings from previous studies, which have shown that models can leverage contextual cues in multi- datasets to achieve strong performance with- out developing robust, generalizing features (Lapuschkin et al., 2019; Geirhos et al., 2019; 2020a).
Natural (multi-domain)
Natural (single-domain)
ImageNet-1k
Flowers
5. Discussion
Perforancc Gap
coeft: -0.31
0.8
coefi:
-0.76
0.5
0.8
The representations of foundation models appear to con- verge canonical representation, irrespective of their training data and objectives (Huh et al., 2024). However, this may just be an artifact of the datasets that the commu- nity commonly uses for evaluation. Thus, in this paper, we presented a systematic way of analyzing pairwise similar- ities of model representations across sets of stimuli. This approach allowed us to examine whether representational similarities generalize across datasets and identify variables influencing their consistency.
0.2
0.6
0.6
0.0
0.0
Specialized
Structured
Diabelic Retinopathy
Performance Gap
0.04
coeff.: -0.40
0.78
0.5
coefl.: -0.69
0.8
0.78
0.02
0.2
0.00
0.74
0.0
0.0
0.5
0.0
0.5
Similarity value
Similarity value
We found that the training objective of models is a central factor for determining the consistency of pairwise represen- tational similarities across datasets, whereas architecture and model size appear less important. Among training paradigms, image-text training leads to the most similar representation spaces, irrespective of the model's training data or architecture. For individual datasets, SSL model representations are not as similar among themselves as the representations of image-text models, but the pairwise sim- ilarities between SSL models and other models are highly consistent across different datasets.
Figure 8: Model similarity (CKA linear) versus absolute difference in downstream task performance (top-1 accuracy) for model pairs across four dataset categories: natural multi- domain, natural single-domain, specialized, and structured. The color of each point indicates the downstream task accu- racy of the better-perfonee perfereprforming model.
Natural multi-domain datasets, such ImageNet- 1k, show weak negative correlation between similarity and per- formance gap. Many model pairs achieve high performance despite low representational similarity (yellow dots in the lower left corner). This suggests that multi-domain datasets might provide richer contextual information, enabling mul- tiple nigh-performance strategies that do not converge to similar representations. However, correlation coefficients in this dataset category show notable variability (see Appx. J).
We hypothesize that both image-text and supervised mod- els learn distinct semantic categories during training that lead to representations whose features overfit to these cate- gories and, thus, do not generalize well to datasets that do not contain them. In contrast, self-supervised pure vision models, which are not constrained to learn explicit semantic categories during training, may develop representations that better accommodate diverse stimuli. This leads to relative pairwise similarities that vary less across datasets than the similarities of image-text supervised model representa-
Natural single-domain and Structured datasets, in con- trast, exhibit a strong negative correlation between similar- ity and performance gap. The absence of nigh-perforning, dissimilar model pairs suggests a close link between per- formance and representational similarity, indicating that
8

---

Objective drives the consistency of representational similarity across datasets
tions.
by training tasks (e.g. Lampinen et al., 2024; Kornblith al., 2021). We believe that pinpointing the variables of model training affecting (dis-)similarities between model represen- tations beyond the training dataset will ultimately help to build more interpretable systems with which humans are more likely to interact (Lake & Baroni, 2023; Sucholutsky et al., 2023; Muttenthaler et al., 2024; Tessler et al., 2024).
Alternatively, the SSL model representations, usually ex- tracted from the average pooling layer (cf., Chen et al., 2020b; Muttenthaler et al., 2023; Tian et al., 2024), may be further away from the models' task behavior than the image encoder and penultimate layer representations from image- text and supervised models. One benefit of our framework that it can be applied to the representations of any model layer, allowing future work to test this hypothesis. Further- more, in contrast to image-text and supervised models, the tasks used to train SSL models are extremely heterogeneous. This may yet be another reason for the higher consistency of representational similarities among SSL models.
A Acknowledgments
I.C is funded by the Hector Fellow Academy and is grateful for their support.
Impact Statement
Finally, we have shown that the pairwise similarities be- tween model representations are only in part predictive of their differences in downstream task behavior. While we observe strong correlation on datasets with limited con- textual information, for others, e.g., ImageNet- Ik, there is no obvious correspondence between the pairwise similari- ties of model representations and the models' differences in task performance. Moreover, the relationship between pairwise similarities and task performance is substantially different between natural multi multi-domain datasets. This is surprising in light of recent findings that have shown con- vergence of model representations as function of task performance (Huh et al., 2024) and a strong linear rela- tionship between a model's ImageNet performance (i.c., performance on the training set distribution) and its down- stream accuracy (Korbbiih al., 2019b). This suggests that the transferability of pairwise similarities across datasets follows more complex relationship than the transferability of task performance. While different objectives and hy- perparameters lead to similar task performances (Kornblith et al., 2021), they seem to change the similarity spaces of the learned representations in a way that affects the relation- ship between pairwise model similarities and differences in downstream behavior between datasets. Therefore, whether the claims of The Platonic Representation Hypothesis hold or not depends on the nature of dataset rather than being true for all sets of stimuli.
This work investigates the consistency of representational similarities across datasets and provides new insights into how stimuli and model characteristics interact. Our findings challenge assumptions about universal representational con- vergence and highlight the nuanced relationship between representations and downstream task performance, paving the way for more robust and aligned models. While these findings contribute to the advancement of machine learning, we do not foresee any immediate negative societal conse- quences requiring specific emphasis.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint M.Xi::2303.0874, 2023.
Alvarez, S. A. Gaussian rbf centered kernel alignment (cka) in the large-bandwing limit. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5) 6593, 2023. doi: 10.109/TPAM.2222.311551
Baek, C., Jiang, Y., Raghunathan, A., and Kolter, J. Z. Agreement-on-the- Predicting the performance of neural networks under distribution shift. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Process- ing Systems, volume 35, pp. 19274-19289, Curran As- sociates, Inc., 2022. URL https: //pooceeig.//....iingg.... neurips. cc wwtp:pps.cppppppppppppppppppeee http:///fieeeeepppp//////////// 40ttaaac88baaa1774000555555 aper sper-Conference.u pdf.
Conclusion. In summary, we presented a blueprint for sys- tematically analyzing pairwise representational similarities across different sets of stimuli. This allowed us demon- strate that pairwise similarities rarely transfer from one datasel to another and depend on both the models' objec- tive functions and the datasets' structure and domain. The community has long optimized for finding the optimal hy- perparameters to improve task performance (c.g. Tolstikhin et al., 2021; Steiner et al., 2022; Liu et al., 2022; Dehghani et al., 2023) but neglected its ramifications for the similarity structure of the models' representation spaces. Only few recent works investigated how representations are affected
Bansal, Y., Nakkiran, P., and Barak, B. Revisiting model stitching to compare neural representations. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 225-236 Curran Associates, Inc., 2021. URL https: : / htt////r/ceeeee htppppoeonenngeee// ngs. neurip

---

Objective drives the consistency of representational similarity across datasets
s /t/.////cc//ppp/feeeee///////// hccc//ppee..fllle/eeee//////////
URL htLpttpss/pprrceednng..eerrrg gee__flle//pperr/DD//flll////ae 9f.6e@228f/411aa2288eee. Paper.pdl
df.
Bao, H., Dong, L., Piao, S., and Wei, F. BEÎT: BERT pre- training of image transformers. International Confer- ence on Learning Representations, 2022. URL https: //openreview. net.eiiwwweeeeee...eeee.... htrr//orummmm/mmm ?icee Bh2Sz5904.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, October 2021.
Barannikov, S., Trofimov, I., Balabin, N., and Burnaev, E. Representation topology divergence: A method for com- paring neural network representations. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1607-1626. PMLR, 17-23 Jul 2022. URL URL r. r.press/ 62 htt.vvvve//avvavieeeeaaee http:/aaaannnnaaa.aaaaaaaa
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual repre- sentations, In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learn- ing, volume 119 Proceedings of Machine Learning Research, pp. 1597-1607, PMLR, 13-18 Jul 2020a. URL https: / httpp/rooeeengg..e ml press/v119/ hen20j html.
Baratin, A., George, T., Laurent, C., Devon Hjelm, R., La- joie, G., Vincent, P., and Lacoste-Julien, S. Implicit regularization via neural feature aligmment. In Banerjee, A. and Fukumizu, K. (eds.), Proceedings of The 24th International Conference Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learn- ing Research, pp. 2269-2277 PMLR, 13-15 Apr 2021. URL htwps:ttp/oomo ptattps:/pproceediiii http///rrceeeen..cccceee ml press/vi 30 000/bbratin2111 html.
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. E. Big self-supervised models are strong semi-supervised learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (cds.). Advances Neural Information Processing Systems, volume 33, pp. 22243-22255, Curran Associates, Inc., 2020b. URL https://ppooeeddagg..eeuuzo paper fitle///pper/000///ieee/aaocosop Bd181207ccc14005555-PP00 pdf.
Bardes, A. Ponce, J., and LeCun, Y. Vicreg: Variance- invariance-covariance regularization for sule-supe learning. In International Conference on Learning Rep- resentations, 2022.
Chen, X., Fan, H., Girshick, R., and He, K. Improved baselines with momentum contrastive learning. arXiv
e-prints, 2020c. https:::: http//aaxivv..vivaa//
Borup, K., Phoo, C. P., and Hariharan, Distilling from similar tasks for transfer learning on budget. In 2023 IEEF/CVF International Conference on Computer Vision (ICCV), pp. 11397-11499 2023. doi: 10. 10.1109/C 070.2023.01050.
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Gryener, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigcerver, J., Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thapliyal, A. V.. Bradbury, J., Kuo, W., Seyedhosseini, M., Jia, C., Ayan, K., Ruiz, C. R., Steiner, P., An- gelova, A., Zhai, X., Houlsby, N., and Soricut, R. PaLI: A jointly-scaled multilingual language-iman image model. In The Eleventh International Conference on Learning Rep- resentations, 2023. URL https: /op..reeeew..n et ffouum ? (t00==mWWoB2440000
Brown, D., Shapiro, M. R., Bittner, A., Warley, J., and Kvinge, H. Wild comparisons: A study of how repre- sentation similarity changes when input data is drawn from a shifted distribution. In ICLR 2024 Workshop on Representational Alignment, 2024. URL https: //openreviww.o net/ forum? id=xapkeSwqf..
Byeon, M., Park, B., Kim, H., Lee, S., Baek, W., and Kim, S. Coyo-700m: Image-text pair dataset. pettss:/git thub. com/
Cherti, M. and Beaumont, R. Clip-like model evaluation, 2022. URL https: thub. com COom/LAION-AAA CILTOONAIICIIO AI/CLIP AI/CLTPbeenchaatts bww.hharnch....
2022.
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P.,
Conwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A., and Konkle, T. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nature Communications, 1538), October 2024. ISSN 2041-1723, doi: 10.1033//41446--024 3147-y. URL https www. nature. com art
and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. In Larochelle, H.,
Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.),
Advances in Neural Information Processing Systems, vol- ume 33, pp. 9912-9924 Curran Associates, Inc., 2020.
10

---

Objective drives the consistency of representational similarity across datasets
icless/41147-0224 531477---317---- Publisher: Nature Publishing Group.
Ding, Y., Tang. J., and Guo, Identification of drug-side effect association via multiple information integration with centered kernel alignment. Neurocomputing, 325: 211-224, 2019. ISSN 0925-2312.
Csiszárik, A., Kōrösi-Szabó, P., Matszangosz, A., Papp, G., and Varga, D. Similarity and matching of neural network representations. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, vol- ume 34, pp.
doi: https://doi. wtg//0..016/j..eecom.2111. 10.028. URL https:
/ //ww.ssciencedirect...... com/ science/art. cle/pp11550922231113333
Curran Associates, Inc., 2021. URL httpsss/prooceecingg.. neurips. cc/p aper_ fetr_filee//pppeee gper_filee/spperr/00////le///// #940F47heb8@11deecb111 2aapaldeeebb16--pp-pp... pdf.
Dosovitskiy. A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition scale. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: http://ppnnrvieeee net net/ htr/m/orummmmmm Yud F FdNTTy.
Cui, T. Kumar, Y., Marttinen, P., and Kaski, S. Decon- founded representation similarity for comparison of neu- ral networks. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D.. Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp- 19138-19151. Curran Associates, Inc., 2022. URL http s: ht cctpppp.cccpppppppp es as/pppptr0002///lee////222166662 pdf.
Duong, I.., Zhou, J., Nassar, J., Berman, J., Olieslagers, J., and Williams, A. H. Representational dissimilarity metric spaces for stochastic neural networks. In The Eleventh International Conference on Learning Representations, 2023. URL https: ?id-xjb563TH /ttp::///ppeeeeeeeeeeeeee wwt.:/iwwwwweeeeeeeeeeeeeeeeeeee forum
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin,, I., Jenatton, R., Beyer, L.. Tschannen, M., Arab. A., Wang, X., Riquelme Ruiz, C., Minderer, M., Puigcerver, J., Evci, U., Kumar, M., Steenkiste, S. V., Elsayed, G. F., Mahendran, A., Yu, F., Oliver, A., Huot, F. Bastings, J., Collier, M., Gritsenko, A. A., Birodkar, V., Vasconcelos, C. N., Tay, Y., Mensink, T., Kolesnikov, A., Pavetic, F., Tran, D., Kipf, T., Lucic, M., Zhai, X., Keysers, D., Harmsen, J. J., and Houlsby, N. Scaling vision transformers to 22 billion parameters. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 Pro- ceedings of Machine Learning Research, pp. 7480-7512. PMLR, 23-29 Jul 2023. URL httts:://proceeetee ngs. m1r wrwsssrssp........ htt///22/////// http////////gg//22a.aaaa
Dwivedi, K. and Roig, G. Representation similarity analysis for efficient task taxonomy & transfer learning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12379-12388, 2019. doi: 10.1 109/CVPR.2019.0122
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wich- mann, F. A., and Brendel, W. Imagenet-trained CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Confer- ence on Learning Representations, 2019. URL https: / //openreviee ew. net. forum? d=Rggh
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Bren- del, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intel- ligence, 2(11):665-673 Nov 2020a. ISSN 2522-5839. doi: 10.11038/42256-00-0022222 //doi. org wtwp///.0.... 1038 10011110000000000000000------111 000022255560000002---222000
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 248 255, 2009. doi:
URL https:
Geirhos, R., Meding, K., and Wichmann, F. A. Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. In Larochelle. H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Sys- tems, volume 33, pp. 13890-13902 Curran Associates, Inc., 2020b. URL URI.ttpps:/pprcceeddrss neur hpppo./ccpppee.llleeeeseerr///// (G925664433aaa0022205cc55------- .pdff
Ding, F., Denain, J.-S., and Steinhardt, J. Grounding repre- sentation similarity through statistical testing. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 1556-1568. Curran Associates, Inc., 2021. http.//RR.tttte//rrreeeessssee.
s. s.cc/paper_ B.ccc//ppee__flleee/pppe///////p 1.5.7779225500088771962222222 df..

---

Objective drives the consistency of representational similarity across datasets
Geirhos, R., Narayanappa, K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F. A., and Brendel, W. Partial success in closing the gap between human and machine vision. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. Advances in Neural Information Processing Systems, volume 34, pp. 23885- 23899. Curran Associates, Inc., 2021. URL https:
NeurIPS, 2022. URL https: / /forum? (errum?iddd/bHlco5Fqqd
/ttp:///oppeeeeeeeeeeee net
Hotelling, H. Relations Benween Two Sets of Variates, pp. 162-190. Springer New York, New York, NY, 1992. ISBN 978-1-4612-4333 380-9 URL https:
doi: 10.1007/998-11461222
wtw..oogg/111.1 1007//97 8-1-4622 4-80-930-99 9_14.
//proceedings.... wtw.:eeeiiing...eeiiie.. wttp:/apppppcccpppp__eeeee files /ppper//00a//Ellee///87222222//2
Hu, J., Shen, L., and Sun, G. Squeeze-andd-xcitation net- works. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132-7141, 2018. doi: 11110//V/..211.000777
Gidaris, S., Singh, P., and Komodakis, N. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018. URL https: /ttp///eeeeeeeeeeeeeeeeeeeeeeeem wtt.eeww..ee..rrreeeeeeeeee forum Pid-s1v4N210-
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely Connected Convolutional Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2269, Los Alamitos, CA, USA, July 2017. IEEE Computer Society. doi: 10.1109/ CVPR.2017.245. URL httpt: / ieeecomput e ersociety: wrg.g.orgg/e http://1.11109// 1w09/01099..P0 2017.22 .243.
Gretton, A., Bousquet, O., Smola, A., and Schölkopf, B. Measuring statistical dependence with hilbert-schmidt norms. In Jain, S., Simon, H. U., and Tomita, E. (eds.), Algorithmic Learning Theory, pp. 63-77, Berlin, Heidel- berg, 2005. Springer Berlin Heidelberg.
Huh, M., Cheung, B., Wang, T., and Isola, P. Position: The platonic representation hypothesis. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 20617-20642. PMLR, 21-27 Jul 2024. URL https: / //proceedings ml wwtppsswws/////.... html.
Hacohen, G., Choshen, L.,, and Weinshall, D. Let's agree to agree: Neural networks share classification order on real datasets. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference Machine Learn- ing, volume 119 of Proceedings of Machine Learning Research, pp. 3950-3960. PMLR, 13-18 Jul 2020. URL https: acohen20a.. html.
mlr. prwss sagess/vlls /v119/h
KakaoBrain. Align- base model, 2023. URL https:// huggingface co ht//kaaacbbain/alignnn Accessed: 01.09.2024.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10. 10.1109/CVR..0066.999
Klabunde, M., Amor, M. B., Granitzer, M., and Lemmerich, F. Towards measuring representational similarity of large language models. In UniReps: the First Workshop on Unifying Representations in Neural Models, 2023. URL https: petps::/opennee ht////pperrveeeeeee ew. net/ forum? d=gG50 GgUzDR.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15979-15988, 2022. doi: 10. N0.1109/CVPRS2288.2222222
Klabunde, M., Schumacher, T., Strohmaier, M., and Lem- merich, F. Similarity of neural network models: A survey of functional and representational measures. ACM Com- pul. Surv., 57(9), May 2025. ISSN 0360-0300. doi: 10.1145/3728458. URL https://doi. org/10.1 145/3728458.
Hermann, K. and Lampinen, A. What shapes feature rep- resentations? exploring datasets, architectures, and train- ing. In Larochelle, H., Ranzato, M., Hadsell, R., Bal- can, M., and Lin, H. (eds.), Advances in Neural In- formation Processing Systems, volume 33, pp. 9995 10006. Curran Associates, Inc., 2020. URL https: //ppoccediigg..coo. .nw.iiis.e..ppppp htcpc/ppppaacccccccccccaccaaetc hwtp:/apper.eee pallppapeelleeees__el files (ppper//000//11e/77/e::. ebe694B40aaaa-Paaeee pdf.
Korbllihh, S., Norouzi. M., Lee, H., and Hinton, G. Sim- ilarity of neural network representations revisited. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceed- ings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learn- ing Research, pp. 3519-3529. PMLR, 09-15 Jun 2019a URL https: http:::pppppooceeeiiee mllr. presspesssss... v kornblithi 9a html.
Hernandez, A., Dangovski, R., Lu, P. Y., and Soljacic, M.
Model stitching: Looking for functional similarity be-
tween representations. In SVRHM 2022 Workshop @
12

---

Objective drives the consistency of representational similarity across datasets
Korbllihh, S., Shlens, J., and Le, Q. V. Do better imagenet models transfer better? In 2019 IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2656-2666, 2019b. doi: 10. 10..1199...2219.00000
Lee. J., Sohl-dickstein, J., Pennington, J., Novak, R., Schoenholz, S., and Bahri, Y. Deep neural networks gaussian processes. In International Conference on Learning Representations, 2018. URL https: open review, www.ew.eeeeeeeeeeeme www. forum d= RA-M-A2 OZ.
Korbllith, S., Chen, T., Lee, H.,, and Norouzi, M. Why do better loss functions lead to less transferable fea- tures? In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp- Curran Associates, Inc., 2021. URL. https://proceddii www.wps..ciicccee........oooo.. pappaper fi filce//apprr/0001//1///002222222 28910047:331bcc22e911---
Li, Y., Yosinski, J., Clune, J., Lipson, H., and Hopcroft, J. Convergent learning: Do different neural networks learn the same representations? In Storcheus, D., Ros- tamizadeh, A., and Kumar, S. (eds.), Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges at NIPS 2015, vol- une 44 of Proceedings of Machine Learning Research, pp. 196-212, Montreal, Canada, 11 Dec 2015. PMLR. URL https: httpt:::tpccceeedaeeeie http::///coccceeeeeeeeeeeeeeeeee www. www..peees........... 1 1 111cconerrentt..c.mm html.
Kriegeskorte, N., Mur, M., and Bandettini, P. A. Representa- tional similarity analesis-connectingg the branches of sys- tems neuroscience. Frontiers in Systems Neuroscience, 2, 2008. ISSN 1662-5137. doi: 10.3389/eeum000600422000 URL httptttttt//tttttttee front wwwp:niieiiiiiinniioioooee
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
hatp://yeccees-nmeurrreceieeaaaa b http://wwwwr.rrrirrrrrrnro/rr/rr 0289/neuro.06..0 06.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Pereira, F., Burges, C., Bottou, I.., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012a. URL https::/prrcceedigg..oo f fflles//opperr//2///ille///33 b76 976c8436e924488ccbbbb
Liu, Z.. Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 688.2022.01167.
2022. doi: 10.1109/CVPR52
ccppper
Loshchilov, I. and Hutter, F. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL https: enreview. net staeet/oooumiiid-kk333332
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Pereira, F., Burges, C.. Bottou, L., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012b. URL paper
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference Learning Representations, 2019. https:/ htt.htstss//opereviii openreview .ne
https://proceeddngg..comm htwp::eeeediinn.....eeeeee _flles fftlee//ppperr//0//////al//39399 b76c8445e9244664555--533 -
Maniparambil, M., Akshulakov, R., Dahou Djilali, A., Seddik, M. E. A., Narayan, S., Mangalam, K., and O'Connor, N. E. Do vision and language encoders repre- sent the world similarly? In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14334-14344 2024. doi: 10. 20.109//CPPR5233.2224..0 1359.
Lake, B. M. and Baroni, M. Human-like systematic general- ization through a meta-learning neural network. Nature, 023(7985):115-12) 121, 2023.
Lampinen, A. K., Chan, S. C., and Hermann, K. Learned fea- ture representations are biased by complexity, learning or- der, position, and more. Transactions Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https: //openreview net uorum? hetmmmidddaananse..mmm
Matthes, A., Hron, J., Rowland, M., Turner, R. E., and Ghahramani, Z. Gaussian process behaviour in wide deep neural networks. International Conference on Learning Representations, 2018. URL https::p. / /open forum? 1id:1dd------ -ndp--
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., and Müller, K.-R. Unmasking clever hans pre- dictors and assessing what machines really learn. Nature Communications, 10(1):1096, Mar 2019. ISSN 2041- 1723. doi: 10. 10.00888441467--119-09997------9 URL ht tps: //doi.orr// 1038
review.ned www./eettuutttto.
Mchrer, J., Spoerer, C. J., Kriegeskorte, N., and Kietzmann, T. C. Individual differences among deep neural net- work models. Nature Communications, 11(1):5725, 2020. ISSN 2041-1723. doi: 1.11038/441467-0225 19632-w.
13

---

Objective drives the consistency of representational similarity across datasets
URL wwtps:ttpt/ooeeee /tts:://ddooo 20-19633 20-19632-w.
http://1...18////// /tt.:tttte///oo.ooro//..////////
Oquab, M., Darcet, T., Moutakanni, T., Vo. H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA,
D., Massa, F,, El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Je- gou, H., Mairal, J., Labatut, P., Joulin, A., and Bo- janowski, P. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856, URL https: / //openreview..o net
Merullo, J., Castricato, L., Eickhofff, C., and Pavlick, E. Lin- early mapping from image text space. In The Eleventh International Conference on Learning Representations, 2023. URL httpt:://////..... peettts://ppereeee /pporeveeeivv net/ forum ?id=8t VR..
Misra, and van der Maaten, L. Self-supervised learning of pretext-invarian representations. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6706-6716, 2020, doi: 10.1109/C 0.2020.00674.
Princeton University. About WordNet. https: setps://word /wwrrd net www.icttoe.dd/... 2010.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8766 PMLR, 18-24 Jul 2021. URL S s/v139/ ht1.3//rafford21a.. html.
Morcos. A., Raghu, M., and Bengio. S. Insights rep- resentational similarity in neural networks with canoni- cal correlation. In Bengio, S.. Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Sys- tems, volume 31. Curran Associates, Inc., 2018. URL paper
https://proceedinss.eeeees fftlee//ppprr/000/////e////72182 73140918996003 032303135600c014400 PaA0011199900000144Paeeerrr....
Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. Sveca: Singular vector canonical correlation analy- sis for deep learning dynamics and interpretability. In Guyon, I,, Luxburg, V., Bengio, S., Wallach, H., Fer- gus, R., Vishwanathan, S., and Garnett, R. (eds.), Ad- vances in Neural Information Processing Systems, vol- une 30. Curran Associates, Inc., 2017. URL https: htpp:///ggnnnersss.cccppooprrrre /pppper/007/filleaa6666665586883 F33e9ee66ccc9-Paper
Muttenthaler, L. and Hebart, M. N. THINGSvision: A python toolbox for streamlining the extraction of activa- tions from deep neural networks. Frontiers in Neuroinfor- matics, 15:45, 2021. ISSN 1662-5196. doi: 10.3389/fnin f.2021.679838. URL https: /www frontiersinn org/article/ 10 3389/fnnnfff////// 2021. 2021.679838...
Muttenthaler, L., Dippel, J., Linhardt, L., Vandermeulen, R. A., and Kornblith, S. Human alignment of neural network representations. The Eleventh International Conference on Learning Representations, 2023. URL tete/foorume RddDRDD
Raghu, M., Unterthiner, T., Komblith, S., Zhang, C., and Dosovitskiy, A. Do vision transformers see like convo- lutional neural networks? In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems, vol- ume 34, pp. 12116-12128. Curran Associates, Inc., 2021. cc/p
10UQROX.
Muttenthaler, I.., Greff, K., Born, F., Spitzer, B., Korbblith, S., Mozer, M. C., Müller, K.-R., Unterthiner, T., and Lampinen, A. K. Aligning machine and human visual representations across abstraction levels. arXiv preprint arXiv:2409.06509, 2024.
URL httphttpp///prcccciing..coo asee_sileee/ppeer////////eeeee// 1a23988302222228811111100222
Ridnik, T.. Ben-Baruch, E., Noy, A., and Zelnik, Imagenet-21k pretraining for the masses. In Vanschoren, J. and Yeung, S. (eds.), Proceedings of the Neural In- formation Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
Nguyen, T., Raghu, M., and Kornblith, S. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference Learning Representations, net forumm
2021. URL httpt::////// /pwp//vveeeeeeeeeee//mee net ? 2id=K.NNcAkYBtY.
t.s tss-banccerrrs--pooeeeeirngggggo
pagpr__flleee/pper///0//////1112 (0119ccc4556771111002244444----3 pdf.
Noroozi, M. and Favaro, P. Unsupervised learning of visual representations by solving jigsaw puzzles. In Computer Vision - ECCV 2016. pp. 69-84. Springer International Publishing, 2016.
Saha, A., Bialkowski. A. N., and Khalifa, S. Distilling representational similarity using centered kernel align-
14

---

Objective drives the consistency of representational similarity across datasets
ment (cka). In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022. BMVA Press, 2022. URL https: http::///bwvveee http///ww2222..2///////// mpi -inf. mpg. de 00355.10 .ddf.
Sun, Q., Fang, Y., Wu, L.., Wang, X., and Cao, Y. Eva- clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303. 2023.
Tan, M. and Le, Q. EfficientNet:
Rethinking model scaling for convolutional neural networks. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th Inter- national Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 6105- 14. PMLR, 09-15 Jun 2019. URL httpss: /ttppps://rroeo/ /proc eedings mll press v27 attnaaa155 aallan9...... aa. html.
Santurkar, S., Tsipras, D., and Madry, A. BREEDS: bench- marks for subpopulation shift. In 9th International Con- ference on Learning Representations, ICLR 2021. Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. net/forum?id
URL htttps://opeereeeeeeeee m@PBmvyAuk.
Schuhmann, C., Kaczmarczyk, R., Komatsuzaki, A., Katta, A., Vencu, R., Beaumont, R., Jitsev, J., Coombes, T., and Mullis, C. Laion-400m: Open dataset of clip-filterd 400 million image-text pairs. In NeurlPS Workshop Datacen- tric AI, number F7J-2022-00923, Jülich Jüllch Supercomputing Center, 2021.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.10 1805, 2023.
Tessler, M. H., Bakker, M. A., Jarrett, D., Sheahan, H., Chadwick, M. J., Koster, R., Evans, G., Campbell- Gillingham, I.., Collins, T., Parkes, D. C., Botvinick, M., and Summerfield, C. Ai can help humans find common ground in democratic deliberation. Science, 386(6719): cadq2852, 2024. doi: 10.1 12l.sciinncc..uq222222 URL https: /www://wwwwww// science. wwt//oo/gooooooooooooooooooooooo /aloo//ab///i 10. 126/ btt./siinnce..cc222 adq2852.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C.,, Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crow- son, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J. Laion-5b: An open large-scale dataset for training next generation image-text models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- ume 35, pp. 25278 25294. Curran Associates, Inc., 2022. URL htw.tttpppp.. httpttts://ppppteee http::::/ppoceeedddege htw.::eeedddng.......uue neurips. cc/p aper fter_fiilespppperr//ee appe_//leee/ppprr//002/////aaaa
Tian, Y., Krishnan, D., and Isola, P. Contrastive representa- tion distillation. In International Conference Learning Representations, 2020. httpttt::/ openreview .net/for http///rrmmmidkkkppp/ee
bibb9d09ff339444eee3223233333333 s_and_ Sond_Benchmarks.co
Tian, Y., Fan, L., Chen, K., Katabi, D.,, Krishnan, D., and Isola, Learning vision from models rivals learning vision from data. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 15887-15898, June 2024.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: http///axxvvvvee org/ abs/14099 155 6.
Tolstikhin, I. O., Houlsby. N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., Lucic, M., and Dosovitskiy, A. Mlp- mixer: An all-mip architecture for vision. Ranzato, M.. Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Process- ing Systems, volume 34, pp. 24261 Curran As- sociates, Inc., 2021. URL https: httpttpss//ppccceeinggg neurips cc ft http://ppppeee.leeeee////rr///// 1e /tt/://feeescc22efaaaeeeaaaaaaa0
Steiner, A. P., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., and Beyer, L. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https: /ttp///pppeeeeeee//////ee net/f orum? 4 kwrk
Sucholussky, I., Muttenthaler, L., Weller, A., Peng, A., Bobu, A., Kim, B., Love, B. C. Grant, E., Groen, I., Achterberg, J., Tenenbaum, J. B., Collins, K. M., Hermann, K. L., Ok- tar, K., Greff, K., Hebart, M. N., Jacoby, N., Zhang, Q., Marjieh, R., Geirhos, R., Chen, S., Kornblith, S., Rane, S., Konkle, T., O'Connell T. P., Unterthiner. T., Lampinen, A. K., Müller, K.RR., Toneva, M., and Griffiths, T. L. Get- ting aligned on representational alignment, 2023. URL https: /tw.ivv..vv/v/////////.////iiavl org/
aper
Touvron, H., Cord, M., Douze, M., Massa, F., Sablay- rolles, A., and Jegou, H. Training data-efficient im- age transformers & distillation through attention. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Re- search, pp. 10347-10357. PMLR, 18-24 Jul 2021. URL
org/abs/////ooooooro.o.oos....br abs 2210 13018.
15

---

Objective drives the consistency of representational similarity across datasets
httpss:/prooeeeingg... mlr. wwwpppressslll wtt.mmtt...sss////////m Ouvron2la.html
Zong, M., Qiu, Z., Ma, X., Yang, K., Liu, C., Hou, J., Yi, S., and Ouyang, W. Better teacher better student: Dy- namic prior knowledge for knowledge distillation. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL / /oormm? URL.https://openeeee http///pppeeeeeeeeeeeeeee net.
Williams, A. H., Kunz, E., Korbblihh, S., and Linderman, S. Generalized shape metrics on neural representations. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (cds.), Advances in Neural Information Processing Systems, volume 34, pp. 4738 4750. Curran Associates, Inc., 2021. URL httptttss////pooeeediii
s sereerrips..cccpppeeefffeeeepppp file #ii1:/552accee322e15222222223333 b-Faper.. pdf.
Xu, Y. and Vaziri-Pashkkm, M. Limits to visual repre- sentational correspondence between convolutional neu- ral networks and the human brain. Nature Commu- nications, 12(1):2065, 2021. ISSN 2041-1723, doi: URL https: /oo i. orgg 201////44466-266 021 0011677722-22------ 22244-7
10.
Yang, G. and Hu, E. J. Tensor programs iv: Feature learning in infhh neural networks. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Confer- ence on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11727-11737 PMLR, 18-24 Jul 2021. URL https::///.... tttttt:://pproeeeeeee http:///ooceeieniee/ee/e ml r.press /vavvvvv//0 139 http://agggnnnnnmmmmm html..
Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In Meila, M. and Zhang. T. (eds.), Proceed- ings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learn- ing Research, pp. 12310-12320, PMLR, 18-24 Jul 2021. URL htwps::t////et htttp:://prrccedingge. mlr press/vl 2992bontarr2a.httt.
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., Beyer, L.., Bachem, O., Tschannen, M., Michalski, M., Bousquet, O., Gelly, S., and Houlsby, N. A large-scale study of represen- tation learning with the visual task adaptation bench- mark. arXiv preprint arXiv: 2020. URL https: /ttts://aatt /ttp///aavvvv/// httpaaawww......////o///aaaaaaaa atgg/bbs/1911 04867.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), pp. -1975-11986
October 2023.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Tor- ralba, A. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4066)::452-1466 2018, doi: 10.1 100/TPAM...0..2772202
16

---

Objective drives the consistency of representational similarity across datasets
Appendix
A. Models and Datasets
We considered 23 downstream datasets (Tab. 1) and 64 pretrained vision models (Tab. 2) in our analysis. For each dataset, we selected the training split, potentially subsetted for large datasets as described in Appx. C. compute the model representational similarities, while
the validation or test split to compute downstream performance.
Table 1: Datasets of tlip-benchmart and breeds and their domains according to (Zhai et al., 2020) with additional separation of natural images into multi- and single-domain.
Natural (multi-domain)
Natural (single-domain)
Specialized
Structured
Caltech101
Cars
Diabetic Retinopathy
FER2013
CIFAR-10
FGVC Aircraft
FuroSAT
Dmlab
CIFAR-100
Flowers
PatchCamelyon (PCam)
DTD
Country211
GTSRB
RESISC45
ImageNet-Ik 1k
Pets
STL-10
SVHN
Pascal VOC 2007
Entity-30 (Breeds)
Living-17 17 (Breeds)
Nonl .ving-2 (Breeds)
Models. The models were chosen to represent a diverse set of training datasets, training objectives, model sizes, and architectures. Training objectives include standard supervised learning (Sup), self-supervised learning (SSL), and image-text alignment Ling-Txt), The models have been trained on datasets spanning four classes of increasing scale and semantic diversity. The most focused class consists of ImageNet- 1k (INIK; (Deng al., 2009)) with 1,000 object categories, followed by models trained on ImageNet-211 kk (IN21K; (Ridnik et al., 2021)) or combination of IN21K and INIK, expanding to 21,000 categories. The "Large" class encompasses COYO-700M (Byeon et al., 2022), LAION400M (Schuhmann et al., 2021), LVD-142M (Oquab et al., 2024), and WIT-400M (Radford et al., 2021), which move beyond curated object categories to include broader visual concepts and text descriptions. Finally, the "XLarge" class includes LAION2B (English subset of LAIONSB (Schuhmann et al., 2022)), Merged2B (merged version of LAION2B and COYO-700M; (Sun et al., 2023)), and WebLI (Chen al., 2023), which contain billions of image-text pairs covering an extremely wide range of visual and semantic content. In terms of architecture, the models employ range of designs, including Convolutional Neural Networks (CNNs) such as ResNet, EfficientNet, ConyNeXt, and VGG, as well as Transformer-basee models like ViT and Further, we divide the models into four different size categories, ranging from small, with around 12 million parameters, to xlarge, with over 1.4 billion parameters. Tab. 3 summarizes the number of models in each category, and Fig. shows the distribution of models across pairs of categories, highlighting the relationships between different model attributes.
Table 3: Number of models in each attribute category.
Category
Nr. models
Category
Nr. models
Training objective
Training data
Image-Tec Text (Img-Txt)
14
ImageNet-Ik (INIK)
37
Self-Supervised (SSL)
20
ImageNet-21k (IN2IK)
9
Supervised (Sup)
30
Large
11
XLarge
7
Architecture Class
Model Size
Convolutional (CNN)
24
small 100M parameter
32
Transformer (TX)
40
medium < 200M parameter
large < 400M parameter
8
xlarge > 400. parameter
10
Preprocessing. To extract the latent representations for each model-dataset combination, we used the Python package
17

---

Objective drives the consistency of representational similarity across datasets
chingsvision sion (Muttenthaler & Hebart, 2021). Images were resized to 256px and center-cropped 224px before applying the model-sppcific normalizations from the pretraining. Tab. 2 specifies exactly which layers used for each model. For VIT models, used the CLS token representations after the final LayerNorm module. We applied L2- 12-normalization to ensure all feature vectors are of unit length.
Training objective vs. Architecture Training objective vs. Training data Training objective vs. Model size
Img-Txt-
2
12
Img-Txt-
7
Img-Txt-
6
7
SSL
4
SSL
0
2
13
Sup
0
Sup
8
CNN
INIK
IN21k
Large
XLarge
small
medium
largo
Marge
Architecture vs. Training data
Architecture vs. Model size
Training data vs. Model size
CNN
22
CNN
IN21K
2
2
4
1
Large
3
3
4
TX
15
9
7
TX
12
10
8
XLarge
3
INIK
IN2IK
Large
XLarge
smail
medium
large
xlarge
small-
medlum
large:
xlarge
Figure 9: Distribution of models across paired categories. Each heatmap shows the count of vision models for different
combinations of model categories: training objective (Sup: supervised, SSL: self-supervised learning, Img-Txt: image-text), architecture (CNN: convolutional neural networks, TX: transformers), training data (INIk: ImageNet-Ik, Ik, IN21K: ImageNet- 21k, Large/XLarre: larger datasets), and model size (small xlarge)
B. Downstream task performance
We evaluated each model's downstream task performance on every dataset by calculating the top-1 accuracy of linear probe trained on features extracted from the corresponding training set. Each model was evaluated with 3 different random seeds on each dataset, and the mean top-1 accuracy across seeds was used. Hyperparameter selection was performed with validation set of 20% of the training set and the remaining training data for optimization. We followed the binary search procedure descrihed in (Radford et al., 2021) and searched for the optimal weight decay parameter À in the interval between 10-6 and 102 in 96 logarithmically spaced steps. This was done for all learning rates E {10') 34 3 After hyperparameter selection, the linear probe was retrained on the full training set and evaluated on the respective test set (validation set for Imagenet-1k), All linear probes trained for 20 epochs, using the AdamW optimizer (Loshchilov & Hutter, 2019) and cosine schedule for learning rate decay (Loshchilov & Hutter, 2017).
C. CKA sensitivity to the number of samples in dataset
Our analysis of model similarities requires reliable CKA measurements while managing computational constraints. To identify the minimum number of samples needed for stable CKA values, we analyzed ImageNet- subsets with varying samples per class e 1,5, 10, 20, 30, 40). We generated pairwise similarity matrices (Fig. 10) and assessed stability by comparing the absolute differences between consecutive sample sizes (Fig. 11). The results show that most CKA variants converged with 10,000 samples (k=10), with only CKA RBF (o 0.2) requiring more samples (30,000, k=30) for stability.
To further assess CKA stability across different samples, we conducted bootstrapping experiments using six representative models (OpenCIIP ViT-L, OpenCLIP RN50, ResNet-50, ViT-L, DINOv2 ViT-L,, and SimCLR RN50 the same anchor models described in Appx. G). For CKA linear, we performed 500 bootstrap iterations with 10, 0,0000 images per subset. For CKA RBF (a 0.2), we performed 500 bootstrap iterations with 30,000 images per subset. For each bootstrap sample, we
18

---

Objective drives the consistency of representational similarity across datasets
extracted features from all six models and computed CKA values between each model pair.
Fig. 12(a) confirms the stability of CKA linear values across all model pairs, while for CKA RBF (a 0.2), Fig. 12(b) reveals that even when using the larger 30,000 samples per subset, we still observe slightly higher variability compared to CKA linear.
These findings indicate that local similarity measures (RBF kernel) are more sensitive to specific stimuli than global similarity measures (linear kernel), This aligns with theoretical expectations, as stimulus-specific fine-grained details have a greater influence on local similarity measurements.
Based these findings, applied stratified subsampling to limit all datasets to 10,000 30,000 samples for CKA linear and CKA RBF (a 0.2), respectively.
CKA RBF 0.2
CKA RBF 0.4
CKA RBF 0.6
CKA RBF 0.8
CKA linear
Sue pa clas samples per samples per
dae 10 Conoona
soyduus aee
5580 02
2 Comnooe (0)
Figure 10: Pairwise model representational similarity matrices using different dataset sizes (1-40 samples per ImageNet-IK Ik class) and metrics (CKA RBF 0.2-0.8 to linear).
19

---

Objective drives the consistency of representational similarity across datasets
CKA RBF 0.2
CKA RBF
Abs. Similarity Difference
0.8
CKA
linear
0.6
0.4
Lممما
0.0
a.a.a....
and 5k
5k 10k
10k and
20k and 30k
30k 40k
Figure 11: Distribution of absolute differences of pairwise model representational similarities for each CKA variant measured on subsets of ImageNet- Ik with consecutive sample sizes per class (1-+5, ... 30~+4), Most CKA variants converge at 10 samples per class, with CKA RBF (a 0.2) requiring larger sample sizes for stability.
linear
0.8
0.8
0.7
0.7
0.5
0.5
OpenCLIP RN50. Open VIT-L
OpenCLI RN50, DINOv2 VIT-L
OpenCurP RN50, SimCLR RNSO
OpencuI RNSO Restett-S 5o
OpenC RN50, VIT-L
OpenCLIP VITL, DINOv2 VIT
OpenCLIP VITL, SimCLR RNSO
OpenCLIP vIr-,, VIT-L
DINOv2 VITL. SimCLR ROISO
DINO22 VITLL ResNet 5o
SimCLR RN50, ResNet-50
SimCLA RA50, VIT-L
ResNes-50, VIT-L
OpenCLIP RNSO. OpenCLiP VITAL
OpenCLI RNSO, DINOv2 VIT-L
OpanCUrP RN50, SimCLR RNSO
OpenCLI LIP RAS0, VIT
OpenCLiP VIT., DINOv2 V.T
OpenCLIP VITL, SimCLR RNSO
OpenCLI ResNet:50 So
OpenCLI VIT-L, VIT-L
DINOv2 VITL, Sim/LR RNIS0
DINOVZ VIT-L, ResNet-50
DINOV2 VIT-L, VIT-L
SimCLR RN50, ResNet-50
SimCLA RA50, VIT-L.
ResNet:50, VIT-L.
OpenCLip Rostiet
OpenCLIP ResNet:
DINOv2
(a) CKA linear
(b) CKA RBF (a = 0.2)
Figure 12: Distribution of CKA values between model representations across 500 randomly sampled ImageNet- 1k subsets
for different model pairs. (a) Linear CKA computed on 10k samples per subset. (b) RBF CKA (a 0.2) computed on 30k samples per subset.
20

---

Objective drives the consistency of representational similarity across datasets
D. Similarity matrices for different datasets
4.2, we showed that similarity measurements are not directly transferable across datasets due to their high variability, Fig. 2 displays representational similarity matrices computed using CKA linear across three selected datasets, along with the mean and standard deviation matrices across all evaluation datasets. Fig. 13, show additional similarity matrices for three representative datasets from each dataset category (see Appx. A for details). The yellow-boxed models, containing image-text models, are most similar on natural image datasets. In contrast, the white-boxed models, containing self-supervised models, show the highest similarity on specialized AuroSAT and PCAM datasets. The cyan-boxed models, containing supervised models, are most dataset dependent, showing high similarity on Pets but low similarity on ImageNet-Ik.
ImageNet-1k
CIFAR-100 100
Caltech-101
1.0
Natural (multi-domain)
0.6
0.4
0.0
Flowers
Pets
Stanford Cars
0.8
Natural
0.6
(single-domain)
0.2
EuroSAT
PCAM
Diabetic Retinopathy
1.0
0.6
Specialized
0.4
0.0
FER2013
Dmlab
DTD
1.0
0.6
Structured
0.4
0.0
Figure 13: Representational similarity of model pairs, calculated with linear CKA. For each dataset category, three evaluation
datasets are displayed.
E. Variation of CKA values near the upper bound
The CKA similarity measure is bounded between 0 and 1, with values close to indicating highly similar representations. However, this upper bound might lead to saturation effects. better differentiate between highly similar model represen- tations, we apply two nonlinear transformations: arccos and tan. Both transformations expand the range of high CKA
21

---

Objective drives the consistency of representational similarity across datasets
values, with lan slightly reducing variation for low similarity values. These transformations amplify the inverse U-shape relationship noted in Section 4.2 (Fig. 14), suggesting that the relationship between mean and variance of similarities is robust across different scales.
Std (CKA linear)
Std tan(CKA linear)
0.25
0.20
0.20
0.15
0.15
0.10
0.10
0.4
1.2 Mean arCcos(CKA linear)
0.6
0.8
1.0
1.4
0.25
0.50
0.75
1.00
1.25
Mean tan(CKA linear)
Figure 14: Mean versus standard deviation of CKA values after nonlinear transformations. Left: arccos transformation inverts the scale, mapping high similarities to low values and vice versa. Right: tan transformation. Both transformations make the inverse U-shape relationship more pronounced.
F. Influence of similarity metric on similarity consistency
In our main analysis, we used CKA linear as a global similarity metric. Fig. 3 shows strong positive correlations between mean representational similarities across datasets when measured with different metries. Here, we investigate whether this stability extends similarity consistency distributions when using alternative metrics.
1.
2:00
1.2
0.F-
piaalee 0.001
9.75
p-oaee
0.001
p-valz 0.091
p-o UE 0.001
T028t AUY 244
FOUTT IISA spcca MANO
9.99
0.E-
3.25-
6.1-
4A-
3.00
02
-0.5
0:50 rcaeff (CKA Fneud)
6.00
635
-0.35
-0.25
8.50 Coolf (CKA (nea))
3.50
-0.5
1.00
-4.-
-025
6.00
ods
650
rcoeft (CKA RSF 0.21
Cooff (CSA (near)
Figure 15: Relationship between similarity consistency values measured with different similarity metrics across all model sets and dataset pairs. From left to right: CKA linear vs. CKA RBF 0.4), CKA linear vs. CKA RBF (a = 0.2). CKA RBF (a 0.2) vs. CKA RBF (a 0.4), and CKA linear vs. RSA Spearman. Correlations are measured using Pearson correlation. Each point represents the similarity consistency value for pair of datasets and model sets.
Local similarity metrics When examining local similarity structures, we leverage the flexibility of CKA with an RBF kernel. The kernel bandwidth parameter a controls sensitivity to local structure, allowing us capture different aspects of representational similarity. Notably, Fig. 10 reveals that pattern differences between CKA linear and CKA RBF (a 0.2) substantially more pronounced than those between CKA linear and CKA RBF (a 0.4). This suggests that the latter partially captures global similarity structures rather than purely local ones. Indeed, the correlation between similarity consistency values measured with CKA linear and CKA RBF (a = 0.4) is strong (Fig. 15, leftmost panel), while is substantially lower between CKA linear vs. CKA RBF (a = 0.2) (Fig. 15, second panel). The correlation between similarity consistency values measured with the two different sigmas (a = 0.2 and a = 0.4) (Fig. 15, third panel) in between, strengthening the hypothesis that CKA RBF (a 0.4) partially captures global structure. Therefore, we use CKA RBF (a 0.2) as local similarity metric.
We observe the same overall pattern as in Fig. 6 of the main text for the CKA RBF (o 0.2) similarity metric: the objective is a main influencing factor for similarity consistency, while network architecture and model size are less important (Fig. 16), However, the distributions of consistency values reveal two interesting observations. For local similarity, supervised model pairs Sup. sup) are more consistent than the mmage-text model pairs (migg-Ta, Immg-Txx) which are better at representing global structure. In addition, models trained on IN21k (fINZIK; PIN2Ix) are more consistent than models trained on IN1k
22

---

Objective drives the consistency of representational similarity across datasets
(PINIK: PINIK). Their ordering flipped compared Fig. 6, where (PINIK: PINIK) model pairs more consistent in their global structure. IN21 contains substantially more classes (21.8 21.8 and higher percentage of the classes leaf nodes in the WordNet (Princeton University, 2010) tree (76.71% for IN21K vs. 65% for INIK), representing more fine-grained entities. The representation must contain more fine-grained details to distinguish these classes, dominating local similarities.
Other global similarity metrics In Fig. 17, the correlation between the two global measures (CKA linear and RSA Spearman) is moderate. Even with this reduced correlation strength, similarity consistency measured using RSA Spearman exhibits the same patterns as CKA linear, particularly regarding the model category we found to be most influential the training objective (see Fig. 17).
The analysis presented in this section suggests that observations on the influence of the training objective similarity consistency hold for multiple metrics.
Training objective
Architecture
Correlation coeff.
0.5
0.0
SSL
SSL sup
Sup Sup
Img-Txt Img-Txt
Img-Txt SSL
Img-Txt Sup
TX
CNN CNN
CNN
5SL
TX
TX
Training
1.0
Correlation coeff.
0.5
0.0
-0.5
IN21K
IN1K IN1K
Large XLarge
INIK IN2.k
Large Large
XLarge XLarge
IN21k Large
INIK Large
IN1k XLarge
IN21K XLarge
Model size
1.0
Correlation coeff.
0.5
-0.5
xlarge
large
small
medium
medium
large
large
large
medium
small
xlarge
large
xlarge
medium
small
xlarge
medium
small
xlarge
Figure 16: Distribution of similarity consistencies for each model set pair based on CKA RBF (a = 0.2) representational similarities. The boxes are sorted in decreasing median correlation. The dotted line indicates the overall median, while the gray spans the 25th 75th percentiles of correlations across all model pairs.
G. Validating training objective effects through controlled anchor model analysis
To validate our findings regarding the influence of different model categories ($4.5), we conducted controlled analysis using six carefully selected anchor models: OpenCLIP RN50, OpenCI.IP ViT-L,, SimCLR RN50, DINOv2 ViT-L,, ResNet-50, and ViT-L. These models were chosen to create systematic variations across training objectives (image-text, self-supervised, and supervised learning), while controlling for architecture (convolutional ResNet-50 vs transformer ViT-L) and model size. This controlled setup helps verify whether the patterns observed in our main analysis persist when comparing individual representative models to broader model sets. each anchor model, we define a single-element model set (e.g..
23

---

Objective drives the consistency of representational similarity across datasets
Correlation coefficient
0.8
0.6
0.4
0.2
0.0
SSL SSL
SSL
Img-Txt Img-Txt
Sup Sup
Img-Txt
Sup
Img-Txt
SSL
Sup
Figure 17: Distribution of similarity consistency values for each model set pair within the training objective category, based on RSA Spearman representational similarities
e OpenCLIP RN50) and compute its similarity consistency with different model groups (e.g., R(P OpennCLIP RN50 Pss
Fig. 18 shows the correlation coefficient distributions across dataset pairs, focusing on training objectives as they emerged as the most influential factor for similarity consistency. The results support our main findings while providing additional insights. Self-supervised models (SimCLR RN50, DINOy2 ViT-L) consistently show the highest median correlations and lowest variations when compared with any anchor model, regardless of the anchor's training objective. Surprisingly, for global similarity, image-text (OpenCLIP RN50, OpenCLIP ViT-L) and supervised (ResNet-50, VIT-L) anchor models correlate more strongly with Pss than with their own categories (fimg-Taand Supp respectively). However, the image-text anchors show high variation of similarity consistency for local similarity measures and all training objectives. We also observe a notable pattern weak correlations between image-text anchors and supervised models (USp and vice versa. These patterns remain remarkably consistent across different architectures and model sizes, suggesting that training objectives, rather than architectural choices or model sizes, primarily drive similarity relationships across datasets.
OpenCLIP RN50 (Img-Txt)
SimCLR RN50 (SSL)
ResNet-50 (Sup)
Correlation coefficient
1.0
0.5
0.0
CKA RBF 0.2
CKA linear
CKA RBF 0.2
CKA linear
CKA RBF 0.2
CKA linear
OpenCLIP ViT-L (Img-Txt)
DINOv2 VIT-L (SSL)
ViT-L (Sup)
Correlation coefficient
1.0
0.5
0.0
pco0
-0.5
CKA RBF 0.2
CKA linear
CKA RBF 0.2
CKA linear
CKA RBF 0.2
CKA linear
Img-Txt
SSL
Sup
Figure 18: Distribution of Pearson correlation coefficients between anchor models (OpenCLIP RN50, OpenCLIP ViT- I., SimCLR RN50, DINOv2 VIT-L,, ResNet-50, ViT-L.) and models trained with different objectives. Each box shows correlations across all dataset pairs between one anchor model and models from specific sets (e.g., Ø Dup)). Results for CKA RBF (a = 0.2) and CKA linear are separated by dotted lines. Subplot titles are colored according to the anchor model's 's subcategory.
24

---

Objective drives the consistency of representational similarity across datasets
H. Effect of evaluation data on similarity consistency
In the main analysis, we examined similarity consistency across model categories for all evaluation datasets ($4.5) and across individual datasets for all models ($4.6). Here, we investigate whether model category effects differ between natural and non-natural (specialized and structured) datasets, as non-natural datasets represent out-of-distribttion data for many models (Fig. 19).
The primary finding from $4.5 holds: training objective remains the dominant factor influencing similarity consisteney regardless of dataset type. However, similarity consistency between image-text and supervised models (Pimg-Txt+ Ssp) shows lower medians and higher variance on natural datasets compared to non-natural ones. For natural datasets, supervised models might encode features tightly coupled to class-label supervision, which diverge from the semantically richer, language-aligned representations of image-text models. Conversely, on non-natural datasets where both model types operate further from their training distributions, this representational gap narrows as both might rely on more generalized features. Complementary, the analysis examining how training dataset composition affects these patterns is presented in the next section.
Training objective
Architecture
Training objective
Arch tecturr
0.5
0.5
0.0
0.0
534
Impilit SSL
SSL Sup
Sup Sup
Imgeeet Img-Ixt
Impilit Sup
14
CNN CNN
CNN
554
Imgelst Imgelxt
SSL Sup
Supp Sup
Img-lixt SSL.
Img sup
CNN CNN
TX
CMN
5SL
59L
TX
Trining data
Training data
1.0
1.5
0.5
0.5
-0.5
-0.5
IN1 I4k
Large Xlange
IN21K I823k
Large Large
Xlarge Xtarge
Bdk X0 Xuarge
NIk Large
IN2TH Xlarge
INO3
N1k
IN21k IN21
Large Large
INC IN21
Large XLarge
IKtK Large
IN1x XLarge
INT1 Large
Xlarge Xlarge
Lange
N1k
30 Maarge
Model size
1.0
1.2
0.5
0.5
0.0-
0,5
0.5
0.5
xarge
brge
Iarge
small
mediu
large
large small
modium
medium small
xarge
small
medium
modum modium
large
large xlange
large medium
large small
small starpe
xarge
brge
xarge
small
Xiarge
medium
medium
small
xarge
xarge
small
xarge
small
medium
large
(a) Natural Images
(b) Non-Natural Images
Figure 19: Distribution of similarity consistencies for natural (single- and multi-domain) and non-natural (specialized and structured) datasets. Representational similarities for each model set pair are computed with CKA linear.
I. Effect of training data domain on similarity consistency
In our main analysis, we restricted our focus to general- sanrroe-purpose vision models trained on large-scale datasets with diverse semantic classes (& 1). This focus was motivated by the Platonic representation hypothesis, which suggests that as the semantic diversity of training data approaches the complexity of reality itself, representations should increasingly converge (Huh al., 2024). However, Conwell et al. (2024) demonstrated that training data is key driver of representational similarity between models, potentially more significant than architecture, training objective, or choice of similarity metric. In this section, we examine how this observation manifests in terms of representational consistency.
To investigate the role of training data domain, we selected four architectures (AlexNet (Krizhevsky et al., 2012a), DenseNet161 (Huang et al., 2017), ResNet18 (He et al., 2016), and ResNet50 (He et al., 2016)) with available pre-trained weights for both the general- purppsse ImageNet- 1 and the scene-specific Places365 (Zhou al., 2018) datasets. Models
25

---

Objective drives the consistency of representational similarity across datasets
sourced from torchvision', using default ImageNet weights and Places365 weights from the official GitHub repository". This selection of models allowed us to isolate the training data effect similarity consisteney while fixing the other characteristics: training objective (Supervised), architecture type (CNN), and model size (small).
We evaluated similarity consistency across the 23 downstream datasets for all model set pairs: (PINIKs PINIK), (PINIK; (Piaces6ss,, (Places365, Pllaces6s) and (Pall Pan)), where aa contains all models.
Our analysis reveals two main observations. First, as shown Fig. 20, Places365-trained model pairs demonstrate more consistent representational similarities compared to ImageNet- Ik-trinned pairs We hypothesize that domain-specing specific training constrains the solution space more tightly, whereas the greater diversity of semantic classes in ImageNet-I4 allows for more varied learned representations across models. This distinction is particularly interesting in light of the platonic representation hypothesis. While domain-speciin models show higher consistency, this may reflect convergence specialized solutions rather than to general representations of reality. In contrast, the lower consistency among ImageNet- I---rrnndd models likely reflects the challenge of learning representations that capture broad semantic diversity. This consistency might be even lower for models trained with supervised objectives since they can exploit different confounders and shortcuts when learning to classify the diverse set of classes.
Second, the distributions R( RININI PINIK), R(PINIK: Plncessss) ). and R Pall, Pan) exhibit similar distribution spreads. Addi- tionally, R( (a Paul shows distribution similar to R( R(PINIK; PINIK) in Fig. 6. This suggests that including domain-specing models alongside general-purpose purpose ones might not fundamentally alter the representational similarity consistency patterns observed in our analysis.
Our analysis shows that the training data domain significantly influences representational similarity consistency, though mixing -emaai--ppecffce and general-ye purposs models preserves the broad distribution patterns of correlation coefficients observed in ImageNet-Ik l models alone.
CKA linear
CKA RBF 0.4
RSA spearman
1.0
140
0.8
0.8
0.6-
0.5
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.0
-02
181c
IN1K
Mocos365
All combs
INLK
INLK
Paces365
All comes
IN1K
Placee3t5 combo
Paces365
Moces365
Paces365
Races365
Placo365 Places365
(a) CKA linear
(b) CKA RBF (a = 0.4)
(c) RSA Spearman
Figure 20: Distribution of Pearson correlation coefficients for model similarities computed with different similarity metrics across dataset pairs, i.e., R(PINIK PINK). R R(PINI, Pplaces66),, R(Praees665 Pllnces66s,, and R( Pan), with INIk containing the four IN1k and Ppiaces365 containing the four Places365 pre-trained models. han contains all eight models.
J. Distribution correlation coefficients downstream task performance vs. model similarity
In $4.7, we examined the correlations between representational similarity and classification performance gaps across three datasets from each of the four dataset categories: Natural (multi-domain), Natural (single-domain), Specialized, and Structured. We observed stronger negative correlations for the Natural (single-domain) datasets. Fig. 22 extends this analysis by illustrating the correlation coefficients for all evaluated datasets. We can confirm that natural -ooglee-doman) datasets exhibit the strongest negative correlations overall on this larger set of datasets, with the SVHN dataset being the sole outlier.
Interestingly, ImageNet- exhibits the weakest negative correlation between performance gap and representational similarity.
htm1
26

---

Objective drives the consistency of representational similarity across datasets
We hypothesize that this is due in part to the high diversity of classes in ImageNet-Ik, along with its rich set of confounders and contextual cues, which enable networks to approach the classification task in multiple ways. This may be exacerbated by the fact that some of the evaluated networks were pretrained ImageNet-Ik or ImageNet-2IK, which could have encouraged the learning of -ptaset-specific features that perform well on ImageNet but fail to generalize to other datasets.
While our categorization of natural datasets into single-domain and multi-domain revealed an interesting difference in the relationship between downstream task performance and model memilaritt-especiall between the ImageNet- Ik and single-domain datasets like Flowers or Pets-the correlation coefficients for these categories still overlap. We hypothesize that our binary sub-categorization of Natural datasets is insufficient for capturing clear distinction for two reasons: (I) Datasets lie spectrum between the two (hypothetical) extremes of encompassing all possible domains and encompassing exactly one. For instance, ImageNet- Ik has a relatively rich set of domains, containing 1 1,000 diverse classes, whereas CIFAR-10 only contains 10 classes of vehicles and animals. (2) Datasets differ in the amount of contextual information (e.g., the object's ' background) they offer. We assume that contextual information plays an important role in the number of viable features a dataset classification can be solved with. A large number of viable features means that multiple perl-peeg but dissimilar strategies may exist. Consequently, multi-domain datasets with limited contextual information may behave more like single-domain datasets. For example, the PASCAL VOC 2007 dataset, when used for single-label classification, makes of tight bounding- -bx crops around the object be classified, thereby reducing the available context. Similarly, in low-resolution datasets such as CIFAR-10 10 and CIFAR- 100, contextual information may be reduced due to the loss of background information. While we have not quantified it, we found the Caltech-101 101 dataset to contain much cleaner object backgrounds than ImageNet-Ik k. also suggesting reduced contextual information.
Naturallmutti - domain) ImageNet-1k
Naturaitsingle - domain) Flowers
Specialized Diabetic Retinopathy
Struclured DTD coaff::
Performance Goip
coeft.
0.9
-0.6
0.04
coeft: -0.40
0.E
-0.69
0.8
0.4
0,7
0.03
0.77
D4
0.7
0.76
0.2
0.52
0.2
0.5
0.8
0.0
0,0
0.00
0.74
0.5
FIFR--100
Pels
EuroSAT
Dmilab
Performncc 8
0.8
cooeft: 0.68
0.9
0.8
coeft: 0.82
0.15
coeff: 00.5
0.96 0.2
cooef.f
0.60
0.6 0.10
0.94
0.2
0.2
0.4
0.4 9.05
0.1
0.0
0.0
9.00
0.58 0.0
Entity- 30
Stanford Cars
PCAM
FER2013
Performance Gap
0.3
coeft: 0.35
coeft: 0.63
0.08
r.ceftt: -0.13
coefff:
0.8
0.96
0.86
0.2
0.6
0.5
0.6
0.04
0.84
0.2
0.5
0.1
0.5
0.2
0.02
0.82
0.0
0.00
1.0
0.5
0.5
0.S Similrity value
1.0
Similarity value
Similarity Vallue
Similarity vallue
Figure 21: Model similarity (CKA linear) vs. absolute difference in downstream task performance (top-1 accuracy) for model pairs for three datasets per dataset category (natural multi-domain, natural single-domain, specialized, and structured). The color of each point indicates the downstream task accuracy of the better-performing model. Pearson correlation coefficients are shown in each subplot.
27

---

Objective drives the consistency of representational similarity across datasets
-0.1
Natural (multi-domain)
-0.2
Natural (single-domain)
Correlation Coefficient
Specialized
-0.3
Structured
-0.4
-0.5
-0.6
-0.7
-0.8
PASCAL VOC 2007
CIFAR-10
Caltech-101
2007
CIFAR 100
STL-10
Living-17
Nonivingg
211
Entity-3 330
ImageN Nk
Flowers
Stanford Cars
FGVC Aircraft
Pets
GTSRB
Diabetic Retinopathy
SVHN
RESISC45
EuroSAT
PCAM
DTD
2ER2013
Dmlab
Figure 22: Pearson correlation coefficients between downstream task performance gap and model representation similarity for all datasets grouped by dataset category.
28

---

Objective drives the consistency of representational similarity across datasets
Table 2: Pretrained neural networks that we considered in our analyses.
Model name
Source
Rep. Layer
Training objective
Training data
Training dara class
Archilecture
Architecture class
Model sipe
size class
Model
KakaobrannAAlgg
KakaoBrain (2023)
pooler
Img--Txi
COYO-700M
Large
EllicienNNet
CNN
62.1M
small
1-pp----------------- mergedZb si lb b114k
Sun et (2023)
visual
Img-Txx
Merged2B
XLarge
VII
TX
1.4B
xlargs
OpenCLIP Be--CLL--EVA------- 14-1aioo4400imm1 1h-b41k
Sun et (2023)
visual
Img-TTx:
LAI0N40IM
Large
VIT
TX
1.18
xlarge
(2023)
Img-Txi
MergalZB
XLarge
VIT
TX
149.7M
medium
bllk
Sun et (2023)
Visual
Img-Ter
Merged288
Xlarge
VIT
TX
427.8M
xlarge
DeasCLIP-R55--ppee
Radiord et al. (2021)
visual
Iog-Tsi
WTT-100M
Large
ResNel
CNN
102.0M
medium
OpeaCLIP- NIT-B- SigLIP-wehli
Zhaä et (2023)
Img-Tx:
XLarge
VII
TX
203.2M
mediam
OpeaCLIPPVVI NIT------------------ 1111---6-lniinnb----
Radfond et al. (2021)
Visual
Img-Txx
LAION2B
XLarge
VIT
TX
149.6M
medium
OpeaCLIP- ViT-B- (-T---16--B
16-laion4400m
Radford el. (2021)
visual
Img Txt
LAI0N400M
Large
VIT
TX
149.6M
medium
Radford (2021)
visual
Ing-Ts:
WTT-40IM
Large
VIT
TX
149.6
mediam
OpeaCLIP:VE DpeaCCLIPVViT-L-114 ----VV----1--innnbbbe
Rudionn et (2021)
visual
Ing-Tai
LAION2B
XLarge
VIT
TX
427 AM
shrrg
OpeaCLIP-VIE NiT--L-16-laoono00
Radford ct. (2021)
visual
Img
LAION40M
Large
VIT
TX
427.6M
slargs
OpeaCL.I-VTTLL---ggeme
Radiond et (2021)
visual
Ing-Txt
WIT-100M
Large
VIT
TX
427.6M
slarge
vit-uuge patchl4ccip-22..mmm Ibi
Radford al. (2021)
norm
Img-Tx: -x: LAION2B
XLarge
VIT
TX
632.1M
slarge
haflowrinns-m50
Zhontar (20221)
aygpool
SSL.
INIK
INIK
ResNet
CNN
23.5M
small
dine-m50
Caron et (2021)
aygool
SSL
INIK
INIK
ResNel
CNN
23.5M
small
dino-vit-base-p16
Caron et al. (2021)
norm
SSL.
INIK
INIK
VIT
TX
85.8M
small
dim--vi--mull-p16
Caron (2021)
VIT
TX
21.7M small
dino-xcit meei-meeiium-222 24
Caron al. (2021)
norm
SSL
INIK
INIK
VII
TX
83.9M
small
dino-xcit- small- 12.p16
Caron et al. (2021)
norm
SSL
INIK
INIK
VIT
TX
25. 25.9M
small
dinov S---ii--bace p14
Oquab al. (2024)
norm
Large
VII
TX
$6A6
small
dinow2-vir-gee aaoo---i--laant-ph plant-gaa-pp4
Oquab et al. (20124)
norm
SSI.
IND-142M
Large
VIT
TX
1.18
xlarge
dinos 2-vil-large--p
Oquab et al. (202-)
SSL
LVD-142M
Largo
VIT
TX
304
lorge
Oquab (2024)
norm
LND-142M
Large
VII
TX
22.1M
small
ligaaw-m50
Narooai & Favaro (2016)
aegpool
SSL.
INIK
INIK
ResNet
CNN
25.5M
small
vis base
He al. (2022)
SSL
INIK
INIk
VIT
TX
86.4M
small
mae-vit-hugee hit--ii--uge-114
al. (2022)
norm
SSL.
INIR
VIT
TX
6508M
xlarge
mac-vit- mae-vit-krge--1o
He ei al. (2022)
normn
SSL
INIK
INIK
VIT
TX
313.3M
large
mocov2 m50
Chen al. (2020c)
aygpool
SSL
INIK
INIK
ResNet
CNN
23.5M
small
pidl-mS0
Misra & van der Musten (2020)
aegpool
SSL
INIK
INIK
ResNet
CNN
23.5M
small
rotner m50
Gidaris al. (2018)
aygpool
SSL
INIK
INIK
ResNet
CNN
23.5M
small
simeli-mm50
Chen (2120a)
argpool
SSL.
INIK
ResNet
CNN
23.5M
small
swa-mrSU
Caron (2020)
aggpool
SSL
INIK
INIK
Re
CNN
23.5M small
vicreg-m500
Randes et al. (2022)
avgrool
SSL
INTK
INIK
ResNet
CNN
23.5M small
beil-baw--aacch6--22
Bau (2022)
Sup
INZIK+INIL
IN2IK
VIT
TX
small
beil beil-base-pacchi6
222.222.2222222 224.in22k-f-in222 in2k
Ban et al. (2002)
norm
Sup
IN2IK
IN2IK
VII
TX
102.5M medium
helt-lange-uuchh16--22
Bao al. (2022)
norm
Sup
IN2IK INIK
IN2IK
VIT
TX
304 AM
large
sei-llrree-atcchee.
Ban (2022)
norm
Sup
IN2IK
IN21K
VII
TX
large
comvieetthsse
Lin al. (2022)
head. flamonn
Sup
INIK
INIK
NeXi
CNN
88.6M small
comvncsl-large
Liu et al. (2022)
head. Batien
INIK
INIK
CoxnNeXi
CNN
197.8M KM
medium
deit3-
saii--baee-aachhl..22
Touvron
norm
Sup
INIK
VII
TX
86.6 small
deit3-base-ppee
Tourron et al. (2021)
norm
Sup
IN2IK INIK
IN2IK
VIT
TX
86.6
small
deir3-large deitt-lange-gucchl6- gatch16-c22
Tourron (2021)
INIK
INIk
VIT
TX
304.4M
large
deltt-lange-uutch1--222
Touvron al. (2021)
norm
Sup
IN2IK INIK
IN2IK
VIT
TX
large
clficientnet-h3
Tan & Le (2019)
aygooul
Sup
INIK
INIK
BlicieniNa
CNN
12.2M
small
afficientnet-h4
(2019)
aygroo
INIK
CNN
ollicientnet-b55
Tan Le (2019)
aygooo
Sup
INIK
INIK
Elicien/Net
CNN
30.1M
small
effciiettet-b6i b6
Tan & Le (2019)
aygoool
Sup
INIK
INIK
EffeisntNet
CNN
43.0M
small
afficientnet-h?
(2019)
aygoool
INIK
EMeeeenNer
CNN
66.3M
small
resmet|52
He et al. (2016)
argpool
Sup
INIK
INIk
ResNet
CNN
60.2M
small
resiet50
He et al. (2016)
aygroal
Sup
INTK
INIK
ResNet
CNN
25.6M
small
roNmox1503322 50
(2016)
global_pool
Sup
ResNeXL
small
seresnet50
Hu et al. (2018)
global poel
Sup
INIK
INIE
SE. ResNet
CNN
25.1M small
swin base---secceeee palch/-winkkw7- 2-22
Liu et al. (2021)
global.pool
INIK
INIK
Swin- Transfomer
TX
87.8M
small
swin hase- patch4- window? ww2.4..m-mmm2222...
(2021)
plobal.pool
Sup
IN2IK
IN21k
Swin-Transformrr
TX
mediam
swin-turge-pucch--wiie wwwp:e-wwwwhhwwwwdww-----.. 7-22
Lin al. (2021)
plobal_pool Sup
INIK
INIK
Swin-Tunsffomer
TX
196.5M
medium
swin-large- sunge--pucch-wwinddww 7-22 mi2k
Liu al. (2021)
pool
INZIK
IN2IK
Swin-Transde Transfomer
TX
228.6M
medium
vgg16
Simonyan Zisserman (201 5)
3lassifier.3
Sup
INTK
INIK
VGG
CNN
138.4M
medium
ygg19
Simonyan & Zisserman (2015)
3
Sup
INIK
INIK
VOG
CNN
143.7M
medium
Miebbse--atte patchahhch222
Dosovisáay (2021)
INIK
INIK
VIT
TX
86.6
small
vi--base-puthh16-222 a6gg22in
Dasavibári al. (2021)
norm
Sup
IN2IK
INZIK
VIT
TX
102.6M
medium
vit- huge patch (4. 244224,arigg-nn ik2
Dasevitkky et al. (2021)
norm
Sup
IN2IK
IN2IK
VIT
TX
630.8M
slarge
vi-laarg--aacch 16-224
Dosovissiiy
VIT
TX
304.3M
large
vie- Marge-patchh patch 2aa.lll62222-1---iiiaa in21
Dosovibkiy et al. (2021)
norm
Sup
IN2IK
IN2IK
VIT
TX
325.7M
large
29


Reviews:
[{"review_id": "g4ZSzgVuOB", "reviewer": "Reviewer_f2Fe", "summary": "The paper sets out to challenge the Platonic representation hypothesis by reexamining similarities between representation of models using multiple datasets. Their key finding is that training objective is a dominant factor driving representations, as opposed to model architecture and model size.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their helpful suggestions and for acknowledging the _relevance of our work_ and the _soundness of our experiments_. We believe that following the reviewer’s suggestions allowed us to notably improve our analyses. Two points stood out in particular, which we address in detail below. We added all additional figures (R1F1-F5) to an anonymized [repository](https://anonymous.4open.science/r/rebuttal_similarity_consistency/README.md)  and refer to the specific Figures detailed in the README.md.\n\nFirst, we would like to clarify the range of the covered training objectives. In representation learning for computer vision, self-supervision (SSL) and image-text alignment are currently the state-of-the-art approaches. Therefore, we selected these two categories alongside supervised learning. The SSL group contains a diverse set of objectives, including self-supervised contrastive losses (such as SimCLR), the mentioned _masked image modeling_ (MAE), self-distillation (DINO), extended self-distillation (DINOv2), but also pretext-task-based (Jigsaw, RotNet), redundancy-reduction (BarlowTwins) and clustering-based (SwAV) losses*. The total set of 64 models contains (most) SOTA models for representation learning in vision. For this diverse set, we identified the training objective as a crucial factor for the consistency of representational similarity, while model architecture and size appear less relevant. \n\nAs we identified the training objective as a main driving factor for similarity consistency, we are convinced that due to its flexibility, our framework can, in future work, easily be applied to analyze other training objectives, e.g., as recommended, testing the effect of robustness losses on the similarity structure of out-of-distribution data. \n\nSecond, we would like to address the effect of taking the _final layer_ of the model by evaluating the representational similarity consistency on intermediate layers. In our work, we followed the standard procedure of extracting features of the _final layer_ (or _penultimate layer_, for classification models), which is commonly referred to as the _representation_ [1]. We remark that these representations are of special interest because they are the ones used for downstream tasks. However, we agree with the reviewer on the potential role of layer choice for our findings, as mentioned in our discussion section.\nOur proposed analysis framework does _not depend on specific layers_; it can also be applied to intermediate layers. Therefore, we followed the reviewer's suggestion and repeated the consistency analysis for the middle layers of _a large subset of our transformer models_. We omitted CNN-based models, as middle-layer extraction is less clear when representations depend on spatial location. We remain consistent in our extraction method across layers for the transformer models: If the model's original representation was derived from the classifier token, we also extracted the classifier token from intermediate layers. Otherwise, we applied avg. pooling. Fig. R1F1 and R1F2 show similarity matrices, analogous to Fig. 2. Here, model representations tend to be more similar across models of the same type. W.r.t. the consistency of similarities, we observe a lower median and larger standard deviation of consistency of representational similarities across all model pairs (grey bar) in Fig. R1F5 compared to Fig. 5. While the variances are relatively large for the training data and model size categories, we see above-median consistency for within-objective model pairs. \nWe speculate that higher consistencies within training objectives in intermediate layers indicate that the training objective has a stronger influence on representational structures already early in the network. For example, supervised models may form structurally similar lower-level representations but are less constrained in the organization of their representations close to the classification output.\n\nIn conclusion, our analyses of intermediate layers support the finding that the training objective is a main influence on representational similarity consistency, though more experiments would be needed to fully characterize layer-specific effects. We thank the reviewer for their insightful comments that have strengthened our analysis.\n\n\\* References to the models can be found in the main paper, in Tab. 2.\n\n[1] Kornblith, Simon, et al. \"Do Better ImageNet Models Transfer Better?\" CVPR, 2019.", "claims_and_evidence": "The claim on objective function needs more support, as currently only SSL versus supervised objectives were examined; there are many other important objective functions—such as robustness to noise or corruptions objectives, generative modeling objectives (e.g., diffusion models), masked image modeling, and reinforcement learning-based objectives—that were not included in the analysis. Without evaluating models trained on these diverse objectives, it is difficult to generalize the conclusion that the objective function is the primary driver of representational similarity consistency.", "methods_and_evaluation_criteria": "A key methodological limitation is that the study focuses exclusively on representations extracted from the final layers of the models (ref table in supplementary: the penultimate layer for supervised models, the average pooling layer for SSL models, and the image encoder output for image-text models). While this is a common practice, it risks biasing the analysis toward the influence of the objective function, since final-layer representations are often more task-specific and reflect the model’s training objective.", "theoretical_claims": "No theoretical claims", "experimental_designs_or_analyses": "Experimental design is sound and standard for measuring similarity of representations.", "relation_to_broader_scientific_literature": "The question of what drives similarity (or dissimilarity) of representations is important for many fields and it directly engages with many recent works on the topic.", "essential_references_not_discussed": "None spotted", "other_strengths_and_weaknesses": "The question and the approach are not novel so the work can really benefit from broadening the evaluations to more models to make a conclusive claim, or refine the claims to the actual results.", "other_comments_or_suggestions": "None", "questions_for_authors": "If possible, could you also report the analysis results on a few internal layers, close to the middle of processing in each model? Do you expect to see more or less the same results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LfDxMruNAJ", "reviewer": "Reviewer_7cXA", "summary": "This paper is more of an analytical paper that analyze the cross-domain representation similarity among models trained with different objectives. The analytical framework is pretty simple as it is a combination of kernalized CKA and a spearman correlation measure. Methodology description is concise. Experiments show the results of analysis, most of the observations are inconclusive due to the confounded factors, but the some insights are interesting. E.g. SSL's representation similarity consistency is higher than that of supervised learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the valuable feedback and appreciate the assessment of our manuscript as being _well-written_ and _interesting_ work. \n\nFirst, we agree that the individual components of our analysis framework are well-established rather than novel by themselves. We see this as a strength rather than a weakness because it allowed us to build a novel meta-framework relying on the methodological soundness of the individual components without introducing subcomponents that still need to be validated by the research community. To the best of our knowledge, our work is the first to propose a structured way of using these components to characterize how similarity relationships vary across datasets. Our main focus lies on introducing and formalizing _similarity consistency_ and conducting a large-scale analysis of this measure across 64 different vision models and 23 datasets.\nWe identify that the learning objective is a crucial driver of similarity consistency, and architectural specifications before training (architecture type and size) are less relevant.\n\nSecond, while confounding factors cannot be avoided when evaluating the limited set of large, well-trained general vision models, we tried to unravel some confounders in Appx. G and H. \n- In Appx. G, we selected two anchor architectures (ResNet-50 and ViT-L) and systematically varied the training objective (Sup., SSL, Img-Txt) while keeping the architecture type and model size constant, resulting in six models. This allowed us to isolate the effect of the objective in a more limited but controlled setting. We observed that even here, the training objective appears to be driving similarity consistency. \n- In Appx. H, we investigate the role of training data on similarity consistency. To that end, we fixed the objective (supervised) and network architectures (AlexNet, DenseNet161, ResNet18, and ResNet50) and varied the training dataset (general-purpose vs. domain-specific). Here, we identified higher consistency in models trained on a domain-specific dataset (Places365), most likely due to a more constrained solution space in comparison to general-purpose models trained on large-scale datasets. These findings complement our main analysis, which deliberately focused on general-purpose vision models trained on datasets with large semantic diversity.\t\n\nLast, we fixed the Fig. reference in Section 4.3. We intended to refer to Fig. 11 in the Appx. D. We will correct this in the camera-ready version by using the extra page to move this figure back to the main text.", "claims_and_evidence": "There is no clear claim of this paper given it is more about analysis. While the paper claimed the \"framework\" of analysis as part of its contribution. It is a combination of pretty standard and well known analytical tools. If that is the key claim of this paper, then the paper's novelty is very low. \n\nI think the biggest contribution of this paper is about the insights it provided on understanding the representation similarity among models in the context of domain-transfer. However, the analysis seems lack depth given multiple observations are less conclusive ( due to the confounded factors). Probably the experimental settings could be adjust to minimize the effect of confounded factors. I think this paper is more suitable for being a position paper rather than a standard ICML submission given its claim is about advocating a particular research direction rather than proposing a new algorithm.", "methods_and_evaluation_criteria": "Method is clearly written and easy to follow. It does make sense to leverage CKI and spearman correlation to quantify the representation similarities. \nThere is no meta-evaluation criteria to quantify the method proposed (given the method itself is a set of measure). However, the experimental setup can be further tuned to reduce confounded observations.", "theoretical_claims": "N/A, there is no theoretical claims I can see unless \"SSL's representation similarity consistency is higher than that of supervised learning\" is the claim, which is empirical not theoretical.", "experimental_designs_or_analyses": "Analysis looks well done. I have learned something from this work. The only complain is about inconclusive observations here and there.", "relation_to_broader_scientific_literature": "This research is one direct extension of representation similarity measure among models (or could be even model vs neuron recordings).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n1. Well written article. Descriptions are concise and clean.\n2. Extending representation similarity research to cross-dataset setting, which looks interesting.\n3. Some observations from the analysis is interesting in terms of justifying the need of SSL in many applications.\n\nWeakness:\n1. The depth of analysis is still shallow. It could be more interesting if the authors can design the analysis with a more precise control on factors.", "other_comments_or_suggestions": "I think the figure reference in section 4.3 is wrong.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FluxF5nfEp", "reviewer": "Reviewer_ZtBX", "summary": "The paper proposes a way of measuring the consistency of pairwise similarities across datasets and transferability of similarities between them. The authors provide many observations regarding these aspects.\n\n## update after rebuttal\n\nThe authors provided some additional discussions and results, which further strengthened my belief that they deserve the high score I initially gave (accept). I hope that as promised the authors will incorporate the main points of their answers into their camera-ready version.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we thank the reviewer for their overall positive feedback and for pointing us to two papers that helped strengthen the integration of our work into the existing literature. We agree with their relevance to our work due to using Representational Similarity Analysis [RSA; 2] or other similarity measures [1] to (pre-)select (downstream) task-specific models. Therefore, we will cite them in the related work section of our manuscript’s camera-ready version: “[...] However, recent work demonstrated that representational similarities (e.g., measured via RSA) can be used to effectively select (downstream) task-specific models [1,2].”\n\n\nSecond, we agree on the lack of clarity of some of the figure captions and will use the extra page in the camera-ready version to expand these captions to improve clarity, i.e., for Fig. 5.\n\n\nIn addition, we thank the reviewer for posing three interesting questions, which we will elaborate on, providing additional results (R3F1-F5) in an anonymized [repository](https://anonymous.4open.science/r/rebuttal_similarity_consistency/README.md).\n\n**Q1**: _Which SSL models are close to Img-Text in Fig. 4, and why?_\n\n\nFig. R3F1 zooms into the area of Img-Txt models for the three dataset combinations of Fig. 4 and labels individual model pairs. We observe that many SSL and image-text model pairs show high CKA values across the datasets, i.e., are located in the upper right corner and therefore in proximity. These models are similar within each objective. The SSL model pairs show high similarities as they are trained with similar datasets,  augmentations, and losses (e.g., BarlowTwins/VicReg and MoCov2/SimCLR). Some image-text models are quite similar as well. However, the proximity of these two model pair sets does not allow us to infer any direct relationship between them.  For this, we must consider the similarities of model pairs containing both model types, as shown in Fig. R3F2. The points, representing cross-type similarities between SSL and Img-Txt models (pink), have lower correlations than within-type pairs. This indicates that despite some individual SSL models appearing close to Img-Txt models, the overall relationship between these two categories of models is less strong and more variable.\n\n**Q2**: _Could Fig. 2 and 4 be provided for more specialized datasets?_\n\n\n\nWe thank the reviewer for their suggestion and have decided to include additional figures extending Fig.2 with datasets of each category (Fig. R3F4) and Fig.4 containing the EuroSAT and DTD (Fig. R3F5). \n\n**Q3.1**: _Do you attribute the inconsistency patterns in CIFAR-10 and CIFAR-100 primarily to their significantly lower image resolution compared to IN-1k?_\n\n\nYes, we indeed think that CIFAR-10/100’s small image size (32×32 pixels) restricts the representation space by severely limiting fine-grained details and surrounding contextual information. Appx. I shows clear differences in correlation coefficients between performance gaps (def. Sec. 4.7) and CKA values: low absolute correlation for IN-1k versus high for CIFAR. This suggests IN-1k’s high-resolution images (224×224 pixels) provide rich contextual cues that support diverse, well-performing representations, whereas CIFAR’s constrained resolution and minimal background details restrict the range of viable representations.\n\n**Q3.2**: _Can you elaborate on the underlying factors causing the visible clustering patterns observed for certain datasets in Fig. 6?_\n\n\nLooking more closely at the clustering patterns of Fig. 6, we observe: \n- CIFAR-10 and CIFAR-100 form a strong cluster, potentially due to their low-resolution format and similar categorical structures, showing also high consistency in the (Img-Txt, Sup).\n- The Breeds datasets, Caltech-101, Country-211, STL-10, and Pets cluster together based on their similar domain properties, resolution profiles, and centered object compositions, showing also high consistency in the (Img-Txt, Sup).\n- Medical imaging datasets do not cluster together, potentially due to their fundamentally different visual patterns and domain-specific features (eye vs. tissue scans).\n- RESISC45 shows stronger correlations with structured datasets than with other specialized datasets (i.e., (Img-Txt, Sup)). This cross-category relationship might stem from satellite imagery's inherent structural properties—regular grid layouts, transportation networks, and geometric patterns—which create feature distributions resembling those in structured datasets. However, it differs from EuroSAT potentially due to RESISC45's higher resolution imagery, greater geographic diversity, and more diverse class set.\n\nA further analysis that isolates natural images from structured and specialized datasets can be found in the first part of Reviewer 9H2r's answer.\n\nWe will incorporate the main points of these answers into our camera-ready version.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods are adequate for the problem (RSA, CKA). I like the simplicity and clear formulation of the framework. Also the datasets and models are adequate.", "theoretical_claims": "N/A - the nature of the paper is rather empirical.", "experimental_designs_or_analyses": "- The authors evaluated numerous models on many datasets. They also analyzed whether the results are not due to the similarity measure of choice. In my opinion, the experiments are convincing. (+)\n- In Section 4.3 and Fig. 4, it is interesting that some SSL models lie very close to the text-image models. Could the authors check what SSL models cluster with text-image models and try to explain why? (question Q1)\n- As the authors use models trained on general purpose datasets (such as ImageNet), it would be nice to present some more qualitative results for a dataset such as EuroSAT due to the fact that this dataset also presents a domain with a large domain gap to the general-purpose datasets (e.g. Fig. 2, 4). Similarly, it would be nice to include some examples for the DTD dataset, as it focuses on textures. (Q2)\n- The authors should better discuss the inconsistencies between relative similarities in Section 4.6 (e.g. ImageNet and CIFAR10/100) - Q3.", "relation_to_broader_scientific_literature": "Key contributions of the paper build on the previous methods (like CKA, RSA) and can be used as an extension of the existing testing procedures.", "essential_references_not_discussed": "The works [1] and [2] could be cited in the paper. They leverage CKA/RSA and task similarities for transfer learning. \n\n[1] Borup, Kenneth, Cheng Perng Phoo, and Bharath Hariharan. \"Distilling from similar tasks for transfer learning on a budget.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Dwivedi, Kshitij, and Gemma Roig. \"Representation similarity analysis for efficient task taxonomy & transfer learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.", "other_strengths_and_weaknesses": "Weaknesses:\n- Some captions of the figures could be more informative. E.g. In Fig. 2, it would be nice to add the legend of the colors used for different boxes (it is only done in the text)\n- In Fig. 5, some terms are difficult to understand (middle, bottom row) i it would be useful to briefly describe what the authors mean by Large etc. when it comes to the dataset size and similarly for model sizes. \n\nStrengths:\n- The authors compare different similarity metrics to minimize the possible impact of a given similarity measure on the results.\n- It is a useful combination of the existing methods.", "other_comments_or_suggestions": "As mentioned before, the authors could once more review their figures and add to their descriptions some better explanations of the contents, especially for Fig. 5 (a brief description of what the labels on the images mean).", "questions_for_authors": "**Q1**: Could the authors check what SSL models are placed close to the text-image models and try to explain why? (Fig. 4)\n\n**Q2**: Would it be possible to add additional results to Fig. 2 and 4 (or somewhere in the Appendix) including the EuroSAT/DTD datasets (as an example of other specialized/structured datasets - as such comparisons are the most interesting due to a domain gap between the general purpose and specialized/structured datasets). \n\n**Q3**: The authors state: “ Interestingly, ImageNet-1k exhibits a milder yet significant pattern of inconsistency. Within the multi-domain category, this is especially pronounced for CIFAR-10 and CIFAR-100” - do the authors think the reason is a significantly lower resolution of the images in the CIFAR datasets? Could the authors dig a little bit deeper to analyze why some datasets for visible clusters in Fig. 6?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DzD9CqueFj", "reviewer": "Reviewer_9H2r", "summary": "To compare representation spaces through representational similarity analysis (RSA) or its close relative in machine learning, centered kernel alignment (CKA), a sample of data is embedded in two different spaces, and the pairwise similarities of all representations in each space is used as a fingerprint for its information content.  The CKA or RSA value critically depends on the dataset from which the sample was drawn.\n\nThe current submission looks at the correlation of CKA values across datasets to draw conclusions about the similarities of vision models, or vice versa (correlation of CKA across models for conclusions about datasets), with motivation in large part to check the recent Platonic representation hypothesis (i.e. that the representation spaces of foundation models are converging).\n\n## Update after rebuttal \nThe authors have addressed all of my main concerns; I appreciate the clarity of the rebuttal and the additional experiments to test the hypotheses I raised.  There are interesting results and the analysis is sound.  I have adjusted my score from 3->4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that _the paper is well-written_ and that our claims are _supported by the results_, which _are presented clearly_. We are grateful for the valuable feedback that helped us improve our paper. We will address each concern point by point. All new figures/tables (R4F1-F7, R4T1-T2) are available [here](https://anonymous.4open.science/r/rebuttal_similarity_consistency/README.md).\n\nFirst, we want to address _[...] metrics might reflect the choice of datasets rather than similarities between the models_. Our analysis included 23 datasets, mostly part of the VTAB [1] and commonly used in CV. We compute the correlation of similarity values for all pairwise combinations of datasets. Therefore, we obtain similarity consistency measures for all dataset combinations (e.g., natural vs natural, structured vs structured, natural vs structured, …). These are responsible for the variance shown in our boxplot (Fig. 5). According to the reviewer’s suggestion, we further analyzed the observed variance in Fig. 5 and isolated the effect of different dataset types. We created two new boxplots (Fig. 5), one containing only correlation coefficients where both datasets contain natural images (Fig. R4F1) and one containing only coefficients for specialized+structured datasets (Fig. R4F2). The main observation remains the same: irrespective of the dataset selection, the training objective is a major driver for representational similarity consistency. However, we observe small differences such as a smaller minimum correlation for natural images  (see IN1k/XLarge & IN1k/Large) and a slightly larger variance for specialized and structure data (Fig. R4F2 vs Fig. R4F1). We will add the new plots to the camera-ready version’s appendix.\n\nSecond, we will address the relevance of sigma for CKA values computed with RBF kernel. We normalize all feature representations, leading to comparable distances over datasets. However, we agree that the strong correlation between CKA RBF 0.4 and CKA linear suggests that $\\sigma=0.4$ (partly) captures global similarity structures. Fig. 9 shows small differences in the upper right corner; however, we agree that the difference between CKA linear and CKA RBF 0.2 is more pronounced. We extend our correlation plot (Fig. 3) by CKA RBF 0.2 as shown in Fig. R4F3. As expected, CKA RBF 0.2 differs more significantly from CKA linear. Based on this evidence, we agree with your perspective that RBF 0.4 still captures some global similarity while RBF 0.2 is a better candidate for analyzing local similarity. Thus, we repeated our experiments with the local kernel (see Fig. R4F4 and R4F5). We observe the same overall pattern, indicating that the objective is relevant for similarity consistency, while network architecture is less important. This analysis reveals two interesting observations. For local similarity, the supervised models are more consistent than the Img-Txt models, which are better at recovering global structure. In addition, models trained on IN-21k are more consistent than models trained on IN-1k, of which the latter set is more consistent in their global structure. IN21k contains substantially more classes (21.843) and a higher percentage of the classes are leaf nodes in the WordNet tree (76.71% vs 65% for IN1k), representing more fine-grained entities. The representation must contain more fine-grained details to distinguish these classes, dominating local similarities. We will include these observations in Appx. F of the camera-ready version.\n\nThird, we investigate the stability of CKA values across different subsets. Fig. 10 shows that CKA linear is stable with 5k samples, showing smaller variance when increasing to 10k samples. To further analyze this stability, we performed bootstrapping over 500 subsets of IN-1k features, each containing 10k samples (as suggested), for 6 example models (the anchor models from Appx. G). Fig. R4F6 and the narrow confidence intervals (CI) in Tab. R4T1 confirm the stability of CKA linear values, demonstrating minimal variation across bootstrapped subsets. For RBF 0.2, Fig. 10 showed instability with a subset of size 10k, while 30k samples are more robust. We verified this with another bootstrapping experiment, Fig. R4F7, and the CIs in Tab. R4T2 showed a larger variance in CKA values. This suggests that local similarity depends more on the specific stimuli than global similarity. This is not surprising as stimulus-specific fine-grained details drive local similarity measurements. To mitigate this effect, we increased the number of samples for experiments analyzing local similarity from 10k to 30k when the dataset size permitted.\n\nLast, we agree that the t-SNE Figure helps us understand our categorization and will move this visualization from the appendix to the main part. \n\n[1] Zhai, Xiaohua, et al. \"A large-scale study of representation learning with the visual task adaptation benchmark.\" arXiv preprint arXiv:1910.04867 (2019).", "claims_and_evidence": "The claims are descriptive of the correlations the authors find and are supported by the results.", "methods_and_evaluation_criteria": "The first major analysis aggregates across ~20 datasets to compare models, but the rationale for this aggregation warrants further scrutiny.  Since the datasets are an arbitrary selection (not necessarily representative of any relevant distribution of images, with ~5% diabetic retinopathy images) and are weighted evenly in the correlation values, the resulting metrics might reflect the choice of datasets rather than similarities between the models.  If, for example, the proportion of datasets whose domain differs significantly from natural images were 10% or 50% instead of the ~25% used in the work, the correlation values could be entirely different.  \n\nTo phrase it differently, while CKA values can be related for any manner of producing different samples, the particular dataset selection appears likely to be a major driver in the observed trends.  In order to interpret the model similarity results (aggregating across datasets), it might be important to investigate the effect of such dataset selection.\n\nAggregating across models to compare datasets seems less problematic -- largely because there does not seem to be the same sense of outliers in the selection of models as there are for datasets.", "theoretical_claims": "I did not see any theoretical claims.", "experimental_designs_or_analyses": "There are relatively few design choices in the work.  One issue I see relates to the selection of a length scale $\\sigma=0.4$ for CKA RBF.  Can the authors justify why using a single value across the board is reasonable?  This assumes that a length of 0.4 has the same relevance in all of the models and for all of the datasets.  Why not use something adaptive, like the median distance between points?  I also did not find the ablation on this parameter (Fig 9) to support CKA RBF with $\\sigma=0.4$ as a local probe -- only 0.2 looks like it captures anything substantially different from CKA linear.", "relation_to_broader_scientific_literature": "The submission can be seen as a direct response to the Platonic representation hypothesis (Huh et al 2024), where the dependence of CKA on dataset immediately raises the question about how dataset factors into the PRH.  While there have been many works on assessing representational similarity, the contributions of this work are not in this direction: it primarily adopts one (the linear variant of CKA) and runs that through various levels of analysis.", "essential_references_not_discussed": "None to my awareness.", "other_strengths_and_weaknesses": "The primary contribution relates to the large-scale analysis of 64 models and 23 datasets, as the methodological innovation is limited: essentially performing correlation of CKA values.  As such, the contribution is somewhat slim, and the insights obtained are not particularly actionable.  Still, the paper is well-written and the results are presented clearly.", "other_comments_or_suggestions": "- I think the boxes in Fig 2 actually make it harder to assess similarity structure across models.  Perhaps signify at the boundary of the heatmap?\n\n- Might a bootstrap-type analysis act to ground the CKA values for each model and each dataset?  To be specific, one could compute CKA for different samples (size 10k, as is currently used in the submission) from each dataset to get a spread of values, and this spread would help shed light on what constitutes a meaningful difference in CKA values when comparing across models/datasets.", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This manuscript investigates why models of different architectures, even ones trained on different modalities, are similar to one another. It concludes that architecture and model size have little impact on model similarity, while the training objective has a significant impact. This is a topic of wide interest to the community at the moment. The reviewers found the manuscript to be interesting, well-written, and the experiments well-executed. One might even imagine practical applications of this work, where one wants to develop maximally different networks to solve problems in different ways; the insights here would help.
Decision: Accept (poster)