Wild Comparisons: A Study of how Representation Similarity Changes when Input Data is Drawn from a Shifted Distribution

Authors: davis  brown
madelyn  shapiro
alyson  bittner
jackson  warley
henry  kvinge
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-09 07:00:13

Abstract: Representation similarity functions, which apply geometric or statistical metrics to the hidden activations of two neural networks on a fixed set of input datapoints, have become important for assessing the extent to which models process and represent data in the same way. Past studies of these functions have largely been model-centric, focusing on varying the models under comparison while keeping the input data fixed. In particular, there have not been comprehensive evaluations of how the results change when either out-of-distribution inputs are used or when the data used in the comparison represents only a small part of the training set diversity (e.g., a single class). These are increasingly important questions as the community looks for tools to assess high-impact models which can be expected to encounter high-diversity, out-of-distribution data at deployment. In this work we explore the ways which input data affects the comparison of model representations. We provide evidence that while two model’s similarity does change when either out-of-distribution data or data representing a subpopulation of the training set is used, relative changes (e.g., ``model $A$ is more similar to $B$ than model $C$ is’’) are small for reasonable datasets. This robustness reinforces the idea that representation similarity functions are a promising tool for analysis of real-world models. Finally, we describe an intriguing depth-based pattern that arises in the representation similarity between different layers of the same model which could provide potential insight into how deep learning models process out-of-distribution data.

Main paper: 
 introduction: 
 In the last two years we have seen an increasing need for capable tools to audit, assess, and analyze deep learning models, either because their popularity and capability means that we can expect them to be deployed in many applications (e.g., foundation models) or because they specifically target high-consequence applications (e.g., medical imaging models). Model similarity functions are a promising tool for this task since they can be used to evaluate whether a new model processes and represents data in a way that is similar to models that may be better understood. For instance, when a new LLM is released, researchers might begin their assessment by comparing it to known LLMs and then use this information to guide downstream analysis.To apply a representation similarity function, a researcher needs a pair of models and a collection of data whose representation (within each of the models) will be compared (see Figure 1 ). Existing research has focused on varying the models being compared. For instance, Ding et al. (2021) study whether a similarity function is sensitive to model's random initialization even when all other aspects of model architecture and training are held constant. On the other hand, little attention has been given to the data that is the basis of the model comparison. The standard procedure would be to assume that both models have the same training/test set and use examples from the test set as input to the similarity function. However, it is increasingly the case that some models (i) are expected to be applied to a broad set of data on deployment that might not be well-represented by the test set or (ii) who have a training set whose diversity is computationally infeasible to capture with current representation similarity functions. Further, the question of not just whether two models are similar, but where they are similar is interesting from the perspective of the emerging science of deep learning. Indeed, from a mathematical point of view, a comparison of two functions f 1 and f 2 based on their value at a small number of points is fundamentally limited, particularly when f 1 and f 2 have the richness and complexity of modern deep learning models. For instance, it is well-known that representation similarity metrics can detect the changes to a model caused by adversarial training. Are these changes purely local to the data distribution where this training occurs, or do they extend to out-of-distribution data? A better understanding of these issues has the potential to help us understand and predict model behavior in the face of distribution shift, a fundamental challenge in real-world machine learning. Hence, we consider the following question:To what extent do model comparisons depend on the distribution we draw input data from?We describe an initial set of experiments to probe this question. We find that model similarities do change (sometimes substantially) as input data drifts further from the training distribution, but changes to relative similarity, by which we mean changes to the ranking (by similarity) between a fixed model f i and all other models under consideration f 1 , . . . , f k , remains relatively minor for most datasets. Even datasets that can be characterized as coming from a different imaging domain, such as satellite images, induce only moderate changes in relative similarity. This is true when we control for differences in architecture, comparing only ResNet50 models with different training routines.We see a similar phenomenon when we compare representations on subpopulations of the data. While the absolute similarity can change a surprising amount from subpopulation to subpopulation, relative similarity changes very little. These results, together with the above finding on out-ofdistribution data suggest that as long as one is only interested in the relative similarity of models, there is flexibility with the data that is used for comparison. In terms of our understanding of deep learning models more generally, our results suggest that changes to model representations induced by training (e.g., adversarial training, different augmentation schemes) have global effects, impacting the way a model represents data well outside its training distribution.Finally, besides looking at changes in representations between models, we also look at changes representations at different layers of the same model. Here we find a curious pattern in which early and late layers change substantially (relative to other layers) as we move to out-of-distribution data, while changes to middle layers are substantially less. We speculate that this is due to OOD data having different input statistics and class outputs, but having largely similar features in the intermediate layers. Altogether our contributions in this paper include:• Formulation of the question of how model representation similarity changes as a function of the input data distribution.• Evidence that while absolute model similarity can change from data distribution to data distribution, the relative representation similarities of models can persist even when input data is taken out-of-distribution or from a subpopulation of the training distribution.• We describe an interesting phenomenon where the representations at early and late layers of a model change more than middle layers when out-of-distribution input data is used. 
related work: 
 There are a range of approaches to analyzing the hidden representations of deep learning models. Prominent examples not used in this work include canonical correlation (Hotelling, 1992) and its relatives such as SVCCA (Raghu et al., 2017) and PWCCA (Morcos et al., 2018) . There are also topology-based similarity functions such as Geometry score (Khrulkov & Oseledets, 2018) , representation topology divergence (Barannikov et al., 2021) , or the similarity function described in (Purvine et al., 2023) . Representations can also be compared via differences in statistics such as intrinsic dimension (Ansuini et al., 2019) .One set of similarity functions that are thematically related to this work are the neighborhood based similarity methods that use Jaccard distance (Schumacher et al., 2021) , (Hryniowski & Wong, 2020) , (Gwilliam & Shrivastava, 2022) , cosine similarity (Hamilton et al., 2016) , or rank similarity (Wang et al., 2020) . Though we also consider relative distances in this work, our relative distances are between models rather than individual representations as in the papers above.Representation similarity has been an important tool in better understanding how and why deep learning models work. For instance, Li et al. (2015) uses similarity functions to show that even with different initializations, models tend to learn the same features during training. Other works look at issues around the use of similarity functions. Sucholutsky et al. (2023) attempts to create a unified framework around representation similarity across machine learning, neuroscience, and cognitive science.We a few works that also concentrate on the input data to similarity functions. Cui et al. (2022) explores a failure mode wherein the similarity or dissimilarity of the input data used confounds measurement of the similarity or dissimilarity of representations. In Section 4 we show that the similarity of models is higher for certain architectures when evaluating on subpopulations. This may be a consequence of the observation in Cui et al. (2022) , as random images from a specific class may tend to be more similar than random images from all of ImageNet. However, we also show that these differences have little impact on relative similarity. Recently, Klabunde et al. (2023a) observe that the relative similarity between language models is dependent on the application of the models and, for example, the difficulty of the task. Finally, Gondhalekar et al. ( 2023) also looks at CKA values between different layers of the same model when out-of-distribution data is used as input. They find that different layers tend to be more similar in models that fail to generalize. This is distinct from our layer-based observation that the early and late layers of a model tend to change more (relative to other layers of the model) when any out-of-distribution data is applied. 
background on representation similarity functions: 
 We briefly review representation similarity functions with a focus on those that we use in the experiments in this paper. For a comprehensive survey of this domain, see Klabunde et al. (2023b) .Let f 1 , f 2 : X → R n be neural networks with f <k i : X → Z k i the function defined by the first k layers of f i where i ∈ {1, 2}. For instance, f <10 1 is the first 10 layers of f 1 with Z 10 1 the space of hidden activations at layer 10. A representation similarity function is a function that quantifies the extent to whichf <k 1 (x 1 ), . . . , f <k 1 (x n ) and f <k 2 (x 1 ), . . . , f <k 2 (x n ) are or are not similar for some x 1 , . . . , x n ∈ X. It is convenient to represent f <k 1 (x 1 ), . . . , f <k 1 (x n ) (re- spectively f <k 2 (x 1 ), . . . , f <k 2 (x n )) as a n × k 1 matrix A k1 1 (x 1 , . . . , x n ) (resp. n × k 2 matrix A k2 2 (x 1 , . . . , x n ))whose rows are the representation of each data point and whose columns are the k 1 (resp. k 2 ) dimensional features. In this context a representation similarity function m takes A k1 1 (x 1 , . . . , x n ) and A k2 2 (x 1 , . . . , x n ) and returns a scalar value quantifying some aspect of the similarity of A k1 1 (x 1 , . . . , x n ) and A k2 2 (x 1 , . . . , x n ). We next briefly review some of the representation similarity functions used in this work.Centered Kernel Alignment (Kornblith et al., 2019) : Centered Kernel Alignment (CKA) captures pairwise similarity within a given set of representations using a representation similarity matrix [S] i,j . This is obtained by applying a kernel function to each pair of hidden activations f <k 1 (x i ) and f <k 2 (x i ) after they have been mean centered. The Hilbert-Schmidt Independence Criterion is then used to evaluate the extent to which to such similarity matrices S 1 , S 2 are the same. This framework means that CKA is invariant to orthogonal transformation and does not require that the representations that are being compared have the same dimension.Orthogonal Procrustes Distance (Ding et al., 2021) : Aims to find orthogonal transformation R from the group O(d) where d = dim(Z k22 ) = dim(Z k1 1 ) such that the Frobenius norm||A k1 1 (x 1 , . . . , x n ) -A k2 2 (x 1 , . . . , x n )R|| F is minimized, i.e., can we make A k2 2 (x 1 , . . . , x n ) look like A k11 (x 1 , . . . , x n ) by rotating the latter? Orthogonal Procrustes is invariant to orthogonal transformations by construction.Permutation CKA and Procrustes Distance: These two similarity functions are analogous to those above but instead of being invariant to any orthogonal rotation in the activation basis, they are only invariant to permutation of activations. The reasoning for this comes from the fact that multiple works have either used theory (Godfrey et al., 2022) or empirical observation (Entezari et al., 2021) , (Bau et al., 2017) as support for the idea that permutations (rather than any orthogonal transformation) are the family of transformations that hidden representations should be invariant to. We refer the reader to Godfrey et al. (2022) for more details on permutation CKA and Williams et al. (2021) for more details on permutation Procrustes. 
experiments: 
 Do relative similarities among models persist when out-of-distribution data is used?: Recall that one of the questions that we are interested in is the extent to which relative similarity changes when out-of-distribution data is used as input to a representation similarity function. We focus our experiments on a collection of models that have fixed architecture and primarily differ in the type of training schemes used, as we would like to know whether relative similarity induced when training on certain distributions persists when measured on other distributions. We restrict ourselves to comparing latent representations for this initial study.Our set of models consists of a range of different ResNets, all trained on ImageNet (Russakovsky et al., 2015) , including torchvision pretrained ResNets (Marcel & Rodriguez, 2010) with both 'version 1' and 'version 2' weights (the latter with more sophisticated augmentation schemes, learning r ate scheduler, etc.). We also include several ResNet50 models trained with different types of heavy augmentation (e.g., PRIME (Modas et al., 2022) , deepaugment (Hendrycks et al., 2021a) , augmix (Hendrycks et al., 2019) ) or data with style transfer applied (Geirhos et al., 2018) , as well as 8 different ResNet50's trained with adversarial training for varying values (Salman et al., 2020) . We summarize these models in Section A.1 of the Appendix.We then choose five datasets that are distinct from ImageNet: vehicle chips from the satellite imaging dataset xView (Lam et al., 2018) , ImageNet-R (Hendrycks et al., 2021a) , MNIST Deng (2012) , ObjectNet (Barbu et al., 2019) , and Describable Textures (Cimpoi et al., 2014) . For each of these datasets D k we (i) choose a random set of 500 images X k , (ii) use these images as input to each pair of models (f i , f j ), (iii) extract the corresponding latent representations f i,latent (X k ) and f j,latent (X k ), and (iv) apply CKA and permutation CKA to each of these pairs f i,latent (X k ) and f j,latent (X k ) to get κ CKA i,j,k and κ pCKA i,j,k . Note that each κ CKA i,j,k (respectively κ pCKA i,j,k ) depends on the pair of models (indices i and k) and the dataset (index k). Figure 3 : Spearman's rank correlation coefficient (y-axis) captures the changes in relative similarity when two models are evaluated on their test data (ImageNet) vs OOD datasets (x-axis) using CKA or permutation CKA. Larger values (with a maximum of 1) indicate that the relative similarity between models is preserved when we evaluate them on a given OOD dataset. A value of 0 would mean there is no correlation between relative similarity on ImageNet vs OOD data and a value of -1 would mean that similarity is exactly reversed on OOD data.In Figure 2 , we plot CKA values for the latent representations of a torchvision ResNet50 model trained on ImageNet (with 'version 1' pretrained weights) and range of other ResNet50 models (xaxis). The different colored bars indicate the dataset from which input data was drawn. We see that CKA values can change substantially when moving from ImageNet (blue bars) to out-of-distribution datasets. There are some patterns in these changes. For instance, models tend to be more similar when data from xView, DTD, or ImageNet-R is used, while they are less similar on ObjectNet. Whether similarity increases on MNIST depends on the model that one uses. We speculate that this pattern may relate to the complexity or semantic variety of the datasets. xView images are low-resolution and have low-semantic variety (being mostly low-resolution chips of satellite images with blurry vehicles. ImageNet-R images are drawings, paintings, etc. and hence tend to have less complexity, and DTD is textures with repeated patterns. On the other hand, ObjectNet has high complexity images with clutter. See Figure 4 for examples. MNIST images are simple, but they may be far enough out-of-distribution to not follow the pattern.To understand how the relative similarity between models changes as we move to out-of-distribution datasets, for each dataset k and model f i , we construct a vector of similarities (for CKA or permutation CKA) between f i and all of the other models ({κ CKA i,j,k } j or {κ CKA i,j,k } j ). Then for each f i , we compare the vector measured on ImageNet with the vector measured on one of the out-of-distribution datasets using Spearman's rank correlation coefficient. If this value is close to 1, it means that the ranking of similarities of f i to other models does not change as we move out-of-distribution, if it is 0, it means there is no correlation between both rankings, and if it is -1 it means that one ranking is the reverse of the other.We show the result of this experiment in Figure 3 . While CKA and permutation CKA values change when out-of-distribution data is used as input, we see that changes in relative similarity are minor. This is the case even when all models have similar architectures and were trained on the same dataset (albeit with different augmentation schemes). The only dataset where relative similarity changes substantially is MNIST, which we would argue is significantly out of distribution.We see similar results when we perform our measurements using Procrustes distance and permutation Procrustes (Figure 9 ). We note that the two similarity functions that are only permutation invariant tend to be more impacted by changes in distribution. We suspect that this is due to the fact that such similarity functions are more sensitive to changes in semantic content given the importance of the activation basis in this context (Godfrey et al., 2022) . • While representation similarity values change when out-of-distribution data is used, changes to the relative similarity between models tend to be minor.What do these results say about a model's representations?: Our experiments support the idea that relative similarity is persistent well-beyond the data distribution where a model is trained. We interpret this as evidence that the way that a model is trained has a substantial impact on how it represents data even when we move outside of the training distribution. While this may not seem surprising at first glance, specifying the value of an arbitrary function at a finite number of points tells us very little about how that function will behave elsewhere, especially in the cases that we consider where the support of these points (the training distribution) is a low-dimensional manifold that does not even take up a sliver of the high-dimensional input space. So just as different neural networks, that could easily learn different representations of data and still solve a problem instead often learn similar representations (Li et al., 2015) , a neural network that in theory should be able to have certain properties on the training distribution and different properties elsewhere ends up being defined largely by learning that is done on the thin slice of input space where the training data sits.Do relative similarities persist when we draw data from a subpopulation of the training distribution?:We ask a similar question to that above but for subpopulations of the training distribution rather than out-of-distribution data. For instance, one might worry that model A would be measured to be more similar to model B than model C when we sample from one class but A would be measured as more similar to C when we use data from another class. To test this we pick 5 random classes from ImageNet: Scorpion, Ant, Otter, Barbell, and Pay-phone. Restricting our input data to each of these in turn, we run CKA on the same pairs of ResNets from the experiment above. We plot CKA values between a ResNet50. Then, as above, we construct vectors with ranked similarity for each fixed model. Using Spearman's rank correlation coefficient we compare rankings corresponding to all of ImageNet and rankings corresponding to a single class. We show our result in Figure 6 , where we again find that even though model similarity shows variance between different classes, changes to relative similarity are minor. Of course, classes are only one way to partition a dataset. It would be interesting to understand the effect of restricting to different types of subpopulations (e.g., different classes but same background, same low-level pixel statistics, etc.).We see similar results when we perform our measurements using Procrustes distance and permutation Procrustes (Figure 9 ) Limiting the input data of a representation similarity function to a single class has a small effect on the relative similarity of models.Do the representations within the layers of a model remain consistent when OOD data is used?:We also examine how the representations between layers in a single model change when evaluated on OOD data. We find that representations at very early and very late layers change the most (with respect to the representations found at other layers). Modifying the notation used above we write κ CKA, i,j,k (resp. κ pCKA, 1 , 2 i,j,k) for the CKA (resp. permutation CKA) value for model f i and f j , on dataset k, at layer 1 in f 1 and 2 in f 2 . Thus, when i = j, we can form the matrix which consists of CKA values for each pair of layers 1 and 2 in f i ,K CKA i,k := [κ pCKA, 1, 2 i,i,k] 1, 2 . To study how this matrix changes when out-of-distribution data is used as input, we explore the representations of timm models (Wightman, 2019) on ImageNet data (the training distribution) and ImageNetA (Hendrycks et al., 2021b) , ImageNetV2 (Recht et al., 2019) , ImageNet-R, and ObjectNet (out-ofdistribution data).We ask how representations change with respect to depth-that is, which layers change the most as we move out-of-distribution? To measure this, we first take the difference between K CKA i,ImageNet and K CKA i,k , which tells us the absolute change from ImageNet and an OOD dataset k in between-layersimilarities. The norm over each row i gives the absolute change in how similar a layer i is to other layers j, which we normalize to get the relative change:EQUATION.(1)We compute this metric for the OOD datasets k, across 428 timm models. Results for ImageNetV2 are shown in Figure 7 , where we bin across relative depth (which varies from 0 to 1 based on the fraction of layers the data has passed through). We find that early and late layers show systematically larger differences on OOD data than middle layers. We include similarity heatmaps for a random subset of models along with the other datasets, which show nearly identical trends, in the Appendix. We speculate that the image-level statistics are different between in-distribution and outof-distribution data, causing representations close to the input to be different. On the other hand, we know that later layers capture much of the class output differences, so these may be different because models are less accurate /class representations are different on OOD data. Further experiments would be necessary to validate these hypotheses. Early and late layers of a model change more with respect to the other layers on OOD data-that is, their relative representation similarities shift more prominently than middle layers. 
conclusion: 
 In this paper we explore how representation similarity values change when either out-of-distribution data or subpopulations of the training data are used as input. We find that while similarity does change in these situations, relative similarity does not, meaning that models that are similar on indistribution data tend to be similar on out-of-distribution data. This not only provides evidence that model comparisons that hold in the lab, often persist when those models are deployed and encounter data from a shifted distribution but also that while training is performed on a specific distribution (usually a thin slice of the input space), the impact of training extends its imprint far beyond to out-of-distribution data.  
 
 Appendix:  



Reviews: review: title: title: Important research question for the representation comparison field, clear experiments
main_review: main_review: **Summary:**

The paper is primarily concerned with the question “To what extent do model comparisons depend on the distribution we draw input data from?” They consider two extreme cases, one where the input data is drawn from a subset of the “in-distribution” and one where the input data is out-of-distribution. They consider ResNet models trained on ImageNet for their experiments and the CKA, permutation CKA, Procrustes distance and permutation Procrustes representation comparison methods. Contributions: while absolute measures of representation similarity can vary widely for OOD data (less so for subset of the training distribution), the relative similiarities (i.e. “model A is closer to model B than to model C”) is relatively robust to the input data. They highlight an interesting phenomenon where OOD data representations at early and late layers change more than for middle layers.

See "Weak points" and "Strong points" for the actual main comments.

**Questions/Feedback/Suggestions:**

- Which representations of the ResNet models are compared in Figures 2 and 3, at which layers of the networks are we comparing the representations? I assume it’s the same layer from two different networks being compared since this is standard, but is this averaged over all possible layers or only one layer is taken? A split of this analysis into “early”, “mid” and “late” layers representation comparison would be interesting.
- “The only dataset where relative similarity changes substantially is MNIST, which we would argue is significantly out of distribution.” → It would be interesting to see if we could have a ranking of the “OOD-ness” of the datasets as well and see if higher drops in similarity is correlated with being more out of distribution.
- What is the difference between the ResNet models used early in the paper and the timm ones mentioned later? This isn’t quite clear.
- There is no reference to Figure 5 in the text.

**Typos:**

- “We a few works…”  beginning of the last paragraph of section 2
- Procrustes results mentioned in OOD section point to subset results (fig 9 but it should be fig 8 I think)
strength_weakness: strength_weakness: **Strong points:**
- The question of interest is highly relevant. So far representation similarity measures have mainly been applied to in-distribution data points. However, modern models are increasingly expected to be robust to large variations in input data distribution. Therefore, better understanding how representation similarity measures behave when OOD data is used as input is an important research direction.
- I like the fact that the authors consider both the local case (subset of the “in” distribution) and the OOD case.
- The relative change of similarity scores is an interesting analysis tool.
- Generally well written and clear text (although maybe a bit repetitive in places).
title: title: Interesting but perhaps not very novel
main_review: main_review: The paper investigates how model representation similarity changes depending on the input distribution. They find that while changes to absolute similarity are substantial, the relative ordering of models in their similarities does not change. 

**Clarity**

The paper is fine to read but I found the writing unintuitive at parts: for instance, in the abstract it says _relative changes (e.g., ''model A is more similar to B than model C is'') are small for reasonable datasets_. I found this confusing and I think something like _changes in the relative similarity between models (e.g., ''model A is more similar to B than model C is'') are small_ would be easier to understand instead. Also, what is a reasonable data set here? 

**Correctness**

The methods seem correct and the results are about what I would expect. 

**Novelty**

While the framing of the investigation is interesting, I think similar work has already been done: Supplementary C.3, Table 9 in [1] calculated CKA on OOD data over layers and also find that early and late layers have a lower rank correlation than later layers. Supplementary D, Figure 10 in [2] shows CKA over layers for both in- and out-of-domain data. Also [3] show in Supplementary I that odd-one out agreements reflect representational similarity and they investigate odd-one out agreements on OOD data. Ideally, the authors could add a section on how these studies relate to their results. 

**Relevance for community**

Even though I don't consider the results to be particularly novel, I do think the work fits well within the scope of the workshop and is still of interest to the community. 


[1] Ding, Frances, Jean-Stanislas Denain, and Jacob Steinhardt. "Grounding representation similarity with statistical testing." arXiv preprint arXiv:2108.01661 (2021).

[2] Mathis, Alexander, et al. "Pretraining boosts out-of-domain robustness for pose estimation." _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 2021.

[3] Muttenthaler, Lukas, et al. "Human alignment of neural network representations." _arXiv preprint arXiv:2211.01201_ (2022).
strength_weakness: strength_weakness: In general the underlying question is interesting: _do representation similarity metrics remain a valid tool for model comparisons even on OOD data_? Also, it fits well within the scope of the workshop and is likely still of some interest to the community.
title: title: I propose acceptance, as the paper generates interesting insights.
main_review: main_review: *Summary:*
- The paper investigates the similarity between model representations when input data is out of distribution.
- The main contributions of the paper are its novel perspective from inspecting out of distribution data and relative similarity (similarity by ranking).

*Strengths:*
- The paper takes a novel perspective by evaluating on out-of-distribution data and quantifying relative similarity.
- The 3 experiments defined are interesting and generate good insights.
- I believe the insights/take is interesting for the wider community and will inspire further research.

*Weaknesses:*
- Some parts of the paper are very hard to read.
- I believe the 'Experiments' section would benefit from a more traditional and clear structure, such as models, datasets, and the experimental setup. 
- The paper lacks an investigation of where the dissimilarities emerge from, allowing for deeper insights.
- The paper only investigates ResNet and a limited set of datasets.

*Recommendations:*
- Accept (Poster)
- I recommend accepting this paper as it presents an interesting investigation into model representations.
- I recommend accepting for a poster, as I believe the conclusions that can be drawn from the results are not clear enough for a talk.

*Supporting Arguments for Recommendation*:
- Within the setting of the workshop, I think the paper presents enough novel results to warrant acceptance.
- I believe the paper needs some restructuring/rewriting and more experiments (read more models) to move towards conference publication level.

*Questions:*
- Can you quantify the changes to relative similarity in a more fine-grained manner? What I am wondering is how close the ranking is to changing.
- Would it be possible to have more controlled toy experiments to check where dissimilarity in out-of-distribution data representations comes from? What I am trying to get at is whether you can make more assertive statements about where the trends emerge from, rather than speculating on dataset similarity/complexity.

*Feedback:*
- I would recommend rereading the paper aloud and shortening sentences. For example, the second sentence of the fourth paragraph in the introduction is very hard to read, but highly important because it introduces relative similarity.
- I think Figure 1 can be made smaller, removed or needs to be improved. While it communicates what the papers research question is, it is not useful beyond this.
-  I think it would also be interesting to investigate how similarity changes within layer with increasing width. I know that this is harder to do, as the representations will be different in dimensionality, but maybe something to think about.
strength_weakness: strength_weakness: I believe the paper is a very good start and interesting to the wider community (e.g. inspiring future research). Therefore, I believe it does not warrant a lower score.