Reliability of CKA as a Similarity Measure in Deep Learning
Authors: MohammadReza Davari
Stefan Horoi
Amine Natik
Guillaume Lajoie
Guy Wolf
Eugene Belilovsky
Venue: ICLR 2023 poster (2023)
Keywords: Representation Learning
Similarity Measures
Centered Kernel Alignment (CKA)

Abstract: Comparing learned neural representations in neural networks is a challenging but important problem, which has been approached in different ways. The Centered Kernel Alignment (CKA) similarity metric, particularly its linear variant, has recently become a popular approach and has been widely used to compare representations of a network's different layers, of architecturally similar networks trained differently, or of models with different architectures trained on the same data. A wide variety of claims about similarity and dissimilarity of these various representations have been made using CKA results. In this work we present analysis that formally characterizes CKA sensitivity to a large class of simple transformations, which can naturally occur in the context of modern machine learning. This provides a concrete explanation to CKA sensitivity to outliers, which has been observed in past works, and to transformations that preserve the linear separability of the data, an important generalization attribute. We empirically investigate several weaknesses of the CKA similarity metric, demonstrating situations in which it gives unexpected or counterintuitive results. Finally we study approaches for modifying representations to maintain functional behaviour while changing the CKA value. Our results illustrate that, in many cases, the CKA value can be easily manipulated without substantial changes to the functional behaviour of the models, and call for caution when leveraging activation alignment metrics.

Published as a conference paper at ICLR 2023
RELIABILITY OF CKA AS A SIMILARITY MEASURE IN DEEP LEARNING
MohammadReza Davari 1,3 Stefan Horoi 2.3 Amine Natik 2,3 Guillaume Lajoie 2,3
Guy Wolf 2 Eugene Belilovsky 1,3
Mila Quebec AI Institute (mohamnadreza davari,, eugene. (aalloovsky@@cccecco (stefan horoi, amine.. miine.natik, guillaume. lajoie, guy. wol f @umentreal, ca
Concordia University Université de Montréal
ABSTRACT
Comparing learned neural representations in neural networks is challenging but important problem, which has been approached in different ways. The Centered Kernel Alignment (CKA) similarity metric, particularly its linear variant, has recently become a popular approach and has been widely used to compare represen- tations of a network's 's different layers, architecturally similar networks trained differently, or of models with different architectures trained the same data, A wide variety of conclusions about similarity and dissimilarity of these various representations have been made using CKA. In this work we present analysis that formally characterizes CKA sensitivity to a large class of simple transformations, which naturally occur in the context of modern machine learning. This provides a concrete explanation of CKA sensitivity to outliers, which has been observed in past works, and transformations that preserve the linear separability of the data, an important generalization attribute. We empirically investigate several sensitivities of the CKA similarity metric, demonstrating situations in which it results. Finally we study approaches for
gives unexpected or counter- modifying representations to maintain functional behaviour while changing the CKA value. Our results illustrate that, in many cases, the CKA value can be easily manipulated without substantial changes to the functional behaviour the models, and call for caution when leveraging activation alignment metrics.
INTRODUCTION
In the last decade, increasingly complex deep learning models have dominated machine learning and have helped us solve, with remarkable accuracy, multitude of tasks across a wide array of domains. Due to the size and flexibility of these models it has been challenging to study and understand exactly how they solve the tasks we use them on. A helpful framework for thinking about these models is that of representation learning, where we view artificial neural networks (ANNs) as learning increasingly complex internal representations as we go deeper through their layers. In practice, it is often of interest to analyze and compare the representations of multiple ANNs. However, the typical high dimensionality of ANN internal representation spaces makes this a fundamentally difficult task.
To address this problem, the machine learning community has tried finding meaningful ways to compare ANN internal representations and various representation (dis)similarity measures have been proposed (Li et al., 2015; Wang et al. [018 Raghu al. 2017 Morcos al. 2018). Recently, Centered Kernel Alignment (CKA) (Krrblith et al.] 2019) was proposed and shown to be able to reliably identify correspondences between representations in architecturally similar networks trained on the same dataset but from different initializations, unlike past methods such as linear regression or CCA based methods (Raghu al. 2017 Morcos al. 2018). While CKA can capture different notions of similarity between points in representation space by using different kernel functions, it was empirically shown in the original work that there are no real benefits using CKA with a nonlinear kernel over its linear counterpart. As a result, linear CKA has been the preferred representation similarity measure of the machine learning community recent years and other similarity measures (including nonlinear CKA) are seldomly used. CKA has been utilized in number of works to make conclusions regarding the similarity between different models and their behaviours such as wide versus deep ANNs (Nguyen et al. 2021) and transformer versus CNN based ANNs (Raghu et al.) 202T). They have also been used to draw conclusions about transfer learning (Neyshabur et al. (020) and catastrophic forgetting (Ramasesh et al.) 2021). Due to this widespread use, is important to
" Equual contribution, name order randomized
Equal senior-author contribution

---

Published as a conference paper at ICLR 2023
understand how reliable the CKA similarity measure is and in what cases it fails to provide meaningful results. In this paper, we study CKA sensitivity to a class of simple transformations and show how CKA similarity values can be directly manipulated without noticeable changes in the model final output behaviour. In particular our contributions are as follows:
In Sec. and with Thm. [ we characterize CKA sensitivity a large class of simple transformations, which can naturally occur in ANNs. With Cor, Cand extend our theoretical results to cover CKA sensitivity to outliers, which has been empirically observed in previous work (Nguyen al. 2021 Ding et al. 2021; Nguyen et al. 2022), and to transformations preserving linear separability of data, an important characteristic for generalization. Concretely, our theoretical contributions show how the CKA value between two copies of the same set of representations can be significantly decreased through simple, functionality preserving transformations of one of the two copies. In Sec. 4 we empirically analyze CKA's reliability, illustrating our theoretical results and subsequently presenting a general optimization procedure that allows the CKA value to be heavily manipulated to be either high or low without significant changes to the functional behaviour of the underlying ANNs. We use this to revisit previous findings (Nguyen et al. 2021; Kornblith et al. 2019).
2 BACKGROUND ON CKA AND RELATED WORK
Comparing representations Let X Raxdd denote a set of ANN internal representations, i.e., the neural activations of specific layer with d1 neurons in network, in response to n E N input examples. Let Y E Raxda be another set of such representations generated by the same input examples but possibly at a different layer of the same, or different, deep learning model. is standard practice to center these representations column-wise (feature or "neuron" wise) before analyzing them. We are interested in representation similarity measures, which try to capture a certain notion of similarity between X'' and Y.
Quantifying similarity Li al. (2015) have considered one-to-one, many-to-one and many-to- many mappings between neurons from different neural networks, found through activation correlation maximization. Waan et al. (2018) extended that work by providing rigorous theory of neuron activation subspace match and algorithms to compute such matches between neurons. Alternatively, Raghu et al.) (2017) introduced SVCCA where singular value decomposition used identify the most important directions in activation space. Canonical correlation analysis (CCA) is then applied to find maximally correlated singular vectors from the two sets of representations and the mean of the correlation coefficients is used similarity measure. In order give less importance to directions corresponding to noise, Morcos et al. (2018) introduced projection weighted CCA (PWCCA). The PWCCA similarity measure corresponds the weighted sum of the correlation coefficients, assigning more importance to directions in representation space contributing more to the output of the layer. Many other representation similarity measures have been proposed based on linear classifying probes (Alain & Bengio 2016 Davari et al. 2022), fixed points topology of internal dynamics in recurrent neural networks (Sussillo & Barak 2013 Maheswaranathan al., (019), solving the orthogonal
Procrustes problem between sets of representations (Ding et al. 2021] Williams et al.] (021) and
many more (Laakso & Cottrell 2000; Lenc & Vedaldi 2018 Arora al. 2017). We also note that a
large body of neuroscience research has focused on comparing neural activation patterns in biological neural networks (Edelman 1998| Kriegeskorte et al. [0088 Williams et al. 2021 Low et al., (021).
CKA Centered Kernel Alignment (CKA) (Kornblith et al. (2199) another such similarity measure based on the Hilbert-Schmidt Independence Criterion (HSIC) (Gretton et al.) 2005) that was presented as means to evaluate independence between random variables in non-parametrie way. For KiJ = (1,,, and Tæ.j = 1yys Yj) where k, are kernels and for H = _11" the centering matrix, HSIC can be written as: HSIC(K, L)- = (n-1)a tr(HHHH CKA can then be computed as:
HSIC(K, L)
CKA(K, / /HSIC(K, K KOHSIC(LL, I.)
In the linear case l: and are both the inner product so K -XT, YYT and we use the Intuitively, HSIC computes the similarity structures of X and Y. as measured by the kernel matrices K and T., and then compares these similarity structures (after centering) by computing their alignment through the trace of KH LH.
notation C'K ("KA(X,Y) CC A YYT).
Recent CKA results CKA has been used in recent years to make many claims about neural network representations. Nguyen et al. (202T) used CKA to establish that parameter initialization drastically impact feature similarity and that the last layers of overparameterized (very wide or deep) models
2

---

Published as a conference paper at ICLR 2023
learn representations that are very similar, characterized by a visible "block structure" in the networks CKA heatmap. CKA has also been used to compare vision transformers with convolutional neural networks and to find striking differences between the representations learned by the two architectures, such as vision transformers having more uniform representations across all layers (Raghu et al. (02T). Ramasesh et al. (202T) have used CKA to find that deeper layers are especially responsible for forgetting in transfer learning settings.
Most closely related to our work, Ding et al. (2021) demonstrated that CKA lacks sensitivity to the removal of low variance principal components from the analyzed representations even when this removal significantly decreases probing accuracy. Also, Nguyen et al. (2022) found that the previously observed high CKA similarity between representations of later layers in large capacity models (so-called block structure) actually caused by a few dominant data points that share similar characteristics. |Wiliiams et al. (2021) discussed how CKA does not respect the triangle inequality, which makes it problematic to use CKA values as a similarity measure in downstream analysis tasks. We distinguish ourselves from these papers by providing theoretical justifications to CKA sensitivity to outliers and to directions of high variance which were only empirically observed in Ding et al. do not only present situations in which CKA gives
(2021); Nguyen et al. (202T). Secondly, unexpected results but we also show how CKA values can be manipulated to take on arbitrary values.
Nonlinear CKA
The original CKA paper (Korbblihh et al. 2019) stated that, in practice, CKA with nonlinear kernel gave similar results as linear CKA across the considered experiments. Potentially as result of this, all subsequent papers which used CKA A as a neural representation similarity measure have used linear CKA A (Maheswaranathan al. 2019
Neyshabur al.. 2020; Nguyen al. T02T 2021: Williams et al [202]] Kornblith et al. 2021). and to our knowledge, no published work besides Kornblith al. (201);; Nguyen al. (2022) has used CKA with nonlinear kernel. Consequently, we largely focus our analysis on linear CKA which is the most popular method and the one actually used in practice. However, our empirical
Raghu et al., [222]] Ramasesh et al. 2002 Ding et al.
results suggest that many of the observed problems hold for CKA with an RBF kernel and we discuss a possible way of extending our theoretical results to the nonlinear case.
3 CKA SENSITIVITY TO SUBSET TRANSLATION
Invariances and sensitivities are important theoretical characteristics of representation similarity measures since they indicate what kind of transformations respectively maintain change the value of the similarity measure. In other words they indicate what transformations "preserve" or "destroy" the similarity between representations defined by the specific measure used. Kornblth et al. (2019) argued that representation similarity measures should not be invariant to invertible linear transformations. They showed that measures which are invariant to such transformations give the same result for any set of representations of width (i.e. number of dimensionssnnerron))) greater than or equal to the dataset size. They instead introduced the CKA method, which satisfies weaker conditions of invariance, specifically it is invariant to orthogonal transformations such as permutations, rotations and reflections and to isotropic scaling. Alternatively, transformations to which representation similarity measures are sensitive have not been studied despite being highly informative as to what notion of "similarity" is captured by a given measure. For example, a measure that is highly sensitive to transformation T is clearly measuring notion of similarity that is destroyed by T. In this section, we theoretically characterize CKA sensitivity to a wide class of simple transformations, namely the translation of subset of the representations. We also justify why this class of transformations and the special cases it contains are important in the context of predictive tasks that are solved using neural networks.
Theorem 1. Consider a set of n. internal representations in p dimensions X E Ruxp that have been centered column-wise let S C X such that p Xs, = Suf + ct : I XXSS.. Then we have:
[X| and t such that | = 1. We define
[S]
| [x]]
lim CK Atm (X, (x, = r(p)) 0-000
dimpp(X)
(2)
E,ex [|=]|]]]
(x))))² Σ. E [1,p] the dimensionality estimate
where l'(p) - E (0,1), and dimpn(X) provided by the participation ratio of eigenvalues { of the covariance of X. Corollary 2. Thm. I holds even if S is taken such that p X
Is]
E =0..5, 1).
3

---

Published as a conference paper at ICLR 2023
X
X X5
S X 36
07
C0
Visual representations of the transformations considered in the theoretical results. a) Thm. The original of neural representations X contains subsets S (red) and X (green). We can then build X v. as a copy of X, where the points X 5 are translated a distance e direction ग. The linear CKA value between it. X and Xs, ste is then computed. b) Cor.3 3 X and st. differ by a single point, which has been translated by ee in Xs. <) Cor S and X (S are linearly separable (red line with orange margins), the transformation made obtain Xg. re preserves the linear separability of the data well the margins.
Figure 1:
Our main theoretical result is presented in Thm. | and Cor. 2 whose proof' are provided in the Appendix (see Appendix () along with additional details and further analysis (see Appendix D) These show that any set of internal neural representations X (e.g., from hidden layers of network) can be manipulated with simple transformations (translations of subset, see Fig. [a] to significantly
reduce the CKA between the original and manipulated set. We note that our theoretical results are entirely class and direction agnostic (except for Cor. which isn't direction agnostic), see the last paragraph of Sec. 4.2 for more details on this point.
Specifically, we consider a copy of X to which we apply transformation where the representations of a subset X of the data is moved distance с along direction J. resulting in the modified representation set Xs.t.e. A closed form solution is found for the limit of the linear CKA value between X and Xs. as tends to infinity. We note that up orthogonal transformations (which CKA is invariant to) the transformation X + Xs. 7. is not difficult to implement when transforming representations between hidden layers in neural networks. More importantly, it is also easy to climinate or ignore in a single layer transformation, as long as the weight vectors associated with the neurons in the subsequent layer are orthogonal 7. Therefore, our results show that from theoretical perspective, CKA can easily provide misleading information by capturing representation differences introduced by shift of the form X Xs, J.c (especially with high magnitude c), which would have no impact on network operations or their effective task-oriented data processing.
The terms in Eq. 2 can each be analyzed individually. (pp) depends entirely on p. the proportion of points in X v. that have not been translated i.e. that are exactly at the same place as in X. Its value between 0 and and tends towards 0 for small sizes of S. The participation ratio, with values in [1. pl. is used as an effective dimensionality estimate for internal representations (Mingzhou Ding Dennis Glanzman 2011 Mazzucato et al.) 2016 Litwin-Kumar al.. 2017). .. has long been observed that the effective dimensionality of internal representations in neural networks is far smaller than the actual number of dimensions of the representation space (Farrell et al. 2019 Horoi et al.. 2020). Exex [[|=]]]]] and [|Ezes(a] are respectively the average squared norms of all representations in and the squared norm of the mean of S. the subset of representations that are not being translated. Since most neural networks are trained using weight decay, the network parameters, and hence the resulting representations as well as these two quantities are biased towards small values in practice.
CKA sensitivity to outliers As mentioned in Sec. 2 it was recently found that the block structure in CKA heatmaps of high capacity models was caused by a few dominant data points that share similar characteristics (Nguyen et al., (022). Other work has empirically highlighted CKA's sensitivity to directions of high variance, failing to detect important, function altering changes that occur in all but the top principal components (Ding et al. 202T). Cor. 3 provides a concrete explanation to these phenomena by treating the special case of Thm. where only a single point, è is moved and thus has different position in Xs, v.e with respect to X, see Fig. [ for an illustration. We note that the term "subset translation" was coined by us and wasn't 't used in past work. However, all the papers referenced in this paragraph and later in this section present naturally occurring examples of subset translations in a set of representations relative to another, comparable set.
Corollary 3. Thm. holds in the special case where S = { } is single point, i.e. an outlier.
Cor. 3 exposes a key sensitivity of linear CKA, namely its sensitivity to outliers. Consider two sets of representations that are identical in all aspects except for the fact that one of them contains an outlier, i.e. representation further away from the others. Cor. then states that as the difference
4

---

Published as a conference paper at ICLR 2023
between the outlier's position in the two sets of representations becomes large the CKA value between the two sets drops dramatically, indicating high dissimilarity. Indeed, as previously noted, [[aes[=]]| [ (X) will be of relatively small value in practice so the whole expression in Eq.
2 will be dominated by T(p) --p. In the outlier case ((p) ~ p== # reprosentations which will be extremely small since for most modern deep learning datasets the number of examples in both the training and test sets is in the tens of thousands or more. This will drastically lower the CKA value between the two considered representations despite their obvious similarity.
CKA sensitivity to transformations preserving linear separability Classical machine learning theory highlights the importance of data separability and of margin size for predictive models generalization (Lee et al., 1995; Bartlett & ThylorTaylor 1999). Large margins, i.e. regions surrounding the separating hyperplane containing no data points, are associated with less overfitting, better generalization and greater robustness to outliers and to noise. The same concepts naturally arise in the study of ANNs with past work establishing that internal representations become almost perfectly linearly separable by the network's 's last layer (Zeiler & Fergus, 2014a Oyallon 2017] Jacobsen et al., 2018 (Belilossky et al. 2019 Davari & |
Furthermore, the quality of the separability, the margin size and the decision boundary smoothness have all been linked to generalization in neural networks (Verma et al. 2019). Given the theoretical and practical importance of these concepts and their natural prevalence in deep learning models it is reasonable assume that meaningful way in which two sets of representations can be "similar" is if they are linearly separable by the same hyperplanes in representation space and if their margins are equally as large. This would suggest that the exact same linear classifier could accurately classify both sets of representations. Cor. 4 treats this exact scenario as a special case of Thm. [ see Fig. [c for an illustration. If X contains two linear separable subsets, S and X' (S, we can create v. by translating one of the subsets in a direction that preserves the linear separability of the representations and the size of the margins while simultaneously decreasing the CKA between the original and the transformed representations, counterintuitively indicating a low similarity between representations.
Corollary 4. Assume S and X are linearly separably i.e. 3w E R,, the separating hyperplane's 's normal vector, and and x E X\S > (w,,x) > k. We can then pick of such that S and {x + + : x E X) are linearly separable by the exact same hyperplane and with the exact same margins as S and X\5 for any value of c E Rgo and Thm. sstill holds.
such that for every representation x E X we have: x E S = (wr, x) S
Extensions to nonlinear CKA As previously noted in Sec. 2 given the popularity of linear CKA, it is outside the scope of our work to theoretically analyze nonlinear kernel CKA. However one can consider extending our theoretical results to the nonlinear CKA case with symmetric, positive definite kernels. Indeed know from reproducing kernel Hilbert space (RKHS) theory that we can write such a kernel as an inner product in an implicit Hilbert space. While directly translating points in the representations space would likely not drive CKA values down as in the linear case, would suffice to find/learn which transformations in representations space correspond to translations in the implicit Hilbert space. Our results should hold if we apply the found transformations, instead of translations. to subset of the representations. Although practically harder to implement than simple translations, we hypothesize that it would be possible to learn such transformations.
EXPERIMENTS AND RESULTS
In this section we will demonstrate several counterintuitive results, which illustrate cases where similarity measured by CKA is inconsistent with expectations (Sec44. or can be manipulated without noticeably changing the functional behaviour of model (Sec. [.2 and 4.3). We emphasize that only Sec. 4.2 is directly tied to the theoretical results presented in the Sec. On the other hand, sections and discuss empirical results which are not explicitly related to the theoretical analysis.
4.1
CKA EARLY LAYER RESULTS
CKA values are often treated as a surrogate metric to measure the usefulness and similarity of a network's 's learned features when compared to another network (Ramasesh et al. (022). In order to analyze this common assumption, we compare the features of: (1) a network trained to generalize on the CIFARIO image classification task (Krizhevsky et al. 2009), (2) a network trained to "memorize" the CIFAR10 images (i.e. target labels are random), and (3) an untrained randomly initialized network (for network architecture and training details see the Appendix). As show in Fig, 2 early layers of these networks should have very similar representations given the high CKA values. Under the previously presented assumption, one should therefore conclude that the learned features at these
5

---

Published as a conference paper at ICLR 2023
1.4
CRA Dipen
0.8
8
0.4
0.2
Layer2
Gewratied Mereer and
0.3
Layor
Figure 2: A layer-wise comparison based on the value of the CKA between a generalized, memorized, and randomly populated network. This comparison reveals that early layers of these networks achieve relatively high CKA values. Mean and standard deviation across 5 soeds are shown.
Figure 3: The convolution filters within the first two layers of a generalized, memorized, and a randomly initialized network. Each of 16 squares corresponds to filter. Layer 1 filter colors correspond to RGB, Layer for each filter we randomly samples channels for visualization. This elucidates that the features are (1) drastically different, and (2) not equally useful despite the CKA results in Fig.
layers are relatively similar and equally valuable. However this is not the case, we can see in Fig. that the convolution filters are drastically different across the three networks. Moreover, Fig. elucidates that considerably high CKA similarity values for early layers, does not necessarily translate to more useful, similar, captured features. In Fig. [f of the Appendix we use linear classifying probes to quantify the "usefulness" of learned features and those results further support this claim.
4.2 PRACTICAL IMPLICATIONS OF THEORETICAL RESULTS
Here we empirically test the behaviour of linear and RBF CKA in situations inspired by our theoretical analysis, first in an artificial setting, then in a more realistic one. We begin with artificially generated representations X E IR Rx to which we apply subset translations to obtain E IR" xxd similar to what is described in Thm. We generate X by sampling 10K points uniformly from the IK-dimensional unit cube centered at the origin and 10K points from a similar cube centered at (1.1, 0, 0, ........)). O), so the points from the two cubes are linearly separable along the first dimension. We translate the representations from the second cube in a random direction sampled from the d-dimensional ball and we plot the CKA values between X and Y function of the translation distance in Fig. 4a. This transformation entirely preserves the topological structure of the representations as well as their local geometry since the points sampled from each cube have not moved with respect to the other points sampled from the same cube and the two cubes are still separated, only the distance between them has been changed. Despite these multiple notions of "similarity" between X and Y being preserved, the CKA values quickly drop below 0.2 for both linear and RBF CKA. While our theoretical results (Thm. [ predicted this drop for linear CKA, it seems that RBF CKA is also highly sensitive to translations of a subset of the representations. Furthermore, it is surprising to see that the drop CKA value occurs even for relatively small translation distances. We note that RBF CKA with equal to 0.2 times the median distance between examples is unperturbed by the considered transformation. However, as we observe ourselves (RBF CKA experiments in the supplement) and as was found in the original CKA paper (see Table 2 of Kornblith et al.) (219)), RBF CKA with a = 0.2 x median is significantly less informative than RBF CKA with higher values of a. With small values of a, RBF CKA only captures very local, possibly trivial, relationships.
Figure 4: a) Linear and RBF CKA values be- tween the artificial representations X and the subset translated version Y as function of the translation distance. b) CKA value between CNN's s internal representations the CIFAR10 training set and modified versions where either class or single point is translated as func- tions of the translation distance. The translation distances are normalized by the average norm of the unmodified representations, Fzex x [| I]| In a more realistic setting we test the practical implications of linear CKA sensitivity to outliers (see Cor. and to transformations that preserve the linear separability of the data well as the margins (see Cor. We consider the 9 layers CNN presented in Sec. 6.1 Komblith et al. (2019) trained on CIFAR10. As argued in Sec. 2 when trained on classification tasks, ANNs tend to learn increasingly complex representations of the input data that can be almost perfectly linearly separated into classes by the last layer of the network. Therefore meaningful way in which two sets of representations can
a) Subset transtation sensitivity tast points sampled from two
Transtation sonsttvity test (IFAR10 representations
10.
CBA volue
Linear
0.8
Where transiation (averiged ac accoss all classest
0..
0.4
RBF ow 3.5*mnnian
RBF ow med an
RBF - 5.2 mmadknn
Lin. CEA
044
Single pate: transaation
0.0
25
50
100
200
Translation distance normalized by [x
Translation distance
normalized by Evexlki1

---

Published as a conference paper at ICLR 2023
Unear Kornel
Origndd Nop
Nortrsang OKA
Marmunng GKA trdeeoon
Mirentong DoA sen potwoon
RBF Karnel CKA
Acc 85 9/a
18 a laveli acs as Layen: als $5.0%
AlL Abc at
Figure 5: Original Map is the CKA map of network trained on CIFARIO. We manipulate this network to produce CKA maps which: (1) maximizes the CKA similarity between the 1 and last layer, (2) maximizes the CKA similarity between all layers, and (3) minimizes the CKA similarity between all layers. In cases (1) and (2), the network experiences only slight loss in performance, which counters previous findings by achieving strong CKA similarity between early and late layers. find similar results are easily achieved in kernel
CKA case.
Target CKA Maps
Original Network CKA Map Acc. 65.9%
Christmas Tree
Goblin
Carrot
Bow Arrow
Sword
Network CKA, Maps
Acc. 85.5%
Acc. 85 3%
Acc. 85.7%
Acc. 85.6%
Acc. 85.7%
Layer
Layer
Luyar
Figure 6: The comical target CKA maps (first row) are used as the objective for the CKA map loss in Eq. while prioritizing network performance (small tolerance for changes in accuracy dacc). The second row shows the CKA map produced by the network.
be "similar" in practice is if they are linearly separable by the same hyperplanes in parameter space, with the same margins. Given X, the network's 's internal representations of 10k training images at SVM classifier to extract the hyperplanes in parameter
the last layer before the output space which best separate the data (with approx. 91 % success rate). We then create Y by translating subset of the representations in direction which won't I cross these hyperplanes, and won't affect the linear separability of the representations. We plot the CKA values between X and Y according the translation distance in Fig. 4p. The CKA values quickly drop to despite the existence of linear classifier that can classify both sets of representations into the correct classes with > 90% accuracy. Fig. 44p we also examine linear CKA's sensitivity outliers. Plotted are the CKA values between the set of training image representations and the same representations but with a single point being translated from its original location. While the translation distance needed to achieve low CKA values is relatively high, the fact that the position of a single point out of tens of thousands drastically influence the CKA value raises doubts about CKA's reliability as a similarity metric.
We note that our main theoretical results, namely Th. I Cor. and Cor. are entirely class and direction agnostic. The empirical results presented in this section are simply examples that we deemed particularly important in the context of ML but the same results would hold with any subset of the representations and translation direction, even randomly chosen ones. This important since the application of CKA is not restricted to be used in unsupervised learning settings (Grigg et al. 202T). Furthermore, the subset translations presented here were added manually to be able to run the experiments transformations can naturally occur in ANNs, as discussed in Sec. 3 and one would not necessarily know that they have occurred. We also run experiments evaluate CKA sensitivity to invertible linear transformations, see the Appendix for justification and results.
where labels are available, for example it can also
controlled fashion but these
4.3 EXPLICITLY OPTIMIZING THE CKA MAP
The CKA map, commonly used to analyze network architectures (Kornblith et al. 2019) and their inter-layers similarities, is a matrix M, where j] is the CKA value between the activations of layers i, and j of network. In many works (Nguyen al. 2021 Raghu et al. 2021 Nguyen et al.

---

Published as a conference paper at ICLR 2023
Original al OKA Maps Zon 93.7%s
Terget CKA Map Witihin
Acc. 93.80%
fon 93.7%
Kaps optimized wat Target CKA Map (vidti x1]
Acc. 93.3%
Acc 94.1%
Acc 93.2%
Figure 7: ResNet-34 networks of different widths and their corresponding CKA Maps are modified to produce CKA maps of thin networks. Top row Original shows the unaltered CKA map of the networks derived from "normal" training. Optimized shows the CKA A map of the networks after their map is optimized mimic the thin net targer CKA map. See Appendix for more networks.
FCA: Representations of The Original Network
FCA: Represontations of The punaptlated Network
100
20
20
-20
10
21
35
40
-52
50
310
150
Figure 8: PCA of the networks presented in Fig. 5 before (left) and after (right) being optimized to manipulate the CKA map with Eq Noticeably to achieve the objective the optimization displaces a subset a single class.
2022) these maps used explicitly to obtain insights how different models behave, compared to one another. However, as seen in our analysis so far it is possible to manipulate the CKA similarity value, decreasing and increasing it without changing the behaviour of the model In this section we set to directly manipulate the CKA map of a trained network so-, by adding the desired CKA map, Mearget. its optimization objective, while maintaining its original outputs via distillation loss (Hinton et al. (015). The goal is to determine if the CKA map can be changed while keeping the model performance the same, suggesting the behaviour of the network can be maintained while changing the CKA measurements. To accomplish this we optimize fo over the training set (X, Y) (note however that the results are shown on the test set) via the following objective:
target task.
8% = argming (Eaistau (fe- (X), + ALmap (Mo((X), Mtarget))
(3)
where Smop (Mar)); Moarget) = Ei, In cosh [Mii)lac - Mii, (targee) The À multiplier in is the weight that balances the two losses. Making X large will favour the agreements between the target and network CKA maps over preservation of the network outputs. In our experiments À allowed to change dynamically at every optimization step. Using the validation set accuracy as a surrogate metric for how well the network's representations are preserved, À is then modulated to learn maps. If the difference between the original accuracy of the network and the current validation accuracy is above certain threshold (face) we scale down À to emphasize the alignment of the network output with the outputs of fo-, otherwise we scale it up to encourage finer agreement between the target and network CKA maps (see Appendix for the pseudo code).
Fig. 5 shows the CKA map of fe- along with the CKA map of three scenarios we investigated: (I) maximizing the CKA similarity between the 1st and last layer, (2) maximizing the CKA similarity between all layers, and (3) minimizing the CKA similarity between all layers (for network archi- tecture and training details see Appendix). In cases (1) and (2), the network performance is barely hindered by the manipulations of its CKA map. This is surprising and contradictory to the previous findings (Kornblith al., 2019 Raghu et al. (02T) as it suggests that is possible to achieve a strong CKA similarity between early and later layers of well-trained network without noticeably changing the model behaviour. Similarly we observe that for the RBF kernel based CKA (Kornblith et al. 2019) we can obtain manipulated results using the same procedure. The bandwidth for the RBF kernel CKA is set to 0.8 of the median Euclidean distance between representations (Komblith et al. 2019). In the Appendix
also show similar analysis on other values. We further experiment with manipulating the CKA map of fo- to produce a series of comical CKA maps (Fig. (6) while maintaining similar model accuracy. Although the network CKA maps seen in Fig. oclooely resemble
8

---

Published as a conference paper at ICLR 2023
their respective targets, it should be noted that we prioritized maintaining the network outputs, and ultimately its accuracy by choosing small Sace- Higher thresholds of accuracy result in stronger agreements between the target and network CKA maps at the cost of performance.
Wider networks Nguyen et al. (202T) and Nguyen et al. (2022) studied the behaviour of wider and deeper networks using CKA maps, obtaining a block structure, which was subsequently used to obtain insights. We revisit these results and investigate whether the CKA map corresponding to a wider network can be mapped to a thin network. Our results for the SIFAR10 dataset and ResNet-34 (He) cet al. 2016) are shown in Fig. (for details on the architecture and training procedure see Appendix). We observe that the specific structures associated with wider network can be completely removed and the map can be nearly identical to the thinner model without changing the performance substantially. Results on OOD datasets and with vision transformers yield similar results (appendix Fig. [0 14]
Analysis of Modified Representations Our focus See 4.3 has been to use optimization to achieve a desired target CKA manipulations without any explicit specification of how to perform this manipulation. We perform an analysis to obtain insights on how representations are changed by optimizing Eq 3 in Fig. 8 Here using the modified network from case (2) of Fig. 5 we compute the PCA of the test set's 's last hidden representation (whose CKA compared to the first layer is increased). We observe that a single class has a very noticeable set of points that are translated in particular direction, away from the general set of classes. This mechanism of manipulating the CKA aligns with our theoretical analysis. We emphasize, that in this case it is completely emergent behavior.
5 DISCUSSION AND CONCLUSION
Given the recent popularity of linear CKA as a similarity measure in deep learning, it is essential to better study its reliability and sensitivity. In this work we have presented formal characterization of CKA's sensitivity to translations of subset of the representations, simple yet large class of transformations that is highly meaningful in the context of deep learning. This characterization has provided theoretical explanation to phenomena observed in practice, namely CKA sensitivity to outliers and to directions of high variance. Our theoretical analysis also show how the CKA value between two sets of representations can diminish even if they share local structure and are linearly separable by the same hyperplanes, with the same margins. This meaningful way in which two sets of representations can be similar, as justified by classical machine learning theory and seminal deep learning results, is therefore not captured by linear CKA. We further empirically showed that CKA attributes low similarity transformations that preserve important functional characteristics and it attributes high similarity to representations from unalike networks. Furthermore, we showed that we can manipulate CKA in networks to result in arbitrarily low/high values while preserving functional behaviour, which we use to revisit previous findings (Nguyen et al., 202T: Kornblith et al., 2019).
sets of representations that are directly linked by simple affine
It is not our intention to cast doubts over linear CKA as representation similarity measure in deep learning, or over recent work using this method. Indeed, some of the problematic transformations we identify are not necessarily encountered in many applications. However, given the popularity of this method and the exclusive way it has been applied to compare representations in recent years, we believe it is necessary better understand its sensitivities and the ways in which it can be manipulated. Our results call for caution when leveraging linear CKA, as well as other representations similarity measures, and especially when the procedure used to produce the model is not known, consistent, or controlled. An example of such a scenario is the increasingly popular use of open-sourced pre-trained models. Such measures are trying to condense a large amount of rich geometrical information from many high-dimensional representations into a single scalar E [0, 1J. Significant work is still required to understand these methods in the context of deep learning representation comparison, what notion of similarity each of them concretely measures, to what transformations each of them are sensitive or invariant and how this all relates to the functional behaviour of the models being analyzed.
In the meantime, as an alternative to using solely linear CKA, we deem it prudent to utilize multiple similarity measures when comparing neural representations and to try relating the results to the specific notion of "similarity" each measure is trying to quantify. Comparison methods that are straightforward to interpret or which are linked to well understood and simple theoretical properties such as solving the orthogonal Procrustes problem (Ding et al.] 2021 Williams et al.) 2021 or comparing the sparsity of neural activations (Kornblih et al. 2021) can be a powerful addition to any similarity analysis. Further, data visualizations methods potentially help to better understand the structure of neural representations in certain scenarios (e.g. Nguyen et al. (2019); Gigante et al. (2019); Horoi et al. (2020); |Recanatesi et al. (021))

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work was partially funded by OpenPhilanthropy [M.D., E.B.J: NSERC CGS D. FRONT BIX & UdeM A scholarships [S.H.]; NSERC Discovery Grant RGPIN-2018-088 0-88-1881 & Samsung Research Support [G.L.]; NSERC Discovery Grant 03267 J.W and Canada CIFAR AI Chairs [G.L., G. G.W.J. This work also supported by resources from Compute Canada and Calcul Quebec. The content solely the responsibility of the authors and does not necessarily represent the views of the funding agencies.
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2016. URL
wttp:///ggaa/./....o.ooo 01644
Sercan Ö Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 6679-6687, 2021.
Sanjeev Arora,
Tengyu Ma. A simple but tough-to-beal baseline for sentence embeddings. In Sth International Conference on Learning Representations, ICLR Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net,
Yingyu Liang, and
2017,
2017. [RLL:ttts://ppeeerreee hwt.://w.eeeeeeeeeeeeeeeeeeeeeee wtt.://eeet.ttooooooooooo.......
Peter Bartlett and John Shawe-Taylor. Generalization performance of support vector machines and other pattern classifiers. Advances in Kernel methods suppoor
support vector learning, pp. 43-54, 1999.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale ImageNet. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 583-593. PMLR, 09-15 Jun 2019. URL httpp:://proeees lovskyl 9a htm
m r. wwwpssssv7//.//
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
MohammadReza Davari and Eugene Belilovsky. Probing representation forgetting in continual learning. In NeurlPS 202/ Workshop Distribution Shifts: Connecting Methods and Applications, 2021.
MohammadReza Davari, Leila Kosseim, and Tien Bui. TIMBERT: Toponym identifier for the medical domain based on BERT. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 662-668, Barcelona, Spain (Online), December 2020. Interna- tional Committee on Computational Linguistics. doi: 10. 1/2863//1/0222.ooingg 0o00200.ooin-mman.55 58. URL https://aclantho.oee org. coli:iin000.c. mttemaain555
MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky, Probing representation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16712-16721, June 2022.
Jacob Devlin, Ming-Wer Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Frances Ding, Stanittassll Denain, and Jacob Steinhardt. Grounding representation similarity through statistical testing. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL htttps://opeereet Eort/Eorum?id= 16V53229
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
10

---

Published as a conference paper at ICLR 2023
Shimon Edelman. Representation is representation of similarities. Behavioral and Brain Sciences, 21 (4):449-461 1998. doi: 10.017/50:445533000022222
Farhood Farahnak, Elham Mohammadi, MohammadReza Davari, and Leila Kosseim. Semantic simi- larity matching using contextualized representations. In Canadian Conference AI, Vancouver, Canada (Online), June 2021.
Matthew Farrell, Stefano Recanatesi, Timothy Moore, Guillaume Lajoie, and Eric Shea-Brown Recurrent neural networks learn robust representations by dynamically balancing compression and expansion. bioRxiv, 2019, doi: 10. 10.1646/5 URL TRLhttps::// https://wwwwww. wttp:www..oo..... www.w/wwww.voorrggg////////////m
[ootten/eearly22011
8 564476
Scott Gigante, Adam $ Charles, Smita Krishnaswamy, and Gal Mishne. Visualizing the phate of neural networks. Advances in neural information processing systems, 32, 2019.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilhert-schmidt norms. In Lecture Notes Computer Science, pp. 63-77. Springer Berlin Heidelberg, 2005. doi: 10.1007/1 1tt11/11640999111 URL https::/aoioorg//11
1007/11564089 7
Tom George Grigg, Dan Busbridge, Jason Ramapuram, and Russ Webb. Do self-supervised and supervised methods learn similar visual representations?, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness common corruptions and surface variations. arXiv preprint 1rXivv1807.01697 2018.
Geoffrey Hinton, Oriol Vinyals. Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0255 02531, 2(7), 2015.
Stefan Horoi, Victor Geadah, Guy Wolf. and Guillaume Lajoie. Low-dimensional dynamics of encoding and learning in recurrent neural networks. In Cyril Goutte and Xiaodan Zhu (eds.), Advances in Artificial Intelligence, pp. 276-282, Cham, 2020. Springer International Publishing. ISBN 778-3-000-47355-7
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer:: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.0667,, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. PMLR, 2015.
Jöm--enrik Jacobsen, Arnold W.M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In International Conference on Learning Representations, 2018. URL https::/
net 21d
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead less transferable features? In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://
Nikolaus Kriegeskorte, Maricke Mur, and Peter Bandettini. Representational similarity analysis connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2, 2008. ISSN 1662-5137. doi: 00338//neuro.006004220000
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, MIT & 2009.
11

---

Published as a conference paper at ICLR 2023
Aarre Laakso and Garrison Cottrell. Content and cluster analysis: Assessing representational similarity in neural systems. Philosophical Psychology, 13(1):47-76, 2000. doi: 10.
09515080050002226. /tt.pt/ts//aaa..oo/////////////0
Wee Sun Lee, Peter L. Bartlett, and Robert C. Williamson. Lower Bounds on the VC Dimension of Smoothly Parameterized Function Classes. Neural Compulation, 7(5):1440-1053, 09 1995. ISSN 7669.7667. doi: 10. 10.1162/nec..1955.7 1.5. Utthttss:: //doi wttp//...//........
1995. 7, 1040
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. International Journal of Computer Vision, 127(5))456 476, May 2018. doi: 10. 1007/s 1263-/11-1------888 URL https: doi org/ 10. 1007 101101001///0000000 1098
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Dmitry Storcheus, Afshin Rostamizadeh, and Sanjiy Kumar (eds.), Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges NIPS 2015, volume 44 of Proceedings of Machine Learning Research, pp. 196-212, Montreal, Canada, 11 Dec 2015. PMLR. URL https //proceedings.nut mir press //t/:///////11SSceeeeeee html
Ashok Litwin-Kumar, Kameron Decker Harris, Richard Axel, Haim Sompolinsky, and L.F. Abbott. Optimal degrees of synaptic connectivity. Neuron, 93(5): 153311113-1164444 2017. ISSN 0896-6273. doi: https://o..nng///....o.onnnoono. com/ http://ccn////tteeeee/////////30
httppp///wwwwws.ceeeeeeee......e URL hrttps://wwe sci encedirect.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv
preprint /608. 039983, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:1711.05100 2017.
Isabel I.C. Low, Alex H. Williams, Malcolm G. Campbell, Scott Linderman, and Lisa M. Gio- como. Dynamic and reversible remapping of network representations in an unchanging environ-
ment. Neuron,
2021. ISSN 0896-6273, doi: http:///oi.org//000016// neuron.2021.07.0000 URL https:/ www. www.scccieeeeiieett.t... http://tttttcccccieeeeceeeeeeeee wwtom/science/artiii 011/80896627321005000
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations of recurrent net- works. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Asso- ciates, Inc., 2019, URL hittps://pel htttt::://proceednggs... nwwrips http://eeeaas..eeerriooo.o/////p /219://///////ee//// Paper.
Luca Mazzucato, Alfredo Fontanini, and Giancario La Camera. Stimuli reduce the dimensionality of cortical activity. Frontiers in Systems Neuroscience, 10, 2016. ISSN 1662-5137. doi: 10. 3389/fnsys.200 2016...000100000000 httpstp::/ /ttps::/wwww.. www.::ooooorrri..oo/oo..........
fnsys. 2016 00011
PhD Mingzhou Ding and PhD Dennis Glanzman. The Dynamic Brain. Oxford University Press, January 2011. doi: 10. 00.0033/hmmmfoo00/77799999999999 .0011. URL httttttss///oi..ot :ww.::ofoooo//..o
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman,
N. Cesa-Bianche and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL wRt.atsss://oooeeeeeeeeeeeeeeeee wew.eeiss..iiii.....see......... www.iieerppp..... cC/
12

---

Published as a conference paper at ICLR 2023
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Ieml, 2010.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in trans- fer learning? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 512-523. Curran Asso- ciates, Inc., 2020. URL httpp:://proceeddingg meer http:///www........o.o////ee//ee httpp//ppp////////LLL... DD97440DSSSSSLLaafff422222-----2
Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding Neural Networks via Feature Visu- alization: A Survey, pp. 55-76. Springer International Publishing, Cham, 2019. ISBN 978-
3-030-28954-6. doi:
URLhtttss///o..
978-3-030-28554-2
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference Learning Representations, 2021. URL/ttps::/
1d=KJNcAKY8LY4
Thao Nguyen, Maithra Raghu, and Simon Kornblith. On the origins of the block structure phe- nomenon in neural network representations. arXiv preprint w2Xi:2200.007884 2022.
Edouard Oyallon. Building a regular decision boundary with deep networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1896,1894, 2017, doi: 10.1 10.1109/CVPR.
2017.204.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singu- lar vector canonical correlation analysis for deep learning dynamics and interpretability. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: http:::s//ppooceedinne dccaa7e655577/33446666616
neurips http://wss.nniiiscncececeeeeeeee http://cccccee.eeeeeeee.c///r/// Paper.pppppppp
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34, 2021.
Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference on Learning Representa- tions, 2021. URL htt.tps::/ppeerreiiiieeee
Stefano Recanatesi, Matthew Farrell, Guillaume Lajoie, Sophie Deneve, Mattia Rigotti, and Eric Shea- Brown. Predictive learning as a network mechanism for extracting low-dimensional latent space representations. Nature Communications, 12(1), March 2021. doi: 10. 10.1008//114467.0222166666 URL 1tttttpp://ooii.og/1000....000.0 11t.:///1..111//111111..11111110 02 -21-21696-1 696-1
Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, 13(5), 2012.
David Sussillo and Omri Barak. Opening the Black Box: Low-Dimensional Dynamics in High- Dimensional Recurrent Neural Networks. Neural Computation, 25(3)6666-499, 03 2013. ISSN 0899-7667. doi: 10. 10.1162/NECO Lac00____00000 a_00409. URL httpp:://oi..rrg:: 111.////0..12///C.C 00409
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivari- ant CNNs for digital pathology. arXiv preprint arXiv: 1806 03962, June 2018.
13

---

Published as a conference paper at ICLR 2023
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Pas and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6438 6447. PMLR, 09-15 Jun 2019. httphhtts://ppr.ee.t. vermal9a.do
ngs ml press/v971
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hoperoft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, Curran As- sociates, Inc., 2018. URL http:ttttt/ppcccediniii wttp:/pp.....pppee////00.0.0 fille// 3
Alex Williams, Erin Kunz, Simon Kornblith, and Scott Linderman. Generalized shape metrics on neural representations. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL pet/forumm 2id pxQ01
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. Com-
puter Vision - ECCV 2014, pp. 318-833. Springer International Publishing, 2014a. doi: 10. 1007/ URL http:://6bb..rrr/1110000///////0
978-3-319- 10590-111100-1111
10590-
53
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014b.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.1 2021.
14

---

Published as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS
A.1 MINIBATCH CKA
In our experiments (with the exception of subsection 4.2 in order reduce memory consumption, we use the minibatch implementation of the CKA similarity Nguyen et al. (021 2022). More precisely, let be a kernel function (we experiment with linear and RBF kernel,, and X E mm*". and Y, E Ruxxa be the minibatches of m samples from two network layers containing ni and n2 neurons respectively. We estimate the value of CKA by averaging the Hilbert-Schmine independence criterion (HSIC), over all minibatches b E B via:
B ELEB HSIC1 (Qb. Zb)
CKAminibatch
[ EBEB HSIC, (Qu, Qo) [8] EBEB HSIC; (Zs, Zb)
Where Qo (X,,,, Z, VYY,, Y)), and HSIC; is the unbiased estimator of HSIC Song et al. (2012), hence the value of the CKA is independent of the batch size.
A.2 NETWORK ARCHITECTURE
In Sec. 4.1
9 layer neural network; the first 8 of these layers convolution layers and the last layer is a fully connected layer used for classification. We use ReLU (Nair & Hinton 2010) throughout the network. The kernel size of every convolution layer is set to (3,3) except the first two convolution layers, which have (7, 7) kernels. All convolution layers follow a padding of
and a stride of 1. Number of kernels each layer of the network, from the lower layers onward
follows: [16, 16, 32, 32, 32, 64, 64). In this network, every convolution layer is followed by batch normalization (loffe & Szegedy 2013). The network we used in Sec. 4.3 to obtain Figures 5 and 6 is similar to the network we just described, except the kernel size for all layers are set to (3, 3). For the
Wider networks experiments Sec. 4.33 we use a ResNet-34 (He al.) (0166) network, where we scale up the channels of the network to increase the width of the network (see Fig. 7).
A.3 TRAINING DETAILS
The models Sec. 4.1 both the generalized and memorized network, were trained for 100 epochs using AdamW W (Loshchilov & Hutter 2017) optimizer with learning rate (LR) of le-3 3 and weight decay of 54-4. The LR is follows cosine LR scheduler (Loshchilov & Hutter 2016 with an initial LR stated earlier.
The training of the base model (original) model in Sec. 4.3 seen in Figures 5 and follows the same training procedure of the models from Sec. except in this setting we train the model for 200 epochs, with an initial LR of 0.01. All other models in Sec. seen in Figures and 6 (with a target CKA map optimize) are also trained with similar training hyperparameters to that of the base model, except the followings: (1) these models are only trained for 30 epochs. (2) the objective function includes hyperparameter À (see Eq. 3q. which we initially set 500 for all models and is changed dynamically following the Algo. during the training by 0.8 on each iteration. (3) The cosine LR scheduler includes warm-up step of 500 optimization steps. (4) the LR is set 1e-3 (4) The distillation loss in the objective function depends on temperature parameter, which we set to 0.2.
The training procedure for the Wider networks experiments in Sec. 4.3 is similar to the previous training procedures in this section (Figures Except that the Original models are trained for only 100 epochs and the Optimized w.r.t Targel Maps models are trained for 15 epochs.
A.4 CKA MAP LOSS BALANCE
Algo. shows the pseudo code the dynamical scaling of the À loss balance parameter seen in Eq. Using the validation set accuracy surrogate metric for how well the network's 's representations are preserved, is then modulated to learn maps. If the difference between the original accuracy of the network and the current validation accuracy is above certain threshold (6ace) we scale down À to emphasize the alignment of the network output with the outputs of fo-, otherwise we scale it up to encourage finer agreement between the target and network CKA maps.
15

---

Published as a conference paper at ICLR 2023
Algorithm 1: Dynamical balancing of Distillation and CKA map loss in Eq.
Data: 100 2 Original Accuracy > 0; 100 2 Current. Validation Accuracy > 0;
Accuracy Threshold 2 0; 2 Scaling Factor >> 0; Initial Lambda > 0
Result: X
acco + Original Accuracy:
acc1 + Current Validation Accuracy;
7 Accuracy Threshold;
a < Scaling Factor;
Initial Lambda;
Aer - ac0n acc1;
if 6 acc - À x a
then
else
+ 1/2
end
Return X;
B ADDITIONAL RESULTS
B.1 CKA SENSITIVITY TO INVERTIBLE LINEAR TRANSFORMATIONS
We also experiment, for linear CKA, with type of trans- formation that isn't considered by our theoretical results but which deemed interesting to analyze empirically, namely multiplications by invertible matrices. Consider matrix M IR Rdd whose elements are sampled from We
Points from multivariate gaussian invertible linear transformation sensitivity test
10
Mean of Minatrix elements (all
08
10
Gaussian with mean μ and standard deviation M since it is not guaranteed and
06
verify the invertibility only keep invertible matrices. We show the CKA values between X and the transformed Y X M Fig. 9 Since this is an invertible linear transformation we would expect it to only modestly change the representations in X and the CKA value to be only slightly lower than I. However, we observe that even for small values of and CKA drops to O, which would indicate that the two sets of representations are dissimilar and not linked by a simple, invertible transformation.
50
100
500
1e3
1e3 Standard deviation of M matrix elements
Figure 9: Linear CKA values between the artificial representations X and with M being am invertible matrix with ele- ments sampled from N N(J, o²) as function of μu and σ. The mean and standard deviation across 10 random instantiations the transla- tion direction and M are shown.
X.M
While Th.1 Kornblith et al. (2019) implies that invari- invertible linear transformations is generally not desirable property for ANN representation similarity mea- sures, there are relatively common scenarios in which the hypotheses of the theorem are not necessarily respected, i.e. width of the layer. Such is the case in smaller ANNs even the last layers of large models which are often fully connected and of far smaller size than the input space or the intermediate layers. Given these situations we see no reason completely dismiss this invariance as being possibly desirable in certain, albeit not all, contexts.
where the dataset size is larger than the
B.2 WIDER NETWORKS
In Sec. 4.3 we investigated whether the CKA of the wide networks can mapped to a thin network (see Fig. 7) using ResNet-34 models and the CIFARI0 10 dataset. In Fig. [0] use the same networks (trained on CIFAR1O) and measure their CKA similarity maps under the Patch Camelyon dataset (Veeling et al. 2018). Patch Camelyon dataset contains histopathologic scans of lymph node sections, which is drastically different from the CIFAR10 dataset both in terms of pixel distribution and the semanties of the data. As
in Fig. 10 even under this drastic shift in data distribution
16

---

Published as a conference paper at ICLR 2023
Original CKA Maps Trained on CIFAR10
Target CKA Map Width x1
Tested on Patch Camelyon Width x4
Trained on CIFAR10
Width x2
Width x5
Camelyon
Maps Optimized w.r.t Tärget CKA Map (width x1) CIFAR10
Tested on Patch Camelyon Width x4
Width x2
Width x5
Figure 10: Fig. we are presented a series of ResNet-34 networks of different widths and their corresponding CKA Maps, which are modified produce CKA maps of thin networks using the CIFARIO dataset. We used these networks and measured their CKA maps using Patch Camelyon dataset. Top Original shows the unaltered CKA map of the networks derived from "normal" training on I0, tested on Patch Camelyon dataset. Optimized shows the CKA map of the networks after their map is optimized mimic the thin network target CKA map using CIFAR 10, tested on Patch Camelyon dataset.
Original CKA Maps Width x8 Acc. 91.9%
Width x2 Acc. 88.6%
Width x4 Acc. 90.9%
Width x10 Acc. 92.3%
Target CKA Map Width x1, Acc. 85.9%
Maps optimized w.r.t Target CKA Map (width x1) Width Acc. 7%
Width x2 Acc. 88.4%
Width Acc. 91.5%
Width
Acc. 92 0%
Figure 11: VGG style networks of different widths and their corresponding CKA Maps are modified to produce CKA maps of thin networks. Top row Original shows the unaltered CKA map of the networks derived from "normal" training. Optimized shows the CKA map of the networks after their map is optimized to mimic the thin net target CKA map.
the CKA maps of the networks Optimized w.r. Target CKA Map resemble the CKA map the thin target network, suggesting the generalizability of the CKA map optimization.
The network architecture presented in the Wider networks experiments seen in Sec. 4.3 is ResNet-33 34, We experimented with VGG style network architecture to broaden findings to other network architectures (see Sec. A.2 for details). As we can see in Fig. II we observe similar results to the ones shown in Fig.
17

---

Published as a conference paper at ICLR 2023
Maximizing CKA Linear Kernel CKA
Original 85.9%
Maximizling CKA Layers: Acc. 85.5%
Minimiziing CKA Between
Last Layer: Acc. 85.4%.
ALL Layers: Acc. 85 45%
Layer Original Map Acc. 85.9%
RBF Kernel CKA x Median Distance Maximizing CKA Sim, Between ALL Layers: Acc. 85.4%
Maximizing CKA Sim. Between 1st Last Layer: Acc. 6%,
Minimizzing CKA Sim. Between ALL Layers: Acc. 85.3%
Layer
RBF Kernel CKA 0.4 x Median Distance Maximizing CKA Maximizing CKA Between Last Layer: Acc. 7%,
Original
Minimizing CKA Between ALL Layers: Acc. 81.
Layers: 85.3%
Layer
RBF Kernel CKA Maximizing CKA Sim. Be Belween 1st Last Layer: Acc. 85.5%,
x Median Distance Maximizing CKA Sim, Belween ALL Layers: Acc. 85.4%
Original Map Acc. 85.9%
Minimizing CKA Sim. Belween ALL Layers: Acc. 84.3%
Laver
Layer
Layer
Layer
Layer
Figure 12: Original Map is the CKA map of network trained on CIFARIO. We manipulate this network to produce CKA maps which: (1) maximizes the CKA similarity between the 1st and last layer, (2) maximizes the CKA similarity between all layers, and (3) minimizes the CKA similarity between all layers. In cases (I) and (2), the network experiences only slight loss in performance, which counters previous findings by achieving strong CKA similarity between early and late layers.
B.3 RBF KERNEL
In Fig. 12 extend results shown in Fig.5tto other bandwidth values commonly used for the RBF kernel CKA (Kornblith et al. 2019). When the CKA values are meaningful, we observe that the RBF kernel CKA values can be manipulated via the procedure described in Sec. 4.3
B.4 CKA MAP OPTIMIZATION VIA LOGISTIC LOss
In Sec. 4.3 manipulated network's s CKA map, while closely maintaining its outputs via the
distillation loss seen Eq. However, a logistic loss also works in this setting, i.e. the substitution of the distillation loss with cross-entropy loss in Eq. yields similar results. In Fig. [3] repeated the linear CKA experiments seen the first row of the Fig. 5 using cross-entropy loss instead of distillation loss.
18

---

Published as a conference paper at ICLR 2023
Original Map
Maximizing Similority Between Layer::
Maximizing Similarity Layers:
Minimizing Simiartty Layers:
54
24
12
64
Layer
Layer
Layer
Layer
Figure 13: Original Map is the CKA map of network trained CIFARIO. We manipulate this network following modified version of Fq. (distillation loss is substituted with cross-entropy loss) produce CKA maps which: (1) maximizes the CKA similarity between the 1 and last layer, (2) maximizes the CKA similarity between all layers, and (3) minimizes the CKA similarity between all layers. In cases (1) and (2), the network experiences only slight loss in performance, which counters previous findings by achieving a strong CKA similarity between early and late layers.
(
Original Map Acc. 68.795
Maximizing CKA Similerity Between 1st & Last Block: Acc. 88. 3%
Maximizing CKA Similarity Between ALL Blocks: Acc. 88.5%
Minimiding CKA Simiarity Between ALL Blocks: Acc. 85.2%
2.0
36
42
42
Tarslamer
Bladm Blo2k
Varofermet Blucks
Varsfura elaces
Tarafor Slocks
Figure 14: Original Map is the CKA map of a VIT Dosovitskiy et al. 2020 network trained on CIFARIO. We manipulate this network following the Eq. to produce CKA maps which: (I) maximizes the CKA similarity between the 1s and last Transformer block, (2) maximizes the
CKA similarity between all Transformer blocks, and (3) minimizes the CKA similarity between all Transformer blocks.
B.5 CKA OPTIMIZATION OF VIT
In Fig. 5 we manipulated the CKA map of a VGG style model trained on CIFARIO in order to: (1) maximize the CKA similarity between the 1st and last layer, (2) maximize the CKA similarity between all layers, and (3) minimize the CKA similarity between all layers.
We further explored this setting at the model architecture level. Given the recent popularity of the Transformer (Vaswani et al.) (017) architecture in a variety of domains such as NLP Devlin et al. (018); Farahnak et al. (2021) b Raffel al. 2020); Davari al. (2020), Computer Vision Dosovitskiy et al. 2020); Zhou et al. (202 1): Liu et al. (2021) I,, and Tabular Huang et al. (2020); Arik & Pfister (2021) we implemented Vision Transformer (ViT) (Dosovitskiy al. (020) style model for the CIFAR 10 dataset, containing 8 Transformer /Vaswani et al.) 2017) blocks (see other architectural details in Tab. I in order to: (1) maximize the CKA similarity between the 1st and last Transformer block, (2) maximize the CKA similarity between all Transformer blocks, and (3) minimize the CKA similarity between all Transformer blocks. As we can see in Fig. 14 these manipulations are achieved with minimal loss of performance, which underlines the model-agnostic nature of our approach.
Transformer Blocks
# Attention Heads
Hidden Size
Epochs
12
256
200
Table 1: Architectural details of out implementation of VIT (Dosovisskiy et al. dataset. Note that the training process (except the number of epochs, which is listed above) follows the Sect. A.3
2020 for the CIFARIO
19

---

Published as a conference paper at ICLR 2023
Seed Same
Seed Different
1.0
0.8
Experiment Generalized vs. Generalized
0.4
Generalized vs. Memorized Generalized vs. Random
0.2
Layer
Layer
Figure 15: layer wise comparison between generalized, memorized, and randomly populated network using either (left figure) the same random seed or (right figure) different random seeds. This comparison reveals that, either case (with same or different random seeds) early layers these networks achieve relatively high CKA values.
Linear Probe Accuracy on Early Layers
Linear Probe Accuracy
Model
Generalized
Memorized
Random
40
20
2
Layer
Figure 16: The linear probe accuracy obtained the CIFARI0 test set for the generalized, memorized, and randomly populated network seen in Fig. 2 shown in Fig. 15 suggests that high values of CKA similarity between networks does not necessarily translate to similarly useful fcatures.
and 3 The results shown in this figure along with the
B.6 CLOSER LOOK AT THE EARLY LAYERS
In this section, extend the study of the behaviour of CKA over the early layers presented in Fig. 2 and In Fig. 15 we can see a layer wise comparison between a generalized, memorized, and randomly populated network using either (Fig. [5tefft the same random seed or (Fig. right different random seeds. This comparison reveals that, in either case (with same or different random seeds) early layers of these networks achieve relatively high CKA values.
However, as it was shown in Fig. 3 high values of CKA similarity between two networks does not necessarily translate more useful, similar, captured features. In order to quantify the usefulness of the features captured by each network in Fig. 2 and 3 we follow the same methodology as used in Self-supervised Learning (Chen al. 2020 and in the analysis of intermediate representations (Zeiler & Fergus) 2014b). We evaluate the adequacy of representations by an optimal linear classifier using training data from the original task, this case the CIFARIO 10 training data. The test set accuracy obtained by the linear probe is used proxy measure the usefulness of the representations. Fig. 16 shows the linear probe accuracy obtained on the CIFAR10 test set for the generalized, memorized, and randomly populated network seen in Fig. [nd The results shown in this figure along with the ones shown Fig. 15 suggests that high values CKA similarity between two networks does not necessarily translate to similarly useful features.
20

---

Published as a conference paper at ICLR 2023
Effect CKA Manipulation on CIFAR1O-C Evaluation
Origial Network. levg. 66.2)
Betlownn Last Layer lauy. 65.7)
Moxtmising CSA Smilbriy
Maximizing CKA Simiarfty Belowan ALL Lapers (aep. 65.0)
Wninizingg CKA Simlianty Bellowen Layats (avg. 65.5)
80
ACC.
40
brightness
contrast
elastic transform
fog
frost
gaussian blur
gaussian noise
Imp ulpulse nolse
Jaag compression
motion blur
pixelate
saturate
shot nolse
snow
speckle noise
z00m blur
Spate
defocus
Nolse
Figure 17: Results of OOD evaluation of models from 17 on CIFAR-10C (Hendrycks & Dietterich 2018) corruptions data. We show results for each corruptions well the average results. We observe the performance remains largely unchanged.
B.7 OUT DISTRIBUTION EVALUATION
In Sec. 4.3 have showed that the CKA maps of models can be modified while maintaining equivalent accuracy for the performance on the test set. We can also consider how the functionality of these models varies under other kinds of inputs. We thus consider evaluating the models created in Sec. [4.3] in particular those corresponding to Fig. 5 on the popular CIFAR-10 Corruptions datasets (Hendrycks & Dietterich] 2018). Figure 17 shows the results of this evaluation schema for the 3 different models explored in Fig. We observe that overall the average performance of these models on this -ff--iirrriuution data remains effectively unchanged. We emphasize that the models
are trained only on uncorrupted data as in Sec. 4.3
21

---

Published as a conference paper at ICLR 2023
C PROOFS
Proof. Theorem 1
First we introduce the notation C₁ = S and C2 = X and note that C₁ and C2 form partition of i.e. X C₁ U2 with C1 n C₂ 0. We note Cýj the set of indices of Cj, meaning that ie + 2, C,. We then rewrite Xs, v.e as being the union of the of points in C1 and the points in C2 translated by in direction dt:
x = (f: :
It is standard practice to center the two sets of representations being compared before using representation similarity measures. is centered by hypothesis but Xs, s, is not. We first note that the mean of Xs,v,e across all representations N, is the vector:
Ng,7,c
Σ
Ij +
Σ
2j +0
by definition of Xs, t.e the
iEC
SEC2
Ei + Σ c7
iECHUC
iE0%
Σ
x
cτ
because C1. C2 form a partition X
n
n
xeX
VEC%
C2|c7
because X is centered by hypothesis
From now on we note Y as being the centered set of representations Xs, where we subtracted the mean X Xs v, (here we used the fact that |Cil + |C2| n)):
Y fa C2lo7
C2 uEC)Jffx
i i C2}
Now that we have workable expressions for X and Xs, we focus on the computation of linear CKA which relies the computation of three HSIC values: between X and itself, between Y and itself and between X and Y:
HSIC (X,
CKA CKAtin(X,Y) (X,Y)
(5)
(X, X)HSIC(nn (Y,Y)
We also remind the reader that linear HSIC takes the form:
7
(SIC(in(X,Y) = (n 1al tr xXxx TY (n 1)2
(o, (1)) v6)
(6)
1=1 1=1
We can split the terms the two sums into three distinct categories and compute the values of the inner products independently in terms of 3,, By and for the three HSIC terms:
22

---

Published as a conference paper at ICLR 2023
1. i E C1 and j E Cí (i.e. x, O1 and 2, C)):
(1i, 2z)² = Ty)²
c/Cal
cOall σ)
(20e 25, (yt, Ms) (1), (2s
v, U2
<02 (2,, 87)
c cO²
03)) + (-cCCl
= (mre,
(x),));)
(8, ity
n
n
n
c cC cC2 (ri, 1J)
(mi,
[5)
( mz) +
n
n2
cCS/ (oe, 2x),)
dC3
c² C22
2,)
(7, x.) (2,, xz) (x.,
(x)
n²
e² 2| (Ii,
= (()) +
n²
c/2a| 1,
clOa 7)²
(yh, 0z) (x:
cC22
clCal
210
(p))))))
(mi; t)
(,,
n²
= O +
n
2. i E. Cj and j E C22 (i.e. as O1 and æy E C2):
(x,,,,))² = (21,
(zi, 2y) 83) (2i, (x: cCC
cC1 1t)
U, T,
(Ii, + cCi (er, t)
cC2á (v,az)
C| ( 5)
= (,),
n
n² 2 (T,z))
(g), + dCi (x,, 7)
cC'l
(m),2,)
n²
cliil (2 23) (2i, 8)
dCa
2' n
Ca
(,,,2,))
(,,,,))
2 &CCI (gi,
O(c)
n22
(2r dC'1 (ye, m3)
c|C| J)
J.HY +
a
= ((01,
dCil
(mi, t)
cC
( xz)
C 21
7
n2
= =0(6) + c (
124
23

---

Published as a conference paper at ICLR 2023
3. i E C22 and j E C222 (i.e. I, C2 and 2j C2):
(0t, (Uk, Us) = (Ir, (Tz + c|Cil 1t, x, + dC'l 1))
n
n
(5), + 20|C (1i, 77 (+ ((C))
2
(g)
(T, T.)
(,)) + 2c|C1| (21, 8)
(x)
n²
2c|Ci ( 8)
e² e2IC11?
= (m
+
(1s,
1/16 TL
n²
(cc)
(g))
(w..w))² = (c: + c n²
c|Ci 7t/
J.zg +
n
n
= (2n,2z) + 2c|C1
e c 7
(zt; T)
n²
=ce)) +
n²
When we take lim CK Atin (X,Y)
= lim
IISI (X, (X.)C it is easy to see that the
HSIC
C-000
terms with the highest powers of will dominate the expression. At the numerator that is o² and at the denominator that is ct inside the square root. To convince oneself of this it suffices to divide by
o² at the numerator and at the denominator, all terms except the higher power ones will then tend to as tends to infinity, the limit we have:
24

---

Published as a conference paper at ICLR 2023
lim CK Atin (X,Y)
6-000
HSIC (X
lim
c
HSIC (X, X, XJHSSICHi(Y,,
Σ Σ (2, (V5.
lim
1=1/=1
Σ Σ (2))
Σ (yis 2112
j=
<'Ic 1
Σ
Σ
(x2,
26 2c Ca
Σ Σ
+
A 2
Σ Σ
n²
n²
sECi
JEC
iEC1 JEC/
8EC JEC2
lim
0-130
(0i x2
Σ Σ
A
Σ Σ
2 C-HCE.
Σ Σ
c Ci
1=
V
iECi JEC
n
iEC JEC
n
JEC
n²a
Σ Σ (1i,
Σ
Σ
( B3) + 3 2
Σ Σ (11, iEC
x²
n²
n²
NEC JECK
iEC JEC2
= -ii
6-100
Σ
(2s nz)²
dlaMe 2d CP
n
o AC
Σ Σ (1)) 20 20 CAIl
Σ Σ 21) A Σ Σ (z))
n²a
iEC, JEC
JEC'
JEC
lim
C-+20
a n
23)2
(1C2/2 + 2|C |C + (0113)
<|c 2
Σ Σ
(I xz)
G0 n²
Σ Σ (1i; +z}} +
GGG 2 n²
Σ Σ (1)
NEC JEC
iEC JEC/
+EC JEC
lim
0-130
010 cil
(ui, VCP 2 C1
C22 + CT
n
C Σ Σ (I 2z) -2 Σ Σ (I, 23)
IC'l
Σ Σ (,),
i JECK
SECi JEC/
SEC% JEC2
lim
0-+00
n Σ 2,) //y/2 + 2 C1 Cy + +Gp
Σ Σ
(It, x3)) (1))2)) - 2 Σ Σ (,,,
Σ Σ (z)
iEC JECK
NECKJEC
iEC
Σ 1-
0,,, 2 | + 2 C + c7
Σ Σ
(g)) -2 Σ (
C Σ Σ (z))
i JECK
iEC JEC₁
JEC JEC
Σ Σ
(ui, 22)2 / (1Cg) + +CN)2²
|=
C Σ Σ
(It, 2,) 2 Σ Σ (I,, +
Σ Σ (g)) LEC)JECy
sECi JECK
iEC JEC/
(Li, 22))2/n
1=
C Σ (1)) -2 Σ Σ (Zt,
C Σ Σ (z),2,)
+
c
C22
sECi JEC
iECK JEC
iECY
Σ
(1t,
32
==
25

---

Published as a conference paper at ICLR 2023
If we look directly at the numerator, by linearity of the inner product have:
C Ic'l
C
(g))
-2
Σ
(x),
(g)
C2
iec
jEC
iECi JECK
iEC JEC
=|
=
x3)) - 2|C |C|
Σ
Σ Rz)
|cil
|C²
iEC1
JEC
sECi
3EC2
+
+|G|||CI
3ii
,3}
iEC2
jEC
[C
2( T2)
T2
|C||Ci|||=T xzll 1
Where xy = Ezec, [x] =
1 Idyl tEO
Σ
I; is the mean of the points in Cj. At the denominator we can
multiply by to obtain:
Σ
Σ
(3,, n²
Σ
ΣΣ n2
(1i,
n n1
Z,E
2,EX
#,EX 2,EX
n X
n²
We note that the term with the square root function is the Frobenius norm of the Gram matrix of the data (matrix inner products) XXT multiplied by which, in turn, is equal to the square root of the sum of 's squared cigenvalues, where A; is the i-th eigenvalue of the matrix 4 However, through singular value decomposition, the Gram matrix (multiplied by 4) has the same eigenvalues as the (biased) covariance matrix of the data, i.e. 1. _ Using the notation X; go over the rows (representations) of X with and over the columns (dimensions or neurons) of X with j we can write the variance in the data the sum of the variances from all dimensions:
Var(X) =
Σ Var(X.s)
1=1
(Xi. - )²
Σ
x2
because the data is centered
n x
n Σ
Σ
x
E [||:||]]]
1=1
26

---

Published as a conference paper at ICLR 2023
We then write the denominator as:
Var(X)
7
x n
Var(X)
Σ x2
=n =n Var(X ()
Σ A2
/=1
Ex'Exxx E=Ex||1011
12
Σ A
= Exe
And we can rewrite the whole expression as:
lim CK (X Cilly J- I PP(X) 0-000
n² [|
Where PR(X) the participation ratio, an effective dimensionality estimator often used in the literature and is defined as:
Σ A
PR(X)
X
With A; being the i-th eigenvalue of the covariance matrix of the data X. We make the replace- 2-|Cil p. Also, because the data is centered have so we have:
ments [Cil |s p and [C2l
C .I C2 C Ty 0 and we can isolate T- =
-
CCa
| 21 ||
x22
xzll
|| +
|C21
|
(1 +
T 2
C22
/||1|
(1 +
2 |
C|C2
P
= (1 +
We then define I to contain all terms of p:
T(p) p(1 (1{1-)(11+
)²
1-p p
1-p p
The following bounds hold: r(p) E (0, 1] for p (0, 0.5) reached when p 2. Finally, we get the final expression by changing उ = Evec, [x] = E.cs[x]:
|B [r] 2
lim CK CKA (X,Y) Y) T(p) 6-000
PR(X)
E x x
27

---

Published as a conference paper at ICLR 2023
Proof. Corollary 2
To prove Corollary it suffices note that the fact that p [s] is in (0, 2 is not used anywhere in
[X]
the proof of Thm. other than to derive the bounds for [(p). We can then conclude that Thm. 1 still holds if S is taken such that p
| | (0.5, 1) only with different bounds for (p)).
We note however that the expression in Thm. | can be written in terms of S and p in terms of
S' = X and p' [x| interchangeably. We first note that p' = 1 1 and for simplicity purposes
we will use notation [xes(æ] and S' = Engs((r) We recall that the expression in Thm. is:
|||
lim CK (X, Xs, = [(p)
PR(X)
BEEx||=]3]
PPR(X) The term Exex [
does not depend on the choice of using S or X S so we can focus on the rest:
/' p1(1 p1)(1 + 1- 2
p¹(1 - -p') is easily found to be equal to (1 - p)p and for the rest we have:
1 ||1-111 = 11(1 + 1 p'
p'
+1+
)511
||S' - S112
= [3 - 1
= || + P 31
1-
(1 p p
131
Where used the fact that the data is centered so | conclude that we have:
so we can isolate = -19'|s Ss'
[5]
|
lim CK (X, Ns, T(pp)
1|=|11=>|1 V PR(X)
Proof. Corollary 4
We already have E S (w, x) I k. Pick any J. E R that is orthogonal to and we have:
(w, + cT) = (w1, x)) + c(w, 1)
by the linearity of the inner product since 7 is orthogonal to w
(w, x))
k
Vx E X
D FURTHER DISCUSSION OF THEORETICAL RESULTS
D.1 ANALYSIS OF THE TERMS IN THEOREM I
In addition the description of the terms that we provide Section (page 4 before the paragraph on CKA sensitivity to outliers) we go here into further detail regarding the terms from Thm. [ The two expectation terms, Exex [[]]]]]]]]]]]]]]]] and [|[]))[[]]] are relatively simple to describe. They respectively represent the average squared norms of all representations in X and the squared norm of the mean of S, the subset of representations that are not being translated. As mentioned in the main text, since most neural networks trained using weight decay, the network parameters, and hence the resulting representations as well as these two quantities are biased towards small values in translated, with respect to the points practice. r(p) Th is the ratio of the points in S, that
28

---

Published as a conference paper at ICLR 2023
in X'S that are translated. Finally, the last term v d is the square root of the participation ratio, which is used as an effective dimensionality estimate for internal representations. It is defined
as (X) & (.)
Σ; [l where } are the cigenvalues of the covariance matrix of X.
These eigenvalues are perhaps best understood through the lens of principal component analysis which uses the eigenvectors of the covariance matrix principal components and the cigenvalues for describing the variability of the data which is found along the direction of it's corresponding eigenvector. In other words, the eigenvalues of the covariance matrix encode the amount of variance in the data in an orthogonal basis of the ambient space of the data points. The participation ratio is the ratio between the squared sum of eigenvalues and the sum of the squared eigenvalues. We describe the two extreme cases for intuition. If all the variance of the data comes from a single direction then only one of the eigenvalues will be non zero (WLOG X)) and therefore both the numerator and the denominator will be equal X so the PR will be 1, i.e. the estimated dimensionality will be 1 which makes sense given that all the variability comes from a single direction. On the other hand, if the variance is equally distributed across all directions, all the eigenvalues will have the same value À and
(Σ.A)² p²A²
the participation ratio will be x pA p. Therefore the effective dimensionality will be the full dimensionality of the ambient space. Since there is an equal amount of variance in all directions, all dimensions of the ambient space "participate" equally in describing the dataset X.
D.2 ADDITIONAL INTUITION ON THEOREM AND COROLLARY 2
question that may arise related Corollary is: what happens when is such that p >> 1 and the value of I quickly grows beyond 1? It turns out that it is the fraction of expectations that will cancel out large values. As side note, before we start explaining the previous sentence, we point out that since X contains a finite number of data points I will never actually tend to infinity even if it can take on large values. Now suppose S is such that p >> 2' i.c. S contains a majority of the points in X, therefore I' will have a large value. Since the points in X are centered and since S contains majority of the points from X, this implies one of two things. Either:
1. The mean of the points in S will be almost zero (Ezes [x] 0), therefore the numerator will be almost zero since it is the squared norm of this mean. This can easily occur if S is a very large part of X (which is centered) and the points in X are not outliers with large norms so the mean of S will be very close to the mean of X, which is 0 in this case. The denominator on the other hand (Raex x [|]==|?] can be large since we take the squared norm of the points in X, which are all positive values, and only afterwards compute the expected value.
2.
The small number of points in XXS will have very large norms relative to those in S and will be positioned on the opposite side of the origin with respect to the mean of S (so the mean of X can be at the origin). The bigger inside X, the bigger the norms of the points in X\S5 will need to be to compensate. Therefore, the numerator ||Emes[<=]]| will again be very small with respect to the denominator Ex [||=||]]] At the denominator, since we take the squared norm of the points in X before computing the expected value, the points in X which have very large norms compared to those in S will dominate the expected value. Consequently, the value of the fraction of these two terms will be very small.
Another insightful way of analyzing this situation, which is perhaps more intuitive, is to look at it through the lens of the proof of Corollary? [ Given the fact that the data centered before applying CKA, translating the points in X in any random direction or translating the points in S in the exact opposite direction will yield the same set of representations post-centering, and thus will give the same CKA value. Therefore, for any S such that p 1 can look at the complementary case where we swap the definitions of S and X\,S in Theorem [ and the translation direction is inverted (opposite direction) and find ourselves with the new p being smaller than 1. the new I smaller than and Theorem still applies. More explicitly we can apply the Theorem 1 to S' = X\, with
p' (1 p) < 1 and the new direction / -
29


Reviews:
[{"review_id": "pkQxv9J9TXw", "reviewer": "Reviewer_YJ8i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission discusses some pitfalls of using CKA as a similarity measure. In particular, the authors show that CKA is sensitive to outliers and some transformations that preserve linear separability and margin.", "summary_of_the_review": "My main concern is that the submission mentions certain properties of CKA as weaknesses without discussing the context in which CKA is used and why these properties are weaknesses in these context. I also have some doubts about the correctness of some theoretical results and novelty of some empirical results that I elaborated on in the strengths and weaknesses section.", "strength_and_weaknesses": "The most interesting point about this paper is characterizing the sensitivity of CKA to outliers. Although sensitivity of CKA to large transformation to a few data points is not surprising to me, characterization of this sensitivity can be helpful for future work.\n\nMajor comments:\n\n1. My main concern is that the submission evaluates reliability of CKA for comparing representations without a clear regard for the exact purpose for using this measure. A measure of (dis)similarity like CKA captures certain information about the representation and is invariant to other transformations. Whether an invariance is a strength or weakness for this measure depends on the context and the purpose for using the similarity measure. For example, Kornblith et al. use CKA for gaining insights about the training process of neural networks (see section 2.1 in their paper) while Ramasesh et al. use CKA to measure representation drift in the context of catastrophic forgetting. Kornblith et al. argue that a proper measure of similarity in their study needs to be invariant to rotation. In the context of Ramasesh et al., the similarity measure should not be invariant to rotation, since a rotated representation will have a different separating hyperplane and will result in forgetting if the classifier is not adjusted for this rotation. The submission would have been stronger if, before stating that a property of CKA is a weakness, it clearly and explicitly stated a context where CKA is used in previous work and discussed why this property of CKA is a weakness in this context.\n\n2. The finding that representations with different separating hyperplanes may have a CKA close to 1 is well-known. Konblith et al. already show that CKA is invariant to orthogonal transformations (which clearly can change the separating hyperplane) and, as I said above, argue that this is a strength of CKA is their context.\n\n3. Corollary 2 states that Theorem 1 holds for values of rho close to 1. In this case, Gamma will tend to infinity. Which factor in Theorem 1 will cancel out this large value of Gamma and keep CKA in [0,1]?\n\n4. It looks like the definition of S has changed in Corollary 3. In theorem 1 it is the set of unmodified data points but corollary 3 it is the set of modified data points.\n\n5. In the experiment for Figure 4a, the points from the second cube are translated in a random direction. The text claims that the cubes are still separated after this transformation. This is not true if the translation direction happens to be towards the first cube.\n\nMinor comment: Please explain in the caption for Figure 3 how the plots are generated and what the colors represent.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fpyQdF-9fjY", "reviewer": "Reviewer_U5e1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on understanding Centered Kernel Alignment (CKA), a common representational similarity metric. In particular, the paper shows that CKA is overly sensitive to a large class of \"simple\" representation-space transformations, which do not change the functional behavior of the trained models.  The theoretical analysis shows that CKA is sensitive to subset translation---a transformation that shifts a subset of data points in representation space. This finding sheds light on why CKA is empirically sensitive to outliers (\"dominant data points\") and to transformations that preserve linear separability.  The paper also shows that one can explicitly optimize the CKA map to match a comical target map without really changing the model behavior (e.g. accuracy) by much. They also show that the block structure phenomenon can be reproduced with this explicit optimization approach. Taken together, these results show that the CKA is unreliable and can easily manipulated without substantial changes to the functional behavior of the models", "summary_of_the_review": "I vote to accept this paper. The paper is easy to read and well organized. The paper focuses on an interesting problem and shows that the CKA metric is not a good proxy for functional behavior of models. Most findings in the paper are original/novel and the practical implications of the theoretical analysis is insightful. ", "strength_and_weaknesses": "Strengths \n\n- The theoretical analysis on subset translations is insightful because it sheds light on why CKA is empirically sensitive to outliers (\"dominant data points\") and to transformations that preserve linear separability. \n\n- The empirical results on explicitly optimizing CKA maps show that CKA is overly sensitive and may not be a useful metric to draw conclusions about general model behavior. \n\n- The empirical findings cast doubt on the \"block structure phenomenon\" that is considered to distinguish representations of wide and narrow networks. \n\n\n Weaknesses:\n\n- Specifics of Theorem 1 is a little hard to interpret.   A more intuitive description of these quantities after stating the theorem (e.g., eigenvalues of cov(X), rho, dim_PR etc) is missing. \n\n- I think section 4.1 needs more work, or can be removed entirely. I am not sure how it relates to subset translation (the class of transformations studied in this paper) and the comparison is weak---comparing weight matrices visually does not make sense (even it \"looks different\")", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "GaFDpGNggSk", "reviewer": "Reviewer_UWFM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors inspect the robustness of centered kernel alignment (CKA) as a measure of representational similarity. They analytically derived the limiting behavior of CKA as a subset of points are translated infinitely far away. They suggest that CKA is sensitive to a small number of \"outlier\" values. They empirically back up their theory by showing it is possible for most networks to achieve a variety of across-layer CKA maps without hurting performance. Further, CKA similarity between networks memorizing and generalizing on CIFAR-10 images was surprisingly high. Thus, their empirically results show that representational similarity as quantified by CKA can be decoupled from network function.", "summary_of_the_review": "Overall, I find myself on the fence. I think in its current form the paper doesn't yet meet the bar for acceptance, but I am open to raising my score if the authors provide further empirical detail on Figure 2A (see comments above), and if they are able to revise their theory section to impart more intuitive insight. ", "strength_and_weaknesses": "I liked the motivation of the paper. CKA is an influential tool in the field and it feels important for us to closely examine its failure cases. I am unaware of any paper within the ML literature which takes a close look into these issues. Figure 2 and Figure 8 show interesting empirical results which I feel are highlights of this work.\n\nOn the downside. I did not feel like the main theorem proved in this paper gave me much intuition, and in fact some of the statements made by the authors were quite confusing to me. For example, at the top of page 5, the authors say \"$\\Vert \\mathbb{E}[x] \\Vert^2  / \\mathbb{E}[ \\Vert x\\Vert^2]) \\sqrt{\\text{dim}_\\text{PR}(X)}$ will be of relatively small value in practice so the whole expression in Eq.\n2 will be dominated by $\\Gamma(\\rho)$.\" Maybe I'm missing something very basic but if $\\Vert \\mathbb{E}[x] \\Vert^2  / \\mathbb{E}[ \\Vert x\\Vert^2]) \\sqrt{\\text{dim}_\\text{PR}(X)} \\approx 0$ then you'd have $\\text{CKA} \\approx 0$ always in the limit of $c \\rightarrow \\infty$? Even when $\\rho = 1/2$? More generally, I think that the limit of very large $c$ isn't too intuitive or sensible to think about (e.g. if some form of weight decay or other regularization is applied). It would be more useful to understand the rate at which the CKA decreases (for finite $c$) in Figure 4. Perhaps the authors can clear up their intuitions in the rebuttal period.\n\nI also think that the results in this paper could be made more rigorous. For example, Figure 2 intrigues me, but I would love to see how much CKA varies across different random seeds. In particular, I would like to see additional lines for `Generalized vs Generalized` and `Memorized vs Memorized` and `Randomized vs Randomized` across random seeds. Also, error bars indicating standard deviations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FMKItGfCo5", "reviewer": "Reviewer_wkkL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work cautions the application of CKA similarity measure in comparing neural representations in practice and points out a few issues including 1) its sensitivity to the family of the subset translations where certain transformations do not change the functional behaviors of the model; 2) its value can be directly manipulated without significant changes in the model behaviors.", "summary_of_the_review": "The CKA similarity has been widely used to draw conclusions in comparing neural representations across model architectures and learning paradigms, thus it is critical to better understand its strengths and weaknesses. I agree with the authors that any similarity measure should not be blindly applied, and multiple measures should be considered to cover multiple notions of similarity. Even though the CKA map manipulations is quite interesting, I am concerned with the similarity assumption advocated in this work: models with similar accuracies (in one dataset) should have similar representations. Since the main part of this work builds on this assumption, I am leaner towards the negative side.\n", "strength_and_weaknesses": "On the strengths,\n- The authors provide a formal characterization of the CKA sensitivity to the family of subset translation transformations (i.e., the translation of a subset of the representations) and demonstrate special cases of this family can preserve \"functional behavior\" of the learned neural representations. Thus, an ideal similarity measure should be insensitive to this family, but the CKA fails to do so.\n- CKA is a family of measures so merely showing the linear case is sensitive is not very conclusive. It is nice that the authors not only conduct analysis on linear CKA but also nonlinear CKA with RBF kernels and show that the nonlinear ones still suffer from the same sensitivity.\n- The demonstration of the CKA map can be manipulated into any target map without changing the model performance is quite interesting and reminds me of the adversarial examples. This result questions the validity of many existing conclusions regarding the similarity between different models and their behaviors.\n\nOn the weaknesses,\n- My biggest concern is about the \"similarity\" advocated in this work. Informally, the similarity assumption is that models that get similar accuracies have similar representations. In order to make this more robust, the authors add more constraints including same linear separating hyperplane and same margin. Even with these additional constraints, I do not think this assumption is a good one. Models learned with different random seeds have almost identical in-distribution accuracies, but vastly different out-of-distribution accuracies. In an analogy, this suggests that two distributions having the same first-order moment should be the same distribution. But we only consider two distributions same if they share the same moments in all orders. Therefore, unless the fixed models have similar accuracies in *all* datasets might we decide them to have similar representations. Indeed, similar assumptions have been considered in the literature, but they are used in a different way. For example, (Ding et al., 2021) uses this as an audit. That is, representations having different in-domain and/or out-of-domain accuracies should be different. This is very different from what used in this work.\n- Some of the conclusions are not well supported. For example, it is argued throughout the paper that the subset translation can *naturally* happen in training neural networks but fails to provide any concrete example. In Section 4.1, the conclusion of \"considerably high CKA similarity values for early layers, does not necessarily translate to more useful, or similar, captured features\" is based on the visual differences of the learned filters without considering the \"functional behaviors\" advocated in this work. It is possible that they are *functionally* similar even though they are visually different.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster