Deconfounded Representation Similarity for Comparison of Neural Networks
Authors: Tianyu Cui
Yogesh Kumar
Pekka Marttinen
Samuel Kaski
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep neural networks
representation similarity
CKA
RSA
functional similarity
covariate adjustment regression

Abstract: Similarity metrics such as representational similarity analysis (RSA) and centered kernel alignment (CKA) have been used to understand neural networks by comparing their layer-wise representations. However, these metrics are confounded by the population structure of data items in the input space, leading to inconsistent conclusions about the \emph{functional} similarity between neural networks, such as spuriously high similarity of completely random neural networks and inconsistent domain relations in transfer learning. We introduce a simple and generally applicable fix to adjust for the confounder with covariate adjustment regression, which improves the ability of CKA and RSA to reveal functional similarity and also retains the intuitive invariance properties of the original similarity measures. We show that deconfounding the similarity metrics increases the resolution of detecting functionally similar neural networks across domains. Moreover, in real-world applications, deconfounding improves the consistency between CKA and domain similarity in transfer learning, and increases the correlation between CKA and model out-of-distribution accuracy similarity.

Deconfounded Representation Similarity for Comparison of Neural Networks
Tianyu Cui Department of Computer Science
Yogesh Kumar Department of Computer Science
Aalto University
Aalto University yogesh. kumar@aalto fi
tianyu. cui @aalto. fi
Pekka Marttinen
Samuel Kaski
Department of Computer Science
Department Computer Science
Aalto University
Aalto University and University of Manchester samuel kaski Saakto0allo.....
pekka. www.mmiennnaaato......
Abstract
Similarity metrics such as representational similarity analysis (RSA) and centered kernel aligmment (CKA) have been used to understand neural networks by compar- ing their layer-wise representations. However, these metrics confounded by the population structure of data items in the input space, leading to inconsistent conclu- sions about the functional similarity between neural networks, such as spuriously high similarity of completely random neural networks and inconsistent domain relations in transfer learning. We introduce simple and generally applicable fix to adjust for the confounder with covariate adjustment regression, which im- proves the ability of CKA and RSA to reveal functional similarity and also retains the intuitive invariance properties of the original similarity measures. We show that deconfounding the similarity metrics increases the resolution of detecting functionally similar neural networks across domains. Morcover, in real-world applications, deconfounding improves the consistency between CKA and domain similarity in transfer learning, and increases correlation between CKA and model out-of-distribution accuracy similarity.
Introduction
Deep neural networks (NNs) have achieved state-of-the-an performance on a wide range machine learning tasks by automatically learning feature representations from data [ 3 4 5]. However, these networks do not offer interpretable predictions on most applications and are seen as "black boxes". It is thus crucial understand the intricacies of neural networks before they deployed on critical applications. Previous work has made progress in understanding how single neural network makes decisions with axiomatic attribution methods 6 7 and understanding how multiple neural networks relate to each other with representation similarity measures [8] Several similarity measures between representations have been proposed with different principles, including linear regression [9), canonical correlation analysis (CCA; [T0 IT), statistical shape analysis [12), and functional behaviors on down-stream tasks 13
14
[5
Another main-stream approach is based on and centered kernel alignment (CKA, (19).
representational similarity analysis (RSA, [8 16 17 (18)
which compute the similarity between (dis)similarity matrices of two neural network representations on the same dataset.
RSA and CKA have been successfully applied understand biological (20 and artificial NNs 21 by studying the similarity of representations of different NNs in a single data domain. However, we find
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

CKA and dCKA bebween two random Rusiluts (NWs MM2)
CKA and dCKA between protrained (PT) and fine-tuned (FT) ReaNets
d600 (ourdnn)
US WERMWik The FUNTOO
Sen
Sawivityy
Sirileriyy
Samibiityy
Ca) Merrus
Samiaarty 17
Samilarty:
Samiluityy
Siilety repreSertation
Samilartyy represerracio
egrestrrziion
repeesercation
*
PIPUE
ripa aorburtyy pai
Figure 1: Demonstration of the confounder in CKA. CKA calculates the similarity between inter- example similarities for two representations. which are confounded by the inter-example similarities in the input space, such that input pairs with high (+) and low (*) input similarities also have high and low representation similarities on both random NNs (Leff) and trained NNs (Right) representations. Moreover, the confounder leads to the counterintuitive conclusion that CKA on random NNs is higher than pretrained and finetuned NNs on similar domains (0.99 vs. 0.95). This is resolved by deconfounding (0.43 vs. 0.72).
that the inter-example (dis)similarity matrices in the representation space of different NNs are highly correlated with shared factor (i.e., a confounder): the (dis)similarity structure of the data items in the input space, especially for shallow layers. This confounding issue limits the ability of CKA to reveal similarity of models on the functional level [15], and can result in spuriously high CKA even between two random NNs. Moreover, this can lead to counter- intuuiive conclusions when using CKA to compare models trained in multiple domains and/or to deduce functional similarities of NNs, such
as in understanding transfer learning, multitask learning, and meta learning 22 23 24 25 26]
In this paper, we propose to adjust the representation similarity to improve its consistency w.r.t. the functional similarity by regressing out the confounder, the inter-example (dis)similaritt matrix in the input space, from the (dis) (diiiariyyyy matrices of two representations. inspired by the covariate adjusted correlation analysis widely studied in biostatistics [27 28 This approach is simple and widely applicable on any similarity measure built on the CKA and RSA framework. Moreover, we study the invariance properties of the deconfounded representation similarity and demonstrate its benefits on public image and natural language datasets with various NN architectures.
Overall, our contributions are:
We study the confounding effect of the input inter-exampll similarity on the representation similarity between two NNs, which limits CKA and RSA from revealing consistent, func- tional similarities. We propose a simple and generally applicable deconfounding fix and discuss the invariance properties of the deconfounded similarities.
We verify that deconfounded similarities can detect functionally similar NNs from random NNs, and small NN changes across domains where previous similarity measures fail.
We show that deconfounded similarities transfer learning, on both image and language datasets, compared with existing methods.
consistent with domain similarities in
We demonstrate that deconfounded similaritics on in-distribution datasets are more corre- lated with -utttr-diittibuttin accuracy than the corresponding original similarities (15).
2
Preliminaries
2.1 Notation and prior work
Let X R=*p denote the input dataset with datapoints and p features, and X E ReXpi and x E m=*x= denote the myth and myth layer representations of two NNs of interest, fX) and f2(X), with Pl and P2 nodes respectively. We center and normalize the representation matrices by first removing the mean of each feature (i.e., each column), and then dividing by the Frobenius norm.

---

A standard approach for comparing representations of two NNs is to compare the similarity structures in each network representation. This can be done by first computing the similarity between every pair of examples in " and fz with a similarity measure k(;;,):
K = k =(X xx)))) ) K = xx xx))
(1)
Here Ky K/ E _ax" are called representational similarity matrices (RSMs) Second, another similarity measure s(;,)) is applied to compare these two similarity structures, K K and K K
fi
mn m f1.f2 s(K m2))
This gives the similarity between the two representations.
The existing approaches vary by using different similarity measures for both levels of comparison. CKA [19] employs a kernel function for the first level similarity k(---), and a Hilbert-Schmidt Independence Criterion (HSIC) estimator for the second, s(.-). On the other hand, RSA-based methods use Euclidean distance to measure the inter-example dissimilarity structure, and apply Pearson's 's correlation [ or Spearman's 's rank correlation [ to quantify the similarity between two dissimilarity structures. Although other approaches, e.g., linear regression [29] and CCA [10), have been proposed to compare NN representations, we focus on CKA and RSA this paper due to their
wide usage in understanding the properties of NNs, such as transfer learning [2 23 30
2.2 Illustration of the confounding in representation similarity
Given dataset X with an inter-example similarity matrix K = xXX, X) in the input space, making predictions with NNs can be seen as modifying k layer-by-layer, such that similarities between data items with different labels decrease and those with the same label increase. In CKA and RSA, the similarity between two RSMs, K and K%²² defines the similarity between two NNs, as shown in Eq However, both K and K7 are affected by the same factor (i.e., a confounder): the similarity structure of input dataset K which can cause spuriously high similarity. Intuitively, (dis)a data points (green stars and red stars in Figure [ in the input space are likely to be (dis)similar in the representation space of the first few layers, regardless of how the NN operates, and the representation similarity structure of different NNs would be similar even for random NNs that have totally different functional behaviors. Hence, CKA/RSA naturally depends on the specifics of the dataset, and for different datasets (c.g., from different domains) with different K comparing CKA/RSA across the datasets may lead to inconsistent comparison results. This is undesirable especially when the goal of calculating the similarity measures for NNs is to quantify how similar the networks are in the functional level, which should not be significantly affected by the specifics of the dataset hand.
We fix this by regressing out the input similarity K from KJ and Kyp i.e. deconfounding. After deconfounding, any similarity in the NN latent space between two data items would be induced by the NN and not because they were similar in the first place. This way, the deconfounded metric focuses more on comparing the functional form of the NNs and is less affected by the structure of the given dataset in the input space.
Figure T illlssratte the spurious similarity using the CKA similarity measure as an example, and compares that with the deconfounded dCKA (defined in the next section) on the first-layer of ResNets
31 with 20 random samples from CIFAR-10 10 (32 test set. We consider two pairs of ResNets: 1.
two random ResNets generated by adding different Gaussian noise, (O, 1), to each parameter of the pretrained ResNet- on ImageNet 333; 2. the pretrained (PT) ResNet-18 and a finetuned (FT)
ResNet- on CIFAR-10. We notice that CKA on random NNs is almost 1, and counterintuitively it is even higher than the CKA between PT and FT ResNets on a similar domain (0.99 vs. 0.95), although we would expect the PT and FT networks learn similar low-level features and hence be more similar than random networks. This happens because the similarities between samples in the input space confound their similarities in the representation space. After adjusting for the confounder with dCKA, the similarity between the two random ResNets is much smaller than the similarity of the PT and FT networks (0.43 vs. 0.72). A more detailed study is provided in Section 4.1
Note that A(-,-) can also be dissimilarity measure, but we call KT a similarity matrix for simplicity. http://///pyoccccoooo//////////s
html
3

---

3 Methods
In this section, we first propose general on the NNs similarities (defined in 2 by regressing out the confounder, i.e., the input similarity, in Section 3.1] and we give two examples of deconfounded similarities: deconfounded CKA and deconfounded RSA in Section 3.2 We then study the invariance
properties of deconfounded similarities in Section 3.3
3.1 Deconfounding representation similarity
We propose simple approach to adjust the spurious similarity caused by the confounder by regressing out the input similarity structure from the representation similarity structure [34] That is:
Th JK K an 1 dK K _ K
where K is the input similarity structure, and ش and f2 are the regression coefficients that minimize the Frobenius norm of d and d respectively. Furthermore, the letter d in front of similarity matrix, e.g. in f, denotes the deconfounded version of /1 and similarly the letter d is applied throughout the text to denote all defounded quantities. To do the deconfounding. we assume that the input similarity structure K has a linear and additive effect on KJ" i.e.,
vec(Ky)) vec(k)) + +ef.
where vec(-) flattens a matrix to a vector. Noise eG is assumed to be independent from the confounder with e = vec(dky) and
aT = (
vec(k ( vec(KT)) vee(k))
(5)
In Appendix A we further verify that the linearity assumption in Eq 4 holds in general, i.e., adding additional nonlinear terms of vec(K to Eq does not improve the fit, and the noise term e/ is not auto-correlated (solution Eq is not misspecified) under the the
test (35).
After the deconfounded similarity structures are obtained with Eq 3 measure to calculate the deconfounded representation similarity:
we use the same similarity
ds mi ma s(dKT) fi .dK.).
(6)
fi.f2
Note that yKy is not always positive semi-defininn even when K is positive semi-definite (PSD). PSD is important for a similarity measure .(:-)) that takes two kernel matrices as input, such as the CKA. We transform dKKm into a positive semi-definite matrix by removing all the negative eigenvalues according to (36.. Specifically, we have the eigenvalue decomposition of dky, such that
dKT - QAQ" Q(A+ A_) A+ diag(max(0,, #AI).... max +Nn)),
where x; is the ith eigenvalue of dKy. We approximate dKy with PSD matrix dRy:
dky dk p²QA4Q"; p (=(()))\((AA)))
(8)
3.2 Examples of deconfounded similarity indices
Deconfounded CKA. In CKA [19], the similarity structure in the feature space is represented with valid kernel I(,, -), i.e., KTh = I(XT) fa X and Km = I(X7 xx%)), ). such as the linear or RBF kernel. Then an empirical estimator of HSIC [37 is used to align two kernels:
1
HIS m2
(n - 1)2 tr(K)) HK f2 H);
(9)
where H/ is the centering matrix. CKA is given by the normalized HSIC such that
CKA(K Kmp)) = HSICT1.m2 fiJJa /
/HSIC HISI M222 Jih f2.
(10)

---

To deconfound the representation similarity matrices i and K we apply the same kernel to measure the inter-example similarity in the input space K X), and adjust its confounding effect with E 3 However, matrices dk fi and d obtained by regressing out one kernel matrix from another kernel, are no longer kernels, and they are not applicable for computing HSIC. f2 with two valid kernels dKj, and dk which are then used to construct the deconfounded CKA (dCKA):
Fortunately, with Eq 8 we can approximate the dk and dk
JCKA(K) TI .K))) = CKA(dK)) ,dky)2)
(II)
We use linear kernels here because Kornblith et al. 19 report similar results with RBF kernels.
Deconfounded RSA. Different from CKA, the similarity structure in RSA g is measured by the pairwise Euclidean distance between examples in the feature space. Specifically, each element of KT and K is obtained by K ||>) mu 112 and Kma f2.1j = ||>> "2 11 where x mi is the mlyyeeer representation of the ith example in NN fi. Thus, the input similarity structure f..
K is measured with the pairwise Euclidean distance in the input space. After k is adjusted with
Eq 3 we apply Spearman's p correlation measure the similarity between the upper triangular part of d. mm and dk i.e., triu(dK). ft and triu(dk fa )2 that is
dRSA(KJ) KKy)) p(triu(dk)))) ), triu(dk)
(12)
Note that rank correlation does not require two similarity matrices to be positive semi-definite. Therefore, we skip the steps of constructing the PSD approximation.
Additional computational complexity. The computational cost of the deconfounding (Eq and PSD approximation (Eq 8) steps are (nn)) and O(r²)) respectively. Therefore, dCKA has the same complexity CKA: O( ). For RSA, only the deconfounding step with O(n²) complexity is needed.
In experiments, computing dCKA between two XLM-RoBERT RoBERTa models [38] takes 0.37 = 0.11s longer than CKA for each layer 3000 random English sentences with single 2080Ti GPU.
3.3 Theoretical properties
In this section, we study the invariance properties of the deconfounded representation similarity. For similarity measures that we studied, i.e., CKA and RSA, the corresponding deconfounded similarity indices have the same invariance properties, such as invariance to orthogonal transformation and isotropic scaling, the original similarity measures.
Proposition 3.1. Deconfounded CKA and deconfounded RSA are invariant to orthogonal transfor- mation, if the (dii)similarity measure k(.--) that compare inter-examples are orthogonal invariant. Proposition 3.2. Deconfounded CKA with a linear kernel and deconfounded RSA are invariant to isotropic scaling.
Intuitively, as long as k(:,)) is invariant to orthogonal transformation, e.g., linear kernels and Eu- clidean distance, the deconfounded representation similarity matrix yKK in Eq is also invariant to orthogonal transformation, because it is defined in terms of the kernel k. Thus all operations on dK3' are invariant to orthogonal transformation. Moreover, if one representation is scaled by scalar, dKT and y' will be scaled by the same scalar, whose effects will be finally eliminated in the normalization step in CKA (Oq[IO) and the rank correlation step in RSA (Eq [2). We give proofs in Appendix B A good functional similarity metric should be independent of the inter-example similar- ity in the input space (because that is dataset specific), but dependent on inter-example similarity in the representation space (because that is affected by the functional form of the NN). Above invariance properties ensure that the deconfounding does not sacrifice the desirable properties of CKA and RSA regarding inter-example similarities in the representation space, which are essential understanding NNs in many cases 19
4 Experiments
In this section, we design experiments to verify that deconfounding can improve the consistency of CKA and RSA with the functional similarity of NNs from various perspectives. Specifically,
5

---

CKA
RSA
RSA
1.0
Proportion
(0.5) (3.02)
Block 1
0.5
diSA (6.25)
0.0
2
4
similarity
Blocks
Blocks
Random NNs
Imagstet NNs
Figure 2: Left: Proportion of ImageNet-CIFAL ResNets pairs identified from random ResNets for each block. The proportions for the last four blocks omitted as they are O for all metrics. The average proportions across layers are shown next to the method's name in the legend. We observe that deconfounding improves the identification of functionally similar NNs from random NNs. Right: Histograms of similarities in the first block. Compared with CKA and RSA, dCKA and dRSA increase separation of ImageNet-CIIee pairs from the random network pairs, even when the proportion of identified network pairs with CKA and dCKA is the same.
we compare NNs layer-by-layer and then average the evaluation metrics about the consistency of layer-wise similarities compare different similarity metrics [T3]. First, in Section we check these representation similarity measures can separate functionally similar NN pairs from random NN pairs. In Section 4.2 study how consistent the similarity measures with small changes of NNs across different domains. We then extend recent framework [15], which uses rank correlation to compare the representation similarity and the functional similarity of NNs, to transfer learning on multi-domain image and natural language datasets (Section 4.3 and to conduct challenging out-of-distribution generalization tests [15 (Section 4.4)
4.1 Ability of detecting functionally similar NN pairs from random NN pairs
Setup: We check if similarity measures can identify functionally similar NN representations from random NN representations. For each model block of ResNets (containing 2-3 convolutional layers), we generate two distributions of similarities: the null distribution Ho and the alternative distribution H1. The Ho contains similarities between 50 pairs of random ResNets on CIFAR-10 test set. We generate random NNs by randomly initializing the weight an untrained ResNet-18 from (O, 10) with different random seeds. We also consider permuting the weight matrix of the pretrained ResNet- 18 to generate random NNs, which preserves parameter distributions, in Appendix [.I] Distribution H. contains similarities between the pretrained ImageNet NN and each of the 50 ResNets trained on CIFAR- 10 from scratch with different random initializations, on the same CIFAR-10 10 test as Ho.
We expect the similarities in Hi to be significantly larger than those in Ho- Intuitively, models trained on ImageNet and CIFAR-10 are functionally similar shallow blocks because their domains are similar; hence similar low-level features are expected (39), while sufficiently randomized untrained NNs are functionally different. We compute the proportion (shown Figure 2 Left) of 50 ImageNet-CIFAR NN pairs in Hi whose similarity is significantly larger than Ho. i.e. larger than the upper bound of its 95% CL. In Figure Left, we observe that in shallow blocks the deconfounded similarity can detect larger proportion of ImageNet-CIFAR pairs from random NN pairs than the original measures. For example, in the third and fourth block, dCKA can still detect all functionally similar pairs whereas the original CKA fails to detect any from random pairs. By averaging proportions of all blocks, dcconfounding increases CKA from 0.24 to 0.5 and RSA from 0.02 to 0.25. In Figure 2 Right, Ho of each similarity measure as histograms in the first block. We observe that although CKA and dCKA can identify the same proportion of IaageNet-CIFAN pairs in block 1, the difference between Hi and Ho is more significant with the deconfounded similarities. Moreover, in deep layers, e.g., method can identify ImageNNetCCIFAR pairs. We hypothesise that ImageNet and CIFAR- 10 contain different classes of images, thus their high-level representations are significantly different. We discuss this more in Appendix D
Results:
visualize the alternative distribution Hi and the null distributions
after block
4.2 Consistency of NN functional similarities across domains
Setup: Ideally, functional similarity between NNs would not depend on the domain which the networks are applied. Here, we study this by constructing set of 6 NNs, {fili (1,2,3,4,5, 6)].

---

(A)
(B)
(overlap:: 28.9%)
dOKA (overlap:
Proportion
0.6
RSA: 0.16
OCKA.0.23
Praportion
density
15-
5.0
dRSA: 0.23
0.0
0.0
0.7
0.8
0.9
0.4
0.6
0.8
0.1
0.2
0.3
Histogram d'm Histogram,
noise level
block
KDE,
Figure 3: (A): Proportion of identified similar NNs across 19 domains summarized for different noise levels (Left) and different NN blocks (Right). We observe that the deconfounded similarity can identify more similar models compared with the corresponding original similarity. (B): histograms and kernel density estimations (KDE) of CKA and dCKA across 19 domains on the first-block representations. We observe that dCKA can separate s s(.22, f*) from s(fa: f*) better than CKA with relative histogram overlap 13.5% and 28.9% respectively.
by adding independent Gaussian noise x (O 0.1) to each parameter of the pretrained ResNet- 18 on ImageNet, f*. Hence, the similarity s(fi, f*) should be higher than s(f+11,f") regardless of the input domain in which the the networks are applied to calculate representations. Thus, we calculate
the similarity s(fi. f*) for every f. on each domain of the corrupted CIFAR-10-4 dataset [40] that contains 19 domains with different types of corruptions to the original CIFAR-IO. We compute the
average similarity ff..f= across the 19 domains and its standard error fi."". more similar to f than to f1+1 across domains, if
We say Si is significantly
Ms.f 1.966 f.. > Pf + 1.960 f:4''""
(13)
We repeat the above experiments 20 times with different random seeds, i.e., generate 20 different sets of NNs, to measure the proportion of cases where fi is significantly more similar to f" than fi+1 for each block, as well as the confidence interval of the proportion.
Results: In Figure 3(A) A) Left, we show the proportion of identified NNs averaged over all blocks for each noise level. We observe that the deconfounded similarity improves the proportion of identified NNs compared with the corresponding original similarity for all noise levels. The averaged proportion increases 59.7% for CKA (from 0.18 to 0.29) and 43.1% for RSA (from 0.16 to 0.23) after deconfounding. We also observe that deconfounding can improve CKA/RSA on different inputs from the same domain, but the improvement is relatively smaller (23% for CKA, from 0.65 to 0.8, and 7% for RSA, from 0.75 to 0.8), shown in Appendix B Moreover, the proportion decreases as the noise level increases for all similarity measures, because for large noise level (large i), both f and f; are far from f In Figure similarities for each block where the results are averaged over different noise levels. In general, we expect to identify fewer similar NNs with deeper layer representations, because the Gaussian noise added each parameter and deeper representations consequently accumulate more noise than shallow layers. We visualize the histogram of CKA and dCKA between fz and f and between fa and fr on the first-block representations of inputs from 19 domains in Figure B),, where we can clearly observe that f2 and fa are more separable in terms of dCKA than CKA. Moreover, the relative overlap areas between two histograms are 13.5% and 28.9% for dCKA and CKA respectively.
Right, we show the the proportion of consistently identified
4.3 Transfer learning: domain similarity vs. the similarity of pretrained and finetuned NNs
Ding et al. 15 argued that the similarity metric must be sensitive to changes that affect the functionality of the networks we compare, and we extend this transfer learning under domain shift. Consider two models with the same initialization from a pretrained (PT) model which are finetuned (FT) on data from different domains. We then expect the similarity of the layer representations between the finetuned and pretrained models to be different for each target domain, and ideally it should be correlated with the similarity between the source and target domains.
Motivation:
Setup: To study this in detail for dCKA and CKA, we choose datasets from two modalities - immae and text that display such domain shift. For text, we use the Multi-lingual STS-B dataset 4] and choose English. Spanish, Polish, and Russian languages as the target domains. For images, we use the Real, Clipart, Sketch, and Quickdraw target domains from the DomainNet dataset [42)
First, we finetune separate models for each domain from both modalities. For text, we initialize a PT model trained on 100 languages where the largest size of data is in English (38 and
7

---

Table I: Rank correlation with standard error (in parentheses) between the domain similarity and the CKA between pretrained and finetuned models. We see that compared to CKA, the dCKA has higher correlation with domain similarity in terms of Spearman's p and Kendall's 7.
CKA
dCKA
Modality
p
T
NomainNet
0.675 (0.020)
0.626 (0.018)
0.751 (0.020)
0.718 (0.018)
Multi STS-B
-0.231 (0.018)
-0.185 (0.016)
0.717 (0.014)
0.641 (0.013)
Domain simiunty
UKA ostwoon and
Decorfounded beteeen and
Engltsh
Spande as4
Speniie 20
di 06
Rolon 0
Spanish
Potis
Russian
858 L850
Marter
Navcer
Figure 4: dCKA adjusts for transfer learning under domain shift. Left shows the ground truth domain similarity (between English and other languages) as measured by test binary cross entropy (BCE) loss of the cross-domain classifier. We plot the CKA (Center) and dCKA (Right) between the pretrained XLM Lo-RoBERTa model and models finetuned for different languages on the STS-B task. The average similarities across layers are shown next to the languages in the legend. We observe that dCKA is better correlated with the domain similarity than CKA.
for images, we pretrain ResNet-50 model on ImageNet 33 We compute the layer-wise CKA and dCKA between each FT model and the corresponding PT model on the test set of the target domain [22. We repeat the above procedure 10 times to construct error bars. Then, we quantify the similarity between two domains according to [43]. Specifically, we build a cross-domain classifier and create a dataset using equal number of samples from each domain. Next, we train a weak discriminator to predict the appropriate true domain for each sample. For the discriminator, we use EfficientNet-BO [45] for text. The discriminator is essentially binary classifier and thus the test binary cross entropy (BCE) loss would be an indicator of the similarity between the two domains (higher meaning more similar). Finally, we measure the Spearman's 's p and Kendall's correlation [15 between the domain similarity (measured by the BCE loss of cross-domain classifier) and the layer-wise CKA A and dCKA between PT and FT models of each domain. We average the rank correlations over all layers.
44 for images and
Results: Table summarizes the averaged rank correlations of CKA and dCKA for each modality. We can see that the dCKA is more correlated with the domain similarity as compared to CKA on both modalities. Figure Left shows the results for the test BCE loss when the cross- doms-nommann classifier was trained to discriminate between the domains on STS-B: B: English-English, Polish, and Bonglish-Russiane and we that the similarity of two English domains is higher than the similarity of English and Russian domains, as expected. The layer-wise CKA and dCKA between the pretrained and finetuned Xo.MRRaERRTa model is shown Figure 4 (Center and Right), respectively. The models finetuned on highly dissimilar domains (e.g.. English- Russian) are expected to have lower layer-wist similarity with the pretrained model. However, the CKA gives counter-intuitively high similarities between PT and FT for Russiinn, and negative rank correlations in Table [ This not because FT close to PT, but because the Russian FT dataset used calculate the CKA, which causes the representations of PT and FT to be similar on that dataset. As expected, dCKA (Figure 4 RRigh)) captures the domain similarity precisely by removing the input structure. We provide results on DomainNet in Appendix F
English-
4.4 Model generalization: in-distribution similarity vs. out-of-distribution accuracy
Setup: Here we use challenge task used in previous work [15,, which evaluates the sensitiveness of similarities to changes that affect the NNs generalizations on out-of-distribution (OOD) data, and
we would expect similar NNs to have similar OOD accuracy. We follow the same setup as [5] 1. We train 50 ResNet- 18, fú, with different random initialization on CIFAR-10; 2. Evaluate the OOD accuracy of each model on CIFAR- 10-C [4), acc(f.), and select the most accurate ResNet as the
8

---

Table 2: Rank correlation with standard error (in parentheses) between CKA and the prediction accuracy similarity of models. We observe that dCKA improves correlations significantly on average, in terms of Spearman's s and Kendall's 7, with prediction accuracy on OOD test sets.
CKA
dCKA
Corruption level
0.147 (0.004)
0.103 (0.003)
0.151 (0.004)
0.105 (0.003)
0.150 (0.004)
0.106 (0.003)
0.157 (0.004)
0.110 (0.003)
3
0.132 (0.004)
0.094 (0.002)
0.140 (0.003)
0.099 (0.003)
4
0.130 (0.003)
0.091 (0.002)
0.138 (0.003)
0.096 (0.003)
5
0.135 (0.003)
0.094 (0.002)
0.140 (0.004)
0.098 (0.003)
Average
0.139 (0.002)
0.098 (0.001)
0.145 (0.002)
0.102 (0.001)
ID accuracy
0. 0,163 (0.020)
0.116 (0.014)
0.167 (0.020)
0.118 (0.014)
(A)
Percentage improvement:
Percentage improvement:
0.15
0.15
zoom
other corraptions
0.00
Corruption types
0.00
Corruption types
CIFAR MD-658.7
Contrast ND-34.7
Zoom blur
MG-51137
Figure 5: dCKA improves CKA each corruption type. (A): percentage improvement of dCKA over CKA and corresponding standard error on each type of corruptions. (B): visualization of corruttion with largest improvement (" ('contrast', red) and smallest improvement ('zoom blur', green). We observe that 'contrast' is more different from the uncorrupted dataset compared with 'zoom blur', in terms of mean pair-wise distance (MD) of images.
reference model, f": 3. Compute the similarity between each f; and f", s(fe, f*), of each block on CIFAR-10 10 test set (in-distributic similarity). and compute the accuracy difference on CIFAR-10-C test set |acc(ff)) - acc(f*) (OOD accuracy similarity): 4. Measure the Kendall's and Spearman's 's p between 1 - (ff, f*) and aacc(fi) acc( for each block. A good similarity should have high rank correlation, meaning that the similarity in the input space correlates with OOD accuracy.
Results: CIFAR-IO-C dataset contains 19 different corruptions and 5 levels for each corruption. We average rank correlations over all blocks ResNet-18 as [15] because the ranking between similarity measures were shown to be consistent across different layers/blocks. We report the averaged rank correlation over all types of corruptions in Table - We observe that dCKA is more correlated with OOD accuracy on all 5 levels of corruption, especially on levels 3-5, compared with CKA. Moreover, we also notice marginal improvements of dCKA in terms of in distribution accuracy (ID accuracy). Figure ([A) shows the improvement of dCKA vs. CKA for each corruption averaged over 5 corruption levels, and we see that deconfounded CKA improves the most in the 'contrast" corrpption (15%) and the least in the 'zoom blur' corruption (1%). We visualize examples of original in-distribution images together with the above two corruptions Figure BJ, and we observe that 'contrast" is very different from the in-distribution 'CIFAR' images whereas 'zoom blur" is more similar to the original images visually. Moreover, we compute the mean pairwise Euclidean distances (MD) between images for each type. We find the MD of the original CIFAR test set (MD= 658.7) 7) is close to the 'zoom blur' (MD=513.77 7), and very far from the 'contrast" corruption (MD= 34.7). Hence, the benefit of dCKA CKA appear greatest when the OOD domain is least similar to the original domain, which aligns with the expectation that dCKA less domain specific due to the correction for the input domain population structure.
5 Discussion
We investigated the confounding effect of the input similarity structure on commonly used similarity measures between NNs representations. The confounder can lead to a high similarity even for completely random NNs and counter-intuitive conclusions when NNs trained in multiple domains are considered. We proposed a simple deconfounding algorithm by regressing out the input similarity from the representation similarity. The deconfounded similarity measures studied in this paper retain

---

the invariance properties of the original measures. Moreover, deconfounding significantly improves the consistency of similarities w.r.t. functionl similarities of NNs. which is especially beneficial in understanding NNs when multiple domains are involved, such as studying the closeness between the finetuned and pretrained models in transfer learning (Figure Although the deconfounded similarity does not improve models' performance directly, some insights can potentially inspire the future development of better machine learning models. For instance, in (low-resource) transfer learning, we could encourage the dCKA between the finetuned and pretrained models to be correlated with the known domain similarity during training, as we observed a high correlation between domain similarity and PT-FT similarity in Section
We further clarify that the original CKA can provide a meaningful representation similarity in applications with a single domain, such as in understanding NN trained with different initializations 19 or with different architectures [2T on the same dataset, and in these cases dCKA is expected to yield similar conclusions with CKA (Appendix G). Moreover, in specific neuroscience applications 46.. the input similarity structure can be feature that should not be removed, and dCKA is not expected to be effective. However, in applications involving multiple domains, such transfer learning, we showed that deconfounding yields more intuitive results, and we expect other applications with multiple domains data sets, such as meta learning, to benefit from the insights in this paper.
There are still a few limitations and open questions. We assumed that the confounder is linearly separable from the representation similarity in Eq and showed that adding higher-order polynomial terms cannot improve the model evidence in Appendix [ However, it still possible that the input similarity structure is not entirely additively separable from representation similarity structures, especially for deeper layers, and this may explain the fact that deconfounding similarities are more beneficial for shallow layers (e.g., Figure 2 Left and Figure 3 One possible solution is regressing out the similarity structure in the previous layer instead of the input layer, which improves detecting similar networks from random networks for deep blocks (Appendix D.2). However, this discards information from all previous layers and eventually loses the ability of representing the similarity between functional behaviors. We consider this as an open question to motivate progress on developing more functionally consistent similarity measures.
Acknowledgements
This work was supported by the Academy of Finland (Flagship programme: Finnish Center for Artificial Intelligence FCAI, and grants 292334, 315896, 336033), EU Horizon 2020 grants of European Network of AI Excellence Centres ELISE (951847) and INTERVENE (101016775), UKRI Turing AI World-Leading Researcher Fellowship (EP/W002973/1). We also acknowledge the computational resources provided by the Aalto Science-IT Project from Computer Science IT.
References
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25:1097-
1 1105, 2012.
[2] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 11.0180444444 004411 2018
[3] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lleus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on Artificial Intelligence, volume 33, pages 590-597, 2019.
[4] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: /66.:/606.00555 2016.
[5] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. European Conference Computer Vision, pages 740-755. Springer, 2014.
10

---

[6] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, In International Conference on Machine Learning. pages 3319-3328. PMLR, 2017.
[7] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the st International Conference on Neural Information Processing Systems, pages 4768 4777, 2017.
[8] Johannes Mehrer, Courtney Spoerer, Nikolaus Kriegeskorte, and Tim C Kietzmann, Individual differences among deep neural network models. Nature Communications, 1(1):1-12, 2020.
[9] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, arXiv preprint arXiv: 14/2.6550 2014.
[10] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. arXiv preprint arXiv:
2017.
[11] Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. arXiv preprint arXiv:/806.05579/ 2018.
[12] Alex Williams, Erin Kunz, Simon Kornblith, and Scott Linderman. Generalized shape metrics on neural representations. Advances in Neural Information Processing Systems, 34, 2021.
[13] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 2016.
[14] Yunzhen Feng, Runtian Zhai, Di He, Liwei Wang, and Bin Dong. Transferred discrepancy: Quantifying the difference between representations. arXiv preprint arXiv:2007. 2020.
[15] Frances Ding, Lean-Stanislar Denain, and Jacob Steinhardt. Grounding representation similarity with statistical testing. arXiv preprint arXiv:2 100.066,,0. 2021.
[16] Shimon Edelman. Representation is representation of similarities. Behavioral and Brain Sciences, 21(4):449-46 1998.
[17] Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2:4, 2008
[18] Mahdiyar Shahbazi, Ali Shirali, Hamid Aghajan, and Hamed Nili. Using distance on the Riemannian manifold to compare representations in brain and in models. Newrolmage, page 1 118271, 2021.
[19] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3522 PMLR, 2019.
[20] James V Haxby. M Ida Gobbini, Maura L Furey, Alumit Ishai, Jennifer L Schouten, and Pietro Pietrini. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293(5539):2425-- 2430, 2001.
[21] Thao Nguyen, Maithra Raghu, and Simon Korbblith. Do wide and deep networks learn the same things? Uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010. 15327, 2020.
[22] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? arXiv preprint arXiv:2008. 1687, 2020.
[23] Simon Korbblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? Advances Neural Information Processing Systems,
34, 2021.
[24] Nicolas Gonthier, Yann Gousseau, and Said Ladjal. An analysis of the transfer learning of convolutional neural networks for artistic images. In International Conference on Pattern Recognition, pages 546-561. Springer, 2021.

---

[25] Yipeng Zhang, Tyler L Hayes, and Christopher Kanan. Disentangling transfer and interference in multi-domain learning. arXiv preprint W7Xiv:210.055455, 2021.
[26] Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriaa Cherepanova, and Tom Goldstein. Unraveling meta-learning Understanding feature representations for few-shot tasks. In International Conference on Machine Learning, pages 3607-3616. PMLR, 2020.
[27] Han-Ming Wu, Yin-Jing Tien, Meng-Ru Ho, Hai-Gwo Hwu, Wen-chang Lin, Mi-Hua Tao, and Chun-houh Chen. Covariate-adjusted heatmaps for visualizing biological data via correlation decomposition. Bioinformatics, 34(20):352-3333 2018.
[28] Qi Liu, Chun Li, Valentine Wanga, and Bryan E Shepherd. adjusteee-ajjusted Spearman's 's rank correlation with probability residuals. Biometrics, 74(2):595 605, 2018.
[29] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James J DiCarlo. Performance-optinle hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 1 (23()8):661 9 8624, 2014.
[30] Maithra Raghu, Thomas Unterthiner, Simon Korbllihh, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34, 2021.
[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on Computer Vision and Puttern Recognition, pages 770-778, 2016.
[32] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[33] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on Computer Vision and Pattern Recognition, pages 248-255 Ieee, 2009.
[34] Damla Şentürk and Hans-georg Müller. Covariate adjusted correlation analysis via varying coefficient models. Scandinavian Journal of Statistics, 32(3):365-383, 2005.
[35] James Durbin and Geoffrey S Watson. Testing for serial correlation in least squares regression.
i. In Breakthroughs in Statistics, pages 237-259. Springer, 1992.
[36] Grace Chan and Andrew TA Wood. Algorithm as 312: An algorithm for simulating stationary Gaussian random fields. Applied Statistics, pages 171-181, 1997.
[37] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bemhard Schölkopf. Measuring statistical dependence with Hilbert- Schbmrt-Schhidt norms. In International Conference on Algorithmic Learning Theory, pages 63-77. Springer, 2005.
[38] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8800 04000,851,445 2020.
[39] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT press, 2016.
[40] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903. 12261, 2019.
[41] Philip May. Machine translated multilingual STS benchmark dataset. 2021.
[42] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1406-1415, 2019.
[43] Stephan Rabanser, Stephan Günnemann, and Zachary C Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. arXiv preprint arXiv:1810.11993, 2018.
12

---

[44] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105 6114. PMLR, 2019.
[45] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv, abs/1910.01 108, 2019.
[46] Nikolaus Kriegeskorte, Marieke Mur, Douglas A Ruff, Roozbeh Kiani, Jerzy Bodurka, Hossein Esteky, Keiji Tanaka, and Peter A Bandettini. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron, 60(6): 126 141, 2008.
[47] Kevin P Murphy, Machine Learning: Probabilistic Perspective. MIT press, 2012.
[48] lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018. 2018.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 2.2 to
(b) Did you describe the limitations of your work? Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
3.3 and Appendix [
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We provide the code the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix C
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cite the original papers.
(b) Did you mention the license of the assets? [Yes] See Appendix
(c) Did you include any assets either in the supplemental material URL? [Yes] We include the code the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [No] All assets are publicly available under licenses.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "lZXnVHxcZPj", "reviewer": "Reviewer_KLP6", "summary": "The paper presents an approach to improve CKA and RSA, which are the representational similarity metrics. Such metrics are used to compare representations from different layers of the same/different networks given the same list of objects.\nAuthors propose to \"regress out\"  inter-object similarity, the idea came from biostatistics.\nExperiments with comparing networks trained from different seeds, fine-tuned networks, transfer learning show that the \"deconfounded\" CKA  (dCKA) is more sensitive than regular CKA and better aligns with the intuitive notion of functional similarity.\nThe paper is well written and easy to follow.", "questions": "1. eq. (4) what does vec(*) mean? I suppose it is matrix flattening? Maybe it makes sense to explain it explicitly.\n2. The paper \"Similarity of Neural Network Representations Revisited\" describes an interesting experiment, \nwhere the goal is to find the most similar layer between two networks trained from different seeds.\nIdeally, the most similar layer should have the same number. \nIt is very interesting to evaluate your dCKA metrics for this problem.\n\n3. line 132: \"we can approximate dK_{f_1}^{m_1} and  dK_{f_1}^{m_1}\" - you repeated dK_{f_1}^{m_1} two times.\n4. in eq (3) you use an additional kernel to \"regress out\" the input similarity.  It should bw different from the original one in CKA. \nWhich kernels you used in the experiments?", "limitations": "No potential negative societal impact from this paper, in my opinion.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* interesting idea regarding deconfounding.\n* convincing experimental results from vision and text domains\n\nWeakness:\n* I wish authors explain the intuition behind deconfounding in more details. \n* some experimental details are not presented - how do you choose kernels in dCKA?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kPCJz0u1h9Z", "reviewer": "Reviewer_4p6w", "summary": "This paper aims at measuring the _functional_ similarity between two given neural networks.\nThe existing approaches naively compute some similarity measure of intermediate outputs of two neural networks, which may be largely entangled with input data, leading to spurious similarity.\nThe authors propose a method to deconfound the effect of input data by simply regressing out their effect.\nThe decounfounded similarity measures admit a good correlation with domain similarity and OOD accuracy, which are observed through experiments.", "questions": "### Questions\n\n1. In Section 2.1 (prior work), the two-staged similarity measurement method is introduced (Eqs. (1) and (2)), but it may be not that evident why the first stage (Eq. (1)) is needed. A few more explanations would be preferred.\n2. I am not sure why the invariance properties discussed in Section 3.3 are important. In the propositions, the proposed similarity measures are shown to be invariant against orthogonal transformations and isotropic scaling, which should not change the inter-example similarity. In my understanding, a good functional similarity should be independent of the inter-example similarity. If the aforementioned operations do not affect the inter-example similarity, the importance of the invariance would be questionable. Hence, I would like to see some clarification.\n\n### Suggestions\n\n1. In Eq. (2), do we need the full generality to introduce the similarity between different layers such as $m\\_1$ and $m\\_2$?\n2. (minor) At l.88, \"decounfounded\" -> \"deconfounded\"\n3. (minor) In Section 3.1, you may define $K^0$ properly right after Eq. (3).\n4. (minor) In Figure 4 (right), having the averaged similarity in the legends (like Figures 2 and 3) should be better.", "limitations": "The authors adequately addressed the limitations in the discussion. The linearity assumption is one of the biggest limitations of this work, but the proposed method still works fairly in several experiments. This work may not be relevant to any negative societal impact.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The spurious correlation between input data and functional similarity is formulated as a linear regression model. Although this simplicity may discard several more intricate structures, the deconfounded solution can be obtained by simply solving the normal equation. This formulation would be sufficient for the first step towards the deconfoundation.\n2. The proposed method basically can be applied to any type of the existing representational similarity measures by applying the deconfoundation step. Through the experiments, we can observe that adding this simple fix is beneficial to improving the performance in most cases.\n\n### Weaknesses\n\n1. The motivation to measure the representational/functional similarity could be made a little bit clearer. For example, Section 4.3 provide the experiments to observe the correlation between the functional similarity and domain similarity, and Section 4.4 to see the correlation between the functional similarity and OOD accuracy. While these results may seem excellent, we would not immediately see what such a functional similarity brings us to improve transfer learning and OOD generalization.\n2. The functional similarity is basically measured between two layers of given neural networks, not the entire neural networks. To measure the similarity between the entire networks, the authors simply average the layer-wise similarity through the experiments. Whereas this works in some situations (for example, Figure 2 tells us the averaged similarity is sufficient to distinguish random and fine-tuned ResNet-18), I have a concern when networks are deeper; most of the high-level features are similar between two networks, and hence the layer-wise averaged similarity would not be sufficient.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FtpbQqyvlzi", "reviewer": "Reviewer_pUes", "summary": "The paper investigates the confounding effect of similarity metrics used to compare neural networks on a layer-by-layer basis. The authors study the effect of this confounding effect. They argue that such measures as CKA and RSA are affected, thus, miscommunicating the functional similarity information by showing that the measures can show high similarity even for random neural networks. The paper proposes to 'deconfound' the metrics using a simple fix by regressing the input similarity structure from the representation similarity structure. Next, the authors conduct extensive experimentation to show the deconfounded measure performance under various settings. ", "questions": "I don't have many questions as mostly the paper is well-written and provides all the necessary information. Just a few comments:\n\n- In 3.3 you do not mention/link to the proofs of the proposition provided in the Appendix\n- In figure 2 x-axis label is missing. Also, the colouring of Random NN-s is a bit difficult to see\n- In 4.1 you perform a lot of experiments that include hypothesis testing and statistical significance. Have you considered multiple testing corrections?\n- In Figure 5 (B) the colours for zoom blur and contrast are mislabeled. ", "limitations": "No limitations detected", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality\nThe paper tackles an interesting problem, though, relatively narrow in scope, but still valuable for a meta or transfer learning domain. While the correction of the similarity metrics is not improving on models' performance, it can provide important insights into the inner workings of these models.   \n\nQuality\nThe paper is well-balanced and has all the necessary components: it introduces the problem and supports it with experiments, it solves the problem and provides necessary proofs without overburdening the paper. A multitude of experiments is introduced showing various aspects of the problem. The code and appendix are also provided. While some experiments are not straightforward and nested (having multiple steps in them on top of each other), I appreciate the complexity of the validation of the similarity metrics and how 'functional' similarity is difficult to define. \n\nClarity\nIt is a very well-written paper that is easy to read. It does not overcomplicate the idea, and the solution is quite elegant. I liked the experiments that provide interesting insights into neural networks' inner workings. \n\nSignificance\nA relatively simple fix of the similarity metrics correction is not necessarily a major contribution that will lead to a groundbreaking impact in ML area, but it is a solid paper that helps to address a problem for future work on models' interpretation and lead to the better model training experience. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept