Grounding Representation Similarity Through Statistical Testing
Authors: Frances Ding
Jean-Stanislas Denain
Jacob Steinhardt
Venue: NeurIPS 2021 Poster (2021)
Keywords: representation similarity
dissimilarity
metric
CKA
CCA
Orthogonal Procrustes
benchmark
representation learning
probing
deep networks

Abstract: To understand neural network behavior, recent works quantitatively compare different networks' learned representations using canonical correlation analysis (CCA), centered kernel alignment (CKA), and other dissimilarity measures. Unfortunately, these widely used measures often disagree on fundamental observations, such as whether deep networks differing only in random initialization learn similar representations. These disagreements raise the question: which, if any, of these dissimilarity measures should we believe? We provide a framework to ground this question through a concrete test: measures should have \emph{sensitivity} to changes that affect functional behavior, and \emph{specificity} against changes that do not. We quantify this through a variety of functional behaviors including probing accuracy and robustness to distribution shift, and examine changes such as varying random initialization and deleting principal components. We find that current metrics exhibit different weaknesses, note that a classical baseline performs surprisingly well, and highlight settings where all metrics appear to fail, thus providing a challenge set for further improvement.

Grounding Representation Similarity with Statistical Testing
Frances Ding,
Jean-Stanislas Denain, Jacob Steinhardt
University of California Berkeley
(frances, js_denain, jsteinhardt] Sberkeley edu
Abstract
To understand neural network behavior, recent works quantitatively compare differ- ent networks' learned representations using canonical correlation analysis (CCA), centered kernel alignment (CKA), and other dissimilarity measures. Unfortunately, these widely used measures often disagree as whether deep networks differing only in random initialization learn similar representations. These disagreements raise the question: which, if any, of these dissimilarity measures should we believe? We provide a framework to ground this question through concrete test: measures should have sensitivity changes that affect functional behavior, and specificity against changes that do not. We quantify this through variety of functional behaviors including probing accuracy and robustness to distribution shift, and examine changes such as varying random initialization and deleting principal components. We find that current metrics ex- hibit different weaknesses, note that a classical baseline performs surprisingly well, and highlight settings where all metrics appear fail, thus providing a challenge set for further improvement.
fundamental observations, such
Introduction
Understanding neural networks is not only scientifically interesting, but critical for applying deep networks in high-stakes situations. Recent work has highlighted the value of analyzing not just the final outputs of network, but also its intermediate representations [20, 29]. This has motivated the development of representation similarity measures, which can provide insight into how different training schemes, architectures, and datasets affect networks' learned representations.
number of similarity measures have been proposed, including centered kernel alignment (CKA) [13], ones based on canonical correlation analysis (CCA) [24, 30), single neuron alignment [20], vector space alignment
32], and others [2, 9, 16, 18, 21, 39]. Unfortunately, these different measures tell different stories. For instance, CKA and projection weighted CCA disagree on which layers of different networks are most similar [13]. This lack of consensus is worrying, as measures are often designed according to different and incompatible intuitive desiderata, such as whether finding one-to-one assignment, or finding few-to-one mappings, between neurons is more appropriate [20]. As community, we need well-chosen formal criteria for evaluating metrics to avoid over-reliance on intuition and the pitfalls of too many researcher degrees of freedom [17].
In this paper we view representation dissimilarity measures as implicitly answering a classification question-whether
representations are essentially similar or importantly different. Thus, in analogy to statistical testing, we can evaluate them based on their sensitivity to important change and specificity (non-responsiveness) against unimportant changes noise.
As a warm-up, we first initially consider two intuitive criteria: first, that metrics should have specificity against random initialization; and second, that they should be sensitive to deleting important principal
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

components (those that affect probing accuracy). Unfortunately, popular metrics fail at least one of these two tests. CCA is not specific random initialization noise overwhelms differences between even far-apart layers in a network (Section 3. CKA on the other hand is not sensitive, failing to detect changes in all but the top 10 principal components of a representation (Section 3.2).
We next construct quantitative benchmarks to evaluate a dissimilarity measure's quality. To move beyond our intuitive criteria, we need ground truth. For this we turn to the functional behavior of the representations we are comparing, measured through probing accuracy (an indicator of syntactic information) [4, 27, 35] and out-of-distribution performance of the model they belong to [7. 23, 25]. We then score dissimilarity measures based on their rank correlation with these measured functional differences. Overall our benchmarks contain 30,480 examples and vary representations across several axes including random seed, layer depth, and low-rank approximation (Section 4)'.
Our benchmarks confirm our two intuitive observations: subtasks that consider layer depth and principal component deletion, we measure the rank correlation with probing accuracy and find CCA and CKA lacking as the previous warm-up experiments suggested. Meanwhile, the Orthogonal Procrustes distance, a classical but often overlooked dissimilarity measure, balances gracefully between CKA and CCA and consistently performs well. This underscores the need for systematic evaluation, otherwise we may fall to recency bias that undervalues classical baselines.
Other subtasks measure correlation with OOD accuracy, motivated by the observation that random initialization sometimes has large effects on OOD performance [23]. We find that dissimilarity measures can sometimes predict OOD performance using only the in-distribution representations, but we also identify a challenge set on which none of the measures do statistically better than chance. We hope this challenge set will help measure and spur progress in the future.
2 Problem Setup: Metrics and Models
Our goal is to quantify the similarity between two different groups of neurons (usually layers). We do this by comparing how their activations behave on the same dataset. Thus for a layer with p neurons, define A RP:x2, the matrix of activations of the pl neurons n data points, to be that layer's raw representation of the data. Similarly, let B E RR2x* be a matrix of the activations of P2 neurons on the same n data points. We center and normalize these representations before computing dissimilarity, per standard practice. Specifically, for raw representation A we first subtract the mean value from each column, then divide by the Frobenius norm, to produce the normalized representation A", used in all our dissimilarity computations. In this work we study dissimilarity measures d(A", B²) that allow for quantitative comparisons of representations both within and across different networks. We colloquially refer to values of d(A", B²) as distances, although they do not necessarily satisfy the triangle inequality required of a proper metric.
We study five dissimilarity measures: centered kernel alignment (CKA), three measures derived from canonical correlation analysis (CCA), and a measure derived from the orthogonal Procrustes problem.
Centered kernel alignment (CKA) uses an inner product to quantify similarity between two representations. It is based on the idea that one can first choose kernel, compute the X kernel matrix for each representation, and then measure similarity as the alignment between these two kernel matrices. The measure of similarity thus depends one's choice of kernel; in this work we consider Linear CKA:
JBB T I
diincar CKA ( A), =
(1)
[AA |B 1F
as proposed in Kornblith et al. [13]. Other choices of kernel are also valid; we focus on Linear CKA here since Kornblith al. [13] report similar results from using either linear or RBF kernel.
Canonical correlation analysis (CCA) finds orthogonal bases (W²) w/y) for two matrices such that after projection onto w wh. the projected matrices have maximally correlated rows. For S sis P1.
I 'Code to replicate our results can be found at h/tps:://gttubb Foor instance, Raghu et al. [30] and Morcos et al. [24] do not mention it, and Komblith et al. [13] relegates it to the appendix; although Smith al. [32] does use it to analyze word embeddings and prefers it to CCA.

---

the ith canonical correlation coefficient is computed as follows:
(wi
A, w B)
P: max " " Iw
(2)
All
B
s.t. (1e') A, w A) = 0, Vj <i,
(wB B, w
B) 0, Vj <i
(3)
To transform the vector of correlation coefficients into scalar measure, two options considered previously [13] are the mean correlation coefficient, PCCA, and the mean squared correlation coefficient, RCCAA defined as follows:
airea (A, =
Σ
Pa:
d dRg (A, =1
P
(4)
P1
P1
To improve the robustness CCA, Morcos et al. [24] propose projection-weighted CCA (PWCCA) as another scalar summary of CCA:
dpwcCA (A, B) =1 E. aipi Σ, 0,
Ihe az)l
ai
where aj is the jth row of A, and h4 w' T " is the projection of A onto the hth canonical direction. We find that PWCCA performs far better than /CCA and ReCA so we focus on PWCCA in the main text, but include results on the other two measures in the appendix.
The orthogonal Procrustes problem consists of finding the left-rotation of A that is closest to B in Frobenius norm, i.c. solving the optimization problem:
min B RAIP: subject to R R RR =1
(6)
The minimum is the squared orthogonal Procrustes distance between A and B, and is equal to dproc( (A, B) = ||A|| + |B 2 Bll,,
where || ||. is the nuclear norm [31]. Unlike the other metrics, the orthogonal Procrustes distance is not normalized between 0 and 1, although for normalized A", B* it lies in [0,
2.1 Models we study
In this work we study representations of both text and image inputs. For text, we investigate representations computed by Transformer architectures in the BERT model family [8] on sentences from the Multigenre Natural Language Inference (MNLI) dataset [40]. We study BERT models of two sizes: BERT base, with 12 hidden layers of 768 neurons, and BERT medium, with 8 hidden layers of 512 neurons. We use the same architectures in the open source BERT release?. but to generate diversity we study 3 variations of these models:
1. 10 BERT base models pretrained with different random seeds but not finetuned for particular tasks, released by Zhong et al. [41]-
2. 10 BERT medium models initialized from pretrained models released by Zhong et al. [41], that further finetuned on MNLI with 10 different finetuning seeds (100 models total),
3. 100 BERT base models that were initialized from the pretrained BERT model in [8] and finetuned on MNLI with different seeds, released by McCoy et al. [23]'.
For images, we investigate representations computed by ResNets [11] on CIFAR-10 test set images [14]. We train 100 ResNet-14 m from random initialization with different seeds the CIFAR-10 training set and collect representations after each convolutional layer.
Further training details, as well checks that our training protocols result in models with comparable performance to the original model releases, can be found in Appendix A.
availlabl at https: /ttps://giiee http://hhhh...ooooooooeeeeeerree
'available at http:////gttuu.ooo.aaaaaaaaaaaaa
Savailable at
frrmm httpss//github. httpat/w....ttaccoooooo/////ss/e

---

0.4
Modell Layer
Model A Layer
0.6
0
Model A
02
9
0.2
11
Layer
Model
Model
Model
Model B
Model
Model
Distnce to Model A Layer
Model B
Distonna to
Model A Layer
Model 8
Distance to
Model A Layer
0..4
Model Layer
0.1
0.1
0.2
91
11
Layer
Layer
Layer
Model A
Model
Model
Model
Model
Model
(a) PWCCA distances
(b) CKA distances
(c) Orthogonal Procrustes distances
Figure 1: PWCCA fails the intuitive specificity test. Top: PWCCA, CKA, and Orthogonal
Procrustes pairwise distances between each layer of two differently initialized networks (Model A and B). Bottom: We zoom to analyze the 7th layer of Model A. plotting this layer's distance to every other layer in both networks; the dashed line indicates the distance to the corresponding 7th layer in Model B. For PWCCA, none of the distances in model exceed this line, indicating that random initialization affects this distance more than large changes in layer depth.
3 Warm-up: Intuitive Tests for Sensitivity and Specificity
When designing dissimilarity measures, researchers usually consider invariants that these measures should not be sensitive [13]; for example, symmetries in neural networks imply that permuting the neurons in a fully connected layer does not change the representations learned. We take this one step further and frame dissimilarity measures answering whether representations the same, or importantly different. We can then evaluate measures based on whether they respond to important changes (sensitivity) while ignoring changes that don't matter (specificity).
essentially
Assessing sensitivity and specificity requires a ground which representations are truly different? To answer this, we begin with the following two intuitions?: 1) neural network representations trained on the same data but from different random initializations are similar, and 2) representations lose deleted. These motivate the following intuitive
crucial information tests specificity and sensitivity: we expect a dissimilarity measure to: 1) assign a small distance between architecturally identical neural networks that only differ in initialization seed, and 2) assign large distance between representation A and the representation Á after deleting important principal components (enough to affect accuracy). We will see that PWCCA fails the first test (specificity), while CKA fails the second (sensitivity).
principal components
3.1 Specificity against changes to random seed
Neural networks with the same architecture trained from different random initializations show many similarities, such as highly correlated predictions on in-distribution data points [23]. Thus seems natural to expect good similarity measure to assign small distances between architecturally corresponding layers networks that are identical except for initialization seed.
To check this property, we take two BERT base models pre-trained with different random seeds and, for every layer in the first model, compute its dissimilarity to every layer in both the first and second model. We do this for 5 separate pairs of models and average the results. To pass the intuitive specificity test, a dissimilarity measure should assign relatively small distances between a layer the first network and its corresponding layer in the second network.
Figure displays the average pair-wise PWCCA. CKA, and Orthogonal Procrustes distances between layers of two networks differing only in random seed. According to PWCCA, these networks' representations are quite dissimilar; for instance, the two layer representations are further apart
Note will see later that these intuitions need refinement.

---

OKA
1.50
CKA
Metric dissimilarity
Procrustes
Metric dissimilarity
1.25
Procrustes
OKA
1.00
PWVCCA between random seeds
1.00
PWCCA between random seeds
seeds
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Fraction of principal components deleted
Fraction of principal components deleted
(a) First layer of BERT
(b) Last layer of BERT
Figure 2: CKA fails to be sensitive to all but the largest principal components. We compute dissimilarities between layer's representation and low-rank approximations that representation obtained by deleting principal components, starting from the smallest (solid lines). We also compute the average distance between networks trained with different random seeds as a baseline (dotted line), and mark the intersection point with star. The starred points indicate that CKA requires almost all the components be deleted before CKA distance exceeds the baseline.
than they are from any other layer in the same network. PWCCA is thus not specific against random initialization, it outweigh even large changes in layer depth.
In contrast, CKA can separate layer in a different network from layers or 10 the same network, showing better specificity to random initialization. Orthogonal Procrustes exhibits smaller but non-trivial specificity, distinguishing layers once they are 4-5 layers apart.
3.2 Sensitivity to removing principal components
Dissimilarity measures should also sensitive to deleting important principal components representation." 8 To quantify which components are important, we fix a layer of a pre-trained BERT base model and measure how probing accuracy degrades as principal components are deleted (starting from the smallest component), since probing accuracy is common measure of the information captured in a representation [4]. We probe linear classification performance on the Stanford Sentiment Tree Bank task (SST-2) [33]. following the experimental protocol in Tamkin al. [34]. Figure 3b shows how probing accuracy degrades with component deletion. Ideally, dissimilarity measures should be large by the time probing accuracy has decreased substantially.
To assess whether dissimilarity measure large, we need a baseline to compare to. For each measure, we define dissimilarity score to be above the detectable threshold if it is larger than the dissimilarity score between networks with different random initialization. Figure plots dissimilarity induced by deleting principal components, as well as this baseline.
For the last layer of BERT, CKA requires 97% of representation's principal components to be deleted for the dissimilarity to be detectable; after deleting these components, probing accuracy shown in Figure 3b drops significantly from 80% to 63% (chance 50%). CKA thus fails detect large accuracy drops and so fails our intuitive sensitivity test.
Other metrics perform better: Orthogonal Procrustes's detection threshold is ~85% of the principal components, corresponding to an accuracy drop 80% to 70%. PWCCA's threshold is ~55% of principal components, corresponding an accuracy drop from 80% to 75%.
PWCCA's failure of specificity and CKA's failure sensitivity on these intuitive tests are worrying. However, before declaring definitive failure, in the next section, we turn to making our assessments more rigorous.
For representation A. define À_ k. the result of deleting the & smallest principal components from A. follows: compute singular value decomposition UEV construct U_K E R by dropping the lowest & singular vectors of U, and finally take Á- UT
5

---

4 Rigorously Evaluating Dissimilarity Metrics
In the previous section, we saw that CKA and PWCCA each failed intuitive tests, based on sensitivity to principal components and specificity to random initialization. However, these were based primarily on intuitive, qualitative desiderata. there some way for us make these tests more rigorous and quantitative?
First consider the intuitive layer specificity test (Section 3. which revealed that random initialization affects PWCCA more than large changes in layer depth. To justify why this is undesirable, we can turn to probing accuracy, which is strongly affected by layer depth, and only weakly affected by random seed (Figure 3a). This suggests a path forward: we can ground the layer test in the concrete differences in functionality captured by the probe.
More generally, we want metries to be sensitive to changes that affect functionality, while ignoring those that don't. This motivates the following general procedure, given distance metric and functionality f (which assigns a real number to a given representation):
1. Collect set S of representations that differ along or axes of interest (e.g. layer depth, random seed).
2. Choose a reference representation A S. When f an accuracy metric, it reasonable to choose A = arg max AES f (A)."
3. For every representation B E S:
Compute [f(A) - fB))
Compute d(A, B)
4. Report the rank correlation between If(A) f(B) and d(A, B) (measured by Kendall's Spearman p).
The above procedure provides a quantitative measure of how well the distance metric d responds to the functionality f. For instance, in the layer specificity test, since depth affects probing accuracy strongly while random seed affects only weakly, a dissimilarity measure with high rank correlation will be strongly responsive to layer depth and weakly responsive to seed; thus rank correlation quantitatively formalizes the test from Section 3.1.
Correlation metries also capture properties that our intuition might miss. For instance, Figure 3a shows that some variation in random seed actually does affect accuracy, and our procedure rewards metrics that pick up on this, while the intuitive sensitivity test would penalize them.
Our procedure requires choosing a collection of models S; the crucial feature of S is that it contains models with diverse behavior according to f. Different sets S, combined with a functional differ- ence J, can be thought of as miniature "benchmarks" that surface complementary perspectives on dissimilarity measures' responsiveness to that functional difference, In the rest of this section, we instantiate this quantitative benchmark for several choices of and S, starting with the layer and principal component tests from Section 3 and continuing on to several tests of OOD performance.
The overall results summarized in Table 1. Note that for any single benchmark, correlation coefficients to be significantly lower than 1, since the metric D must capture all important axes of variation while measures only one type of functionality. A good metric is one that has consistently high correlation across many different functional measures.
expect the
Benchmark 1: Layer depth. We turn the layer test into benchmark for both text and images. For the text setting, we construct a set S of 120 representations by pretraining 10 BERT base models with different initialization seeds and including each of the 12 BERT layers We separately consider two functionalities f: probing accuracy on QNLI [37] and SST-2 [33]. To compute the rank correlation, we take the reference representation A be the representation with highest probing accuracy. We compute the Kendall's r and Spearman's 's rank correlations between the dissimilarities and the probing accuracy differences and report the results in Table 1.
representation.
"Choosing the highest accuracy model as reference makes more likely that as accuracy changes, models are on average becoming more dissimilar. A low accuracy model may be on the "periphery" of model space, where it is dissimilar to models with high accuracy, but potentially even more dissimilar to other low accuracy models that make different mistakes.

---

Probing accuracy on QNLI
Mean across random seeds
Probing accuracy on SST-2
0.80
0.73
0.55
0.50
Moan across random seads
100
200
300
400
500
700 750
Depth probed layer
Number principal directions deleted
(a) Probing variation across BERT base layers
(b) Variation across BERT base PC delctions
CIFAR 10C: contrast
Accuracy on stress tests
0.425
STRESS, NUMERICAL STRESS, STRESS_ANTONYYY ANTONYMY
0.400
0.00
0.375
0.55
0.350
Diffeeett
fineturing weeds
0:300-
0.45
0.275
0:250
0.25 0.250 0.275
0.100
0,325 0,350 0,375 0.400
10
Accuracy on CIFAR-10C: gaussian noise
Pretraining seed
(c) Variation across ResNet-14 14 training seeds
(d) Variation across BERT medium training seeds
Figure 3: Our perturbations induce substantial variation on probing tasks and stress tests: (3a): Changing the depth of the examined BERT base layer strongly affects probing accuracy on QNLI. The trend for each randomly initialized model is displayed semi-transparently. and the solid black line is the mean trend. (3b): Truncating principal components from pretrained BERT base significantly degrades probing accuracy on SST-2 (BERT layer shown here). (3c): Training ResNet- 14 on CIFAR- 10 with different seeds leads to variation in accuracies on CIFAR-106 corruptions (here Gaussian noise and contrast). (3d): Pretraining and finetuning BERT medium with 10 different pretraining seeds and 10 different finetuning seeds per pretrained model leads variation in accuracies on the Antonymy (yellow scatter points) and Numerical (blue scatter points) stress tests [25].
For the image setting, we similarly construct a S of 70 representations by training 5 ResNet- 14 models with different initialization seeds and including each of the 14 layers' representations. We also consider two functionalities f for these vision models: probing accuracy on CIFAR-100 [14] and on SVHN [26], and compute rank correlations in the same way.
We find that PWCCA has lower rank correlations compared CKA and Procrustes for both language probing tasks. This corroborates the intuitive specificity test (Section 3.1), suggesting that PWCCA registers large of dissimilarity across random initializations. For the vision tasks, CKA and Procrustes achieve similar rank correlations, while PWCCA cannot be computed because n <
Benchmark Principal component (PC) deletion. We next quantify the PC deletion test from Section 3.2, by constructing a set S of representations that vary in both random ini- tialization and fraction of principal components deleted. We pretrain 10 BERT base mod- els with different initializations, and for each pretrained model we obtain 14 different rep- resentations by deleting that representation's k smallest principal components, with (0, 100, 200, 300, 400, 500, 600, 650, 700, 725, 750, 758, 763, 767). Thus S has 10 x 14 140 elements. The representations themselves are the layer-& activations, for E E (8, 1889,.... 12). so there are 5 different choices of S. We use SST-2 probing accuracy the functionality of interest and select the reference representation A as the element in S with highest accuracy. Rank correlation
10 Farller layers have near-chance accuracy on probing tasks, ignore them.
7

---

Table 1:
Summary of rank correlation results. For Benchmarks #1-3 in both language and vision, all dissimilarity measures successfully achieve significant positive rank correlation with the functionality of both CKA and PWCCA dominate certain benchmarks and fall behind on others, while Procrustes is more consistent and often close to the leader. Benchmark #4 is more challenging. and no dissimilarity measure achieves high correlation. The vision experiments do not have results for PWCCA because n < d.
Perturbation
Subtask Size
Functionality
Procrustes
CKA
PWCCA
P
T
Modality: Language
Pretraining seed,
120
Probe: QNLI
0.862
0.670
0.876
0.685
0.763
0,564
layer depth
120
Probe: SST-2
0.890
0.707
0.905
0.732
0.829
0.637
Pretraining seed, PC deletion
2
140 x
Probe: SST-2
0.860
0.677
0.751
0.564
0.870
0.690
OOD: HANS
Finetuning seed
100 x 12
0.551
0.398
0.462
0.329
0.568
0.412
Lexical non-entailed
OOD: Antonymy stress test
100 x 8
0.243
0.178
0.227
0.160
0.204
0.152
Pretraining and finetuning seeds
OOD: Numerical stress test
100 x
0.071
0.049
0.122
0.084
0.031
0.023
Total (language)
3740
Average
0.580
0.447
0.557
0.426
0.544
0.413
Modality: Vision
Training seed,
70
Probe: CIFAR-100
0.485
0.376
0.507
0.359
layer depth
70
Probe: SVHN
0.363
0.272
0.372
0.255
Training seed
1900 x
OOD: CIFAR-10C
0.060
0.057
0.041
0.038
Total (vision)
26740
Average
0.303
0.235
0.307
0.217
results are consistent across the 5 choices of (Appendix C), so we report the average as summary statistic in Table 1.
We find that PWCCA has the highest rank correlation between dissimilarity and probing accuracy, followed by Procrustes, and distantly followed by CKA. This corroborates the intuitive observations from Section 3.2 that CKA is not sensitive principal component deletion.
4.1 Investigating variation in OOD performance across random seeds
So far our benchmarks have been based on probing accuracy, which only measures in-distribution behavior (the train and test set of the probe are typically i.i.d.). In addition, the BERT models were always pretrained on language modeling but not finetuned for classification. To add diversity to our benchmarks, we next consider the -iitt----ttttttioon performance of language and vision models trained for classification tasks.
Benchmark 3: Changing fine-tuning seeds. McCoy et al. [23] show that single pretrained BERT base model finetuned MNLI with different random initializations will produce models with similar in-distribution performance, but widely variable performance on out-of-distrinnn off-istribution data. We thus create a benchmark S out of McCoy et al.'s 100 released fine-tuned models, using OOD accuracy on the "Lexical Heuristic (Non-entailmentt" subset of the HANS dataset [22] as our functionality f. This functionality is associated with the entire model, rather than an individual layer (in contrast to the probing functionality), but consider one layer time measure whether dissimilarities

---

between representations at that layer correlate with f. This allows us to also localize whether certain layers are more predictive of f.
We construct 12 different S (one for each of the 12 layers of BERT base), taking the reference representation A to be that of the highest accuracy model according f. As before, report each dissimilarity measure's 's rank correlation with f in Table I, averaged over the 12 runs.
All three dissimilarity measures correlate with OOD accuracy, with Orthogonal Procrustes and PWCCA being more correlated than CKA. Since the representations in our benchmarks were computed on in-distribution MNLI data, this has the interesting implication that dissimilarity measures can detect OOD differences without access to OOD data. It also implies that random initialization leads to meaningful functional differences that are picked up by these measures, especially Procrustes and PWCCA. Contrast this with our intuitive specificity test in Section 3.1, where all sensitivity to random initialization was seen as shortcoming. Our more quantitative benchmark here suggests that some of that sensitivity tracks true functionality.
To check that the differences in rank correlation for Procrustes, PWCCA, and CKA are statistically sigmificant, we compute bootstrap estimates of their 95% confidence intervals. With 2000 bootstrapped samples, we find statistically significant differences between all pairs of measures for most choices of layer depth S, so we conclude PWCCA > Orthogonal Procrustes > CKA (the full results are in Appendix E). We do not apply this procedure for the previous two benchmarks, because the different models have correlated randomness and so any p-value based on independence assumptions would be invalid.
Benchmark 4: Challenge sets: Changing pretraining and fine-tuning seeds. We also construct benchmarks using models trained from scratch with different random seeds (for language, this is pretraining and fine-tuning, and for vision, this is standard training). For language, construct benchmarks from a collection of 100 BERT medium models, trained with all combinations of 10 pretraining and 10 fine-tuning seeds. The models are fine-tuned on MNLI, and consider two different functionalities of interest f: accuracy on the OOD Antonymy stress test and on the OOD Numerical stress test [25], which both show significant variation accuracy across models (see Figure 3d). We obtain 8 different sets S (one for each of the 8 layer depths in BERT medium), again taking A be the representation of the highest-accuracy model according to f. Rank correlations for each dissimilarity measure are averaged over the 8 runs and reported in Table 1.
For vision, construct benchmarks from collection of 100 ResNet- 14 models, trained with different random seeds on CIFAR-10. We consider 19 different functionalities of interest- 19 types of corruptions in the CIFAR-10C dataset [12], which show significant variation in accuracy across models (see Figure 3c). We obtain 14 different sets S (one for each of the 14 layers), taking A to be the representation of the highest-accuracy model according f. Rank correlations for each dissimilarity measure are averaged over the 14 runs and over the 19 corruption types and reported in Table Results for each of the 19 corruptions individually can be found in Appendix D..
None of the dissimilarity measures show large rank correlation for either the language or vision tasks, and for the Numerical stress test, at most layers, the associated p-values (assuming independence) are non non-significant at the 0.05 level (see Appendix C). Thus we conclude that all measures fail to be sensitive to OOD accuracy in these settings. One reason for this could be that there is less variation in the OOD accuracies compared to the previous experiment with the HANS dataset (there accuracies varied from 0 to nearly 60%). Another reason could be that it is harder to correctly account for both pretraining and fine-tuning variation at the same time. Either way, we hope that future dissimilarity measures can improve upon these results, and we present this benchmark as challenge task to motivate progress.
5 Discussion
In this work we proposed quantitative measure for evaluating similarity metries, based on the rank correlation with functional behavior. Using this, we generated tasks motivated by sensitivity to
See Appendix C for p-auuues as produced by sci-kit leam. Strictly speaking, the p-values are invalid because they assume independence. but the pretraining seed induces correlations. However, correctly accounting for these would tend to make the pavuues larger, thus preserving our conclusion of non-significance.

---

deleting important directions, specificity to random initialization, and sensitivity to performance. Popular existing metrics such as CKA and CCA often performed poorly on these tasks, sometimes in striking ways. Meanwhile, the classical Orthogonal Procrustes transform attained consistently good performance.
Given the success of Orthogonal Procrustes, it is worth reflecting on how it differs from the other metrics and why it might perform well. To do so, we consider simplified case where A and B have the same singular vectors but different singular values. Thus without loss of generality A = A and B Ag. where the A, are both diagonal. In this case, the Orthogonal Procrustes distance reduces to ||A - Azl1? ||?-, or the sum of the squared distances between the singular values. We will see that both CCA and CKA reduce less reasonable formulae in this case.
Orthogonal Procrustes vs. CCA. All three metrics derived from CCA assign zero distance even when the (non-zero) singular values are arbitrarily different. This is because CCA correlation coefficients are invariant to all invertible linear transformations. This invariance property may help explain why CCA metries generally find layers within the same network to be much more similar than networks trained with different randomness. Random initialization introduces noise, particularly in unimportant principal components, while representations within the same network more easily preserve these components, and CCA may place too much weight on their associated correlation coefficients.
Orthogonal Procrustes CKA. In contrast to the squared distance of Orthogonal Procrustes, CKA actually reduces to a quartic function based on the dot products between the squared entries of A1 and 22. As consequence, CKA is dominated by representations' largest singular values, leaving it insensitive to meaningful differences in smaller singular values as illustrated in Figure 2. This lack of sensitivity moderate-sized differences may help explain why CKA fails to track diitr--distrbbution error effectively.
In addition to helping understand similarity measures, our benchmarks pinpoint directions for improvement. No method was sensitive to accuracy on the Numerical stress test in our challenge set, possibly due lower signal-to-ne -oisee ratio. Since Orthogonal Procrustes performed well on most of our tasks, it could be a promising foundation for a new measure, and recent work shows how to regularize Orthogonal Procrustes to handle high noise [28]. Perhaps similar techniques could be adapted here.
An alternative to our benchmarking approach is to directly define two representations' dissimilarity as their difference in a functional behavior of interest. Feng et al. [9] take this approach, defining dissimilarity difference in accuracy on handful of probing tasks. One drawback of this approach is that small set of probes may not capture all the differences in representations, so it is useful to base dissimilarity measures on representations' intrinsic properties. Intrinsically defined dissimilarities also have the potential to highlight new functional behaviors, as we found that representations with similar didiiiribtttion probing accuracy often have highly variable OOD accuracy.
A limitation of our work is that we only consider a handful of model variations and functional behaviors, and restricting our attention to these settings could overlook other important considerations. To address this, we envision a paradigm in which a rich tapestry of benchmarks are used to ground and validate neural network interpretations. Other axes of variation in models could include training on more or fewer examples, training on shuffled labels vs. real labels, training from specifically chosen initializations [10], and using different architectures. Other functional behaviors to examine could include modularity and meta-learning capabilities. Benchmarks could also be applied to other interpretability tools beyond dissimilarity. For example, sensitivity to deleting principal components could provide an additional sanity check for saliency maps and other visualization tools [1].
More broadly, many interpretability tools are designed as audits of models, although it is often unclear what characteristics of the models are consistently audited. We position this work as a counter-audit, where by collecting models that differ in functional behavior, we can assess whether the interpretability tools CKA, PWCCA, etc., accurately reflect the behavioral differences. Many other types of counter-audits may be designed to assess other interpretability tools. For example, models that have backdoors built into them to misclassify certain inputs provide counter-audits for interpretability tools that explain model predictions- these explanations should reflect any backdoors present [5, 15, 19, 38). We are hopeful that more comprehensive checks on interpretability tools will provide deeper understanding of neural networks, and more reliable models.
10

---

Acknowledgments and Disclosure of Funding
Thanks to Ruiqi Zhong for helpful comments and assistance in finetuning models, and thanks to Daniel Rothchild and anonymous reviewers for helpful discussion. FD is supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 1752814 14 and the Open Philanthropy Project AI Fellows Program. JSD is supported by the NSF Division of Mathematical Sciences Grant No. 2031985.
References
[1] Adebayo, J. Gilmer, M. Muelly. Goodfellow, M. Hardt, and B. Kim. Sanity checks for saliency maps. Advances in Neural Information Processing Systems, 31:9505-9515 2018.
[2] G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes, 2018.
[3] S. Arora, Y. Liang, and T. Ma. A simple but tough-to-beat baseline for sentence embeddings. 2017.
[4] Y. Belinkov, N. Durrani, F. Dalvi, H. Sajjad, and J. Glass. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 861-872, 2017.
[5] X. Chen, C. Liu, B. Li, K. Lu, and D. Song. Targeted backdoor attacks deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017.
[6] A. Conneau, G. Lample, M. Ranzato, L. Denoyer, and H. Jégou. Word translation without parallel data, 2018.
[7] A. D' O'mour K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, Deaton,
J. Eisenstein, M. D. Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:201 1.22001110335.. 2020.
[8] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint wwXXv:18100048800 2018
[9] Y. Feng, R. Zhai, D. He, Wang. and B. Dong. Transferred discrepancy: Quantifying the difference between representations. arXiv preprint arXiv:2007.124466 2020.
[10] J. Frankle and M. Carbin, The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference Learning Representations, 2018.
[11] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770- 778, 2016.
[12] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representa- tions, 2019.
[13] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529, 2019.
[14] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[15] K. Kurita, P. Michel, and G. Neubig. Weight poisoning attacks on pre-trained models. arXiv preprint arXiv:2004. 06660, 2020.
[16] A. Laakso and G. Cottrell. Content and cluster analysis: assessing representational similarity in neural systems. Philosophical psychology, 13(1):47-76, 2000.
[17] M. L. Leavitt and A. Morcos. Towards falsifiable interpretability research. arXiv preprint arXiv:2010. 12016, 2020.
[18] K. Lene and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence, 2015.
[19] S. Li, S. Ma, M. Xue, and 7. H. Zhao. Deep learning backdoors. arXiv preprint arXiv:2007.08273. 2020.

---

[20] Y. Li, J. Yosinski, J. Clune, H. Lipson, and J. Hoperoft. Convergent learning: Do different neural networks learn the same representations? In Feature Extraction: Modern Questions and Challenges, pages 196-212. PMLR, 2015.
[21] R. Liang, T. Li, L.. Li, J. Wang, and Q. Zhang. Knowledge consistency between neural networks and beyond. In International Conference on Learning Representations, 2019.
[22] R. T. McCoy, E. Pavlick, and T. Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint
2019.
[23] R. T. McCoy, J. Min, and T. Linzen. Berts of feather do not generalize together: Large variability in generalization across models with similar test set performance. In Proceedings of the Third BlackboxNI.P Workshop on Analyzing and Interpreting Neural Networks for NI.P, pages 217-227, 2020.
[24] A. Morcos, M. Raghu, and S. Bengio. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems, pages 5727-5736, 2018.
[25] A. Naik, A. Ravichander, N. Sadeh, C. Rose, and G. Neubig. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340-2353, 2018.
[26] Y. Netzer, T. Wang. A. Coates, A. Bissacco, B. Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[27] M. F. Peters, M. Neumann, I.. Zettlemoyer, and W.-t Yih. Dissecting contextual word embed- dings: Architecture and representation. arXiv preprint arXiv: 08949, 2018.
[28] T. Pumir, A. Singer, and N. Boumal. The generalized orthegonal procrustes problem in the high noise regime. Information and Inference: A Journal of the IMA, Jan 2021. ISSN 2049-8772. doi: 10. 10109//maia//iaa003. URL http: //dx. ht//xxxxoiii.. coi.org//i 10. 1et9/00mamaaa hta.aiaaaaaa/////aa//iia/iaaiiia imaiammaaaa03...... 1aaa035.
[29] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards under- standing the effectiveness of maml. In International Conference on Learning Representations, 2019.
[30] M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pages 6076-6085, 2017.
[31] P. H. Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 3 31(1):1-10, 1966.
[32] S. L. Smith, D. H. P. Turban, S. Hamblin, and N. Y. Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax, 2017.
[33] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642. 2013. [34] A. Tamkin, T. Singh, D. Giovanardi, and N. Goodman. Investigating transferability in pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Findings, pages
Language Processing:
2020.
[35] I. Tenney, P. Xia, B. Chen, A. Wang. A. Poliak, R. T. McCoy, N. Kim, B. V. Durme, S. Bowman, D. Das, and E. Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations, 2019. URL https: / opwnreview.eeeeeeeo net. htt////uummnddddssmiiiee
[36] I. Ture, M.-W. Chang, K. Lee, and K. Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint 1rXiv:1908.08962 2019.
[37] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. Glue: A multi-tase task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, 2018.
[38] B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, and B. Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEFE Symposium on Security and Privacy (SP), pages 707-723. IEEE, 2019.
12

---

[39] L. Wang, L. Hu, J. Gu, Z. Hu, Y. Wu, K. He, and J. Hoperoft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Resa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, pages 9584-9593. Curran Associates, Inc., 2018. URL https::/ppoceeeeeeeees http://roceed..s...........ppppp cc/paper/2018, eef.feeeee/acclaa8aa011919eeeee0
pdf.
[40] A. Williams, N. Nangia, and S. Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long Papers), pages 1112-1122, Association for Computational Linguistics, 201 8. URL BoLLhttp//acleee org/ werg/anthology////o 1aroggyyN/1110110001011111111111 1101.
[41] R. Zhong, D. Ghosh, D. Klein, and J. Steinhardt. Are larger pretrained language models uni- formly better? comparing performance at the instance level. arXiv preprini arXiv: 17Xiv:2105.06020 2021.
13


Reviews:
[{"review_id": "_IpTrDNdMGf", "reviewer": "Reviewer_swz6", "summary": "The authors present a number of benchmarks and stress tests for similarity measures of learned representations of neural networks. Three similarity metrics are compared for their sensitivity and specificity of the similarity measure under a number of conditions, such as out-of-domain data or low rank approximations of hidden layer activation. Empirical results suggest that there is no clear trend for one or the other similarity metric that would be robust against all types of tests proposed.  \n\n**Update**\nAfter reading the authors responses I found that they addressed most of my concerns sufficiently. My main concern was that CCA a) is not evaluated with cross-validation and instead canonical correlations are computed on training data and b) that the CCA projections are not regularized. These two points together can lead to overfitted CCA projections and biased estimates of the canonical correlations, especially in the settings addressed in the present study. This could explain the poor results of CCA in the comparisons. Th\n\nThe authors responded that this is how the PWCCA method was applied and evaluated and the overfitted and not-cross-validated canonical correlations are a 'feature' of PWCCA. I wouldn't agree to that personally, but I think it's a valid point -- after all many researchers do not evaluate unsupervised methods on held-out test data, so there's probably some value in examining and comparing how these (wrong and biased) metrics computed on training data behave. \n\nSo I think it's a valuable contribution and I increased my score accordingly. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\n**Originality**: \n\nThe probing tasks proposed are novel to the best of my knowledge, however they are a combination of previously proposed methods. It’s an important contribution to evaluate these different methods, and the idea to compare these methods under a variety of different conditions is also interesting. \n\n**Quality**: \n\nThe submission is technically sound for most parts, but there are some technical details and  some motivations/intuitions that could be made clearer. \n\nFor instance in line 99 the authors write that \n\n*We find that PWCCA performs far better than …*, here it would be helpful to see what performance is measured. \n\nIf canonical correlations are not “working right” it could also be due to how they are evaluated. With high dimensional low rank data sets (and some large layers might be of that kind), CCA requires regularisation, and it makes sense to avoid overfitted canonical correlations (ccs) by evaluating the ccs on held-out data, just like with any supervised model. In line 302 the authors also suggest that CCA might have overfitted, but if it’s really overfitted it would not result in high held-out ccs. So both the hyper parameter optimisation for the CCA regularisation parameter as well as the evaluation on held out data might change the CCA results and I would assume the ccs are more trustworthy when cross-validated. \nThat said, the other alternative similarity measures are probably more attractive as they do not require fitting parameters. \n\nThe idea with most probing tests makes a lot of sense, but I’m not sure I understand what exactly is measured in the experiments with removal of PC components, or rather how that measure would help to understand something about representations - it's hard to relate that transformation to something that happens to networks in applications. A minor general remark on the selection of components, I think it could be helpful to discard components based on the amount of variance they explain, not their absolute number. \n\nIn general it seems the authors often use the term “performance” and it is not really clear what is meant by that. It seems the methods are better or worse on different tasks, but it could be better explained what exactly is measured across all tasks, and what we can conclude from the results, beyond the observations for the single experiments. \n\nAlso there are number of packages for perturbations and augmentations that would be worth integrating into these tests. \n\n**Clarity**: \n\nThe manuscript is well written and structured, but there could be a bit more structure in the interpretation of results or the experimental design. I was asking myself what the results imply and what specificity and sensitivity results tell us about the networks if the probing tasks are so specific and lead to different results for most similarity metrics. Maybe one could evaluate just robustness of the similarity metrics under perturbations. Or maybe there is a way to add more probing tests or modify the metric to see more consistent differences between similarity metrics. In the end one is interested in things like which metric should I use to evaluate which aspect. \n\n\n**Significance**: \n\nDeveloping tests to evaluate is a relevant research direction and the tests proposed in this work are an important contribution. There are some aspects about the evaluation that could be improved and the heterogeneity of the empirical results suggest that it could be difficult to draw general conclusions about neural representations. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ZfJ6NbDTGnB", "reviewer": "Reviewer_DsGH", "summary": "This paper evaluates recent efforts to analyze learn representations obtained by neural networks using canonical correlation analysis (CCA), centered kernel alignment (CKA), and orthogonal Procrustes distance. The authors pointed out on some inconsistencies across these metrics and suggested a method to measure of how well the distance metric is with respect to some functionality.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written, easy to follow, and the findings regarding CCA, CKA, and orthogonal Procrustes are interesting and valuable.\nI'm a little bit puzzled by the quantitative measure and its contribution. \nIf I understand it correctly, the authors first claim that the current metrics are not consistent concerning sensitivity and specificity. The authors demonstrated it with an example. Then, the authors presented a different method, grounded by some downstream task, using the same example to assess the same thing. Can we use this method to better understand representations similarities or only to compare different distance metrics? \n\nOverall, the method and analysis is interesting. I have some questions to the authors: \n1) I agree with the authors that according to Figure 1 two different networks’ representations at layer 7 have higher PWCCA distance than that between layer 7 and any other layer within the same network. However, we still see the same behavior within different layers for different models, meaning the distance is the smallest to layer 7 than any other layer. Can the authors say something about that? Maybe this is a normalization issue? Will it be different if we do it with randomly initialized network / network that was trained for other task? \n2) In line 257-259 the authors wrote: \"The representations were computed on in-distribution MNLI data, meaning that the dissimilarity measures can detect OOD differences without access to OOD data\". However, the authors did use OOD data (HANS) to get accuracies and learn f, no? Am I missing something? \n3) The authors mentioned that they trained several models for each setting and averaged the results, can the authors also share the STDs / variance?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations: See above.\nSocietal impact: I do not see any.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-5 hours"}, "extra_text": {}}, {"review_id": "BR33deqzUu", "reviewer": "Reviewer_L7iZ", "summary": "This paper focuses on giving a consistent meaning to (dis)similarity metrics among neural networks based on representations out of intermediate layers. \n\nThis consistent meaning is given by comparing the ordering that a dissimilarity measure gives to a \"functional behavior\". The functional behavior is defined as any function on the representations including but not limited to the accuracy of a given task. \n\nThe authors focus on three dissimilarity measures. First, they explain intuitively how these measures behave when there different initialization for the network. Then they give intuitions around what happens when they remove principal components of the representations. Then they show that they can arrive at similar conclusions using their rank comparison proposal with a functional behavior of the representations.\n\nThe authors then verify that the three measures can detect out-of-distribution differences when the pre-trained weights are the same but fine-tuning seed is different. However, they construct a benchmark to show that when the pre-trained weights are also different these three metrics do not show a statistically significant rank correlation, posing an open question around what metric is suitable in such a scenario.\n\n", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "What is the motivation behind using orthogonal Procrustes besides the fact that it's invariant to left orthogonal transformation? What is it capturing intuitively? Geometrically it seems to be relevant for representations that need to be compared modulo a rotation factor, but neural network representations do not necessarily get optimized to produce such rotation effect.\n\nThe first paragraph of page 5 and the footnote on page 5 together are a little confusing. Do you mean the smaller principle components are removed first or the largest ones? Or by quantifying you mean you sort them based on how much they decrease probing accuracy?\n\nBesides these, I think the paper is novel enough and has good clarity. In terms of significance, I think it provides an intuitive framework for giving meaning to dissimilarity measures which is beneficial for future research in this field.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "2M7GY-4USC4", "reviewer": "Reviewer_SGnL", "summary": "The work is motivated by a desire to better understand representational similarity metrics and ensure they obey certain desirable properties. Specifically, representational similarity measures should be sensitive changes that affect functional behavior, and specific (invariant) to changes that do not affect functional behavior. The authors operationalize these desiderata as 1) sensitivity to deletions of principal components in activation matrices, and 2) invariance to variations between models that arise from different random seeds. They find that CCA-based methods are less specific than CKA and Orthogonal Procrustes Analysis (OPA), while CKA is less sensitive than CCA and OPA. The authors then establish “benchmarks” by examining how certain variables and interventions (e.g. deleting principal components, changes in random seed, fine-tuning) affect the correlation between representational similarity metrics and performance metrics (mainly linguistic probes). The authors find that the benchmarks generally recapitulate the initial findings re: specificity and sensitivity. OPA seems to fall in between CKA and CCA with respect to the trade-off between sensitivity and specificity, leading the authors to encourage its adoption.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Excellent response. Score updated from 3 to 7. Details below**\n\n**Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?**\n\nIt takes a set of established representational similarity methods, prescribes desirable operating principles, and attempts to understand and benchmark them with regards to the principles. The clear articulation of desirable operating principles are somewhat novel, and the principal component deletion and benchmarking components of this work are novel combinations of known techniques. The effects of random seeds/initialization on representational similarity metrics has already been examined by Kornblith et al. (2019), though they did not examine OPA. Otherwise, related work appears adequately cited. \n\n**Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?**\n\nIn its current form, I think the scope of the paper is limited. The analyses are limited to transformer models and linguistic probing tasks. Extending the analyses to CNNs and even vision transformers (and multiple datasets) would allow the authors to make stronger claims about generality and increase the relevance of this work. This could be accomplished with layer-wise probing tasks similarly to with the language models. Comparisons should also be made between different model architectures.\n\nMore generally, I am curious about the following issue: The authors state that “Metrics should have specificity against random initialization” (L41-42). If two different initializations drawn from the same distribution result in two networks with very different behavior (i.e. different outputs for the same inputs), do we want a representational similarity metric to detect this? My assumption is yes, but perhaps I’m mistaken. This should be discussed.\n\nThe authors state “Neural network representations trained on the same data but from different random initializations are similar” (L125-126). Is this axiomatic, or a claim that representational similarity measures are intended to test? If it’s the latter, can you expand on what Kornblith et al. (2019) report?\n\nI believe it’s implicit that “similarity” between networks always means representational similarity, but I think it’s important that you explicitly dissociate representational vs. functional similarity. If two networks have very different representations but very similar outputs, would these networks be “similar”? By the definition of Sundararajan et al. (2018; Axiomatic Attribution for Deep Networks), these networks are functionally equivalent. To this end, comparing distributions (of outputs/classifier layers) could be more informative than just comparing task accuracies.\n\nDoing more probe-representation benchmarks seems like an easy way to strengthen your work, as probing tasks are cheap to run and there are a lot of them.\n\n**Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)**\n\nThe submission is sufficiently clear and organized, but I have a few questions and suggestions:\n\nLooking at Table 1, CKA seems seems to be most consistent, yet the authors state “the classical Orthogonal Procrustes transform attained consistently good performance” (ln 289-290). Can these opinions be reconciled? Perhaps by quantification? \n\nThe results in Lines 169-172 should be presented in a figure, e.g. plotting PC-deletion vs. accuracy curves.\n\nMore information should be provided about the rows in Table 1. For example, the OOD conditions should be explicitly labeled as such.\n\n**Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\nI think the work is well-motivated.**\n\nThe authors state “As a community, we need well-chosen formal criteria for evaluating metrics to avoid over-reliance on intuition and the pitfalls of too many researcher degrees of freedom” (L30-31), and I strongly agree with this. But as stated in the “Quality” section of this review, in its current form the work’s primary (and severe) limitation is that it only uses transformer models and language tasks. However, I am optimistic about the potential relevance and impact of this work if my concerns are addressed.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors state “A limitation of this work is that we only consider a handful of functional behaviors”, which I think is true, and I provide a number of suggestions in the “Quality” section of the review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper evaluates several representational similarity measures on a wide variety of benchmark tasks, and concludes that orthogonal Procrustes performs slightly better on average than other methods. All reviewers agreed that the empirical evaluation was novel and valuable, particularly with the additional results for vision models that the authors provided during the rebuttal period. Two reviewers noted that the suggestion that orthogonal Procrustes is superior to other methods was not entirely convincing given that different representational similarity methods seemed to perform best for different tasks. Nonetheless, all reviewers recommended acceptance.

The AC has some technical comments that may assist the authors in improving the camera-ready paper:

1. The definition of CCA in Eq. 2 isn’t sufficient to specify the value of PWCCA, since different scalings of the weights will lead to the same canonical correlations but different values of $\alpha_i$. If the components are assumed to have unit norm, then it does not seem like PWCCA is invariant to left orthogonal transformations as claimed.
2. The statement at the beginning of the appendix that the representations are normalized seems important for interpreting and reproducing the results in this work, and although L104-105 says that orthogonal Procrustes distance is not normalized between 0 and 1, because the representations are normalized, the Procrustes distance is normalized between 0 and 2.
3. The discussion suggests that orthogonal Procrustes gives better correlations than linear CKA for some tasks because, for diagonal matrices, orthogonal Procrustes reduces to a function of the squared distances between singular values whereas linear CKA reduces to the sum of the squared distances between the squared singular values. This seems possible, but this argument is not entirely convincing, because the difference in weighting of singular values is not the only difference between the methods: If all singular values are 1 and representations are equal-sized, orthogonal Procrustes distance reduces to the mean CCA correlation distance (up to a factor of 2), whereas linear CKA reduces to the mean squared CCA correlation distance. The AC suggests that the authors try manipulating (squaring/taking square roots of) the singular values of real representations before computing orthogonal Procrustes/CKA to ensure that the difference in efficacy between the methods can be explained by differences in their treatment of the singular values alone.

Finally, the biggest strength of this paper is its thorough empirical evaluation of the correlation between representational similarity and various functional properties of networks. Releasing the code for performing this evaluation would likely increase the paper's impact.
Decision: Accept (Poster)