Grounding Representation Similarity Through Statistical Testing

1 Introduction . Understanding neural networks is not only scientifically interesting , but critical for applying deep networks in high-stakes situations . Recent work has highlighted the value of analyzing not just the final outputs of a network , but also its intermediate representations [ 20 , 29 ] . This has motivated the development of representation similarity measures , which can provide insight into how different training schemes , architectures , and datasets affect networks ’ learned representations . A number of similarity measures have been proposed , including centered kernel alignment ( CKA ) [ 13 ] , ones based on canonical correlation analysis ( CCA ) [ 24 , 30 ] , single neuron alignment [ 20 ] , vector space alignment [ 3 , 6 , 32 ] , and others [ 2 , 9 , 16 , 18 , 21 , 39 ] . Unfortunately , these different measures tell different stories . For instance , CKA and projection weighted CCA disagree on which layers of different networks are most similar [ 13 ] . This lack of consensus is worrying , as measures are often designed according to different and incompatible intuitive desiderata , such as whether finding a one-to-one assignment , or finding few-to-one mappings , between neurons is more appropriate [ 20 ] . As a community , we need well-chosen formal criteria for evaluating metrics to avoid over-reliance on intuition and the pitfalls of too many researcher degrees of freedom [ 17 ] . In this paper we view representation dissimilarity measures as implicitly answering a classification question–whether two representations are essentially similar or importantly different . Thus , in analogy to statistical testing , we can evaluate them based on their sensitivity to important change and specificity ( non-responsiveness ) against unimportant changes or noise . As a warm-up , we first initially consider two intuitive criteria : first , that metrics should have specificity against random initialization ; and second , that they should be sensitive to deleting important principal 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . components ( those that affect probing accuracy ) . Unfortunately , popular metrics fail at least one of these two tests . CCA is not specific – random initialization noise overwhelms differences between even far-apart layers in a network ( Section 3.1 ) . CKA on the other hand is not sensitive , failing to detect changes in all but the top 10 principal components of a representation ( Section 3.2 ) . We next construct quantitative benchmarks to evaluate a dissimilarity measure ’ s quality . To move beyond our intuitive criteria , we need a ground truth . For this we turn to the functional behavior of the representations we are comparing , measured through probing accuracy ( an indicator of syntactic information ) [ 4 , 27 , 35 ] and out-of-distribution performance of the model they belong to [ 7 , 23 , 25 ] . We then score dissimilarity measures based on their rank correlation with these measured functional differences . Overall our benchmarks contain 30,480 examples and vary representations across several axes including random seed , layer depth , and low-rank approximation ( Section 4 ) 1 . Our benchmarks confirm our two intuitive observations : on subtasks that consider layer depth and principal component deletion , we measure the rank correlation with probing accuracy and find CCA and CKA lacking as the previous warm-up experiments suggested . Meanwhile , the Orthogonal Procrustes distance , a classical but often overlooked2 dissimilarity measure , balances gracefully between CKA and CCA and consistently performs well . This underscores the need for systematic evaluation , otherwise we may fall to recency bias that undervalues classical baselines . Other subtasks measure correlation with OOD accuracy , motivated by the observation that random initialization sometimes has large effects on OOD performance [ 23 ] . We find that dissimilarity measures can sometimes predict OOD performance using only the in-distribution representations , but we also identify a challenge set on which none of the measures do statistically better than chance . We hope this challenge set will help measure and spur progress in the future . 2 Problem Setup : Metrics and Models . Our goal is to quantify the similarity between two different groups of neurons ( usually layers ) . We do this by comparing how their activations behave on the same dataset . Thus for a layer with p1 neurons , we define A 2 Rp1⇥n , the matrix of activations of the p1 neurons on n data points , to be that layer ’ s raw representation of the data . Similarly , let B 2 Rp2⇥n be a matrix of the activations of p2 neurons on the same n data points . We center and normalize these representations before computing dissimilarity , per standard practice . Specifically , for a raw representation A we first subtract the mean value from each column , then divide by the Frobenius norm , to produce the normalized representation A⇤ , used in all our dissimilarity computations . In this work we study dissimilarity measures d ( A⇤ , B⇤ ) that allow for quantitative comparisons of representations both within and across different networks . We colloquially refer to values of d ( A⇤ , B⇤ ) as distances , although they do not necessarily satisfy the triangle inequality required of a proper metric . We study five dissimilarity measures : centered kernel alignment ( CKA ) , three measures derived from canonical correlation analysis ( CCA ) , and a measure derived from the orthogonal Procrustes problem . Centered kernel alignment ( CKA ) uses an inner product to quantify similarity between two representations . It is based on the idea that one can first choose a kernel , compute the n⇥ n kernel matrix for each representation , and then measure similarity as the alignment between these two kernel matrices . The measure of similarity thus depends on one ’ s choice of kernel ; in this work we consider Linear CKA : dLinear CKA ( A , B ) = 1 kAB > k2F kAA > kF kBB > kF ( 1 ) as proposed in Kornblith et al . [ 13 ] . Other choices of kernel are also valid ; we focus on Linear CKA here since Kornblith et al . [ 13 ] report similar results from using either a linear or RBF kernel . Canonical correlation analysis ( CCA ) finds orthogonal bases ( wiA , wiB ) for two matrices such that after projection onto wiA , w i B , the projected matrices have maximally correlated rows . For 1 i p1 , 1Code to replicate our results can be found at https : //github.com/js-d/sim_metric . 2For instance , Raghu et al . [ 30 ] and Morcos et al . [ 24 ] do not mention it , and Kornblith et al . [ 13 ] relegates it to the appendix ; although Smith et al . [ 32 ] does use it to analyze word embeddings and prefers it to CCA . the ith canonical correlation coefficient ⇢i is computed as follows : ⇢i = max wiA , w i B hwiA > A , wiB > Bi kwiA > Ak · kwiB > Bk ( 2 ) s.t . hwiA > A , wjA > Ai = 0 , 8j < i , hwiB > B , wjB > Bi = 0 , 8j < i ( 3 ) To transform the vector of correlation coefficients into a scalar measure , two options considered previously [ 13 ] are the mean correlation coefficient , ⇢̄CCA , and the mean squared correlation coefficient , R2CCA , defined as follows : d⇢̄CCA ( A , B ) = 1 1 p1 X i ⇢i , dR2CCA ( A , B ) = 1 1 p1 X i ⇢2i ( 4 ) To improve the robustness of CCA , Morcos et al . [ 24 ] propose projection-weighted CCA ( PWCCA ) as another scalar summary of CCA : dPWCCA ( A , B ) = 1 P i ↵i⇢iP i ↵i , ↵i = X j |hhi , aji| ( 5 ) where aj is the jth row of A , and hi = wiA > A is the projection of A onto the ith canonical direction . We find that PWCCA performs far better than ⇢̄CCA and R2CCA , so we focus on PWCCA in the main text , but include results on the other two measures in the appendix . The orthogonal Procrustes problem consists of finding the left-rotation of A that is closest to B in Frobenius norm , i.e . solving the optimization problem : min R kB RAk2F , subject to R > R = I . ( 6 ) The minimum is the squared orthogonal Procrustes distance between A and B , and is equal to dProc ( A , B ) = kAk2F + kBk2F 2kA > Bk⇤ , ( 7 ) where k · k⇤ is the nuclear norm [ 31 ] . Unlike the other metrics , the orthogonal Procrustes distance is not normalized between 0 and 1 , although for normalized A⇤ , B⇤ it lies in [ 0 , 2 ] . 2.1 Models we study In this work we study representations of both text and image inputs . For text , we investigate representations computed by Transformer architectures in the BERT model family [ 8 ] on sentences from the Multigenre Natural Language Inference ( MNLI ) dataset [ 40 ] . We study BERT models of two sizes : BERT base , with 12 hidden layers of 768 neurons , and BERT medium , with 8 hidden layers of 512 neurons . We use the same architectures as in the open source BERT release3 , but to generate diversity we study 3 variations of these models : 1 . 10 BERT base models pretrained with different random seeds but not finetuned for particular tasks , released by Zhong et al . [ 41 ] 4 . 2 . 10 BERT medium models initialized from pretrained models released by Zhong et al . [ 41 ] , that we further finetuned on MNLI with 10 different finetuning seeds ( 100 models total ) . 3 . 100 BERT base models that were initialized from the pretrained BERT model in [ 8 ] and finetuned on MNLI with different seeds , released by McCoy et al . [ 23 ] 5 . For images , we investigate representations computed by ResNets [ 11 ] on CIFAR-10 test set images [ 14 ] . We train 100 ResNet-14 models6 from random initialization with different seeds on the CIFAR-10 training set and collect representations after each convolutional layer . Further training details , as well as checks that our training protocols result in models with comparable performance to the original model releases , can be found in Appendix A . 3available at https : //github.com/google-research/bert 4available at https : //github.com/ruiqi-zhong/acl2021-instance-level 5available at https : //github.com/tommccoy1/hans/tree/master/berts_of_a_feather 6from https : //github.com/pytorch/vision/blob/master/torchvision/models/resnet.py 3 Warm-up : Intuitive Tests for Sensitivity and Specificity . When designing dissimilarity measures , researchers usually consider invariants that these measures should not be sensitive to [ 13 ] ; for example , symmetries in neural networks imply that permuting the neurons in a fully connected layer does not change the representations learned . We take this one step further and frame dissimilarity measures as answering whether representations are essentially the same , or importantly different . We can then evaluate measures based on whether they respond to important changes ( sensitivity ) while ignoring changes that don ’ t matter ( specificity ) . Assessing sensitivity and specificity requires a ground truth–which representations are truly different ? To answer this , we begin with the following two intuitions7 : 1 ) neural network representations trained on the same data but from different random initializations are similar , and 2 ) representations lose crucial information as principal components are deleted . These motivate the following intuitive tests of specificity and sensitivity : we expect a dissimilarity measure to : 1 ) assign a small distance between architecturally identical neural networks that only differ in initialization seed , and 2 ) assign a large distance between a representation A and the representation Â after deleting important principal components ( enough to affect accuracy ) . We will see that PWCCA fails the first test ( specificity ) , while CKA fails the second ( sensitivity ) .

Summary: The authors present a number of benchmarks and stress tests for similarity measures of learned representations of neural networks. Three similarity metrics are compared for their sensitivity and specificity of the similarity measure under a number of conditions, such as out-of-domain data or low rank approximations of hidden layer activation. Empirical results suggest that there is no clear trend for one or the other similarity metric that would be robust against all types of tests proposed.    **Update** After reading the authors responses I found that they addressed most of my concerns sufficiently. My main concern was that CCA a) is not evaluated with cross-validation and instead canonical correlations are computed on training data and b) that the CCA projections are not regularized. These two points together can lead to overfitted CCA projections and biased estimates of the canonical correlations, especially in the settings addressed in the present study. This could explain the poor results of CCA in the comparisons. Th  The authors responded that this is how the PWCCA method was applied and evaluated and the overfitted and not-cross-validated canonical correlations are a 'feature' of PWCCA. I wouldn't agree to that personally, but I think it's a valid point -- after all many researchers do not evaluate unsupervised methods on held-out test data, so there's probably some value in examining and comparing how these (wrong and biased) metrics computed on training data behave.   So I think it's a valuable contribution and I increased my score accordingly.