Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper focuses on giving a consistent meaning to (dis)similarity metrics among neural networks based on representations out of intermediate layers. 

This consistent meaning is given by comparing the ordering that a dissimilarity measure gives to a "functional behavior". The functional behavior is defined as any function on the representations including but not limited to the accuracy of a given task. 

The authors focus on three dissimilarity measures. First, they explain intuitively how these measures behave when there different initialization for the network. Then they give intuitions around what happens when they remove principal components of the representations. Then they show that they can arrive at similar conclusions using their rank comparison proposal with a functional behavior of the representations.

The authors then verify that the three measures can detect out-of-distribution differences when the pre-trained weights are the same but fine-tuning seed is different. However, they construct a benchmark to show that when the pre-trained weights are also different these three metrics do not show a statistically significant rank correlation, posing an open question around what metric is suitable in such a scenario.


main_review: What is the motivation behind using orthogonal Procrustes besides the fact that it's invariant to left orthogonal transformation? What is it capturing intuitively? Geometrically it seems to be relevant for representations that need to be compared modulo a rotation factor, but neural network representations do not necessarily get optimized to produce such rotation effect.

The first paragraph of page 5 and the footnote on page 5 together are a little confusing. Do you mean the smaller principle components are removed first or the largest ones? Or by quantifying you mean you sort them based on how much they decrease probing accuracy?

Besides these, I think the paper is novel enough and has good clarity. In terms of significance, I think it provides an intuitive framework for giving meaning to dissimilarity measures which is beneficial for future research in this field.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper evaluates recent efforts to analyze learn representations obtained by neural networks using canonical correlation analysis (CCA), centered kernel alignment (CKA), and orthogonal Procrustes distance. The authors pointed out on some inconsistencies across these metrics and suggested a method to measure of how well the distance metric is with respect to some functionality.

main_review: The paper is clearly written, easy to follow, and the findings regarding CCA, CKA, and orthogonal Procrustes are interesting and valuable.
I'm a little bit puzzled by the quantitative measure and its contribution. 
If I understand it correctly, the authors first claim that the current metrics are not consistent concerning sensitivity and specificity. The authors demonstrated it with an example. Then, the authors presented a different method, grounded by some downstream task, using the same example to assess the same thing. Can we use this method to better understand representations similarities or only to compare different distance metrics? 

Overall, the method and analysis is interesting. I have some questions to the authors: 
1) I agree with the authors that according to Figure 1 two different networks’ representations at layer 7 have higher PWCCA distance than that between layer 7 and any other layer within the same network. However, we still see the same behavior within different layers for different models, meaning the distance is the smallest to layer 7 than any other layer. Can the authors say something about that? Maybe this is a normalization issue? Will it be different if we do it with randomly initialized network / network that was trained for other task? 
2) In line 257-259 the authors wrote: "The representations were computed on in-distribution MNLI data, meaning that the dissimilarity measures can detect OOD differences without access to OOD data". However, the authors did use OOD data (HANS) to get accuracies and learn f, no? Am I missing something? 
3) The authors mentioned that they trained several models for each setting and averaged the results, can the authors also share the STDs / variance?

limitations_and_societal_impact: Limitations: See above.
Societal impact: I do not see any.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The work is motivated by a desire to better understand representational similarity metrics and ensure they obey certain desirable properties. Specifically, representational similarity measures should be sensitive changes that affect functional behavior, and specific (invariant) to changes that do not affect functional behavior. The authors operationalize these desiderata as 1) sensitivity to deletions of principal components in activation matrices, and 2) invariance to variations between models that arise from different random seeds. They find that CCA-based methods are less specific than CKA and Orthogonal Procrustes Analysis (OPA), while CKA is less sensitive than CCA and OPA. The authors then establish “benchmarks” by examining how certain variables and interventions (e.g. deleting principal components, changes in random seed, fine-tuning) affect the correlation between representational similarity metrics and performance metrics (mainly linguistic probes). The authors find that the benchmarks generally recapitulate the initial findings re: specificity and sensitivity. OPA seems to fall in between CKA and CCA with respect to the trade-off between sensitivity and specificity, leading the authors to encourage its adoption.
main_review: **Excellent response. Score updated from 3 to 7. Details below**

**Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?**

It takes a set of established representational similarity methods, prescribes desirable operating principles, and attempts to understand and benchmark them with regards to the principles. The clear articulation of desirable operating principles are somewhat novel, and the principal component deletion and benchmarking components of this work are novel combinations of known techniques. The effects of random seeds/initialization on representational similarity metrics has already been examined by Kornblith et al. (2019), though they did not examine OPA. Otherwise, related work appears adequately cited. 

**Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?**

In its current form, I think the scope of the paper is limited. The analyses are limited to transformer models and linguistic probing tasks. Extending the analyses to CNNs and even vision transformers (and multiple datasets) would allow the authors to make stronger claims about generality and increase the relevance of this work. This could be accomplished with layer-wise probing tasks similarly to with the language models. Comparisons should also be made between different model architectures.

More generally, I am curious about the following issue: The authors state that “Metrics should have specificity against random initialization” (L41-42). If two different initializations drawn from the same distribution result in two networks with very different behavior (i.e. different outputs for the same inputs), do we want a representational similarity metric to detect this? My assumption is yes, but perhaps I’m mistaken. This should be discussed.

The authors state “Neural network representations trained on the same data but from different random initializations are similar” (L125-126). Is this axiomatic, or a claim that representational similarity measures are intended to test? If it’s the latter, can you expand on what Kornblith et al. (2019) report?

I believe it’s implicit that “similarity” between networks always means representational similarity, but I think it’s important that you explicitly dissociate representational vs. functional similarity. If two networks have very different representations but very similar outputs, would these networks be “similar”? By the definition of Sundararajan et al. (2018; Axiomatic Attribution for Deep Networks), these networks are functionally equivalent. To this end, comparing distributions (of outputs/classifier layers) could be more informative than just comparing task accuracies.

Doing more probe-representation benchmarks seems like an easy way to strengthen your work, as probing tasks are cheap to run and there are a lot of them.

**Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)**

The submission is sufficiently clear and organized, but I have a few questions and suggestions:

Looking at Table 1, CKA seems seems to be most consistent, yet the authors state “the classical Orthogonal Procrustes transform attained consistently good performance” (ln 289-290). Can these opinions be reconciled? Perhaps by quantification? 

The results in Lines 169-172 should be presented in a figure, e.g. plotting PC-deletion vs. accuracy curves.

More information should be provided about the rows in Table 1. For example, the OOD conditions should be explicitly labeled as such.

**Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
I think the work is well-motivated.**

The authors state “As a community, we need well-chosen formal criteria for evaluating metrics to avoid over-reliance on intuition and the pitfalls of too many researcher degrees of freedom” (L30-31), and I strongly agree with this. But as stated in the “Quality” section of this review, in its current form the work’s primary (and severe) limitation is that it only uses transformer models and language tasks. However, I am optimistic about the potential relevance and impact of this work if my concerns are addressed.

limitations_and_societal_impact: The authors state “A limitation of this work is that we only consider a handful of functional behaviors”, which I think is true, and I provide a number of suggestions in the “Quality” section of the review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors present a number of benchmarks and stress tests for similarity measures of learned representations of neural networks. Three similarity metrics are compared for their sensitivity and specificity of the similarity measure under a number of conditions, such as out-of-domain data or low rank approximations of hidden layer activation. Empirical results suggest that there is no clear trend for one or the other similarity metric that would be robust against all types of tests proposed.  

**Update**
After reading the authors responses I found that they addressed most of my concerns sufficiently. My main concern was that CCA a) is not evaluated with cross-validation and instead canonical correlations are computed on training data and b) that the CCA projections are not regularized. These two points together can lead to overfitted CCA projections and biased estimates of the canonical correlations, especially in the settings addressed in the present study. This could explain the poor results of CCA in the comparisons. Th

The authors responded that this is how the PWCCA method was applied and evaluated and the overfitted and not-cross-validated canonical correlations are a 'feature' of PWCCA. I wouldn't agree to that personally, but I think it's a valid point -- after all many researchers do not evaluate unsupervised methods on held-out test data, so there's probably some value in examining and comparing how these (wrong and biased) metrics computed on training data behave. 

So I think it's a valuable contribution and I increased my score accordingly. 


main_review: 

**Originality**: 

The probing tasks proposed are novel to the best of my knowledge, however they are a combination of previously proposed methods. It’s an important contribution to evaluate these different methods, and the idea to compare these methods under a variety of different conditions is also interesting. 

**Quality**: 

The submission is technically sound for most parts, but there are some technical details and  some motivations/intuitions that could be made clearer. 

For instance in line 99 the authors write that 

*We find that PWCCA performs far better than …*, here it would be helpful to see what performance is measured. 

If canonical correlations are not “working right” it could also be due to how they are evaluated. With high dimensional low rank data sets (and some large layers might be of that kind), CCA requires regularisation, and it makes sense to avoid overfitted canonical correlations (ccs) by evaluating the ccs on held-out data, just like with any supervised model. In line 302 the authors also suggest that CCA might have overfitted, but if it’s really overfitted it would not result in high held-out ccs. So both the hyper parameter optimisation for the CCA regularisation parameter as well as the evaluation on held out data might change the CCA results and I would assume the ccs are more trustworthy when cross-validated. 
That said, the other alternative similarity measures are probably more attractive as they do not require fitting parameters. 

The idea with most probing tests makes a lot of sense, but I’m not sure I understand what exactly is measured in the experiments with removal of PC components, or rather how that measure would help to understand something about representations - it's hard to relate that transformation to something that happens to networks in applications. A minor general remark on the selection of components, I think it could be helpful to discard components based on the amount of variance they explain, not their absolute number. 

In general it seems the authors often use the term “performance” and it is not really clear what is meant by that. It seems the methods are better or worse on different tasks, but it could be better explained what exactly is measured across all tasks, and what we can conclude from the results, beyond the observations for the single experiments. 

Also there are number of packages for perturbations and augmentations that would be worth integrating into these tests. 

**Clarity**: 

The manuscript is well written and structured, but there could be a bit more structure in the interpretation of results or the experimental design. I was asking myself what the results imply and what specificity and sensitivity results tell us about the networks if the probing tasks are so specific and lead to different results for most similarity metrics. Maybe one could evaluate just robustness of the similarity metrics under perturbations. Or maybe there is a way to add more probing tests or modify the metric to see more consistent differences between similarity metrics. In the end one is interested in things like which metric should I use to evaluate which aspect. 


**Significance**: 

Developing tests to evaluate is a relevant research direction and the tests proposed in this work are an important contribution. There are some aspects about the evaluation that could be improved and the heterogeneity of the empirical results suggest that it could be difficult to draw general conclusions about neural representations. 
limitations_and_societal_impact: yes
review_title: null

Meta-review: This paper evaluates several representational similarity measures on a wide variety of benchmark tasks, and concludes that orthogonal Procrustes performs slightly better on average than other methods. All reviewers agreed that the empirical evaluation was novel and valuable, particularly with the additional results for vision models that the authors provided during the rebuttal period. Two reviewers noted that the suggestion that orthogonal Procrustes is superior to other methods was not entirely convincing given that different representational similarity methods seemed to perform best for different tasks. Nonetheless, all reviewers recommended acceptance.

The AC has some technical comments that may assist the authors in improving the camera-ready paper:

1. The definition of CCA in Eq. 2 isn’t sufficient to specify the value of PWCCA, since different scalings of the weights will lead to the same canonical correlations but different values of $\alpha_i$. If the components are assumed to have unit norm, then it does not seem like PWCCA is invariant to left orthogonal transformations as claimed.
2. The statement at the beginning of the appendix that the representations are normalized seems important for interpreting and reproducing the results in this work, and although L104-105 says that orthogonal Procrustes distance is not normalized between 0 and 1, because the representations are normalized, the Procrustes distance is normalized between 0 and 2.
3. The discussion suggests that orthogonal Procrustes gives better correlations than linear CKA for some tasks because, for diagonal matrices, orthogonal Procrustes reduces to a function of the squared distances between singular values whereas linear CKA reduces to the sum of the squared distances between the squared singular values. This seems possible, but this argument is not entirely convincing, because the difference in weighting of singular values is not the only difference between the methods: If all singular values are 1 and representations are equal-sized, orthogonal Procrustes distance reduces to the mean CCA correlation distance (up to a factor of 2), whereas linear CKA reduces to the mean squared CCA correlation distance. The AC suggests that the authors try manipulating (squaring/taking square roots of) the singular values of real representations before computing orthogonal Procrustes/CKA to ensure that the difference in efficacy between the methods can be explained by differences in their treatment of the singular values alone.

Finally, the biggest strength of this paper is its thorough empirical evaluation of the correlation between representational similarity and various functional properties of networks. Releasing the code for performing this evaluation would likely increase the paper's impact.
Decision: Accept (Poster)