Grounding High Dimensional Representation Similarity by Comparing Decodability and Network Performance
Authors: Lucas Hayne
Heejung Jung
Abhijit Suresh
R. McKell Carter
Venue: Submitted to ICLR 2023 (2023)
Keywords: ablation
representation
semantic decoding
linear decoding
representation similarity
neural network interpretability
activation space

Abstract: To understand and interpret neural networks, representation similarity metrics have been used to compare learned representations between and across networks. Recent experiments have compared these similarity metrics to find the best performing and the most robust metrics, noting that classic baselines perform surprisingly well. These experiments are mostly constrained to studying relatively low-dimensional representations because of the computational cost of prominent representation similarity metrics. We extend previous work to test representation similarity metrics on larger convolutional networks processing larger images. In order to make this work possible, we employ reformulated representation similarity metrics for use on very high-dimensional representations. Using these reformulated similarity metrics, we test how well each metric captures changes to representations induced by ablations in two popular convolutional networks. In order to ground the effects of changes to representations in function, we use linear decoding probes and network performance measures. These measures of function allow us to test how well similarity metrics capture changes in decodable information versus changes in network performance. Linear decoding methods index available information in the representation, while network performance measures index the information used by the network. We show that all the tested representation similarity metrics significantly predict changes in network function and decodability. Within these metrics, on average, Procrustes and CKA outperform regularized CCA-based methods. All metrics predict decodability changes significantly better than they do network function. Procrustes and CKA do not outperform regularized CCA-based metrics for all network and functionality measure combinations. We add to the growing literature on representational similarity metrics to facilitate the improvement of current metrics for network interpretability.

Under review as a conference paper at ICLR 2023
CAUSAL TESTING OF REPRESENTATION SIMILARITY METRICS
Anonymous authors Paper under double-blind review
ABSTRACT
Representation similarity metrics are widely used to compare learned representa- tions neural networks, as evident in extensive literature investigating metries that accurately captures information encoded in the network. However, aiming capture all of the information available in the network may have little to do with what information is actually used by the network. One solution is to experiment with causal measures function. By ablating groups of units thought to carry information and observing whether those ablations affect network performance, we can focus on an outcome that causally links representations to function.
In this paper, we systematically test representation similarity metrics to evaluate their sensitivity to causal functional changes induced by ablation. We use network performance changes after ablation way causally measure the influence of representation on function. These measures of function allow us to test how well similarity metries capture changes in network performance versus changes lin- ear decodability. Network performance measures index the information used by the network, while linear decoding methods index available information in the representation.
We show that all of the tested metries more sensitive to decodable features than network performance. Within these metrics, Procrustes and CKA outper- form regularized CCA-based methods average. Although Procrustes and CKA outperform on average, for AlexNet, Procrustes and CKA no longer outperform CCA methods when looking at network performance. We provide causal tests of the utility of different representational similarity metrics. Our results suggest that interpretability methods will be more effective if they are based on representa- tional similarity metrics that have been evaluated using causal tests.
INTRODUCTION
Neural networks already play critical role in systems where understanding and interpretation paramount like self-driving cars and the criminal justice system. To understand and interpret neural networks, representation similarity metrics have been used compare learned representations between and across networks (Korbbith et al. (0019; Raghu et al. (2017): : Morcos et al.) 8b);
Wang et al. (2018); Li al. (2015); Feng et al. (2000) Nguyen et (2220)) Using these similarity metrics, researchers evaluate whether networks trained from different random initializations learn the same information, whether different layers learn redundant complementary information, and how different training data affect learning (Kornblith et al.] (2019); Li et al. (2013); Wang et al. (2018). ) Apart from helping to answer these fundamental questions, similarity metrics have the potential to provide a perrpoe metric over representations Boix-Adsera et al. (2022).
What it means for two representations to be similar, however, not straightforward. Many similar- ity metrics have been proposed with different underlying assumptions and strategies for comparing representation spaces. For example, some similarity metrics are invariant under linear transforma- tions while others are not (see Kornblith et al. (2019) for theoretical comparison). These different assumptions and strategies can lead to quantitatively different predictions. For instance, Ding et al. (2021) show that certain metrics insensitive to changes to the decodable information present in representations. In another study, Davari et al. (2022) demonstrate that the centered kernel alignment metric predicts high similarity between random and fully trained representations. It is therefore

---

Under review as a conference paper at ICLR 2023
unclear which representation similarity metrics capture the most important information from repre- sentations and further tests are needed to evaluate them.
What important pieces of information do similar representations share? Previous studies into simi- larity metrics have assumed that similar representations share lincarly decodable information Boix- Adsera al. (2022); Ding et al. 22021); Feng et al. (2020). To measure the linearly decodable infor- mation in a representation, researchers usually train linear probes for downstream tasks on learned representations and compare the results. However, the features of representation that carry the most information may not be those actually used by the network during inference. Studies that remove features from representations trained networks have revealed weak link between the relevance of feature for decoding and its effect when removed from the network [Meyes et al. (2020); (Zhou et al. (2018) Donnelly& & Roegiest (2019); Morcos et al. (2018b). Hayne et al. (2022) recently showed that linear decoders specifically cannot single out the features of representations actually used by the network. Consequently, two representations that are equally decodable using linear probes may not actually be equal from the point of view of network performance. This distinction is crucial for neural network interpretability where the aim is to develop human-understandabb descriptions of how neural networks actually rely their internal representations.
For the purpose of interpreting neural network function, we suggest that representations should be judged as similar if they cause similar effects in trained network. To observe these causal effects, previous studies have removed features from a representation, a process called ablation, and observed the effects LeCun et al. (1989). In this paper, representation similarity metrics are related to causal function. We first ablate groups of units from AlexNet and MobileNet, compare the original representations the ablated representations using representation similarity metrics, and then compare metric outputs to the changes seen for linear probe decoding or network performance. This way we can test how well representational changes from ablations are captured by representational similarity metrics by comparing those metrics to changes in linear decoding and causal differences in network performance.
ablation evaluate how closely
Linear probes measure how much task-specific information is directly decodable from given rep- resentation. On the other hand, network performance measures quantify how the network trained on the same task uses given representation. Finally, we test how well representation similarity metries capture these two changes. By directly comparing linear probe accuracies and network per- formances on the same task tion similarity metrics to the non-causal linear properties of representations compared to the causal non-linear properties used by the network? Answering these questions may help in the development of interpretability methods that are increasingly sensitive to actual network function.
answer questions like: how much more sensitive representa-
In this work, we show that CKA, Procrustes, and regularized CCA-based representation similarity metrics predict causal network performance changes significantly worse than non-causal decoding changes. We also show that, on average, Procrustes and CKA outperform regularized CCA-based methods. However, Procrustes and CKA do not outperform regularized CCA-based metrics on all network and functional measure combinations. Overall, our results suggest that interpretability methods will be more effective if they are based on representational similarity metrics that have been evaluated using causal tests. In general, this paper documents the following contributions:
We introduce causal test of the utility of representation similarity metrics. We find that five popular representation similarity metrics are significantly less sensitive to network per- formance changes induced by ablation than linearly decodable changes.
Within the tested metrics, we show that Procrustes and CKA tend to outperform regularized CCA-based methods, but that tests using linear probes and network performance based functional measures can produce different results in different networks.
2
METHODS
In Section 2.1 we describe the statistical testing methodology used in our experiments. In Section
2.2
we introduce the representation similarity measures evaluate and reformulate them for use on high dimensional representations. In Section 2.3] we describe how we use ablation to produce representations with different functional properties. Finally, in Section 2.4 we describe how
2

---

Under review as a conference paper at ICLR 2023
linear probe decoding deficits and class-specift performance deficits to measure decodable and downstream network changes, respectively, in the ablated representations.
Features
Features
3
A vs. X
CCA
Imputs
Representation
XX)
Inputs
Ablated Representation (A)
CKA
Procrustes
6
Linear Decoding
A
A
Change
2.1%
5.6%
Network Performance Deficit
X
X
X
Linear Decoding Change (Non-causal function)
Network Performance Deficit (Causal Function)
Correlate Distance and Function
Figure 1: Elements of the experimental design. 1) A representation matrix X is generated from a layer in trained network. 2) Ablations are applied by deleting features from the representation generate the ablated matrix A. Representation matrices X and A compared in three ways. 3) A representation metric similarity calculated between them. 4) Linear probes are fit to each to decode target class and the linear probe accuracies are compared. 5) The representations are fed back into the network and the network performance difference between them is calculated. 6) By comparing metric similarities with both linear probe decoding changes and network performance changes across many ablations (represented with multiple points in the correlation plots in panel 6), we can assess to what extent each metric captures non-causal and causal measures of function.
2.1 STATISTICAL TESTING
Assume A RexpI represents matrix of activations for P1 neurons given n examples, and B Raxps represents a matrix of activations for pa neurons given the same n examples. The matrices A and B called representation matrices and have been preprocessed to have centered columns. Let d(A, A, B) denote representation similarity metric that returns zero if and only if A = B and for which d(A, B) d(B, These metrics do satisfy the triangle inequality and are therefore not formal distance metrics. For simplicity, we will refer to them as metrics in this work.
To quantify functional differences between representation matrices, we use functional behavior mea- sures. Formally, let f R=xp > IR denote functional behavior measure that, given representation matrix, returns a scalar measure the representation's 's role function. this study, we utilize two functionality measures, class-specific linear decoding accuracy (ʃ (/iecoding) and class-specific network performance (peeformmanc)) In the case of linear decoding accuracy, fdecoding returns the average lincar probe accuracy achieved from decoding target class identity from representation matrix. On the other hand, Sperfonmance returns the average classification performance for a target class achieved by feeding the representation matrix to the network at the appropriate layer. More details presented in Section 2.4
aim to statistically test representation similarity metries using the same As in Ding al. 2021
methodology according to their paper:
Gather S, a set of representation matrices that vary along one or more dimensions. We note that S contains unablated as well ablated representation matrices (see Section 2.3
2. Select an unablated reference matrix X E S.
3. Compute the following for all A, E S where A, + X
D d(x,A,)
F= - f(A)
4. Compute the correlation between D and F using Spearman's 's correlation.
3

---

Under review as a conference paper at ICLR 2023
This procedure quantifies the extent which the representation similarity metries, d(-). capture the functionality differences, as measured by f()), produced by ablating the representation matrix. A high Spearman's 's correlation value between metric's computed representation similarities and the functionality differences produced by ablation implies that the chosen metric is sensitive to the chosen functionality. Whereas, a low correlation implies the opposite: that the chosen metric is not sensitive to the chosen functionality.
2.2
REPRESENTATION SIMILARITY METRICS
As in Ding et al. (021)) we study three main representation similarity metrics: centered kernel alignment (CKA), Procrustes, and segjection-weighter canonical correlation analysis (PWCCA).
Centered kernel alignment (CKA) is based on the idea that similar representations also have sim- ilar relations between examples. In other words, representation matrices that store images of lettuce and rabbits using similar vectors should be more similar to each other than with representation matrices that encode images of lettuce and dinner plates using similar vectors. This idea leads Korn- blith et al. (2019) to formulate linear CKA, which uses linear kernel to compare example vectors (henceforth referred to as just CKA):
dekA (A, B) =1
(1)
|| B
where ||
||| is the Frobenius norm and || derives from the following relation:
(recAAA), vec BB") = tr (BAA - B
(2)
Relation shows that the similarity between pairwise example similarity matrices (far left) is equal to the squared Frobenius norm of the feature covariance matrix between representations (far right). Kornblith et al. (2019) use this relation to form Equation which measures the normalized similarity between the example similarity matrices of and B. Unfortunately, computing and storing either A B, A A, B B can be prohibitively expensive when both pi and p2 grow too large. Therefore we reformulate Equation using relation 2 and the fact that |
X||: IF = |XX*|le into:
dckA ( (A, B) trace( AA BB
TAAT|
BB |
This reformulation allows us to use CKA on layers with at least 100 times more neurons than previ- ous studies.
Procrustes is analytical solution to the orthogonal Procrustes problem which involves finding right rotation matrix that is as close as possible to A as measured by the Frobenius norm:
drrocrustes (A, B) = A 2 + BB] 2 A
(4)
where || /l. nuclear norm. As with CKA. A needs to be replaced to lighten the compu- tational cost of working with large layers. Therefore, utilize the fact that the nuclear norm of matrix is the sum of its singular values to reformulate Procrustes:
dprocrustes ( () | BB 2 Σ ((ATBB
(5)
where (X) represents the 7th eigenvalue of matrix X. Again, this reformulation saves from manipulating the large P1 X P2 matrix by replacing it with a much smaller n x n matrix (assuming < P2).
Projection- weighted canonical correlation analysis (PWCCA) a special case of canonical cor- relation analysis (CCA) proposed by Morcos al. 2018a). CCA itself provides solution to the problem linearly projecting A and B into a shared subspace where their correlations are max- imized. CCA finds min(p1,P2) pairs of weight vectors (w (.B and the resulting correlation
4

---

With the Under review as a conference paper at ICLR 2023
induced by projecting A and B using the hth weight vector is:
Pi corr(Aw'A, Bw
max
win " subject to Vjei Aw'A 1 Aw
(6)
Bw's 1 Bw
where the P; is maximized subject to the constraint that the subspace features be thogonal. Equation 6 can be solved for by performing singular value dccomposition on (A" 1/2 B 1/2 where the singular values are equal the correlations (p: A [1, [1, pin(p),P2)))
However, the inverses of the feature covariance matrices do not exist when the number of neurons exceeds the number of examples. For these cases, we can use regularized or "ridge" CCA (Vinod (976)), which applies an L2 penalty to the weight vectors and can be solved by performing SVD on [A*A+KAT) 1/2 A + KB 1/2 However, we again run into the problem that the feature covariance matrices are too costly to compute for large layers. So, we employ the "kernel trick" introduced by Kuss & (raappe](2003) and et al. (2004) and refined by Tuzhilina et al.) 202II which allows us to substitute into the above expression RA for where RA represents an X matrix recovered by applying SVD on A. i.e. A RA VA. The same trick can be applied to B. These substitutions allows us to work with much smaller matrices and make CCA computationally tractable for large layers. The only drawback is that if we apply the "kernel trick" to both matrices, we recover only n canonical correlations rather than min(p1, P2).
Mean CCA (as used by Raghu al. (2177) and mean squared CCA (Ramsay al. (984)) average raw and squared correlations recovered through CCA, respectively. However, PWCCA re- re-weights each correlation value by its importance for the underlying representation. Formally, if representa- tion matrix A has neuron activation vectors [Z,,..., Zp,] and CCA vectors [h]..., hn), then PWCCA computes weighted mean as:
dPwCCA (A, CRNCCA(A,B'r A.
Σ 1m
asps
s.t. a, = Σ (h zz)l
where di is normalized version of a; The preceding CCA reformulations represent regularized forms of CCA-based metrics useful for nigh-dimensional representations, which we refer to as reg- ularized CCA-CaCS metrics in the text, but just CCA metrics in the figures for sake of brevity.
2.3 ABLATION
To study representation functionality changes, we needed a method for manipulating representations to reliably produce network performance deficits at the output the network. To reliably produce network performance deficits, we followed the procedure of Hayne al. (2022). Specifically, for each the 10 or 50 randomly chosen target classes and layer of the CNNs we tested, AlexNet and MobileNetV2, first projected every neuron onto two dimensions: class selectivity and activation magnitude. Then, we constructed grid to overlay on the activation space so that each cell of the grid contained the same number of neurons. To supply the set S of representation matrices from Section 2.1 we ablated one cell of neurons at a time by setting the activation values for those neurons to zero.
2.4 FUNCTIONALITY MEASURES
After collecting the set S of ablated representation matrices, we sought to compare two function- ality measures. First, we linear probe decoders to each representation matrix linear probes aimed to decode the identity of one target class, we fit simple logistic regression classifiers to distinguish the target class representations from all other representations the repre- sentation matrix. Because many of the CNN layers contained thousands of features, randomly selected 100 neurons as features in the logistic regression model and averaged the training accuracy over 200 repetitions as in Alain Bengio (2066). Intuitively, the accuracies measure how much
5

---

Under review as a conference paper at ICLR 2023
information representation matrix contains about the target class. We refer to the changes induced in the decodability of the target class by ablation as decoding accuracy deficits.
In contrast linear decoding accuracies, also measured the network's class-specific classifica- tion performance when utilizing the representations in the representation matrix during inference. Specifically, recorded the average classification rank of the target class. For instance, if the CNN's softmax layer predicted that the Junco bird class was the third most likely class given an image of Junco bird, then that image received a rank of three. The average class-specific ranks intuitively measure how functionally useful the representations from the representation matrix were the network. We refer to the changes induced class-specific classification rank by ablation as network performance deficits.
RESULTS
In this section, we detail the results of aggregating our statistical tests across each layer and class from AlexNet and MobileNetV2. We conducted a hierarchical linear model to evaluate the predic- tion of the rank correlation values as a function of two factors: metric, i.e. CKA, Procrustes, and regularized ICAA-aaee methods and 2) functionality, i.e. network performance deficits and decod- ing accuracy deficits. To allow for comparable correlation coefficients, we fisher-z transformed each rank correlation value, then, averaged them across layers for each each metric and functionality. Af- terwards, we inverse transformed the fisher-z back to correlation coefficients. We modeled classes as random units. To sum, the transformed correlation values were modeled type and functionality measure. All analyses were conducted using the "Ime4" package in R (Bates) generated using "ggplot2" (Wickham 2016), "raincloudplots" (Allen
function of metric
al. (.144). Figures
et al. 2021)) ), and "smplot" package (Min Zhou (021 in R.
3.1
REPRESENTATION SIMILARITY METRICS ARE SIGNIFICANTLY LESS SENSITIVE TO CAUSAL FUNCTION
Figure Za shows the distribution of correlation values across functionality measures. In both cases, the rank correlation values of both functionality measures are significantly different from 0, suggest- ing that the correlation values are sensitive to functional behavioral changes in the representations, (F(1,49) = 477.37, p < .001 for AlexNet, F(1,9) 1464.06, < ,001 for MobileNett2).. The anal- ysis of interest demonstrates these correlation values are significantly different between network performance deficits and decoding accuracy deficits, when averaging across the five similarity met- rics- CKA, Procrustes, and regularized CCA-based metries (F(1,441) = 633.85 85, p < .001). In other words, there is significant main effect of functionality. Likewise, Figure 2b shows the same results for MobileNetV2 (F(1,81) ((1,81)= 5770.04, p < .001).
3.2 CKA AND PROCRUSTES TEND TO OUTPERFORM OTHER METRICS
Figures Ba and 3b show that CKA and Procrustes have, on average, higher rank correlation values compared the other CCA-based metries (1(441) 6.33, p .001 for AlexNet; ((81) 15.70, < .001 for MobileNetV2). Again, higher rank correlation values indicate a higher coupling be- tween functionality measures and representation similarity metrics, thereby suggesting that CKA and Procrustes are better metrics at capturing functionality.
3.3 FUNCTIONALITIE PRODUCE DIFFERENT RESULTS FOR DIFFERENT NETWORKS
Figure 4 (top) shows a significant interaction between functionality measures and similarity met- rics for AlexNet (F(4,441)= = 12,21, < .001). Figure (bottom) shows the same model for Mo- b22eNetV2, however, interaction is non-significant (F(4,81) -88, p .48) In the case of AlexNet, with the plotted means, it evident that the interaction is driven by the metric differences within de- coding accuracy deficits. As for the similarity metrics within network performance deficits, single metric outperforms the other; instead, each metric achieves similarly low correlation amongst the network performance deficit group. As for the decoding accuracy deficit group, the other hand, CKA and Procrustes outperform the CCA-based metrics (r(441) = 9.31, p < ,0001). In the case of MobileNetV2, there is no significant interaction. CKA and Procrustes outperform the CCA-based

---

Under review as a conference paper at ICLR 2023
metrics both decoding (x(1)) 12.41, .0001) and network performance tests (r(81) 9.79, < .,0011,, thereby canceling out an interaction effect.
DISCUSSION
In this work, we systematically test representation similarity metrics on ImageNet-trained CNNs determine how sensitive they are network performance and decoding changes in representa- tions. To facilitate these tests, we reformulate previously proposed similarity metrics to make them computationally tractable for use on high- nigh-dimensional representations. Our tests revealed that, while similarity metrics are significantly sensitive to network performance and decoding measures of func- tionality, they are significantly less sensitive to network performance. Additionally, in most of our tests CKA and Procrustes significantly outperform CCA-based methods. However, do find that different networks and different functionality choices can eliminate their advantages.
Previously, Ding et al. ) performed some general stress tests representation similarity met- rics. In their work they showed that PWCCA distances were easily influenced by changes to random seed, predicting that the same layers in two differennt networks were more similar to other distant layers in the network than to each other. another test, the CKA metric suffered from a different problem where failed distinguish representations from their low rank counterparts. Davari al.) (022) pointed out other weaknesses associated with CKA. Among other findings, they demon- strated that CKA judged random and fully trained representations to highly similar, especially in early layers of network. Both studies used functional and intuitive tests for evaluation.
Although our work utilizes framework introduced by Ding al. (202T), we aim to funda- mentally different hypothesis not addressed by their tests. Specifically ly, we aim to test how sensitive representation similarity metrics are to the functional properties of a representation actually used by a trained network during inference. In our tests we directly compare linear prohe accuracy changes on in-distribution inputs to network performance changes on the same inputs after ablation. On the other hand Ding et al. (022 do not test this direct comparison. Instead, they perform more general stress tests on similarity metries using either in-distribution linear probes or oul-of-distribution net-
Correlation between metric and functionality
Correlation between metric and functionality
0.6
0.6
0.3
0.3
0.0
0.3
Network performance deficits accuracy deficits
Decoding
Network performance deficits
Decoding accuracy deficits
Functionality
Functionality
(a) AlexNet
(b) MobileNetV2
Figure 2: Correlation values overall significantly sensitive to ablation changes in the repre- sentations, however, decoding accuracy has a higher value than network performance deficits. This figure shows the distribution of rank correlation values between functionality measures, i.c.. the main effect of functionality measure on AlexNet (a) and MobileNetV2 (b). Each data point repre- sents value from the ten classes; within each class, we average the five metrics of CKA, Procrustes, PWCCA, mean CCA and mean squared CCA. The distribution is further depicted by the violin plots and boxplots, which illustrates the median and upper/lower quartile of the distribution. Overall, both functionality measures are significantly different from 0, indicating that the rank correlation values are sensitive changes produced by ablations the representations. Further examining the differ- ence between functionality measures, deficit are higher than that network performance deficits. Note that all CCA- -based metrics are regularized in our formulations.
that the rank correlation values of decoding accuracy

---

Under review as a conference paper at ICLR 2023
Correlation between
metric and functionality titonallty
0.5
w-g-- d.
" woteeee +2
Correlation between metric and functionality
0.5
. 14 N0h
0.3
0.2
0.1
0.0
0.0
Mean
Mean
PWCCA
CKA
Procrustes
Mean
Mean PWCCA
CKA. Procrustes
CCA squared CCA
CCA squared CCA
Metric
Metric
(a) AlexNet
(b) MobileNetV2
Figure 3: CKA and Procrustes outperform CCA-based metries. This figure shows the distri- bution of rank correlation values across each metric, averaged between functionality measures on AlexNet (a) and MobileNetV2 (b). Within each distribution, each data point represents the aver- age rank correlation value from the ten classes, collapsed across network performance deficits and decoding accuracy deficits. Higher rank correlation values the CKA and Procrustes condition indicate that these metrics sensitive to the perturbations in the representations compared to other CCA-based metrics. Note that all CCA-based metrics are regularized our formulations.
work performance scores and different methods of perturbing representations. Through this direct comparison, we able show that similarity metries significantly less sensitive the features of representations that are actually used by the network compared to its decodable features.
All of the representation similarity metrics tested significantly less sensitive causal changes induced by ablation than non-causal decoding changes. This finding reflects the considerations made in developing these similarity metries. Representation similarity metries were designed compare the linear geometric properties of two representation spaces. So, it is not surprising that similarity metries correlate with changes in decoding accuracies. On the other hand, network performance measures of function reflect how the network utilizes representations. In this case, functionally sim- ilar representations are those representations that remain similar after a series of non-linear trans- formations through layers of the network. Perhaps it is not surprising that this non-linear notion of similarity is harder to capture using current similarity metrics. However, the ultimate goal of interpretability to link representation and non-linear network function.
Like previous studies, we show that some metrics tend to outperform others. Ding et al. (2021) previously showed that Procrustes tends to outperform CKA and CCA-based methods language models. our experiments on larger image classification models, both CKA and Procrustes tend perform better than regularized CCA-based methods adapted for larger representations. These results are hinted at by Ding et al. (2021) who show that CKA and Procrustes perform well on image decoding tests, but omit comparisons CCA-based metrics. Interestingly, CKA and Procrustes did not outperform other methods in all of our experiments. For AlexNet, network performance tests show that all the similarity metrics equally capture network function (Figure D In other words, the best representation similarity metric can change according to the network tested and the functionality used. To employ representation similarity metrics for interpretability, metries should be developed that can capture the functional properties of representations across many networks. These causal tests can help future interpretability studies identify similarity metries that are sensitive to causal function.
5 LIMITATIONS
We acknowledge the following limitation in this work. Reformulating CCA-based measures to accommodate representations with more neurons than examples required using regularized version of CCA called "ridge" CCA. In utilizing "ridge" CCA we had to choose regularization penalties to apply each representation matrix. These penalties were chosen be consistent across all tests, but
8

---

Under review as a conference paper at ICLR 2023
AlexNet
0.4
Correlation between metric and functionality
0.0
Mean CCA x Network deficit
Mean Squared CCA Network deficit
0..4
PWCCA Network deficit
Network deficit
MobileNetv2
Network deficit
Decoding
Mean Squared CCA Decoding deficit
PWCCA Decoding deficit
CKA. Decoding deficit
Procrustes Decoding deficit
0.4-
0.24
0.0
Network
Decoding accuracy deficits
performance deficits
Functionality
Figure Pattern of similarity metries differs depending on functionality measure. This figure shows significant interaction between functionality and metric for Alexnet (top) and a non- significant interaction MobileNetV2 (bottom). In the case of AlexNet, each of the five metrics behave differently depending on functionality type: the five similarity metrics within the network performance deficits condition are not significantly different, whereas metrics differ within decoding accuracy deficits condition. Specifically. the CKA and Procrustes metric is significantly different from the CCA--bsed metrics, only in the decoding accuracy deficit condition. In contrast, in the of MobileNetV2, this interaction is longer prominent: CKA and Procrustes consis- tently outperform (CA-based measures on both functionality tests MobileNetV2. Note that all CCA-based metries regularized in our formulations.
were not cross-validated. Future works would benefit from testing more penalty settings to explore their effect similarity results.
CONCLUSION
Taken altogether, the results of this study suggest that representation similarity metrics may already serve well as tools for comparing the geometries of representational spaces, but could be enhanced order to capture non-linear network function. Similarity metries achieve high correlations with linear probe decoding accuracy changes in representation induced by ablation. This sensitivity reveals that existing similarity metrics do a good job predicting how useful two representations will be for linear downstream tasks. However, trained networks do not necessarily use the features of a representation that are relevant for linear decoding during inference Hayne et al. (2022): Zhou et al. (2018); Meyes et al. (2020); Donnelly & Roegiest (2019). This discrepancy reveals that rep- resentation similarity metrics could be improved by taking into account the non-linear features of representations used during inference.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of the current work, we have included the code used to conduct and run each of the experiments with the submission. addition to the code, we have also included in the submission the raw data collected during each of our conducted experiments. At the request of reviewers or other interested parties, additional data sources (i.e. the exact image set used, the representation matrices, etc.) can be made available.
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint 1rXiv:1610.016444 2016.
Micah Allen, Davide Poggiali, Kirstie Whitaker, Tom Rhys Marshall, Jordy van Langen, and Ro- gier A. Kievit. Raincloud plots: multi-plattorm tool for robust data visualization. Wellcome Open Research, 4(63), 2021.
Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. Fitting linear mixed-effects models using 1me4. arXiv preprint urXiv: 2014.
Enric Boix-Adsera, Hannah Lawrence, George Stepaniants, and Philippe Rigollet, Gulp: prediction- based metric between representations. arXiv preprint arXiv:2210.06545, 2022.
MohammadReza Davari, Stefan Horoi, Amine Natik, Guillaume Lajoie, Guy Wolf, and Eugene Belilovsky. On the inadequacy of cka as a measure of similarity in deep learning. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022.
Frances Ding, lean-Stanislas Denain, and Jacob Steinhardt. Grounding representation similarity with statistical testing, arXiv preprint arXiv: (tt::2108.00100 000601 2021.
Jonathan Donnelly and Adam Roegiest. On interpretability and feature representations: an anal- ysis of the sentiment neuron. In European Conference on Information Retrieval, pp. 795-809 802. Springer, 2019.
Yunzhen Feng, Runtian Zhai, Di He, Liwei Wang, and Bin Dong. Transferred discrepancy: Quanti- fying the difference between representations. arXiv preprint arXiv:2007.12446. 2020.
David Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural computation, (6(12):2639-266 2004.
Lucas Hayne, Abhijit Suresh, Hunar Jain, Rahul Kumar, and McKell Carter. Much easier said than done: Falsifying the causal relevance of linear decoding methods. arXiv preprint arX1v:2211.04367, 2022.
Simon Korbllith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Malte Kuss and Thore Graepel. The geometry of kernel canonical correlation analysis. 2003.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hoperoft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint arXiv: 155551300433 2015.
Richard Meyes, Constantin Waubert de Puiseau, Andres Posada-Moreno, and Tobias Meisen. Un- der the hood of neural networks: Characterizing learned representations by functional neuron populations and network ablations. arXiv preprint wrXbv:2004.01254, 2020.
Seung Hyun Min and Jiawei Zhou. smplot: An package for easy and elegant data visualization. Frontiers in Genetics, 12, 2021. doi: 22.89//fenee.222.802333.3
10

---

Under review as a conference paper at ICLR 2023
Ari Morcos, Maithra Raghu, and Samy Bengio. networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018a.
Insights representational similarity in neural
Ari S Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. arXiv preprint arXiv: 180.00800, 2018b.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. Interna- lional Conference on Learning Representations; 2020.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
JO Ramsay, Jos Berge, and GPH Styan. Matrix correlation. Psychometrika, 49(3):403-423, 1984.
Elena Tuzhilina, Leonardo Tozzi, and Trevor Hastie. Canonical correlation analysis in high dimen- sions with structured regularization. Statistical Modelling, pp. 1471082X211041133, 2021.
Hrishikesh D Vinod. Canonical ridge and econometrics joint production. Journal of econometrics, 4(2):147-166, 1976.
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. Advances neural information processing systems, 31, 2018.
Hadley Wickham. g8plot2: Elegant Graphics for Data Analysis. Springer- New York, 2016. ISBN 978-3-319-242772 URL
oir
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the importance of individual units in cnns via ablation. arXiv preprint arXiv://806.02891, 2018.

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
Network performance deficit
bookcase
brain_com coral
electric_far
fountain_ _een
grey. whole
layer
conv_
(z-score)
-2-
conv 2
conv
junco bird
schooner
siberian, husky
snail
toaster
conv 4
conv
dense 1
dense_2 2
-2
2
0
2
2
0
2
Decoding accuracy deficit
(z-score)
Figure 5: Decoding accuracy and network performance deficits correlate in later layers of AlexNet. This figure shows the the relationship between network performance deficit and decoding accuracy deficit across layers and classes in AlexNet, using Local Polynomial Regression Fitting. In later layers of the network, ablations that produce high decoding accuracy deficits also tend to produce high network performance deficits (as illustrated by the fitted lines in the later layers). This relationship does not typically hold for early layers. Note that some points are missing in the case where ablations for certain layer and class combinations did produce decoding deficits.
12


Reviews:
[{"review_id": "qLxiUN_ic3", "reviewer": "Reviewer_TGPW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper performs a comparative study of three representation similarity metrics (CKA, Procrustes and PWCCA) evaluated on representations generated by AlexNet and MobileV2Net. This work extends Ding et al. (2021) (which studies the same metrics on ResNet with CIFAR-10 test images and language models). The main contribution is the reformulation of these metrics to enable efficient computation on larger models with larger images. Further contributions include the comparison of the evaluation of these metrics on two functionality measures: linear probing accuracies and network performance deficits.", "summary_of_the_review": "This work provides some interesting insights on representation similarity metrics evaluated on CNNs. The novelty/originality needs further clarification, especially with respect to Ding et al. (2021). ", "strength_and_weaknesses": "Strengths\n\nThe paper is well written - the theory and results are presented clearly. The paper presents interesting insights into the performance of various similarity metrics on representations generated by CNNs with two functionality measures.\n\nWeaknesses\n\nThe novelty/originality is unclear.  The paper appears to extend the work of Ding et al. (2021) who performs similar comparisons of these metrics on ResNet with CIFAR-10 (omitting the analysis of these two functionality measures).  If the main contributions is the reformulation of the metrics and the inclusion of these two functionality measures - it would be useful to see how this compares to the work of Ding et al. (2021)  e.g. the computational resources used under the original formulations on ResNet or AlexNet vs the reformulation. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YkXCJvijBu", "reviewer": "Reviewer_u5hR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper address a representation similarity metric analysis for high-dimension features by reformulating the representation simiarity metrics such as CKA, Procrustes, and CCA-based methods. Also the authors validate the changes by ablated representation by comparing linear decoding and non-linear classification performances. They present the analysis results on two CNNs such as AlexNet and MobileNetV2 traind on ImageNet. \n", "summary_of_the_review": "Despite the topic importance, this paper has some room to be improved (See the weakness). However, I am not an expert in this topic, so I mainy decided my score based on paper organization. ", "strength_and_weaknesses": "### Strength\n- In-depth analysis on representation similarity is an important and fundmental topic on representation learning. It can contribute to model interpretability.\n- Representation similarity metrics for high-dimensional features is also a significant topic.\n- Experiments and analysis on ImageNet-scale data are valuable.\n\n### Weakness\n- [Major] This paper seems to be basically based on [Ding et al. 2021] and its extension to high-dimensionality by reformulating metrics and linear decoding analysis. However, the advantage of the high-dimensionality is not effectively showed. The relevant information on high-dimensionality is that the authors validate their method on AlexNet and MobileNetV2 trained with ImagNet only. It is hard to find the specific details on high-dimensionality despite the emphasis in the title.\n- [Major]  I am not sure high-dimensionality is mandatory in represenation similarity metric analysis because a feature dimension of a single layer of many Transformer-based models is from 768 to 2048. To show the efficacy, experiments on higher dimensional feature similarity might be required. For example, to emphasize the effects of high-dimensionality, the aughors can concate two layers features over several tens of dimensionality. \n- [Major]  The authors presented 10 classes results in Figures 2 and 3. It is not clear how to select 10 classes from 1k classes of ImageNet. Overall, the experiemental setup is not clear for reproducibility and understanding even if they provided the source code. \n- [Major]  Following [Ding et al. 2021], experiments on multiple seeds are required. \n- [Minor] Although the authors argue they validate their method on popular CNN models, AlexNet and MobileNetV2 are not popular nowadays. Because [Ding et al. 2021] presents the results on BERT, the authors need to consider the similar experiments. The experiments on ViT are ok. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7biI7BVNIR", "reviewer": "Reviewer_ZbX5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper conducts a series of experiments to study how well each metric captures changes to representations. Linear decoding probes and network performance are choosen to be the measurements. Authors also propose several modification to existing metrics to make it possible to evaluate high-diemntional representations with low computational cost. Experiments are conducted on AlexNet and MobileNetV2 pre-trained with ImageNet.\n", "summary_of_the_review": "- This paper has merits on reformulating existing metrics for advancing calculating high-dimentional representations and conducting experiments to observe the correlation between metric and functionality. Based on the experiments, authors find some new observations. However, I think the experiments are not sufficient to support the claims. The overall contribution is not enough for accepting this paper. ", "strength_and_weaknesses": "**Strength:**\n\n- The reformulation of CKA, Procrustes can be applied to high-dimentional representations.\n- Some observations are new compared to existing studies.\n\n**Weakness:**\n\n- I think this paper is a simple extension of [1] which follows the same testing protocal. The observations are also empirical without deeper explanation.\n- Apart from Image-net pretraining and classification, more experiments should be performed on different tasks to further verify the claims in this paper.\n- The organization can be improved. For example, Page 8 has so much blank areas. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject