Does Representation Similarity Capture Function Similarity?
Authors: Lucas Hayne
Heejung Jung
R. Carter
Venue: Accepted by TMLR ()

Abstract: Representation similarity metrics are widely used to compare learned representations in neural networks, as is evident in extensive literature investigating metrics that accurately capture information encoded in representations. However, aiming to capture all of the information available in representations may have little to do with what information is actually used by the downstream network. One solution is to experiment with interventions on network function. By ablating groups of units thought to carry information and observing whether those ablations affect network performance, we can focus on an outcome that mechanistically links representations to function. In this paper, we systematically test representation similarity metrics to evaluate their sensitivity to functional changes induced by ablation. We use network performance changes after ablation as a way to measure the influence of representation on function. These measures of function allow us to test how well similarity metrics capture changes in network performance versus changes to linear decodability. Network performance measures index the information used by the downstream network, while linear decoding methods index available information in the representation. We show that all of the tested metrics are more sensitive to decodable features than network performance. When comparing these metrics, Procrustes and CKA outperform regularized CCA-based methods on average. Although Procrustes and CKA outperform on average, these metrics have a diminished advantage when looking at network performance. We provide ablation tests of the utility of different representational similarity metrics. Our results suggest that interpretability methods will be more effective if they are based on representational similarity metrics that have been evaluated using ablation tests.

Lucas Hayne Published in Transactions on Machine Learning Research (04/2024)
Does Representation Similarity Capture Function Similarity?
lucas.hayyne@ccolruudocom eduu
University of Colorado Boulder
Heejung Jung University of Colorado Boulder
crejjung.uun@Baalrddd. edu
R. McKell Carter
mckell. careell.crrerree wwwloorrrdo..ooooooo.ooooooo.... edu
University of Colorado Boulder
Reviewed on OpenReview: https: //
// opeereveew..ee.... net/ forum? id= 1Y21A0hfia
Abstract
Representation similarity metrics are widely used to compare learned representations in neural networks, as is evident in extensive literature investigating metrics that accurately capture information encoded in representations. However, aiming to capture all the infor- mation available in representations may have little to do with what information is actually used by the downstream network. One solution network function. By ablating groups of units thought to carry information and observ- ing whether those ablations affect network performance, we can focus on outcome that mechanistically links representations to function.
experiment with interventions on
In this paper, we systematically test representation similarity metrics to evaluate their sensitivity to functional changes induced by ablation. We use network performance changes after ablation as a way to measure the influence of representation on function. These measures of function allow to test how well similarity metrics capture changes in network performance versus changes to linear decodability, Network performance measures index the information used by the downstream network, while linear decoding methods index available information in the representation.
We show that all of the lested metrics are more sensitive to decodable features than network performance. When comparing these metrics, Procrustes and CKA outperform regularized CCA-based methods average. Although Procrustes and CKA outperform average, these metrics have diminished advantage when looking at network performance. We provide ablation tests of the utility of different representational similarity metrics. Our results suggest that interpretability methods will be more effective if they are based on representational similarity metrics that have been evaluated using ablation tests.
1 Introduction
Neural networks already play critical role in systems where understanding and interpretation are paramount like in self-driving cars and the criminal justice system. To understand and interpret nenral networks, learned representations are compared using representation similarity metrics (RSMs) (Kornblith et al. (2019):
Raghu et al. (017)); Morcos et al. (2018b); Wang et al. (2018); Li et al. (2015); Feng et al. (2020): Nguyen et al. (020 Using these similarity metrics, researchers evaluate whether networks trained from different random initializations learn the same information, whether different layers learn redundant or complementary
information, and how different training data affect learning (Kornblith et al. (2019); Li et al. (2015): Wang
et al. (2018)). Furthermore, RSMs are also used compare computational models and biological networks [Kriegeskorte et al. (2008)) to determine the ability of networks to model brains.

---

Published in Transactions on Machine Learning Research (04/2024)
In a prototypical example comparing neural systems, Raghu et al. (2021) used RSMs to help answer the question: "Do vision transformers see like convolutional neural networks?" Put another way, the researchers aim to answer whether the two networks function in a similar way. Two networks can be shown to function in a similar way when two criteria are met. First, the two networks must compute similar representations. What it means for two representations to be similar, however, not straightforward. In response to this vagueness, many RSMs have been developed with different notions of similarity over the past decade. Dif- ferent similarity metrics employ different strategies and make different assumptions, meaning that different similarity metrics might not always agree. For example, some similarity metrics are invariant under invertible linear transformations while others are not (see Kornblith et al. (2019) for a theoretical comparison). Cap- turing the various notions of representational similarity has been the primary driver behind developing and lesting RSMs. These different assumptions and strategies can lead to quantilatively different predictions. For instance, Davari et al. (2022) demonstrate that the centered kernel alignment metric predicts a high similarity between random and fully trained representations. It is unclear which representation similarity metrics capture the most important information from representations, further tests are needed to evaluate them.
However, to address the original question: whether two networks function similarly, an additional sccond criteria must be met. Apart from computing similar representations, the two networks must also use their representations in a similar way. This second criterion has been mostly neglected when considering the similarity of network function using RSMs. In this paper, we aim to investigate whether RSMs can tell us when this second criterion is met. In other words, how likely is it that two representations deemed similar by RSMs are actually used the same way by trained networks?
In this work, we aim to quantify how likely is it that two representations deemed similar by RSMs are actually used the same way by trained networks. Previous studies have tackled this question by making assumptions about what kinds of information networks use. If this information is detectable using RSMs, then RSMs can be used to determine if two reprosontations are used by the network in the same way. For instance, RSMs have been evaluated for their ability to track class separability in representations (Boix-Adserr) el al. 2002 Ding et al.) (202I); Feng et al. (2020)). To measure the class separability in a representation, researchers usually train linear probes for downstream tasks on learned representations and compare the results. However, the features of a representation that carry the most information for linear probes may not be those actually used by the network during inference. Studies that remove features from representations in trained net nerworks have revealed a weak link between the relevance of a feature for decoding and its effect when removed from the network (Meyes et al. (2020] : Zhou et al. (2018); Donnelly & Roegiest (2019); Morcos et al. (2018b) ). Hayne et al. 2022) recently showed that linear decoders specifically cannot. single out the features of representations actually used by the network. Consequently, two representations that are equally decodable using lincar probes may not actually be equal from the point of view of network performance.
If networks neglect linearly decodable information, then RSMs that track linear class separability may not satisfy the second criterion mentioned previously. Remember the second criterion says that to infer the functional similarity of two networks from RSM scores, one must also show that networks use similar representations in similar ways. In this work, we propose using ablations to directly test if RSMs satisfy the second criterion. We use ablation to evaluate how closely changes in RSM scores are related to changes in network performance. We first ablate groups of units from cither AlexNet, MobileNet, or ResNet, compare the original representations to the ablated representations using RSMs, and then compare scores to the changes seen for network performance (see Figure 1). As baseline, we also compare RSM scores to changes in class scparability in the representation using lincar probes. This process reveals the extent to which RSMs satisfy the second criterion, tracking functional similarities as opposed to just linear similarities between representations. This distinction is crucial for neural network interpretability where the aim is to develop human-undeestandaabl descriptions of how neural networks actually rely on their internal representations.
In this work, we show that CKA, Procrustes, and regularized bCsed representation similarity metrics predict network performance changes significantly worse than linear decoding changes. We also show that, average, Procrustes and CKA outperform regularized CCA-based methods. The advantage of using Procrustes and CKA is significantly diminished, however, when considering network performance tests. In fact, Procrustes and CKA fail to outperform regularized CCA-based metrics in AlexNet. Overall, our results
2

---

Published in Transactions on Machine Learning Research (04/2024)
suggest that interpretability methods will be more effective if they are based representational similarity metrics that have been evaluated using ablation tests. In general, this paper documents the following contributions:
We introduce a new test of the utility of representation similarity metrics. We find that five popular representation similarity metrics are significantly less sensitive network performance changes induced by ablation than linearly decodable changes.
Within the tested metrics, we show that Procrustes and CKA tend to outperform regularized CCA- based methods for capturing functional similarities between representations, but that tests using linear probes and network performance based functional measures can produce different results in different networks.
2 Methods
Here, we described the methods necessary for correlating representational metries with changes in decod- ability and ablation effects. In Section 2.1 describe the statistical testing methodology used in experiments. In Section 2.2 we introduce the representation similarity measures we evaluate and reformu- late them for use on high dimensional representations. In Section 2.3 we describe how we use ablation to produce representations with different functional properties. Finally, in Section 2.4] we describe how we use linear probe decoding deficits and class-specific performance deficits to measure decodable and downstream network changes, respectively, the ablatod representations.
2.1 Statistical testing
Assume E Rexp: represents a matrix of activations for neurons given n examples, and B E RXxp= represents a matrix of activations for P2 neurons given the same n examples. The matricos and B are called representation matrices and have been preprocessed to have centered columns. Let RSM(A, A, B) denole representation similarity metric that returns zero and only if B and for which RSM(A, B) RSM(B, A). These metrics do not satisfy the triangle inequality and are therefore not formal distance metrics. For simplicity, will refer to them metrics in this work.
Separate from representational differences, we also seek to quantify how changes in representation affect function. To quantify functional differences between representation matrices, we use functional behavior measures. Formally, let f maxp + IR denote a functional behavior measure that, given a representation matrix, returns a scalar measure of the representation's role in function. In this study, we utilize two functionality measures, class-specific linear decoding accuracy (fDee) and class-specific network performance (fret). In the case of linear decoding accuracy, fDee returns the average linear probe accuracy achieved from decoding a target class identity from representation matrix. On the other hand, fPeet returns the average classification performance for target class achieved by feeding the representation matrix to the network at the appropriate layer. More details are presented in Section 2.4
To change the representations themselves, we perform interventions on a given representation Pearl (2009h). Interventions on Z come in the form of zero ablations where entire features (columns of Z) are set zero using mask producing an ablated representation A where A M Z. The functionality of the masked representation is evaluated using linear decoder (fDee) and the downstream neural network (Spert). We represent the zero ablation intervention on functionality with the do(.) operator. For each image 2, we compute the predicted target class rank re(r) for the target class t. For instance, if the CNN's softmax layer predicts that the Junco bird class was the third most. likely class given an image of a Junco bird, then that image receives a rank of three (ri(z) = 3). The impact of the ablation on function registered as the difference predicted target class ranks produced by the intervention on the representation.
Aublate [rx(x) (o)()
(1)
3

---

Published in Transactions on Machine Learning Research (04/2024)
Collect reprasentations
Measure representation sirrilarity & function
Generate measure of representation similarity & function
Functionality. Network Performance Penti - 91.3%
PertX PerfiA)
(0.17.5
PertA)
Features
Per((X) - Perl(A) 7.5%
...and Conduct statistical tests
rm
Inguu
X
Representation
Correlation between metrij and funitiionaltt
Functionality: Linoar Docoding
RSMX A]
DeepQ
Deo/A 82.7%
Fisatures
Dec() Dec(A) 6.5%
Decp0 Dec(A)
irouu
A
(0.1.
Ablated Reprosentation
Metric: Representational Similarity
Network perf deficits
Decoding acc. deficits
CKA
e.g. using CKA metric
Procrustes
RSM(X, A) 0.1
RSMX, A)
Figure 1: Elements of the experimental design. In the first phase, a representation matrix X is extracted from layer in a trained network and we then generate ablation matrices A by delcting features from the representation. In the second phase, representation matrices and are compared in three ways. To measure how the representations differ the context. of the network, the representations are fed back into the network and the network performance difference between them is calculated. Next, linear probes are also fit to X and A to decode target class and the linear probe accuracies are compared. Additionally, representation metric similarity is calculated between X and A. In the last phase, representation similarity and function are compared. By comparing metric similaritics with both lincar probe decoding changes and network performance changes across many ablations (represented with multiple points in the correlation plots), we can assess to what extent each metric can be used to infer functional similarity from representation similarity or whether they merely capture changes in class separability. Finally, statistical tests are performed the correlation values in each experiment.
Similarly, for linear decoders, the impact of ablation on class separability is registered as the drop in linear decoder accuracy after retraining on the ablated representation (sce Scction 2.4 for more details).
As in Ding et al. ) aim statistically test. representation similarity metrics using similar method- ology according to their paper:
Extract a representation matrix X XPppptt from a given layer in neural network.
Collect a set representations by masking groups of neurons in to produce A, E P.PPaalltedd
where Pablated < Pinput (see Section 2.3
3. Compute the following for all A:
D, RSS((X.,)
FPe (X,,) |f (X) fFerr (A:)l
F (,A.)) = (X) - (Ae)
4. Compute the Spearman rank correlation between and Frerf and D and F
4

---

Published in Transactions on Machine Learning Research (04/2024)
This procedure quantifies the extent to which the representation similarity metries, RSM(-), capture the functionality differences, as measured by (')rr(+) and SDea()), produced by ablating the representation matrix. A high Spearman's 's rank correlation value between a metric's computed representation similarities and the functionality differences produced by ablation implics that the chosen metric is sensitive to the chosen funct ionality. Whereas, low correlation implies the opposite: that the chosen metric not sensitive the chosen functionality.
2.2 Representation similarity metrics
As in Ding et al. (021]; we study three main representation similarity metrics: centered kernel alignment (CKA), Procrustes, and canonical correlation analysis (CCA).
Centered kernel alignment (CKA) is based on the idea that similar representations also have similar relations between examples. In other words, representation matrices that store images of lettuce and rabbits using similar vectors should be more similar to each other than with representation matrices that encode images of lettuce and dinner plates using similar vectors. This idea leads Kornblith al. (2019) to formulate linear CKA, which uses a linear kernel to compare example vectors (henceforth referred to as just CKA):
A AT B (A, || A B
(2)
where || || is the Frobenius norm and || derives from the following relation:
(vec(AAT), vee BB tr (rAAABBB) B|
(3)
Relation 3 which was derived by Kornblith et al.) (2019) shows that the similarity between pairwise example similarity matricos (far left) equal to the squared Frobenius norm of the feature covariance matrix between representations (far right). Kornblith et al. (2019) use this relation to form Equation 2 which measures the normalized similarity between the example similarity matrices of A and B. Unfortunately, computing and storing either B, ATA, or 8 B can be prohibitively expensive when both p1 and p2 grow too large. Therefore we reformulate Equation using relation from Kornblith et al. (2019) and the fact that |X ||F = |XXXXX x into:
=1 trace AAT
(4)
|AAT BBT |F
This reformulation allows us to use CKA on layers with n < P1.P2
Procrustes an analytical solution to the orthogonal Procrustes problem which involves finding a right rotation of matrix B that is as close possible to A measured by the Frobenius norm:
dprocrustes (A, B) = || 1 + BI - -2| AT
(5)
where I|. is the nuclear norm. As with CKA, T needs to be replaced to lighten the computational cost of working with large layers. Therefore, we utilize the fact that the nuclcar norm of a matrix is the sum of singular values reformulate Procrustes:
(onteruuttssAAABB dprocrustes (A, = =| + B 2 Σ V A(AATBBT)
(6)
where Xx represents the ith eigenvalue of matrix X. Again, this reformulation saves us from manipulating the large Pı x P2 matrix by replacing it with a much smaller n x n matrix (assuming n < P1.P2).
Canonical correlation analysis (CCA) provides a solution to the problem of linearly projecting and B into shared subspace where their correlations maximized. CCA finds min(p1,P2) P2) pairs of weight
5

---

Published in Transactions on Machine Learning Research (04/2024)
vectors (WA and the resulting correlation induced by projecting A and B using the 1th weight vector 18:
A max (uw s.t. Aeei Aw's Aw', Bwi Bw "A.'B
(7)
where the is maximized subject to the constraint that the subspace features be orthogonal. Equation can be solved for by performing singular value decomposition on (ATA) 1/2 ATB((BB) 1/2 where the singular values are equal to the correlations (p: v E [I,.., pin(p
However, the inverses of the feature covariance matrices do not exist when the number of neurons use regularized or "ridge" CCA (Vinod)
exceeds the number of examples. For these cases, (9766), which applies an La penalty to the weight vectors and can be solved by performing SVD on A -1/2 A B -1/2 However, we again run into the problem that the feature co- variance matrices are too costly to compute for large layers. So, we employ the "kernel trick" introduced by Kuss & Graepel (2003) and Hardoon et al. (2004) and refined by Tuzzilina et al. (2021) which allows us substitute into the above expression RA for A where RA represents an n X n matrix recovered by applying SVD on A, i.e. A = R The same trick can be applied to B. The kernel trick (Tuzhilina et. al. (2021) makes CCA computationally tractable for large layers. The only caveat is that if we apply the "kernel trick" to both matrices, we recover only n canonical correlations rather than min P2).
Mean CCA (as used by Raghu et al. (017 and mean squared CCA Ramsay et al. 1984) average raw and squared correlations recovered through CCA, respectively. Projection-weightee canonical correlation analysis (PWCCA) is special case of canonical corrclation analysis (CCA) proposed by Morcos et al. (2018a). PWCCA re-weights each correlation value by its importance for the underlying representation. Formally, if representation matrix A has neuron activation vectors (21,..., Ep,] and CCA vectors [h1, hn): then PWCCA computes weighted mean
dPwCCA (A, BA B) 1- Σ Tapi s.I. ai =
Σ (hi 1z)l
where āj is a normalized version of oz. The preceding CCA reformulations represent regularized forms of CCA-based metrics useful for high-dimensional representations, which we refer to as regularized CCA-based metrics in the text, but just CCA metrics in the figures for sake of brevity.
There is some concern that the regularized approximation could introduce errors into our results that interfere with the conclusions of the paper. Observing different penaltics for regularizing the CCA metrics, we find that the exact CCA values do change when the penalties change. However, in the current study, we are not concerned with the exact CCA values, only the relative order of scores given an ablation. We find that CCA metrics regularized with different penalties highly correlated (Spearman rank correlation > 0.95) and therefore the chosen penalty does not interfere with the main results.
2.3 Ablation
To study representation functionality changes, needed method for manipulating representations to reliably produce network performance deficits at the output of the network. To reliably produce network performance deficits, we followed the procedure of Hayne et al. (2022). Specifically, for cach of the 10 or 50 randomly chosen (argel classes and layer of the CNNs we lested, AlexNet, MobileNet and ResNet55, we first projected every neuron onto two dimensions: class selectivity and activation magnitude. Then, we constructed a grid overlay on the activation space so that each cell of the grid contained the same number of neurons. To supply the set A of representation matrices from Section 2.1 we ablated one cell of neurons at time by setting the activation values for those neurons to zero.

---

Published in Transactions on Machine Learning Research (04/2024)
2.4 Functionality measures
After collecting the set & of ablated representation matrices, we sought to compare two functionality mea- sures. First, we fit linear probe decoders to each represenlation malrix in A. With the linear probes we aimed to decode the identity of one target class, so we fit simple logistic regression classifiers to distinguish the target class representations from all other representations in the representation matrix. Because many of the CNN layers contained thousands of Teatures, we randomly selected 100 neurons as features the lo- gistic regression model and averaged the training accuracy over 200 repetitions as in Alain Bengio (2016). Intuitively, the accuracies measure how linearly separable the target class is from other classes on average given the features in representation. We refer to the changes induced in the decodability of the target class by ablation as decoding accuracy deficits.
In contrast to linear decoding accuracies, we also measured the network's class-specific classification per- formance when utilizing the representations in the representation matrix during inference. Specifically, we recorded the average classification rank of the target class. For instance, if the CNN's softmax layer pre- dicted that the Junco bird class was the third most likely class given an image of a Junco bird, then that image received rank of three. The average class-specilic ranks intuitively measure how the representation was used by the network. We refer to the changes induced in class-specific classification rank by ablation network performance deficits.
2.5 Networks and data
For experiments, we investigated AlexNet (Krizhevsky et al. (0012) ), Ve
(Sandler et al. (0188)
and ResNet50 (He et al. (2016)) all pre-trained on ImageNet (Deng et al. (2009) > F We chose these three networks as a representative sample of three convolutional networks with different architectures that are still widely studied for producing, lesting, and analyzing representation similarity metrics (Kornblith et al. (2 Raghu et al. (2021): Boix-Adsera et al. (2022)) ). AlexNet and MobileNet were built using Keras and included lambda masking layers after each parameterized layer to selectively ablate unit groups. ResNet50 was built using Pytorch with forward hooks applied the output of each block and used to perform ablation. For MobileNetV2 and ResNet50, 10 classes were randomly chosen from ImageNet and all the images from the validation set were used for our analysis. For AlexNet, 50 random classes were chosen. We choose 50 classes for AlexNet and 10 classes for MobileNet and ResNet to strike a balance between convenience and breadth. Given the large layer sizes of MobileNet and ResNet (~100,000 neurons), adding more classes increases the cost of computing RSM scores. To help avoid bias in our small sample of classes, we sample the classes completely randomly from ImageNct. We note that previous studics include a comparable number of images in their analysis (Raghu et al. (2021)).
3 Results
In this section, we detail the results of aggregating our statistical tests across each layer and class from AlexNet,, MobileNet and ResNet50. We conducted a hierarchical linear model to evaluate the prediction of the rank correlation values as a function of two factors: 1) metric, i.c. CKA, Procrustes, and rogularized CCA-based methods and 2) functionalily, i.e. network performance deficits and decoding accuracy deficits. To allow for comparable correlation coefficients, we Fisher-Z transformed each rank correlation value. This transformation normalizes the distribution of coefficients, which makes it suitable for averaging and executing further statistical analyses. After Fisher-Z transformation, we implement hicrarchical linear model function of funalit and metric, while allowing for random intercepis and random slope of functionality. The model is as follows:
1 AlexNet pre-trained weights from http:://ttthhi wwtpc///urrtteec/coceeteteeeeeee MobileNet weights downloaded from http://eeraa..taappppapppcccoooo ResNet55 weights were downloaded from org/hub/pytoorc. visioo_resned

---

Published in Transactions on Machine Learning Research (04/2024)
FishorZzy = 9o + 31 x Functionality, ij + x Metrici + x (Functionalityy, x Metricc)
(8)
+ Uuj + W1j x Functionality, +R,
Unj Uij
~ N (00
Top
Pog2s Tuj T1j] )
(9)
P15.01 x Fiz x Tos
Ti
Rig ~ N(0,
(10)
Class j: the unit of analysis, also referred to as clusters. Given that observations from the same cluster are bound to be non-independent, we model classes as our unit of analysis. FisherZij: the dependent variable for the i-th observation in the j-th class. Functionality.y: the contrast code to represent diffcrent levels of functionality for the i-th observation the j-th class. Metricy;: the rost codes to represent levels of metrics CKA, Procrustes, and regularized CCA-based methods of mean CCA, mean squared CCA, and PWCCA for the i-th observation in the j-th class. Bo B1, 82, A3: the fixed effect cocfficients for the intercept, Functionality, Metric, and their interaction respectively. For instance, 31 represents the average difference between Network performance deficits and Decoding accuracy deficits across classes. Moj: the random intercept for the j-th class. U1j: the random slope for the Functionality factor, for the j-th class. In other words, we allow for the functionality effects to vary across classes, c.g.; the average difference between Network performance deficits and Decoding accuracy deficits can differ across classes. The random effects vector uoj U1j 1 is multivariate normal distribution with a mean of 0 respectively and covariance structure with 1) Toj the variance for the random intercepts (uoz), 2) Tig. the variance for the random slopes (W1j), and 3) the covariance of the random intercepts and slopes, indicated as the correlation between the random slopes and the random intercepts Pij.0j multiplied with the standard deviation of the random intercepts Toj and random slopes TIj. Ray: the residual error term for each observation, after fitting the model with aforementioned parameters.
For visualization, we plot the raw correlation coelficients, for better interpretability. We average the correla- tion coelficients within classes, i.e., the unit of analyses, across layers for each each metric and functionality. Each data point represents unique class, plotted as function of the factor of interest. Distribution of the class-average correlation coefficients are plotted to visualize the spread of the data per factor. Boxplots indicate the median, and the 25th and 75th percentile of the class data distribution (i.e. inter-quartile range; IQR). The black whiskers represent 1.5 IQR.
3.1 Representation similarity metrics are significantly less sensitive to network function
Figure 2 shows the distribution of correlation values across functionality measures. In all cases, the rank correlation values of both functionality measures are significantly different from O, suggesting that the cor- relation values are sensitive to functional behavioral changes in the representations. The analysis of interest demonstrates these correlation values are significantly different between network performance deficits and decoding accuracy deficits, when averaging across the five similarity metries CKA, Procrustes, and regu- larized CCA-based metries. other words, there is significant main effect of functionality. The distribution of Spearman correlation values between the metrics and each functionality for each network are shown in
Figure 2
3.2 CKA and Procrustes tend to outperform other metrics
Figure 3 shows that CKA and Procrustes have, on average, higher rank correlation values compared to the other CCA-based metrics. Again, higher rank correlation values indicate a higher coupling between functionality measures and representation similarity metrics, thereby suggesting that CKA and Procrustes are better metrics at capturing functionality in general.
8

---

Published in Transactions on Machine Learning Research (04/2024)
3.3 Functionalities produce different results for different networks
Figure (left) shows significant interaction between functionality mcasures and similarity metrics for Ale..el.. Figure (middle) shows the same model for MobileNet however, interaction is non-signilicant. Figure (right) shows the conceptually identical interaction, which is significant for ResNet.
In the case of AlexNet, with the plotted mcans, it is evident that the interaction is driven by the metric for the similarity metrics within network performance
differences within decoding accuracy deficits. deficits, no single metric outperforms the other; instead, each metric achieves a similarly low correlation amongst the network performance deficit group. As for the decoding accuracy deficit group, on the other hand, CKA and Procrustes outperform the CCA-based metrics. In the case of MobileNetV2 and ResNet, there is a significant interaction where the difference between CKA/ CKA/Procrusies and the CCA-base metrics for network performance tests is significantly different from decoding tests.
4 Discussion
this work, we systematically test representation similarity metrics on ImageNet-trained CNNs to determine how sensitive they are to network performance and decoding changes in representations. These tests help answer: how likely is it that two representations deemed similar by RSMs actually used the way by trained networks? Our tests revealed that, while similarity metries are significantly sensitive to network performance and decoding measures of functionality, they are significantly less sensitive to network performance. In other words, RSM scores are somewhat predictive of what ablations will have the largest effects on network function, but are comparatively much better at predicting changes in class separability due to ablation. Additionally,
most of our tests CKA and Procrustes significantly outperform CCA- based methods at predicting functional changes. However, do find that tests of network function tend significantly reduce and, in the case of Ale eliminate their advantages.
AlexNet
MobileNetV2
ResNet50
Correlation latinn between metric and functionality
0.6
Netwarkkarfarmmnnc deficiss Decoding accuracy deficis Network Functionality
Network performance deficts Decoding accumcy deficiis Networks Functionality
Netwarkk performance defitss Decading acouracy deficts
Functionality
Figure 2: Correlation values overall are significantly sensitive to ablation changes in the repre- scntations, however, decoding accuracy has a higher value than network performance deficits. This figure shows the distribution of rank correlation values between functionality measures, i.e., the main effect of functionality measure on AlexNet (a), MobileNetV2 (b), and ResNet50 (c). Each data point rep- resents value from the ten classes; within each class, we average the five metrics of CKA, Procrustes, PWCCA, mean CCA and mean squared CCA. The distribution is further depicted by the violin plots and boxplots, which illustrates the median and upper/ upper/lower quartile of the distribution. Overall, both function- ality measures are significantly different from O, indicating that the rank correlation values are sensitive to changes produced by ablations in the aepsesentations (F(1,49) = 437.31, < .001 for AlexNet, F(1,9) = 1139.43, p < ,001 for MobileNetV2.. F(1,9) = 222.62, p < .001 for ResNet55). Further examining the differ- ence between functionality mcasures, we see that the rank correlation values of decoding accuracy deficit are higher than that of network performance deficits (F(1,49) 79.22, p < .001 for AlexNet, F(1,9) 669.25 p < .001 for NeV,, and F(1,9) 31.60, p .001 for ResNet50). Note that all CCA-based metrics are regularized in our formulations.

---

Published in Transactions on Machine Learning Research (04/2024)
AlexNet
MobileNetV2
ResNet50
Correlation between metric and functionality
0.8
0.6
0.4
0.2
0.0
Mean CCA
Mean
PWCCA
CKA Procrustes
Mean
Mean
PWCCA
CKA Procrustes
Mean
Mean
PWCCA
CKA Procrustes
squared
CCA
squared
CCA
squared
Metric
Metric
Metric
CCA
CCA
CCA
Figure 3: CKA and Procrustes outperform CCA-based metrics. This figure shows the distribution of rank correlation values across each metric, averaged between functionality measures AlexNet (a), MobileNetV2 (b), and ResNet50 (c). Within each distribution, each data point represents the average rank correlation value from the ten classes, collapsed across network performance deficits and decoding accuracy deficits. Higher rank correlation values in the CKA and Procrustes condition indicate that these metrics are sensitive to the perturbations the representations compared to other CCA-based metrics (I(3392) = 9.00, p ,001 for AlexNet; (101322) = 13.38, p < .001 for MobileNet f(1572) 25.41, p < .001 for ResNet550. Note that all CCA-based metrics are regularized in our formulations.
Previously, Ding et al. (2021) performed some general stress tests on representation similarity metrics. In their work they showed that PWCCA distances were easily influenced by changes to random scod, predicting that the same layers in two different networks were more similar to other distant layers in the network than to each other. In another test, the CKA metric suffered from different problem where it failed to distinguish representations from their low rank counterparts. Davari et al. (2022) pointed out other weaknesses associated with CKA. Among other findings, they demonstrated that CKA judged random and fully trained representations to be highly similar, especially in early layers of a network. Both studies used functional and intuitive tests for evaluation.
Although our work utilizes a framework introduced by Ding et al. (2021). we aim to test a fundamentally different hypothesis not addressed by their tests. Specifically, we aim to test how sensitive representation similarity metrics are to the functional propertics of reprosentation actually used by trained network during inference. In our tests we directly compare linear probe accuracy changes on in-distribution inputs to network performance changes on the same imputs after ablation. On the other hand, Ding et al. (2021) do not test this direct comparison. Instead, they perform more general stress tests on similarity metrics using either in-distributtin linear probes or out-of-distribution network performance scores and different methods of perturbing representations. Through this direct comparison, we are able show that similarity metrics are significantly less sensitive to the features of representations that are actually used by the network compared to its decodable features.
All of the RSMs we tested are significantly less sensitive to network performance changes induced by ablation than changes in class separability induced by ablation. This finding reflects the considerations made in developing these similarity metrics. RSMs were designed to compare the linear geometric properties of two representation spaces. So, it is not surprising that similarity metrics correlate with changes in decoding accuracies. On the other hand, network performance measures of function reflect how the network utilizes representations. In this case, functionally similar representations similar after series of non-linear transformations through layers of the network. Perhaps it is not surprising that this non-lincar notion of similarity is harder to capture using current similarity metrics. However, it is the ultimate goal of interpretability to link representation and non-linear network function.
those representations that remain
Previously, Bansal et al. (0021 proposed model stitching as a method for measuring the difference between two representations. Model stitching works by freezing two models and "stitching" the bottom half of one the top half of another. Using this procedure the difference between the representation learned by the first model and the representation learned by the second at particular layer is the drop in performance observed
10

---

Published in Transactions on Machine Learning Research (04/2024)
AlexNet
NebileNetV2
ResNet50
Correlation between metric and functionality
0.9
0.6
0.3
0.0
-0.3
Network performance
Decoding accuracy deficits
Network performance deficits
Decoding accuracy deficits
Network performance deficits Functionality
Decoding accuracy deficits
deficits
Functionality
Functionality
Mean Network performance detich
Network parfermence defich
Moan squared CCA x Linear decading detit
Moan squared ODA Network performance defici
Procrustes X performanos defitt
PWCCA X Linear decading defisi
PWOCA Network
CCA Lineer decoding defici
decoding
Linear decoding
Figure 4: Pattern of similarity metrics differs depending on functionality measure. This figure shows a significant interaction between functionality and metric for Alexnet (top; F(4, 3392) = 23.67,, p < .001.), for MobileNett2 (middle; F(4, 10132) 4.59, p < .005.), and for ResNet50 (bottom: F(4, 1572) 98.69, p .001). In the case of AlexNet, each of the five metrics behave differently depending on functionality type. The five similarity metrics within the network performance deficits condition are not significantly different, whereas metrics do differ within the decoding sccuracy deficits condition. Specifically, the CKA and Procrustes metric is significantly different from the CCA-based metrics, only in the decoding accuracy deficit condition (t(3392) = 13.10, < .0001). In both MobileNetV2 and ResNet50 models, there is also a significant interaction effect. Notably, CKA and Procrustes consistently outperform CCA-based measures, with a pronounced difference in decoding-basse based tests compared to network- ones. Specifically, in decoding- bassed tests, the average correlation differences are 0.16 for MobileNet and 8 0.65 for ResNet50 for CKA/Procrustos vs. CCA-based mcasures. In contrast, for network-basod tests, these differences are smaller, with 3 0.0865 for MobileNedV2 and B 0.0817 for ResNet50 with diminished advantage for CKA and Procrustes based on network performance deficits compared to decoding accuracy deficits. However, CKA and Procrustes still significantly outperform CCA-based metrics on tests of network performance (f(10132) = 6.534, p < ,0001 for MobileNetV2 and t(1572) = 4.01, p < .0005 for ResNet50). Note that all CCA-based metrics are regularized our formulations.
for the stitched model compared to the original models. However issues arise when naively plugging one network into another. For instance, knowing which neurons in one network correspond to which neurons in another network a non-trivial problem. This especially true cases where the two network architectures differ. These problems can be overcome by retraining some portion of the downstream network, as done in Model Stitching. However, retraining brings with the danger that the downstream function will change given the new representation. This can cause the decoding network to hallucinate features that were not used by the upstream network.
The problem of overly powerful decoders is especially pertinent in neuroscience where decoding studies have been performed for many decades. Neuroscientists take extreme cantion to limit the power of their decoders dissentangle functions performed by the brain from those hallucinated by the decoder (Tong Pratte (2012)). By limiting the power of decoders, neuroscientists take the brain-centric perspective, restricting their search for brain function to computations the brain actually performs not those could perform under different cireumstances. We take similar approach in this study by not retraining the downstream network the ablated representation to reveal what information is actually used by the network and test RSMs to predict when this information is removed. Instead we find that RSMs are more attuned to changes in class separability revealed by linear decoders, the kind of computation that could be hallucinated by overly powerful decoders, but not used by the network.
Like previous studies, we show that some metrics tend to outperform others. Ding al. (2021) previously showed that. Procrustes tends to outperform CKA and CCA-based methods language models. Tn
11

---

Published in Transactions on Machine Learning Research (04/2024)
experiments on larger image classification models, both CKA and Procrustes tend to perform better than regularized CCA-based methods adapted for larger representations. These results are hinted at by Ding et al.] (2021) who show that CKA and Procrustes perform well on image decoding tests, but omit comparisons to CCA-based metrics.
Interestingly, the improved ability of CKA and Procrustes to capture function shrinks when tests of network function are applied. This reduced advantage occurs in every tested network and in AlexNet the advantage is completely eliminated. For AlexNet, network performance tests show that all the similarity metrics equally capture network function (Figure 4. This suggests that the advantages of CKA and Procrustes may be be overstated in previous literature. Furthermore, the best representation similarity metric may depend on the network used. To employ representation similarity metrics for interpretability, metrics should be developed that can capture the functional properties of representations across many networks. These tests can help future interpretability studies identify similarity metrics that can be used to infer the functional similarity of two networks based on their representational similaritics.
5 Limitations
We acknowledge the following limitations in this work. Reformulating CCA-based measures to accommodate representations with more neurons than examples required using a regularized version of CCA called "ridge" CCA. In utilizing "ridge" CCA we had to choose regularization penalties to apply to each representation matrix. These penalties were chosen to consistent across all tests, but were not cross-validated. Future works would benefit from testing more penalty settings to explore their effect on similarity results. In future work, would like to explore these hyperparameter settings to establish best practices for computing RSM scores on large layers. Additionally, the current study focuses exclusively on investigating convolutional neural networks trained on ImageNet. Additional experiments should be conducted to determine how well the current findings generalize to new tasks and networks.
6 Conclusion
Taken altogether, the results of this study suggest that representation similarity metrics may already serve well tools for comparing the geometries of representational spaces. However, RSMs are powerful tools that we would like to use to infer the functional similarity of two networks from their representational similarity, In other words, we would like to make the claim that two vision models "see" the same way if they have similar representations. In this work, we test RSMs using ablations on representations to determine if similar reprosentations in CNNs are actually used the same way by downstream networks. Similarity metrics achiove high correlations with linear probe decoding accuracy changes in a representation induced by ablation. This sensitivity reveals that existing similarity metrics do a good job of predicting how useful two representations will be for linear downstream tasks. However, trained networks do not necessarily use the features of a reprosentation that are relevant for linear decoding during inference (Hayne et al. 2022); Zhou et al. (0018): Meyes et. al. (2020); Donnelly & Roegiest (2019h). This discrepancy reveals that representation similarity metrics could be improved by taking into account the features of representations used during inference.
References
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arxiv preprint arXio:1610. 01644, 2016.
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural represen- tations. Aduances in neural information processing systems, 34:225-236 2021.
Enric Boix- Adsera, Hannah Lawrence, George Stepaniants, and Philippe Rigollet. Gulp: a prediction-based metric between representations. arXiv preprint crXiv:2210.065/5. 2022.
MohammadReza Davari, Stefan Horoi, Amine Natik, Guillaume Lajoie, Guy Wolf, and Eugene Belilovsky. On the inadequacy of cka as a measure of similarity in deep learning. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022.
12

---

Published in Transactions on Machine Learning Research (04/2024)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pullern recognilion, pp. Teee, 2009.
Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. Grounding representation similarity with sta- tistical testing. arXiv preprint arXiv:2108. N11:2208.001611, 2021.
Jonathan Donnelly and Adam Roegiest. On interpretability and feature representations: an analysis of the sentiment neuron. In European Conference on Information Retrieval, pp. 795 802. Springer, 2019.
Yunzhen Feng, Runtian Zhai, Di He, Liwei Wang, and Bin Dong. Transferred discrepancy: Quantifying the difference between representations. arXiv preprint urXiv:2007.12440 2020.
David R. Hardoon, Sandor Szedmak, and John Shawe- Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Newral compulation, 16(12))2639 2664, 2004.
Lucas Hayne, Abhijit Suresh, Hunar Jain, Rahul Kumar, and R McKell Carter. Much casier said than done: Falsifying the causal relevance of linear decoding methods. arXiv preprinl. Xrxi::2211 1222221111366., 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and paltern recognition, pp. 770- (780-777--0 2016.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519-3529. PML.R 2019.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity mallssi-cooneectng the branches of systems neuroscience. Frontiers in systems neuroscience, pp- 4, 2008.
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in newral information processing systems, 25, 2012.
Malte Kuss and Thore Graepel. The geometry of kernel canonical correlation analysis. 2003.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hoperoft. Convergent learning: Do different neural networks learn the same representations? arXie preprint urX1::1511.07543, 2015.
Richard Meyes, Constantin Waubert de Puiseau, Andres Posada-Moreno, and Tobias Meisen. Under the hood of neural networks: Characterizing learned representations by functional neuron populations and network ablations. arXiu preprint Xix::200400125, 2020.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018a.
Ari S Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. arXiv preprint arXiv:1803.0699 2018b.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference on Learning Representations, 2020.
Judea Pearl. Causality. Cambridge university press, 2009.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexcy Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Newral Information Processing Systems, 34:12116 12128, 2021.
JO Ramsay, Jos ten Berge, and GPH Styan. Matrix correlation. Psychometrika, 49(3):403 423,, 1984.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chicr Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on compuler vision and pattern recognition, pp. 4510 4520, 2018.
Frank Tong and Michael S Pratte. Decoding patterns of human brain activity. Annual review of psychology. 63:483-509, 2012.
Elena Tuzhilina, Leonardo Tozzi, and Trevor Hastie. Canonical correlation analysis in high dimensions with structured regularization. Statistical Modelling, pp. 1471082X211041088, 2021.
Hrishikesh D Vinod. Canonical ridge and econometrics of joint production. Journal of econometrics, 4(2): 147-166, 1976.
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. Towards under- standing learning representations: To what extent do different neural networks learn the same represen- tation. Advances in neural information processing systems, 31, 2018.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the importance of individual units in cnns via ablation. Xiee preprint orXiv: 106.1880.00881 2018.
14


Reviews:
[{"review_id": "KfONqyHXqb", "reviewer": "Reviewer_vUVK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. Well written and easy to follow paper\n\n2. Systematic comparison of representation similarity metrics\n\n3. To the best of my knowledge, these results are novel.\n\n## Limitations\n\n1. The paper is primarily focused on vision models (AlexNet, MobileNet, or ResNet50). Their results may not generalize to other models.\n\n2 The scale of problem (in terms of the number of classes) is quite small.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper systematically tests representation similarity metrics to evaluate their sensitivity to causal functional changes induced by ablation. The authors use changes in network performance after ablation as a way to causally measure the influence of representation on function. They compare this to changes in linear decodability, which measures the available information in the representation.\n\nThe results show that all tested metrics are more sensitive to decodable features than network performance. Among the metrics, Procrustes and Centered Kernel Alignment (CKA) outperform regularized Canonical Correlation Analysis (CCA)-based methods on average. However, these metrics have a diminished advantage when considering network performance. \n\nThe paper concludes that interpretability methods will be more effective if they are based on representational similarity metrics that have been evaluated using causal tests. They also suggest that further tests are needed to evaluate representation similarity metrics and to understand what important pieces of information similar representations share.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Include results with non-vision models and with larger problems (more classes)", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kyg8U33TvR", "reviewer": "Reviewer_x8XN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written, although it was not clear to me how the changes the\nauthors are inducing in the networks are causal and to what extent the tests\nthat the authors use are causal. This deserves more discussion as it is the\ncentral claim of the paper.\n\nFor the presentation of results, it would be great to see the actual similarity\nvalues in addition to the correlation coefficients. In particular, it is unclear\nwhether the similarity values are what you would expect at the moment, i.e. more\nsimilar networks have higher similarity scores. There could be a good\ncorrelation even if the actual values do not make sense, so it is important to\nshow them.\n\nThe statistical tests the authors run are unclear. The authors say that they\ncompute the Spearman correlation (rank?) coefficient, but also report p-values\n-- how were those computed? Did you correct for multiple testing?\n\nSimilarly, it was unclear to me how the performance deficits were produced.\nPlease elaborate.\n\nWhat is the impact of regularization on the CCA-based metrics? The authors note\nthat they perform worse than CKA and Procrustes; could this be a consequence of\nthe normalization?\n\nFor Figure 4, please use the same scale for each sub-plot -- it looks like the\ndifferences for MobileNetV2 are largest, but this is just an artifact of the\ndifferent scale.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an empirical comparison of representation similarity metrics\nfor deep neural networks. The authors draw conclusions with respect to the\nsensitivity of the metrics, and how well they capture the underlying semantics.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please explain in detail how what you are considering is causal.\n- Please present the similarity values.\n- Please clarify the details of the statistical testing.\n- Please describe in detail how performance deficits were computed.\n- Please evaluate the impact of normalization on the metrics.\n- Please make the scales in Figure 4 consistent.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g6YUhqyLyJ", "reviewer": "Reviewer_DRWf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe work sets out to measure quantities of broad interest to the community: What are different RSMs capturing and not capturing? The work is set out as an empirical study, with an experimental protocol of how quantities of interest will be measured and aggregated up.\n\nThe work contains a nice discussion and summary of some widely-used RSMs, some related numerical approximations, and results on what aspects of the data those RSMs are capturing.\n\n# Weaknesses\n\nThe work focuses on evaluating sensitivity to causal function changes. However, the work only investigates the causal form of the network performance functionality, whilst opting for a non-causal test for linear decoding functionality. One of the core conclusions of the paper is that the RSMs are not capturing the causal non-linear network functions. Yet, because in their analysis the linear decoding benchmark is both non-causal and linear, it is not possible to say whether the ineffectiveness of the RSM is due to the network non-linearly, or due to a swap from causal to non-causal tests. This test chosen testing protocol obfuscates any general conclusions that can be drawn. \n\nIndependent of evaluation, the general importance of causal versus non-causal tests of representations is not discussed independently to the importance of capturing network performance.\n\nAdditionally, the tasks are different across these choices (full classification, and one-versus rest classification respectively), potentially introducing further confounding into any overall statements regarding precisely which aspects RSMs are failing to capture.\n\nI am happy to see the acknowledgement of the limitations of the CCA approximation used. Using the approximation itself is not a weakness, however, the work does not contain any discussion regarding the extent to which the approximation is valid, and in the scenarios being tested, the size of approximation errors that should be expected (and crucially, whether these approximation errors could dominate the statistical analysis, either in terms of systematic bias or noise).\n\nReasons are not given for the specific experiments performed (linear probe vs non-linear probe), architectures chosen (why AlexNet, MobileNetV2, ResNet50), hyperparameters chosen (e.g. 50 classes chosen for AlexNet compared to 10- classes for MobileNetV2 and ResNet50). Are the results sensitive to these hyperparameters? Does the number of samples chosen, features chosen (100, randomly sampled - each time or fixed?), or repetitions (200) meet some numerical convergence criteria? The paper does not contain the details that convince a reader these choices are natural or that the key conclusions do not depend on their choice.\n\nThe conclusion contains “representation similarity metrics may already serve well as tools for comparing the geometries of representational spaces, but could be enhanced in order to capture causal, non-linear network function.”.\nThe model comparison Model Stitching (https://arxiv.org/abs/2106.07682) was introduced for this reason. A discussion of how Model Stitching would fare under the process outlined in Figure 1, or better, an evaluation of this representation comparison methodology feels missing.\n\nAnalysis is performed on a single dataset (ImageNet) with three similar (conv-based) architectures, yet general conclusions are being drawn.\n\nIt is unclear what a reader should do given the conclusions (should we stop using RSMs/should we develop others/does an RSM exist than can meet the requirements of the test?). To what extent is a non-correlation a problem (what numbers do we need for the RSM to be useful in practice?).\n\nThe paragraph under “3 Results” is difficult for me to follow and contains key steps required for reproducing the remainder of the section, which obfuscate details required to meet reproducibility standards.\n\nKey experimental quantities leading to the conclusions drawn are buried within paragraphs, and accompanied with a number of distracting undefined (in text) quantities. The experimental discussion would be easier to follow if statistical quantities were associated with figure captions, and the prose mostly reserved for discussions of the results, highlighting the main points.\n\nThe claim around Equation (3) as their derivation enabling computation of CKA on larger layers than prior works is confusing to me, as the original CKA paper (https://arxiv.org/abs/1905.00414) contains this property as the second equality in Equation (1). Similarly, the way the kernel trick for calculating Equation (6) as presented in the paper presents the contribution as one of the current work, however, enabling CCA in high dimensions is the work of the (cited) https://arxiv.org/abs/2011.01650. This should be clearer.\n\nThe writing could be made more succinct, with the important concepts pulled out, numbered, labeled, and cross-referenced. For example, the definitions of Linear Decoding Functionality Measures, and Network Performance measures are mentioned (but not linked, numbered, or referable to in an unambiguous way).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work investigates what aspects are captured by representation similarity metrics (RSMs) of the form $\\textrm{RSM}(A, B)$ where $A\n\\in\\mathbb R^{n\\times p_1}$ is a $p_1$-dimensional activation matrix of $n$ samples, and $B\n\\in\\mathbb R^{n\\times p_2}$ is a $p_2$-dimensional activation matrix of the same $n$ samples.\n\nSpecifically, the work seeks to understand to what extent RSMs capture the properties of:\n\n* *Network performance* (i.e. what aspect of a given representation contributes to network performance), which is a measure of the “useful information” in the representation, and\n* *Decoding accuracy* (i.e. what information about a given class is present in a representation in a linear dense), which is a measure of the “information present, independent of whether the network uses it” in the representation.\n\nThe work formulates this question in a statistical way:\n\n* To what extent is *network performance* (rank) correlated with an RSM,\n* To what extent is *decoding accuracy* (rank) correlated with an RSM, and\n* Is the (rank) correlation of network performance with RSM higher than that of decoding accuracy with RSM.\n\nIn order to generate different representations to base the statistical analysis on, the authors perform interventions on the representation place, taking $A\\rightarrow M\\circ A=\\widehat A$ where M is an element-wise masking matrix. This procedure can be performed many times. For any given mask $M$ instantiation, one can measure $RSM(A, M\\circ A)$ as well as network performance deficit as $\\textrm{Acc}(A) - \\textrm{Acc}(M\\circ A))$, which provides one sample for a correlation analysis. In summary, the authors are looking for statistical relationships between data generated by a causal mechanism of their design.\n\nThe authors investigate this for a number of different RSMs:\n\n* Mean Canonical Correlation Analysis (CCA)\n* Mean Squared CCA\n* Projection Weighted CCA (PWCCA)\n* Centered Kernel Alignment (CKA)\n* Procrustes\n\nIn order to compute some of these RSMs, the authors introduce reduced complexity versions.\n\nThe authors find that:\n\n* Similarity measures in general correspond more closely to linear decidability (i.e. information presence), than model performance (i.e. used information)\n* CKA and Procrustes better correspond to aggregate model functionality", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Required (critical)\n\n* Perform a casual linear decoding experiment, to remove the possibility of the decoder and task modifications as a confounder to statements being made regarding causal versus non-causal. For a given layer, train a linear decoder on $X$. Now use the masking strategy to produce the $A$, and evaluate the linear decoder on these $A$ in the same way as you do in the non-causal setup through $\\textrm{Dec}(X)-\\textrm{Dec}(A)$.\n* Explain the importance of a causal vs non-causal test (potentially use a causal DAG), and clearly distinguish this with a test of model downstream performance versus linear decoding.\n* Expand the paragraph under “3 Results” to describe the method proposed in detail, to the level which the results can be reproduced. Please include:\n    * What a hierarchical linear model is (with references)\n    * How it is being used in your method (preferably presented alongside any relevant equations/cross-references)\n    * Explain what exactly the Fisher Z transformation is being applied to.\n    * Provide equations for the averaging process.\n    * Explain what it means to “model classes as random units”.\n* Explain all architectures and hyperparameters choices (see weaknesses section), and dependence (or insensitivity) of the drawn conclusions to these choices.\n* Include a discussion on Model Stitching (see weaknesses section).\n* Clarify in 2.2 which are contributions of this paper, and which are cited pre-existing contributions.\n* Define all symbols introduced, including those of Section 3 which are statistics standards like $F$ and $t$. Explain what these symbols mean and how they relate to the analysis.\n\n# Recommended for strengthening but not critical \n\n* In the same spirit as the casual linear decoding experiment, perform a non-causal network performance experiment, and report $\\textrm{Perf}(X) - \\textrm{Perf}(A)$. In addition to the causal linear above, this would complete the question regarding whether the issue is the RSMs are having difficulty capturing causal measures or the semantics of the information. The non-causal network performance experiment may be prohibitively compute intensive, however, as one would need to (for example) fine-tune $f_{\\textrm{Perf}}$ given a sampled masking $M$ for each $M$, and so this may be out of scope. Fine-tuning hyperparameter choices may also introduce additional confounders in this setting.\n* Evaluate the Model Stitching method using the testing protocol.\n* Page 1, 3rd line of 2nd paragraph. “... invariant under linear transformations while others ...” → “... invariant under invertible linear transformations while others ...”\n* Page 3, Increase the resolution of Figure 1, or use vector graphics.\n* Page 4, point 1. re-add the space $\\mathbf X$ lives in.\n* Page 4, point 2. Define $A$. Is $A$ a set? If so, use set notation, e.g. $\\mathbb A$.\n* Page 4, point 3.1, for $D$ to be defined as given, it needs an $i$ subscript to correspond to a given $A_i$.\n* Page 4, point 3.2, $F_{Perf}$ →  $F_{Perf}(\\mathbf X, \\mathbf A_i)$ (it is not a global quantity) and add subscript i inside mod.\n* Page 4, point 3.3, $F_{Dec}$ →  $F_{Dec}(\\mathbf X, \\mathbf A_i)$ (it is not a global quantity) and add subscript i inside mod.\n* Page 4, point 3.4, be clear this is Spearman rank which you use later (and do not have a linear relationship prior).\n* Page 5, Equation (6) $\\rho_i$ → $\\rho_i(\\mathbf A, \\mathbf B)$, this equation can go on a single line.\n* Page 7, Increase the resolution of Figure 2, or use vector graphics. Increase the size of the font for the y-ticks. You can use a shared y label here to give you additional horizontal space. Consider tying the y-axes in order to make the different models comparable.\n* Page 8, Increase the resolution of Figure 3, or use vector graphics. Increase the size of the font for the y-ticks. You can use a shared y label here to give you additional horizontal space. Consider tying the y-axes in order to make the different models comparable. Alternatively, the ResNet y-axis can be rescaled as the data currently occupies only a fraction of the plot.\n* Page 9, Increase the resolution of Figure 4, or use vector graphics. I found it challenging to understand the information being conveyed in this plot. I can see the statistical difference between CKA + procrustes and the other methods for network performance in the case of non-AlexNet architectures, however, this many cross-references against legend and color matching. It would be helpful for the reader if this plot could be more simply presented without information loss.\n* Throughout, be careful about tildes and indices, for example top of page 6. $\\tilde{\\alpha_i}$ → $\\tilde{\\alpha}_i$.\n* Move statistical results to figure captions, and reserve main prose for discussion of the results.\n* Check for different architecture types within problem. E.g. perform the same analysis with a non-conv architecture, for example a PyTorch ViTB-16/224 ImageNet pre-trained model can be found in the time library https://timm.fast.ai/create_model.\n* Check a different problem setting. E.g. perform the same analysis in a non-image setting, for example, using a pre-trained BERT model from HuggingFace on a small-medium sized text corpus.\n* Give clear, numbered, reference-able (and then reference) definitions for:\n    * Causal test\n    * Non-causal test\n    * Functionality\n    * Network Performance\n    * Linear Recoding\n    * RSM\n* Provide discussion around why AlexNet behaves qualitatively differently to MobileNetV2 and ResNet50.\n* Give guidance on how researchers going forward should use RSMs based on your conclusions. Discuss what quantitative conditions are necessary for an RSM to meet in order for an RSM to have a given utility.", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper were reviewed by three reviewers. After rebuttal and revision, two reviewers were positive (Accept and Leaning accept), while one reviewer was negative (Leaning reject). 

Reviewer DRWf wrote a very detailed review. This reviewer was initially concerned with the wordings about "causal" and "non-causal", but was satisfied with the authors' revision with improved and clarified terminology. Other suggestions by the reviewer in terms of the presentation were also incorporated in the revision. In the end, Reviewer DRWf was positive about the revision. 

Reviewer vUVK was concerned with the restriction to the vision models and the limited scale of the experiments. The authors addressed the concerns in the rebuttal. 

Reviewer x8XN was negative after rebuttal, primary because the revision did not include results for the logit deficit calculations. The authors reported rank deficit calculations, which can be considered discretized versions of logit deficit calculations. This reviewer was also concerned with the wording "causal" in the title, but it has since been removed from the title. 

Overall, I feel that the paper could be made more substantive by enlarging the scope of the experiments. However, I concur with Reviewer DRWf and Reviewer vUVK that the current version is acceptable.
Decision: Accept (poster)