Equivalence between representational similarity analysis, centered kernel alignment, and canonical correlations analysis

Authors: alex h williams
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-20 18:17:18

Abstract: Centered kernel alignment (CKA) and representational similarity analysis (RSA) of dissimilarity matrices are two popular methods for quantifying similarity in neural representational geometry. Although they follow a conceptually similar approach, typical implementations of CKA and RSA tend to result in numerically different outcomes. Here, I show that these two approaches are largely equivalent once one incorporates a mean-centering step into RSA. This connection is quite simple to derive, but appears to have been thus far overlooked by the community studying neural representational geometry. By unifying these measures, this paper hopes to simplify a complex and fragmented literature on this subject.

Main paper: 
 introduction: 
 Representational similarity analysis (RSA) is a decades-old framework for comparing neural response patterns across systems. It was developed in the cognitive science and computational neuroscience communities, within which it remains a very popular technique [25, 17, 24, 13] . While the RSA framework is quite general, most practical applications involve the construction of representational dissimilarity matrices (RDMs). For an experiment where neural responses are measured across M conditions, an RDM is a symmetric M × M matrix that captures response dissimilarities across all unique condition pairs. Similarity between networks is quantified by similarity in their RDMs (e.g. by cosine similarity or Pearson correlation). I refer to this class of methods as RDM-RSA.RDM-RSA bears a close resemblance to centered kernel alignment (CKA), a more recently developed framework for comparing neural representations [6, 23] . CKA is massively popular within the deep learning community, garnering over 1250 citations as of the time of this writing. In place of RDMs, CKA constructs M × M kernel matrices that capture neural response similarities through a positive definite kernel function. CKA quantifies similarity between networks as similarity in their kernel matrices, in exact analogy to how RSA quantifies similarity between RDMs.To what extent are these methods quantifying the same thing? Here, I document several connections:• First, a popular variant of RDM-RSA is to use squared Euclidean distance to construct RDMs and then use cosine similarity to compare RDMs [45] . I show that if one applies a centering operation before comparing the RDMs, the result is identical to linear CKA, which is the most popular variant of CKA. • It is also popular to construct RDMs with Mahalanobis distance [45] . Here, I show that incorporating the centering operation on RDMs leads to a connection with canonical correlations analysis (CCA). Specifically, under a particular choice of covariance matrix, the centered Mahalanobis RSA score equals the mean squared canonical correlation-a quantity sometimes called Yanai's generalized coefficient of determination [31, 47] . • Finally, I comment on nonlinear extensions of CKA and RSA. In CKA, this is achieved by using nonlinear kernel functions [23] , while a nonlinear extension of RDM-RSA was recently introduced by Lin and Kriegeskorte [26] . I point out a simple way to construct an RDM using a nonlinear kernel function. Here again, RSA on the centered RDM yields the same result as CKA. This new approach conceptually mirrors that of Lin and Kriegeskorte [26] . Thus, nonlinear variants of CKA and RSA are also quite similar. Moreover, I note that performing RSA on centered Euclidean RDMs (instead of squared Euclidean RDMs) is equivalent to a form of nonlinear CKA.These relationships are straightforward to derive, but I have not seen them laid out explicitly in the context of comparing neural representations. Kornblith et al. [23] documented a relationship between linear CKA and CCA, which will be leveraged as part of point 2 above. Additionally, Diedrichsen et al. [8] describe a relationship between linear CKA and RSA with whitened RDMs in the presence of independent and identically distributed measurement noise. The relationships between RSA and CKA I describe here are more basic-in fact, my exposition will treat neural responses as noise-free. Finally, Sejdinovic et al. [37] characterize the equivalence of kernel-based and distance-based hypothesis tests for variable independence. Similar results also appear in the context of multidimensional scaling algorithms [5] and broader mathematical literature [34] . However, it is easy for practitioners to overlook this prior literature because (a) existing papers are focused on distinct motivating applications, and because (b) many presentations assume an audience with strong mathematical background. I therefore believe there is utility in digesting and interpreting these results in plain terms to the neuroscience and interpretable AI communities. 
summary of rdm-rsa: 
 RSA is motivated by decades-old concepts from psychology and philosophy. In particular, Shepard and Chipman [38] posited that similar external objects (e.g. a square and rectangle) are mapped onto mental representations that are also, in some sense, close together-more so than mental representations of dissimilar external objects (e.g. a square and a cat). Or, as Edelman [12] succinctly put it, "Representation is representation of similarities."RDM-RSA is a quantitative framework that enables practitioners to concretely apply these concepts to neural data analysis [24] . Formally, given M stimulus conditions and N -dimensional neural response vectors x 1 , . . . , x M ∈ R N X , the first step of RDM-RSA is to compute an M × M representational dissimilarity matrix (RDM). For example, if the squared Euclidean distance is used to compare neural responses, the elements of the RDM will be given by:D X ij = ∥x i -x j ∥ 2 2 .Then, given responses from a second system, a second RDM, D Y ij = ∥y i -y j ∥ 2 2 , is computed from neural responses y 1 , . . . , y M ∈ R N Y across the same M stimulus conditions. Finally, similarity between the RDMs is quantified by, for example, computing the Spearman or Pearson correlation between the upper triangular elements of D X and D Y . The overall workflow can be summarized as follows:Procedure to compute RDM-RSA similarity scores:• Given neural responses, x 1 , . . . , x M ∈ R N X and y 1 , . . . , y M ∈ R N Y . • Specify distance functions, d X and d Y . • Specify an RDM comparison function, s : R M ×M × R M ×M → R + . • Compute RDMs, D X ij = d X (x i , x j ) and D Y ij = d Y (y i , y j ). • Finally, compute the RDM similarity S(D X , D Y ).Throughout this paper we will use cosine similarity to compare RDMs. Thus we define:EQUATIONas the comparison function between any two symmetric matrices A ∈ R M ×M and B ∈ R M ×M .In machine learning, there has been a swell of recent interest on the topic of comparing neural representations across networks. Popular approaches include centered kernel alignment (CKA; [6, 23] ), canonical correlations analysis (CCA; [30] ), and Procrustes shape distance [9, 46] .Of these approaches, CKA is most obviously similar to RDM-RSA. In fact, one can consider CKA to be a special case of RSA that involves computing and comparing representational similarity matrices (RSMs) instead of RDMs. However, both historically and in current practice, RSMs are used considerably less frequently than RDMs by cognitive neuroscientists and psychologists. Thus, I have focused the narrative of this paper on RDM-RSA to construct a foil of CKA.Before describing the procedure for computing CKA, two definitions must be introduced. First, a positive definite kernel function is, informally, a similarity function that, when applied pairwise to a set of M neural response patterns, is guaranteed to produce a symmetric M × M matrix with nonnegative eigenvalues (i.e. a positive semidefinite matrix). Positive definite kernels are fundamental to modern machine learning theory, and a more deep and formal treatment is provided, for example, by [39] . We will use k X and k Y to denote positive definite kernels, and we will see that these functions play an analogous role to the distance/dissimilarity functions d X and d Y in RDM-RSA.Next, the M × M centering matrix as is a matrix given by C = I -1 M 11 ⊤ where 1 ∈ R M is a vector full of ones. The reader can verify that multiplying any M × N matrix on the left by C results produces another M × N matrix whose columns sum to zero. Furthermore, for a symmetric matrix A ∈ R M ×M the centered matrix CAC has rows and columns that sum to zero. Moreover,ij [CAC] ij = 0.With these definitions in hand, we are ready to state the procedure for computing CKA scores.Procedure to compute CKA similarity scores:• Given neural responses, x 1 , . . . , x M ∈ R N X and y 1 , . . . , y M ∈ R N Y . • Specify positive definite kernel functions, k X and k Y .• Compute kernel matrices, K X ij = k X (x i , x j ) and K Y ij = k Y (y i , y j ). • Compute centered cosine similarity, S(CK X C, CK Y C).The kernel matrices K X and K X in CKA are analogous to the RDMs D X and D Y . Furthermore, the kernel matrices are guaranteed to be positive semidefinite (i.e. be symmetric with nonnegative eigenvalues)-this guarantee comes from our definition of positive definite kernel functions, given above. The centered kernel matrices CK X C and CK Y C are also positive semidefinite because: (a) the centering matrix, C, is positive semidefinite, and (b) positive semidefinite matrices are closed under matrix multiplication.It may not be entirely obvious to some readers why the entries of K X and K Y should be interpreted as similarity scores between neural population responses. This is due to a result called Mercer's theorem which states, in essence, that any positive definite kernel can be interpreted as an inner product on some feature space (see [39] for a more rigorous introduction). Inner products are a measure of similarity-they increase as the angle between vectors decreases (i.e. as the vectors become more aligned with each other).To summarize, there are two major differences between RDM-RSA and CKA. First, in place of RDMs, CKA uses kernel matrices K X ij and K Y ij which can be interpreted as M × M representational similarity (instead of dissimilarity) matrices. Second, to quantify similarity between kernel matrices, CKA computes the cosine similarity between centered kernel matrices. This centering step is typically absent in implementations of RSA, which will turn out to be critical.Canonical correlation analysis (CCA) is a classical multivariate analysis method that identifies a sequence of maximally correlated one-dimensional projections from a pair of datasets [18] . When applying CCA to neural representations, x 1 , . . . , x M ∈ R N X and y 1 , . . . , y M ∈ R N Y as introduced above, the outcome of CCA will be a sequence ofN canonical correlation coefficients 1 ≥ ρ 1 ≥ ρ 2 ≥ • • • ≥ ρ N ≥ 0 where N = min(N X , N Y ).The canonical correlations are defined as the solutions to sequence of optimization problems. The top coefficient, ρ 1 , is given by: maximizeEQUATIONwherew ∈ R N X , h ∈ R N Y parameterize linear projections, x = 1 M ix i denotes the mean neural response for the first system, and ȳ = 1 M i y i denotes the mean neural response of the second system. Subsequent canonical correlation coefficients are found by solving the same optimization problem subject to an appropriate orthogonality constraint on the projection vectors. See Eaton [11] for more detailed background.Larger canonical correlation coefficients indicate greater alignment between neural representations, and past work in both machine learning [30, 27] and neuroscience [41, 14] has used CCA as a framework for comparing representations across neural systems. The average canonical correlation, 1 N i ρ i , and the average squared canonical correlation,1 N i ρ 2i , can be used to summarize the overall similarity between two multivariate datasets [47, 30, 46] . Superficially, CCA does not ressemble RSA or CKA. But it turns out that they can be related by a simple change of variables. Specifically, let Σ X and Σ Y denote the N X × N X and N Y × N Y covariance matrices across the M stimulus conditions within each neural system:EQUATIONThen we define a linearly transformed set of responses as:x i = Σ -1/2 Xx i andEQUATIONfor i = 1, . . . , M . It is common to refer to this change of variables as a whitening transformation (see e.g. [1] ). Note that the whitening transformation assumes that Σ X and Σ Y are invertible; it is possible to incorporate regularization into this change of variables and achieve similar results.The following lemma states that performing linear CKA on the transformed variables yields1 N i ρ 2 ias a similarity measure. This was previously noted by Kornblith et al. [23] ; the lemma below is only a slight reformulation of the statement in their paper.Lemma 1. When using linear kernel matrices on whitened neural responses, K X ij = x ⊤ i x j and K Y ij = y ⊤ i y j , the CKA similarity score is equal to the average squared canonical correlation coefficient between {x 1 , . . . , x M } and {y 1 , . . . , y M }. That is,EQUATION)We will make use of this relationship in section 3.2 to show that performing RSA with centered squared Mahalanobis RDMs also yields 1 N i ρ 2 i as a measure of network similarity. 
results: 
 It is clear that RDM-RSA and CKA are conceptually similar methods, but do they yield quantitatively similar outcomes? I now document several instances where they coincide exactly. All of these are special instances of the following result, stated formally below as proposition 1.Proposition 1. Let k X and k Y be positive definite kernel functions associated with kernel matrices:EQUATIONFurther, let D X and D Y be RDMs defined in terms of this kernel function:D X ij = K X ii + K X jj -2K X ij and D Y ij = K Y ii + K Y jj -2K Y ij (7)Then, the centered cosine similarity scores between these matrices agree:EQUATIONThis result follows from straightforward algebraic manipulations. A step-by-step derivation is provided in appendix A. As mentioned in section 1, equivalences between distance metrics and kernels are already established in the broader literature [34, 4, 37] , but they appear to be overlooked, or at least underappreciated, within the context of comparing neural representational geometry.The rest of this section discusses three specific cases of interest. In each, proposition 1 is used to show a near equivalence between a popular form of RDM-RSA and CKA or CCA.We first consider RDMs that are constructed using the squared Euclidean distance:D X ij = ∥x i -x j ∥ 2 2 and D Y ij = ∥y i -y j ∥ 2 2 (9) Since ∥x i -x j ∥ 2 2 = x ⊤ i x i + x ⊤ j x j -2x ⊤ ix j , we see that eq. ( 7) implies that the corresponding kernel matrices are given by:EQUATIONThe kernel function associated with these matrices is called the linear kernel, and the CKA score between linear kernel matrices is called linear CKA. Proposition 1 implies that performing RSA on the centered RDMs in eq. ( 9) is an equivalent to performing CKA on the kernel matrices in eq. (10) .We remark that this choice of distance (squared Euclidean) and kernel function (linear) are among the most popular variants of RDM-RSA and CKA, respectively. As of this writing, the squared Euclidean distance is currently the default option for constructing an RDM in the rsatoolbox Python package [33] . Furthermore, recent work has leveraged the mathematical tractability of squared Euclidean RDMs to establish statistical inference frameworks for RSA [36, 8] . Similarly, the predominant form of CKA within the deep learning community uses linear kernels eq. ( 10). Indeed, the paper popularizing CKA advocated explicitly for using linear kernels as a default choice [23] .Given the popularity of these two methods, it is somewhat surprising that their near equivalence has not been previously documented.Next, we consider RDMs that are constructed by the squared Mahalanobis distance. Formally, let P X ∈ R N ×N and P Y ∈ R N ×N be two arbitrary positive definite matrices and define:EQUATIONas two RDMs. To achieve the desired relation in eq. ( 7), we choose the kernel matrices to be:EQUATIONProposition 1 implies that performing RSA on the centered RDMs in eq. ( 11) is an equivalent to performing CKA on the kernel matrices in eq. ( 12) for any choice of P X and P Y .Moreover, notice that the kernel matrices in eq. ( 12) can be interpreted as linear kernel matrices under the change of variablesx i = P -1/2 Xx i and y i = P -1/2 Y y i . This change of variables corresponds to a whitening transformation when we choose, P X = Σ X and P Y = Σ Y using definitions from eq. ( 3). Thus, by lemma 1, the CKA score computed from the kernel matrices in eq. ( 12) is equal to the average squared canonical correlation coefficient. Therefore, by proposition 1, the cosine similarity RSA score computed from the centered RDMs in eq. ( 11) is also equal to this quantity. (c-d ) Linear kernel matrices computed from the representations in panels (a) and (b). The network similarity is 0.595 according to linear CKA. (e-f) Nonlinear kernel matrices using RBF kernel functions with a bandwidth parameters of ℓ = 0.3. The kernel matrices look more similar and indeed the network similarity score is higher, 0.982, according to this nonlinear extension of CKA. (g) Lin and Kriegeskorte [26] proposed a nonlinear extension of RSA in which RDMs are transformed elementwise by a monotonically increasing piecewise linear function.(h) When we translate the nonlinear CKA procedure in panels (e-f) into an equivalent RDM-RSA procedure according to proposition 1, we observe that the nonlinear RBF kernel induces a similar "geo-topological transform" on the Euclidean distances between neural responses. The shape of this transform is modulated by the bandwidth parameter, ℓ, plotted as different colors.The squared Mahalanobis distance is a popular method for constructing RDMs. It is supported by the rsatoolbox package [33] and discussed in multiple recent papers [45, 8, 36] . We must mention, however, a key difference between these works and our above analysis with respect to CCA. Typically, Mahalanobis RDMs are motivated by choosing P X to be the covariance of "noise" in the neural response. To follow this motivation, P X and P Y are often set to the covariance matrices computed from residuals of a simple model [45] . The choice of P X = Σ X and P Y = Σ Y was made to elucidate a connection to CCA. This does not capture "noise" per se, as it applies equally well to a complete noiseless, deterministic system (e.g. a feedforward deep network). The choice of P X = Σ X and P Y = Σ Y is nonetheless similar in the sense that one could replace the average neural responses, x and ȳ in eq. ( 3), with a condition-specific model prediction.One of the nice features of CKA is that it nicely extends to nonlinear kernel functions. For example, the standard radial basis function (RBF) kernel (also known as the squared exponential kernel) is a positive definite kernel with a tuneable bandwidth parameter, ℓ. Using this function leads to the nonlinear kernel matrices:EQUATION)where we have allowed for the possibility of separating tuning different bandwidth parameters, ℓ X and ℓ Y , for each network. Kornblith et al. [23] briefly commented on using this approach to compute nonlinear CKA similarity scores, but only a few works have seriously followed up on this possibility [2] .A practitioner may be interested in using nonlinear CKA to achieve a similarity measure between neural representations that have dissimilar shapes but have similar topological features. More precisely, nonlinear CKA using RBF kernel matrices will characterize neural representations as similar when short-range distances between neural responses are preserved, but long-range distances are potentially quite different. This results in a similarity measure that is mostly insensitive to continuous deformations that do not change the topology such as bending. A toy example where nonlinear CKA succeeds at capturing topological similarities, but linear CKA fails to do so, is shown in Figure 1a -f. The idea of developing metrics that capture topological (and not geometric) similarity between neural representations has garnered recent interest within the community, but requires more research to be fully fleshed out and understood [26, 29, 3, 19] .Interestingly, Lin and Kriegeskorte [26] recently proposed a nonlinear extension to RDM-RSA that involves applying an monotonic and saturating transform to the elements of an RDM (see Figure 1g ). They show that this results in similarity measures that are sensitive to topological features of the neural representation. It is easy to see that this procedure is closely related to nonlinear CKA. In particular, the form of the squared exponential kernel means that K X ii = 1 and K Y ii = 1 for all i = 1, . . . , M . Thus, by eq. ( 7), the nonlinear RDMs associated with the RBF kernel take the form:EQUATION)Inspecting these expressions carefully, we realize that D X ij and D Y ij are saturating, monotonically increasing functions of the squared Euclidean distances, ∥x i -x j ∥ 2 2 and ∥y i -y j ∥ 2 2 respectively. The bandwidth parameter ℓ 2 controls the steepness of these nonlinear functions, as shown in Figure 1h .In summary, a nonlinear kernel can be used to construct a RDM using eq. ( 7). This construction of an RDM via a nonlinear kernel can be viewed as applying an elementwise nonlinearity to a squared Euclidean RDM (as in eq. 14 for the case of a RBF kernel). By proposition 1, the centered cosine similarity between nonlinearly transformed RDMs will be equivalent to performing nonlinear CKA. Qualitatively, this approach resembles the topological RSA method introduced by Lin and Kriegeskorte [26] .RDMs is also a form of nonlinear CKA Equation ( 7) shows how we can use a positive definite kernel function to create a notion of distance for which RDM-RSA (with centering and cosine similarity comparison) is equivalent to CKA. It is also possible to go on in the other direction-i.e. we can use a distance function 1 to define a positive definite kernel. For instance, consider the possibility of constructing RDMs using a fractional Euclidean distance:EQUATION)where 0 < q ≤ 2 is a user-defined hyperparameter. Of course, when q = 2 we recover the squared Euclidean distance, for which RDM-RSA is equivalent to linear CKA. However, the choice of q = 1 (i.e. the classic Euclidean distance) is also popular within the RSA literature and it may not be immediately clear how to map this onto a form of CKA.It turns out that computing the centered RDM-RSA score with the distance matrices in eq. ( 15) is equivalent to performing CKA on the following kernel matrices (see, e.g., Example 15 in [37] ):K X ij = 1 2 (∥x i ∥ q 2 + ∥x i ∥ q 2 -∥x i -x j ∥ q 2 ) and K Y ij = 1 2 ∥y i ∥ q 2 + ∥y i ∥ q 2 -∥y i -y j ∥ q 2 (16)Indeed, it is easy to verify that eqs. ( 15) and ( 16) satisfy the relationship in eq. ( 7), which is the main requirement of proposition 1. It is less obvious that the expressions in eq. ( 16) define positive definite kernels, but it can be shown that they correspond to the covariance function of fractional Brownian motion [28] , which is positive definite. This reduces to classical Brownian motion (or a Wiener process) when q = 1, which is a popular choice within RSA literature.Interestingly, tuning the parameter 0 < q ≤ 2 can result in a family of nonlinear similarity scores similar to the nonlinear RBF kernel with different bandwidth parameters, ℓ, that was discussed above in section 3.3. To show this, Figure 2 revisits the toy example shown in Figure 1a -b, using fractional Euclidean distance RDMs in place of RBF kernel matrices. Figure 2a visualizes the raw RDMs, D X and D Y , for various values of q. Qualitatively, we see that these RDMs appear more similar as q decreases. A similar trend is seen in the centered RDMs, shown in Figure 2b . 15), for various choices of 0 < q ≤ 2. (b) Centered RDM matrices from panel a. By proposition 1, these centered RDMs are equal to negative two times the centered kernel matrices given in eq. ( 16). (c) The hyperparameter q can be interpreted as applying an elementwise, monotonically increasing function to the squared Euclidean RDM (similar to the distance induced by the RBF kernel in Figure 1h ). (d) The cosine similarity between CD X C and CD Y C, which is equivalent to the CKA score on the kernel matrices in eq. ( 16), is plotted as a function of q. When q = 2, this converges to the linear CKA score (shown as black dashed line). Smaller values of q result in higher similarity scores, emphasizing topological similarity between the response patterns in Figure 1a-b .Intuitively, the fractional exponent q applies an elementwise nonlinearity to the squared Euclidean RDM matrices (Figure 2c ), which is similar to the RBF-RDM transform previously highlighted in Figure 2h . In the limit that q → 0, the nonlinearity is a step function, equal to one everywhere except zero. Because of this step function behavior, the cosine similarity between centered RDMs will equal one in the limit that q → 0 because every RDM will have zeros along the diagonal and ones on the off diagonals. On the other extreme, when q = 2, we recover squared Euclidean RDMs, and the resulting RDM-RSA score after centering will be equal to linear CKA. Intermediate values of q will smoothly interpolate between these outcomes, as shown in Figure 2d .In summary, this section has shown a new interpretation of RSA with Euclidean RDMs (q = 1) as a form of nonlinear CKA with a kernel function defined by Brownian motion or Wiener process. More generally, one can choose any value of 0 < q < 2, which can be interepreted as a form of nonlinear CKA with a kernel related to fractional Brownian motion. When q = 2, we recover linear CKA or squared Euclidean RDM-RSA with centering. 
importance and interpretation of centering: 
 We have seen that the key difference between CKA and commonly used RDM-RSA methods is the presence of the centering operation, A → CAC for a symmetric marix A. Beyond deriving an equivalence between CKA and RDM-RSA, are there desirable reasons for this centering operation?There is a simple justification for centering in the case of linear CKA. Specifically, consider translating the neural responses, x i = x i + α and y i = y i + β for some arbitrary vectors α ∈ R N X and β ∈ R N Y . Translating the responses in this manner is akin to choosing a different origin for the coordinate system defining neural responses. It is easy to show that the linear CKA score computed on x 1 , . . . x M and y 1 , . . . y M is invariant to the value of α and β, while the uncentered cosine similarity is sensitive and can be made arbitrarily close to one. Thus, the centering operation on kernel matrices is necessary if one desires a translation-invariant measure of representational similarity. Readers seeking further intuition should take a closer look at Cortes et al. [6] , who originally introduced the CKA score, and who argue that the centering step is "critical" at length in their paper. The intuition behind centering RDMs is different. Unlike linear kernel matrices, the elements of RDMs are already invariant to the translations. That is, ∥ x i -x j ∥ 2 2 = ∥x i -x j ∥ 2 2 for any transformation of the form x i = x i + α. On the other hand, because distances are nonnegative, the cosine similarity between uncentered RDMs can be inflated above zero. Incorporating the centering operation results in matrices with positive and negative entries, which intuitively can be "more orthogonal" resulting in cosine similarity scores closer to zero. Figure 3 illustrates this intuition in a simple toy example. Briefly, two neural systems are defined in N = 2 dimensions across M = 3 stimulus conditions (fig. 3a ). The centered RSA and linear CKA scores are essentially zero, indicating that the two neural systems are maximally dissimilar; however, the uncentered RDM-RSA score is roughly 1/3 (fig. 3b ). In fact, these two triangular point configurations (visualized in fig. 3c ) are maximally different shapes, as shown for example in [22] . Thus, in this setting where M = 3 conditions and N = 2 neural dimensions, uncentered RDM-RSA can only output a similarity score on the interval [1/3, 1]. Once centering is incorporated, the resulting score, equivalent to CKA, is normalized to lie on [0, 1].Applying the centering operation to distance matrices appears to have deeper importance in other contexts. For example, the distance covariance statistic [43, 42] , which can be used to test for independence among random variables, is computed from centered distance matrices. One can show that removing the centering step is undesirable-the resulting statistic can fail to detect dependencies among random variables, effectively resulting in false negatives when hypothesis testing for independence. A concrete example of this failure is described in [32] .Centering is also included in distance covariance analysis [7] , a method that leverages the distance covariance framework for dimensionality reduction. Kernel principal components analysis (kernel PCA) is a similar method to achieve nonlinear dimensionality reduction, and this too typically includes a centering step (see [35] , appendix B). Intuitively, PCA fits a hyperplane passing through the origin to approximate high-dimensional data. Centering around the origin is therefore a sensible and important preprocessing step for this analysis.The centering operation is also applied to squared Euclidean distance matrices in the context of multidimensional scaling [15, 5] . Here, the problem is to reconstruct a set of points, x 1 , . . . , x M , that are consistent with a given pairwise distance matrix, D X . There are of course many solutions to this problem-any valid configuration of points can be freely rotated, reflected, and translated. If the configuration generating the squared distance matrix spans the full space, then one can perform an eigendecomposition of CD X C to obtain a solution. Specifically, one obtains CD X C = GG ⊤ where G is a matrix with orthogonal columns (scaled eigenvectors). The rows of G are taken as M points in an N X -dimensional space, and one can show that they indeed recover the appropriate pairwise Euclidean distance scores. Of course, GG ⊤ is a linear kernel matrix, which is the core insight highlighted in this paper. Gower [15] remarks that the origin of the configuration will be the centroid of the points (i.e. a centered kernel matrix). 
conclusion: 
 The contribution of this paper is to document some precise equivalences between two popular frameworks for quantifying representational similarity between neural systems: CKA and RDM-RSA. This was done by exploiting one-to-one relationships between positive kernel functions and distance functions that are already well-established in mathematical literature, tracing back to work by Schoenberg [34] (for more background, see [4] ). Closely related work has highlighted similar equivalences within in the context of statistical tests for independence [37] . Nonetheless, to my best knowledge, the relationship between CKA and RDM-RSA has not been explicitly spelled out in prior work and is not widely understood by researchers in this area. Indeed, while conceptual similarities between CKA and RDM-RSA are often acknowledged, they are mostly treated as being distinct methods (e.g. in [21] ). Moreover, CKA and RDM-RSA are preferred by different research communities in machine learning and cognitive neuroscience for historical reasons. By illustrating deeper mathematical connections, I hope to encourage more exchanges and cross-citations between these communities. For example, Cortes et al. [6] provide a detailed theoretical analysis of CKA including results on bounds on how many sampled stimuli, M , are needed to achieve good estimates. Our analysis shows that these results can be immediately applied to RSA with centered squared Euclidean RDMs. Likewise, our results may also enable statistical frameworks developed for RDM-RSA (e.g. [8, 36] ) to be adapted and applied to CKA-based analysis.More broadly, the literature on comparing neural representations is complex and federated. A recent review by Klabunde et al. [20] catalogues over thirty methods for quantifying similarity. It is difficult for practitioners to choose among this large menu of options, many of which give different numerical outputs [40] . Cases where methods are truly identical ought to be widely appreciated and highlighted. Harvey et al. [16] previously showed that the Procrustes shape distances (advocated by [9, 46] ) are equivalent to the normalized Bures similarity score (advocated in [44] ) up to a monotonic transformation. This paper adds to this list by documenting several additional examples where RSA on centered RDMs coincides with linear CKA, CCA, and nonlinear CKA.The analyses detailed in this paper focus on deterministic (i.e. noise-free) and static (i.e. nondynamical) neural representations. This is a limitation, and there is growing interest within the literature to characterize the stochastic [10] and dynamical [29] aspects of neural representations. Future work that connects these emerging methodologies to CKA and RSA-based analyses would be of great interest. 
 
 Appendix: Preprint.More precisely, the distance function must be a semimetric of negative type. See lemma 12 in Sejdinovic et al.[37] for a formal statement. 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Instructive work bridging representation similarity measures from different communities
main_review: main_review: This contribution draws an equivalence between CKA and RSA. It outlines CKA, RSA, and CCA (canonical correlation analysis), which are representation similarity frameworks that take choices of, e.g., similarity or distance functions between neural responses. The authors show in a series of propositions that, under certain realizations (choices of the mentioned functions) of each framework, they produce identical scores on the same neural responses. 

I found this work to be very clear and instructive. Given the frequent use of representation similarity frameworks now in the neuroscience and machine learning communities, this contribution can be a practical guide as to when similarity measures are equivalent. Given its timeliness and broad relevance to the workshop, I would strongly recommend the paper for acceptance.
title: title: Review of Submission 16
main_review: main_review: **Strengths:**
1. The writing is clear and lucid, and the relevant literature has been discussed adequately.
2. Documenting the similarity between different similarity metrics is particularly useful and relevant to this workshop, especially given the deluge of such metrics – it would be helpful to both practitioners and theoreticians to be aware of these equivalences, in order to guide their decision making when choosing similarity metrics for their work.

**Suggestions:**
I checked the mathematical results and wasn't able to spot anything amiss. Perhaps one "weakness" is that the paper treats neural responses as noise-free/non-stochastic, but this is not the case in neuroscience or in many works in theoretical/computational neuroscience. This is acknowledged in the paper but it might be interesting to have a brief discussion on metrics that take stochasticity into account, e.g., work from Duong et al. (2023). Nevertheless, this is not a negative -- several works consider deterministic neural networks or condition-averaged responses, and I reiterate that showing the equivalences between various metrics explicitly is useful.

Duong, L., Zhou, J., Nassar, J., Berman, J., Olieslagers, J., & Williams, A. H. Representational Dissimilarity Metric Spaces for Stochastic Neural Networks. In The Eleventh International Conference on Learning Representations.

I just have some minor suggestions on fixing typos/grammar issues in certain places (this list may not be exhaustive):
1. Line 28: "then use cosine similarity is used to compare" -> remove "is used"
2. Line 35: "Yanai's generalized generalized" -> remove duplicated "generalized"
3. Line 75: "Specify also a RDM" -> might be better to use "[an](https://blog.apastyle.org/apastyle/2012/04/using-a-or-an-with-acronyms-and-abbreviations.html) RDM"
4. Line 94: $\mathbf{C} = \mathbf{I} - \frac{1}{M} \mathbf{11}$ -> should be $\mathbf{11^\top}$, right?
5. Line 175: Should use `\citep` for reference 21, i.e., without explicit mention of "Kornblith et al." (unless you say "Kornblith et al. [21] advocated ..." in line 174).
6. Line 190: "_rsatoolbox_ package" -> "$\texttt{rsatoolbox}$ package" (to be consistent with Line 170)
7. Line 314: "practicioners" -> "practitioners"
8. Inconsistent usage of "I" and "we" throughout the paper: it might be better to just stick with "we" throughout

**Quality:** I think this is quality work with a good fit to this workshop. While it is not proposing something remarkably novel, this is a very useful, explicit documentation of some equivalences the UniReps community should be aware of (those with a strong mathematical background may already be familiar with these relationships, as the paper admits, but it is always beneficial to have explicit documentation of such knowledge). The math and overall presentation are great.

**Clarity:** As I stated earlier, I'm a fan of the writing of this paper and think it reads clearly.

**Originality:** While the work builds upon existing knowledge of equivalences between CCA and CKA from Kornblith et al. and Sejdinovic et al. ([21, 34] in the paper), I am not aware of any other work explicitly documenting the equivalences mention here.

**Significance:** This work is of great fit to UniReps and will be of interest/use to both the cognitive/computational neuroscience community and the machine learning community alike.
title: title: A clear demonstration of cases where computing RDM-RSA is equivalent to CKA
main_review: main_review: Comparing the similarity between representations is a core problem in many fields such as Artificial Intelligence (AI), Neuroscience & Cognitive Science. However, different fields have different preferences for the metrics they commonly use. For example, the field of AI uses Centered Kernel Alignment (CKA), Neuroscience uses Representational Similarity Analysis (RSA), and Cognitive Science uses Canonical Correlation Analysis (CCA). This paper demonstrates and provides mathematical proofs of the cases where RSA, CKA, and CCA are equivalent. A mean-centering step is crucial in RSA for this equivalence, providing benefits such as translational invariance and testing for independence. Demonstrating this is important to enable cross-talk between the fields and can accelerate the development of such measures through interdisciplinary research.

The authors show three cases where RSA is equivalent to computing CKA (and CCA in some cases). Note that cosine similarity is used to compare matrices, the responses are assumed noise-free, and centering is applied to Representational Dissimilarity Matrices (RDMs) when using RSA.

- Result 1:  Linear CKA is equivalent to using squared Euclidean RDMs
- Result 2: Average squared canonical correlation coefficient, linear CKA (with a kernel involving whitening transformation), and squared Mahalanobis RDMs are equivalent
- Result 3: Non-linear CKA with Radial Bias Function (RBF) kernels is equivalent to squared Euclidean RDMs with elementwise non-linearity applied

Overall, the paper is very well written with clear mathematical proofs. With the advancements and increased interest in computing representational similarity in different fields, this paper shows that different fields are converging on equivalent metrics, and can benefit from cross-talk. Additionally, while these equivalences might be intuitive to researchers with a mathematical background, a clear demonstration was required to make it accessible to all researchers. 

Suggestions that can improve the paper:
1. In the paper, the systems are assumed noise-free. I believe discussing if these results hold in noisy systems, what accommodations are required to extend these results to noisy systems, recommendations when comparing representational similarity in noisy systems, and the relevant literature would boost the usefulness and impact of the paper. 
2. Typo: Line 227 "a RDM using eq(7). Many kernel functions". 
3. Typo: Line 308 "are needed to achieve a good estimates"
title: title: The author's claim is simple and intuitive but appears incremental.
main_review: main_review: This paper establishes the equivalence between representational similarity analysis (RSA), centered kernel alignment (CKA), and canonical correlation analysis (CCA),  as the manuscript title states. CKA and RSA are popular techniques that can be applied to quantify the similarity between neural networks. The basis for proving the equivalence is CCA, a classical method for analyzing multivariate data; CCA finds the hidden correlation between two sets of variables. In the manuscript's setting, the two sets of variables correspond to the neural responses from different networks.

The author's central claim is Proposition 1, the equivalence between CCA and RSA. When calculating distance matrices and gram matrices from two neural responses, the centered cosine similarity (CCS) between the distance matrices is identical to that between the gram matrices. The arguments supporting this claim are quite clear because Proposition 1 is a straightforward application of Lemma 1 presented by Kornblith et al., which claims that the CCS between gram matrices is identical to the average of canonical correlations. 

Pros:
- The main claim and supporting arguments are clear.

Cons:
- The main claim is straightforward and intuitive, but it is not so surprising. The manuscript's progress seems incremental compared to the importance of Lemma 1.

- The manuscript seems redundant compared to the simplicity of the main claim, and it becomes more concise and focused if sections 3.4 and 4 are moved to the appendix.

- No practical application was presented. Practitioners will get
more interested if the author sufficiently explains that CKA and RSA are 
useful for quantifying similarity in neural representational geometry.

Minor comments:
- Section 2.3: The description might be misleading. Readers unfamiliar with CCA cannot readily understand that the optimizer of Eq. (2) becomes a ``sequence'' of canonical correlations. The section would read better if the author explained canonical correlations clearly. They should be obtained from the eigendecomposition of Sigma_{XX}^{-1} Sigma_{XY} Sigma_{YY}^{-1} Sigma_{YX}.

- page 3, line 94: The transpose symbol in the centering matrix is missing.

- First-person singular (I) and plural (we) are mixed.