Generalized Shape Metrics on Neural Representations
Authors: Alex H Williams
Erin Kunz
Simon Kornblith
Scott Linderman
Venue: NeurIPS 2021 Poster (2021)
Keywords: Representational Similarity Analysis
Neural Representations
Shape Analysis
Metric Space

Abstract: Understanding the operation of biological and artificial networks remains a difficult and important challenge. To identify general principles, researchers are increasingly interested in surveying large collections of networks that are trained on, or biologically adapted to, similar tasks. A standardized set of analysis tools is now needed to identify how network-level covariates---such as architecture, anatomical brain region, and model organism---impact neural representations (hidden layer activations). Here, we provide a rigorous foundation for these analyses by defining a broad family of metric spaces that quantify representational dissimilarity. Using this framework, we modify existing representational similarity measures based on canonical correlation analysis and centered kernel alignment to satisfy the triangle inequality, formulate a novel metric that respects the inductive biases in convolutional layers, and identify approximate Euclidean embeddings that enable network representations to be incorporated into essentially any off-the-shelf machine learning method. We demonstrate these methods on large-scale datasets from biology (Allen Institute Brain Observatory) and deep learning (NAS-Bench-101). In doing so, we identify relationships between neural representations that are interpretable in terms of anatomical features and model performance.

Generalized Shape Metrics on Neural Representations
Alex H. Williams Statistics Department Stanford University shrilliadstanford.ccmm
Erin Kunz
Electrical Engineering Department
Stanford University
ekunz@stanford edu
Simon Kornblith Google Research, Toronto skorablith@google... com
Scott W. Linderman
Statistics Department
Stanford University
scott htw.:iiiieeaaanttaaaooddddddddd
Abstract
Understanding the operation of biological and artificial networks remains difficult and important challenge. To identify general principles, researchers are increas- ingly interested in surveying large collections of networks that are trained on, or biologically adapted to, similar tasks. A standardized set of analysis tools now needed to identify how network-level covariates such as architecture, anatomical brain region, and model organism -mpact neural representations (hidden layer activations). Here, we provide rigorous foundation for these analyses by defining broad family of metric spaces that quantify representational dissimilarity. Using this framework, we modify existing representational similarity measures based on canonical correlation analysis and centered kernel alignment to satisfy the triangle inequality, formulate novel metric that respects the inductive biases in convolu- tional layers, and identify approximate Euclidean embeddings that enable network representations to be incorporated into essentially any off- off-the-shelf machine learn- ing method. We demonstrate these methods on large-scale datasets from biology (Allen Institute Brain Observatory) and deep learning WNAS-Bench- 101). In doing so, we identify relationships between neural representations that are interpretable in terms of anatomical features and model performance.
Introduction
The extent to which different deep networks or neurobiological systems use equivalent represen- tations support of similar task demands is a topic of persistent interest in machine learning and neuroscience [1-3]. Several methods including linear regression [4, 5], canonical correlation analysis (CCA; [6, 7)), representational similarity analysis (RSA; [8)), and centered kernel align- ment (CKA; [9]) have been used to quantify the similarity of hidden layer activation patterns. These measures are often interpreted on an ordinal scale and are employed compare limited number of networks-ec e.g. they can indicate whether networks A and B are more or less similar than networks and C. While these comparisons have yielded many insights [4-12], the underlying methodologies have not been extended to systematic analyses spanning thousands of networks.
To unify existing approaches and enable more sophisticated analyses, we draw on ideas from statistical shape analysis [13-15] to develop dissimilarity measures that are proper metrics-i.e. measures that are symmetric and respect the triangle inequality. This enables several off-the-shelf methods with theoretical guarantees for classification (e.g. k-nearest neighbors, [16]) and clustering (e.g. hierarchical clustering [17]). Existing similarity measures can violate the triangle inequality, which complicates these downstream analyses [18-20]. However, we show that existing dissimilarity
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

activations
feature space
K'x matrix of pairwise distances
Manifold Interpretation (K points on smooth manifold)
Approximate Euclidean embedding (K points dimensions)
Knewwokk
neurons
features
Riemannian k-means, principal geodesic analysis, etc.
high
hierachical clustering
k-nearest neighbors
PCA
Nonlinear Regression
Density Estimation
class
class
class
Figure 1: Machine learning workflows enabled by generalized shape metrics.
measures often be modified satisfy the triangle inequality and viewed special cases of the framework we outline. We also describe novel metrics within this broader family that are specialized convolutional layers and have appealing properties for analyzing artificial networks.
Moreover, we show empirically that these metric spaces on neural representations can be embedded with low distortion into Euclidean spaces, enabling even broader variety of previously unconsidered supervised and unsupervised analyses. For example, we can use neural representations as the inputs to linear nonlinear regression models. We demonstrate this approach neural representations in mouse visual cortex (Allen Brain Observatory: [21]) in order to predict each brain region's 's anatomical hierarchy from its pattern of visual responses- i.e., predicting feature of brain structure from function. We demonstrate a similar approach to analyze hidden layer representations in a database of 432K deep artificial networks (NAS-Bench-101; (22]) and find surprising degree of correlation between early and deep layer representations.
Overall, provide theoretical grounding which explains why existing representational similarity measures are useful: they are often close to metric spaces, and can be modified to fulfill metric space axioms precisely. Further, draw new conceptual connections between analyses of neural representations and established research areas [15, 23], utilize these insights to propose novel metrics, and demonstrate general-purposs machine learning workflow that scales datasets with thousands of networks.
2 Methods
This section outlines several workflows (Fig. to analyze representations across large collections of networks. After briefly summarizing prior approaches (sec. 2.1), we cover background material on metric spaces and discuss their theoretical advantages over existing dissimilarity measures (sec. 2.2). We then present a class of metrics that capture these advantages (sec. 2.3) and cover special case that is suited convolutional layers (sec. 2.4). We then demonstrate the practical advantages of these methods in Section 3, and demonstrate empirically that Euclidean feature spaces can approximate the metric structure of neural representations, enabling broad set of novel analyses.
2.1 Prior work and problem setup
Neural network representations are often summarized over a set of m reference inputs (e.g. test set images). Let X; mxx, and X, Rmxx, denote the responses of two networks (with n; and n, neurons. respectively) to collection of these inputs. Quantifying the similarity between X, and X; complicated by the fact that, while the m inputs are the same, there no direct correspondence between the neurons. Even if n; = n2. the typical Frobenius inner product, (X,, X,,X) X,) and metric, |,,-Xx,1
Tr[X; X;), s]) (X; X,, X; ,,2,/2 fail to capture the desired notion of dissimilarity. For instance, let II denote some should consider X; ; and X, to be identical in this case since the ordering of neurons arbitrary. Yet. clearly ||X,- X Ill a O, except in very special cases.
permutation matrix and let X; = X II. Intuitively, we

---

One way to address this problem is to linearly regress over the neurons to predict X, from X, Then, one can use the coefficient of determination (R²) as a measure of similarity [4, 5]. However, this similarity score is asymmetric-if one instead treats X; as the dependent variable that predicted from X; i, this will result in different R². Canonical correlation analysis (CCA; [6. 7J) and linear centered kernel alignment (linear CKA; 19, 24)) also search for linear correspondences between neurons, but have the advantage of producing symmetric scores. Representational similarity analysis (RSA; [8]) is yet another approach, which first computes an m x m matrix holding the dissimilarities between all pairs of representations for each network. These representational dissimilarity matrices (RDMs), are very similar to the m x m kernel matrices computed and compared by CKA. RSA traditionally quantifies the similarity between two neural networks by computing Spearman's 's rank correlation between their RDMs. A very recent paper by Shahbazi et al. [25]. which was published while this manuscript was undergoing review, proposes use the Riemannian metric between positive definite matrices instead of Spearman correlation. Similar to our results, this establishes metric space that can be used to compare neural representations. Here, we leverage metric structure over shape spaces [13- instead of positive definite matrices, leading to complementary insights.
In summary, there Without a unifying theoretical framework it is unclear how to choose among them, use their outputs for downstream tasks, or generalize them new domains.
diversity of methods that one can use compare neural representations.
2.2 Feature space mapping, metrics, and equivalence relations
Our first contribution will be establish formal notions of distance (metries) between neural representations. To accommodate the common scenario when the number of neurons varies across networks (i.e. when n, * nj), we first map the representations into a common feature space. For each set of representations, X,, we suppose there is a mapping into a p-dimensional feature space, X, + X where X E Rmxp, In the special case where all networks have equal size, 71 = n2 = = n, we can express the feature mapping as a single function ő : R + R"* so that X / = i)). When networks have dissimilar sizes, we can map the representations into a common dimension using, for example, PCA [6].
Next, we seek to establish metrics within the feature space, which are distance functions that satisfy:
Equivalence:
d d(x), x3) = 0 = x X3
Symmetry: d(x; x²) d(x). x,))
(2)
Triangle Inequality: d(x) x S d(x x + d k x3)
(3)
for all X Xy. and X in the feature space. The symbol '~' denotes an equivalence relation
between two elements. That is, the expression X x means that "X. is equivalent to X Formally, distance functions satisfying Eqs. (1) to (3) define a metric over a quotient space defined by the equivalence relation and a pseudometric over IR mxp (see Supplement A). Intuitively, by specifying different equivalence relations we can account for symmetries in network representations, such as permutations over arbitrarily labeled neurons (other options are discussed below in sec. 2.3).
Metrics quantify dissimilarity in a way that agrees with our intuitive notion of distance. For exam- ple, Eq. (2) ensures that the distance from X to X² is the same as the distance from X² to X Linear regression is an approach that violates this condition: the similarity measured by R2 depends on which network treated as the dependent variable.
Further, Eq. (3) ensures that distances are self-consistent in the sense that if two elements (X and
X9 are both close to a third (X then they are necessarily close to each other. Many machine learning models and algorithms rely on this triangle inequality condition. For example, in clustering, it ensures that if X and X are put into the same cluster as X { then X and X cannot be too far apart, thus implying that they too can be clustered together. Intuitively, this establishes an appealing transitive relation for clustering, which can be violated when the triangle inequality fails to hold. Existing measures based on CCA, RSA, and CKA, are symmetric, but do not satisfy the triangle inequality. By modifying these approaches to satisfy the triangle inequality, we avoid potential pitfalls and can leverage theoretical guarantees on learning in proper metric spaces [16-20].
3

---

A
Xi
X
0,(X, x
B
X
XK
rotational alignment
9
whitening feature map
6o(X, x,)= 0.032
flattened metric (x xx) 8,(X, x = (x xx) =
rotational alignment
convolutional metric
8x = ( x 0,(X, xx)
1.519
Figure 2: (A) Schematic illustration of metrics with rotational invariance (top), and linear invariance (bottom). Red and blue dots represent a pair of network representations X; and X, which correspond to m points in w-dimensional space. (B) Demonstration of convolutional metric on toy data. Flattened metrics (e.g. [6. 9]) that ignore convolutional layer structure treat permuted images (X right) equivalent images with coherent spatial structure (X, and X,, left and middle). convolutional metric, Eq. (11), distinguishes between these cases while still treating X, and X, as equivalent (obeying translation invariance).
2.3 Generalized shape metrics and group invariance
In this section, we outline a new framework to quantify representational dissimilarity, which leverages a well-developed mathematical literature on shape spaces [13-15]. The key idea is to treat X ~ X and only if there exists a linear transformation T within a set of allowable transformations G. such that X X Although 9 only contains linear functions, nonlinear alignments between the raw representations can be achieved when the feature mappings X; + X are chosen to be nonlinear. Much of shape analysis literature focuses on the special case where p n and G is the special orthogonal group (O(n) = {R E RAxR R R = I, det R) = 1}, meaning that X and X equivalent if there is re-dimensional rotation (without reflection) that relates them. Standard shape analysis further considers each X to be mean-centered (X))) 0) and normalized (| 1) version of the raw landmark locations held in X; E Rmxn (an assumption that we will relax). That is, the feature map Ø R=** + Smxxn transforms the raw landmarks onto the hypersphere, denoted gmxx of X n matrices with unit Frobenius norm. In this context, X Smxx is called a "pre-shape." By removing rotations from pre-shape, [X*] = {S E Smxa IS~ [ for pre-shape X we recover its "shape."
To quantify dissimilarity in neural representations, feature mappings and alignments. The minimal distance within the feature space, after optimizing over aligmments, defines metric under suitable conditions (Fig. 2A). This results in a broad variety of generalized shape metrics (see also, ch. 18 of [15]), which fall into two categories as formalized by the pair of propositions below. Proofs are provided in Supplement B.
generalize this notion of shape to include other
Proposition Let X o E IRx and let be a group of linear isometries on mmxp, Then,
d(x x
min
X
XITII
(4)
TEg
defines a metric, where X X if and only if there is T G such that X X
Proposition 2. Let X E smxp, and let & be group of linear isometries on @mxp, Then,
x
min Teg
arccos (x T)
(5)
defines a metric, where X x if and only if there is a T E G such that X X²T.
Two key conditions appear in these propositions. First, 9 must group functions. This means g is set that contains the identity function, is closed under composition (T (T22 G for any T2 G and To G), and whose elements are invertible by other members of the set T E G then T E G).

---

Second, every T E G must be an isometry, meaning that |X x; = |X x - for all T E G and all elements of the feature space. On mx and xm 7, all linear isometries are orthogonal transformations. Further, the set of orthogonal transformations, O(p) {Q RP*P Q Q I defines a well-known group. Thus, the condition that G is a group of isometries equivalent to g being subgroup of (pp) -i.., subset of (p)) satisfying the group axioms.
Intuitively, by requiring G to be group of functions, we ensure that the alignment procedure is symmetric-i.e it is equivalent to transform X to match X or transform the latter to match the former. Further, by requiring each E 9 to be an isometry, we ensure that the underlying metric (Fuclidean distance for Proposition I; angular distance for Proposition 2) preserves its key properties.
Together, these propositions define broad class of metrics as we enumerate below. For simplicity, we assume that n, = nt in the examples below, with the understanding that a PCA or zero-padding preprocessing step has been performed in the case of dissimilar network sizes. This enables to express the metrics as functions of the raw activations, i.c. functions I | X R + R+.
Permutation invariance neurons are one-to-one matched across networks. If we set the feature map to be the identity function, i.e., X = X; for all i, then:
The most stringent notion of representational similarity is to demand that
( Xi xi) = min ITP()))
X xX,1
(6)
defines a metric by Proposition since the set of permutation matrices, P(n), is subgroup of ((n). To evaluate this metric we must optimize over the set of neuron permutations to align the two networks. This can be reformulated (see Supplement C) fundamental problem in combinatorial optimization known as the linear assignment problem [26]. Exploiting an algorithm due to Jonker and Volgenant [27, 28] can solve this problem in ( time. The overall runtime for evaluating Eq. (6) is Omn? + n n²),, since we must evaluate X X, to formulate the assignment problem.
Rotation invariance Let C m (1/m)111 denote an m m centering matrix, and consider the feature mapping di which mean-centers the columns, d1 (X, X = C.,,- Then,
(X X,) min
x x Q
QEO
defines metric by Proposition 1, and is equivalent to the Procrustes size-and-shape distance with reflections [15]. Further, by Proposition 2,
X
0x x))
min QEO
arecos
defines another metric, and is closely related to the Riemannian distance on Kendall's shape space [15]. To evaluate Eqs. (7) and (8). we must optimize over the set of orthogonal matrices find the best aligmment. This also maps onto a fundamental optimization problem known as the orthogonal Procrustes problem [29, 30], which can be solved in closed form O(n)) time. As in the permutation- invariant metric described above, the overall runtime is O(mn² + n²
Linear invariance
Consider a partial whitening transformation, parameterized by 0 S a S 1:
X C + (1 a) 1/2
Note that X TC the empirical covariance matrix of X. Thus, when a = 0, Eq. (9) corresponds to ZCA whitening [31], which intuitively removes invertible linear transformations from the repre- sentations. Thus, when a the metric outlined below treats X, ~ X, there exists an affine transformation that relates them: X; X, W + b for some W E Ruxn and b E I.. When a = 1, Eq. (9) reduces to the mean mean-centering feature map used above.
Using orthogonal alignments within this feature space leads to a metric that is related to CCA. First, let # 2... 2 Pn 2 0 denote the singular values of (X &o T (xxx) )/1 X " ||| One can show that
X dn
0a (X X,) = min arccos
arccos (2 Pr).
(10)
QEO
X
24
5

---

A
B
and we can see from Proposition 2 that this defines a metric for any S a S When a - the values Pl..... Pn are proportional to the canonical correlation coefficients, with 1/n being the factor of proportionality. When a > 0, these values can viewed as ridge regularized canonical correlation coefficients [32]. See Supplement C for further details. Past works [6, 7] have used the average canonical correlation as a measure of representational similarity. When a = 0, the average canonical correlation is given by Ee Pe cos 0 (X; X2). Thus, if we apply arccos(-) to the average canonical correlation, we modify the calculation to produce a proper metric (see Fig. 4A). regularize the est. distance (6,)
loyer
est. distance (8,
distance
network layer
perm.
orth.
conv
0.5
Mattened
layer layer layer
101
10a 10*
10
10
10
10a
network 2
num inputs (m)
/
Figure 3: (A) Each heatmap shows a brute-force search over the shift parameters along the width and height dimensions of pair of convolutional layers compared across two networks. The optimal shifts typically close zero (red lines). (B) Impact of sample size, m, on flattened and convolutional metrics with orthogonal invariance. convolutional metric approaches final value faster than flattened metric, which is still increasing even at the full size of the CIFAR-I0 test set (m = 10*). (C) Impact of sample density, m m/n, on metrics invariant to permutation, orthogonal, regularized linear (a 0.5). and linear transformations. Shaded regions mark the 10th and 90th percentiles across shuftled repeats. Further details provided in Supplement E,
Since the covariance is often ill-conditioned or singular in practice, setting calculation also typically necessary.
Nonlinear invariances We discuss feature maps that enable nonlinear notions of equivalence, and which relate to kernel CCA [33] and CKA [9], in Supplement C.
2.4 Metrics for convolutional layers
In deep networks for image processing, each convolutional layer produces a h x X c array of activations, whose axes respectively correspond image height, image width, and channels (number of convolutional filters). If stride-1 circular convolutions are used, then applying a circular shift along either spatial dimension produces the same shift in the layer's output. natural to reflect this property, known as translation equivariance [23], in the equivalence relation on layer representations. Supposing that the feature map preserves the shape of the activation tensor, we have X R h xxxu wxxe xe for neural networks indexed by E 1,..., K. Letting S(n) denote the group of n-dimensional circular shifts (a subgroup of the permutation group) and 'ø" denote the Kronecker product, we propose:
X x + vec( = S C S₂ @ Q)vec((9))
(11)
for some S1 E S(h), S, E S(w), Q E O(c), the desired equivalence relation. This relation allows for orthogonal invariance across the channel dimension but only shift invariance across the spatial dimensions. The mixed product property of Kronecker products, (A & BC & D) AB & CD, ensures that the overall transformation maintains the group structure and remains an isometry. Figure 2B uses toy dataset (stacked MNIST digits) to show that this metric is sensitive to differences spatial activation patterns, but insensitive coherent spatial translations across channels. In contrast, metrics that ignore the convolutional structure (as in past work [6, 9]) treat very different spatial patterns as identical representations.
Evaluating Eq. (11) requires optimizing over spatial shifts in conjuction with solving a Procrustes alignment. If we fit the shifts by an exhaustive brute-force search, the overall runtime is O(mh² 2 hwe²), ), which is costly if this calculation is repeated across a large collection of networks. In practice, we observe that the optimal shift parameters are typically close to zero (Fig. 3A). This motivates the more stringent equivalence relation:
X X² + vec(X?))
Q)vec(X)) for some Q Q
(12)

---

which has more manageable runtime of O(nhwe? + c²). To evaluate the metries implied by Eq. (12), simply reshape each & from (m x h x e) tensor into (mhw c) matrix and apply the Procrustes alignment procedure as done above for previous metrics. In contrast, the "flattened metric" in Fig. 2B reshapes the features into (m x hwre) matrix, resulting in more computationally expensive alignment that runs in O(mh 2 w 2e? + 11 1 time.
2.5
How large of a sample size is needed?
An important issue, particularly in neurobiological applications, is to determine the number of network inputs, m, and neurons, n, that one needs to accurately infer the distance between two network representations [12]. Reasoning about these questions rigorously requires probabilistic perspective of neural representational similarity, which is missing from current literature and which we outline in Supplement D for generalized shape metrics. Intuitively. looser equivalence relations achieved by having more flexible alignment operations (e.g. nonlinear instead of linear alignments). Thus, looser equivalence relations require more sampled inputs prevent overfitting. Figure 3B-C show that this intuition holds in practice for data from deep convolutional networks. Metrics with looser equivalence relations the "flattened" metric in panel B, or e.g. the linear metric in panel C-converge slower to stable estimate as m is increased.
2.6 Modeling approaches and conceptual insights
Generalized shape metrics facilitate several new modeling approaches and conceptual perspectives. For example, a collection of representations from K neural networks can, in certain cases, be interpreted and visualized as K points the quotient manifold theorem [34] so long as G is not a finite set (c.g. corresponding to permutation) and all matrices are full rank in the feature space. This geometric intuition can be made even stronger when 9 corresponds to a connected manifold, such as (O(p). In this case, can be shown that the geodesic distance between two neural representations coincides with the metrics defined in Propositions and 2 (see Supplement C, and [15]). This result extends the well-documented manifold structure of Kendall's shape space [35].
smooth manifold (see Fig. 1). This holds rigorously due to
Viewing neural representations as points on a manifold is not a purely theoretical
models can be adapted to manifold-valued data (e.g. principal geodesic analysis [36] provides generalization of PCA), and additional adaptions are an area of active research [37]. However, there is generally no simple connection between these curved geometries and the flat geometries of Euclidean or Hilbert spaces [38].' Unfortunately, the majority of off-the-shelf machine learning tools are incompatible with the former and require the latter. Thus, we can resort to heuristic approach: the set of K representations can be embedded into a Euclidean space that approximately preserves the pairwise shape distances. One possibility, employed widely in shape analysis. is to embed points in the tangent space of the manifold at a reference point [41, 42). Another approach, which we demonstrate below with favorable results, optimize the vector embedding directly via multi-dimensional scaling [43, 44).
Applications and Results
We analyzed two lcrge-scale public datasets spanning neuroscience (Allen Brain Observatory, ABO; Neuropixels visual coding experiment; [21]) and deep learning (NAS-Bench-Iug [22]). We constructed the ABO dataset by pooling recorded neurons from K = 48 anatomically defined brain regions across all sessions; each X & E Rmxn was a dimensionally reduced matrix holding the neural responses (summarized by n = 100 principal components) to m = 1600 movie frames (120 second clip, "natural movie three"). The full however, we analyze a subset of = 2000 networks for simplicity. In this application each X E Rmxn is representation from specific network layer, with (m,n) E {(32 10 128), (162 x 10 256), (82 x 10", 5122), (10*, 512)}. Here, n corresponds to the number of channels and m is the product of the number of test set images (10*) and the height and width dimensions of the convolutional layer-i.e., we use equivalence relation in Eq. (12) to evaluate dissimilarity.
101 dataset contains 423,624 architectures;
'However, see [39] for conjectured relationship and [40] for result in the special of 2D shapes.
7

---

A
8 metric
B
metric clustering dendrogram
C
ABO
NAS-Bench-101
linear heuristic
2.0
data shuffle
1.8-
shuffle
distance
distortion
1.6
1-
1.2
embeoding
1.0
Σ Pt 1=1
40 embedding dim.
20
20
30 40
5
embedding dim.
heuristic clustering dendrogram
Figure 4: (A) Comparison of metric and linear heuristic. (B) Metric and linear heuristic produce discordant hierarchical clusterings of brain areas in the ABO dataset. Leaves represent brain areas that are clustered by representational similarity (see Fig. IC), colored by Allen reference atlas, and ordered to maximize dendrogram similarities of adjacent leaves. In the middle, grey lines connect leaves corresponding the same brain region across the dendrograms. (C) ABO and NAS-Bench- 101 datasets can be accurately embedded into Euclidean spaces. Dark red line shows median distortion. Light red shaded region corresponds to 5th 95th percentiles of distortion, dark red shaded corresponds to interquartile range. The mean distortion of a null distribution over representations (blue line) was generated by shuffling the m inputs independently in cach network.
Triangle inequality violations can occur in practice when using existing methods. As mentioned above, dissimilarity measure based the mean canonical correlation, 1 1-E, Pt, has been used in past work [7, 10]. We refer to this as the "linear heuristic." A slight reformulation of this calculation, arccos (E, Pr). produces metric that satisfies the triangle inequality (see Eq. (10)). Figure 4A compares these calculations as a function of the average (regularized) canonical correlation: one can that a-ccos(-) is approximately linear when the mean correlation is near zero, but highly nonlinear when the mean correlation is near one. Thus, we reasoned that triangle inequality violations are more likely occur when is large and when many network representations are close each other. Both ABO and NAS-Bench- 101 datasets satisfy these conditions, and in both cases we observed triangle inequality violations by the linear heuristic with full regularization the ABO dataset had at least one triangle inequality violation, while 1): 17/1128 network pairs in 00000 randomly sampled network pairs contained violations in the L...-Bench-lle -01 Stem layer dataset. We also examined a standard version of RSA that quantifies similarity via Spearman's 's rank correlation coefficient [8]. Similar to the results above, we observed violations in 14/1128 pairs networks in the ABO dataset.
Overall, these results suggest that generalized shape metrics correct for triangle inequality violations that do occur in practice. Depending on the dataset, these violations may be rare (~1% occurrence in ABO) or relatively common 100% in the Stem layer of MAS-Bench-101) These differences can produce quantitative discrepancies in downstream analyses. For example, the dendrograms produced by hierarchical clustering differ depending on whether one uses the linear heuristic or the shape distance (- 85.1 1% dendrogram similarity as quantified by the method in [45]; see Fig. 4B).
Neural representation metric spaces can be approximated by Euclidean spaces. Having estab- lished that neural representations can be viewed elements in metric space, it natural ask if this metric space is, loosely speaking, "close to" Euclidean space. We used standard multidimensional scaling methods (SMACOF, [43]; implementation in [46)) to obtain set of embedded vectors, y: E RR for which 81 (X. x R: lly; y, for i,j E .,. K. The embedding dimension is a user-defined hyperparameter. This problem admits multiple formulations and optimization strategies [44], which could be systematically explored in future work. Our simple approach already yields promising results: we find that moderate embedding dimensions (L ~ 20) is sufficient to produce high-quality embeddings. We quantify the embedding distortions multiplicatively [47]:
max (X X x)) yy 1 1111/01(X x)))
(13)
for each pair of networks i,j E ..... K. Plotting the distortions as a function of L (Fig. 4C), we see that they rapidly decrease, such that 95% of pairwise distances are distorted by. at most, -5% (ABO data) or 10% (NAS-Bench-101) for sufficiently large Past work [10] has used multidimensional scaling heuristically to visualize collections of network representations in I= 2 dimensions. Our results here suggest that such a small value of L,, while being amenable to visualization, results in highly distorted embedding. It is noteworthy that the situation improves dramatically when
8

---

A
Lov
5928
B
5p4
Golesh
Vise
SSoo
predicted score
Sata
Rear
DGmmo
Visung
SGN
AUC
go go
vise
550
PC
Shr
Sap troe
hierarchy score
-0.5
0.0
hiecarctty ecore
PC
Sip-hi22g
D
C
Stem
Stack 0
Stack
Stack
Avg Pool
0.6
Ridge
0.4
Kernel Ridge
Stem
Stack
stack 1
Stack
Avg Pool
2
Figure 5: (A) PCA visualization of representations across 48 brain regions in the ABO dataset. Areas colored by the reference atlas (see inset), illustrating a functional clustering of regions that maps onto anatomy. (B) Left, kemel regression predicts anatomical hicrarchy [48] from embedded representations (sce Supplement E). Righr, PCA visualization of 31 areas labeled with hicrarchy scores. (C) PCA visualization of 2000 network representations (a subset of NAS- S----Bnnc--111 101) across five layers, showing global structure is preserved across layers. Each network is colored by its position in the "Stack 1" layer (the middle of the architecture). (D) Embeddings of NAS-Bench- representations are predictive of test set accuracy, even in very early layers.
L is even modestly increased. While cannot easily visualize these higher-dimensional vector embeddings, we can use them as features for downstream modeling tasks. This is well-motivated as an approximation performing model inference in the true metric space that characterizes neural representations [47].
Anatomical structure and hierarchy is reflected ABO representations. We can now collect the L-dimensional vector embeddings of K network representations into matrix Z E R k The results in Fig. 4C imply that the distance between any two rows, I - 2,||, closely reflects the distance between network representations. i and j in shape space. We applied PCA to Z to visualize the K = 48 brain regions and found that anatomically related brain regions indeed were closer together in the embedded space (Fig. 5A): cortical and sub-cortical regions are separated along PC 1, and different layers of the same region (e.g. layers 2/3, 4, 5, and 6a of VISp) are clustered together. As expected from Fig. 4C, performing multidimensional scaling directly to low- dimensional space (L = 2, as done in [10]) results in a qualitatively different outcome with distorted geometry (see Supplement E). Additionally, we used Z fit an ensembled kernel regressor predict an anatomical hierarchy score (defined in [48]) from the embedded vectors (Fig. 5B). Overall, these results demonstrate that the geometry of the learned embedding scientifically interpretable and can be exploited for novel analyses, such as nonlinear regression. To our knowledge, the fine scale anatomical parcellation used here novel the context of representational similarity studies.
NAS-Bench-101 representations show persistent structure across layers. Since we collected representations across five layers in each deep network, the embedded representation vectors form set of five K x matrices, (Z,, Zg, Z3. Z4, Z5)- We aligned these embeddings by rotations in RR.. via Procrustes amalysis, and then performed PCA to visualize the K 2000 network representations from each layer in a common low- low-dimensional space. We observe that many features of the global structure are remarkably well-preserved- two networks that are close together in the Stack1 layer are assigned similar colors in Fig. 5C, and are likely to be close together in the other four layers. This preservation of representational similarity across layers suggests that even early layers contain signatures of network performance, which we expect to be present in the AvgPool layer. Indeed, when we fit ridge and RBF kernel ridge regressors to predict test set accuracy from representation embeddings, we see that even early layers support moderately good predictions (Fig. 5D). This is particularly surprising for the Stem layer. This is the first layer in each network, and its architecture is identical for all networks. Thus, the differences that detected in the Stem layer result only from

---

differences in backpropagated gradients, Again, these results demonstrate the ability of generalized shape metries incorporate neural representations into analyses with greater scale (K corresponding to thousands of networks) and complexity (nonlinear kernel regression) than has been previously explored.
Conclusion and Limitations
We demonstrated how to ground analyses of neural representations in proper metric spaces. By doing so, we capture number of theoretical advantages [16-20]. Further, we suggest new practical modeling approaches, such using Euclidean embeddings metric spaces. An important limitation of our work, as well as the past works we build upon, the possibility that representational geometry is only loosely tied higher-level algorithmic principles of network function [10]. On the other hand, analyses of representational geometry may provide insight into lower-level implementational principles [49]. Further, these analyses highly scalable, demonstrated by analyzing thousands of networks-a much larger scale than is typically considered.
approximate the representational
We used simple metrics (extensions of regularized CCA) in these analyses, but metries that account for nonlinear transformations across neural representations are also possible as we document in Supplement C. The utility of these nonlinear extensions remains under-investigated and it is possible that currently popular linear methods are insufficient to capture structures of interest. For example, the topology of neural representations has received substantial interest in recent years [50-5 (3). Generalized shape metrics do not directly capture these topological features, and future work could consider developing new metrics that do analysis may be useful towards this end [54-56].
variety of recent developments in topological data
Finally, several of the metrics we described can be viewed geodesic distances on Riemannian manifolds [35]. Future work would ideally exploit methods that are rigorously adapted to such manifolds, which are being actively developed [37]. Nonetheless, we found that optimized Euclidean embeddings, while only approximate, provide a practical off off-the-shelf solution for large-scale surveys of neural representations.
Acknowledgments
We thank Ian Dryden (Florida International University), Søren Hauberg (Technical University of Denmark), and Nina Miolane (UC Santa Barbara) for fruitful discussions. A.H.W. was supported by the National Institutes of Health BRAIN initiative (1F32MH122998-00) and the Wu Tsai Stanford Neurosciences Institute Interdisciplinary Scholar Program. E. K. supported by the Wu Tsai Stanford Neurosciences Institute Interdisciplinary Graduate Fellows Program. S. W.L.L. was supported by grants from the Simons Collaboration the Global Brain (SCGB 697092) and the NIH BRAIN Initiative (U19NS1 13201 and ROINS113119).
References
[1] David GT Barrett, Ari S Morcos, and Jakob H Macke. "Analyzing biological and artificial neural networks: challenges with opportunities for synergy?" Current Opinion in Neurobiology 55 (2019). Machine Learning, Big Data, and Neuroscience, pp. 55-64.
[2] Nikolaus Kriegeskorte and Xue-Xin Wei. "Neural tuning and representational geometry".
Nature Reviews Neuroscience (2021).
[3] Geoffrey Roeder, Luke Metz, and Durk Kingma. "On Linear Identifiability of Learned Repre- sentations".. Proceedings of the 38th International Conference on Machine Learning. Ed. by Marina Meila and Tong Zhang. Vol. 139, Proceedings of Machine Learning Research. PMI.R,
2021, pp. 9030-9039.
[4] Daniel L. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and
James J. DiCarlo. "Performance-optiiizee hierarchical models predict neural responses in higher visual cortex". Proceedings of the National Academy of Sciences 1 111.23 (2014),
pp. 8619-8624.
10

---

[5] Santiago A. Cadena, Fabian H. Sinz, Taliah Muhammad, Emmanouil Froudarakis, Erick Cobos, Edgar Y. Walker, Jake Reimer, Matthias Bethge, Andreas Tolias, and Alexander S. Ecker. "How well do deep neural networks trained on object recognition characterize the mouse visual system?" NeurlPS Workshop Neuro Al (2019).
[6] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. "SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability" Advances in Neural Information Processing Systems 30. Ed. by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett Curran Associates, Inc.,
2017, pp. 6076-6085.
[7] Ari Morcos, Maithra Raghu, and Samy Bengio. "Insights on representational similarity in neural networks with canonical correlation". Advances Neural Information Processing Systems 31. Ed. by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and Garnett. Curran Associates, Inc., 2018, pp. 5727-5736.
[8] Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. "Representational similarity analy- sis connecting the branches of systems neuroscience". Frontirrs in Systems Neuroscience (2008), p. 4.
[9] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. "Similarity of Neural Network Representations Revisited", Ed. by Kamalika Chaudhuri and Ruslan Salakhut- dinov. Vol. 97. Proceedings of Machine Learning Research. Long Beach, California, USA: PMLR, 2019. pp. -3229.
[10] Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sus- sillo. "Universality and individuality in neural dynamics across large populations of recurrent networks". Advances in Neural Information Processing Systems 32. Ed. by H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett. Curran Associates, Inc., 2019, pp. 15629-15641.
[11] Thao Nguyen, Maithra Raghu, and Simon Korbblith. Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth. 2020.
[12] Jianghong Shi, Eric Shea-Brown. and Michael Buice. "Comparison Against Task Driven Artificial Neural Networks Reveals Functional Properties in Mouse Visual C Neural Information Processing Systems 32. Ed. by H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buce E. Fox, and R. Garnett. Curran Associates, Inc., 2019, pp. 5764-5774.
Advances in
[13] Christopher G. Small. The statistical theory of shape. Springer series in statistics. New York:
Springer, 1996.
[14] David George Kendall, Dennis Barden, Thomas Carne, and Huiling Le. Shape and shape theory. New York: Wiley, 1999.
[15] Ian L. Dryden and Kantilal Mardia. Staristical shape analysis with applications in R. Chichester, UK Hoboken, NJ: John Wiley & Sons, 2016.
[16] Peter N Yianilos. "Data structures and algorithms for nearest neighbor search in general metric spaces". Proceedings of the fourth annual ACM-SIAM Symposium on Discrete algorithms. 1993, pp. 311-321.
[17] Sanjoy Dasgupta and Philip M Long. "Performance guarantees for hierarchical clustering". Journal of Computer and System Sciences 70.4 (2005), pp. 555-569.
[18] Saaid Baraty. Dan A. Simovici, and Catalin Zara. "The Impact of Triangular Inequality Violations on Medoid-Based Clustering". Foundations of Intelligent Systems. Ed. by Marzena Kryszkiewicz, Henryk Rybinski, Andrzej Skowron, and Zbigniew W. Raš. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 280-289.
[19] Fei Wang and Jimeng Sun. "Survey on distance metric learning and dimensionality reduction in data mining". Data Mining and Knowledge Discovery 29.2 (2015), pp. 534-564. [20] C. Chang, W. Liao, Y. Chen, and L. Liou. "A Mathematical Theory for Clustering in Metric Spaces". IEEE Transactions on Network Science and Engineering 3.1 (2016), pp. 2-16.
[21] Joshua H. Siegle et al. "Survey of spiking in the mouse visual system reveals functional hierarchy". Nature 592.7852 (2021), pp. 86 86-92.
11

---

[22] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. "NAS-Bench- 101: Towards Reproducible Neural Architecture Search". Proceedings of the 36th International Conference on Machine Learning. Fd. by Kamalika Chaudhuri and Ruslan Salakhutdinov. Vol. 97. Proceedings of Machine Learning Research. PMIR, 2019, pp. 7105- 7114.
[23] Taco Cohen and Max Welling. "Group Equivariant Convolutional Networks". Proceedings of The 33rd International Conference on Machine Learning. Ed. by Maria Florina Balean and Kilian Q Weinberger. Vol. 48. Proceedings of Machine Learning Research. New York, New York, USA: PMLR, 2016, pp. 2990-2999.
[24] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. "Algorithms for learning kernels based on centered alignment". The Journal of Machine Learning Research 13.1 (2012),
pp. 795-828.
[25] Mahdiyar Shahbazi, Ali Shirali, Hamid Aghajan, and Hamed Nili. "Using distance on the Riemannian manifold to compare representations in brain and in models". Neurolmage 239 (2021), P. 118271.
[26] Rainer Burkard, Mauro Dell' Amico, and Silvano Martello. Assignment Problems. Society for Industrial and Applied Mathematics, 2012.
[27] R Jonker and A Volgenant. "A shortest augmenting path algorithm for dense and sparse linear assignment problems". Computing 38.4 (1987), pp. 325-340.
[28] David F. Crouse. "On implementing 2D rectangular assignment algorithms". IFEEE Transac- tions on Aerospace and Electronic Systems 52.4 (2016), pp. 1679-1696.
[29] Peter H. Schönemann. "A generalized solution of the orthogonal procrustes problem". Psy- chometrika 31.1 (1966), pp. 1-10.
[30] J. C. Gower and Garmt B. Dijksterhuis. Procrustes problems. Oxford New York: Oxford
University Press, 2004.
[31] Agnan Kessy, Alex Lewin, and Korbinian Strimmer. "Optimal whitening and decorrelation". The American Statistician 72.4 (2018), pp. 309-314.
[32] Hrishikesh Vinod. "Canonical ridge and econometrics of joint production". Journal of econometrics 4.2 (1976), pp. 147-166.
[33] P. Lai and C. Fyfe. "Kernel and Nonlinear Canonical Correlation Analysis". International Journal of Neural Systems 10.05 (2000). PMID: 11195936, pp. 365-377.
[34] John M. Lee. Introduction to smooth manifolds. 2nd ed. Graduate texts in mathematics 218.
New York; London: Springer, 2013.
[35] David G. Kendall. "Shape Manifolds, Procrustean Metrics, and Complex Projective Spaces".
Bulletin of the London Mathematical Society 16.2 (1984), pp. 81-121.
[36] P. Thomas Fletcher and Sarang Joshi. "Riemannian geometry for the statistical analysis of diffusion tensor data". Signal Processing 87.2 (2007). Tensor Signal Processing, pp. 250-262.
[37] Nina Miolane, Nicolas Guigui. Alice Le Brigant, Johan Mathe, Benjamin Hou, Yann Thanwer- das, Stefan Heyder, Olivier Peltre, Niklas Koep, Hadi Zaatiti, Hatem Hajri, Yann Cabanes, Thomas Gerald, Paul Chauchat, Christian Shewmake, Daniel Brooks, Bernhard Kainz, Claire Donnat, Susan Holmes, and Xavier Pennec. "Geomstats: Python Package for Riemannian Geometry in Machine Learning".. Journal of Machine Learning Research 21.223 (2020), pp. 1-9.
[38] Aasa Feragen, Francois Lauze, and Soren Hauberg. "Geodesic Exponential Kernels: When Curvature and Linearity Conflict". Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2015.
[39] Aasa Feragen and Spren Hauberg. "Open Problem: Kernel methods on manifolds and metric spaces. What is the probability of a positive definite geodesic exponential kernel?" 29th Annual Conference on Learning Theory. Ed. by Vitaly Feldman, Alexander Rakhlin, and Ohad Shamir. Vol. 49. Proceedings of Machine Learning Research. Columbia University, New York, New York, USA: PMLR, 2016, pp. 1647-1650.
[40] Sadeep Jayasumana, Mathieu Salzmann, Hongdong Li, and Mehrtash Harandi. "A Framework for Shape Analysis via Hilbert Space Embedding". Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2013.
[41] Ian L. Dryden and Kanti V Mardia. "Multivariate shape analysis". Sankhya: The Indian Journal of Statistics, Series A (1993), pp. 480-
12

---

[42] F. James Rohlf. "Shape Statistics: Procrustes Superimpositions and Tangent Spaces". Journal of Classification 16.2 (1999), pp. 197-223.
[43] Ingwer Borg and Patrick JF Groenen. Modern multidimensional scaling: Theory and applica- tions. Springer Science & Business Media, 2005.
[44] Akshay Agrawal, Alnur Ali, and Stephen Boyd. "Minimun-Disttrtiion Embedding". arXiv
(2021).
[45] Alexander J. Gates, Ian B. Wood, William P. Hetrick, and Yong-Yeol Ahn. "Element-centric clustering comparison unifies overlaps and hierarchy". Scientific Reports 9.1 (2019), p. 8574. [46] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. "Scikit-learn: Machine Learning in Python" Journal of Machine Learning Research 12 (201 I), pp. 2825-2830.
[47] Leena Chennuru Vankadara and Ulrkee von Luxburg. "Measures of distortion for machine learning". Advances in Neural Information Processing Systems. Ed. by S. Bengio, H. Wallach, Larochelle, Grauman, N. Basa-Bianchi, and Garnett. Vol. 31. Curran Associates, Inc., 2018.
[48] Julie A. Harris et al. "Hierarchical organization of cortical and thalamic connectivity". Coanectivit"".Nattuie Natuure 575.7781 (2019), pp- 195-202.
[49] Jess B. Hamrick and Shakir Mohamed. "Levels of Analysis for Machine Learning". Proceed- ings of the ICLR 2020 Workshop on Bridging Al and Cognitive Science. 2020.
[50] Erik Rybakken, Nils Baas, and Benjamin Dunn. "Decoding of Neural Data Using Cohomolog- ical Feature Extraction". Neural Computation 31.1 (2019), pp. 68-93.
[51] Rishidev Chaudhuri. Berk Gerçek, Biraj Pandey, Adrien Peyrache, and Il Fiete. "The intrinsic attractor manifold and population dynamics of a canonical cognitive circuit across waking and sleep". Nature Neuroscience 22.9 (2019), pp. 1512-1520.
[52] Tevin C. Rouse, Amy M. Ni, Chengcheng Huang, and Marlene R. Cohen. "Topological insights into the neural basis of flexible behavior". bioRxiv (2021).
[53] Richard J. Gardner, Erik Hermansen, Marius Pachitariu, Yoram Burak, Nils A. Baas, Benjamin A. Dunn, May-Britt Moser, and Edvard I. Moser. "Toroidal topology of population activity in grid cells". bioRxiv (2021).
[54] Genki Kusano, Kenji Fukumizu, and Yasuaki Hiraoka. "Kernel Method for Persistence Di- agrams via Kernel Embedding and Weight Factor". Journal of Machine Learning Research 18.18 189 (2018), pp. 1-41.
[55] Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. "Topological Autoencoders". Proceedings of the 37th International Conference on Machine Learning. Ed. by Hal Daumé III and Aarti Singh. Vol. 119. Proceedings of Machine Learning Research. PMLR, 2020, pp. 7045-7054.
[56] Kristopher Jensen, Ta-Chu Kao, Marco Tripodi, and Guillaume Hennequin. "Manifold GPLVMs for discovering non-Euclidean latent structure in neural data". Advances in Neural Information Processing Systems. Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin. Vol. 33, Curran Associates, Inc., 2020, pp.
13


Reviews:
[{"review_id": "s_JAHbIKqmb", "reviewer": "Reviewer_2CzF", "summary": "This paper proposes a relatively novel method using shape metrics to measure similarity across representations in both neural networks and in the brain, which is an important problem in both fields. The paper compares its method to existing representational metrics and shows that it is able to overcome some of the previous limitations as well as to provide new insights from representations. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The incorporation of shape metric into analyzing representation similarity is novel to my knowledge. \n\nQuality: The proposed shape metrics is well motivated and mathematically sound. Given that the authors shows that their method could give different results in hierarchical clustering of representations, which by itself is very interesting, it is unclear how differently it does in other applications comparing to existing methods. The persistent structure across layers, as well as the relationship from test set accuracy and representations might be visible using existing representation similar metrics. I would like to see more experiments done to further validate the method. For example, it can be used to compare representations learned for different tasks to inform multi-task learning, or on a different brain dataset. \n\nClarity: The paper is clear and well written. \n\nSignificance: Application of the method proposed by this paper is very general and extremely useful to both neuroscience and machine learning. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "jmUEvyPVfxF", "reviewer": "Reviewer_TdwX", "summary": "The paper is concerned with issues that are fundamental to basic research on understanding natural and artificial neural networks. Such research often involves measuring the similarity of neural representations (patterns of neural activity). The authors propose desirable attributes for the methods used for such comparisons, e.g. that the similarity function should define a metric metric space: it should satisfy the constraints of equivariance, symmetry and the triangle inequality. They show that many commonly used methods do not satisfy these constraints but can be easily modified to do so. They also construct a metric especially suited to convolutional layers, which traditionally have not been straightforwardly analyzed by existing methods. The authors also discuss practical issues of general importance: They discuss what sample size is required for such metrics to be accurate/robust, which is highly important for neuroscience research where it can be difficult to obtain large sample sizes. They also talk about dimensionality reduction, showing that approximating neural embeddings with low dimensional Euclidean spaces, while suitable for visualization, can highly distort the embedding. They validate their new metrics by showing that it can recover known structure in large scale datasets.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is important, timely, clearly written, and likely to be impactful in both neuroscience and machine learning. It addresses an important issue that I have previously noticed in the work on similarity of neural representations. The analysis appears to be appropriate and correct. The work is clearly motivated and the results actionable. I'm afraid I have nothing constructive to add. My only notes on the paper are positive. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have identified important limitations of their work and highlighted directions for future work. They do not address potential negative societal impact but I can offer no suggestion in that regard. The work does not appear to have any obvious potential negative societal impacts. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.25"}, "extra_text": {}}, {"review_id": "a7ZP2YKHCS9", "reviewer": "Reviewer_2qso", "summary": "The current manuscript presented a unifying framework to analyze the representation similarity between neural networks. The authors first established that the distance metrics between neural representations should satisfy equivalence, symmetry and triangle inequality. And the authors showed that a simply modification of canonical correlation analysis would satisfy these reequipment. Next, the authors identified the transformation matrix that best conform the two representation spaces under various conditions. The current framework revealed the hierarchy of multiple brain areas and visualized the representation structure of a deep neural network.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In the current manuscript, the authors provided theoretical foundation for representation similarity analysis, as well as demonstrated its application in both real and artificial neural networks. While the methods seem to be not novel, the theory was valuable and the application results were intuitive. However, it was confusing that mathematical foundations did not lay out in a way to guide the applications. It seems extra math were provided without giving an example (eg. the permutation invariance, rotation invariance and so on), while the real-world application seems to only relevant to a small fraction of the theory.   ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed that limitation of representation analysis and acknowledged that the representation similarity was only loosely related to network function. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "GCGwBJrS4ci", "reviewer": "Reviewer_Fmhp", "summary": "This paper develops metrics for comparing different (yet presumably related) representations. The emphasis is on representations generated by artificial or biological neural networks. The paper relates this development to previously proposed approaches. Experimental results on recordings of neuronal activity in the mouse brain and representations generated by different layers of different deep neural networks are presented.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- It is not clear why representations of different networks should be linearly related. It seems identifying topological similarity would be a better fit for this problem, although it may be computationally impractical. I find the reasoning behind the equivalence classes being defined through linear transformations missing. At least, a topology-centered introduction which discusses its practicality before settling on linear transformations would be a better exposition.\n- It takes a while to realize that this work concerns the alignment of representations and these representations do not have to come from networks. Network representations is the application here, although the development does not have much to do with having an underlying biological/artificial neural network.\n- The paper devotes ~5 pages to reviewing the basics and setting up the problem. As a result, the experimental section is somewhat rushed and whether the approach offers practical improvements is not clear. Although, of course, having a principled approach is inherently advantageous.\n-  On a related note, I couldn’t tell if the authors claim novelty about the propositions. They are rather elementary, including the generalized version in the supplement, which covers both cases. It wouldn’t surprise me if they appear in a textbook etc although I can’t cite one here. Please clarify this point.\n- The experiments would be much more meaningful in the presence of baselines. For instance, if the ABO experiment were repeated, except with non-metric embeddings, but following the same multidimensional scaling and PCA parameters, how different would the embedding of the brain regions be?\n- Why does it make a big difference to embed in L-dim space first and then perform dim. reduction rather than embedding immediately?\n\nMinor:\n- (line 120) Please define the equivalence relation.\n- (line 31) As written, the definition corresponds to a semi-metric. Please include the equivalence condition here as well.\n\nClarity: The exposition is generally clear and it is not hard to follow the development.\n\n------------------------------ UPDATE ------------------------------\n\nI decided to increase my score from 5 to 6. Please refer to the discussion with the authors for details on the topological aspects of the problem. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, but I don't think this mathematical development carries foreseeable potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new similarity measure for representations. Given the wide recent use of comparing representations both in analyzing neural networks and in analyzing brain activity. There was enthusiasm from the reviewers about the fact that the metrics introduced are formally based, but however, there was a discussion about whether the paper should include a more formal topological analysis. 

A consensus was reached by having the authors discuss their specific assumptions, the potential connections to topological analysis and what could be addressed in future work. Given that the authors have promised to implement these changes, I recommend acceptance. 

Decision: Accept (Poster)