Neural Networks as Paths through the Space of Representations
Authors: Richard D Lange
Devin Kwok
Jordan Kyle Matelsky
Xinyue Wang
David Rolnick
Konrad Kording
Venue: Submitted to ICLR 2023 (2023)
Keywords: representational similarity
metrics
geodesic
neural network expressivity
visualization

Abstract: Deep neural networks implement a sequence of layer-by-layer operations that are each relatively easy to understand, but the resulting overall computation is generally difficult to understand. We develop a simple idea for interpreting the layer-by-layer construction of useful representations: the role of each layer is to reformat information to reduce the "distance" to the desired outputs. With this framework, the layer-wise computation implemented by a deep neural network can be viewed as a path through a high-dimensional representation space. We formalize this intuitive idea of a "path" by leveraging recent advances in metric representational similarity. We extend existing representational distance methods by computing geodesics, angles, and projections of representations, going beyond mere layer distances. We then demonstrate these tools by visualizing and comparing the paths taken by ResNet and VGG architectures on CIFAR-10. We conclude by sketching additional ways that this kind of representational geometry can be used to understand and interpret network training, to describe novel kinds of similarities between different models, and for representation-learning without backpropagation.

NEURAL NETWORKS AS PATHS THROUGH THE SPACE OF REPRESENTATIONS
An Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks implement a sequence of layer-by-layer operations that are each relatively easy to understand, but the resulting overall computation is gen- erally difficult to understand. We consider simple hypothesis for interpreting the layer-by-layer construction of useful representations: perhaps the role of each layer is to reformat information to reduce the "distance" to the desired outputs. With this framework, the layer-wise computation implemented by a deep neural network can be viewed as a path through a high-dimensional representation space. We formalize this intuitive idea of "path" by leveraging recent advances in metric representational similarity. We extend existing representational distance methods by computing geodesics, angles, and projections of representations, going beyond mere layer distances. We then demonstrate these tools by visualizing and compar- ing the paths taken by ResNet and VGG architectures on CIFAR-10 10. We conclude by sketching additional ways that this kind of representational geometry be used to understand and interpret network training, and to describe novel kinds similarities between different models.
Keywords:
representational similarity, metric spaces, geometry, neural network expressivity, vi-
sualization
INTRODUCTION
core design principle of modern neural networks is that they process information serially, pro- gressively transforming inputs until the information is in format that is immediately usable for some task (Rumelhart et al.) 1988 LeCun et al.) 2015). This idea composing sets of simple units to construct more complicated functions is central to both artificial neural networks and how roscientists conceptualize various functions in the brain (Kriegeskorte) 2015 Richards et al. 2019 Barrett al. 2019).
Our work is motivated by a spatial analogy for information informatioo-proocssinng we imagine that outputs are "far" from inputs if the mapping between them is complex, or "close" if it is simple. In this spatial analogy, any one layer of a neural network contributes a single step, and the composition of many steps transports representations along path towards the desired target representation. Formalizing this intuition requires a method to quantify if any two representations are "close" (similar) or "far" (dissimilar) (Kriegeskorte, 2009 Kornblith al. 2019). order to use this kind of spatial or ge- ometric analogy for neural representations, we need some way to quantify the "distance" between representations. We build on recent work introducing metrics for quantifying representational dis- similarity (Williams et al. 2021 Shahbazi et al. 2021). Representational dissimilarity is quantified using a function d(X, nonnegative value for their dissimilarity. Here, X U R x is the space of all m matrices for all The matrices X and Y could be, for instance, the values of two hidden layers in a network with n. and ny units, respectively, response to m inputs.
X x x that takes in two matrices of neural data and outputs
What are desirable properties of such representational dissimilarity function? Previous work has argued that any sensible dissimilarity function should be nonnegative, so d(X, Y) 2 O, and should return zero between any equivalent representations, means that X and Y are in the same equivalence class. For example, we may wish to design the function d that d(X, Y) 0 if Y is a shifted copy of X, or if it is non-degenerate scaling,
d(X, Y) 0 X Y, where X ~ Y

---

A
H
(iii)
(iv)
(v)
(vi)
(vii) (viii)
Conv
BatchNorm
ReLU
Project
Skip
Pool
Linear
SoftMax
Targets
B
Hidden layer activity
Layer distances
F
Path & geodesic
(ii)
(v)
(viil)
m imputs
(iii)
(iv)
embedding function (Gram matrices)
C
42
(vi)
(Vili)
PC1
(vii)
X
Y
distance from input
G
distance to targets
H
projected progress dist from
dist to targets
D
Angular CKA
1.5
M
0.5
projected progress
Y
dos X,
internal
layer (color coded)
x
angle
target angle
J
Interiar
target
n/2
layer (color coded)
Figure 1: Representation paths of Resnet- 14 model trained on CIFAR-10, evaluated using the An- gular CKA metric with linear kernel. A) Schematic of Resnet-14 14 architecture with color-coded layers. Gray boxes correspond to residual blocks. B, C, D) Matrices of neural data are converted into points in the metric space of Angular CKA. The outputs of each block form X matrix for m examples and n channels (B); data from layers (ii). (v), and (vii) in panel are shown. Out- puts are transformed into centered and normalized Gram matrices by the embedding function (C). The resulting points are on a spherical manifold where the are distance between points provides measure of the distance between representations (D). E) Pairwise distance between layers com- puted using Angular CKA. F) 2D embedding of the network's 's path using multi-dimensional scaling (MDS) down to 15D followed by PCA. Includes points for the input pixels (black square), target class labels (purple star), and points calculated from the geodesic between input and labels (black dashed line). G) Three types of distances plotted in (H): distance of layer from input (orange A), distance to target (green V), and projected distance along the geodesic from input to target (blue). Two types of angles plotted (J): "internal angle" or the angle between adjacent path segments (red), and "target angle" or the angle between each segment and the geodesic from segment to tar- gets (green). Note that we treat each residual block as a single step or segment the path.
rotation, or affine transformation X (Krnblith et al. 2019; Williams et al., 2021 Shahbazi et al., 2021). A second desirable property is that d is symmetric, so d d Y) d(Y, X). A third is that satisfies the triangle inequality, or d(X, Y) s d(x, Z) + dZZ Y). As argued by Williams et al. 2021 representational dissimilarity function that fails to satisfy the triangle inequality lead
2

---

to errant results when, for instance, clustering embedding representations based their pairwise dissimilarity. A dissimilarity function that satisfies all of the above properties equivalence, sym- metry, and the triangle inequality qualifies as a metrid on X (Burago et al. 2001). Examples of metrics between neural representations were recently developed independently by Williams et al. (202T) and (Shahbazi et al. (201))
We are interested in using metrics between neural representations to explore how representations evolve spatially as they are transformed through the hidden layers of deep networks. Not all metrics are sufficient for the kind of spatial reasoning that is, not all metrics can be interpreted as distances. For example, consider the trivial metric
fo 1
i Y
d(x, Y)
otherwise
This is valid metric according to the equivalence, symmetry, and triangle inequality criteria, but it is useless as a tool for characterizing distances. To be interpretable as a measure of distance, d(X, Y) must satisfy intuitive fourth condition called rectifiability: the distance between any two points must be realizable as the (infimum of the) sum of distances of segments along path between them (Burago et al., 2001T. While not all metrics perhaps unsurprising that this condition is met by many sensible metrics, including those already developed by Williams et al. (202I) and /Sahbazii et al. (2021). In fact, all metrics considered in this paper are Riemannian metries, which not only implies that they are rectifiable, but further requires that points live on smooth manifold (Burago et al.) 2001). Each restriction on the type of metric comes with additional structure that we can use to inspect and visualize neural representations: rectifiability allows us to smoothly interpolate neural representations along a geodesic as well as compute projections and angles, and Riemannian structure allows to meaningfully compare the direction of steps taken by different layers.
rectifiable (such the trivial metric above), it is
The main contributions of this paper are
We put forward the spatial "path" analogy for deep neural networks, and quantitatively evaluate using recently developed methods for representational distance.
We develop a toolbox for analyzing geometric properties of sequences of neural network layers, and show how existing representational distance measures imply a rich set of geo- metric concepts beyond mere pairwise distance.
We create novel visualizations of how representations transformed through the layers of deep networks.
We apply these techniques to compare paths taken by wide and deep residual networks, as well as four VGG architectures, all trained CIFAR-10, finding differences in both the magnitude and direction of steps taken by layers in wide versus deep models Nguyen et al. (2021).
PRELIMINARIES
2.1 RELATED WORK
One motivation for thinking of neural networks paths is that it provides a compelling analogy for the way that complex functions (deep networks) can be composed out of simple parts (layers). In- deed, it well known that both deeper (Poole al. 2016 Raghu al. 2017 Rolnick and Tegmark 2017) and wider (Hornik et al. 1989) neural network architectures can express a larger class of functions than their shallower implementing a particular complex function constrains the role of individual layers and interme- diate representations the intervening layers between input and output. Our work is in line with other recent efforts to characterize the features learned in hidden layers smoothly varying between inputs and outputs (Chan et al., 2020 Yang et al., 2022 He and Su 2022). In our path framework,
narrower counterparts. However, much less is known about how
It would be more precise to say it is a "metric" on the quotient space of the equivalence class X ~ Y. "pseudometric" X, but we suppress this distinction throughout avoid excess verbiage. For more thorough treatment of equivalence classes, see (Williams et al. 2021
3

---

relatively narrow layers take shorter steps, and chaining them together increases the total achievable length, while wide layers are capable of taking a single large step, and all such steps contribute by moving "closer" to the targets. Our work is first step, to speak, towards formalizing this notion of composing simple functions in geometric terms.
There is rich literature applying geometric concepts like distance between representations for-
malize notions of "similarity" in neuroscience and psychology (Edelman 1998 Täkel et al., 2008
Rodriguez and Granger 2017 Hénafff al. 2019 Kriegeskorte and Wei 2021). However, there is a crucial difference between measuring similarity or distance between points in a given space, and measuring distances between representational spaces themselves. The former includes questions like. "how far apart are the activation vectors for two inputs in a given layer?" The latter, which is the subject of this paper, asks instead, "how far apart are two layers' representations, considering all inputs?"
Our work is most closely related to, and draws much inspiration from recent advances in represen- tational similarity analysis (RSA). In particular, Kornblith et al. (2019) showed that kernel method for testing statistical dependence, known as CKA (Gretton et al. 2005 Cortes et al. [2012), J, is closely related to classic RSA [Kriegskkorte (2009). In follow-up work, Nguyen et al. (2021) used CKA to make layer-by-layce comparisons between wide and deep networks which we elaborate on in sec- tion 3.2 below. Independently, both WWlliams et al. (202I) and [Shahbazi et al. (2021) developed
methods to compute metrics between neural representations. [Shahbazi et al. (2021) proposed us- ing the so-called Affine-Invarian Riemannian Metric on the space of symmetric positive-definit matrices (Pennec) 2006 2019). [Williams et al. (202I) derived a metric variation of CKA which we call Angular CKA, well as a family of Generalized Shape Metries. We extend this prior work by computing not just pairwise distances, but by also introducing a suite of tools for analyzing the geometry of representation space induced by each of these distance functions. Finally, whereas Williams et al. (202I) and Shahbazi et al. (202T) compare representations across models, we com- pare representations within a single model to study the transformation of information from inputs to outputs through hidden layers.
2.2 DISTANCE METRICS BETWEEN NEURAL REPRESENTATIONS
In our notation, a deep neural network processes an input x" through a sequence of hidden layers, fxt x--1), and produces outputs x trained to match some target outputs (LeCun et al. 2015). In the common example of image classification, x" is the input image, targets are the class label, and x set of (log) class probabilities. We will use subscripts to index inputs, so x/ is the mn-dimensional response of the Ith hidden layer to the ith input, xf, with i E (1...., m]. We use capital letters like X', X or Y to refer to m X n matrices of neural representations in response to m inpute and we treat target labels as one vectors (i.e. if Y is the target representation, then it is a m x 10 matrix for the CIFAR- 10 dataset (Krizhevsky) 2009)). We will adopt the convention that I < A if x' is direct ancestor of x" through the layers of a network (layers will in general be partially but not strictly ordered). We are interested in functions d(X, Y) for that quantify the "dissimilaritt"" between two neural representations; when d(X, Y) satisfies the four conditions of a length metric - equivalence, symmetry, triangle inequality, and rectifiability we will say that it measures their "representational distance." " Note that X and Y may be different layers of the same model or layers from different models, as long as they are evaluated on the same inputs. We use d(X, Y) to refer to the distance function between arbitrary representations X and Y.
We can unify and organize existing representational distance metrics by recognizing that distance metries use a two-stage approach to defining d(X, Y): first, X and Y are mapped to a common space M through an embedding function f: X - M,, then distance is computed using a distance metric defined on M. More precisely,
(I) where X = f(X) is the result of mapping X from X to M. In all cases we consider here, M is a Riemannian manifold with metric dy. Equivalence relations on X can be built into this two-stage approach in either stage: d(X, Y) can be made invariant to changes in the scale of X either by imposing a canonical scale in f, or by preserving scale in f but using a ocale-invariant metric dag.
dx, Y) dm (X, Y):
We assume convolutional layers are flattened, in which case n is the product of height, width, and feature dimensions.
4

---

We implemented three existing families of representational distance, each of which can be under- stood as different choices for f, M,, and dy. and extended them to further compute various geometric quantities including geodesics, projections, tangent vectors, and angles. The representational dis- tance metrics we investigate here are Angular CKA (with a linear kernel), Shape Metrics (with angular distance), and the Affine Invariant Riemannian Metric (with squared exponential kernel and ridge regularization) (Williams et al., 2021] Shahbazi et al. 2021]. We have found that results using Angular CKA are the most interpretable among the metrics we have tested, our main re- sults are presented using Angular CKA. Results using other metrics are shown in Figure C.1 In the [..aand and Appendix [A] provide mathematical details on all metrics in one place, including discussion of their invariances.
following section, we will give mathematical details for Angular CKA; Table A..1
2.3 THE GEOMETRY OF ANGULAR CKA
Angular CKA was introduced by Williams et al. (002)) (eq (60) in their supplement). It is defined as the arccosine of CKA, which itself derived from the Hilbert- Schmidt independence criterion (HSIC) (Gretton et al. 2005; Cortes et al. 2012 : Kornblith et al.) 2019). Because HSIC and CKA measure statistical dependence, distance measured by Angular CKA is small when the rows of X and Y are strongly statistically dependent, and large when they are independent. While Angular CKA was originally introduced simply sentations, here we exploit the fact that Angular CKA is the arc-length on Hypersphere to compute additional geometric properties of the space.
method for computing a metric between neural repre-
Angular CKA is equivalent to arc-length distance on the spherical manifold consisting of centered and normalized m m Gram matrices. Let Gx denote the Gram matrix of X, i.e. Gxiş given by the inner-product between the ith and jth rows of X. Optionally, this inner-product may be computed using a kernel; following previous work (Kornblith al.) 2019; Nguyen et al.] (02T). results in this paper use Linear CKA, i.c. admissible and available in our python package at [link redacted]. The normalized and centered Gram matrix is given by
Gx XX" Other kernels are in principle
HGxH
Gx = |GG H|
where H = -11 is the xm centering matrix, and ||-Ile the Frobenius norm of matrix.
The Riemannian manifold M for Angular CKA consists of all centered and normalized symmetric
positive definite matrices; is a sphere because (x Gx 1, where (A, B)F F Tr(A T B) is the Frobenius mmner-product.
Comparing to equation []] we can see that the embedding function for Angular CKA is f(X) = Gx. Distance according to Angular CKA is
d(X, Y) = dr (I(X), = dm (Gx, Gx)
arccos ((xx,Gx)
(2)
(Figure B-D))
Because Angular CKA is are length hypersphere, easily compute its geodesics: the shortest path connecting Gx Gy is
geodesic Gxx GY sim((1- - t]
sin(tS)) sin
Gx sin(
GY
where E [0, 1] is the fraction of distance along the geodesic from X to Y, and Ω = drc (Gx, GY). This is a direct application of the SLERP formula for hyperspheres, applied to G-space.
numerical optimization methods to project point in MI onto the geodesic spanning two other points. The projection of X onto the geodesic spanning Y and Z can be found
For all metrics,
pttsa:///en.wiixxpdaaaa.o./////
5

---

by minimizing the distance from X geodesic(Y, Z, t) with respect to t. In practice, we solve this as an optimization problem with respect to t. Figure [ demonstrates one way that this idea of projecting neural representations can be used: we project each hidden layer onto the geodesic connecting inputs (raw pixels) to targets (one-hot labels). (Note that we keep only the representations labeled (i) through (vii) in Figure TA that the resulting path consists only of steps that perform comparable operations). This provides more gradual picture of the "progress" being made by cach layer towards the targets than mere distance between layers and inputs or targets (Figure [H).
For all metrics, we can compute angles between any three neural representations using the tangent space defined by the metric. The logarithmic map is a function that takes in two points in M and returns a tangent vector that "points" from one to the other (Burago et al. 2001). The logarithmic map for Angular CKA from Gx to Gy is
logox (Gx) W Warccos
(x)
(4)
where W is the unit tangent vector at Gx pointing towards Gr, given by
GY Gx (Gx (xx.Gx).
W
|| - Gx (āx, GY Gx), 1|:
We then compute angles between triplets of points by computing the inner-product of their tangent vectors. In the case of Angular CKA in particular, let Wxy denote the tangent vector pointing from X to Y, i.c the result of logex (Gy). Then,
0(Gx, Gy, Gz) = arccos ( (Yx WYZ)F
(5)
|| le WYZllF
the angle of the XYZ triangle Y. Equations (3 are simply applications of the geometry of hypersphere to the space of m x m centered and normalized Gram matrices. Figure demonstrates two types of angles that are of interest: "internal angles," which quantify how straight a path is, and "target angles," which quantify to what extent each step points in the direction of the targets. Interestingly, we find that nearly all internal angles are orthogonal, and only later layers begin to take steps in the direction of the targets (Figure
2.4 COMPARISONS WITH OTHER METRICS
The main results in this paper are presented using Angular CKA for ease of exposition. Mathemat- ical details for other metrics can be found in Table A.I and Appendix A Figure C.I gives results like those in Figure [[ using the metrics proposed by Williams et al. (021) (Angular Shape and Euclidean Shape) Shahbazi et al. 2021 (Affine-Invariant Riemannian).
3 EXPERIMENTS
3.1 HOW PATHS CHANGE OVER TRAINING
Whereas Figure ! visualizes properties of the path taken by well-trained Resnet-14 model, here we ask how properties of its path emerge over training. For this, we trained new model of the same architecture, and logged model parameters every 10 batches. Since the vast majority of changes occur in the first epoch, Figure 2 shows how distances, projected distances, and angles change for every 10 steps in the first epoch, plus single additional point after 20 epochs to show where values converge after longer training.
At initialization, the distance from inputs initially increases (Figure 3)). ) but the distance to targets constant (Figure (A). This implies that the network's 's representations are changing in some direction that is orthogonal to making "progress" towards the targets. Indeed, we find that plotting projected progress is useful way visualize training (Figure At initialization, projected progress is zero for all layers, and it quickly begins to ramp upwards after a few training steps.
We hypothesized that training would cause the network's 's path to become more straight, but surpris- ingly we found that this is not the case. In fact, internal angles begin slightly above π 7/2 at initializa- tion, and concentrate towards /2 as training progresses (Figure 2)). This means that the function

---

distance targets
B
distance from inputs
C
projected progress
1.
6.5
0.5-
0.5
6.0
0.9
internal angles
target angles
step 0
step
step
step 7800
layer (color coded)
layer (color coded)
Figure 2: Evolution of paths over training for a Resnet-14 model trained on CIFAR-IO. We plot lines every 10 batches for the first epoch (blue to yellow), and an additional line at epoch 20 (ma- genta). The first row plots the same distances as in Figure A) Distance to targets (one-hot labels) from hidden layers
function of layer depth. B) Distance from inputs (pixels) to hidden layers. C) Projected progress, or the distance from inputs to the projection of hidden layers onto the geodesic connecting inputs to targets. D) Internal angles, angles between adjacent path segments versus increasing layer depth. E) Target angles, or angles between each segment and the geodesic from its starting point the targets.
implemented by each residual block applies an orthogonal operation (in representation space) com- pared to the blocks before or after it. To our knowledge, this kind of orthogonality between layers has not been previously observed.
Figure _ shows how "target angles" evolve over training. Consistent with past work, this visual- ization shows that the emergence of layers that point in the direction of the targets does not happen until later in training.
3.2
COMPARING PATHS OF WIDE VERSUS DEEP MODELS
A
distance targets
B
distance from inputs
C
projected progress
1.5
1.5
dszince
1.0
1.0
1.0
0.5
0.5
0.0
0.0
0.4
0.2
1.0
0.4
0.8
1.0
(faciion)
(haction)
(fraction)
D
E
Resnet
interior angles
target angles
width depth
VGG
1x
13
I22
62
26
16
8x
32
10x
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Depth (fraction)
Depth (fraction
Figure 3: Comparison between paths of different model architectures. We compare residual models by width (brightness) and by depth (hue), well as VGG networks (yellows). Plots follow layout of 2 2 A) distance to targets, B) distance from inputs, C) projected distance to targets, D) angle between path segments, and E) angle between segment and target.
Inspired by the results of Nguyen et al. (2021). we compare paths taken by "wide" versus "deep" models. In 3 we show the same quantities computed in Figure but comparing different Resnet widths and depths. Our results confirm the main findings of Nguyen et al. (2021). namely that wide and deep Resnets compute similar functions in other words, wide-model paths and deep-model paths all follow roughly the same trajectory in representational space. Note that the x-axis for these plots is scaled by the depth of each model; the fact that all Resnets are superimposed

---

on each other means that all of them distribute the function of each layer relatively evenly over their depth. This is not universally true of all models - we also compare with VGG, and see salient differences between the VGG and Resnet architectures, where the progress made in VGG (without skip connections) is much more dependent on absolute rather than relative depth.
3.3 DECOMPOSING EACH STEP INTO "PROGRESS" AND "DEVIATION"
We next asked to what extent each step in the path, i.e. each residual block, moves representations towards the targets (progress) or in orthogonal directions (deviation; ( Note that unlike "projected progress" above, which was measured relative to the input-targets geodesic, here we compare each step from X to X to the geodesic connecting current position X' to the targets. A model with D blocks provides D-1 measures of both progress and deviation one for each block. Figure T summarizes the net contribution of progress and deviation for all layers in models with varying architectures. Note the following salient effects: first, there is strong overall upward trend for every block that makes 0.1 radians of progress towards the targets (recall that Angular CKA is an arc-length), it incurs some deviation of about 0.3 radians in an orthogonal direction, and this trend is remarkably uniform for both Resnet and VGG architectures. Second, deeper models are lower and to the left; deeper models parcel out both progress and deviation among their layers. Third, we can identify an effect of width at constant depth; at a constant depth, wider models incur less deviation than their narrower counterparts.
DISCUSSION
In this work, we interpret neural networks spatially as paths from inputs to outputs through a space of intermediate representations. These paths have rich geometric structure, inherited from computing distances between representations, and we can use that structure to build intuitions about network structure, training, and comparisons between models.
We investigated three families of representational distance metrics - Angular CKA, Shape Met- rics, and the Affine Invariant metric on symmetric positive definite matrices (Willias et al. 2021 Shahbazi et al., 2021). Surprisingly, we found that trained networks take circuitous paths according all of these metrics, and deviate far from the shortest paths from inputs to targets which are de- fined geometrically by each representational distance metric. There are three potential explanations for this. First, networks may be taking short paths according to some metric other than those we investigated here, implying that our metrics may not reflect the distance between representations in terms of ease of computation. Second, neural networks may fail to take efficient paths. The distance metrics we consider are all differentiable, and so an interesting question for future work is whether networks can be regularized to take shorter paths, and whether such regularization will improve or reduce their performance or generalization ability. Third, it could be that networks take the shortest and most direct path which is possible under some architectural constraints, which may prevent the hidden layers of the network from moving directly along the geodesic. This explanation must be at least partially true, since the dimensionality of the representation space M generally exceeds the number of parameters in each layer/block of the network.
We were also surprised to discover that according to all metrics we investigated, network paths tended be jagged, and consist predominantly of 902 angle turns. Although it is well known that random directions in high-dimensional spaces such as the representation space M tend to be nearly always orthogonal, this does not explain why we found that path angles are straighter at initialization and become more orthogonal during training. This is surprising because the training objective ostensibly should encourage all layers to point in the same direction towards the targets. This means that steps become more orthogonal through training. This is an especially surprising result in light of recent work by Chan al. (2020) that suggests that a sequence of residual blocks can be interpreted as sequence of small gradient steps optimizing a rafe reduction objective; in this interpretation, all layers ought to be moving in the same general direction. However, this is not what we find in our models trained by backpropagation and where angles are quantified using Angular CKA (nor using other metrics - see Figure C.I. Ultimately, ours is an empirical finding which suggests that future theoretical work is needed to interpret the direction of steps taken in representation space in the context of a given representational distance metric, and to understand which directions realizable by given network architecture.
8

---

A
B
0.6
deviation
X
0.5
progress
0.4
Resnet_ _4________ 14_16
Resnet_32 _2__66
Deviation
Resnet_ Resnet_14_32
Resnet_32 _2_64
0.3
Resnet_14 14_64
Resmet_38_16
Resnet_ 14_128
Resnet 38,32
Resnet 160
Resnet 38
0.2
Resnet_ 20_16 16
Resnet_ _a_tt____122__ 128
Resnet_20 20_64
Desmet_38_160
Resnet_28 26, 16
Resnet_44, 44_16
0.1
Resnet,
Resnet 44_64
Resnet_56.
Resnet_56_ce 56_64
0.0
VGG_16
0.1
0.2
0.5
0.6
VGG_13
VGG_19
Progress
Figure 4: A) For each model, we decompose every path segment into "progress" and "deviation" components, where "progress" is the component along the geodesic from the segment start to the targets, and "deviation" is the component which orthogonal the geodesic. B) plots the average (circle) and + standard error (horizontal and vertical lines) "progress" and "deviation" over all layers of variety of model architectures. Note that the names of Resnets indicate depth followed by width, and the names of VGG models indicate depth.
As described in the introduction, we are motivated to develop general spatial analogy for information-processe where complex transformations of representations cover more "distance" than simple ones. this sense, a measure representational distance ought to reflect the function complexity of transforming X into Y. We chose extend and compare existing representational dis- tance metrics order to build directly on previous work, but the metrics we evaluated here may not be interpretable measures of function complexity. An exciting avenue for future work is thus to derive a measure of representational distance directly from a measure of function complexity. Such measure will likely violate the axioms of a distance metric. For example, the complexity of a func- tion and its inverse are in general not equal, and so it may be desirable to have d(X, Y) < d(Y, the transformation from to can be implemented by simpler function than the inverse. No- tably, many of the geometric properties studied here (shortest paths, angles, projection, etc.) can be extended the asymmetric case using the theory of Finsler rather than Riemannian manifolds (Burago et al. 2001).
The analogy neural networks as paths in a representation space brings together ideas about repre- sentational similarity and the expressivity of deep networks, marrying these techniques with intuitive and mathematically rigorous geometric concepts. Our work takes first step exploring the pos- sibilities of this new geometric framework, and we anticipate that it will spark new insights about model design, model training, and model comparison.
REPRODUCIBILITY STATEMENT
Our model specifications, training recipe, and data augmentation are all based on standard research practice: default training options for all models taken from the OpenLTH framework") Our standalone PyTorch library for metric representational distance is available at [redacted github link], and our repository to train and analyze models specific this paper is available [redacted github link].

---

REFERENCES
David GT Barrett, Ari S. Morcos, and Jakob H. Macke. Analyzing biological and artificial neural networks: challenges with opportunities for synergy? Current Opinion in Neurobiology, 55: 55 64, 2019.
Dimitri Burago, Yuri Burago, and Sergei Ivanov. Course in Metric Geometry. American Mathe- matical Society, 2001.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi. John Wright, and Yi Ma. Deep Networks from the Principle of Rate Reduction, October 2020. arXiv:2010. 14765 [cs, math, stat].
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. Journal of Machine Learning Research, 13:795-828, 2012.
Jörn Diedrichsen and Nikolaus Kriegeskorte. Representational models: common framework for understanding encoding, pattern-omponne and pppreesentationaa-smmiaariii analysis. PLoS Com- putational Biology, 134::13-3 2017.
Shimon Edelman. Representation is representation of similarities. Behavioral and Brain Sciences, 21(4):449-498, 1998.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical de- pendence with Filbeet-Schhidl norms. In S. Jain, H. U. Simon, and E. Tomita, editors, Lecture Notes in Artificial Intelligence, volume 3734, pages 63-77. Springer- Verlag, Berlin, 2005.
Hangfeng He and Weijie J. Su. A Law of Data Separation in Deep Learning, October 2022. arXiv:2210.1 17020 [cs, math, stat].
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. CVPR, 2016.
Olivier J. Hénaff, Robbe L.T. Goris, and Eero P. Simoncelli. Perceptual straightening of natural videos. Nature Neuroscience, 22(6):984-991, 2019.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural Networks, 2(5):359-366, 1989.
Frank Jäkel, Bernhard Schölkopf, and Felix A. Wichmann. Similarity, kernels, and the triangle inequality. Journal of Mathematical Psychology, 52(5):297-303, 2008
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Representations Revisited. ICMI., 36, 2019.
Nikolaus Kriegeskorte. Relating population-code representations between man, monkey, and com- putational models. Frontiers Neuroscience, 3(3):363-373, 2009.
Nikolaus Kriegeskorte. Deep Neural Networks: A New Framework for Modeling Biological Vision and Brain Information Processing, Annual Review of Vision Science, 1(1):417-446, 2015.
Nikolaus Kriegeskorte and Xue-Xin Wei. Neural tuning and representational geometry. Nature Reviews Neuroscience, 22111):7,--718, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. Deep learning. Nature, 521((553):436 444, 2015.
Nina Miolane, Nicolas Guigui, Alice Le Brigant, Johan Mathe, Benjamin Hou, Yann Thanwerdas, Stefan Heyder, Olivier Peltre, Niklas Koep, Hadi Zaatiti, Hatem Hajri, Yann Cabanes, Thomas Gerald, Paul Chauchat, Christian Shewmake, Daniel Brooks, Bernhard Kainz, Claire Donnat, Susan Holmes, and Xavier Pennec. Geomstats: A python package for riemannian geometry in machine learning. Journal of Machine Learning Research, 21(223):1 2020.
10

---

Esfandiar Nava-Yate Yazdani, Wans-Christian Hege, Timothy John Sullivan, and Christoph Tycow- icz. Geodesic analysis in kendall's ' shape space with epidemiological applications. Journal of Mathematical Imaging and Vision, 62(4):549-559 2020.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth. ICLR, pages 1-25, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
Xavier Pennec. Intrinsic statistics on Riemannian manifolds: Basic tools for geometric measure- ments. Journal of Mathematical Imaging and Vision, 25(1):227-154, 2006.
Xavier Pennec. Munifold-valueed image processing with SPD matrices. Elsevier Ltd, 2019.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. arXiv, 2016.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Soh. On the Expressive Power of Deep Fully Circulant Neural Networks. ICML, 70:2847-2854, 2017.
Blake A. Richards, Timothy P. Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, Colleen Gillon, Danijar Hafner, Adam Kepecs, Nikolaus Kriegeskorte, Peter Latham, Grace W. Lind- say, Kenneth D. Miller, Richard Naud, Christopher C. Pack, Panayiota Poirazi, Pieter Roelfsema, João Sacramento, Andrew Saxe, Benjamin Scellier, Anna C. Schapiro, Walter Senn, Greg Wayne, Daniel Yamins, Friedemann Zenke, Joel Zylberberg, Denis Therien, and Konrad P. Kording. A deep learning framework for neuroscience. Nature Neuroscience, 22(11): 17611-1777 2019.
Antonio M Rodriguez and Richard Granger. The differential geometry of perceptual similarity. arXiv preprint arXiv: 1708. 1708.0013, 2017.
David Rolnick and Max Tegmark. The power of deeper networks for expressing natural functions. ICLR, pages 14, 2017.
David E Rumelhart, James L McClelland, PDP Research Group, al. Parallel distributed process- ing, volume 1. IEEE New York, 1988.
Mahdiyar Shahbazi, Ali Shirali, Hamid Aghajan, and Hamed Nili. Using distance on the Rieman- nian manifold to compare representations in brain and in models. Neurolmage, 239())))): 1 118211, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint 1rXiv:1409.1556 2014.
O. Tange. Gnu parallel the command-line power tool. ;login: The USENIX Magazine, 36(1): )): 42-47, Feb 2011.
Pauli Virtanen, Ralf Gommers, Travis F. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nel- son, Eric Jones, Robert Kern, Eric Larson, C J Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Natur Methods, 17:261-272, 2020.

---

Alex H Williams, Erin Kunz, Simon Kornblith, and Scott Linderman. Generalized shape metrics neural representations. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 4738- 4750. Curran Associates, Inc., 2021.
Adam X. Yang. Maxime Robeyns, Edward Milsom, Nandi Schoots, and Laurence Aitchison. A theory of representation learning in deep neural networks gives deep generalisation of kernel methods, July 2022. arXiv:2108.13097 [cs, stat].
12

---

A
DETAILED INFORMATION ON METRICS
Distance Metric
Manfiold M
Distance dm (X, Y)
Angular CKA
Gramm
arccos
(X,Y)
V(X.X):
Angular Shape!
Rmxp
ming arccos
(X.YR))
(Xx,X))
Euclidean Shape?
Rmxp
minR m7 Σ 7
|X:; Y,R
Affine Invariant Riemannian
Symth or Sym
E, log(d))²
where de is the ith eigenvalue of X YX
Table A.1: Summary of representational distance metrics. Gramm is the set of m m centered Gram matrices. Sym is the set of m x m symmetric positive definite matrices. Distm is the set of m m pairwise-distance matrices. p is a hyperparameter used to set the dimensionality of points in M. || denotes the Frobenius norm, and (X, Y)F denotes the Frobenius inner-product. R is p p orthonormal matrix, found by solving the orthogonal Procrustes problem to maximally align X and Y. Metrics marked with "1" are due to Williams et al. (022)) The Affine Invariant Riemannian metric due to oPPennec (2006) 2019); Shahbazi et al. (2021). For further details on each metric and its implied geometry, see Appendix
As summarized in equation ( all distance metrics between neural representations operate in two stages: first, layer's activity X X is transformed into a point in some canonical space M through an embedding function I, and second distances are measured in that shared space. In the following subsections we give details for each metric in Table A.1
We say a metric is scale-invariant if d(X, ax) = O for all scalars a * O. A metric is shift- invariant if dXX, X + ++bb)) 0 for any n dimensiooa vector b. metric rotation-invarian d(X, X = 0 for any n n orthonormal matrix R. A metric is alline-invarian if d(X, X.A) = for any full-rank
n matrix A.
For all geometric calculations, we drew subset of m 1000 items randomly from the test set, and used the same subset throughout. We verified that m 1000 was sufficient reliably esti- mate the geometric quantities of interest in both the AngularCKA and Angular Shape metrics by randomly resampling m = 1000 points from the test set multiple times and inspecting the variance of computed quantities (Figure embedding.
Targets (class labels) are converted to 1-hot vectors before
Our Python implementation of various quantities from Riemannian geometry draws much inspira- tion from the geomstats Python package (Miolane et al. 2020). Our analyses in the main paper were done using
Angular CKA with m 1000 and the linear kernel k(x;, xj) = x] xxx
The Angular Shape metric with 1000, p 100, a
The Gram-matrix version of the Alfine-Invaarin Riemannian metric with m 1000, 0.1, and squared exponential kernel with length scale set to the median Euclidean distance of X.
We begin with an introduction Angular CKA, where we also review some key terms from Rie- mannian geometry.
A.1 ANGULAR CKA
Angular CKA was introduced by Williams et al. (2221 (eq (60) in their supplement). is defined as the arccosine centered kernel alignment (CKA), which is itself derived from the Schmit
independence criterion (HSIC) (Gretton al. 2005 Cortes et al. 2012 Kornblith al. 2019
13

---

Because HSIC and CKA measure statistical dependence, distance measured by Angular CKA is high when the rows of X and Y are statistically independent, and low when they are highly correlated.
Angular CKA is equivalent to arc-length distance the spherical manifold consisting of centered and normalized m X m Gram matrices. Let Gx denote the Gram matrix of X, i.c. Gxij is given by the inner-product between the ith and jth computed using a kernel; following previous work (Kornblith et al.) 2019 Nguyen et al.] [2021). results in this paper use Linear CKA, i.e. use Gx -XX but our python package supports the use of other kernels. The normalized and centered Gram matrix is given by
of X. Optionally, this inner-product may be
HGxH
Gx
||
where H = Im -1m---11 is the im centering matrix, and |1-||P is the Frobenius norm of matrix. Note that both G and G are symmetric and positive definite matrices. The Riemannian manifold M for Angular CKA consists of all such centered, normalized, symmetrie positive definite matrices; it sphere in sense that (Gx,Cr Gx), = 1, where (A, B) = Tr(AT is the Frobenius inner- product.
The embedding function for Angular CKA is f(X) Gx. Distance according to Angular CKA is
equal to are length on the sphere consisting of centered and normalized Gram matrices:
d(X, Y) ((()))),
= (G,, Gx)
= arccos ((xx,Gx)
(A.1)
Because Angular CKA is an are length, its geodesics lie along great circles G-space. We therefore can compute points along the geodesic in closed-form using the SLERR formula
geodesic(Gxx Gx sin((1- - () Gx
sin((12)
GY.
restated)
sin(
sin(1)
where E [0, L] is the fraction of distance along the geodesic from X to Y, and Ω = dgc(Gx. GY).
The tangent space for Angular CKA the space of all symmetric m x m matrices, and the inner- product in the tangent space is simply the Frobenius inner- -roduct. The logarithmie map computes tangent vectors from base point that point towards another point. In the case of Angular CKA, the logarithmic map from Gx to Gy is a tangent vector (symmetric matrix) at Gx given by
logax (Gx)
W Warccos
((xx,ax),)
restated)
where Wi is the unit tangent vector at Gx pointing towards GY, given by
Gy Gx (xx:Gx)
W
G - Gx (G Gy
||F
The exponential map the inverse of the logarithmic map - it function that "extrapolates" tangent vector W from point to give another point the manifold. In the case of Angular CKA, the exponential map given by
expox (W) cos (I) Gx sine ([))) W
(A.2)
where sinc(x) sin(x)
For all metrics, we compute angles between triplets of points by computing the inner-product of their tangent vectors. In the case of Angular CKA in particular, let Wxy denote the tangent vector pointing from X to Y, i.e. the result of logax (Gx). Then,
(Wyx (yz)
0( 0(xx, Gy, Gz) arccos
restated)
[ WYZlIF/
the angle of the XYZ triangle.
https://en. wi wttp://wwwwpiiiaa..o..iiiiiii.i. /tg.
14

---

A.1.1 INVARIANCES OF ANGULAR CKA
The invariances of Angular CKA depend on the kernel used to compute the Gram matrix. In the simplest case of Linear CKA, the Gram matrix is simply Gx XX The resulting metric is
thift-nnvarian due to centering the Gram matrix.
scale-invariant due to normalizing the Gram matrix.
rotation-invariant since (XR)(XR) XRR x XX for any orthonormal R.
However, Angular CKA with is not invariant to arbitrary affine transformations - a feature it inherits from CKA and has been argued to be an important feature of CKA (Kombllth et al.] [2019). Note that when using nonlinear kernel to compute the Gram matrix, the resulting metric may lose these invariances. However, Angular CKA with nonlinear kernel may still be shift-, scale-, and rotation- invariant the kernel itself has those invariances. For example the squared exponential kernel
G. kxx,,x x,) = exp (I|x; x)ll//T²)
(A.3)
is naturally shift- and ootation-invariant, and it can me made further scale-invariant by setting the length scale T automatically based on the scale of the data.
A.2 SHAPE METRICS
Williams al. 2021) proposed using generalization of Procrustes distance and Kendall's shape space to measure metric distance between neural representations. Shape-space space and Procrustes dis- tance are well-studied case of Riemannian manifold between point clouds (Nava- Yazdani al. 2020). WWlliams et al. (202T) consider two different shape metrics one angular shape metric and one Euclidean shape metric. The key idea behind both of these metries is follows:
n matrices of neural data are first transformed into a common m X p space, and interpreted as a point cloud consisting of p dimensional points. Then, any two point clouds are scaled and rotated so that they maximally align with each other. The final distance is then computed as some measure of discrep- point clouds. The behavior of these shape metrics is tuned
ancy between these maximally- using two hyperparameters: the dimensionality p. and partial whitening parameter a.
The role of the embedding function for shape metrics is to convert n- dimeensiona neural data into a canonical zero-mean p- dimeensionnl space (i.e. MI = Rmxp is the space of all m x p matrices whose column means are all zero). |Willias et al. (20TT) also include partial whitening stage as part of the embedding. This space of p zero-mean (and sometimes scaled) matrices is called the pre shape space Yazdai et al.) 2020). In the case where n < p, this conversion from n to p dimensions is done by simply padding X with p n columns of all zeros. In the case where p n, we reduce the dimensionality of X by keeping only the top p principal components. Formally, let 17 Σm X; be the matrix of neural data with its mean subtracted, then
X X
whiten X, O),a)
S if n
X = f(X)
(A.4)
whiten XU ppy a)
where U.p stands for the first p principal components of X, unit column vectors, and m x (p n) matrix of all zeros. The partial whitening function begins by computing the eigen- decomposition of its input, m I VEVT (here, V is a x p orthonormal matrix containing the top principal components of X, and Σ is diagonal matrix of variances). Then, the partial whitening stage
whiten X, a) XV (all, + +1 (1- VT.
Note that when a = 0, this equivalent to ZCA whitening, and when a it leaves X unchanged. All shape metric results we report are with p 100 and O. We use 0 because this entails further invariances, making metrics more interpretable across disparate layer shapes and types (see section A.2.1 below for details shape metric invariances).
Both the angular and Euclidean shape metrics require aligning by rotating the embedded points by minimizing | - RlIF where R is a x orthonormal matrix. This is known as the orthogonal Procrustes problem, and its solution is given by
R
V
15

---

where UEVT =X is singular value decomposition of X Y. The generalized shape metrics introduced by Williams et al. (202I) include further restrictions on R. such as considering rotations across channel but not spatial dimensions of convolutional layers, but omit these restrictions in our work.
In the case of angular shape metries, distance is defined
X (XYYR),
dm X, Y) = arccos
(A.5)
|
In the of Euclidean shape metries, distance is defined
dm X. Y)
X Y,R
(A.6)
772
We compute geodesics in shape space after finding R to align Y X. Then, the geodesic from X
to Y R in the angular case is given by the SLERP formula in (3
geodesic(X, Y,t) sin((1 - 1)52) X sin [12
YR.
(A.7)
sin(
sin
where (X, Y) is the angular shape distance. Note that this means that geodesic [X Y, results in point that equivalent but not identical Y.
Tangent vectors for Euclidean shape metrics can be any m matrix whose column- -wiee mean zero. In the case of angular shape metrics, the tangent space further restricted to the tangent space of the hypersphere of unit-Frobenius-orm matrices (i.e. a tangent vector W X must satisfy (x,
0 in the angular case). The tangent space further divided into so-called
horizontal and vertical subspaces, where the vertical subspace captures changes to X that leave distance invariant, i.c. rotations that are removed by alignment by and the horizontal subspace captures changes that affect the metric (Nava- Yazdani et al. (020). The vertical component of tangent vector W at point X is given by vert (W) X.A, where A E RRxp is the solution to the following Sylvester equation:
A
the solve colve.de sylvester function from Seipy to compute this (Virtanen et al. [2020). The horizontal component of a tangent vector is given by simply subtracting the vertical part of W:
Following the example of Miolane al. 2020),
(vert& (W), W)F verty (W)ll:
h (W)= W vert W)
To compute the angle between any triplet of representations, vectors, as in (5). but using only the horizontal part each tangent vector. As in Angular CKA, we compute horizontal tangent vectors from to Y using the logarithmie map, which in the case of shape metrics is given by
the inner-product of tangent
horizontal log (Y) YR-X
(A.8)
in the Euclidean case,
{ (X
horizontal logx Warccor
(A.9)
|
where W is the unit tangent vector at X pointing towards Y, given by
ÝR X YR
W
ÝR X X ||F
Nava- Yazdani al. 2020 As in (A.5 and A.6)
R the rotation matrix that optimally aligns Y
to X.
16

---

A.2.1 INVARIANCES OF SHAPE METRICS
The invariances of the shape metrics depend on a variety of hyperparameter settings.
All shape metries are shift-invariant because the embedding function X ((X) subtracts the mean.
All shape metrics are notation-invarianl because of the Procrustes alignment procedure, and because rotation does not affect the principal component projection nor the zero- padding step of (A.4).
The angular shape metric is scale-invariant because A.5 divides by the norms of and Y.
The Fuclidean shape metric not ocale-invariant in general, but it for the special case of
Qx 0, since scale is removed by whitening.
Neither angular nor Euclidean shape metrics is affine-invariant in general, but both can since full-rank affine transforms
become affine-invariant for the special case of are removed by whitening as long as n S p. However, in the n > p case, an affine transformation may amplify or suppress the principal components of the data, and as a result it can affect the embedding stage (A.4).
A.3 AFFINE INVARIANT RIEMANNIAN METRIC
The Affine Invariant Riemannian (AIR) metric is a metric between symmetric positive definite (SPD) matrices, originally derived for use in image processing (Pennec) (0066) 2019). and recently it was proposed to use it as a metric between neural representations by first converting neural data into a SPD matrix (Shahbazi et al.) 2021). The embedding function can therefore be any function that maps m X n matrices in X int M = Sym for some k. Shahbazi et al. 2021) considered two possibilities for the embedding stage: either using the m m Gram matrix [[X] Gx, or using the n X n data covariance matrix f(X) = cov(X) = m _ HX. These correspond to comple- mentary perspectives on the nature of neural representation, analogous to the difference between representational similarity analysis and pattern component analysis (Diedrichsen and Kriegeskorte 2017].
The challenge when using the m Gram matrix approach is that, without further regularization, a m x m Gram matrix has rank n when n < m, which implies that it cannot be SPD (and the metric considers all rank-deficient matrices to be infinitely far away). To address this, toolbox implements the AIR metric between neural representations with additional regularization options. In the Gram matrix case, we regularize in two ways: first, we compute the Gram matrix using a kernel that implicitly has an infinite feature space (so that m is much less than the number of features). This alleviates the rank-deficiency problem in cases where < m but rows are unique. However, when rows of X contain duplicates (notably, this is true for the target labels), Gk is still rank-deficied To address this, we include a second regularization stage where we add a small diagonal ridge with magnitude e. The full embedding function in the Gram matrix case is given by
f(X) Gx + eIm
(A.10)
where the ijth element of Gx given by k(X, x)). For our results in the paper, we use e 0.05
and a squared exponential kernel for k as in A.3 setting the length scale 7 automatically to the median pairwise Euclidean distance between rows of X.
The main challenge when using the covariance matrix approach is that it cannot be directly applied to compare layers with different numbers of neurons n. To address this, we first convert from m x n matrices of neural data into common metries as in (A.4), but without the whitening stage. We can then embed all layers into common space of p issues when n < m (c.g. for the one-hot embedding of targets for which n 10), and so we again regularize by adding diagonal ridge to the resulting covariance matrices. The full embedding function in the covariance matrix case is given by
p size, using the same method as we use for the shape
covariance matrices. As in the Gram matrix case, we again run into refk-cefic
ccvv ([X, 0)) + elIp in S p if > p
f(X)
(A.11)
ccov (XU.p) + ellp
17

---

(compare with A.44
Let P f(x) and Q = ( be SPD matrices (we are using P and Q instead of X and Y to use consistent notation with Pennec (2019). The AIR metric distance is defined as
d(X, Y) = dx (P, Q) = Σ log(d,)²
(A.12)
where d; is the ith eigenvalue of P QP value decomposition can be written P VEV where Σ is a diagonal matrix and V is orthonor-
Pennec 2006 2019). Since P is SPD, its singular
mal. Following Pennec (2019), we element-wise square root, exp, and log operations the singular values to define the matrix square root, matrix exponential, and matrix logarithm:
pow (P, k) = Vpow Σ, k) V
exp (P) = Vexp ( V
log (P) Vlog ()) V
where the operations the left hand side are matrix power, exponential, and log, whereas pow, exp, and log operations are performed element-wise on the diagonal of E. P equivalent to pow (P, A)).
The geodesics from P to Q is given by
geodesic Q. = P P PP-QPP- ( )) p
(A.13)
(combining equations (3.12) and (3.13) in Pennec (2019)).
Tangent vectors in this space are symmetric matrices, and the logarithmic map is given by
logp QQ
po (p- QP :) p
(A.14)
(see equation (3.12) in Pennec 2019 by
The exponential map of the tangent vector W at P given
expp
(W)
P
(A.15)
(see equation (3.13) in Pennec (2019)). One can easily verify that expp (logp Q
As before, we compute angles between triplets of representations X, Y, Z by computing the inner product of the logy (X) and logy (Z) tangent vectors, but unlike the previous metries the definition of inner products for the AIR metric is not simply the Frobenius inner product. For the AIR metric, the inner product of tangent vectors W and at P defined as
(W, V)p (P- (P-'W, P VV) Tr (W - -P-'V
(A.16)
A.3.1 INVARIANCES OF AFFINE INVARIANT RIEMANNIAN METRIC
We will treat the Gram matrix (0.10) and the covariance matrix
A.11 cases separately. In the Gram
matrix case,
The AIR metric is shift-invariant, scale-invariant, and/or rotation-invariant if and only if the kernel used to compute Gx has the corresponding invariance. Because we use squared-exponential kernel with a length scale that adapts to the data scale, we have all three invariances.
The AIR metric is, despite its name, not affine-invariant in the sense we are interested in, since affine transformations of X will in general affect Gx through the nonlinear kernel (e.g. the squared exponential kernel with an isotropic length scale is sensitive to non- isotropic scaling of X).
In the covariance matrix case,
The AIR metric is shift-invariant because covariance subtracts the mean.
18

---

The AIR metric is scale- and rotation-invariant due to the eponymous Saffine-invariannn of the metric itself (Pennec 2006 2019).
As in the case of shape metrics discussed above, the AIR metric may or may not be variant to arbitrary affine transformations of X due to the restriction to the top p principal components the embedding stage. Only the case where n > p and A is a matrix such that XA changes the subspace of the top p principal components, then the resulting metric is not invariant to A.
B MODELS AND TRAINING DETAILS
We trained a collection of convolutional networks including both residual networks (He et al, 2016) and VGG (Simonyan and Zisserman, 2014) CIFAR-10 (Krizhevsky 2009) using Py' PyTorch
Paszke et al.) 2019). managing compute jobs using GNU Parallel (Tange 2011). We used the
OpenLTH framework for training and checkpointing models, using the default hyperparameters' for
each model.
Following Nguyen et al. (221), ). trained Residual networks of varying widths and depths. The "width" refers to the number of feature channels per convolutional layer, and took on values of {16,32 32, 64, 128, 160) (corresponding to the base size of 16 multiplied by {1x, (1x,2x, 4x, 8x, 10x)). The "depth" controls the number of residual blocks, according to the formula #blocks = (depth 2)/2, since each block contains two convolutional layers, and there are two addi- tional preprocesiingprroeccioo layers before/after the blocks. We trained models of depths {14, 20, 26, 38, 44, 56, 110). The VGG architecture supports "depths" of {11, 13, 16, 18), all at the same width. The test accuracy of all models is shown in Table B.2 We analyzed the represen- tational distances and geometry of subset these, focusing on depths 14 and 38 (for all widths), and widths 16 and 64 (for all depths).
All models were trained using the default training hyperparameters of OpenLTH Specifically, all models were trained by stochastic gradient descent for 160 epochs with batch size of 128 (390.6 batches per epoch of 50k training items), an initial learning rate of 0.1 reducing to 0.01 and 0.001 after 80 and 120 epochs respectively, momentum of 0.9, and weight decay of 0.0001. During train- ing, images were augmented by random horizontal flips and random +4 pixel left/right or up/down shifts (padding with zeros).
For the training analysis in Figure we trained second Resnet- 14 with default width (16), and analyzed the representational distance and geometry once every batches for the epoch, since the vast majority of the changes to the network performance and geometry occur in the first epoch.
19

---

Architecture (depth/width)
CIFAR- 10 test accuracy (%)
Resnet 14/16
90.78
Resnet 14/32
92.67
Resnet 14/64
93.88
Resnet 14/128
94.57
Resnet 14/160
94.91
Resnet 20/16
91.28
Resnet 20/32
93.76
Resnet 20/64
94.84
Resnet 20/128
95.13
Resnet 20/160
95.22
Resnet 26/16
91.92
Resnet 26/32
93.99
Resnet 26/64
94.69
Resnet 26/128
95.24
Resnet 26/160
95.07
Resnet 32/16
93.04
Resnet 32/32
94.18
Resnet 32/64
94.83
Resnet 32/128
95,47
Resnet 32/160
94.79
Resnet 38/16
92.28
Resnet 38/32
94.07
Resnet 38/64
95.05
Resnet 38/128
94.71
Resnet 38/160
94.99
Resnet 44/16
93.06
Resnet 44/32
94.14
Resnet 44/64
95.34
Resnet 44/128
94.85
Resnet 44/160
94.91
Resnet 56/16
93.06
Resnet 56/32
94.46
Resnet 56/64
94.82
Resnet 56/128
95.22
Resnet 56/160
95.32
Resnet 110/16
93.37
Resnet 10/32
94.91
Resnet 110/64
94.87
Resnet 110/128
95.15
Resnet 10/160
95.02
VGG 11
91.93
VGG 13
93.37
VGG 16
93,45
VGG 19
93.33
Table B.2: Model architectures and performance.
20

---

C
ADDITIONAL FIGURES
Angular Shape Metric
Euclidean Shape Metric
Affine-Invariant Riemannian Metric
6.1
12
0.30
16
0.2
08
6.3
PC2
PC2
PC1
PC1
PC1
target
n/2
layer (colar coded)
layer (color coded)
layer (color coded)
Figure C.1: Comparison of paths taken by the same Resnet- 14 model in Figure but computed with the Angular Shape Metric and Euclidean Shape Metric of Williams et al. (202I), and the Affine-Invariant Riemannian Metric of [Sahbaaii al. (202T). A) Pairwise distance between layers according to the Angular Shape Metric, computed with p 100 and to Figure E B) 2D embedding of the network's 's path using MDS, as Figure IT. C) Internal angles and target angles, as in Figure D]. D-F) Same as A-C but for the Euclidean Shape Metric with 100 and a = 0.O. G-I) Same as A-C but for the Affine-Invariant Riemannian Metric using a squared exponential kernel and 0.1.
0.0, equivalent
Notably, all metrics agree with the finding that "internal angles" are close to orthogonal throughout the network, and that "target angles" close orthogonal for early layers, with the later layers being the only ones that appreciably point in the direction of the targets. Both the Angular Shape Metric and the Affine-Invariant Riemannian metric measure large distances in the last few layers as the last convolutional layer is projected to logits and then passed through a softmax function. For both of these metrics, the first principal component deseribing the network path primarily describes just these last few layers (Figure C. A.H.G.HH.. This suggests that, while these metrics give sensible results for distances between hidden layers, they may ill-suited for measuring distances between internal representations and targets.
21

---

examples
examples
2.0
2.0
2.0
3nges
1.5
1.5
1.5
1.0
1.0
st. dev.
range
50
150
50
100
150
50
100
1.5
1.5
1.5
1.0
1.0-
1.0
0.5
0.5
1000
3000
1000
2000
3000
2000
3000
0.5
0.0
0.0
0.0
100
200
300
100
200
300
100
200
300
1.5
1.5
1.0
1.0-
1.0
0.5
0.0
2000
4000
2000
4000
0
2000
4000
1.5
0.0
0.0
0.0
500
1.5
3.5
1.0
1.0-
1.0
projectd
0.5
0.5
0.5
100
50
100
50
100
1.0
0.5
0.5
0.5
0.0
0.0
0.0
1000
2000
3000
1000
2000
3000
1000
3000 rank of moan
2000
rank of mcan
rank mean
Figure C.2: 10-fold cross validation for quantities computed using the Angular CKA metric on ran- dom subsets CIFAR-10 10 test examples. Rows correspond to different types computed quantity, and columns correspond to the number of examples used. Quantities are ordered by the rank of their mean over CV folds (x-axis). Each plot shows the mean (line), standard deviation (blue region), and minimum to maximum range (grey region) per quantity over CV folds (y-axis).
22

---

50 examples
250 examples
1000 examples
2.5
2.5
2.5
2.0
2.0
2.0
1.5
1.5
1.5
mean
0.5
St. dev.
0.0
0.0
0.0
150
1.5
1.5
1.5
1.0
1.0
1.0
0.5
0.5
0.5
0.0
1000
2000
3000
1000
2000
3000
1000
2000
3000
painwise
0.0
0.0
0.0
300
100
1.5
1.5
1.5
pairuize path accoeiic painiise and
1.0
1.0
1.0
0.5
0.5
0.5
2000
4000
2000
4000
2000
4000
1.5
1.5
0.0
1.5
1.5
1.5
1.0
1.0
1.0
projected
0.5
0.5
0.5
0.0
0.0
0.0
50
100
50
100
50
100
iangorediineer products
1.5
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
-.0
1000
mean
Figure C.3: 10-fold cross validation for quantities computed using the Angular Shape metric on random subsets CIFAR-10 test examples. Rows correspond to different types computed quan- tity, and columns correspond to the number of examples used. Quantities are ordered by the rank of their mean over CV folds (x-axis). Each plot shows the mean (line), standard deviation (blue region), and minimum maximum range (grey region) per quantity over CV folds (y-axis).
23


Reviews:
[{"review_id": "wP32gL6_tZ", "reviewer": "Reviewer_GUMC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "[Summary]\n\nThe submission uses different measures of dissimilarity to characterize the sequence of a neural network's hidden representations at a point in training. They observe that, as the layers progress toward the output layer, the representation gets further from the input representation and closer to the output representation. They also compare th \"path\" made from this sequence of representations to the shortest path between the input and output representations.", "summary_of_the_review": "[Decision]\n\nIt is hard to evaluate this paper as there is no specified end goal (designing a certain new algorithm or architecture or improving deep learning theory) for this analysis and I cannot determine to what extent the experiment design is appropriate without an ultimate goal. Nevertheless, as long as a study shows clear and reproducible patterns with potential for future work, it is valuable to me. Right now I'm putting a low score since (1) the experiments are on one task and (2) there is no discussion on the implications of the chosen dissimilarity measures but I'll be happy to raise my score if these limitations are addressed unless other reviewers reveal other issues.", "strength_and_weaknesses": "[Strengths/Weaknesses]\n\nThe most interesting finding to me was that hidden representations in deeper neural networks are closer to the straight line between the input and output representation. While I'm not certain at this point, it is likely that this insight will lead to better generalization guarantees for deeper networks.\n\nThe main weakness is that all the experiments are conducted on Cifar-10. I suggest repeating the experiments on at least one more dataset. This dataset has to be sufficiently complex (this excludes MNIST and Fashion MNIST) and sufficiently different from Cifar-10 (this excludes Cifar-100). Some suggestions are STL-10, Caltech, and Mini ImageNet.\n\nAnother thing missing in this paper is discussions on the implications of the dissimilarity measures. What does each dissimilarity measure capture and what changes is it invariant to? A nice example is the discussion on CKA by [1] that shows CKA captures alignment between the top singular vectors of the two representations.\n\nI would also like to see what the purpose of the \"projection\" experiments (Like Fig 1 H-I-J) is and what conclusions one can get from its results.\n\nThe discussion section says that the paths are more straight (the angles are closer to pi) at initialization. This is not what I would conclude from the results in Figure 2. Only one dissimilarity measure shows this pattern and even in that subplot there is a lot of variability in the angles. With the other two measures all the angles are close to pi/2.\n\n[Minor comments]\n\nFig 3 is too crowded and some curves are basically invisible. I suggest breaking it down into multiple figures and moving some to the appendix.\n\nThe caption for Fig 4 says that the marker size indicates width. Does this mean that the network width keeps growing all through the legend from top to bottom?\n\n[1] Kornblith, Simon, et al. \"Similarity of neural network representations revisited.\" International Conference on Machine Learning, 2019.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "fY6ECMqXMCB", "reviewer": "Reviewer_dSZX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose thinking about neural network layers as steps in a path that describes how a representation changes steadily towards the target output.  In order to analyze this, they use existing representational distance metrics between the outputs of different layers of the same network.  The authors introduce new visualization methods to understand how a representation changes as it is passed through the network; they propose looking at how the path differs from an shortest-path interpolation of the input and output, projecting onto geodesics, and considering the straightness of the angle that occurs when traveling through two layers.   ", "summary_of_the_review": "Overall, I believe this paper is an interesting step towards the direction of understanding how layers interact with each other to compose a complex neural network.  The authors propose such a spatial path analogy, which they set about visualizing layer by layer.  They also propose extra visualizations to understand these paths; interpolating to generate the shortest path, projection, and showcasing the angles between layers.  However, the results that the authors achieved on these proposed visualizations fly against their hypotheses for expected behavior (e.g. the paths do not follow the shortest interpolated paths, nor do the angles suggest a straight path).  It is therefore highly unclear if these visualizations are correct or useful to use, if there are inherent limitations in the representational metrics utilized that inhibit their ability to provide a clear picture into neural network behavior, or some other reason.  Without aligning our hypothesized understanding of how the neural network behaves with the actual behavior according to these visualizations in an interpretable and understandable manner, I believe it would be difficult for the general community to adopt these proposed methods for analysis.  I therefore suggest the authors think deeper about why their results differ greatly from their hypotheses regarding their constructed visualizations, what the current discovered implications are, and if there are other natural metrics that might achieve the expected behavior, before being considered for acceptance.", "strength_and_weaknesses": "A strength of this paper is its proposed geometric visualizations for understanding the internal behavior of neural networks, which is rather novel.  Comparing the paths taken by a wide versus a deep model was also an interesting insight.\n\nA main weakness of this paper is in its inability to explain or hypothesize much of the results it found through its analysis.  Whereas the hypothesis is that training a neural network would cause the representational path to be straight, it happens that they turn more orthogonal instead.  The paths taken by the network also greatly differ from the shortest interpolated path, across all representational metrics.   Such mysteries are not expected or understood, and it is unclear if this is because the metrics are unable to provide a clear picture on the behavior (as the authors mention in the discussion) or if there is something more fundamental about the behavior of a neural network itself.  There is therefore no clean takeaway regarding these proposed metrics - without understanding why the behavior is the way it is (or why it is different from what is expected from these metrics), one might be unable to derive much insights from them and therefore hesitate to use them.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JOZgLNfBL1", "reviewer": "Reviewer_m1JB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors leverage recent work (Williams et al., 2021; Shahbazi et al., 2021) that defines metric spaces over neural representations to analyze multi-layer networks as paths through this representational shape space. These metrics were recently developed, and are similar to CKA (Kornblith et al., 2019). The main novelty in this paper comes in section 2.3 where the authors describe how to interpolate along geodesics within this space of representations, measure the angles at two incident geodesic paths, and to perform projection of a point onto a geodesic path. They apply these novel interpretive tools to understand how computations progress in ResNet and VGG architectures. They study how these paths change over training and across \"wide\" versus \"deep\" networks. The most interesting result is that the geodesic angles are close to orthogonal at each layer.", "summary_of_the_review": "I really like the ideas in this paper and I think it could grow into an important form of analysis for the field. Currently I have recommended \"borderline reject\" due to some of the missing details in the Appendix, and because I am only moderately enthusiastic about the experimental results in this paper (even though I think they are promising). Some of these points are addressable within the rebuttal period and I am very much open to raising my score.", "strength_and_weaknesses": "The main strength of this paper is to introduce conceptual advances, and the main weaknesses are the empirical demonstrations. While some of the experiments are interesting, I just wish there was \"more\" in this section of the paper. I also have some concerns about technical details missing from the Appendix (see next section on \"clarity\").\n\nIt is nice that the authors use three different metrics to illustrate their points. I would also be interested in seeing the Euclidean shape metric (not just the angular shape metric) as was introduced in Williams et al. 2021.\n\nI would have loved to see more results on \"toy\" tasks and models to help build intuition -- e.g. do fully connected networks trained on a simple task like MNIST show similar results to the very deep networks?\n\nI have some reservations about the practice of flattening convolutional layers as the authors describe in footnote 2. Could the authors comment on the alternative procedure in section 2.4 of Williams et al. (2021) for conv layers? I think that paper makes a reasonably compelling case against flattening the conv layers.\n\nI wish the authors were able to dig further into some of their interesting observations. For example, what do the orthogonal jumps at each layer really mean? Can further intuition be developed through targeted experiments?\n\nFinally &mdash; and this is not so much a weakness as a suggestion for future work &mdash; I would ***love*** to see this analysis applied to neural ODEs ([Chen et al. 2018](https://arxiv.org/abs/1806.07366)). The idea is that these will be truly continuous paths through representation space rather than jagged, discrete jumps. I think this may really help shed light on the approximate orthogonal angles observed by the authors.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8FZA4jOG2M9", "reviewer": "Reviewer_kK2H", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper looks at the paths (as defined by their representation spaces) taken by models through their training. There is prior work on measuring the distance between two embedding spaces, and the authors expand on this by computing other properties (geodesics, angles, and projections), and show empirical results of comparing ResNet (wide vs deep) and VGG architectures.", "summary_of_the_review": "The paper presents a novel framing of the networks as moving through a \"space\", but I'm not sure that's a significant enough contribution.  I also found it generally hard to read and understand, especially the figures, and felt like conclusions were drawn from the figures without enough support.  All in all, I would say reject. That being said, I'm coming from a visualization background, rather than a theory background, so there is a chance that there are other aspects that I couldn't fully evaluate.", "strength_and_weaknesses": "Strengths:\n- At a high level, I appreciate that the authors are approaching this from a theory standpoint-- the field of ML is really missing this.\n- \"representations are not merely separated by some distance, but also by some direction\". This is true-- I think this is an interesting reframing and I haven't heard it said like this before.\n\nWeaknesses:\n- In general, this paper is interesting, but I'm missing the motivation-- why is it important to trace the paths of the network as it trains? The only answer I could find was in the discussion where the authors discuss the fact that the path is not straight, and maybe we could normalize better to make training more efficient. But the path not being straight isn't that surprising to me, given that gradient descent has no such guarantees. \n- The figures are hard to read, and I'm not that convinced that you can draw conclusions from them. For example, \"We observe that deep networks have significantly less deviation and slightly less progress per step compared with shallow networks\": it would be great to compute some actual correlation coefficients on this.\n- Similarly, \"wide-model paths and deep-model paths all follow roughly the same trajectory in representational space\" and \"see salient differences between the VGG and Resnet architectures\": there should be numbers to back these up too.\n- \"This grants us the ability to interpret neural network layers not as discrete jumps from one representation to another, but as a piece-wise smooth curve from inputs to network outputs.\". Is this a better mental model? I would think that former might be more accurate-- I am not really sure what the interpolation between two layers really means, given that it's computationally a single step.\n\nClarity nits:\n- nit: Figure 4 is hard to read. Suggestions: make the X axis range smaller, color code resnet_n to be similar colors (how VGG is shades of yellow).\n- What is the data on the x axis of E/F/G in Figure 1 (the small squares and dots)?\n- in Figure 1, what does \"the hypothetical representation at the midpoint of the geodesic between inputs and targets\" mean precisely? ie, how did you calculate the pixels in the blue square?\n- Figure 3 has too many lines to read. Would recommend separating them out or simplifying.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject