Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper looks at the paths (as defined by their representation spaces) taken by models through their training. There is prior work on measuring the distance between two embedding spaces, and the authors expand on this by computing other properties (geodesics, angles, and projections), and show empirical results of comparing ResNet (wide vs deep) and VGG architectures.
strength_and_weaknesses: Strengths:
- At a high level, I appreciate that the authors are approaching this from a theory standpoint-- the field of ML is really missing this.
- "representations are not merely separated by some distance, but also by some direction". This is true-- I think this is an interesting reframing and I haven't heard it said like this before.

Weaknesses:
- In general, this paper is interesting, but I'm missing the motivation-- why is it important to trace the paths of the network as it trains? The only answer I could find was in the discussion where the authors discuss the fact that the path is not straight, and maybe we could normalize better to make training more efficient. But the path not being straight isn't that surprising to me, given that gradient descent has no such guarantees. 
- The figures are hard to read, and I'm not that convinced that you can draw conclusions from them. For example, "We observe that deep networks have significantly less deviation and slightly less progress per step compared with shallow networks": it would be great to compute some actual correlation coefficients on this.
- Similarly, "wide-model paths and deep-model paths all follow roughly the same trajectory in representational space" and "see salient differences between the VGG and Resnet architectures": there should be numbers to back these up too.
- "This grants us the ability to interpret neural network layers not as discrete jumps from one representation to another, but as a piece-wise smooth curve from inputs to network outputs.". Is this a better mental model? I would think that former might be more accurate-- I am not really sure what the interpolation between two layers really means, given that it's computationally a single step.

Clarity nits:
- nit: Figure 4 is hard to read. Suggestions: make the X axis range smaller, color code resnet_n to be similar colors (how VGG is shades of yellow).
- What is the data on the x axis of E/F/G in Figure 1 (the small squares and dots)?
- in Figure 1, what does "the hypothetical representation at the midpoint of the geodesic between inputs and targets" mean precisely? ie, how did you calculate the pixels in the blue square?
- Figure 3 has too many lines to read. Would recommend separating them out or simplifying.
clarity,_quality,_novelty_and_reproducibility: As discussed in the strengths/weaknesses section, I found the figures of this paper pretty hard to read and draw conclusions from. I think the paper presents a novel framing of the networks as moving through a "space", but I'm not sure that's a significant enough contribution.
summary_of_the_review: The paper presents a novel framing of the networks as moving through a "space", but I'm not sure that's a significant enough contribution.  I also found it generally hard to read and understand, especially the figures, and felt like conclusions were drawn from the figures without enough support.  All in all, I would say reject. That being said, I'm coming from a visualization background, rather than a theory background, so there is a chance that there are other aspects that I couldn't fully evaluate.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors leverage recent work (Williams et al., 2021; Shahbazi et al., 2021) that defines metric spaces over neural representations to analyze multi-layer networks as paths through this representational shape space. These metrics were recently developed, and are similar to CKA (Kornblith et al., 2019). The main novelty in this paper comes in section 2.3 where the authors describe how to interpolate along geodesics within this space of representations, measure the angles at two incident geodesic paths, and to perform projection of a point onto a geodesic path. They apply these novel interpretive tools to understand how computations progress in ResNet and VGG architectures. They study how these paths change over training and across "wide" versus "deep" networks. The most interesting result is that the geodesic angles are close to orthogonal at each layer.
strength_and_weaknesses: The main strength of this paper is to introduce conceptual advances, and the main weaknesses are the empirical demonstrations. While some of the experiments are interesting, I just wish there was "more" in this section of the paper. I also have some concerns about technical details missing from the Appendix (see next section on "clarity").

It is nice that the authors use three different metrics to illustrate their points. I would also be interested in seeing the Euclidean shape metric (not just the angular shape metric) as was introduced in Williams et al. 2021.

I would have loved to see more results on "toy" tasks and models to help build intuition -- e.g. do fully connected networks trained on a simple task like MNIST show similar results to the very deep networks?

I have some reservations about the practice of flattening convolutional layers as the authors describe in footnote 2. Could the authors comment on the alternative procedure in section 2.4 of Williams et al. (2021) for conv layers? I think that paper makes a reasonably compelling case against flattening the conv layers.

I wish the authors were able to dig further into some of their interesting observations. For example, what do the orthogonal jumps at each layer really mean? Can further intuition be developed through targeted experiments?

Finally &mdash; and this is not so much a weakness as a suggestion for future work &mdash; I would ***love*** to see this analysis applied to neural ODEs ([Chen et al. 2018](https://arxiv.org/abs/1806.07366)). The idea is that these will be truly continuous paths through representation space rather than jagged, discrete jumps. I think this may really help shed light on the approximate orthogonal angles observed by the authors.
clarity,_quality,_novelty_and_reproducibility: From a clarity standpoint I would like to see section 2.3 expanded with explicit references to sections in the Appendix that describe how geodesics, angles, and projections are actually calculated in practice. I think the main text buries too many details.

Can equation (A.10) be found in prior literature on shape space geodesics? If so, could the authors provide a citation. If not could the authros provide a more formal proof that this is indeed a geodesic? What do the geodesics look like for the Euclidean shape metric? Is it true that you only need to fit $\mathbf{R}$ once at the endpoint of the geodesic?

Many expressions in the appendix are asserted without a very clear reference or a self-contained proof. See, for example, equations (A.3) and (A.4). Please flesh this out.
summary_of_the_review: I really like the ideas in this paper and I think it could grow into an important form of analysis for the field. Currently I have recommended "borderline reject" due to some of the missing details in the Appendix, and because I am only moderately enthusiastic about the experimental results in this paper (even though I think they are promising). Some of these points are addressable within the rebuttal period and I am very much open to raising my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: [Summary]

The submission uses different measures of dissimilarity to characterize the sequence of a neural network's hidden representations at a point in training. They observe that, as the layers progress toward the output layer, the representation gets further from the input representation and closer to the output representation. They also compare th "path" made from this sequence of representations to the shortest path between the input and output representations.
strength_and_weaknesses: [Strengths/Weaknesses]

The most interesting finding to me was that hidden representations in deeper neural networks are closer to the straight line between the input and output representation. While I'm not certain at this point, it is likely that this insight will lead to better generalization guarantees for deeper networks.

The main weakness is that all the experiments are conducted on Cifar-10. I suggest repeating the experiments on at least one more dataset. This dataset has to be sufficiently complex (this excludes MNIST and Fashion MNIST) and sufficiently different from Cifar-10 (this excludes Cifar-100). Some suggestions are STL-10, Caltech, and Mini ImageNet.

Another thing missing in this paper is discussions on the implications of the dissimilarity measures. What does each dissimilarity measure capture and what changes is it invariant to? A nice example is the discussion on CKA by [1] that shows CKA captures alignment between the top singular vectors of the two representations.

I would also like to see what the purpose of the "projection" experiments (Like Fig 1 H-I-J) is and what conclusions one can get from its results.

The discussion section says that the paths are more straight (the angles are closer to pi) at initialization. This is not what I would conclude from the results in Figure 2. Only one dissimilarity measure shows this pattern and even in that subplot there is a lot of variability in the angles. With the other two measures all the angles are close to pi/2.

[Minor comments]

Fig 3 is too crowded and some curves are basically invisible. I suggest breaking it down into multiple figures and moving some to the appendix.

The caption for Fig 4 says that the marker size indicates width. Does this mean that the network width keeps growing all through the legend from top to bottom?

[1] Kornblith, Simon, et al. "Similarity of neural network representations revisited." International Conference on Machine Learning, 2019.
clarity,_quality,_novelty_and_reproducibility: The paper is overall well-written.

The analysis is novel to the best of my knowledge.

Code is not provided.
summary_of_the_review: [Decision]

It is hard to evaluate this paper as there is no specified end goal (designing a certain new algorithm or architecture or improving deep learning theory) for this analysis and I cannot determine to what extent the experiment design is appropriate without an ultimate goal. Nevertheless, as long as a study shows clear and reproducible patterns with potential for future work, it is valuable to me. Right now I'm putting a low score since (1) the experiments are on one task and (2) there is no discussion on the implications of the chosen dissimilarity measures but I'll be happy to raise my score if these limitations are addressed unless other reviewers reveal other issues.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose thinking about neural network layers as steps in a path that describes how a representation changes steadily towards the target output.  In order to analyze this, they use existing representational distance metrics between the outputs of different layers of the same network.  The authors introduce new visualization methods to understand how a representation changes as it is passed through the network; they propose looking at how the path differs from an shortest-path interpolation of the input and output, projecting onto geodesics, and considering the straightness of the angle that occurs when traveling through two layers.   
strength_and_weaknesses: A strength of this paper is its proposed geometric visualizations for understanding the internal behavior of neural networks, which is rather novel.  Comparing the paths taken by a wide versus a deep model was also an interesting insight.

A main weakness of this paper is in its inability to explain or hypothesize much of the results it found through its analysis.  Whereas the hypothesis is that training a neural network would cause the representational path to be straight, it happens that they turn more orthogonal instead.  The paths taken by the network also greatly differ from the shortest interpolated path, across all representational metrics.   Such mysteries are not expected or understood, and it is unclear if this is because the metrics are unable to provide a clear picture on the behavior (as the authors mention in the discussion) or if there is something more fundamental about the behavior of a neural network itself.  There is therefore no clean takeaway regarding these proposed metrics - without understanding why the behavior is the way it is (or why it is different from what is expected from these metrics), one might be unable to derive much insights from them and therefore hesitate to use them.
clarity,_quality,_novelty_and_reproducibility: There are no issues regarding the clarity or quality of this submission.  I believe there to be no issues regarding reproducibility in this work.  I also believe the proposed approach to be decently novel; whereas they reuse many existing representational metrics, they apply them to the layers within a consistent network and also propose new, additional visualizations about the geometry of the path.
summary_of_the_review: Overall, I believe this paper is an interesting step towards the direction of understanding how layers interact with each other to compose a complex neural network.  The authors propose such a spatial path analogy, which they set about visualizing layer by layer.  They also propose extra visualizations to understand these paths; interpolating to generate the shortest path, projection, and showcasing the angles between layers.  However, the results that the authors achieved on these proposed visualizations fly against their hypotheses for expected behavior (e.g. the paths do not follow the shortest interpolated paths, nor do the angles suggest a straight path).  It is therefore highly unclear if these visualizations are correct or useful to use, if there are inherent limitations in the representational metrics utilized that inhibit their ability to provide a clear picture into neural network behavior, or some other reason.  Without aligning our hypothesized understanding of how the neural network behaves with the actual behavior according to these visualizations in an interpretable and understandable manner, I believe it would be difficult for the general community to adopt these proposed methods for analysis.  I therefore suggest the authors think deeper about why their results differ greatly from their hypotheses regarding their constructed visualizations, what the current discovered implications are, and if there are other natural metrics that might achieve the expected behavior, before being considered for acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this work, authors use the presentation at different layers of deep networks to visualize how representations evolve across layers of the network. This line of work is interesting and useful. The paper is also well-written and easy to follow. Based on reviewers feedback and my own judgement of the paper, I think the following two are the main weaknesses of the paper:

- All experiments are run on CIFAR-10 dataset. This is a very limiting factor for an empirical paper. I suggest authors to repeat their experiments on more datasets including CIFAR-100, SVHN, ImageNet or even NLP tasks.

- I think authors should go beyond the visualization and observations and suggest how this finding is related to or explaining existing phenomena in deep learning or how it can be potentially used to develop better architectures or training algorithms.

Given the above, I recommend rejection but I suggest authors to resubmit after taking the above points into account.
justification_for_why_not_higher_score: Empirical results are only reported on CIFAR-10 and it is not clear how the findings of the paper is connected to existing phenomena.
Decision: Reject