Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
The authors propose to visualize training trajectories with the dimension reduction and visualization tool - PHATE. The authors try to reconstruct manifolds on which the training trajectories lie. It argues that the method captures data characteristics from all dimensions and shows consistent geometric patterns for loss landscape surrounding good and bad generalization optima.


Strength
- The approach embeds multiple training trajectories in one visualization by adopting dimension reduction approaches, which is not explored before. The comparison of multiple dimension reduction approaches is also lacking in previous literatures.
- The paper has a nice review and summarization of previous approaches on loss surface visualization.

Weakness

My major concern is that the insights observed from the methods are not quite straightforward for interpretation and some conclusions are not clear or novel. 

- In page 4, the authors claimed “We note that methods that pick random directions...are not able to visualize entire trajectories as the spaces and planes change” this observation is also made in previous work[1] and PCA is used for visualizing the training trajectories. However, this is not necessarily a disadvantage for visualizing the flatness of the minima as discussed follows.

- To characterize the region of loss landscape, the authors propose to adopt “jump and retrain” (setting 1) and this involves visualizing multiple trajectories together. However, the rationale to do this is not quite clear and the interpretation of the visualization is hard. It is not quite clear about the advantage of proposed visualization (Figure 2) on explaining the “flatness” in comparison with previous 1D or 2D visualization with randomized directions. On the other hand, the computation cost (multiple initialization, multiple steps and retraining) for generating those figures could be significant.

- The proposed visualization should be valuable for comparing different trajectories produced by different optimizers as all trajectories can be plotted together, however, the visualization (figure 4) does not convey a clear message about which optimizer (trajectory) is better in performance (generalization). 

- The individual trajectory visualization (Figure 3) in sec 5.2 seems not to differ too much with the PCA based visualization [1][2]. What is the takeaway message from Figure 3? It is not clear how the observation “the resulting parameters appear to be further away from the initialization” is made from the figure.

Questions

In section 5.3, the authors claim there are consistent and distinct patterns for “good” and “bad” minima. Is there a smooth transition between the “good” and “bad”? Will the pattern differ with a much better or worse initialization? e.g., what is the pattern for random initialization (which is the “worst”)? 


[1] Li et al, Visualizing the Loss Landscape of Neural Nets, NIPS 2018.
[2] Eliana Lorch. Visualizing deep network training trajectories with pca. ICML Workshop on Visualization for Deep Learning, 2016.
review_title: Interesting visualization but the interpretation is not quite clear

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### 1. Brief summary:
The authors use a new dimensionality reduction technique called PHATE (that was first introduced in a different paper) to study the weight-space positions and training trajectories of several DNN architectures (ResNet, WideResNet) on vision classification tasks (CIFAR-10, CIFAR-100). They use different optimizers (SGD, SGD+Momentum, Adam), study very good and decent (they call them bad) optima, and the solutions to memorization of random labels. They perform two kinds of experiments: 1) perturbing a single optimum and retraining from there, visualizing those trajectories, and 2) visualizing several random init -> optimum trajectories from different inits. They then draw some conclusions from the PHATE projections for the two kind of experiments and different optimizers and kinds of optima.

### 2. Strengths
* I like the introduction of the PHATE algorithm and the authors explanation of its advantages and why it might be a good fit for the weight space trajectory visualization of DNNs. 
* The paper is well motivated. Understanding the loss landscape of DNNs is likely very important and still very underexplored.
* I like the comparison to other algorithms (PCA, t-SNE) and the synthetic tree-like structure they visualize in addition to real DNNs to demonstrate the advantageous properties of PHATE.
* I like that the others study the stability of their visualization to retraining and seed change -- that is very good and should be the norm in all papers that make claims that depend on stochasticity.

### 3. Weaker points
1. The scope of the paper is, in my opinion, a bit too limited. A lot of space is used to introduce the PHATE algorithm that is, as I understand it, /not/ a novel contribution of this work.

2. While the visualizations look compelling, the conclusions that are drawn from them are often too strong. I don't see how the visualization can be connected to flatness, dimensionality etc directly. They do serve as a good visual guide, but I feel the paper doesn't establish the connection to those quantities very well. I see that let's say for the random labels and real labels, the projections look different, but how do I link this to e.g. the size of the low-loss basin around those two?

3. The authors flip between a language suggesting that there are some specific dimensions that are being visualized (e.g. "potentially flying off to an orthogonal subspace" and appreciating that the PHATE technique doesn't preserve dimensions in any meaningful way. I would advise to soften the claims to reflect the non-linear, adaptive nature of PHATE and the difficulty of connecting the embeddings to any particular directions.

4. The experiment type 1 = going in random directions off an optimum and retraining back doesn't seem to push far enough. What would happen if I perturbed more? Would the solutions glide back as they do? How does this compare between the real labels and random labels? I'd say this would be a nice validation at least -- if you go to far, you should not go back. If you do, the PHATE projections do not project the way you believe. If you don't, that's interesting on its own and can be used for comparison between cases (Adam, SGD, random labels ...).

5. The Figure 3 results for a single trajectory look uninformative -- what would happen if you did this multiple times from different seeds? Are the Adam paths smoother generically, or just this particular random seed? You should either make the experiment statistically meaningful in some way, or explain what the reader is supposed to take away from it. As is, it doesn't do much.

6. The experiments of type 2: random inits -> training -> optima, look promising. However, they really break my intuition for what PHATE does and if it is at all relevant. My main worry is this: why do the inits end up coinciding? From [1] and [4] it is quite certain that a) the inits are mutually orthogonal to a high degree and b) so are their optimized endpoints. How come your inits get mapped to the same point sometimes, but other times they do not? This really puts your results into question for me, especially because you use the visualizations to make pretty strong claims about the loss landscape structure.

7. (correct me if I am wrong) Only WideResNet and ResNet on CIFAR-10/100 were used in this paper. Those are two pretty similar architectures and two very similar vision tasks. I would appreciate a much broader set of experiments to make sure the claims here hold. For example, 
a. what do skip connections do? both architectures here have them, and
b. How would MNIST, FASHION MNIST, SVHN do? Those are cheap to train and easy to use datasets and I would expect the authors do use them
c. How would a simple feed-forward CNN do?
d. What about fully-connected nets? Would they behave the same way?
Empathically, I am /not/ asking for ImageNet -- I know that it is very hard to run and while it would be nice to have, there is no need to use it in every paper. But I would want to see a much broader set of experiments  on other CNN-based architectures as a well as other, potentially weaker datasets. As is, the paper doesn't have the sufficient experimental support to see the generality of the interpretation claims. 

### 4. Relevant papers that might be worth adding/exploring
There are some papers that I think might be relevant here and that the authors might want to wish to add / read / explore. 

[1] /Large Scale Structure of Neural Network Loss Landscapes/ by Stanislav Fort, Stanislaw Jastrzebski (https://arxiv.org/abs/1906.04724) at NeurIPS 2019 build a geometric model of the low-loss manifolds of DNNs incorporating the observations of 1. connectivity of init to optimum, the high-dim nature of the manifolds and their connectedness. They also visualize the loss landscape on sections that might be relevant here. They also show that they can find N-dimensional surfaces connected N+1 independent optima together, going beyond the 2 optima on a 1-d path.

[2] You might be missing the second paper that established the connectivity between different modes: Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson,  A. G.  /Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs/ at http://arxiv.org/abs/1802.10026

[3] /Measuring the Intrinsic Dimension of Objective Landscapes/ by Chunyuan Li, Heerad Farkhoor, Rosanne Liu, Jason Yosinski (https://arxiv.org/abs/1804.08838) establishes that the loss landscape low-loss manifolds have a low /intrinsic dimension/ that can be interpreted as high d of the manifolds (done in https://arxiv.org/abs/1906.04724)

[4] /Deep Ensembles: A Loss Landscape Perspective/ by Stanislav Fort, Huiyi Hu, Balaji Lakshminarayanan (https://arxiv.org/abs/1912.02757) look at the cosine similarity of the weight space positions of multiple runs as well as a single run. They also visualize the loss landscape between optima on different affine and non-affine sections.

In your justification for why PHASE might be good at visualizing the trajectories you mention that they are low dimensional. There is a number of works showing that it might or might not be the case:
[5] Gradient Descent Happens in a Tiny Subspace by Guy Gur-Ari, Daniel A. Roberts, Ethan Dyer (https://arxiv.org/abs/1812.04754) shows the gradients in a mostly low-D subspace, but the actual learning happens orthogonal to that.
[6] Emergent properties of the local geometry of neural loss landscapes by Stanislav Fort, Surya Ganguli (https://arxiv.org/abs/1910.05929) dissects the Hessian structure and finds low-D signal + high-D noise, where the signal is not in the learning directions but rather forms constraints.
[7] Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra by V. Papyan (arXiv:2008.11865) has a on overview of the low-D structures in the DNN Hessians that might be relevant.

 
### 5. Summary
I like that you introduced a new dimensionality reduction technique to visualizing loss landscape trajectories and positions for DNNs. The reasons why it might be better than others are compelling. However, I think that the scope of the experiments you provide is limited, the claims you make a bit stronger than would be justified by the results shown, and there are some worrisome features of some of the embedded trajectories that make my question the validity of your overall conclusions. I think the paper has a promise, but it needs a bit more work. **I am open to revising my score** if you address my questions well.

review_title: A promising dimensionality reduction technique is applied to DNN trajectories in the weight space, however, more experimental validation is needed to draw meaningful loss landscape implications

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper suggests using PHATE, a modern dimensionality reduction method, for visualizing the training trajectories of deep networks. It argues that PHATE visualizations can bring to light interesting aspects of the training dynamics that are missed by other dimensionality reduction algorithms because PHATE does a better job at preserving both local and global structure in the data. This is not the first application of PHATE in the context of deep learning, but to my knowledge it is the first application to deep learning trajectories.

The paper shows that PHATE visualizations of train-and-jump trajectories (where the model is repeatedly pushed away from a minimum and then allowed to retrain) can have features that are correlated with how well a model generalizes. It also shows that different optimization algorithms can lead to distinguishable visual features.

While the use of better visualization techniques may lead to better understanding of neural network dynamics, I am not convinced that the paper succeeds in making this case:

1. The generalization results (Figure 2) are interpreted using known results about the generalization properties of flat vs. sharp minima. These results are nice, and are potentially useful. However, they were only demonstrated in a few settings. Do these results hold more generally, when using other architectures and different data sets?

2. I felt that Section 5.2 on visualizing the trajectories of different optimizers did not have a clear take away message.

3. Except for Figure 1, PHATE is not compared against more commonly used reduction algorithms such as tSNE and UMAP. For example, regarding Figure 2: Could we draw similar conclusions about generalization using other techniques?

review_title: Using PHATE to visualize gradient descent trajectories

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Paper Summary

This paper uses PHATE to visualize the progression of neural net parameters during learning in neural networks to provide insight into generalizable vs. non-generalizable minima, and the behaviors of different optimization algorithms.  PHATE is an improvement over previous visualization techniques due to its approach to finding a manifold, allowing it to plot in two dimensions multiple trajectories that do not otherwise share a plane.  This is then used to plot trajectories in "jump and retrain" experiments, in which a minimum is found, and then perturbations made to the network parameters, before restarting training.  It is shown that in the networks experimented on, minima with good test set performance reliably funnel the new learning trajectories into the same minimum, while minima with poor test set performance see the perturbed initializations find other minima, thus demonstrating the "flatness vs sharpness" of minima.  Trajectories produced by SGD, SGD with momentum, and Adam are also compared.  Adam is shown to travel further, but along a smoother trajectory.

### Originality

The visualization approach is not new, but is new to neural network trajectories.  The visualizations effectively confirm suspected properties of neural networks, but do not offer anything particularly new.  The behaviors of learning algorithms are demonstrated, but not much discussed, so little new is learned.

### Significance

Visualization is important to understanding the behaviors of high-dimensional learning trajectories, and this is better for plotting multiple trajectories than previous approaches.  It is an effective approach, and is superior at demonstrating known neural network behaviors than others.  Insights are not significant, but the visualization of them is improved.

### Clarity

Very clearly written, and easy to understand.

### Quality

Enjoyable paper, which offers a new, effective tool, but no real new insights into neural networks.  The jump-and-retrain experiments are effective and well-chosen.  The demonstrations of learning algorithm are not as effective, and I'm not sure what to take away from them.  I invite the authors to help me find the significance of these experiments.
review_title: Presents an improved approach to visualizing multiple learning trajectories, which is useful in understanding NN generalization. Approach is correct and useful, but resulting insights are a little underwhelming.

Meta-review: The reviewers and I agree that the paper is well motivated and that there are good comparisons to prior work. However, the scope of the paper is rather limited, and there were some doubts about the overall conclusions and whether the current results fully support them. As such, I cannot recommend the paper for publication.
Decision: Reject