Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper takes a large number of neural networks that span range of different parameters (e.g. width), train them in a large number of ways (e.g. dataset, batch size), and studies several metrics that characterize the local (e.g. Hessian spectral norm) and global (weight vector pair-wise connectivity on potential non-linear paths). By doing a large scale study of this kind, the authors are able to identify several distinct phases of loss landscapes, dividing them phenomenologically into (globally {well,poorly}-connected) x (locally {flat, sharp}). The authors also develop a simple mathematical toy model that can model this taxonomy. 
main_review: Overall, I really like this paper. I am very happy that the a large scale study of this kind has been done and that the different local and global aspects of loss landscapes of DNNs have been investigated in this way. I enjoyed reading the paper, the message is clear, and so are the experiments. I have a number of points of confusion and clarifications but overall I'm a fan! Even if the paper potentially doesn't get accepted I'd strongly suggest the authors put it on arXiv so that the community can benefit from their results.

STRONG POINTS:
1) a clear theoretical backing of the questions
2) a well written paper
3) looking at both local and global aspects of the loss landscape (the global aspects are often understudied)
4) a nice mathematical toy model allowing us to conceptualize the taxonomy the authors present

WEAKER POINTS AND QUESTIONS:

1) In your experiments you vary a lot of parameters such as the network width, batch size, etc. Once you change a hyperparameter, the optimal value of the other hyperparamters also changes. This is certainly the case when you're spanning orders of magnitude as you do. In your results and plots, is each datapoint the optimal set of hyperparameters for that specific choice of the controlled hyperparameters, or are all of them kept constant. For example, let's say you have batch size, learning rate, width, and L2 regularization, and let's say you're looking at mode connectivity vs the learning rate. Are you, for a specific choice of learning, choosing a new, optimal (through a grid search let's say) values of (batch size and L2 regularization). If not, this could be a serious confounder for your experimental conclusions. I think this point as the key question I have that needs to be addressed.

2) In the introduction, you mention that the best test accuracy is achieved when the ensembles of trained models are more similar to each other. This seems to go both against 1) the received wisdom that ensembles work well because of function diversity, 2) many theoretical results in this direction, and 3) empirical work on this (for example see https://arxiv.org/abs/1912.02757 which might be a relevant reference for you as well, since they also deal with mode connectivity a lot). How do you reconcile your conclusion with the previous results?

3) Your toy model is one dimensional? Some papers argue that it's the high dimensional nature of the loss landscape that plays a key role in forming many of its properties. For example https://arxiv.org/abs/1906.04724 (NeurIPS'19) looks at a similar question to you but with a distinctly high-D lens. Does the predictive power of your toy model depend on its dimension, or are you just trying to capture the relevant phenomenology with 1-D?

4) Bigger and better models? The results you show are on the weaker side of CIFAR-10 test accuracies. In Figure 2 it seems that the color scale doesn't even go to 90% test acc. It might be better to add stronger, better optimizer models, and at least CIFAR-100 as well to make the case for your taxonomy stronger.

5) I am a bit worried that the large "phase changes" you see in e.g. Figure 2 Panel e) are caused by the really bad losses you have there. You use cross entropy loss on 10 classes, so the default value should be ln(10) ~ 2.3. But your barriers are often of the order ~10. How is that possible? And would the transitions look as sharp if you didn't have these really high losses to compare to?

RELEVANT PAPERS
I also thought of a few more relevant papers. 
- On large scale structure of loss landscapes: https://arxiv.org/abs/1906.04724
- On ensembles from the loss perspective: https://arxiv.org/abs/1912.02757
- On linear mode connectivity and even init - optimum connectivity from an unrelated init: http://proceedings.mlr.press/v139/lucas21a.html
- On the large scale structure of the Tr(H): https://arxiv.org/abs/1807.02581
Feel free to add them if you see that they are relevant, but no pressure for sure.

Overall I really like this paper! It could be made better by potentially addressing the suggestions I made above, but as it is it's certainly valuable to the community and should get published one way or another (and arXived for sure as soon as the authors see fit).

UPDATE AFTER REBUTTAL:
I am sorry for being slightly late into the reply game. I am very happy with the answers the authors provided and would like to keep my score of 7 = accept. I think this paper is a valuable contribution to the subfield. Good job!
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposed an experiment framework for evaluating loss landscape of neural networks during training. The experimental setup systematically vary two types of hyperparameters: 1) temperature type, where at high temperatures, larger learning rate or smaller batch size will induce more noise to training, and 2) load type, including data quantity/quality and model width. Neural networks are trained with different hyperparameter combinations according to the load-temperature pairs. Then 3 metrics including mode connectivity, Hessian, and output CKA similarity evaluated on perturbed training data. Jointly consider the 3 metrics (local and global) evaluated on all loss landscapes, the authors classify the loss landscapes into different phases. One of the phases (IV with high connectivity, low Hessian and high CKA) corresponds to best test accuracy. 

main_review: The paper jointly considers three types of metrics that were considered in literature to evaluate both global and local characteristics of loss landscapes. It explains some of the counter-intuitive phenomena observed such as why in some low hessian loss landscape, the model does not generalize well. Experiments were run with 3 types models (ResNet, VGG and Transformer) on different data sets. However comparing the results figures, ResNet vs Transformer, it is not clear if the defined phases are universally useful. On the other hand, the effect of optimizer was not discussed. 
Overall, the paper is well written.

Minor comments:
In Figure 4, when evaluating ‘global nice’, global connectivity seems the only thing that matter, as highest accuracy is at 0 mode connectivity and low (but not lowest) Hessian.  
Figure 3 should the y axis label be learning rate instead of  batch size, since learning rate is used as temperature? Otherwise, what do the ‘batch size’ in both subtitle and axis labels mean? The figure captions need to be more clear.

The authors clarified how to interpret the results and made corrections to figures in their responses. I am more convinced. 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper explores factors that influence the loss landscape. In particular, it categorizes the loss landscape in different phases, each corresponding to different status of key factors such as Hessian spectrum and mode connectivity. It also tries to establish an "effective loss landscape" model that depends on key parameters.
main_review: Exploring key factors that affect the global landscape and training has been a hot research area. Many work has focused on the influence of one particular factor (such as Hessian, connectivity, etc). The advantage of this work is that it combines several factors together and looks at the comprehensive influence. I believe a single factor is not enough to explain all about landscape and training, and a phase diagram is a good trial along multi-factor direction. Another interesting point is the category of load and temperature-like parameters. It seems that different load/temperature-like parameters have similar effect on the landscape. This may help unify the influence of different hyper-parameters.

The following are my questions and concerns about this paper.

1. This work shows the phase transition of some key parameters. What is the cause of the transition, and is it possible that these parameters are still intermediate parameters that depend on some other key factors?

2. Hessian spectrum and CKA similarity perform similarly on most results but different in a few. What is the essential difference between these two factors, and what standard should we choose if we have to choose one?

3. Is there any empirical guidance this phase diagram can provide? The empirical results show the landscape and key parameters at the end of the training. Is it possible that we acknowledge the current phase during early iterations?

4. I'm a bit confused about the effective loss landscape. I'd like to see more about why defining effective loss landscape is reasonable (and necessary) for observing the influence of temperature and load.

5. The explanation of (4) when discussing mode connectivity is contradictory in the main paper and the appendix. In the main paper, it claims that there is a barrier if $mc>0$, while in the appendix both the description and the figures show that there is a barrier when $mc<0$.
limitations_and_societal_impact:  The authors addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The author(s) perform an empirical study of which (and how) properties of the landscape affect the test accuracy, obtaining results that are consistent over a large number of deep neural network models over several credible datasets. The study puts into practice this idea that the generalization is not uniquely determined by local properties of the landscape, but rather also by global properties. They argue that different kinds of acting on the system and on the dynamics can be grouped into two control parameters that they call 'load' and 'temperature', and show that according to the position in the (load, temperature) plane one has different phases. They observe that each phase seems related to a different test accuracy, and realize that to obtain the best test accuracy, in addition to being locally flat, the landscape should also be globally "nice". By "nice", they mean that the landscape is well-connected (no large barriers separating regions) and that different trained models have a large CKA similarity.

main_review: The paper addresses a long-due question, trying to explain that local flatness cannot be used as a sole indicator of a good minimization. They address this question numerically, through large-scale experiments over a wide range of parameters. The results seem quite robust across architectures and datasets. The main message of the paper is simple and compelling, and will likely have a good resonance throughout the community.

I do have a series of concerns and comments, however, that I would like the authors to address.

I will mostly refer to Fig.2, but many of the comments apply to several figures.

Despite being conceptually very simple, the results along the temperature axis are extremely confusing. This is  due to the way labels, axes and captions are presented. The y axis of Fig.2 is labeled "temperature (batch size)". However, the batch size is the inverse temperature. So, is the quantity on the y axis the batch size or its inverse? Further, the direction of the axis is reversed (bigger values are below), which makes everything even more confusing. Finally, it took me a long time before I realized that the axis was inverted, because the fonts are so small that they can bearly be read with a 200% zoom (and impossible to read when printed).
So what is the quantity on the y axis? Is zero temperature at the top or bottom (an intuitive figure would put it at the bottom)? I think that in the y axis temperature goes upwards and batch size goes downwards. This makes sense but it took me quite a while to understand the figure, and the reader should be able to grasp it immediately.
The captions and labels should clearly explain what is plotted, and help the reader in the interpretation.
What do the subplot titles in Fig2 mean? I read batch size, and I assume that it is because the temperature parameter here is the batch size. If so, this is redundant (thus confusing), since it is already written on the y axis. However, the same "batch size" labels appear in figure 3, where the caption says that this figure is "using learning rate as temperature". Yet further confusion is caused by the main text, which instead states that a learning rate decay was used. In this case, the authors rightly state that the temperature is not well-defined. Besides the contradiction with the figure caption, I assume then that what the authors are stating is that the y axis stands for the initial temperature value. This independence from the learning rate is interesting, though I would be more conservative, because in many complex systems the rate of annealing (how fast the learning rate decreases) can be crucial to this effect [].


I do not understand why Fig.2e reveals a transition on a vertical line. By looking at the figure I see a vertical line in the bottom part of the plot, but this line then curves and possibly even splits. How are the authors identifying this phase transition?

The authors talk of phases, but the word phase has a specific meaning related to a non-analyticity in a (free) energy functional. For sharpness and connectivity a phase transition seems possible (actually, it was shown in https://arxiv.org/abs/1809.09349 that the one along the load axis is actually a phase transition), but the one from phase IV-A to phase IV-B seems more a smooth crossover, since I see no sharp change in the CKA similarity nor in any other observable. Am I right?

It seems to me that the CKA similarity alone is enough to explain the test accuracy (the two quantities correlate extremely well), without the need of the other metrics. Can the authors elaborate on this?

In all shown examples, the interpolation threshold (where the training loss becomes zero) seems to be at a higher load than the MC connectivity. From Ref https://arxiv.org/abs/1809.09349, which identifies a jamming transition at the interpolation threshold, I would have expected the two to coincide. Can the authors say anything on this? Can it be due to the way the MC was computed (both because of the method and the k=2 choice)?

In the supplemental material the authors state that most experiments were run until convergence, but they stopped at 150 epochs when the convergence criterion was not met. I think that this can influence the results, so I would least specify in the main text which runs did not reach convergence. Does this apply to the runs (and only those) with a negative mode connectivity?

The authors talk about "poor" mode connectivity, but this term is not defined. Is a poor MC positive? Is it non-zero (so also negative)?

Line 206. The sentence is not clear, also due to the "poor" term.

The results are shown for VGG and Resnets. I wonder whether they also hold for simpler models such as multi-layer perceptrons and simply convolutional networks, since these models are more tractable analytically. Moreover, can the authors say anything about linear models?

Would the authors have found equivalent results if they studied the mode connectivity through the Nudged Elastic Band method, as done in http://proceedings.mlr.press/v80/draxler18a/draxler18a.pdf?

Fruitful examples of the study of loss landscapes can be found in https://www.annualreviews.org/doi/pdf/10.1146/annurev-conmatphys-031119-050745, from random landscapes, to jamming, to spin glasses. Ref http://proceedings.mlr.press/v80/draxler18a/draxler18a.pdf (cited elsewhere in the text) is a great example.
When mentioning sharp vs flat minima, I would have expected a citation to https://arxiv.org/abs/1609.04836.
The concept of temperature is not new as well as the interpretation in terms of a Gibbs distribution. See e.g. https://arxiv.org/pdf/1711.04623.pdf and references therein.
The concept of load is not new either (except for the name).

On line 295 they can cite e.g. 
- Q. Li, C. Tai, and E. Weinan, Stochastic modified equations and adaptive stochastic gradient algorithms,inInternational Conference on Machine Learning(PMLR, 2017) pp. 2101–2110.
- W. Hu, C. J. Li, L. Li, and J.-G. Liu, On the diffusion approximation of nonconvex stochastic gradientdescent, Annals of Mathematical Sciences and Applications4(2019), arXiv:1705.07562.
- Y. Zhang, A. M. Saxe, M. S. Advani, and A. A. Lee, Energy–entropy competition and the effectivenessof stochastic gradient descent in machine learning, Molecular Physics116, 3214 (2018)

Anyhow, I am not sure that section 4 adds any knowledge at all. The authors have a picture in their mind, and they draw it schematically, in one way out of infinitely many that they could have chosen. It might even be wrong, since it assumes the existence of only two relevant scales of ordering (local and global), while they could be infinite (e.g. with spin glasses, jamming, ...). The concept of effective landscape already exists, and it is called Free Energy. This quantity comprehends the competition between energy (loss) and thermal agitation, it can in principle be obtained by legendre-transforming the energy (= the loss), and was studied for decades (one reference is the just given Zhang et al paper, but we can even resort to papers from the 80s from Gardner and coworkers). 

There is something awkward with the batch indices at line 116. The concept is simple and clear, but the authors introduce some names of indices which it is not clear to what they correspond unless one already knows what the authors mean.

Line 162: which typically occurs if training failed to locate a reasonable optimum. This statement is true for a strict inequality, but the authors are referring it to mc<=0. 

Line 338: "we provide theoretical explanations". I disagree. The paper is purely empirical and this is fine.

Line 344: the statement seems naif.

limitations_and_societal_impact: One limitation that I see is that the temperature and load axes are (perhaps deliberately) defined in a fuzzy manner. They do this to emphasize that different drivers have qualitatively similar effects. I am not sure that all the drivers that are collected under the same umbrella actually represent the same kind of behavior (e.g. weight decay as temperature). I don't see this as a problem, since these issues fall out of the scope of the paper and can be addressed and understood in future work.
review_title: null

Meta-review: This paper investigates how the structure of the loss landscape of neural networks affects their generalization performance.  The authors categorize loss landscapes into four different phases, globally {well,poorly}-connected) x (locally {flat, sharp}), where global connectivity is measured by mode connectivity and local flatness is measured using Hessian. Through a large-scale empirical investigation of neural network loss landscapes, the authors present an analysis of how these categories affect generalization.

Overall, I think this is a solid paper. The reviewers provided a lot of useful detailed feedback and the authors also did a great job of addressing the reviewer concerns. During the discussion phase, the consensus decision leaned towards acceptance. 

I recommend acceptance and encourage the authors to address the reviewer comments in the final revision.


Decision: Accept (Poster)