Taxonomizing local versus global structure in neural network loss landscapes
Authors: Yaoqing Yang
Liam Hodgkinson
Ryan Theisen
Joe Zou
Joseph E. Gonzalez
Kannan Ramchandran
Michael W. Mahoney
Venue: NeurIPS 2021 Poster (2021)
Keywords: Loss landscape
energy landscape
deep learning

Abstract: Viewing neural network models in terms of their loss landscapes has a long history in the statistical mechanics approach to learning, and in recent years it has received attention within machine learning proper. Among other things, local metrics (such as the smoothness of the loss landscape) have been shown to correlate with global properties of the model (such as good generalization performance). Here, we perform a detailed empirical analysis of the loss landscape structure of thousands of neural network models, systematically varying learning tasks, model architectures, and/or quantity/quality of data. By considering a range of metrics that attempt to capture different aspects of the loss landscape, we demonstrate that the best test accuracy is obtained when: the loss landscape is globally well-connected; ensembles of trained models are more similar to each other; and models converge to locally smooth regions. We also show that globally poorly-connected landscapes can arise when models are small or when they are trained to lower quality data; and that, if the loss landscape is globally poorly-connected, then training to zero loss can actually lead to worse test accuracy. Our detailed empirical results shed light on phases of learning (and consequent double descent behavior), fundamental versus incidental determinants of good generalization, the role of load-like and temperature-like parameters in the learning process, different influences on the loss landscape from model and data, and the relationships between local and global metrics, all topics of recent interest.

Taxonomizing local versus global structure in neural network loss landscapes
Yaoqing Yang', Liam Hodgkinson 1.2 Ryan Theisen's .oe Zou',
Joseph E. Gonzalez', Kannan Ramchandran's Michael W. Mahoney
1 University of California, Berkeley
International Computer Science Institute
fyqyang, 1iam. hodgkinson, theisen, joezou, jegonzal, kannanr, mahoneymux&berrkeley edu
Abstract
Viewing neural network models in terms of their loss landscapes has a long history in the statistical mechanics approach learning, and in recent years it has received attention within machine learning proper. Among other things, local metrics (such as the smoothness of the loss landscape) have been shown to correlate with global properties of the model (such as good generalization performance). Here, we perform detailed empirical analysis of the loss landscape structure of thousands of neural network models, systematically varying learning tasks, model architectures, and/or quantity/uuality of data. By considering range of metrics that attempt to capture different aspects of the loss landscape, we demonstrate that the best test accuracy obtained when: the loss landscape is globally well-connected; ensembles of trained models are more similar to each other; and models converge locally smooth regions. We also show that globally poorly-connected landscapes can arise when models are small or when they are trained to lower quality data; and that, if the loss landscape is globally poorly-connected, then training to zero loss can actually lead to worse test accuracy. Our detailed empirical results shed light on phases of learning (and consequent double descent behavior), fundamental versus incidental determinants of good generalization, the role of load-like and temperature-lin parameters in the learning process, different influences on the loss landscape from model and data, and the relationships between local and global metrics, all topics of recent interest.
1
Introduction
Among the many approaches to understanding the behavior of neural network (NN) models, the study of their loss landscapes 2 has proven to be particularly fruitful. Indeed, analyzing loss landscapes has helped shed light on the workings of many popular techniques, including large-batch training
3/4 4. adversarial training [5], residual connections [6), and BatchNorm 7 One particular concept
of recent interest the so-called sharpness local minima 3 5 1 While sharpness can be
measured by first-order sensitivity measures, such as the Jacobian Lipschitz constant, it is more appropriately measured by second-order sensitivity measures, typically via the Hessian spectrum
[T]] It has been observed that in some cases NNs generalize well when they converge to relatively flat, i.c., non-sharp, local minimum
While such local sharpness measures can provide insight, their focus on the local geometry of the loss landscape neglects the global structure the loss landscape (namely, precisely the sort of structure that statistical mechanics approaches to learning aim to quantify [3). Indeed, it is well-known that existing sharpness- based metrics can be altered (trivially) by reparameterization tricks (more interestingly) by taking algorithmic steps which have the effect of changing the local structures on the loss landscape [5 [4]. For example, ( shows that adversarial training can decrease the magnitude
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Globally poorly-connected
Globally well-connected
Phase I high harrier
Phase II
Locally sharp
low-energy path
wwwwwn
Phase III
Phase IV-A
Phase IV-B
high barrier
Locally flat
trained are less similar
trained models similar
Figure (Caricature of different types of loss landscapes), Globally well-connected versus globally poorly-connected loss landscapes; and locally sharp versus locally flat loss landscapes. Globally well-connected loss landscapes can be interpreted in terms of global "rugged convexity"; and globally well-connected and locally flat loss landscapes can be further divided into two sub-cases, based on the similarity of trained models.
of Hessian eigenvalues and bias the model towards locally smooth area, even though adversarial training can reduce clean test accuracy [15]. Similarly, [4] shows that Hessian eigenvalues become smaller with reduced l regularization, even though increased E2 regularization is known to reduce overfitting and improve training. used properly. More general considerations would suggest (and indeed our own empirical results, e.g., as reported in Figure 7 demonstrate) that by training to data with noisy labels, one can find models that generalize poorly and yet simultaneously lie very "flat" regions of the loss landscape, with small Hessian eigenvalues, and vice versa. These observations (and other observations we describe below) indicate that the nreviously-observer empirical correlation between very local metrics like sharpness and more global properties like generalization performance may be correlative and not causative, i.e., they may be due to the confounding factor that results in the published literature are reasonably- due to some fundamental properties of deep NNs. They also raise the question of how to capture more global properties of the loss landscape.
models trained to reasonably-gon data, rather than
Motivated by these considerations, we are interested in understanding local properties/structure versus global properties/structure of the loss landscape of realistic NN models. While similar ideas underlie work that adopts a statistical mechanics perspective [2] 12 [6]. here we are interested in adopting an operational machine learning (ML) perspective, where we employ metrics that have been used within ML as "experimental probes" to gain insight into local versus global properties. To do so, we employ the following metrics.
First, we consider
metrics, including the largest eigenvalue and the trace of the Hessian. These metrics try to capture local curvature properties of the loss landscape [T].
Second, we use mode connectivity [17 [8 in particular, the connectivity between trained models.
This metric tries to capture how well-connected different local minima are to each other.
Third, we use CKA similarity [19 to capture correlation-like similarity between the outputs of different trained models. Averaging the CKA over several pairs of models can be thought as approximating so-called overlap integrals frequently appearing in statistical mechanics 12 13 20
We have considered many other metrics, but these three seem to be particularly useful for identifying global structure versus local structure in loss landscapes. Informally, mode connectivity, its name suggests, captures connectiviry, where well-connected models exhibit a single "rugged basin" " with low-energy / low-loss, potentially non-linear, paths through the loss landscape, (i.e., continuous chains of models) all achieving a small loss value. We expect this property to be important since the connectivity of local minima indicates efficiency of the training dynamics to explore the loss landscape, without becoming stuck at saddle points or in a "bad" local minimum. Similarly, CKA captures similarity, where an ensemble of good models will produce roughly similar outputs. These two types of metrics are different and complementary; and both of them are very different than baased metrics, which clearly capture much more local information.
Here we briefly summarize our main contributions.
We design an experimental setup based on two control parameters, temperature-lik parameter that correlates with the magnitude of SGD noise during training, e.g., batch size (in most figures),

---

learning rate, or weight decay. and a load-like parameter that measures the relationship between model size and data quantity and/or quality, e.g., the amount of data, size of intermediate layers. amount of exogenously-introoueee
label noise, etc. By training thousands of models, under variety of settings, and by measuring local and global metrics of the loss landscape, we identify four distinct phases in temperature-load space, with relatively sharp transitions between them. Using global connectivity (measured by mode connectivity) and local flatness (measured by the Hessian), we taxonomize loss landscapes into four categories, which are pictorially represented in Figure T labelled Phase I through Phase IV. For reasons observed in our empirical results in Section it is often convenient to further divide Phase IV into two subcategories, depending on whether the trained models produce similar representations (as measured by CKA similarity). If the loss landscape satisfies the first property, we say it is globally well-connected and if the loss landscape also satisfies the second property, we say it is globally nice. Depending on whether the Hessian eigenvalues are large or small, we say the loss landscape is locally sharp locally flat. Based on these results, as well as measured model quality, e.g., test accuracy, we empirically demonstrate that the global (but not necessarily local) structure of loss landscape well-correlated with good generalization performance, and that the best generalization occurs in the phase associated with a locally flat, globally nice loss landscape. We demonstrate these results on range of computer vision and natural language processing benchmarks 100 CIFAR- 100, SVHN, and IWSLT 2016 De-En) and various models (ResNet, VGG, and Transformers). We also vary the amount of data, the number of noisy labels, etc., to study both the effect of the quantity of data and the quality of data on changing the loss landscape.
We observe the well-known double descent phenomenon 21 22 in our experiments, which exhibits itself as a "bad fluctuation" between the different phases (e.g., see the transition that separates Phase I and II from Phase III and IV in Figure 4a Our empirical observations on double descent corroborates recent theoretical analysis [23 which views the phenomenon as a consequence of transition between qualitatively different phases of learning [3..
Computing connectivity and similarity requires comparing multiple distinct models. This could be computationally expensive, especially if model training is expensive. For many reasonably-sized models, however, the metrics we consider are sufficiently tractable so as to be useful, e.g., during model training. Moreover, the phase transitions and the metrics that use to determine the phases lead to practical tools that can diagnose typical failure modes in training, which we will discuss towards the end of the paper. In this short conference version, we focus on the main message, and we provide a much more thorough discussion on prior work in the full version online (25). in which we also talk about related papers in the study of loss landscapes and statistical mechanics of learning. In order that our results can be reproduced and extended, we have open-sourced our code
2 Setup
In the sequel, we consider training NN fe -Rd + Rdw,, with trainable parameters 0, dataset consisting of n datapoint/label pairs Strain ((11,81),. (Zn, }. Our nominal training objective is to minimize a loss function of the form
(())=
Σ (
+ ())(z),,)+
(1)
Here l is loss function, typically chosen be the cross entropy loss. The parameter À is the weight decay parameter, which controls the level of &y regularization. We consider optimizing NN models using standard minibatch SGD, with iterates of the form
B - 0 (g(()) g = Σ Vol(f((z.)..... 1=1
where v is the learning rate, 0 < B < n the batch size, and the indices i ..,..., in of each minibatch are sampled without replacement from .n]. For classification tasks, we consider also the training/testing accuracy, which is simply the fraction of correctly classified points, (8) = 1 n ( ((fe(x) y). and similarly for accust (0) given test set Sest
hnaps:/githubb.com/sssiyyy/sssss landscape taxonomy
3

---

We now briefly introduce the main metrics and control parameters which we will consider.
Temperature and load. In the sequel, a load-like parameter of loss landscape refers to some quantity related to the amount and/or quality of data, relative to the size of the model. Specifically, we vary either i) model size (e-g., width, which captures the size of an internal representation of the data), for fixed training set size n. ii) training set size n. for fixed model size, or iii) the "quality" of training data, which is varied by randomizing a fraction a of the training labels. Each of these control parameters directly induces a different loss landscape by changing the data Stain and/or architecture fe for which the loss (()) is being computed. For example, we expect that increasing width will result in smoother loss landscape [26]; ; we shall see this effect with CKA similarity in the transition from Phase IV-A to IV-B.
The second control parameter we vary in our experiments is temperature-like parameter, repre- senting the amount of noise introduced in the SGD iterates (2). Most commonly, we take this to be the batch size B, although we will also use the learning rate and the weight decay parameter X. Increasing temperature corresponds smaller batch size, and large learning rate or weight decay, Varying the temperature does not directly define a different loss function [()), but rather it indirectly induces a different effective loss function. This is because, at different temperatures, the iterates of SGD concentrate on different regions of the loss landscape. Due to the noise in the stochastic optimization, the training dynamics may not be able "see" certain features of the loss landscape.
CKA similarity. To measure the similarity of two NN representations, we use the centered kernel alignment (CKA) metric, proposed in [1I. For a NN fa. let Fe = [fo(21)
fo( E R.x denote the concatenation of the output of the network over set of m randomly sampled datapoints. Then the (linear) CKA similarity between two parameter configurations 0, O' is given by
Cov(Fo. For)
000 = /Cov(Fo. Cov Fa.)'
where for X, Y E IRdd we define Cov(X,Y) (m - 1xxXX HY Hm), and Hm
Im m mm1111 T is the centering matrix. The CKA similarity is known to be an effective way to compare the overall representations learned by two different trained NNs [19]. Rather than computing the similarity directly on the original training points, we measure CKA a perturbed training set comprised of Mixup samples (27]: this can reduce trivial similarity that occurs when the models are trained to exactly or approximately zero training error. See also Appendix A.4.1 in the full paper for the ablation study on different perturbed training sets.
Mode connectivity. For two parameter configurations 8, 0. computing mode connectivity involves finding a low-energy curve (f).. E [0, 1], for which (0) = 0, 7(1) = 6', such that J (()(t))dt is minimized 17 [8] A number of techniques have been proposed to find such curves 7. In this work, we use the technique proposed in [7], which parameterizes the Bezier curve with + 1 bends, given by 7o = ΣØ (1) t) for t E [0, 1], where Og = 0, 0x = 8', and é = - Bx-13 are trainable parameters of additional models, defining "bends" on the curve 7+ (4). We use Bezier curves with three bends (k = 2). Given the curve 7a (t), we define the mode connectivity of the models 0 0' to be
mc(0, 8-) 7 (C(0) + L(0)) - [(ts(t)),
(4)
where 1 maximizes the deviation ( 1+
L(ra (1)). There are three possibilities for mode connectivity. If mc(0, 0') 0, then (C(()) + C((')) < (()o(t)),, which means there is a
"barrier" of high loss between 0, 0': in this case, we will say that the loss landscape is poorly-connected or simply say that mode connectivity is poor. If mc(0, 0') > O, then this implies a curve of low loss connecting 0, 0', but it also implies that the training failed to locate a reasonable optimum, i.e., L()) and L(0') are large. If mc(0, 0') R then we will say that the loss landscape is well-connected or simply say that the mode connectivity is good. Note that for all the experiments except neural machine translation, we use the training error (0-1 1 loss) when computing mode connectivity, so that mode connectivity is always normalized to the range of [-100, 100]. We provide additional details
'For this work, we focus on the similarity of representations at the output layer, i.c. after the softmax is applied, although the CKA similarity can be used to compare the representations at any layer.

---

Traning loss
Los Feesian eigencnlio
log llessian Iroee
Obatch soel
(aand soel
Test accuracy
(owter
Phase II
0.8
10
+0003
10"
10
T (batch
Width model
Width model
Whath model
Phase
(b) Training loss
(c) Hessian eigenvalue
(d) Hessian trace
10a
0.6
connechistyy
simlwttt
Kidell distane
120
T00000 2000
107
Width of model
(a) Test accuracy
100
100
03
200
(c) Mode connectivity
(f) CKA similarity
E2 distance
Figure 2: (Standard setting). Partitioning the 2D load-like- terikk--eeeerrtuue--iiiiii diagram into different phases of learning, using batch size the temperature and varying model width to change load. Models are trained with ResNet18 on CIFAR-10. All plots are on the same set of axes. We note that batch size inverse temperature, and thus it has smaller values at the top of the y-axis and larger values at the bottom.
on this procedure, as well as an ablation study on different mode connectivity hyperparameters, in Appendix A.4.2 of the full paper (25)
Hessian. The Hessian at a given point Og parameter space represented by the matrix V2L(0)).
To summarize the Hessian single scalar value, we report the dominant eigenvalue Amax (2)(()))
and/or the trace tr( ((,)), calculated using the PyHessian software 11
l2 distance. We will also occasionally report the l distance between two parameter configurations 110 0' 2 as measure of similarity between models, although we typically find that the CKA similarity more informative measure.
Empirical results on taxonomizing local versus global structure
In this section, we present our main empirical results. Among other things, our results will highlight the presence of globally nice, globally well-commeced//ooolyccoomm landscapes, and the phase transitions which separate them. In addition to test accuracy, results on six other metrics presented, including training loss, leading Hessian eigenvalue, trace of Hessian, CKA similarity, mode connectivity, and & distance measured between model weights. For each metric, the results presented in 2D diagram, in which the horizontal dimension the load (with increasing load to the right), and the vertical dimension is the temperature (with increasing temperature to the top).
and locally flat/sharp loss
We will illustrate our main results in simple setting, and then consider several variants of this setting to illustrate how these results do or do not change when various parameters and design decisions modified. To start, we will consider ResNets (28) trained on CIFAR- 10 (29] as the standard setting to demonstrate different loss landscapes. We will scale the network width to change the size of the network. For ResNet18 which contains four major blocks with channel width (k, 2k, 4k, 8k), we select different values of to obtain ResNet models with different widths. In the standard setting, batch size, learning rate, and weight decay are kept constant throughout training to study interactions between temperature-like parameters, load-like parameters, and the loss landscape. Below, will apply learning rate decay and consider other variations of this standard setting, in separate experiments. More details on the experimental setup can be found Appendix B of (25).
3.1 Types of loss landscapes and phase transitions
In this subsection, we discuss our standard setting, in which we vary model width as the load-like parameter and batch size as the temperature-like parameter. A summary of the results is displayed in Figure Each pixel represents specific training configuration tuple (width, batch size), averaged over five independent runs. Observe that there are two phase transitions (identified by different metrics) that separate each plot into four primary regions (corresponding to those shown in Figure
5

---

Hessian distinguishes locally sharp versus locally flat loss landscapes. The first phase transition 22 2d] separating Phase I/II from Phase III/IV. larger Hessian eigenvalue or Hessian trace (darker color) represents a sharper local loss landscape [1 In Figure 26 we find this transition coincides with significant decrease in the training loss. Indeed, the training loss experiences more than tenfold reduction when transitioning from the upper side to the lower side on the right of the figure. Comparing Figures 2a and 2d|2d| categorizing loss landscapes based solely on the Hessian (or, from other results, other local flatness metries) is insufficient to predict test accuracy, e.g., the test accuracy in Phase III is lower than Phase IV-A but the Hessian eigenvalues are almost the same.
is displayed in Figure2ccand]
loss landscapes. The second phase transition is shown in Figure 2c The white region represents near- zero mode connectivity which, according to our definition, implies a flat curve in the loss landscape between
Mode connectivity distinguishes globally well-connected versus globally poorly-connest
weights; the blue region represents negative mode connectivity which implies a high barrier between weights; and the red region represents positive mode connectivity which implies a low-lost curve between weights, although the weights are not trained to reasonable optimum. The loss along individual mode connectivity curves can be found in Appendix A.5 of the full version [25]. In contrast to training loss, test accuracy only appears to show significant improvements after this transition. In particular, for well-connected loss landscapes, one can improve the test accuracy with suitable choice of temperature. This phase transition forms a curve separating Phase I from II, and separates Phase III from IV.
Based on the two transitions, we now classify the loss landscapes into the following phases.
Phase I: Globally poorly-connected and locally sharp: Training loss is high; Hessian eigenvalue
and trace are large; and mode connectivity is poor.
Phase I:: Globally well-connected and locally sharp: Training loss is high; Hessian eigenvalue and trace are large; and mode connectivity is poor because the trained weights fail to locate reasonable minimum.
Phase III: Globally poorly-connected and locally flat: Training loss is small; Hessian eigenvalue and trace are small: yct mode connectivity still remains poor. Phase IV: Globally well-connected and locally flat: Training loss is small; Hessian eigenvalue and trace are small; and mode connectivity is good (near-zero).
We remark that in Figure 2 (and subsequent figures below) the load-like and temperature-like parameters are on the X and Y axes, respectively, and we have, to the extent possible, kept other control parameters (in particular, those which are also load-like and temperature-like fixed, to isolate the effect of load-like and temperature- behavior on trained models. One might wonder (or even criticize our experimental setup, if one were not realize that we are trying to isolate the effects of load-like and temperature-like parameters) what would be the effect of varying learning rate (which is another temperature-like parameter) during the training process. Thus, we include the setting with decaying learning rate during training in Section 3.2
Here are two additional observations can make from Figure
CKA further distinguishes two subcategories in Phase IV. From Figure 21 CKA can be used to further divide Phase IV into Phase IV-A and Phase IV-B, with the latter exhibiting larger CKA similarity. We remark that the transition from Phase IV-A to Phase IV-B is more like a smooth crossover than sharp transition. Thus, we name both of them Phase IV.
Simple &g distance is not enough. challenge in measuring similarity between models that the same model can be realized using different weights (8) To reconcile this effect, the distance between two models is commonly defined in terms of their predictions instead weights. Indeed, the representationn-assed CKA similarity is seen to be preferable to the weights-based fy distance. For example, from Figure 2g the Eg distance provides some limited information, but it is not as informative as CKA similarity.
Based on these results, we assert the following central claim of this work: optimal test accuracy is obtained when the loss landscape is globally nice and the trained model converges to a locally flat region; and we can diagnose these different phases in the tead-iik-teepeeatuure-ikik phase diagram with Hessian, mode connectivity, and CKA metrics. Importantly, both similarity and connectivity metrics are required for a globally nice loss landscape. Phase IV-B precisely the region with globally nice landscapes, exhibiting the highest test accuracies.

---

Triniin lows
Los Flession eigenonlie
Log llessian froee
Test accuracy
Thatch sare
Bulch sioe
0.9
Phase
Batch size
0.8
10
10a
Phase IV-8
10
Width of model
Width model
whath model
0.7
(b) Training loss
(c) Hessian eigenvalue
(d) Hessian trace
10a
Phose IV-A
0.6
connectioly
C.A simlwtty
Kiddel distane
120
Betch oze
Butchssie
086ch
10
Width of model
(a) Test accuracy
100
10k
100
100
0.3
200
(c) Mode connectivity
(1) CKA similarity
Eg distance
Figure 3: (Learning rate decay). Partitioning the 2D load-like- tealte-tttmrrrtue--iiiee diagram into different phases of learning, varying batch size to change temperature and varying model width to change load. Learning rate decay is applied during training. Models are trained with ResNet18 on CIFAR-10. All plots are on the same of axes.
3.2 Corroborating results
In this subsection, we consider initial corroborating results, modifying the setup of Section 3.1 to train with learning rate decay, or to data with cxogenously-introuucee noisy labels, etc. Still more results can be found in Section 3.3dnd in the Appendix of the full paper (25]
Training with learning rate decay. Next, we consider a similar experimental setup and the same phase diagram, except with the same learning rate decay schedule applied in the middle of training rather than with a fixed learning rate throughout. We still vary batch size to change temperature. The results are presented in Figure Comparing Figure with Figure we see that the four phases are still present, and the test accuracy maximized when the loss landscape is globally nice and locally flat. Therefore, our central claim is unaffected by the learning rate decay schedule. In Figures 3c [d] smaller temperatures (or larger batch size) in Phase IV-A appear to increase the size of the Hessian. This is well-known issue with large-batch training (3). Finally, note that the optimal test accuracy achieved improves in the presence of learning rate decay.
Training to noisy labels and double descent. Next, we consider similar experimental setup and the same phase diagram, except that we randomize 10% of the training labels (similar to (0). The results are presented in Figure Comparing with Figure 2 we see that our main conclusion still holds, i.e., the loss landscape which both globally nice and locally flat achieves the best test accuracy, shown in Phase IV-B. However, an additional observation can be made: if we compare Figure 4a with Figure 2a] a "dark band" arises between different learning phases. particular, from that the test accuracy exhibits both width-wise and temperature-wii double descent 24 30), for certain parameter choices. In particular, the shape of the dark band matches that of
Figure 4a
the transitions shown in Figure and 4d
Double descent and phases of learning. The significance of this "dark band" is the following.
A central prediction when viewing different phases of optimization landscapes from a statistical
mechanics perspective [12 1 is that there should be "bad fluctuations" between qualitatively
different phases of learning (e.g., see the transition that separates Phase and II from Phase III and IV
Figure 44)). The connection between phases and fluctuations the popular double descent [21 22 was made precise in analyzable settings 23 24 Here, we complement 23 24 by exhibiting the
same type of transitions empirically between different phases in our taxonomy, and demonstrating that empirical double descent is consequence of qualitatively different phases of learning.
Training to zero loss. Next, we use Figure 4 to discuss whether to train (approximately) zero loss, which is popular in recent work. From Figure 4b we observe that Phase III and Phase IV achieve almost exactly zero loss, while Phase and Phase do not. Once again, the loss experiences a more than tenfold decay when transitioning from Phase I/II to Phase III/IV. However, if we restrict to globally poorly-connectee
regions and we restrict to a particular width value, i.e., selecting one column slice in the diagram that cuts through Phase III, such as the red block shown in Figure 4a we see that the best test accuracy obtained Phase I/II, instead of Phase III. Note that Phase / not only does not achieve zero loss, but it also has locally sharp minima (observed from Figure 4c and
7

---

Traniing lnss
Log Hessian eigencilal
Hossian trace
Tlbbtch staa)
Thbbrchh stoet
Test accuracy
Phase II
(batch size)
124
bar
101
10"-
0.7
300
300 Widh
Width model
Width
0.6
(b) Training loss
(c) Hessian eigenvalue
(d) Hessian trace
Mode connectivty
C-A simiiantty
Mocel distance (L2)
10
100
0.5
Toatch 500
T.atth 536)
Ub
130
Width model
100
(a) Test accuracy
-20
Wiith
Witth modal
Width
(c) Mode connectivity
(f) CKA similarity
f distance
Figure 4: (Training to noisy labels and double descent). Partitioning the 2D load-like -eeeeratuue like diagram into different phases of learning, using batch size as the temperature and varying model width to change load. 10% of labels are randomized, and double descent is observed between different phases. For an arbitrary column slice that through Phase III (e.g., the red block), optimal accuracy achieved Phase I/Il with locally sharp minima. Models are trained with ResNet18 on CIFAR-I 10.
All plots are on the same set of axes.
accuracy
Log
connectivity
similerity
Twwoight decavd
(wwo docay?
0.7
(weiggtt
30
10
102
10
15
100
100
0.3
Width model
Withh of model
Witth model
Width of model
(a) Test accuracy
(b) Hessian trace
(C) Mode connectivity
(d) CKA similarity
Figure 5: (Weight decay temperature). Partitioning the 2D load-like into different phases of learning, using weight decay as the temperature and varying model width to change load. Models are trained with ResNet18 on CIFAR-10. All plots are on the same set of axes.
load-liee-temperatuuue temperature-
diagram
4dp. This means that for globally
loss landscapes, it is possible that converging to locally flat region achieves lower accuracy than locally sharp region. More interestingly, this locally sharp region does not even converge to close-to-zero training loss. Thus, one will wrongly predict that Phase IIII outperforms Phase I/II if one only looks local sharpness.
3.3 Ablation study
Different temperature parameters. First, we study weight decay as an alternative temperature parameter, in addition to batch size. We change the temperature parameter from batch size used in Figure 2 to weight decay, and we report the results in Figure [ The results shown in Figure 5 are similar to those seen in Figure 2 One observation is that, once again, the best test accuracy is obtained when the loss landscape is both globally nice and locally flat. Another observation with Figure 5b is that when training a wide model with small weight decay (which is shown on the bottom of the figure), the Hessian trace becomes extremely small. This matches observations in [4] that decreasing weight decay reduces the size of the Hessian. Since weight decay is known to improve generalization, this also demonstrates that local metrics alone are insufficient to predict test performance.
Different amount of training data. Next, we vary the amount of training data (as another way of changing load) and see how that affects our results. We vary the number of training samples in CIFAR-10 by factor of ten. Results are shown in Figure Again, the optimal test accuracy achieved when the Hessian eigenvalue and trace are small, mode connectivity near-zero, and CKA similarity is large. Perhaps unsurprisingly. better test accuracy is achieved with more data. Here, CKA provides useful complementary information to the Hessian and mode connectivity for explaining the utility of larger data. The Hessian alone cannot predict the correct trend, as it increases

---

Number of samples
5000
10000
20000
30000
40000
50000
AUURIYY
Test accuracy
witn
Hessian trace
Wib strani
vitn steade
Wilh aimee
Vith
viin rval
Malt
Mode connectivity
celuaty
CKA similarity
Figure 6: (Varying amount of training data). Partitioning the 2D load-like- hew.lii---temmprruur---ii-ii diagram into different phases of learning, using batch size as the temperature and varying model width to change load. We vary quantities of training data from CIFAR-10 different columns. All plots are on the same set of axes.
Tesk accuracy
log Mosuen crecs
CCA am
T.wwohh soos
0.8
Random
Rancom perce stage
Random Isbel DeTENTIS
(c) Mode connectiv-
Randomm
(e) Hessian trace ver-
(a) Test accuracy
(b) Hessian trace
ity
(d) CKA similarity
sus batch size
Figure 7: (Proportion of randomized labels as load). Partitioning the 2D load-like load-liee-temperaaeee temperature- like diagram into different phases of learning, using batch size as the temperature and varying proportion of randomized training labels to change load. Models are trained with ResNet18 on CIFAR-10. 10. All plots on the same set of axes. (e) shows that the Hessian trace changes slowly with the proportion of noisy labels when training loss is small.
in magnitude with data. Mode connectivity alone cannot predict the correct trend either, becoming increasingly poor with larger data (see the shrinking white region). Indeed, it appears that larger models are required to keep the loss landscape well-connected with increasing data. In contrast, CKA precisely captures the relationship of increasing test accuracy with additional data.
These observations also imply that the utilities of extra data and larger models are different: larger models increase connectivity in the loss landscape (e.g., Figure 2e); while increasing data boosts signal the landscape, enabling trained models to become more similar to each other. Clearly, researchers have been increasing both the size of data and the size of models in recent years; methodology suggests obvious directions for doing this in more principled ways.
Different quality of data by changing the amount of noisy labels. Next, we vary the proportion of randomized labels to simulate the change the quality of data, as another way to change load. To generate randomized labels, a percentage of the training data is randomly selected and altered to an incorrect target class. Results are shown in Figure 7 Once again, local information alone fails to measure the quality of training data. We significantly affect the Hessian - see Figure 7b particular, as the temperature decreases (down to the bottom of Figure 7b). the Hessian becomes smaller, independent of the quantity of noisy labels. This is especially evident in Figure 7e where we plot Hessian trace against batch size. However, looking instead at mode connectivity Figure [ and CKA in Figure 7 training with more noisy labels leads to more poorly-connecteo loss landscapes.
that training with a large amount of noise does not
easily deduce that
Different datasets, architectures, load/temperature parameters, and training schemes. We have performed a wide range other experiments, only a subset of which we report here. These additional experiments can be found in the Appendix of the full paper (25). In Appendix D, cover additional

---

datasets, including SVHN, CIFAR-100, and IWSLT 2016 German to English (De-En) (a machine translation dataset), as well as additional NN architectures, including VGG11 I and Transformers. While there are many subtleties in such a detailed analysis (several of which point to future research directions), all experiments support our main conclusions. Here, we briefly summarize these results.
In Appendix D.3, we study an analogous plot to Figure training with 10% noisy labels but replacing the lemperature-like parameter from batch size to learning rate. Again, we observe the double descent phenomenon. Using this experiment, we infer that the decision to train to zero loss (traditionally a rule-of-thumb in computer vision tasks, although note that recent work has highlighted how the difference between exactly zero versus approximately zero can matter [II) should depend on the global connectivity of the loss landscape. Indeed, for small models with poor connectivity, we find that training to zero loss can harm test accuracy. This suggests that the common wisdom to fit training data to zero loss is derived from experiments involving relatively high-quality data and models, and is not a principle of learning more generally.
In Appendix D.4, we show that in the setting of machine translation, the loss landscape remains poorly- connected (i.e., the mode connectivity remains negative) even for a reasonably large embedding dimension up to 512 (see Figure 19). In this case, generalization can be quite poor when training to zero loss. This conclusion matches (with hindsight) the observations in practice, e.g., dropout and early stopping can improve test loss 33 33.. It also suggests that an embedding size of dimension 512 (for six-layer Transformers with eight attention heads used in our experiments) is still not large enough for baseline machine translation, and that certain (different) training schemes should be designed to improve the optimization on these loss landscapes.
alternative temperature parameter, which produces analogous results to Figure 2 In Appendix D.7.1, we study large-batch training and show that it increases local sharpness. Note that for most experiments, we intentionally keep a constant learning rate when varying the batch size to study the change in the landscape with a changing temperature; thus, in Appendix D.7.2, we provide additional results on tuning learning rate with changing batch size, including the commonly used "linear scaling rule" [34]
In Appendix D.5, we study learning rate
4 Conclusions
Motivated by recent work in the statistical mechanics of learning, we have performed a detailed empirical amalysis of the loss landscape of realistic models, with particular attention to how properties vary as load-like and temperature-lic like control parameters are varied. In particular, local propertics (such as those based on Hessian) are relatively easy to measure; and while more global properties of a loss landscape are more challenging to measure, we have found success with a combination of similarity metrics and connectivity metrics. This complements recent work that uses tools from statistical mechanics and heavy-tailed random matrix theory, as we can perform large-scale empirical evaluations using metrics (CKA, mode connectivity, Hessian eigenvalues, etc.) that more familiar to the ML community. We interpreted these metrics in terms of connectivity and similarity, and we used them obtain insight into the local versus global properties of NN loss landscapes.
Here, we summarize a few observations from our connectivity and similarity plots (that we expect will be increasingly relevant as larger data sets and models are considered). i) A larger width improves mode connectivity; ii) more data improves similarity; iii) better data quality improves mode connectivity; iv) a larger width and a higher temperature in Phase IV improves similarity. These observations can be restated in a way to guide the operations in training:
A negative mode connectivity suggests that the data quality is low the model size is small. A large positive mode connectivity or a large Hessian leading eigenvalue/trace indicates that the training fails converge to the bottom of a local minimum.
A small CKA similarity suggests that generalization is not good, which can be caused by various factors. However, if, in addition, the mode connectivity is close zero, and the Hessian small, a large CKA similarity indicates the "lack of signal" from the data. In other words, one should get more nigh-quality data for training.
In future work, we aim to provide more detailed study on using the metrics for improved training, and we will look at phase diagrams outside of the load/temperature form, especially in the low- connectivity regime, which is most challenging according to our taxonomy.
10

---

Acknowledgements. We want to thank Charles Martin, Rajiv Khanna, Zhewei Yao, and Amir Gholami for helpful discussions. Michael W. Mahoney would like acknowledge the UC Berkeley CLTC, ARO, IARPA (contract W91 W NF20C0035) NSF, and ONR for providing partial support of this work. Kannan Ramchandran would like to acknowledge support from NSF CIF-2007669, CIF-1703678, and CIF-2002821. Joseph E. Gonzalez would like to acknowledge supports from NSF CISE Expeditions Award CCF-1730623 and gifts from Alibaba Group, Amazon Web Services, Ant Group, CapitalOne, Ericsson, Facebook, Futurewei, Google, Intel, Microsoft, Nvidia, Scotiabank, Splunk and VMware. Our conclusions do not necessarily reflect the position or the policy of our sponsors, and no official endorsement should be inferred.
References
[1] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Conference on Neural Information Processing Systems, pages 6389-6399 2018.
[2] Andrew J Ballard, Ritankar Das, Stefano Martiniani, Dhagash Mehta, Levent Sagun, Jacob D Stevenson, and David Wales. Energy landscapes for machine learning. Physical Chemistry Chemical Physics, 19(20): 15885-22033 2017.
[3] Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017.
[4] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Large batch size training of neural networks with adversarial training and second-orde information. Technical Report Preprint: urXiv:1810.0102.. 2018.
[5] Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian- -ased analysis of large batch training and robustness to adversaries. In Conference on Neural Informa- fion Processing Systems, volume 31, pages 4949-4959, 2018.
[6] Yuanzhi Li and Yang Yuan. Convergence analysis of two-layer neural networks with ReLU activation. In Conference on Neural Information Processing Systems, pages 597-607, 2017.
[7] Shibani Santurkar, Dimitris Tsipras. Andrew Ilyas, and Aleksander Mądry. How does batch normalization help optimization? In Neural Information Processing Systems, pages 2488-2498, 2018.
[8] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima generalize for deep nets. In International Conference on Machine Learning, pages 1019-1028, 2017.
[9] Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC-Bayesian approach to ppectrally- margin bounds for neural networks. In International Conference on Learning Representations, 2018
[10] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mini- mization for efficiently improving generalization. arXiv preprint arXiv:2010001412,, 2020.
[11] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael Mahoney. PyHessian: Neural networks through the lens of the hessian. In IEEE International Conference on Big Data (Big Data), pages 581-590, 2020.
[12] Andreas Engel and Christian P. L. Van den Broeck. Statistical mechanics of learning. Cambridge
University Press. New York, NY, USA, 2001.
[13] Charles H Martin and Michael W Mahoney. Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior. Technical Report Preprint:
2017.
[14] Diego Granziol. Flatness a false friend. Technical Report Preprint: arXiv:2006.0900 099999 2020.
[15] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations, 2019.
[16] Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165).1-73, 2021.

---

[17]
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Conference on Neural Information Processing Systems, pages 8803-8812, 2018.
[18) Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pages 1309-1318, 2018.
[19] Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, pages 3519-3529, 2019.
[20] Madhu Advani, Subhaneil Lahiri, and Surya Ganguli. Statistical mechanics of complex neural systems and high dimensional data. Journal of Statistical Mechanics: Theory and Experiment, 2013(03):P03014, 4, mar 2013. doi: 10. (1188/17772554622200000000030000 org/10. 1088/ 1741114422211221 54085246832333 00/ p03014
URL httphhttt///do.
[21] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias variinee trade-off. Proceedings of National Academy of Sciences, 116(32)115849-11 158544, 2019.
[22] Mikhail Belkin, Daniel Hsu, and Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science 2(4): 2020.
[23] Zhenyu Liao, Romain Couillet, and Michael W Mahoney. A random matrix analysis of random fourier features: beyond the Gaussian kernel, a precise phase transition, and the corresponding double descent. In Conference on Neural Information Processing Systems, 2020.
[24] Michat Derezinski, Feynman Liang, and Michael W Mahoney. Exact expressions for double descent and implicit regularization via surrogate random design. In Conference on Neural Information Processing Systems, volume 33, 2020.
[25] Yaoqing Yang, Liam Hodgkinson, Ryan Theisen, Joe Zou, Joseph E Gonzalez, Kannan Ram- chandran, and Michael W Mahoney. Taxonomizing local versus global structure in neural network loss landscapes. Technical Report Preprint: arXiv:2107.11222 2021.
[26] Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste- Julien, and loannis Mitliagkas. A modern take on the bias-variance tradeoff in neural networks. Technical Report Preprint: arXiv: 18 1111110.0000 10.00055111 2018.
[27] Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[30] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2019.
[31] Charles H Martin and Michael W Mahoney. Post- Post-mortem on deep learning contest: Simpson's paradox and the complementary roles of scale metrics versus shape metrics. Technical Report Preprint: arXiv:2106.000334 2021.
[32] Antonio Valerio Miceli-Barone Barry Haddow, Ulrich Germann, and Rico Sennrich. Regular- ization techniques for fine-tuning in neural machine translation. In Conference on Empirical Methods in Natural Language Processing, pages 1489-1494, 2017.
[33] Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In Conference on Neural Information Processing Systems, volume 29, pages 1019-1027, 2016.
[34] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training 2017.
ImageNet in I hour. Technical Report Preprint:
12


Reviews:
[{"review_id": "zmGUnRqEJTu", "reviewer": "Reviewer_EVPJ", "summary": "The paper proposed an experiment framework for evaluating loss landscape of neural networks during training. The experimental setup systematically vary two types of hyperparameters: 1) temperature type, where at high temperatures, larger learning rate or smaller batch size will induce more noise to training, and 2) load type, including data quantity/quality and model width. Neural networks are trained with different hyperparameter combinations according to the load-temperature pairs. Then 3 metrics including mode connectivity, Hessian, and output CKA similarity evaluated on perturbed training data. Jointly consider the 3 metrics (local and global) evaluated on all loss landscapes, the authors classify the loss landscapes into different phases. One of the phases (IV with high connectivity, low Hessian and high CKA) corresponds to best test accuracy. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper jointly considers three types of metrics that were considered in literature to evaluate both global and local characteristics of loss landscapes. It explains some of the counter-intuitive phenomena observed such as why in some low hessian loss landscape, the model does not generalize well. Experiments were run with 3 types models (ResNet, VGG and Transformer) on different data sets. However comparing the results figures, ResNet vs Transformer, it is not clear if the defined phases are universally useful. On the other hand, the effect of optimizer was not discussed. \nOverall, the paper is well written.\n\nMinor comments:\nIn Figure 4, when evaluating ‘global nice’, global connectivity seems the only thing that matter, as highest accuracy is at 0 mode connectivity and low (but not lowest) Hessian.  \nFigure 3 should the y axis label be learning rate instead of  batch size, since learning rate is used as temperature? Otherwise, what do the ‘batch size’ in both subtitle and axis labels mean? The figure captions need to be more clear.\n\nThe authors clarified how to interpret the results and made corrections to figures in their responses. I am more convinced. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "_Y87L5zfbKq", "reviewer": "Reviewer_qCke", "summary": "This paper takes a large number of neural networks that span range of different parameters (e.g. width), train them in a large number of ways (e.g. dataset, batch size), and studies several metrics that characterize the local (e.g. Hessian spectral norm) and global (weight vector pair-wise connectivity on potential non-linear paths). By doing a large scale study of this kind, the authors are able to identify several distinct phases of loss landscapes, dividing them phenomenologically into (globally {well,poorly}-connected) x (locally {flat, sharp}). The authors also develop a simple mathematical toy model that can model this taxonomy. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I really like this paper. I am very happy that the a large scale study of this kind has been done and that the different local and global aspects of loss landscapes of DNNs have been investigated in this way. I enjoyed reading the paper, the message is clear, and so are the experiments. I have a number of points of confusion and clarifications but overall I'm a fan! Even if the paper potentially doesn't get accepted I'd strongly suggest the authors put it on arXiv so that the community can benefit from their results.\n\nSTRONG POINTS:\n1) a clear theoretical backing of the questions\n2) a well written paper\n3) looking at both local and global aspects of the loss landscape (the global aspects are often understudied)\n4) a nice mathematical toy model allowing us to conceptualize the taxonomy the authors present\n\nWEAKER POINTS AND QUESTIONS:\n\n1) In your experiments you vary a lot of parameters such as the network width, batch size, etc. Once you change a hyperparameter, the optimal value of the other hyperparamters also changes. This is certainly the case when you're spanning orders of magnitude as you do. In your results and plots, is each datapoint the optimal set of hyperparameters for that specific choice of the controlled hyperparameters, or are all of them kept constant. For example, let's say you have batch size, learning rate, width, and L2 regularization, and let's say you're looking at mode connectivity vs the learning rate. Are you, for a specific choice of learning, choosing a new, optimal (through a grid search let's say) values of (batch size and L2 regularization). If not, this could be a serious confounder for your experimental conclusions. I think this point as the key question I have that needs to be addressed.\n\n2) In the introduction, you mention that the best test accuracy is achieved when the ensembles of trained models are more similar to each other. This seems to go both against 1) the received wisdom that ensembles work well because of function diversity, 2) many theoretical results in this direction, and 3) empirical work on this (for example see https://arxiv.org/abs/1912.02757 which might be a relevant reference for you as well, since they also deal with mode connectivity a lot). How do you reconcile your conclusion with the previous results?\n\n3) Your toy model is one dimensional? Some papers argue that it's the high dimensional nature of the loss landscape that plays a key role in forming many of its properties. For example https://arxiv.org/abs/1906.04724 (NeurIPS'19) looks at a similar question to you but with a distinctly high-D lens. Does the predictive power of your toy model depend on its dimension, or are you just trying to capture the relevant phenomenology with 1-D?\n\n4) Bigger and better models? The results you show are on the weaker side of CIFAR-10 test accuracies. In Figure 2 it seems that the color scale doesn't even go to 90% test acc. It might be better to add stronger, better optimizer models, and at least CIFAR-100 as well to make the case for your taxonomy stronger.\n\n5) I am a bit worried that the large \"phase changes\" you see in e.g. Figure 2 Panel e) are caused by the really bad losses you have there. You use cross entropy loss on 10 classes, so the default value should be ln(10) ~ 2.3. But your barriers are often of the order ~10. How is that possible? And would the transitions look as sharp if you didn't have these really high losses to compare to?\n\nRELEVANT PAPERS\nI also thought of a few more relevant papers. \n- On large scale structure of loss landscapes: https://arxiv.org/abs/1906.04724\n- On ensembles from the loss perspective: https://arxiv.org/abs/1912.02757\n- On linear mode connectivity and even init - optimum connectivity from an unrelated init: http://proceedings.mlr.press/v139/lucas21a.html\n- On the large scale structure of the Tr(H): https://arxiv.org/abs/1807.02581\nFeel free to add them if you see that they are relevant, but no pressure for sure.\n\nOverall I really like this paper! It could be made better by potentially addressing the suggestions I made above, but as it is it's certainly valuable to the community and should get published one way or another (and arXived for sure as soon as the authors see fit).\n\nUPDATE AFTER REBUTTAL:\nI am sorry for being slightly late into the reply game. I am very happy with the answers the authors provided and would like to keep my score of 7 = accept. I think this paper is a valuable contribution to the subfield. Good job!", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "QavPQF_N9mm", "reviewer": "Reviewer_guMx", "summary": "The author(s) perform an empirical study of which (and how) properties of the landscape affect the test accuracy, obtaining results that are consistent over a large number of deep neural network models over several credible datasets. The study puts into practice this idea that the generalization is not uniquely determined by local properties of the landscape, but rather also by global properties. They argue that different kinds of acting on the system and on the dynamics can be grouped into two control parameters that they call 'load' and 'temperature', and show that according to the position in the (load, temperature) plane one has different phases. They observe that each phase seems related to a different test accuracy, and realize that to obtain the best test accuracy, in addition to being locally flat, the landscape should also be globally \"nice\". By \"nice\", they mean that the landscape is well-connected (no large barriers separating regions) and that different trained models have a large CKA similarity.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses a long-due question, trying to explain that local flatness cannot be used as a sole indicator of a good minimization. They address this question numerically, through large-scale experiments over a wide range of parameters. The results seem quite robust across architectures and datasets. The main message of the paper is simple and compelling, and will likely have a good resonance throughout the community.\n\nI do have a series of concerns and comments, however, that I would like the authors to address.\n\nI will mostly refer to Fig.2, but many of the comments apply to several figures.\n\nDespite being conceptually very simple, the results along the temperature axis are extremely confusing. This is  due to the way labels, axes and captions are presented. The y axis of Fig.2 is labeled \"temperature (batch size)\". However, the batch size is the inverse temperature. So, is the quantity on the y axis the batch size or its inverse? Further, the direction of the axis is reversed (bigger values are below), which makes everything even more confusing. Finally, it took me a long time before I realized that the axis was inverted, because the fonts are so small that they can bearly be read with a 200% zoom (and impossible to read when printed).\nSo what is the quantity on the y axis? Is zero temperature at the top or bottom (an intuitive figure would put it at the bottom)? I think that in the y axis temperature goes upwards and batch size goes downwards. This makes sense but it took me quite a while to understand the figure, and the reader should be able to grasp it immediately.\nThe captions and labels should clearly explain what is plotted, and help the reader in the interpretation.\nWhat do the subplot titles in Fig2 mean? I read batch size, and I assume that it is because the temperature parameter here is the batch size. If so, this is redundant (thus confusing), since it is already written on the y axis. However, the same \"batch size\" labels appear in figure 3, where the caption says that this figure is \"using learning rate as temperature\". Yet further confusion is caused by the main text, which instead states that a learning rate decay was used. In this case, the authors rightly state that the temperature is not well-defined. Besides the contradiction with the figure caption, I assume then that what the authors are stating is that the y axis stands for the initial temperature value. This independence from the learning rate is interesting, though I would be more conservative, because in many complex systems the rate of annealing (how fast the learning rate decreases) can be crucial to this effect [].\n\n\nI do not understand why Fig.2e reveals a transition on a vertical line. By looking at the figure I see a vertical line in the bottom part of the plot, but this line then curves and possibly even splits. How are the authors identifying this phase transition?\n\nThe authors talk of phases, but the word phase has a specific meaning related to a non-analyticity in a (free) energy functional. For sharpness and connectivity a phase transition seems possible (actually, it was shown in https://arxiv.org/abs/1809.09349 that the one along the load axis is actually a phase transition), but the one from phase IV-A to phase IV-B seems more a smooth crossover, since I see no sharp change in the CKA similarity nor in any other observable. Am I right?\n\nIt seems to me that the CKA similarity alone is enough to explain the test accuracy (the two quantities correlate extremely well), without the need of the other metrics. Can the authors elaborate on this?\n\nIn all shown examples, the interpolation threshold (where the training loss becomes zero) seems to be at a higher load than the MC connectivity. From Ref https://arxiv.org/abs/1809.09349, which identifies a jamming transition at the interpolation threshold, I would have expected the two to coincide. Can the authors say anything on this? Can it be due to the way the MC was computed (both because of the method and the k=2 choice)?\n\nIn the supplemental material the authors state that most experiments were run until convergence, but they stopped at 150 epochs when the convergence criterion was not met. I think that this can influence the results, so I would least specify in the main text which runs did not reach convergence. Does this apply to the runs (and only those) with a negative mode connectivity?\n\nThe authors talk about \"poor\" mode connectivity, but this term is not defined. Is a poor MC positive? Is it non-zero (so also negative)?\n\nLine 206. The sentence is not clear, also due to the \"poor\" term.\n\nThe results are shown for VGG and Resnets. I wonder whether they also hold for simpler models such as multi-layer perceptrons and simply convolutional networks, since these models are more tractable analytically. Moreover, can the authors say anything about linear models?\n\nWould the authors have found equivalent results if they studied the mode connectivity through the Nudged Elastic Band method, as done in http://proceedings.mlr.press/v80/draxler18a/draxler18a.pdf?\n\nFruitful examples of the study of loss landscapes can be found in https://www.annualreviews.org/doi/pdf/10.1146/annurev-conmatphys-031119-050745, from random landscapes, to jamming, to spin glasses. Ref http://proceedings.mlr.press/v80/draxler18a/draxler18a.pdf (cited elsewhere in the text) is a great example.\nWhen mentioning sharp vs flat minima, I would have expected a citation to https://arxiv.org/abs/1609.04836.\nThe concept of temperature is not new as well as the interpretation in terms of a Gibbs distribution. See e.g. https://arxiv.org/pdf/1711.04623.pdf and references therein.\nThe concept of load is not new either (except for the name).\n\nOn line 295 they can cite e.g. \n- Q. Li, C. Tai, and E. Weinan, Stochastic modified equations and adaptive stochastic gradient algorithms,inInternational Conference on Machine Learning(PMLR, 2017) pp. 2101–2110.\n- W. Hu, C. J. Li, L. Li, and J.-G. Liu, On the diffusion approximation of nonconvex stochastic gradientdescent, Annals of Mathematical Sciences and Applications4(2019), arXiv:1705.07562.\n- Y. Zhang, A. M. Saxe, M. S. Advani, and A. A. Lee, Energy–entropy competition and the effectivenessof stochastic gradient descent in machine learning, Molecular Physics116, 3214 (2018)\n\nAnyhow, I am not sure that section 4 adds any knowledge at all. The authors have a picture in their mind, and they draw it schematically, in one way out of infinitely many that they could have chosen. It might even be wrong, since it assumes the existence of only two relevant scales of ordering (local and global), while they could be infinite (e.g. with spin glasses, jamming, ...). The concept of effective landscape already exists, and it is called Free Energy. This quantity comprehends the competition between energy (loss) and thermal agitation, it can in principle be obtained by legendre-transforming the energy (= the loss), and was studied for decades (one reference is the just given Zhang et al paper, but we can even resort to papers from the 80s from Gardner and coworkers). \n\nThere is something awkward with the batch indices at line 116. The concept is simple and clear, but the authors introduce some names of indices which it is not clear to what they correspond unless one already knows what the authors mean.\n\nLine 162: which typically occurs if training failed to locate a reasonable optimum. This statement is true for a strict inequality, but the authors are referring it to mc<=0. \n\nLine 338: \"we provide theoretical explanations\". I disagree. The paper is purely empirical and this is fine.\n\nLine 344: the statement seems naif.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One limitation that I see is that the temperature and load axes are (perhaps deliberately) defined in a fuzzy manner. They do this to emphasize that different drivers have qualitatively similar effects. I am not sure that all the drivers that are collected under the same umbrella actually represent the same kind of behavior (e.g. weight decay as temperature). I don't see this as a problem, since these issues fall out of the scope of the paper and can be addressed and understood in future work.\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "HeTofdZNmT", "reviewer": "Reviewer_4CJS", "summary": "This paper explores factors that influence the loss landscape. In particular, it categorizes the loss landscape in different phases, each corresponding to different status of key factors such as Hessian spectrum and mode connectivity. It also tries to establish an \"effective loss landscape\" model that depends on key parameters.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Exploring key factors that affect the global landscape and training has been a hot research area. Many work has focused on the influence of one particular factor (such as Hessian, connectivity, etc). The advantage of this work is that it combines several factors together and looks at the comprehensive influence. I believe a single factor is not enough to explain all about landscape and training, and a phase diagram is a good trial along multi-factor direction. Another interesting point is the category of load and temperature-like parameters. It seems that different load/temperature-like parameters have similar effect on the landscape. This may help unify the influence of different hyper-parameters.\n\nThe following are my questions and concerns about this paper.\n\n1. This work shows the phase transition of some key parameters. What is the cause of the transition, and is it possible that these parameters are still intermediate parameters that depend on some other key factors?\n\n2. Hessian spectrum and CKA similarity perform similarly on most results but different in a few. What is the essential difference between these two factors, and what standard should we choose if we have to choose one?\n\n3. Is there any empirical guidance this phase diagram can provide? The empirical results show the landscape and key parameters at the end of the training. Is it possible that we acknowledge the current phase during early iterations?\n\n4. I'm a bit confused about the effective loss landscape. I'd like to see more about why defining effective loss landscape is reasonable (and necessary) for observing the influence of temperature and load.\n\n5. The explanation of (4) when discussing mode connectivity is contradictory in the main paper and the appendix. In the main paper, it claims that there is a barrier if $mc>0$, while in the appendix both the description and the figures show that there is a barrier when $mc<0$.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " The authors addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates how the structure of the loss landscape of neural networks affects their generalization performance.  The authors categorize loss landscapes into four different phases, globally {well,poorly}-connected) x (locally {flat, sharp}), where global connectivity is measured by mode connectivity and local flatness is measured using Hessian. Through a large-scale empirical investigation of neural network loss landscapes, the authors present an analysis of how these categories affect generalization.

Overall, I think this is a solid paper. The reviewers provided a lot of useful detailed feedback and the authors also did a great job of addressing the reviewer concerns. During the discussion phase, the consensus decision leaned towards acceptance. 

I recommend acceptance and encourage the authors to address the reviewer comments in the final revision.


Decision: Accept (Poster)