Entropic gradient descent algorithms and wide flat minima
Authors: Fabrizio Pittorino
Carlo Lucibello
Christoph Feinauer
Gabriele Perugini
Carlo Baldassi
Elizaveta Demyanenko
Riccardo Zecchina
Venue: ICLR 2021 Poster (2021)
Keywords: flat minima
entropic algorithms
statistical physics
belief-propagation

Abstract: The properties of flat minima in the empirical risk landscape of neural networks have been debated for some time. Increasing evidence suggests they possess better generalization capabilities with respect to sharp ones. In this work we first discuss the relationship between alternative measures of flatness: The local entropy, which is useful for analysis and algorithm development, and the local energy, which is easier to compute and was shown empirically in extensive tests on state-of-the-art networks to be the best predictor of generalization capabilities. We show semi-analytically in simple controlled scenarios that these two measures correlate strongly with each other and with generalization. Then, we extend the analysis to the deep learning scenario by extensive numerical validations. We study two algorithms, Entropy-SGD and Replicated-SGD, that explicitly include the local entropy in the optimization objective. We devise a training schedule by which we consistently find flatter minima (using both flatness measures), and improve the generalization error for common architectures (e.g. ResNet, EfficientNet).

Published as a conference paper at ICLR 2021
ENTROPIC GRADIENT DESCENT ALGORITTMM AND WIDE FLAT MINIMA
Fabrizio Pittorino 1 Carlo Lucibello', Christoph Feinauer's Gabriele Perugini', Carlo Baldassi', Elizaveta Demyanenko', Riccardo Zecchina'
'AI Lab, Institute for Data Science and Analytics, Bocconi University, 20136 Milano, Italy
2 Dept. Applied Science and Technology, Politecnico di Torino, 10129 Torino, Italy
ABSTRACT
The properties of flat minima in the empirical risk landscape of neural networks have been debated for some time. Increasing evidence suggests they possess better generalization capabilities with respect to sharp ones. In this work we first discuss the relationship between alternative measures of flatness: The local entropy, which is useful for analysis and algorithm development, and the local energy. which is easier to compute and was shown empirically in extensive tests state-of-the-art networks be the best predictor of generalization capabilities. We show semi semi-analytically in simple controlled scenarios that these two measures correlate strongly with each other and with generalization. Then, we extend the analysis to the deep lcarning scenario by extensive numerical validations. We study two algorithms, Entropy-SCh local entropy in the optimization objective. We devise a training schedule by which we consistently find flatter minima (using both flatness measures), and improve the generalization error for common architectures (e.g. ResNet, EefiiieetNett.
and Replicated-SGD, that explicitly include the
1 INTRODUCTION
The geometrical structure of the loss landscape of neural networks has been a key topic of study for several decades (Hochreiter & "Schmidhuberr 1997 Keskar et al. 2016). One area of ongoing rescarch is the connection between the flatness of minima found by optimization algorithms like stochastic gradient descent (SGD) and the generalization performance of the network (Baldassi et al.] 2020 Keskar et al. 2016). There are open conceptual problems in this context: On the one hand, there is accunulating evidence that flatness a good predictor of generalization (Jiang el al. 2019) On the other hand, modern deep networks using ReLU activations are invariant in their outputs with respect to rescaling of weights in different layers (Dinh et al. 2017). which makes the mathematical picture complicated! General results are lacking. Some initial progress has been made in connecting PAC- PAC-Bayes bounds for the generalization gap with flatness (Dziugaite & Roy 2018).
The purpose of this work is to shed light on the connection between flatness and generalization by using methods and algorithms from the statistical physics of disordered systems, and to corroborate the results with a performance study on stale-of-thhearl deep architectures.
Methods from statistical physics have led to several results in the last years. Firstly, wide flat minima have been shown to be a structural property of shallow networks. They exist even when training on random data and are accessible by relatively simple algorithms, even though coexisting with exponentially more numerous minima (Baldassi et al. 2015 2016a
We note, in passing, that appropriate framework for theoretical studies would be to consider networks with binary weights, for which most ambignities are absent.
1

---

Published as a conference paper at ICLR 2021
2020). We believe this to be an overlooked property of neural networks, which makes them particularly suited for learning. In analytically tractable settings, it has been shown that flatness depends on the choice of the loss and activation functions, and that it correlates with generalization (Baldassi et al. 2020 2019).
In the abovo-mentioned works, the notion of flatness used was the so-called local entropy (Baldassi et al. 2015 2016a). It mcasures the low-loss volume in the weight space around a minimizer, as function of the distance (i.e. roughly speaking it measures the amount of "good" configurations around a given one). This framework is not only useful for analytical calcuations, but it has also been used to introduce a variety of efficient learning algorithms
that focus their search on flat regions (Baldassi et al. 2016a; Chaudhari et al. 2019 2017 In this paper we call them entropic algorithms.
A different notion of flatness, that we refer to as local energy in this paper, mcasures the average profile of the training loss function around a minimizer, as a function of the distance (i.e. it measures the typical increase in the training error when moving away from the minimizer). This quantity is intuitively appealing and rather easy to estimate via sampling, even in large systems. In Jiang et al. (2019), several candidates for predicting generalization performance were tested using an extensive numerical approach on an array of different networks and tasks, and the local energy was found to be among the best and most consistent predictors.
The two notions, local entropy and local energy, are distinct: in given region of complex landscape, the local entropy measures the size of the lowest valleys, whereas the local energy measures the average height. Therefore, in principle, the two quantities could vary independently. It seems reasonable, however, to conjecture that they would be highly correlated under mild assumptions on the roughness of the landscape (which is another way to say that they are both reasonable measures to express the intuitive notion of "flatness").
In this paper, we first show that for simple systems in controlled conditions, where all relevant quantities can be estimated well by using the Belief Propagation (BP) algorithm (Mezard & Montanari (2009)), the two notions of flatness are strongly correlated: regions high local entropy have low local energy, and vice versa. We also confirm that they are both correlated with generalization.
This justifies the expectation that, even for more complex architect and datasets, those algorithms which are driven towards high-local-entropy regions would minimize the local energy too, and thus (based on the findings in Jiang et al. (2019)) would find minimizers that generalize well. Indeed, we systematically applied two entropic algorithms, Entropy-SGD (eSGD) and Roplicated-SGD (rSGD), to satto-of-tho-at deep architectures, and found that we could achiove an improved generalization performance, at the same computational cost, compared to the original papers where those architectures were introduced. We believe these results to be an important addition to the current state of knowledge, since in (Baldassi el al. (016b)) iSGD was applied only to shallow networks with binary weights trained on random patterns and the current work represents the first study of rSGD in a realistic deep neural network setting, Together with the first reported consistent improvement of eSGD over SGD on image classification, these results point to very promising direction for further rescarch. While we hope to foster the application of entropic algorithms by publishing code that can be used to adapt them easily to new architectures, we also believe that the numeric results are important for theoretical research, since they are rooted in a well-defined geometric interpretation of the loss landscape.
We also confirmed numerically that the minimizers found in this way have a lower local energy profile, expected. Remarkably, these results go beyond even those where the eSGD and rSGD algorithms were originally introduced, thanks to a general improvement in the choice for the lcarning protocol, that also discuss; apart from that, we used little no parameer tuning.
2

---

Published as a conference paper at ICLR 2021
2 RELATED WORK
The idea of using the flatness of a minimum of the loss function, also called the falness of the posterior and the local area estimate of guality, for evaluating different minimizers is several decades old Hochreiter & Schmidhuber 1997 Hinton & van Camp 1993 Buntine & Weigend 1991p. These works connect the Haatness of minimum to information tion theoretical concepts like the minimum description length of its minimizer: flatter minima correspond to minimizers that can be encoded using fewer bits. For neural networks, a recent empirical study (Keskar et al. 2016) shows that. large-batch methods find sharp minima while sinall-bacch ones lind Natter ones, with positive effect on generalization performance.
PAC-Bayes bounds can be used for deriving generalization bounds for neural networks (Zhou et al. 2018). In Dziugaite & Roy (2017), method for optimizing the PAC-Bayes bound directly is introduced and the authors note similarities between the resulting objective
function and an objective function that searches for flat minima. This connection is further analyzed in Dziugaite & Roy (2018).
In Jiang et al. (2019), the authors present large-scale empirical study of the correlation between different complexity measures of neural networks and their generalization perfor- mance. The authors conclude that PAC-Bayes bounds and flatness measures (in particular, what we call local energy in this paper) are the most predictive mcasures of generalization.
The concept of local entropy has been introduced in the context of statistical mechanics approach to machine learning for discrete neural worksrks in Baldassi et al. (2015), and subsequently extended to models with continuous weights. We provide detailed delinition in the next section, but mention here that it measures a volume in the space of configurations, which poses computational difficulties. On relatively tractable shallow networks, the local entropy of any given configuration can be computed efficiently using Belief Propagation, and it can be also used directly as a training objective. In this setting, detailed analytical studies accompanied by numerical experiments have shown that the local entropy correlates with the generalization error and the eigenvalues of the Hessian (Baldassi et al. 2015 2020). Another interesting finding is that the cross-entropy loss (Baldassi et al. 2020 and ReLU transfer functions [Blldassi et al. 2019), which have become the de-facto standard for neural networks, tend to bias the models towards high local entropy regions (computed based on the error loss).
Extending such techniques for general architectures is open problem. However, the local entropy objective can be approximated to derive general algorithmic schemes. Replicated stochastic gradient descent (rSGD) replaces the local entropy objective by an objective involving several replicas of the model, each one moving in the potential induced by the loss while also attracting each other. The method has been introduced in Baldassi el al.] (2016a), but only demonstrated on shallow networks. The rSGD algorithm is closely related to Elastic Averaging SGD (EASGD), presented in Zhang et al. (2014), even though the latter was motivated purely by the idea of enabling massively parallel training and had no theoretical basis. The substantial distinguishing feature of rSGD compared EASGD when applied to deep networks is the focusing procedure, discussed in more detail below. Another difference is that in rSGD there is no explicit master replica.
Entropy-SGD (eSGD), introduced in Chaudhari et al. 2019 is a method that directly optimizes the local entropy using stochastic gradient Langevin dynamics (SGLD) (Welling & Teh 2011). While the goal of this method is the same as rSGD, the optimization techniques involves a double loop instead replicas. Parle (Chaudhari et al. 2017), combines eSGD and EASGD (with added focusing) to obtain a distributed algorithm that shows also excellent generalization performance, consistently with the results obtained in this work.
FLATNESS MEASURES: LOCAL ENTROPY, LOCAL ENERGY
The general definition of the local entropy loss LLE for a system in a given configuration w (a vector of size N) can be given in terms of any common (usually, data-dependent) loss L
3

---

Published as a conference paper at ICLR 2021
1 LL (w) = 3 log I du'
(1)
The function measures distance and is commonly taken to be the squared norm of the difference of the configurations w and w':
1 2
d(w', w)
(w) w;)
(2)
The integral is performed over all possible configurations w'; for discrete systems, it can be substituted by a sum. The two paramcters 3 and 9 = By are Legendre conjugates of the loss and the distance. For large systems, N > 1, the integral is dominated by configurations having a certain loss value C* (w, 3.7) and a certain distance d (w, 5,7) from the reference configuration w. These functiinal dependencies can be obtained by saddle point approximation. In general, increasing & reduces S and increasing 7 reduces d"
While it is convenient to use Eq- 1 theoretical analysis of shallow networks, it is more natural to use a normalized definition with explicit parameters when we want measure the flatness of minimum. We thus also introduce the normalized local entropy PLE (w, d), which, for given configuration w E R', measures the logarithm of the volume fraction of configurations whose training error is smaller or equal than that of the reference in a ball of squared-radius 2d centered in w:
objective function in algorithms and for the
log I dw' θ (Eerain (w) Errain (w))) d
PE (w, d)
(3)
ddw' θ (d - (w', w))
Here, Eraain (w) is the error on the training set for a given configuration w and O (x) is the Heaviside step function, O (x) = 1 if by zero and tends to zero for d + 0 (since for almost any except for a set with mull measure, there is always sufficiently small neighborhood in which Errain is constant). For sharp minima, it is expected to drop rapidly with d, whereas for flat regions it is expected to stay close zero within some range.
and 0 otherwise. This quantity upper-bounded
A different notion of flatness is that used in Jiang et al. (2019), which we call local energy. Given a weight configuration w E RR, we define Berain (w, o) as the average training error difference with respect to Furai(w) when perturbing by (multiplicalive) noise proportional a parameter a:
& (w, σ) E, Euain az w) Ewriinw)),
(4)
where O denotes the Hadamard (element-wise) product and the expectation is over normally distributed 2 ~ NO, Ix). In Jiang et al.] (2019), a single, arbitrarily chosen value of a was used, whereas compute entire profiles within some range [0, Tmax in all our tests.
4 ENTROPIC ALGORITTMS
For our numerical experiments we have used two entropic algorithms, rSGD and eSGD, mentioned in the introduction. They both approximately optimize the local entropy loss LLE as defined in Fq. [1), for which an exact evaluation of the integral is in the general case intractable. The two algorithms employ different but related approximation strategics.
Entropy-SGD. Entropy-SGD (eSGD), introduced in Chaudhari et al. 2019 minimizes the local entropy loss Eq. ( by approximate evaluations of its gradient. The gradient can be expressed
VLLE(W) = 7 (w-(w'))
(5)
where (.) denotes the expectation over the measure Z 2-1c-B0(w()) ----(((""==((((( -vd((w where Z a normalization factor. The eSGD strategy is to approximate (w") (which implicitly depends w) using L steps of stochastic gradient Langevin dynamics (SGLD). The resulting double-loop algorithm is presented Algorithm 1
The noise parameter in the algorithm is linked to
4

---

Published as a conference paper at ICLR 2021
Algorithm 1: Entropy-SGD (cSGD)
Algorithm 2: Replicated-SGD (rSGD)
Input:
10
Input
:(w"}
Hyper-parametens: I,,7, 7 n', 6 for t = 1, 2..... do
Hypor-parameters: 73 K
for t = 1.2,... do
2
w', μ + w
W +- y - w
for = l..... L do
for
L, ...., do
t sample minibatch
E レ sample minibatch
dw' t VL (w'; =) + (w' w)
du VL (w): E)
w' t- w' - 1/ q'' + +V cN,0
if 1 = mod K then dw dw Kr (w 1
I t ap + a1 a)
7
w 7J p)
w f w ll dw
2/3. In practice always set it to the small value 10 as in Chaudhari et al. 20119] For 0, eSGD approximately computes a proximal operator (Chaudhari et al. 2018 For = a - = 0, eSGD reduces to the recently introduced Lookahead optimizer (Zhang et al. 2019
the inverse temperature by the usual Langevin relation
Replicated-SGD. Replicated-SGDD (rSGD) consists in replicated version of the usual stochastic gradient (SGD) method. In rSGD, a number y of replicas of the same system, each with its own parameters Wa where a = 1,...... y, are trained in parallel for K iterations. During training, they interact with each other indirectly through an attractive term towards their center of mass, As detailed in Baldassi et al. (2016a) 2020) in the simple case of shallow networks (committee machines), the replicated system, when trained with a stochastic algorithm such as SGD, collectively explores an approximation of the local entropy landscape without the need to explicitly estimate the integral in Eq. (1). In principle, the larger y the better the approximation, but already with 3 the effect of the replication is significant. To summarize, rSGD replaces the local entropy Eq. (I with the replicated loss Lx:
y CR((Wa]a) Σ ((w²) +
y
Σ d aww "wu..)) 1)
(6)
Here, w a center replica defined as w == 1 wa. The algorithm is presented as Algorithm 2 Thanks to focusing (see below), any of the replicas or the center can be used after training for prediction. This procedure is parallelizable over the replicas, so that wall-clock time for training is comparable to SGD, excluding the communication which happens every parallel optimization steps. In order to decouple the communication period and the coupling hyperparameter 7, we let the coupling strength take the value Ky. In our experiments, we did not observe degradation in generalization performance with K up to 10.
Focusing. A common feature of both algorithms is that the parameter y in the objective LLE changes during the optimization process. We start with a small 7 (targeting large regions and allowing wider exploration of the landscape) and gradually increase it. We call this process focusing. Focusing improves the dynamics by driving the system quickly to wide regions and then, once there, gradually trading off the width in order to gel. to the minima of the loss within those regions, see Baldassi et. al. (2016b We adopt exponential schedule for 7, where its value at epoch T is given by 7 (1 iSGD, we fix no by balancing the distance and the data term in the objective before training starts, ic. we set To = Σ. L(w")/ / E. d(w", w) for rSGD. The parameter ?1 is chosen such that y increases by a factor 10a. For eSGD, we were unable to find a criterion that worked for all experiments and manually tuned it.
For
Optimizers. Vanilla SGD updates in Algorithms and 2 can be replaced by optimization steps of any commonly used gradient-based optimizers.
5

---

Published as a conference paper at ICLR 2021
5 DETAILED COMPARISON OF FLATNESS MEASURES IN SHALLOW NETWORKS
In this section, we explore in detail the connection between the two flatness measures and the generalization properties in a one-hidden-layer network that performs a binary classification task, also called committee machine. This model has symmetry that allows to fix all the weights in the last layer to 1, and thus only the first layer is trained. It also invariant to rescaling of the weights. This allows to study its typical properties analytically with statistical mechanics techniques, and it was shown in Baldassi et al. (2020) that it has a rich non-convex crror-loss landscape, in which rare flat minima coexist with narrower ones. It is amenable to be studied comi-analytically: for individual instances, the minimizers found by different. algorithms can be compared by computing their local entropy elficiently with the Belief Propagation (BP) algorithm (see Appendix [B.1. bypassing the need to perform the integral in Eq. 4 explicitly. Doing the same for general architectures is an open problem.
For a network with K hidden units, the output predicted for a given input pattern x reads:
(ww, = sign
sign
WkeTi
K
We follow the numerical sctting of Baldassi al. (0200 and train this network to perform binary classification on two classes of the Fashion-MNIST dataset with binarized patterns, comparing the results of standard SGD with cross-entropy loss (CE) with the entropic counterparts rSGD and eSGD. All these algorithms require differentiable objective, thus we approximate sign activation functions on the hidden layer with tanh(3x) functions, where the B parameter increases during the training. The CE loss is not invariant with respect to weight rescaling: we control the norm of the weights explicitly by keeping them normalized and introducing overall scale parameter that we insert explicitly in the loss:
((w) = Ex f(a ((w, w)
(8)
Here, we have defined [(x, w) - - + parameter is increased gradually in the training process in order to rontrol the growth rate of the weight norms. Notice that the parameter 3 could also be interpreted as a norm that grows over time.
2u log(2 cosh(wwz) as in Baldassi et al. (2020). The w
As shown in Baldassi et al. (2020), slowing down the norm growth rate results in better generalization performance and increased flatness of the minima found at the end of the training. To approciate this effect we used two different parameters settings for optimizing the loss in (8) with SGD, that we name "SGD slow" and "SGD fast". In the fast setting both 3 and w start with a large value and grow quickly, while in the slow setting they start from small values and grow more slowly, requiring more epochs to converge. For rSGD, we also used two different "fast" and "slow" settings, where the difference is in faster or slower increase of the 7 parameter that controls the distance between replicas.
The results are shown in Fig. 1. In the left panel, we report PLE computed with BP around the solutions found by the different algorithms, as a function of the distance from the solution. Even if the slow SGD setting improves the flatness of the solution found, entropy-driver algorithms are biased lowards flatter minima, in the sense of the local entropy, as expected. In the central panel we plot the local energy profiles & for the same solutions, and we can see that the ranking the algorithm is preserved: the two flatness measures agree. The same ranking is also clearly visible when comparing the generalization errors, in the right panel of the figure: flatter minima generalize better
2In the appendix B.3 generalization holds also setting where we do not explicitly increase the local entropy.
we show that the correlation between local entropy. local energy and

---

Published as a conference paper at ICLR 2021
0.000
0.30
350
-0.002
tain erfor differance
0.25
SGD slow
300
SGD slow
local entropy
-0.004
0.20
ISGD slow
frequency
250
SGD slow
200
#S3D
-0.006
0.15
SGD tast
160
-0.008
SGD
0.10
reGD fast
-0.010
slow
0.05
#SGD
-0.012 7.00 0.05 0.10 0.15
0.00
0.20
0.25
0.30
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.04
0.05
0.06
0.07
0.08
Normalized local entropy LE as function of the squared distance d (left), training error difference & as a function of perturbation intensity (center) and test error distribution (right) for committce machine as defined in Eq. trained with various algorithms the reduced version of the Pashion-MNIST dataset. Results are obtained using 50 random restarts for each algorithu.
Figure 1:
6 NUMERICAL EXPERIMENTS ON DEEP NETWORKS
6.1 COMPARISONS ACROSS SEVERAL ARCHITECTURES AND DATASETS
In this section we show that, by optimizing the local entropy with eSGD and rSGD, we are able to systematically improve the generalization performance compared standard SGD, We perform experiments on image classification tasks, using common benchmark datasets, state-of-the-art deep architectures and the usual cross-entropy loss. The detailed settings of the experiments are reported in the SM. For the experiments with eSGD and rSGD, we use the same settings and syper-parameter (architecture, dropout, learning rate schedule,...) as for the baseline, unless othriie stated in the SM and apart from the syper-parameten specific to these algorithms. While it would be interesting to add weight normalization did for committee machine, none of
(Salimans & Kingma (2016)) with frozen norm, the baselines that we compare against uses this method. We also note that for the local energy defined in Eq. 4 the noise is multiplicative and local energy is norm-invariant if the model itself is norm-inve anmniivvriant.
While we do some little hyper-parameter exploration obtain a reasonable baseline, we do not aim to reproduce the best achievable results with these networks, since we are only interested in comparing different algorithms in similar contexts. For instance, we train PyramidNet-Shannn
for 300 epochs, instead of the 1800 epochs used in Cubuk et al. (2018), and we start from random initial conditions for EfficientNet instead of doing transfer learning as done in Tan & Te (2019). In the case of the ResNet 10 architecture instead, we use the training specification of the original paper (He et al.
2016
Dataset
Model
Baseline
rSGD
eSGD
FSGDxy
CIFAR-10
SmallConvNet
16.5 0.2
15.6
0.3
0.37
14.9 0.2
ResNet-18
13.1
12.4 0.3
0.3
11.8 12.1
ResNet-110
6.4 0.1
6.2
6.2
5.3 0.1 1.8
PyramidNet | ShakeDrop
2.1 0.2
2.2
CIFAR-100
PyramidNNar+ShaaeDDDop
0.1
0.15=0.1
12.7
EfficientNet-Bi
20.5
20.6
2.21
19.5
Tiny ImageNet ResNet-50
1.2
4..
41.7
39.2 0.3 38.9
DenseNet-121
0.3
39.8
38.6 0.4
Test set error (%) for vanilla SGD (baseline), eSGD and rSGD, The first three columns show results obtained with the same number of passes over the training data. In the last column instcad, each replica in the parallclizable rSGD algorithm consumes the same amount of data as the bascline.
Table 1:

---

Published as a conference paper at ICLR 2021
ResNet-18, CIFAR-10
ResNet-50, Tiny ImageNet
000 tast error (86)
top3 test error (56)
17
44
16
15
14
SGD
SGD
()=3)
(y:3)
eSGD (L=5)
eSGD (L=5)
12
rSGDXy (y=3)
rSGDXy (y=3)
11
20
40
60
80
100
120
140
180
88
50
100
150
200
230
epochs
epochs
Test error of ResNet-18 on CIFAR-10. Right: Test error of ResNet-50 on Tiny NetgeNet. The curves averaged over 5 runs. Training data consumed is the same for SGD, iSGD and eSGD. Epochs are rescaled by for rSGD and by L for eSGD (they are not rescaled for rSGDxy).
Figure 2: Left:
All combinations of datasets and architectures tested reported in Table [ while representative test eгтоr curves are reported in Fig. 2 Blanks correspond to untested combinations. The first 3 columns correspond to experiments with the same number of effective epochs, that is considering that in each itcration of the outer loop in Algorithms and we sample L and y mini-batches respectively. In the last column instoad, cach replica consumes individually the same amount of data the bascline. Being a distributable algorithm, rSGD enjoys the same scalability as the related EASGD and Parle (Zhang et al. 2014 Chaudhari et al. 2017).
replicas and the scoping schedules described Sec. In our explorations, rSGD proved to be robust with respect to specific choices of the hyper- parameters. The error reported that of the center replica w. We note here, however, that the distance between the replicas at the end of training is very small and they effectively collapse to a single solution. Since training continues after the collapse and we reach a stationary value of the loss, we are confident that the minimum found by the replicated system corresponds to minimum of a single system. For eSGD, we set L = = le and 0.75 in all experiments, and we perform little tuning for the the other hyper-parameters. The algorithm is more sensitive to parameeassessers than rSGD, while still being quite robust. Moreover, it misses an automatic ? scoping schedule.
For rSGD, we use y
Results in Table show that entropic algorithms generally outperform the corresponding baseline with roughly the same amount of parameter tuning and computational resources. In the next section we also show that they end up in flatter minima.
6.2 FLATNESS VS CENERALIZATION
For the deep network lests, we measured the local energy profiles (see Eq. 44) of the configurations explored by the three algorithms. The estimates of the expectations were computed by averaging over 1000 perturbations for each value of We did not limit ourselves to the end result, but rather we traced the evolution throughout the training and stopped when the training error and loss reached stationary values. In our experiments, the final training error is close to O. Representative results are shown in Fig. 3 which shows that the eSGD and rSGD curves are below the SGD curve across a wide range of values, while also achieving better generalization. Similar results are found for different architectures, as reported in Appendix B.3 This confirms the results the shallow networks experiments: entropic algorithms tend to find flatter minima that generalize better, even when the eoper-parameter: the standard SGD algorithms had already been tuned for optimal generalization (and thus presumably to end up in generally flatter regions).
8

---

Published as a conference paper at ICLR 2021
SGD
eSGD
iSGD
comparison upoch=260
3.0
3.0
3.0
3.0
82 (2.89.8.8 16.0)
92 (2 18:2)
52
13.2)
SGp (0.35, 13.3)
0.2 13.5)
123 (0.00 12.00
12.0)
esGD
2.5
100 00.23, 13.7)
2.5
2.5
(0.06) 11.9)
2.5
7SGD (1.00,11.))
train effor difference
260 (0.15, 13.50
difference
2801 11.8)
train error difference
260
11.5)
2.0
2.0
2.0
1.5
1.5
1.5
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0,00
0.30
98.00
0.30 0.02
0
Figure 3: Evolution of the flatness along the training dynamics, for ResNet-18 trained CIFAR-10 with different algorithms. Figures show the train error difference with respect to the unperturbed configurations. The value of the epoch, unperturbed train and test. errors (%) are reported the legends. The last, panel shows that minima found at the end of an entropic training are flatter and generalize better. The value of the cross-entropy train loss of the final configurations is: 0.005 (SGD), 0.01 (eSGD), 0.005 (rSGD).
7 DISCUSSION AND CONCLUSIONS
We studied the connection between notions of flatness and generalization, We have performed detailed studies on shallow networks and extensive numerical study on state of the art deep architectures. Our results suggest that local entropy is a good predictor of generalization performance. This is consistent with its relation to another flatness measure, the local energy, for which this property has already been established empirically. exploit this fact and be effective in improving the
Furthermore, entropic algorithms generalization performance on existing architectures, at fixed computational cost and with little hyper-parametts tuning. Our future efforts will be devoted to studying the connection between generalization bounds and the existence wide flat regions in the landscape of the classifier.
REFERENCES
Carlo Baldassi, Alessandro Ingrosso, Carlo Lucibello, Luca Saglictti, and Riccardo Zecchina. Subdominant dense clusters allow for simple learning and high computational perfor- mance in neural networks with discrete synapses. Phys. Rev. Lett., 115:128101, Sep 2015. doi: 10.1103/ PhysRevLett. 121.10 URL https:t//inkk aps. org//oi/ 1103/ PhysRevLett 115. 128101
Carlo Baldassi, Christian Borgs, Jennifer T. Chayes, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, and Riccardo Zecchina. Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes. Proceedings of the National Academy of Sciences, 113 F7655-5)7765577666 0027-8424. doi: 10.1073/ pnas. 1aaas.660103111 URL https: www. pnas ortooogg//ooooooooooooooooo htt///nneenn113 48/E7655
2016a. ISSN
Carlo Baldassi, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, and Riccardo Zecchina. Local entropy as a measure for sampling solutions in constraint satis- faction problems. Journal of Statistical Mechanics: Theory and Experiment, 2016 (2):P023301, February 2016b. ISSN 1742-5468.
doi: 10.1088 1742-5468 2016 022 023301. URL http://taacckiioooooo/oro/o/e//e org/1742 / wwwee220000000oeeeeeeeee
iop

---

Published as a conference paper at ICLR 2021
Carlo Baldassi, Enrico M. Malatesta, and Riccardo Zecchina. Properties of the geometry of solutions and capacity of multilayer neural networks with rectified linear unit activations. doi: 10. 10.1103// www.seeeLett....111.... 173.17 URL
Phys. Rev. Lett., 123:170602, Oct 2019.
https://link aps wwtps/ogg//oo///ooo 11133.1..111//111111111111111111
1103 www.s//ysyesstt..ttersee 123. 170602
Carlo Baldassi, Fabrizio Pittorino, and Riccardo Zecchina. Shaping the learning landscape in neural networks around wide flat minima. Proceedings of the National Academy of Sciences, 117(1) (77(1)):61111111 2020. ISSN 0027-8424. doi: 10.1073, ppas./pssss... apaas.998666117 URL.
https://www pnas org/content/117///6//
Wray L Buntine and Andreas S Weigend. Bayesian back-propagation. Compler syslems, 5 (6):603 643, 1991.
Pratik Chaudhari, Carlo Baldassi, Riccardo Zecchina, Stefano Soatto, and Ameet Talwalkar. Parle: parallelizing stochastic gradient descent. CoRR, abs/1707.000 hos/1777.00024, 2017. URL httt:://axizi http://aaxivv..ggbbb////ss
00424
Pratik Chaudhari, Adam Oberman, Stanley Osher, Stefano Soatto, and Guillaume Carlier. Deep relaxation: partial differential equations for optimizing deep neural networks. Research in the Mathematical Sciences, 5(3):30, 2018.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Chris- tian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Eutropy-sgd: Biasing gradient descent into wide valleys. Journal of Stalistical Mechanics: Theory and Experi- 2019.
ment, 2019
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data. CoRR, Abs/1885.09501 2018. URL http://arxiv org/ abs/ 1800.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. CoRR, abs/ 1708.04552, 2017.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. 34th International Conference on Mochine Learning, ICML 2017, 3:1705-1714, 2017.
Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data, 2017.
Gintare Karolina Dziugaite and Daniel M. Roy. Entropy-SGD optimizes the prior of PAC-bayes bound: Data-dependent PAC-bayes priors via differential privacy, 2018. URL https://openrevier http://wwweettooomm////eeyru...
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. CoRR,
abs/161 1610.02915, 2016. URL http: httt:://ardd..org/abb////e orgg//b///11/..////o/o// 02915
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016. doi: 10.1109/ CVPR.2016.90.
Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference
on Computational Learning Theory, COLT '93, pp. 5-13, New York, NY, USA, 1993. Association for Computing Machinery. ISBN 0897916115, doi: 10.1145/168304.168830 URL https://dolt org/10. 1145/168304. 168306
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Newral Computation, 9(1):1-42 42. 1997. doi: 10.1162 /eco//cco.nn. http://co..9999.0 1997.9.111 11112//..0..199..1111.. URL https:///oi. www.oo.go... 1162..1............ 1997.9.999.. 1.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them, 2019.
10

---

Published as a conference paper at ICLR 2021
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-bater batch training for deep learning: Generalization gap and sharp minima. CoRR, abs/1609. 1609.04836, 2016. URL http:://aaa.v..orgaabbb///////// wwwgg//gaa///.oaareeergg 04836
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. CoRR, abs/ Abs/1995.00097 2019. URL
00397
Marc Mezard and Andrea Montanari. Information, physics, and compulation. Oxford University Press, 2009.
Marc Mézard, Giorgio Parisi, and Miguel Virasoro. Syin glass theory and beyond: An Introduction to the Replica Method and Its Applications, volume 9. World Scientific Publishing Company, 1987.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Newral Informalion Processing Systems, pp. 8024 8035, 2019.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparamcterization to accelerate training of deep neural networks. In Advances in neural information processing systems, pp. 901-909, 2016.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, 2019.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-11), pp. 681-688, 2011.
Yoshihiro Yamada, Masakazu Twamura, and Koichi Kise. Shakedrop regularization. CoRR, abs 1bs:1800.02375, 2018. URL http://arxiv
aa//a/ass//1.... 02375
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey Hinton. Lookahead optimizer: steps forward, 1 step back. In Advances in Neurol Information Processing Systems, pp. -693-9604, 2019.
Sixin Zhang, Anna Choromanska, and Yann LeCun. Deep learning with elastic averaging sgd, 2014.
Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P. Adams, and Peter Orbanz. Non- vacuous generalization bounds at the imagenet scale: A pac-bayesian compression approach, 2018.
A LOCAL ENTROPY AND REPLICATED SYSTEMS
The analytical framework of Local Entropy was introduced in Ref. Baldassi et al. while the connection between Local Entropy and systems of real replicas (as opposed to the
2015
"fake" replicas of spin glass theory (Mézard el al.] 1987)) was made in Baldassi el al. 201 2016a)) For convenience, we briefly recap here the simple derivation.
We start from the definition of the local entropy loss given in the main text:
3 log / du'' e
LL (w)
(9)
We then consider the Boltzmann distribution of a system with energy function SLLE (w) and with inverse temperature y, that is
p(w) PyLLE(w)
(10)
11

---

Published as a conference paper at ICLR 2021
where equivalence is up to normalization factor. If we restrict y to integer values, can then use the definition of LLE to construct an equivalent but enlarged system, containing y + 1 replicas. Their joint distribution p(w, {w")a) is readily obtained by plugging Eq. 9 into Eq. (1). We can then integrate out the original configuration w and obtain the marginal distributional for the y remaining replicas
Ja( x e -La(((")))),
(11)
where the energy function is now given by
1 I00
Cn(tw")a) Ja) Σ ((w")
Σ
w||
(12)
2
with w = w². We have thus recovered the loss function for the replicated SGD (1SGD) algorithm presented in the main text.
B FLATNESS AND LOCAL ENTROPY
B.1 LOCAL ENTROPY ON THE COMMITTEE MACHINE
In what follows, we describe the details of the numerical experiments on the committee machine. We apply different algorithms to find zero error configurations and then use Belief Propagation (BP) to compute the local entropy curve for each configuration obtained. We compare them in Fig. D along with the local energy computed by sampling and their test errors.
We define reduced version of the MasHion-MNIST dataset following Baldassi et al. 2020 we choose the classes Dress and Coat as they are non-trivial to discriminate but also different enough that small network as the one used can generalize. The network is trained a small subset of the available examples (500 patterns) binarized to +1 by using the median of each image threshold on the inputs; we also filter both the training and test sets to use only images in which the median is between 0.25 and 0.75.
The network has input size N 784 and a single hidden layer with K = 9 hidden units. The weights between the hidden layer and the output are fixed to 1. It is trained using mini-batches of 100 patterns. All the results are averaged over 50 independent restarts. For all algorithms we initialize the weights with a uniform distribution and then normalize the weights of the hidden units norm before the training starts and after each weight update. The 3 and paramcters are updated using exponential schodules, 5(t) = 3u (1 + 31)² and w()) = wi (1 + w1)', where the current epoch. An analogous exponential schedule is used for the elastic interaction y for rSGD and eSGD, as described in the main text. In the SGD fast case, we stop as soon as a solution with zero errors is found, while for SGD slow we stop when the cross entropy loss reaches value lower than 10 10-7 For rSGD, stop training as soon as the distance between the replicas and their center of mass is smaller than 10-8, 8, For eSGD, stop training soon the distance between the parameters and the mean (μ in Algorithu
I is smaller than 10-8
We used the following hyper-parameters for the various algorithms:
SGD fast: n= 2- 10-4 P. = 2.0, 3 = 10-1, we = 5.0, w1 = 0.0;
SGD slow: 1) 3- 10-5 B. = 0.5, 8 10-3 wo 0.5, 41 10-3;
rSGD fast: 1 = 10 10-4, 10, = 2. : = 2- 3u = 110, 31 2- We = 0.5,
w = 10-3,
rSGD slow: = 10 3 y = 10, % = - 11 = 10-4, 4 3 = 1.0, 3 = 10 wp = 0.5,
wi = 10-3.
eS 7 = 10 // 5- 5 c 10-6, L = 20, 70 = 10.0, m = 5 10 10-5, 80 = 1.0,
81 = 10 wp 0.5, Wi =
5- 5-
For a given configuration w obtained by one of the previous algorithms, we compute the local entropy through BP. The message passing involved is similar to the one accurately
12

---

Published as a conference paper at ICLR 2021
detailed in Appendix IV of Ref. Baldassi et al. (2200). the only difference here being that we apply it to a fully-connected committee machine instead the tree-like one used in Ref. Baldassi et al. (020). We thus give here just brief overview of the procedure. We frame the supervised learning problem as constraint satisfaction problem given by the network architecture and the training examples. In order to compute the local entropy, add a Gaussian prior centered in to the weights, (w, A). The variance A of the prior will be related later a specific distance from w. We run BP iterations until convergence, then compute the Bethe free-energy fethh (w, A) as function of the fixed point messages. Applying the procedure for different values of A and then performing a Legendre transform, we finally oblain the local entropy curve (1 d).
B.2 FLATNESS CURVES FOR DEEP NETWORKS
In this Section we present flatness curves, Eraain (w, o) from Eq. networks architecture examined in this paper.
some of the deep
Results are reported Figs 4 and 5 for different architectures and datasets. The expectation in Eq. 4 is computed over the complete training set using 100 and 400 realizations of the Gaussian noise for each data point in Figs and respectively. In experiments where data. augmentation used during training, is also used when computing the flatness curve.
EfficientNet-BO, CIFAR-100
PyramidNet=ShakkeDrgg CIFAR-100
35
20.5)
S50
(L5, 10.5)
(2.2, 12.7)
train error difference
20
ResNet-50, Tiny ImageNet
DenseNet-122 Tiny ImageNet
(1.2, 45.4)
(28.
ISGD (1.53, 38.1)
17.5
ISGD [L.1, 38.2)
train error differencce
15.0
12.5
10.0
7.5
0.0
0.00
0.02
0.04
0.06
0.08
0.10
0.00
0.02
0.04
0.06
0.08
0.10
Figure Train error difference ò'raiin from Eq. 4 for minina obtained on various architectures, datasets and with different algorithms, as a function of the perturbation intensity a. Unperturbed train and test errors (%) are reported in the legends. The values of the train cross-entropy loss for the final configurations are: EfficientNet--O0 0.08 (SGD), 0.06 (rSGD); PyramidNet 0.07 (SGD), 0.07 (rSGD): ResNet-50: 0.04 (SGD), 0.006 (eSGD), 0.1 (rSGD); DenseNet: 0.1 (SGD), 0.2 (eSGD), 0.12 (rSGD).
The comparison is performed between minima found by different algorithms, at a point where the training error is near zero and the loss has reached stationary value. We note
13

---

Published as a conference paper at ICLR 2021
epoch=80
epoch=92
opoch=160
3.0
3.0
3.0
5K30 (1.5.13.3)
SGD (0.24, 6.4)
5500 (0.008. 6.3)
ISGD
rSGD (0.04,
2.5
eSGD (5.85 21.50
2.5
#SGD (.)1,
2.5
65GD (0.0.5.4) 5.4)
train effor differencee
2.0
2.0
1.5
1.5
1.0
1.0
1.0
0.5
0.5
0.5
8.00
98.00
0.00
Figure 5: Train error difference EEaaii from Eq. 44 for ResNet-110 on Cifar-10. Values are computed along the training dynamics of different algorithms and as a function the perturbation intensily
Unperturbed train and test errors (%) are reported in the legends. The values of the train cross-entropy loss for the final configurations are: 0.001 (SGD), 0.0005 (eSGD), 0.0005 (rSGD).
here that this kind of comparison sensitive to the fact that the training errors at a are close to cach other. If the comparison is made for minima that show different train errors, the correlation between flatness and Lest error not observed.
We report generally good agreement. between the flatness of the 8 Farain curve and the generalization performance, for a large range of a values.
B.3 CORRELATION OF FLATNESS AND GENERALIZATION
In this scction test more thoroughly the correlation of our definition of flatness with the gencralization error. The entropic algorithms that we tested resulted in both increased flatness and increased generalization accuracy, but this does not imply that these quantities are correlated in other settings.
For the committee machine we tested in different settings that the local entropy provides the same information as the local energy, and that they both correlate well with the generalization error, even when entropic algorithms are not used. An example can be seen in Fig. where the same architecture has been trained with SGD fast (see B.I] and different values of the dropout probability. The minima obtained with higher dropout have larger local entropy and generalize better.
In order test this correlation independently in deep networks, we use models provided in the framework of the PGDL competition (see These models have different generalization errors and have been obtained without using local entropy in their training objective.
seriitessggggl.....oooo google. woogle.ccm//vee http///wwww/dlle11//
We notice that the local entropy and the local energy are only comparable the context of the same architecture, the loss landscape may be very different if the network architecture varies. We therefore choose among the available models a subset with the same architecture (VGG-like with convolutional blocks of width 512 and 1 fully connected layer of width 128) trained on CIFAR-10. Since the models were trained with different hyperparameters (dropout, batch size and weight decay), they still show wide range of generalization errors.
Since we cannot compute the local entropy for these deep networks, we restrict the analysis to the computationally cheaper local energy (see Eq.
mcasure of flatness.
As shown in Fig. 7 we report a good correlation between flatness as measured by the train error difference (or local energy, at given value of the perturbation intensity o, namely
14

---

Published as a conference paper at ICLR 2021
0.000
0.30
pdrop
160
pdrop 0
-0.002
0.25
pdrop m. 0.25
140
pdrop 0.25
Sain erfor difference
porop
120
pdrop
local entropy
-0.004
0.20
frequency
-0.006
0.15
80
-0.008
0.10
pdrop 0
-0.010
porop
0.05
pdrop 0.4
20
0.0 -0.0
0.00
0.05 0.10 0.15
0.20
0.25
0.30
0.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.04
0.05
0.06
0.07
0.08
Figure 6: Normalized local entropy PLE as function of the squared distance (left), training error difference E'raii as a function of perturbation intensity (center) and test error distribution (right) for a committee machine trained with SGD fast and different dropout probabilities the reduced version of the Pashion-MNIST dataset. Results are obtained using 50 random restarts for each algorithm.
0.5) and test error, resulting in Pearson correlation coefficient r(12) 0.90 with p-value le-4.
y
0.8
train error difference
0.7
0.6
0.5
0.4
0.3
0.2
0.14
0.22 0.24
0.26
0.30
test error
Figure 7: Train error difference Etrain from Eq. 4 in function of test error for a fixed value of the perturbation intensity 0.5, for minima obtained the same VGG-like architecture and dataset (CIFAR-10) and with different values of dropout, batch size and weight decay. Each point shows the mean and standard deviation over 64 realizations of the perturbation. The models are taken from the public data of the PGDL competition.
C DEEP NETWORKS EXPERIMENTAL DETAILS
In this Section we describe in more detail the experiments reported in the Table 1 of the main text. all experiments, the loss C is the usual cross-entropy and the paramcter initialization is Kaiming normal. We normalize images in the train and test sets by the mean and variance over the train set. We also apply random crops (of width w image size is w x w, with zero-padding of size 4 for CIFAR and for Tiny ImageNet) and random horizontal flips. In
15

---

Published as a conference paper at ICLR 2021
the following we refer to the latter procedure by the name "standard preprocessing". All experiments are implemented using PyTorch (Paszke et al. 2019
For the experiments with eSGD and rSGD, use the same settings and ppperpparaeetess used for SGD (unless otherwise stated and apart from the hyperparameters specific to these two algorithms).
For rSGD and unless otherwise stated, we set y = 3, K 10 and use the automatic exponential focusing schedule for y reported in the main text.
For eSGD, we use again an exponential focusing protocol. In some experiments, we use value of "Io automatically chosen by computing the distance between the configurations w' and w after a loop of the SGLD dynamics (i.e. in the first I steps with y = 0) and setting "o [(w)/d(w', w). Unfortumately, this criterion is not robust. Therefore, in some experiments the value of To was manually tuned. However, we found that eSGD is not sensitive to the precise value but to the order of magnitude.
We choose such that y is increased by factor of 10 by the end of the training. Unless otherwise stated, we set the the number of SGLD iterations to L 5, SGLD noise to < 10-4 and a = 0.75. Moreover, the internal and external loop. As for the learning rate schedule, when we rescale the total number of epochs for eSGD and rSGD, we use a rescaled schedule giving comparable final learning rate and with consequently rescaled learning rate drop times as well.
use 0.9 Nesterov momentum and weight decay in both
C.1 CIFAR-10 AND CIFAR-100
SmallConyNet
The smallest architecture we use in our experiments is a LeNet-like
network LeCun al. 1998):
Conu(5 20) -MarPool(2) Conu(5 x 5, 50) Max Dense(500)- Softmor
Each convolutional layer and the dense layer before the final output layer are followed by ReLU non-lincaritics.
We train the SmallConvNet model CIFAR-10 for 300 epochs with the following settings: SGD optimizer with Nesterov momentum 0.9; learning rate 0.01 that decays by a factor of 10 at epochs 150 and 225; batch-size 128; weight decay 1e-4; standard preprocessing is applied; default parameter initialization (PyTorch 1.3). For rSGD we set Ir = 0.05 and to = 0.001. For eSGD, train for 60 epochs with: n 0.5 that drops by factor of 10 at epochs 30 and 45; n = 0.02; 7o = 0.5; = 10-5
ResNet-18 In order to have fast baseline network, we adopt a simple training procedure for 18 on CIFAR-10, without further optimizations. We train the model for 160 epochs with: SGD optimizer with Nesterov momentum 0.9; initial learning rate 0.01 that decays by factor of 10 at epoch 110; batch-size 128; weight decay 5e-4; standard preprocessing.
For rSGD we set K and learning rate 0.02. For eSGD, we train for 32 epochs with initial learning rate 1 0.25 that drops by factor of 10 at epochs 16 and 25; n/ 0.01. In the case which we drop the learning rate at certain epochs, we notice that is important not to schedule it before that the training error has reached plateau also for eSGD and rSGD.
ResNet-110 We train the ResNet- 110 model on CIFAR-10 for 164 epochs following the original settings of He et al. (016):: SGD optimizer with momentum 0.9; batch-size 128; weight decay 1e-4. We perform learning rate warm-up starting with 0.01 and increasing it at 0.1 after epoch; then is dropped by a factor of at epochs 82 and 124; standard preproccssing is applicd.
For both eSGD and rSGD, we find that the learning rate warm-up is not necessary. For rSGD set no 5e-4. For eSGD, train for 32 epochs with initial learning rate n 0.9 that drops at epochs 17 and 25, SGLD learning rate r/ = 0.02 and we set % = 0.1 and
m 5 10-4
16

---

Published as a conference paper at ICLR 2021
2016; Yamada et al. 2018 together with AutoAngment or Fast-AntoAuumennt is currently the state-of-the- art CIFAR-10 and CIFAR-100 without extra training data. We train this model
PyramidNett-Sakee
CrramiddeettShhaeDeaa (Han et al.
CIFAR-10 and CIFAR-100 following the settings of Cubuk et al. (2018); Lim et al. (2019): PyramidNet272-02000 SGD optimizer with Nesterov momentum 0.9; batch-size 64; weight decay 5e-5. At variance with Cubuk et al. (2018); ; Lim et al. (2019) we train for 300 epochs and not 1800. We perform a cosine annealing of the learning rate (with a single annealing cycle) with initial learning rate 0.05. SrakeDrop is applied with the same parameters as
in the original paper (Yamada et al., 2018). For data augmentation we add to standard preprocessing Aut AuloAugment with the policies found on CIFAR-10 (Cubuk et al. 2018) (for both CIFAR-10 and CIFAR- 100) and CutQut. (Devries & Taylor 2017) with size 16.
For rSGD, we use cosine focusing protocol for 7, defined at epoch by 7 0m5x cos (IT with 7max 0.1. On CIFAR-10, we decrease the interaction step K from 10 to 3 towards the end of the training (at epoch 220) in order to reduce noise and allow the replicas to collapse.
EfficientNet-BO Eeficient.Net-BO is the base model for the EfficientNet family, In this section we train FfficientNNet-OO on CIFAR- 100, starting from random initial conditions. We follow the same settings as Tan & Le (2019), with some differences:
train for 350 epochs with RMSprop optimizer with momentum 0.9; batch-size 64; weight decay 1e-5; initial learning rate 0.01 that decays by 0.97 every 2 epochs. We rescale image size 224 224 and as data augmentation we apply standard preprocessing (with zero-padding of size 32) adding AutoAugment with the policies found on CIFAR-10 (Cubuk et al.) 2018). For rSGD we set = 5e 6. For eSGD we used initial learning rate 17 = 0.5 that decays by 0.92 every epochs and o/ 0.05.
C.2 TINY IMAGENET
ResNet-50 Entropic algorithms are effective also on more complex datasets. We train ResNet-50 on Tiny ImageNet (dala downloaded from: Tiny ImageNet Visual Recognition Challenge") for 270 epochs with: SGD optimizer with Nesterov momentum 0.9; initial learning rate 0.05 that decays by a factor of 10 at epochs 90, 180 and 240; batch-size 128; weight decay 1e-4. Standard preprocessing is applied together with Fast-AutoAugman with the policies found on ImageNet (Lim et al. 2019).
For eSGD train the model for 50 epochs with n 0.8 that drops by factor of 10 at epochs 18, 36, 48 and rf 0.02.
DenseNNtt121 For DenseNet-121 Tiny ImageNet, the setting is the same ResNet-50, except that we train the model for 200 epochs with learning rate drops at epochs 100 and 150.
For eSGD we train the model for 40 epochs with 0.5 that drops by a factor of 10 at epochs 25 and 30, r/' 0.02 and set % 1.0 and ?1 10
17


Reviews:
[{"review_id": "zazvip4H6A", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies local entropy measures for characterizing flat regions in the energy landscape of deep networks. The paper discusses, at length, two previously proposed algorithms named Entropy-SGD and Replicated-SGD and demonstrates, using (i) controlled experiments where Belief Propagation (BP) can be used to estimate the local entropy integral precisely, and (ii) empirical results on deep networks that flatter minima generalize better.\n\nThis paper presents a systematic analysis of the hypothesis that flat minima generalize better. This has been the subject of much debate in the recent literature because flatness is often characterized using spectral norm of the Hessian. The paper revisits the line of work that initiated this debate and shows that flatness, as measured by local entropy instead, indeed correlates with good generalization.\n\nThe paper lacks in terms of novelty in the sense that it does not propose a “new” algorithm, but the reviewer believes that not all publications need to. A paper like this with carefully constructed constructed and a clear conclusion is also a valuable addition to this debate. I am however concerned about the incremental value of this manuscript in view of the long line of work that it builds upon. I am recommending a weak accept but I am willing to increase my score if the authors make a convincing case against this concern.\n\nComments.\n\n1. The paper, as noted above, heavily builds upon existing work, in particular series of works of Baldassi et al. and Chaudhari et al. The papers introduce local entropy, focussing, replicated SGD, BP for calculating the local entropy etc. Entropy-SGD/Parle were also shown to work well for state-of-the-art deep networks then. The authors should state clearly what the concrete contributions of this manuscript are.\n2. I like the section on measuring flatness for shallow architectures using BP. It would be good to include the BP calculations in the Appendix.\n3. The idea of making the cross-entropy loss invariant to the scale of the weights by keeping them normalized can also be used with deep networks using weight normalization https://arxiv.org/abs/1602.07868.\n4. Can you characterize the shape of the wide region? I believe this will lend more insight in why wide minima generalize better for deep networks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z5OMSX91aaU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors essentially study the generalization properties of networks trained via two different algorithms, Entropy SGD (eSGD) and Replicated SGD (rSGD)  against networks trained via SGD. Both eSGD and rSGD are algorithms that have been previously designed using the notion of entropy guided modified loss function. In this paper the authors compare the performance of broadly these three categories of networks in terms of Local Entropy (Eq. 3) and local energy (Eq. 4) (i.e. difference in training error when the weights are perturbed by factor \\sigma), which was developed in [1]. With this the authors aim to find a correlation between networks with close to zero entropy and networks with lower local energy. \n\nThe first experiment uses a simple two layer neural network with fixed final layer weights and studies both local entropy and local energy as a function of weight perturbation. (I tried hard to find where Figure 1 is explicitly mentioned but I simply couldn't. I assume the experiments in Figure 1 correspond to the network eq (7) - please make this explicit). \n\nSecond set of experiments include simple classification test accuracies on deeper architectures such as ResNet110, DenseNet, EfficientNet and PyramidNet [this is not novel, just a replication and extension to newer architectures] (not including local energy, if I understand correctly.)\n\nThird set of experiments are the main contribution of this paper - testing the correlation of local energy and local entropy - this is done on ResNet-18 trained on CIFAR10 dataset. While this does shed some light on the correlation of local entropy and local energy in promoting flatness, I'm not sure if one experiment on CIFAR10/Resnet18 is sufficient to justify correlations in deeper architectures that have been trained on larger datasets. What would have been more useful is to see if the same results translate for the models discussed in Section 6.1. \n\nIn conclusion, I do think this is an interesting correlation but not enough experiments are provided to justify this correlation. \n\nReferences\n\n[1] Fantastic Generalization Measures and Where to Find Them, Jiang et al ICLR 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XThqxFd7lVn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies (1) the relationship between the flatness of minima and their generalization properties, and (2) the connection between two measures of flatness, known as local entropy and local energy. Through a series of experiments, the authors show that the two measures are highly correlated and correlate well with generalization. They also empirically show that Entropy-SGD and Replicated-SGD, when used to explicitly optimize the local entropy, are able to flatter and better minima (in terms of lower generalization errors).\n\nClearly, the contributions of this work are not on either new algorithms or new measures, but rather it is a comprehensive empirical study that nicely brings seemingly different things together. I find the results interesting and the findings influential.\n\nI have two clarifying questions:\n\n(1) About activation functions in Section 5 and Section 6: For the results presented in Table 1, the deep networks use ReLU non-linearities, while in Section 5, the sign activation is shown in Equation 7 but then switched to an approximation by tanh(beta x). Can the authors clarify why using ReLU leads to an issue here? Please explain this “All these algorithms require a differentiable objective” and the use of a new loss function.\n\n(2) Figure 3 and Figure 5, the training error difference of eSGD consistently decrease and gradually becomes smaller than that of SGD and rSGD. Does this have anything to do with the Langevin sampling? Can the authors give an explanation?\n\nMinor comments:\n\n+) In Equation (1), the loss L(w) is with respect to w, but what is L*(w, beta, gamma)? The same with d*(w, beta, gamma).\n\n+) In Equation (3), E_train(w) is not defined. When \\it{d}=0, then Omega(-d(w’, w)) in the denominator seems to be 0, which is strange?\n\n+) Last paragraph before Section 6: do you mean Figure 1 instead of 6?. Also, there are some inconsistencies in the referencing figures: “Fig. 6”, “fig. 3” and “Figures 4 and 5”. The same problem is with equations: “(1)” vs “eq. (4)” and “Eq. (4)”.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-ssXagImwDA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update after response: I appreciate the authors making their contributions clearer, and adding details about the training loss and error. I have increased my score accordingly.\n\nOriginal Review:\n\nThis paper presents an empirical evaluation of whether flatness correlates with generalization using a few different definitions of flatness - local energy and local entropy. The authors study two training procedures - entropy sgd and replicated sgd, and show that deep networks trained using these procedures are able to locate flatter solutions that also generalize better.\n\nWhile the paper presents some interesting empirical confirmation of the correlation between local entropy/energy and generalization, the algorithms presented in this paper have also been defined in previous work, and this phenomenon has been repeatedly observed with different definitions of flatness [1,2]. The authors also do not present any reasons to expect that generalization is related to flatness that are grounded in theory. The contributions of this paper thus seem to be confirmation of previously observed phenomena [3,4].\n\nMoreover, in the experiments it is not clear whether the solutions that are reached by the training procedures actually correspond to local or global minima of the cross-entropy loss function (There is also the issue that minimizers of the cross-entropy loss function occur at infinity). The authors do not report the training loss at the solutions that they choose to plot, and the training error also does not seem to be zero (plots corresponding to Figure 3). I am not sure whether it makes sense to call these solutions minima, and compare their flatness.\n\n\n[1] Neyshabur, B., Bhojanapalli, S., McAllester, D., & Srebro, N. (2017). Exploring generalization in deep learning. In Advances in neural information processing systems (pp. 5947-5956). \n\n[2] Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., & Tang, P. T. P. (2016). On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836.\n\n[3] Baldassi, C., Pittorino, F., & Zecchina, R. (2020). Shaping the learning landscape in neural networks around wide flat minima. Proceedings of the National Academy of Sciences, 117(1), 161-170.\n\n[4] Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C., ... & Zecchina, R. (2019). Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12), 124018.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the link between generalization behavior and "flatness" of the loss landscape in deep networks. Specifically, the authors study two measures of flatness (local entropy and local energy), and show that these two measurements are strongly correlated with one another. Moreover they show via a careful set of numerical experiments that two previously proposed algorithms (entropy SGD and replica SGD) that optimize for local entropy tend to both find flatter minima as well as provide better generalization.

Despite the fact that the paper proposes no new models or algorithms, the experiments are compelling and provide non-trivial insights into predicting generalization behavior of deep networks, as well as solid evidence on the benefits of entropy regularization in SGD. The authors also seem to have satisfactorily answered the (numerous) initial concerns raised by the authors. Overall, I recommend an accept.
Decision: Accept (Poster)