Flatness is a False Friend
Authors: Diego Granziol
Venue:  (2021)

Abstract: Hessian based measures of flatness, such as the trace, Frobenius and spectral norms, have been argued, used and shown to relate to generalisation. In this paper we demonstrate that, for feed-forward neural networks under the cross-entropy loss, low-loss solutions with large neural network weights have small Hessian based measures of flatness. This implies that solutions obtained without L2 regularisation should be less sharp than those with despite generalising worse. We show this to be true for logistic regression, multi-layer perceptrons, simple convolutional, pre-activated and wide residual networks on the MNIST and CIFAR-$100$ datasets. Furthermore, we show that adaptive optimisation algorithms using iterate averaging, on the VGG-$16$ network and CIFAR-$100$ dataset, achieve superior generalisation to SGD but are $30 \times$ sharper. These theoretical and experimental results further advocate the need to use flatness in conjunction with the weights scale to measure generalisation \citep{neyshabur2017exploring,dziugaite2017computing}. 

Under review as a conference paper at ICLR 2021
FLATNESS IS A FALSE FRIEND
A Anonymous authors
Paper under double-blind review
ABSTRACT
Hessian based measures of flatness, such as the trace, Frobenius and spectral norms, have been argued, used and shown to relate to generalisation, In this paper we demonstrate that, for feed-forward neural networks under the cross- entropy loss, low-loss solutions with large neural network weights have small Hessian based measures of flatness. This implics that solutions obtained without L2 regularisation should be less sharp than those with despite generalising worse. We show this to be true for logistic regression, multi- layer perceptrons, simple convolutional, pre-activated and wide residual networks on the MNIST and CIFAR-100 datasets. Furthermore, we show that adaplive optimisation algorithms using ilerate averaging, on the VGG- 16 network and CIFAR-100 dataset, achieve superior generalisation to SGD but are 30x sharper. These theoretical and experimental results further advocate the need to use flatness in conjunction with the weights scale to measure generalisation (Neyshabur al. 2017 Dziugaite and Roy 2017
1 INTRODUCTION
Deep Neural Networks (DNNs), with more parameters than data-points, trained with many passes of the same data, still manage perform exceptionally on test data. The reasons for this remain laregly unsolved (Ncyshabur et al. 2017). However, DNNs are not completely immune the classical problem of wver-fitting. Zhang et al. (2016) show that DNNs can perfectly fit. random labels. Schedules with initially lly low sharply decaying learning rates. lead identical training but much higher testing error (Berrada et al. 2018 Granziol et al., 2020a Jastrzebski et al. 2020). In Wilson et al. (2017) the authors argue that specific adaptive gradient optimisers lead to solutions which don't generalise. This has lead to a significant development in partially adaptive algorithms (Chen and Gu 2018 Keskar and Sochor 2017). Given the importance of accurate predictions on unseen data, understanding exactly what helps deep networks generalise has been fundamental arca of rescarch. A key concept which has taken a foothold in the community, allowing for the comparison of different training loss minima using only the training data, is the concept of flainess. From both Bayesian and minimum description length framework, flatter minima should generalize better than sharp minima (Hochreiter and Schmidluber 1997). Sharpness is usually measured by properties of the second derivative of the loss,, the Hessian H=V²L(u [Kekarr et al. 2016 Jastrzcbski et al.) (2017b] Chaudhari al. 2016 Wu H
ct al.] 2017 2018), such as the spectral norm or trace. The assumption is that duc to finite numerical precision (Hochreiter and Schmidhuber, 1997) or from Bayesian perspective
MacKay
2003), the test surlace is shajled from the training surface. The difference wetn
train and Lest loss for a shift Aw given by
P L(w" + Aw) L(w")= =0 AT HAW +... Σ Ao Aw
Tr(H) P
||w < A Au
(1) in which w is the final training point and [Ai, Φ.] are the eigenvalue beteenaallleeeignnveee /igenveccoor pairs of H E PxP. We have dropped the terms beyond second-order by assuming that the gradient at training end is small. In general we have no priori reason to assume that shift should preferentially lie along any of the Hessian eigenvectors, hence by taking a maximum entropy prior (MacKay) 2003
Jaynes 1982) we expect strong high dimensional concentration results
1

---

Under review as a conference paper at ICLR 2021
/Vershynin 2018) to hold, hence 10 2 ~ 1/P, where v is simply the normalised version of w. This justifies the trace as a measure of sharpness. In the worst case scenario the shift is completely aligned with the eigenvector corresponding to the largest eigenvalue A Le. Aw" Φb = 1. Hence the spectral norm X1 of H serves as a loca upper bound to the loss change. The idea of a shift between the training and testing loss surface is prolific in the literature and regularly related to generalisation (He et al.) 2019 Izmailov et al. 2018 Maddox al., 2019). Alternative, yet closely related, measures of Hatnnes are also used. Keskar ct al. (2016) define a sharp minimiser as one "with a significant number of large positive cigenvalues" in fact as can be seen by the Rayleigh-Rite Ritz thcorem, the metric which is proportional to the largest eigenvalue.
they propose, shown in Equation 2
L., A) == (maxyec. L(w + Ay)) - I(wr) I x(e))11
(2)
1 ((w)
C. is the constraint box as defined in (Keskar et al. 2016), where e controls the box size. As shown by Dinh et al. (2017), this definition of sharpness is approximately given by ATe2/2(1 + L (w)), proportional to the largest eigenvalue. This result can be explained intuitively as within small vicinity of the largest change in loss is along the leading cigenvector and is proportional to the largest cigenvalue. Wu et al. (2017) consider the logarithm of the product of the top k cigenvalues as proxy mcasure the volume of the minimum (a truncated log delerminant). In this paper we will exclusively consider the Hessian Irace, spectral and Frobenius norm as measures of sharpness.
Motivation: There have been numerous positive empirical results relating sharpness and generalisation. Keskar et al. (2016) ; Rangamani et al. (2019) consider how large batch vs small batch stochastic gradient descent (SGD) alters the sharpness of solutions, with smaller batches leading to convergence
flatter solutions, leading to better generalisation. Jastrzebski et al. (2017a) look at the importance of the ratio learning rale and batch sive in terms of generalisation, finding that large ratios lead flatter minima (as measured by the spectral norm) and better generalisation. Yao et al. (0088) investigated flat regions of weight space (small spectral norm) showing them to be more robust under adversarial attack. Zhang et al. (2018) show that SGD concentrates in probability on flat minima. Certain algorithmic design choices, such as Entropy-SGD (Chaudhari et al.] 2016) and the use of Polyak averaging (Izmailov et al.] 2018) have been motivated by considerations of flatness. However Dinh el al.] (2017) show That by exploiting ReLUs (Rectified Linear Units) positive homogeneity property f(ax) af(()), any flat minima can be mapped into a sharp minimum, without altering the loss. As these measures can be arbitrarily distorted, this implies they serve little value as generalisation measures. However such transformations alter other properties, such as the weight norm. In practice the use of L2 regularisation, which penalises weight norm, means that optimisers are unlikely to converge to such a solution. It can even be shown that unrogularised SGD converges to the minimum norm solution for simple problems (Wilson et al.) 2017), further limiting the practical relevance of such reparameterisstioo arguments. The question which remains and warrants investigation, is are Hessian based sharpness metrics at the end of training meaningful metrics for generulisation? ? We demonstrate both theoretically and experimentally that the answer to this question is affirmative no.
To the best of our knowledge, this is the first work which demonstrates theoretically motivated empirical results contrary to purely flatness based generalisation measures. For the fully connected feed-forward network with ReLU activation and cross entropy loss, we demonstrate in the limit of O training loss, that the spectral norm and trace of the Hessian also go to O. The key insight is that in order for the loss to go to O, the weight vector components w. must tend to infinity. Conversely, this implies that methods which reduce the weight magnitudes extensively used aid generalisation (Bishop) 2006): Krogh and Hertz 1992p, makes solutions sharper. We present the counter-intuitive result that adding L2 regularisation increases both sharpness and generalisation, for Logistic Regression, MLP, simple CNN, PreResNet-161 and WideResNet-288 x 10 for the MNIST and CIFAR-100
Contributions:
1 'we use the word local here because the largest eigenvalue /eigaleeeeegeeeectorr pair may change along the path taken
2

---

Under review as a conference paper at ICLR 2021
datasets. We also present and discuss various amendments to the Hessian, which robust against the arguments presented here and those of Dinh et al. (2017).
Related work: Empirically negative results on flatness and its effect on generalisation have been previously observed. Neyshabur et. al. (2017) show that it caplures generalisation for large but not small networks. Goolakkar et al. 2019 show that the maximum of the trace of the Fisher information correlates better with generalisation than its final value. [Jastrzebski et al.) (2018) show that it is possible to optimise faster and attain better generalisation performance whilst finding a final sharper region. For small networks, those trained on random labels (with no generalisation) are less sharp than those trained on the true labels. However this does not rule out that for the same network trained on true labels, solutions which are flatter generalise better. Instead the main focus has centered around the Hessians lack of reparamcterisation invariance (Neyshabur et al., 2017 Tsuzuku et al. 2019 Rangamani et al. 2019 This has been a primary motivator for normalised definitions of Hatnnss Tsuzuku et al. (2019): Rangamani et al. (2019) often in a PAC- PAC-Bayesian framework. In Ballard et al. (001)) Mehta et al. (2018), it. shown that adding the 1.2 regularization
weights including the
bias weights removed singular modes of the Hessian matrix for a feed-forward artificial neural network with one hidden layer, with tanh activation function, employed to fit the XOR data. In Mehta et al. (2018), with the help of an algebraic geometry interpretation of the loss landscape of the deep lincar networks, was proven that a generalized L2 regularization guaranteed to remove all singular solutions leaving the Hessian matrix strictly non-singular at every critical point.
2 GEDANKEN EXPERIMENT: WHY THE HESSIAN WON'T DO
For simple illustration let consider the deep linear model, with exponential loss. The deep lincar model is often employed as a thcoretical tool for its analytical tractability (Kawauuchi 2016 Lu and Kawaguchi 2017). In Scction 3 we formalise the results to the fully connected Teed lorward network with cross entropy loss. Intuitively we can think of feed forward network a sum of deep linear networks and the cross entropy as an approximation to the exponential loss. For 3 parameters and single datum X, the loss is given by exp (w1W2wyX). The Hessian, its trace and spectral norm H, Tr(H), (H) are given by
w wh W2wl w wfwg
w W1w2ws wfug
w w wa w Wawsy wfw?
H
X exp (wi (WIW2w3)
(3)
Tr(H) = (H) (wgng + uane + wfw3) X exp (WIW2w3X)
(4)
Smaller losses imply flatter Hessians: Equation 4 shows that under this model the trace and maximum eigenvalue are products a polynomial function of the weights and an exponential in the weights. As the optimiser drives the loss L + 0 expect the exponential to dominate the polynomia?] This implies that methods to reduce the weight magnitude, such as L2 regularisation, which has been extensively shown to aid generalisation (Krogh and Hertz 1992 Bishop, 2006) should increase Hcssian measures of sharpness. Wc show that this is the case experimentally in Section 4
3 THEORETICAL FRAMEWORK
In this Section we extend our intuition developed under the deep linear network with exponential loss, to more realistic scenarios. Similar to the prior work of Choromanska et al. (2015); Milne (2019), we consider a neural network with a d, dimensional input æ. Our network has II hidden layers and we refer to the output as the H'th layer and the input as the 0'th layer. We denote the ReLU activation function as (xx) where f(x) max(0,
consider the function x" (xp"" + Vn, as
3

---

Under review as a conference paper at ICLR 2021
Let W; be the matrix of weights between the (i - 1) and i'th layer. For dy dimensional output our q'th component of the output can be written as
H
z(<;; w) =
ΣΣ A.A ==
(5)
k=1
where the indices i.j denote the sum over network inputs and paths respectively and y is the number of paths. As, E [0, 1] denotes whether the path active or not and w denotes the the weight of the path segment which connects node i in layer q 1 with node j in layer
9. layer has n, nodes and y = II - ny We formalise our intuition from Section 2 with the following theorem:
Theorem 1. For any feed forward neural network with ReLU output activation functions I (() = max(0, x), coupled with softmar output in the final layer and cross entropy loss, in the limil that the training loss L(w)= + the spectral norm X of the empirical Hessian H- VVL(w) E RxP also lends to O.
Proof. The cross-entropy loss E(h(a;; y) of single sample æ; defined by:
dy
E(h(2; vi)
Σ 1 [h(,;; w)e # vel) log h(zg;
(6)
Where dy is the number of classes and Il is the indicator function which takes the value of 1 for the incorrect class and 0 for the correct class, 2(,;;; w) is the softmax input. The softinx output 2α0;;; w) for class is given by
dy
h(æ,, w). = o(2) exp 2 (a:; w)
1
Σ exp (=(#;; w)x - 2(0);;))))
(7)
1= exp 2(0;;; w)
Hence combining Equations
and the loss per sample be written
dy
E( ((((z;; yi) = log 1+
Σ exp (2E Agf((((
(8)
in which c(i) denotes the correct class for the data point a; and =k,o(() = 2(x;;; w)x (z; w)((()- Note that, for the per sample loss f(h(w, w) v.) + O, exp(2x,.(()
vk A c(i). Using the chain rule
((((((z;;) w), v.) Swrdwm
Expare exp(zx,c())
azc.eri Ozp,a(s) Own
Jor Twr Own
Ow
1 Lije exp(zx,0())
Enze(H)ppo()) exp (2,c) (1+ Expe(e exp ())
exp(za,e(0)
Batm
As shown in Milne 2019), the network is differentiable at the majority of points in weight space. Specificllyy the zero set of non-zero real analytic functions (the network is piecewise analytic in the weights) has Lebesgue mcasure zero. Hence all we need to show is that the output derivatives tend to oo more slowly than the loss tends to 0. To do this we consider
9m
de A
7/I1m
(b) (1) 1 [I o why
3
(10)
Own mm
We are interosted in the limit where the output for the correct class z(2;; w))()) + co. Although any individual weight segment w may be zero, this simply deactivates that path segment and reduces the contributing sum. Hence we can absorb such zero weights into the Aig and without loss of generality assume that the weights are bounded to minimum
4

---

Under review as a conference paper at ICLR 2021
absolute value of [ > 0. Note from Equation 10 that only paths containing the weight segment corresponding the differentiated nodes contribute, hence the bracket on the RHS of this equation is also upper bounded by 1. We can upper bound the norm of the diffcrentiated output
(K)
mm
de
H
j=1 Ty Ai, ===1 wis
A
(k)
S
i=
(11)
II Ow th Um i=1 5=1
e IIm Tpm
1=1
Hence
8mh (1; w)g
exp (=(x;;; w)) |z(0,;; w))l
exp (z(00;;)))
(12)
I Bw pm.00m
II 7m
and hence in the limit E(((a;; w), y.) O, all terms in Equation have norms tending to zero. Hence 8 (((((m,))))))) + 0. Taking I = m and summing over m we have O trace Swdeem
and using the Frobenius norm identity, i.e. taking the sum of squares over I, m we have Σ X + 0 and hence \ 0.
Remark. By writing the loss in terms of the activation a at the output of the final layer f(w), ie L(w)= of of((w).. The Hessian may be expressed using the chain rule as
1 N
N
dy
dy
Po(s(w) 0f((w) 0S((w) 0f((w) (f( Ow, dw.
du (o(s(w)) 0² (e((w) owy Sw,dwy
H(w) jk: =
n=1
(13)
Where, for the cross-entropy loss and softmax output exactly 0 loss, s'a(f(w))
[ 0 and
Dws
3f((w)ff(()) = 0. However, in practice, since the weights are finite, we never have 0 loss. Hence, unlike our proof which shows that the Hessian is given by a product of a polynomial and exponential in the weights which we expect to go to in the limit of large weights and low loss, this simple result does yield information prior to the loss being exactly O.
WEIGHT DECAY AND SHARPNESS
Section more formally demonstrates what was already hypothesisod Scction weights, required to drive the loss to very low values, are expected to give small Hessian based mcasures of sharpness, despite potentially wildly over-fitting the data and generalising poorly. To show that this is relevant in practice we evaluate the effect of weight norm reducing techniques, such as L2 regularisation on spectral sharpness. L2 is regularly used to help generalisation, having been showed to reduce the effect of static noise on the target (Koogh and Hertz 1992).
2 Larger
Experimental Sctup:
We use the deep visualisation suite (Granziol al. 2019 package
to visualise the spectrum of the Hessian and calculate the largest cigenvalues. We train all networks using SGD with momentum p = 0.9 and varying levels of L2 regularisation 3 3|/w|] || [0, 0.0001, 0.0005]. For further experimental details, such as the learning rate schedule ( we a linear decay schedule with terminal learning rate of 0.01 the initial) employed and the finer details of the spectral visualisation method see Appendix A Since adding L2 regularisation naturally adds y to each eigenvalue, as H + H +71, in our results we do not calculate the Hessian the regularised loss. For simplicity but include results on the Frobenius norm and trace in the Appendix. Furthermore since for certain simple architectures the inclusion of L2 regularisation also increases optimisation performance, wherever this is the case we will relate Hessian based measures of sharpness to the generalisation gap. For which we just report the difference in validation and training accuracy. For more modern networks using batch normalisation (Ioffe and Szegedy 2015). L2 regularisation reduces convergence speed but negligibly alters optimisation performance, so we can relate measures of sharpness to the validation performance directly3
focus on the spectral norm,
since we substracting essentially the same amount
5

---

Under review as a conference paper at ICLR 2021
The simplest Neural Network model, corresponding to a hidden layer feed forward neural network, is the Softmax Regressor By the diagonal dominance theorem QCover and Thomas 2012) the Hessian of the Softmax Regressor is positive semi-definite, so the loss surface is convex. For a convex objective all local minima global, hence do not have the complication of different minima. We this model the MNIST dataset (LeCun 1998), splitting the training set into 45, 000 training and 5, 000 validation samples. The total parameter count 7850. We run for 1000 (we specifically use an abnormally large number of epochs to make sure that convergence is not an issue) epochs with learning rate 0.01. The validation accuracy increases incrementally with increased weight decay (93.48, 94, 94.08). We plot the spectra of the final solution in Figure I We note here that for increasing weight decay coefficient, which corresponds to higher performing testing solutions, the spectral norm increases, from a [2.26, 14.17, 15.84]. This shows that greater Hessian based measures of sharpness, occur for solutions with improved generalisation.
Logistic Regression MNIST:
10
10
10
10
10
10
10
10
10 11
10
10
10 10
0.00
12.26
0.00
14.17
0.00
15.84
(a) Acc 93.48 A=
(b) Acc 94 X 10
(c) Acc 94.08 À 5 10
Figure 1: Hessian spectrum for Softmax regression after 1000 epochs of SGD on the MNIST dataset, for various 1.2 regularisation coefficients X
We now consider single hidden layer MLP on the MNIST dataset, with a hidden layer of 100 units, parameter count 9960, trained for 50 epochs with an identical schedule and learning rate of 0.01. We similarly find that the addition of weight decay both increases the generalisation accuracy (from 94.4 + 96.46 + 96.7 as increase the regularisation coefficient y from > 0.0001 > 0.0005). Similar to the Softmax example, this also increases the spectral norm shown in Figure 2 The training accuracy increases slightly with the introduction of regularisation, but decreases over the unregularised network when the regularisation is increased to 0.0005. The generalisation gap for the various levels of regularisation [0.48%, 0.51%, 0.04%), where the smallest generalisation gap corresponds to the largest spectral norm.
MLP:
10-1
10-1
10
1
10-3
10-3
1
10-5
10-5
10-7
10-7
10-7
-0.01
14.37
-0.01
14.99
0.00
17.76
(a) Ace 94.4, =0
(b) Acc = 96.46, 7 10
(c) Acc = 96.7, 7 10
Acc=
Figure 2: Hessian spectrum for MLP after 50 epochs of SGD the MNIST dataset, for various L.2 regularisation coefficients
CNN: We consider a 9 layer simple convolutional neural network on the CIFAR-100 dataset (Dangel al. 2019), with parameter count 187,7, 108 and learning rate of a 0.01 for 300 epochs. We also observe that adding weight decay increases the spectral norm, as shown Figure For this network, training is also improved by the addition of little 7.2 regularisation, but performance decreases for over regularisation, i.c. as the weight decay parameter increases from [o, 10-4, 55xx 10-4] the training performance is [86.3%, 87.9%, 86.0%]. In this particular example the training accuracy is quite low, but there is still generalisation gap [32.4%, 32.5%, 31.1%]. Furthermore the generalisation gap is smaller for the network with the largest spectral norm, seen by comparing Figures 3c to Figures and 3b
multi-class equivalent of Logistic Regression

---

Under review as a conference paper at ICLR 2021
10
10a
10
10 2
10
10
10-4
10-4
10
-0
10
10
+
10-8
10
-3.58
49.01
33.6
65.67
4,5
127.3
(a) Val 53.9, 7
(b) Val 55.4, 10
10-
(c) Val = 54.8, 7 7= x
Figure 3: Hessian spectrum for CNN after 300 epochs of SGD on the CIFAR-100 dataset, for various 7. regularisation coefficients
PreResNect-66 We use a pre-activated residual network on the CIFAR-100 dataset with parameter count 1, 726, 388. Our training performance decreases with increased level of regularisation (0. 10-1, 5 10-1) from 999.987% 99.985% 99.87% but our testing performance increases significantly (72.8%, 75.56%, 76.76% The generalisation gap decreases increase the regularisation, yet the Hessian spectral norm continues to increase.
10
10a
10
10
10
10
10
10
10
10
-0-8
10 8
-00490
0.510
1.520
2.525
-4.42
11.49
-7.57
2.63
12.83
23.04
(a) Vol = 73.12, 7
(b) Val 75.96, 7 1e-4
(c) Val = 77.36, 7 =
Figure 4: Hessian spectrum for PreResNet-166 after 300 epochs of SGD the CIFAR-100 datasel, for various L2 regularisation coefficients 7
WideResNet-28 10: We use a wide residual network on the CIFAR-100 dataset, with parameter count. 36, 546, 980, we observe the training accuracy remains roughly constant 999.984%, 99.984%, 99. (9.982%] are now the regime where the optimisation benefit of regularisation the generalisation benefit is significant. As shown in Figure 5 the Hessian spectral norm continues to increase with the increased regularisation coefficient y. The valicatiio set accuracies are [75.2%, 79.5%, 80.6%] and again the spectral norm increases as the
we increase the regularisation from [0, 10-1,5 x 10-1). We
negligible, but
generalisation gap decreases.
10
10
10
10
10
10
10
-5
10
10
10
IT.
10
0.220
0.410
1.050
1.693
10
115.38
39.13
10
-18.55
0.77
20.39
40.00
(a) Val = 75.2,
=
(b) Vol = 79.50, A=
(c) Val 80.6,
5e
Figure 5: Hessian spectrum for WideRessNet28x 10 after 300 epochs of SGD the CIFAR-100 dataset, for various L2 regularisation coefficients 7, Batch Norm Train mode
Note on relationship between learning rate and spectral norm: Under stability analysis Wu et al. (2018); Lewkowycz et al. (2020) argue that gradient descent must find
A1 < 2/ao, whereas SGD satisfies a more restrictive condition, i.c the minima must be even flatter due to the condition of non-uniformity specifically À < 1 1 /1 a X1 (Var H)). Where A₁ (Var(HF ())) is simply the largest eigenvalue the variance of the Hessian. Although the authors Wu et al. (2018) remark that this bound is tight in deep learning experiments. We note for our ResNet experiments, with initial learning rate a 0.1, as we increase the weight decay, even the GD bound of 20 is overhee Whilst potentially the learning rate decrease in training by a factor of 100, brings the optimiser to a new region within the loss surface, the bound of 2000 is certainly not tight.
7

---

Under review as a conference paper at ICLR 2021
5 SHARPNESS AND ADAPTIVE OPTIMISATION
Given that all high performing solutions use some form weight regularisation, we consider whether sharpness be useful indicator in the wild for the same neural network trained on the same datasct, but with alternative optimisors and schodules. Since out of the box adaptive gradient methods perform more poorly on the validation/ /tesot test sets than SGD Wilson ee al. (2017), we compare the sharpness of solutions of the Adam optimiser when combined with decoupled weight decay (Loshchilov and Hutter 2018) and iterate averaging (Granzio] et al. 2020b which has been shown to generalise better than SGD. We use the VGG-16 with nottalaioonniiitiionn on the CTFAR-10 100 datasets. We use a decoupled weight decay of 0.35/0.25 and a learning rade of a 0.0005. For SGD we use weight decay of 3/5 x 10-1 and a lcarning rate of a 0.1. We plot the validation accuracy curve for CIFAR-100 in Figure [c] whilst we see that Adam clearly generalises better than SGD, As shown in Figures 6a and (aaand] 6b the spectral norm of the better performing Adam solutions is almost 40x larger than the SGD solution, the Frobenius norm of Adam is 0.02 opposed to 0.0001 for SGD. Both solutions give similar training performance, with Adam 99.81 and SGD 99.64. For similar picture, shown in Figure To investigate the fragility of other commonly employed metrics for this practical scenario, the Frobenius norm of the Adam solution is 1.55 10- opposed to 1.43 10-5, Furthermore we note from Figures 6c and 7c that not only are these solutions sharper, but they also have higher norms.
CIFAR-10 although the generalisation gap smaller,
10
10
0.8
Garaurz
SGD
125
12 norm
10
10
0.6
10-6
10
10
-1.43
10.11
10-8
62.2
387.8
0.2
50
150
250
50
(a) Val = 73.5, SCD
(b) Val = 75.8, Aadamn
(c) Test Error L2 norm
Figure 6: Hessian spectrum for VGG-16BN after 300 epochs of SGD the CIFAR-100
dataset, for varions optimisation algorithms [SGD, Adam], batch norm train mode
10a
1.00
10-4
10
10-4
10
0.50
0.75
Gadam
SGB
0
80
100
L2 morm
10-6
10-6
10-0
10-8
0.00
40
3.296
6.28
200
(a) Val = 94.3, SGD
(b) Val = 95.1, Adam
(c) Test Error L2 norm
Figure Hessian spectrum for VGG-16BN after 300 epochs of SGD on the CIFAR-10 dataset, for various optimisation algorithms [SGD, Adaml, batch norm train mode
CONCLUSION
In this paper we show that the Hessian be written as the product of polynomial and exponential in the weights, which in the limit of large weights, which overfit, give rise to flat minima despite poor generalisation. We show that adding L2 regularisation, significantly increases the spectral norm whilst improving generalisation. We also show that certain heuristics used to improve adaptive optimisation generalisation, give high weight norm sharp solutions, which generalise better than the corresponding flatter lower norm SGD solutions. This shows that whilst intuitive, pure Hessian based flatness measures are not relevant for generalisation Solutions found in practical schedules which generalise significantly better can be much sharper.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Andrew J Ballard, Ritankar Das, Stefano Martiniani, Dhagash Mehta, Levent Sagun, Jacob T) Stevenson, and David J Wales. Energy landscapes for machine learning. Physical Chemistry Chemical Physics, (9(20):12885-11
2017.
Leonard Berrada, Andrew Zisserman, and M Pawan Kumar. Deep Prank-Wolfe for neural network optimization. arXiv preprint arXiv.811..07591 2018.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Tony Cai, Jianqing Fan, and Tiefeng Jiang. Distributions of angles in random packing on spheres. The Journal of Machine Learning Research, 14(1):1837-1864, 2013.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Chris- tian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD, Biasing gradient descent into wide valleys. arXiv preprint ar 1rxi:1661..0888 2016.
Jinghui Chen and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXiv: 2018
Anna Choromanska, Mikacl Henaff, Michael Mathicu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics, pages 192-20 204, 2015.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Felix Dangel, Frederik Kunstner, and Philipp Hennig, Backpack: Packing more into backprop. arXiv preprint arXiv: 1912. 10985, 2019.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1019-1028 JMLR. org, 2017.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. Xie preprinl Xive 103: 701110031111000 11008, 2017.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via Hessian cigenvalue density. arXiv preprint arXiv:1901. 2019.
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect carly learning dynamics, matter little near convergence. In Advances in Newral Information Processing Systems, pages 10678 10688, 2019.
Gene Н Golub and Gérard Meurant. Matrices, moments and quadrature. Pitman Research Notes in Mathematics Series, pages 105 105, 1994.
Gene H Golub and Charles F Van Loan. Matriz computations, volume 3. JHU press, 2012.
Diego Granziol, Xingchen Wan, Timur Garipov, Dmitry Vetrov, and Stephen Roberts. MLRG deep curvature. Xi preprint orXiv:1912.0963: 2019.
Diego Granziol, Timur Garipov, Dmitry Vetrov, Stefan Zohren, Stephen Roberts, and Andrew Gordon Wilson. Towards understanding the true loss surface of deep neural networks using random matrix theory and iterative spectral methods, 2020s. URL
Diego Granziol, Xingchen Wan, and Stephen Roberts. Iterate averaging helps: An alternative perspective in deep learning. arXiu preprint orXim:2003.022 2020b.
Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. arXiv preprint ar Xiv:1 1002. 002.00744, 2019.

---

Under review as a conference paper at ICLR 2021
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1):1-42, 1997.
Michael F Hutchinson. stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in Statistiss-Simulation and Computation, (922):4334450, 1990.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ar preprint arXi::1502.091 031677 2015.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXi prepriat or arXin: 05407, 2018.
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey Three factors influencing minima in sgd. arXiv preprint vrxii:1711.. 022,, 2017a.
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. ar preprint xrxio:1711.00 0432,, 2017b.
Stanislaw Jastrzebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey On the relation between the sharpest directions DNN loss and the SGD step length. 2018.
Stanislaw Jastrzebski, Maciej Szymozak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on the optimization trajec- tories of deep neural networks. In International Conference on Learning Representations, 2020. URL htttps://ppereet htt.:aettttttooorrdddrggggr///.
Edwin T Jaynes. On the rationale of maximum-entrropy methods. Proceedings of the IEEE, 70(9):939 952, 1982.
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in neural informa- lion processing systems, pages 586 594, 2016.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from Adam to SGD, arXiv preprint xix::17112.07625 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04$5 2016.
Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In Advances in neural information processing systems, pages 950 957, 1992.
Yann LeCun. The MNIST database of handwritten digits. http://yann. lecun. com/eadb) /eadb/mmmssss mmist// 1998.
Aitor Lewkowyez, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint ar Xio:20003. 02218, 2020.
Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 2018.
Haihao Lu and Kenji Kawaguchi. Depth creates bad local minima. arXie preprint arXiv:1 702. 08580, 2017.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for Bayesian uncertainty in deep learning. In Advonces in 2019.
Newral Information Processing Systems, pages
10

---

Under review as a conference paper at ICLR 2021
Dhagash Mehta, Xiaojun Zhao, Edgar A Bernal, and David J Wales. Loss surface of xor artificial neural networks. Physical Review E, 97(5):052307, 2018.
Gérard Meurant and Zdenëk Strakoš. The Lanezos and conjugate gradient algorithms in finite precision arithmetic. Acta Numerica, 15:471-542, 2006.
Tristan Milne. Piecewise strong convexity of neural networks. In Advances in Neural Information Processing Systems, pages 12973 12983, 2019.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester and Nati Srebro. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems, pages 1947-5956, 2017.
Vardan Papyan. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size. arXiv preprint arXiv:1811.0006: 2018.
Barak A Pearlmutter. Fast exact multiplication by the Hessian. Neural computation, 6(1): 147 160, 1994.
Akshay Rangamani, Nam H Nguyen, Abhishek Kumar, Dzung Phan, Sang H Chin, and Trac D Tran. scale invariant flatness measure for deep network minima. ar prepriat or Xrxiv: 02434, 2019.
Farbod Roosta- Khorasann and Uri Ascher. Improved bounds on sample size for implicit matrix trace estimators. Foundations of Computational Mathematics, 15(5):1187 1212, 2015.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Normalized flat minima: Exploring scale invariant definition of flat minima for neural networks using bac-bayesian analysis. Xiv preprint 1rXiv:1901.44653 2019.
Roman Vershynin. High- dimensoamal probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht, The marginal value of adaptive gradient methods in machine learning. In Advonces in Neural Information Processing Systems, pages 4148-4158, 2017.
Lei Wu, Zhanxing Zhu, al. Towards understanding generalization of deep learning: Perspective of loss landscapes. arXiv preprint arxiv:1706. 10239, 2017.
Lei Wu, Chao Ma, and Weinan. How sgd selects the global minima in over over-vvranenne parameterizze learning: A dynamical stability perspective. In Advances in Neural Information Processing Systems, pages 8279 8288, 2018.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian- Hessian-based analysis of large batch training and robustness to adversaries. In Advances in Neural Information Processing Systems, pages 4949 4959, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Under- standing deep learning requires rethinking generalization. arXiv preprint arXic:1611.0885 2016.
Chiynan Zhang, Qianli Liao, Alexander Rakhlin, Brando Miranda, Noah Golowich, and Tomaso Poggio. Theory of deep learning iib: Optimization properties of sgd. arxiv preprint orXiv: 1801. 18010025,,, 2018.
11

---

Under review as a conference paper at ICLR 2021
A EXPERIMENT DETAILS
A.1 IMAGE CLASSIFICATION EXPERIMENTS
Hyper parameter Tuning For SGD and Gadam, set the momentum parameter to be 0.9 whereas for Adam, we set (B1, .22 = (0.9, 0.999) and <= 10-8 their default values. For SGD, we use grid scarched initial learning rates in the range of [0.01, 0.03, 0.1] for all experiments with a fixed weight decay; for Adam and all its variants, we use grid searched initial learning rate range of [10-4,3 x 10-3, 10 After the best learning rate has been identified, we conduct a further search on the weight decay, which we find often leads to a trade off between the convergence speed and final performance. For CIFAR experiments, we search in the range of [10 4 10 10-$1 whereas for ImageNet experiments, we search in the range of [10-6; 10-9). For decoupled weight decay, we search the same range for the weight decay scaled by initial learning rate.
A.2 EXPERIMENTAL DETAILS
For all experiments with SGD, we use the following learning rate schedule for the learning rate at the 1-th epoch, similar to Izmailov el. al. (2018):
00:
if + 0.5
02
ao (1-r)(+ -0.5) 0f < 0.9 otherwise
(14)
0.4
where ag the initial learning rate. In the motivating logistic regression experiments on MNIST, we used T' = 50. T = 300 is the total number of epochs budgeted for all CIFAR experiments. We set r = 0p01 for all experiments. For experiments with iterate averaging, use the following lcarning rate schedule instead:
if 1_ 0.5
Taco
2au2 ) 0.5)
a1
if 0.5 < Taeg S 0.9
(15)
Olangy
otherwise
where Gaag refers to the (constant) learning rate after iterate averaging activation, and in this paper we set Gaar = 2gg- Tavg is the epoch after which iterate averaging is activated, and the methods to determine Tavg was described in the main text. This schedule allows us to adjust learning rate smoothly in the epochs leading to iterate averaging activation through similar linear decay mechanism in the experiments without iterate averaging, as described above.
B LANCZOS ALGORITHM
In order empirically analyse properties of modern neural network spectra with tens of millions of parameters N = O(10), we use the Lanczos algorithm [Meurant and Strakoš 2006), provided for deep learning by Granziol products, for which we use the Pearlmutter trick (Pearlmutter 1994) with computational cost ONP)), where N is the dataset size and 1 is the number of parameters. Hence for steps the total computational complexity including ce-orthogonalisationn is D(NPm) and memory cost of O(Pm). In order to obtain accurate spectral density estimates we re-orthogonalise at every step (Meurant and Strakos], 2006). We exploit the relationship between the Lanczos method and Gaussian quadrature, using random vectors to allow us to learn discrete approximation of the spectral densily. A quadrat ure rule is a relation of the form,
al. (2019). It requires Hessian vector
S. -ANd)(A)-AAAA = Eerf(t)) + R[f] J-1
M
(16)
for a function f, such that its Stemaan-Stieltjes integral and all the moments exist on the measure dj(A), the interval [a, b) and where R[]] denotes the unknown remainder. The
12

---

Under review as a conference paper at ICLR 2021
nodes t of the Gauss quadrature rule are given by the Ritz values and the weights (or mass) Pj by the squares of the first elements the normalised eigenvectors the Lanczos tri- diagonal matrix (Golub and Meurant 1994.. The main properties of the Lanezos algorithm are summarized in the theorems 23
Theorem Let HN*N be a symmelric matria with eigenvalues À 2..2 An and corre- sponding orthonormal eigenvectors 2in. If 01 2 2 Om the eigenvalues of the matriz Tm obtained after m Lanezos steps and q1....qx the corresponding Ritz eigenvectors then
(A1 ), (01)
A 2 0 2 A1
(ck-1 + 2p1))²
(17)
(A1 - Xn) (0n) (c 2p1))²
An - 0x S Am +
where Ok is the chebysher polyomial of order
Proof: (oolu and Van Loan 2012
Theorem 3. The eigenvalues T& Te are the nodes t, of the Gauss quadrature rule, the weights w/ are the squares of the first elements of the normalised eigenvectors of Th
Proof: See (Golub and Meurant 1994). The first term on the RHS of equation 16 using
Theorem 3 can be seen discrete approximation to the spectral density matching the first
m moments uHHH", H (Golub and Meurant 1994 Golub and Van Loan 2012), where e is the initial seed vector. Using the expectation of quadratic torms, for zero mcan, unit variance random vectors, using the linearity of trace and expectation
N Tr H") = (vE. TY(H") = - JXED Adp(A)
(18)
The error between the expectation over the of all zero mcan, unit variance vectors and the monte carlo sum used in practice can be bounded (Hutchinson 1990, Roosta-Khorasan and Ascher 2015). However in the high dimensional regime overlap of each random vector with eigenvector of H, lu 6,1² v y Vi, with high probability.
+ 00, we expect the squared
This result can be seen by computing the moments of the overlap between Rademacher vectors, containing elements P(u, +1) 0.5. Further analytical results for Gaussian vectors have been obtained CCai et al. 2013).
C MATHEMATICAL PRELIMINARIES
For an /oput/output pair (ar E RRd,, y E Rd,] and given model h(::)) RRd x R. -> Rdd Without loss of generality, we consider the family of models functions parameterized by the weight vector w, Le., H == (h(;:w) w) : w RR), with a given loss (h((;; w), y) Rdd x Rd R.
The empirical risk (often denote the loss in deep learning), its gradient and Hessian are given by
Remp
Σ e(h(:; y:), (emp(() = V Remps (e) 22 R2Remp N
(19)
The Hessian describes the curvature at that point in weight space w and hence the risk surface can be studied through the Hessian. By the spectral theorem, we can rewrite (wm)(w)= = S == A. terms its eigenvalue, eigenvector pairs [Ax Φ.J. In order to characterise Hemp(1)) (w) by single value, authors typically consider the spectral norm, which is given by the largest cigenvalue of Hemp(w) or the normalised trace, which gives the mean cigenvalue. The Hcssian contains p2 elements, so cannot be stored or cigendecomposed for all but the simplest of models. Stochastic Lanczos Quadrature can be used Meurant and Strakoo] (2006), with computational complexity ((P) to give tight bounds on the extremal eigenvalues and good estimations of Tr(H) and Tr(H2), along with moment matched
13

---

Under review as a conference paper at ICLR 2021
approximation of the spectrum. We use the Deep Learning implementation provided by Granziol et al. (2019). DNNs are typically trained using stochastic gradient descent with momentum, where iteratively update the weights
2h++ + pzz + Vf((w))
(20)
wk+1 Wk a#k+1
Where p is the momentum. The gradient is usually taken randomly selected sub-sample of size B < N. An epoch is defined as a full training pass of the data, so comprises ze N B iterations. Often L2 regularisation (also termed weight decay) is added to the loss, which corresponds to Remp(w) + Remp(2) + p/21|w/|2
How docs batch normalisation affect curvature? During training both the mean and variance of the batch normalisation layers are adapted to the specific batch, whereas at evaluation they are fixed (to their exponentially moving averago). This done so that the transforms can function even if the prediction set. is only sample 5l Previous works investigating neural nelwork Hessians (Papyan 2018 Ghorbani el al.] 2019) do not consider this free parameter in batch-normallsation and its effect on the spectrum. From sharpness and generalisation perspective, we would consider that it is the model that is making predictions that should evaluate. Changing batch normalisation to the evaluation mode, we find that a somewhat different curvature profile, as shown in Figure g In this case the sharpness of the regularised solution in terms of the spectral norm is nearly 1000 times larger than that of the regularised, better generalising solution. The Frobenius norm, for the regularised solution is 4.9 x 10- opposed to 9.8 10 12, O(107) larger.
10
10
10
10
0.75
150
L2 nomm
ATO
10
10" 7
10
10 -7
9.50
-00
19.00013
0.02694
10
-0.66
21.20
0.25
0
50
100
150
200
250
(a) Val = 75.2, =
(b) Val 80.6, X 5e -4
(c)
Test Ervor & L.2 norm
Figure 8: Hessian spectrum for WideResNet288x 10 after 300 epochs of SGD on the CIFAR-100
dataset, for various L2 regularisation coefficients X, batch norm evaluation mode
aa 1 sample has variance
14


Reviews:
[{"review_id": "ofnyVOKyhC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### 1. Brief summary:\nThe authors look at the Hessian related measures of flatness such as the spectral norm, trace and the Frobenius norm. They ask the question of whether flatness at the end of training is a meaningful metric of generalization and answer no. To to do that they use a simplified model and a mathematical derivation and several numerical experiments where they evaluate the flatness measure for networks trained with varying amounts of L2 generalization, showing that solutions with lower flatness can generalize better.\n\n### 2. Strengths\n* The question the authors are asking is interesting and very relevant.\n* The introduction is very good in clarifying prior work, the reasons for why some people believe flatness is relevant and the assumptions it rests on.\n* The range of architectures and datasets tested is pretty good.\n* The paper does both theory and empirical experiments, which is a great sign and gives more gravity to the theoretical claims.\n* The paper likely dispels some preconceptions that presumably a part of the community holds about the role of flatness.\n* I like that you notice that the regularized Hessian has eigenvalues offset by the regularization strength -- a small thing, but made my appreciate the level of care you take.\n\n\n### 3. Points of my confusion\n\n#### a) The validity of the shift motivation for flatness as a relevant measure. \nIn the intro in page 1 you mention that the original motivation for considering flatness is the assumption that the train and validation minima are offset by delta w. This is not a weakness of your paper, I would just like to know why is it that people assume this would be the case?\n\n#### b) I do not get the Gedanken experiment. \nI am not sure I understand the point of the Gedanken experiment. Why is the loss exponential dependent on the output of the net? Where is the target y for the (X,y) pair? Are you just trying to make the output of the net as small as possible and exponentially punishing the deviations from that? I get that the you get the X W1 W2 ... WL polynomial structure there the same way you'd get in normal cases, but what is the motivation for this experiment? And what is the argument here? Is it that for L -> 0 the polynomial w1w2w3 must go to -infty, and in the expression ww exp(wwwX) the exponential decay kills off the powerlaw growth, taking the Trace -> 0? I am just generally confused about this setup and what it shows us. If you wouldn't mind clarifying this, I'd be grateful.\n\n#### c) How general is the derivation in Section 3? \nI see that you are using the fact that the weights go to infinity for the loss to go to 0. How general are the results you derive here for the Loss = epsilon. To my mind that is actually way more interesting because as I rightly say practically we're not going to get to L=0 anyway.  I see that you derive that as L->0 so do the second derivatives, but what about L = epsilon << log(# number of classes)?\n\n### 4. Relevant papers that could be of interest\n[1] I found a paper that looks at the Trace(H) and the ration Trace(H) / |H|_F and links this to the weight space norm of the network. It seems very relevant to what you are doing here. The paper is The Goldilocks zone: Towards better understanding of neural network loss landscapes by Stanislav Fort, Adam Scherlis (AAAI 2019, https://arxiv.org/abs/1807.02581). They link this measure to the easy of optimization on random low-dimensional sections of the weight space.\n\n[2] There is a nice overview of the structure of the Hessian in Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra\nby Vardan Papyan (https://arxiv.org/abs/2008.11865). In your spectra you seem to be observing (C-1) outlying eigenvalues as observed / expected in many papers. \n\n[3] Emergent properties of the local geometry of neural loss landscapes by Stanislav Fort, Surya Ganguli (https://arxiv.org/abs/1906.04724) forms a model of the Hessian based on the logit gradient clustering properties and also looks at measures of flatness in that regime and its dependence on the weight norm.\n\n### 5. Summary\nOverall I like the paper. I am a bit unsure about the generality of the derivations and the gedanken model, but the experiments show that at least in some cases flatness does not correlate well with generalization. I think it is important to appreciate the role of the weight norm and given that (possibly) some do not, this is a useful addition to that effort. I'm not clear on whether this insight is original (I am ready to be corrected) but if it were I think this is a valuable paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HZ0JSmwUgmf", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of this article experimentally investigate whether flatness of the loss surface can be a good measure for generalization capabilities of neural networks. They present theoretical reasoning that suggests that weight regularization can lead to sharper local minima and better generalization although it is expected that flatter minima generalize better. Several experiments are conducted that support this interplay of weight regularization with sharpness. Finally it is demonstrated that also different optimization techniques lead to results that question the validity of purely flatness based measures.\n\nThe experiments are well-described and show the interesting behavior how weight regularization affects generalization and flatness for several models and datasets (logistic regression on MNIST, 2-layer neural network on MNIST, 9-Layer CNN on Cifar100, ResNets on Cifar100). The experiments take advantage of an open-source Pytorch implementation (described in a preprint and based on Laczos algorithm) to find the spectrum of the Hessian. The theoretical reasoning is nicely presented first for a linear 1D model explaining the intuition behind the approach and then considered in the more general setting. The theoretical argument is based on large weights during convergence and supports that regularization by weight reduction increases generalization and sharpness. The conclusion of the article that flatness alone is a misleading measure and cannot explain the generalization performance of deep neural networks.\n\nDespite these interesting experiments, I recommend to reject for the following two reasons:\n\n1.) The conclusion of the article is not novel and therefore does not offer any new insight. There are several articles, also mentioned in the paper, that realize the same fact and suggest different measures (not considered in this submission) to address this short-coming of purely flatness-based measures. Taking the results of other articles into account (e.g. Neyshabur et al., 2017; Tsuzuku et al., 2019; Rangamani et al., 2019) the result that adding L2 regularization increases sharpness and generalization is not counter-intuitive. Therefore, the paper only offers additional experimental evidence that flatness alone is not a good measure.\n\n2.) The proof of the theoretical result (Theorem 1) appears to be incorrect in its current form (see below). If the theory can be saved, the theoretical application does not seem to explain the observations as the argument is based on (the product of) weight norms converging to infinity, while the L2 norm is bounded and decreasing in some experiments (see Figure 6 and 7).\n\nI would appreciate a clarification on the extent of the contribution and clarification of the theoretical part.\n\n\nIssue in the proof to Theorem 1:\nAt the bottom of page 4, the weights cannot be bounded below by a value strictly larger than epsilon without a loss of generality: The symmetries of Dinh hold at every finite point, so at each weight setting we can reduce one weight arbitrarily close to zero while accommodating the change in another weight. \n\n\n\nOther issues:\n- Equation (1) only holds at a critical point of the loss surface\n- Below equation (1). It is not because of small gradient that higher order terms can be dropped in an approximation. Instead, bounded higher derivatives and small delta w are necessary for the approximation to hold\n- End of Motivation: The argument that reparameterizations are not an issue under L2 regularization is incorrect. The flatness measure can still be altered in that setting.\n- Below equation (9): A ReLU network is not real analytic, and it is unclear what the \"specifically\" sentence should say. It is obvious that ReLU networks are differentiable everywhere except at a set of measure zero, is this all that is meant here?\n- Equation (10) is incorrect at several places (A_{i,j} - what is j? What is gamma \\setminus a product? ....)\n- Equation (10) To denote a higher derivative with products in the denominator is nonstandard\n- Third line on page 3: The bracket is bounded by 1 by definition, this requires no explanation about contributing paths. Maybe this can be explained by the typos in (10)?\n- Figure 6&7: Does it show the val accuracy (according to text), val error (according to axis label) or test error (according to caption)?  \n- Figure 6&7, please label the two curves of identical color (although one can deduce which curve shows what)\n\n\n\nTypos\n- Around equation (5) both layer and vector components are indexed with ‘q’\n- After equation (5), typo „the the“\n- one line below that, the dot is at the wrong position\n- before equation (7), z denotes both softmax input and softmax output\n- Equation (10)\n - Line 5 of Experimental setup, verb is missing\n- Figure 6 caption „Aadam“", "extra_scores": {}, "extra_text": {}}, {"review_id": "9kxrAz64ez-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper argues that as the cross-entropy loss goes to zero, since the correct logit increases in magnitude the entries of the Hessian diminish to zero. Such overfitting on the training set and a small spectral norm of the Hessian should result in poor generalization error. Motivated by this, the paper experimentally evaluates the effect of weight decay on the Hessian controls the magnitude of weights, increases the spectral norm of the Hessian and improves the generalization.\n\nThis paper presents the beginning of an interesting argument but the thesis of the study is unclear. Hessian-based measures of sharpness come with caveats, as has been explored in the literature cited in the paper. The conclusions of the paper have been widely discussed in these existing papers and it is therefore the incremental value of this paper’s results is difficult to ascertain.\n\nI do not believe I understand what the paper is arguing here. It is clear that exponential losses such as the cross-entropy loss result in weights that can have a large magnitude. It is also clear from Section 3 that entries of the Hessian go to zero if the weights increase. It is also known that the spectral norm of the Hessian can be made arbitrarily small/large without changing the input-output mapping of the deep network and thereby the generalization performance. So although the experiments are sound, the conclusion that “flatness is a false friend” does not follow from these experiments.\n\nThe experiments are sound but the numerical accuracies are very poor; Fig 2 shows 3.3% validation error on MNIST, Fig 3 shows 45.2% error on CIFAR-100, Fig. 4 does have a stronger error of 19.4%. The authors are advised to use state of the art data augmentation and regularization techniques in these experiments.\n\nThe paper advocates the need to understand the magnitude of weights in addition to spectral properties of the Hessian. The reviewer agrees with this point of view but the present paper does not offer any concrete insights into this issue.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The novelty of the claims of the paper has been challenged by one of the reviewers, and in addition, one reviewer also raised concerns about the validity of the proof of the main result (Theorem 1). I looked into the proof myself, and I agree with these
concerns. The authors did not use the rebuttal phase to clarify these issues.
Decision: Reject