On Flat Minima, Large Margins and Generalizability
Authors: Daniel Lengyel
Nicholas Jennings
Panos Parpas
Nicholas Kantas
Venue:  (2021)

Abstract: The intuitive connection to robustness and convincing empirical evidence have made the flatness of the loss surface an attractive measure of generalizability for neural networks. 
     Yet it suffers from various problems such as computational difficulties, reparametrization issues, and a growing concern that it may only be an epiphenomenon of optimization methods. 
     We provide empirical evidence that under the cross-entropy loss once a neural network reaches a non-trivial training error, the flatness correlates (via Pearson Correlation Coefficient) well to the classification margins, which allows us to better reason about the concerns surrounding flatness. 
     Our results lead to the practical recommendation that when assessing generalizability one should consider a margin-based measure instead, as it is computationally more efficient, provides further insight, and is highly correlated to flatness. 
     We also use our insight to replace the misleading folklore that small-batch methods generalize better because they are able to escape sharp minima. Instead, we argue that large-batch methods did not have enough time to maximize margins and hence generalize worse. 

Under review as a conference paper at ICLR 2021
ON FLAT MINIMA, LARGE MARGINS AND GENERAL- IZABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
The intuitive connection to robustness and convincing empirical evidence have made the flatness of the loss surface an attractive measure of generalizability for neural networks. Yet it suffers from various problems such as computational dif- ficulties, reparametrization issues, and growing concern that it may only be an epiphenomenon of optimization methods. We provide empirical evidence that un- der the cross-entropy loss once neural network reaches a non-trivial training error, the flatness correlates (via Pearson Correlation Coefficient) well to the clas- sification margins, which allows us to better reason about the concerns surround- ing flatness. Our results lead to the practical recommendation that when assessing generalizability one should consider a margin- measure instead, as it is com- putationally more efficient, provides further insight, and is highly correlated flatness. We also use our insight to replace the misleading folklore that small- batch methods generalize better because they are able to escape sharp minima. Instead we argue that large-batch methods did not have enough time to maximize margins and hence generalize worse.
INTRODUCTION
Understanding under which conditions a neural network will generalize from seen unseen data is crucial, as it motivates design choices and principles which can greatly improve performance. Com- plexity or generalization measures are used quantify the properties of a neural network which lead to good generalization. Currently however, established complexity measures such as VC-Dimension (Vapnik, 1998) or Rademacher Complexity (Bartlett & Mendelson, 2002) do not correlate with the generalizability of neural networks (e.g. see Zhang et al. (2016)). Hence many recommendations, such as reducing model complexity, early stopping, or adding explicit regularization are also not applicable or necessary anymore. Therefore, there is ongoing effort to devise new complexity measures that may guide recommendations on how to obtain models that generalize well.
A popular approach is to consider the flatness of the loss surface around a neural network. Hochre- iter & Schmidhuber (1997) used the minimum description length (MDL) argument of Hinton & Van Camp (1993) to claim that the flatness of minimum can also be used as generalization measure. Motivated by this new measure Hochreiter & Schmidhuber (1997), and more recently Chaudhari et al. (2019), developed algorithms with explicit regularization intended to converge to flat solutions. Keskar et al. (2016) then presented empirical evidence that flatness relates to improved generalizability and used it explain the behavior of stochastic gradient descent (SGD) with large and small- batch sizes. Other works since have empirically corroborated that flatter minima general- ize better (e.g. Jiang al. (2019); Li al. (2018): Bosman al. (2020)).
There are however various issues that are still unresolved, which makes using flatness for construct- ing practical deep learning recommendations difficult. For one, flatness is computationally expen- sive to compute. The most common way to compute the flatness is via the Hessian, which grows quadratically in the number of parameters; this becomes too large when used with modern networks containing millions of parameters. It is also not clear to what extent flatness is a true measure of generalizability, capable of discerning which neural network will or will not generalize. Dinh et al. (2017) showed that reparametrizations affect flatness and flat model can be made arbitrarily sharp without changing any of its generalization properties. In addition Probably Approximately Correct (PAC-Bayes) bounds that bound the generalizability in terms of the flatness are also either affected

---

Under review as a conference paper at ICLR 2021
by rescaling, impossible to evaluate loose (Neyshabur et al., 2017; Arora et al., 2018; Petzka et al., 2020). While there have been solutions attempting to prevent issues around reparametrization (Liang et al., 2019; Tsuzuku et al., 2019). it remains to establish whether flatness is an epiphe- nomenon of stochastic gradient descent or other complexity measures as Achille et al. (2018) and Jastrzebski et al. (2018) suggesting. This motivates investigating possible correlations more well-understood measures of generalization that may help alleviate issues surrounding flat minima, while allowing flat minima to be used when appropriate.
In this paper we will demonstrate a correlation to classification margins, which are a well-understood generalization measure. Margins represent the linearized distance the decision boundaries of the classification region (Elsayed et al., 2018). An immediate consequence of such a relationship is that to assess generalizability. we could now simply use a computationally cheap and more robust margin based complexity measure. Our contributions will demonstrate further practical implications of the relationship between margins and flatness which open doors to valuable future work such as a better understanding of why and when model generalizes and more principled algorithm design.
We prove that under certain conditions flatness and margins are strongly correlated. We do so by deriving the Hessian trace for the affine classifier. Based on its form, we derive an expression in terms of classification margins which we show correlates well with the Hessian trace, with increasing training accuracy for various neural network architectures. By being able relate the two complexity measures, we are now able to provide various practical recommendations, and offer different perspectives phenomena that may not be explainable without such a view. These are shown in the following contributions.
We use our insight to replace the misleading folklore that, unlike large-batch methods, small-batch methods are able to escape sharp minima (Keskar al., 2016). We instead employ margin perspective and use our empirical results along with recent results by Banburski al. (2019) and Hoffer al. (2017) to argue that a large batch method was unable to train long enough to maximize the margins. With our explanation, we help re- frame the small and large-batch discussion and build further intuition.
We show that once a neural network is able to correctly predict the label of every element in the training set it can be made arbitrarily flat by scaling the last layer. We are motivated by the relationship to margins which suffer from the same issue. We highlight this scaling issue because, in some instances, it may still be beneficial for algorithm design to be guided by convergence to flat regions. Hence, we need to account for scaling issues which make it difficult to use flatness to assess whether network generalizes better than another.
Other works have made connections between flatness and well- well-behaved classification margins via visualizations (see Huang al. (2019); Wang et al. (2018)), but they have not demonstrated quan- tifiable relationship. Further work has used both the classification margins and flatness to construct PAC-Bayes bounds (Neyshabur al., 2017; Arora et al., 2018), and have related flatness increased robustness (Petzka et al., 2020; Borovykh et al., 2019) however they did not show when and to what extent these quantities are related.
We structure the paper as follows. In Section 2, we discuss both our notation and our motivation choosing the cross-entropy loss and the Hessian trace as the flatness measure and provide further background on the classification margins. In Section 3, we present our contribution showing a strong correlation between the margins and flatness by deriving. In Section 4, we combine recent results based on classification margins to offer a different perspective on the misleading folklore on why larger-batch methods generalize worse. In Section 5, we highlight that networks can be made arbitrarily flat. Lastly, we offer our thoughts and future work in the Section 6.
PROBLEM SETTING
We first define the basic notation that we use for a classification task. We let X' represent the input space and y ..., C} the output space where C are the number of possible classes. The network architecture is given by ø : e X' RD'l where θ is the corresponding parameter space. We measure the performance of a parameter vector by defining some loss function & : RC x x + R. If we have have joint probability distribution D relating input and output space then would
2

---

Under review as a conference paper at ICLR 2021
y)]. Since usually only have access to some finite dataset D, we denote the empirical loss by (D) = ( El ((o(0, zz), ye)- If LD and ED are close, then we would say model generalizes well, as we were able to train on a finite dataset and extrapolate to the true distribution. We will use the cross-entropy loss which is given by L[0(0, x), y) = - log( ( (0(( x))) where the softmax function S : R + R is given by 2a (aa): Σ ΣS=re ." (see Goodfellow et al. (2016)).
like minimize the expected loss (0 ((r.))-D((((((.2)))))))
The choice of the cross-entropy function as the loss function has significant impact on how the flat- ness measure behaves. Unlike the multiclass mean squared error (MMSE), exponential type losses such as the cross-entropy loss neural networks have been shown to include implicit regularization which leads to margin maximizing solutions for neural networks (Banburski et al., 2019). Also, var- ious properties for flat minima which have been proven for the MMSE loss by Mulayoff & Michaeli are not applicable to the cross-entropy loss, further highlighting the fundamental differences be- tween the loss functions. While the MMSE loss has shown some promise for many classification tasks (Hui& & Belkin, 2020) the cross-entropy loss is still the loss which is most used and was primar- ily used for the empirical evidence around flat minima (Keskar et al., 2016; Chaudhari al., 2019), which motivates our choice.
The qualitative description of a flat region was given by Hochreiter & Schmidhuber (1997) as "a large connected region in parameter space where the error remains approximately constant". We measure the flatness by the trace of the Hessian of the loss with respect the parameters (in short the Hessian trace) denoted by Tr Tr(Ho (CD() (Dinh et al., 2017). Since the Hessian is symmetric, the Hessian trace is equivalent to the sum of its eigenvalues which for a fixed parameter space is proportional the expected increase of the second order approximation of the loss around fixed minimum 8 in a random direction 6' with 6' ~ (6, I). Since we apply flatness arguments only close to minima, we assume that all eigenvalues are positive and that the Hessian trace is a good measure of flatness Sagun et al. (2017). Even though the Hessian is only an approximation of flatness, the Hessian is often preferred as it allows us to reason about various directions in parameter space via its cigenvectors and eigenvalues (see Sagun et al. (2017); Chaudhari et al. (2019)) and alleviates the issue of infinitely long but sharp ridges making a minimum infinitely flat (Dinh et al., 2017; Freeman & Bruna, 2016). The Hessian has also been linked to feature robustness via its use in the second order approximation of the loss (e.g. Petzka et al. (2020); Borovykh et al. (2019)) and a promising quantity to relate the margins.
As we are working with non-linear functions it is intractable to compute exact distances to the de- cision boundary, therefore we use measure which is related to the linearized distance as described in Elsayed et al. (2018). Under this view, larger margins are better because the data is further from the decision boundary. Specifically, we define the margins as in Neyshabur et al. (2017): for some vector CRC and label y we let the margin of be (F, = l0g maxj Uyl- Since we use the margin in different contexts we define the output margins ((((f, x)) y) and the margins of the model output after the softmax layer (S()(0,x))) x)), v). Due to the intuition of margins relating to the regularity of the classification regions, they have been proven and shown to be good generalization measure for linear networks (Langford & Shawe-
2003) and later for neural networks (see Bartlett et al. (2017); Jiang et al. (2018; 2019)) when correctly adjusted. Due to results by Banburski et al. (2019) and Soudry et al. (2018), Poggio et al. (2019) claimed that large part of the mystery around generalizability has been solved, since standard optimization methods are maximizing the margin instead of memorizing data.
THE MARGIN AND HESSIAN TRACE RELATIONSHIP
3.1 THE AFFINE CROSS-ENTROPE ENTROPY HESSIAN TRACE
Generally, it is difficult to derive a closed form solution of the Hessian trace due to the non-linear nature of neural networks. To gain insight into what may determine the flatness or sharpness of a solution we consider an affine prediction function for which we derive the following simple and insightful expression for the Hessian trace:
3

---

Under review as a conference paper at ICLR 2021
Proposition 3.1 (Affine Cross-Entrops Hessian Trace (ACEHT)). Assume affine predictor given by (( b)) = fx + where (8, E RCxd x EC = e. Then the rrace of the Hessian under the 2)
cross-entropy loss assuming our fancctio is:
C
Tr(H(((6((0))))))).))) = (1==3 + 1)(1 Σ ()()
j=1
-1x1 + 1)(L=1(((00.2))
The derivation is in Appendix C. We immediately observe that the trace the Hessian a product of both the size of the input and - ((S(((0, ())) where ()((( x))) =
S - ()(O,))) where we can view 1 - ((S(((( (0, x))) as a confidence measure. In the visualization provided in Figure we clearly see that 1 (((( is only zero when the predictor predicts one class with probability 1, regardless of whether it is the correct class or not. When the model is least confident, namely when every entry is predicted with probability 1/C, then (((((0, x))) is also highest. Hence, in the affine case with cross-entropy loss the Hessian trace can be seen as an indication of the model confidence in its prediction. This confidence interpretation is also connected to classification margins by observing that S, 2 ((S(6(0, x)), y) and hence (1 - Eÿ S ()((, x)) S - (2((((O, s 1 11--7 (S)(((0)
Therefore, the margins are large then the region will also flat. The intuition for this is that the error in the upper bound becomes smaller as Sy becomes larger, i.e. when the model predicts correctly and confidently. We will also provide evidence for converse, i.e. flat minimum has large margins, in the following experimental sections. Finally, we note that without the expression in Proposition 3.1 we would not have been able to derive the upper bound - 2 (S((0 x)), y) without guesswork.
LO
To visualize how 1 7/( (((((, xx))) represents the confidence of a model's prediction we plot 1 - (aa for all a E R&3 such that is a valid probability distribution over three classes (i.e. for all the elements of the standard 2- 2immpe). Since there only two free variables, x and y in the plot represent a by a = [z. y, - x y/. We see that - r7 (a) is only zero when a = e, for some i, namely when model would be most confident. We also note that 1 nn(a) is largest when model would be least confident in its prediction-i.e when a = [1/3, 1/311/3)
Figure 1:
3.2 EXTENSION TO THE NON-LINEAR CASE
Now we will attempt extend the derivation of the previous section to the non-linear case. This is challenging undertaking so we will resort to numerical evidence. To extend the results from the affine case we will consider both the ACEHT and the upper bound AC (CEHT((((((,,))) |x|(1 - (S tities to the empirically derived Hessian trace. To compute the empirical Hessian trace we use the PyHessian package (Yao et al., 2019) which implements Hutchinson's method (Bai et al., 1996; Avron Toledo, 2011).
s y)) to which we refer as the "margin bound". We will compare both quan-
To compare the quantities we will compare them in terms of their distributions over the data. Specifically,, let (X, Y) ~ D and fix then we compute the Pearson Correlation Coefficient (r- value) (Lee Rodgers & Nicewander, 1988) between the random variables 7 Tr Tr(H(e(á(O, X)), Y)) and ACEHT(S((() ()), and similarly for the margin bound. The choice of the r-value natural because in the affine case the ACEHT and the Hessian trace equivalent, therefore linear rela- tionship should be expected. Our method is also more general than just comparing some statistic, such the average (which is generally used for flatness measures), of the above random variables.
4

---

Under review as a conference paper at ICLR 2021
For example, while the smallest margin over the dataset is commonly used generalization measure (Bartlett et al., 2017; Jiang et al., 2019; Neyshabur et al., 2017). Jiang et al. (2018) showed that higher moments of the distribution see in Section 4.
much better predictor for generalizability as will also
Figure 2 is examples of such fit for an affine predictor. While the high r-value of 0.97 confirms our analytic results, we also observe that the is not perfect, as would be expected due to the exact relationship. The inaccuracies are due to the numerical methods used and become more pronounced the higher the Hessian trace is. To avoid outliers heavily impacting the linear regression model in the non-linear case. we will use the SciPy function LocalOutlierFactoo (Breunig et al., 2000) to remove outliers before fitting the line. With this we prevent hand picking points to skew the results and will also stabilize our results.
Figure 2: We consider the afiin predictor ((00,)),a) = 0' + with arbitrarily chosen parameters (9, b) on 1, 000 randomly sampled datapoints from the MNIST dataset. Each scatter point represents a datapoint for which we compute the ACEHT and the empirical Hessian trace. The linear relation- ship between the ACEHT distribution and empirical Hessian trace both confirms our derivation of the ACEHT and provides a baseline for the numerical methods used for the empirical Hessian trace.
3.2. EMPIRICAL EVIDENCE
We present our results using the convolutional neural network LeNet on the MNIST dataset as they representative of what have observed on other architectures, hyperparameters, and datsets (see Appendix B). Our results use stochastic gradient descent with fixed learning rate and batch size achieve an appropriate performance on the classification task. Because of the computational difficulty of computing the empirical Hessian trace for every single element in the input data, we consider 1,000 randomly selected datapoints from the training-set. To highlight the computational difficulty of using even very optimized numerical tools, such as PyHessian, we note that it takes us roughly 1,5 hours to compute the Hessian trace for the whole MNIST dataset while it only takes seconds for the margins.
In Figure 3 we present the plots related to the correlation of the empirical Hessian trace to the ACEHT and margin bound over the randomly sampled datapoints. Figures 3a and 3b show that for most of training, the correlation is between 0.8 and 1. Combining Figures 3a and 3c can be seen that the r-value increases with the model training accuracy. Furthermore, the datapoint which incorrectly predicted do not show a correlation.
With that confirm the intuition that indeed, flatter solution more robust and have larger mar- gins. While we have found flatness and margins to be highly correlated in scenarios in which others have identified flatness to be good generalization measure (Jiang et al., 2019; Keskar al., 2016; Chaudhari et al., 2019), it may just be that this is also an epiphenomenon of stochastic gradient descent or some other process and there may be situations in which the relationship does not hold. However, our general advice to consider margins more is not impacted by this. In the scenario where generalizability and flatness have been linked, we have also shown that margins and flatness are cor- related, hence it is advantageous to use margins instead due to computational reasons or for more complete intuition. The only situation in which it is more likely that margins and flatness are not correlated is when flatness has not yet been linked to generalizability. In such a situation it may also be better to use the better understood margin measure instead using a flatness measure to assess generalizability. In the next section will consider the first case, where examine general scenario in which flatness has been used to reason about generalizability and offer a more insightful margin perspective.
5

---

Under review as a conference paper at ICLR 2021
Broekeyy Probler
600
(a) ACEHT to HT
(b) SM Bound to HT
(c) Training Accuracy
Carect
(d) Initialization
(c) Step 5000
(I) Final Step (10,000)
Figure 3: We observe how ACEHT and the softmax margin (SM) bound relate. via the Pearson Correlation Coefficient, to the observed Hessian trace (HT) of LeNet trained on MNIST. In Figures 3a 3b see that for correctly classified datapoints (orange) the empirical Hessian trace correlates well with the ACEHT and SM bound. In Figure 3c we observe that the increase in correlation occurs with an increase in training accuracy. To demonstrate the evolution of the distributions throughout training we plot the ACEHT and empirical HT distribution against each other in Figures 3d 3e 3f. We observe that while the most apparent outliers were removed, some still skew the linear regression.
PERSPECTIVE ON LARGE AND SMALL-BATCR METHODS
We now show how our results lead to a better understanding of phenomena which have been mis- leadingly attributed flat minima. To do so, we consider the experiments which rekindled the debate around flat minima by Keskar et al. (2016), where flatness was used to explain why small- batch methods tend to generalize better than large-batch methods. The idea was that small-bater methods converge to flatter minima due to them being able to "escape" sharp minima more easily. However, it has been shown that the minima of both methods appear to be the same attractive basin (Sagun et al., 2017; Freeman & Bruna, 2016; Draxler et al., 201 8), meaning that small-bater methods do not seem to escape any attractive basin but are merely in a different area the same attractive basin. While the results gave credence to flatter minima generalizing better, flatter minima do not seem to provide the full picture for why large-batt batch methods tend to do worse and we believe that explanation in terms of the margins is more illuminating.
4.1 EXPERIMENT SETUP
We will replicate the experiment by Keskar et al. (2016) for a fully connected network with batch- normalized layers on the MNIST dataset described in Appendix A. We chose the large-batch size to be 4096 and the small-batch size to be 256. To have a fair comparison, we use the same seed and take 10,000 gradient steps for both methods, instead of basing the stopping time on epochs. We also used stochastic gradient descent without Momentum. With our setup we observe a similar phenomenon as Keskar et al. (2016) in Table 1. The small and large-batch method both attain the same training accuracy and comparable training loss. However, the small-batch method is at considerably flatter minimum and generalizes better than the large-batch method. We will now show that instead of considering the flatness. it would be more insightful to consider margins to explain the difference in generalizability.
While the upper bound of ACEHT is in terms of the softmax margins, we consider the output mar- gins in this section. The reason is that most margin based generalization measures use the output margins. Another more practical reason is that towards the end of training, the softmax margins all very close to making it difficult to visualize and observe the distribution. We also do not use a normalized version of the margins (such as Bartlett et al. (2017); Jiang et al. (2018)). Our reasoning

---

Under review as a conference paper at ICLR 2021
is that because use the same architecture, the same dataset, and train in similar manner the margin distributions will be comparable.
Loss
Accuracy
Trace Training
Batch Size
Training
Test
Training
Test
256
2.1 x 10-3
7.28 x 10-2
1.0
0.9834
1.01
4096
3.4 x 10-5 9.45 x 10-2
1.0
0.9794
6.62
Table 1: The results of our trained fully connected network with batch-normalized layers on MNIST optimized with SGD and a 0.1 learning rate. The results reflect the observations made by Keskar al. (2016). Le. the
method has a smaller Hessian trace and generalizes better.
4.2 A MARGIN PERSPECTIVE ON LARGE AND SMALL-BATCH SIZES
In Figure 4 we see that the output margins and the Hessian trace are correlate as expected from Section 3. We can also roughly see that the small-batch method has fewer low margins than the large- batch methods. To emphasize this difference we consider Figure 4c where box-plot of the output margin distribution for both the large-batch and small-batch method. We also display the skewness of each, which is the third moment centered around the mean. The box-plots and the skewness confirm that the small-batch method is dominated by large margins indicating better generalizability (as discussed in Bartlett al. (2017); Jiang et al. (2018)). The idea with left-skewed margin distribution is that the tail with low margin datapoints is mostly compromised of outliers and will not massively affect the robustness to input perturbations. This soft coft-margin SVM perspective is in contrast to ward-nargin SVMs where the margin is defined to be the minimum of all the distances to the decision boundary "Shalev-Shwards was adopted, then the small-batch method would be predicted to generalize worse, because it has the smallest margin as we see in Figure 4c. However, the distribution of the small-batch method is also more left skewed, which would point to this minimum being outlier rather than being indicative of generalizability.
plot the histogram and
Ben-David, 2014). a hard-margin view
We now want to explain why the small-batch method generalizes well. As observed in Jastrzębski et al. (2017) a smaller batch-size is similar to larger learning-rate, hence at every step the process will advance further than large batch-method would. It has already been noted by Hoffer et al. (2017) that training longer leads the large-batch method to generalize just as well as the -malh-hatch method because it had time "catch up". even though the decrease in training loss may be barely noticeable. We have also seen that SGD converges to margin maximizing solutions by Banburski al. (2019). Therefore, method that is able train or advance further, will also be closer margin maximizing solution. We therefore expect that large-batch methods not having had enough time to maximize margins is the driving force behind the large vs small-batch phenomenon.
Sutiel Megen
(a) Batch Size: 256
(b) Batch Size: 4096
(c) Box-Plot
Figure 4: We plot both small-batch method (orange) and large-batch method (blue). In Figures 4a and 4b we plot the output margins against the Hessian trace for each datapoint. We observe strong relationship between the Hessian trace and the output margins. In Figure 4c we plot both the histogram and box-plot and display the skewness (the third standardized moment) for both the large and bathh method's 's margin distributions. We observe that the distribution of the small-bater method is more left skewed which would indicate better generalizability independent of the flatness.
7

---

Under review as a conference paper at ICLR 2021
5 BECOMING FLATTER WITH INCREASING MARGINS
Reparametrization problems such as shown by Dinh et al. (2017) are neither a new phenomenon nor should they necessarily discourage the design of algorithms which attempt find flat minima. Rather they inform on what aspects of generalization measure need to be adjusted to allow them to be used in a practical setting. For SVMs, the problem of scaling the hyperplane normal to increase margins of correctly classified points is solved by scaling the normal to make it a unit vector, trans- forming the functional margin into the geometric margin "Shalev-Shwarts & Ben-David, 2014). In the case of neural networks, it is also known that scaling the last layer leads to an increase in the margins for data which has been correctly predicted (Neyshabur et al., 2017). This scaling issues has been successfully addressed (see Bartlett et al. (2017); Elsayed et al. (2018); Jiang et al. 20) Due to the relationship to the classification margins it is natural to ask if flatness suffers from a similar problem. We confirm this with the following Proposition:
Proposition 5.1. For a given neural network & let Ta θ +
O be such that for all x E & and E θ we have (To (0), = av(10, x). Now assume that O' E and a datapoint (x', y d') for which argnazze(L.....r (0(0, x ))& = y then
Vs,t {L...., (O) lim 0o ( 2())) =
a-000
The proof is in the Appendix D. From the Proposition we immediately derive the following Corol- lary: Corollary 5.2. Assume that o and f predict every datapoint in a set D correctly then
Vs, I E dim( lim Oo, 0o (T( =
Due to the Corollary, if network has achieved full training accuracy. then the network is equivalent under the Ta transformation to an arbitrarily flat network. We note that there exists such a Ta trans- form for most networks. Scaling the last layer is one simple instance of such a transform. Another is that for fully connected and convolutional networks with ReLU non-linearities we observe that by the non non-negative homogeneity scaling each layer also results in a valid Ta transformation. The crucial property of the To map is that it does not change the relative order of the model outputs and therefore, given two networks which have achieved full training accuracy we can not determine which network should generalize better based solely the flatness of the local-geometry We note that Banburski et al. (2019) mentioned such an issue but they did not discuss it in the context of flat minima and their arguments relied on further structure which we believe is less illuminating than our presentation and proofs.
6 CONCLUSIONS
In this paper, we have related flatness to the classification margins in a principled manner, in contrast to other works that have made more intuitive or less quantifiable connection (Huang et al., 2019; Wang et al., 2018; Neyshabur et al., 2017: Petzka et al., 2020). Our results lead to the immediate practical recommendation of using margins instead of the computationally expensive flatness to assess generalizability. We also use our results to replace the misleading notion that small-batch methods generalize better because they "escape" sharp minima, instead arguing that small-batch methods have more time to maximize margins. We were also motivated by the flatness and margin relationship to highlight that neural networks can be made arbitrarily flat. This implies that the generalizability of two networks can not be distinguished based on flatness and hence needs to be addressed to make flatness a viable generalization measure. Based on our results, future work may assess whether flatness is an epiphenomenon of the optimization methods, because now recent work on margins (e.g. Banburski et al. (2019); Soudry et al. (2018) can be applied to reason about flatness. Furthermore, by relating properties of the parameter space (flatness) properties of the input space (margin) there is now an opportunity to further explore results such as by Sagun et al. (2017), where they found that the Hessian, with respect to the parameters of a neural network upon convergence, has as many positive eigenvalues as the number of classes in the dataset used. Overall, our results enable more principled discussion on how flatness may contribute generalizability.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations, 2018.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:/80..05296, 2018.
Haim Avron and Sivan Toledo. Randomized algorithms for estimating the trace of an implicit sym- metric positive semi-definite matrix. Journal of the ACM (JACM), 58(2):1-34, 2011.
Zhaojun Bai, Gark Fahey, and Gene Golub. Some large-scale matrix computation problems. Journal of Computational and Applied Mathematics, 74(1-2):71-89, 1996.
Andrzej Banburski, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Jack Hidary, and Tomaso Pog- gio. Theory ii:: Dynamics and generalization in deep networks-a simple solution. arXiv preprint 1903.0999. 2019.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482 2002.
Peter L Bartlett, Dylan Foster, and Matus Telgarsky. Spertrally-normalizedd margin bounds for
neural networks. In Advances in Neural Information Processing Systems, pp. 6240-6249, 2017.
Anastasia Borovykh, Cornelis W Oosterlee, and Sander M Bohté. Generalization in fully- mulle-connected neural networks for time series forecasting. Journal of Computational Science, 36: 101020, 2019.
Anna Sergeevna Bosman, Andries Engelbrecht, and Mardé Helbig. Visualising basins of attraction for the cross-entropy and the squared error neural network loss functions. Neurocomputing. 2020.
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93-104, 2000.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 124018, 2019.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1019-1028 JMLR. org. 2017.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A Hamprecht. Essentially no barri- ers in neural network energy landscape. arXiv preprint arXiv:1803.068 00088888... 2018.
Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. Advances in neural information processing systems, pp. 842-852, 2018.
C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. arXiv preprint 161 1.01500, 2016.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5-13, 1993.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1):1-42, 1997.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, pp. 1731-1741, 2017.

---

Under review as a conference paper at ICLR 2021
W Ronny Huang. Zeyad Emam, Micah Goldblum, Liam Fowl, Justin K Terry, Furong Huang, and Tom Goldstein. Understanding generalization through visualizations. arXiv preprint 1rXXv::906.03291,, 2019.
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross- entropy in classification tasks. arXiv preprini arXiv:2006.0732) 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Stanistaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623, 2017.
Stanislaw Jastrzebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. On the relation between the sharpest directions of dnn loss and the sgd step length. arXiv preprint 180. 1, 2018.
Yiding Jiang. Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. arXiv preprint arXiv:1810.00113, 2018.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic 2019.
generalization measures and where to find them. arXiv preprint
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint a6Xiv: 16000008669 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
John Langford and John Shawe-Taylor. Pac-bayes & margins. In Advances in neural information processing systems, pp. 439-446, 2003.
Yann LeCun, Léon Bottou, Yoshua Bengio. and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2322, 1998.
Joseph Lee Rodgers and W Alan Nicewander. Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1):59-66 66, 1988.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In Advances Neural Information Processing Systems, pp. 6389-6399, 2018.
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metrie, ge- ometry, and complexity of neural networks. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 888-896, 2019.
Rotem Mulayoff and Tomer Michaeli. Unique properties of flat minima in deep networks.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization deep learning. In Advances neural information processing systems, pp. 5947-5956, 2017.
Henning Petzka, Linara Adilova, Michael Kamp, and Cristian Sminchisescu. Feature-robustness, flatness and generalization error for deep neural networks. arXiv preprint arXiv:2001.00939, 2020.
Tomaso Poggio, Andrzej Banburski, and Qianli Liao. Theoretical issues in deep networks: Approx- imation. optimization and generalization. arXiv preprint urXiv:1908.09373 2019.
Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of parammaraeerrized neural networks. arXiv preprint (rXiv:/706.0445: 2017.
10

---

Under review as a conference paper at ICLR 2021
Shai and Shai Ben-David. Understanding machine learning: From theory algo- rithms. Cambridge university press, 2014.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19 (1):2822-222 2878, 2018.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Normalized flat minima: Exploring scale invariant definition of flat minima for neural networks using pac-bayesian analysis. arXiv preprint arXiv:1901.0465: 2019.
Vladimir Naumovich Vapnik. Statistical learning theory. Adaptive and learning systems for signal processing, communications, and control. Wiley, New York, 1998. ISBN 978-0-471-03003--
Huan Wang, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. Identifying generalization properties in neural networks. arXiv preprint (rXivv1800.074000 2018.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael Mahoney. Pyhessian: Neural networks through the lens the hessian. arXiv preprint arXiv: 1rXiv:1912.07145, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 161 2016.
A APPENDIX: NETWORK ARCHITECTURE AND DATASETS
A.1 NETWORK ARCHITECTURE
We implement the convolutional neural network LeNet-5 described in LeCun al. (1998). Our fully connected neural network with batch normalized layers (FCNBN) is inspired by Keskar et al. (2016). has 784-dimensional (MNIST) 1024-dimensional (CIFARIO) input layer followed by three narch-normmalied Ioffe & Szegedy (2015) layers with ReLU non-lincarities and a 10- dimensional output layer.
A.2 DATASETS
Dataset
Training Points
Test Points
Features
Classes
MNIST (LeCun et al., 1998) CIFARIO (Krizhevsky al., 2009)
60,000
10,000
28 x 28
10
50,000
10,000
32 32
10
Table 2: The datasets used for this paper.
B APPENDIX; FLATNESS AND MARGIN CORRELATION
Here we present further evidence of the flatness and margin correlation discussed in Section 3. Like in Section 3 we have used appropriate learning rates and batch sizes to get reasonable performance for the task, and have observed our results to hold for different hyperparameters. One instance where demonstrate two different batch-sizes is for the Fully Connected Network with Batch Normaliza- tion on MNIST (Section B. where we present results for a batch size 256 and 4096. We again only consider 1,000 randomly selected datapoints from the training-set due the computational difficult of computing the Hessian trace. If the network achieves full training accuracy and there are no incorrectly classified datapoints, set the r-value zero.
Overall, we observe the same results as in Section 3 and a correlation between 0.8 and 1. As before, the correlation increases with increasing training accuracy for correctly predicted datapoints.

---

Under review as a conference paper at ICLR 2021
B.1 FULLY CONNECTED NETWORK WITH BATCH NORMALIZATION ON MNIST
B.1.1 BATCH SIZE: 256
(a) ACEHT to HT
(b) SM Bound to HT
(c) Training Accuracy
Ceaet
Caenk oat
Oenest
Cenccl 6.95
Sueelt 0.00
(d) Initialization
Step 5000
Final Step (10,000)
B.1.2
BATCH SIZE: 4096
0.4
(a) ACEHT to HT
(b) SM Bound to HT
(c) Training Accuracy
Oemect
(d) Initialization
(c) Step 5000
(f) Final Step (10,000)
B.2 LENET ON CIFAR1O
(a) ACEHT HT
(b) SM Bound HT
(c) Training Accuracy
Corect
0.00
2001
8.77
0.33
(d) Initialization
(c) Step 5000
Final Step (10,000)
12

---

Under review as a conference paper at ICLR 2021
B.2.1
FULLY CONNECTED NETWORK WITH BATCH NORMALIZATION ON CIFARIO
1/4
04
Octadt Pralece
(a) ACEHT HT
(b) SM Bound HT
(c) Training Accuracy
Cirrec
(d) Initialization
(e) Step 5000
(I) Final Step (10,000)
C APPENDIX: DERIVATIVES OF THE CROSS Loss
C.I GENERAL FORM
For the general form we consider the cross-entropy loss for predictor function which is scaled by some scalar a. Specifically, we assume an arbitrary input-output pair (x, y) E X' x y and will compute the partial derivatives with respect to the parameters 0 of the predictor function ad(0, x)) Since the equations can become very long we will declutter the notation by letting S = S(ab(0, x)), @00 x) and for two d-dimensional vectors z, y Rd we write (z, v)
Σ -=1 Tit.. We also denote elementwise multiplication by @ and let be a matrix such that (Φ), Øj-
Lemma C.1. The first partial derivative of the cross-entropy loss with respect to an element 0; is:
0o, (0.(((((0,zz,,))) -a(8s, - Σ 0o. (i())) E
= =aa - 0..,0,
Proof.
0o. (acd,))))) = 0o. - log
0g. S,
(3)
S
With some manipulation compute 0o, Sy:
0g. a
S,,
Do,e adi
Σ ビm= caoe
R=1 cuda )2
eaps
eady
adi
= =ad, 0,
a
Og. 01
Ek=1 Eade
1 eaoh
eads
SSa( Dy
On, Si).
(4)
1=1
Combining Equations 3 and 4 we obtain Lemma C.I:
0g 00.((ap,1) 4) = -a(00, - Σ
0g P(Si).
13

---

Under review as a conference paper at ICLR 2021
Lemma C.2. The second partial derivative of the cross-entropy loss with respect to elements 8, and
8, is:
Do,0s, 0g (()((0)))) 1) -c(00,00, (0,,00, S) (0,, 0, as o (0g, Φ (0g. 4)5)))
Proof. Differentiating the first order derivative given by Lemma C.I we obtain by the multi-variable chain rule:
0o Do l(ad, v) -0(b, 0o. By (0p, 0o o, - (8s, 0, S)).
(5)
To compute Sg,5 S in Equation 5 Equation and obtain:
(do, S) do,
S,a (0,,0,5))
which after some simplification reduces to:
0g S(Ø) aS @ (da, Φ (0s, 4)
(6)
Combining Equations 5 and 6 we obtain Lemma C.2: de 0o, llao, v) = a (0o, 0o oy (0o, 0, o, (8o, o, as @ (06, Φ (0b, +)5)))
C.2 AFFINE
HESSIAN TRACE
We now present the proof of Proposition 3.1:
Proof. Throughout the proof we make use of Lemma C.2 and let a 1. We also notice that any second derivative with respect o(00, b), x)) is zero since is an affine classifier.
We first consider the derivatives with respect to elements of 0 where we use 05,j to denote the element in the ith row and jth column of the matrix 0. Notice that aa., Φ æje; which we write as æ).. The second order derivatives given by:
Da., So,, ((o, y) = (x7 @ (x) r,
=S,((,,(r))) S,)),
when computing the trace we only compute the elements on the diagonal hence we get:
, (0o,,((000 v) =x, (,,(S - x.)):)
= -1,
Now we consider derivatives with respect to elements of b and notice that Ob = e;. For the second derivative we then get:
0, Or, ((b.)) = (ej, S (e: ee)::)
= S.( S.).
Finally, summing up the diagonal of the total Hessian we get:
Tr(H(((,,,,))) (|x) + 1)(1 Σ $9)
where used the fact that Σ. S, 1.
14

---

Under review as a conference paper at ICLR 2021
D APPENDIX: SCALING PROOF
To prove Proposition 5.1 we first prove the following lemma:
Lemma D.1. Assume that the argmax of é is the correct class y and is unique then for k E N, 21 and # y:
lim at Si(a)) = 0
Proof. Let y be such that dy makke (1..... OR. For i # we have:
lim a Ea0,
a
lim a ca(dy-di) + E Ek= (a(
a0+00
Σx=rear eaor
k!
lim a-x (Øy c dil Ee E Ealte -b))
where the last line follows from applying I.' L'Hopital's rule k times. Since we assumed that is the only y E (1,-..N N) such that Øy max.e( O2, we have that ox By for all l: o y. Hence, as a + oc we have c - bu) + Therefore:
(0 (b))
lim (dy 0i)² + =K== (on - th (())ea((s-05)
We are now ready to prove Proposition 5.1:
Proof. We first show that the term -a² (0,, S @ (0o. Ø (0o, $, S)))) always goes to zero. Ex- panding get:
a²
Σ 0o, di (S)
a² 00. 0
t Sg. +
Σ
0o, di (S)(0,,
Σ do. ox
1m1,19
We now show that each term in the sum goes Consider I y::
C
2
Si do. S 02 Σ
di - Oo, Sx) (b) the Triangle Inequality and S S S
k=1
< S,C
We let M = Ee 0r 0, Sx) and note that M < oo for all 0 < a < 00 since 5 S < 0g Pi, do. Φi, Og. Φb are constants, and is a finite sum. By Lemma D.1 as - 0o have a S.C - 0 and hence a² E 0o, (80, Sx) + 0.
We now consider 1 y:
la? by By
Σd
dy Sx < a3)00, 100,0y - 0o,0y Σ k=1.4/
100 SKSA
1=1
15

---

Under review as a conference paper at ICLR 2021
C a² 0y - 0b, Sy 10b, 1a
2m=1 gaom
18,, la2
Σ
Sg since S, >0
1,
and using similar arguments and Lemma D.1 follows that this term zero in the limit.
is also obvious that 2 ΣE 10, S& goes to zero.
We are left with showing that (ddg 0 (āg, 0g o, S) goes to zero, this is only guaranteed when y is the true label). We will use the same method above:
0o.0,.0..0. (0o, p. S))l S al10,, 00.00 oy
Σ
00,,0o.00 01
1=1,6
and the result follows using again Lemma D.1.
16


Reviews:
[{"review_id": "x7OZ7eXveU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "----------- Overview of the paper --------------\n\nThis paper studies the correlation of flat minima and large margins, as well as their impact on the generalization abilities of neural networks. The main tool in the main is the relation between the trace of Hessian and the functional margin. The empirical study in the paper reveals that both in linear and nonlinear settings, there exists an intimate connection between the trace of Hessian and the margin. This idea is further utilized to explain learning with large/small batch sizes.\n\n----------- Contributions and strength--------------\n\nProperties of the stationary point found by training algorithms (e.g., SGD) are sweet spot topics, especially in deep learning. Flatness and the corresponding margins are believed to be influential factors on the generalizability of neural networks. The paper makes efforts in explaining the interplay of these two factors, which is helpful in forming a deep understanding of neural networks. The message in the paper is rather clear.\n\n----------- Weakness --------------\n\nSome claims in the paper are conjectures and lack a serious treatment. For example, in section 4, the margin perspective on large and small-batch sizes are based on some folklore and beliefs without empirical/theoretical confirmation. The experiments in section 4 is used to support the observation in existing works, rather not for further explanation of the observation. Extra experiments corroborate that ``large-batch methods not having had enough time to maximize margins is the driving force behind the large-batch vs small-batch phenomenon'' should be provided.\n\nFor the theory part, there seem to be some typos.\n\n----------- Questions and comments --------------\n\nIn proposition 3.1 and the following discussion, \\eta is undefined or it should be replaced by \\gamma. It may be better to write the bound using the margin in the proposition statement. I am also curious why there is a dependence on the norm of input in the trace of the Hessian, and what are the implications of such a dependence?\n\nI am a bit confused of the message in section 5. The concluding remark seems to argue that when the data is perfectly classified, simple scaling can alter the margin yet does not change the generalization. I think this is a direct consequence of the positive homogeneous property of classifiers. \n\nThe paper is prepared in a nonstandard ICLR template. The margin of the current draft is much smaller than the ICLR template. Although the content is fit in 8 pages, it will definitely be around 10 pages in the ICLR template.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "j-Dqbwsn8_E", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents empirical evidence that the output margin - as a measure of the confidence of a multiclass predictor - is strongly correlated to the Hessian trace when using cross-entropy loss with softmax. Moreover, the paper presents a method for estimating the Hessian trace using the input norm and softmax output. This estimation is inspired by linear classifiers and shows a strong correlation with the Hessian trace. \n\nI think the paper presents an interesting take on generalization by relating flatness and output margins. I would argue that it might be even more fruitful to also investigate input margins, or margins in some representation (i.e., some layer of the network). Nonetheless, the direction is interesting and the paper is well-written and pleasant to read. However, the analysis is a bit limited and the claimed contributions of this paper are not well substantiated. \n\nThe paper claims that it proves that flatness and output margins are strongly correlated. The paper proves this relation for linear models with cross-entropy loss and softmax, but only presents empirical indication for neural networks. While those experiments are interesting, they are very limited (only MNIST). Thus, they do not fully substantiate the claim.\n\nMoreover, the paper claims to show that small-batch SGD is not better in escaping local minima than large-batch SGD. To that end, the paper performs experiments in which small-batch SGD indeed generalizes better than large-batch SGD. The paper explains this by the distribution of output margins. This alone does not support the claim. The paper than argues that Hoffer et al., 2017 have shown that large-batch training generalizes just as well (given a similar number of parameter updates) and from that concludes their claim is true. There are two issues with this argument: (i) it remains unclear whether the empirical results of Hoffer et al., 2017 show that large-batch training always generalizes as well as small-batch training given the same number of weight updates, since Hoffer already found that the generalization is also connected to their batch normalization technique and other works (e.g., Goyal et al, 2017 [1]) report that large-batch training generalizes not as well as small-batch training. (ii) even if Hoffer et al., 2017 would support the claim that large-batch training generalizes as well as small-batch training, I don't see why the arguments made by the authors add to this. To give compelling empirical evidence the paper could, for example, show that the distribution of margins is different for small and large batch training over a wide selection of different datasets and network architectures. This would not prove the claim, but make it more plausible. In general, I am unsure whether this claim can be true for arbitrary learning problems. The argument made for small-batch training is that SGD in expectation optimizes the risk, while gradient descent optimizes the empirical error (see Chapter 14.5.1 in Shalev-Shwartz and Ben-David, 2014). Thus, using a small-batch size (and thus performing something closer to SGD) should optimize the true risk and thus generalize better than using a large-batch (thus being closer to GD) which should overfit. Of course, it is unclear how this observation on SGD translates to neural networks which often are trained in the interpolation regime. However, this paper in its current form does not add insights to this discussion.\n\nLastly, the paper claims to show that neural networks can be scaled to be arbitrary flat and that this issue relates to the output margin. The scaling issue itself is well-known (Dinh et al., 2917) and reparameterization-invariant flatness measures have been proposed (Petzka, et al., 2019, 2020, Tsuzuku, et al. 2019) that avoid this issue. The difference of those measures to the one proposed in this paper is not discussed. Thus, the contribution of this paper remain unclear.\n\nTherefore, I think this paper is not ready for publication, yet.\n\nDetailed comments:\n\n- it seems that the major advantage of using the margin measure instead of the Hessian trace is computational complexity. If this is indeed the selling point, please add an analysis of the computational complexity and a runtime analysis (e.g., when regularizing using the measure). In that case, please also compare to flatness measures of only one layer of the network (e.g., Petzka et al., 2020).\n- it seems that the margin measure does not suffer from the reparameterization curse which is a huge plus for this method. I suggest adding a short section that proves this.\n- in Sec. 4.2 the paper relates its results on output margins to input margins (mentioning SVMs). It is unclear to me how output margins relate to input margins. I do think that this relation is interesting, but the brief discussion in that section is not compelling.\n\n[1] Goyal, Priya, et al. \"Accurate, large minibatch sgd: Training imagenet in 1 hour.\" arXiv preprint arXiv:1706.02677 (2017).\n\n------- After Discussion --------\nThe authors agreed that the paper is not ready for publication, yet. Thus I keep my original score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eghSaXsv2V", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update**: Since there's no substantial author response, I'm keeping my score as it is. All the best for your future submissions!\n\n### Summary of paper\n\n- This paper argues that the flatness of a deep network at an input --- quantified by the Hessian trace of the cross-entropy loss --- roughly corresponds to the margin of the network on that point. The paper argues this by first explicitly deriving a relation between the two for a linear classifier. Then, for non-linear networks, it demonstrates this relation empirically via correlations between the two terms, on two architectures (LeNet and fully connected network) and datasets (MNIST and CIFAR10).\n\n- Then the paper uses the above insight to explain why large batch methods appear to generalize worse than small batch methods. The observation is that the margin of small batch methods are on average larger than small batch methods, and this implies that the large batch method has not trained long enough to maximize its margins and generalize well.\n\n- Finally, the paper shows that rescaling the weights of the network can lead to arbitrarily flat loss landscapes and arbitrarily large minima.\n\n### Strengths\nThe paper attempts to make a variety of interesting points connecting flatness and margins. The claims made are easy to understand, and are supported by a variety of illuminating experiments.\n\n### Weaknesses\n0. The paper violates the ICLR margins. Would be great if the authors can resubmit a version of the paper with the correct margins.\n\n1. The argument that \"flatness is related to margins\" as such I'm afraid is not novel to me and has appeared in PAC-Bayesian analyses. Here, one thinks of flatness in terms of how much random perturbation a trained deep network can withstand until a non-negligible fraction of the training data end up being misclassified. If the training data is classified by a larger margin, the classifier would withstand larger perturbations, and hence implying more flatness. For example Neyshabur et al., '18 has a nice formalization of this. Having said that, I agree that the relationship between the trace of the Hessian of cross-entropy loss to its margin is strictly speaking, novel. However, I'm not sure I see much new insight here since this insight is claimed to be a fundamental contribution of this paper.\n\n\n2. There are many existing \"normalized\" flatness measures, which I feel this paper seems should have engaged with more deeply than it has. In particular, this paper argues that the trace of the Hessian or the margin of the deep network can be scaled arbitrarily without affecting the performance of the network -- therefore these quantities in themselves are bad measures of generalization. Isn't this exactly the point made in Neyshabur et al., '17,18, Bartlett et al., '17 etc., in the context of generalization bounds, and in Dinh et al., '17 in the context of Keskar et al., '17's work? The argument as to what is fundamentally new in this paper when compared to those, is lacking.\n\n3. I'd also want to note that Hessian-based flatness measures capture only limited local information given that the landscape is non-convex and we've ReLU units. Would be nice to note this in the paper. (This is however not a problem with PAC-Bayesian measures).\n\n4. Unfortunately, the section explaining why small-batch training generalizes better was not any more illuminating to me than what we understood from Hoffer et al 2017 and Jastrze ̨bski et al. (2017). \n  \n5. I'd also like to note that  \"Flatness is a False Friend\", Diego Granziol \nhttps://arxiv.org/abs/2006.09091 makes a similar point about how training longer naturally leads to flatter minima in terms of the Hessian. Depending on how concurrent the authors think that work is, the authors may want to consider citing it. \n\n### Overall opinion\nWhile understanding connections between flatness and generalization is an important direction, unfortunately, I feel that this paper does not provide significantly new insights into these terms. Hence, I'd recommend rejection at this point.\n\n\n### Clarification questions\n\nI'm not sure I understood the architecture that is used in Fig 2. Have you trained a linear classifier?\n\n#### Minor suggestions\n\n- Page 4 \"Fig 2 is an examples of\"\n\n#### References\n\n- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks ICLR 2018\nBehnam Neyshabur, Srinadh Bhojanapalli, Nathan Srebro https://arxiv.org/pdf/1706.08947.pdf\n\n- Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.\n\n- Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv:1912.02178, 2019.\n\n- Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1019–1028. JMLR. org, 2017.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cVY7QPCT0ex", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the correlation between the flatness of the converged local minimum and the margin. The authors report experimental results that verify the positive correlation. They suggest using margin-based measures to assess the generalizability. Also, the authors argue that large-batch optimization does not have enough time to maximize margins and hence generalize worse and suggest using it to replace the “misleading folklore” that small-batch methods generalize better because they are able to escape sharp minima. In addition, the authors significantly narrowed the margin which would have violated the policy: “Tweaking the style files may be grounds for rejection.”\n\nOverall, I vote for rejection. The experiments are described in detail and seem correct. However, I was worried that (1) the reported results are not new; and (2) the authors argue existing results are misleading but did not give enough establishment to support their argument. Extraordinary claims require extraordinary evidence. It would be good if the authors can address thesis concerns in the rebuttal session.\n\nPros:\n\n+ The authors conduct experiments which verify a positive correlation between the margin and the flatness.\n\nCons:\n\n- The results are not new. It is well-known that (1) margin is a good measure for assessing the generalizability [1-4]; and (2) flatness has a strong correlation with the generalizability as the authors have stated. Combining (1) and (2), it is not surprising margin and flatness has a strong correlation.\n\n- The authors argue that large-batch optimization does not have enough time to maximize margins and hence generalize worse. This argument lacks evidence from either theoretical or empirical aspect.\n\n- The authors argue that it is a misleading folklore that small-batch methods generalize better because they are able to escape sharp minima, still without evidence. In contrast, this has been established in many works; e.g., Sagun et al. (2017) as the authors mentioned.\n\n- The authors significantly narrowed the margin. As stated in the template: “Tweaking the style files may be grounds for rejection.”\n\nQuestions: It would be good if the authors can address the cons.\n\n[1] Vladimir Vapnik and Vlamimir Vapnik. Statistical learning theory. Wiley New York, 1:624, 1998.  \n[2] Peter Bartlett and John Shawe-Taylor. “Generalization performance of support vector machines and other pattern classifiers.” In Advances in Kernel Methods: Support Vector Learning, pages 43–54, 1999.  \n[3] Vladimir Koltchinskii, Dmitry Panchenko. “Empirical margin distributions and bounding the generalization error of combined classifiers.” The Annals of Statistics, 30(1):1–50, 2002.  \n[4] Ben Taskar, Carlos Guestrin, and Daphne Koller. “Max-margin Markov networks.” In Advances in Neural Information Processing Systems, pages 25–32, 2004.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers explain in detail, why they think the paper should not be accepted. Besides fixing an initially criticized format violation, the authors did not respond to any of the concerns raised the reviewers, and in fact, they partially agree that more work in another direction needs to be done.  
Decision: Reject