Stochastic Training is Not Necessary for Generalization
Authors: Jonas Geiping
Micah Goldblum
Phil Pope
Michael Moeller
Tom Goldstein
Venue: ICLR 2022 Poster (2022)
Keywords: Optimization
Generalization
Stochasticity
SGD
full-batch
implicit regularization
implicit bias

Abstract: It is widely believed that the implicit regularization of SGD is fundamental to the impressive generalization behavior we observe in neural networks.  In this work, we demonstrate that non-stochastic full-batch training can achieve comparably strong performance to SGD on CIFAR-10 using modern architectures. To this end, we show that the implicit regularization of SGD can be completely replaced with explicit regularization. Our observations indicate that the perceived difficulty of full-batch training may be the result of its optimization properties and the disproportionate time and effort spent by the ML community tuning optimizers and hyperparameters for small-batch training.

Published as a conference paper at ICLR 2022
STOCHASTIC TRAINING IS NOT NECESSARY FOR GENERALIZATION
Jonas Geiping
Micah Goldblum
Phillip E. Pope
University of Siegen jgeipinge umd edu
University of Maryland go 1 db um@umd. edu
University of Maryland
pepopeEcs umd. edu
Michael Moeller
Tom Goldstein
University of Siegen
University of Maryland
mi michael www.eeeeeeeeeeeeeeeeeeeeeeee
t tomg uudd edu
ABSTRACT
It is widely believed that the implicit regularization of SGD is fundamental to the impressive generalization behavior we observe in neural networks. In this work, we demonstrate that non-stochastic full-batch training can achieve comparably strong performance to SGD on CIFAR-10 using modern architectures. To this end, we show that the implicit regularization of SGD can be completely replaced with explicit regularization. Our observations indicate that the perceived difficulty full-batch training may be the result its optimization properties and the dis- proportionate time and effort spent by the ML community tuning optimizers and hyperparameters for small-batch training.
INTRODUCTION
Stochastic gradient descent (SGD) is the backbone of optimization for neural networks, going back least far as LeCun et al. (1998a), and SGD is the de-facto tool for optimizing the parameters of modern neural networks (Krizhevsky et al. 2012 He et al.) 2015a Brown et al. 2020). A central reason for the success stochastic gradient descent its eficiency in the face of large datasets noisy estimate of the loss function gradient is generally sufficient to improve the parameters of neural network and can be computed much faster than a full gradient over the entire training set.
At the same time, folk wisdom dictates that small-batch SGD is not only faster but also has unique bias towards good loss function minima that cannot be replicated with full batch gradient descent. Some even believe that stochastic sampling is the fundamental force behind the success of neural networks. These popular beliefs are linked to various properties of SGD, such as its gradient noise, fast escape from saddle points, and its uncanny ability avoid sub-optimal local minima (Hendrik) 2017 LeCun 2018). These properties are also attributed in varying degrees to all mini-batched first-order optimizers, such as Adam (Kingma & Ba 2015) and others (Schmidt et al. 2020).
But why does stochastic mini-batching really aid generalization? In this work, we set out to isolate mechanisms which underlie the benefits of SGD and use these mechanisms to replicate the empirical benefits of SGD without stochasticity. In this way, we provide counterexample to the hypothesis that stochastic
which leads to noisy estimates of the gradient of the loss function, is fundamental for the strong generalization success of over-parameterized neural networks.
We show that standard 18 can be trained with batch size 50K (the entire training dataset) and still achieve 05.67%(=0.06 validation accuracy on CIFAR-10, which is comparable to the same network trained with a strong SGD baseline, provided data augmentation is used for both methods. We then extend these findings to train without (random) data augmentations, for an entirely non- stochastic full-batch training routine with exact computation of the full loss gradient, while still achieving over 95% accuracy. Because existing training routines batch SGD, the success of our experiments requires us to eschew standard training parameters in favor of more training steps, aggressive gradient clipping, and explicit regul arizaatin terms.
heavily optimized for small-

---

Published as a conference paper at ICLR 2022
The existence of this example raises questions about the role of stochastic mini-batching, and by extension gradient noise, in generalization. In particular, it shows that the practical effects of such gradient noise can be captured by explicit, non-stochastic, regularization. This shows that deep learning succeeds even in the absence of mini-batched training. A number of authors have studied relatively large batch training, often finding trade-offss between batch size and model performance (Yamazaki et al., 2019; Mikami et al., 2019; You et al. (2020). However, the goal of these studies has been first and foremost to accelerate training speed (Goyal et al. 2018 Jia et al. 2018), with maintaining accuracy as secondary goal. In this study, we seek to achieve high performance on full-batch training at all costs. Our focus is not on fast runtimes or altra-efficient parallelism. but rather on the implications of our experiments for deep learning theory. In fact, the extremely high cost of each full-batch update makes GD far less efficient than a conventional SGD training loop.
We begin our discussion by reviewing the literature on SGD and describing various studies that have sought to explain various successes of deep learning through the lens of stochastic sampling. Then, explain the present benchmark results using a range of settings, both with and without data augmentation.
needed to achieve strong results in the full-batch setting and
PERSPECTIVES ON GENERALIZATION VIA SGD
The widespread success of SGD in practical neural network implementations has inspired theorists to investigate the gradient noise created by stochastic sampling as potential source of observed generalization phenomena in neural networks. This section will cover some of the recent litera- ture concerning hypothesized effects of stochastic mini-batch gradient descent (SGD). We explicitly focus on generalization effects of SGD in this work. Other possible sources of generalization for neural networks have been proposed that do not lean stochastic sampling, for example general- ization results that only require overparametrization (Neyshabur et al.] 2018 Advani et al. 2020),
large width (Golubeva et al. 202T), and well-behaved initialization schemes (Wu et al.][20 [017 Mehta
it al. 2020). We will not discuss these here. Furthermore, because we wish to isolate the effect of stochastic sampling in our experiments, we fix an architecture and network hyperparameters in our studies, acknowledging that they were likely chosen because of their synergy with SGD.
Notation. We denote the optimization objective for training neural network by L(z, 0). where represents network parameters, and x is a single data sample. Over dataset X of N data points, {x; ( the neural network training problem is the minimization of
L(O)= = N
Σ C(x,0). ØEX
(1)
This objective can be optimized via first-order optimization, of which the simplest form is descent in the direction of the negative gradient with respect to parameters 0 on a batch B of data points and with step size Ty:
g++1 = 0+ Th [B
[v((x 0*).
(2)
REB
Now, full-batch gradient descent corresponds to descent on the full dataset B = X, stochastic
gradient descent corresponds to sampling single random data point B {z) X (with or without replacement), and mini-batch stochastic gradient descent corresponds to sampling S data points B 1,,),51,2 2 X at once. When sampling without replacement, the set commonly reset after all elements are depleted.
Although stochastic gradient descent has been used intermittently in applications of pattern recog- nition as far back as the 90's, its advantages were debated as late as Wilson & Martinez (2003), who in support of SGD discuss its efficiency benefits (which would become much more prominent the following years due to increasing dataset sizes), in addition to carlier ideas that stochastic training can escape from local minima, and its relationship to Brownian motion and "quasi-annealing"" both of which are also discussed in practical guides such as LeCun et al. (1998b).
SGD and critical points. While early results from an optimization perspective were concerned with showing the effectiveness and convergence properties of SGD (Bottou) 2010), later ideas focused on the generalization of stochastic training via navigating the optimization landscape, finding global minima, and avoiding bad local minima and saddlepoints. [e al. (2015) show that stochastic
2

---

Published as a conference paper at ICLR 2022
descent is advantageous compared to full-batch gradient descent (GD) in its ability to escape saddle points. Although the same conditions actually also allow vanilla gradient descent to avoid saddle- points (Lee et al., 2016), full-batch descent is slowed down significantly by the existence of saddle points compared to stochastically perturbed variants (Du et al. (017)) Random perturbations also appear necessary to facilitate escape from saddle points in Jin et al.) (20T9). It is also noted by some authors that higher-order optimization, which can alleviate these issues, does perform better in the large-batch regimes (Martens & Grosse, 2020; Yaday 2020 Anil et al., 202T). Related works further study a critical mini-batch size (Ma et al. 2018] Jain et al., 2018) after which SGD behaves similarly to full- gradient descent (GD) and converges slowly. is unclear though whether the analysis of sub-optimal critical points can explain the benefits of SGD, given that modern neural networks can generally be trained to reach global minima even with deterministic algorithms (for wide enough networks (Du et al. 2.19)). It has been postulated that "good" minima that generalize well share geometric properties that make it likely for SGD to find them (Huang et al. 2020).
Flatness and Noise Shapes, One such geometric property of global minimizer is its flatness (Hochreiter & Schmidhuber) 1997). Empirically, Keskar et al. (2016) discuss the advantages of small-batch stochastic gradient descent and propose that finding flat basins is benefit of small- batch SGD: Large-batch training converges to models with both lower generalization and sharper minimizers. Although flatness difficult to measure (Dinh et al., 2017), flatness based measures appear to be the most promising tool for predicting generalization in Jiang et al. (2019).
The analysis of such stochastic effects is often facilitated by considering the stochastic differential equation that arises for small enough step sizes r from Eq. (2 under the assumption that the gradient noise is effectively a Gaussian random variable:
d0.- = ---VLLCE ((.)) dt + TE,
where E, represents the covariance of gradient noise at time 6, and W; is a Brownian motion model- ing it. The magnitude of E, is inversely proportional to mini-batch size (Jastrzebski al. 2018). and it is also connected to the flatness of minima reached by SGD in Dai & Zhu (2018) and Jastrzębski ct al. (2018) if 2i is isotropic. Analysis therein as well as Le (2018) provides evidence that the step size should increase linearly with the batch size to keep the magnitude of noise fixed. However, the anisotropy of Σ, strong enough to generate behavior that qualitatively differs from Brown- ian motion around critical points (Chaudhari & Soatto, 2018 Siisekli et al. 2019) and isotropic diffusion is insufficient to explain generalization benefits in Saxe et al. (2019).
The shape of E, is thus further discussed in Zhu et al. (0199) where anisotropic noise induced by SGD is found to be beneficial to reach flat minima in contrast to isotropic noise, Zhou et al. (0020) where it is contrasted with noise induced by Adam (Kingma & Ba 2015), and HaoChen ct al. 2020 who discuss that such parameter-dependent noise, also induced by label noise, biases SGD towards well-generaliiingg minima. Empirical studies in Wen et al. (2020); Wu et al. (2020) and Li et al. 2221 show that large-batch training can be improved by adding the right kind of anisotropic noise.
Notably, in all of these works, the noise introduced by SGD is in the end both unbiased and (mostly) Gaussian, and its disappearance in full-batch gradient descent should remove its beneficial effects. However, Eq. (3) only approximates SGD first-order, while for non-vanishing step sizes T, Li
ct al. (017)) find that a second-order approximation,
+ d0, -v ( [IVV dt + VTE,
(4)
does include an implicit bias proportional to the step size. Later studies such as Li et al. (2020) discuss the importance of large initial learning rates, which are also not well modeled by first-order SDE analysis but have a noticeable impact on generalization.
An explicit, non-stochastic bias? Several of these theoretical investigations into the nature of gen- eralization via SGD rely on earlier intuitions that this generalization effect would not be capturable by explicit regularization: Arora et al. (019a) write that "standard regularizers may not be rich enough to fully encompass the implicit regularization brought forth by gradient-based optimization" and further rule out norm- baand regularizers rigorously. Similar statements have already been shown for the generalization effects of overparametrization in Arora et al. (2018) who show that no regular- izer exists that could replicate the effects of overparametrization in deep linear networks. Yet, (arrettr & Dherin 2020); Smith al. (2020b) find that the implicit regularization induced by GD and SGI)
3

---

Published as a conference paper at ICLR 2022
can be analyzed via backward-ert analysis and scalar regularizer can be derived. The implicit generalization of aani-bachhed gradient descent with batches B E B can be (up to third-order terms and sampling without replacement) described explicitly by the modified loss function
L(O) + 4
Σ FEB
VL(x,
(5)
B
BEB
which simplifies for gradient descent to
( |L(0)
(6)
as found in Barrett & Dherin (2020). Training with this regularizer can induce the generalization benefits of larger learning rates, even if optimized with small learning rates, and induce benefits in generalization behavior for small batch sizes when training moderately larger batch sizes. However, Smith et al. (2020b) "expect this phenomenon to break down for very large batch sizes". Related are discussions in Roberts (2018) and Poggio & Cooper (2020), who show setting in which SGD can be shown to converge to a critical point where VL(a, ; 0) 0 holds separately for each data point r, condition which implies that the regularizer of Eq.
zero
Large-bateh training in practice. In response to Keskar et al. (2016), Hoffer et al. (017 show that the adverse effects of (moderately) large batch training can be mitigated by improved hyperparame-
ters ters-tunee tuning learning rates, optimization steps, and batch normalization behavior. A resulting line of work suggests hyperparameter improvements that successively allow larger batch sizes, (You al., 2017) with reduced trade-offs in generalization. Yet, parity in generalization between small and large batch training has proven elusive in many applications, even after extensive hyperparameter studies in De et al. (017): Golmant et al. (2018) (018): [asterss & Luschi (2018) and (mith et al. (2020a). Golmant et al. 201 8) go on to discuss that this is not only a problem of generalization in their ex- periments but also one of optimization during training, as they find that the number of iterations it takes to even reach low training loss increases significantly after the critical batch size is surpassed. Conversely, Shallue et al. (2019) find that training in large-batch regime is often still possible, but this is dependent on finding an appropriate learning rate that is not predicted by simple scaling rules, and it also depends on choosing appropriate hyperparameters: and momentum that may differ from their small-batch counterparts. A reduction of possible learning rates that converge reliably is also discussed in Masters & Luschi (2018), but significant gap in generalization is observed in Smith et al. 2020a even after grid-searching for an optimal learning rate.
Empirical studies continue to optimize hyperparameters for large-batch training with reasonable sacrifices in generalization performance, including learning rate scaling and warmup (Goyal et al.) 2018 You et al.) 2019a), adaptive optimizers (You al. 2017 7 (019b), omitting weight regulariza- tion on scales and biases (Jia et al., 2018), adaptive momentum Miikami et al. 2019), second-order optimization (Osawa et al. 2019)) and label smoothing Yamazaki al. (2019). Yet. You al. (2020) find that full-batch gradient descent cannot be tuned to reach the performance of SGD, even when optimizing for long periods, indicating fundamental "limit of batch size". The difficulty of achiev- ing good generalization with large batches has been linked to instability of training. As discussed in Cohen et al. (202)); Gilmer et al. (202I), training with GD progressively increases the sharpness of the objective function until training destabilizes in a sudden loss spike. Surprisingly however, the al- gorithm does not diverge, but quickly recovers and continues to decrease non-monotonically. while sharpness remains close to stability threshold. This phenomenon of non-monotone, but effective training close to a stability threshold is also found in Lewkowycz et al. (2020).
2.1 A MORE SUBTLE HYPOTHESIS
From the above literature, we find two main advantages of SGD over GD. First, its optimization behavior appears qualitatively different, both in terms of stability and in terms of convergence speed beyond the critical batch size. Secondly, there is evidence that the implicit bias induced by large step size SGD on mini batches can be replaced with explicit regularization as derived in Eq. (4) and Eq. (5) a bias that approximately penalizes the per- per-example gradient norm of every example. In light of these apparent advantages, we hypothesize that can modify and tune optimization hy- perparameters for GD and also add an explicit regularizer in order to recover SGD's generalization performance without injecting any noise into training. This would imply that gradient noise from
4

---

Published as a conference paper at ICLR 2022
mini-batching is not necessary for generalization, but intermediate factor; while modeling the bias of gradient noise and its optimization properties is sufficient for generalization, mini biiibbachhing by itself is not necessary and these benefits can also be procured by other means.
This hypothesis stands in contrast to possibilities that gradient noise injection is either necessary to reach state-of the-tarrtt performance (as in Wu al. (2020); Li al. (202T) that regularizing function exists with the property that its gradient replicates the practical effect of gradient noise (Arora et al., 2018). A "cultural" roadblock in this endeavor is further that existing models and hyperparameter strategies have been extensively optimized for SGD, with a significant number of hours spent improving performance on CIFAR- 10 for models trained with small batch SGD, which begets the question whether these mechanisms are by now reifrreinfor
3 FULL-BATCH GD WITH RANDOMIZED DATA AUGMENTATION
We now investigate our hypothesis empirically, attempting to set up training so that strong gener- alization occurs even without gradient noise from mini-batching. We will thus compare full-batch settings in which the gradient of the full loss is computed every iteration and mini-batch settings in which a noisy estimate of the loss computed. Our central goal is to reach good full-batch per- formance without resorting to gradient noise, via mini- miii-bbatching or explicit injection. Yet, we will occasionally make remarks regarding full-butch in practical scenarios outside these limitations.
For this, we focus well-understood case in the literature and train ResNet model on CIFAR-10 for image classification. We consider a standard ResNet-18 (He et al. 2015a 2019) with randomly initialized linear layer parameters (He et al.) 20155) and batch norma izzatin parameters initialized with mean zero and unit variance, except for the last in each residual branch which is initialized to zero (Goyal et al. 2018). This model and its initialization were tuned to reach optimal performance when trained with SGD. The default random CIFAR-10 10 data ordering is kept as is.
We proceed in several stages from baseline experiments using standard settings to specialized schemes for full-batch training. comparing stochastic gradient descent performance with full-batch gradient descent. Over the course of this and the next section we first examine full-batch training with standard data augmentations, and later remove randomized data augmentations from training as well evaluate completely noise-less pipeline.
3. BASELINE SGD
We start by describing our baseline setup, which is well-tuned for SGD. For the entire Section 3 every image is randomly augmented by horizontal flips and random crops after padding by 4 pixels.
Baseline SGD: For the SGD baseline, we train with SGD and a batch size of 128, Nesterov momen- tum of 0.9 and weight decay of 0.0005.
are drawn randomly without replacement in every epoch. The learning rate is warmed up from 0.0 to 0.1 over the first epochs and then reduced via cosine annealing 0 over the course of training (Loshchilov & Hutter 2017). The model is trained for 300 epochs. In total, 390 x 300 = 117, 000 update steps occur in this setting.
With these hyperparameters, mini-batch SGD (sampling without replacement) reaches a validation accuracy of (00700%%(llll) tecture. Mini-batch SGD provides this strong baseline largely independent from the exact flavor of mini-batching, more details can be found in the appendix.
which we consider very competitive modern baseline for this archi-
With the same settings, we now switch to full-batch gradient descent. We replace the mini-bated updates by full batches and accumulate the gradients over all mini- batches. To rule out confounding effects of batch normalization, batch normalization is still computed over blocks of size 128 (Hoffer et al. 2017). although the assignment of data points to these blocks is kept fixed throughout training so that no stochasticity is introduced by batch normalization. training, applying full-batch gradient descent with these settings reaches validation accuracy of only 75.42%(++00 (+00..33) yielding a ~ 20% gap in accuracy between SGD and GD. In the following experiments, we will close the gap between full-batch and mini-batch training. We do this by chewing common training apper-parameters used for small batches, and re-designing the training pipeline to maintain stability without mini-batching.
line with literature on large-batch
5

---

Published as a conference paper at ICLR 2022
Experiment
Mini Mini-batching
Epochs
Steps
Modifications
Val. Acc,%
Bascline SGD
300
117,000
95.70(=0.11)
SGD regularized
300
1 117000
reg
95.81(+0.18)
Baseline FB
X
300
300
75.42(10.13)
FB train longer
X
3000
3000
87.36(+1.22)
FB clipped
X
3000
3000
clip
93.85(=0.10)
FB regularized
X
3000
3000
clip+reg
(5.04
FB strong reg.
3000
3000
clip+reg+bs32
95.68 +0.09)
FB in practice
X 3000
3000
cliptreg-bs32+shufii
95.91 (10.1)))
Table I: Validation accuracies on the CIFAR-10 validation set for each experiment with data aug-
mentations considered in Section All validation accuracies are averaged over runs.
3.2 STABILIZING TRAINING
Training with huge batches leads to unstable behavior. As the model is trained close to its edge of stability (Cohen et al., 2020), we soon encounter spike instabilities, where the cross entropy objec- tive C(0) suddenly increases value, before quickly returning to its previous value and improving further. While this behavior can be mitigated with small-enough learning rates and aggressive learn- ing rate decay (see supp. material), small learning rates also mean that the training will firstly make less progress, but secondly also induce smaller implicit gradient regularization, i.e. Eq. ( Ac- cordingly, we seek to reduce the negative effects of instability while keeping learning rates from vanishing. In our experiments, found that very gentle warmup learning rate schedules combined with aggressive gradient clipping enables us to maintain stability with a manageable learning rate.
Gentle learning rate schedules. Because full-batch training notoriously unstable, the learning rate is now warmed up from 0.0 to 0.4 over 400 steps (each step is now an epochs) to maintain stability, and then decayed by cosine annealing (with a single decay without restarts) to 0.1 over the course of 3000 steps/epochs.
The initial learning rate of 0.4 is not particularly larger than in the small-batch regime, and it is extremely small by the standards of linear scaling rule (Goyal et al., 2018), which would suggest a learning rate of 39, or even a square-scaling rule (Hoffer et al.) 2017). which would predict a learning rate of 1.975 when training longer. As the size of the full dataset is certainly larger than any critical batch size, we would not expect to succeed in fewer steps than SGD. Yet, the number of steps, 3000, is simultaneously huge, when measuring efficiency in passes through the dataset, and tiny, when measuring parameter update steps. Compared to the baseline of SGD, this approach requires ten-fold increase in dataset passes, but it provides 39-fold decrease in parameter update steps. Another point of consideration is the effective learning rate of Li & Arora (2019). Due to the effects of weight decay over 3000 steps and limited annealing, the effective learning rate is not actually decreasing during training.
Training with these changes leads full-batch gradient descent performance of 87.36%(+11.25 which is 12% increase over the baseline, but still ways off from the performance of SGD, We summarize validation scores in Table as we move across experiments.
Gradient Clipping, We clip the gradient over the entire dataset to have an C noomm of at most 0.25
before updating parameters.
Training with all the previous hyperparameters and additional clipping obtains a validation accuracy
of 93.85(#0.10)) This is significant increase of 6% over the previous result due surprisingly
simple modification, even as other improvements suggested in the literature (label smoothing (Ya-) mazaki et al. 2019), partial weight decay (Jia et al. 2018), adaptive optimization (You al.) (017). sharpness-aww aware minimization (Foret et al. 2021) fail produce significant gains, appendix).
Gradient clipping is used in some applications to stabilize training (Pascanu et al. 2013). However in contrast to its usual application in mini-batch SGD, where a few batches with high gradient contributions might be clipped in every epoch, here the entire dataset gradient is clipped. As such, the method is not tool against heavy-tailed noise (Gorbunov al.) 22000), but it is effectively limit on the maximum distance moved in parameter space during a single update. Because clipping simply changes the size of the gradient update but not its direction, clipping is equivalent to choosing

---

Published as a conference paper at ICLR 2022
Low sube))
Optinilation Strpe
Optiniaaionn Strge
Figure| 1:
Cross-Entropy Loss on the training and validation set and full loss (including weight decay) during training for full-batch gradient descent. Left: training as described in Section 3.2 without clipping, right: with gradient clipping. Validation computed every 100 steps.
a small learning rate when the gradient is large. Theoretical analysis of gradient clipping for GD in Zhang et al. (2019b) and and Zhang al. (2020) supports these findings, where it is shown that clipped descent algorithms can converge faster than unclipped algorithms for a class of functions with a relaxed smoothness condition. Clipping also does not actually repress the spike behavior entirely. To do so would require a combination of even stronger clipping and reduced step sizes, but the latter would reduce both training progress and regularization via Eq. 6
3.3 BRIDGING THE GAP WITH EXPLICIT REGULARIZATION
Finally, there is still the bias of mini-batch gradient descent towards solutions with low gradient norm per batch described in Eq. 4 and Eq. { This bias, although 2nd-order effect, is noticeable in our experiments. We can replicate this bias as explicit regularizer via Eq. (5). However, computing exact gradients of this regularizer directly is computationally expensive due to the computation of repeated Hessian-vedt vector products in each accumulated batch, especially within frameworks without forward automatic differentiation which would allow for the method of Pearlmutter (1994) for fast approximation of the Hessian. As such, we approximate the gradient of the regularizer through finite-differences approximation and compute
0112 VL (Vl,0+vV((z,0)) -((x,,
This approximation only requires one additional forward-backwant pass, given that VL(x, O) is
already required for the main loss function. Its accuracy is similar to a full computation of the Hessian-vector products (see supplementary material). In all experiments, we set [VC(-,)) similar to (Liu et al.] 2018a). To compute Eq. (5), the same derivation is applied for averaged Z VL(r,)).
0.01
gradients
Gradient Penalty. We regularize the loss via the gradient penalty in Eq. 5 with coefficient ATT We set a = 1.0 for these experiments.
We use this regularizer entirely without sampling, computing over the fixed mini-batch blocks B E B, already computed for batch normalization, which are never shuffled. We control the strength of the regularization via parameter a. Note that this regularizer can be computed in parallel across all batches in the dataset. Theoretical results from Smith et al. (2020b) do not guarantee that the regularizer can work this setting, especially given the relatively large step sizes we employ. However, the regularizer leads to the direct effect that not only ||VI(() is small after optimization, but also || >>Ee (z(,,, O)]l, i.e. the loss on each mini-batch. Intuitively, the model is optimized so that it is still optimal when evaluated only on subsets of the training set (such these mini-batches)
Applying this regularizer on top of clipping and longer training leads to a validation accuracy of 94. (4..77 for regularizer accumulated over the default batch size of [B] 128. This can be further increased to 94.97(1:0.05) if the batch size is reduced to 32 (reducing the SGD batch size in the same way does not lead additional improvement, beneficial effect as the regularizer Eq. (5) is moved closer to a direct penalty on the per-example gradient norms of Poggio & Cooper 20200) yet computational effort increases proportionally.
supp. material). Reducing [B] is
7

---

Published as a conference paper at ICLR 2022
Experiment
ResNet-18
ResNet-550
Resnet-152
DenseNet-12)
VGG-16
Baseline SGD
95.65(=0.18) 95.70(=0.11)
06.80(+0.11)
95.80
94.4((+0.89)
Baseline FB
75.42(=0.13) 87.36(=1.23)
55.03(=3.89)
77.26( +0.28)
77.26(10.2g)
46.95(#19.51)
FB train longer
85.12(=0.00)
91.67
+0.35
$9.56(10.000
89.36(#1.19)
FB clipped
93.85
92.55(=0.36)
92.48(
+0.46)
92.59(+0.16)
(2.521+
FB regularized
95.54(=0.00)
95.84(=0.09)
95.98(
95.92(+0..0)
93.86(+0.18)
FB strong reg.
95.68(=0.09)
96.06( 00.04)
96.21 (0.12)
96.08( +0)13)
93.91 (0.17)
FB in practice
95.91
96.50(=0.13)
96.37 (0.56) 46.43((0..10)
94.44(+0.07)
Table 2: Validation accuracies on the CIFAR-10 validation set for each of the experiments with data augmentations considered in Section for multiple modern CNNs.
Double the learning rate. We again increase the initial learning rate, now to 0.8 at iteration 400, which then decays to 0.2 over the course of 3000 steps/epochs.
This second modification of the learning rate is interestingly only an advantage after the regularizer included. Training with this learning rate and clipping, but without the regularizer (i.e. in Section 3.2). reduces that accuracy slightly to 93. (5.(3..13... However, the larger learning rate does improve the performance when the regularizer is included, reaching 95.54(+0..09) if [ 128 and (5.68(=0.09) if B| = 32, which is finally fully on par with SGD.
Overall, we find that after all modifications, both full-batch (with random data augmentations) and SGD behave similarly, achieving significantly more than 95% validation accuracy. Figure 4 vi- sualizes the loss landscape around the found solution throughout these changes. Noticeably both clipping and gradient regularization correlate with flatter landscape.
Remark (The Practical View). Throughout these experiments with full-batch GD, we have decided not to shuffle the data in every epoch to rule out confounding effects of batch normalization. If we turn on shuffle again, we reach 95.91 %[ (1t0.10 (=..14) validation accuracy (with separate runs ranging be- tween 96. 11%1 and 95.71 95)),, which even slightly exceeds SGD. This the practical view, given that shuffling is nearly free in terms of performance. but of course potentially introduces meaningful source of gradient noise- which is why is not our main focus.
Furthermore, to verify that this behavior is not specific to the ResNet- 18 model considered so far, we also evaluate related vision models with exactly the same hyperparameters. Results are found in Table where we find that methods generalize the ResNer-50, ResNet- 152 and a DenseNet- 121 without any hyperparameter modification. For VGG-16, we do make a minimal adjustment and increase the clipping to 1.0, as the gradients there are scaled differently, to reach parity with SGD,
FULL-BATCH GD IN THE TOTALLY NON-STOCHASTIC SETTING
A final question remains if the full-batch experiments shown so far work to capture the effect of mini-batch SGD, what about the stochastic effect of random data augmentations on gradient noise? It is conceivable that the generalization effect is impacted by the noise variance of data augmentations. As such, we repeat the experiments of the last section in several variations.
No Data Augmentation. If we do not apply any data augmentations and repeat previous experi- ments, then GD with clipping and regularization at 89.17%, substantially beats SGD with default hyperparameters at 24.32%((11.1) 90.07(=0.48), see Table Interestingly, not only does the modified GD match SGD, the modified GD is even more stable, as it works well with the same hyperparameters as described in the previous section, and we must tune SGD even though it benefits from the same regularization implicitly.
and nearly matches SGD with newly tuned hyperparameters at
Enlarged CIFAR-10 To analyze both GD and SGD in setting were they enjoy the benefits of augmentation, but without stochasticity, replace the random data augmentations with a fixed in- creased CIFAR- 10 dataset. This dataset is generated by sampling N random data augmentations for each data point before training. These samples are kept fixed during training and never resampled, resulting in an N-times larger CIFAR-10 dataset. This dataset contains the same kind of variations that would appear through data augmentation, but is entirely devoid of stochastic effects training.
If we consider this experiment for a 10x enlarged CIFAR-10 10 dataset, then we do recover value of 95.11% %. Note that we present this experiments only because of its implications for deep learning
8

---

Published as a conference paper at ICLR 2022
Experiment
Fixed Dataset
Mini-batching
Steps
Modifications
Val. Acc,
Baseline SGD
CIFAR-I0
117,000
84.32( 90.07(#0.48)
Baseline S
CIFAR-I0
117,000
FB strong reg. Baseline SGD
CIFAR-10
X
3000
clip+reg+bs32
89.17(#0.24)
10x CIFAR-I0
117,001
95.20 (+009)
FB train longer
10x CIFAR-I0
3000
88. 88.44( -
FB strong reg.
10x CIFAR-10
X
3000
clip+reg+bs32
95. 95.11(-)
Table 3: Validation accuracies on the CIFAR-IO validation set for fixed versions of the dataset with random data augmentations in Section Hyperparameters fixed from the previous section except for SGD marked", where the learning rate is doubled to 0.2 for a stronger baseline.
theory: computing the gradient over the enlarged CIFAR-10 is N-times as expensive, and there are additional training expenses incurred through increased step numbers and regularization. For this reason we do not endorse training this way as a practical mechanism. Note that SGD still have an advantage over 10x CIFAR - SGD sees 300 augmented CIFAR-I0 10 datasets, once each, over its 300 epochs of training. If we take the same enlarged CIFAR-10 10 dataset and train SGD by selecting one of the 10 augmented versions in each epoch, then SGD reaches 95. (0.2%%%(..000
Overall, we find that we can reach more than 95% validation accuracy entirely without stochasticity, after disabling gradient noise induced via mini-batching, shuffling well as via data augmentations. The gains of ~ 6% compared to the setting without data augmentations are realized only through the increased dataset size. This shows that noise introduced through data augmentations does not appear to influence generalization in our setting and is by itself also not necessary for generalization.
5 DISCUSSION & CONCLUSIONS
SGD, which was originally introduced to speed up computation, has become mainstay of neural network training. The hacks and tricks at our disposal for improving generalization in neural models are the result of millions of hours of experimentation in the small batch regime. For this reason, should come as no surprise that conventional training routines work best with small batches. The heavy reliance of practitioners on small batch training has made stochastic noise prominent target for theorists, and SGD is and continues to be the practical algorithm of choice, but the assumption that stochastic mini-batching by itself is the unique key to reaching the impressive generalization performance of popular models may not be well founded.
In this paper, we show that full-batch training matches the performance of stochastic small-batch training for a popular image classification benchmark. We observe that (i) with randomized aug- mentations, full-batch training can match the performance of even a highly optimized SGD base- line, reaching 95.67% for ResNet-18 fully non-stochastic training beats SGD with standard sypper-araamtter, matching it when optimiz- ing SGD hyperparameters, and (iii) after 10x fixed dataset expansion, full-batch training with no stochasticity exceeds 95%, matching SGD on the same dataset. Nonetheless, our training routine is highly inefficient compared SGD (taking far longer run time), and stochastic optimization remains a great practical choice for practitioners in most settings.
CIFAR-10, (ii) without any form of data augmentation,
The results this paper focus on commonly used vision models. While the scope may seem narrow, the existence of these counter-examplee is enough to show that stochastic mini-batching, and by extension gradient noise, not required for generalization. also strongly suggests that any theory that relies exclusively on stochastic properties to explain generalization is unlikely to capture the true phenomena responsible for the success of deep learning. Stochastic sampling has become focus of the theory community in efforts to explain generalization. However, experimental evidence in this paper and others suggests that strong generalization is achievable with large or even full batches in several practical scenarios. If stochastic regularization does indeed have benefits in these settings that cannot be captured through stochastc,,,,, regularized training, then those benefits are just the cherry on top of large and complex cake.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
We foresee no direct social impact of this work at the moment.
REPRODUCIBILITY STATEMENT
We detail all hyperparameters in the main body and provide all additional details in Appendix A Our open source implementation can be found at /tttps://gitnn com/ Saam/JJoaaGiippli Fullbatchtraining and contains the exact implementation with which these results were com- puted and we further include all necessary scaffolding we used to run distributed experiments on arbitrarily many GPU nodes, as well as model checkpointing to run experiments on only a single machine with optional GPU. Overall, we thus believe that our experimental evaluation is accessible and reproducible for a wide range of interested parties.
FUNDING STATEMENT
This research was made possible by the OMNI cluster of the University of Siegen which contributed notable part of its GPU resources the project and thank the Zentrum für Informations- und Medientechnik of the University of Siegen for their support. We further thank the University of Maryland Institute for Advanced Computer Studies for additional resources and support through the Center for Machine Learning cluster. This research was overall supported by the universities of Siegen and Maryland and by the ONR MURI program, AFOSR MURI Program, and the National Science Foundation Division of Mathematical Sciences.
REFERENCES
Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High nigh-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428 446, December 2020. ISSN 0893-6080. doi: 10.1016/ 2nccunet2020008...
Idan Amir, Yair Carmon, Tomer Koren, and Roi Livni. Never Go Full Batch (in Stochastic Convex Optimiza- tion). 1XXVv:107.00066 [cs, math], June 2021a. URL htt.:/apppppx...ooo///o/////////
60469
Idan Amir, Tomer Koren, and Roi Livni. SGD Generalizes Better Than GD (And (Ad Doesn't Help).
1102::2002201111: [cs, stat), June 2021b.
Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable Second Order Optimization for Deep Learning. arXiv:2002.09018 [cs, math, stat), March 2021. URL http::::/aaww..or///aaaaa//
2002.09013 09018
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the Optimization of Deep Networks: Implicit Acceleration by Overparameteriizaion.. In International Conference on Machine Learning, pp. 244-253. PMI.R July 2018. URI.http://proceee
ngs. m1 wrw.sprsssse...... parress/80//aarraa
htm1
Sanjeev Arora, Naday Cohen, Wei Hu, and Yuping Luo. Implicit Regularization in Deep Matrix Factorization. arXiv:1905.1365 [cs, star). May 2019a. URLhttp: http:::/axxx...ec httpggg/ab//911. 13655
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, Ruosong Wang. Fine-Grained Analysis Optimiza- tion and Generalization for Overparameteriiee Two-Layer Neural Networks. In International Conference on Machine Learning, pp. 322-332. PMLR, May 2019b. URL http::/prooceedingg www.ian...mm....l.mm..mm...m..me p v91 199//rora10 9a, html
David Barrett and Benoit Dherin. Implicit Gradient Regularization. In International Conference on Learning Representations, September 2020. URL https: http:///ppeeeeeeeeee/mmmm net / wttp://orunmdddeeoemm 3dt.eiiddddssssudsee.
Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stnchastic gradient descent on nonsmooth convex losses. In Advances in Neural Information Processing Systems, volume 33, pp. 4381- 4391. Curran Associates, Inc., 2020. URL https:
ngs neur wttperrppa.cpppppeare/eeee
file/2e2c4bé/cese 712a 2dddee 3 wwc.
Léon Bottou. Large-Scale Machine Learning with Stochastic Gradient Descent. In Proceedings of COMP- 2010, pp. 177-186, Heidelherg, 2010. Physica- HD. ISBN 978-3-7908-2600 doi: 10.1007/ 978-3-7908- 2--7999-2604-3116
10

---

Published as a conference paper at ICLR 2022
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretehen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeftrey Wu, Clemens Win- ter. Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Suiskever, and Dario Amodei. Language Models Few-Shot Learners. In 34th Conference Neural Information Processing Systems (NeurlPS 2020), December 2020. URL http:tt/aa//iii.oooomm orge abs http://bb//0000012 20050 14165
Satrajit Chatterjee. Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent- based Optimization. In Eighth International Conference on Learning Representations, April 2020. URL
Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to
limit cycles for deep networks. In International Conference on Learning Representations, February 2018.
URL httptttp:://peeeeiiiitte
net /ese//ooriee Forum By IgWOW
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cuDNN: Efficient Primitives for Deep Learning. arXiv:1400.073. [es], December 2014. URL
attp://arxil: org/ http:///ss/11..... 3759
Jeremy Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient Descent on Neural Networks Typically Occurs the Edge of Stability. In International Conference on Learning Representa- tionx, September 2020. URL htttpss://pperrriiiii http::ttt//oommidddd---------
Xiaowu Dai and Yuhua Zhu. Towards Theoretical Understanding of Large Batch Training in Stochastic Gra- dient Descent. arXiv:1812.00542 [cs, stat, December 2018. URL TTttkeE D0542
wttpobrggg/bbbaeeeeeee 1812.
Assaf Dauber, Meir Feder, Tomer Koren, and Roi Livni. Can Implicit Bias Explain Generalization? Stochastic Convex Optimization as a Case Study. arXiv:2003.6615: [cs, stat], December 2020. URL http://
arxiv. httgg/ab/22003 2003. 001000001122....
Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated Inference with Adaptive Batches. In Artificial Intelligence and Statistics, pp. 1504-1513, PMLR, April 2017. URL http::/pproceedingg mlr w.lrsprcss//ee v54 de 7a. html
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp Minima Can Generalize For Deep Nets. arXiv:1003.04933 [es]. March 2017. URL http:::///aai...rrtttt///// /abg///////// 11001/70300000 04933
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai Gradient Descent Finds Global Minima of Deep Neural Networks. In International Conference on Machine Learning, pp. 1675-1685, May 2019. URL
http://proceedinng
mir. wwwps//vese.... http//9///ll// dul.c.
Simon S. Du Chi Jin, Jason D. Lee, Michael I. Jordan, Aarti Singh, and Barnabas Poczos. Gradient Descent Can Take Exponential Time to Escape Saddle Points. Advances in Neural Information Pro- cessing Systems, 30, 2017. URL htttps:/prroceel.ne nwwrsps..rrpecccoceeoo pttp.rpp...////pr///00.......... 179921bbac 1atobaac40a5700 77w.11077777777770 (al22266222cccccaaaaaaaaaao -aweoddc2a--sstaaccccee
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-Aware Minimization for Efficiently Improving Generalization. arXiv:2010.01412 [cs, stat]. April 2021. URL http://arxiv. arxiv..
orq/abs/2010 01412
Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping From Saddle Points Online Stochastic Gradient for Tensor Decomposition. In Conference Learning Theory, pp. 797-842. PMLR, June 2015, URL
http://proceedi wttps//.m....p.////////////
h
Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Cardoze, George Dahl, Zachary Nado, and Orhan Firat. Loss Curvature Perspective on Training Instability Deep Learning 1rXiv:2110.04330 [cs), October 2021. URL/aatpp//axxx/ wrg. abs/ 2110. 04369
Micah Goldblum, Jonas Geiping. Avi Schwarzschild. Michael Moeller. and Tom Goldstein. Truth or backpro- paganda? An empirical investigation of deep learning theory. In Eighth International Conference on Learn- ing Representations (ICLR 2020, Oral Presentation), April 2020. URL U http:ss///ccrrroco http:///ccccc.....tttt. httt0/ooster_Hyyyy..oo html html
Noah Golmant, Nikita Vemuri, Zhewei Yao, Vladimir Feinberg, Amir Gholami, Kai Rothauge, Michael Ma- honey, and Joseph Gonzalez. On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent. Septemher 2018. URL http:pps//ppenreeiee.neeeeeeeeee
d- 1 en0

---

Published as a conference paper at ICLR 2022
Anna Golubeva, Behnam Neyshabur, and Guy Gur-Ari. Are wider better given the same number of parameters? arXiv:2010.1440 14495 fcs, stat), April 2021. UK//ttps//aaxxv..rrr//b///CCC..
Eduard Gorbunov, Marina Danilova, and Alexander Gasnikov. Stochastic optimization with heavy-tailed noise via accelerated gradient clipping. Advances in Neural Information Processing Systems. volume 33. pp. 15042-15053. Curran Associates, Inc., 2020, URL /tttps:/pprocoe 2020/fi 1e abd (7w7788888809777 759551441110 11w.1112faaaommm 0b6 Pap.rr..pppppppp..
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tul- loch, Yangging Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv: 706.02667 [cs], April 2018. URI, http:httt/////ii..cto
706. 02677
Jeff Z. HaoChen, Colin Wei, Jason D. Lec, and Tengyu Ma. Shape Matters: Understanding the Implicit Bias of the Noise Covariance. (1Xiv:2006008660 fcs, stat), June 2020. http:::///aa.....oooa/////////
08680
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. 11Xiv:1112.03: /cs/, December 2015a. URL httthh/tp///xxe or abs 12..2..0 03385
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human-
Level Performance on ImageNet Classification. arXiv:1502.00 01852 [cs]. February 2015b. URL http:
//arxiv.crm abs/ 1502. 1852
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of Tricks for Image Classification with Convolutional Neural Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 558-567, 2019. URLhttps:: /ttp///pppaccsceeeeooe/e.. //tp://///tttt////ooaaaeeeessss thecvf. com/ conteet CVPR_ 2019/html /He_ Bag of Tricks for Image Classification
with Convelutionsl Neural Networks Swtporrkksssss____e CVPR 2019 1ac19___pp______ .tmll
Hendrik. Machine learning- Why mini batch size is better than one single "batch" with all training data?,
February 2017. URL https:://datacciieneee
why-mini-ban wtw.--ssise------eet---eeeeee
Sepp Hochreiter and Jürgen Schmidhuber. Flat Minima. Neural Computation, 9(1):142,, January 1997. ISSN 0899-7667. doi: 10.1 1221/eco..9997...1... 1997.9.1.1.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: Closing the gener- alization gap large batch training of neural networks. Advances in Neural Information Pro- cessing Systems, 30, 2017. URL https: /tettp::/prrceete /ttp:////ccceeeings/c.o. www.unngg...uiiisp.eeee.m.. www.neeu.rccccco.oo httpr//pp..ccp/ppp///////a////// (BBDFF62220003355ffff.eeee.. 2a Abstract. .ttml
Wenqing Hu, Chris Junchi Li, Lei Li, and Juan-Guo Liu. On the diffusion approximation of nonconvex stochas- tic gradient descent. 170.v:170 [cs, stat], March 2018.
07562
W. Ronny Huang, Zeyad Emam, Micah Goldblum, Liam Fowl, Justin K. Terry, Furong Huang, and Tom Goldstein. Understanding Generalization through Visualizations. arXiv: 190.0339 [cs, stat), November 2020. URLtttt:://axxx wwwp://wwwooooooooooooo /tt..////...... 03292
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel:: Convergence and generalization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, 18, pp. 3580-8589 Red Hook, NY, USA, December 2018. Curran Associates Inc.
Prateek Jain, Sham M. Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Parallelizing Stochastic Gradient Descent for Least Squares Regression: Mini-batching, Averaging, and Model Mis- specification. Journal of Machine Learning Research, 18(223):1-42 515.1/195116111mm111
2018. ISSN 1533-7928. URL
http: jmmr..rrooro wwwpp/.rggpppppp.pepp pwwppsappssss//oe wwt.///v///1-1111/
Stanislaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Width of Minima Reached by Stochastic Gradient Descent is Influenced by Learning Rate to Batch Size Ratio. Artificial Neural Networks and Machine Learning ICANN 2018. Lecture Notes in Computer Science, pp. 392-402, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01424-77 doi: 0000/978--3030-01424--
Xianyan Jia, Shutao Song. Wei He, Yangzihao Wang, Haidong Rong. Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, Tiegang Chen, Guangxiao Hu, Shaohuai Shi, and Xiaowen Chu. Highly Scalable Deep Leanning Training System with Mixed-Precisson: Training ImageNet in Four Minutes. arXiv: / /205 [cs, stat), July 2018. UW./http.//arxivvor////a
11205
12

---

Published as a conference paper at ICLR 2022
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic Gen- cralization Mcasures and Where to Find Them. arXiv:1112.02178 fes, stat], December 2019. URL
Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M. Kakade, and Michael I. Jordan. On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points. arXiv: arXiv: 1902.04818 [es, math, stat), February 2019. URL http:://aaxxx..rrr/bbb// 902. 04811
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In International Conference on Learning Representations (ICLR). November 2016. URL hnttps:///ppnreeetet
net/
forum?id Aoloully
Diederik P. Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. International Conference on Learning Representations (ICLR), San Diego, May 2015.
6980
Lingkai Kong and Molei Tao. Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multi- scale Objective Function. In NeurlPS, Jamuary 2020. URL htt.:cccsss//ppeenvvveeeeeeee/ee
CWb CWb1DTh
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. 2009. URL Ittpp: cs. toronto. wttp:ntt..edo/krii///eee httprr///eaaaannngeeeeeeeuuuueee 2003-TRR. pdf.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Daniel Kunin, Javier Sagastuy-Brent Lauren Gillespie, Eshed Margalit, Hidenori Tanaka, Surya Ganguli, and Daniel L. K. Yamins. Rethinking the limiting dynamics of SGD: Modified loss, phase space oscillations, and anomalous diffusion. arXiv:2107.09133 [cond-mal, q-bio, stat], July 2021. URL http:
0rg/abs/2107.09113
Samuel L. Smith and Quoc V. Le. A Bayesian Perspective on Generalization and Stochastic Gradient Descent. In Internationaa Conference on Learning Representations, Februuu 2018. URLAatps::/openccvi.. net /
www.ooouuuuiii.rooo 4yg02
Yann LeCun. Yann LeCun on Twitter, April 2018. httpps///eerttter...c////leeeeee 98961 0208443368366
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11) (6(11):2278-233) 1998a.
Yann LeCun, Leon Bottou, Genevieve B. Orr, and Klaus Robert Müller. Efficient BackProp. In Neural Net- works: Tricks of the Trade, Lecture Notes in Computer Science, pp. 9-50. Springer, Berlin, Heidelberg. 1998b. ISBN -78-3-540--4440-00 doi: 1111007333540-444555
Jason D. Lee, Max Simchowitz, Michael I. Jordan, and Benjamin Recht. Gradient Descent Only Converges to Minimizers. In Conference on Learning Theory, pp. 1246-1257. PMLR, June 2016. URL http://
proceedings.c http:ssmm....r.ss.eeeeeeeeeeeeee wwtppppssssspee.eee
html
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein and Guy Gur-Ari. The large learning rate phase of deep learning: The catapult mechanism. arXiv:2003.02218 [cs, stat], March 2020. URL
http:/ awwaaaiiiivv..e. org abg/2bs//0033 02218
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the Loss Land- scape of Neural Nets. In Advances Neural Information Processing Systems, volume 31, Cur- ran Associates, Inc., 2018. URL 1es663b11-bbbsaatt...e.eee Abstract. html
/hash/
Qianxiao Li, Cheng Tai, and Weinan E. Stochastic Modified Equations and Adaptive Stochastic Gradient Algorithms. International Conference on Machine Learning, pp. 2101 10000 PMLR, July 2017, URL
.ml. press/v70/1111/
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks. arXiv: 990000009555 (cs, stat), April 2020. URI.http: http:///axxvv.ro///o
abs/ 907.00.0000 04595
13

---

Published as a conference paper at ICLR 2022
Zhiyuan Li and Sanjeev Arora. An Exponential Learning Rate Schedule for Deep Learning. In Interna- tional Conference on Learning Representations, September 2019, URL httppps////eeeereivv.e.eet
[orum?Id=rJg8TeSPPP
Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the Validity of Modeling SGD with Stochastic Differeenial Equations (SDEs). arXiv:2102.12470 [cs, stat], June 2021. URL 12470
Hanxiao Liu, Karen Simonyan, and Yiming Yang- DARIS: Differentiable Architecture Search.
In International Conference on Learning Representations, September 2018a.
URL https:
7/
medi
Yanli Liu. Yunbei Xu, and Wotao Yin. Acceleration of Primal-Dur Methods by Preconditioning and Fixed Number of Inner Loops. / 1.09337 [math], November 2018h, URL htt.httt://aaxi..org////./ 181 08937
Ilya Loshchilov and Frank Hutter, SGDR: Stochastic Gradient Descent with Warm Restarts. arXiv:1608.03985 [cs, mathj, May 2017. URL http:://taa/i/l/// org/ 13398
Siyuan Ma, Raef Bassily, and Mikhail Belkin. The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-pare parammermeere Learning. In International Conference on Machine Learning, pp. 3325-3334. PMLR, July 2018. URL htth::ttpptpeeeeee ngs m prw..//v88 ma 1 8a html
James Martens and Roger Grosse. Optimizing Neural Networks with Kromecker-factoree Approxmmate Curva- ture. arXiv:1503.0567) [cs, stat), June 2020. htt.t///aaa.....or//aaa/////////
Dominic Masters and Carlo Luschi.
Revisiting Small Batch Training for Deep Neural Networks.
arXiv:1804.07612 [cs, stat), April 2018. URL/attp://axxiv.oo http:/abss//a..//. 07612
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An Empirical Model of Large- Batch Training. UrXiv:1812.06164 [cs, stat], December 2018. URL 06162 62
org/ abs/l8l2.
Harsh Mchta, Ashok Cutkosky, and Behnam Neyshabur. Extreme Memorization via Scale of Initialization. In International Conference on Learning Representations, September 2020. URLAttpp:://pperew...
net/ www.ooodddnnell-e-...
Hiroaki Mikami, Hisahiro Suganuma, Pongsakorn U-chupala, Yoshiki Tanaka, and Yuichi Kageyama. Mas- sively Distributed SGD: ImageNel/esNNet500 Training in Flash. arXiv:1811.05233 [es], March 2019.
httt::t/t///xii..t////// org/ abw 1etp///11111131
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards Understand- ing the Role Over-Parametriization Generalization Neural Networks. 1805 12076 (cs, stat), May 2018. URL [ttp::// http://xxxiv.ooo/o wwwp://wogg/////..oo.....
NVIDIA. NVIDIA cuDNN, 2022. URL
http://www..oeeeeeiiiiaaaaaaaooa www.iiiiiii........ com/ cudnn
Kazuki Osawa, Yohei Tsuji. Yuichiro Ueno, Akira Naruse, Rio Yokota, and Satoshi Matsuoka. Large- Scale Distributed Second-Order Optimization Using Cromecker-Factorre Approximate Curvature for Deep Convolutional Neural Networks. In Proceedings of the IEEF/VV Conference on Computer theevf.
Vision and Pattern Recognition, pp. 12359-12367, 2019. URL com/ content CVPR 20119/ttm/Dsaawe... Large-Scale Distributed Secona-Ordes Opt imi zation Using
Approximate Curvature for Deep
CVPR 2019 paper. html
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training Recurrent Neural Networks. arXiv:1211.506, [cs], February 2013. URL/tttp://arxiv wttp://ww...//b///...e/// 5063
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, ban Desmaison, Luca Antiga, and Adam Lever. Automatic differemtiation in Py Torch. In NIPS 2017 Autodiff Workshop, Long Beach, CA, 2017. URL
Barak A. Pearlmutter. Fast Exact Multiplication by the Hessian. Neural Computation, 6:147-160, 1994.
Tomaso Poggio and Yaim Cooper. Loss landscape: Sgd has better view. Technical report, CBMM Memo, 2020.
14

---

Published as a conference paper at ICLR 2022
Daniel A. Roberts. SGD Implicitly Regularizes Generalization Error. Neural Information Processing Systems Workshop on Integration of Deep Learning Theories, 2018. htLLittt///ax....rr////a/ 04874
Andrew M. Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D. Tracey, and David D. Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124000, December 2019. ISSN 1742-5468. doi: 10. 10.1088//742-54400
ab3985.
Robin M. Schmidt, Frank Schneider, and Philipp Hennig. Descending through a Crowded Valley Bench- marking Deep Learning Optimizers. arXiv:2007.01547 (cs, stat], July 2020. URL http://ataat org/ org/abs/ abs/
01547
Christopher J. Shallue, Jachoon Lee, Joseph Antogmini, Jascha Sohl-Dickstein, Roy Frostig, and George F. Dahl. Measuring the Fffects of Data Parallefism Neural Network Training. Journal of Machine Learning Research, 2011 12):1-49, 2019. ISSN 1533-7928. URL http::/ wttpp//pppp///////////
html
Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks. In International Conference on Machine Learning, pp. 5827-8877 PMLR, May 2019.
wwt.mmlr.pprre.. mww.ppees//v9. www.ssmmiiii...... hhtml
htmm
Samuel Smith. Erich Elsen, and Soham De. On the Generalization Benefit of Noise in Stochastic Gradient Descent. In International Conference Machine Learning, pp. 9058-9061 PMLR, November 2020a.
URL / www.iigggsssr....... http://weeeedaan........r.eee//a ptrress/v11///mmtt2aaa....
Samuel L. Smith, Benoit Dherin, David Barrett, and Soham De. On the Origin of Implicit Regularization in Stochastic Gradient Descent. In International Conference on Learning Representations, September 2020b.
URL https::://ppereviieee hwtp:/eneettttttttteeee forum?ld=rg CrbelEyo
Yeming Wen, Kevin Luk. Maxime Gazeau, Guodong Zhang, Harris Chan, and Jimmy Ba. An Empirical Study of Stochastic Gradient Descent with Structured Covariance Noise. In International Conference on Artificial Intelligence and Statistics, pp. 3621-3631. PMLR, June 2020. URL
mlr r. wwwpseesslvvee.. http////wwwwweeaa..1ammm www.wa.nnwa.... html
D. Randall Wilson and Tony R. Martinez. The general inefficiency of batch training for gradient descent learn- ing. Neural Networks, (6(10):1429- 14511 December 2003. ISSN 0893-6080 doi: (0016//0893-0000000 00138-2.
Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the Noisy Gradient Descent that Generalizes SGD, International Conference on Machine Learning, pp. 10367- 10376. PMLR, November 2020. URL
Lei Wu, Zhanxing Zhu, and Weinan E. Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes. 1xXiv:/706 10239 [cs, stat), June 2017. URI,
10239
Lei Wu, Chao Ma, and Weinan E. How SGD Selects the Global Minima in Over-parameterrzed Learning: Dynamical Stability Perspective. In Advances in Neural Information Processing Sys- tems, volume 31, 2018, URL https: htttps:///rcceeding.. 000.0666608820000000000000
wwwpseeeurrips..eo.o.e het.ripp..cc//pprr//0////...ea Abstract. httl
Yuxin Wu and Kaiming He. Group Normalization. In Proceedings of the Eumppan Conference on Com- puter Vision (ECCV), pp. 3-19, 2018. httphhttt///peeeeee
thecvf strevff.cmm wwt. cttom//ontett_cttco 2018/html Yuxin Nu Group Normal aiemallizaiion zatiion ECCV 2018 paper. paper.html
Zeke Xie, Issei Sato, and Masashi Sugiyama. Diffusion Theory For Deep Leaming Dynamies: Stochastic Gradient Descent Exponentially Favors Flat Minima. International Conference on Learning Representa- tions, September 2020. SRRl/tttts://peerev/re net.:/iwwwwweeeeeeeeeeeeeeeeeeee wttpr/iwwwwee//ouuuneeme. www.mmodooworrorrorrrro iCTYGo
Zeke Xie, Issei Sato, and Masashi Sugiyama. Understaading and Scheduling Weight Decay. 2..1122011.11152
[cs]. September 2021. htllattp::/aax....cg////////////
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A Walk with SGD. arXiv: [es, stat], May 2018. Bul.atp:///rrss..og//a/////////
Abhay Yadav. Making L-BFGS Work with Industrial-Strength Nets. In BMVC 2020, pp. 13, 2020.
15

---

Published as a conference paper at ICLR 2022
Masafumi Yamazaki, Akihiko Kasagi. Akihiro Tabuchi, Takumi Honda, Masahiro Miwa, Naoto Fukumoto, Tsuguchika Tabaru, Atsushi Ikc, and Kohta Nakashima. Yet Another Accelerated SGD: 5e Training on ImageNet in 74.7 seconds. 1eXiv:1903.1266. fcs, stat), March 2019. URL httpp:/aaxxv..rrg
abs//900 12650
Dong Yin, Ashwin Pananjady, Max Lam, Dimitris Papailiopoolos, Kannan Ramchandran, and Peter Bartlett. Gradient Diversity: A Key Ingredient for Scalahle Distributed I Learning. In International Conference on Artificial Intelligence and Statistics, pp. 1998-2007. PMIR, March 2018. URI
html
Yang You, Igor Gitman, and Boris Ginsburg, Large Batch Training of Convolutional Networks.
arXiv: / 708. 03888 [cs], September 2017. URL CRLLtttp://ec http::/arxiv..rrg/////o wttpoorr///aa///a////////// 03888
Yang You, Jonathan Hseu, Chris Ying, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. I Large-Batch Training for LSTM and Beyond. arXiv: 08256 [es, stat], January 2019a. URL http: http::/arxiv..rm awwaaaaiivv.oo..... www.vvorr/aaae. /bbg/bb//
abs/
1901. 09256
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. In International Conference on Learning Representations, September 2019b. URL/sttps: //openreview net Eorumoirumssoo. -dx.mmm2--xxxxxxxxWWW
Yang You, Yuhui Wang, Huan Zhang, Zhao Zhang. James Demmel, and Cho-Jui Hsich. The Limit of the Batch Size. 205020006.88177 [cs, star), June 2020. URL /tatpi/ wtt. abwpg//bbb//00.../. 08511
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. A Critical View of Global Optimality in Deep Learning. vrXiv:1802.03487 [cs, math, stat], February 2018. URL http:tt::/a/....rrrt/ halg:///s////00 03487
Bohang Zhang. Jikai Jin, Cong Fang, and Liwei Wang. Improved Analysis of Clipping gorithms for Non-convex Optimization. Advances in Neural Information Processing Systems, 2020. URL httpos://proceedingg wwt..ppp.cc/ppprr////00.. 4ww.eeeee4ebeeee.... 393ce Abstract. html html
33:15511-15521, b282d1
Guodong Zhang. Lala Li. Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B. Grosse. Which Algorithmic Choices Matter Which Batch Sizes? In- sights From Noisy Quadratic Model. In Advances Neural Information Procesingg Sys- tems, volume 32, 2019a. URL Abs.bb6c221--.ttttc.........
neur tneurips.com ps..cc/paper//000 9/00199hashhh
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity. In International Conference on Learning Representations, Septem- 2019b. URL [tL.attsss//peeeeeaate
Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong. Steven Chu Hong Hoi, and Weinan To- wards Theoretically Understanding Why Sgd Generalizes Better Than Adam in Deep Learn-
ing. In Advances in Neural Information Processing Systems, volume 33. pp. 31285-21296 Cur- Associates, Inc., 2020. URL
neur
neur ps. cctpppppss//22//aaaa...
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The Anisotropic Noise in Stochastic Gradient Descent: Behavior Escaping from Sharp Minima and Regularization Effects. In International Con- ference on Machine Learning, pp. 7654-7663. PMLR, May 2019. URI. http: http::htttt:tceeeeet httt:::/ppocceedngss.it
press/ / / zhu1 9e. ht.m1
A EXPERIMENTAL SETUP
A. EXPERIMENTAL DETAILS
As mentioned in the main body, all experiments are evaluated on the CIFAR-IO dataset. The data is normalized per color channel. When augmented, the augmentations are random horizontal flips and random crops of size 32 x 32 after zero paddopg by 4 pixels in both spatial dimensions. For the experiments with fixed Nx CIFAR- 10, the data is fully written to a database (LMDB) in N rounds, to guarantee that the dataset is fixed. The same fixed dataset is used for all experiments using that dataset. The ResNet- 18 model used for most of the experiments is the default model as described in
16

---

Published as a conference paper at ICLR 2022
GD
SGD hyperparams
GD-tuned hyperparams
+ Clip + Grad Penalty
5
100
100
4
80
4
80
Accuracy
Loss
3-
60
3
60
2-
40
2
40
1
20
20
-1.0
-0.5
0.0
0.5
1.0
00.5
0.0
0.5
1.0
Figure 2: One-dimensional loss landscapes (random direction) of models trained with gradient de- scent. Default full-batch gradient descent (left) produces sharp models that do not train and general- ize well, yet it can be modified converge to flatter minima with longer training, gradient clipping and appropriate regularization (right).
He et al. (2015a 2019) with the usual CIFAR-10 adaption of replacing the ImageNet stem (7 x convolution and max- sox-pooling) with 3 convolution. All experiments in float 32 precision.
As mentioned further, the batch size during gradient accumulation is 128 not otherwise mentioned. Gradients averaged over all machines and batches using running mean. The optimizer is always gradient descent with Nesterov momentum (m = 0.9) with learning rates as specified in the main body. Weight decay is applied to all layers. The learning rate in the basic full-batch setting with 3000 steps decays from 0.4 to 0.0 (after the 400 step linear warmup from 0.0 to 0.4) via cosine annealing (with a single cycle in (He al., (209)) over 4000 ticks, so that 0. 0093 is reached the final iterate 3000 (The algorithm could be run for the additional 1000 steps to anneal to O, but we did not find that this hurt or hindered generalization performance and such iterate only up 3000 steps for efficiency reasons). For the initial learning rate of 0.8 used later, this corresponds to the same schedule with warmup, but starting from 0.8 iteration 400, which decays to 0.2187 iteration 3000. The gradient clipping is computed based on the l norm of the fully accumulated gradient vector (after addition of regularizer gradients if applicable) and the gradient vector is divided by this norm value with fudge factor of le-6, if the target norm value is exceeded. This is also the Py Torch (Paszke et al.) 2017) gradient clipping fudge factor. Batch normalization statistics are accumulated sequentially. If multiple GPUs are used then these accumulated statistics are averaged over all machines before each validation.
Gradient regularization described in the main body via forward differences approximation is implemented by in-place addition the already computed batch gradient to the model parameters with differential length
0.01/1| [ Eien n L(z,, 0)]l suggested in (Liu et al. 2018a). The gradient at this offset location is computed through automatic differentiation usual and the finite difference of both gradients is added to the loss gradient with the factor 4 Afterwards the original values of the model parameters in- place are restored exactly from copy.
All reported statistics are based on averaged results over 4-5 trials in all cases where standard de- viation reported. Numbers without standard deviation correspond to single runs. In a few edge cases where spike behavior such as in Figure T is seen at the final iteration 3000, and training and validation loss are accordingly large, we report maximal validation accuracy, although the value of validation accuracy minimal full training loss at the last iterate with gradient norm less than the clip value would also be a sensible fallbacks.
The loss landscape visualizations in Figure and Figure are computed by sampling the loss land- scape of a fixed model (with batch normalization in evaluation mode) in a fixed random direction, which is drawn by filter normalization described in [i et al. (2018) in single dimension.
We provide the code to repeat these experiments with our PyTorch implementation at github. com/Jonastas / ful battea/llaaatttatttttttaaaaaa rbatchttriinnnn
17

---

Published as a conference paper at ICLR 2022
A.2 COMPUTATIONAL SETUP
This experimental setup is implemented in PyTorch (Paszke et al. 2017), version 1.9. All experi- ments are run on an internal SLURM cluster of 4 + 8 NVIDIA Tesla 1100-CCIE-16GB GPUs. Jobs in the data-augmented setting were mainly run on the single GPUs one at a time, whereas the Nx CIFAR-10 10 jobs and in the fixed dataset section and jobs with gradient regularization were run distributed over 2 x 4 GPUs. In both settings, this amounts to 16-32 GPU hours (depending on hy- perparameters, especially gradient regularization) of computation time for each experiment, times N for the repeated Nx CIFAR-10 variants.
0.31
3001
Opiiniotios
Optisiotion
10
Tom 4ale)
Twen 0vg (o00e)
3.4
6:30
4001
1.031
14:80
4.01
Uptiniostion Steps
Optintcatin Stepo
Pani 160 (000
6.01
4.00
200
1000
1508
20
1200
12:6
2000
Optiniation Strps
Optintzatios Strpe
Leo solee)
Opelesiaaiine Stege
Optiniiaation Steges
Figure 3: Cross-Entropy Loss on the training and validation set and full loss (including weight decay) during training for full-batch gradient descent. Clipped steps are marked in black. Validation computed every 100 steps. From top to bottom: Top: training as described in Section 3.2 with and without clipping. All other rows: training with gradient regularization: FB regularized) on the left and FB strong reg. on the right. Second from the top: Training with Ir=0.4. Third from the top: Training with 1r=0.8 (this is the main setting investigated in this work). Bottom: Training with 1r=1.6.
18

---

Published as a conference paper at ICLR 2022
B FURTHER RELATED WORK
The phenomenon of effective non-existence of sub-optimal local minima is itself puzzling sub- optimal local minima do exist and can be found by specialized optimization techniques (Yun et al.) 2018 Goldblum et al. 2020), but they are not found by first-order descent methods with standard initialization
The update equation Eq. is often analyzed update of the full-batch gradient that is con- taminated by gradient noise arising from the stochastic mini-batch sampling, in this setting gradient noise is defined via:
gh-1 8h -T
VL(Bk)
+TR
Σ FEX
Vl(x,6')
1 B
Σ JEB
VL(x, 0K)
X
full loss gradient
gradient ge.
Analysis of flatness through other means, such dynamical system theory (Wu et al. 2018 Hu et al. 2018), also derives stability conditions for SGD and GD, where among all possible global minima, SGD both converges flatter minima than GD and also escape from sharp minima. Xing et (2018) analyze SGD and GD empirically in response to the aforementioned theoretical findings about noise shape, finding that both algorithms (without momentum) significantly differ in their exploration of the loss landscape and that the structure of the noise induced by SGD is closely related this behavior. Yin et al. (2018) introduce gradient diversity as measure of the effectiveness of SGD:
AD Σxex |/ (x;, N VL])|
which works well up critical batch size proportional to A (D))). Crucially gradient diversity is ratio of per-example gradient norms to the full gradient norm. This relationship is also investigated gradient coherence in Chatterjee (000) as it depends the amount of alignment of these gradient
vectors. Additional analysis of SGD as a diffusion process is facilitated in Xie et al. (2020).
Another angle theoretical understandings of the effects of SGD is the theoretical analysis based on convex optimization theory proposed in a series of works (Dauber et al.) [2020] Bassily et al.) 2020 Amir al. 202 2021a b). This line of work shows that convex loss functions can be constructed on which SGD converges to optimal generalization error orders of magnitude fast than full-batch GD.
As mentioned, critical mini batch size are studied in (Ma et al. 2018 Jain et al. 2018) after which SGD behaves similarly to full-batch gradient descent (GD) and converges slowly. This idea of critical batch size is echoed for noisy quadratic models in Zhang et al. (2019a), and empirical measure of critical batch size is proposed in McCandlish et al. (2018). There are also hypothe- (HaoChen et al. 2020) that GD necessarily overfits at sub-optimal minima linearized neural tangent kernel regime of Jacot et al. (2018); Arora et al. (2019b). Additional regu- larization, for example via clever weight decay scheduling as in Xie et al. in large(r)-batch) -batc settings, possibly by alleviating such overfitting effects.
trains in the
(2021) improves training
C ABLATION STUDIES
C.1 ADDITIONAL. VARIATIONS
We add additional information for Table in Table 4 listing training loss well as full loss (which includes regularizations). Tableaccontiin several ablation studies centered around number of steps and learning rate. We visualize the experiments from Table I as well variations in learning rate in Fig. 3
Overall, find that the hyperparameters of the final variant (gradient descent with both clipping and regularization) are surprisingly stable. For example, we find that any learning rate within [0.4, 1.6) ends up generalization performance around 95% [94.70, 95.69] which is large stable interval,
19

---

Published as a conference paper at ICLR 2022
S0D hygerparams
SGD reparparams
CD-saned hypergarams
CD (TD-uned Reperpermms
GD-raad typerperams.
Clip
Clip Grad Penalty
1.00
1.00
100
100
100
60
20
20
3
20
20
-1.0
-6.5
00
0.5
1.
9.0
-05
n.a
1d
-0.5
0.5
1.8
-5.5
0.0
0.5
1.
-1.0
-0.5
0.0
1.8
Figure 4: One-dimensional loss landscapes visualizations (random direction) of models trained with gradient descent, going from SGD (left) to GD with successive modifications (right).
compared even to baseline stochastic gradient descent. Relatedly, although we do decay the learn- ing rate slightly (i.e. from 0.8 0.2), this leads only a minor benefit and models generalizing almost as well can also be found with fixed learning rate. For clipping, we mostly stuck to the gradient norm value of 0.25 suggested by related literature for ResNet architectures (although any clip value in [0.25, 1] appeared decent) and did not spend significant efforts tuning it further (aside from experiments with VGG, where we find that 0.25 was indeed too small to allow the model to make sufficient training progress and where main body). A coefficient a for the strength of the regularizer was introduced in the main body, yet from brief exploratory analysis, we found that the value of 1 suggested by theory optimal (Note that this is implemented as 0.5 in the code to account for the factor in Eq. D The finite differences approximation via 0.01/IIVL(0")) from Liu et al. (2018a) worked well out of the box, although we found similar results for [5e - 2, 1e 3J. We also experimented with a higher- precision finite difference (see below). Several other parameters, such as batch norm momentum, Nesterov momentum, weight decay and model initialization were adopted directly from the SGD ResNet18 bascline model and not investigated.
upped the clipping value to 1.0 described in the
indeed
Experiment
Steps
Modifications
Val. Acc. %
Train Loss
Full Loss 0.0976(15.022. 4)
Baseline SGD
117'000
95.70(=0.11)
0.0017(14..43:
Baseline FB
300
75.42(=0011)
0.321(
FB train longer
3000
87.36(+1.23)
0.0353(+2..17
0.0115(12.224
FB clipped
3000
clip clip+reg
93.85(=0.10))
0.00173(4:3.26e-
0.0170
FB regularized
3000
95.54(=0.009
0.00161
0.0259 0.00158(+1.25e44)
FB strong reg.
3000
clip+reg+bs32
95 (5.688(0)) (0.099
0.0381(#1.00ee2) 0.00236(+2.01e-4) 17.600
FB in practice
3000
clip+reg+bs3++shuffe
(=0.14)
Table 4: Experiments considered in Table [ with additional information in training loss (cross en-
tropy loss on the CIFAR-10 training set) and full loss, including regularizations. Validation accu- racies on the CIFAR-IO validation set for each experiment with data augmentations considered in Section 3
Best viewed screen.
C.2 VARIOUS TYPES OF MINIBATCHING
As deseribed in the main body, mini-batch SGD (sampling without replacement) reaches a valida- tion accuracy of 95. 75..0% (...0 which we consider very competitive modern baseline for this architecture. Mini-batch SGD provides this strong baseline largely independent from the exact fla- of mini-batching as can be seen in Table 6. reaching the same accuracy when sampling with replacement. In both cases the gradient noise induced by random mini-batching leads to strong generalization. If batches are sampled without replacement and in the same order every epoch, i.c. without shuffling in every epoch, then mini-batching still provides its generalization benefit. The apparent discrepancy between both versions of shuffling not actually a SGD effect, but shuffling benefits the batch normalization layers also present in the ResNet- 18. This can be seen by replac- ing batch norm with group normalization (Wu& & He 2018), which has no dependence on batching. Without shuffling we find 94. 01.44%(=0.2 normalized ResNets; a difference of less than la. Overall all of these variations of mini-batched stochastic gradient descent lead to strong generalization after training. As validation of previous work, we also note that the gap between SGD and GD is not easily closed by injecting simple forms of gradient noise, such additive multiplicative noise, can also be seen in Table 6
for SGD and with shuffling 64.55%(11..
for group
20

---

Published as a conference paper at ICLR 2022
Experiment
Steps
Modifications
Val. Acc.%
SGD regularized
117'000
reg
00..18 (5.81%(=0.18)))
SGD all changes
117'000
clip+reg
93.23%
SGD all changes strong reg
117'000
clip+reg+bs32
84.35%
FB clipped
300
clip
84.40% (-)
FB strong reg.
300
clip+reg+bs32
(6.00)(=
FB strong reg. long
40'000
reg+bs32+1r=0.08
92.19%(-
FB strong reg
6000
cliptreg+bs32
05.68%((0.100))
FB regularized
6000
cliptreg
95.37%(=0.12))
FB strong reg no clip
3000
reg+bs32
91.08%
FB regularized double
3000
cliptre++Ir=1.6
95.69%((0.006)))
FB strong reg double Ir
3000
clip#reg=bs33++1=111
95. 95.23%(-
FB regularized half Ir
3000
cliptreg+1r=0.4
94.70%(++.17)) 94.97(+0.05)
FB strong reg half Ir
3000
clip#reg=hs3++110044
Table 5: Hyperparameter ablation studies for experiments considered in Table I Validation accu- racies the CIFAR-IO validation for each experiment with data augmentations considered in Section
The number of trials for each experiment is included in parentheses.
Source of Gradient Noise
Batch size
Val. Accuracy %
Sampling without replacement
128
95.70(=0.11) 95.70(=0.05)
Sampling with replacement
128
Sampling without replacement (fixed across epochs)
128
95.25(=0.07)
Additive = 0.01
50'000
61.41(=0.09)
Multiplicative m = 0.01
50'000
(9.25(=0.14)
50'000
75.42( +0013)
Table 6: Summary of validation accuracies on the CIFAR-10 10 validation dataset for baseline types of gradient noise experiments with data augmentations considered Section
C.3 NUMERICAL STABILITY
Technically, the computations in this paper may still contain residual errors in the gradient com- putation. This is due to of GPUs central computational units which operations deterministic by default (Chetlur et al. 2014 NVIDIA 2022). We measure the amount of error between consecutive evaluations of single gradient Table However, the errors introduced by non-determinism are notably small in magnitude, especially compared to the effect size of noise induced by normal mini-batching (which highlight the first row). Even over a full training run, where the baseline SGD would take 117'000 steps with this magnitude of gradient noise (totaling an upper bound z 5534 points of relative gradient error), these errors estimates would remain small as only 3000 steps are taken (totaling an upper bound of x= 2.77e-4 units of relative gradient error). We note that we implement the accumulation of gradients over the whole dataset with numerically stable online mean, instead of relying the naive accumulation in gradient leafs as in default orch. We also experimented with accumulating into higher precision (double), but did not find any effects in comparison to accumulation in single precision.
We verify the minimal impact of these findings through a sanity check: We repeat experiments from Table in Table g in the slow deterministic mode of cudnn (and further disable all possibly non-determimistcc operations in pytorch), but find no significant differences. We also check for confounding effeets introduced by single floating point precision used in experiments by running experiments in double precision. Both variations fail to find significant differences to the non-deterministic, single precision experiments that form the remainder of this work.
C.4 OTHER TECHNIQUES GENERALIZATION
Several other strategies to increase generalization performance have been proposed in literature about latch-batch training. Here we enumerate some of the strategies as alternative to gradient clipping in the main body. The baseline here is 87.36, i.e. FB with long training. Label
21

---

Published as a conference paper at ICLR 2022
Experiment
Batching
Modifications
Total Error
Relative Error
Baseline SGD
0.0988 + 0.0055)
0.0173(+0.01074)
Baseline FB
X
6.57e-7( - 44200e-8
3.40e-7 -
FB clipped
X
clip
8.93e-8 -
1.57e-7( - 1.22e
FB regularized FB strong reg.
clip+reg
1.15e-7
2
4.60e-7 7 8.83e-7
cliptreg+bs32
9.24e 8 3.97e
3.70e 7 1.59e
FB strong reg. (det.)
X
clip+reg+bs32
0(10)
0 ( + 0)
FB strong reg. (fp64)
X
clip+reg+bs32
6.25e 8.58e-13)
2.50e 3.4
Table 7: Numerical Stability of Gradient Computations in several settings for a single sample of data, controlling for data augmentations. Shown are total and relative euclidean emoor averaged 5 gradient computations on randomly initialized model. For baseline SGD this is the gradient noise between randomly sampled mini-batches batch size 128 plus numerical effects. For fullbatch gradient descent this is noise produced solely by numerical approximation due to non-determinism when computing the gradient by accumulation over the entire dataset.
Experiment
Mini-batching
Epochs
Steps
Modifications
Val. Ace.
Baseline SGD (det.)
300
117,000
95.66
FB regularized (det.)
x
3000
3000
clip+reg
05.40(+0.000
FB strong reg. (det.)
X
3000
3000
cliptreg+bs32
45.66(+-.100
Baseline SGD (fp64)
/
300
117,000
95.66(#0.07 95.54
FB regularized (fp64)
X
3000
3000
cliptreg clip+reg+bs32
FB strong reg. (fp64)
X
3000
3000
95.62(+0.00)
Table 8: Validation accuracies for each experiment considered in Section 3 for deterministic runs (referring to disabled cudnn non-deterninism) and runs double (fp64) floating point preci- sion. All validation accuracies are averaged over 2-5 runs. These experiments show significant difference to the non-deterministic runs in single floating point precision.
smoothing with smoothing value of 0.1 leads to full-batch performance of 55.94%((18.8)) (which is lower on average, due to one run at 70%. If that run were treated as outlier and removed, we would find 90.34%(++..23)) 93.85 93.85%(=0.10 with which label smoothing does not stack. Applying weight decay only to linear (convolutional and fully-connectedd layers leads performance of 85 (53.94))3.8))). Sharpness- aware minimization (SAM) on the full gradient level leads to 68. (8.77%((+7..1)) even with our in- creased budget to 3000 iterations and gentle learning rate scheduling. however note the connection of gradient regularization and SAM when accumulated over mini-batcher discussed in Appendix C.7
This is still significantly lower than the gradient clipping value of
Furthermore, regarding discussions about reducing the batch size [B] for SGD, we find a validation accuracy of 94.99( 19..19) at [B] 32 without hyperparameter adaptation. When the learning rate scaled to 0.05, SGD reaches 95. (00556(000))) which is slightly below the value for [ = 128. Using [B| 32 for accumulation is further not advantage for variants without regularization, FB train longer in Table [ reaches 77.42% accuracy with this batch size, and FB clipped reaches 93.43%. Both numbers are below the performance accumulation batch size 128.
C.5 TRAINING WITHOUT SPIKES
Figure 5 shows training curves for stable full-batch training without spiking behavior. However, the optimization does not reach levels of performance shown for = 0.4 in the main body within the allotted 3000 steps.
C.6 HESSIAN-VECTOR PRODUCT APPROXIMATION
The Hessian-vector product necessary to compute the gradient of the gradient regularization of Equation (6) can also be compute by automatic differentiation instead of using finite differences. Using Pytorch's 's automatic differentiation via autograd. grad lead to a validation performance of (9.434%((- (-) for preliminary experimental setup with label smoothing and 0.1, for which the forward differences approximation we otherwise employ found 94.20% (), so that both approaches performed near identical.
22

---

Published as a conference paper at ICLR 2022
Opendbaton
Openobadio
Figure 5: Cross Cooss-Entropy Loss on the training and validation set during training for full-batch gra- dient descent in direct comparison Figure 2 in the main body but with reduced learning rates. Top Row: T = (0.05, 0.01, 0.005}. Bottom Row: Same step sizes but with gradient clipping of 0.25. T 0.4 pictured with and without clipping in the main body. Training behavior is stabilized at lower learning rates, but significantly slowed in most cases. In the first case, some overfitting appears in the validation loss, possibly corresponding to the reduced regularization of the full gradi- ent, i.e. |Barrett & Dherin (2020) and catapult behavior of Lewkowycz et al. (2020). Weight decay regularization not pictured. Validation computed every 100 steps.
Another alternative would be improve the precision of the finite-differences approximation. Al- though we employ scheme which has beneficial approximation properties. Using central differences instead of cen- tral differences leads to 94.80(-) in another preliminary FB regularized setting in which forward- differences yielded 94. 74.00. 17). i.e. similar performance.
scheme, it is also possible to utilize sentral-differenee
However, the additional computational effort is significant is both cases. Computing the FB regular- ized experiment on x 4 GPUs takes about hours, 14 minutes. The same experiment with central differences takes about 4 hours, 27 minutes. Finally, with automatic differentiation this takes about 8 hours and 40 minutes so that we employ the forward differences approximation only, especially in preparation for the 10x and 40x CIFAR- experiments.
C.7 RELATIONSHIP OF HVP APPROXIMATION AND SHARPNESS-AWAR MINIMIZATION
The gradient regularization of Smith et al. (2020b) is related to the sharpness-aware minimization Foret al. (2021) if the latter is computed on a mini-batch level and accumulated over the entire dataset. This relationship is especially apparent when using the approximation of the Hessian-vector product with step size selection as proposed Liu al. (2018b). In our notation, the sharpness- aware minimization update consists of an update step based on the gradient
gSAM VC (0+
P
VC(0)
(10)
[VC(O
In comparison the update via loss gradient plus derivative of the regularizer can be be written as
Igradreg = VL()) + aT VL + eVL(()) V2(0)
(11)
2
If we consider the differential step size of ε 0.01 ( p 0.01, the can rewrite this update to
(Liu al. 2018b) and identify
(1 -11-2 2
|| [V(()]]
VC(0)+ at VC(())
VL ||VC())] VZ(0))
P
Ogradreg
(12)
2p
This shows that from the point of view of Foret et al. (2021). gradient regularization is an interpo- lation between the normal loss gradient and the adversarial gradient that depends on the step size.
23

---

Published as a conference paper at ICLR 2022
From the point of view of Smith al. (2020b), SAM minimization accumulated over mini-batchin finite-difference approximation the gradient regularization for a fixed step size. Both are e training in experiments), this happens whenever ||
equivalent iff [(V) 1. If p 0.01, = 0.5, 7 = 0.1 (the hyperparameters at the end of
0.2. At the beginning of training, i.e. = 0.4, equivalence reached at || = 0.05. If the gradient norm is greater than this equivalence, then the adversarial gradient dominates, if it smaller the loss gradient V (()) domi- nates. However we note that according to the experiments in Appendix [C6, gradient regularization can also be implemented via automatic differentiation, so that (in the spirit of this work) the finite differences approximation itself is not necessary for the generalization effect of this regularizer.
C.8 CHAOS THEORY WHAT OUR RESULTS DO NOT SHOW
We would like to point out that while our results show that stochastic mini-batching (or even non- stochastic minibatching) in gradient descent is not necessary achieve state-of-the-arr generaliza- tion behavior, this does not entirely rule out stochastic modeling of the behavior of GD for deep neural networks proposed in works such Chaudhari & Soatto (0088); Kunin al. (202T) and Simsekli et al. (201)). Even a full-batch gradient descent algorithm could potentially exhibit chaotic behavior the loss surface of deep neural networks (Kong & Tao 2020), which could be modelled by statistical techniques. In this work, we can make no statement about whether chaotic behavior exists for these examples of gradient descent and whether it has impact on model performance.
D ADDITIONAL DETAILS
D.I ASSET LICENCES
We use only CIFAR-10 data (Krizhevsky, 2009) in our experiments, for more information refer to https: /www. cs toronto......oo..o... edw/~kriz/cia included within their respective files and can be found as part of our code release.
html Code licenses for submodules are
24


Reviews:
[{"review_id": "_3InxIo7T6_", "reviewer": "Reviewer_o29S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper reveals an important message to the field. Recent works on SGD are more or less all based on the idea that minibatch training improves generalization and is worth studying on its own, and this work makes the surprising discovery that the minibatch sampling process is, in fact, unnecessary for a good generalization. The same effect can be reproduced by a series of deterministic tricks on GD\n\nHowever, I feel that the way the authors present the conclusion is a little too negative against SGD. In fact, from a practitioner's point of view, I become much more convinced of the advantage of minibatch training after reading the paper (instead of finding it unnecessary). The reason is this: the paper essentially shows the following:\n - GD + minibatch sampling = GD + learning rate schedule + large learning rate + gradient clipping + gradient penalty\n\nNamely, disregarding the computational efficiency that the minibatch technique brings, simply using mini batching alone already gives me the advantage of all the later four tricks taken together! Not to mention that a simpler technique tends to be more robust in more complicated settings. \n\nTherefore, I think one additional conclusion that should be reached is that the minibatch sampling technique in SGD is indeed a really good technique -- but this point is not pointed out in the paper, and missing this point I think can be highly misleading for the general audience, given that the paper mainly has a negative tone against SGD. I thus think the authors need to revise their conclusion to reflect this point (or convince me why this additional conclusion is incorrect).\n\nAdditionally, I think that one crucial experiment is lacking: if one adds minibatch sampling on top of GD + learning rate schedule + large learning rate + gradient clipping + gradient penalty, what will happen? The authors are obliged to show that when the minibatch sampling is added, there is no further improvement\n\n\n", "summary_of_the_paper": "The paper shows what the title says; to paraphrase, the paper shows that GD with a bunch of non-stochastic tricks can be as good as SGD with minibatch noise ", "summary_of_the_review": "Update: I am satisfied with the author's reply. I think the message is sufficiently novel and important to be published at ICLR. Though I actually would like to give a 7, due to ICLR's scoring constraints, I can only give a 6. I do not feel that giving 8 is appropriate because this paper only suggests the possibility of SGD noise being non-essential, but does not give a definitive answer. \n\nTo be specific, the main speculation/conjecture is only validated for a single example, it is completely unclear whether these results are generalizable or not. While I do find meaning and insight in the single example the authors give, I feel that more convincing empirical evidence is required to obtain a score higher than 7 and, more importantly, to reduce the speculative nature of the present conclusion.\n\n--------------\n\nI lean towards acceptance because I find the discovery surprising and important. However, I give a 5 because I feel that the authors missed one crucial conclusion, which might mislead the readers, and missed one crucial control experiment. I will change the score if these two points are addressed satisfactorily", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "RjQuEfsIFUX", "reviewer": "Reviewer_XRTg", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "== Updated score ==\n \nI am satisfied by the results of the control experiments, and the explanations of the authors. I have read the other reviewer comments, and I believe the work will be quite valuable for the ICLR community and thus have changed the score to a strong accept. \n\n========\n\nAuthors do thorough experiments to bring out the nuances around the topic with the CIFAR-10 dataset, and Residual Networks. \n\nStrengths:\n+ Very thorough set of experiments ablating over various sources of stochasticity\n+ Detailed explanation of experimental protocol and reproducibility\n+ Shows explicit regularization can bridge the gap, and shows efficient implementation and also discusses prior work (SAM)\n\nWeakness/Questions:\n+ Does explicit regularization help SGD (stochastic mini-batch training) (This is an important control experiment)? If it changes the target solution quality - how does one reason about matching generalization of stochastic mini-batch training. \n+ Do the results hold across wider range of datasets, While I can see how much larger datasets could become more expensive - it would have been very satisfying to verify.\n\nSome other comments: Relevant citations that are missing, which hopefully authors can easily address:\nSecond order optimization that seem to work well for Neural Nets:\nhttps://arxiv.org/abs/1503.05671 (KFAC)\nhttps://arxiv.org/abs/2002.09018 (Shampoo)\nhttps://www.bmvc2020-conference.com/assets/papers/0479.pdf (L-BFGS)\n\nRecent work on curvature and gradient clipping (and other techniques): \nhttps://arxiv.org/pdf/2110.04369.pdf\n\n", "summary_of_the_paper": "Paper presents empirical results that show that full-batch training of neural networks can still generalize with appropriate explicit regularization. Authors propose explicit regularization to minimum norm solution and discusses techniques for efficient implementation as well as relation to prior work.", "summary_of_the_review": "This paper provides convincing contrary evidence that shows stochastic mini-batching by itself is not unique, and can be substituted with explicit regularization, and techniques such as gradient clipping.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "PcX9vy1Wnzl", "reviewer": "Reviewer_g1Z8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has strong merits on the technical side, offering several well-executed novel experiments:\n- First full-batch training regime of cifar10 with competitive validation accuracy\n- Removal of any noise affecting training (full-batch, fixed batch normalization slices)\n- \"Augmentation aware\" full-batch training -- by pre-enlarging the dataset with fixed augmentation, thus making sure comparison to practice is fair.\n\nThe authors also offer some practical methods that may prove useful in the future, by offering gradient-clipping and regularization to combat large batch induced accuracy degradation and failures. \nThe paper is also well written and easy to follow with a good cover of past efforts on this topic. \n\nMy main concern with the work is that several choices seem arbitrary and not well explained:\n- Number of iterations: as seen in past works (Hoffer et al. 17',  Shallue et al. 19'), adapting the training regime with more training steps can close the gap between large and small batch training. This is also apparent in this work, as authors increase the number of steps to 3000 (10x the number of epochs) to reach the comparable accuracy milestone. However, this number is not justified in any way and no further exploration in that regard is made. Some possible questions that arise: (1) can we apply the same number of iterations as in small batch (117K) to close the gap completely with no need for additional regularization? (2) what is the tradeoff in terms of accuracy? (3) in terms of computational effort?\n- Regularization - the authors choose a specific regularizer to replace the implicit bias offered by SGD. However, several other choices can be offered instead (e.g SAM, preconditioning). More importantly, does the regularization really replaces SGD? what accuracy do we get when the same regularizer is applied to baseline SGD training?\n- Learning rate adjustment - authors use x2 learning rate instead of previous scaling methods (sqrt/linear scaling), in what seems like a completely arbitrary choice. As previous works used signal-to-noise ratio of gradients to suggest scaling of the learning rate, I expect authors to suggest and demonstrate why (and if) these break at the full-batch limit, when no noise is introduced.\n", "summary_of_the_paper": "The paper discusses large batch training at its limit -- full-batch cifar10 training when no stochasticity is introduced. This setting allows authors to examine and discuss the common conception about the generalization benefits of SGD. The authors show that replacing implicit bias of SGD with explicit regularization can eliminate the generalization gap.", "summary_of_the_review": "As explained in the review, the paper has several technical merits and interesting experiments but is clouded with ill-justified and what seems like arbitrary choices and solutions. Currently, I feel these hurt the potential value of the paper to the field. \n\nMy view is that the paper in its current form should be rejected, but I suggest the following improvements:\n- Discuss regime adaptation (increasing number of steps) similar to Hoffer et al. by performing additional experiments with a varying number of epochs. I understand that a complete 117K steps training in full-batch is too cumbersome, but the current single datapoint of 3000 epochs seems arbitrary and not justified in any way.\n- Regularization on baseline SGD -- does it help? will be very interested to see a discussion either way.\n- LR adjustment and gradient clipping justification -- perhaps additional measurements of gradient noise may show why these are needed over \"traditional\" scaling techniques.\n\nIf authors work to lift these concerns in their rebuttal, I will consider raising my score.\n\n\n\n----- Post rebuttal update -----\nThe authors answered most of my concerns, so I'm updating my score as promised. I still feel some of the choices are arbitrary (or at least not substantiated enough) and may apply to the specific case presented only (dataset, model), so I will raise my score 5->6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PFzbfwcb-lt", "reviewer": "Reviewer_Tx6S", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The theoretical implications of the experiments are not significant. SGD is an empirical regularization technique.  So models trained with SGD *may* empirically overfit less than models trained in other ways.  However, SGD is neither necessary nor sufficient for models to generalize.\n\nSome theory papers argue that SGD is sufficient for generalization.  For example, the paper, Stability and Generalization, Bousquet and Elisseeff,  shows that a technical form of stability is sufficient for generalization.  The paper, Train faster, generalize better: Stability of stochastic gradient descent, Moritz Hardt,  Benjamin Recht, Yoram Singer, argues that SGD introduces stability.  However, no one has argued that SGD is *theoretically necessary* for generalization.\n\nThe paper offers \"A MORE SUBTLE HYPOTHESIS *we can modify and tune optimization hyperparameters for GD and also add an explicit regularizer in order to recover SGD’s generalization performance without injecting any noise into training.*\"\n\nTo me this is an empirical technique, not a theory hypothesis.  A theory hypothesis would be something along the lines of : perform this type of regularization, and we can prove a bound on the generalization gap of the model.  An empirical theory hypothesis would be something like: perform this type of regularization, and in practise, we obtain similar generalization performance to SGD.  However, this would require more substantial empirical evidence, than is presented in the paper. \n\n", "summary_of_the_paper": "This paper shows that it is possible to train CIFAR 10 models with full batches, and still obtain get test accuracy.  While training with SGD leads to accuracy 95+ percent, training with baseline full batch degrades accuracy in a range of 30-77 percent.  However, introducing a number of regularization discussed in section 3 can close the validation accuracy gap between SGD and FB.\n\nThis paper spends about half the time discussing theory heuristics for SGD regularization, and half the time discussion regularization heuristics which are compatible with full batch optimization.   \n\nAs an empirical paper, there are a number of regularization techniques discussed in section 3: baseline sgd, stabilizing training, finite difference regularization of the Hessian of the loss equation (7), learning rate schedules, gradient clipping, gradient penalty, data augmentation. However, CIFAR-10 is no longer a challenging dataset. \n\nFrom the paper: \"A number of authors have studied relatively large batch training, often finding trade-offs between batch size and model performance (Yamazaki et al., 2019; Mikami et al., 2019; You et al., 2020). However, the goal of these studies has been first and foremost to accelerate training speed (Goyal et al., 2018; Jia et al., 2018), with maintaining accuracy as a secondary goal. In this study, we seek to achieve high performance on full-batch training at all costs. Our focus is not on fast runtimes or ultra-efficient parallelism, but rather on the implications of our experiments for deep learning theory.\"", "summary_of_the_review": "The achievement of the paper is an empirical result: good validation accuracy on CIFAR 10 using full batch training and a number of regularization techniques.  But the paper makes a claim of implications for deep learning theory which is not substantiated. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ef1wSj7NPOh", "reviewer": "Reviewer_UbWk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In the deep learning community, it is widely *believed* that the noise in SGD is a necessary component of any performant neural network. Over the past few years, theory papers have emerged that explicitly attribute the success of neural networks to the special properties of SGD. As most of the effort in the community has been focused on models trained with SGD-type algorithms, the conclusions of these papers so far have been mostly unchallenged. By showing that simple full-batch gradient descent can indeed achieve results similar to SGD, the paper provides an impactful contribution to the theory community. The paper is well-written and the experimental set-up is carefully designed and communicated in detailed. The fact that the authors can replicate their findings across a number of different architectures (Table 2) increases my confidence in the presented results.  \n\nI list my questions / concerns below:\n\n* One can attribute variations in the validation accuracy to either issues in the optimization or generalization aspects of the model. When going through Tables 1 & 2 and Appendix C.2, it was not clear to me where exactly the improvements are coming from. I strongly suggest reporting metrics such as training loss & accuracy with the validation metrics in order to avoid confusion. \n\n* I wonder if adding the explicit regularization provides the same kind of regularization as the noise in gradient descent. In particular, do SGD and GD models make similar predictions (and similar mistakes)? Will adding the explicit regularization on top of SGD improve the model?\n\n* Figure 2 compares the loss flatness across a random direction in parameter space for different models. Can we reliably state anything regarding the flatness of the loss landscape by using 1-D projections of a O(1M) dimensional loss manifold? Moreover, note that the curvature of these models depend on the normalization parameters of the BN layer. By playing with these normalization parameters, one can change the curvature without changing the predictions of the model. This makes the cross-model curvature comparison that Figure 2 is attempting highly unreliable. ", "summary_of_the_paper": "The paper examines the role of SGD noise in the generalization performance of neural networks. In particular, the paper examines if the generalization performance of SGD-trained models can be replicated with explicitly regularized models trained with full-batch gradient descent. The authors show that this is indeed possible for a number of different architectures (ResNet, DenseNet, VGG) trained on CIFAR-10 dataset. Based on these observations, the authors conclude that any theory that relies exclusively on stochasticity of training to explain generalization in neural networks is unlikely to capture the true phenomena responsible for the success of deep learning.", "summary_of_the_review": "The paper provides a strong contribution to the community. The main conclusions of the paper are supported with strong empirical evidence. There are a number of improvements / clarifications that can be added to the paper (which I have listed above).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper carefully shows how all the stochastic elements in neural network training could be removed (by using full batch, and a dataset with fixed augmentation) and still maintain good performance, by adjusting hyper-parameters and adding explicit regularization. 
All reviewers were eventually positive and recommended acceptance, except one reviewer, who was initially not aware of the recent theoretical interest in this question, and was therefore less surprised.

There are three remaining issues with the current version: 
1) Operations in cuDNN are, by default, non-deterministic, but can be made deterministic. Though I believe this will not affect the final results, without it, the title and conclusions of this paper are technically unjustified. The authors have agreed to add this to the camera ready version of the paper.
2) Deterministic training was only shown on CIFAR10. I understand ImageNet would be too heavy for this task, but there are many other small and medium-sized datasets, and I think showing that this on several such datasets would strengthen the message of this paper, and convince more readers. 
3) The question of how to achieve good performance with deterministic training is still mostly unknown, as it seems to require significant hyperparameter search (with unknown sensitivity), and no conclusion was reached regarding the how to properly adjust them. However, I agree that a good answer to this question might not exist.

Lastly, I recommend the authors to mention in the main paper the new baseline experiment, where the explicit regularization is added to SGD. I saw it in the appendix, but I didn't see it mentioned in the main paper (maybe I missed it). I think without it, many readers will not be fully convinced (as several reviewers requested it).
Decision: Accept (Poster)