Disentangling the Mechanisms Behind Implicit Regularization in SGD
Authors: Zachary Novack
Simran Kaur
Tanya Marwah
Saurabh Garg
Zachary Chase Lipton
Venue: ICLR 2023 poster (2023)
Keywords: deep learning
generalization
implicit regularization
sgd

Abstract: A number of competing hypotheses have been proposed to explain why small-batch Stochastic Gradient Descent (SGD) leads to improved generalization over the full-batch regime, with recent work crediting the implicit regularization of various quantities throughout training. However, to date, empirical evidence assessing the explanatory power of these hypotheses is lacking. In this paper, we conduct an extensive empirical evaluation, focusing on the ability of various theorized mechanisms to close the small-to-large batch generalization gap. Additionally, we characterize how the quantities that SGD has been claimed to (implicitly) regularize change over the course of training. By using micro-batches, i.e. disjoint smaller subsets of each mini-batch, we empirically show that explicitly penalizing the gradient norm or the Fisher Information Matrix trace, averaged over micro-batches, in the large-batch regime recovers small-batch SGD generalization, whereas Jacobian-based regularizations fail to do so. This generalization performance is shown to often be correlated with how well the regularized model’s gradient norms resemble those of small-batch SGD. We additionally show that this behavior breaks down as the micro-batch size approaches the batch size. Finally, we note that in this line of inquiry, positive experimental findings on CIFAR10 are often reversed on other datasets like CIFAR100, highlighting the need to test hypotheses on a wider collection of datasets.

Published as a conference paper at ICLR 2023
DISENTANGLING THE MECHANISMS BEHIND IMPLICIT REGULARIZATION IN SGD
Zachary Novack
Simran Kaur
UC San Diego
Princeton University
znovackeucsd.com
skaur@princeton..dd
Tanya Marwah
Saurabh Garg
Zachary Lipton
Carnegie Mellon University
Carnegie Mellon University sgarg2@andrew. cmu edu
Carnegie Mellon University zliptoneanden cmm, edu
tmarwah@andrew, cmu, edu
ABSTRACT
A number of competing hypotheses have been proposed to explain why small- batch Stochastic Gradient Descent (SGD) leads to improved generalization over the full-batch regime, with recent work crediting the implicit regularization of var- ious quantities throughout training. However, to date, empirical evidence assess- ing the explanatory power of these hypotheses is lacking. In this paper, con- duct an extensive empirical evaluation, focusing on the ability of various theo- rized mechanisms close the small-to-large batch generalization gap. Addition- ally, we characterize how the quantities that SGD has been claimed to (implicitly) regularize change over the course of training. By using micro-batches, i.e. dis- joint smaller subsets of each mini-batch, we empirically show that explicitly pe- nalizing the gradient norm the Fisher Information Matrix trace, averaged over micro-batcher in the large-batch regime recovers amall-batch SGD generaliza- tion, whereas Jacobian- regularizations fail to do so. This generalization per- formance is shown to often be correlated with how well the regularized model's gradient norms resemble those of small-batch SGD. We additionally show that this behavior breaks down the micro-batch size approaches the batch size. Fi- nally, we note that in this line of inquiry. positive experimental findings on CI- FARI0 10 are often reversed other datasets like CIFARI00) highlighting the need to test hypotheses on a wider collection of datasets.
INTRODUCTION
While small-batch SGD has frequently been observed to outperform large-batch SGD (Geiping al., 2022; Keskar et al., 2017: Masters and Luschi, 2018; Smith et al., 2021; Wu al., 2020; Jastrzebski et al., 2018; Wu et al., 2018; Wen et al., 2020; Mori and Ueda, 2020), the upstream cause for this generalization gap is a contested topic, approached from variety of analytical perspectives (Goyal et al., 2017; Wu et al., 2020; Geiping et al., 2022; Lee et al., 2022). Initial work in this field has generally focused on the learning rate to batch-size ratio (Keskar et al., 2017: Masters and Luschi, 2018; Goyal et al., 2017; Mandt et al., 2017; He et al., 2019; Li et al., 2019) or on recreating stochastic noise via mini-batching (Wu et al., 2020; Jastrzebski et al., 2018: Zhu et al., 2019; Mori and Ueda, 2020; Cheng et al., 2020; Simsekli et al., 2019; Xie et al., 2021), whereas recent works have pivoted focus on understanding how mini-batch SGD may implicitly regularize certain quantities that improve generalization (Geiping al., 2022; Barrett and Dherin, 2020; Smith et al., 2021; Lee et al., 2022; Jastrzebski et al., 2020).
In this paper, we provide careful empirical analysis of how these competing regularization theories compare to each other as assessed by how well the prescribed interventions, when applied in the large batch setting, recover SGD's performance. Additionally, study their similarities and differences by analyzing the evolution of the regularized quantities over the course of training.
Our main contributions are the following:

---

Published as a conference paper at ICLR 2023
CIFARI00
CIFARIO
Micro batch Gredient Norm
Averaae M betch Gradient Norm
Validation Accurscy
Mcc batch Gradlent Norm
Validation Accuracy
Mccr-batal Gradient Norm
Veltdation Accumey
Velldeii Accube
80
60
85
10
20
60
20
10303
25020
30329
10
1D
10200
20000
30000
10
10
lteration
Iteration
Iteratonn
Iteration
SB SGD
LB SGD
LB SGD GN
LB SGD + FT LB SGD A]
LB SGD UJ
Figure Validation Accuracy and Average Micro-batch (MM) 128) Gradient Norm for CI- FAR10/100 Regularization Experiments, averaged across runs (plots also smoothed for clarity). In both datasets, Gradient Norm (GN) and Fisher Trace (FT) Regularization mimic the average micro- batch gradient norm behavior of SGD during early training and effectively recover generalization performance (within small margin of error), whereas both Average and Unit Jacobian (AJ and UJ) fail to do so.
1. By utilizing micro-batches (i.e. disjoint subsets of each mini-batch), we find that explicitly regularizing either the average miero-batch gradient norm (Geiping et al., 2022; Barrett and Dherin, 2020) or Fisher Information Matrix trace (Jastrzebski et al., 2020) (equivalent to the average gradient norm when labels are drawn from the predictive distribution, detailed in Section 2.2) in the large-batch regime fully recovers small-batch SGD generalization performance, but using Jacobian-bars basad regularization (Lee et al., 2022) fails to recover small-batch SGD performance (see Figure 1).
2. We show that the generalization performance is strongly correlated with how well the tra- jectory of the average micro-batch gradient norm during training mimics that of small-batch SGD,, but that this condition is not necessary for recovering performance in some scenar- ios. The poor performance of Jacobian regularization, which enforces either uniform or fully random weighting on each class and example (see Section 2.3), highlights that the beneficial aspects of average micro-batch gradient norm or Fisher trace regularization may come from the loss gradient's ability adaptively weight outputs on the per example and per class basis.
3. We demonstrate that the generalization benefits of both successful methods no longer hold when the micro-batch size closer to the actual batch size. We too subsequently show that in this regime the average micro-batch gradient norm behavior of both previously success- ful methods differs significantly from the small-batch SGD case.
4. We highlight a high-level issue in modern empirical deep learning research: Experimental results that hold on CIFAR10 10 do not necessarily carry over to other datasets. In particular, we focus on a technique called gradient grafting (Agarwal al., 2020), which has been shown to improve generalization for adaptive gradient methods. By looking its behavior for normal SGD and GD, we show that gradient grafting recovers small-batch SGD gen- eralization's 's performance CIFAR 10 but fails in CIFARION 100, arguing that research in this line should prioritize experiments on a larger and diverse range of benchmark datasets.
2 PRIOR WORK AND PRELIMINARIES
In neural network training, the choice of batch size (and learning rate) heavily influence generaliza- tion. particular, researchers have found that opting for small batch sizes (and large learning rates) improve a network's 's ability to generalize (Keskar et al., 2017; Masters and Luschi, 2018; Goyal et al., 2017; Mandt et al., 2017; He et al., 2019; Li et al., 2019) Yet, explanations for this phe- nomenon have long been debated. While some researchers have attributed the success of small-batch SGD to gradient noise introduced by stochasticity and hanibiathhnn (Wu et al., 2020: Jastrzebski et al., 2018; Zhu et al., 2019; Mori and Ueda, 2020; Cheng et al., 2020; Simsekli et al., 2019; Xie et al., 2021), others posit that small-batch SGD finds "flat minima" with low non-uniformity, which in turn boosts generalization (Keskar et al., 2017; Wu al., 2018; Simsekli et al., 2019). Mean-
2

---

Published as a conference paper at ICLR 2023
while, some works credit the implicit regularization of quantities such loss gradient norm, the Jacobian norm (i.e., the network hyppt-to-weight gradient norm), and the Fisher Information Ma- trix (Geiping et al., 2022; Barrett and Dherin, 2020; Smith et al., 2021; Lee et al., 2022; Jastrzebski et al., 2020).
Recent works have shown that one recover SGD generalization performance by training modified loss function that regularizes large loss gradients (Geiping et al., 2022; Barrett and Dherin, 2020; Smith et al., 2021). While Smith et al. (2021) and Barrett and Dherin (2020) expect that training on this modified loss function with large micro-batch sizes will be unable to recover SGD generalization performance, we empirically verify this is the case.
To our knowledge, we are the first to introduce the mmicro-batch" terminology to denote component disjoint sub-batches used in accumulated mini-batch SGD, This choice was made avoid overload- ing the term "mini-batch" and thus clarify the work done by Smith et al. (2021) and Barrett and Dherin (2020). Note that here and for the rest of this paper, we use Large-Batch SGD as an approx- imation for full-batch GD due to the computational constraints of using the full training set on each update. We emphasize that throughout this paper. micro-batches are not meant as any sort of "al- ternative" to mini-batches, as they are purely implementation feature of gradient accumulation- based large-batch SGD, We additionally leverage the work done by Agarwal et al. (2020), who pro- pose the idea of grafting meta-optimization algorithm, though in the paper their focus mostly rests on grafting adaptive optimization algorithms together, not plain mini-batch SGD.
As a whole, our paper is situated as comparative analysis of multiple proposed regularization mechanisms (Geiping et al., 2022; Barrett and Dherin, 2020; Smith et al., 2021; Lee et al., 2022; Jastrzebski al., 2020) in side-by-side empirical context, with additional ablations over how minor design choices may affect the efficacy of these proposed methods to close the generalization gap. We now discuss various implicit and explicit regularization mechanisms in more depth.
Setup and Notation We primarily consider the case of softmax classifier f R2 + RC (where C' is the number of classes) parameterized by some deep neural network with parameters 0. We let f(x, y; 0) denote the standard cross entropy loss for example x and label and let LB(0) [18] Σ (x.g)) {(x, y; 0) denote the average loss over some batch B. Note that throughout this paper, the terms "batch" and "mini-batch" are used interchangeably to refer to B.
2.1 AVERAGE MICRO-BATCH GRADIENT NORM REGULARIZATION
As proposed by Smith et al. (2021), we attempt to understand the generalization behavior of mini- batch SGD by how it implicitly regularizes the norm of the micro-batch gradient, || ar (0)] for some micro-batch M C B. In large- SGD, we accomplish this through gradiens accumulation (i.e. accumulating the gradients of many small-batches to generate the large-batch update), and thus can add an explicit regularizer (described in Geiping et al. (2022)) that penalizes the average micro- batch norm. Formally, for some large-batch B and component disjoint micro-batches M C B, let [M ((.))MM Vol(x, y; 0). The new loss function is:
VeL (0)
JM B
(B( A
(o)(1
MEB
While this quantity can be approximated through finite differences, it can also be optimized directly by backpropagation using modern deep learning packages, as we do in this paper.
Note that by definition, we can decompose the regularizer term into the product of the Jacobian of the network and the gradient of the loss with respect network output. Formally, for some network f with parameters, if we let z = f(x; 0) E RC be the model output for some input x and denote its corresponding label as y, then:
Vol(x, y, = (Voz)
Where Vaz E RPXC is the Jacobian of the network and the second term is the loss-output gradient. We explicitly show this equivalence for the comparison made in section 2.3.
3

---

Published as a conference paper at ICLR 2023
2.2 AVERAGE MICRO-BATCH FISHER TRACE REGULARIZATION
One noticeable artifact of Equation 1 is its implicit reliance on the true labels to calculate the regularizer penalty. Jastrzebski al. (2020) shows that
derive similar quantity in the mini-batch SGD setting by penalizing the trace of the Fisher Information Matrix F, which is given by Tr(F) Ex-xXi- (x--xxx-=p((l>)) ( y; 0)] where po x) the predictive distribution of the model at the current iteration and X' is the data distribution. We thus extend their work to the accumulated large-batch regime and penalize an approximation of the average Fisher trace over
micro-batchen we (0)
M ExEM.S<po(ulxx {(x, 9: 0), then our penalized loss is
M B
(B(
Σ ( MEB
(3)
The only difference between the expressions in Equation and Equation 3 is that the latter now uses labels sampled from the predictive distribution, rather than true labels, to calculate the regularizer term. As with Equation 1, directly backpropagate using this term in our loss equation.
Like, in equation 2, we can decompose the regularizer term as:
((x, ÿ: 0) = (Voz) ;;
(4)
Where the second term is another loss-output gradient.
Jastrzebski al. (2020) observes that models with poor generalization typically show a large spike in the Fisher Trace during the early phases of training, which they coin as Catastrophic Fisher Explosion. Figure we show that this behavior also occurs when looking at the average Micro- Batch gradient norm.
2.3 JACOBIAN REGULARIZATION
Given the decompositions shown in equations and 4, unclear in either case whether the Jaco- bian term is the sole source of possible generalization benefit, if the loss-output gradient is also needed. To disentangle this effect, we borrow from Lee et al. (2022) and use the average and unit Jacobian regularization losses:
M
CB(0)
A
[Mwg((M))
(avg(M)
Σ
(Voz)
(5)
B
IM
MCB
(x.jjM
(s(0) +
M
(M))) Junin (M)
Σ (Vez)(u). (x.))EM
(6)
B
M
MCB
where E RCC is randomly sampled from the unit hypersphere (i.e. ||u||| 1), and u is sam- pled once per micro-batch. In words, the average Jacobian case penalizes the Jacobian with equal weighting on every class and every example, while the unil Jacobian case penalizes the Jacobian with different but random weighting an unbiased estimator of the Frobenius norm of the Jacobian || ez which is an upper bound on its spectral norm V oz (see Lee et al. (2022) for more detailed theoretical analysis).
each class and example. Note that the unit Jacobian penalty
EXPLICIT REGULARIZATION AND GRADIENT NORM TRAJECTORY
These aforementioned explicit regularization mechanisms have previously been investigated in lim- ited empirical settings. the best of our knowledge, Jastrzebski al. (2020) is the only work that has directly compared some of these regularization mechanisms, but only did so in the context of improving small-batch performance. Like our work, Geiping et al. (2022) is centered on the small- to-large batch generalization gap, but they do not focus solety on the explicit regularization they pro- pose and do not include any analysis of the micro-batee batch gradient norm behavior during training. In this work, we investigate (i) how these regularizers effect generalization for array of benchmarks and (ii) how such performance may correlate with the evolution of the micro-batel gradient norm during training.
4

---

Published as a conference paper at ICLR 2023
Table 1: ResNet18 Test Performance for Regularizer Penalties. Values shown are average test ac- curacies across two to three different initializations per experiment, with corresponding confidence intervals.
Experiment
CIFARIO
CIFARI00
Imageee
SVHN
SB SGD
92.33 (=0.10)
71.01 (#0.27)
39.64 (#0.18)
93.69 (#0.12)
LB SGD
90.00 (=0.11)
66.45 (#0.29)
27.71 (=0.09)
90.37 (#0.33)
GN
91.98 (#:0.33
70.22 (+:0.77)
37.78 (+:0.7)
92.77 (=0.01)
FT
91.79 (+0.05)
71.19 (+0.16)
40.25 (+0.02)
93.72 (+0.16)
AJ
90.41 (+0.01)
65.95 (+-0333)
22.86 (+0.95)
91.76 (+0.11)
UJ
90.46 (=0.20)
66.41 (#0.06)
26.07 (#0.54)
92.08 (=0.01)
3.1 EXPERIMENTAL SETUP
We first focus our experiments on the case of using a ResNet-18 (He et al., 2015), with standard ini- tialization and batch normalization, the CIFARIO, CIFARI00, Tiny-ImageNet, and SVHN im- age classification benchmarks (Krizhevsky, 2009; Le and Yang. 2015; Netzer et al., 2011). l). Addi- tional experiments
small-batch (B = 128) and large-batch (B = 5120) SGD baselines, we also train the networks in the large-batol regime using (i) average Micro-batch Gradient Norm Regularization (GN); (ii) average Micro-batch Fisher Trace Regularization (FT); and (iii) average and unit Micro-batch Jacobian Regularizations (AJ and UJ). Note that for all the regularized experiments, we use component micro-batch size equal to the small-batch size (i.e. 128). In order to compare the best possible performance within each experimental regime, we separately tune the optimal learning rate 1) and optimal regularization parameter À independently for each regime. Additional experimental details can be found in Ap- pendix A.5.
different architectures are detailed in Appendix A.1. Besides
3.2 RESULTS
(i) Average micro-batch Gradient Norm and average Fisher trace Regularization recover SGD generalization For CIFAR 100, Tiny-ImageNet, and SVHN we find that we can fully recover small-batch SGD generalization performance by penalizing the average micro-batch Fisher trace and nearly recover performance by penalizing the average micro-batch gradient norm (with an optimally tuned regularization parameter X, see Figure and Table 1). In CIFARIO, neither penalizing the gradient norm nor the Fisher trace completely recovers small-batch SGD performance, but rather come within 2s 0.3% and se 0.4% (respectively) the small-batch SGD performance and significantly improves over large-bate batgh SGD.
We additionally find that using the micro-batch gradient norm leads to slightly faster per-iteration convergence but less stable training noted the tendency for the model to exhibit random drops in performance), while using the Fisher trace leads to slightly slower per-iteration convergence but much more stable training (see Figure 1). This behavior may be due the Fisher trace's ability to more reliably mimic the small-batch SGD micro-batch gradient norm behavior throughout training, whereas penalizing the gradient norm effectively curbs the initial explosion but collapses to much smaller norm values as training progresses.
(ii) Average and Unit Jacobian regularizations do recover SGD generalization Observe in Table that we are unable to match SGD generalization performance with either Jacobian regu- larization. In Section 2 we showed that each regularization method can be viewed as penalizing the norm of the Jacobian matrix- gradient norm and Fisher trace regularizers use some form of loss-output gradient, which is data- dependent and has constraint on the weighting of each class, while both Jacobian regularizers use intepeedddend and comparatively simpler vectors. Given the noticeable difference in generaliza- tion performance between the regularization methods that weight the Jacobian with the loss-output gradient and those that do not, we indicate that the loss-output gradient may be crucial to either ap- plying the beneficial regularization effect itself and/or stabilizing the training procedure.
product with some C-dimensional vector. Crucially, both the
5

---

Published as a conference paper at ICLR 2023
4 SHORTCOMINGS AND EXTENSIONS OF GRADIENT NORM REGULARIZATION
4.1 GENERALIZATION FAILURE LARGE MICRO-BATCH SIZES
In both successful regularization regimes, namely the average micro-batch gradient norm and aver- age fisher trace regularizers, there is an implicit hyperparameter: the size of the miero-batch used to calculate the regularization term. Note that this hyperparameter is practical artifact of modern GPU memory limits, as efficiently caleulating higher-order derivatives for large batch sizes is not feasible in standard autodifferentiation packages. Consequently, gradient accumulation (and the use of the average micro-batch regularizer, rather than taking the norm over the entire batch) must be used on most standard GPUs (more detailed hadware specificatiins can be found in appendix A.5).
This restriction, however, may actually be beneficial, as Geiping et al. (2022); Barrett and Dherin (2020); Smith al. (2021) have noted that they expect the benefits of gradient norm regularizers to break down when the micro-batch size becomes too large. To test this hypothesis, we return to the ResNet- 18 in CIFAR 100 and Imageee settings and increase the micro-batch size large as we could reasonably fit on a single GPU at MM| 2560 in both the gradient norm and Fisher trace experiments. Additionally,
run experiments using a VGGII (Simonyan and Zisserman, 2015) on CIFAR 10, interpolating the micro-batch size from the small to large regimes. In both settings, we separately tune the learning rate 1) and regularization coefficient in each experiment find the best possible generalization performance in the large micro-batee regimes.
Table 2: Test Performance for R8 with Increased Micro-Batch Size. Small-batch SGD per- formances: CIFAR100 = 71.01, Tiny-ImageNet 39.64.
Dataset
GN (IM 128)
FT (IM 128)
GN ((M 2560)
FT ([M) 2560)
CIFARIO0
70.22 (#0.27)
71.19 (t:0.66)
64.23 (#0.49)
65.44 (=:0.76)
Iny-ImmgeNNet
37.78 (=0.07)
40.25 (=0002)
31.96 (=0.56)
37.71 (=0.31)
ResNet18 CIFAR100
VGG11 CIFAR10
Avarage Micro Grozlent Nom
140
Average Micro Grodient Nom
120
50
100
40-
80
30
20
40
20
20
10a
164
Iteration
kmp (M) 25661
5m
90t (N (ovenar M) 25-7
QM1 2281
9TD FT 23001
50000
Jeveagee (M] 130
Figure 2: Average Micro-batch Gradient Norm for varying micro-batch sizes. In all experimental regimes, increasing the micro-batch size leads to worse reconstruction of the SGD average micro- batch gradient norm behavior, especially in early training.
Table Test Performance for VGGI (no batch batch-norrmalizationn) CIFARIO with Increased Micro- Batch Size
SB SGD
LB SGD
GN (IM) = 100) GN (IM) = 1000)
GN (M 2500)
78.19
73.90
76.89 (0.7)
75.19 (=0.10)
75.11 (=0.29)
Results We successfully show that such hypotheses mentioned in Geiping et al. (2022); Smith et al. (2021); Barrett and Dherin (2020) hold true: the micro-batch size approaches the mini-batch size,

---

Published as a conference paper at ICLR 2023
both regularization mechanisms lose the ability recover small-batch SGD performance (see Tables 2 and 3). Additionally, we note that using large micro-batch sizes no longer effectively mimics the average micro-batch gradient norm behavior of small-batch SGD, thus supporting our claim that matching this quantity throughout training is of key importance to recovering generalization performance (Figure 2).
4.2 SAMPLE MICRO-BATCH GRADIENT NORM REGULARIZATION
One potential practical drawback of these gradient-basee based regularization terms is the relatively high computation cost needed to calculate the second-order gradients for every component micro-batch. Instead of penalizing the average micro-batch gradient norm, we can penalize one micro-batel gra- dient norm. For some large batch B and fixed sample micro-batch S from batch B, we define the modified loss function
Cs(0) + N\Voss(0))
VGG11 on CIFAR10
Final Average batch Gradlent Norm
76.5
Final Tet Accuracy
16
14
12-
74.5
10
74.0
73.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Regularization on Strength (A)
Regularization Strength (A)
13500 OK (emple.
GiiSanss.18
isargie 1000
02) -100
Figure 3: Explicitly regularizing the sample loss gradient norm recovers SGD test accuracy
Results In Figure 3, plot the final test accuracy (left column) and the average gradient norm (right column) as a function of A. We observe that both a larger À and a smaller micro-batch size [S] boost test accuracy. Furthermore, test accuracy for sample micro-batch gradient norm regularization is close to (and sometimes better) than the final test accuracy for SGD. Just Norm regularization, generalization benefits diminish as the sample micro-batch size approaches the mini-batch size.
find that with the "optimal" À and micro-batch size |S], the final
observed with the average Micro-batee Gradient
Is MIMICKING SGD GRADIENT NORM BEHAVIOR NECESSARY FOR CENERALIZATION?
As seen in Figure 1, the trajectory of the average micro-batch gradient norm during training, and its similarity to that of small-batch SGD especially in the early stages of training, is strongly correlated with generalization performance. Furthermore, we have observed that models with poor general- ization performance tend to exhibit the characteristic "explosion" during the early phase of train- ing and quickly plummet to average micro-hatch gradient norm values much smaller than seen in small-bater SGD. That being said, it is not immediately clear whether recreating the micro-batch norm trajectory small-batch SGD necessary for ensuring good generalization performance (i.c. whether good generalization directly implies gradient norm behavior similar to SGD).
To test this hypothesis, we empirically validate an orthogonal vein of optimization methods that do not explicitly regularize the micro-baie batch gradient norm during training for their ability to close the small-to large batch generalization gap, and whether they too mimic the average micro-bate norm trajectory of mall--btth SGD.

---

Published as a conference paper at ICLR 2023
5.1 EXTERNAL AND ITERATIVE GRAFTING AND NORMALIZED GRADIENT DESCENT
Inspired by the work of Agarwal et al. (2020), we proposed to use gradient grafting in order to control the loss gradient norm behavior during training. Formally, for any two different optimization algorithms M, D, the grafted updated rule is arbitrarily:
IM (((0),
gD D(Ox)
(8) In this sense, .M controls the magnitude of the update step and D controls the direction. We first propose Iterative Grafting, wherein M(2 = v7VCar and D(0,) = VLn(Ox), where e B is sampled uniformly from the component miiro-bbtchees at every update. In words, at every update step we take the large batch gradient, normalize it, and then rescale the gradient by the norm of one of the component micro-batch gradients.
k+1 0%
1l
9D
|
Additionally, we propose External Grafting, where .M(0;) = 7L and D(0)) = (f&)) Here, use VL (Ok) to denote the gradient norm step k from separate small- batch SGD training run. We propose this experiment to make a comparison with the Iterative Graft- ing case, since here the implicit step length schedule is independent of the current run, while with Iterative grafting the schedule depends upon the current training dynamics.
Aside from grafting algorithms, which define the implicit step length schedule at every step, we also consider the situation where the step length is fixed throughout training through normalized gradient descent (NGD) (Hazan et al., 2015), wherein M(0x) 17 and D(O;) (),)))
Table 4: Test Performance for Grafting NGD Experiments
Dataset
Model
SB SGD
LB SGD
EG
IG
NGD
ResNet18
92.33
89.99
92.12
92.16
92.10
CIFARIO
VGG16 w/Batch-Norm
89.56
86.97
88.65
89.06
89.39
ResNet18
71.21
66.17
68.3
68.4
66.83
CIFARIO0
VGG16
w/Batch-Norm
64.26
55.94
59.71
63.48
58.05
CIFAR100
CIFAR10
Micro-balch Gradient Norm
140
Micro-bass batch Gradient Norm
60
120
50-
100
40
80
30
60
40
20
10
0-
10
10a
Iteration
Iteration
5B
Iterative Grafting
SGD Extemal Grafting
Figure 4: Average Micro-batch Gradient Norm Grafting Experiments for ResNet-18 (Plots smoothed for clarity). In both scenarios, irrespective of generalization performance, the grafting ex- periments do not mimic the SGD average micro-batel gradient norm behavior.
Results We find that both forms of grafting and NGD can recover the generalization performance of SGD some model-dataset combinations (see Table 4). Namely, though grafting NGD seems to work quite well in CIFAR10, no amount of hyperparameter tuning was able to recover the SGD performance for either model in CIFARIOK That being said, in the CIFARIO case we see (in Figure 4) that the grafting experiments (and NGD, not pictured) do not replicate the same average mini- batch gradient norm behavior of small-batee SGD despite sometimes replicating its performance. This thus gives solid empirical evidence that while controlling the average mini-batch gradient norm behavior through explicit regularization may aide generalization, it is not the only mechanism in which large-batch SGD can recover performance.
8

---

Published as a conference paper at ICLR 2023
5.2 WIDER IMPLICATIONS
The stark disparity in performance between the CIFARIO and CIFAR100 benchmarks are of key importance. These differences may be explained by the much larger disparity between the mid-stage average micro-batch gradient norm behavior in the CIFAR 100 case than in the CIFAR10 case (see Figure 4). This situation highlights a possible cultural issue within the deep learning community: there is a concerning trend of papers in the deep learning field that cite desired performance CIFARIO, and no harder datasets, as empirical justification for any posed theoretical results (Orvieto al., 2022; Geiping et al., 2022; Agarwal et al., 2020; Smith et al., 2021; Barrett and Dherin, 2020; Cheng et al., 2020). Given the continued advancement of state-of-the-art deep learning models, argue that it is imperative that baselines like CIFAR 100 and ImageNet adopted as the main standard for empirical verification, so that possibly non-ge non-generalizable results (as the grafting / NGD results would have been had we stopped at CIFAR10) do not fall through the cracks in the larger community (see Appendix A.3 for more information).
DISCUSSION & CONCLUSION
In this paper, we provide holistic account of how the proposed regularization mechanisms (Geip- ing et al., 2022; Barrett and Dherin, 2020; Smith al., 2021; Lee et al., 2022: Jastrzebski et al., 2020) compare to each other in performance and gradient norm trajectory. and additionally show the limitations of this analytical paradigm for explaining the root cause of generalization. Our results with regards to the relative poor performance of the Jacobian-basee -based regularizations somewhat con- flict with the results of Lee al. (2022), which shows positive results on using the unit Jacobian reg- ularization with respect to improving performance within the same batch-size regime. We attribute this difference to the fact that Lee et al. (2022) is not concerned with cases where the small-to-large batch generalization gap exists, which is main focus.
In light of this prior work, more research should be done to disentangle the exact effect that implic- itly regularizing the loss-output gradient has on generalization performance. Next, given the suc- cess of average micro- batch gradient norm and average micro-batch Fisher trace regularization (es- pecially with small niiro-batchess), future work should leverage these regularization mechanisms to investigate the possibility of ameliorating generalization, while improving time efficiency, by taking advantage of high resource, parallelizable settings. We also show that experimental findings CI- FAR10 may no longer hold in CIFAR 100, which sheds light on wider implication for the research community. Namely, we urge researchers to adapt the practice of evaluating empirical hypotheses on more widespread, complex set of benchmarks.
We acknowledge that performance in each experiment could possibly be improved by progressively finer hyperparameter tuning, though we are confident that our core results would continue to hold in such situations given the extensive hyperparameter searches performed for each experiment. As whole, the present research helps to shed light on the mechanisms behind SGD's generalization properties through implicit regularization, and offers robust fixes to the generalization issue at high batch-sizes.
REPRODUCIBLITY STATEMENT
The source code for reproducing the work presented here, including all hyperparameters and random seeds, is available at ht https: : / experimental details are available in Appendix A.5.
//gtttb. com/ acmi- imp- Additional
7 ACKNOWLEDGEMENTS
We thank Jeremy Cohen for their useful insights and helpful discussions throughout the course of this project. TM is supported in part by CMU's Software Engineering Institute (SEI) via Department of Defense under contract 18800-11-D-0000 SG acknowledges JP Morgan AI PhD Fellowship and Amazon Graduate fellowships for their support. ZL acknowledges Amazon AI, Salesforce Research, Facebook, UPMC, Abridge, the PwC Center, the Block Center, the Center for Machine

---

Published as a conference paper at ICLR 2023
Learning and Health, and the CMU Software Engineering Institute (SEI) via Department of Defense contract 1-8702215---0002 for their generous support of ACMI Lab's ' research.
REFERENCES
Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling adaptive gradient methods from learning rates. CoRR, abs/2002.11803. 2020. URL https:// //rxiv. org/abs 2002. 11 1803.
David G. T. Barrett and Benoit Dherin. Implicit gradient regularization. CoRR, abs/2009.1116. 2020. URL https: / //ariiv.aiiiv////a org/abs/ /2000 11112.
Xiang Cheng, Dong Yin, Peter L. Bartlett, and Michael I. Jordan. Stochastic gradient and langevin processes. In International Conference on Machine Learning (ICML), 2020.
Jonas Geiping, Micah Goldblum, Phillip E. Pope, Michael Moeller, and Tom Goldstein. Stochastic training not necessary for generalization. In International Conference on Learning Represen- fations (ICLR), 2022.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training im- agenet in hour. CoRR, abs/1 706.02677, 2017. URL http::/arxiv...r org org/bb/////// 02677.
Elad Hazan, Kfir Y. Levy, and Shai Shalev-Shwariz. Beyond convexity: Stochastic quasi-convex optimization, 2015. URL httpp::/arxiv..rtt orw. abs /107 02030.
Fengxiang He, Tongliang Liu, and Dacheng Tao. Control batch size and learning rate to generalize well: Theoretical and empirical evidence. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015, URL https::////// /tgp//.tiii.//.aaaa.../oo/oo.aat abs 1512.111111 03385.
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Ben- gio, and Amos Storkey. Three factors influencing minima in sgd. In International Conference on Artificial Neural Networks (ICANN), 2018.
Stanislaw Jastrzebski, Devansh Arpit, Oliver Astrand, Giancarlo Kerg, Huan Wang, Caiming Xiong, Richard Socher, Kyunghyun Cho, and Krzysztof J. Geras. Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. International Conference on Machine Learning (ICML), 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In In- ternational Conference on Learning Representations (ICLR), 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015.
Sungyoon Lee, Jinseong Park, and Jaewook Lee. Implicit jacobian regularization weighted with impurity of probability output, 2022. URL httttp:://openevvee..o http:/ewwww.e...reeeeeeeeeeeeeee wtr./oormm?mm ?id= Rg3xUXjzWMa
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint ar0iv:/608. /60.:039888. 2016.
10

---

Published as a conference paper at ICLR 2023
Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent approxi- mate bayesian inference. 2017. doi: 1044550AARXX..774.000288. URL pettps://areee org/abs/ 1704. 04289.
Dominic Masters and Carlo Luschi. Revisiting small batch training for deep neural networks. CoRR, abs/1804.07612, 2018. URL http: / arxiv. org/ hagp/ab///te 1804. 076 1761..
Takashi Mori and Masahito Ueda. Improved generalization by noise enhancement, 2020. URL
https: //axivv...ivvvv//a///aa org/ als 2009 13094.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NewrIPS), 2011.
Antonio Orvieto, Hans Kersting, Frank Proske, Francis Bach, and Aurelien Lucchi. Anticorrelated noise injection for improved generalization, 2022. URL htpp:///axiv.. wbwgg///////////////// 02831.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015.
Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis stochastic gradient noise in deep neural networks. In International Conference Machine Learning (ICML), 2019.
Samuel L. Smith, Benoit Dherin, David G. T. Barrett, and Soham De. On the origin of implicit reg- ularization in stochastic gradient descent. In International Conference on Learning Representa- tions (ICLR), 2021.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. the importance initial- ization and momentum in deep learning. In International conference machine learning, pages 1139-1147, PMLR, 2013.
Yeming Wen, Kevin Luk, Maxime Gazeau, Guodong Zhang, Harris Chan, and Jimmy Ba. An empirical study of large-batch stochastic gradient descent with structured covariance noise. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, and Zhanxing Zhu. The multiplicative noise in stochastic gradient descent: Data-dependent regularization, continuous and discrete approxima- tion. In International Conference on Machine Learning (ICML), 2020.
Lei Wu, Chao Ma. and Weinan E. How sgd selects the global minima in wer-paramettrized learn- ing: A dynamical stability perspective. Advances Neural Information Processing Systems (NeurIPS), 2018.
Zeke Xie, Li Yuan, Zhanxing Zhu, and Masashi Sugiyama. Positive-negative momentum: Manip- ulating stochastic gradient noise to improve generalization. In International Conference on Ma- chine Learning (ICML), 2021.
Liu Yang, Jifan Zhang, Joseph Shenouda, Dimitris Papailiopoulos, Kangwook Lee, and Robert Nowak. A better way to decay: Proximal gradient training algorithms for neural nets. arXiv preprint arXiv:2210.03065 2022.
Zhanxing Zhu, Jingfeng WU, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In Inter- national Conference on Machine Learning (ICML), 2019.
11

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1
ADDITIONAL. REGULARIIZATINN EXPERIMENTS
Aside from the main results using ResNet-18, additionally the regularization experiments with a VGG11 (Simonyan and Zisserman, 2015) without batch normalization on CIFARIO 10. Results shown below:
Table 5: VGGII (no batch-normalization) Test Performance for Regularizer Penalties
Dataset
SB SGD
LB SGD
GN
FT
AJ
UJ
CIFARIO 78.19
73.90
77.62
79.10
74.09
N/A
Consistent with our earlier observations (see Section 3), we find that average micro-batch gradi- ent norm and average Fisher trace regularization nearly recover SGD generalization performance, whereas average Jacobian regularization does not.
A.2
SAMPLE MICRO-BATES GRADIENT NORM REGULARIZATION (CONTINUED)
Resnet18 on CIFAR100
VGG11 on CIFAR10
80.0
77.5
70
75.0
68
Test Accuracy
Test Accuracy
72.5
66
70.0
64
67.5
65.0
62.5
60.0
58
10000
20000 30000
40000
50000
5000
10000 Iteration
15000
20000
Iteration
SBISGD
LB SGD + GN (Average)
LB SGD
LB SGD (Average, [M] = 100)
LB SGD
L8 SGD GN (Sample)
SB SED
LB SGD
GN (Sam pie. [S] 100)
Figure 5: Explicitly regularizing the average loss gradient norm over sample recovers SGD test accuracy
We see that switching from using the average micro-bated gradient norm to using a single sample micro-batch gradient norm as a regularizer does not impact final generalization performance. How- ever, we do see in the CIFAR 100 case that while using the sample-based regularizer is faster in terms of per iteration wall-clock time, the average-based regularizer converges to an optimal performance in considerably fewer gradient update steps (Figure 5).
A.3 LIMITATIONS OF ANTICORRELATED PERTURBED GRADIENT DESCENT
Orvieto et al. (2022) proposes method for improving generalization by injecting spherical Gaussian noise (with variance o² as a hyperparameter) at each gradient update step that is anticorrelated between concurrent time steps, which they term Anti-PGD. They empirically show that CIFAR 10, training a ResNet18 in the large- regime with Anti-PGD and then shutting off the noise (to allow for convergence) allows them to beat small-batch batch SGD generalization performance.
However, when we extended their methodology to the CIFAR100 regime (while removing possible confounding factors such as momentum), hyperparameter combination in the large-batch regime
12

---

Published as a conference paper at ICLR 2023
was able to recover SGD generalization performance. This thus represents just one example of the possible endemic issue described in 5.2. Hyperparameter combinations and final test accuracy are show below.
Table 6: ResNet18 w/Anti-PGD CIFARI00 (SB SGD Test Accuracy = 71.21), No hyperparam- eter combination comes close to recovering SGD performance.
Learning Rate (17)
o²
Test Accuracy
0.5
0.01
67.54
0.5
0,001
65,44
0.1
0.01
64.55
0.1
0,001
64.90
0.05
0,01
62.52
0.05
0.001
62.78
A.4 EXPLICIT REGULARIZATION WITH SOTA OPTIMIZATION TOOLS
In the present work, concerned with understanding the generalization performance of explicit regularization mechanisms in SGD with no other modifications. However, in practice many different heuristics used to improve SGD, including momentum (Sutskever al., 2013), weight decay (Yang et al., 2022), and learning rate scheduling (Loshchilov and Hutter, 2016). To verify that the behavior throughout the paper holds in more standard conditions, return the set-up of training a ResNet18 8 on CIFAR R1FAR100, this time with momentum, weight decay, and cosine annealed learning rate. Here, we focus on the two successful regularization algorithms (i.e. Gradient Norm and Fisher Trace regularization). In Table 7, we see that the behavior shown in the main paper still holds: with large gap between small-batch and large-batch SGD, Fisher Trace regularization able to recover small-batch performance, while Gradient Norm regularization is able to beat large- batch but not fully recover small-batch performance. Specific values for the added hyperparameters are detailed in Appendix A.5.
Table 7: R Test Accuracy with momentum, weight decay, and cosine annealed learning rate. The relative performance of regularized training is maintained when adding additional optimization tools.
SB SGD
LB SGD
LB + GN
LB + FT
72.59
67.47
70.58
72.53
A.5 EXPERIMENTAL SETUP DETAILS
All experiments for the present paper performed using the Pytorch deep learn- ing API, and source code can be found here: htt..aa...cco///oom22ma.... imp-regularizer..
Values for our hyperparameters in our main experiments are detailed below:
Table 8: Learning rate (1)) used in main experiments
Model/Dataset
SB SGD
LB SGD
LB GN
LB FT
LB + AJ
LB UJ
17 0.1
1) 0.1 0.1.1
0.1
7= 0.1
001.1
17 1= 0.1
7 0.1
BesNet-18/CIFAA 100
= 0.1
n =055
= 0.1
1= 0.1 001.1
1 7=0.1
= 0.1
ResNet-18/Tiny-maaee
17 0.1
7 =.55
" 7= =01
7 =.1.
17 7= 00.5
" 0.1
ResNet-I8/SVHN
0.1
" 7.1.1
7= 0.1
=0.1
== =0.11 -0.1
7= 0.1
= 0.1
VGG11/CIFARI0
17 -..55
7 -0.01
7 -0.01
== 0.01
= 0.01
N/A
13

---

Published as a conference paper at ICLR 2023
Table Regularization strength (A) used main experiments
Model/Dataset
LB
LB+
LB AJ
LB+ UJ
0.01
==0.0
0.001
X 0.001
ResNet-18/CIIFA
À 0.01
À 0.01
5 x 10
À 0.001
ResNet-18/Tiy--maeeee
À = 0.01
À = =00011
1 10
À = 0.001
ResNet-18SSVHN
0.01
=00.1
0.0001
À 0.001
VGGI1/CIFARIO
À 0.5
À 0.5
À 2 X 10
N/A
Table Hyperparameters for large micro-batch experiments
Model
Microbatch
Learning Rate (1))
Regularization
Experiment
Dataset
Size
Strength (A)
ResNet-18/CIFAR100
LB + GN
2560
0,5
0,0025
ResNet-18/CIFARIO
LB FT
2560
0.1
0.01
LB + GN 2560
0.5
0.1
ResNee-18/Tnn--maggee
LB FT 2560
0.1
0.1
VGGI1/CIFARI0
LB GN
1000
0,01
0.25
VGGI1/CIFARI0
LB + FT
2500
0.01
0.25
ResNet-18 For all ResNet-18 experiments, use the standard He initialization (He et al., 2015), and the default Pytorch batch normalization initialization. Additionally, we use the standard data augmentations for CIFARIO and CIFARIOM 100; that is, random cropping, horizontal flipping, and whitening. For SVHN, we performed only whitening on the dataset. For Tiny-ImageNet, no data augmentations were made aside from rescaling the input images (which are 64 x 64) be 32 32. Additionally, for Tiny- Inn--mmgeNee the sample regularization penalties were used rather than the nor- mal average regularization penalties given compute constraints (see Section 4.2 for the documented similarities between the sample and average regularizations).
All experiments were run for 50000 update iterations. In this case, all models are trained well past the point of reaching 100% training accuracy. No weight decay or momentum was used in any of the experiments. We use large-batch size of 5120 for all experiments, and thus have 40 micro-batchd batches of 128 examples each for the regularization experiments. We calculate the penalty term at every update step, which is different from the procedure in Jastrzebski et (2020), which recalculates the penalty term only every 10 update steps. For the external grafting experiments, norm data from separate run of small-batch SGD with batch size equal to the same micro-bart batch size used for iterative grafting (i.e. 128). All experiments
use the gradient
single RTX A6000 NVidia GPU.
For the experiments with added optimization tools in Appendix A.4, we take inspiration from Jas- trzebski al. (2020) and Geiping et al. (2022) and use momentum = 0.9, weight decay 1 x 10-5 and a cosine annealing schedule that anneals the initial learning rate to every 300 epochs. This set-up is used for all experiments in this section.
VGG16 The set-up for the VGG16 experiments are identical to the ResNet- 18 experiments, in- cluding the usage of batch normalization within the architecture.
VGG11 without batch-nowe nomichhimmmiiitiionn For all VGG-11 experiments, train the network with fixed learning rate (and no momentum) until we reach 99% train accuracy. Note that we do not use any form of data augmentations. We use small batch size of 100 and a large batch size of 5000.
14

---

Published as a conference paper at ICLR 2023
Table 11: Hyperparameters for sample micro-batch experiments
/
Microbatch
Learning Rate (1)
Regularization
Experiment
Dataset
Size
Strength (A)
VGGII/CIFARIO
SB SGD
N/A
0.01
N/A
VGGI1/CIFAR10
LB + FT
50
0.01
0.25
VGGI1/CIFAR1O
LB + FT
100
0.01
0.5
VGG11/CIFAR10
LB + FT
1000
0.01
0.5
VGGII/CIFARIO
LB + FT
2500
0.01
0.5
Table 12: Hyperparameters for Grafting Experiments
Model/Dataset
SB SGD
LB SGD
Iterative Grafting
External Grafting
NGD
1 = 0.1
17 7= 0.1
1) -= 0.1
r 0.1
n 0.2626
100
0.1
17 0.5
-0.1
0.1
0.3951
VGG-16/CIFAR10
7 == 0.05
=-1.1 0.1
7= 0.05
=00.5
7 0.05
n 0.2388
VGG-16/CIFARI00
= 0.1
0.1
== 0.1
0.1.1
= 0.1
0.4322
15


Reviews:
[{"review_id": "lfhYZ4HqY54", "reviewer": "Reviewer_fWma", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the implicit bias of SGD by investigating which kind of explicit regularization can help large batch SGD/GD match the performance of (small batch) SGD. Several new insights are drawn from a systematic set of experiments. \n\nFirstly, the effect of several explicit regularizer are studied, including (1) micro-batch gradient norm regularization, (2) micro-batch fisher trace regularization, (3) micro-batch average Jacobian regularization and (4) micro-batch random and unit Jacobian regularization. It is shown that the first two regularizers are more effective than the last two. \n\nSecondly, the work turns to looks at some extreme use cases of micro-batch gradient norm regularization: (1) when the micro batch size approaches the scale of the batch size of \"large batch SGD\", the regularizer no longer works well; (2) when the micro-batch gradient norm regularization is approximated by a sample micro-batch gradient norm regularization, the latter regularizer works well when he regularization strength is property turned. \n\nFinally, the paper considered a very interesting set of experiments that involve gradient grafting, which mixes the direction and magnitude from two separate (or related) optimizers. It is shown that, for some CIFAR-10 settings, gradient grafting can recover the performance of SGD, indicating that perhaps the direction of large batch SGD update is good enough but its magnitude could have been improved. Yet, this conclusion does not carry over to harder settings, where improving the magnitude of large batch SGD update might not be sufficient for it to generalize well. \n\nBased on the last result, an inconsistent performance of one algorithm in easy/hard deep learning application, the paper discusses some limitations of current scientific study of deep learning, where most experimental evidence is from relatively easy tasks and might not migrate to harder tasks.  ", "summary_of_the_review": "Please see above. At this point I would like to vote for a weak acceptance. I will consider increasing the score based on the authors reply to my questions. I might also consider decreasing the score if I or other reviewers find a major vulnerability in the experimental design that cannot be satisfactorily addressed. ", "strength_and_weaknesses": "# Strength\n+ The main contributions are summarized in the Summary section. Some other strength of this work includes:\n+ The paper is overall very well organized. I have a good time readying most of the paper (except for a few places please see below). \n+ As a scientific paper, more experiments are always welcome. With that being said, I think the conclusions presented in this work are well backed up by its experimental design. At this point I did not find a major shortcoming in terms of experimental design. \n+ I really like the design of Section 5 which disentangles the effect of the magnitude and direction of the update. Such a perspective is novel to my knowledge for understanding SGD (despite it has been studied from a more theory work), and I think it may inspire a deeper understanding of SGD. \n\n\n# Weakness\n- A large portion of the work focuses on verifying which type of explicit regularization for large batch SGD/GD can recover the performance of small batch SGD. I would like to see some discussions on why finding the right explicit regularization (that corresponds to the implicit regularization of SGD) is important or useful. \n- Note that the explicit regularization term introduces several new hyperparameters. For example the penalty strength $\\\\lambda$. I would like to see some discussion on how hard it is to tune the penalty strength $\\\\lambda$. Partly due to the new hyperparameters, I somehow feel the finding of this work might not be very useful for practitioners. \n- I find Section 3.2, paragraph (ii) is very hard to interpret. Would you mind elaborate a bit? Perhaps written down some math could help. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g6PRStUZrs", "reviewer": "Reviewer_i18E", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper does a thorough simulations to clarify and verify the existing hypothesis about why small batch SGD outperform its large batch counterpart. They derive many interesting results. First, they show that adding regularizer based on gradient norm or the Fisher Information Matrix trace can recover small-batch SGD generalization, while Jacobian-based regularizations fail. On the other hand, they show that the gradient norm, in some case, is not a good indicator of generalization, i.e., there exists some algorithms achieving good generalization but with large gradient norm. Last, they show that some tricks work on small dataset like CIFAR10 might fail on large dataset like CIFAR100, Anti-PGD, for example.", "summary_of_the_review": "Please see the strength and weaknesses part.", "strength_and_weaknesses": "## Strength\n- Though this paper is purely empirical, it provides many important intuitions for future work. For example, the failure of the Jacobian-based regularizations shows that this might not the reason for the good generalization of small-batch SGD. On the other hand, the counterexample in Figure 4 shows that we should seek other explanations beyond gradient norm regularization. The last observation that some tricks might fail on large datasets calls for verifying all the previous tricks only verified on small datasets and rethinking the importance of dataset structure. \n- The paper is well structured. I enjoy the process of reading this paper.  \n\n\n## Weaknesses\n- The performance reported in the paper is not the SOTA. I suspect the reason is that they do not use weight decay and momentum, as mentioned in the appendix. I can understand that weight decay or momentum can be regarded as some kind of regularization. Moreover, I did not find the stepsize regime. Do you use constant stepsize? In the reality, we always add weight and momentum, together with stepsize decay (or cosine learning rate) to improve the generalization. I will be very happy to increase my score if you can provide simulations showing that gradient norm regularization can also recover the generalization after adding all these common tricks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Gu6PN6eS30", "reviewer": "Reviewer_rQEx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers micro-batches, which are disjoint smaller subsets of mini-batch. Empirical evidence is provided to suggest that, although a large mini-batch does not lead to generalization as good as a small mini-batch, the introduction of micro-batches together with a large mini-batch helps recover the generalization corresponding to a small mini-batch.", "summary_of_the_review": "Overall, I feel this paper definitely proposed an interesting idea. However, I hope to understand better what is the gain, as well as to see more evidence of its effectiveness (if not theoretically, at least empirically, and more comprehensively).", "strength_and_weaknesses": "Strength: using micro-batch as an estimator of the gradient norm and a regularizer is a very interesting idea. It seems to have a lot of potential.\n\nWeakness: \n\n(1) very little theoretical justification is provided, but the empirical results are not comprehensive enough to support the claims. \n\n(2) if I didn’t misunderstand, one central goal of this paper is to suggest that small-batch SGD has an implicit bias that regularizes gradient norm. I’d like to better understand why this wasn’t investigated directly, but rather via an indirect route based on the extra complication of micro-batch. I’m relatively convinced that micro-batch has this implicit bias, but curious about why this serves a demonstration of the small-batch SGD story.\n\n(3) a traditional belief is that smaller batch size leads to behaviors similar to those under larger learning rate. The implicit bias of large learning rate, in terms of regularizing the gradient, for example, has been studied already, not only empirically but also theoretically. In addition, the implicit bias of small batch size has also been studied, see e.g., [Smith et al. ICLR 2021], which was already cited. Therefore, it is not completely clear to me what’s new in terms of SGD’s implicit bias. The micro-batch part is still interesting though.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A8kTnzHP42", "reviewer": "Reviewer_QhHW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides an empirical study comparing a range of proposed mechanisms by which small batch SGD enhances generalization. \n\nTo achieve this, the authors split large mini-batches into a set of \"micro-batches\". They then evaluate a range of proposed implicit regularization terms on each micro-batch before summing across micro-batches to form a single large batch update. They show that applying the gradient norm/Fisher trace regularizers proposed by Smith et al. [2021] and Jastrzebski et al. [2020] to these micro-batches consistently closes the generalization gap between small and large batch training, while Jacobian regularization does not. Additionally, as predicted by prior work, they confirm that the benefits of gradient norm regularization diminish as the micro-batch size rises.", "summary_of_the_review": "The authors present some useful experiments clarifying which proposed mechanisms for the implicit regularization of small mini-batches are most effective in practice at closing the generalization gap between small and large batch training.\n\nI am currently recommending marginal accept since I think this is a helpful contribution. The experiments are reasonably thorough but no large scale datasets are included, and the novelty/originality of the submission is not very high.", "strength_and_weaknesses": "Strengths:\n\n1) The micro-batch concept enables the authors to directly minimize the modified loss proposed in Smith et al. while increasing the batch size. This is also practical, since large batch training is usually achieved by parallelizing across multiple devices.\n2) The authors provide experiments across a range of models/datasets, which consistently show that gradient norm and Fisher trace regularization closes the gap between small and large batch training, thus supporting the claim that small batch SGD implicitly regularizes these quantities.\n3) The authors confirm empirically that these regularizers perform poorly when the micro-batch size is too large. \n\nWeaknesses:\n1) The novelty of the work is quite limited, however I think this is fine for an empirical evaluation.\n2) It would have been nice to include an ImageNet experiment.\n3) I didn't understand how the grafting experiments connected to the main theme of the paper?\n4) A longer background section or appendix describing the relevant prior work in detail would help readers less familiar with the topic.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster