Hyperparameter Tuning and Implicit Regularization in Minibatch SGD
Authors: Samuel L Smith
Erich Elsen
Soham De
Venue:  (2020)
Keywords: SGD
momentum
batch size
learning rate
noise
temperature
implicit regularization
optimization
generalization

Abstract: This paper makes two contributions towards understanding how the hyperparameters of stochastic gradient descent affect the final training loss and test accuracy of neural networks. First, we argue that stochastic gradient descent exhibits two regimes with different behaviours; a noise dominated regime which typically arises for small or moderate batch sizes, and a curvature dominated regime which typically arises when the batch size is large. In the noise dominated regime, the optimal learning rate increases as the batch size rises, and the training loss and test accuracy are independent of batch size under a constant epoch budget. In the curvature dominated regime, the optimal learning rate is independent of batch size, and the training loss and test accuracy degrade as the batch size rises. We support these claims with experiments on a range of architectures including ResNets, LSTMs and autoencoders. We always perform a grid search over learning rates at all batch sizes. Second, we demonstrate that small or moderately large batch sizes continue to outperform very large batches on the test set, even when both models are trained for the same number of steps and reach similar training losses. Furthermore, when training Wide-ResNets on CIFAR-10 with a constant batch size of 64, the optimal learning rate to maximize the test accuracy only decays by a factor of 2 when the epoch budget is increased by a factor of 128, while the optimal learning rate to minimize the training loss decays by a factor of 16. These results confirm that the noise in stochastic gradients can introduce beneficial implicit regularization.

Under review as a conference paper at ICLR 2020
HYPERPARAMETER TUNING AND IMPLICIT REGULARIZATION IN MINIBATCH SGD
Anonymous authors Paper under double-blind review
ABSTRACT
This paper makes two contributions towards understanding how the hyperparame- ters of stochastic gradient descent affect the final training loss and test accuracy of neural networks. First, we argue that stochastic gradient descent exhibits two regimes with different behaviours; a noise dominated regime which typically arises for small or moderate batch sizes, and a curvature dominated regime which typi- cally arises when the batch size is large. In the noise dominated regime, the optimal learning rate increases the batch size rises, and the training loss and test accuracy are independent of batch size under a constant epoch budget. In the curvature dominated regime, the optimal learning rate is independent of batch size, and the training loss and test accuracy degrade as the batch size rises. We support these claims with experiments on a range of architectures including ResNets, LSTMs and autoencoders. We always perform a grid search over learning rates at all batch sizes. Second, we demonstrate that small moderately large batch sizes continue to outperform very large batches on the test set, even when both models are trained for the same number of steps and reach similar training losses. Furthermore, when training Wide-ResNets on CIFAR-I0 with constant batch size of 64, the optimal learning rate to maximize the test accuracy only decays by a factor
when the epoch budget is increased by a factor of 128, while the optimal learning rate minimize the training loss decays by factor of 16. These results confirm that the noise in stochastic gradients can introduce beneficial implicit regularization.
INTRODUCTION
Stochastic gradient descent (SGD) is the most popular optimization algorithm in deep learning, but it remains poorly understood, A number of papers propose simple scaling rules that predict how changing the learning rate and batch size will influence the final performance of popular network architectures (Hoffer et al.] 2017]] Goyal et al., 2017: Smith et al., 2017 Jastrzebski et al. (017). Some of these scaling rules are contradictory. and Shallue et al. (2018) argue that none of these simple prescriptions work reliably across multiple architectures. Some papers claim SGD with Momentum significantly outperforms SGD without Momentum (Sutskever et al. (013). but others observe little difference between both algorithms in practice (Kidambi al. 2018 Zhang al. 2019).
We hope to clarify this debate. We argue that minibatch stochastic gradient descent exhibits two regimes with different behaviours: noise dominated regime and a curvature dominated regime (Ma ct al. 2017b McCandlish et al.) 2018 Liu & Belkin 2018). The noise dominated regime typically arises for small or moderate batch sizes, while the curvature dominated regime typically arises when the batch size is large. The curvature dominated regime may also arise if the epoch budget is small or the loss is poorly conditioned (McCandlish et al. 2018). Our extensive experiments demonstrate that,
1. In the noise dominated regime, the final training loss and test accuracy are independent of batch size under constant epoch budget, and the optimal learning rate increases as the batch size rises. In the curvature dominated regime, the optimal learning rate is independent of batch size, and the training loss and test accuracy degrade with increasing batch size. The critical learning rate which separates the two regimes varies between architectures.
2. If specific assumptions are satisfied, then the optimal learning rate is proportional to batch size in the noise dominated regime. These assumptions hold for most tasks. However

---

Under review as a conference paper at ICLR 2020
observe a square root scaling rule when performing language modelling with an LSTM. This is not surprising, since consecutive gradients in a language model are not independent.
3. SGD with Momentum and learning rate warmup do not outperform vanilla SGD in the noise dominated regime, but they can outperform vanilla SGD in the curvature dominated regime.
There also an active dehate regarding the role of stochastic gradients in promoting generalization. has been suspected for a long time that stochastic gradients sometimes generalize better than full batch gradient descent (Heskes & Kappen 1993; LeCun et al., (012). This topic was revived by Keskar et al.] (2016), who showed that the test accuracy often falls if one holds the learning rate constant and increases the batch size, even if one continues training until the training loss ceases to fall. Many authors have studied this effect (Jastrzebski et al. 2017 Smith & Lel 2017 Chaudhari & Soatto 2018), but to our knowledge no paper has demonstrated a clear generalization gap between small and large batch training under constant step budget on challenging benchmark while simultaneously tuning the learning rate. This phenomenon has also been questioned by a number of authors. [Shallue] ct al. (2018) argued that one can reduce the generalization gap between small and large batch sizes if one introduces additional regularization (we note that this is consistent with the claim that stochastic gradients can enhance generalization). Zhang et al. (2019) suggested that noisy quadratic model is sufficient to describe the performance of neural networks on both the training set and the test set.
In this work, we verify that small or moderately large batch sizes substantially outperform very large batches on the test set in some cases, even when compared under constant step budget. However the batch size at which the test accuracy begins to degrade can be larger than previously thought. We find that the test accuracy of a 16-4 Wide-ResNe (Zagoruyko & Komodakis] 2016) trained on CIFAR-10 for 9725 updates falls from 94.7% at a batch size of 4096 to 92.8% at a batch size of 16384. When performing language modelling with an LSTM the Penn TreeBank dataset for 16560 updates (Zaremba et al,, 2014), the test perplexity rises from 81.7 to 92.2 when the batch size rises from 64 to 256. We observe degradation in the final training loss the batch size rises in either model.
These surprising results motivated us to study how the optimal learning rate depends on the epoch budget for fixed batch size. As expected, the optimal test accuracy is maximized for finite epoch budget, consistent with the well known phenomenon of early stopping (Prechelt 1998). Meanwhile the training loss falls monotonically classical optimization theory. More surprisingly, the learning rate that maximizes the final test accuracy decays very slowly as the epoch budget increases, while the learning rate that minimizes the training loss decays rapidly| hapidly[TTheee These results provide further evidence that the noise in stochastic gradients can enhance generalization in some cases, and they suggest novel hyper- aeee-parmetter tuning strategies that may reduce the cost of identifying the optimal learning rate and optimal epoch budget.
the epoch budget increases, consistent with
We describe the noise dominated and curvature dominated regimes of SGD with and without Mo- mentum in section 2 We focus on the analogy between SGD and stochastic differential equations (Gardiner et al., 1985; Welling & Teh 201 | et al. 2017 Li et al.. 2017), but our primary contributions are empirical and many of our conclusions can be derived from different assumptions (Ma et al. 201 7b Zhang et al. 2019). In section we provide an empirical study of the relationship between the optimal learning rate and the batch size under a constant epoch budget, which verifies the existence of the two regimes in practice. In section 4 we study the relationship between the optimal learning rate and the batch size under a constant step budget, which confirms that stochastic gradients can introduce implicit regularization enhancing the test set accuracy. Finally in section 5), we fix the batch size and consider the relationship between the optimal learning rate and the epoch budget.
2 THE TWO REGIMES OF SGD
Full batch gradient descent is in the "curvature dominated" regime, where the optimal learning rate is determined by the curvature of the loss function. The full batch gradient descent update on the 7th step is given by wi+1 = Wi de w=w. where the loss C(w)) = 1 Σx C(w, x;)) is function of the parameters and the training inputs (2,),, and denotes the learning rate. In order to minmmize the loss as quickly as possible, we will set the learning rate at the start of training as large as
'These conclusions might change if the number of updates is very small, in which case the optimal learning rate schedules on the training test and the test may agrec.
2

---

Under review as a conference paper at ICLR 2020
we can while avoiding divergences instabilities. To build our intuition for this, we approximate the loss by a strictly convex quadratic, C(w) = 2 Hw. For simplicity, we assume the minimum lies at w = O. Substituting this approximation into the parameter update, we conclude wi+1 = w. Ew]" H. In the eigenbasis of II, wi+1 = wi A). where I denotes the identity matrix and A denotes a diagonal matrix comprising the eigenvalues of H. The iterates will converge so long as the learning rate 6 < Cerst. where Cerit = 2/ Amaz is the critical learning rate above which training diverges, and Amaz is the largest Hessian eigenvalue. We call this inequality the curvature constraint, and the optimal initial learning rate with full batch gradients will be just below ferit- For clarity, although the critical learning rate will perform poorly for high curvature directions of the loss, we can introduce learning rate decay to minimize the loss along these directions later in training (Ge et al. 2019). Of course, in realistic loss landscapes this critical learning rate might also change during training.
Acceleration methods such as Heavy-Ball Momentum (referred to as Momentum from here on) (Polyak] 1964) were designed to enable faster convergence on poorly conditioned loss landscapes. Momentum works by taking an exponential moving average of previous gradients, Wi+1 = Wi Ei m' m dC
du w=w,' where m denotes the momentum coefficient. Gradients in high curvature directions, which often switch sign between updates, partially cancel out. This enables Momentum to take larger steps in low curvature directions while remaining stable high curvature directions. On quadratic losses for example, Momentum increases the critical learning rate: exit < 2(1 + mm/ //maz Goh 2017), and can minimize the training loss in fewer steps than full batch gradient descent.
In practice we do not compute full batch gradient, we estimate the gradient over minibatch (Robbins & Monro 1951; Bottou 2010). This introduces noise into our parameter updates, and this noise will play crucial role in the training dynamics in some cases. However when the batch size is large, and the number of training epochs is finite, the noise in the parameter updates is low, and so typically most of training is governed by the curvature of the loss landscape (similar to full batch gradient descent). We call this large batch training regime curvature dominated. When the batch size in the curvature dominated regime, we expect the optimal initial learning rate to be determined by the critical learning rate Ecrit, and for SGD with Momentum to outperform SGD without Momentum. On the other hand, when the batch size small, typically most of the training process is governed by the noise in the parameter updates, and we call this small batch training regime noise dominated.
In order to build a model of the training dynamics in the noise dominated regime, we must make some assumptions. Following previous work (Mandt et al. [017] [i et al. [017]] [mmth & Le 2017] Jastrzębski et al., 2017,; Simsekli et al., 2019), we assume the gradients of individual examples are independent samples from an underlying distribution, and that this distribution is not heavy tailed.
When the training set size N >>>>/ B, the batch size B >> 1, and B < N, we can apply the central limit theorem model the noise in gradient update by Gaussian noise source 8, whose covariance is inversely proportional to the batch size. We therefore approximate the SGD update by,
dC
8,
(Wi+1 wi) = -E
+
(1)
dw
To interpret this update, we introduce the temperature T
E/B to obtain,
(Wi+1 - wi) R - dC
+ VET
dw luma
Equation 2 describes the discretization of stochastic differential equation with step size c and temperature T (Gardiner et al. 1985), and we expect the dynamics of SGD to follow this underlying stochastic differential equation, so long as the learning rate e < Cerit and the assumptions above are satisfied. When equation hholds and the learning rate e < ferit, any two training runs with the same temperature and the same epoch budget should achieve similar performance on both the training set and the test set. Consequently, we usually expect the learning rate to scale linearly with the batch size in the noise dominated regime, and this was observed many empirical studies (Krizhevsky) 2014, Goyal et al., [017]] (Smith et al., 2017 Jastrzzbbb et al., 2017 McCandlish et al. 2018 8). For completeness, we derive this linear scaling rule in appendix B, and we demonstrate that the linear scaling rule can be derived without assuming that the batch size B >>>>>>>>>> 1. However the remaining assumptions above are required, and they are not always satisfied. This linear scaling rule is therefore not valid in all cases (Shallue et al. 20188] Simsekli et al. 2019). Empirically however, we have found that all batch sizes in the noise dominated regime achieve similar test accuracies and training losses under constant epoch budget, even when the optimal learning rate does not obey linear scaling.
3

---

Under review as a conference paper at ICLR 2020
Further observations on the two regimes: Many previous works have established that SGD with and without Momentum are equivalent in the small learning rate limit when m is fixed (Oir & Leen Qian 1999 Yuan et al. 2016). In this limit, the speed of convergence of SGD with Momentum 1994
is governed by the effective learning rate ceff = </(1 m), and the temperature T = feft/ / B (Mandt et al. 2017; Smith & Le, 2017). We therefore expect SGD with and without Momentum to achieve the same final training losses and test accuracies in the noise dominated regime (where the optimal learning rate smaller than éerit). Supporting this claim, Shallue et al. (2018) confirmed in a recent empirical study that SGD with and without Momentum achieve similar lar test accuracies in the small batch limit, while SGD with Momentum outperforms vanilla SGD in the large batch limit.
In recent years, a number of authors have exploited large batch training and parallel computation to
minimize the wallelock time of training deep networks (De et al. 2016)] Smith et al.) 2017
2017 [Akiba et al..
2017: You et al. 2018 McCandlish et al. 2018 [Shallue et al. 2018). Goyal et al. (2017) succeeded in training ResNet-50 to over 76% accuracy in under one hour, and since then this has fallen to just a few minutes (Mikami al.) 2018 Ying et al. 2018). Goyal al. (2017) also introduced learning rate warmup, and found that it enabled stable training with larger batch sizes; a result which we confirm in this work. This procedure has straightforward interpretation within the two regimes. If the critical learning rate increases early in training, then learning rate warmup will increase the largest stable learning rate, which in turn enables efficient training with larger minibatches.
Learning rate schedules: In the noise dominated regime, the learning rate increases as the batch size rises, and therefore we increase the batch size will eventually invalidate the assumption e Eorit and enter the curvature dominated regime. There may be a transition phase between the two regimes (Liu& & Belkin, 2018). although our experiments suggest this transition can be surprisingly sharp in practice. We note that, with the optimal learning rate schedule, many batch sizes might exhibit both the curvature dominated regime (typically early in training) and the noise dominated regime (typically towards the end of training). For example, on simple quadratic loss landscapes at any batch size, the optimal learning rate schedule for minimizing the training loss begins with an initial learning rate close to farit, followed by learning rate decay (Zhang et al. [209)]. However in practice it is not possible to identify the optimal learning rate schedule within a realistic computation budget. Practitioners prefer simple learning rate schedules, often parameterized by initial learning rate and a few sharp drops (He et al. 2016). These schedules are easy to tune and are also thought to generalize well. For these popular schedules, the optimal initial learning rate would be determined by whether most of the training process is noise dominated or curvature dominated. Furthermore, Smith & Le (0017) suggested that there may be an optimal temperature at early times that promotes generalization. If this speculation is correct, then the optimal learning rate schedule to maximize the test acccuracy would select the noise dominated regime throughout the whole of training when the batch size is small. Of course, if the loss is extremely poorly conditioned, the critical learning rate may already be optimal when the batch size is 1, although we have never seen this in practice
Implicit regularization: In the noise dominated regime, the temperature T defines the influence of gradient noise on the dynamics. In the noisy quadratic model, our goal during training is to minimize the effect this noise has on the final parameters (Polyak & Juditsky 1992 Zhang et al. 2019).
However recently many authors have argued that minibatch noise can be beneficial, helping us to select final parameters which perform well on the test set (Keskar et al. 2016 Jastrzebski et al. 2017
Chaudhari & Soatto| 2018). In this alternative perspective, there may be an "optimal temperature early in training, which drives the parameters towards regions that generalize well (Smith et al. 2017 Park et al. 2019). The noisy quadratic model predicts that increasing the batch size may increase the final training loss under a constant epoch budget, but that it should not increase the final training loss under a constant step budget. Note that this model does not make explicit predictions about the test accuracy. We note that Jin et al. (2119) argued noise enables SGD to escape saddle points in non-convex landscapes, which could enable small batch sizes to achieve lower training losses under both constant epoch and constant step budgets. The optimal temperature perspective predicts that increasing the batch size may increase the final training loss under a constant epoch budget, but it does not predict what will happen to the training loss under constant step budget. Crucially, predicts that beyond some threshold batch size (which may be very large), increasing the batch size will decrease the test accuracy under both constant epoch and constant step budget
2 We assume the epoch budget is large enough to the training data. Training may occur in the curvature dominated regime at all batch sizes when the epoch budget is very small (McCandlish ct al. 2018).
We emphasize that both perspectives agree that learning rate decay should he used at the end of training.
4

---

Under review as a conference paper at ICLR 2020
0.95
Ogkimell elleelliiie beatinng Tele
et-
Optimal test eccurate
C.
Warm up Momentum SGD
27-
Warm up Nomentum SGD
28-
0.94
treining
C.20
6.943
C.13
Warm
0.92
Momentum
SGD
64
120
256
512
1024
2048
4296
RL52
64
129
256
512
1924
2040
4036
6292
0.25
120
256
512
1724 2048
4096 3092
Batcht size
Batcc size
Banch sime
(a)
(b)
(c)
Figure 1: A 16-4 Wide-ResNet, trained with ghost batch normalization on CIFAR-10 10 for 200 epochs. We report the performance of SGD with and without Momentum, and the combination of SGD with Momentum and learning rate warmup. We perform a grid search to identify the optimal learning rate which maximizes the test accuracy, and report the mean performance of the best 12 of 15 runs. a) The test accuracy is independent of batch size when the batch size is small, but begins to fall when the batch size exceeds 512. b) Similarly, the training loss at the optimal effective learning rate is independent of batch size when the batch size is small, but rises rapidly if acceleration techniques are not used when the batch size is large. c) The optimal effective learning rate is proportional to batch size when the batch size is small, while it is independent of batch size when the batch size is large.
3 TRAINING UNDER A CONSTANT EPOCH BUDGET
In order to verify the existence of the two regimes of SGD, we study how the performance on the training and test sets, and the optimal learning rate, depend on the batch size under constant epoch budget (when using a realistic learning rate decay schedule). To explore this, we must first select a set of model architectures and datasets, and then identify single learning rate decay schedule that performs well across all of these tasks, matching the baseline performance of the schedules reported in the original papers. For clarity, in the main text only report experiments using Wide-ResNets on CIFAR-I0 ZZgoruyko & Komodakis 2016). however we provide additional experiments using ResNet-50, LSTMs and autoencoders in appendix D (Sutskever et al. 2013) Zaremba al. 2014 He et al., 2016). For each experiment in this section, we train for the same number of epochs N Nepocks reported in the original papers (e.g.. 200 epochs on CIFAR-10), 10). Our chosen schedule the following. We hold the learning rate constant for the first Nepochs/2 epochs. Then for the remainder of training, we reduce the learning rate by a factor of 2 every Nepochs/ /20 epochs. This scheme has a single syper-parameter, the initial learning rate c, and we found this schedule to reliably meet the performance reported by the authors of the original papers. In some of our experiments we also introduce learning rate warmup (Goyal et al. 2017), whereby the learning rate is linearly increased from 0 to e over the first 5 epochs of training. We illustrate these schedules in appendix A
We will evaluate the optimal test accuracy and the optimal learning rate for range of batch sizes. At each batch size, we train the model 15 times for a range of learning rates on a logarithmic grid. For each learning rate in this grid, we take the best 12 runs and evaluate the mean and standard deviation of their test accuracy. The optimal test accuracy is defined by the maximum value of this mean, and the corresponding learning rate is the optimal learning rate. This procedure ensures our results are not corrupted by outliers or failed training runs. To define error bars on the optimal learning rate, we include any learning rate whose mean accuracy was within one standard deviation of the mean accuracy of the optimal learning rate, and we always verify that both the optimal learning rate and the error bars are not at the boundary of our learning rate grid. We apply data augmentation including padding, random crops and left-right flips. The momentum coefficient m 0.9, the 1.2 regularization coefficient is 10-4, and we use ghost batch normalization with a ghost batch size of 64 (Hoffer et al. 2017).. We also report the mean final training loss at the optimal learning rate. We note that although we tune the learning rate on the test set, our goal in this paper is not to report state of the art performance. Our goal is to compare the performance different batch sizes and with different training procedures. We apply the same experimental protocol in each case (Shallue et al. 2018). We also provide the full results of learning rate sweep at two batch sizes in appendix D
In figure la, we plot the optimal test accuracy for a range of batch sizes with a 16-4 Wide-ResNet, trained with batch normalization using SGD with and without Momentum, and also with learning
5

---

Under review as a conference paper at ICLR 2020
rate warmup. All three methods have the same optimal test accuracy when the batch size is small, but both SGD with Momentum and learning rate warmup outperform SGD without Momentum when the batch size is large. The optimal test accuracy is independent of batch size when the batch size is small, but begins to fall when the batch size grows. A very similar trend is observed for the final training loss at the optimal effective learning rate in figure 1b. To understand these results, we plot the optimal effective learning rate against batch size in figure Ic (for SGD, cefs = e). Looking first at the curve for vanilla SGD, the learning rate is proportional to the batch size below B Re 512, beyond which the optimal learning rate is constant. SGD with Momentum and warmup have the same optimal effective learning rate as SGD in the small batch limit, but their optimal effective learning rates are larger when B > 512. All of these results exactly match the theoretical predictions we made in section 2
The behaviour of SGD strongly influenced by batch normalization (Bjorck et al. 2018 Santurkar
et al 2018 Sankararaman et al. 2019) Park et al. (011)). We therefore repeat this experiment without normalization in appendix [D] To ensure training is stable we introduce a simple modification to the initialization scheme, "Zerolnit", introduced in parallel submission (Anonymous) 2020) and
defined for the reader's convenience in appendix C This modification enables the training of very deep networks, and it reduces the gap in optimal test accuracy between networks trained with and without batch normalization. We observe remarkably similar trends, although the critical learning rate, beyond which the optimal learning rate of SGD is independent of batch size, is significantly smaller. We also provide similar experiments in appendix D for a range of model architectures. In all cases, we observe a transition from a small batch regime, where the learning rate increases with the batch size and SGD with Momentum does not outperform SGD, to a large batch regime, where the learning rate independent of the batch size and SGD with Momentum outperforms SGD, Under a constant epoch budget, both the training loss and the optimal test accuracy are independent of batch size in the noise dominated regime, but they begin to fall when one enters the curvature dominated regime. In most cases the optimal learning rate in the noise dominated regime was proportional to batch size, however for an LSTM trained on the Penn TreeBank dataset, the optimal learning rate was proportional to the square root of the batch size. To understand this, we note that consecutive minibatches in language model are correlated, which violates the linear scaling assumptions discussed in section 2
4 TRAINING UNDER A CONSTANT STEP BUDGET
In the previous section, we studied how the optimal learning rate depends on the batch size under a constant epoch budget. As predicted in section 2, we found that SGD transitions between two regimes with different behaviours in a range of popular architectures. However, these results can be explained from a number of different perspectives, including both the interpretation of small learning rate SGD as the discretization of stochastic differential equation (Li et al., 2017] Mandt et al.) 2017). and also a simple noisy quadratic model of the loss landscape (McCandlish et al. 2018) Zhang et al. 2019). Crucially, the results the previous section do not tell us whether minibatch noise introduces implicit regularization that selects parameters that perform better the test set, since under a constant epoch budget, when we increase the batch size we reduce the number of training steps.
In order to establish whether minibatch noise enhances the test accuracy, now evaluate how the optimal test accuracy depends on the batch size under a constant step budget. In this scheme, the number of training epochs is proportional to the batch size, which ensures that large batch sizes are able to minimize the training loss. In table 1, we report the optimal test accuracy of our 16-4 Nese-ResNet on CIFAR-10 batch sizes ranging from 1024 to 16384. For each batch size, we train for 9765 updates using SGD with Momentum. Note that this corresponds to 200 epochs when the batch size is 1024 (to ensure these experiments did not require an unreasonably large epoch budget, we intentionally selected a batch size just below the boundary of the curvature dominated regime in figure 1). Following our previous schedule, we hold the learning rate constant for 4882 updates, and then decay the learning rate by factor of 2 every 488 steps. We conclude that the optimal test accuracy initially increases with increasing batch size, but it then begins to fall. The optimal test accuracy at batch size 4096 is 94. 7%, but the optimal test accuracy at batch size 16384 is just 92.8%. We also report the final training loss, which falls continuously with batch size. Notice that this occurs despite the fact that the optimal learning rate is defined by the test set accuracy. We observed similar results when training without batch normalization using zerolnit, which are shown in table 2, and we also provide similar results the autoencoder and LST tasks in appendix F.

---

Under review as a conference paper at ICLR 2020
Table 1: The optimal test accuracy and final training loss for range of batch sizes under a constant step budget. For each batch size, we train a 16-4 Wide-ResNet with ghost batch normalization for 9765 updates, and we perform grid search to identify the optimal learning rate which maximizes the test set accuracy. We provide the average performance of the best 12 out of 15 training runs. The final training loss falls as the batch size increases, but the optimal test accuracy drops significantly for batch sizes greater than 4096. This strongly suggests that minibatch noise can enhance generalization.
Batch size
Optimal test accuracy (%)
Final training loss
Optimal effective learning rate
1024
94.6 = 0.1
0.107 = 0.001
2 (2 to 2*)
4096
94.7 + 0.1
0.025 = 0.000
24 (2 24)
16384
92.5 + 0.6
0.019 = 0.004
2 (2 to 29)
Table 2: The optimal test accuracy and final training loss for a range of batch sizes under a constant step budget. For each batch size, we train 16-4 WiieeRReNett without batch normalization using "zerolnit" (defined in appendix C for 1 156,250 updates. We perform a grid search to identify the optimal learning rate which maximizes the test accuracy, and we provide the average performance of the best 12 out of 15 training runs. The final test accuracy falls for very large batches.
Batch size
Optimal test accuracy
Final training loss
Optimal effective learning rate 2- (2-2 to 2 2-1) 2-1 (2 to 29)
64
94.4 + 0.1
0.192 + 0.002
256
94.4 + 0.1
0.071 + 0.001 0.028 + 0.000
1024
93.8 + 0.1
2-1 (2-1 to 29)
These results demonstrate that stochastic gradient noise can enhance generalization, increasing the test accuracy here by nearly 2%. This shows that while the noisy quadratic model may help describe the evolution of the training loss, it does not capture important phenomena observed on the test set in popular networks (McCandlish et al. 2018 Zhang et al., 2019). While many previous authors have observed that stochastic gradient noise enhances generalization (Keskar et al.) 2016 Smith Le 2017 Dastrzebski al., 2017; Chaudhari & Soatto 2018, Park et al.] 2019), we believe our experiment is the first to demonstrate this effect when training well-respected architecture to the expected test accuracy with properly tuned learning rate schedule. Finallly, we emphasize that the implicit regularization introduced by stochastic gradients should be considered complementary to the | 701 Gunasekar et al., | [2088a bb [Soudry et al. 2018 Jacot et al., 2018; Allen-Zhu et al. 2018: Oymak & Soltanolkotabi] 2018 Azizan et al. 2019).
implicit bias of gradient descent (Ma et al.]
THE DEPENDENCE OF THE LEARNING RATE ON THE EPOCH BUDGET
We established in section 4 that, in some architectures and datasets, the noise introduced by stochastic gradients does enhance generalization. This motivates the following question; if the batch size is fixed, how does the optimal test accuracy and optimal learning rate depend on the epoch budget? In particular, is the optimal training temperature, defined by the ratio of the learning rate to the batch size, independent of the epoch budget, or does it fall as the number of training epochs increases.
To answer this question, we select a batch size of 64, and we evaluate both the optimal test accuracy and the optimal training loss for a range of epoch budgets using SGD with Momentum. As before, we use our standardized learning rate schedule for each epoch budget, described in appendix [ However, to study the effect of the optimal training temperature, we now independently measure both the optimal learning rufe to maximize the test accurucy, and the optimal learning rate to minimize the training loss. The optimal test accuracy and optimal training loss are shown in figures 2a and 2b. We train both with and without batch normalization, and provide the optimal learning rates with batch normalization in figure 2c, and the optimal learning rates without batch normalization in figure 2d.
Considering first figure 2a, the optimal test accuracy initially increases as we increase the epoch budget, however with batch normalization it saturates for epoch budgets beyond 800 epochs, while without batch normalization it falls for epoch budgets beyond 400 epochs. This is similar to the well- well-known phenomenon of early stopping (Prechelt 19988 Caruana et al., 200T). As expected. in figure 2b, find that the optimal training loss falls monotonically the epoch budget increases.
7

---

Under review as a conference paper at ICLR 2020
Oytima ofhectvee tem Fip rafo bett: mommmaattinn
Optine artective ear noa pat
with batch nermakzsion without batch normaliaaiion
000-
052-
with batch commilerass
Eotch nomollzation
Epech budget
Epoch bucgee
Gooch oudget
Epech busger
(a)
(b)
(c)
(d)
Figure 2: The performance of 16-4 Wide-ResNee on CIFAR-10 using SGD with Momentum and a batch size of 64, We train both with batch normalization, and also without batch normalization using "zeroInit". We identify both the optimal effective learning rate which maximizes the test accuracy and the optimal effective learning rate which minimizes the training loss, and we present the mean performance of the best 12 out of 15 runs. a) Initially the test accuracy rises as the epoch budget increases, however when training without batch normalization it begins to fall beyond 400 training epochs. b) The training loss falls monotonically as the epoch budget rises. e) With batch normalization, the learning rate which minimizes the training loss falls rapidly as the epoch budget rises, while the learning rate which maximizes the test accuracy only varies by a factor 2 when the epoch budget rises over two orders of magnitude. d) Similarly, without batch normalization using zerolnit, the learning rate which minimizes the training loss falls as the epoch budget rises while the learning rate which maximizes the test accuracy is constant for all epoch budgets considered.
Figures 2c and 2d are more surprising. Both with and without batch normalization, the learning rate which minimizes the training loss falls rapidly as the epoch budget rises. This is exactly what one would expect from convergence bounds or the noisy quadratic model (Ma al. 2017bj Zhang] (et al. 2019). Strikingly however, when training with batch normalization the learning rate which maximizes the test accuracy only falls by factor of 2 when we increase the epoch budget from 50 to 6400 epochs. Mcanwhile when training without batch normalization using zerolnit, the learning rate which maximizes the test accuracy constant for all epoch budgets considered. These results support the claim that when training deep networks on classification tasks, there is an optimal temperature scale early in training (Smith & Le 2017 Park et al.) 2019), which biases small batch SGD towards parameters which perform well on the test set. Our results also suggest that one might be able to reduce the cost of hyper-parameter tuning by first identifying the optimal learning rate for a modest epoch budget, before progressively increasing the epoch budget until the test accuracy saturates. We provide additional experimental results on the LSTM and autoencoder in appendix F. To further investigate whether there is an optimal temperature early in training, we provide an additional experiment in appendix where we independently tune both the initial learning rate and the final learning rate in our schedule. Our main claims in this section still hold in this experiment.
6 CONCLUSIONS
The contributions of this work are twofold. First, we verified that SGD exhibits two regimes with different behaviours. In the noise dominated regime which arises when the batch size is small, the test accuracy is independent of batch size under constant epoch budget, the optimal learning rate increases as the batch size rises, and acceleration techniques do not outperform vanilla SGD, Meanwhile in the curvature dominated regime which arises when the batch size is large, the optimal learning rate is independent of batch size, acceleration techniques outperform vanilla SGD, and the test accuracy degrades with batch size. If certain assumptions are satisfied, the optimal learning rate in the noise dominated regime is proportional to batch size. These assumptions hold for most tasks. Second, we confirm that a gap in test accuracy between small or moderately large batch sizes and very large batches persists even when one trains under constant step budget. When training a 16-4 Wide-ResNet on CIFAR-10 for 9765 updates, the test accuracy drops from 94.7% at a batch size of 4096 to 92.5% at a batch size of 16384. We also find that the optimal learning rate which maximizes the test accuracy of which minimizes the training loss falls rapidly as the epoch budget increases. These results confirm that stochastic gradients introduce implicit regularization which enhances generalization, and they provide novel insights which could be used to reduce the cost of identifying the optimal learning rate.
depends very weakly on the epoch budget while the learning rate
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Extremely large minibatch sgd: training resnet-50 on imagenet 15 minutes. arXiv preprint xrXiv:1711.04325. 2017.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. arXiv preprint arXiv: 10tn::9111.0,,, 2018.
Anonymous. Batch normalization has multiple benefits: An empirical study residual networks. Under review at ICLR, 2020.
Navid Azizan, Sahin Lale, and Babak Hassibi. Stochastic mirror descent on overparameterized nonlinear models: Convergence, implicit regularization, and generalization. arXiv preprint arXiv:1906.03830, 2019.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normaliza- tion. In Advances in Neural Information Processing Systems, pp. 7694-7705, 2018.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT pp. 177-186. Springer, 2010.
Rich Caruana, Steve Lawrence, and C Lee Giles. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In Advances in neural information processing systems, pp. 402-408, 2001.
Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges limit cycles for deep networks. In 2018 Information Theory and Applications Workshop (ITA), pp. 1-10. IEEE, 2018.
Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Big batch sgd: Automated inference using adaptive batch sizes. arXiv preprint 110.5716,, 2016.
Crispin W Gardiner et Handbook of stochastic methods, volume springer Berlin, 1985.
Rong Ge, Sham Kakade, Rahul Kidambi, and Praneeth Netrapalli, The step decay schedule: near optimal, geometrically decaying learning rate procedure. arXiv preprint 1904:1280.283 2019.
Gabriel Goh. Why momentum really works. Distill, 2(4): 2017.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 1eXivv/706.000 706707777,777 2017.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. arXiv preprint arXiv:/802.08244 2018a.
Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, pp. 9461-9471 2018b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Tom M Heskes and Bert Kappen. On-line learning processes artificial neural networks. In North-Holland Mathematical Library, volume 51, pp. 199-233. Elsevier, 1993.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza-
tion gap in large batch training of neural networks. Advances in Neural Information Processing Systems, pp. 1731-1741 2017.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization neural networks. In Advances neural information processing systems, pp. 8571-8580, 2018.

---

Under review as a conference paper at ICLR 2020
Stanistaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint 2017.
Chi Jin, Praneeth Netrapalli, Rong Ge, Sham Kakade, and Michael Jordan, Stochastic gradient descent escapes saddle points efficiently. arXiv preprint arXiv: 1902.048/ 2019.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 1609. 044366 2016.
Rahul Kidambi, Pranecth Netrapalli, Prateek Jain, and Sham Kakade. On the insufficiency of existing momentum schemes for stochastic optimization. In 2018 Information Theory and Applications Workshop (ITA), pp. 1-9. IEEE, 2018.
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprini arXiv:1404.5997 2014.
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
Müller. Efficient backprop. In
Qianxiao Li, Cheng Tai, et al. Stochastic modified equations and adaptive stochastic gradient algorithms. Proceedings of the 34th International Conference on Machine Learning-Vois Volume 70, pp. 1101-2110. JMLR. org, 2017.
Chaoyue Liu and Mikhail Belkin. Mass: an accelerated stochastic method for over wer-parametrized learning. arXiv preprint arXiv: 13395, 2018.
Cong Ma, Kaizheng Wang, Yuejie Chi, and Yuxin Chen. Implicit regularization in nonconvex statistical estimation: Gradient descent converges linearly for phase retrieval, matrix completion and blind deconvolution. arXiv preprint arXiv: 10467, 2017a.
Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern weerpparamttrized learning. arXiv preprint arXiv:1712.06559, 2017b.
Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent approximate bayesian inference. The Journal of Machine Learning Research, (6(1):4873-490) 2017.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint urXiv:1812.061 2018
Hiroaki Mikami, Hisahiro Suganuma, Yoshiki Tanaka, Yuichi Kageyama, et al. Imagenet/resnet-50 training 224 seconds. arXiv preprint arXiv:/8/1.05233. 2018.
Genevieve Orr and Todd Leen. Momentum and optimal stochastic search. In Proceedings of the 1993 Connectionist Models Summer School, pp. 351-357. Psychology Press, 1994.
Samet Oymak and Mahdi Soltanolkotabi. Overparameteriized nonlinear learning: Gradient descent takes the shortest path? arXiv preprint / 2018.
Daniel S Park, Jascha Sohl-Dickstein, Quoe Le, and Samuel I. Smith. The effect of network width stochastic gradient descent and generalization: empirical study. arXiv preprint arXiv:1905.03776, 2019.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838-88) 885, 1992.
Lutz Prechelt. Early stopping-bus but when? In Neural Networks: Tricks of the trade, pp. 55-69, Springer, 1998.
10

---

Under review as a conference paper at ICLR 2020
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12(1): 145-151, 1999.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Karthik Sankararaman, Soham De, Zheng Xu, Ronny Huang, and Tom Goldstein. The impact of neural network overparameterization on gradient confusion and stochastic gradient descent. arXiv preprint
2019.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Alcksander Madry. How does batch normal- ization help optimization? In Advances in Neural Information Processing Systems, pp. 2483-2493,
2018.
Christopher Shallue, Jachoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv: 2018.
Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise deep neural networks. arXiv preprint arXiv:/901 0e0010909.... 2019.
Samuel L Smith and Quoe V Le. bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451/ 2017.
Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don't decay the learning rate, increase the batch size. arXiv preprint Sttv:1711..04899 2017.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1): 2822-2878, 2018
Iya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139-1147, 2013.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688 2011.
Chris Ying, Sameer Kumar, Dehao Chen, Tao Wang, and Youlong Cheng. Image classification at supercomputer scale. arXiv preprint
2018.
Yang You, Zhao Zhang, Cho-Jui Hsich, James Demmel, and Kurt Keutzer. Imagenet training in minutes. In Proceedings of the 47th International Conference on Parallel Processing, pp. 1. ACM, 2018.
Kun Yuan, Bicheng Ying, and Ali H Sayed. On the influence of momentum acceleration on online learning. The Journal of Machine Learning Research, 17(1):6602-660 2016.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 16X.0:660..07146. 2016.
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv: 011.44499222222 2014.
Guodong Zhang, Lala Li. Zachary Nado, James Martens, Sushant Sachdeva, George E Dahl, Christo- pher Shallue, and Roger Grosse. Which algorithmic choices matter at which batch sizes? insights from noisy quadratic model. arXiv preprint arXiv:I907 04 04164, 2019.
11

---

Under review as a conference paper at ICLR 2020
A OUR LEARNING RATE DECAY SCHEDULE
For clarity, we illustrate our standardized learning rate decay schedule in figure 3. As specified the main text, if the epoch budget is N Nepochas we hold the learning rate constant for Nepochs /2, before decaying the learning rate by factor of 2 every Nepocha/ 20. When learning rate warmup is included, we linearly increase the learning rate to its maximal value over the first epochs of training.
1.0
Learning rate multiplier
without learning rate warmup
0.8
with learning rate warmup
0.6
0.2
0.0
5 epochs
0.5 0.55 0.6 0.65 0.7 0.75
0.85
0.9
0.95
10
Fraction compute budget (a)
Our standardized learning rate schedule, both with and without learning rate warmup. Note that while the learning rate decays points defined by the fraction of the total epoch budget completed, we perform learning rate warmup for 5 epochs, irrespective of the epoch budget.
Figure 3:
B DERIVING THE LINEAR SCALING RULE FOR SMALL BATCH SIZES
In the main text, we applied the central limit theorem to approximate single SGD step by,
Aw; (Wi41 wi) -E dC
+ VETó,
(3)
di w
The temperature 7 e/B, R (5;) = O and E (6.d7) F(w;) where F(w) is the empirical Fisher information matrix. Equation holds so long as the gradient each training example is an independent and uncorrelated sample from an underlying short tailed distribution. Additionally, it assumes that the training set size N > 1, the batch size > 1, and B < N. To derive the linear scaling rule, consider the total change in the parameters
consecutive parameter updates,
dC
Awl-
Awi+s
VneT'
dis w=ww43
The noise & = (1/v7) En 824j. When the product of the number of steps n and the learning rate e is much smaller than the critical learning rate, ne & Ccrit, the parameters do not move far enough for the gradients significantly change, and therefore for all {j,ī'} greater than 0 and less than n,
dC
dC
(5)
dw w-
dw
(6.4 814g')
F(w.)ojz
(6)
Equation implies that,
dC
Aw! ~ nE
+ V VnCTE.
(7)
dw w=w
While equation implies that E (Ex) = and E (EGET) = Fw:)). We therefore conclude that & and both Gaussian random variables from the same distribution. Comparing equation and equation we conclude that n SGD updates at temperature T with learning rate e is equivalent to a single SGD step at temperature 7 with learning rate ne. Since the temperature T 7==6 e/ B, this implies that
12

---

Under review as a conference paper at ICLR 2020
c30
635
Warn up
Wamn up
023
50D
652-
600
6:35
025
653
6:5
Jeamiin rate
Leariing rate
Laarning 1abe
Learning note
(a) Batch size 64
(b) Batch size 1024
(c) Batch size 64 (d) Batch size 1024
Figure 4: A 16-4 Wide-ResNer trained with batch normalization CIFAR-10 10 for 200 epochs. For completeness, we provide the full results of a learning rate sweep at two batch sizes, 64 and 1024. The smaller batch size is in the noise dominated regime, while the larger batch size is in the curvature dominated regime. We provide the final test accuracy at a range of learning rates in figures a and b, and we provide the final training loss at range of learning rates in figures с and d. We note that SGD, SGD with Momentum, and SGD with Momentum and learning rate warmup always achieve similar final performance in the small learning rate limit, while SGD performs poorly when the learning rate is large. When the batch size is small, the optimal learning rate is also small, and so all three methods perform similarly. When the batch size large, the optimal learning rate also large, and SGD underperforms both SGD with Momentum and SGD with Momentum and learning rate warmup.
when < Cerit, then simultaneously doubling both the learning rate and the batch size should draw samples from the same distribution over parameters after the same number of training epochs.
This prediction is known as the linear scaling rule (Krizhevsky 2014] Goyal et al. 2017: Mandi et al. 2017 Smith& & Le 2017 Dastrzebski et al. [017] Chaudhari & Soatto [2018] McCandlish et al. 2018 Shallue et al. 2018). 1). Since this linear scaling rule assumes that < Cerit, it usually holds when the batch size is small, which appears to contradict the assumption B > 1 above. Crucially however, the distribution of & does not matter in practice, since dynamics is governed by the combined influence of noise over multiple consecutive updates, & = (1/Vn) Σド= Pezj. In other words, do not require that equation 3 is accurate model of an single SGD step, only require that equation 7 is an accurate model of n SGD steps. We therefore conclude that 6, does not need to be Gaussian, we only require that & is Gaussian. The central limit theorem predicts that, if 6, is an independent random sample from a short-tailed distribution, & will be Gaussian if N >> 1, nB > 1 and nB33 < N. If & Cerit, then we can choose < n < N, and discard the assumption B >>
C ZEROINIT, SIMPLE INITIALIZATION SCHEME FOR RESIDUAL NETWORKS
In the main text, we provided number of experimental results on wide residual networks, trained without batch normalization using "Zerolnit". This simple initialization scheme, introduced in parallel submission (Anonymous) 2020), is now presented here for the reader's convenience. The scheme comprises three simple modifications to the original Wide-ResNet architecture (Zagoruyko & Komodakis 2016):
1. It introduces a scalar multiplier the end of each residual branch, just before the residual branch and skip connection merge. This scalar multiplier is initialized to zero
2. It introduces biases to each convolutional layer, which are initialized to zero.
3. It introduces dropout the final fully connected layer.
Modification 1 is sufficient to train very deep residual networks without batch normalization, while modifications 2 and 3 slightly increase the final test accuracy. We emphasize that dropout is included solely to illustrate that additional regularization is required when batch normalization is removed. Dropout itself is not required to train without batch normalization. and similar or superior results could be obtained with alternative regularizers. Throughout this paper, the drop probability is 60%.
13

---

Under review as a conference paper at ICLR 2020
10.
Optinal test accuracy
0.943
Finr training laps
0.8
Warm up Momentum
learning
Momentum
We'm
SGD
SGD
3.94
0.6-
0.823
0.4
0.25
Warm Momentum
0.325-
2.8
0.2
0.0855
SGD
0,03125
0.925
26
32
128 Betch size
64
250 512
1024
8
26
123 Beuch size
64
256
512
1024
0.015225
16
32
64
128
256
512
1024
Ratch 62M
(a)
(b)
(c)
Figure 5: A 16-4 Wide-ResNet, trained without batch normalization using zeroInit on CIFAR- 10 for 200 epochs. We report the performance of SGD, SGD w/ Momentum, and SGD w/ momentum using learning rate warmup. We perform grid search to identify the optimal learning rate which maximizes the test accuracy. and report the mean performance of the best 12 of 15 runs. a) The test accuracy of SGD w/ Momentum independent of batch size when the batch size is small, but begins to fall when the batch size exceeds 256. The test accuracy of vanilla SGD is falling for all batch sizes considered. b) The training loss at the optimal effective learning rate is independent of batch size when the batch size is small, but rises rapidly if acceleration techniques are not used when the batch size is large. e) The optimal effective learning rate is proportional to batch size when the batch size small for SGD Momentum, while is independent of batch size for vanilla SGD.
D ADDITIONAL RESULTS UNDER A CONSTANT EPOCH BUDGET
Here we provide some additional results studying how the optimal effective learning rate depends on the batch size under constant epoch budget.
Additional results on 16-4 Wide-ResNet on CIFAR with batch normalization. In figure we provide additional results with the 16-4 Wide-ResNet, trained with batch normalization on CIFAR-10 for 200 epochs. Here we provide the final test set accuracies and the final training set losses for full learning rate sweep two batch sizes, 64 and 1024. From figure we see that SGD, SGD with Momentum and warm up learning rates always achieve similar final performance in the small learning rate limit. This confirms previous theoretical work showing the equivalence of SGD and SGD with momentum in the small learning rate limit when the momentum parameter is kept fixed (Orr & Leen 1994. Qian 1999 Yuan et al., 2016). Further, we see that SGD performs poorly compared to SGI with Momentum and warmup learning rates when the learning rate is large. When the batch size is small, the optimal learning rates for all three methods are also small, and so all three methods perform similarly. On the other hand, when the batch size is large, the optimal learning rates of SGD with Momentum and warmup learning rates are higher than the optimal learning rate for SGD, and see Momentum and warmup learning rates start outperforming vanilla SGD. These results entirely consistent with the two regimes of SGD and SGD with Momentum discussed in section 2
16-4 Wide-ResNet CIFAR without batch normalization. In figure 5 we present results when training our 16-4 Wide-ResNet (Zagoruyko & Komodakis 2016). We follow the same setup and learning rate schedule described in section 3, and we train for 200 epochs. However we remove batch normalization, and introduce Zerolnit (described in appendix ( The performance of SGD degrades with increasing batch size on both the test set and the training set, while the performance of Momentum with or without learning rate warmup is constant for batch sizes B S 256. Above this threshold, the performance of both methods degrades rapidly. These observations are explained by the optimal effective learning rates in figure 5p. Momentum has similar optimal learning rates with and without warmup. In both cases the learning rate initially increases proportional to the batch size before saturating. The optimal learning rate of SGD is curvature bound all batch sizes considered.
ResNet-50 on ImageNet. In table 3, we provide results for ResNet-50 trained on ImageNet for 90 epochs at small range of batch sizes. We follow the modified ResNet-50 implementation of Goyal et al. (2017). and we use our standardized learning rate schedule without warmup (see appendix Ap. We train single model each batch size-learning rate pair. SGD with and without Momentum achieve similar test accuracies when the batch size is small, but SGD with Momentum outperforms SGD without Momentum when the batch size is large. The optimal effective learning
14

---

Under review as a conference paper at ICLR 2020
SCD Momentur
Optmmal MSE
Mumenturn
Ootimal Effective Learnin
2
24
Nomentum
Finel Train
2%
24
2
27
21
Besch Size
Datch Size
Batch Sice
(a)
(b)
(c)
Figure 6: fully connected autoencoder, trained on MNIST for 200 epochs. report the perfor- mance of SGD and SGD w/ Momentum. We perform grid search to identify the optimal learning rate which maximizes the mean-squared error (MSE) on the test set, and report the mean performance of the best 5 of runs. a) The test MSE of SGD w/ Momentum is initially independent of batch size, but it begins to rise when the batch size exceeds 128. The test MSE of vanilla SGD starts rising for batch sizes exceeding 16. b) We similar phenomena effective learning rate is proportional to batch size when the batch size is small for both vanilla SGD and SGD w/ Momentum, while it becomes independent of batch size for larger batch sizes. The optimal effective learning rate in the curvature dominated regime is larger for SGD w/ Momentum.
the training set MSE. c) The optimal
rate is proportional to batch size for all batch sizes considered when using SGD with Momentum, but when using SGD without Momentum.
Fully connected autoencoder on MNIST. In figure we present results when training a fully- connected autoencoder on the MNIST dataset (Sutskever al. 2013). Our network architecture is described by the sequence of layer widths (784, 1000, 500, 250, 30, 250, 500, 1000, 784), ) where 784 denotes the input and output dimensions. For more details on this architecture, refer (Sutskever et al. (2013). Because of the bottleneck structure of the model, it is known to be difficult problem to optimize and has often been used as an optimization benchmark (Sutskever al. 2013 Kidambi et al., 2018). The 1.2 regularization parameter was set at 10-5 We use the learning rate decay schedule described in appendix A without learning rate warmup, and we train each model for 200 epochs. As before, notice that for small batch sizes, the performance of both SGD and SGD w/ momentum independent of batch size, while performance begins to degrade when the batch size is large. On this model, the performance of SGD begins to degrade at much smaller batch sizes than we observed in residual networks, and consequently SGD w/ momentum starts outperforming SGD much smaller batch sizes. This is likely due the poor conditioning of the model due the bottleneck structure of its architecture.
LSTM on Penn TreeBank. Finally in figure language model on the Penn TreeBank dataset (PTB), following the implementation described in
we present results when training a word-level LSTM
Table 3: R0sNet-50, trained on ImageNet for 90 epochs. We follow the implementation of Goyal al. (2017), however introduce modified learning do not use learning rate warmup. We perform a grid search to identify the optimal effective learning rate and report the performance of single training Momentum are equal when the batch size is small, but Momentum outperforms SGD when the batch size is large, For SGD with Momentum, the optimal effective learning rate is proportional batch size for all batch sizes considered, while this linear scaling rule breaks at large batch sizes for SGD.
schedule defined in appendix A We
The test accuracies achieved by SGD and
Batch size
Optimal test accuracy (%)
Training loss
Optimal effective learning rate
256
77.0
2.25
1.0
SGD
1024
76.7
2.25
4.0
4096
76.1
2.30
8.0
256
77.0
2.25
1.0
Momentum
1024
76.8
2.25
4.0
4096
76.8
2.25
16.0
15

---

Under review as a conference paper at ICLR 2020
90.
Optimaa Effective Learnin Raba
Optiral Perpienty
SGD Momernuum
Fitel Thain Perplexity
SGD Momeniumm
SGD Momencum
22
06-
28
27-
2*
2
2
2'
27
2
21 Baich Size
21
27
27
27
Batch Size
Dettr1 Size
(a)
(b)
(C)
Figure A word-level LSTM language model trained on PTB for 40 epochs. We report the performance of SGD and SGD w/ Momentum. We perform grid search to identify the optimal learning rate which maximizes the test set perplexity, and report the mean performance of the best o runs. a) The test set perplexity of SGD w/ Momentum is independent of batch size when the batch size is small, but begins to rise when the batch size exceeds 128. The test set perplexity of vanilla SGD starts rising for batch sizes exceeding 64. b) We similar phenomena set perplexity, <) The optimal effective learning rate is proportional to square root of the batch size when the batch size is small, while levels off for larger batch sizes. The gradients of consecutive minibatches in a language model are not independent, violating the assumptions behind linear scaling.
the training
Zaremba et al. (2014). The LSTM used has two layers with 650 units per layer. The parameters are initialized uniformly in [-0.05, 0.05]. We apply gradient clipping at 5, as well as dropout with probability 0.5 on the non-recurrent connections. We train the LSTT for 40 epochs using an unroll step of 35, and use the learning rate decay schedule described in appendix A without learning rate warmup. As with the other models tested in this paper, this learning rate schedule reaches the same test perplexity performance as the original schedules reported in Zaremba et al. (2014). Once again, we see that SGD and SGD w/ momentum have similar performance for small batch sizes. Performance for SGD starts degrading for batch sizes exceeding 64, whereas performance for SGD w/ momentum starts degrading for batch sizes exceeding 128. However, as mentioned in section 3, the optimal learning rate increases as square root of the batch size for small batch sizes, before leveling off at a constant value for larger batch sizes. This is likely due to correlations consecutive data samples when training the LSTM, which violate the assumptions used derive the linear scaling rule section 2.
E ADDITIONAL RESULTS UNDER CONSTANT STEP BUDGET
Here we provide some additional results studying how the optimal test accuracy depends on the batch size under constant step budget. In table 4, we train a fully connected auto-encoder on MNIST for 156,250 updates (Sutskever et al. 2013). This corresponds to 200 epochs when the batch size is 64. We described this model in appendix D, and we train using the learning rate schedule defined in appendix A using SGD with Momentum without learning rate warmup. The test set MSE increases as the batch size increases, while the training set MSE falls the batch size rises. Although the training set MSE appears to rise for a batch size of 4096, this only occurs because the optimal effective learning rate is measured on the test set. The optimal effective learning rate is independent of the batch size, suggesting that the learning rate may be close to curvature dominated regime.
In table 5, we train a word-level LSTM on the Penn TreeBank (PTB) dataset (Zaremba et al. 2014) for 16560 updates. This corresponds to 40 epochs when the batch size is 64. We described this model in appendix D, and we train using the learning rate schedule defined in appendix A using SGD with Momentum without learning rate warmup. The test perplexity increases significantly the batch size increases, while the training perplexity falls as the batch size rises. The optimal effective learning rate increases as the batch size rises, suggesting that we are inside the noise dominated regime.
16

---

Under review as a conference paper at ICLR 2020
Table 4: The optimal test set MSE and final training set MSE for range of batch sizes under constant step budget. For each batch size, we train fully connected autoencoder on MNIST for 156,250 updates. We perform grid search to identify the optimal learning rate which maximizes the test set MSE, and we provide the average performance of the best 5 out of 7 training runs. The final test MSE falls for large batch sizes, although this effect rather weak this model.
Batch size
Optimal test set MSE
Final training set MSE
Optimal effective learning rate
64
2.91 + 0.01
2.017 + 0.003
0.08 (0.08 to 0.08)
256
2.95 + 0.01
2.010 + 0.005
0.08 (0.08 to 0.08)
1024
2.96 - 0.01
2.005 = 0,011
0.08
(0.08 to 0.08)
4096
2.98 - 0.01
2.018 = 0.008
0.08
(0.08 to 0.08)
Table 5: The optimal test set perplexity and final training set perplexity for a range of batch sizes under constant step budget. For each batch size, we train word-level LSTM on PTB for 16560 updates. We perform a grid search to identify the optimal learning rate which maximizes the test set perplexity, and we provide the average performance of the best 5 out of 7 training runs. The optimal test perplexity increases considerably as the batch size rises, while the training perplexity falls.
Batch size
Optimal test perplexity
Final training perplexity
Optimal effective learning rate
64
81.67 = 0.26
23.25 + 0.05
2.8 (2.8 to 2.8)
128
86.04 = 0.49
21.16 0.06 15.74 + 0.03
5.6 (5.6 to 5.6)
256
92.19 = 0.26
5.6 (5.6 to 5.6)
F ADDITIONAL RESULTS WITH FIXED BATCH SIZE AND VARIABLE EPOCH BUDGET
We now provide additional experimental results to accompany those provided in section 5] where we study whether the optimal training temperature is independent of the epoch budget. We use SGD with Momentum with the momentum parameter m 0.9 for all our experiments in this section. In figure g we present results on word-level LSTM on the PTB dataset for batch size of 64 and for varying epoch budgets. Note that the original LSTM model in Zaremba et al. (2014) was trained for 39 epochs. The results in figure g are remarkably similar to those presented in figure 2 As the epoch budget rises, the test set perplexity first falls but then begins to increase, The training set perplexity falls monotonically as the epoch budget increases. Finally, the optimal learning rate which minimizes the test set perplexity is independent of the epoch budget once this epoch budget is not too small, while the optimal learning rate which minimizes the training set perplexity falls.
In figure 9 we present results on a fully connected autoencoder trained on MNIST for a batch size of 32 and for range of epoch budgets. Note that the autoencoder results presented in section D were trained for 200 epochs. Figures 9h and are similar to figures 2a and 2b in the main text. Initially the test set MSE falls as the epoch budget increases, but then it starts increasing. The training set MSE falls monotonically as the epoch budget rises. In figure DC however, we notice that the learning rate that minimizes the test set MSE decreases as the epoch budget rises. This is the opposite of what we observed in figures 2 and g To further investigate this, in figure 99 we plot the mean test set MSE during training for an epoch budget of 800 for learning rates = 0.004 and = 0.002. We notice that for the larger learning rate c = 0.004, the model overfits faster on the training set, causing the test set MSE to rise by the time of the first learning rate drop at 400 epochs. This is consistently the case for all epoch budgets over 200 epochs. To avoid the test set MSE from rising, the optimal learning rate for the test MSE drops to slow down training sufficiently such that there is no overfitting before the first learning rate decay. Meanwhile the optimal learning rate to minimize the training loss is more or less constant. This suggests that early stopping is particularly important in this architecture and dataset, and that it has more influence the final test performance than stochastic gradient noise.
17

---

Under review as a conference paper at ICLR 2020
04-
40
Optinal Parplisiti
92-
Optimal Perpicetty
00-
25
Optimm Effective
06-
Teat Perplexity Train Perplexity
10
2' Epoch Dudget
2 Epoch budget
Cpoch Dudget
(a)
(b)
(C)
Figure 8: The performance a word-level LSTM language model trained on the Penn TreeBank dataset using SGD with Momentum and batch size of 64 at range of epoch budgets. We identify both the optimal effective learning rate which minimizes the test set perplexity and the optimal effective learning rate which minimizes the training set perplexity, and we present the mean performance of the best 5 out of 7 runs. a) Initially the test set perplexity falls as the epoch budget increases, however it begins rise beyond 56 training epochs. b) The training set perplexity falls monotonically as the epoch budget rises. c) The learning rate that minimizes the training set perplexity falls the epoch budget rises, while the learning rate that minimizes the test set perplexity only varies by factor of 2 when the epoch budget rises over two orders of magnitude.
33
opor effocte LOO NOMC
3.2-
Derine Trea
335-
Epech Budget
Epsch Susqet
Epech Budget
Numbee oftrainnn spechts
(a)
(b)
(c)
(d)
Figure 9: The performance of fully connected autoencoder MNIST using SGD with Momentum and a batch size of 32 with varying training epoch budget. We identify both the optimal effective learning rate which minimizes the test set MSE and the optimal effective learning rate which minimizes the training set MSE, and we present the mean performance of the best out of 7 runs. a) Initially the test set MSE falls the epoch budget increases, and it only starts going up very slightly for large epoch budgets. b) The train set MSE falls monotonically as the the epoch budget rises. c) The learning rate that minimizes the test set MSE decreases, while the learning rate that minimizes the train set MSE remains constant as the epoch budget rises. This is contrary to what we observe in figures and The reason for this is apparent from figure d), where we plot the test set MSE during training for all 7 runs for an epoch budget of 800 for learning rate 0.004 and notice that for a larger learning rate, the model overfits on the training set faster, causing the test set MSE to rise by the time of the first learning rate drop at 400 epochs. This suggests that early stopping has more influence on the final test performance this architecture than stochastic gradient noise.
0.002. We
G ADDITIONAL RESULTS SWEEPING BOTH THE INITIAL AND FINAL LEARNING RATES
In section 5 and appendix we study how the optimal learning rate depends on the epoch budget. In find evidence that there may be an optimal temperature early in training which is beneficial for good generalization performance on classification tasks. However the learning rate schedules used for these experiments have the property that the initial learning rate, which denote in this section with eo. is coupled with the final learning rate, which we denote in this section with ey. More specifically, the final learning rate e = cu y 10 where 7 denotes the decay factor, which we set to 2 in the bulk of our experiments (this schedule is described in detail in section AA.
these experiments,
Coupling the initial and final learning rates make it less clear whether this optimal temperature is important the start or the end of training, and it is also not clear what extent our conclusions
18

---

Under review as a conference paper at ICLR 2020
0099
Cittmns affectiv IeEESSI
Oytima affectivv
2.12
Optinaledd
0.45-
mang
leaming leamiin rate ale
ale ado Epech Bodget.
Epoch budiged
Epsch sudget
Spech budget
(a)
(b)
(c)
(d)
Figure 10: Performance of 16-4 ReseeR with batch normalization trained on 10
using SGD with Momentum at a batch size of 64. We tune the initial and the final learning rates independently, as described in section G We plot both the optimal initial and final learning rates for maximizing the test set accuracy, as well as the optimal initial and final learning rates for minimizing the training set loss, and we present the mean performance of the best 5 out of 7 runs. a) The test accuracy initially increases with increasing compute budget before saturating for epochs budgets greater than 800. b) Meanwhile the training loss falls monotonically as the epoch budget rises. c) The optimal initial learning rate which maximizes the test accuracy is constant for epoch budgets greater than 400, while the optimal final learning rate decays rapidly as the epoch budget increases. d) The optimal initial learning rate which minimizes the training loss decays slowly as the epoch budget increases, while the optimal final learning rate decays more rapidly.
are influenced by our choice of decay factor. Therefore in this section, we perform experiments on varying epoch budgets where we tune both the initial and the final learning rates independently. As in the schedule presented in section A when training for an epoch budget of N Nepochs. we use the initial learning rate for the first Nepochs/2 /2 epochs, and we then decay the learning rate by a factor of y every N Nepochs/ /20. To define y, we select both an initial learning rate co and final learning rate cy, and we then set 7 = (c0/E))2/10 We do not use learning rate warmup. These experiments require a very only study our 16-4 Wide-ResNe model with batch normalization.
large compute budget, and
In figure T0 we show results when training this 16-4 Wide-Resnet on CIFAR- 10 at a batch size of 64 using SGD with Momentum. We train for range of epoch budgets from 50 to 6400 epochs, and we evaluate the optimal initial and final learning rates independently for both maximizing the test set accuracy and minimizing the training loss. From figure T0h and T0p, we see the same trends as observed in section 5 and appendix [F] Specifically, when we increase the compute budget, the optimal test set accuracy first increases, and then saturates for epoch budgets greater than 800, while the optimal training loss falls monotonically as the epoch budget grows.
In figures 10 and I0 we plot the optimal initial and final learning rates for maximizing the test set accuracy 100) and minimizing the training set loss (TO)). We make several observations from these plots. First, the optimal initial learning rate for maximizing the test set accuracy decays very slowly as the epoch budget rises, and it is constant for epoch budgets greater than 400. This supports the existence of an optimal temperature early in training which boost generalization performance. Meanwhile, the optimal final learning rate for maximizing the test set accuracy does decay rapidly as the epoch budget increases, which is likely helpful to prevent overfitting at late times. We note that the error bars the final learning rate are much larger than those the initial learning rate, suggesting that it is the initial learning rate which is most important to tune in practice.
The optimal initial learning rate for minimizing the training loss also decays slowly as the epoch budget rises (decreasing by a factor of 4 to 8 when the epoch budget rises by factor of 128). while the optimal final learning rate for minimizing the training loss decays much more quickly (roughly by a factor of 128 over the same range). The optimal initial learning rate for maximizing the test accuracy is consistently higher than the optimal initial learning rate for minimizing the training loss, while the optimal final learning rate for maximizing the test accuracy is consistently lower than the optimal final learning rate for minimizing the training loss. These two observations support the widely held belief that learning rate schedules which maintain a high temperature at early times, and then decay the learning rate rapidly at late times generalize well the test set in some architectures.
19


Reviews:
[{"review_id": "rJxkq6waYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the properties of SGD as a function of batch size and learning rate. Authors argue that SGD has two regimes:  a noise dominated regime (small batch size) and curvature dominated regime (large batch size). Authors conduct through numerical experiments highlighting how learning rate changes as a function of batch size (initially linear growth and then saturates). The critical contribution of this work appears to be the observation that large batch size can be worse than small under same number of steps demonstrating implicit regularization of small batch size.\n\nThe two regime claim of the paper is not really novel. These regimes are fairly well covered by previous works (e.g. Belkin et al as well as others). When it comes to experiments, constant epoch budget is also fairly well understood and the behavior in Figure 1 is not really surprising (as the eventual training performance gets worse with large batches).\n\nThe interesting part in my opinion is the experiments on constant steps. Authors verify large batch size reduces test accuracy while improving train. I believe these experiments are novel and the results are interesting. Besides CIFAR 10, authors test this hypothesis in two other datasets while tuning the learning rate. On the other hand, contribution is somewhat incremental given observations made by related literature (Keskar et al and others).\n\nSome remarks:\n1) In Table 1, batch size 16k has effective LR of 32. However in Figure 1c SGD with momentum at batch size 8k uses an effective LR of 4. Can you explain this inconsistency i.e. why is there such a huge jump from 4 to 32 (in reality we expect the effective LR to stay constant in the curvature regime). I also understand that one is constant epoch and other is constant step. However 4 to 32 seems a bit inconsistent.\n\n2) Does momentum help in constant step budget (with sufficiently large steps so that training loss is small)?\n\n3) Readability: Consider explaining what is meant by \"warm-up\", \"epoch budget\", \"step budget\" clearly and upfront.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1l1CEFwKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper attempts to clarify the debate on large-batch neural network training, particularly on the relationship between learning rate, batch sizes and test performance. The authors claim two contributions towards understanding how the hyper-parameters of SGD affect final training and test performance: (1) SGD exhibits two regimes with different behaviours and (2) large-batch training leads to degradation of test performance even with same step budgets.\n\nOverall, the authors did a comprehensive study on large-batch training with the support of extensive experiments. But I'm concerned with the novelty and contributions of this paper. I tend to reject this paper because (1) the first contribution of the paper is not new as it has already been recognized by a few paper that SGD exhibits two different regimes; (2) this paper makes the debate of large-batch training even muddier.\n\nMain argument:\nThe paper does not do a great job in clarify the debate. Particularly, the authors mixed their observations up with the results of published works, making it hard to identify the contributions of this paper. For example, the two regimes mentioned in the paper has been identified by a few other works and the contribution of this paper is just to verify them again. Also, I find the experiments done in section 3 and 4 are similar to previous works and even the conclusions are similar. The only new observation I'm aware of in these two sections is that the training loss and test accuracy are independent of batch size in the noise dominated regime.\n\nBack to introduction section, the goal of this paper (as claimed in the beginning of second paragraph) is to clarify the debate. But does this paper really achieves this goal? In terms of learning rate scaling, this paper gets similar conclusions as Shallue et al. (2018). In terms of the difference between vanilla SGD and SGD with momentum, Zhang et al. (2019) already argued that the difference depends on specific batch sizes and SGD with momentum only outperforms SGD in the curvature dominated regime. \n\nI think the authors should instead focus on the discussion of generalization performance and the observation that training loss and test accuracy are independent of batch size in noise dominated regime. To my knowledge, this part is novel and interesting. \n\nIn summary, I'm inclined to reject this paper given the current version. However, I think the paper is still worth reading if the authors can reorganize the paper and I might increase my score if my concerns get resolved.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgmhEfTcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is an empirical contribution regarding SGD arguing that it presents two different behaviors which the authors name a noise dominated regimen, and a curvature dominated regime. They observe that the behaviors seem to arise in different batch sizes \n\nThe authors derive empirical conclusions and perform experiments in different settings. The paper is well-written and the experimental setup seems to be carefully carried out. \n\nI find the observations interesting, but the contribution is empirical and not entirely new. It would be nice if there were some theoretical results to back up the observations. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors provide an empirical evaluation of batch size and learning rate selection and its effect on training and generalization performance. As the authors and reviewers note, this is an active area of research with many closely related results to the contributions of this paper already existing in the literature. In light of this work, reviewers felt that this paper did not clearly place itself in the appropriate context to make its contributions clear. Following the rebuttal, reviewers minds remained unchanged. 
Decision: Reject