A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
Authors: Zachary Nado
Justin Gilmer
Christopher J Shallue
Rohan Anil
George Edward Dahl
Venue: NeurIPS 2021 Submitted (2021)
Keywords: neural networks
deep learning
neural network optimization
hyperparameter tuning
optimizer comparison

Abstract: Recently the LARS and LAMB optimizers have been proposed for training neural networks faster using large batch sizes. LARS and LAMB add layer-wise normalization to the update rules of Heavy-ball momentum and Adam, respectively, and have become popular in prominent benchmarks and deep learning libraries. However, without fair comparisons to standard optimizers, it remains an open question whether LARS and LAMB have any benefit over traditional, generic algorithms. In this work we demonstrate that standard optimization algorithms such as Nesterov momentum and Adam can match or exceed the results of LARS and LAMB at large batch sizes. Our results establish new, stronger baselines for future comparisons at these batch sizes and shed light on the difficulties of comparing optimizers for neural network training more generally. 

A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch
Sizes
Anonymous Author(s)
Affiliation
Address
email
Abstract
Recently the LARS and LAMB optimizers have been proposed for training neural networks faster using large batch sizes. LARS and LAMB add layer-wise normal- ization to the update rules of Heavy-ball momentum and Adam, respectively, and have become popular prominent benchmarks and deep learning libraries. How- ever, without fair comparisons to standard optimizers, remains open question whether LARS and LAMB have any benefit over traditional, generic algorithms. this work demonstrate that standard optimization algorithms such as Nesterov momentum and Adam can match or exceed the results of LARS and LAMB at large batch sizes. Our results establish new, stronger baselines for future comparisons at these batch sizes and shed light on the difficulties of comparing optimizers for neural network training more generally.
2
4
6
10
11
Introduction
12
In recent years, hardware systems employing GPUs and TPUs have enabled neural network training 14 programs to process dramatically more data in parallel than ever before. The most popular way to exploit these systems is to increase the batch size in the optimization algorithm (i.e. the number
16 of training examples processed per training step). On many workloads, modern systems can scale larger batch sizes without significantly increasing the time per step (Jouppi al.) [017]] Wang 18 et al. 2019], thus proportionally increasing the number of training examples processed per second. researchers can this increased throughput reduce the time required to train each neural 20 network, then they should achieve better results by training larger models, using larger datasets, and
21 by exploring new ideas more rapidly.
22 As the capacity for data parallelism continues to increase, practitioners can take their existing, well-tuned training configurations and re- re-train with larger batch sizes, hoping to achieve the same 24 performance in less training time [e.g. Ying et al. 2018). On an idealized data-parallel system with negligible overhead from increasing the batch size, they might hope achieve perfect scaling, proportional reduction in training time as the batch size increases.
26
However, achieving perfect scaling is not always straightforward. Changing the batch size changes training dynamics, requiring the training hyperparameters (e.g. learning rate) be carefully tuned in order to maintain the same level of validation performance| In addition, smaller batch 30 sizes provide implicit regularization from gradient noise that may need to be replaced by other forms of regularization when the batch size is increased. Finally, even with perfect tuning, increasing
28
Although there heuristies for adjusting the learning rate the batch size changes, these heuristics inevitably break down sufficiently far from the initial batch size and is also not clear how apply them to other training hyperparameters (c.g. momentum).
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

73 with Adam also improves upon the wall-time record of LAMB reported in You et al. 2019). Taken together, our results establish stronger training speed baselines for these tasks and batch sizes, which 32 the batch size eventually produces diminishing returns. After a critical batch size, the number of training steps cannot be decreased in proportion to the batch size - the number of epochs must increase to match the validation performance of the smaller batch size. See (Saaluu et al. 2019 for survey of the effects of data parallelism on neural network training. Once these effects are taken into account, there is no strong evidence that increasing the batch size degrades the maximum achievable performance on any workload. At the same time, the ever-increasing capacity for data parallelism presents opportunities for new regularization techniques that can replace the gradient noise of smaller batch sizes and new optimization algorithms that can extend perfect scaling to larger batch sizes by using more sophisticated gradient information [Zhang et al. 2019).
33
34
35
36
37
38
39
41
You et al. [2017) proposed the LARS optimization algorithm in the hope of speeding up neural network training by exploiting larger batch sizes, LARS is variant of stochastic gradient descent (SGD) with momentum [Polyak] 1964) that applies layer-wiee wise normalization before applying each gradient update. Although it is difficult to draw strong conclusions from the results presented in the Training benchmark"
42
43
44
45
LARS as one of two allowed algorithms in the closed division for ResNet-50 on ImageNet and it became the de facte standard algorithm for that benchmark task. With MLPerf entrants competing to find the fastest-training hyperparameters for LARS, the first place submissions in the two most recent MLPerf Training competitions used LARS to achieve record training speeds with batch sizes of 32,678 and 65,536, respectively. No publications or competitive submissions to MLPerf have attempted to match these results with standard optimizer (e.g. Momentum or Adam). However, MLPerf entrants do not have a strong incentive (nor necessarily permitted by the rules) to explore other algorithms because MLPerf Training is systems benchmark that requires algorithmic equivalence between submissions to make fair comparisons. Moreover, since the main justification for LARS is its excellent performance on ResNet-50 at large batch sizes, more work is needed to quantify any benefit of LARS over standard algorithms at any batch size.
LARS paper, the ML.Per Mr.PerTTaainig
46
47
48
49
51
59
54
55
56
57
You et al. [2019] later proposed the LAMB optimizer to speed up pre-training for BERT (Devlin et al. 2018 using larger batch sizes after concluding that LARS was not effective across workloads.
58
59
LAMB is a variant of Adam Kingmma and Ba 2014] that adds a similar layer-wise wise normalization step
60
to LARS. You et al. (2019) used LAMB for BERT pre-training with batch sizes up to 65,536 and claimed that Adam cannot match the performance of LAMB beyond batch size 16,384.
61
62
In this paper, we demonstrate that standard optimizers, without any layer-wise normalization tech- niques, can match or improve upon the large batch size results used to justify LARS and LAMB. In Section 2 we show that Nesterov momentum [Nesterov] 1983] matches the performance of LARS on the ResNet-50 benchmark with batch size 32,768. We are the first to match this result with a standard optimizer. In Section 3 contradicting the claims in You et al. [219)) we show that Adam obtains better BERT pre-training results than LAMB at the largest batch sizes, resulting in better downstream performance metrics after fine-tuning.
63
64
65
66
67
69 In addition, we establish new state-of ofe-the-art for BERT pretraining speed, reaching an Fl score of
70 90.46 in 7,818 steps using Adam at batch size 65,536 (we report training speed in steps because our
71 focus is algorithmic efficiency, but since we compare LARS and LAMB to simpler optimizers, fewer
72 training steps corresponds to faster wall-time in an optimized implementation our BERT result
75 we hope will assist future work aiming to accelerate training using larger batch sizes.
78 In addition to the contributions mentioned above, we demonstrate several key effects that are often overlooked by studies aiming to establish the superiority of new optimization algorithms. We show that future work must carefully disentangle regularization and optimization effects when comparing a new optimizer to baselines. We also report several inder-documented details used to generate the
77
78
79
80 best LARS and LAMB results, a reminder that future comparisons should document any novel tricks and include them in baselines. Finally, our results add to existing evidence in the literature on the difficulty of performing independently rigorous hyperparameter tuning for optimizers and baselines.
81
82
2 The modified AlexNet on ImageNet benchmark did not have well-established accuracy targets from prior work and LARS used a more general lcarning rate schedule than the momentum baseline. For ResNet-50 on ImageNet, LARS achieved sub-par accuracy numbers and was not compared any other optimizer at the same batch size, lcaving open the possibility that generie optimizer would scale just as well LARS. ML.Perf a trademark of MLCoomonn.oog.
https: attps://mlperr: hegg/training-vver

---

(in 83 In particular, we show that the optimal shape of the learning rate schedule is addition to the scale), and that differences in the schedule can dominate optimizer comparisons at smaller step budgets and become less important at larger step budgets.
84
85
86
1.1
Related work
87
Shallue et al.] [2019] and Zhang et al. [2019] explored the effects of data parallelism on neural network training for different optimizers, finding no evidence that larger batch sizes degrade performance and demonstrating that different optimizers can achieve perfect scaling up to different critical batch sizes. You et al. (2017) 2019] developed the LARS and LAMB optimizers in the hope of speeding up training by achieving perfect scaling beyond standard optimizers. Many other recent papers have proposed new optimization algorithms for generic batch sizes or larger batch sizes [see [Schmidt et al. 2020]. Choi et al. [2019] and /Schmidt et al. [2020] demonstrated the difficulties with fairly comparing optimizers, showing that the hyperparameter tuning protocol is a key determinant of optimizer rankings. The MLPerf Training benchmark (Mattson et al. 2019] provides competitive ranking of neural network training systems, but does not shed much light on the relative performance of optimizers because entrants are limited in the algorithms they can use and the hyperparameters they can tune.
88
89
90
91
92
94
96
98
2
Matching LARS on ImageNet
99
100
The MLPerf training benchmark for ResNet-50 v1.5 on ImageNet [Mattson et al. 2019] aims to reach 75.9% validation accuracy in the shortest possible wall-clock time. In the closed division of the competition, entrants must choose between two optimizers, SGD with momentum or LARS, and are only allowed to tune a specified subset of the optimization hyperparameters, with the remaining hyperparameter values set by the competition rules F| The winning entries in the two most recent competitions used LARS with batch size 32,768 for 72 training epoch and LARS with batch size 65, ,36 for 88 training epochs respectively. Kumar et al. [2019] later improved the training time for batch size 32,768 by reaching the target accuracy in 64 epochs. These are currently the fastest published results on the ResNet-50 benchmark. However, it has been unclear whether LARS was necessary to achieve these training speeds since no recent published results or competitive MLPerf submissions have used another optimizer. In this section, describe how matched the 64 epoch, 32,768 batch size result of LARS using standard Nesterov momentum
101
102
103
104
105
106
107
109
111
112
A fair benchmark of training algorithms or hardware systems must account for stochasticity in individual training runs. In the MI.Perf competition, the benchmark metric is the mean wall-clock time of 5 trials after the fastest and slowest trials are excluded. Only 4 out of the 5 trials need to reach the target accuracy and there is no explicit limit on the number of times an entrant can try a different set of 5 trials. Since our goal is to compare algorithms, rather than systems, we aim to match the LARS result in terms of training steps instead (but since Nesterov momentum is computationally simpler than LARS, this would also correspond to faster wall-clock time on optimized system). Specifically, we measure the median validation accuracy over 50 training runs with a fixed budget of 2,512 training steps" at batch size of 32.768. When ran the published LARS training pipeline To
113
114
115
117
119
121
LARS achieved a median accuracy of 75.97% and reached the target in 35 out of 50 trials. We consider the LARS result to be matched by another optimizer if the median over 50 trials exceeds the
122
123
target of 75.9%.
124
2.1
Nesterov momentum at batch size 32k
125
This section describes how we used the standard Nesterov momentum optimizer to train the ResNet- 50 v1.5 on ImageNet to 75.9% validation accuracy in 2,512 update steps at a batch size of 32,768, matching the best published LARS result this batch size. Although we implemented our own training program, the only logical changes we made to the published LARS pipeline were to the optimizer and the optimization hyperparameters. Our model implementation and data pre-processing pipeline were identical to those required under the MLPerf closed division rules (see Appendix B
126
127
128
129
130
0-6 "The 88 epoch. 65,536 batch size result is faster in terms of wall-clock time hut requires more training epachs, indicating that it is beyond LARS's perfect scaling regime. Although LARS obtains diminishing returns when increasing the batch size from 32,768 to 65,536, future work could investigate whether Nesterov momentum drops off more or less rapidly than LARS.
https::/gitt.iit 10/ JtknD
https://mlperf.
httpss//mlppef org/ http:/trininnggtteeeee http:/raiinnn--euutteeeeeee results-0-7
9 Corresponding to 64 training epochs in Kumar et al. 20019
10 htttps://gt /tt//t/tttt//tt
3

---

131
We present two Nesterov momentum hyperparameter configurations that achieve comparable per- formance to LARS. Configuration achieved a median accuracy of 75, 55.97% (the same as LARS) and reached the target accuracy in 34 out of 50 trials. Configuration B is modified version of Configuration A designed to make as few changes as possible to the LARS hyperparameters; achieved a median accuracy of 75.92% and reached the target in 29 out of 50 trials. See Appendix D. for the complete hyperparameter configurations.
132
133
134
135
136
137
To achieve these results, we tuned the hyperparameters of the training pipeline from scratch using Nesterov momentum. We ran a series of experiments, each of which searched over a hand-designed hyperparameter search space using quasi-randon search [Bousquet et al. 2017). Between each experiment, we modified the previous search space and/or tweaked the training program to include optimization tricks and non-default hyperparameter values we discovered in the state-of-the- LARS pipeline. The full sequence of experiments we ran, including the number of trials, hyperparameters tune, and search space ranges, are provided in Appendix [D4] Once we had matched the LARS result with Configuration A, we tried setting each hyperparameter to its value the LARS pipeline in order to find the minimal set of changes that still achieved the target result, producing Configuration B. The remainder of this section describes the hyperparameters we tuned and the techniques we applied on the journey to these results.
138
139
140
142
143
144
146
148
2.1.1
Nesterov Momentum Optimizer
149
Nesterov momentum is a variant of classical or "heavy-ball" momentum defined by the update rule
U(+1 = j40; + vE(O,),
0.+1 = 02 - T) (10++1+ + VE(O:)),
150
where ipp = O, 8 is the vector of model parameters after steps, Vε(+:) is the gradient of the loss function E(O) averaged over batch of training examples, μ is the momentum, and 7) is the learning rate for step t. We prefer Nesterov momentum over classical momentum because it tolerates larger values of its momentum parameter (Sutskever al. 2013) and sometimes outperforms classical momentum, although the two algorithms perform similarly on many tasks [Shallue et al.] 2019 Choi
152
153
154
155
al. 2019]. We tuned the Nesterov momentum y in Configurations A and B. We discuss the learning
156
rate schedule } separately in Section 2.1.4
157
2.1.2
Batch normalization
156
The ResNet-50 v1.5 model uses batch normalization [loffe and Szegedy 2015). defined as
(x()) m))n(x(0) x 7()) + =8()
() =
())() +
where (1() vector of pre-normalization outputs from layer I, mean(.) and var(-) denote the demeent-wsee sample mean and variance across the batch of training examples and y()) and z
159
180
are trainable model parameters.
162
Batch normalization introduces the following tuneable hyperparameters: e, the small constant added to the sample variance; the initial values of (()) and g(0); and p. which governs the exponential moving averages of the scaling factors used in evaluation. The LARS pipeline uses c = 10-5 and
164
p = 0.9. It sets the initial value of 300) to 0.0 everywhere, but the initial value of 70)) depends on
165
the layer: it sets (0))) to 0.0 in the final batch normalization layer of each residual block, and to 1.0 everywhere else. In Configuration A, we tuned e, P. and 7o. the initial value of 700) in the final batch normalization layer of each residual block. In Configuration B, we used the same values as LARS for and p. but we found that choosing Yo between 0.0 and 1.0 was important for matching the LARS result with Nesterov momentum.
186
167
188
169
170
2.1.3 Regularization
172
In Configuration A, we tuned both the L2 regularization coefficient À and label smoothing coefficient T (Szegedy et al.) 2016). The LARS pipeline uses X = 10-4 and T 0.1.
173
In a distributed training environment the mean and variance are commonly computed over a subset of the full batch. The LARS pipeline uses a "virtual batch size" of 64, which we also use to avoid changing the training objective [Hoffer et al. 2017.

---

174
Crucially, the LARS pipeline does not apply L2 regularization to the bias variables of the
ResNet model nor the batch normalization parameters 7(()) and 8() (indeed, the published uses Heavy-ball momen-
175
178
LARS pipeline does not even apply LARS these parameters
177
tum). This detail is extremely important for both LARS and Nesterov momentum to achieve
178
the fastest training speed. Configuration B used the same and as Configuration A.
179
Nesterov
LARS
180
2.1.4
Learning rate schedule
Pwarnup
2
1
181
The LARS pipeline uses piecewise polynomial schedule
7ppaak
7.05
29.0
Thuit + (Mpaak Mimit)
Ponamp
twarmup
Thiaal
6 x 10-6
10-4
Fromuu T-E founcc
Th:
Discary
0.02397
0.071
Minal + (Mpeak Minal
T
twarmups
5.8 10
10
with Jinit = 0.0, Vppaak 29.0, 7/imal 10-1, Pwarmup 1,
0.15
0.10
183
Pdecuy 2, and wwarmup = 706 steps. Configuration A, we re- tuned all of these hyperparameters with Nesterov momentum. In Configuration B, we set Minits Pdccay, and Ewammup to the same values as LARS, changing only Pwarmup from 1 to 2 and re- scaling Tpaak and Thinal-
0.4138
0.0
184
185
Table I: The hyperparameters of Configuration that differ from state-of-the-ar LARS at batch size 32,768 Kumar et al. 2019].
187
188
2.1.5
Comparing Nesterov momentum and LARS
Table [ shows the hyperparameter values for Configuration that differ from the state- of-the-art LARS pipeline. Aside from re-tuning the momentum, learning rate scale, and regularization hyperparameters (whose optimal values are all expected to change with the
189
191
optimizer), the only changes are setting Pwarmup to 2 instead of and re-tuning Yo-
193
194
Figure T shows the LARS learning rate schedule com-
LARS
195
pared the Nesterov momentum schedule. Even though these schedules are similar, we found that each optimizer had a different optimal value the warmup polynomial power As Table 2 shows, Nesterov momentum performs better with Pwammup = instead of while the opposite is true with LARS. As discussed in (Agaraal et al. [2020). optimizers can induce implicit step size schedules that strongly influence their training dynamics and solution quality, and it appears from Table 2 that the implicit step sizes of Nesterov momentum and LARS may evolve dif- ferently, causing the shapes of their optimal learning rate schedules to differ.
Retatve Stop Size
keawnr
196
197
199
530
1030 1500
2000
2506
Step
201
Figure 1: The learning rate schedules of LARS and Nesterov momentum Con- figuration B. Aside from re-scaling, the only difference is setting the warmup polynomial power to instead 1.
203
205
207
Although the main concern of a practitioner validation performance, the primary task of an optimization algorithm is to minimize training loss. Table[]shows that Nesterov momentum achieves higher training accuracy than LARS, despite similar validation performance. Thus, may more appropriate to consider the layerwise normalization of LARS be regularization technique, rather than an optimization technique.
209
211
Spending even more effort tuning LARS or Nesterov momentum would likely further improve the current sfate-of-the-wee for that optimizer. Meaningful optimizer comparisons are only possible with independent and equally intensive tuning efforts, and we do not claim that either optimizer 214
212
213
215
outperforms the other on this benchmark. That said, if the main evidence for LARS's s utility as a batche-batch optimizer" is its performance on this particular benchmark, then more evidence is needed to quantify any benefit has over traditional, generic optimizers like Nesterov momentum.
217
218
2.2
Lessons learned
In hindsight, it was only necessary to make a few changes to the LARS pipeline match its performance at batch size 32,768 with Nesterov momentum. However, Table I does not accurately represent the effort required when attempting to match highly tuned training- speed benchmark.
220
5

---

Pwarmup
Nesterov
LARS
Optimizer
Train Ace
Test Acc
75.79%
75.97%
Nesterov
78.97%
75.93%
2
75.92%
75.69%
LARS
78.07%
75.97%
Table 2: (Left) The best warmup schedule differs for Nesterov momentum and LARS. Values are medians over 50 training runs after setting Pwarmup without retuning other hyperparameters. (Right) Median train and test accuracies over 50 training runs for Nesterov momentum Configuration B and LARS.
Firstly, described in Sections 2.1.2 and 2.1.3 the strong results of LARS depend partly on few subtle optimization tricks and non-default values of uncommonly-tumm -ouuned hyperparaneters. Fortunately, in this case we could discover these tricks by examining the open-source code required for MIPPrf submissions, but machine learning research papers do not always report these important details. Researchers can easily waste lot of experiments and produce misleading results hefore getting all of these details right. We demonstrate the importance of adding these tricks to our Nesterov momentum pipeline in Appendix C without these tricks (or some new tricks), we likely would not have been able to match the LARS performance.
223
224
225
226
227
228
229
Secondly, the learning rate schedule really matters when trying maximize performance with relatively small step budget. Both LARS and Nesterov momentum are sensitive to small deviations from the optimized learning rate schedules in Figure T and neither schedule works as well for the other optimizer. Although relatively minor changes were sufficient to match LARS with Nesterov momentum, there is no way to know priori how the optimal schedule will look for new optimizer
231
232
233
234
Wu et al.] 2018). Even in toy settings where the optimal learning rate schedule can be derived, it does not fit into commonly used schedule families and depends strongly on the optimizer Zhhag et al. [019]] Indeed, this problem applies to the other optimization hyperparameters as well: it 237
235
236
238
extremely difficult to know which are worth considering ahead of time. Finally, even when we narrowed down our hyperparemeter search spaces around the optimal point, the volume of our search
239
240
spaces corresponding to near-peak performance was small, likely due to the small step budget (Shallue
241
et al. 2019]. We investigate how these effects change with a less stringent step budget in Section
3
Stronger BERT pretraining speed baselines
242
You et al. 2019 developed the LAMB optimizer in the hope of speeding up training for BERT-
244
Bidirectional Encoder Representations from Transformers, Devlin et al.) 2018). BERT training consists of two phases. The "pretraining" phase has two objectives: (1) predicting masked tokens based on the rest of the sequence (a masked language model), and (2) predicting whether two given sentences follow one from another. Finally, the "fine-tuning" phase refines the model for downstream task of interest. BERT pretraining takes a considerable amount of time (up to 3 days on 16 Cloud TPU-v3 chips Jouppi et al. (2017). whereas the fine-tuning phase is typically much faster. Model quality is typically assessed on the downstream metrics, not on pretraining loss, making BERT training a somewhat awkward benchmark for optimization research.
246
248
249
250
251
252
You et al. 2019) used LAMB for BERT pretraining with batch sizes up to 65,536 and claimed that LAMB outperforms Adam batch size 16,384 and beyond. The LAMB optimizer has since appeared
254
in several NLP toolkits, including as Microsoft DeepSpeed and NVIDIA Multi-node BERT training, and as a benchmark task in MLPerf vO.7 As shown in Table 3 we trained Adam (with decoupled weight decay) baselines that achieve better results than both the LAMB and Adam results reported in You et al. 2019). Our new Adam baselines obtain better Fl scores on the development set of the SQuaD vI.1 task in the same number
255
256
258
of training steps as LAMB for both batch size 32,768 and the hybrid 65.536-then-32,766 batch
280
size training regime in You et al. [019]. We also ran Adam at batch size 65,536 to reach nearly the same Fl score as the hybrid batch size LAMB result, but in much fewer training steps. We believe 7,818 steps is a new ofate-of-the-art for BERT pretraining speed [in our experiments, it
282
also improves upon the 76-minute record claimed in You et al. 2019]. Additionally, at batch
264
size 32,768 our Adam baseline got a better pretraining loss of 1.277 compared to LAMB's 1.342.
We do not consider the MLPerf task in this paper since it is a warm-start, partial training task.

---

285
Batch size
Step budget
LAMB
Adam
266
We used the same experimental setup You et al. [2019]. including two pretraining phases with max sequence lengths of 128 and then 512.
32k
15,625
91.48
91.58
65k/32k
8,599
90.58
91.04
267
65k
7,818
90.46
268
289 In order to match You et al. [2019) we reported
Table 3: Using Adam for pretraining exceeds the reported performance of LAMB in You et al. [019] in terms of F1 score on the downstream SQuaD v1.1 task.
270
the Fl score on the downstream SQuaD v1.1 task as the target metric, although this metric introduces potential confounds: optimization efficiency should be measured on the training task using training and held-out data sets. Fortunately, in this case better pretraining performance correlated a with higher Fl score after fine-tuning, See Appendix [B.2 for additional experiment details. We tuned Adam hyperparameters independently for each pretraining phase, specifically learning rate 7) 31. 82, the polynomial power for the learning rate warmup Pwarmmpp: and weight decay X, using quasi- quasi-random search [Bousquet et al., 2017]. See Appendix [D22 the search spaces.
271
272
273
274
275
276
277
278
279
In addition to hyperparmeter tuning, our improved Adam results at these batch sizes are also likely due to two implementation differences. First, the Adam implementation in You et al.) 2019] comes
280
from the BERT open source code base, in which Adam is missing the standard bias correction 13
281
282
The Adam bias correction acts as an additional step size warm-up, thereby potentially improving the
283
stability in the initial steps of training. Second, the BERT learning rate schedule had a discontinuity
284
at the start of the decay phase due to the learning rate decay being incorrectly applied during warm-up up
T4
(see Figure 2 in Appendix B). This peculiarity is part of the official BERT release and is present in
285
286
+ copies of the BERT Training code on GitHub.
287
4
Investigating less stringent step budget
288
Part of what makes comparing optimizers so difficult is that the hyperparameter tuning tends to dominate the comparisons [Choi et al. 2019]. Moreover, tuning becomes especially difficult when
289
290
we demand a fixed epoch budget even when dramatically increasing the batch size (Shallue et al.
281
2019]. Fixing the epoch budget as the batch size increases is equivalent to demanding perfect scaling
(i.e that the number of training steps decreases by the same factor that the batch size is increased). 292
293
We can view the role of hyperparameter tuning for large batch training as resisting the inevitable end of perfect scaling, For example. it might be possible to extend perfect scaling using delicately tuned learning rate schedules, but comparing optimizers under these conditions can make the learning rate schedule dominate the comparison by favoring some algorithms over others. Therefore, in order to better understand the behavior of LARS and LAMB compared to Nesterov Momentum and Adam, we ran additional ResNet-50 experiments with a more generous 6,000 step budget (vs 2,512 in Section 2 and a more simplistic cosine learning rate schedule. At batch size 32,768, this budget should let us reach better validation accuracy than the MLPerf target of 75.9%.
294
295
296
297
298
299
301
Although not mentioned in You et al. [017]. the state-of-the-art MLPerf pipeline for "LARS" actually uses both LARS and Heavy-ball Momentum, with Momentum applied to the batch normalization and ResNet bias parameters and LARS applied to the other parameters. You et al. [2019] does not mention
302
303
304
whether LAMB was only applied to some parameters and not others. If layerwise normalization can
305
be harmful for some model parameters, this is critical information for practitioners using LARS or LAMB, since it might not be obvious which optimizer to apply to which parameters. To investigate this, we trained both pure LARS and LAMB configurations, as well as configurations that did not apply layerwise normalization to the batch normalization and ResNet bias parameters. Moreover, LAMB's 's underlying Adam implementation defaults to e = 10 6 rather than the typical 10-7 or In some cases, can be a critical hyperparameter for Adam [Choi et al. 2019 so we included Adam configurations with both = 10 and = 1
306
307
309
10
311
312
Table shows the validation accuracy of these different configurations after training for 6,000 steps with batch size 32.768. In every case, we used a simple cosine decay learning rate sched- ule and tuned the initial learning rate and weight decay using quasi-random search. We used momentum parameters of 0.98 for Nesterov momentum and 0.929 for LARS, respectively, based
313
314
315
316
on the tuned values from Section 2 We used default hyperparameters for Adam and LAMB except where specified. We set all other hyperparameters to the same values as the ofate-of-the- art LARS pipeline, except we set 70 = 1.0. See Appendix D.3 for more details. As expected,
317
https://git..o/U/
See Stetttss///gt..1//// and https::/gttt.iae
7

---

319
highly tuned learning rate schedules and optimizer hyperparameters are no longer necessary with a less stringent step budget. Multiple optimizer configurations in Table 4 exceed the MLPerf target accuracy of 75.9% at batch size 32,768 with minimal tuning. Training with larger batch sizes is not fundamentally unstable: stringent step budgets make hyperparameter tuning trickier.
320
322
323
324
Weights
Bias/BN
In Table 4 "pure LAMB" performs extremely poorly: LAMB only obtains reasonable results when is nor used on the batch normalization and ResNet bias parameters, suggesting that lay- erwise normalization can indeed be harmful on some parameters. "Pure LARS" and Nesterov momentum perform roughly the same at this step budget, but the MLPerf LARS pipeline, which is tuned for a more stringent step bud- get, does not use LARS on all parameters, at least suggesting that the optimal choice could be budget-dependent.
Top-1
325
Optimizer
Optimizer
326
Nesterov
Nesterov
76.7
327
328
LARS
Momentum
76.9
329
LARS
LARS
76.9
330
Adam (c 10-8)
Adam 10 8)
76.2
332
Adam (c 10-9)
Adam (c = 10 10-0)
76.4
LAMB
LAMB
27.3
334
LAMB
Adam = 10-8)
76.3
336
Many new neural net optimizers, including LAMB, are introduced alongside claims that the new optimizer does not require any-or -r at least minimal-tuning claims require a lot of work to support, since they require trying the optimizer on new prob- lems without using those problems during the development of the algorithm. Although our ex- periments here are not sufficient to determine which optimizers are easiest to tune, experiments like these that operate outside the regime of highly tuned learning rate schedules can serve as a starting point. In this experiment, LARS and LAMB do not appear to have advantage in how easy they are to tune
LAMB
Adam (c (c=110 10 10-9)
76.3
337
Table 4: Validation accuracy of ResNet-50 on Ima- geNet trained for 6,000 steps instead of 2.512. The second column is the optimizer that was applied to the batch norm and ResNet bias variables. We report the median top-1 accuracy over 5 seeds of the best hyperparameter setting in a refined search space. See Appendix D.3 for details.
338
339
Unfortunately, these
340
341
342
344
346
a dataset and model that were used in the development of both of those algorithms. LAMB is a variant of Adam and performs about the same Adam with the same value of e; LARS is more analogous Momentum and indeed Nesterov momentum and LARS have similar performance.
348
349
350
351
5
Discussion
352
Our results show that standard, generic optimizers suffice for achieving strong results across batch sizes. Therefore, any research program to create new optimizers for training at larger batch sizes must start from the fact that Momentum, Adam, and likely other standard methods work fine at batch sizes as large as those considered in this paper. The LARS and LAMB update rules have no more to do with the batch size (or "large" batches) than the Momentum or Adam update rules. Although You et al. 2019 presented convergence rate bounds for LARS and LAMB to support their claims of superior performance, we show in Appendix A that Adam satisfies similar bound to LAMB. These bounds all rely on very unrealistic assumptions 15 Most of all, they are loose upper bounds on the worst case behavior of the algorithms, not accurate reflections of optimizer performance in reality. Whether layer-wise normalization can be useful for optimization or regularization remains an open question. However, if LARS and LAMB have any advantage over standard techniques, it is not that they work dramatically better on the tasks and batch sizes in You et al. 2017 2019). This is
353
354
355
357
359
361
362
363
364
not to suggest that there is nothing interesting about studying neural network optimization at larger batch sizes. For example, as gradient noise decreases, there may be opportunities to harness curvature
385
366
information and extend the region of perfect scaling (Zhang et al. 2019]. However, there is currently no evidence that LARS and LAMB scale better than Momentum and Adam.
387
Our primary concern in this paper has been matching the state of the ant-and establishing new baselines- for training speed measurements of the sort used to justify new techniques and algorithms for training with larger batch sizes. In contrast, many practitioners the best possible validation error with a somewhat flexible training time budget. Part of the reason why matching LARS at batch size 32,768 was non-trivial is because getting state of the art training
369
more concerned with obtaining
371
372
15 All convergence bounds assume no momentum is used, and the Lavg bound for LAMB also assumes B3 = 0, when it is typically 0.999. Additionally, Lang could still be large if Loo is large, but we leave an empirical analysis of this to future work.
8

---

373
speed requires several tricks and implementation details that are not often discussed. It was not obvious to us a priori which ones would prove crucial. These details do not involve changes to the optimizer, but they interact with the optimizer in regime where all hyperparameters need to be well tuned to stay competitive, making it necessary to re-tune everything for a new optimizer.
374
375
376
377 In neural network optimization research, training loss is rarely discussed in detail and evaluation centers on validation/test performance since that what practitioners care most about. However, although we shouldn't 't only consider training loss, it is counter-intuitive and counter-productive to elide a careful investigation of the actual objective of the optimizer. If new optimizer achieves better test performance, but shows no speedup on training loss, then perhaps it is not a better optimizer so much as an indirect regularizer. [6] Indeed, in our experiments we found that Nesterov momentum
378
379
380
381
382
achieves noticeably better training accuracy on ResNet-50 than the LARS configuration we used,
383
384
despite reaching roughly the same validation accuracy. Properly disentangling possible regularization benefits from optimization speed-ups is crucial if we are to understand neural network training, especially at larger batch sizes where we lose some of the regularization effect of gradient noise. Hypothetically, if the primary benefit of training procedure is regularization, then it would be better to compare the method with other regularization baselines than other optimizers.
385
386
388
389
Ultimately, we only care about batch size to the extent that higher degrees of data parallelism lead to faster training. Training with a larger batch size is a means, not the end goal. New optimizers whether designed for generic batch sizes or larger batch sizes- have the potential to dramatically improve algorithmic efficiency across multiple workloads, but our results show that standard opti-
390
391
392
383
mizers can match the performance of newer alternatives on the workloads we considered. Indeed, despite the legion of new update rule variants being proposed in the literature, standard Adam and Momentum remain the workhorses of practitioners and researchers alike, while independent empirical
394
395
396
comparisons consistently find no clear winner when optimizers are compared across a variety of workloads (Schmidt et al. 2020). Meanwhile, as Choi et al. [2019] and our results underscore, comparisons between optimizers crucially depend on the effort spent tuning hyperparameters for each optimizer. Given these facts, we should regard with extreme caution studies claiming to show the superiority of one particular optimizer over others. Part of the issue stems from current incentives in the research community: we overvalue the novelty of new methods and undervalue establishing strong baselines to measure progress against. This is particularly problematic in the study of optimizers, where the learning rate schedule is arguably more important than the choice of the optimizer update rule itself! As our results show, the best learning rate schedule is tightly coupled with the optimizer, meaning that tuning the learning rate schedule for new optimizer will generally favor the new optimizer over a baseline unless the schedule of the baseline is afforded the same tuning effort.
398
400
402
404
405
406
Conclusion
407
408
In this work, we demonstrated that standard optimizers, without any layer-wise normalization techniques, can match or exceed the large batch size results used to justify LARS and LAMB. Future work attempting to argue that new algorithm is useful by comparing to baseline methods or results, including those established in this paper, faces a key challenge in showing that the gains are due the new method and not merely due to better tuning or changes to the training pipeline (e.g. regularization tricks). Although gains from tuning will eventually saturate, we can, in principle, always invest more effort in tuning and potentially get better results for any optimizer. However, our goal should be developing optimizers that work better across many different workloads when taking into account the amount of additional tuning they require.
410
412
413
414
415
416
Moving forward, if we are reliably make progress need to rethink how we compare and evaluate new optimizers for neural network training. Given how sensitive optimizer performance is to the hyperparameter tuning protocol and how difficult it is to quantify hyperparameter tuning effort, can't 't expect experiments with self self-reported baselines to always lead to fair comparisons. Ideally, new training methods would be evaluated in standardized competitive benchmark, where submitters of new optimizers do not have full knowledge of the evaluation workloads. Some efforts in this direction have started, for instance the MLCommons Algorithmic Efficiency Working Group [77] but more work needs to be done to produce incentives for the community to publish well-tuned baselines and to reward researchers that conduct the most rigorous empirical comparisons.
418
419
420
422
423
424
425
16 Deep learning folk wisdom is that "any method to make training less effective can serve as regularizer." whether it is bug in gradients or clever algorithm.
17 https: //ml:///////mmmmmmmmmmmmmmmmm org/ wwwgg/n/ggrops/reeeeeeeee wwahaaarrchhaaarorree algorithms/

---

Checklist
428
427
1. For all authors...
428
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Sections 2
3
430
(b) Did you describe the limitations of your work? Yes] We had lengthy discussion of
431
the limitations and scope of the work in Section 5
432
(c) Did you discuss any potential negative societal impacts of your work? [No] We did not discuss this in the main text. Our primary contribution is to improve experimental protocols for other methodological work, which is removed from specific machine learning applications that is hard to determine the net impact. That said, more effective experimental protocols should lead to more effective science which in turn should lead to more effective machine learning applications. Whether this development is positive or negative for society will depend who stands to gain from the use of machine learning in future applied contexts. Additionally, although our work should, in the long run, save computational resources for individual researchers, in net across the community this may or may not produce an aggregate savings hecause more efficient machine learning training, by making larger scale projects more accessible, can lead to an increased demand for compute resources [York] 2006). which can have varying degrees of negative environmental impacts (Patterson et al. 2021).
433
435
437
439
440
441
443
444
445
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Appendix for a comprehensive description of the problem setting.
A
449
450
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A
451
3. If you ran experiments...
452
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] We will include
453
454
a link to all code and all possible reproducibility instructions after the anonymized reviewing period is over.
455
456
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We extremely detailed about our tuning procedures and dataset
458
details, see Appendices B D
459
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] While we do not report error bars in the tables in the main
460
text, Appendices B.2 C contains box plots showing the quartiles of the distribution
462
over random seeds.
463
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No] In Appendix [B we state that we run on Google TPUs, however we do not tally up the total number of experiments run (although an interested reader could compute it from the information we provided in our detailed appendices given that we list all intermediate experiments, no matter how silly in hindsight).
464
485
466
467
468
469
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
470
(a) If your work uses existing assets, did you cite the creators? [Yes] We reference the relevant citations for all models, datasets, and techniques.
471
472
(b) Did you mention the license of the assets? [No]
475
(c) Did you include any new assets either the supplemental material or as a URL? [No]
474
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
475
using/curating? [N/A]
476
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
10

---

479
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
480
481
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
484
References
485
486
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin. Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL ttttpa://wwwe tensorflox org/ Software available from tensorflow.org.
488
490
481
482
483
434
485
Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling adaptive gradient methods from learning rates. arXiv preprint arXiv:2002. /8003, 2020.
496
497
Olivier Bousquet, Sylvain Gelly, Karol Kurach, Olivier Teytaud, and Damien Vincent. Critical hyper- parameters: No random, no cry. arXiv, 2017. URL h 1etp:/17..... 03200
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL cow goog///o
500
502
503
Dami Choi, Christopher Shallue, Zachary Nado, Jachoon Lee, Chris J Maddison, and George Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint 19X.v:1900.05555 2019.
505
506
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rX.v:1810.04805, 2018.
507
508
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the gen- eralization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.
509
510
511 Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03166 2015.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of tensor processing unit. In Proceedings of the 44th Annual International Symposium Computer Architecture, pages 1-12, 2017.
514
516
517
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1412.6:412.69980 2014.
519
Sameer Kumar, Victor Bitorff, Dehao Chen, Chiachen Chou, Blake Hechtman, HyoukJoong Lee, Naveen Kumar, Peter Mattson, Shibo Wang. Tao Wang, et al. Scale mlperf-0.6 models on google tpu-v3 pods. arxiv preprint (tti::/999.097.00 2019.
520
521
522
Peter Mattson, Christine Cheng. Cody Coleman, Greg Diamos, Paulius Micíkevicius, David Patterson, Hanlin Tang, Gu- YeoY Wei, Peter Bailis, Victor Bittorf, David Brooks, Dehao Chen, Debojy- oti Dutta, Udit Gupta, Kim Hazelwood, Andrew Hock, Xinyuan Huang, Atsushi Ike, Bill Jia, Daniel Kang, David Kanter, Naveen Kumar, Jeffery Liao, Guokai Ma, Deepak Narayanan, Tayo Oguntebi, Gennady Pekhimenko, Lillian Pentecost, Vijay Janapa Reddi, Taylor Robie, Tom St. John, Tsuguchika Tabaru, Carole-Jean Wu, Lingjie Xu, Masafumi Yamazaki, Cliff Young, and
523
524
526
11

---

528
Matei Zaharia. MLPerf training benchmark. arXiv preprint arXiv:1910000/ 2019. URL https://arxiv.org/ wwwgg//b////.....//// 01500
529
530
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate O(1/k2). In Dokl. akad. nauk Sssr, volume 269, pages 543-547, 1983.
David Patterson, Joseph Gonzalez, Quoe Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint ar 100.50000. 2021.
533
535
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
536
537
Robin M Schmidt, Frank Schneider, and Philipp Hennig. Descending through crowded valley- benchmarking deep learning optimizers. arXiv preprint arXiv:2007.01547, 2020.
538
539
Christopher Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism neural network training. Journal of Machine Learning Research, 20(112): 1-4,, 2019.
541
542
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initialization and momentum in deep learning. In ICML, 2013.
543
5M4
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe. Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
545
546
547
Yu Emma Wang, Gu- Yeon Wei, and David Brooks. Benchmarking tpu, gpu, and cpu platforms for deep learning. arXiv preprint arXiv: 10701, 2019.
548
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. arXiv preprint arXiv:1803.0202.1 2018.
550
551
Chris Ying, Sameer Kumar, Dehao Chen, Tao Wang, and Youlong Cheng. Image classification at supercomputer scale. arXiv preprini arXiv:18/ 1.06992, 2018.
552
553
Richard York. Ecological paradoxes: William stanley jevons and the paperless office. Human Ecology Review, pages 143-147 2006.
554
555
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint Xiviv 1et./700..03800 2017
556
557 Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song. James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2019.
559
Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. In Advances in Neural Information Processing Systems, pages 8 (96-8207,2019)
562
583
584
585
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the 2015 HEEE International Conference on Computer Vision (ICCV). ICCV '15, page 19-27, USA, 2015. IEEE Computer Society. ISBN 978146738391..
566
567
568
doi: 10. 10.1109/1CV.2205..I 11. htLp:tts::/// //doi org/ 110.//111111110
1ttp///VCV..2.1..111 2015.11
12


Reviews:
[{"review_id": "xYBKtnaNVgm", "reviewer": "Reviewer_JpZL", "summary": "The authors detail the significant effort required to reproduce results from the original LARS and LAMB papers.  Having reproduced results for LARS (for ResNet-50 training on ImageNet) and LAMB (for BERT pretraining), the authors then show that standard/older optimizers (Nesterov momentum and Adam) can match or even exceed LARS and LAMB at large batch sizes.  The authors provide in-depth details on both the difficulties required for them to reproduce the original results (i.e., fixing discrepancies between details in the publications and available source-code from online implementations) and their hyperparameter optimization for Nesterov Momentum and Adam on the datasets evaluated.  Also, insights and lessons are provided, both on the importance of hyperparameter tuning for comparison of DL solvers and on practitioners attempting to demonstrate performance gains for a new method (are not merely due to better tuning or regularization tricks).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the authors have done an impressive job both recreating the results of the LARS and LAMB papers (including all necessary hyperparameters and implementation tricks) and showing that Adam and Nesterov Momentum may be used in place of LARS and LAMB for the two tasks presented.  Furthermore, the lessons detailed are important and the paper is a valuable addition to researchers putting together new baselines.\n\nI spent a long time combing through the paper.  On the one hand, this is an unfiltered account of the struggles researchers often encounter when attempting to reproduce state-of-the-art (including the extreme detective work necessary to find the undocumented tricks necessary to achieve some results).  However, in the end, I did not provide a higher score due to the lack of novel contributions and the narrow scope of the paper.  For the former, there are no new methods presented (or novel applications), and the lessons gleaned-e.g., optimizer-performance depending on the learning rate schedule and the effect of large batch sizes on DL training schedules-have been extensively explored in numerous previous works (see below for recent examples).  For the latter, the paper narrowly focuses on the results in previously published papers, rather than, for instance, showing how to optimize existing solvers for general downstream tasks.  Furthermore, there is no cohesive method presented with which readers may takeaway to aid them, i.e., how can these lessons help researchers develop new methods to push the state-of-the-art?  \n\nUltimately, the paper raises an important question \"comparing optimizers is difficult since hyperparameter tuning tends to dominate the comparisons.\"  However, the paper does not provide an answer or general method to attack this difficult problem.\n\nOther comments:\n----------------------\n\"In particular, we show that the optimal shape of the learning rate schedule is optimizer-dependent (in\naddition to the scale), and that differences in the schedule can dominate optimizer comparisons at\nsmaller step budgets and become less important at larger step\nbudgets.\" <- This has been observed in several previous studies.E.g., see:\n-Lee, Namhoon, et al. \"Understanding the effects of data parallelism\nand sparsity on neural network training.\" arXiv preprint\narXiv:2003.11316 (2020).\n-Schmidt, Robin M., Frank Schneider, and Philipp Hennig. \"Descending\nthrough a crowded valley-benchmarking deep learning optimizers.\"\nInternational Conference on Machine Learning. PMLR, 2021.\n\n\"they are loose upper bounds\non the worst case behavior of the algorithms\" <- You have not\nshown the lack of tightness (i.e., achievability), how have\nyou characterized these bounds as loose?\n\n\"Although there are heuristics for adjusting the learning rate as the batch size changes, these heuristics inevitably\nbreak down sufficiently far from the initial batch size and it is also not clear how to apply them to other training\nhyperparameters (e.g. momentum).\" <- The former claim is too broad;\nthere are importance sampling based approaches which allow automatic\nlearning rate adjustments, e.g.:\nJohnson, Tyler B., and Carlos Guestrin. \"Training deep models faster\nwith robust, approximate importance sampling.\" Advances in Neural\nInformation Processing Systems 31 (2018): 7265-7275.\n\n-Please include more discussion (in the main paper) of the Appendix\n results displaying LARS performance with and without the\n \"optimization tricks and non-default values of uncommonly-tuned\n hyperparameters.\"\n\n\"our BERT result\nwith Adam also improves upon the wall-time record of LAMB reported in\nYou et al. 2019)\" <- I agree that reporting training speed in steps is\na fair comparison between different training schedules and solvers.\nHowever, due to compute environment issues, comparing wall-times\nreported in another paper is not a fair comparison (the LAMB\nexperiment from You et al 2019 would have to be repeated on the same\ncompute environment used to run the presented Adam experiments).\n\n\"Moving forward, if we are to reliably make progress we need to rethink how we compare and evaluate\nnew optimizers for neural network training.\" <- Once again, the paper focuses on asking this question, rather than providing answers to it.\n\n\"However, achieving perfect scaling is not always straightforward. Changing the batch size changes\nthe training dynamics, requiring the training hyperparameters (e.g. learning rate) to be carefully\nre-tuned in order to maintain the same level of validation\nperformance.\" <- Citation required for this claim\n\n\"In addition, smaller batch sizes provide implicit regularization from\ngradient noise that may need to be replaced by other forms of\nregularization when the batch size is increased.\" <- Citation required\nfor this claim\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "13"}, "extra_text": {}}, {"review_id": "cs7di7MJwy", "reviewer": "Reviewer_x9rX", "summary": "The paper performs a detailed hyperparameter study of stochastic optimizers specifically designed for large-batch training (LARS and LAMB). With appropriate hyperparameter tuning, simpler optimizers such as Nesterov SGD and Adam match and sometimes outperform those specialized optimizers on ResNet-50 and BERT. The authors attribute the relative success of large-batch methods to minute undocumented aspects in their implementations, such as regularization and scaling on certain operators, which can be back-ported to the other optimizers, and advocate for more relaxed definitions (e.g. number of steps) for benchmarks such as MLPerf Training.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper neatly showcases the feebleness of highly-tuned stochastic optimization schedules on two standard deep learning benchmarks. While the points of the paper are mention-worthy, there is nearly no scientific contribution in the paper. Even as an empirical study, it hard to draw tangible conclusions from the paper. The only takeaway from the paper is that LAMB/LARS are very well-tuned, and simpler optimizers' hyperparameter schedules could also be aggressively tuned to achieve the same result.\n\nThe paper reads like a direct attack on LAMB and LARS, though the same claims could be made on other optimizers, including Adam and its hyperparameters (with enough tuning, one could probably attain similar convergence properties to Adam with momentum SGD). Similar claims about benchmarking stochastic optimizers w.r.t. tunability have already been made, e.g. by Sivaprasad et al. in \"Optimizer Benchmarking Needs to Account for Hyperparameter Tuning\", which is not cited here. \n\nIn the paper, there is very limited (sometimes absent) theoretical/empirical reasoning for the importance of certain parameter values: for example, in line 169: \"we found that choosing $\\gamma_0$ between 0.0 and 1.0 was important\" --- why is that? What happens to the training process otherwise? Additionally, would the tuned Nesterov parameters for ResNet-50 transfer to other neural architectures (such as ViT), or other datasets? \nAs a reader, I would be interested in a sensitivity analysis of the hyperparameters (or of the same algorithm to varying neural architectures/datasets), theoretical justification, or some sort of guidelines on how to tune those parameters in reasonable time (i.e. without exploring the entire space). \n\nIn-depth comments:\n\nSince most of the paper is spent discussing LAMB and LARS, I find it strange that their update rule definitions cannot be found in the manuscript.\n\nLine 322: \"Training with larger batch sizes is not fundamentally unstable: stringent step budgets make hyperparameter tuning trickier.\" --- Even with an unlimited step budget, it was shown on multiple occasions (with theoretical justification) that larger batch sizes tend to overfit.\n\nWhat about non-Nesterov momentum SGD? is there a configuration that works similarly well? If not, why not?\n\nLine 164: moving averages not explained well in the context of batch normalization.\n\nThroughout: the paper uses too many footnotes to my taste, which detracts from the flow of the text.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The answer to question 3d in the checklist is not satisfactory. Given that this is a work that performs extensive hyperparameter tuning on compute-intensive datasets, the time and overall energy spent on training are of utmost importance to report.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "QYpdJi0Afu", "reviewer": "Reviewer_St34", "summary": "In this work, the authors compared the standard optimizers (i.e., Nesterov momentum, Adam) and optimizers with layer-wise normalization (i.e., LARS, LAMB) in training neural networks with large-batch sizes.\nAlthough LARS and LAMB were proposed and known as \"large-batch optimizers,\" the authors showed that the standard optimizers with careful hyperparameter tuning could match or exceed the state-of-the-art results by LARS (for training ResNet-50 on ImageNet, batch size = 32K) and LAMB (for pre-training BEAT, batch size = 65K) at large-batch settings within the same step budgets.\nThe authors pointed out that the strong results depend on several optimization/regularization tricks, non-default values of uncommonly-tuned hyperparameters, and a careful learning rate schedule, but such details are not often discussed. With this point, the difficulties of comparing optimizers and the importance of rigorous empirical comparisons and open-sourcing in deep learning optimizer research are highlighted.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-organized, and the background, experimental methods, results, and lessons are clearly presented.\nBased on their rigorous empirical comparisons, they showed the standard optimizers (i.e., Nesterov momentum, Adam) could much or exceed the results of the optimizers designed for large-batch settings (i.e., LARS, LAMB).\nSince it has been believed that we cannot use such standard optimizers in large-batch settings, their results are surprising and make us re-examine how we should compare optimizers and whether we really need new optimizers. \nAlthough their results are only in large-batch settings, it is expected that we can observe the same phenomena (i.e., standard optimizers can meet or exceed the results of optimizers tailored for a specific task) in other settings.\nThis observation is beneficial, especially for practitioners.\nIn particular, I agree with the authors' claim: the importance of uncommonly-tuned hyperparameters, learning rate schedule, and disentangling optimization speed and regularization has not been discussed in detail.\nAnd I appreciate this work for shedding light on this point with convincing empirical results.\nI believe the community should appreciate the points made in this paper. \n\nMinor comments:\n- L93: the citation to Schmidt et al. [2020] is repeated.\n\n++++++++++++++++++++++++++++++++\n\nUpdate after seeing the other reviews and the authors' response\n\nAs some reviewers (and the authors) mentioned, the importance of hyperparameter tuning and the fact that “there is no clear winner” are already well-known (and this is why standard optimizers such as Momentum SGD and Adam are still the most popular choices because of their simplicity). However, LARS/LAMB has been used as the first choice in large-batch settings, but there is not enough evidence and discussion that it generally works better than the standard optimizers. This paper is the first study to point this out and show that the standard optimizers still work even in large-batch settings.\n\nFor the above reason, I believe this paper provides an important lesson for readers, and I wish to keep recommending to accept. But I also agree with other reviewers that \"contribution is too narrow\" (the readers interested in large-batch training may be limited.) So I updated my score from 8 to 7. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5 hours"}, "extra_text": {}}, {"review_id": "FB3YhBXUwHO", "reviewer": "Reviewer_tvWs", "summary": "Authors study the problem of large-batch training of neural networks and claim an important (negative) result: specialized large-batch algorithms are not necessarily better than traditional ones. Authors consider two standard optimizers (Adam and SGD w/ Nesterov momentum) and find hyperparameter configurations under which these standard optimizers match and sometimes outperform LAMB/LARS for large-batch training of ResNet50 and BERT respectively.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Disclaimer:** I am writing this (probably) additional review without reading prior discussion and other reviews. My review may or may not raise questions that were already addressed in prior discussion.\n\nThis was a difficult paper to review. On one hand, it claims an important negative result that may affect an entire line of prior work. At the current state of the field, we badly need more papers like this to reduce the “research debt”, and hence the complexity of future research in the field. On the other hand, this paper is lacking in the way it supports its main claim, and overall the paper is not refined enough to recommend acceptance to NeurIPS. I would also argue that it does not stand up to the standards that authors themselves suggest in Section 6. For simplicity, I summarize strong and weak points of the paper in two separate sections.\n\n## Reasons to accept\n* Outstanding technical competence: authors work with highly tuned baselines with great attention to detail; L279-286 is pure brilliance\n* Reporting the search space chronology (Appendix D.4) is a great practice that deserves wider adoption.\n\nPresentation-wise, the paper is clearly written and easy to read, even though it follows a somewhat unconventional structure. The only recommendation I can give is to avoid wrapping figures with low-margin text (e.g. Figure 4). That said, this is an optional request that did not affect my score.\n\n## Reasons to reject\n\nMy main issue with this paper is that it leaves an open question where it was clearly possible to give an answer with strong evidence. As authors rightfully state in L5-6, “it is an open queston” whether LARS/LAMB has any benefit for large batches. Unfortunately, providing specific configurations of hyperparameters where SGD/Adam outperforms LARS does not make us much closer to answering that question.\n\nThere is a wide range of prior work [1,2,3] on properly benchmarking optimizers and their sensitivity to hyperparameters. For instance, [1] proposes a general approach that could have been used to test whether or not LAMB requires less hyperparameter tuning than Adam, as authors wonder in L336-340. In fact, it would not require much more compute than what was used for finding the optimal hyperparameters. Some prior work[2] already measured the tuning-aware efficiency of the same optimizers and demonstrates similar issues with LARS/LAMB, although not in a large-batch setting. It comes to the same conclusion that none of the specialized algorithms consistently outperform SGD/Adam.\n\nTo summarize, this paper does indeed propose novel and potentially important “anecdotal” evidence, but I would expect a more rigorous analysis from a high profile venue such as NeurIPS. Aside from that, the discussion and conclusions made in the paper are insightful, but already well-known: the importance of hyperparameter configuration is better addressed in [1,2,3], the fact that an optimizer is also an indirect regularizer is also known [4].\n\n\n## Questions, minor concerns\n\n**MLPerf closed division**\nAuthors refer to the closed division multiple times; however, there is a separate “open” division in MLPerf where participants can tune the optimizer hyperparameters. Currently, the submissions on that track use either slightly modified LAMB or a completely different approach based on second order optimization[5]. If there is some intuitive explanation as for why these submissions still rely on LAMB?\n\n**Nesterov momentum**\nAs far as I understood, you use nesterov momentum for ResNet while baselines and other experiments use traditional momentum (or Adam’s “momentum” for Adam/LAMB). Based on your reasoning in S2 this may be important. To better convey the importance of this change of this change\n- Does LARS benefit from using Nesterov momentum instead of classical one? (or explain why it is impossible to modify LARS with nesterov momentum)\n- [optional] Does Adam/LAMB benefit from switching to Nadam-like momentum?\n\n**Hyperparameter stability to batch size**\nDo configurations A/B maintain their advantage over LARS if we apply them to adjacent batch sizes without re-tuning? (e.g. 65536 samples, 16384 samples)\n\n**Page 1 footnote:** \"... these heuristics inevitably break down ...\"\n[optional] this would benefit a citation or further discussion\n\n**Maybe typo:** Table 14, 15: t_{decay} is 2.815 is reported with a dot, as if it is approximately 3 steps. Did you mean 2,815?\n\n[1] Optimizer Benchmarking Needs to Account for Hyperparameter Tuning. Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, François Fleuret. 2019\n\n[2] How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers. Yuanhao Xiong, Xuanqing Liu, Li-Cheng Lan, Yang You, Si Si, Cho-Jui Hsieh. 2020\n\n[3] Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers. Robin M. Schmidt, Frank Schneider, and Philipp Hennig, 2020\n\n[4] Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks. Yuanzhi Li, Colin Wei, Tengyu Ma. 2019\n\n[5] Scalable Second Order Optimization for Deep Learning. Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, Yoram Singer. 2020\n\np.s. regardless of whether the paper is accepted, I must express my deepest respect for pinpointing the widespread implementation issues with Adam and learning rate decays in BERT.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is a fundamental paper that critiques large-batch optimizers. It is difficult to forsee its potential societal impact. Thus, I have no issue with how authors address it.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "8FcCt2WYNNZ", "reviewer": "Reviewer_amk6", "summary": "This is a very interesting work focused on a critical examination of LARS and LAMB optimizers, and the comparison of these with more established \"standard\" optimizers such as Adam.  The authors' results and analysis are insightful and of wide interest to the deep learning community.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "FIrstly, the focus of this work is timely, relevant, and well motivated, making it significant and of interest to the deep learning community.  The quality of writing is good, so quality standards are high in this respect too.  The main contribution comes in the form of a series of insights pertaining to why LARS and LAMB perform well (or appear to), when they do not, and how they should be meaningfully compared with Adam and the like.  The authors' arguments are convincing as is the evidence presented.  Overall, this is a high quality paper which should be accepted. The key contributions are summarized well and succinctly by the authors themselves, so I quote:\n\n\"standard optimizers, without any layer-wise normalization techniques, can match or exceed the large batch size results used to justify LARS and LAMB. Future work attempting to argue that a new algorithm is useful by comparing to baseline methods or results [...] faces a key challenge in showing that the gains are due to the new method and not merely due to better tuning or changes to the training pipeline\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.75"}, "extra_text": {}}, {"review_id": "-zdx7rTZZoD", "reviewer": "Reviewer_Ucu1", "summary": "I carefully read the responses from the authors. However, the authors did not address my concern. I'd like to keep my score.\n\n###########################################################################################################\n\nThe authors used a huge amount of computing resources to tune hyperparameters of Adam/SGD and claimed that they can match the performance of LARS/LAMB for large-batch training. I think the comparison is not fair.", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The performance of LARS will be much higher if they use the same amount of computing resources to tune hyper-parameters.\n\nFor ImageNet, the authors did not report the results of a larger batch size (e.g. 64K or 128K).\nMy experiments show that the performance of LARS would be much better than Nesterov for larger batch size.\n\nWith unlimited computing resources, the performance of LARS will still be better than Nesterov. \nThe authors probably will eventually get results like Figure 7 of https://arxiv.org/pdf/2006.08517v1.pdf\n\n\nIf we take a look at Table 1, we can see that the hyper-parameters of Nesterov are meticulously selected or cherry-picked.\nSince the tuning cost is so high, it is almost impossible for users to consider this solution.\n\nFor example, to my knowledge, the authors of LARS did not tune the hyper-parameters of Batch Normalization.\nThe authors of this paper used a lot of computing resources to tune the hyper-parameters of Batch Normalization.\n\n\n\"We ran a series of experiments, each of which searched over a hand-designed hyper-parameter search space using quasi-random search [Bousquet et al., 2017].\"\n\nTo my knowledge, the LARS authors did not use any advanced hyper-parameter tuning methods. This means the comparison is not fair.\n\n\n\"Table 2 shows that Nesterov momentum achieves higher training accuracy than LARS, despite similar validation performance. Thus, it may be more appropriate to consider the layerwise normalization of LARS to be a regularization technique, rather\nthan an optimization technique.\"\n\nThis conclusion is doubtful. Firstly, the authors probably need to sample many different convergence points (including low-accuracy points) to study of optimization pattern of LARS/Nesterov. \nSecondly, the generalization performance of the different minima may have different implicit properties, which can not be explained in such a simple way.\n\n\nBTW, I can't reproduce the authors' results even though I totally trust the correctness of this paper.\nThe reason is that we often need to use different hyper-parameters on different systems/hardware for large-batch training.\nA high tuning cost makes this process very hard.\n\n\nFor BERT, I suspect the authors tuned the hyper-parameters of fine-tuning process, which means the comparison is very unfair as the LAMB authors did not tune it.\n\nEven if the authors did not tune the fine-tuning process, this comparison is still unfair because LAMB can work without tuning for changing batch size.\n\nAs mentioned by the LAMB authors (caption of Table 4 in https://openreview.net/pdf?id=Syx4wnEtvH): \"We can achieve an even higher F1 score if we manually tune the hyperparameters\"\n\nThe LAMB authors reported untuned results in Table 4 and Table 5 of https://openreview.net/pdf?id=Syx4wnEtvH\n\nThe authors of this paper not only tuned common hyper-parameters (e.g. learning rate) but also tuned some uncommon hyper-parameters (e.g. beta1, beta2 in Adam).\n\nI suspect only big companies can do this kind of hyper-parameter searching for BERT per-training.\n\n\nThe novelty of this paper is very low. The key technical part is just tuning hyper-parameters. The authors also did not provide any deep analysis.\nIf we don't need deep analysis, there are actually many blogposts on this topic (e.g. https://medium.com/fenwicks/tutorial-2-94-accuracy-on-cifar10-in-2-minutes-7b5aaecd9cdd).\nThey can also significantly improve the performance of SGD with different optimization tricks.\nSimilar tricks will likely improve the performance of LARS/LAMB.\n\nThe key advantage of this paper is that the authors can use a significant amount of computing resources while other researchers can not. \nI worry that this paper may mislead the ML community and have a negative impact on academia's budget/planning.\n\nIn my humble opinion, this paper is not a fit to top ML conferences like NeurIPS/ICML/ICLR.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The novelty of this paper is very low. The key technical part is just tuning hyper-parameters. The authors also did not provide any deep analysis.\nIf we don't need deep analysis, there are actually many blogposts on this topic (e.g. https://medium.com/fenwicks/tutorial-2-94-accuracy-on-cifar10-in-2-minutes-7b5aaecd9cdd).\nThey can also significantly improve the performance of SGD with different optimization tricks.\nSimilar tricks will likely improve the performance of LARS/LAMB.\n\nThe key advantage of this paper is that the authors can use a significant amount of computing resources while other researchers can not. \nI worry that this paper may mislead the ML community and have a negative impact on academia's budget/planning.\n\nIn my humble opinion, this paper is not a fit to top ML conferences like NeurIPS/ICML/ICLR.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9 hours"}, "extra_text": {"ethical_concerns": "The key advantage of this paper is that the authors can use a significant amount of computing resources while other researchers can not. \nI worry that this paper may mislead the ML community and have a negative impact on academia's budget/planning.\n"}}]

Author rebuttal:

Meta-review: This paper provides a critical examination of the LARS and LAMB optimizers, and a comparison to the more established optimizers such as momemtum or Adam. The paper provides an extensive set of experiments to show that Nesterov momentum and Adam can match or exceed the results of LARS and LAMB when using large batch sizes. Momentum techniques are known to have problems with high-levels of noise so this in some sense provides another empirical confirmation that these methods can work well with moderate amounts of noise.

There is a wide variance in the scores attributed by the different reviewers. Overall, the contribution of the paper is quite narrow and focused on a specific problem. The discussion of related work is very incomplete. I don't see any reference to the optimization literature on stochastic momentum methods, including for instance: Mahmoud Assran and Michael Rabbat. On the convergence of nesterov’s accelerated gradient method in stochastic settings. arXiv preprint arXiv:2002.12414, 2020. Nicolas Loizou and Peter Richtárik. Momentum and stochastic momentum for stochastic gradient, newton, prox- imal point and subspace descent methods. arXiv preprint arXiv:1712.09677, 2017.
(and many others)

Finally, some reviewers are very critical regarding some of the details of the empirical evaluation. In response, the authors provided more follow-up experimental results. The changes required are I believe too significant and require another round of review.

Overall, I therefore do not recommend acceptance at this stage.


Decision: Reject