A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
Authors: Zachary Nado
Justin Gilmer
Christopher J Shallue
Rohan Anil
George Edward Dahl
Venue: ICLR 2022 Submitted (2022)
Keywords: neural networks
deep learning
neural network optimization
hyperparameter tuning
optimizer comparison

Abstract: Recently the LARS and LAMB optimizers have been proposed for training neural networks faster using large batch sizes. LARS and LAMB add layer-wise normalization to the update rules of Heavy-ball momentum and Adam, respectively, and have become popular in prominent benchmarks and deep learning libraries. However, without fair comparisons to standard optimizers, it remains an open question whether LARS and LAMB have any benefit over traditional, generic algorithms. In this work we demonstrate that standard optimization algorithms such as Nesterov momentum and Adam can match or exceed the results of LARS and LAMB at large batch sizes. Our results establish new, stronger baselines for future comparisons at these batch sizes and shed light on the difficulties of comparing optimizers for neural network training more generally.

Under review as a conference paper at ICLR 2022
A LARGE BATCH OPTIMIZER REALITY CHECK: TRADITIONAL, GENERIC OPTIMIZERS SUFFICE
ACROSS BATCH SIZES
Anonymous authors Paper under double-blind review
ABSTRACT
Recently the LARS and LAMB optimizers have been proposed for training neural networks faster using large batch sizes. LARS and LAMB add layer-wise normal- ization to the update rules of Heavy-ball momentum and Adam, respectively, and have become popular in prominent benchmarks and deep learning libraries. How- ever, without fair comparisons to standard optimizers, remains open question whether LARS and LAMB have any benefit over traditional, generic algorithms. this work demonstrate that standard optimization algorithms such as Nesterov momentum and Adam can match or exceed the results of LARS and LAMB at large batch sizes. Our results establish new, stronger baselines for future comparisons at these batch sizes and shed light on the difficulties of comparing optimizers for neural network training more generally.
INTRODUCTION
In recent years, hardware systems employing GPUs and TPUs have enabled neural network training programs to process dramatically more data in parallel than ever before. The most popular way to exploit these systems is to increase the batch size in the optimization algorithm (i.e. the number of training examples processed per training step). On many workloads, modern systems to larger batch sizes without significantly increasing the time per step (Jouppi et al. 2017] Wang al. 2019), thus proportionally increasing the number of training examples processed per second. If researchers can use this increased throughput to reduce the time required to train each neural network, then they should achieve hetter results by training larger models, using larger datasets, and by exploring new ideas more rapidly.
scale
As the capacity for data parallelism continues to increase, practitioners can take their existing, well-tuned training configurations and re-train with larger batch sizes, hoping to achieve the same performance in less training time (e.g. Ying et al., 2018). On an idealized data-parallel system with negligible overhead from increasing the batch size, they might hope to achieve perfect scaling, proportional reduction in training time as the batch size increases.
However, achieving perfect scaling is not always straightforward. Changing the batch size changes the training dynamics, requiring the training hyperparameters (e.g. learning rate) be carefully re-tuned in order to maintain the same level of validation performance In addition, smaller batch sizes provide implicit regularization from gradient noise (Smith et al. 2021) that may need to be replaced by other forms of regularization when the batch size is increased. Finally, tuning, increasing the batch size eventually produces diminishing returns. After a critical batch size, the number of training steps cannot be decreased in proportion to the batch size the number of epochs must increase to match the validation performance of the smaller batch size. See Shallue et al.) for survey of the effects of data parallelism neural network training. Once these effects taken into account, there is no strong evidence that increasing the batch size degrades the maximum achievable performance on any workload. At the same time, the ever-increasing capacity for data parallelism presents opportunities for new regularization techniques that can replace the gradient
with perfect
1. Although there are heuristics for adjusting the learning rate as the batch size changes, these heuristics inevitably break down sufficiently far from the initial batch size and it is also not clear how apply them to other training hyperparameters (e.g. momentum), detailed in (hallue al. 2019

---

Under review as a conference paper at ICLR 2022
noise of smaller batch sizes and new optimization algorithms that can extend perfect scaling to larger batch sizes by using more sophisticated gradient information (Zhang et al. 2019).
You et al. (2017) proposed the LARS optimization algorithm in the hope of speeding up neural network training by exploiting larger batch sizes. LARS is a variant of stochastic gradient descent (SGD) with momentum (Polyak 1964) that applies layer-wise normalization before applying each gradient update. Although it is difficult to draw strong conclusions from the results presented in the LARS paper. P the MLPer Training benchmark adopted addotarraadopted LARS as one of two allowed algorithms
MLPerrTraining
in the closed division for ResNet-50 on ImageNet and it became the de facto standard algorithm for that benchmark task. With MLPerf entrants competing to find the mastest-training hyperparameters for LARS, the first place submissions in the two most recent MLPerf Training competitions used LARS to achieve record training speeds with batch sizes of 32,678 and 65,536, respectively. No publications or competitive submissions to MLPerf have attempted to match these results with standard optimizer (e.g. Momentum or Adam). However, MLPerf entrants do not have strong incentive (nor are necessarily permitted by the rules) to explore other algorithms because MLPerf Training is a systems benchmark that requires algorithmic equivalence between submissions to make fair comparisons. Moreover, since the main justification for LARS is its excellent performance on ResNet-50 at large batch sizes, more work is needed to quantify any benefit of LARS over standard algorithms at any batch size.
You et al. (2019) later proposed the LAMB optimizer to speed up pre-training for BERT (Devlin et al. (0188) using larger batch sizes after concluding that LARS was not effective across workloads. LAMB is a variant of Adam (Kingma & Ba 2014) that adds a similar layer-wise normalization step to LARS. You et al. (2019) used LAMB for BERT pre-training with batch sizes up 65,536 and claimed that Adam cannol match the performance of LAMB beyond batch size 16,384.
In this paper, we demonstrate that standard optimizers, without any layer-wise normalization tech- niques, can match or improve upon the large batch size results used to justify LARS and LAMB. In Section [] we show that Nesterov momentum [Nesterov] [983) matches the performance of LARS on the ResNet-50 benchmark with batch size 32,768. We are the first to match this result with standard optimizer. In Section D contradicting the claims in You et al. (201)) we show that Adam obtains better BERT pre-training results than LAMB at the largest batch sizes, resulting in better downstream performance metrics after fine-tuning.
In addition, we establish new tate-of-theearr for BERT pretraining speed, reaching an F1 score of 90.46 in 7,818 steps using Adam at batch size 65,536 (we report training speed in steps because our focus is algorithmic efficiency, but since we compare LARS and LAMB to simpler optimizers, fewer training steps corresponds to faster wall-time in an optimized implementation our BERT result with Adam also improves upon the wall-time record of LAMB reported in You et al. 2019). Taken together, our results establish stronger training speed baselines for these tasks and batch sizes, which we hope will assist future work aiming accelerate training using larger batch sizes.
In addition to the contributions mentioned above, we demonstrate several key effects that are often overlooked by studies aiming establish the superiority of new optimization algorithms. We show that future work must carefully disentangle regularization and optimization effects when comparing a new optimizer to baselines. We also report several -oduumdeeeeeteed details used to generate the best LARS and LAMB results, a reminder that future comparisons should document any novel tricks and include them in baselines. Finally, our results add to existing evidence the literature on the difficulty of performing independently rigorous hyperparameter tuning for optimizers and baselines. In particular, we show that the optimal shape the learning rate schedule is optimizer-dependent (in addition to the scale), and that differences in the schedule can dominate optimizer comparisons at smaller step budgets and become less important at larger step budgets.
1.1 RELATED WORK
Shallue et al. 2019) and Zhang et al. (019 explored the effects of data parallelism on neural network training for different optimizers, finding no evidence that larger batch sizes degrade performance and
The modified AlexNet on ImageNet benchmark did not have well-established accuracy targets from prior work and LARS used more general learning rate schedule than the momentum baseline. For ResNet-50 on ImageNet, LARS achieved sub-par accuracy numbers and was not compared any other optimizer the same batch size, lcaving open the possibility that generic optimizer would scale just as well as LARS. ML.Prrf is a trademark of MLCommons.org.
2
overview
2

---

Under review as a conference paper at ICLR 2022
demonstrating that different optimizers can achieve perfect scaling up to different critical batch sizes. Geiping et al. (202I) found that full batch gradient descent can achieve competitive performance on CIFAR-IO I0. You et al. (2017) 2019) developed the LARS and LAMB optimizers in the hope of speeding up training by achieving perfect scaling beyond standard optimizers. Many other recent papers have proposed new optimization algorithms for generic batch sizes larger batch sizes (see Schmidt et al. 2020). Choi et al. (2019), Sivaprasad et al. (2020). and |Schmidt et al. (2020) demonstrated the difficulties with fairly comparing optimizers, showing that the hyperparameter tuning protocol is a key determinant of optimizer rankings. The MLPerf Training benchmark (Mattson al., 2 provides a competitive ranking of neural network training systems, but does not shed much light on the relative performance of optimizers because entrants are limited in the algorithms they can use and the hyperparameters they can tune.
2
MATCHING LARS ON IMAGENET
The MLPerf training benchmark for ResNet-50 v1.5 on ImageNet (Mattson et al. 2019) aims to reach 75.9% validation accuracy in the shortest possible wall-clock time. In the closed division of the competition, entrants must choose between two optimizers, SGD with momentum or LARS, and are only allowed to tune a specified subset of the optimization hyperparameters, with the remaining hyperparameter values set by the competition rules F The winning entries in the two most recent competitions used LARS with batch size 32,768 for 72 training epoch and LARS with batch size 65,536 for 88 training epochs respectively. Kumar et al. (2019) later improved the training time for batch size 32,768 by reaching the target accuracy in 64 epochs. These are currently the fastest published results on the ResNet-50 benchmark. However, it has been unclear whether LARS was necessary to achieve these training speeds since no recent published results or competitive MLPerf submissions have used another optimizer. In this section, we describe how we matched the 64 epoch, 32,768 batch size result of LARS using standard Nesterov momentum
A fair benchmark of training algorithms or hardware systems must account for stochasticity in individual training runs. In the MLPerf competition, the benchmark metric is the mean wall-clock time of 5 trials after the fastest and slowest trials are excluded. Only 4 out of the 5 trials need to reach the target accuracy and there is no explicit limit on the number of times an entrant can try a different set of trials. Since our goal is to compare algorithms, rather than systems, we aim to match the LARS result in terms of training steps instead (but since Nesterov momentum is computationally simpler than LARS, this would also correspond to faster wall-clock time on an optimized system). Specifically, we measure the median validation accuracy over 50 training runs with a fixed budget of 2,512 training steps at batch size of 32.768. When we ran the published LARS training pipeline To) LARS achieved a median accuracy of 75.97% and reached the target in 35 out of 50 trials. We consider the LARS result be matched by another optimizer if the median over 50 trials exceeds the target of 75.9%.
2.1 NESTEROV MOMENTUM AT BATCH SIZE 32K
This section describes how we used the standard Nesterov momentum optimizer to train the ResNet- 50 v1.5 on ImageNet to 75.9 75.9% validation accuracy in 2,512 update steps a batch size of 32,768, matching the best published LARS result at this batch size. Although we implemented our own training program, the only logical changes we made to the published LARS pipeline were to the optimizer and the optimization hyperparameters. Our model implementation and data pre-processing pipeline were identical to those required under the MLPerf closed division rules (see Appendix (B)
We present two Nesterov momentum hyperparameter configurations that achieve comparable per- formance to LARS. Configuration achieved a median accuracy of 75.97% (the same as L.ARS) and reached the target accuracy in 34 out of 50 trials. Configuration B is modified version of Configuration A designed to make as few changes as possible to the LARS hyperparameters;
https://git.io//tkk
The 88 epoch, 65,536 batch size re- sult is taster terms of wall-clock time but requires more trainin epochs, indicating that it is beyond LARS's perfect scaling regime. Although LARS obtains diminishing returns when increasing the batch size from 32,768 65,536, future work could investigate whether Nesterov momentum drops off more or less rapidly than LARS. Corresponding to 64 training epochs in Kumar et al. (019))
10
io/Jtsl.
3

---

Under review as a conference paper at ICLR 2022
achieved a median accuracy of 75.92% and reached the target in 29 out of 50 trials. See Appendix E. for the complete hyperparameter configurations.
To achieve these results, we tuned the hyperparameters of the training pipeline from scratch using Nesterov momentum. We ran a series of experiments, each of which searched over a hand-designed hyperparameter search space using quasi-random search (Bousquet al. 2017). Between each experiment, we modified the previous search space and/or tweaked the training program to include optimization tricks and non-default hyperparameter values discovered in the state-of-the-art LARS pipeline. The full sequence of experiments we ran, including the number of trials, hyperparameters tuned, and search space ranges, are provided in appendix[E.4 Once we had matched the LARS result with Configuration A, we tried setting each hyperparumeter to its value in the LARS pipeline in order to find the minimal set of changes that still achieved the target result, producing Configuration B. The remainder of this section describes the hyperparameters we tuned and the techniques we applied on the journey to these results.
2.1.1 NESTEROV MOMENTUM OPTIMIZER
Nesterov momentum is a variant of classical or "heavy-ball" momentum defined by the update rule
V(+1 = fuùt + v((O.).
0++1 = 02 Th (pmtt1 + Ve(O.)),
where vo = 0, Or is the vector of model parameters after steps, Vε(O,) is the gradient of the loss function ((O) averaged over batch of training examples, is the momentum, and 7he is the learning rate for step t. We prefer Nesterov momentum over classical momentum because it tolerates larger values of its momentum parameter (Sutskever al. 2013) and sometimes outperforms classical momentum, although the two algorithms perform similarly on many tasks (Shallue et al.] 2019) Choi al. 2019). We tuned the Nesterov momentum μ in Configurations A and B. We discuss the learning rate schedule [mt) separately in Section 2.1.4
2.1.2 BATCH NORMALIZATION
The ResNet-50 v1.5 model uses batch normalization (loffe & Szegedy 2015), defined as x0() mean(x!))
()
vvar(z()) + x 70(0 3
where z!)) is a vector of pre-normalization outputs from layer 4, mean(-) and var(-) denote the element-wise sample mean and variance across the batch of training examples 11 and y(()) and gtl) are trainable model parameters.
Batch normalization introduces the following tuneable hyperparameters: e, the small constant added to the sample variance; the initial values of (0()) and g(0; and p. which governs the exponential moving averages of the scaling factors used in evaluation. The LARS pipeline uses e = 10-5 5 and P 0.9. It sets the initial value of (30h to 0.0 everywhere, but the initial value of (0)) depends on the layer: it sets (0)) to 0.0 in the final batch normalization layer of each residual block, and to 1.0 everywhere else. In Configuration A, we tuned c, p. and Yoo the initial value of 70()) in the final batch normalization layer of each residual block. In Configuration B, we used the same values as LARS for and p. but we found that choosing To between 0.0 and 1.0 was important for matching the LARS result with Nesterov momentum.
2.3.3 REGULARIZATION
In Configuration A, we tuned both the L2 regularization coefficient À and label smoothing coefficient T (Szegedy et al. 2016). The LARS pipeline uses À = 10 and T = 0.1. Crucially, the LARS pipeline does not apply L2 regularization to the bias variables of the ResNet model nor the batch normalization parameters (1) and 8(9) (indeed, the published LARS pipeline does not even apply LARS to these parameters it uses Heavy-ball momentum). This detail extremely important for both LARS and Nesterov momentum achieve the fastest training speed. Configuration B used the same À and 7 as Configuration A.
In distributed training environment the and variance commonly computed over subset of the full batch. The LARS pipeline uses a "virtual batch size" of 64, which we use avoid changing the training objective (Hoffer et al. 2017]
4

---

Under review as a conference paper at ICLR 2022
2.1.4 LEARNING RATE SCHEDULE
The LARS pipeline uses piecewise polynomial schedule
Nesterov
LARS
Pozmup
Thmit + (ppeak 1/niit
twarmup
Pwannup
1
T/
Thinal + (ppaa Thiaa
T
T-t
Plesor
twarmup
7ppcak
7.05
29.0
Thial
6 x 10
10
with Tinit = 0.0, Vpeakk = 29.0, Thámal = 10-1, Pwarmup 1, Pdecay 2. and Ewarmup 706 steps. In Configuration A, we re- tuned all of these hyperparameters with Nesterov momentum. In Configuration B. set 7nit Pdecay. and twarmup to the same values as LARS, changing only Pwarmup from 1 to 2 and re- scaling Upeak and T/mmm
0.02397
0.071
5.8 x 10-6
10-4
T
0.15
0.10
70
0.4138
0.0
Table 1: The hyperparameters of Configuration B that differ from state-of-the-art LARS batch size 32,768 (Kumar et al.) 2019).
2.1.5 COMPARING NESTEROV MOMENTUM AND LARS
Table [ shows the hyperparameter values for Configuration that differ from the state-of-the-art LARS pipeline. Aside from re-tuning the momentum, learning rate scale, and regularization hyperparameters (whose optimal values are all expected to change with the optimizer), the only changes are setting Pwarmup to 2 instead of and re-tuning 70-
Figure shows the LARS learning rate schedule compared to the Nes- terov momentum schedule. Even though these schedules are similar, we found that each optimizer had different optimal value of the warmup polynomial power. As Table 2 shows, Nesterov momentum performs better with Pwarmup 2 instead of 1. while the opposite is true with LARS. As discussed in |Agarwal et al. (2020), optimizers can induce implicit step size schedules that strongly influence their training dynamics and solution quality (Li et al. 2019), and it appears from Table that the implicit step sizes of Nesterov momentum and LARS may evolve differently, causing the shapes of their optimal learning rate schedules to differ.
305
1300
300
Figure
The learn- ing rate schedules of LARS and Nesterov mo- mentum Configuration B. Aside from re-scaling, the only difference is set- ting the warmup polyno- mial power to 2 instead of 1.
Although the main concern of practitioner is validation performance, the primary task of an optimization algorithm is to minimize training loss. Table that Nesterov momentum achieves higher training accuracy than LARS, despite similar validation performance. Thus, it may be more appropriate to consider the layerwise normalization of LARS to be regularization technique, rather than an optimization technique. This also explains its strong performance on ImageNet where regularization is critical (Bello et al. 2021).
Spending even more effort tuning LARS or Nesterov momentum would likely further improve the current state-of-the-art for that optimizer. Meaningful optimizer comparisons are only possible with independent and equally intensive tuning efforts, and we do not claim that either optimizer outperforms the other on this benchmark. That said, the main evidence for LARS's s utility as a "large-batcl" optimizer" is its performance this particular benchmark, then more evidence is needed to quantify any benefit it has over traditional, generic optimizers like Nesterov momentum.
Pwamup
Nesterov
LARS
Optimizer 78.97%
Train Acc
Test Acc
75.79%
75.97%
Nesterov
75.93%
75.92%
75.69%
LARS
78.07%
75.97%
Table 2: (Left) The best warmup schedule differs for Nesterov momentum and LARS. Values are medians over 50 training runs after setting Pwammup without retuning other hyperparameters. (Right) Median train and test accuracies over 50 training runs for Nesterov momentum Configuration B and LARS.
5

---

Under review as a conference paper at ICLR 2022
2.2 LESSONS LEARNED
In hindsight, it was only necessary to make a few changes to the LARS pipeline to match its performance at batch size 32,768 with Nesterov momentum. However, Table I does not accurately represent the effort required when attempting to match a highly tuned training- spedd benchmark.
Firstly, as described in Sections 2.1.2 and 2.1.3 the strong results of LARS depend partly on a few subtle optimization tricks and non-default values of imcommonly-tuned hyperparameters. Fortunately, in this case we could discover these tricks by examining the open-source MLPerf code, but machine learning research papers do not always report these important details. Researchers can easily waste experiments and produce misleading results before getting all of these details right. We demonstrate the importance of adding these tricks to our Nesterov momentum pipeline in Appendix C without these tricks (or some new tricks), we likely would not have been able to match the LARS performance.
Secondly, the learning rate schedule really matters when trying to maximize performance with relatively small step budget. Both LARS and Nesterov momentum are sensitive to small deviations from the optimized learning rate schedules in Figure I and neither schedule works as well for the other optimizer. Although relatively minor changes were sufficient to match LARS with Nesterov momentum, there is no way to know priori how the optimal schedule will look for new optimizer Wu et al. (2018). Even in toy settings where the optimal learning rate schedule can be derived, does not fit into commonly used schedule families and depends strongly on the optimizer ZZhng et al. (2019). Indeed, this problem applies to the other optimization hyperparameters as well: it is extremely difficult to know which are worth considering ahead of time. Finally, even when we narrowed down our hyperparemeter search spaces around the optimal point, the volume of our search spaces corresponding to near-pe near-peak performance was small, likely due to the small step budget (Shallue et al. 2019).
We investigate how these effects change with less stringent step budget in Section
3
STRONGER BERT PRETRAINING SPEED BASELINES
You et al. (2019) developed the LAMB optimizer in the hope of speeding up training for BERT-Large (Bidirectional Encoder Representations from Transformers, [Deliin al. 2018). BERT training consists of two phases. The "pretraining" phase has two objectives: (1) predicting masked tokens based on the rest of the sequence (a masked language model), and (2) predicting whether two given sentences follow one from another. Finally, the "fine-tuning" phase refines the model for a downstream task of interest. BERT pretraining takes a considerable amount of time (up to 3 days on 16 Cloud TPU-v3 chips Jouppi et al. (2017)) whereas the fine-tuning phase is typically much faster. Model quality is typically assessed on the downstream metrics, not on pretraining loss, making BERT training a somewhat awkward benchmark for optimization research.
You et al. 2019) used LAMB for BERT pretraining with batch sizes up to 65,536 and claimed that T.AMB outperforms Adam batch size 16,384 and beyond. The LAMB optimizer has since appeared in several NLP toolkits, including as Microsoft DeepSpeed and NVIDIA Multi-node BERT training, and as a benchmark task in MLPerf vo.7
As shown in Table S we trained Adam (with decoupled weight decay) baselines that achieve better results than both the LAMB and Adam results reported in You et al. 2019 Our new Adam baselines obtain better Fl scores on the development set of the SQuaD v1.1 task in the same number of training steps as LAMB for both batch size 32,768 and the hybrid 65.536-then-32,768 batch size training regime in You et al. 2019). We also ran Adam at batch size 65,536 to reach nearly the same F1 score as the hybrid batch size LAMB result, but in much fewer training steps. We believe 7,818 steps is a new state-of-the-an for BERT pretraining speed (in our experiments, it also improves upon the 76-minute record claimed in You al. 2019). Additionally, at batch size 32,768 our Adam baseline got a better pretraining loss of 1.277 compared to LAMB's B 1.342.
We used the same experimental setup as You et al. (2019). including two pretraining phases with max sequence lengths of 128 and then 512. In order to match You et al. (019). we reported the F1 score on the downstream SQual v1.1 task as the target metric, although this metric in- troduces potential confounds: optimization efficiency should be measured on the training task using training and held-out data sets. Fortunately, in this case better pretraining performance corre- lated with higher Fl score after fine-tuning, See Appendix B.2 for additional experiment details.
We do not consider the MI.Per task in this paper since it a warm-start, partial training task.
6

---

Under review as a conference paper at ICLR 2022
We tuned Adam hyperparameters independently for each pretraining phase, specifically learning rate 77. 31, $2, the polynomial power for the learning rate warmup Pwormup: and weight decay X, using quasi-random search (Bousquet et al. 2017). See Appendix E.2 for the search spaces.
In addition to hyperparmeter tuning, our im- proved Adam results at these batch sizes are also likely due to two implementation differences. First, the Adam implementation in You et al. (2019) comes from the BERT open source code base, in which Adam is missing the standard bias correction The Adam bias correction acts an additional step size warm-up, thereby potentially improving the stability in the initial steps of training. Second, the BERT learning rate schedule had a discontinuity at the start of the decay phase due to the learning rate decay being incorrectly applied during warm-up (see Figure in Appendix [B. This peculiarity is part of the official BERT release and is present in 3000+ copies of the BERT Traiinn code on GitHub.
Batch size
Step budget
LAMB
Adam
32k
15,625
91.48
91.58
65k/32k
8,599
90.58
91.04
65k
7,818
90.46
Table 3: Using Adam for pretraining exceeds the reported performance of LAMB in You et al. (2019) in terms of F1 score the downstream SQuaD v1.1 task.
4 INVESTIGATING A LESS STRINGENT STEP BUDGET
Part of what makes comparing optimizers so difficult is that the hyperparameter tuning tends to dominate the comparisons (Choi et al. 2019). Moreover, tuning becomes especially difficult when we demand a fixed epoch budget even when dramatically increasing the batch size (Shallue et al. 2019). Fixing the epoch budget as the batch size increases is equivalent to demanding perfect scaling (i.e. that the number of training steps decreases by the same factor that the batch size is increased). We can view the role of hyperparameter tuning for large batch training as resisting the inevitable end of perfect scaling. For example, it might be possible extend perfect scaling using delicately tuned learning rate schedules, but comparing optimizers under these conditions can make the learning rate schedule dominate the comparison by favoring some algorithms over others. Therefore, in order to better understand the behavior of LARS and LAMB compared to Nesterov Momentum and Adam, we ran additional ResNet-50 experiments with a more generous 6,000 step budget (vs 2,512 in Section ( and more simplistic cosine learning rate schedule. At batch size 32,768, this budget should let us reach better validation accuracy than the MLPerf target of 75.9%.
Although not mentioned in You et al. (0177) the state-of-the-art MLPerf pipeline for "LARS" actually uses both LARS and Heavy-ball Momentum, with Momentum applied to the batch normalization and ResNet bias parameters and LARS applied to the other parameters. You et al. (201)) does not mention whether LAMB was only applied to some parameters and not others. If layerwise normalization can be harmful for some model parameters, this is critical information for practitioners using LARS or LAMB, since it might not be obvious which optimizer to apply to which parameters. To investigate this, we trained both pure LARS and LAMB configurations, apply layerwise normalization to the batch normalization and ResNet bias parameters. Moreover, LAMB's 's underlying Adam implementation defaults e 10-6 rather than the typical 10-7 or
well configurations that did not
10 10-8, 8 In some cases, can be a critical hyperparameter for Adam (Choi et al. 2019). so we included Adam configurations with both = 10 - and 10
Table Ishows the validation accuracy of these different configurations after training for 6,000 steps with batch size 32,768. In every case, we used a simple cosine decay learning rate schedule and tuned the initial learning rate and weight decay using quasi-random search. We used momentum parameters of 0.98 for Nesterov momentum and 0.929 for LARS, respectively, based on the tuned values from Section 2 We used default hyperparameters for Adam and LAMB except where specified. We set all other hyperparameters to the same values as the state-of-the-ar LARS pipeline, except we set yo = 1.0. See Appendix B.33 for more details. As expected, highly tuned learning rate schedules and optimizer hyperparameters are no longer necessary with a less stringent step budget. Multiple optimizer configurations in Table exccee the MIPerf target accuracy of 75.9% at batch size 32,768 with minimal tuning. Training with larger batch sizes is not fundamentally unstable: stringent step budgets make hyperparameter tuning trickier.
https: pttppp///gtt... l/JttYd
14 See Scchttts://git.ioo wttpt///gg.t..oo///////// and httpss////tt..ioo/uegg
7

---

Under review as a conference paper at ICLR 2022
In Table 4 "pure LAMB" performs extremely poorly: LAMB only obtains reasonable results when it is nor used on the batch normalization and ResNet bias parameters, suggesting that lay- erwise normalization can indeed be harmful on some parameters. "Pure LARS" and Nesteroy momentum perform roughly the same at this step budget, but the MLPerf LARS pipeline, which is tuned for a more stringent step bud- get, does not use LARS on all parameters, at least suggesting that the optimal choice could be budget-dependent.
Weights
Bias/BN
Top-1
Optimizer
Optimizer
Nesterov
Nesterov
76.7
LARS
Momentum
76.9
LARS
LARS
76.9
Adam (c = 10-6) 8)
Adam (c = 10 8)
76.2
Adam (e = 10 10-")
Adam (6 = 10-0) ")
76.4
LAMB
LAMB
27.3
LAMB
Adam (e 10 8)
76.3
Many new neural net optimizers, including LAMB, are introduced alongside claims that they do not require any-or at least minimal- tuning. Unfortunately, these claims require lot of work to support, since they require try- ing the optimizer on new problems not used during the development of the algorithm. Al- though our experiments here are not sufficient to determine which optimizers are casiest to tune, experiments like those in Table 4 that operate outside the regime of highly tuned learning rate schedules can serve as a starting point, and we include preliminary "tunability" experiments using a simple grid search for LARS and Nesterov (details and more figures in Appendix D)..
LAMB
Adam (e = 10 10-6)
76.3
Table 4: Validation accuracy of ResNet-50 on Ima- geNet trained for 6,000 steps instead of 2,512. The second column is the optimizer that was applied to the batch norm and ResNet bias variables. We report the median top-1 accuracy over 5 seeds of the best hyperparameter setting a refined search space. See Appendix E.3 for details.
These grid search experiments sweep over 2 orders of magnitude of learning rate (1)) and weight decay (A), using 100 trials centered around the best settings from Table Figure shows the fraction of hyperparameter settings that reach a target validation accuracy. For very competitive accuracies (> 76.6%). LARS reaches the target slightly more often than Nesterov, but otherwise the optimizers similar or Nesterov more frequently reaches the target. Additionally, we analyze the best y for each À in the grid search, and Nesterov appears have an inverse relation- ship between and X while LARS has the benefit of an optimal 1) independent of X. This raises the question if we could make Nesterov "easier" to tune by reparameterizing the search space to be 17 by X. similar to how LARS incorporates À into its step size denominator. Overall, LARS and LAMB do not appear to have an advantage in how easy they are to tune even on dataset and model that were used in the development of both of those algorithms. LAMB a variant of Adam and performs about the same as Adam with the same value of in Table 4 LARS is more analogous to Momentum and indeed Nesterov momentum and LARS have similar performance.
The fraction of hyperparameter settings that achieve better than target validation error rate.
Figure 2:
DISCUSSION
Our results show that standard, generic optimizers suffice for achieving strong results across batch sizes. Therefore, any research program to create new optimizers for training at larger batch sizes must start from the fact that Momentum, Adam, and likely other standard methods work fine batch sizes as large as those considered in this paper. The LARS and LAMB update rules have no more to do with the batch size (or "large" batches) than the Momentum or Adam update rules. Although You et al. (2019) presented convergence rate bounds for LARS and LAMB to support their claims of superior performance, we show in Appendix A that Adam satisfies a similar bound to LAMB, These bounds all rely very unrealistic assumptions 15 Most of all, they are loose upper bounds on the worst case behavior of the algorithms, not accurate reflections of optimizer performance in reality. Whether layer-wise normalization can be useful for optimization regularization remains open
All convergence bounds assume no momentum is used, and the Lovg bound for LAMB also assumes 82 0. when it is typically 0.999. Additionally, Lavg could still be large if Loo is large, but we leave an empirical analysis of this to future work.
8

---

Under review as a conference paper at ICLR 2022
question. However, if LARS and LAMB have any advantage standard techniques, it is not that they work dramatically better on the tasks and batch sizes in You et al. (2017: 2019).
Our primary concern in this paper has been matching the state of the art-and establishing new baselines- for training speed measurements of the sort used to justify new techniques and algorithmss for training with larger batch sizes. In contrast, many practitioners the best possible validation error with a somewhat flexible training time budget. Part of the reason why matching LARS at batch size 32,768 was non-trivial is because getting state of the art training speed requires several tricks and implementation details that are not often discussed. It was not obvious to us a priori which ones would prove crucial. These details do not involve changes to the optimizer, but they interact with the optimizer in regime where all hyperparameters need to be well tuned to stay competitive, making it necessary to re-tune everything for a new optimizer.
more concerned with obtaining
In neural network optimization research, training loss is rarely discussed in detail and evaluation centers on validation/test performance since that what practitioners care most about. However, although we shouldn't It only consider training loss, it is counter-intuitive and counter-productive to elide a careful investigation of the actual objective of the optimizer. If new optimizer achieves better test performance, but shows no speedup on training loss, then perhaps it is not a better optimizer so much as an indirect regularizer. Indeed, in our experiments we found that Nesterov momentum achieves noticeably better training accuracy on ResNet-50 than the LARS configuration we used, despite reaching roughly the same validation accuracy. Properly disentangling possible regularization benefits from optimization speed-ups is crucial if we to understand neural network training, especially at larger batch sizes where we lose some of the regularization effect of gradient noise. Hypothetically, if the primary benefit of training procedure is regularization, then it would be better to compare the method with other regularization baselines than other optimizers.
Ultimately, we only care about batch size to the extent that higher degrees of data parallelism lead to faster training. New optimizers have the potential to dramatically improve algorithmic efficiency across multiple workloads, but our results show that standard optimizers can match the performance of newer alternatives on the workloads we considered. Indeed, despite the legion of new update rule variants being proposed in the literature, standard Adam and Momentum remain the workhorses of practitioners and researchers alike, while independent empirical comparisons consistently find no clear winner when optimizers are compared across a variety of workloads (Schmidt et al. 2020). Meanwhile, as Choi et al. (2019) and our results underscore, comparisons between optimizers crucially depend on the effort spent tuning hyperparameters for each optimizer. Given these facts, should regard with extreme caution studies claiming show the superiority of one particular optimizer over others.
6 CONCLUSION
In this work, demonstrated that standard optimizers, without any layer-wise normalization techniques, can match or exceed the large batch size results used to justify LARS and LAMB. Future work attempting argue that new algorithm is useful by comparing baseline methods results, including those established in this paper, faces a key challenge in showing that the gains are due to the new method and not merely due better tuning or changes the training pipeline (c.g. regularization tricks). Although gains from tuning will eventually saturate, we can, in principle, always invest more effort in tuning and potentially get better results for any optimizer. However, our goal should be developing optimizers that work better across many different workloads when taking into account the amount of additional tuning they require.
Moving forward, if we are to reliably make progress we need to rethink how we compare and evaluate new optimizers for neural network training. Given how sensitive optimizer performance is to the hyperparameter tuning protocol and how difficult it is to quantify hyperparameter tuning effort, can't 't expect experiments with self self-reported basclines to always lead to fair comparisons. Ideally, new training methods would be evaluated in standardized competitive benchmark, where submitters of new optimizers do not have full knowledge of the evaluation workloads. Some efforts in this direction have started, for instance the MLCommons Algorithmic Efficiency Working Groug [6] but more work needs to be done to produce incentives for the community to publish well-tuned baselines and to reward researchers that conduct the most rigorous empirical comparisons.
ht nttpps:/ /ml commons. http:///oooooorrrrree////rrrrrre
thms/

---

Under review as a conference paper at ICLR 2022
Reproducibility Statement
We will include a link to all code and all possible reproducibility instructions after the anonymized reviewing period is over. However, we extremely detailed about our tuning procedures and dataset details, see Appendices B F
We also include hosted Tensorboards for almost all of our tuning runs, available as ftensorboarr rd. dev links in the captions of the tables describing the hyperparameter searches. This allows readers to see, among other data, the exact points generated for each search in the "hparams" tab. All of this data can also be downloaded via the Tensorboard DataFrames API as CSV files or pd. DataameFrs for future analysis.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jía, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: /www www.seensor.. fow. org/ Software available from tensorflow.org.
Olah,
Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling adaptive gradient methods from learning rates. arXiv preprint arXiv:2002.10 1803, 2020.
Irwan Bello, William Fedus, Xianzhi Du, Ekin D Cubuk, Aravind Srinivas, Tsung- Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. arXiv 2021.
preprint
Olivier Bousquet, Sylvain Gelly, Karol Kurach, Olivier Teytaud, and Damien Vincent. Critical papee--arameters: No random, no cry. arXiv, 2017. URL 03200
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miin and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL hamm//oogle//pt
http:://iithie
Dami Choi, Christopher J Shallue, Zachary Nado, Jaehoon Lee, Chris J Maddison, and George E Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint 1111v:191..05446. 2019.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480 04805, 2018.
Jonas Geiping, Micah Goldblum, Phillip E Pope, Michael Moeller, and Tom Goldstein. Stochastic training is not necessary for generalization. arXiv preprint arXiv:2109.144 14019,
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the gen- eralization gap in large batch training of neural networks. arXiv preprini arXiv:1705.08744 2017.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 1502.0330 2015.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 1- 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
10

---

Under review as a conference paper at ICLR 2022
Sameer Kumar, Victor Bitorff, Dehao Chen, Chiachen Chou, Blake Hechtman, HyoukJoong Lee, Naveen Kumar, Peter Mattson, Shibo Wang, Tao Wang, et al. Scale mlperf-0.6 models on google tpu-v3 pods. arXiv preprint arXiv:1909.0971 2019.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. arXiv preprini vrXiv:1907.04555 2019.
Peter Mattson, Christine Cheng, Cody Coleman, Greg Diamos, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu- Wei, Peter Bailis, Victor Bittorf, David Brooks, Dehao Chen, Debojy- oti Dutta, Udit Gupta, Kim Hazelwood, Andrew Hock, Xinyuan Huang, Atsushi Ike, Bill Jia, Daniel Kang. David Kanter, Naveen Kumar, Jeffery Liao, Guokai Ma, Deepak Narayanan, Tayo Oguntebi, Gennady Pekhimenko, Lillian Pentecost, Vijay Janapa Reddi, Taylor Robie, Tom St. John, Tsuguchika Tabaru, Carole-Jean Wu, Lingjie Xu, Masafumi Yamazaki, Cliff Young, and Matei Zaharia. MIPerf training benchmark. arXiv preprint arXiv: 1910001500 2019. URL https://arxxi>> https//xxxivooog/aa/.... owg/ 1910. 01500
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate 0(1/k2)). In Dokl. akad. nauk Sssr, volume 269, pp. 543 1983.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR
Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Robin M Schmidt, Frank Schneider, and Philipp Hennig. Descending through a crowded valley- benchmarking deep learning optimizers. arXiv preprint arXiv:2007.01547 2020.
Christopher Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein. Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20 1-99, 2019.
Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, and Francois Fleuret. Optimizer benchmarking needs to account for hyperparameter tuning. In International Conference on Machine Learning, pp. 9036-9045. PMLR, 2020.
Samuel Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit
regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176. 2021.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. ICML, 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-26886, 2016.
Yu Emma Wang, GeoYeon Wei, and David Brooks. Benchmarking tpu, gpu, and cpu platforms for deep learning. arXiv preprint 1907 10701, 2019.
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse Understanding short-horizon bias in stochastic meta-optimization. arXiv preprint 1r03iv:8003.200 2018.
Chris Ying, Sameer Kumar, Dehao Chen, Tao Wang, and Youlong Cheng. Image classification at supercomputer scale. arXiv preprint arXiv:1811 1,69922,, 2018.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arviv: / 03888, 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2019.
Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. In Advances in Neural Information Processing Systems, pp. 8196-8207, 2019.
11

---

Under review as a conference paper at ICLR 2022
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings the 2015 JEEE International Conference Computer Vision (ICCV), ICCV '15, pp. 19-27, USA, 2013. IEEE Computer Society. ISBN 9781467383912. doi: 10.1 10.1109/ICC..2005.. URL doi wow.ioorg//loo00 100110111111TTT TCCV 2015. 11
12


Reviews:
[{"review_id": "v5jOjhjQrKo", "reviewer": "Reviewer_9Uka", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The central message of the paper is expressed quite clearly and also backed up by substantial experiments, i.e., **LARS and LAMB don't seem to have a fundamental optimization benefit as opposed to simpler methods**. I do have some comments on the writing/results:\n\n1) At many places in the paper the techniques are not fully explained, and their usage is taken for granted. This forces the reader to do a lot of back and forth between different papers to disambiguate and doesn't keep the work self-contained. It is ironic because the paper wants to promote rigor while specifying hyper-parameters. For instance, layerwise normalization is never formally written down in an equation. Similarly, how is L2 regularization performed? This is crucial to know how to interpret $\\lambda$. What does the label smoothening coefficient $\\tau$ do? What do you mean by decoupled weight decay with Adam?  All of this could have been avoided by using simple mathematical descriptions or through pseudocode.\n\n2) LARS and LAMB were proposed to improve the batch size scaling while training. In all the experiments in the paper, the intent is to compare against them for a specific batch size that obtained the best performance on a benchmark task. How do the simpler methods compare against them in terms of the linear speed-up and critical batch sizes? Are they more or less scalable? Having a speed-up curve v/s batch size would have made the comparison more transparent as the authors claim that,\n> The 88 epoch, 65,536 batch size result is faster in terms of wall-clock time but requires more training epochs, indicating that it is beyond LARS’s perfect scaling regime. Although LARS obtains diminishing returns when increasing the batch size from 32,768\nto 65,536, future work could investigate whether Nesterov momentum drops off more or less rapidly than LARS.\n\n    Also, how does robustness to hyper-parameter tuning change with the batch size (if it does)? Moreover, how do these behaviors vary across learning tasks (and not just Resnet with Imagenet)? These are important questions. After reading the paper, it was not clear to me which optimizer should I actually use for a given scale of the problem and systems constraints. As a paper, which tries to establish a new baseline this should have happened. Overall, the comparison was not as comprehensive as [this paper](https://arxiv.org/abs/1811.03600). \n\n\n3) How did the authors come with the 75.9% threshold? It would have been more useful to give some second-moment statistics such as standard deviation or confidence intervals for hypothesis testing, instead of saying how many runs successfully crossed the threshold. \n\n4) The paper takes credit for initiating discourse about things, which are already common knowledge. For instance, \n>We show that future work must carefully disentangle regularization and optimization effects when comparing a\nnew optimizer to baselines.\n\n    Any good paper should do that, and many bad papers don't understand that simple thing. This paper doesn't add anything new to that discourse. In table 2 the authors show that LARS has a slightly better test accuracy than SGD w/ momentum, but performs worse on optimization. There are many issues here. \n    * First of all, how should I interpret these differences in performance, are they significant or not? What is the standard deviation across the different runs or how big are the confidence intervals for some p-value? \n    * More importantly, how were the hyper-parameters tuned: to get the best validation performance or the best optimization performance? And in either case, how was this measured, through average performance at the tail of the optimization curve or just looking at a single number at the end? \n    * If the hyper-parameters were indeed tuned using a validation data-set, how can the paper claim that the aim of the optimizers is to optimize sample loss? If I just cared about optimization, I would tune the hyper-parameters to minimize the optimization loss.\n    * If I were to believe the assertion that layer-wise normalization should be interpreted as regularization, then what is the implicit bias it introduces to SGD? Can any differences in train and test performance now, be pushed under the rug as implicit regularization? Why couldn't this simply be explained in terms of the bias-variance trade-off/overfitting? Are there functional differences between the models learned through different optimizers, in terms of let's say prediction disagreement? How should I think about these issues in face of (less or more) overparameterization? These are all questions that are left unanswered. It seems the authors just want to show the marginal utility of LARS, instead of investigating what is happening under the surface. \n    > Although the main concern of a practitioner is validation performance, the primary task of an optimization algorithm is to minimize training loss.\n    * In light of the nuances I highlight above, this is not a correct statement. It depends on how the optimizer is sampling, tuning its hyper-parameters, etc.  \n\n5) **Hyper-parameters:** the warm-up schedule used for configuration B seems rather ad-hoc, have some other work used a quadratic warm-up before? Is there any rationale behind this? \n\nThus, I feel the paper can greatly improve by introducing more details rigorously, qualifying its statements, not overselling its contributions, and investigating some of its assertions further. Even then, I am not convinced that just showing that a previously accepted baseline, is not justifiably better is a big enough contribution. Specifically, it is unclear to me how useful is the entire line of work that tries to squeeze the most of data-parallel training for Resnets on Imagenet. There are other learning problems (like say in speech), and I don't understand how generalizable are any insights in this paper to neural network optimization.  ", "summary_of_the_paper": "In data-parallel distributed training, increasing the batch size of the optimizer's updates is the most natural way to reduce wall-clock training time. Prevalent first-order stochastic methods such as SGD, provide a linear speed-up in the batch size, but only up to some critical batch size (c.f., [smooth convex optimization bounds](https://arxiv.org/abs/1106.4574), and [an empirical work suggesting the same](https://arxiv.org/abs/1811.03600)). This critical batch size is typically very different for different learning architectures and optimization algorithms, and it is often difficult to decouple the effect of,\n* the bias-variance terms in optimization, \n* improper or insufficient hyper-parameter tuning, and \n* implicit regularization of the optimizer for extremely over-parameterized learning problems. \n\nAs a result, many existing works that propose new optimization algorithms, miss important baselines or don't compare against them fairly. This paper highlights such an issue with the recently proposed layerwise normalization techniques [LARS](https://arxiv.org/abs/1811.03600) and [LAMB](https://arxiv.org/abs/1904.00962), which build upon the update rules for SGD w/ Polyak momentum and Adam respectively. These methods were proposed to speed up the large batch (pre-)training of Imagenet and BERT respectively, and have gained a lot of attention in benchmark competitions. This paper underlines that these techniques improve optimization either marginally or not at all when compared to their vanilla first-order counterparts. Moreover, it highlights nuances in their hyper-parameter choices which are very important to consider. Most importantly, it establishes simpler baselines for improving future optimization algorithms for the considered learning tasks.  ", "summary_of_the_review": "The paper does enough to justify to me that LARS and LAMB are not strictly better than simpler optimizers. But it doesn't do much more. There aren't many generalizable insights here. I highlight some areas where the paper can improve. I might be open to increasing my score if those concerns are addressed.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hwESdx5k6gO", "reviewer": "Reviewer_uA2C", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The performance of LARS will be much higher if they use the same amount of computing resources to tune hyper-parameters.\n\nFor ImageNet, the authors did not report the results of a larger batch size (e.g. 64K or 128K). My experiments show that the performance of LARS would be much better than Nesterov for larger batch size.\n\nWith unlimited computing resources, the performance of LARS will still be better than Nesterov. The authors probably will eventually get results like Figure 7 of https://arxiv.org/pdf/2006.08517v1.pdf\n\nIf we take a look at Table 1, we can see that the hyper-parameters of Nesterov are meticulously selected or cherry-picked. Since the tuning cost is so high, it is almost impossible for users to consider this solution.\n\nFor example, to my knowledge, the authors of LARS did not tune the hyper-parameters of Batch Normalization. The authors of this paper used a lot of computing resources to tune the hyper-parameters of Batch Normalization.\n\n\"We ran a series of experiments, each of which searched over a hand-designed hyper-parameter search space using quasi-random search [Bousquet et al., 2017].\"\n\nTo my knowledge, the LARS authors did not use any advanced hyper-parameter tuning methods. This means the comparison is not fair.\n\n\"Table 2 shows that Nesterov momentum achieves higher training accuracy than LARS, despite similar validation performance. Thus, it may be more appropriate to consider the layerwise normalization of LARS to be a regularization technique, rather than an optimization technique.\"\n\nThis conclusion is doubtful. Firstly, the authors probably need to sample many different convergence points (including low-accuracy points) to study of optimization pattern of LARS/Nesterov. Secondly, the generalization performance of the different minima may have different implicit properties, which can not be explained in such a simple way.\n\nBTW, I can't reproduce the authors' results even though I totally trust the correctness of this paper. The reason is that we often need to use different hyper-parameters on different systems/hardware for large-batch training. A high tuning cost makes this process very hard.\n\nFor BERT, I suspect the authors tuned the hyper-parameters of fine-tuning process, which means the comparison is very unfair as the LAMB authors did not tune it.\n\nEven if the authors did not tune the fine-tuning process, this comparison is still unfair because LAMB can work without tuning for changing batch size.\n\nAs mentioned by the LAMB authors (caption of Table 4 in https://openreview.net/pdf?id=Syx4wnEtvH): \"We can achieve an even higher F1 score if we manually tune the hyperparameters\"\n\nThe LAMB authors reported untuned results in Table 4 and Table 5 of https://openreview.net/pdf?id=Syx4wnEtvH\n\nThe authors of this paper not only tuned common hyper-parameters (e.g. learning rate) but also tuned some uncommon hyper-parameters (e.g. beta1, beta2 in Adam).\n\nI suspect only big companies can do this kind of hyper-parameter searching for BERT per-training.\n\nThe novelty of this paper is very low. The key technical part is just tuning hyper-parameters. The authors also did not provide any deep analysis. If we don't need deep analysis, there are actually many blogposts on this topic (e.g. https://medium.com/fenwicks/tutorial-2-94-accuracy-on-cifar10-in-2-minutes-7b5aaecd9cdd). They can also significantly improve the performance of SGD with different optimization tricks. Similar tricks will likely improve the performance of LARS/LAMB.\n\nThe key advantage of this paper is that the authors can use a significant amount of computing resources while other researchers can not. I worry that this paper may mislead the ML community and have a negative impact on academia's budget/planning.\n\nIn my humble opinion, this paper is not a fit to top ML conferences like NeurIPS/ICML/ICLR.", "summary_of_the_paper": "This paper was previously rejected by NeurIPS and ICML. I was one of the NeurIPS reviewers for this paper.\nAlthough the authors made some changes, after carefully reading this paper, I found it did not change too much for key techniques and key contributions.\nSo I decided to use my previous NeurIPS review.\n\n###############################################################################################################\n\nThe authors used a huge amount of computing resources to tune hyperparameters of Adam/SGD and claimed that they can match the performance of LARS/LAMB for large-batch training. I think the comparison is not fair.", "summary_of_the_review": "The novelty of this paper is very low. The key technical part is just tuning hyper-parameters. The authors also did not provide any deep analysis. If we don't need deep analysis, there are actually many blogposts on this topic (e.g. https://medium.com/fenwicks/tutorial-2-94-accuracy-on-cifar10-in-2-minutes-7b5aaecd9cdd). They can also significantly improve the performance of SGD with different optimization tricks. Similar tricks will likely improve the performance of LARS/LAMB.\n\nThe key advantage of this paper is that the authors can use a significant amount of computing resources while other researchers can not. I worry that this paper may mislead the ML community and have a negative impact on academia's budget/planning.\n\nIn my humble opinion, this paper is not a fit to top ML conferences like NeurIPS/ICML/ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "HBVAiwlM0_V", "reviewer": "Reviewer_F1kk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "~~**A quick note in advance:** Apparently, the authors forgot to attach the appendix to the submitted version.\nFor my review, I didn't see a strong need to consult the appendix material,\nwhich mostly contains additional explanation and experimental results.\nSo I won't consider this missing material problematic for this review,\nbut will conduct a final check of the appendix material once it has been \nadded during revision.~~\n\n*Sorry for the erroneous comment on the appendix. I didn't realize that the appendix was in the supplements rather than attached to the main pdf (which was the case in other papers I have been reviewing).*\n\nThe paper is well-written. The experiment that have been conducted are\nwell-motivated and the experimental protocol is explained comprehensively and\nclearly.\nThe authors do a good job explaining the lessons learned and make various\nimportant points regarding the difficulty of farily evaluating and comparing\ndeep learning optimizers, e.g., the entanglement of optimization speed and\nregularization effects.\n\nI have one crucial criticism of this work. The paper puts significant effort in\nthe tuning of Nesterov/Adam to show that it can match the performance\nachieved by LARS/LAMB. That answers one question: Is LARS/LAMB *necessary* to achieve\nstate-of-the-art results in large-batch optimization. While potentially eye-opening, this is a very limited\nresult. As the authors themselves write in the conclusion, one \"can, in principle,\nalways invest more effort in tuning and potentially get better results\nfor any optimizer\". A much more interesting and helpful question to ask would be:\nIs LARS/LAMB *beneficial* in large-batch optimization? This would require to\nassign a similar tuning protocol and budget to both methods. The hyperparameters\nused for LARS are taken from published work (Kumar et al., 2019), which ostensibly\nalready put some effort into tuning those. However, a paper that critiques inadequate\ncomparisons in prior work should go the extra mile and guarantee an apples-to-apples\ncomparison. It seems quite likely that an equally well-tuned LARS/LAMB could see\nsome improvements.", "summary_of_the_paper": "This paper takes a closer look at deep learning optimization methods tailored to \nthe large batch size regime, namely LARS and LAMB. It shows that generic optimizers\n(SGD with Nesterov momentum and Adam, respectively) can achieve similar results in the large\nbatch regime given a careful tuning of their hyperparameters.", "summary_of_the_review": "As stated above, the paper is well-written and clear. All claims are adequately\nscoped and supported by the experiments provided in the paper.\nThis paper is not novel and it isn't trying to be. I expressly welcome papers\nthat double-check and critique prior work and try to disentangle effects with\ncarefully-designed experiments. However, I feel that the limited result of matching LARS/LAMB\ninstead of a full-fledged comparison of between LARS/LAMB and Nesterov/Adam greatly\ndiminishes the significance of the paper. With this shortcoming, in my opinion,\nthe paper narrowly misses the mark for publication at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AiIuXT5X8LW", "reviewer": "Reviewer_Fjz9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Understanding large batch training is quite important in this days of large data, and setting up a proper baseline result is an important. And yet, my main concern is the novelty or significance of this work, in that the main message of this work is pretty much shared and revealed by previous works including Shallue et al. (2019), Zhang et al. (2019), Schmidt et al. (2020) and Choi et al. (2019): it highlights the importance of tuning the hyperparameters involved when comparing optimization algorithms; standard optimization algorithms are no worse in generalization at large batches. But unlike the previous works this work does not provide results for an extensive range of batch sizes (ok may it’s fine considering only large batch optimizers), any theoretical guarantee (I checked the proof but the convergence rate looks worse to me compared to that of LARS -- although I agree with you that it’s worst case bound, but still) or concrete evidence to support the idea of implicit regularization effects for small batch sizes.\n\nMaybe a minor issue but I’d like to point out that the manuscript is quite unnecessarily lengthy; the main message is repeating, the details to secure evaluation validity, and the presentation of their results can all be more organized and cleaned up.", "summary_of_the_paper": "This paper revisits the effectiveness of the optimizers designed for large-batch training such as LAMB and LARS by You et al. (2017, 2019) respectively. While it has been claimed and demonstrated (in perhaps limited settings though) that such optimizers can achieve better performance compared to other generic optimizers such as SGD or ADAM (in the sense that they don’t require a specific batch size), this paper re-evaluates these optimizers while fine-tuning all hyperparameters involved to potentially affect the result and finds that they do not work better as claimed; or, more precisely the standard optimization algorithms including Nesterov and Adam can match or outperform LARS as long as they are properly tuned. The paper provides empirical evidence obtained from Imagenet and BERT experiments to support their finding.", "summary_of_the_review": "I appreciate the authors effort in correcting the common misconception that large batch training optimizers works better than standard optimizers through experiments, and I give a weak accept for now, but honestly I am not entirely certain if the paper contains substantial improvements or novelty compared to previous relevant works.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper experimentally shows that the commonly used standard solvers such as Nesterov momentum or Adam can achieve the same performance as optimizers such as LARS and LAMB specially proposed for large batches training.

Large batch training is a very important topic, and if the author's argument is true, it might be an interesting discovery.

However, all reviewers were concerned about its limited technical contribution. Not only that, but above all, when tuning the optimizer using the same computational resource (for a new task), it seems unclear whether the standard optimizer can achieve as much performance as large batch optimizers (currently they tune the standard optimizers, fixing the hyperparameter for large batch solvers to match their performances). The authors did not answer the reviewer's questions about it, and they did not answer the reviewer's other questions in great detail. Through discussion among reviewers, all reviews agreed on this concern and agreed to reject this paper.

The quality of the paper will be greatly improved if this concern is resolved.
Decision: Reject