Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors detail the significant effort required to reproduce results from the original LARS and LAMB papers.  Having reproduced results for LARS (for ResNet-50 training on ImageNet) and LAMB (for BERT pretraining), the authors then show that standard/older optimizers (Nesterov momentum and Adam) can match or even exceed LARS and LAMB at large batch sizes.  The authors provide in-depth details on both the difficulties required for them to reproduce the original results (i.e., fixing discrepancies between details in the publications and available source-code from online implementations) and their hyperparameter optimization for Nesterov Momentum and Adam on the datasets evaluated.  Also, insights and lessons are provided, both on the importance of hyperparameter tuning for comparison of DL solvers and on practitioners attempting to demonstrate performance gains for a new method (are not merely due to better tuning or regularization tricks).
main_review: The paper is well written and the authors have done an impressive job both recreating the results of the LARS and LAMB papers (including all necessary hyperparameters and implementation tricks) and showing that Adam and Nesterov Momentum may be used in place of LARS and LAMB for the two tasks presented.  Furthermore, the lessons detailed are important and the paper is a valuable addition to researchers putting together new baselines.

I spent a long time combing through the paper.  On the one hand, this is an unfiltered account of the struggles researchers often encounter when attempting to reproduce state-of-the-art (including the extreme detective work necessary to find the undocumented tricks necessary to achieve some results).  However, in the end, I did not provide a higher score due to the lack of novel contributions and the narrow scope of the paper.  For the former, there are no new methods presented (or novel applications), and the lessons gleaned-e.g., optimizer-performance depending on the learning rate schedule and the effect of large batch sizes on DL training schedules-have been extensively explored in numerous previous works (see below for recent examples).  For the latter, the paper narrowly focuses on the results in previously published papers, rather than, for instance, showing how to optimize existing solvers for general downstream tasks.  Furthermore, there is no cohesive method presented with which readers may takeaway to aid them, i.e., how can these lessons help researchers develop new methods to push the state-of-the-art?  

Ultimately, the paper raises an important question "comparing optimizers is difficult since hyperparameter tuning tends to dominate the comparisons."  However, the paper does not provide an answer or general method to attack this difficult problem.

Other comments:
----------------------
"In particular, we show that the optimal shape of the learning rate schedule is optimizer-dependent (in
addition to the scale), and that differences in the schedule can dominate optimizer comparisons at
smaller step budgets and become less important at larger step
budgets." <- This has been observed in several previous studies.E.g., see:
-Lee, Namhoon, et al. "Understanding the effects of data parallelism
and sparsity on neural network training." arXiv preprint
arXiv:2003.11316 (2020).
-Schmidt, Robin M., Frank Schneider, and Philipp Hennig. "Descending
through a crowded valley-benchmarking deep learning optimizers."
International Conference on Machine Learning. PMLR, 2021.

"they are loose upper bounds
on the worst case behavior of the algorithms" <- You have not
shown the lack of tightness (i.e., achievability), how have
you characterized these bounds as loose?

"Although there are heuristics for adjusting the learning rate as the batch size changes, these heuristics inevitably
break down sufficiently far from the initial batch size and it is also not clear how to apply them to other training
hyperparameters (e.g. momentum)." <- The former claim is too broad;
there are importance sampling based approaches which allow automatic
learning rate adjustments, e.g.:
Johnson, Tyler B., and Carlos Guestrin. "Training deep models faster
with robust, approximate importance sampling." Advances in Neural
Information Processing Systems 31 (2018): 7265-7275.

-Please include more discussion (in the main paper) of the Appendix
 results displaying LARS performance with and without the
 "optimization tricks and non-default values of uncommonly-tuned
 hyperparameters."

"our BERT result
with Adam also improves upon the wall-time record of LAMB reported in
You et al. 2019)" <- I agree that reporting training speed in steps is
a fair comparison between different training schedules and solvers.
However, due to compute environment issues, comparing wall-times
reported in another paper is not a fair comparison (the LAMB
experiment from You et al 2019 would have to be repeated on the same
compute environment used to run the presented Adam experiments).

"Moving forward, if we are to reliably make progress we need to rethink how we compare and evaluate
new optimizers for neural network training." <- Once again, the paper focuses on asking this question, rather than providing answers to it.

"However, achieving perfect scaling is not always straightforward. Changing the batch size changes
the training dynamics, requiring the training hyperparameters (e.g. learning rate) to be carefully
re-tuned in order to maintain the same level of validation
performance." <- Citation required for this claim

"In addition, smaller batch sizes provide implicit regularization from
gradient noise that may need to be replaced by other forms of
regularization when the batch size is increased." <- Citation required
for this claim

limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: Authors study the problem of large-batch training of neural networks and claim an important (negative) result: specialized large-batch algorithms are not necessarily better than traditional ones. Authors consider two standard optimizers (Adam and SGD w/ Nesterov momentum) and find hyperparameter configurations under which these standard optimizers match and sometimes outperform LAMB/LARS for large-batch training of ResNet50 and BERT respectively.
main_review: **Disclaimer:** I am writing this (probably) additional review without reading prior discussion and other reviews. My review may or may not raise questions that were already addressed in prior discussion.

This was a difficult paper to review. On one hand, it claims an important negative result that may affect an entire line of prior work. At the current state of the field, we badly need more papers like this to reduce the “research debt”, and hence the complexity of future research in the field. On the other hand, this paper is lacking in the way it supports its main claim, and overall the paper is not refined enough to recommend acceptance to NeurIPS. I would also argue that it does not stand up to the standards that authors themselves suggest in Section 6. For simplicity, I summarize strong and weak points of the paper in two separate sections.

## Reasons to accept
* Outstanding technical competence: authors work with highly tuned baselines with great attention to detail; L279-286 is pure brilliance
* Reporting the search space chronology (Appendix D.4) is a great practice that deserves wider adoption.

Presentation-wise, the paper is clearly written and easy to read, even though it follows a somewhat unconventional structure. The only recommendation I can give is to avoid wrapping figures with low-margin text (e.g. Figure 4). That said, this is an optional request that did not affect my score.

## Reasons to reject

My main issue with this paper is that it leaves an open question where it was clearly possible to give an answer with strong evidence. As authors rightfully state in L5-6, “it is an open queston” whether LARS/LAMB has any benefit for large batches. Unfortunately, providing specific configurations of hyperparameters where SGD/Adam outperforms LARS does not make us much closer to answering that question.

There is a wide range of prior work [1,2,3] on properly benchmarking optimizers and their sensitivity to hyperparameters. For instance, [1] proposes a general approach that could have been used to test whether or not LAMB requires less hyperparameter tuning than Adam, as authors wonder in L336-340. In fact, it would not require much more compute than what was used for finding the optimal hyperparameters. Some prior work[2] already measured the tuning-aware efficiency of the same optimizers and demonstrates similar issues with LARS/LAMB, although not in a large-batch setting. It comes to the same conclusion that none of the specialized algorithms consistently outperform SGD/Adam.

To summarize, this paper does indeed propose novel and potentially important “anecdotal” evidence, but I would expect a more rigorous analysis from a high profile venue such as NeurIPS. Aside from that, the discussion and conclusions made in the paper are insightful, but already well-known: the importance of hyperparameter configuration is better addressed in [1,2,3], the fact that an optimizer is also an indirect regularizer is also known [4].


## Questions, minor concerns

**MLPerf closed division**
Authors refer to the closed division multiple times; however, there is a separate “open” division in MLPerf where participants can tune the optimizer hyperparameters. Currently, the submissions on that track use either slightly modified LAMB or a completely different approach based on second order optimization[5]. If there is some intuitive explanation as for why these submissions still rely on LAMB?

**Nesterov momentum**
As far as I understood, you use nesterov momentum for ResNet while baselines and other experiments use traditional momentum (or Adam’s “momentum” for Adam/LAMB). Based on your reasoning in S2 this may be important. To better convey the importance of this change of this change
- Does LARS benefit from using Nesterov momentum instead of classical one? (or explain why it is impossible to modify LARS with nesterov momentum)
- [optional] Does Adam/LAMB benefit from switching to Nadam-like momentum?

**Hyperparameter stability to batch size**
Do configurations A/B maintain their advantage over LARS if we apply them to adjacent batch sizes without re-tuning? (e.g. 65536 samples, 16384 samples)

**Page 1 footnote:** "... these heuristics inevitably break down ..."
[optional] this would benefit a citation or further discussion

**Maybe typo:** Table 14, 15: t_{decay} is 2.815 is reported with a dot, as if it is approximately 3 steps. Did you mean 2,815?

[1] Optimizer Benchmarking Needs to Account for Hyperparameter Tuning. Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, François Fleuret. 2019

[2] How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers. Yuanhao Xiong, Xuanqing Liu, Li-Cheng Lan, Yang You, Si Si, Cho-Jui Hsieh. 2020

[3] Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers. Robin M. Schmidt, Frank Schneider, and Philipp Hennig, 2020

[4] Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks. Yuanzhi Li, Colin Wei, Tengyu Ma. 2019

[5] Scalable Second Order Optimization for Deep Learning. Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, Yoram Singer. 2020

p.s. regardless of whether the paper is accepted, I must express my deepest respect for pinpointing the widespread implementation issues with Adam and learning rate decays in BERT.
limitations_and_societal_impact: This is a fundamental paper that critiques large-batch optimizers. It is difficult to forsee its potential societal impact. Thus, I have no issue with how authors address it.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This is a very interesting work focused on a critical examination of LARS and LAMB optimizers, and the comparison of these with more established "standard" optimizers such as Adam.  The authors' results and analysis are insightful and of wide interest to the deep learning community.
main_review: FIrstly, the focus of this work is timely, relevant, and well motivated, making it significant and of interest to the deep learning community.  The quality of writing is good, so quality standards are high in this respect too.  The main contribution comes in the form of a series of insights pertaining to why LARS and LAMB perform well (or appear to), when they do not, and how they should be meaningfully compared with Adam and the like.  The authors' arguments are convincing as is the evidence presented.  Overall, this is a high quality paper which should be accepted. The key contributions are summarized well and succinctly by the authors themselves, so I quote:

"standard optimizers, without any layer-wise normalization techniques, can match or exceed the large batch size results used to justify LARS and LAMB. Future work attempting to argue that a new algorithm is useful by comparing to baseline methods or results [...] faces a key challenge in showing that the gains are due to the new method and not merely due to better tuning or changes to the training pipeline"
limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this work, the authors compared the standard optimizers (i.e., Nesterov momentum, Adam) and optimizers with layer-wise normalization (i.e., LARS, LAMB) in training neural networks with large-batch sizes.
Although LARS and LAMB were proposed and known as "large-batch optimizers," the authors showed that the standard optimizers with careful hyperparameter tuning could match or exceed the state-of-the-art results by LARS (for training ResNet-50 on ImageNet, batch size = 32K) and LAMB (for pre-training BEAT, batch size = 65K) at large-batch settings within the same step budgets.
The authors pointed out that the strong results depend on several optimization/regularization tricks, non-default values of uncommonly-tuned hyperparameters, and a careful learning rate schedule, but such details are not often discussed. With this point, the difficulties of comparing optimizers and the importance of rigorous empirical comparisons and open-sourcing in deep learning optimizer research are highlighted.

main_review: This paper is well-organized, and the background, experimental methods, results, and lessons are clearly presented.
Based on their rigorous empirical comparisons, they showed the standard optimizers (i.e., Nesterov momentum, Adam) could much or exceed the results of the optimizers designed for large-batch settings (i.e., LARS, LAMB).
Since it has been believed that we cannot use such standard optimizers in large-batch settings, their results are surprising and make us re-examine how we should compare optimizers and whether we really need new optimizers. 
Although their results are only in large-batch settings, it is expected that we can observe the same phenomena (i.e., standard optimizers can meet or exceed the results of optimizers tailored for a specific task) in other settings.
This observation is beneficial, especially for practitioners.
In particular, I agree with the authors' claim: the importance of uncommonly-tuned hyperparameters, learning rate schedule, and disentangling optimization speed and regularization has not been discussed in detail.
And I appreciate this work for shedding light on this point with convincing empirical results.
I believe the community should appreciate the points made in this paper. 

Minor comments:
- L93: the citation to Schmidt et al. [2020] is repeated.

++++++++++++++++++++++++++++++++

Update after seeing the other reviews and the authors' response

As some reviewers (and the authors) mentioned, the importance of hyperparameter tuning and the fact that “there is no clear winner” are already well-known (and this is why standard optimizers such as Momentum SGD and Adam are still the most popular choices because of their simplicity). However, LARS/LAMB has been used as the first choice in large-batch settings, but there is not enough evidence and discussion that it generally works better than the standard optimizers. This paper is the first study to point this out and show that the standard optimizers still work even in large-batch settings.

For the above reason, I believe this paper provides an important lesson for readers, and I wish to keep recommending to accept. But I also agree with other reviewers that "contribution is too narrow" (the readers interested in large-batch training may be limited.) So I updated my score from 8 to 7. 

limitations_and_societal_impact: The authors adequately addressed the limitations of their work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper performs a detailed hyperparameter study of stochastic optimizers specifically designed for large-batch training (LARS and LAMB). With appropriate hyperparameter tuning, simpler optimizers such as Nesterov SGD and Adam match and sometimes outperform those specialized optimizers on ResNet-50 and BERT. The authors attribute the relative success of large-batch methods to minute undocumented aspects in their implementations, such as regularization and scaling on certain operators, which can be back-ported to the other optimizers, and advocate for more relaxed definitions (e.g. number of steps) for benchmarks such as MLPerf Training.
main_review: The paper neatly showcases the feebleness of highly-tuned stochastic optimization schedules on two standard deep learning benchmarks. While the points of the paper are mention-worthy, there is nearly no scientific contribution in the paper. Even as an empirical study, it hard to draw tangible conclusions from the paper. The only takeaway from the paper is that LAMB/LARS are very well-tuned, and simpler optimizers' hyperparameter schedules could also be aggressively tuned to achieve the same result.

The paper reads like a direct attack on LAMB and LARS, though the same claims could be made on other optimizers, including Adam and its hyperparameters (with enough tuning, one could probably attain similar convergence properties to Adam with momentum SGD). Similar claims about benchmarking stochastic optimizers w.r.t. tunability have already been made, e.g. by Sivaprasad et al. in "Optimizer Benchmarking Needs to Account for Hyperparameter Tuning", which is not cited here. 

In the paper, there is very limited (sometimes absent) theoretical/empirical reasoning for the importance of certain parameter values: for example, in line 169: "we found that choosing $\gamma_0$ between 0.0 and 1.0 was important" --- why is that? What happens to the training process otherwise? Additionally, would the tuned Nesterov parameters for ResNet-50 transfer to other neural architectures (such as ViT), or other datasets? 
As a reader, I would be interested in a sensitivity analysis of the hyperparameters (or of the same algorithm to varying neural architectures/datasets), theoretical justification, or some sort of guidelines on how to tune those parameters in reasonable time (i.e. without exploring the entire space). 

In-depth comments:

Since most of the paper is spent discussing LAMB and LARS, I find it strange that their update rule definitions cannot be found in the manuscript.

Line 322: "Training with larger batch sizes is not fundamentally unstable: stringent step budgets make hyperparameter tuning trickier." --- Even with an unlimited step budget, it was shown on multiple occasions (with theoretical justification) that larger batch sizes tend to overfit.

What about non-Nesterov momentum SGD? is there a configuration that works similarly well? If not, why not?

Line 164: moving averages not explained well in the context of batch normalization.

Throughout: the paper uses too many footnotes to my taste, which detracts from the flow of the text.
limitations_and_societal_impact: The answer to question 3d in the checklist is not satisfactory. Given that this is a work that performs extensive hyperparameter tuning on compute-intensive datasets, the time and overall energy spent on training are of utmost importance to report.
review_title: null

final_score: rating: 2: Strong rejection
review_content: summary: I carefully read the responses from the authors. However, the authors did not address my concern. I'd like to keep my score.

###########################################################################################################

The authors used a huge amount of computing resources to tune hyperparameters of Adam/SGD and claimed that they can match the performance of LARS/LAMB for large-batch training. I think the comparison is not fair.
main_review: The performance of LARS will be much higher if they use the same amount of computing resources to tune hyper-parameters.

For ImageNet, the authors did not report the results of a larger batch size (e.g. 64K or 128K).
My experiments show that the performance of LARS would be much better than Nesterov for larger batch size.

With unlimited computing resources, the performance of LARS will still be better than Nesterov. 
The authors probably will eventually get results like Figure 7 of https://arxiv.org/pdf/2006.08517v1.pdf


If we take a look at Table 1, we can see that the hyper-parameters of Nesterov are meticulously selected or cherry-picked.
Since the tuning cost is so high, it is almost impossible for users to consider this solution.

For example, to my knowledge, the authors of LARS did not tune the hyper-parameters of Batch Normalization.
The authors of this paper used a lot of computing resources to tune the hyper-parameters of Batch Normalization.


"We ran a series of experiments, each of which searched over a hand-designed hyper-parameter search space using quasi-random search [Bousquet et al., 2017]."

To my knowledge, the LARS authors did not use any advanced hyper-parameter tuning methods. This means the comparison is not fair.


"Table 2 shows that Nesterov momentum achieves higher training accuracy than LARS, despite similar validation performance. Thus, it may be more appropriate to consider the layerwise normalization of LARS to be a regularization technique, rather
than an optimization technique."

This conclusion is doubtful. Firstly, the authors probably need to sample many different convergence points (including low-accuracy points) to study of optimization pattern of LARS/Nesterov. 
Secondly, the generalization performance of the different minima may have different implicit properties, which can not be explained in such a simple way.


BTW, I can't reproduce the authors' results even though I totally trust the correctness of this paper.
The reason is that we often need to use different hyper-parameters on different systems/hardware for large-batch training.
A high tuning cost makes this process very hard.


For BERT, I suspect the authors tuned the hyper-parameters of fine-tuning process, which means the comparison is very unfair as the LAMB authors did not tune it.

Even if the authors did not tune the fine-tuning process, this comparison is still unfair because LAMB can work without tuning for changing batch size.

As mentioned by the LAMB authors (caption of Table 4 in https://openreview.net/pdf?id=Syx4wnEtvH): "We can achieve an even higher F1 score if we manually tune the hyperparameters"

The LAMB authors reported untuned results in Table 4 and Table 5 of https://openreview.net/pdf?id=Syx4wnEtvH

The authors of this paper not only tuned common hyper-parameters (e.g. learning rate) but also tuned some uncommon hyper-parameters (e.g. beta1, beta2 in Adam).

I suspect only big companies can do this kind of hyper-parameter searching for BERT per-training.


The novelty of this paper is very low. The key technical part is just tuning hyper-parameters. The authors also did not provide any deep analysis.
If we don't need deep analysis, there are actually many blogposts on this topic (e.g. https://medium.com/fenwicks/tutorial-2-94-accuracy-on-cifar10-in-2-minutes-7b5aaecd9cdd).
They can also significantly improve the performance of SGD with different optimization tricks.
Similar tricks will likely improve the performance of LARS/LAMB.

The key advantage of this paper is that the authors can use a significant amount of computing resources while other researchers can not. 
I worry that this paper may mislead the ML community and have a negative impact on academia's budget/planning.

In my humble opinion, this paper is not a fit to top ML conferences like NeurIPS/ICML/ICLR.
limitations_and_societal_impact: The novelty of this paper is very low. The key technical part is just tuning hyper-parameters. The authors also did not provide any deep analysis.
If we don't need deep analysis, there are actually many blogposts on this topic (e.g. https://medium.com/fenwicks/tutorial-2-94-accuracy-on-cifar10-in-2-minutes-7b5aaecd9cdd).
They can also significantly improve the performance of SGD with different optimization tricks.
Similar tricks will likely improve the performance of LARS/LAMB.

The key advantage of this paper is that the authors can use a significant amount of computing resources while other researchers can not. 
I worry that this paper may mislead the ML community and have a negative impact on academia's budget/planning.

In my humble opinion, this paper is not a fit to top ML conferences like NeurIPS/ICML/ICLR.
review_title: null

Meta-review: This paper provides a critical examination of the LARS and LAMB optimizers, and a comparison to the more established optimizers such as momemtum or Adam. The paper provides an extensive set of experiments to show that Nesterov momentum and Adam can match or exceed the results of LARS and LAMB when using large batch sizes. Momentum techniques are known to have problems with high-levels of noise so this in some sense provides another empirical confirmation that these methods can work well with moderate amounts of noise.

There is a wide variance in the scores attributed by the different reviewers. Overall, the contribution of the paper is quite narrow and focused on a specific problem. The discussion of related work is very incomplete. I don't see any reference to the optimization literature on stochastic momentum methods, including for instance: Mahmoud Assran and Michael Rabbat. On the convergence of nesterov’s accelerated gradient method in stochastic settings. arXiv preprint arXiv:2002.12414, 2020. Nicolas Loizou and Peter Richtárik. Momentum and stochastic momentum for stochastic gradient, newton, prox- imal point and subspace descent methods. arXiv preprint arXiv:1712.09677, 2017.
(and many others)

Finally, some reviewers are very critical regarding some of the details of the empirical evaluation. In response, the authors provided more follow-up experimental results. The changes required are I believe too significant and require another round of review.

Overall, I therefore do not recommend acceptance at this stage.


Decision: Reject