Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In data-parallel distributed training, increasing the batch size of the optimizer's updates is the most natural way to reduce wall-clock training time. Prevalent first-order stochastic methods such as SGD, provide a linear speed-up in the batch size, but only up to some critical batch size (c.f., [smooth convex optimization bounds](https://arxiv.org/abs/1106.4574), and [an empirical work suggesting the same](https://arxiv.org/abs/1811.03600)). This critical batch size is typically very different for different learning architectures and optimization algorithms, and it is often difficult to decouple the effect of,
* the bias-variance terms in optimization, 
* improper or insufficient hyper-parameter tuning, and 
* implicit regularization of the optimizer for extremely over-parameterized learning problems. 

As a result, many existing works that propose new optimization algorithms, miss important baselines or don't compare against them fairly. This paper highlights such an issue with the recently proposed layerwise normalization techniques [LARS](https://arxiv.org/abs/1811.03600) and [LAMB](https://arxiv.org/abs/1904.00962), which build upon the update rules for SGD w/ Polyak momentum and Adam respectively. These methods were proposed to speed up the large batch (pre-)training of Imagenet and BERT respectively, and have gained a lot of attention in benchmark competitions. This paper underlines that these techniques improve optimization either marginally or not at all when compared to their vanilla first-order counterparts. Moreover, it highlights nuances in their hyper-parameter choices which are very important to consider. Most importantly, it establishes simpler baselines for improving future optimization algorithms for the considered learning tasks.  
main_review: The central message of the paper is expressed quite clearly and also backed up by substantial experiments, i.e., **LARS and LAMB don't seem to have a fundamental optimization benefit as opposed to simpler methods**. I do have some comments on the writing/results:

1) At many places in the paper the techniques are not fully explained, and their usage is taken for granted. This forces the reader to do a lot of back and forth between different papers to disambiguate and doesn't keep the work self-contained. It is ironic because the paper wants to promote rigor while specifying hyper-parameters. For instance, layerwise normalization is never formally written down in an equation. Similarly, how is L2 regularization performed? This is crucial to know how to interpret $\lambda$. What does the label smoothening coefficient $\tau$ do? What do you mean by decoupled weight decay with Adam?  All of this could have been avoided by using simple mathematical descriptions or through pseudocode.

2) LARS and LAMB were proposed to improve the batch size scaling while training. In all the experiments in the paper, the intent is to compare against them for a specific batch size that obtained the best performance on a benchmark task. How do the simpler methods compare against them in terms of the linear speed-up and critical batch sizes? Are they more or less scalable? Having a speed-up curve v/s batch size would have made the comparison more transparent as the authors claim that,
> The 88 epoch, 65,536 batch size result is faster in terms of wall-clock time but requires more training epochs, indicating that it is beyond LARS’s perfect scaling regime. Although LARS obtains diminishing returns when increasing the batch size from 32,768
to 65,536, future work could investigate whether Nesterov momentum drops off more or less rapidly than LARS.

    Also, how does robustness to hyper-parameter tuning change with the batch size (if it does)? Moreover, how do these behaviors vary across learning tasks (and not just Resnet with Imagenet)? These are important questions. After reading the paper, it was not clear to me which optimizer should I actually use for a given scale of the problem and systems constraints. As a paper, which tries to establish a new baseline this should have happened. Overall, the comparison was not as comprehensive as [this paper](https://arxiv.org/abs/1811.03600). 


3) How did the authors come with the 75.9% threshold? It would have been more useful to give some second-moment statistics such as standard deviation or confidence intervals for hypothesis testing, instead of saying how many runs successfully crossed the threshold. 

4) The paper takes credit for initiating discourse about things, which are already common knowledge. For instance, 
>We show that future work must carefully disentangle regularization and optimization effects when comparing a
new optimizer to baselines.

    Any good paper should do that, and many bad papers don't understand that simple thing. This paper doesn't add anything new to that discourse. In table 2 the authors show that LARS has a slightly better test accuracy than SGD w/ momentum, but performs worse on optimization. There are many issues here. 
    * First of all, how should I interpret these differences in performance, are they significant or not? What is the standard deviation across the different runs or how big are the confidence intervals for some p-value? 
    * More importantly, how were the hyper-parameters tuned: to get the best validation performance or the best optimization performance? And in either case, how was this measured, through average performance at the tail of the optimization curve or just looking at a single number at the end? 
    * If the hyper-parameters were indeed tuned using a validation data-set, how can the paper claim that the aim of the optimizers is to optimize sample loss? If I just cared about optimization, I would tune the hyper-parameters to minimize the optimization loss.
    * If I were to believe the assertion that layer-wise normalization should be interpreted as regularization, then what is the implicit bias it introduces to SGD? Can any differences in train and test performance now, be pushed under the rug as implicit regularization? Why couldn't this simply be explained in terms of the bias-variance trade-off/overfitting? Are there functional differences between the models learned through different optimizers, in terms of let's say prediction disagreement? How should I think about these issues in face of (less or more) overparameterization? These are all questions that are left unanswered. It seems the authors just want to show the marginal utility of LARS, instead of investigating what is happening under the surface. 
    > Although the main concern of a practitioner is validation performance, the primary task of an optimization algorithm is to minimize training loss.
    * In light of the nuances I highlight above, this is not a correct statement. It depends on how the optimizer is sampling, tuning its hyper-parameters, etc.  

5) **Hyper-parameters:** the warm-up schedule used for configuration B seems rather ad-hoc, have some other work used a quadratic warm-up before? Is there any rationale behind this? 

Thus, I feel the paper can greatly improve by introducing more details rigorously, qualifying its statements, not overselling its contributions, and investigating some of its assertions further. Even then, I am not convinced that just showing that a previously accepted baseline, is not justifiably better is a big enough contribution. Specifically, it is unclear to me how useful is the entire line of work that tries to squeeze the most of data-parallel training for Resnets on Imagenet. There are other learning problems (like say in speech), and I don't understand how generalizable are any insights in this paper to neural network optimization.  
summary_of_the_review: The paper does enough to justify to me that LARS and LAMB are not strictly better than simpler optimizers. But it doesn't do much more. There aren't many generalizable insights here. I highlight some areas where the paper can improve. I might be open to increasing my score if those concerns are addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper revisits the effectiveness of the optimizers designed for large-batch training such as LAMB and LARS by You et al. (2017, 2019) respectively. While it has been claimed and demonstrated (in perhaps limited settings though) that such optimizers can achieve better performance compared to other generic optimizers such as SGD or ADAM (in the sense that they don’t require a specific batch size), this paper re-evaluates these optimizers while fine-tuning all hyperparameters involved to potentially affect the result and finds that they do not work better as claimed; or, more precisely the standard optimization algorithms including Nesterov and Adam can match or outperform LARS as long as they are properly tuned. The paper provides empirical evidence obtained from Imagenet and BERT experiments to support their finding.
main_review: Understanding large batch training is quite important in this days of large data, and setting up a proper baseline result is an important. And yet, my main concern is the novelty or significance of this work, in that the main message of this work is pretty much shared and revealed by previous works including Shallue et al. (2019), Zhang et al. (2019), Schmidt et al. (2020) and Choi et al. (2019): it highlights the importance of tuning the hyperparameters involved when comparing optimization algorithms; standard optimization algorithms are no worse in generalization at large batches. But unlike the previous works this work does not provide results for an extensive range of batch sizes (ok may it’s fine considering only large batch optimizers), any theoretical guarantee (I checked the proof but the convergence rate looks worse to me compared to that of LARS -- although I agree with you that it’s worst case bound, but still) or concrete evidence to support the idea of implicit regularization effects for small batch sizes.

Maybe a minor issue but I’d like to point out that the manuscript is quite unnecessarily lengthy; the main message is repeating, the details to secure evaluation validity, and the presentation of their results can all be more organized and cleaned up.
summary_of_the_review: I appreciate the authors effort in correcting the common misconception that large batch training optimizers works better than standard optimizers through experiments, and I give a weak accept for now, but honestly I am not entirely certain if the paper contains substantial improvements or novelty compared to previous relevant works.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper was previously rejected by NeurIPS and ICML. I was one of the NeurIPS reviewers for this paper.
Although the authors made some changes, after carefully reading this paper, I found it did not change too much for key techniques and key contributions.
So I decided to use my previous NeurIPS review.

###############################################################################################################

The authors used a huge amount of computing resources to tune hyperparameters of Adam/SGD and claimed that they can match the performance of LARS/LAMB for large-batch training. I think the comparison is not fair.
main_review: The performance of LARS will be much higher if they use the same amount of computing resources to tune hyper-parameters.

For ImageNet, the authors did not report the results of a larger batch size (e.g. 64K or 128K). My experiments show that the performance of LARS would be much better than Nesterov for larger batch size.

With unlimited computing resources, the performance of LARS will still be better than Nesterov. The authors probably will eventually get results like Figure 7 of https://arxiv.org/pdf/2006.08517v1.pdf

If we take a look at Table 1, we can see that the hyper-parameters of Nesterov are meticulously selected or cherry-picked. Since the tuning cost is so high, it is almost impossible for users to consider this solution.

For example, to my knowledge, the authors of LARS did not tune the hyper-parameters of Batch Normalization. The authors of this paper used a lot of computing resources to tune the hyper-parameters of Batch Normalization.

"We ran a series of experiments, each of which searched over a hand-designed hyper-parameter search space using quasi-random search [Bousquet et al., 2017]."

To my knowledge, the LARS authors did not use any advanced hyper-parameter tuning methods. This means the comparison is not fair.

"Table 2 shows that Nesterov momentum achieves higher training accuracy than LARS, despite similar validation performance. Thus, it may be more appropriate to consider the layerwise normalization of LARS to be a regularization technique, rather than an optimization technique."

This conclusion is doubtful. Firstly, the authors probably need to sample many different convergence points (including low-accuracy points) to study of optimization pattern of LARS/Nesterov. Secondly, the generalization performance of the different minima may have different implicit properties, which can not be explained in such a simple way.

BTW, I can't reproduce the authors' results even though I totally trust the correctness of this paper. The reason is that we often need to use different hyper-parameters on different systems/hardware for large-batch training. A high tuning cost makes this process very hard.

For BERT, I suspect the authors tuned the hyper-parameters of fine-tuning process, which means the comparison is very unfair as the LAMB authors did not tune it.

Even if the authors did not tune the fine-tuning process, this comparison is still unfair because LAMB can work without tuning for changing batch size.

As mentioned by the LAMB authors (caption of Table 4 in https://openreview.net/pdf?id=Syx4wnEtvH): "We can achieve an even higher F1 score if we manually tune the hyperparameters"

The LAMB authors reported untuned results in Table 4 and Table 5 of https://openreview.net/pdf?id=Syx4wnEtvH

The authors of this paper not only tuned common hyper-parameters (e.g. learning rate) but also tuned some uncommon hyper-parameters (e.g. beta1, beta2 in Adam).

I suspect only big companies can do this kind of hyper-parameter searching for BERT per-training.

The novelty of this paper is very low. The key technical part is just tuning hyper-parameters. The authors also did not provide any deep analysis. If we don't need deep analysis, there are actually many blogposts on this topic (e.g. https://medium.com/fenwicks/tutorial-2-94-accuracy-on-cifar10-in-2-minutes-7b5aaecd9cdd). They can also significantly improve the performance of SGD with different optimization tricks. Similar tricks will likely improve the performance of LARS/LAMB.

The key advantage of this paper is that the authors can use a significant amount of computing resources while other researchers can not. I worry that this paper may mislead the ML community and have a negative impact on academia's budget/planning.

In my humble opinion, this paper is not a fit to top ML conferences like NeurIPS/ICML/ICLR.
summary_of_the_review: The novelty of this paper is very low. The key technical part is just tuning hyper-parameters. The authors also did not provide any deep analysis. If we don't need deep analysis, there are actually many blogposts on this topic (e.g. https://medium.com/fenwicks/tutorial-2-94-accuracy-on-cifar10-in-2-minutes-7b5aaecd9cdd). They can also significantly improve the performance of SGD with different optimization tricks. Similar tricks will likely improve the performance of LARS/LAMB.

The key advantage of this paper is that the authors can use a significant amount of computing resources while other researchers can not. I worry that this paper may mislead the ML community and have a negative impact on academia's budget/planning.

In my humble opinion, this paper is not a fit to top ML conferences like NeurIPS/ICML/ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper takes a closer look at deep learning optimization methods tailored to 
the large batch size regime, namely LARS and LAMB. It shows that generic optimizers
(SGD with Nesterov momentum and Adam, respectively) can achieve similar results in the large
batch regime given a careful tuning of their hyperparameters.
main_review: ~~**A quick note in advance:** Apparently, the authors forgot to attach the appendix to the submitted version.
For my review, I didn't see a strong need to consult the appendix material,
which mostly contains additional explanation and experimental results.
So I won't consider this missing material problematic for this review,
but will conduct a final check of the appendix material once it has been 
added during revision.~~

*Sorry for the erroneous comment on the appendix. I didn't realize that the appendix was in the supplements rather than attached to the main pdf (which was the case in other papers I have been reviewing).*

The paper is well-written. The experiment that have been conducted are
well-motivated and the experimental protocol is explained comprehensively and
clearly.
The authors do a good job explaining the lessons learned and make various
important points regarding the difficulty of farily evaluating and comparing
deep learning optimizers, e.g., the entanglement of optimization speed and
regularization effects.

I have one crucial criticism of this work. The paper puts significant effort in
the tuning of Nesterov/Adam to show that it can match the performance
achieved by LARS/LAMB. That answers one question: Is LARS/LAMB *necessary* to achieve
state-of-the-art results in large-batch optimization. While potentially eye-opening, this is a very limited
result. As the authors themselves write in the conclusion, one "can, in principle,
always invest more effort in tuning and potentially get better results
for any optimizer". A much more interesting and helpful question to ask would be:
Is LARS/LAMB *beneficial* in large-batch optimization? This would require to
assign a similar tuning protocol and budget to both methods. The hyperparameters
used for LARS are taken from published work (Kumar et al., 2019), which ostensibly
already put some effort into tuning those. However, a paper that critiques inadequate
comparisons in prior work should go the extra mile and guarantee an apples-to-apples
comparison. It seems quite likely that an equally well-tuned LARS/LAMB could see
some improvements.
summary_of_the_review: As stated above, the paper is well-written and clear. All claims are adequately
scoped and supported by the experiments provided in the paper.
This paper is not novel and it isn't trying to be. I expressly welcome papers
that double-check and critique prior work and try to disentangle effects with
carefully-designed experiments. However, I feel that the limited result of matching LARS/LAMB
instead of a full-fledged comparison of between LARS/LAMB and Nesterov/Adam greatly
diminishes the significance of the paper. With this shortcoming, in my opinion,
the paper narrowly misses the mark for publication at ICLR.
review_title: null

Meta-review: This paper experimentally shows that the commonly used standard solvers such as Nesterov momentum or Adam can achieve the same performance as optimizers such as LARS and LAMB specially proposed for large batches training.

Large batch training is a very important topic, and if the author's argument is true, it might be an interesting discovery.

However, all reviewers were concerned about its limited technical contribution. Not only that, but above all, when tuning the optimizer using the same computational resource (for a new task), it seems unclear whether the standard optimizer can achieve as much performance as large batch optimizers (currently they tune the standard optimizers, fixing the hyperparameter for large batch solvers to match their performances). The authors did not answer the reviewer's questions about it, and they did not answer the reviewer's other questions in great detail. Through discussion among reviewers, all reviews agreed on this concern and agreed to reject this paper.

The quality of the paper will be greatly improved if this concern is resolved.
Decision: Reject