Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: 
This paper addresses the issue of parallel, large mini-batch SGD, and how to adapt learning rate  and batch size. The aim is to find a sweet spot with batches large enough reduce gradient estimator variance while not too large, with incremental gains in gradient variance reduction, and potentially harming generalization.  Specifically, they consider a fast way to compute the gradient variance.  The issue is that computing the covariance for the gradient estimator is very expensive, especially if done so with different batch sizes (as is done, eg in [16]).  While prior methods for adapting LR or batch size have computed the gradient variance relatively infrequently, e.g., every epoch, this paper argues that gradient variance can vary significantly on finer time scales, and hence they aim to estimate gradient variance and adapt batch at every mini-batch.  The experiments in the paper are extensive, and they show promising results with extremely large batch sizes.


main_review: I enjoyed reading this paper.  The topic will be of interest to a large fraction of the NeurIPS community, and the innovations and experiments in the paper appear promising.

I do have several concerns and suggestions that the authors may wish to address.

1) Section 4.1 begins 'Existing works typically estimate gradient variance via collecting gradients with different batch sizes'.  I think it might be worth explaining that for IID data, the estimator variance for the mean should decreases like 1/N multiplied by the variance of the RV. So from a single batch one can in estimate the gradient covariance. A noisy estimate of the mean can, however be problematic, in which case it is sometimes effective to use two samples.  This is explained in Appendix A.1 in [16] for example (ie https://arxiv.org/pdf/1812.06162.pdf).

Indeed, there are strong similarities between the approach proposed in the current paper and in reference [16].  In particular, Appendix A.1 in [16] appears to describe a method similar to SimiGrad that they say is "free in a data-parallel environment".  It outlines a similar idea to SimiGrad that uses averages before and after allreduce.  This I think a more detailed discussion of similarities and differences with [16] may be useful.

2) The relation between normalized gradients, cosine similarity, and the gradient covariance is not well explained in the paper, but should be.  This is a major omission, but it can be corrected reasonably straightforwardly.  In particular, one can show that the cosine similarity in this case is affinely related to the trace of the covariance matrix for the normalized gradient vector.  In this sense, cosine similarity is closely related to the total variance of the normalized gradients in the batch.  Explaining this relation between cosine similarity of the aggregated gradients, and the trace of the covariance, and the assumptions under which it holds, will take a few steps to explain clearly, but I believe it will greatly help to clarify and motivate SimiGrad.

That said, I note that a proper estimate of the total variance (or trace of the covariance) is not strictly necessary as SimiGrad requires the user to specify a target value of the cosine similarity (gamma in the paper).  One only requires a quantity that is related to some measure of the variance, up to a constant for example, since one only compares this quantity to a threshold whose units are not critical.

3) It would be useful to clarify the hardware used for the experiments in the paper.  In particular, what size of GPUs do the workers use, and how much VRAM is available.  More details about the parameters used in the experiments would also be appreciated.  Section 5.1 states that the maximum batch size is 2K for CIFAR10, 32K for ImageNet, and 32K for text data with sequence length 512 when training BERT.  The maximum batch size for data with a sequence length of 128 is not specified.  The issue here is that the average batch size in Figure 4 goes far beyond the maximum batch sizes listed.  Perhaps it would be helpful to explicitly specify experiment parameters values in terms of the variables used in the algorithms (ie b_max, m_max, n, and s, etc).

I also have a few minor points:
1) In several places the paper uses the term variance where the correct quantity of interest is the covariance.

2) pg 9, line 391: Given that the experiment discussed just before the conclusion section is in simulation, I don't think the paper should claim a demonstration of real-world impact, but rather potential real-world impact.

3) pg 4, line 190: should be g_i in the summations instead of r_i.


limitations_and_societal_impact: Given the cost of training large models, work like this addressing faster convergence through batch size adaptation is important.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Large batch sizes are required to accelerate the large-scale training of deep learning (DL) models, which often comes at the cost of generalization performance. This paper introduces SimiGrad, an automated method that adapts the batch size at the mini-batch level during DL training. The proposed method achieves state-of-the-art performance on BERT-Large when employing very large batch sizes. 

The first contribution of the paper is the use of the cosine similarity metric to measure the gradient variance of the gradient of a mini-batch. This replaces the typical expensive step that needs to calculate multiple batches and collect gradients from different batch sizes. 

SimiGrad also automatically optimizes the learning rate of the training procedure. The automated adaptation of the learning rate used by SimiGrad is shown to out-perform other learning rate scaling rules when using large batch sizes. 

SimiGrad is integrated into existing ML systems and the experimental results show that state-of-the-art results can be obtained when SimiGrad is applied to BERT-Large training using a record-breaking batch size of 77K.
main_review: In terms of significance, the paper addresses an important and very-well studied problem, namely the optimization of the batch size and learning rate in DL training. It is shown that the proposed method can achieve state-of-the-art results when employing very large batch size. More precisely, the test accuracy of the BERT-Large model is improved over the state-of-the-art accuracy by using 77K batch size.

In terms of originality, the paper uses a simple but effective and novel idea to measure the gradient variance using cosine similarity. Moreover, it introduces an algorithm that automatically adapts the batch size and the learning rate during training. In contrast to many approaches from the state-of-the-art, the learning rate is not adapted using a linear or a squared scaling rules. The learning rate is computed based on the ratio between the new and the old batch size.

The paper is fairly easy to read and includes a comprehensive related work section and a clear explanation of the motivation and challenges of the addressed problem. The clarity and quality could be improved as suggested in the following paragraphs.

- Figure 1 should show that the similarity is used to update the weights of the model.
- In section 4.2, what are the `n` steps in line 217? Is `n` related to `s`? The similarity threshold is a hyper-parameter of the proposed methodology. How is it tuned and how it relates to `n`?
- For consistency with the description of Algorithm 1, the authors should use `B` instead of `b` in Algorithm 2.
- When mentioning modern ML frameworks, please exemplify and provide references. otherwise it is too generic.
- How many nodes have been used for running the experiments, how many GPUs per node?
- How were the similarity threshold and the maximum batch size chosen for the experiments.
- It would be good if the authors would include information about the training times of the different methods. This would allow a good accuracy-time trade-off analysis. Is the time model proposed in (1) validated by the experiments?
- In Table 1, why only 5M samples were used for training AdaBatch? Moreover, why not writing SimiGrad instead of `Ours` in the BERT sub-table.
- What are the integration challenges when implementing the proposed solution into existing ML frameworks?
- How can the model in (1) be extended to support hierarchical scenarios? For examples, two nodes each with X GPUs, the local GPUs are interconnected via a higher bandwidth than the inter-node connection bandwidth.
- Figure 5 shows that with no warmup the batch size explodes to sizes larger than 256K, why this difference between warmup and no warmup?
limitations_and_societal_impact: The authors do not discuss the potential negative societal impact of their work, but that looks reasonable to me. I do not see any negative impact of their proposed work.

In terms of technical limitations of the work, the authors analyze the robustness of SimiGrad to the mandatory gradient similarity hyper-parameter. The authors could further discuss the impact of the optional hyper-parameters on the performance of their scheme.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper study the scaling problem of large batch training.
This paper propose an adaptive method that can automatically adjust batch size to achieve good performance while scaling the training with large batch size.
Different from previous method that gather gradient variance information and adjust batch size at epoch level, the proposed method adjust the batch size at iteration level, which is more adaptive. 
main_review: Strength:
- The idea is straight forward.
- The empirical performance is strong.

Weakness & Questions:
- I have a general question: It seems like the connection between bathsize and learning rate is strong [1]. Can the same intuition be applied to just adjusting learning rate? Such that, we can always fully utilize the hardware with large batch size. 
- I don't see a clear connection between gradient variance and cosine similarity. 
- Is SimiGrad sensitive to adjust interval $n$? The paper claim that iteration level batch-size adjustment is superior to epoch-level adjustment. An ablation study showing that a smaller $n$ leads to better performance is necessary. 
- For CV tasks, the similarity threshold is 0.1, while it is 0.6 for BERT. I am curious what happens if  0.1 is used  for BERT. 
- In line 312, does SimiGrad really achieve 3x speedup? The average batchsize of AdaBatch is merely 1/6 of SimiGrad. I expect the per-iteration speed of AdaBatch will be faster. 

Minor:
- In line 142, where the data "441ms" comes from? which data/model?
- Figure 4. What is 'sota'

Reference:
[1] Don't Decay the Learning Rate, Increase the Batch Size
limitations_and_societal_impact: N/A
review_title: null

Meta-review: All reviewers like the simplicity of the proposed approach and think this paper makes a good contribution towards large batch training of DNNs. Reviewers also raised a number of concerns about methodology and comparisons. Authors have clarified some of them in the response. Overall I think the paper makes some interesting contributions and suggest acceptance. I encourage authors to update the final version addressing all the comments form the reviewers.
Decision: Accept (Poster)