SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement
Authors: Heyang Qin
Samyam Rajbhandari
Olatunji Ruwase
Feng Yan
Lei Yang
Yuxiong He
Venue: NeurIPS 2021 Poster (2021)
Keywords: adaptive batching
large batch size
gradient similarity measure
gradient variance
gradient noise
large scale training

Abstract: Large scale training requires massive parallelism to finish the training within a reasonable amount of time. To support massive parallelism, large batch training is the key enabler but often at the cost of generalization performance. Existing works explore adaptive batching or hand-tuned static large batching, in order to strike a balance between the computational efficiency and the performance. However, these methods can provide only coarse-grained adaption (e.g., at a epoch level) due to the intrinsic expensive calculation or hand tuning requirements. In this paper, we propose a fully automated and lightweight adaptive batching methodology to enable fine-grained batch size adaption (e.g., at a mini-batch level) that can achieve state-of-the-art performance with record breaking batch sizes. The core component of our method is a lightweight yet efficient representation of the critical gradient noise information. We open-source the proposed methodology by providing a plugin tool that supports mainstream machine learning frameworks. Extensive evaluations on popular benchmarks (e.g., CIFAR10, ImageNet, and BERT-Large) demonstrate that the proposed methodology outperforms state-of-the-art methodologies using adaptive batching approaches or hand-tuned static strategies in both performance and batch size. Particularly, we achieve a new state-of-the-art batch size of 78k in BERT-Large pretraining with SQuAD score 90.69 compared to 90.58 reported in previous state-of-the-art with 59k batch size.

SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement
Heyang Qin
Samyam Rajbhandari Microsoft samy Sunnyammmmicrrooftttcomm com
Olatunji Ruwase
University Nevada, Reno heyang-qin qin@nevada unr edu
Microsoft
Aaruaas@@microsoff..cm
Feng Yan
Lei Yang University of Nevada, Reno leiy@unr... edu
Yuxiong He Microsoft
University of Nevada, Reno fyanuunr.. edu
wwwhe@mi www.oed.miicrroooo.oooooo
Abstract
Large scale training requires massive parallelism to finish the training within a reasonable amount of time. support massive parallelism, large batch training is the key enabler but often at the cost of generalization performance. Existing works explore adaptive batching or hand-tuned static large batching, in order strike a balance between the computational efficiency and the performance. However, these methods can provide only coarse-grain gessenerrinedd adaption (e.g., at epoch level) due to the intrinsic expensive calculation or hand tuning requirements. In this paper, we propose fully automated and lightweight adaptive batching methodology enable fine-grained batch size adaption (e.g., at a mini-batch level) that can achieve state- of-the-art performance with record breaking batch sizes. The core component of our method is a lightweight yet efficient representation of the critical gradient noise information. We open-source the proposed methodology by providing plugin tool that supports mainstream machine learning frameworks. Extensive evaluations popular benchmarks (e.g., CIFAR1O. 10. ImageNet, and BERT-Large) demonstrate that the proposed methodology outperforms state-of-the fhe-hhtert methodologies using adaptive batching approaches or band-tuned static strategies in both performance and batch size. Particularly, achieve new state-of-the-art batch size of 78k in BERT-Large pretraining with SQuAD score 90.69 compared to 90.58 reported in previous state-of-the-art with 59k batch size.
Introduction
Recent years have witnessed the success of deep learning in variety of applications, such as computer vision [I] and natural language processing []. To push the performance boundaries, deep learning models many applications have become increasingly complicated, and training such models requires large amounts of data as well as high cost. Taking GPT3 3 (an autoregressive language model that uses deep learning produce human-like text) model has 175 billion parameters and is trained on 300 billion data, which costs $12 million for a single training run 4 Therefore, it is of critical importance to study scalable and efficient training for large scale machine learning [5] (].
example, the state-of-the-art
The core idea of large scale machine learning is to parallelize model training across vast amounts of computing resources speed up the training process. However, the overhead of the model synchronization due to massive parallelism in training can be the bottleneck for large scale training. To reduce the model synchronization overhead, large-batch training is a commonly employed strategy that can reduce the synchronization frequency. Taking BERT training as an example, increasing batch size from 256 4k can increase the training speed 2.5x faster 16-GPU cluster. OpenÅl uses a batch size of 3.2 million to train GPT3 over supercomputing cluster B3, which accelerates the training speed by approximately over 500x compared to sample-by -ammpl run 16-GPU cluster.
35th Conference on Neural Information Processing Systems (NeurIPS 202.)).

---

Despite the benefits of large batch size, large batch size may degrade the model generalization performance 77 9 To address this issue, existing works [TO 12 demonstrate that a carefully tuned learning rate scheduling can help to mitigate such performance loss caused by large batch size. However, manually tuning learning rate requires significant amounts of efforts, which limits its potential use for wide adoption. [33 14 develop automated learning rate tuning methods for large batch size, which, however, rely on users to set appropriate batch sizes. [15] develops adaptive batching using second order information, aiming to reduce the generalization impact due to large batch sizes. [16] uses the gradient variance information to guide the batch size and learning rate configurations. Due to the high cost of analyzing the gradient variance and the second order information, existing methods can adapt the batch size only at coarse- coarse-grained level (i.e., at the end of each epoch). It worth noting that within each epoch, the variance of gradient may change swiftly and significantly [17 [8] (also see Section 4.1). Therefore, the course-graid -rrained methods cannot find a suitable batch size to reduce the performance loss, and thus miss the opportunity to achieve larger batch sizes with less performance impact.
To address the above challenges, we propose fine-grained batch size adaption approach SimiGrad to optimize the batch size at the iteration level. SimiGrad is driven by novel gradient similarity mea- surement that can accurately capture the gradient variance information in a lightweight manner. We use the proposed gradient similarity measurement to derive the optimized batch size and learning rate with an automated algorithm. We integrate SimiGrad into mainstream machine learning frameworks by providing an open-sourced plugin tool.
We evaluate SimiGrad via extensive experiments using popular benchmarks, such as CIFARIO 10, ImageNet, and BERT-Large. The experimental results demonstrate that SimiGrad can consistently outperform state-of-the-ar methods in both the average batch size and the model generalization performance. Particularly, training with the SQuAD score 90.69, compared to 90.58 reported in the ttate-of-the-art LAMB [9] with 59k batch size. We also achieve better Pareto frontier for the trade-off between the batch size and the model performance, compared with state-of-the-an methods.
achieve new state-of-the-an batch size of 78k in BERT-Large pre-
We summarize our main contributions as follows:
We develop an accurate gradient similarity measurement for capturing the gradient variance the iteration level. This measurement can be directly inferred from the gradients of replicas with low computation overhead.
We develop a fully automated fine-grained adaptive batching algorithm that can identify the optimal batch size and learning rate using the gradient similarity measurement.
We integrate SimiGrad into mainstream machine learning frameworks and open-source it SimiGrad enables a record breaking large batch size of 77k for BERT-Large pretraining. With very large batch sizes, SimiGrad able to achieve better performance than state-of-the-ard methods.
2
Background and Related Work
2.1
Mini-batching Training
The objective of training a neural network is to optimize the neural network parameters 0 by minimiz- ing the loss function (()). One typical method of training a neural network is the gradient descent method [20). which computes the gradient of (0) with respect to 0 over the entire dataset; however, could be too expensive for training deep neural networks with large training datasets. typical solution is based on mini-batch stochastic gradient descent (SGD) [2I]. which samples a small subset, namely mini-batch, from the dataset and approximates the gradient using this batch instead. Such approximation introduces inevitable noise to the gradients. By increasing the batch size (i.e., the number of samples), we can reduce the noise level, but is diminishing return, meaning that after sweet point we can gain almost no more information by further increasing the batch size [22]. In other words, the gradient computed on a too small batch is biased with noise, whereas too large batch leads to a waste of time and insufficient updates to the model (33]. This makes the batch size a key control knob, and the choice of the batch size is of critical importance. In practice. a large batch may be too large to fit into GPU memory. The solution [24] 25 is to break it into many micro batches and accumulates the gradients of all micro batches. This method is called gradient accumulation, which is widely used large scale training.
2.2 Large Batch Size Training
For large scale neural network training, it is desirable to have a large batch size, as it improves hardware utilization and system scalability as well as reduce wall clock training time, However, there
'Code repo is available http://ggihbbbboomttttgggguiiinn

---

is a dispute as to how exact large batch size would impact the model performance. In 77, it is reported that training neural network model with a large batch size leads a non-trivial generalization gap compared with a small batch size. [10] suggests that this gap could be closed by adjusting the training regime. [ observes from a large number of training runs that this gap could be mitigated if the learning rate is well-tuned for a large batch size.
[12] further observes that the range of "good"
learning rates is shrinking as the batch size grows. M further argues that with carefully tuned hyper parameters, the smaller batch sizes still outperform large ones in terms of test accuracy.
Despite the disagreement over large batch size's generalization performance, much effort has been made on how to choose an appropriate batch size and the corresponding learning rate for neural network training. [26] formulates SGD as stochastic differential equation and suggests that there exists an optimal noise scale related the batch size, the learning rate and the dataset. Along this line, [16] formulates the noise scale in a different way and tests it in a variety of applications. Another line of research explores the potential of large batch size training from practical perspective by fine tuning the training setting, especially the learning rate. [24] suggests that the learning rate should be increased proportionally to the square root of the increase ratio of the batch size. [13] proposes the warmup learning rate scheduling that can scale up the batch size to 8k for ResNet--0 on ImageNet.
The most closely related works to this paper are AdaScale SGD 14 and gradient noise scale [16]. AdaScale SGD [4] proposes a novel learning rate scaling rule based on the gradient variance for training with a large batch size; however, its performance hinges on the choice of suitable batch size, which is challenging to determine in practice. Gradient noise scale [T6] proposes to predict the batch size by comparing the gradients before and after the allreduce call in the data parallelism as an alternative implementation. SimiGrad differs from gradient noise scale (16) in three ways: 1) the approach in [16) highly depends on the parallelism degree of the system, whereas SimiGrad supports flexible parallelism degree; 2) the measurement proposed in [16] is noisy and needs to use a hand-tuned exponential decay parameter to stabilize it; and 3) [16] assumes the learning rate is well-tuned and close enough to optimal, whereas SimiGrad has no assumptions on learning rate. In particular, by evaluating [16] in a common cluster of 16 GPUs, we observe a very high noise level. We also observe that the noise level decreases as the number of GPUs increases, which indicates that the design in [16) highly depends on the parallelism degree. In comparison, SimiGrad can derive relatively stable measurements in a parameter free and system independent manner as shown in Fig.
3
2.3 Adaptive Batching
Using fixed batch size throughout the training may not be the best practice. A desirable approach is to use adaptive batch sizes during training, which is suggested by research from both theoretical [22] and practical [27] 28) perspectives. [22] converts the existing learning rate schedule into the batch
size adjustment schedule. Yet the optimal learning rate is difficult to obtain in practice and the batch
size adjusted in this way has limited scalability. [27 28 use fixed multi-step batch size schedulers that closely resemble the multi-step learning rate scheduler. Recent works focus on utilizing the gradient noise variance information to adjust batch sizes by various rules and measurements. [29] proposes coupled criterion for adjusting both the batch size and the learning rate. [15) adjusts the batch size using the second order information and adversarial training. (16) proposes a gradient noise scale to adjust the batch size.
3 Motivation and Challenges
Due to the high cost of analyzing the gradient variance and the second order information, existing methods can adapt the batch size only at a coarse-grained level (i.e., at the end of each epoch). Our key observation is that the gradient variance may change swiftly across iterations within one epoch (see Fig. 2 in [17] and Fig. 3 in (18)). which cannot be captured by the existing methods. Taking [16] as an example, to get the gradient variance information, it needs to at least 20 batches with different batch sizes. To perform such analysis at each iteration, the extra overhead due to gradient variance analysis would be more than 20 times of the original training, which is infeasible in practice. Due to these issues, the batch size usually needs to be compromised and large scale training methods that achieve state-of-the-ar performance still heavily rely on manual tuning efforts [. O).
To address these issues, one promising solution is by improving the granularity of gradient analysis, which, however, is challenging due to the following reasons.
To compute the gradient variance, we need to first collect the gradient information from batches at different sizes and then take large amounts of computing time and resources. For example, computing the gradient variance for a single 8k batch in BERT training takes 44 14 and the time grows significantly with the batch size.
3

---

Forward
GPU Replica
Replica
Backward
workflow
Propagation
For each half repliccas
Reduce
Similarity
Reduce
Compute Cosine Similarity
Reduce
Calculation
Reduce
replicas
GPU Replica
GPU Replica
Weights Update
batch size
Group 1
Group
SimiGrad Workflow
Figure Workflow how SimiGrad is built into normal data parallelism pipeline.
Figure 2: Cosine similarity computation from two groups of replicas.
Even if we can obtain the gradient variance, it is not straightforward to derive the optimal batch
size as the gradient covariance is a big matrix.
Existing batch adaption methods usually require a fine-tuned learning rate scheduler to achieve good performance. Fine tuning the learning rate at the iteration level requires significant amounts of manual efforts, which may not be feasible in practice.
Due to the frequent gradient analysis, the system deployment is no longer trivial engineering problem as the analysis and tuning overhead can disrupt the well optimized training pipeline in existing machine learning frameworks. For example in [16], executing multiple batches with different batch sizes to compute the gradient variance is not supported by the training pipeline and may lead to inefficient memory use.
4 SimiGrad: Fine-Grained Adaptive Batching Methodology
To address the above challenges, we propose fine-grained adaptive batching methodology SimiGrad that supports swift batch size adaption at the iteration level. The core component of SimiGrad novel lightweight gradient variance measurement that can capture the gradient variance change per iteration without expensive analysis. Our adaptive batching algorithm uses the gradient variance measurement to guide the batch size configuration so that the batch size is maximized while the gradient variance is maintained scheduling and requires little manual effort. Finally, we integrate SimiGrad into mainstream machine learning frameworks with optimized real system performance and open-source for easy adoption.
steady level. SimiGrad also supports automated learning rate
4.1 Measure Gradient Variance via Cosine Similarity of Batching
Existing works typically estimate gradient vari- ance via collecting gradients with different batch sizes [16], which, however, is quite expensive due the calculation of multiple batches well
Similarity between Batch Gradients in BERT Training
batch size 4 batch size 1024
batch size 65536 batch size 262144
1
Cosine Similarity
batch size 4096
as the process of collecting gradients from dif- ferent batch sizes. In this paper, we propose new way to measure the gradient variance by computing the cosine similarity of the gradient of mini-batch.
0.8
0.6
0.4
0.2
Specifically, let G = (g1, 92, 9m) be the
0 0
gradients of batch of size 32 The indi- vidual gradient 91, I...... Im is assumed i.i.d. given a large enough training set [30]. De- note G1 (gl1.912, gl....., 9.. glm) and G2 (g21, observations from the distribution of G. The cosine similarity of aggregated gradients G1, G2:
200
Number of Sample (Million)
Figure Cosine similarity in BERT pretrain.
002,,922,-0 9222..2, 92m. as the aggregated gradients collected by SimiGrad, which are two independent
G1 G2
cos cos(G1, G1, G2)
||G||
|G22|
m
m
G1 G2
gl:g2,
(gl, + 92.)² (91. - g2.)²
(2)
1=1

---

Taking the expectation on both sides, we have
1 m
1 m
B[G1 .2) - Σ
E(gl++9244
Σ
Eggl - -222)²
1=1
==1
For random variable X, we have Var(X) - IE(x)) - [[(]] Using this formula, we rewrite the above equation as follows:
B(G1-G22
: Σ Var(gl, 4
_ [[gl:+922) Σ
Var(s1;-924 >
2m
(4)
As gl, g2 are assumed to follow the same distribution, we have (arggl++221) Var(gl; 924) and E(gl; 92,) Thus, we have
1 m
m
B(G1 G2)
Σ
B(g1; + g2;)l
> [Egg)]]²
(5)
4
E[G1 - G2)
>m [E{(:)]]
E(cos(G1 (G1 G2))
(6)
E
G[
[G2
TE BO
T (9:)²)
We can derive the relationship between the trace of covariance (total variance) Tr(Cou (G) of gradients and the cosine similarity measurement cos(G1, G2) proposed in SimiGrad as follows:
Tr(Cou(G)) Σ Var(g.) - 2
m
m
E(gi)
> [(()))) (1 E-os (G1, G2))) Σ E(g?). (7)
Note that the Tr(Cou (G)) are computed with half batch size 4 To get a relationship between trace of covariance Tr(Cou( G full) ) on full batch size and the cosine similarity, we could use the conclusion from 30 to scale the variance as
b
m
Tr(Cou(Gyuut)
- Tr(Cou(C))
E(cos G2)
Σ
E(g?),
(8)
2n
2n
where n is the total data samples and is the batch size.
Thus, the cosine similarity is closely related to the gradient covariance. As the cosine similarity does require computing additional results using different batch sizes, our approach allows to evaluate the gradient variance at each iteration with little overhead.
At each iteration, the cosine similarity is computed using aggregated gradients, where utilize the replicas in distributed training systems. A replica a local copy of model and optimizer on certain hardware (typically GPU) 31 32). As shown in Fig [ in each training step, all replicas r1, ..., Tn. start with the same weights but with different mini-batches. After the feed forward and backward propagation of loss, each replica has its own gradient information G G2, .... G,, Then allreduce, a collective operation [33], called aggregate gradients from replicas, thanks to its excellent capability of utilizing network bandwidth to achieve fast communication.
aggregated by allreduce, split replicas equally into two groups, denoted as (r1, rs, and (r2, r.,...). We perform aggregation using allreduce in each group. This process
Before the gradients
is visualized in Fig. 2 The aggregated gradients in each group denoted G1 2 Σ G2i-
and G2 Σ = Gzs respectively. These are equivalent to the gradients of two mini-batches with half of the batch size. Then we compute the cosine similarity of the two group gradients as the measurement of gradient variance & = cos(G1, G2). Once the gradient variance is computed, we aggregate the two group gradients again using allreduce compute the final aggregated gradients. At this point, we reach the step as the normal workflow.
When computing the cosine similarity, the training pipeline is not interrupted and the main overhead is an additional allreduce call and a cosine similarity computation. This is a significant cost reduction, compared with the gradient variance evaluation in previous works that need to collect multiple additional batches with different batch sizes or use heuristic algorithmss Taking BERT-Large pretrain on 16 GPU cluster as example, the overhead of an extra allreduce call and cosine similarity
5

---

computation is only around 10-15 ms, which is irrelevant to the batch sizes. In comparison, the gradient computation of a single 8k batch of BERT takes about 44 and the time grows with the batch size and the need of computing multiple batch sizes [16].
To demonstrate the proposed cosine similarity is effective measure for tracking the gradient variance, we plot the cosine similarity over the BERT pretraining for different batch sizes in Fig. It is observed that the similarity of the gradients increases with the batch size. For the small batch size of 4, due to the high "noise" in the gradients, the cosine similarity is almost always O. For the batch sizes of 1024 and 4096, the gradients are stable initially. However, the gradient quickly becomes unstable and thus the cosine similarity drops. For the large batch sizes of 65536 and 262144, 144, the cosine similarity is relatively stable over the time, which corresponds to the stable gradients we get.
4.2 Automated and Fine-grained Adaptive Batching Algorithm
Using the cosine similarity, we propose an automated and fine- ane-grained adaptive batching algorithm to derive optimized large batch sizes. The total batch size is denoted as B m d, where is the micro batch size on each replica, s is the gradient accumulation steps, and is the number of replicas. In practice, the number of replicas d depends on the number of available GPUs and the micro batch size m is limited by GPU memory. Thus the factor is given and m has an upper bound Mmax-
Algorithm| D provides an overview of our adaptive batching algorithm. For every steps, we compare the measured gradient similarity & with the user defined similarity threshold y and update the target batch size B accordingly using a predefined incremental step (e.g.. 10%) to make the batch gradient similarity as close the threshold as possible. Then, Algorithm 2 is invoked to update the micro batch size m and the gradient accumulation steps to make the effective batch size as close to the target batch size as possible. Based on the updated batch size, we update the learning rate accordingly.
Note that in Algorithm T the only required parameter the similarity threshold yy and the other parameters are optional to improve the flexibility of the algorithm. Specifically, the adjust interval p is used when training on small batch sizes or slow networks to reduce overhead. The batch size lower bound Bmin is introduced for special scenarios where a well-tuned learning rate scheduling is provided. The batch size upper bound Biax is used when training with only limited epochs to ensure a reasonable number of updates to the model. In Sec. 5.3.2) we perform an ablation study to analyze the impact of the parameters in Algorithm [ The results indicate that Algorithm [ performs well with a broad range of even without any optional parameters.
Algorithm I: Adaptive Batching Algorithm Overview
User specifications: similarity threshold 7, adjust interval (optional), batch size lower bound Buin (optional), batch size upper bound Brax (optional), micro batch size upper bound mmax (optional).
Initialization: learning rate modifier a 1. If mmax not set, Mmmax m.. If p not set, p 1. For every steps in training
1) Retrieve similarity measurement +
2) If +- < Y, target batch size B = 0.9B, else, B = 1.1B.
3) Update the batch size to approximate B based on Algorithm 2
4) Update the learning rate modifier according to Oun Dord
Bold
Algorithm 2: Adaptive Batch Size
Input: Target batch size B. micro batch size upper bound Mmaa. batch size lower bound Buin (optional), batch size upper bound Bmax (optional). (Here, '\" means integer division) 1) If Bina set, bound the batch size B min B, Binax).
2) If Bmi set, bound the batch size B- = max B, Bmin).
3) If mmax set, update micro batch size
min B',d, mmax).
4) Update gradient accumulation steps s = B/mldd.
4.3 Learning Rate Scheduling Support
[3] suggest that the learning rate should be adjusted while adapting batch sizes in order to achieve good performance. We employ a learning rate scaling rule from [10 24 [9] to adjust the learning rate automatically using a square root factor of the batch size
Existing works [10
12
26
24

---

is worth noting that SimiGrad works with or without warmup [3] If warmup is not used (i.e., a random initial learning rate is used), SimiGrad can still achieve good performance. It is interesting to observe that the learning rate automatically adjusted by the learning rate scaling rule resembles similar pattern as the fine-tuned warmup scheduling (see Fig. 5). This indicates SimiGrad is robust and provides good support for the learning rate adaption.
change (see Step 4 of Algorithm{]] D).
4.4 Machine Learning System Integration
We can further optimize SimiGrad runtime performance by instrumenting the allreduce operation. An allreduce process can be decomposed into: reduce-scatter process, an allreduce among groups, and an allgather process inside each group. After the reduce-scatter process, each replica holds one portion of the reduced gradients in each group. Thus we can directly compute the cosine similarity from the gradients. Then the allreduce is executed among groups but only exchange the portion of the reduced gradients. Here the transferred data volume size is inversely proportional to the degree of parallelism and thus the overhead is small. Finally, the allgather is executed inside each group. In this way. we can avoid the overhead of an extra allreduce to further reduce the overhead of SimiGrad.
Modern machine learning frameworks are built with fixed micro batch sizes. Even if we can instrument the framework to make support flexible batch sizes, it may disrupt the optimized existing training pipelines, such as the widely used NVIDIA DALI[34). Therefore, we opt to design a batch dispatch queue that accumulates the data samples until the miero batch size reached before aggregating them into a micro batch to be executed. This general design enables our approach to be easily integrated into almost any machine learning framework.
Furthermore, we do not directly change the learning rate as the client scripts may operate or depend on it (e.g., custom learning rate scheduler). Instead, we use a learning rate modifier factor to indicate the learning rate scale with the adaptive batch size, which is applied during model weights update.
5 Evaluation
We evaluate SimiGrad via extensive experiments using popular benchmarks and demonstrate the superiority of SimiGrad over these benchmarks using the following criteria:
Performance: we compare the performance of SimiGrad with state-of-the-ar adaptive batching and large batch training techniques to demonstrate superior results on ResNet18, ResNet50, and BERT. Generalization: we demonstrate that SimiGrad establishes a better Pareto frontier of average batch size under different evaluation metrics, and sustains good generalization at very large batch sizes. Robustness: we show that SimiGrad does not need manual learning rate warm-up schedule required by most large batch training techniques or careful tuning of hyperparameters, which makes SimiGrad robust against hyper- aperrpprameter variations.
Scalability: we demonstrate that SimiGrad allows deep learning training to scale more efficiently on a wide range of GPU clusters with both slow and fast interconnect.
5.1 Evaluation Setup
Evaluations are performed Ubuntu 18.04 on NVIDIA DGX-2 nodes (16 V100 GPUs). The applications are evaluated with DeepSpeed 0.2, Pytorch 1.6+, and CUDA 10.1 and 1 11.1. To ensure fair comparison, the hyperparameters, including learning rate and seeds, are kept the same between the baseline and our approach evaluations. Gradient accumulation is used when the batch size exceeds the graphic memory limit. For SimiGrad, the optional adjust interval p is by default for all evaluations and the gradient accumulation steps is adjusted during the training using Algorithm 2
88 on CIFARIO: The training lasts 200 epochs with SGD optimizer. The initial learning rate is 0.1, which decays at epoch 80 and 120 by factor 0.1, respectively. For SimiGrad, we set the similarity threshold 0.1 and the maximum batch size 2048
ResNer50 on ImageNet: The training lasts 90 epochs with SGD optimizer. The initial learning rate is 0.1, which decays at epoch 30, 60, and 80 by factor 0.1, respectively. For SimiGrad, we set the similarity threshold 0.1 and the maximum batch size 32k.
BERT Pretrain on WiKi and BookCorpus Datasets: We pretrain BERT Large 2 from scratch for 170 epochs on WiKi and BookCorpus datasets with fixed sequence length 128 and 512 using LAMB optimizer. The training procedure follows the recommendation from the BERT paper (2]. based on which we train epoch 1-150 using the sequence length 128 and epoch 151-170 using the sequence length 512. We evaluate it for two cases: (i) well-tuned warmup learning rate scheduling is used and (ii) a simple learning rate decay is used. For the warmup case, the learning rate follows an exponential increase to 0.011 at epoch 9 and then decays with factor of 0.9 for every 250 steps until epoch 150. For the simple case, a fixed learning rate 0.0006 with a linear decay is used until epoch 150. For both cases, the sequence length 512 training follows a warmup scheduling
7

---

Table 1
ADAPTIVE BATCH RESULTS
ResNet18 CIFARIO (SGD)
Number of Updates
Average Batch Size
Number of Samples
Test Accuracy (%)
Baseline
78125
10M
94.28
AdaBatch
15137
330
5M
94.38
AdaptDL
7489
1335
10M
93.07
SimiGrad
5094
1963
10M
94.33
(y=0.1, 2048)
ABSA
4327
2311
10M
85.85
ResNet50 ImageNet (SGD)
Number of Updates
Average Batch Size
Number of Samples
Test Accuracy (%)
Baseline
56250
2048
115M
76.56
Hand-tuned Baseline
7038
16384
115M
75.3
SimiGrad
23284
4952
115M
76.85
(y=0.11, bmin= =2k. Drn 32k)
Gimi (y=0.4, mmin=2KK 1maa=320
7007
16456
115M
75.71
BERT Large Pre-Training
Number of Updates
Average Batch Size
Number of Samples
SQuAD score
BERT Paper Baseline
1000000
256
256M
90.9
LAMB Paper Baseline
8599
59542
512M
90.58
NoWarmup Baseline
106793
4096
437M
86.9
SimiGrad with warmup (1=0.5, =64k)
5620
77831
437M
90.69
SimiGrad with warmup (1=0.8, bmann
3425
127711
437M
89.82
SimiGrad without warmup (ym0.7, mma 128k) SimiGrad without warmup (7=0.8)
6604
66236
437M
90.87
4779
91530
437M
90.39
SimiGrad without warmup (n=066
7207
60694
437M
90.52
until learning rate 0.02 and then decay with factor of 0.9 for every 150 steps. The batch size for sequence length 512 is fixed at 32k.
State-of-the-Art Adaptive Batching Methods. We cross-compare the performance of SimiGrad with other approaches that can adaptively adjust the batch size during training as well as the hand- tuned baselines. The evaluation metrics the average batch size and the final generalization performance. The approaches we compared with are:
2288atch(28): a hard tuned strategy that gradually increases the batch size b and the learning rate a along the training to keep b/a constant.
AdaptDL: a library that adaptively sets the batch size based on the gradient noise scale 16 and the learning rate according AdaScale [14].
ABSA/75): an adaptive batching scheme using second order methods and adversarial training.
5.2 Performance Analysis
For CIFAR-10, Table [ indicates that SimiGrad achieves a higher test accuracy than the baseline, while keeping the largest average batch size among the other approaches that have similar test accuracies. Compared with AdaBatch that uses 15k model updates, SimiGrad uses only 5k model updates, which indicates almost 3x speedup in terms of the number of updates.
For ImageNet, Table T that SimiGrad achieves better test accuracy with twice large the baseline batch size. Note that we use the vanilla SGD without tricks (such as smoothing and mixup data augmentation) and sophisticated learning rate scaling (such as LARS). Under this setup, the state-of -ote-----hhhrrr the-art approach (35) achieves 75.3% test accuracy with 16k batch size under well-tuned learning rate scaling, warmup, and decay. To show the performance of SimiGrad with larger batch

---

(a) ResNet18 on CIFAR10
(b) ResNet50 on ImageNet
(c) BERT Large Pretrain
SimiGrad
sota
90
lr
75.0
SimiGrad
80
linear Ir warmup
sqrt warmup
72.5
SQUAD 1.1 Score
*
70.0
90.4
70-
Test
67.5
00.2
60
65.0
50
62.5
sota
SimiGrad
90.0
60.0
1k 4k 8k
16k
32k
4k8k
16k
32k
48k
60k
80k
100k
120k
Average Batch Size
Avarage Batch
Average Batch Size
Figure 4: Pareto frontier of the average batch size under different performance metrics, For the same average batch size, SimiGrad always performs better than the other benchmarks. "sota' (state-of-the-art) in (b) is from 35 and 'sota' in (c) is from [9).
sizes, we test more aggressive settings with similarity threshold 0.4, minimum batch size 2k, and maximum batch size 32k. Under this setting, the average batch size of SimiGrad reaches 16k, while keeping the test accuracy at 75.71% % which is 0.4% higher than the hand-tuned results in [35..
For BERT Large pretraining, the model has 340 million parameters. The results in Table I shows the advantage of SimiGrad in large scale training over other approaches (the original BERT paper and the state-of-the-art I LAMB). Specifically, even with an ultra large batch size 128k (using similarity threshold 0.8), SimiGrad can achieve comparable SQuAD scores. With less extreme batch size 66k (using similarity threshold 0.5), SimiGrad can achieve a better SQuAD score 90.85 than the tfth-offthheart LAMB that uses an average batch size 59k and achieves SQuAD score 90.58.
5.3 Generalization and Robustness
To demonstrate the generalization and robustness of SimiGrad with large batch sizes, we generate the Pareto frontier of the average batch size under different performance metrics for different applications by grid search. As illustrated in Fig. 4 it can be observed that the model performance degrades as the batch size increases, which agrees with the observations in related work. It is worth noting that SimiGrad is able to keep the model performance reasonably good for wide range of batch sizes and constantly outperforms the of-the--rrr art approaches.
5.3.1 Robustness under Learning Rate Schedules
Learning rate warmup and learning rate scaling are two common techniques used improve the model generalization for large batch training. Fig. [aa shows the impact of two common learning rate scaling rules (the linear scaling and squared root scaling) with large batch size; for both scenarios, we also show the impact of learning rate warm up. As illustrated in Fig. a)), the results with warmup have better test accuracies than the counterpart without warmup, and the squared root scaling is better than the linear scaling. However, despite the effectiveness of the learning rate warmup and the learning rate scaling, identifying an optimal schedule that works well for a certain batch size requires a non-trivial tuning effort.
Notably, SimiGrad can mitigate this effort and outperform even well tuned schedules with minimal tuning. To show the robustness of SimiGrad with untuned learning rate, we evaluate SimiGrad's performance for BERT pretrain without using any warmup but with a uniuned learning rate with linear decay (see the details in Sec. 55. The results are presented in Table Compared with the scenario of well-tumed learning rate, SimiGrad with untuned learning rate chooses more conservative batch sizes (still larger than state-of of-the-at art batch sizes) but has better SQuAD scores. These results also outperform the state-of-the-an art results. Compared with SimiGrad, No Warmmu Baseline in Table (the same untuned learning rate with LAMB without any warmup schedule using a fixed batch size 4k) shows a non-trivial performance degradation.
The reason why SimiGrad is robust against the learning rate warm-up is because it uses the gradient similarity metric automatically adjust the effective learning rate at run time. From Fig. B(a). can be observed that even when no learning rate warmup schedule is provided, SimiGrad is able to come up with warmup-like learning rate scaling based the adaptive batch size, which not only saves considrrable tuning efforts but also leads to better model performance than the hand-tuned warmup schedules. When highly tuned warmup schedule is available for large batch size, SimiGrad can leverage it to start the training from a pre-tuned large batch size, and increase it further over the training. which ultimately reaches larger average batch size.
9

---

(a) Learning Rate
(b) Batch Size
(c) Cosine Similarity
(d) Train Loss
256k
1.0
12.5
No Warmup
0.010
10.0
Warmup
0.5
7.5
0.005
5.0
64k
32k
0.0
0.000
100M200M300M400M
100M200MB00M400M
100M200M300M4000
100M200M300M400M
Figure 5: Details of BERT Large pretrain with adaptive batch size in sequence length 128.
5.3.2
Robustness Ablation Study
We perform the ablation study to evaluate SimiGrad's robustness against different parameters, First, get rid of all optional parameters (such batch size bounds) and extra learning rate scheduling (warmup), and run the BERT pretrain with the gradient similarity 7 = 0.6. The results are shown in the bottom of Table 1 which shows comparable performance as the other experiments and outperforms the baselines. This indicates that SimiGrad can work well without any optional parameters nor extra learning rate scheduling. Then, in the same setting we evaluate the impact of gradient similarity values [0.6, 0.7, 0.8] on the performance of SimiGrad. The corresponding SQuAD scores are 90.52, 90.22, and 90.38 with the average batch size 60k, 75k, and 91k, respectively. This further indicates that SimiGrad can perform well under different gradient similarity thresholds.
We also evaluate how the adjustment interval impacts SimiGrad performance for ResNet18 on CIFAR- 10. The adjustment intervals tested are 5,10,100,20 100,200,0000 The corresponding number of updates are 6053, 6456, 9333, 1 11299, 15732 and the test accuracy are 94.27, 94.23, 94.34, 94.39, 94.36. The ablation study results suggests that the scalability (batch size) gets worse as the adjustment interval grows while the accuracy remains the tta
5.4 Real-world Scalability and Speed Up Indication
Large batch size enables training the model by the all-reduce communication. If we assume that a GPU can compute samples per second, then batch size of B can be processed by N gpus in
large number of GPUs without getting bottle-necked
B NX
B NX
N - N
M
Aleration_limee
+ allreduce_ _cott
+2x
bw
where, the absolute allreduce cost can be written as function of N, network bandwidth (bw) between GPUs and the model size (MM). But relatively, more time is spent in allreduce as N gets larger B gets smaller. Therefore, large batch size is necessary to achieve good efficiency.
Using (9). we can evaluate the potential impact of batch sizes on end-to-end training time as a function of GPU cluster size and
Bandwidth 1GB/s
Bandwidth 5GB/s
Bandwidth 60GB/s
30
500
2000
GPUNN Number
400
1500
GPU-GPU bandwidth within the
300
cluster. We measure X using NVIDIA V100 GPU, and simu- late the end-to-end training time for BERT-Pretraining on a small, medium and large cluster with slow (1GB/s), moderate (5GB/s) Figure 6: Cosine similarity of differrnt batch sizes in BERT pretrain.
200
500
500
750
1000
1250
40
60
100
10
20
Run Time (mins)
Run Time (mins)
Run Time (mins)
SimiGred
SimiGrad
and fast interconnect (60 GB/s). Our simulation in Fig. Gishows that SimiGrad achieves 23.67%. 40.44%, and 32.97% reduction in training time compared to baseline on 32, 512 and 2048 GPUs on the small, medium and large clusters with slow, moderate and fast interconnect, respectively. This demonstrates the potential real world impact of our approach in improving training scalability across a wide range of actual GPU clusters with varying network bandwidth.
6 Conclusion
In this paper, we propose SimiGrad, a fine-grained adaptive batching methodology for enabling swift batch size adaption the iteration level. The core component of SimiGrad is lightweight gradient similarity measure based on cosine similarity and an enhanced synchronization approach. Our optimized machine learning system integration allows SimiGrad to be executed with negligible overhead in practice. SimiGrad achieves record breaking large batch size of 78k for BERT- Large pretraining with SQuAD score of 90.69.
10

---

Acknowledgments and Disclosure of Funding
This work is supported in part by the National Science Foundation grants CAREER-2048044 and IIS-1838024, We thank the anonymous reviewers for their insightful comments.
References
[1] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6105 61 14. PMLR, 2019.
[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLL 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Pupers), pages 1176.. Association for Computational Linguisties, 2019.
[3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Burbert-Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[4] Kyle Wiggers, 2020. http:://eenuubeee...o///220000/0 htw.://acaiieeaaaaiiiii caapllngg-ppnniigiiee size-isat-everrtthnnn,; [Online].
[5] Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang. Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang. Liwei Yu, Tiegang Chen, Guangxiao Hu, Shaohuai Shi, and Xiaowen Chu. Highly scalable deep learning training system with mixed-precision Training imagenet in four minutes. CoRR, abs/1807 1205, 2018.
[6] Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Extremely large minibatch SGD: training resnet-50 on imagenet in 15 minutes. CORR, abs/171 1..0112,,, 2017.
[7] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On larch-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[8] Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus Roburt Müller. Efficient backprop. In Grégoire Montavon, Genevieve B. Orr, and Robert Müller, editors, Neural Nenworks: Tricks of the Trade Second Edition, volume 7700 of Lecture Notes in Computer Science, pages 9-48. Springer, 2012.
[9] Samuel L. Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 0058-9067. PMLR, 2020.
[10] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 17411, 2017.
[11] Christopher J. Shallue, Jachoon Lee, Joseph M. Antognini, Jascha Sohl-Dicksten Roy Frostig, and George E. Dahl. Measuring the effects of data parallclism on neural network training. J. Mach. Learn. Res., 20:112:1-112:49 2019.

---

[12] Dominic Masters and Carlo Luschi. Revisiting small batch training for deep neural networks CoRR, abs/1804.07612 2018.
[13] Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training imagenet in 1 hour. CoRR, abs/1 706.1006777 2017.
[14] Tyler B. Johnson, Pulkit Agrawal, Haijie Gu, and Carlos Guestrin. Adascale SGD: A user- friendly algorithm for distributed training. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4911 1-200. PMLR, 2020.
[15] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W. Mahoney. Large batch size training of neural networks with adversarial training and second-orde information. CoRR, abs/18 10.00022,, 2018.
[16] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. CoRR, abs/ 18 11.06616,, 2018.
[17] Liyuan Liu, Haoming Jiang. Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[18] Fartash Faghri, David Duvenaud, David J. Fleet, and Jimmy Ba. A study of gradient variance in deep learning. CoRR, abs/2007. .04522, 2020.
[19] Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[20] Haskell B Curry. The method of steepest descent for non-linear minimization problems. Quarterly of Applied Mathematics, 2(3):258-260 1944.
[21] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Yves Lecheval- lier and Gilbert Saporta, editors, 19th International Conference on Computational Statistics, COMPSTAT 2010, Paris, France, August 22-27, 2010 Keynole, Invited and Contributed Papers, pages 177-186. Physica-Verlas 2010.
[22] Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V. Le. Don't 't decay the learning rate, increase the batch size. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[23] Fengxiang He, Tongliang Liu, and Dacheng Tao. Control batch size and learning rate to generalize well: Theoretical and empirical evidence. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1141-1150, 2019.
[24] Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. CoRR, abs/1404.5997, 2014.
[25] John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12:2121-21 -2599, 2011.
[26] Samuel Smith and Quoc V. Le. bayesian perspective on generalization and stochastic gradient descent. In 6th International Conference on Learning Representations, ICLR 2018, Van- couver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[27] Soham De, Abhay Kumar Yadav, David W. Jacobs, and Tom Goldstein. Big batch SGD: automated inference using adaptive batch sizes. CoRR, abs/1610.05792, 2016.
12

---

[28] Aditya Devarakonda, Maxim Naumov, and Michael Garland. Adabatch: Adaptive batch sizes for training deep neural networks. CoRR, abs/1712.02022, 2017.
[29] Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. In Gal Elidan, Kristian Kersting, and Alexander T. Ihler, editors, Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence, UAI 2017, Sydney, Australia, August 11-15, 2017. AUAI Press, 2017.
[30] Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes SGD. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10367-10376. PMLR, 2020.
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Bus Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 8024-8035. 2019.
[32] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In Kimberly Keeton and Timothy Roscoe, editors, 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2-4, 2016, pages 265-283. USENIX Association, 2016.
[33] NVIDIA. The nvidia collective communications library (necl). https://githut com/
NVIDIA/ncc1]
[34] NVIDIA. The nvidia data loading library (dali). http:://githuu..cmm//DDDD
[35] Chris Ying. Sameer Kumar, Dehao Chen, Tao Wang, and Youlong Cheng. Image classification at supercomputer scale. CoRR, abs//811.0699/ 1.06992, 2018
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section
Did you include the license to the code and datascts? [No] The code and the data are proprictary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We provide ablation study in section 5,3.2
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
13

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Our framework and all training code can be found at the anonymous Github repo weggghhwwooooo/eyggnnnnnniiiiiie
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please section 5.1
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] All our results (except the GLUE finetuning as suggested not to by BERT paper) using fixed seed
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see section 5.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "zyw8Kpz1_xg", "reviewer": "Reviewer_mR3m", "summary": "Large batch sizes are required to accelerate the large-scale training of deep learning (DL) models, which often comes at the cost of generalization performance. This paper introduces SimiGrad, an automated method that adapts the batch size at the mini-batch level during DL training. The proposed method achieves state-of-the-art performance on BERT-Large when employing very large batch sizes. \n\nThe first contribution of the paper is the use of the cosine similarity metric to measure the gradient variance of the gradient of a mini-batch. This replaces the typical expensive step that needs to calculate multiple batches and collect gradients from different batch sizes. \n\nSimiGrad also automatically optimizes the learning rate of the training procedure. The automated adaptation of the learning rate used by SimiGrad is shown to out-perform other learning rate scaling rules when using large batch sizes. \n\nSimiGrad is integrated into existing ML systems and the experimental results show that state-of-the-art results can be obtained when SimiGrad is applied to BERT-Large training using a record-breaking batch size of 77K.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In terms of significance, the paper addresses an important and very-well studied problem, namely the optimization of the batch size and learning rate in DL training. It is shown that the proposed method can achieve state-of-the-art results when employing very large batch size. More precisely, the test accuracy of the BERT-Large model is improved over the state-of-the-art accuracy by using 77K batch size.\n\nIn terms of originality, the paper uses a simple but effective and novel idea to measure the gradient variance using cosine similarity. Moreover, it introduces an algorithm that automatically adapts the batch size and the learning rate during training. In contrast to many approaches from the state-of-the-art, the learning rate is not adapted using a linear or a squared scaling rules. The learning rate is computed based on the ratio between the new and the old batch size.\n\nThe paper is fairly easy to read and includes a comprehensive related work section and a clear explanation of the motivation and challenges of the addressed problem. The clarity and quality could be improved as suggested in the following paragraphs.\n\n- Figure 1 should show that the similarity is used to update the weights of the model.\n- In section 4.2, what are the `n` steps in line 217? Is `n` related to `s`? The similarity threshold is a hyper-parameter of the proposed methodology. How is it tuned and how it relates to `n`?\n- For consistency with the description of Algorithm 1, the authors should use `B` instead of `b` in Algorithm 2.\n- When mentioning modern ML frameworks, please exemplify and provide references. otherwise it is too generic.\n- How many nodes have been used for running the experiments, how many GPUs per node?\n- How were the similarity threshold and the maximum batch size chosen for the experiments.\n- It would be good if the authors would include information about the training times of the different methods. This would allow a good accuracy-time trade-off analysis. Is the time model proposed in (1) validated by the experiments?\n- In Table 1, why only 5M samples were used for training AdaBatch? Moreover, why not writing SimiGrad instead of `Ours` in the BERT sub-table.\n- What are the integration challenges when implementing the proposed solution into existing ML frameworks?\n- How can the model in (1) be extended to support hierarchical scenarios? For examples, two nodes each with X GPUs, the local GPUs are interconnected via a higher bandwidth than the inter-node connection bandwidth.\n- Figure 5 shows that with no warmup the batch size explodes to sizes larger than 256K, why this difference between warmup and no warmup?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss the potential negative societal impact of their work, but that looks reasonable to me. I do not see any negative impact of their proposed work.\n\nIn terms of technical limitations of the work, the authors analyze the robustness of SimiGrad to the mandatory gradient similarity hyper-parameter. The authors could further discuss the impact of the optional hyper-parameters on the performance of their scheme.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "_s_kYozhuhL", "reviewer": "Reviewer_44BP", "summary": "This paper study the scaling problem of large batch training.\nThis paper propose an adaptive method that can automatically adjust batch size to achieve good performance while scaling the training with large batch size.\nDifferent from previous method that gather gradient variance information and adjust batch size at epoch level, the proposed method adjust the batch size at iteration level, which is more adaptive. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The idea is straight forward.\n- The empirical performance is strong.\n\nWeakness & Questions:\n- I have a general question: It seems like the connection between bathsize and learning rate is strong [1]. Can the same intuition be applied to just adjusting learning rate? Such that, we can always fully utilize the hardware with large batch size. \n- I don't see a clear connection between gradient variance and cosine similarity. \n- Is SimiGrad sensitive to adjust interval $n$? The paper claim that iteration level batch-size adjustment is superior to epoch-level adjustment. An ablation study showing that a smaller $n$ leads to better performance is necessary. \n- For CV tasks, the similarity threshold is 0.1, while it is 0.6 for BERT. I am curious what happens if  0.1 is used  for BERT. \n- In line 312, does SimiGrad really achieve 3x speedup? The average batchsize of AdaBatch is merely 1/6 of SimiGrad. I expect the per-iteration speed of AdaBatch will be faster. \n\nMinor:\n- In line 142, where the data \"441ms\" comes from? which data/model?\n- Figure 4. What is 'sota'\n\nReference:\n[1] Don't Decay the Learning Rate, Increase the Batch Size", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "Pjmq6_AXgP_", "reviewer": "Reviewer_xSCv", "summary": "\nThis paper addresses the issue of parallel, large mini-batch SGD, and how to adapt learning rate  and batch size. The aim is to find a sweet spot with batches large enough reduce gradient estimator variance while not too large, with incremental gains in gradient variance reduction, and potentially harming generalization.  Specifically, they consider a fast way to compute the gradient variance.  The issue is that computing the covariance for the gradient estimator is very expensive, especially if done so with different batch sizes (as is done, eg in [16]).  While prior methods for adapting LR or batch size have computed the gradient variance relatively infrequently, e.g., every epoch, this paper argues that gradient variance can vary significantly on finer time scales, and hence they aim to estimate gradient variance and adapt batch at every mini-batch.  The experiments in the paper are extensive, and they show promising results with extremely large batch sizes.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this paper.  The topic will be of interest to a large fraction of the NeurIPS community, and the innovations and experiments in the paper appear promising.\n\nI do have several concerns and suggestions that the authors may wish to address.\n\n1) Section 4.1 begins 'Existing works typically estimate gradient variance via collecting gradients with different batch sizes'.  I think it might be worth explaining that for IID data, the estimator variance for the mean should decreases like 1/N multiplied by the variance of the RV. So from a single batch one can in estimate the gradient covariance. A noisy estimate of the mean can, however be problematic, in which case it is sometimes effective to use two samples.  This is explained in Appendix A.1 in [16] for example (ie https://arxiv.org/pdf/1812.06162.pdf).\n\nIndeed, there are strong similarities between the approach proposed in the current paper and in reference [16].  In particular, Appendix A.1 in [16] appears to describe a method similar to SimiGrad that they say is \"free in a data-parallel environment\".  It outlines a similar idea to SimiGrad that uses averages before and after allreduce.  This I think a more detailed discussion of similarities and differences with [16] may be useful.\n\n2) The relation between normalized gradients, cosine similarity, and the gradient covariance is not well explained in the paper, but should be.  This is a major omission, but it can be corrected reasonably straightforwardly.  In particular, one can show that the cosine similarity in this case is affinely related to the trace of the covariance matrix for the normalized gradient vector.  In this sense, cosine similarity is closely related to the total variance of the normalized gradients in the batch.  Explaining this relation between cosine similarity of the aggregated gradients, and the trace of the covariance, and the assumptions under which it holds, will take a few steps to explain clearly, but I believe it will greatly help to clarify and motivate SimiGrad.\n\nThat said, I note that a proper estimate of the total variance (or trace of the covariance) is not strictly necessary as SimiGrad requires the user to specify a target value of the cosine similarity (gamma in the paper).  One only requires a quantity that is related to some measure of the variance, up to a constant for example, since one only compares this quantity to a threshold whose units are not critical.\n\n3) It would be useful to clarify the hardware used for the experiments in the paper.  In particular, what size of GPUs do the workers use, and how much VRAM is available.  More details about the parameters used in the experiments would also be appreciated.  Section 5.1 states that the maximum batch size is 2K for CIFAR10, 32K for ImageNet, and 32K for text data with sequence length 512 when training BERT.  The maximum batch size for data with a sequence length of 128 is not specified.  The issue here is that the average batch size in Figure 4 goes far beyond the maximum batch sizes listed.  Perhaps it would be helpful to explicitly specify experiment parameters values in terms of the variables used in the algorithms (ie b_max, m_max, n, and s, etc).\n\nI also have a few minor points:\n1) In several places the paper uses the term variance where the correct quantity of interest is the covariance.\n\n2) pg 9, line 391: Given that the experiment discussed just before the conclusion section is in simulation, I don't think the paper should claim a demonstration of real-world impact, but rather potential real-world impact.\n\n3) pg 4, line 190: should be g_i in the summations instead of r_i.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Given the cost of training large models, work like this addressing faster convergence through batch size adaptation is important.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers like the simplicity of the proposed approach and think this paper makes a good contribution towards large batch training of DNNs. Reviewers also raised a number of concerns about methodology and comparisons. Authors have clarified some of them in the response. Overall I think the paper makes some interesting contributions and suggest acceptance. I encourage authors to update the final version addressing all the comments form the reviewers.
Decision: Accept (Poster)