Large-batch Optimization for Dense Visual Predictions: Training Faster R-CNN in 4.2 Minutes
Authors: Zeyue Xue
Jianming Liang
Guanglu Song
Zhuofan Zong
Liang Chen
Yu Liu
Ping Luo
Venue: NeurIPS 2022 Accept (2022)
Keywords: Large-batch Training
Dense Visual Predictions
Object Detection and Segmentation

Abstract: Training a large-scale deep neural network in a large-scale dataset is challenging and time-consuming. The recent breakthrough of large-batch optimization is a promising way to tackle this challenge. However, although the current advanced algorithms such as LARS and LAMB succeed in classification models, the complicated pipelines of dense visual predictions such as object detection and segmentation still suffer from the heavy performance drop in the large-batch training regime. To address this challenge, we propose a simple yet effective algorithm, named Adaptive Gradient Variance Modulator (AGVM), which can train dense visual predictors with very large batch size, enabling several benefits more appealing than prior arts. Firstly, AGVM can align the gradient variances between different modules in the dense visual predictors, such as backbone, feature pyramid network (FPN), detection, and segmentation heads. We show that training with a large batch size can fail with the gradient variances misaligned among them, which is a phenomenon primarily overlooked in previous work. Secondly, AGVM is a plug-and-play module that generalizes well to many different architectures (e.g., CNNs and Transformers) and different tasks (e.g., object detection, instance segmentation, semantic segmentation, and panoptic segmentation). It is also compatible with different optimizers (e.g., SGD and AdamW). Thirdly, a theoretical analysis of AGVM is provided. Extensive experiments on the COCO and ADE20K datasets demonstrate the superiority of AGVM. For example, AGVM demonstrates more stable generalization performance than prior arts under extremely large batch size (i.e., 10k). AGVM can train Faster R-CNN+ResNet50 in 4.2 minutes without losing performance. It enables training an object detector with one billion parameters in just 3.5 hours, reducing the training time by 20.9×, whilst achieving 62.2 mAP on COCO. The deliverables will be released at https://github.com/Sense-X/AGVM.

Large-batch Optimization for Dense Visual Predictions: Training Faster R-CNN in 4.2 Minutes
Zeyue Xue" The University of Hong Kong xuezeyue@connect. hku hk
Jianming Liang" Beihang University
1 jmmm1 jmmm19970gmaill com
Guanglu Song
Zhuofan Zong"
Liang Chen"
Sensetime Research
Beihang Univerisity
Peking University
songguangLudseenetiiine. com
aongzhuofan@gmaiin com
clandzyy@pku. edu. cn
Yu Liu
Ping Luo
Sensetime Research liuyuisanal Sgmmillcom com
The University of Hong Kong, Shanghai AI Laboratory pluoêcs hku.hk
Abstract
Training a large-scale deep neural network in large-scale dataset is challeng- ing and time-consuming. The recent breakthrough of large-batch optimization is a promising way to tackle this challenge. However, although the current ad- vanced algorithms such LARS and LAMB succeed in classification models, the complicated pipelines of dense visual predictions such as object detection and segmentation still suffer from the heavy performance drop in the large-batch training regime. To address this challenge, we propose simple yet effective algorithm, named Adaptive Gradient Variance Modulator (AGVM), which can train dense visual predictors with very large batch size, enabling several benefits more appealing than prior arts. Firstly, AGVM can align the gradient variances between different modules in the dense visual predictors, such as backbone, feature pyramid network (FPN), detection, and segmentation heads. We show that training with a large batch size can fail with the gradient variances misaligned among them, which a phenomenon primarily overlooked in previous work. Secondly, AGVM is alug-and-po module that generalizes well to many different architectures (e.g., CNNs and Transformers) and different tasks (e.g., object detection, instance segmentation, semantic segmentation, and panoptic segmentation). It is also com- patible with different optimizers (e.g., SGD and Adam W). Thirdly, a theoretical analysis of AGVM is provided. Extensive experiments datasets demonstrate the superiority of AGVM. For example, it can train Faster R- Ne5+ in 4.2 minutes without losing performance. AGVM demonstrates more stable generalization performance than prior arts under extremely large batch size (i.e., 10k). It enables training an object detector with one billion parameters just 3.5 hours, reducing the training time by 20.9x, whilst achieving 62.2 mAP on COCO. The deliverables are released at https //gthtittbb.. com /ee///nneee2 X/ AGVM.
the COCO and ADE20K
Work done during an internship at Sensetime Research. CCorrsponniig authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Backbone
FPN
RPN
Detection Head
Mask Head
1.5
Variacce
1.0
Verience
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
0.0
20000
40000
2500
5000
2000
1000
Iterations
nerations
Iterations
Iterations
0.za
IMOLAA
OuT
CAM
Oura Magghtl
5MM
LAUD
Taiiing 1ime
Veg
Thinng
MegDe:
Trining 1re
MAP
MAP 35.0 2
PQ
mloU
40
39
Minutes
2
2'
38
20
27
30.0
37
2
2
27.5
30
32
256 512 10241536
32
256
512
1024
32
256
512
1024
32
512 1024 2048
Batch size
Batch
Batch
Figure 1: First row: Comparisons of the gradient variances (omitting learning rate in $66h referred to Eq. (3)) of different network modules in Mask R-CNN, including backbone, FPN, RPN, and heads. From left right, the models are trained using SGD with mini-hatch size of 32, 256, 512, and 1024, respectively. Note that smaller batch size (32 in the first figure) produces similar " between different modules. When the batch size increases from 256 to 1024 (2nd ~ 4th figures), the gradient variance curves suffer from heavy misalignment between modules. Specifically, the gradient variances are significantly small in the RPN, FPN, detection head, and mask head. We find that the larger the variance gap, the lower the model performance (the best performance is achieved when batch size equals 32). Second row: In figures from left to right, we compare the performance (right vertical axis) and training time of AGVM (bar diagram, left vertical axis) in different visual tasks, including object detection (1at figure), instance segmentation (2"dd, panoptic segmentation (3 ), and semantic segmentation (1th), where the models are trained using different methods with different batch sizes. The "x" "x" indicates training failure when using previous methods. Our method outperforms the recent approaches in all tasks with various batch sizes, significantly reducing training time.
Introduction
The recent successes in many tasks of dense visual predictions rely on the large-scale datasets [1, 2, 3]. the increase of computational power (e.g., GPUs), and the parallel training paradigm with large sample batches. Sufficient computational resource enables large- batch training, greatly reducing the training time [4]. However, although simply scaling the batch size allows fewer iterations to update the parameters of deep neural networks, it often leads to dramatic drop of generalization performance [5, 6, 7].
To reduce the generalization gap in the large-batch training paradigm, LARS [8] scales the batch size of plain ResNet50 from 8k to 32k without losing accuracy. enabling to train an image classification model on ImageNet in a few minutes. However, different from the plain network architectures in ImageNet classification [9, 10, 11], many tasks of dense visual predictions, such as object detection [12, 13, 14, 15, 16] and segmentation [17, 18, 19, 20], are solved by more complicated pipelines, which consist of multiple different modules, such as region proposal network (RPN) [12], feature pyramid network (FPN) [21], detection head, and segmentation head. Nevertheless, the recent advanced large-batch optimization methods such as LARS [8] and LAMB [6] are typically not sufficient to achieve good generalization performance in dense visual predictions. The long training time of dense predictors greatly limits the researchers from making full use of the increasing computational power and large-scale datascts.
To address the above challenge, we present novel latch-batch training algorithm, named Adaptive Gradient Variance Modulator (AGVM), which can train different complicated dense predictors with very large batch size, significantly reducing their training time while maintaining the generalization performance. The design of AGVM is motivated by a training phenomenon overlooked in prior arts. We call it gradient variance misalignment, which would present when visual dense prediction pipeline contains many different modules and is trained with a large mini-batch, where different modules (e.g., backbone, RPN, FPN, and heads) can have different gradient variance magnitudes, impeding the generalization ability.

---

As shown in the first row of 1ig. where Mask R-CNN [17] with ResNet50 [22] as the backbone is trained using different batch sizes, we compare the gradient variances of different network modules, including backbone, FPN, RPN, detection head, and mask head. We see that when the batch size small (32 in the first figure), the gradient variances of different network modules are similar throughout the training process. When the batch size increases from 256 to 1024 (2nd ~ 4th figures), the gradient variances misalign in different modules whose variance gap enlarges during training. Training fails when batch size equals 1024. More importantly the gradient variances have significantly smaller values in the RPN, FPN, detection head, and mask head compared to that in the backbone, and their gradient variances change sharply in the late stage of training (two figures in the middle). We find that such misalignment undesirably burdens the large- batch training, leading to severe performance drop and even training failure. More observations on various visual tasks and networks can be found in Appendix A.2.
The above empirical analysis naturally inspires us to design simple yet effective method AGVM for training dense visual predictors with multiple modules using very large batch size. AGVM directly modulates the misaligned variance of gradient, making it consistent between different network modules throughout training. As shown in the second row of Fig. I, AGVM significantly outperforms the recent approaches of large-batch training in four different visual prediction tasks with various batch sizes from 32 to 2048. For example, AGVM enables us to train an object detector with a huge batch size 1536 (where prior arts may fail), reducing training time by more than 35x compared to the regular training setup.
This work makes three main contributions. Firstly, we carefully design AGVM, which to our knowledge, the first large-batcl optimization method for various dense prediction networks and tasks. We evaluate AGVM in different architectures (e.g., CNNs and Transformers), solvers (e.g. SGD and Add and tasks (e.g., object detection, instance segmentation, semantic segmentation, and panoptic segmentation). Secondly, we provide convergence analysis of AGVM, which converges to stable point in a general non-convex optimization setting. We also conduct an empirical analysis that reveals an important insight: the inconsistency of effective batch size between different modules would aggravate the gradient variance misalignment when batch size is large, leading to performance drop and even training failure. We believe this insight may facilitate future research for large-scale training of complicated vision systems. Thirdly, extensive experiments are conducted to evaluate AGVM, which achieves many new state-of-the-an performances large-batch training. For example, AGVM demonstrates more stable generalization performance than prior arts under extremely large batch size (i.e., 10k). In particular, it enables training of the widely-used Faster R-CNN+ResNett within 4 minutes without performance drop. More importantly, AGVM can train a detector with one billion parameters within just 3.5 hours, which reduces the training time by 20.9x, while achieving a top-ranking mAp 62.2 on the COCO dataset.
2 Preliminary and Notation
Let S = =[(x,,, =((ar,):))) denote a dataset with n training samples, where x; and y/s represent a data point and its label respectively. We can estimate the value of a loss function L Rd + E using a mini-batch of samples that are randomly sampled, and obtain I(w,)
Ejes, L (w,, (zz, 4,)). where S, denotes the mini-batch at the 1-th iteration with batch size |S.| = b and w; represents the parameters of a deep neural network. We can apply stochastic gradient descent (SGD), one of the most representative algorithms, to update the parameters w.. The SGD update equation with learning rate 17 is:
W1+1 W/ - Te Vl(w:),
(1)
where v((w)) represents the gradient of the loss function with respect to Wt-
Layerwise Scaling Ratio. In large-batch training, You et al. [8] observe that the ratio between the norm of the layer weights and the norm of the gradients is unstable (i.e., oscillate a lot), leading to training failure. You et al. [8] present the LARS algorithm, which adopts a layerwise scaling ratio, [ (/ + []. to modify the magnitude of the gradient of the i-th layer Vl(un(")) where (6) and À indicate the parameters of the i-th layer and the weight decay coefficient, respectively. Furthermore, LAMB [6] improves LARS by combining the AdamW optimizer with the layerwise scaling ratio. It can be formulated as T₁ me/ /vr + where m, f,m;-1 + (1 B1) ((w))
Xua
3

---

and = -B20 + (1 V()w)( The layerwise scaling ratio of LAMB can be computed by Juo (9)
11/1/r/
(3) + Aw, 1
Sharpness-aware Minimization. Large-batch training often converges to a sharp local minima, resulting in undesired generalization performance. The sharpness-aware minimization (SAM) [23] algorithm explicitly penalizes the sharp minima and finds the parameters whose neighbors (in an by-ball) have low training loss function values using the following objective function:
(wx))
max (ww, + +). ||
To solve the above equation, SAM applies one-step gradient ascent to determine é pV[((,))[ Vi(w))l Its gradient is then approximated by VISAM (we) = VI(w)]lu.ee How- ever, SAM involves two sequential gradient computations each iteration and thus doubles the computational cost.
Gradient Variance Estimation. Qin et al. [24] utilize the cosine similarity between aggregated gradients from the replicas in a distributed training system, to estimate the gradient variance between SGD and GD efficiently. Specifically, we can compute the gradient for each sample in the t-th mini-batch St of batch size denoted by ri,t:--, -.,,, Pb.t We have VI((w) = Ej= "j We
split the above gradients into two groups and average each group, obtaining Gt,1 / 3 1,
and G1,² = 2 Ej T2j,t, respectively. Then the gradient variance can be measured by 4; = 1 - cos(Gr,11 Gt,2), where cos(",)) is the cosine similarity function.
b
Our Approach
Our goal is to perform large-batch batch training for dense visual predictors with many different network modules. As illustrated in Fig. the inconsistency of gradient variances among different modules need to be modulated.
Gradient Variance across Modules. We derive an updated (considering learning rate) gradient variance to delve into the difference of network modules in complicated dense visual prediction pipelines. The updated gradient variance of the i-th network module the t-th iteration can be formulated as:
Var (neg. (1) b
2n 11 BBcoos(. G(22)) Billg," (0) [|2),
(3)
where and are the number of training samples and the mini-batch size, respectively. is the learning rate. 9: (f) indicates the gradient of the i-th network module. G( and G two (4)
groups of the gradient estimation discussed above. Since each entry in the vector gr could be assumed i.i.d. in a massive dataset following [24, 25], 4 is thus proportional to the above updated gradient variance. At each training iteration, we can approximate the updated gradient variance by ( = (1 - cos( ( G(()). Note that (0 for i-th module has been normalized by the number of parameters, so (²) of different modules are comparable. For consistency of presentation, we still call / gradient variance, which enables to estimate the gradient variance of cach network module at each training iteration. More discussions can be found in Appendix A.I.
Adaptive Gradient Variance Modulator (AG. Let M be a set of modules in a complicated dense prediction pipeline, where M has h different modules. At the t-th iteration, we have set of learning rates, {n( li E
corresponding to different modules. We treat the Backbone = 1) the anchor and modulate other modules making their gradient variances consistent with the Backbone. Specifically, we adjust the module learning rates f)! by using the ratio between ((1)) and
(
$ ( The update rule for each network module can be written
4
!I)
(6) (4) =w?") -1) gi where (t) mepu and Pt (4)
(1) (3)
(8)
W
T (
(4)

---

Table 1: Comparisons between different methods. "Generalization" indicates methods' generalization ability for dense visual prediction tasks. The number of "4" in the column "stable to batch size scaling" means the degree of stability when batch size is increased, whereas the number in the bracket means the maximum applicable batch size without divergence object detection. We measure the average extra overhead of the Faster CNN+ResNNet5 detector each iteration using 128 NVIDIA A100 GPUs (total batch size 1024). The number in the column "extra overhead" indicates the ratio extra overhead (an extra all-reduce call) compared to the original computations. "N/A" means no extra averhead.
Method
Solution
Cieneralization
Less hyperparam. tuning
Stable to batch size scaling
Extra
overhead
MegDet [28]
Accumulate statistics of BN
(1024)
N/A
SAM [23]
Penalize sharp minima
x
X
(2048)
100%
LARS [8]
Rectily layerwise gradient
x
X
+ (1024)
N/A
LAMB
Rectify layerwise gradient
X
X
(4096)
N/A
PMD-LAMB [29]
Reduce historical effect
X
++ (4096)
N/A
AGVM
Balance gradient variance
+++ (10k)
0.12%
where T/ is the global learning rate. However, simply adjusting the learning rates on-the-fly would easily yield training failure due to the transitory large variance ratio that impedes the optimization. We propose a momentum update address this problem. Let a E [0, 1) be a momentum coefficient, we have:
(1) api- + (1 a))u
(5)
which can reduce the influence of unstable variance. Note that we update Hi
(1) each iterations.
Discussion on Momentum and Weight Decay. In practice, the weight decay is widely used as regularizer and is tightly coupled with the learning rate and the momentum. For instance, the gradient
Si ()) will be replaced by the momentum, such as m, 31 mm + (1 Bi (g + Xw)(") [6. 26), where 31 and À indicate the momentum coefficient and the weight decay coefficient, respectively.
We observe that it's also important modulate the learning rate by Eq.(4) when weight decay is
presented. In addition, since the above m, is a momentum- based moving average of (g." + xww ()) can directly apply 1i
((i) onto m. (4)
Extensions Different Optimization Algorithms. AGVM can be easily embedded into different optimization algorithms such as SGD and AdamW. We demonstrate the details in Appendix A.6: Alg.1 and Alg.2, respectively. They can be easily implemented using a deep learning framework egg. PyTorch [27].
Discussion on Convergence Rate. With AGVM, the SGD and the AdamW optimizers still have appealing convergence properties in the general non- non-convex settings. Considering some mild as- sumptions in stochastic optimization and the case without heavy-ball momentum (81 = 0). SGD and AdamW achieve O(1/VT) and ((n(T)/VT convergence rate respectively with appropriate choice of the learning rate т)- We present the analysis in Appendix A.4.
Comparisons with Existing Works. The purpose of exploring large-batch training is to speed up model training with increasing computational power, as well as enabling us to explore the larger dataset. As shown in Table 1, the seminal works such as LARS [8], LAMB [6]. and SAM [23] have made great contributions to large-batch training for plain vision pipelines e.g., image-level prediction, despite that they often require hyper-paramete tuning by experienced engineers. For complicated pipelines of dense visual predictions, they are typically not sufficient to achieve desired generalization performance. MegDet [28] and PMD-LAMB [29] contribute the preliminary attempts by applying large- batch training on object detection. Different from these approaches, we revisit the design paradigm of the complicated dense visual perception pipelines and present a simple yet effective solution, AGVM, which is insensitive to hyperparameter tuning and can be easily plugged into many visual perception pipelines. For example, AGVM can perform stable training with an unprecedented batch size 10K, which could greatly reduce the training time. Moreover, AGVM adds negligible computational overhead training, unlike SAM which involves two sequential paraleeall))) gradient computations at each iteration, resulting in significant increase of the training time.
5

---

Table 2: Comparisons different tasks (ie., object detection, instance segmentation, semantic segmentation, and panoptic segmentation) and pipelines (f.e. Faster R-CNN, Mask R-CNN, Semantic FPN, and Panoptic FPN). All pipelines use ResNet50 as the backbone and we use SGD as optimizer. We see that previous methods' performances drop lot when scaling the batch size and even result in training fäilure when batch size is 1024 ("NaN"). Since LARS always leads to huge performance drop in large-batch settings, performance Mask R-CNN. We also report the comparisons with MegDet and SAM. The hest-performing models are shown in bold. Surprisingly, AGVM can alleviate the training difficultics in large-batch settings.
only report its
Performance
Pipeline
Dataset
Task
Batch size
Iterations
MegDet
SAM
LARS
Ours
32
36.8
36.0
36.8
58640
256
36.1
36.5
36.7
7344
Faster R-CNN
COCO
Detection
512
35.8
35.7
36.7
3680
1024
34.2
33.0
35.4
1840
32
33.9
33.7
34.0
33.9
51310
256
33.7
33.9
32.0
34.1
6426
Mask R-CNN
COCO
Instance Seg
512
33.1
33,0
30.4
33,9
3220
1024
NaN
31.0
25.1
32.6
1610
32
37.5
38.8
37.5
160000
512
36.7
37.6
37.3
10000
Semantic FPN
ADE20K
Semantic Seg
1024
36.4
37.5
37.3
5000
2048
36.2
36.2
37.0
2500
32
38.9
39.0
38.9
$1310
256
39.2
39.3
39.3
6426
Panoptic FPN
COCO
Panoptic Seg
512
38.7
38.7
39.5
3220
1024
NaN
NaN
38.8
1610
Experiments
Dataset. We conduct comprehensive experiments on the MS-CMSSCCOS 2017 [2] and the ADE20K [30] datasets. Specifically, we perform various tasks of object detection, instance segmentation, and panoptic segmentation on COCO, and conduct semantic segmentation on ADE20K.
Baselines. Since the prior arts of large-batch optimization methods can be divided into two types, SGD-based methods (i.e., LARS [8], MegDet [28]) and AdamW-based methods (i.e., LAMB [6],
PMD-LAMB [29]). For fair comparison, we introduce two training configurations using SGD and AdamW with AGVM, respectively. The details of the hyper-paramen pprampprreettr settings can found in Appendix A.5.
Pipelines and Models. To evaluate the generalization ability of AGVM, conduct extensive experiments on different pipelines, including RetinaNet [31], Faster R-CNN [12], Mask R-CNN [17], Panoptic FPN [32], and Semantic FPN [32]. For the backbone networks, use ResNet [22] and Swin Transformer [33]. We strictly follow the official implementations of these pipelines and models.
Implementation Details. We implement AGVM in PyTorch and reproduce PMD-LAMB with the official implementation of LAMB [6]. We also evaluate LARS [8] and SAM [23] by borrowing their official implementations. To make fair comparisons, we follow the same learning rate scaling method all experiments. For SGD optimizer, we use linear learning rate scaling when batch size is less than 128 (256 on semantic segmentation). When the batch size is greater than 128, we the square root of learning rate scaling to avoid divergence in the training process. For PMD-LAMB and LAMB, we follow the learning rate scaling scheme in [29]. We apply learning rate warm-up scheme avoid divergence when the learning rate is large. The implementation details can found in Appendix A.5.
4.1 Comparisons to the State-of- The-AArt Methods
Table 3 compares the results of object detection on the COCO dataset with different backbones and batch sizes. We compare the mAP and the number of iterations of LAMB, PMD-LAMB, and AGVM using the AdamW optimizer. To our knowledge, AGVM reports the first result that successfully scales the batch size to 1536 with negligible performance drop compared to small-batch training using LAMB. We also see that AGVM contributes significant improvements along with the continuous increase of the batch size. By scaling the batch size larger than 1024 for different backbones, AGVM

---

Table 3: Comparisons of performance for object detection the COCO dataset with different backhones batch sizes. compare the mAP the number of iterations AdamW, LAMB, PMD-LAMB. and AGVM+Adam The best performing models are shown in bold. The underlined numbers indicate the results are borrowed directly from [29].
Performance
Pipeline
Backbone
Batch size
Iterations
Adam W
LAMB
PMD-LAMB
AGVM (ours)
32
37.1
36.7
36.7
37.1
43980
256
36.9
36.2
36.7
37.2
5508
Fuster R-CNN
ResNet50
512
36.2
35,5
36.5
36,8
2760
1024
36.2
34.8
35.3
37.0
1380
1536
35.9
33.2
33,5
36,6
924
32
43.6
42.9
40.2
43.7
47645
256
43.4
43.5
42.4
43.5
5967
Fuster R-CNN
Swin-Tiny
512
42.7
42.9
41.3
43.2
2990
1024
42.4
41.6
39.4
42.8
1495
can still achieve 36.6 and 42.8 mAP without heavy hyper-parameter tuning. conclusion, compared with LAMB and PMD-LAMB, AGVM achieves more accurate results whilst reducing training iterations and runtime, AGVM can be embedded CNN and Transformer models.
Generalize to various pipelines, architectures, and optimizers. AGVM can be generalized to different tasks, pipelines, architectures, and optimizers. Table compares MegDet, SAM, LARS, and AGVM in different dense visual prediction tasks, including object detection, instance segmentation, semantic segmentation, and panoptic segmentation on COCO and ADE20K. We evaluate four representative pipelines (e.g., Faster R-CNN, Mask R-CNN, Semantic FPN, and Panoptic FPN) with different batch sizes from to 1024. We see that scaling the batch size only allows fewer iterations to update weights in previous methods, whose performances drop a lot and even have training failure when the batch size 1024 (denoted by "NaN"". In contrast, AGVM yields surprising results in all tasks when increasing the batch size. Table 6 reports the performances of AGVM trained with different optimizers, SGD and AdamW, AGVM works well with both of them.
Table Training time of Faster R-CNN with batch size per NVIDIA A100.
Table Scaling the batch size to RetinaNet with ResNet18.
Batch size
32
256
512
1024
1536
Batch size
32
4k
10k
GPUs
128
256
512
768
PMD-LAMB
31.4
23.5
NaN
Time (min)
148
20.8
11.8
6.0
4.2
Ours
32.8
28.7
26.7
Table 6: AGVM-different optimizers on Faster R- CNN. AGVM works well with both these optimizers.
Table 7: Anchor module selection. We report the segmentation mAP with different anchor modules.
Optimizer
AGVM
Batch
Backbone
mAP
Pipeline
Modules
mAP
SGD
512
ResNet50
35.8
Backbone
33.9
SGD
/
ResNet50
36.7
FPN
33.3
AdamW
X
512
ResNet50
36.2
Mask R-CNN
Detection Head
33.1
AdamW
ResNet50
36.8
RPN
33.1
Mask Head
32.9
AdamW
512
Swin-Tiny 42.7
Swin-Tiny
Training COCO in 4.2 minutes. With AGVM, we can push the frontier of fast training time COCO. We employ Faster R-CNN with ResNet50-FPN as the detector and use the same experimental setting [29]. Then we explore how fast AGVM can reach the 36.6 mAP 0.5.0.0.9 reported in [29] (which needs 12 minutes to train). Different from the hardware setup in Fig. (batch size per GPU), this experiment is conducted on 768 NVIDIA A 100 GPUs. As shown in Table 4, we reduce the original bathh training time from 2.5 hours to only 4.2 minutes, which is the fastest record our knowledge.
Scaling the batch size 10k. We also try to push the frontier of large batch size in dense visual prediction tasks. We choose RetinaNet with ResNet18 the detector, which is trained for 24 epochs

---

Table 8: Extending UniNet [34] to one billion parameters. Both AdamW and PMD-LAMB do not converge when the batch size is 960. On the contrary, our method achieves a top-ranking mAP 62.2 on the COCO dataset, while reducing the training time by 20.9x.
Optimizer
Batch size
Box mAP
Seg mAP
Iterations
Wall-clock time
Adam W
62.6
53.8
43980
73 hours
Adam W
960
NaN
NaN
PMD-LAMB
960
NaN
NaN
Ours
960
62.2
53.4
1349
3,5 hours
Table Insensitive
arper-parmmete r and a. We gradually decrease the update frequency of p from left to right and report the Detection mAP and Segmentation mAP of Mask R-CNN. These results indicate AGVM is not sensitive to these two parammerre..rss. However, when we don't introduce moving average coefficient, the training fails in the early stage.
T o
None
5/9.5
5/0.97
10/0.97
0.97
20/0.98
MAP
NaN
37,5/337 33,9
37.5/34.0
37.57339 33.9
37.67
34.0
(2x) using the AdamW optimizer. For batch size 4k and 10k, the learning rates are 0.001 and 0.0015, respectively. The mAP results on COCO are shown in Table 5. Without bells and whistles, the batch size is successfully scaled to 10k while maintaining generalization ability, but PMD-LAMB fails ("NaN").
Scaling the detector to 1-Billion parameters. We evaluate AGVM on an extremely- detector using the UniNet [34]. We extend it one billion parameters by following the design in [34]. The detailed settings are released in Appendix A.5. Table 8 shows that AGVM still stabilizes and accelerates the training process in such large model regime. Both AdamW and PMI-LAMB diverge in the early training stage. AGVM can reduce the training time from 3 days (batch size 32) to 3.5 hours using 480 NVIDIA A100 GPUs, achieving a 62.2 box mAP on COCO test-dev benchmark. whilst reducing the training wall-clock time by more than 20 times.
4.2 Ablation Study
Insensitive hyper-parameter and a. We study the effect of the interval parameter т, which means we update 14," every T iterations, as well as the coefficient of moving average a using Mask R-CNN. The experimental results in Table 9 indicate that AGVM is not sensitive to these two hyper-parameters. In practice, we employ = 10 and a 0.97 by default. When the batch size is significantly large (e.g., larger than 1K), reduce the interval to 5 to update 14, faster.
Anchor module selection. In AGVM, we choose the backbone network as the anchor and modulate other modules to make their gradient variances consistent with the backbone. To deeply investigate this selection, we choose different modules as the anchors. As shown in Table we see that the backbone is the optimal anchor because the backbone network plays the most important role in dense visual predictions.
Delving into the gradient variance misalignment. We answer an important question: what causes the gradient variance misalignment for dense visual predictors? To tackle this question, we revisit the data flow of dense prediction pipelines and find that the effective batch size is not consistent between different network modules. For instance, due to the shared detection head (i.e., classifiers and regressors) in all the levels of the FPN and different region proposals, the detection head has different effective batch size compared to the backbone. Similarly, the RPN (or detection head in RetinaNet) shared by all FPN levels and pixel-wise loss computation lead to the increased effective batch size in RPN. Similar to a previous work [25], we find that larger effective batch size leads to lower gradient variance of modules ge. RPN, detection head).
To explore these analyses, we conduct a progressive ablation study using the RetinaNet, as shown by the different gradient variance curves in 2 We have three observations. (1) Intuitively, the shared head leads the unavoidable batch size misalignment between the backbone and the detection head. For example, given an input mini-batch size B, the valid mini-batch size for the detection head NB, where N the pyramidal feature number. This motivates us to directly replace the shared detection head by independent detection heads. As illustrated by the second figure in Fig.2, the gradient variance misalignment between the detection head and the backbone has been significantly reduced. (2) Furthermore, compared with the plain network architecture, argue that the effective
8

---

Backbone
FPN
Detection Head
1.5
1.5
1.5
1.5
Variance
1.0
Variance
1.0
Variance
1.0
Variance
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0
2000
0
2000
0
2000
0
2000
Iterations
Iterations
Iterations
Iterations
Figure 2: Ablative experiments exploring the gradient variance misalignment. To validate our result on effective batch size, we progressively use independent detection heads, remove FPN, and mask 75% pixels to reduce the effective batch size on the detection head. Finally, we find near-constant trend of variance throughout training towards convergence between the backbone and the detection head.
batch size is also related to the bottom-up and top-down pathways in FPN. To evaluate this, we remove FPN and only adopt the final-level feature map perform detection. As shown in the third figure in Fig.2, this alleviates the variance difference between the backbone and the detection head. (3) In the fourth figure, we randomly ignore 75% pixels for loss computation in the predictions generated by detection head. This leads to a near- near-constant trend of variance throughout training towards convergence between the backbone and the detection head. We have done a similar study using Faster R-CNN, whose results and discussions can be found in Appendix A.3.
5
Related Work
We review related work on two aspects including dense visual predictions and large-batch optimization. We defer the related work on dense visual predictions to Appendix A.7.
Large-batch Optimization. For large scale deep model training, it is significant to adopt larger batch size for better hardware utilization and system scalability. However, large-batch training prone converge to sharp minima, resulting in undesired generalization ability [7]. The main reason that the number of iterations will decrease when we the number of epochs in large-batch settings. Researchers [35, 36] try to carefully tune the hyper-parametert to narrow this generalization gap. In detail, by incorporating learning rate warm-up and linear scaling, Goyal et al. [5] successfully train ResNet50 with batch size 8192 without loss in generalization performance. Recently, to avoid these hand-tuned methods, the adaptive learning rate on large-batch training has gained enormous attention from researchers. For example, LARS and LAMB algorithms [8. 6] enable researchers to scale the batch size for ResNet50/BERTT to 32k/64k. Both LARS and LAMB leverage the norm of weights and gradients to adjust the learning rate of each layer. These adaptive methods enable researchers to train ImageNet in a few minutes [37, 38, 39]. Johnson et al. [40] propose AdaScale SGD, a novel learning rate schedule rule for stabilizing the warm-up stage. However, it highly depends on the parallelism degree of the system. Liu et al. [41] use adversarial learning to further scale the batch size to 96k. More recently, sharpness-aware minimization (SAM) [23] introduces a procedure to minimize the loss value and loss sharpness to close the generalization gap. However, SAM suffers from training efficiency since the update rule of SAM involves two sequential gradient computation at each iteration. There are few works [42, 43] towards improving the efficiency of SAM. Recently, effort [44] has been made on how to choose an appropriate batch size and corresponding learning rate for batge training. And Qin et al. [24] propose Simigrad, which utilizes a lightweight and automated adaptive batching method to enable fine- fine-grained adaptive batch size. However, rather than classification tasks, there are few works towards batch training for object detection. Peng et al. (28) implement cross-GPU batch normalization to stabilize the training process and Wang et [29] propose PMD-LAMB to reduce the negative effects of the lagging historical gradients. They can scale the training of widely used Faster R- Re5 Detector with batch size 256/1056 with small performance drop.

---

Conclusion
The complicated pipelines of dense visual predictions suffer from heavy performance drop in large- batch training. In this paper, we propose and fully study AGVM, which enables module wise learning rate scaling and successfully scales the batch size larger than 10K with desired generalization per- formance. We also provide a convergence analysis, showing that AGVM+SGD and AGVM+Adam both converge to stable point in the general non-convex setting. Furthermore, we have conducted extensive experiments to show that AGVM can generalize to different complicated pipelines and challenging tasks, including object detection, instance segmentation, semantic segmentation, and panoptic segmentation. We report unprecedented better performance very large batch size. For example, AGVM trains Faster R- R-CNN+ResNee in 4.2 minutes without loss of performance. By increasing the object detector UniNet to one billion parameters, AGVM can achieve 62.2 mAP on COCO using a batch size of 960 in just 3.5 hours, reducing the training time by 20.9 compared to the normal small-batch training.
lurge-batch training with using batch size of 1536
Limitation and Potential Negative Societal Impact. Module partitioning is important estimate the effective batch size quantitatively. For some pipelines without explicit modularity such as the heatmap-based pose estimation, we need to more empirical analysis. We will investigate future. The potential negative social impact is to use the proposed algorithm to speed up the training of fraud models such as DeepFake [45].
the
Acknowledgments and Disclosure of Funding
Ping Luo is supported by the General Research Fund of HK No.27208720, No. 17212222, and No. 1720620.
References
[1] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, 2009, pp. 248-255.
[2] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft coco: Common objects in context," in European Conference on Computer Vision, 2014, pp. 740-755.
[3] M. Cordrs, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, "The cityscapes dataset for semantic urban scene understanding." in Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
[4] Y. You, I. Gitman, and B. Ginsburg, "Large batch training of convolutional networks," arXiv preprint arXiv: 1708.03888,,8 2017.
[5] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, "Accurate, large minibatch sgd: Training imagenet in 1 hour," arXiv Preprint arXiv: arXiv:1706.0266 1706.02677 2017.
[6] Y. You, Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.-J. Hsieh, "Large batch optimization for deep learning: Training bert in 76 minutes," in International Conference Learning Representations, 2020.
[7] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. Tang, "On large-batch train- ing for deep learning: Generalization gap and sharp minima," arXiv Preprint 160.v:1000.04000 2016.
[8] Y. You, Z. Zhang, C.-J. Hsich, Demmel, and K. Keutzer, "Imagenet training in minutes," in Proceedings of the 47th International Conference on Parallel Processing, 2018, pp. 1-10.
[9] L. Chen, Y. Lou, J. He, T. Bai, and M. Deng, "Geometric anchor correspondence mining with uncertainty modeling for universal domain adaptation," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 134-16 143.
10

---

[10] L. Chen, Q. Du, Y. Lou, J. He, T. Bai, and M. Deng, "Mutual nearest neighbor contrast and hybrid prototype self-traing soff-training for universal domain adaptation," Proceedings of the AAAI Conference Artificial Intelligence, 2022.
[11] L. Chen, Y. Lou, J. He, T. Bai, and M. Deng. "Evidential neighborhood contrastive learning for universal domain adaptation," in Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[12] Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," in Advances in Neural Information Processing Systems, vol. 28, 2015.
[13] T.-.. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, "Focal loss for dense object detection," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[14] Z. Tian, C. Shen, H. Chen, and T. He, "Foos: Fully convolutional one-stage object detection," " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp- 9627-9636.
[15] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-end object detection with transformers," in European Conference on Computer Vision, 2020, pp. 213-229.
[16] G. Song, Y. Liu, and X. Wang, "Revisiting the sibling head in object detector," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 563- 11 572.
[17] K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask r-cnn," in Proceedings of the IEEE International Conference Computer Vision, 2017, pp. 2961-2969.
[18] D. Bolya, C. Zhou, F. Xiao, and Y. Lee, "Yolact: Real-time instance segmentation," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 9157- 9166.
[19] Y. Fang, S. Yang, X. Wang. Y. Li, C. Fang, Y. Shan, B. Feng, and W. Liu, "Instances as queries." in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 69 1910-
[20] E. Xie, W. Wang. Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, "Segformer: Simple and efficient design for semantic segmentation with transformers," in Advances in Neural Information Processing Systems, vol. 34, 2021.
[21] T.-Y. Lin, Dollár, Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2117-2125
[22] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778.
[23] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, "Sharpness-awarr minimization for effi- ciently improving generalization," in International Conference on Learning Representations, 2021.
[24] H. Qin, S. Rajbhandari, O. Ruwase, F. Yan, L. Yang, and Y. He, "Simigrad: Fine-grained adaptive batching for large scale training using gradient similarity measurement," in Advances Neural Information Processing Systems, vol. 34, 2021.
[25] J. Wu, W. Hu, H. Xiong, J. Huan, V. Braverman, and Z. Zhu, "On the noisy gradient descent that generalizes sgd," in International Conference on Machine Learning. 2020, pp. 10 107-6110 376.
[26] I.. Smith, "A disciplined approach to neural network hyper-parameters: Part 1-learning rate, batch size, momentum, and weight decay," arXiv Preprint 1rXiv:1803.098220 2018.

---

[27] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, serh-peffrrance deep learning library," in Advances in Neural Information Processing Systems, vol. 32, 2019.
[28] C. Peng, T. Xiao, Z. Li, Y. Jiang, X. Zhang, K. Jia, G. Yu, and J. Sun, "Megdet: A large mini-batch object detector," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 6181-6189.
[29] Wang, Y. Zhu, C. Zhao, W. Zeng. Y. Wang. J. Wang, and M. Tang, "Large batch optimization for object detection: Training coco in 12 minutes," in European Conference Computer Vision, 2020, pp. 481-496.
[30] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, "Scene parsing through ade20k dataset," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017, pp. 633-641.
[31] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1980-2988.
[32] A. Kirillov, R. Girshick, K. He, and P. Dollár, "Panoptic feature pyramid networks," in Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1399-6408
[33] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012-10022.
[34] J. Liu, H. Li, G. Song, X. Huang, and Y. Liu, "Uninet: Unified architecture search with convolution, transformer, and mlp," arXiv Preprint arXiv:2/ 2021.
[35] J. J. Shallue, J. Lee, J. Antognini, J. Sohl-Dickstein, R. Frostig, and G. E. Dahl, "Measuring the effects of data parallelism on neural network training," " Journal of Machine Learning Reseurch, vol. 20, pp. 1-49, 2019.
[36] D. Masters and C. Luschi, "Revisiting small batch training for deep neural networks," arXiv preprint arXiv: 1804 07612, 2018.
[37] X. Jia, S. Song, W. He, Y. Wang, H. Rong, F. Zhou, L. Xie, Z. Guo, Y. Yang, L. Yu et al., "Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes," arXiv preprint arXiv: /205, 2018.
[38] C. Ying, S. Kumar, D. Chen, T. Wang, and Y. Cheng, "Image classification at supercomputer scale," arXiv preprint arXiv: 1.1199..99..... 2018.
[39] M. Yamazaki, A. Kasagi, A. Tabuchi, T. Honda, M. Miwa, N. Fukumoto, T. Tabaru, A. Ike. and K. Nakashima, "Yet another accelerated sgd: Resnet-50 training on imagenet in 74.7 seconds." arXiv preprint arXiv:/903 / 12650, 2019.
[40] T. Johnson, P. Agrawal, H. Gu, and C. Guestrin, "Adascale sgd: A frienfriendly algorithm for distributed training," in International Conference on Machine Learning. 2020. pp. 4911 19200.
[41] Y. Liu, X. Chen, M. Cheng, C.-J. Hsieh, and Y. You, "Concurrent adversarial learning for large-batch training," in International Conference on Learning Representations, 2022.
[42] Y. Liu, S. Mai, X. Chen, C.-J. Hsieh, and Y. You, "Towards efficient and scalable sharpness- aware minimization," arXiv Preprint M.XXx:2203.027144 2022.
[43] Du, H. Yan, Feng, T. Zhou, L. Zhen, S. M. Goh, and V. Y. Tan, "Efficient sharpness- aware minimization for improved training of neural networks," in International Conference on Learning Representations, 2022.
[44] S. McCandlish, J. Kaplan, D. Amodei, and O. D. Team, "An empirical model of large-batch training," " arXiy Preprint 1eXx::8822.0662. 2018.
[45] S. Lyu, "Deepfake detection: Current challenges and next steps," " in 2020 IEEE International Conference on Multimedia & Expo workshops (ICMEW), 2020, pp. 1-6.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes) Please Appendix A.4.
(b) Did you include complete proofs of all theoretical results? [Yes] Please see Appendix A.4.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [No] Codes and models will be released in the future.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Please see Section 4 and Appendix A.5.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No] All our results are using fixed seed since the uncertainty in dense visual predictions is very small.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Section 4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] Please see Section
(b) Did you mention the license of the assets? [Yes] The data used in our work is open source and can be used for adademic research.
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "vh9tDLrExpA", "reviewer": "Reviewer_8KtA", "summary": "This paper proposes an Adaptive Gradient Variance Modulator (AGVM) to achieve large-batch optimization for dense visual tasks, i.e., object detection and segmentation. The paper clearly claims the motivation, which is the high gradient variance in the different modules in large-batch dense visual tasks. To overcome the issue in large-batch dense visual tasks, the authors propose the AGVM to make the gradient variance of different modules consistent and moving average to have stable optimization. The AGVM has a theoretical guarantee, and sufficient experiments have also proved its effectiveness. ", "questions": "See the weakness. I will consider raising the score if the authors address the above issues.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The paper is clear and well written. The paper clearly studies the difference between dense visual tasks and normal classification tasks.\n2. The method AGVM proposed has a theoretical guarantee.\n3. The method is evaluated in different tasks and different optimizers. The experiments show comparable or better performance compared to the previous state-of-the-art optimization.\n\n\nWeakness\n1. The Equ(4) shows the AGVM uses Backbone as the anchor, so the ratio is \\sqrt{\\frac{\\Phi^1}{\\Phi^i}}. Why choose backbone? Will using other modules lead to training failure?\n2. Table 8 shows the results in different batch sizes. What are the experimental results of other batch sizes, e.g., 128, 512, and the batch size increased 30 times from 32 to 960, but the training time was shortened 20 times, why does AGVM not achieve linear speed? If the bottleneck is data loader IO, please report its time.\n3. Table 3 shows the performance in different batch sizes with different optimizers. We wonder about the results of AdamW since the AdamW works well in small batch sizes, i.e., 16, 32.\n4. The paper claims that the high gradient variance is the reason for the failure of large-batch dense visual training. It’s recommended to report the same variance-iteration figure as the Figure1 with AGVM to complete the paper.\n5. Recently, the work [r1] also study the large-scale DNN training from the perspective of gradient variance misalignment. The authors should give a fair comparison and discussion on this related work.\n6. About theoretical analysis.  Can the authors provide a rigorous guarantee for Eq (7) in the appendix, which is merely numerical observation rather than a theoretical analysis? \n7. The theoretical contribution is mild. We recommend the authors provide the linear speedup property of the proposed AGVM with respect to the number of works or mini-batch size. \n\n[r1] Ko, Yunyong, Dongwon Lee, and Sang-Wook Kim. \"Not All Layers Are Equal: A Layer-Wise Adaptive Approach Toward Large-Scale DNN Training.\" Proceedings of the ACM Web Conference 2022. 2022.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cUjnhTO9-fc", "reviewer": "Reviewer_KpQK", "summary": "This work aims to the heavy performance drop that takes place in the large-batch training regime for object detection and segmentation. Specifically, the authors propose an algorithm named Adaptive Gradient Variance Modulator (AGVM) that is able to work with a very large batch size. This work provides extensive experiments on experiments on MS COCO and ADE20K, which verify the superiority of the proposed method.", "questions": "Please refer to the weaknesses. I will check other reviewers' comments and the authors' responses. I will adjust my rating accordingly.", "limitations": "It seems that the authors addressed the limitations and potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The paper is easy to follow and well-structured.\n\n+ The experiments are well executed. They show that the proposed method can work with different deep learning backbones (e.g., CNNs and Transformers) and different optimization methods on MS COCO and ADE20K.\n\nWeaknesses:\n\n- It seems that $\\mu_{t}^{(i)}$ is sensitive to $\\Phi_{t}^{(i)}$ in Equation (4). When $G_{t,1}^{i}$ is similar to $G_{t,2}^{i}$, the expectation of the cosine of the two groups of the gradient estimation is close 0. Then $\\Phi$ will be a large value.\n\n- According to Section 3, it seems that the proposed method is generic to most computer vision tasks, like image classification. Why does the proposed method target the object detection and segmentation tasks? Can the proposed method generalize to the image classification task? \n\n- The paper may lack a discussion about if the proposed method is able to generalize to the imbalance problems in object detection [r1].\n\nReferences:\n[r1] Oksuz, Kemal, et al. \"Imbalance problems in object detection: A review.\" IEEE transactions on pattern analysis and machine intelligence 43.10 (2020): 3388-3415.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_cLFnqNu5vL", "reviewer": "Reviewer_q5yJ", "summary": "The paper presents a new approach called Adaptive Gradient Variance Modulator （AGVM）for large batch-size training. The approach is well motivated with simple implementation. Experimental results on different kinds of dense prediction tasks like object detection, instance segmentation, panoptic segmentation validate the effectiveness of the approach. More specifically, the paper claims to be able to train with a batchsize of 10K. ", "questions": "1. According to Table 2, the results with batch-size 1024 (and 512) are usually lower than the setting with batchsize of 32 (or 256), expecially for the case of object detection 35.4 vs 36.8. The performance loss is not negligible with large-batchsize setting. \n\n2. In the abstract, the paper claims that the batchsize can be scalable to 10K. But the experiments in Table 5 cannot fully support the claim. First, the training setting is compromised with small backbone like Resnet18. How about the performance of the Res50 backbone, which is widely used in other experimental setting? Also, how about other detection methods rather than RetinaNet. Moreover, the reported results of 10K batchsize is much lower than batch-size of 32. \n\n3. For the results discussed in Table 4, only training time is reported without the converence results. Also, when more GPUs have been adopted, the accelerator is a little bit lower. For example, the training time is 4.2 mins with 1536 GPUs vs 148 mins with 32 GPUs.", "limitations": "The authors addresed the limitations when the batchsize cannot be well estimated effectively like heatmap based pose estimation. It also addressed the negative social impact when used for deepfake training. \n\nOne limitation which has not reported is that the performance drop when large batch-size has been utilized. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n1. The proposed approach for large batchsize training is simple and easy to implement. \n2. The proposed AGVM can be generally applied to different visual prediction tasks like instance segmentation, panoptic segmentation.\n\n\nweakness:\n1. Training with large batchsize with the proposed approach leads to performance drop. The performance gap is obvious if larger batch-size is adopted.\n2. The speed-up ratio is not constant with more GPUs are utilized. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WLLtGj3ZObn", "reviewer": "Reviewer_8EAD", "summary": "Authors study the problem of large-batch optimization for various \"dense\" computer vision tasks, such as object detection or instance segmentation. Their primary stated objective is to enable training these models with extremely large batches using multiple GPUs.\nTo achieve this objective, authors analyze the behavior of gradient variance and observe that the variance of different network sub-modules (e.g. detector backbone vs FPN) becomes dissimilar during training with large batch sizes. Based on this observation, authors propose Adaptive Gradient Variance Modulator (AGVM) - an optimizer-agnostic technique meant to balance the gradient variance between the sub-modules. Authors conduct experiments on several vision tasks and demonstrate that AGVM can scale to very large batch sizes.", "questions": "\n\n### Minor comments / typos:\n\n> L159 they can be easily implemented using the popular deep learning platform e.g., PyTorch\n\nPerhaps it would be better to paraphrase?\n(A) **a / any** … platform **e.g.** PyTorch\n(B) **the** … platform **i.e.** PyTorch\n\n> Figure 1\n\nThe first row colors can be unduly associated with the second row colors. Would recommend using different color schemes.\n\n>Table 3:\n\nWhy is batch 1536 in bold? (the caption indicates that bold denotes best results) If this is a deliberate formatting choice, I would recommend changing the caption or highlighting that batch size in a different manner.\n\n> L192 Pytorch\n\nPy**T**orch (consistency)\n\n> The deliverables are released at https://anonymized-agvm.github.io/.\n\nAs of the first week of July, the above link only contains figures from the paper.\n[Nit] While it does not affect my recommendation, i would still recommend to either use \"deliverables **will be** released\" or actually provide them during submission.\n\n", "limitations": "To the best of my understanding, the proposed problem (variance mismatch) and the solution (AGVM) appear more general than authors position them to be. As such, I believe it would be best to explain how is AGVM specific to dense visual predictions - or remove this limitation and evaluate it more generally. Furthermore, to the \"extra overhead\" could be addressed in more detail.\nI elaborate on both these concerns above, in the \"Strengths And Weaknesses\" section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposes a practically useful solution to large-batch training with convincing practical experiments and hyperparameter sensitivity analysis.\nBy extending the applicability of large-batch training, authors not only allow training models in minutes on GPU clusters, but also make it feasible to train \"dense prediction\" tasks outside of compute clusters, such as in federated learning[1] or even volunteer computing[2], where large-batch training allows one to mitigate high communication latency. While i have many low-level concerns about the paper positioning and quality, I am strongly in favor of paper acceptance, since the proposed method could help \"democratize\" training in several computer vision problems. I list my concerns below.\n\n### [conceptual] Dense vision tasks vs modular tasks\n\nThe paper positions AGVM as a solution to dense visual prediction tasks. However, the method itself does not seem specific to dense prediction, (and does necessarily work for all dense prediction tasks as authors admit in L319-320). Instead, AGVM appears to rely on the fact that a given task contains multiple sub-modules. Naturally, there are many \"modular\" tasks outside the area of dense prediction:\n- NLP pre-training with ELECTRA[3] and derivative work\n- reinforcement learning: DDPG[4] or SAC[5] (actor and critic networks)\n- generative adversarial networks[6], (generator vs discriminator) cyclic GAN[7] contains 4 sub-modules\n\nIn short, there is a plethora of other tasks that appear to fit AGVM's motivation of sub-module variance.\nEither the proposed AGVM fits those tasks - in which case, it is not specific to dense prediction - or there is some reason why it doesn't.\nI believe that the paper would be more clearly positioned if authors explain why AGVM is specific to dense predition, or evaluate it on more general tasks.\n\n\n### [practical] On measuring overhead\n\nIn Table 1 (and later), authors report \"extra overhead\" of less than 1%. In its current form, it is unclear what exactly does this overhead mean - and hence, predict how it will generalize to other hardware setups. At the very least, i would recommend clarifying the following issues:\n\n- __communication per step:__ does AGVM double the required communication (MB / step) from the additional gradient all-reduce?\n- __maximum memory:__ does AGVM need additional gradient buffers? (and hence, extra gpu memory in proportion to the model size)\n- __measured quantity:__ does the overhead from Table 1 represent time, communication, flops, or energy overhead?\n\nIdeally, it would be insightful to evaluate how the proposed overhead scales with network bandwidth and the number of GPUs.\nIf needed, the former can be emulated using `tc qdisc` on the active network interface.\n\n[1] https://arxiv.org/pdf/1902.01046.pdf\n\n[2] https://arxiv.org/pdf/2106.10207.pdf\n\n[3] https://arxiv.org/pdf/2003.10555.pdf\n\n[4] https://arxiv.org/pdf/1509.02971.pdf\n\n[5] https://arxiv.org/pdf/1801.01290.pdf\n\n[6] https://arxiv.org/pdf/1406.2661.pdf\n\n[7] https://arxiv.org/pdf/1703.10593.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept