Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Authors study the problem of large-batch optimization for various "dense" computer vision tasks, such as object detection or instance segmentation. Their primary stated objective is to enable training these models with extremely large batches using multiple GPUs.
To achieve this objective, authors analyze the behavior of gradient variance and observe that the variance of different network sub-modules (e.g. detector backbone vs FPN) becomes dissimilar during training with large batch sizes. Based on this observation, authors propose Adaptive Gradient Variance Modulator (AGVM) - an optimizer-agnostic technique meant to balance the gradient variance between the sub-modules. Authors conduct experiments on several vision tasks and demonstrate that AGVM can scale to very large batch sizes.
strengths_and_weaknesses: The paper proposes a practically useful solution to large-batch training with convincing practical experiments and hyperparameter sensitivity analysis.
By extending the applicability of large-batch training, authors not only allow training models in minutes on GPU clusters, but also make it feasible to train "dense prediction" tasks outside of compute clusters, such as in federated learning[1] or even volunteer computing[2], where large-batch training allows one to mitigate high communication latency. While i have many low-level concerns about the paper positioning and quality, I am strongly in favor of paper acceptance, since the proposed method could help "democratize" training in several computer vision problems. I list my concerns below.

### [conceptual] Dense vision tasks vs modular tasks

The paper positions AGVM as a solution to dense visual prediction tasks. However, the method itself does not seem specific to dense prediction, (and does necessarily work for all dense prediction tasks as authors admit in L319-320). Instead, AGVM appears to rely on the fact that a given task contains multiple sub-modules. Naturally, there are many "modular" tasks outside the area of dense prediction:
- NLP pre-training with ELECTRA[3] and derivative work
- reinforcement learning: DDPG[4] or SAC[5] (actor and critic networks)
- generative adversarial networks[6], (generator vs discriminator) cyclic GAN[7] contains 4 sub-modules

In short, there is a plethora of other tasks that appear to fit AGVM's motivation of sub-module variance.
Either the proposed AGVM fits those tasks - in which case, it is not specific to dense prediction - or there is some reason why it doesn't.
I believe that the paper would be more clearly positioned if authors explain why AGVM is specific to dense predition, or evaluate it on more general tasks.


### [practical] On measuring overhead

In Table 1 (and later), authors report "extra overhead" of less than 1%. In its current form, it is unclear what exactly does this overhead mean - and hence, predict how it will generalize to other hardware setups. At the very least, i would recommend clarifying the following issues:

- __communication per step:__ does AGVM double the required communication (MB / step) from the additional gradient all-reduce?
- __maximum memory:__ does AGVM need additional gradient buffers? (and hence, extra gpu memory in proportion to the model size)
- __measured quantity:__ does the overhead from Table 1 represent time, communication, flops, or energy overhead?

Ideally, it would be insightful to evaluate how the proposed overhead scales with network bandwidth and the number of GPUs.
If needed, the former can be emulated using `tc qdisc` on the active network interface.

[1] https://arxiv.org/pdf/1902.01046.pdf

[2] https://arxiv.org/pdf/2106.10207.pdf

[3] https://arxiv.org/pdf/2003.10555.pdf

[4] https://arxiv.org/pdf/1509.02971.pdf

[5] https://arxiv.org/pdf/1801.01290.pdf

[6] https://arxiv.org/pdf/1406.2661.pdf

[7] https://arxiv.org/pdf/1703.10593.pdf
questions: 

### Minor comments / typos:

> L159 they can be easily implemented using the popular deep learning platform e.g., PyTorch

Perhaps it would be better to paraphrase?
(A) **a / any** … platform **e.g.** PyTorch
(B) **the** … platform **i.e.** PyTorch

> Figure 1

The first row colors can be unduly associated with the second row colors. Would recommend using different color schemes.

>Table 3:

Why is batch 1536 in bold? (the caption indicates that bold denotes best results) If this is a deliberate formatting choice, I would recommend changing the caption or highlighting that batch size in a different manner.

> L192 Pytorch

Py**T**orch (consistency)

> The deliverables are released at https://anonymized-agvm.github.io/.

As of the first week of July, the above link only contains figures from the paper.
[Nit] While it does not affect my recommendation, i would still recommend to either use "deliverables **will be** released" or actually provide them during submission.


limitations: To the best of my understanding, the proposed problem (variance mismatch) and the solution (AGVM) appear more general than authors position them to be. As such, I believe it would be best to explain how is AGVM specific to dense visual predictions - or remove this limitation and evaluate it more generally. Furthermore, to the "extra overhead" could be addressed in more detail.
I elaborate on both these concerns above, in the "Strengths And Weaknesses" section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work aims to the heavy performance drop that takes place in the large-batch training regime for object detection and segmentation. Specifically, the authors propose an algorithm named Adaptive Gradient Variance Modulator (AGVM) that is able to work with a very large batch size. This work provides extensive experiments on experiments on MS COCO and ADE20K, which verify the superiority of the proposed method.
strengths_and_weaknesses: Strengths:

+ The paper is easy to follow and well-structured.

+ The experiments are well executed. They show that the proposed method can work with different deep learning backbones (e.g., CNNs and Transformers) and different optimization methods on MS COCO and ADE20K.

Weaknesses:

- It seems that $\mu_{t}^{(i)}$ is sensitive to $\Phi_{t}^{(i)}$ in Equation (4). When $G_{t,1}^{i}$ is similar to $G_{t,2}^{i}$, the expectation of the cosine of the two groups of the gradient estimation is close 0. Then $\Phi$ will be a large value.

- According to Section 3, it seems that the proposed method is generic to most computer vision tasks, like image classification. Why does the proposed method target the object detection and segmentation tasks? Can the proposed method generalize to the image classification task? 

- The paper may lack a discussion about if the proposed method is able to generalize to the imbalance problems in object detection [r1].

References:
[r1] Oksuz, Kemal, et al. "Imbalance problems in object detection: A review." IEEE transactions on pattern analysis and machine intelligence 43.10 (2020): 3388-3415.

questions: Please refer to the weaknesses. I will check other reviewers' comments and the authors' responses. I will adjust my rating accordingly.
limitations: It seems that the authors addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes an Adaptive Gradient Variance Modulator (AGVM) to achieve large-batch optimization for dense visual tasks, i.e., object detection and segmentation. The paper clearly claims the motivation, which is the high gradient variance in the different modules in large-batch dense visual tasks. To overcome the issue in large-batch dense visual tasks, the authors propose the AGVM to make the gradient variance of different modules consistent and moving average to have stable optimization. The AGVM has a theoretical guarantee, and sufficient experiments have also proved its effectiveness. 
strengths_and_weaknesses: Strength
1. The paper is clear and well written. The paper clearly studies the difference between dense visual tasks and normal classification tasks.
2. The method AGVM proposed has a theoretical guarantee.
3. The method is evaluated in different tasks and different optimizers. The experiments show comparable or better performance compared to the previous state-of-the-art optimization.


Weakness
1. The Equ(4) shows the AGVM uses Backbone as the anchor, so the ratio is \sqrt{\frac{\Phi^1}{\Phi^i}}. Why choose backbone? Will using other modules lead to training failure?
2. Table 8 shows the results in different batch sizes. What are the experimental results of other batch sizes, e.g., 128, 512, and the batch size increased 30 times from 32 to 960, but the training time was shortened 20 times, why does AGVM not achieve linear speed? If the bottleneck is data loader IO, please report its time.
3. Table 3 shows the performance in different batch sizes with different optimizers. We wonder about the results of AdamW since the AdamW works well in small batch sizes, i.e., 16, 32.
4. The paper claims that the high gradient variance is the reason for the failure of large-batch dense visual training. It’s recommended to report the same variance-iteration figure as the Figure1 with AGVM to complete the paper.
5. Recently, the work [r1] also study the large-scale DNN training from the perspective of gradient variance misalignment. The authors should give a fair comparison and discussion on this related work.
6. About theoretical analysis.  Can the authors provide a rigorous guarantee for Eq (7) in the appendix, which is merely numerical observation rather than a theoretical analysis? 
7. The theoretical contribution is mild. We recommend the authors provide the linear speedup property of the proposed AGVM with respect to the number of works or mini-batch size. 

[r1] Ko, Yunyong, Dongwon Lee, and Sang-Wook Kim. "Not All Layers Are Equal: A Layer-Wise Adaptive Approach Toward Large-Scale DNN Training." Proceedings of the ACM Web Conference 2022. 2022.

questions: See the weakness. I will consider raising the score if the authors address the above issues.
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a new approach called Adaptive Gradient Variance Modulator （AGVM）for large batch-size training. The approach is well motivated with simple implementation. Experimental results on different kinds of dense prediction tasks like object detection, instance segmentation, panoptic segmentation validate the effectiveness of the approach. More specifically, the paper claims to be able to train with a batchsize of 10K. 
strengths_and_weaknesses: strengths:
1. The proposed approach for large batchsize training is simple and easy to implement. 
2. The proposed AGVM can be generally applied to different visual prediction tasks like instance segmentation, panoptic segmentation.


weakness:
1. Training with large batchsize with the proposed approach leads to performance drop. The performance gap is obvious if larger batch-size is adopted.
2. The speed-up ratio is not constant with more GPUs are utilized. 
questions: 1. According to Table 2, the results with batch-size 1024 (and 512) are usually lower than the setting with batchsize of 32 (or 256), expecially for the case of object detection 35.4 vs 36.8. The performance loss is not negligible with large-batchsize setting. 

2. In the abstract, the paper claims that the batchsize can be scalable to 10K. But the experiments in Table 5 cannot fully support the claim. First, the training setting is compromised with small backbone like Resnet18. How about the performance of the Res50 backbone, which is widely used in other experimental setting? Also, how about other detection methods rather than RetinaNet. Moreover, the reported results of 10K batchsize is much lower than batch-size of 32. 

3. For the results discussed in Table 4, only training time is reported without the converence results. Also, when more GPUs have been adopted, the accelerator is a little bit lower. For example, the training time is 4.2 mins with 1536 GPUs vs 148 mins with 32 GPUs.
limitations: The authors addresed the limitations when the batchsize cannot be well estimated effectively like heatmap based pose estimation. It also addressed the negative social impact when used for deepfake training. 

One limitation which has not reported is that the performance drop when large batch-size has been utilized.
review_title: null

Meta-review: metareview: The authors describe a new method of large-batch optimisation for dense prediction computer vision tasks. The reviewers appreciate the simplicity of the method, convincing experiments and the potential practical importance. AC recommends acceptance.
confidence: Less certain
Decision: Accept