Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a simple algorithm named ConAdv to incorporate adversarial training into the large-batch training setting such that one can further increase the batch size without harming too much accuracy while maintaining the high utilization of the hardware. The core idea is to use adversarial training to improve the accuracy for large batch training and at the same time use stale weights to allow parallel computation of the adversarial example and the normal gradient updates. With his simple yet novel approach, the paper has demonstrated good performance on ImageNet with batch size as large as 96k while maintaining the accuracy above MLPerf's 75.9.
main_review: ## Strength

- Well-written. The motivation is clearly depicted and so is the thought process to the proposed solution.

- The proposed approach is simple yet novel and useful. As demonstrated in the experiments, using stale gradients can achieve a great speed up over the DisAdv baseline with no noticeable accuracy differences.

## Weaknesses

- It seems that the local machine has to support two parallel models. From what I understand, Fig 1b demonstrates the case where the local machine supports a stale model and a current model so both can be executed by different workers in parallel. If a model is so large that a single machine can't fit two models, then it creates extra communication overhead with other workers for communicating stale weights and adversarial examples. Long story short, using stale weights as a solution might lead to additional communication overhead that might be an issue depending on the setting of the training system.
summary_of_the_review: Overall, I think this paper is well-written and it provides a novel, simple, yet useful algorithm for further increasing the batch size for large-batch training.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel strategy which adding adversarial data for training to improve the performance of large batch training. Experiments also verify its efficiency.
main_review: My main concerns are the scalability of the proposed augmentation strategy and the gap between theorems and experiments.

Details:
1. The experiments mainly focus on ResNet-50/ImageNet training. How to set a large batch size for a new training problem? Whether 96k, 64k or 32k batch size is suitable for other problems?
2. The authors actually prove a non-asymptotic convergence rate and I also notice that LARS is used for the proposed methods in experiments. Such a gap may lead to many unknown problems. Furthermore, according to my experience, LARS is sensitive to the hyper-parameters and NaN often occurs for large batch training.

summary_of_the_review: Above all, the authors should improve the theory for a better understand of the adversarial data. It would be better if the authors propose the strategy on how to choose a suitable large batch size for a given problems.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This manuscript empirically show that adversarial training in large-batch training scenario has better performance than traditional data augmentation. And furthermore, the authors proposed a simple method to conduct adversarial example generation and gradient computation w.r.t. to weights concurrently to accelerate the adversarial training in distributed setting. The key strategy is to use staled weights to generate adversarial examples, and then decoupled the bi-level optimization. 
main_review: The paper is clearly motivated and well written. To the best of my knowledge, the findings regarding the adversarial training for improving generalization performance in **large-batch training** is new. Although the proposed method is simple and straightforward, and the theoretical analysis is easy to conduct, the empirical results are nice. 

The authors conducted various experiments, in which many empirical findings are quite interesting. However, the results lack of reasonable interpretations. For example, 
1.	About the perturbation magnitude. From the experiment result in Table 5, it shows we should increase the attack intensity as the batch size increasing and augmentation used. Could you elaborate a little bit why should be this? It is a well known conclusion that there exists accuracy-robustness trade-off when using adversarial training, which means that larger perturbation might ruin the clean accuracy. This appears contrary to your finding. It is a really interesting point. 
2.	Another important side regarding the training details is that whether the pure adversarial training helps, i.e. not mixing the clean data and adversarial data for training the networks. I think the authors should provide this result to deepen the understanding the role of adversarial examples for generalization. 

summary_of_the_review: New empirical findings regarding the advesarial training helps generalization in large-batch training scenarios, but some results lack of deeper understanding.
review_title: null

Meta-review: Thank you for your submission to ICLR.  This paper presents a straightforward but reasonable approach to (slightly) improving the performance of large-batch training via adversarial training.  The basic approach is to apply (small epsilon) adversarial training, shown to help performance in small-batch settings, but accelerate the method using stale parameters to allow for parallel computation of the perturbations.  This speeds up adversarial training while improving performance, enough to enable it to be more effective than existing techniques for this large-batch setting.

The reviewers are not entirely in agreement about this paper, but I personally found the objections of the reviewer to be fairly generic, and not really addressing the core contributions of the paper.  However, I also felt that the overall contribution of this work seems somewhat incremental, using a not-particularly-unexpected result (that we can use stale gradients for this form of adversarial training) to achieve moderate speedup in what ultimately seems like one point in hyperparameter space.

That all being said, though, clearly the authors are working within standard benchmark frameworks, and "simple" algorithms here are indeed a positive rather than a negative.  So I am inclined to slightly recommend the paper for acceptance.
Decision: Accept (Poster)