Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors group common data augmentation into two groups: simple and complex.
The authors propose to use separate batch normalization layers for the two types of data augmentations, simple and complex. They argue that this helps to reconcile the distribution shift between the two types of augmentations. They also propose minimizing the JSD between the output of a network with perturbed weights when given different augmentation of a sample as an input. They show improvements in the clean and robust accuracy on multiple datasets. 
strengths_and_weaknesses: Strengths:

- The authors' approach shows emperically gains in robustness and accuracy on multiple datasets.
- The methods work well on small datasets.
- The method is relatively computationally efficient.
- The authors try to explain the results using the transfer learning theory proposed in [4].



Weaknesses and Questions:
1. The paper is not very easy to follow.
2. The distintion between simple and complex is not very clear. For example: flip results in a large jump in the pixel space. Pad and crop is similar to translate in AA. I'm not sure how the authors draw the line and sperate the two augmentations. Many AA have little effect on the pixel space statistics.
3. Line 143-151:  I don't think the results shown conclusively support the arguments presented. I think the section should be revised to better explain what the authors mean. For example, the authors say (Line 145) `There is a large difference between the augmented
 data and test data in pixel space, although they may be similar in feature space. `. I think the differnce between the augmented data and the  test data is similar to that between the augmented data and the adverserial test data in the pixel space, since the adversarial examples are only an $\epsilon$ away from the real test data.
4. Table 1 should include baseline models trained with neither of the data augmentation types and both. 
5. Lines 221-223: I don't understand the reasoning here. $\gamma$ and $\beta$ are learned parameters and expected to be different when trained on different data, even if the data disterbution is similar. $\gamma$ and $\beta$  scale and add a bias to the normalized data (equation in line 208). Figure 1 shows the the mean and var are the same for both augmentation types (in cos similarity) which hints the the statstics of both augmentation types are similar. This contracdicts the argument the authors presenting ? 

**UPDATE:**
I appreciate the authors' comprehensive response and clarification of the ambiguous points. I must admit that I am still not convinced that the authors' method of splitting the augmentation is completely sound. However, the authors provide strong empirical evidence to support their approach.

questions: Please address the points listed under Weaknesses and Questions above.
limitations: n/a
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tackles the problem of Lp-norm robustness for classification. More specifically, previous papers have shown that data augmentations are helpful for adversarial robustness for certain conditions: network with big capacity and data augmentations have to preserve low-level features. Here, this paper shows that these conditions can become unnecessary by using data augmentation specific batch normalization layers. In this way, complex data augmentations will not impact the network capacity. The authors also propose an optimised training strategy with: 1) increasing perturbation radius throughout training and 2) using JS divergence.
strengths_and_weaknesses: Strengths: 
* The paper is very clear and well-written. This work starts from previous results in the literature to come up with a simple but effective approach regarding the use of data augmentation for adversarial training.
* Very thorough experimental section and appendix. This type of paper with many ablation studies and in-depth analysis is very useful for the community as it can be a great source of inspiration for future work.
* Experimental results speak for themselves with clear improvement on robust and clean accuracies.

Weaknesses:
1) The usefulness of the ascending perturbation radius could be more extensively checked in the paper. Currently, in Figure 2, panel (a) shows better performance on clean accuracy but not really on robust accuracy in panel (b) as the robust accuracy at 8/255 seems the same when training at 8/255 with or without ACAT.  Hence, it is not clear that it is a key element in DAJAT as there is no ablation study of DAJAT without ascending perturbation radius (sorry if I missed it) and it would have been nice to have in Table 2 ACAT vs fixed perturbation radius (besides just having Figure 2 for this comparison).
2) It is not clear whether this approach stills helps for larger architectures as split batch normalization layers is already a way of cleverly increasing the network expressiveness with few parameters. Maybe, this approach has diminishing returns when the network has more capacity. This would require further investigation but I understand that it might require too much compute to check it.
questions: * Please see above the weaknesses section: especially point 1) which could be easily addressed.

* Just a quick related work comment: split batch normalization layers have been used for supervised learning setting (page 3) in AdvProp and similar line of work (Pyramid Adversarial Training) and not only self-supervised learning setting.
limitations: The authors did not mention any negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper combines many previous robust training techniques together: adversarial training, Auto Augmentation, Split batch normalization, smoothness regularization (JSD loss), progressively hardened adversarial training, etc.
strengths_and_weaknesses: Strengths:
The paper is well written, and the method achieves good empirical performance. 

Weakness:
1. The technical novelty is limited. 
This proposed method is just a simple combination of many previous robust training techniques together: adversarial training, Auto Augmentation, Split batch normalization, smoothness regularization (JSD loss in AugMix), progressively hardened adversarial training [1,2], etc. It is undoubtedly a good technical report but hard to justify its technical novelty as a research paper. 

[1] Curriculum Adversarial Training. IJCAI, 2018.
[2] Improving Adversarial Robustness Through Progressive Hardening. 2020.

2. Unfortunately, I find some claims in the paper to be wrong. 

For example, in lines 211-214: "Prior works use separate batch normalization layers for clean and adversarial images to improve the performance of adversarial training, specifically in a self-supervised learning scenario [46, 27, 14]. However, in a supervised learning setting, state-of-the-art defenses often use single batch normalization layers for both clean and adversarial images."

This is wrong. Unfortunately, your paper is not the first to use separate BN in supervised learning. In fact, separate BN is originally proposed and most widely used in supervised learning. Just to name a few: [3,4,5,6,7,8]

[3] Intriguing properties of adversarial training. ICLR, 2020. 
[4] Adversarial examples improve image recognition. CVPR, 2020.
[5] Does data augmentation benefit from split batchnorms. 2020.
[6] Once-for-all adversarial training: In-situ tradeoff between robustness and accuracy for free. NeurIPS, 2020.
[7] Shape-texture debiased neural network training. ICLR, 2021.
[8] AugMax: Adversarial composition of random augmentations for robust training. NeurIPS, 2021.



questions: Please see above. 
limitations: Please see above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an augmentation strategy for adversarial training by combining simple and complex augmentations to improve the effectiveness of the augmentation and introducing ACAT that gradually increases adversarial epsilon with 2 adversarial defence steps to allow the method to work efficiently. Three claims of justification are listed based on the connection with domain generalization theory. Two important conjectures are proposed and verified through detailed analysis and solid experiments. Comprehensive experiments conducted on several benchmarks for comparisons and ablation studies are provided in the main paper and the supplementary materials. With comprehensive experiments and substantial improvement, this study may be an important milestone for future work.
strengths_and_weaknesses: Strengths:

-The paper is well-written and easy to follow. Claims are supported by detailed analysis and convincing experiments.

-The proposed method gains substantial improvement over the state-of-the-art baselines and beats the concurrent work. 

-The proposed method is novel, effective, efficient and convincing. The paper provides better understanding of augmentations in adversarial training.

-The authors conduct comprehensive experiments considering different settings and different comparisons to verify their ideas.

Weakness:

-The author claims that the method is efficient, could they provide FLOPs, number of parameters and training time of the model?
questions: NA
limitations: NA
review_title: null

Meta-review: metareview: The reviewers found the paper well written and were satisfied with the experimental setting, which shows clear improvements. The authors made a thorough rebuttal and carefully answered the reviewer's questions and I recommend for acceptance as I believe this will be useful to the community. 

I recommend the authors to carefully go over the reviewers’ comments and incorporate them into the final manuscript, along with the additional experiments from the rebuttal. 

confidence: Certain
Decision: Accept