Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a method to improve the generalization of neural networks by training them to be robust to adversarial perturbations in the statistics of the batch normalization (BN) layers. The approach combines gradients computed on unperturbed BN statistics with gradients computed on perturbed statistics. Perturbations or noise in the BN statistics are obtained through 1) signed gradients from the first update and 2) reductions in the batch size for the second update.
Experiments demonstrate improvements over standard training, especially in the case of smaller-scale datasets, i.e., CIFAR and Time-ImageNet. The method can also be combined with other techniques, such as Mixup and SAM optimization, typically leading to further improvements.   
strengths_and_weaknesses: Strengths:
- The method benefits the generalization of neural networks trained on smaller datasets considerably
- The technical presentation of the method in Section 3.2 is detailed and sufficiently clear
- The method can be combined with other training methods, such as SAM. 


Weaknesses:
- The paper claims to bridge the gap between robustness and generalization. Experiments are focused mainly on the generalization ability of the learned networks, and robustness experiments are restricted to perturbations of the BN statistics. This is quite limited, and it is unclear if the learned networks are robust to various other adversarial attacks. Indeed, it is unclear what the relevance of Sections 4.4 and 4.5 are regarding the robustness of the networks in practice. 
- Another contribution of the paper is "a new AT paradigm, termed model-based AT." It appears that the main idea of perturbing model parameters has been explored in various prior works (e.g., [8, 28]). It is not clear what the generic formulation in Eq 2 contributes or what novel insights are provided. 
- The benefits of the method seem to disappear during large-scale experiments on ImageNet. This is somewhat concerning, and it might be good to investigate this issue further. 
- Section 3.3 is somewhat confusing: L206 claims \mathcal{R}=0, but then \mathcal{R} appears in the perturbation computation of (7). It is also unclear if a term similar to g_\phi exists in this case. 
questions: I would appreciate it if the authors could address the weakness listed above, especially the first three points, i.e., 
- How is the robustness to other typical adversarial attacks?
- How is the introduced AT paradigm different from what prior works proposed, or what novel insights are provided?
- What are the issues with large-scale training?
limitations: As mentioned above, it might be good to further address the performance on larger scale datasets if this turns out to be a limitation. Also, depending on how robust the method is to other adversarial perturbations, this could also be mentioned in the limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: While Adversarial Training is one of the most successful methods to increase robustness, it usually degrades performance of the models on clean images. The authors attribute this to distributional discrepancy in Batch Norm statistics. They propose Adversarially Perturbed bAtch noRmalizaTion (APART) to achieve robustness against BN statistics noise, and to bridge the gap between models’ generalization and robustness. They perform backward passes twice over each batch of clean samples. The first backward pass produces two gradient computations: a normal gradient that helps update parameters of model, and a statistics gradient that is used to perturb the statistics parameters in BN. The second pass is performed to generate the defensive gradient that helps the model resist the adversarial statistics perturbation. The normal and defensive gradients are combined to improve both generalization and robustness of the model. Experiments are performed on CIFAR, Tiny-ImageNet and ImageNet, and show improved clean accuracy over standard training and SAM [28].  



strengths_and_weaknesses: **Originality and Significance**:
- The paper presents a new way of bridging the gap between models’ generalization and robustness. It is known in the literature that there is discrepancy between Batch Norm statistics of clean and adversarial examples [13] (as well as the statistics from different batches). AdvProp proposes using two batch norm statistics, one for clean images and one auxiliary for adversarial examples [13]. Rather than creating a separate layer to deal with this discrepancy, the paper attempts to make the models robust to the BN statistics noise. This approach is interesting and novel to the best of my knowledge.   
- The method can be combined with other augmentations to further boost performance. The proposed combination with SAM (as one of the state-of-the-art methods) is particularly promising. 

**Quality**:
- Overall the paper is well-structured and well-written.
- The proposed approach is sound, and is described clearly. 
- Experiments are performed on various datasets including CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet. Overall, experimental results are convincing. They demonstrate improvements on clean accuracy over the baselines as well as robustness against perturbed BN statistics. Comparing with baselines using the same budget is important given the additional cost of the proposed approach.   
- The authors report detailed experimental results in the supplementary material, and show that ARAPT is relatively insensitive to hyper-parameters. 

**Clarity**:
- Scalability of ARAPT to large datasets and models is not clearly supported in the experiments. The authors use the relatively small ResNet-18 model on ImageNet-1K. ARAPT underperforms standard training on ImageNet at 2x budget and outperforms it at 4x budget (Table 2). The authors note that "APART employed on the large-scale dataset requires more steps to show its promise", but do not provide further explanation or experiments on this. 
- All experiments are performed on the ResNet family. On ImageNet the achieved accuracy of 72.14% (Table 2) is far from the state-of-the-art. It'd be good to include experiments on other architectures (e.g. EfficientNet), and see if the gains are significant. 
questions: See the Clarity section above. 
limitations: - The authors have addressed limitation of the work in terms of suffering from potential degeneration in case of the combination with other training methods implicitly involving BN. 
- The authors can address potential limitation of their work on large-scale datasets and models. 
- There are no potential negative societal impact that need to be specifically addressed.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes to add adversarial noise on the BN statistics to improve classification accuracy on in-distribution images. 
strengths_and_weaknesses: Strength:
1. The paper is well-written and easy to follow. The related works are thoroughly discussed. 

Weakness:
1. The novelty is limited. The proposed method is almost identical with AdvBN [15] (NeurIPS'21). Although the authors mentioned three differences in related work section, I still think they are all minor differences. 
2. In experiments, no results on [11] or [15] are reported. This makes it hard to evaluate whether the proposed method can outperform previous works. 
questions: Please see above
limitations: Please see above
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduces an ‘Adversarially Perturbed Batch Normalization’ to improve the model’s generalization and robustness. Experiments on CIFAR, Tiny-ImageNet, and ImageNet show that the proposed methods can improve the models’ performance, compared with the baseline model. 
strengths_and_weaknesses: 
Strengths:
Compared with the previous AdvBN [15], the proposed APART is more appliable and easy-training. 
The paper is well-written, and the theoretical analysis is clear.


Weaknesses:
Experiments
From the reviewer’s view, the experiments in this paper are not sufficient.
(1)	As mentioned in Lines 62-63, the author mentioned that they want to bridge the gap between the model’s generalization and robustness. The reviewer thinks experiments on ImageNet-C or Stylized ImageNet are needed to show the advantages of robustness. 
(2)	The comparison with other methods is missing. The reviewer thinks a comparison with normalization methods [18-20] and adversarial methods [11, 15] is needed.
(3)	‘Mix-Up’ experiments on ImageNet are missing.
(4)	Similar to the experiments on CIFAR-10 and CIFRA-100, the authors are suggested to conduct the experiments on one more backbone on Tiny-ImageNet and ImageNet.

questions: I would like to see the experimental results on the robustness benchmarks (e.g., ImageNet-C). As the main contribution lies in the model's generalization and robustness, the robustness result is must-needed, in my opinion.
limitations: For me, the current experiments are not sufficient. The authors are suggested to add more experiments to show the advantages of their paper.
review_title: null

Meta-review: metareview: The paper presents a new way of bridging the gap between models’ generalization and robustness, by combining gradients computed on unperturbed BN statistics with gradients computed on perturbed statistics. The main goal is to improve the standard generalization, but the authors should clarify their definition of "robustness" as it seems to confuse all reviewers (e.g., questioning adversarial attacks). Moreover, the method itself is very simple, and the idea of using adversarial perturbation to stabilize model training isn't new (AdvProp, etc.). Reviewers are further concerned about the lack of large-scale experiments or on state-of-the-art architectures. Besides, there are no comparisons with some of the competing methods such as AdvProp. Therefore, I find no sufficient ground to recommend acceptance in this paper's current shape.
confidence: Certain
Decision: Reject