Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new method for augmenting training based on worst-case perturbations of batch statistics, batch mean and variance. The idea is that these statistics encode style information thus making a model robust to small changes of their values will promote style invariance. The authors evaluate their method on variants of the ImageNet dataset and compare its performance with existing data augmentation alternatives. They find that while the method itself does not always outperform these alternatives, combining it with them can improve performance.
main_review: The paper studies an important question: how can simple synthetic transformations used during training improve robustness to unforeseen transformations of the data. The proposed method is rather simple and intuitive. At the same time, the performance improvement from using this method alone is too small to justify using it over other methods. Also, the improvement from combining it with other methods is still rather small in most cases. Nevertheless, I still believe that the findings could be of interest to the robustness community.
limitations_and_societal_impact: Yes, they do.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies the problem of making predictive models robust to distributional shifts. This is achieved using an adversarial training strategy that adversarially perturbs batch norm statistics of a pre-trained model, which effectively captures variations in image style such as color, texture, etc. This worst-case optimization strategy exposes the model to larger variations in image statistics and semantics rather than just just pixel-based perturbations, and as a result is expected to produce better robustness under large distribution shifts. 

main_review: **Strengths**
+ The paper adds to emerging evidence that BN statistics can play a critical role in designing classifiers that are robust to unknown noisy changes and domain changes. 
+ Interesting technique to visualize the adversarially perturbed features, which many adversarial methods are not able to do. This is useful in understanding the kinds of attributes being perturbed during optimization. That the most relevant are color, and texture are also intuitive, corroborating previous studies that argue CNNs are very reliant on these properties for classification. 
+ I like the formulation that it is generic (network agnostic), and simple to implement.

**Weaknesses**
+ There are a few concerns with the work with regard to novelty — the idea of using adversarial training to obtain robustness to large semantic shifts instead of pixel-wise perturbations has been explored in the past (Gokhale et al., 2020 https://arxiv.org/abs/2012.01806). More specifically, the idea of adapting BN statistics to obtain robustness to distribution shifts has also been explored (Schneider et al., 2020, https://arxiv.org/abs/2006.16971; Nado et al., https://arxiv.org/abs/2006.10963); and other kinds of layer statistics (Wang et al., https://arxiv.org/abs/2006.10726)  albeit for test time adaptation, which is a different application than those considered in this paper. However, the broad idea that robustness in this manner can be achieved has been somewhat established. This paper is using the same approach in an adversarial training setting.
+ Next, it seems the benchmarks on which robustness is measured are better suited for AdvBN (like stylized imageNet, sketch etc.) which mostly vary along texture and color anyway. On ImageNetC it is clear AdvBN on its own is unable to match the performance of existing baseline techniques like Adv Prop, Augmix etc. Many methods using adv robustness have also considered larger domain shifts (like in a domain generalization setting) which is more challenging, and will perhaps be a more fairer evaluation for AdvBN as well.
+ Most evaluations are performed like ablations, with robustness methods mainly based on augmentation — whereas there are several recent approaches that have shown promise even otherwise which will help provide context to AdvBN’s performance benchmarks. This is the same issue on the semantic segmentation results — no other baselines are reported other than baseline and adv training, which are weak for this task which involves large semantic shifts — advBN is better by design. 
+ Question about the approach:  Why is only a single AdvBN layer used? How does the performance vary if more than one AdvBN layers are used or do they end up being redundant? 
limitations_and_societal_impact: Yes, the paper addresses some of its weaknesses in the broader problem of robustness under distributional shifts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to improve robustness on ImageNet classification by adversarially optimizing the batch-norm statistics of a pretrained classifier. The initial layers of the classifier are frozen, and only the upper subnetwork of the classifier is finetuned after the addition of the adversarial batch-norm layer. The benefits of the method often appear in conjunction with image augmentation methods, where applying AdvBN on top of an image augmentation method tends to result in better performance.
main_review: The methodology of this paper is fairly simple, and the results are consistent across the board on various domains (several ImageNet variants, several classification architectures, and on a segmentation task). Ablations show the impact of altering the position of the AdvBN layer and on number of PGD steps. Overall, the paper reads smoothly and is clearly written.

A few points that would be helpful to clarify:
-  in A.3 it is mention that the original BN parameters are used for some datasets, and auxiliary BN is for others. It would be helpful to provide more details on how this choice is made, and full results on using both BN option to see how sensitive the performance is for this choice. Is main or auxiliary BN used for the segmentation experiment? 
- in Table 7, why are the reconstructed classifications without adversarial perturbation much worse than clean ImageNet classifications? It would be helpful to see what the reconstructions look like. 
- in Eqn 1, the notation $\delta f$ seems a bit unclear. Is it standard deviation of the features f?


limitations_and_societal_impact: Limitations and impact are addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper propose AdvBN as a variant of batch normalization, to increase the robustness to distributional shifts of existing CNNs.

main_review: The authors propose to improve DNNs' robustness to different image styles by adversarially perturbing the features statistics during training, and fine-tuning the parameters to make them robust to the perturbed features.
The authors implement this idea as a DNN layer named AdvBN, which could be integrated into different DNN architectures.

Strengths:
* The intuition (perturbing feature statistics during fine-tuning to increase the robustness) makes sense.The authors first visualize the effects of perturbing feature statistics by an autocoder, to show that the feature statistics can indeed encode the "style" of different images. Then the authors propose to adversarially modify the feature statistics which could yield generate different image styles, and then make the classifier to be robust to these styles in the training time.
* The authors integrate the proposed AdvBN on five data augmentation methods, and all test cases on four datasets show that the AdvBN can indeed improve the robustness to distribution shifts.
* I particularly like the experiment designed in Section 5.3, which demonstrates that the ResNet-50 model with AdvBN can learn more robust feature representations across different image styles, when compared with a standard ResNet-50 model.

Weakness:
* In Table 1, since the authors perform fine-tuning for AdvBN, the total training time seem to be longer than baseline methods . The number of training steps should be carefully controlled to make a fair comparison.


limitations_and_societal_impact: Yes.
review_title: null

Meta-review: After a thorough discussion (particularly with reviewer 4gjW) there are now four reviews that unanimously recommend acceptance.
Thus, I will also recommend acceptance.
Decision: Accept (Poster)