Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes an interesting perspective that BatchNorm may introduce the adversarial vulnerability, and probes why BatchNorm performs like that (the tracking part in BatchNorm). In experiment, the robustness of the networks increases by 20% when removing the tracking part, but the test accuracy on the clean images drops a lot. Afterwards, the authors propose RobustNorm, which performs better than BatchNorm for both natural and adversarial scenarios.

Detailed Comments: 
+ The paper is well written. The paper structure is clear and figures are well illustrated.
+ The paper understands and carefully investigates BatchNorm in a interesting and important direction. After the investigation, the improved version RobustNorm shows more potential.
+ The experimental results seem good. The RobustNorm performs better than BatchNorm for both natural and adversarial scenarios.
- More results on ImageNet would be better to verify the proposed RobustNorm method.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Review: This paper investigates the reason behind the vulnerability of BatchNorm and proposes a Robust Normalization. They experimentally show that it is the moving averages of mini-batch means and variances (tracking) used in Normalization that cause the adversarial vulnerability. Based on this observation, they propose a new normalization method not only achieves significantly better results under a variety of attack methods but ensures a comparable test accuracy to that of BatchNorm on unperturbed datasets. The paper is clearly written, easy to read.
 
Strengths:
 
Explore the cause of adversarial vulnerability of the BatchNorm and assume that the tracking mechanism used in original BatchNorm leads to the vulnerability from experiment results.
Propose a new and simple normalization method and perform extensive experiments to validate the efficacy of proposed method.
 
Weaknesses:
Though extensive experiments have been done by revealing what leads the vulnerability and the effectiveness of proposed method. The results seem unconvincing with respect to different datasets, since Cifar10 and Cifar100 are inherently connected. Would you mind performing some experiments on ImageNet? Since adversarial training on ImageNet is time-consuming, can you show us the result of Natural Training of different models with different norms on ImageNet and compare their robustness under different attack?

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper addresses a limitation of BatchNorm: vulnerability to adversarial perturbations. The authors propose a possible explanation of this issue and correspondingly an alternative called RobustNorm to tackle this problem. Specifically, the authors observe that the statistics of BatchNorm for training and inference are different, resulting in different data distributions for training and inference. To solve this problem, the authors propose to use min-max rescaling instead of normalization. In addition, the running average is calculated with mean and the running mean of the denominator during inference. Experimental results show significant improvement of robustness and also comparable accuracy for clean data.

The paper is well-written and the contributions are stated clearly. The explanation of vulnerability is reasonable. The proposed solution is simple but effective.

However, I have several concerns:
*The authors verify that the running average is the main culprit of vulnerability to adversarial attack, but provide no further investigation of why this happens. A possible solution is the drift in input distributions, but the manuscript does not state clearly how is the distribution changed. Further experiments would have made this claim more convincing.
*The proposed method involves a hyper-parameter \rho, but it may result in problematic issues. The variance of input is of the same order of magnitude as (max(x)-min(x))^2. If \rho is set to other value, the magnitude of gradient will change drastically during back-propagation. Although \rho can be set to 0.2, it still seems ad-hoc. Experiments on more datasets and the sensitivity of the proposed method to \rho would have validated the claims of the authors.

review_title: Official Blind Review #2

Meta-review:  This paper presents an empirical analysis of the reasons behind BatchNorm vulnerability to adversarial inputs, based on the hypothesis that such vulnerability may be caused by using different statistics during the inference stage as compared to the training stage. While the paper is interesting and clearly written, reviewers point out insufficient empirical evaluation in order to make the claim more convincing.
Decision: Reject