Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Overview:
This is an interesting work. The paper is dedicated to studying the effect of BN to network robustness. The author shows that BN can reduce network robustness to small adversarial input perturbations and common corruptions by double-digit percentages. Then, they use a linear "toy model" to explain the mechanism that the actual cause is the tilting of the decision boundary. Moreover, the author conducts extensive experiments on popular datasets to show the robustness margin with or without the BN module. Finally, the author finds that substituting weight decay for BN is good enough to nullify a relationship between adversarial vulnerability and the input resolution.

Strength Bullets:
1. I like the linear toy example. For that binary classification example, the author explicitly explains the boundary tilting, which increases the adversarial vulnerability of the model. It is clear.
2. The paper conducts extensive experiment on SVHN, MNIST, CIFAR10 (C) datasets. And they show performance margin with or without the BN module. And for the attacker setting, they do use the popular setting (i.e. Mardy's PGD setting) in this field which makes the results more convincing.

Weakness Bullets:
1. Why do not visualize the decision boundary of networks (used in this work) to valid the boundary tilting "theory". The toy example is clear but not convincing enough. There exist several techniques may be helpful to the visualization. (i.e. Robustness via curvature regularization, and vice versa). I think it is one of the important parts of this work. The observation of BN causes adversarial vulnerability is interesting but the main focus should be offering more convincing explanations.
2. I do run experiments for VGG11,13,16,19 on cifar10 with PGD 3 attack (Mardy's setting). There exist ~20 ATA performance gaps between networks with BN and BN networks without BN. But for adversarial trained models, the gaps don't exist anymore, at least for VGG11,13,16,19 on cifar10 with PGD 3 attack. (The performance gap is less than 0.5). In other words, without the BN layers, the robustness of adversarially trained models will not increase in my experiments. I see you report some results in Appendix C. But it is not enough to convince me. Could you provide more implementation details about the adversarial training and attacker? And more experiment results about this point are needed. If adversarial training can fix the vulnerability by BN and BN can give a TA boost, there is no reason we need to remove BN in our adversarial training setting. I see there are similar concerns in the OpenReview.
3. [Minior] The experiments need to be organized better. Especially for section 3, it will be better to divide different experiments or observations into the different subsections.


Recommendations:
For the above weakness bullets, this is a week reject.

Suggestions:
1. To solve the weakness bullets;
2. minor suggestion: add the reference mention in the OpenReview, they are related to this work.

Questions:
1. You mention that you run PGD for 20-40 iterations in the experiment at the bottom of page three. But at each table, you only report one number. So my question is for that accuracy number, you run how many iterations for PGD?
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: Summary:
In this empirical study, the authors identify that batch normalization -- a common technique for accelerating training -- leads to brittle representations that exhibit a lack of robustness and are more susceptible to adversarial attacks. The authors demonstrate their results on SVHN, CIFAR-10, CIFAR-100 CIFAR-10.1 using a variety of network architectures including VGG, BagNet, WideResNet, AlexNet, etc.

Major Concerns:

1. As presented, the experiments are not convincing.

I do not know how much of the changes in adversarial vulnerability are due to batch normalization as opposed to other facets of the training procedure that may have changed in their BN vs no-BN experiments.

For instance, batch normalization usually accommodates higher learning rates and it is not clear if the authors adjusted the initial learning rate, learning rate schedule or training schedule accordingly. If so, it would be important to run a set of experiments with these parameters fixed as per the baseline no-BN models. 

That said, even if the authors did run these experiments, it is still not clear if the cause of adversarial vulnerability is due to BN. Consider that what is truly important in model training is not the learning rate (i.e. step size), but rather the magnitude of the changes in each weight (or the ratio of weight change to the weight). By swapping in batch normalization, the authors may just be altering the norm of the weight change in the (re-parameterized) weights. In this scenario, the gains of removing batch normalization could just as well be explained by the effective change in the learning rate, and not about batch normalization itself, c.f.
  Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
  Yuanzhi Li, Colin Wei, Tengyu Ma
  https://arxiv.org/abs/1907.04595
If the differences in the adversarial vulnerability could be ascribed to effective changes in gradient updates, then this would change the interpretation of these results notably.

2. The underlying hypothesis is specious.

I have several reservations about the underlying hypothesis that requires stronger evidence to overcome. In particular, I have reservations in believing that BN itself is a cause of adversarial vulnerability because BN is just a factorization of a network's weights. That is, there is nothing "special" nor unique about BN-networks; instead, the BN factorization merely permits accelerated training efficiency.

Consider the fact that a BN model may be re-expressed by merely folding in the parameters (i.e. applying the matrix multiplications) into the MLP weights or CNN filters. Thus, the numerical function approximated by the BN and the "folded" non-BN model is identical. What would it mean to say that the BN is "causing" adversarial vulnerability in the BN model given that both the BN and non-BN model perform the identical function?

Another way to say this is to pretend we train a non-BN MLP or CNN model. After training the model, we could apply a BN factorization of the weights. Thus, the non-BN model may be factorized into a BN model. If the resulting BN model were adversarial vulnerable (which I suspect is the case), it would seem very hard to believe that BN was the cause of the vulnerability given it was a post-hoc factorization of the weights.

That said, I could definitely imagine that the training procedure itself could lead to adversarial vulnerability (e.g. citation above) and by employing a BN factorization, one may be encouraged to use a training procedure which leads to increased vulnerability. I would encourage the authors to consider this line of attack and thus, re-orient their analysis and discussion accordingly.

3. The title is poorly worded.

Not withstanding the point above, adversarial vulnerability predates BN. Likewise, non-BN models exhibit adversarial vulnerability. Thus, this title is not a great reflection of the findings of the paper. I would strongly suggest replacing "is a cause" with "increases" or "exacerbates".
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper identifies an important weakness of batch normalization: it increases adversarial vulnerability. It is very well written and the claims are theoretically sound. In the experiments, the authors demonstrated a significant difference in robustness between networks with or without batch normalization layers, in varies settings against both random input noise and adversarial noise. This weakness of batch norm was explained due to the "decision boundary tilting" effect caused by the normalization. Overall, this paper has done solid work to reveal an interesting phenomenon. If it is true, this finding will impact almost all DNN models. 

My concern is that this phenomenon is just another effect of "gradient masking " (as pointed out by Athalye, et al.). Batch norm is a well-known technique to avoid overfitting, without batch norm the network can be easily trained to be saturated with almost zero gradients, demonstrating a false signal of "robustness" to noise. The random noise and real-world corruption experiments are definitely helpful to clear this doubt, but only partially. My concern remains because of two obvious signs of  gradient masking: 
1. The accuracy on PGD-li (epsilon=0.031) attacks are suspiciously too high (20% - 40% Table 3/4). For this level of attack, the acc should be nearly zero. This is likely caused by the gradient masking effect, considering the cifar-10 networks were trained for longer time with larger learning rate (150 epochs, fixed lr 0.01). Training on MNIST is much easier to get zero gradients.  
2. The weight decay discussion is not helpful at all, on the contrary, it confirms my concern on the gradient masking effect. In Table 8, the robustness was increased ~40% by just using large weight decay. This is not the "real robustness", and can be easily evaded by adaptive attack (see Athalye's paper).
With the above two concerns in mind, I doubt the phenomenon revealed in this paper is just "one can easily train a saturated model without batch norm" or equivalently "it's hard to train a saturated model with batch norm". It is hard to say if this is a bad thing for batch norm.

I am quite surprised that the authors ignore this completely. Here are a few things that can be done to rule out the possibility of gradient masking. The masked gradient can be identified by: 1) One-step attacks perform better than iterative attacks; 2) Unbounded attacks do not reach 100% success., etc (see Section 3.1 of Athalye's paper).
1. Including FGSM in the experiments and show the same trends as PGD-li. 
2. Show two networks have similar gradient norms.
3. Apply cw-l2 attack, and show batch norm has forced large perturbation.

Two other suggestions:
1. Summarize the different angles/steps taken to verify the phenomenon, somewhere before the experiments.
2. Cannot see why the input dimension discussion contribute to explanations of the batch norm weakness.

============
My rating stays the same after rebuttal. 

My original concerns are like the other reviewers: why BN, not other techniques such as structure of DNNs MLP vs CNN vs ResNet, activation functions, weight decay, learning rates, softmax etc. My initial suspect was that it is caused by gradient masking likely caused by the l2 weight regularization, so asked the authors to look at the gradient norms and run some testes to rule this out. Yes, the weight norm is directly related to the Lipschitz continuity of the function represented by the network, but it often becomes more complicated on complex nonlinear neural networks. 

According to the new experiment results, the vulnerability is indeed not an effect of gradient masking, thanks for the clarification. However, the new results also indicate that the finding is susceptible to both weight decay and learning rate: in Figure 16 (a): "Un PGD" < "BN PGD" before learning rate decay, andFigure 17 (a) vs (b), doubling the weight decay penalty to 1e-3 also increases the vulnerability of BN. Overall, I believe the phenomenon exists, but the reasons behind requires more explanations, at least not just the batch norm.
review_title: Official Blind Review #3

Meta-review: This article studies the effects of BN on robustness. The article presents a series of experiments on various datasets with noise, PGD adversarial attacks, and various corruption benchmarks, that show a drop in robustness when using BN. It is suggested that a main cause of vulnerability is the tiling angle of the decision boundary, which is illustrated in a toy example. 
The reviewers found the contribution interesting and that the effect will impact many DNNs. However, they the did not find the arguments for the tiling explanation convincing enough, and suggested more theory and experimental illustration of this explanation would be important. In the rebuttal the authors maintain that the main contribution is to link BN and adversarial vulnerability and consider their explanation reasonable. In the initial discussion the reviewers also mentioned that the experiments were not convincing enough and that the phenomenon could be an effect of gradient masking, and that more experiments with other attack strategies would be important to clarify this. In response, the revision included various experiments, including some with various initial learning schedules. The revision clarified some of these issues. However, the reviewers still found that the reason behind the effect requires more explanations. In summary, this article makes an important observation that is already generating a vivid discussion and will likely have an impact, but the reviewers were not convinced by the explanations provided for these observations. 

Decision: Reject