Batch Normalization is a Cause of Adversarial Vulnerability
Authors: Angus Galloway
Anna Golubeva
Thomas Tanay
Medhat Moussa
Graham W. Taylor
Venue:  (2020)
Keywords: batch normalization
adversarial examples
robustness

Abstract: Batch normalization (BN) is often used in an attempt to stabilize and accelerate training in deep neural networks. In many cases it indeed decreases the number of parameter updates required to achieve low training error. However, it also reduces robustness to small adversarial input perturbations and common corruptions by double-digit percentages, as we show on five standard datasets. Furthermore, we find that substituting weight decay for BN is sufficient to nullify a relationship between adversarial vulnerability and the input dimension. A recent mean-field analysis found that BN induces gradient explosion when used on multiple layers, but this cannot fully explain the vulnerability we observe, given that it occurs already for a single BN layer. We argue that the actual cause is the tilting of the decision boundary with respect to the nearest-centroid classifier along input dimensions of low variance. As a result, the constant introduced for numerical stability in the BN step acts as an important hyperparameter that can be tuned to recover some robustness at the cost of standard test accuracy. We explain this mechanism explicitly on a linear ``toy model and show in experiments that it still holds for nonlinear ``real-world models.

Under review as a conference paper at ICLR 2020
BATCH NORMALIZATION INCREASES ADVERSARIAL VULNERABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
Batch normalization (BN) is often used in an attempt to stabilize and accelerate training in deep neural networks. In many cases it indeed decreases the number of parameter updates required to achieve low training error. However, also reduces robustness to small adversarial input perturbations and common corruptions by double-digit percentages, as we show on five standard datasets. Furthermore, we find that substituting weight decay for BN is sufficient to nullify relationship between adversarial vulnerability and the input dimension. A recent mean-field analysis found that BN induces gradient explosion when used multiple layers, but this does not fully explain the vulnerability we observe, given that it occurs for single BN layer. We argue that the main cause is the tilting of the decision boundary with respect to the nearest-centroid classifier along input dimensions of low variance. As result, the numerical stability constant of BN acts as an important meta-pareee parameter that can be tuned to recover some robustness at the cost of standard test accuracy. We explain this mechanism explicitly for linear models but find that it still holds for nonlinear models.
INTRODUCTION
BN is standard component of modern deep neural networks, and tends to make the training process less sensitive to the choice of hyperparameters in many cases (Ioffe & Szegedy 2015). While ease of training is desirable for model developers, an important concern among stakeholders that of model robustness during deployment to plausible, previously unseen inputs. The adversarial examples phenomenon has exposed unstable predictions across state-of-the-art models (Szegedy et al. 2014). This has led to variety of methods that aim to improve robustness, but doing so effectively remains a challenge (Athalye et al., 2018 Schott et al., 2019 Hendrycks & Dietterich 2019 Jacobsen et al. (019a). We believe that prerequisite to developing methods that increase robustness is an understanding of factors that reduce it.
Approaches for improving robustness often begin with existing neural network architectures- BN-and patch them against specific attacks, e.g., through inclusion of adversarial examples during training (Szegedy et al. 2014 Goodfellow al., [2015: Kurakin al. 2017 Madry al. (0188). An implicit assumption is that BN itself does not reduce robustness, however, recent initialization-time analyses have shown that causes exploding gradients, and increased sensitivity input perturbations as the network depth increases YYang et al., 2019 Labatie 2019). In this work, we consider the impact of BN practical scenarios terms of robustness common corruptions (Hendrycks & Dietterich 2019) and adversarial examples (Szegedy et al. (2014), finding that BN induced sensitivity remains a concern even cases where its use appears benign on the basis of clean test accuracy, and when only one BN layer is used.
The frequently made observation that adversarial vulnerability can scale with the input dimen- sion (Goodfellow et al., 2015: Gilmer et al.) 2018 Simon-Gabriel et al. 2019) highlights the importance of identifying regularizers as more than merely a way to improve test accuracy. In particular, BN initialization-ting analysis hold after training. By adding & regularization and removing BN, we show that there is no inherent relationship between adversarial vulnerability and the input dimension.
confounding factor in "Sionn-Gabie et al. (2019). making the results of their

---

Under review as a conference paper at ICLR 2020
2 BATCH NORMALIZATION
We briefly review how BN modifies the hidden layers' pre-activations h of a neural network. We use the notation of Yang al. 2019), where a is an index for units in a layer and i for mini-batch of B samples from the dataset; Nr denotes the number of units in layer 1, W is the matrix of weights and b' is the vector of biases that parametrize layer I. The batch mean is defined as Ma = E, has- and the variance is o² the pre-activation of each unit hyg (consistent with loffe & Szegedy (2015)), the result is divided by the standard deviation aa plus a small constant e to prevent division by zero, then scaled and shifted by the learned parameters "/o and Ba, respectively. This is described in equation T where per-unit nonlinearity o e.g., ReLUU, is applied after the normalization.
Σi (hai Ha)² In the BN procedure, the mean (4a is subtracted from
Ro: = Ta hod Ho + Ba.
h = (h( +
(1)
Voa +
This procedure introduces complications, however. Consider two mini-batches that differ by only a single example: due to the induced batch-wise wise nonlinearity, they will have different representations of all examples. These differences are amplified by stacking BN layers, and were shown to cause exploding gradients at initialization (Yang et al. 2019). Conversely, normalization of intermediate representations for two different training inputs impairs the ability to distinguish definite examples that ought to be classified with large prediction margin (as judged by an "oracle"), from more ambiguous instances. The last layer of a discriminative neural network, in particular, is typically a linear decoding of class abhl-homogennous clusters, and thus makes use of information contained in the mean and variance at this stage for classification. In light of these observations, we begin in our analysis by adding single BN layer models trained by gradient descent (GD). This is the most favorable scenario according to the analysis of Yang et al. (2019), where more layers and a smaller mini-batch size exacerbate the exploding gradients.
BOUNDARY TILTING
Tanay& & Griffin (016) relate the adversarial vulnerability of linear classifiers to the tilting angle of the decision boundary w.r.t. the nearest-centres classifier. Following their setup, examine how BN affects this angle in a simple linear model, and then show that increasing model complexity cannot "undo" this vulnerability.
Consider the binary classification task of iden- tifying two different types of imput x subject to Gaussian noise with a linear classifier w x + b. This can be modeled by the class-conditional distribution p(rly j) = N(v Σ) with la- bel y (0)... The Bayes-optimal so- lution to this problem is given by the weight vector w- = Σー T 0 + log p(y=0). where p(v) denotes the marginal probability for the label (see e.g. (Jordan 1995)). ). while the nearest- centroid classifier is defined by w² -'!
u. GD(x)
b. BNGD(s)
BN(x), GD
w²), and = [
Figure 1: A dataset with one task-relevant (a = 1) and one task-irrelevant dimension (a 2). Nor- malization aligns the decision boundary with the Bayes solution (indicated by arrows in "BNGD"), but this minimizes the averaged distance between the points and the boundary, maximizing adver- sarial vulnerability. Compared with the decision boundary of a linear model (0 zs 09), the batch- normalized model has 0 66.7%. On the right is the dataset seen by the BNGD classifier. We use Σn o = |-5,,00 0J. and 1d = [5,0]
We analyze the effect of batch-normalizing the input to the classifier for this problem (i.e., h.i = R.i), first in the simplest setting where To = 1, 3, = 0 Va. We select the class distribution means w to be symmetric around zero, so that the batch mean computed by BN is Ma 0 Ya. The norta-zorm linear clas- sifier is thus defined as: f(x) w w+b By construction of our synthetic dataset, the vari- ance of the batch can be deduced from the data
Σ22 0.01, Σ12 Σz1 0.05.
Vo*te"
2

---

Under review as a conference paper at ICLR 2020
Table I: As predicted by the theory, batch-normalized gradient descent (BNGD) yields a tilted decision boundary w.r.t. the nearest-centroid classifier, regardless of the affine parameters being learned or fixed. We report the tilting angle (0) and accuracies of linear models trained on MNIST vs. 7 for vanilla GD, GD with L2 weight decay "WD"(A = 0.1), and BNGD. Affine = "F" indicates 7 y=1 and 8 = 0, whereas "T means they are randomly initialized and learnable. AWGN = -0, 1),
FGSM used with e = 1/10. Entries are the mean and its standard error over five random seeds.
Model
Test Acc.
AWGN Acc.
FGSM Acc.
0 E [0, 90%
GD
96.94 + 0.08
90.08 = 0.07
66.96 + 0.49
49.04 + 0.46
GD + WD
96.93 + 0.05
91.93 + 0.14
74.20 + 0.35
40.83 + 0.46
BNGD Affine F
97.75 + 0.03
49.67 = 0.18
0.15 0.02
90.00 0.00
BNGD Affine T
97.40 + 0.07
49.50 = 0.20 0.13 + 0.02 90.00 - 0.00
+2 Eoo. The tilting angle 0 of the batch- nuuch-norallized decision boundary w.r.t. the one given by w² (note that the boundary is perpendicular to w)) is therefore approximately equal to the angle between the datasets before and after normalization. To compute 0, we divide the weights w by V + c, and then normalize w/ ||w 112, such that cos 'w"""w"" "=') w²). From this analysis follows that the order of magnitude of is important relative to the data variance: if c > og then the effective weight value wa is reduced, and if e < o² and o² is small, then Wo increases greatly, causing boundary tilting along direction a.
parameters: =2 (12) +
We depict simulations of the toy model in Figure | We use constant learning rate GD, which is known to converge to the max-margin solution oolution-equivalenn to the nearest centroid classifier in this case-for linear models on separable data (Soudry et al. 2018). Batch-normalized GD (BNGD) converges for arbitrary learning rates for linear models (Cai et al. 2019);
value of 0.1 for 1000 epochs.
Next, we train linear models on the MNIST 3 vs. 7 dataset with 5000 training samples (drawn uniformly per class) using learning rate of 0.1 for 50 epochs. We compute the angle 9 w.r.t. the nearest-centroid classifier, which is obtained by subtracting the "average 3" from the "average 7" of the full training set. Although this may seem like a crude reference point, the nearest-centroid classifier is much more robust than the linear model of Goodfellow et al. (2015), achieving 40% accuracy for the fast gradient sign method (FGSM) at E = 1/1 vs. R 0%. Results consistent with the boundary tilting theory are shown in Table 1 which not only shows that BN causes tilting, but that this is unaffected by the parameters y and B. Post noss--ommaiiztion, there is no signal to y and about the variances of the original dataset. This is consistent with other works that observe y and ,3 do not influence the studied effect (van Laarhoven 2017: Zhang et al. 2019a Yang et al. 2019)
Increasing the numerical stability constant e increases robustness in terms of absolute test accuracy for additive white Gaussian noise (AWGN) on MNIST and CIFAR-10 10 datasets by 33% and 41% respectively (at the cost of standard accuracy). We defer the details of this experiment to Appendix A This loss of accuracy, and the effect of are consistent with Labatie (201)), which remarks that under BN, directions of high signal variance are dampened, while directions of low signal variance are amplified. This preferential exploration of low signal directions reduces the signal-to-nun to-ioee ratio and increases sensitivity w.r.t. the input. Increasing e reduces the sensitivity along "low signal directions".
4 EMPIRICAL RESULTS
For the main practical results on MNIST, SVHN CIFAR-10, and ImageNet, we evaluate the robustness- measured as a drop in test accuracy under various input perturbations-ot of convolutional networks with and without BN
As a white-box adversarial attack we use projected gradient descent (PGD) in lac- and fg-mrrm variants, for its simplicity and ability to degrade performance with little change to the input (Madry) ct al. 2018). The PGD implementation details are provided in Appendix B We report the test
Unless stated otherwise, we leave the internal parameters of BN to their default values as in modern deep learning frameworks, =1c-5 with y and enabled.
3

---

Under review as a conference paper at ICLR 2020
Table (Small learning rate) Test accuracies of VGG8 and WideResNet-28--1 on CIFAR-10 and CIFAR-10.1 (v6) in several variants: clean, noisy, and PGD perturbed. We evaluate models achieving the highest validation accuracy after training for 150 epochs.
CIFAR-10
CIFAR-10.1
Model
BN
Clean
Noise
PGD-L3o
PGD--2
Clean
Noise
VGG x
87.9 + 0.1
79 + 1
52.9 + 0.6
65.6 + 0.3
75.3 + 0.2
66 +
VGG V
88.7
0.1
73 + 1
35.7 : 0.3
59.7 + 0.3
77.3 0.2
60 + 2
WRN F
94.6
0.1
69 + 1
20.3 + 0.3
9.4 0.2
87.5 + 0.3
68 + 1
WRN
J
95.9
14.9 = 0.6
8.3 0.3
89.6 = 0.2
58 =
accuracy for additive Gaussian noise of zero mean and variance 1/16, denoted as "Noise", as well as using the CIFAR- - common corruption benchmark (Hendrycks & Dietterich] 2019). We found these methods were sufficient to demonstrate considerable disparity in robustness due to BN, but this is not intended as a complete security evaluation.
Standard vetaaparaameters from the literature were used to train models with and without BN from scratch over several random seeds. All uncertainties are the standard error of the mean accuracy For SVHN and CIFAR-10, we examine two different learning rate schemes, given that it has been suggested that one of the primary mechanisms of BN is to facilitate training with larger learning rate (Toffe Szegedy 2015 Bjorck et al. 2018):
1. A fixed "small" learning rate of 1e-2 (SVHN, CIFAR-10).
2. An initial "large" learning rate of le- with subsequent drops by a factor of ten (CIFAR-10).
In the SVHN experiment, VGG variants (Simonyan & Zisserman 2015) are trained using using five random seeds. BN increased clean test accuracy by 1.86 0.05%. % but reduced test accuracy for additive noise by 5.5 + 0.6%, for PGD-Loo by 17 + 1%, and for PGD-L2 by 20 + 1%. We defer the full reta-pprrameters and results to Appendix E
For the CIFAR- 10 experiments we trained models with a similar procedure as for SVHN, but with random 32 x 32 crops using four-pixel padding, and horizontal flips. We evaluate two families of contemporary models: one without skip connections (VGG) and a WideResNets (WRN) using "Fixup" initialization (Zhang al. 2019b) reduce the use of BN.
Results of the first experiment are summarized in Table 2 In this case, inclusion of BN for VGG reduces the clean generalization gap (difference between training and test accuracy) by 1.1 + 0.2%. For additive noise, test accuracy drops by 6 + 1%, and for PGD perturbations by 17.3 + 0.7% and 5.9 0.4% for foo and l variants, respectively.
Very similar results are obtained on new test set, CIFAR-10.1 v6 (Recht et al., 20188: BN slightly improves the clean test accuracy (by 2.0 0.3%), but leads to a consider- able drop in test accuracy of 6 + 1% for the case with additive noise, and 15 = 1% and 3.4 0.6% respectively for l. and f2 PGD variants (PGD absolute values omitted for CIFAR-10.1 in Table 2 for brevity).
Table 3: (Large initial learning rate) Robustness of VGG models of increasing depth on CIFAR-10, with and without BN. See text for meta-parameters.
Model
Test Accuracy (%)
L
BN
Clean
Noise
PGD-(
8
x
89.29 0.09
81.7
0.3
55.6 + 0.4
8
/
90.49 + 0.01
77 77:1
40.6
In the second "large" learning rate exper- iment summarize in Table 3 we prolong
13
x
91.74 + 0.02
77.8 + 0.7
00.30.77
13
/
93.0 + 0.1
67:1
28.5 0.4
training for up to 350 epochs, and drop the learning rate at epoch 150 and 250 in both cases. This increases clean test accuracy rel- ative to results in Table The deepest model that could be trained without BN using this
16
/
92.8 + 0.1
66 + 2
28.9 0.2
19
92.65 + 0.09
68 + 2
30.0 0.1
Fach experiment has unique uncertainty, hence the numher of decimal places varies.
4

---

Under review as a conference paper at ICLR 2020
Table 4: Robustness of three modern convolutional neural network architectures with and without BN on the CIFAR- C-C common "noise" corruptions (Hendrycks & Dietterich 2019). We use "F" to denote the Fixup variant of WRN. Values were averaged over five intensity levels for each corruption.
Model
Test Accuracy (%)
Variant
BN
Clean
Gaussian
Impulse
Shot
Speckle
X
87.9 0.1
65.6 1.2
58.8 0.8
71.0 = 1.2
70.8 + 1.2
VGG8
V
88.7 + 0.1
56.4 + 1.5
51.2 + 0.1
65.4 + 1.1
66.3 + 1.1
X
91.74 + 0.02
64.5 + 0.8
63.3 + 0.3
70.9 = 0.4
71.5 + 0.5
VGG13
V
93.0 : 0.1
43.6 + 1.2
49.7 = 0.5
56.8 + 0.9
60.4 + 0.7
F
94.6 0.1
63.3 0.9
66.7 0.9
71.7 = 0.7
73.5 + 0.6
WRN28
95.9 :: 0.1 51.2 + 2.7
56.0 2.7
63.0 + 2.5 66.6 + 2.5
procedure and He et al. (0055) initialization was VGG13. None of the deeper models recover the robustness of the most shallow, or same-depth equivalents, nor does the higher learning rate recover the performance of the baselines. Results for deeper models are in Appendix Other learning rate schedules, and robustness vs. training epoch curves are in Appendix K
Next, we evaluate robustness on the common corruption benchmark comprising 19 types of real- world distortions from four high-level categories: "noise", "blur", "weather", and "digital" ef- fects (Hendrycks & Dietterich, 2019). Each corruption has five "severity" or intensity levels. We report the mean error on the corrupted test (mCE) by averaging over all intensity levels and corruptions. We summarize the results for two VGG variants and a WideResNet on CIFAR-10-C, trained from scratch on the default training set for three and five random seeds, respectively. Accuracy for the "noise" corruptions, causing the largest difference in accuracy with BN, are outlined in Table
The key takeaway is: For all models tested, the norma-normllized variant has higher error rate for all corruptions of the "noise" category, every intensity level.
Averaging over all 19 corruptions, BN increases mCE by 1.9 + 0.9% for VGG8, 2.0 + 0.3% for VGG13, and 1.6 + 0.4% for WRN. There is a large disparity in accuracy when modulating BN for different corruption categories, we examine these in more detail in Appendix G
Table 5: Robustness of pre-trained ImageNet mod- els with and without BN. Note: The numeric suffix indicates number of layers, or the spatial patch width pixels (of 224) for BagNet.
Interestingly, some corruptions that led to pos- itive gap for VGG8 show negative gap for the WRN, i.e., BN improved accuracy to: Contrast- 1.1%, Snow-22 2.8 = 0.4%. Spatter-22 -23 0.8%. These are the same corruptions for which VGG13 loses, does not improve its robust- ness when BN is removed. We suspect accuracy for these corruptions correlates with standard test accuracy, which is highest for the WRN. Visually, these corruptions appear to preserve texture information. Conversely, noise is ap- plied in a spatially global way that dispropor- tionately degrades these textures, emphasizing shapes and edges. It is now known that mod- CNNs trained standard image datasets have a propensity to rely heavily on texture in addition shape and edge cues for object recog- nition Geirhos et al. 2019). We evaluate pre-
Top 5 Test Accuracy (%)
Model
BN
Clean
Noise
PGD-foc
VGG11
88.63
49.16
37.12
VGG11
89.81
49.95
26.12
VGG19
x
90.88
64.86
34.19
VGG19
/
91.84
68.79
24.49
AlexNet
79.07
41.41
39.12
ResNet18
88.65
79.62
31.07
BagNet-9
70.39
1.25
7.42
BagNet-17
81.16
5.09
16.66
BagNet-33
/
86.99
14.62
24.34
For which one of ten random seeds failed achieve better than chance accuracy on the training set, while others performed expected. We report the first three successful runs for consistency with the other experiments.
5

---

Under review as a conference paper at ICLR 2020
Clean Acc.
Noise Acc.
BIM Acc.
60
30
Batch Dorm
L
30
20
60
Unnormalizer
L
30
50
100
150
200
250
50
100
150
200
250
50
100
150
200
250
B
B
B
0.1
U.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
We extend the experiment of Yang et al. (2019) by training fully-connected nets of depth and wonttant-width (N; 384) ReLU layers by SGD, batch size B, and learning rate 1 10- on MNIST. The BN parameters y and 3 were left as default, momentum disabled, and c = 10-3, The dashed line is the theoretical maximum trainable depth of aatch-normalized networks a function of the batch size. We report the clean test accuracy, and that for additive Gaussian noise and BIM perturbations. The aatch-normalize models ones were trained for 40 epochs as they took longer to converge. The 40 epoch batch-normalized plot was qualitatively similar with dark blue bands for BIM for shallow and deep variants. The dark blue patch for 55 and 60 layer unnormalized models at large batch sizes depicts a total failure to train. These networks were trainable by reducing 7), but for consistency we keep ry the same in both cases.
Figure 2:
trained for 10 epochs, while the unnormalized
trained bag-of-local-feature models (BagNets) on ImageNet with an architecture that discards spatial information between patches and is thus considered to make extensive of texture patterns for classification (Brendel & Bethge 2019). For patch sizes {9. 17, 33), the top-5 accuracies of the BagNets are reduced to just 1.25%. 5 5.09%. and 14.62% for AWGN, respectively. Compared with Table 5 where all models obtain over 40%, these figures suggest that robustness to Gaussian noise is a good proxy for the use of texture for ImageNet classification. These results support the hypothesis that BN may be exacerbating this tendency to use superficial texture-like information.
Next, we evaluate the robustness of pre-trained ImageNet models from the torchviss wii.iiiiiiinii.o...ii modee repository, which conveniently provides models with and without BN Results are shown in Table 5 where BN improves top-5 accuracy on noise in some cases, but consistently reduces it by 8.54% to 11.00% (absolute) for PGD. The trends the same for top-1 accuracy, only the absolute values smaller; the degradation varies from 2.38% to 4.17%. Given the discrepancy between noise and PGD for ImageNet, include black-box transfer analysis in the Appendix E.2 that is consistent with the white-box analysis.
Finally, explore the role of batch size and depth in Figure 2 We find that BN limits the maximum trainable depth, which increases with the batch size, but quickly plateaus as predicted by Theorem 3.10 of (Yang et al., 2019). Robustness decreases with the batch size for depths that maintain reasonable test accuracy, at around 25 or fewer layers. This tension between clean accuracy and robustness as a function the batch size is not observed in unnormalized networks.
http:////poorooooo/////////tttta htmr1

---

Under review as a conference paper at ICLR 2020
5 VULNERABILITY AND INPUT DIMENSION
A recent work Simon-Gabriel al. 2019) analyzes adversarial vulnerability of natch-normalized networks at initialization time and conjectures based on a scaling analysis that, under the com- monly used He et al. (2055) initialization scheme, adversarial vulnerability scales as ~ vd. They also show in experiments that indepen- dence between vulnerability and the input dimension can be approximately recovered through adversarial training by projected gra- dient descent (PGD) (Madry et al. 2018), with a modest trade-off of clean accuracy. Intuitively, the input dimension should be relevant as it does not affect the image com- plexity (Shafahi et al. 2019).
Table 6: Evaluating the robustness of MLP with and without batch norm. See text for architecture. We observe 61 + 1% reduction in test accuracy due to batch norm for v = 84 compared to vd = 28.
Model
Test Accuracy (%)
vd
BN
Clean
Noise
0.1
We show that this can be achieved by sim- pler means and with little to no trade-off through lg weight decay, where the regular- ization constant À corrects the loss scaling as the norm of the input increases with d. We increase the MNIST image width va from 28 to 56, 84, and 1 112 pixels. The loss C is predicted to grow like vd for e-sized attacks by Thm. 4 of (Simon-Gabriel (2019). We confirm that without regulariza- tion the loss does scale roughly as predicted (see Table of Appendix F.. Training with &s weight decay, however, we obtain adversarial test
X
97.95 0.08
93.0 0.4
66.7 + 0.9
28
97.88 0.09
76.6 1.6:100 0.7
22.9 0.7
x
98.19 0.04
93.8 0.1
53.2 + 0.7
56
98.22 + 0.02
79.3 + 0.6
8.6 + 0.8
X
98.27 + 0.04
94.3 + 0.1
47.6 + 0.8
84
98.28 + 0.05
80.5 + 0.6
6.1 + 0.5
al.
accuracy ratios of 0.98 + 0. O., 0.96 and 0.00 and clean accuracy ratios of +.999 0.002, 0.996 + 0.003, and 0.987 + 0.004 for vd of 56, 84, and 112, respectively, relative to the original vd 28 dataset. A more detailed explanation and results are provided in Appendix
Next, we repeat this experiment with two- hidden-layer ReLU MLP, with the number of hidden units equal to the half the input dimension, and optionally use one hidden layer with batch norm To evaluate robust- ness, 100 iterations of BIM- were used with step size of 1e-3, and 6oc = 0.1. We also report test accuracy with additive Gaus- sian noise of zero mean and unit variance, the same first two moments as the clean im- ages]
Table 7: Evaluating the robustness of a MI.P with & weight decay (same À as for linear model, see Table [3 of Appendix FF. See text for architecture. Adding batch norm degrades all accuracies.
Model
Test Accuracy (%)
Vd
BN
Clean
Noise
== 0.1
X
97.62 0.06
95.93 + 0.06
87.9 + 0.2
56
/
96.23 + 0.03
90.22 + 0.18
66.2 + 0.8
Despite difference in clean accuracy of only 0.08 + 0.05%, Table 6 shows that for the original image resolution, batch norm reduced accuracy for noise by 16.4 0.4%, and for BIM-2x by 4..5%. Robustness keeps decreasing as the image size increases, with the at all sizes.
X
96.99 + 0.05
95.69 + 0.09
87.9 + 0.1
84
93.30 + 0.09
87.72 + 0.11
65.1 + 0.5
network having ~ 40% less robustness to BIM and 13 16% less to noise
We then apply the l regularization constants tuned for the respective input dimensions on the linear model to the ReLU MLP with no further adjustments. Table shows that by adding sufficient & regularization (A = 0.01) to recover the original (vd- = 28, no BN) accuracy for BIM of z 66% when using batch norm, we induce test error increase of 1.69 + 0.01%, which is substantial on
$ This choice of architecture is mostly arbitrary, the trends were the same for constant width layers.
We first apply the noise to the original 28 28 pixel images, then resize them to preserve the appearance of the noise.
7

---

Under review as a conference paper at ICLR 2020
MNIST. Furthermore, using the same regularization constant and no batch norm increases clean test accuracy by 1.39 + 0.04%, and for the BIM- perturbation by 21.7 + 0.4%.
Finally, following the guidance in the original work batch norm (loffe & Szegedy 2015 to the extreme (A = 0): when we reduce weight decay when using batch norm, accuracy for the 6 0.1 perturbation is degraded by 79.3 + 0.3% for vd = 56, and 81.2 + 0.2% for v 84.
In all cases, using batch norm greatly reduced test accuracy for noisy and adversarially perturbed imputs, while weight decay increased accuracy for such inpuss.
As supplementary evidence, we contrast the "Fooling images" (Nguyen al. 2013) and Carlini & Wagner 2017) examples of BN vs. L--regularized models on MNIST and SVHN in Appendix
RELATED WORK
Our work examines the effect of batch norm model robustness at test time. References with an immediate connection to our work were discussed in the previous sections; here we briefly mention other works that do not have direct relationship experiments, but relevant to batch norm
in general.
The original work loffe & Szegedy (2013) that introduced batch norm technique for improving neural network training and test performance motivated it in terms of "internal covariate shift" referring to the changing distribution of layer outputs, effect that requires subsequent layers to steadily adapt to the new distribution and thus slows down the training process. Several follow-up works started from the empirical observation that batch norm usually accelerates and stabilizes training, and attempted to clarify the mechanism behind this effect. One argument is that batch- normalized networks have a smoother optimization landscape due smaller gradients immediately before the effect of stacking many batch-normalized layers and prove that this causes gradient explosion that exponential in network depth for networks without skip connections and holds for any non-linearity. In practice, relatively shallow batch- normalzeddl networks seem to benefit from the "helpful smoothing" of the loss surface property Santurkar et al. (2018), while very deep networks not trainable (Yang) et al. 2019). In our work, we found that a single batch-normalized layer suffices to induce severe adversarial vulnerability.
layer (Santurkar et al., 2018). However, Yang et al. (2019) study the
concurrent submission suggests that BN induced vulnerability may be due to a mismatch between the tracked mean and variance values training versus test time (Anonymous) 2020). We investigate this hypothesis in Appendix D and find that the use of tracked statistics can play a similar role as tuning the numerical stability constant c, thus this does not completely account for the vulnerability.
Weight decay's 's loss scaling mechanism is complementary to other mechanisms identified in the literature, for instance that it increases the effective learning rate (van Laarhoven 2017 Zhang et al.) 2019ap. Our results are consistent with these works in that weight decay reduces the generalization fiich-oormalized networks where is presumed to
gap (between training and test error), even have no effect. Given that batch norm is not typically used on all layers, the loss scaling mechanism persists, although to a lesser degree in this case.
Shafahi et al. (2019) performed similar input dimension scaling experiments in this work and came to similar conclusion that the input dimension irrelevant to adversarial vulnerability. However, like Gabriel al. (2019), they use PGD rather than weight decay prevent vulnerability from increasing with input dimension. Although it can be shown that robust optimization is equivalent to parameter norm regularization for linear models if we allow the b-bal (aka disturbance 8) to vary with each example (Xu et al. 2009), we maintain that the latter a more efficient approach.
CONCLUSION
We found that there is no free lunch with batch norm when model robustness is concern: the accelerated training properties and occasionally higher clean test accuracy come at the cost of increased vulnerability, both to additive noise and for adversarial perturbations. We have shown that there is no inherent relationship between the input dimension and vulnerability. Our results highlight the importance of identifying the disparate mechanisms of regularization techniques.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anonymous. Towards an adversarially robust normalization approach. In Submitted to International Conference Learning Representations, 2020. under review.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing Defenses to Adversarial Examples. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 274-283, Stackholmsmássan, Stockholm Sweden, 2018, PMLR.
Nils Bjorek, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding Batch Nor- malization. In Advances in Neural Information Processing Systems 31, pp. 7705-7716. Curran Associates, Inc., 2018.
Wieland Brendel and Matthias Bethge. Approximating CNNs with Cae-of-local-Fatturee models works surprisingly well ImageNet. In International Conference on Learning Representations, 2019.
Yongqiang Cai, Qianxiao Li, and Zuowei Shen. A Quantitative Analysis of the Effect of Batch Normalization Gradient Descent. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 882-890, Long Beach, California, USA, 2019. PMLR
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, pp. 39-57. IEFF, 2017.
Gavin Weiguang Ding, Luyu Wang, and Xiaomeng Jin. AdverTorch v0.1: An Adversarial Robustness Toolbox based on Py PoTchc. arXiv preprint arXiv:1902.07665 2019.
Angus Galloway, Thomas Tanay, and Graham W. Taylor. Adversarial Training Versus Weight Decay. arXiv preprint arXiv: 1804.03380 2018.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
Justin Gilmer and Dan Hendrycks. A discussion of Adversarial examples are not bugs, they are features': Adversarial example researchers need to expand what is meant by "Robustness'. Distill, 2019. doi: 111391////iiiil0000019
Justin Gilmer, Luke Metz, Fartash Faghri, Sam Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow. Adversarial Spheres. In International Conference on Learning Representations Workshop Track, 2018
Ian. J. Goodfellow, Jonathon. Shlens, and Christian. Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human-Levar Level Performance on ImageNet Classification. In International Conference on Computer Vision, pp. 1026-1034. IEEE Computer Society, 2015.
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In International Conference on Learning Representations, 2019.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: Closing the generaliza- tion gap in large batch training of neural networks. In Advances in Neural Information Processing Systems 30, pp. 1731-1741. Curran Associates, Inc., 2017.
Sergey Toffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
Jöm-Henrik Jacobsen, Jens Behrmann, Nicholas Carlini, Florian Tramèr, and Nicolas Papernot. Exploiting Excessive Invariance Caused by Norm-Bounde Bounded Adversarial Robustness. In Safe Machine Learning Workshop at ICLR, 2019a.

---

Under review as a conference paper at ICLR 2020
Forn-Henrik/. Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive Invariance Causes Adversarial Vulnerability. In International Conference on Learning Representations. 2019b.
Michael L. Jordan. Why the logistic function? A tutorial discussion on probabilities and neural networks. MIT Computational Cognitive Science Report, 1995.
Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial Machine Learning at Scale. International Conference Learning Representations, 2017.
Antoine Labatie. Characterizing Well-Behaved vs. Pathological Deep Neural Networks. In Proceed- ings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3611-3621, Long Beach, California, USA, 2019. PMLR.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks. Neural Information Processing Systems, 2019. To appear.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference Learning Representations, 2018.
Norman Mu and Justin Gilmer. MNIST-C: A robustness benchmark for computer vision. In International Conference on Machine Learning Workshop on Uncertainty and Robustness in Deep Learning, volume abs/1906 023337 2019.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Computer Vision and Pattern Recognition, pp. 427-436. IEEE Computer Society, 2015.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Cclik, and Ananthram Swami. Practical Black-Box Attacks Against Machine Learning. In Asia Conference on Computer and Communications Security, ASIA CCS, pp. 506-519, Abu Dhabi, UAE 2017. ACM.
Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Ku- rakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang, Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, and Rujun Long. Technical Report on the CleverHans v2. 1.0 Adversarial Examples Library. 161.:1611000768. 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do CIFAR-10 Classifiers Generalize CIFAR-10? arXiv:/806.0000 2018.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How Does Batch Normalization Help Optimization? In Advances in Neural Information Processing Systems 31, pp. 2488-2498. 2018.
Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST. In International Conference on Learning Representations, 2019.
Ali Shafahi, W. Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples inevitable? In International Conference on Learning Representations, 2019.
Carl-Johann Sabon-Gabriel, Yann Ollivier, Leon Bottou, Bernhard Schölkopf, and David Lopez-Pat -Paz. First-Order Adversarial Vulnerability of Neural Networks and Input Dimension. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5817. PMLR, 2019.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference Learning Representations, 2015.
10

---

Under review as a conference paper at ICLR 2020
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. In International Conference on Learning Representations Workshop, 2014.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, and Nathan Srebro. The Implicit Bias of Gradient Descent Separable Data. In International Conference on Learning Representations, 2018.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin-Ye Chen, and Yupeng Gao. Robustness the Cost of Accuracy? A Comprehensive Study on the Robustness of 18 Deep Image Classification Models. Computer Vision ECCV 2018, pp. 644-661. Springer International Publishing, 2018.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014.
Thomas Tanay and Lewis D. Griffin. Boundary Tilting Persepective on the Phenomenon of Adversarial Examples. arXiv:/608.011 2016.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Ro- bustness May Be at Odds with Accuracy. In International Conference on Learning Representations, 2019.
George L. Turin. An Introduction to Matched Filters. IRE Transactions on Information Theory, pp. 311 311-329, 1960.
Twan van Laarhoven. 1.2 Regularization versus Batch and Weight Normalization. arXiv:1706.05350 2017.
Huan Xu, Constantine Caramanis, and Shie Mannor. Robustness and Regularization of Support Vector Machines. Journal of Machine Learning Research, 10:1485-1510, 2009, ISSN 1532-4435.
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel Schoenholz. A Mean Field Theory of Batch Normalization. In International Conference on Learning Representa- tions, 2019.
Guodong Zhang, Chaoqi Wang. Bowen Xu, and Roger Grosse. Three Mechanisms of Weight Decay Regularization. In International Conference on Learning Representations, 2019a.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Residual Learning Without Normalization via Better Initialization. In International Conference on Learning Representations, 2019b.
A THE NUMERICAL STABILITY CONSTANT
The constant e originally added to the mini-batch variance in the denominator for numerical stability (named e in loffe & Szegedy (2015)) turns out be an important hyperparameter in terms of robustness. It acts as threshold on the variance of all input dimensions or neurons. When is much less than the minimum variance over dimensions, induces boundary tilting along the low- variance dimensions. In Figure 3 we sweep e for MNIST 3 vs. 7 and CIFAR-10, and compare the corresponding clean test accuracy with FGSM and AWGN accuracy for MNIST, and AWGN for CIFAR-10 10. For MNIST, increasing allows to trade-off clean accuracy for robustness to FGSM, but is suboptimal compared to L2 weight decay, For these experiments we fixed Tor and 3a
For CIFAR-10, eight-layer VGG models were trained with constant learning rate of 0.01 with no drops, momentum of 0.9, a batch size of 128, and 50 epochs (for computational reasons) over four random seeds. As for BNGD, for this particular experiment consistent trend is observed where robustness to noise increases greatly as c is increased, but we note that this occurs for e several orders of magnitude greater than default settings.
apply BN only to the input layer. A
11

---

confirm that the clipping itself Under review as a conference paper at ICLR 2020
100%
95%
80%
FGSM (e=0.1) Accuracy
70%
80%
85%
AWON Accuracy
AWGN Accuracy
60%
60%
50%
7636
40%
40%
BNGD FGSM
BNGD AWGN
65%
30%
20%
GD-L2 AWGN
BNSGD
GD-L2 FGSM
20%
SGD
0%
65%
10%
96.0%
96.5%
97.0%
97.5%
98.0%
66%
70%
74%
78%
82%
86%
Clean Test Accuracy
Clean Test Accuracy
aa)
(b)
Figure 3: Sweeping the BN numerical stability constant for (a) the MNIST 3 vs. 7 dataset with & e
[lc-3, 2c+1] for BNGD, and bascline the L2 regularization constant À E [le-3, 9] (shown as "GD-L2") Sweeping с for narch-normalized SGD (BNSGD) on the CIFAR-I0 dataset with c e [1e-6, 3 for the VGG8 architecture. Increasing either or À has similar effect to trade clean test accuracy for increased robustness, until the effect is too large and both accuracies degrade. The absolute accuracies are consistently higher without BN. Error bars indicate standard error of the mean over four and five random seeds for MNIST and CIFAR-I0, respectively. We recommend following each curve from right left be consistent with our description above. The default setting (highest clean test accuracy, lowest robustness) starts in the bottom right corner and the initial trade-off between clean test accuracy and robustness is traced up and leftwards until the curves inflect.
B PGD IMPLEMENTATION DETAILS
We used the PGD implementation from (Ding al., 2019), with example code snippet below. Because we normalize the data to zero mean and unit variance for SVHN, CIFAR-IO, and ImageNet, the pixel range clipped to {+1}. [=2}. and {+2] respectively These ranges were obtained by visual inspection of the image histograms and confirming that most of the pixels fall within this range. As sanity check, set the perturbation magnitude had negligible impact on test accuracy.
As result, absolute accuracy comparisons with other works where the input is not centered, e.g. [o, 1], should be compared to 6 = 4/255 in those works for e = 8/255 in this work.
B.1 FOR EVALUATION
During the evaluation of robustness with PGD, we use 6x0 = 0.03 and a step size of 6x /s0 for SVHN, CIFAR-10, and 6x 0.01 for ImageNet. For PGD-6z we set 62 Coo Vd, where d is the input dimension. To reduce the random error of the evaluation no random start was used at test-time, iee rand init=False. A code snippet for the PGD-E evaluation on SVHN is shown below:
from advertorch. import LinfPGDAttack
adversary = LinfPGDAttack (net, loss_fn=nn fn=nn.n=n....m www.snrrrosooollllooooooot (rer:/utssssscccsutro eps=0. 03, nb _b_tter=00, eps_iter=0._r Sand_init=False clip_min=-1.0, clip_max-1.0 targeted-False)
We report 20 iterations in most cases unless stated otherwise, but confirmed that using 40 iterations did not significantly improve upon this to within the measurement random error, i.e., the accuracy was not degraded further. For example, for VGG16 CIFAR-10 evaluated using 40 iterations of PGD with 6 = 0.03 and step size of /20, instead of 20 iterations with 6 / 10, accuracy changed from 28.9 + 0.2% to 28.5 + 0.3%, which is difference of only 0.4 + 0.5%, less than the random (standard) error.
thub. For CIFAR-10, see e.g. http://hhh..hgocoo/////go///oooo
main.pytla3 for the preprocessing scheme used.
12

---

Under review as a conference paper at ICLR 2020
Toxt Accuncy (60
50
Tea Arcunao (90
Tal Autuaee
Sroup 233
00
320
Loyens
Layers
versus unnormalized (Fixup) residual networks of varying depth. Models were trained with standard hyperparameters and evaluated on CIFAR-I0: [a] clean test set, [b) noisy test set, [c) PGD & and (d) PGD lz. Error bars denote the standard error of the mean over five random seeds. The unnormalized networks obtain higher accuracy for tests and depths.
Figure 4: Accuracy of
B.2 FOR PGD ADVERSARIAL TRAINING
For PGD training, the initial random perturbation (rand iol__initt (hii--True) was used improve the diversity of adversarial examples generated as per Madry et al. (2018). The meta-parameten are provided on case-by aase-by-case basis Appendix D
C ON AN ACCURACY VS. ROBUSTNESS TRADE-OFF
is natural to wonder if the degradation in robustness arising from the use of BN is simply due to BN increasing the standard test accuracy, given known trade-off between the (Tanay & Griffin 2016 Galloway et al. 2018 8 Su et al. 2018 Tsipras et al. 2019). Note that if the relationship between input X and label free of noise, e.g., in Gilmer et (2018), then there is no such trade-off and increasing accuracy corresponds to increasing robustness. For the toy problem we studied in $ BN actually aligned the decision boundary with the Bayes-optimal solution, so increasing standard accuracy may be intrinsic to the normalization itself in some cases.
Given that BN does typically increase clean test accuracy by some small amount on commonly used datasets, we thought it was most representative to not intentionally limit the performance of BN. We did, however, find natural cases where BN did not improve clean test accuracy. We trained ResNets 20,32,44,566 10J using Fixup initialization CIFAR-10: all consistently obtain about 0.5% higher clean test accuracy than their marmh-normllized equivalent, and are also more robust to noise (s= 15%) and PGD & and & perturbations (= 30%), as shown in Figure 4
For MNIST, the results of Tables 6 & also show compatible clean accuracy irrespective of BN, and yet vastly different robustness. Thus, the vulnerability induced by BN is not merely consequence of increasing standard test accuracy.
D PGD TRAINING YIELDS UNFORTUNATE ROBUSTNESS TRADE-OFFS
We believe that it is most informative to evaluate corruptions or perturbations that are not presented models during training, which why we did not use PGD adversarial training in the main text given that we evaluate on the same. In practical setting we cannot assume awareness of potential attacks a priori.
For brevity, we opted to report accuracy for an arbitrary small value of in the main text. In general, however, is more informative to plot accuracy vs. e to ensure the accuracy reaches zero for reasonably large to help rule out gradient masking issues (Papernot et al. 2017 Athalye al. 2018). This also shows that e was not averry-picked. Figure 5(b) shows that PGD-L training recovers much of the BN-vulnerability gap when tested on PGD-La perturbations only, although there is still a non-trivial improvement at e = 8/255 from 38.84% to 41.57% (recall that we only trained with mmax 4/255, absolute accuracy is slightly lower than in Madry et al. (018)).
13

---

Under review as a conference paper at ICLR 2020
B-mis
B-mis
80
P-mile
P-mic
Accurscy
B-igt-ll
Accuracy
60-
B-gg--1
E-ggt-ll
F-tgt-11
40
40
20
0
0
2
10
Figure Test accuracy vs. 6 for (a) naturally trained ResNet32 (b)ja state-of-the-art WiteResNet 28-10 10 CIFAR-10 baseline (Madry et al. (008)) trained with PGD-L (<max = 4, 5 iterations, step size 1 out of 255, w/rand. start) in aatch-normalized ('B') and unnormalized ('F' for Fixup) variants. At test-time, 20 PGD iterations used (the "strong" adversary from Madry et al.) for the loos and lz norms. A misclassification objective ("mis') is compared to targeting the least-likely label ('tgt-ll").
Ultimately, adversarial robustness concerned with robustness the worst attack in our threat model. If we consider the cont rast corruption from CIFAR-10-C (Hendrycks & Dietterich 2019), PGD-& training with 4, iterations, and step size of in Figure 5(b).. , this reduces the absolute test accuracy by 23.5% and 28.5% for WideResNet 28-10 in Fixup and BN variants, respectively. We do not believe it is reasonable for a threat model natural image classifiers to exclude natural changes in image contrast (Gilmer& & Hendrycks 2019). Thus, the recipe of PGD training a high capacity model alone can exacerbate other vulnerabilities (Jacbseen et al. 201 9a Mu & Gilmer 2019).
Similar results are observed on an MNIST-C benchmark (Mu & Gilmer 2019), to which we add an additional Gaussian noise corruption. We PGD- train the LeNet-5 variant from the AdverTorch library (Ding et al. 2019) using 20 inner-loop iterations, step size of 0./in, and <max = 0.3/1.0 with random initial perturbation in Madry al. 2018). Table 8 shows that BN reduces the mean test accuracy on MNIST-C by 11 + 2%; the accuracy is reduced from 87.0 + 0.5% (Per-img) to 76 2% (BN). This is compatible with Mu Gilmer (019), which reported 11.15% degradation in absolute test accuracy for PGD training, although they did not report the variance over multiple runs. Note that Mu & Gilmer (2019) use slightly different CNN architecture that obtains 91.21% mean test accuracy, slightly higher than our 87.0 + 0.5% baseline.
In particular, the performance of BN and the PGD variants is decimated by altering the brightness of the images, whereas the baseline shows little performance degradation. It is known that PGD training yields a highly sparse network that implements a thresholding operation in the first layer, see the Appendix C Madry al. (2188), which forces the model to be invariant to perturbations with -norm < e. This excessive invariance itself another cause of adversarial vulnerability (Jacobsen et al., 2019a|b). The PGD trained batch- normaazzdd model (PGD BN) performs worse than the baseline by double digit percentages for each of the: "Brightness", "Fog", "Impulse Noise", "Stripe", and "Zigzag" corruptions, with 7 + 2% higher mCE.
14

---

Under review as a conference paper at ICLR 2020
Table MNIST-C corruption benchmark results for four models: A naturally trained baseline using per-image standardization, "Per-img". naturally trained batch-normalized baseline, "BN", PGD trained model, "PGD", and PGD trained batch-normalized model, "PGD BN" See text for PGD details. Values are the accuracy after applying each corruption to the test set, and the standard enror over three random initializations each model, to one or two significant figures. Cells for which the "Per-img" baseline dramatically outperforms all other columns emphasized in grey. For "BN", "PGD", and "PGD BN", corruption accuracies can fluctuate considerably with the random seed, despite low variance in clean test accuracy. Conversely, the accuracy of Per-img has low variance in all cases.
Model
Corruption
Per-img
BN
PGD
PGD BN
Clean
98.50 + 0.05%
99.12 + 0.03%
98.51 + 0.05%
98.86 + 0.02%
Brightness
98.46 + 0.06%
48 + 12%
27 + 3%
40 = 9%
Canny Edges
77 + 2%
79 + 3%
86 + 2%
84 + 2%
Dotted Line
96.7 +: 0.1%
95.5 + 0.5%
90 + 2%
90 - 1%
Fog
82.0 + 0.2%
23 + 5%
58 + 3%
69 + 2%
Glass Blur
91 + 1%
53 + 2%
94.6 + 0.3%
95.0 = 0.3%
Gaussian Noise
+ 1%
84.7 + 0.6%
2%
94.2 0.6%
Impulse Noise
96.5 + 0.2%
50 2%
50 4%
57 2%
Motion Blur
80.5 + 0.6%
86 + 2%
93.4 + 0.2%
93.6 - 0.3%
Rotate
89.1 + 0.2%
91.6 + 0.2%
91.1 + 0.3%
92.5 + 0.1%
Scale
87.9 + 0.5%
94.3 + 0.2%
88 + 1 1%
91.8 = 0.5%
Shear
95.2 + 0.2%
97.3 + 0.2%
97.2 + 0.1%
97.63 + 0.07%
Shot Noise
97.8 + 0.1%
83.7 + 0.8%
98.2 0.1%
98.60 0.04%
Spatter
90.4 + 0.5%
97.60 + 0.08%
96.2 0.2%
96.44 + 0.08%
Stripe
85.0 + 0.2%
92 + 1%
52 + 3%
51 + 8%
Translate
46.7 + 0.5%
56.6 + 0.7%
51 + 1%
54.6 = 0.8%
Zigzag
88.0 + 0.8%
81 + 1%
71 + 1%
76.3 + 0.3%
Overall
87.0 + 0.5%
76 + 2%
77 + 1%
80 = 2%
15

---

Under review as a conference paper at ICLR 2020
Table 10: (Small learning rate) VGG variants on SVHN with BN.
Test Accuracy (%)
L
Clean
Noise
PGD--
PGD-12
11
95.31 + 0.03
80.5 + 1
20.2 + 0.2
6.1 + 0.2
13
95.88 + 0.05
77.2 + 7
21.7 + 0.5
5.4 + 0.2
16
94.59 + 0.05
78.1 + 4
19.2 + 0.3
3.0 + 0.2
19
95.1 + 0.3
78 +
24.2 : 0.6
4.1 + 0.4
Table 11: (Large learning rate) Accuracies of WideResNet-28-10 on CIFAR-10 and CIFAR-10. 10.1 (v6).
CIFAR-10
CIFAR-101 10.1
Model
Clean
Noise
PGD-L
PGD-&2
Clean
Noise
Fixup
94.6 + 0.1
69.1 + 1.1
20.3 + 0.3
9.4 + 0.2
87.5 0.3
67.8 + 0.9
BN
95.9 + 0.1
57.6 + 1.5
14.9 + 0.6
8.3 + 0.3
89.6 + 0.2
58.3 + 1.2
E ADDITIONAL EMPIRICAL RESULTS
This section contains supplementary explanations and results to those of Section
E.1 DETAILED SVHN AND CIFAR-10 RESULTS
For SVHN, models were trained by stochastic gradient descent
Table 9: (Small learning rate) Test accuracies of VGG8 on SVHN.
(SGD) with momentum 0.9 for 50 epochs, with batch size of 128 and initial learning rate of 0.01, which was dropped by factor of ten at epochs 25 and 40. Trials were repeated over five random seeds. We show the results of this experiment in Table finding that BN increased clean test accuracy by 1.86 0.05%, and reduced test accuracy for additive noise by 5.5 0.6%, for PGD- by 17 = 1%. and for PGD-L2 by 20 + 1%.
BN
Clean
Noise
PGD-Loo
PGD-&
X
92.60
0.04
83.6 1 0.2
27.1 0.3
22.0 1 0.8
94.46 0.02
78.1 = 0.6
10
1.6 0.3
Our first attempt to train VGG models on SVHN with more than eight layers failed, therefore for a fair comparison we report the robustness of the deeper models that were only trainable by using BN in Table 10 None these models obtained much better robustness in terms of PGD-f2, although they did better for PGD-Ex
Fixup initialization was recently proposed to reduce the use of normalization layers in deep residual networks (Zhang et al. 2019b). As natural test we compare WideResNet (28 layers, width factor 10) with Fixup versus the default architecture with BN. Note that the Fixup variant still contains one BN layer before the classification layer, but the number of BN layers is still greatly reduced
We train WideResNets (WRN) with five unique seeds and report their test accuracies Table Consistent with (Recht al. 2018), higher clean test accuracy CIFAR-10, i.e. obtained by the WRN compared to VGG, translated to higher clean accuracy on CIFAR-10.1 However, these gains were wiped out by moderate Gaussian noise. VGG8 dramatically outperforms both WideResNet variants subject to noise, achieving 78.9 18.9900.6. vs. 1.11 Unlike for VGG8, the WRN showed little generalization gap between noisy CIFAR-10 and 10.1 variants: 69.1 + 1.1 is reasonably comparable
'We used the implementation from battps://gls 150 epochs for consistency with the VGG8 experiment. Both models had already fit the training set by this point.
cww. vac 11 melllieenkee Lenk/ fixup but stopped training
16

---

Under review as a conference paper at ICLR 2020
Table 12: ImageNet validation accuracy for adversarial examples transfered between VGG variants of various depths, indicated by number, with and without BN ("V",, "x"). All adversarial examples were crafted with BIM-& using 10 steps and step size of 5e-3, which higher than for the white- analysis to improve transferability. The BIM objective was simply misclassification, i.e., it was not targeted attack. For efficiency reasons, we select 2048 samples from the validation set. Values along the diagonal in first two columns for Source Target indicate white-box accuracy.
Target
11
13
16
19
Acc. Type
Source
X
X
/
X
/
X
V
X
1.2
42.4
37.8
42.9
43.8
49.6
47.9
53.8
Top
11
/
58.8
0.3
58.2
45.0
61.6
54.1
64.4
58.7
X
11.9
80.4
75.9
80.9
80.3
83.3
81.6
85.1
Top 5
11
87.9
6.8
86.7
83.7
89.0
85.7
90.4
88.1
with 07.8=0.9 and 1.5 with 1.2. The Fixup variant improves accuracy by 1 for noisy CIFAR-10, 10, 9.5 = 1.5% for noisy CIFAR-10.1, 5.4 + 0.6% for PGD- and 1.1 = 0.4% for PGD-22.
We believe our work serves as compelling motivation for Fixup and other techniques that aim to reduce usage of BN. The role of skip-connections should be isolated in future work since absolute values were consistently lower for residual networks.
E.2 IMAGENET BTACK-BOOX TRANSFERABILITY ANALYSIS
The discrepancy between the results additive noise and for white-box BIM perturbations for ImageNet in Section 3 raises natural question: Is gradient masking factor influencing the success of the white-box results on ImageNet? No, consistent with the white-box results, when the target unnormalized but the source is, top accuracy 10.5% 16.4% higher, while top accuracy is 5.3% - 7.5% higher, than vice versa. This can be observed in Table 12 by comparing the diagonals from lower left upper right. When targeting an unnormalized model, we reduce top accuracy by 16.5% 20.4% using source that is also unnormalized, compared to difference of only 2.1% 4.9% by matching narch-normalized networks. This suggests that the features used by unnormalized networks are more stable than those of batch-normalized networks.
Unfortunately, the pre-trained ImageNet models provided by the PyTorch developers do not include hyperparameter settings other training details. However, we believe that this speaks to the generality of the results, i.e., that they are not sensitive to hyperparameters.
E.3 BATCH NORM LIMITS MAXIMUM TRAINABLE DEPTH AND ROBUSTNESS
In Figure we show that BN not only limits the maximum trainable depth, but robustness decreases with the batch size for depths that maintain test accuracy, at around 25 or fewer layers (in Figure 6(a)). Both clean accuracy and robustness showed little to no relationship with depth nor batch size in unnormalized networks. A few outliers are observed for unnormalized networks at large depths and batch size, which could be due the reduced number of parameter update steps that result from higher batch size and fixed number of epochs (Hoffer et al. 2017).
Note that in Figure Tig)re(o(a)aa the bottom row- without batch norm appears lighter than the equivalent plot above, with batch norm, indicating that unnormalized networks obtain less absolute peak accuracy than the mach-normaliizd network. Given that the unnormalized networks take longer to converge, we prolong training for 40 epochs total. When they do converge, see more configurations that
achieve higher clean test accuracy than aatch-normalized networks in Figure 6(b) Furthermore, good robustness can be experienced simultaneously with good clean test accuracy in unnormalized networks, whereas the regimes of good clean accuracy and robustness are still mostly non-overlapping in Figure ((b)
17

---

Under review as a conference paper at ICLR 2020
Clean Accuracy (%)
Noise Accuracy (%)
Clean Accuracy (%)
Noise Accursey (%)
60
50
40
Botch norm
40
Batch norn
L
L
30
20
20
10
50
Umoomna lised
Unmornalize
L
L
30
20
10
B3
B
B
B
0.4
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
(a)
Figure 6: We repeat the experiment of Yang et al. (2019) by training fully-connected models of depth I. and constant width (N/=384) with ReLU units by SGD, and learning rate 10-5 B for batch size B MNIST. We train for 10 and 40 epochs in and 3 were left as default, momentum disabled, and le-3. Each coordinate is first averaged over three seeds. Diamond-she -shped artefacts for unnormalized case indicate one of three seeds failed to train note that we show an equivalent version (a) with these outliers removed and additional batch sizes from 5-20 in Figure Best viewed in colour.
(a) and (b) respectively. The BN parameters
F WEIGHT DECAY AND INPUT DIMENSION
Consider a logistic classification model represented by a neural network consisting of a single unit, parameterized by weights E I2 and bias b E I, with input denoted by x E IRd and true labels y E [+1]. Predictions are defined by s = w x + b, and the model is optimized through empirical risk minimization, i.., by applying stochastic gradient descent (SGD) to the loss function equation 2 where ((=) = log(1 + e=)):
Eg.y-ran Cly(w' x b))-
(2)
We note that w by our model. If define d(x:) the signed Euclidean distance between and the boundary, then we have: w x + b = | d(xr). Hence, minimizing equation 2 is equivalent to minimizing
T x + is a scaled, signed distance between r and the classification boundary defined
E Eg.yropm <([lw||e d(x)).
(3)
We define the scaled loss as
S/ (2) = ([|w z)
and note that adding & regularization term in equation 3 resulting in equation 5 can be understood as a way of controlling the scaling of the loss function:
Ez.er-aa S (y d(x)) A||w||2
we develop intuition for the different quantities contained in equation 2 with respect to a typical binary classification problem, while Figures 7(d) 7(f) depict the effect of the regularization parameter on the scaling the loss function.
In Figures 7 7
To test this theory empirically we study model with single linear layer (number of units equals input dimension) and cross-entropp loss function on variants of MNIST of increasing input dimension, to approximate the toy model described in the "core idea" from (Simon Gabriel et al.) 2019) as closely as possible, but with a model capable of learning. Clearly, this model is too simple to obtain competitive test accuracy, but this is helpful first step that will be subsequently extended ReLUU
18

---

Under review as a conference paper at ICLR 2020
(a) w T x
(b) y(w x)
(c) Clo(w"ar x)) (d) (o(yd(a)) (c) (o.()d((x)) ( (o
For a given weight vector w and bias b, the values of w= + over the training set typically follow bimodal distribution (corresponding to the two classes) centered on the classification boundary. bb) Multiplying by the label y allows us to distinguish the correctly classified data the positive region from misclassified data in the negative region.
Figure 7:
[c] We can then attribute a penalty to each training point by applying the loss to y(w x b). (d) For a small regularization parameter (large || the misclassified data is penalized linearly while the correctly classified data is not penalized. (e) medium regularization parameter (medium || corresponds to smoothly blending
the margin. For large regularization parameter (small [[=]|2), all data points are penalized almost
linearly.
networks. The model was trained by SGD for 50 epochs with constant learning rate of le-2 and a mini-batch size of 128. In Table [3 we show that increasing the input dimension by resizing MNIST from 28 x to various resolutions with PIL. Image NEAREST interpolation increases adversarial vulnerability in terms of accuracy and loss. Furthermore, the "adversarial damage", defined as the average increase of the loss after attack, which is predicted to grow like vd by Theorem of (Simon-Gabriel et al. 2019), falls in between that obtained empirically for e = 0.05 and 0.1 for all image widths except for 12, which experiences slightly more damage than anticipated.
SimoneGabriel et al. (2019) note that independence between vulnerability and the input dimension can be recovered through adversarial-examen augmented training by projected gradient descent (PGD), with a small trade-off terms of standard test accuracy. We find that the same can be achieved through much simpler approach: l weight decay, with parameter À chosen dependent d to correct for the loss scaling. This way we recover input dimension invariant vulnerability with little degradation of test accuracy, e.g., see the result for vd = 112 and = 0.1 in Table 13 the accuracy ratio 1.00 + 0.03 with weight decay regularization, compared to 0.10 = 0.09 without.
Compared to PGD training, weight decay regularization i) does not have an arbitrary hyperparameter that ignores inter-sample distances, ii) does not prolong training by a multiplicative factor given by the number of steps in the inner loop, and 3) is less attack-specific. Thus, we do not use adversarially augmented training because we wish to convey a notion of robustness to unseen attacks and common corruptions. Furthermore, enforcing robustness to e-perturbations may increase vulnerability to invariance-bused examples, where semantic changes are made to the input, thus changing the Oracle label, but not the classifier's 's prediction (Jacobsen et al. 2019a). Our models trained with weight decay obtained 12% higher accuracy (86% vs. 74% correct) compared to batch norm a small sample of 100 &o invariance-batt based MNIST examples We make primary of traditional ép perturbations as they are well studied in the literature and straightforward to compute, but solely defending against these is not the end goal.
more detailed comparison between adversarial training and weight decay can be found in (Galloway al. 2018). The scaling of the loss function mechanism of weight decay is complementary to other mechanisms identified in the literature recently, for instance that it also increases the effective learning rate (van Laarhoven, 2017 Zhang al. 2019a). Our results are consistent with these works in that weight decay reduces the generalization gap, even in batch-normalized networks where it is presumed have no effect. Given that batch norm not typically used on the last layer, the loss scaling mechanism persists in this setting, albeit to lesser degree.
Invariiace based adversarial examples downloaded from ht.ps.//gitthbb.comffrraaa
Excess ve- Invarianne
19

---

Under review as a conference paper at ICLR 2020
Table 13: Mitigating the effect of the input dimension on adversarial vulnerability by correcting the margin enforced by the loss function. Regularization constant À is for & weight decay. Consistent with Saionn-Gabrian et al. 2019). we use c-FGSM perturbations, the optimal loc attack for linear model. Values in rows with d > 28 are ratios of entry (accuracy or loss) wrt the vd = 28 baseline. "Pred." is the predicted increase of the loss L due small e-perturbation using Thm. of (Simon-Gabriel et al. 2019).
Model
(Relative) Test Accuracy
(Relative) Loss
vd
À
Clean
c 0.1
Clean
0.1
Pred.
92.4 0.1%
53.9
0.3%
0.268 0.001
1.410 + 0.004
1.001 + 0.001
0.33 + 0.03
1.011 0.007
2.449 + 0.009
0.01
0.999 = 0.002
0.98 = 0.01
1.010 = 0.007
1.01 = 0.01
84
0.998 + 0.002
0.10 + 0.09
1.06 + 0.01
4.15 + 0.02
84
0.0225
0.996 + 0.003
0.96 + 0.04
1.05 + 0.02
1.06 + 0.03
112
0.992 = 0.004
0.1 = 0.2
1.18 + 0.03
5.96 = 0.02
112
0.05
0.987 + 0.004
1.00 + 0.03
1.14 + 0.04
1.04 + 0.03
Table 14: Two-hidden-layer ReLU MLP (see main text for architecture), with and without batch norm (BN), trained for 50 epochs and repeated over five random seeds. Values in rows with vd > 28 are ratios wrt the vd = 28 baseline (accuracy or loss). There a considerable increase of the loss, or similarly, a degradation of robustness in terms of accuracy, due to batch norm. The discrepancy for BIM- with 0.1 for vd 84 with batch norm represents 61 + 1% degradation in absolute accuracy compared to the baseline.
Model
(Relative) Test Accuracy
(Relative) Loss
Vd
BN
Clean
0.1
Clean
e 0.1
28
97.95 + 0.08%
66.7 + 0.9%
0.0669 + 0.0008
1.06 0.02
28
0.9992 + 0.0012
0.34 + 0.03
1.06 + 0.04
3.18 + 0.03
56
1.0025 = 0.0009
0.80 + 0.02
0.87 + 0.02
1.68 + 0.03
56
/
1.0027 + 0.0008
0.13 + 0.09
0.91 + 0.03
5.83 + 0.03
84
x
1.0033 + 0.0009
0.71 + 0.02
0.86 + 0.02
2.15 + 0.03
84
/
1.0033 + 0.0010
0.09 + 0.08
0.88 + 0.02
7.34 + 0.02
20

---

Under review as a conference paper at ICLR 2020
Inpat
Layer
Layer 14
Impal
Layer COOK
Layer 14
000 G0O9
(a)
(b)
Figure 8: Two mini-batches from the "Adversarial Spheres" dataset (2D variant), and their representa- tions in deep linear network at initialization time (a) with batch norm and [b) without batch norm. Mini-batch membership is indicated by marker fill and class membership by colour. Each layer is projected to its two principal components. In [b) we scale both components by factor of 100, as the dynamic range decreases with depth under default initialization. We observe in that some samples are already overlapping at Layer 2, and classes are mixed at Layer 14.
G COMMON CORRUPTION ROBUSTNESS
For VGG8, the mean generalization gaps due batch norm for noise were: Gaussian- = 1.9%, Impulse- 7.5 0.8%, Shot-3.6
3.6 = 1.6%, and Speckle-42 4.5 = 1.6%. After the "noise" category the next most damaging corruptions (by difference accuracy due batch norm) were: Contrast- 4.4 1.3%. Spatter-2... -.4 0.7%, JPEG- 2.0 + 0.4%, and Pixelate- 3 + 0.5%. Results for the remaining corruptions were coin toss whether batch norm improved degraded robustness, as the random eтror was in the same ballpark as the difference being measured.
For VGGI3, the batch norm accuracy gap enlarged 26 28% for Gaussian noise severity levels 3, 4, and 5; and over 17% for Impulse noise at levels 4 and 5. Averaging over all levels, we have gaps for noise variants of: Gaussian- 20.99 + 1.4%, Impulse- 1..e--3.6--- + 0.6%. Shot-14.1 1.0% and Speckle-11.1
1ickle--1..1t 1.1.1111 : 0.8%. Robustness to the other corruptions seemed to benefit from the slightly higher clean test accuracy of 1.3 0.1% for the notch-normalize VGG13. The remaining generalization gaps varied from (negative) 0.2 + 1.3% for Zoom blur, to 2.9 + 0.6% for Pixelate.
For the WRN, the mean generalization gaps for noise were: Gaussian 12.1 2.8%, Impulse 10.7 + 2.9%, Shot-8.7 + 2.6%, and Speckle-6.9 + 2.6%. Note that the large uncertainty for these measurements is due to high variance for the model with batch norm, on average 2.3% versus 0. 7% for Fixup. JPEG compression was next at 4.6 + 0.3%.
H ADVERSARIAL SPHERES
The "Adversarial Spheres" dataset contains points sampled uniformly from the surfaces of two concentric n-dimensional spheres with radii task is to attribute given point to the inner outer sphere. We consider the case = 2, that is, datapoints from two concentric circles. This simple problem poses a challenge to the conventional wisdom regarding batch norm: not only does batch norm harm robustness, it makes training less stable. In Figure we show that, using the same architecture as in (Gilmer et al. 20188) the batch- normalized network is highly sensitive to the learning rate vy. We use SGD instead of Adam avoid introducing unnecessary complexity, and especially since SGD has been shown to converge to the margin solution for linearly separable data (Soudry et al.) (0188.. We use finite dataset of 500 samples from (00, 7) projected onto the circles. The unormalized network achieves zero training error for 1 up to 0.1 (not shown), whereas the batch-normalized network is already untrainable at 0.01. To evaluate robustness, we sample 1 10,000 test points from the same distribution for each class (20k total), and apply noise drawn from (0, 0.005 x 1). We evaluate only the models that could be trained to 100% training accuracy with the smaller learning rate of 7/ 0.001. The model with batch norm classifies 94.83% of these points correctly, while the unnormalized net obtains 96.06%.
and R = 1.3 respectively, and the classification
21

---

Under review as a conference paper at ICLR 2020
0.6
0.001
0.8
0.001 BN
Train
0.01
0.01 BN
00
A33
0.0
50
100
150
200
50
100
150
200
Epoclss
Epochs
(a)
(b)
Figure 9: We train the same two-hidden-layer fully connected network of width 1000 units using ReLU activations and mini-batch size 50 on 2D variant of the "Adversarial Spheres" binary classification problem (Gilmer al. (2088). Dashed lines denote the model with batch norm. The batch-normalized model fails to train for a learning rate of = 0.01, which otherwise converges quickly for the unnormalized equivalent. We repeat the experiment over five random seeds, shaded regions indicate 95% confidence interval.
22

---

Under review as a conference paper at ICLR 2020
88.9
75.2
0 = 50.0
0 18.7
0 0.0
(a) 97.9%
(b) 97.9%
(c) 96.8%
(d) 96.9%
(e) 94.4%
(f)
Figure 10: Weight matrices of various logistic regression models annotated with their tilting angles w.r.t. the nearest centroid classifier (above) and clean test accuracy (below). Plots are (a) BNGD, tested in eval mode, (b) BNGD, tested in train mode, i.c. without tracking and using the full test set (batch size of 5000). (c) GD baseline with no preprocessing other than division by 255 for pixels in range [0, 1I. (d) GD baseline with per-image normalization, (e) the nearest centroid classifier, the standard deviation over the training set. For BN we use c=le-5.
88.9
86.8
0 -=3 81.8
0 76.1
0 73.4
0 70.3
3
(a) le-5
(b) le-4
(c) le-3
(d) le-2
(c) le-1
(f)
Figure 11: By increasing the numerical stability constant to between 1e-2 and Ic-1, we can achieve the same tilting angle as obtained by not using tracking (TOb)). but which works with arbitrary batch sizes at test time. We test BNGD in eval mode (with tracked mean and variance from the training set) and sweep c from the pytorch default value le-5 to 1. The weights are then visualized after training.
T ALTERNATIVE EXPLANATIONS OF THE VULNERABILITY
In concurrent submission, it is suggested that the main reason for BN induced vulnerability is mismatch between the mean and variance statistics at training versus test time (Anonymous 2020).
We investigate this hypothesis using the well controlled MNIST 3 vs. testbed from
Figure shows that by climinating tracking. i.c. going from plot [o(a) to I0(b the boundary tilting angle 8 w.r.t. the nearest-centroid classifier is reduced by 13.79. Eliminating BN altogether reduces by an additional 25.2 and using per-image normalization (rather than per- per-pixel as in BN) achieves further reduction of by 31.3". In terms of FGSM at 1/10, model 10(b) (BN, no tracking) achieves 38. 38%7% accuracy whereas the unnormalized baseline I0(c) achieves 66.1%.
The observation regarding tracking is insightful and contributes of BN. However, this experiment is a counterexample to the claim that the tracking aspect of BN the main source of vulnerability, compared to boundary tilting arising from the per-dimension normalization procedure itself. As acknowledged in Anonymous 2020). it is worth re-iterating that using BN without tracked statistics at test-time restricted to large batch sizes, which may not be practical.
refined understanding
In Figure we show that the reduction in vulnerability achieved by not using tracked statistics at test time can be achieved by increasing the numerical stability constant Here, the value can be used interpolate between the configurations achieved when tracking used (shown in Figure TO(a)). or not used (shown in Figure 10(b) for fixed
There remains distinct qualitative difference between the weight matrices visualized in Figures 10(b) (c=le-5, no - (c=1, with tracking)], which both resemble the hand-drawn digit "3", compared to the case without BN Figures [0(c) (e) which contain both a "3" as dark pixels and a "7" bright pixels. This is consequence of the BN procedure not being class-aware, i.e., all pixels of a digit "7" are normalized by the variance of both class "3" and class "7", which more closely resembles "3" for this dataset shown in Figure TO())
23

---

Under review as a conference paper at ICLR 2020
J ADVERSARIAL EXAMPLES
To alleviate possible concerns that gradient masking effects (Papernot et al., 2017; Athalye et al. 2018) may explain the apparent difference robustness between batch-normalized and unnormalized networks, we provide qualitative evidence that BN degrades the visual appearance of adversarial examples for strong unbounded attacks. In other words, we show that these unbounded adversarial examples remain "adversarial" for batch-normalized networks, while semantically relevant features are introduced for inputs classified with high confidence by the baselines, such that human oracle would agree with the predictions. If the gradients were obfuscated in any way, we would be unable to meaningfully increase the accuracy or prediction confidence of the models beyond random chance, and the examples that we initialize from white noise would remain noisy. Note: this is not the sole source of evidence we provide that gradient masking is not factor, the common corruption benchmarks and unbounded attacks reaching 100% success also support this.
J.1 MNIST
For the MNIST experiments, we train LetNet-5 CNN variant from the AdverTorch library (Ding) c al. 2019). All models (batch-normalize or not) are trained for 90 epochs of SGD with a constant learning rate of 0.01, and mini-batch size of 50. We compare three variants of this model:
1. L2: Uses an L2 weight decay regularization constant À = 0.01.
2. BN: The two convolution layers of the LetNet-5 are batch- nommaliedd, with default settings
(7 and / enabled, numerical stability constant set le-5, momentum of 0.1).
3. Per-img: Subsumes "L22 and additionally uses per-image normalization, i.e., subtract the global mean of each image to correct the average intensity, then divide by the standard deviation of all features in the image
The models obtain similar clean test accuracy: "I.2" 98.0%, "BN"-99.25, "Per-ing"-98.A
To construct adversarial examples, two unbounded attack schemes are used: "fooling im- ages" (Nguyen al. 2015) and the CCalini & Wagner (2017) (CWL2) method. The fooling images procedure starts with white noise, then minimizes the cross entropy loss wrt a target class until it is predicted with high confidence. "Confidence" here means the softmax margin, i.e., the difference between the max softmax output and the second from max output. Formally, we sample a starting 100
point 20 (0, 0.1) E 2828 then update according x" zn + 8, using n iterations of PGD-&z w.r.t. each class label. For "L2" and "BN", we clip pixel values to [0, 1]. and for "Per-img" to 11. We use a PGDD-2 step size of 0.2.
Figure 12 shows the fooling images, their predicted class labels, and prediction confidence for the three configurations. The procedure bears similarity to "Activation Maximization" techniques for interpreting neural networks, but this usually requires post-hoc regularization of the input images (Si-
monyan et al. 2014). This was not required here, owing to the competitive robustness of models (2()) and [2(b) which makes them somewhat interpretable by default. The aatch-normalize model classi- fies images that only faintly resemble natural digits with full confidence, while those of the baselines contain more clearly class-relevant features. In particular, the "Per-img" case in Figure T2(b) makes no fully saturated confidence predictions, and its fooling images resemble matched filters (Turnn 1960) w.r.t. task relevant edges.
The Carlini & Wagner (2017) (CWL2) Adam optimizer- aptimizer-based attack seeks to find the smallest perturbation in terms of &a distance which achieves either arbitrary, or targeted, misclassification. We use the arbitrary misclassification objective, confidence parameter k = 4 to encourage reasonably high confidence predictions, "learning rate" of 1e-2, nine (9) binary search steps, and an initial value for the constant c of le-3, which balances the minimization of the mn-mrrm of the perturbation 8, and prediction confidence I We allow up 10,000 attack iterations, and show the resulting examples For the same confidence threshold, the mean & norm of the perturbations required
in Figure 13
1hhis contrasts sharply with BN which computes the same statisties but over samples, and then normalizes each dimension.
"l Note that c here is meta-parameter of the CWT.2 attack and is not to be confused with the BN numerical stability constant.
24

---

Under review as a conference paper at ICLR 2020
0: 1.00
1: 1.00
2: 1.00
3: 1.00
4: 1.00
5: 1.00
6: 1.00
1.00
8: 1.00
9: 1.00
(a)
0: 0.55
0.85
0.22
0:94
5:
0.77
0,87
8: 0.87
9: 0.79
Per-ing
(b)
1.00
4:
1.00
1.00
BN
(c)
Figure 12: Fooling images crafted on three variants LeNet-5 Model. Case (a) is the Le weight decay
baseline, case (b) subsumes (a) and includes per-image normalization. In case [c) the convolution layers the LeNet are batch- normm-nzda and no other explicit regularization is used. See text for detailed training and evaluation meta- parameearssssss.
to change the predictions of the batch-normalized model was ||ō]| 0.95, while ||ö]|2 2.89 was required for the unnormalized model. This is a three fold improvement in terms of the relevant performance metric for this attack, which always obtains 100% success by virtue being unbounded.
J.2 SVHN
For SVHN, visualize the fooling images in Figure 14 and all combinations of source to targeted misclassification crafted using a basic iterative gradient method (BIM-(2) for a random sample of each class in Figure 15
We use simple CNN architecture deseribed in Table 15 The model trained for 50 epochs of SGD on the SVHN dataset (excluding the "extra" split) with constant learning rate of 0.01, and mini-batch size of 128 in both cases. Since SVHN is imbalanced, the loss is weighted by the inverse L2 weight decay regularization
frequency of each class. As in I1. for the no-BN baseline constant À = 0.01, and per-image preprocessing.
Although both models obtain similar a accuracy of about 90%, the fooling images for the unnormalized model contain task relevant edges, like those of Figure 12bb whereas the batch- normalized model yields difficult to interpret images. Similarly, the examples of the target class in many cases, while those of (5bb) for the batch-normalized model have a subtle textured effect, and preserve the semantic cues the source image.
resemble
K ON THE INITIAL LEARNING RATE
is known that using a high initial learning rate, which annealed during training, often achieves higher test accuracy than training with smaller fixed learning rate, even though the latter approach optimizes the training loss more quickly. To explain this phenomenon, Li et al. (2019) introduce the concept of learning order, which training time property that was found to be predictive of generalization ability, in contrast with post- cosi-trainnng notions of model complexity that are typically believed to govern generalization. Learning order refers different examples being learned at
25

---

Under review as a conference paper at ICLR 2020
7: 1.00
2: 0.90
0.99
1.00
0.98
1:
4: 0.99
0.98
5: 0.77
9: 0.96
7
2_
/
O
4
/
4
(a)
3: 0.94
I: 0.94
4: 0.94
0.95
9: 0.96
7: 0.95
9: 0.82
4: 0.95
6: 0.94
7: 0.96
7
2
+
9
7
(b)
0.55
0.96
0.95
Per-ing -ng
7
2_
7
9
7
4
9
(c)
2: 0.96
0:
0.96
0.96
9: 0.96
7:
0.96
0.96
6: 0.96
4: 0.96
BN
7
2
/
O
4
/
4
a
9
(d)
Figure 13: Candidate adversarial examples crafted with the Carlini Wagner 2017 (CWL2) attack originally correctly classified images (a) For CWL2 set the confidence parameter k 4, and use the "arbitrary misclassification' objective, i.e., the adversary only needs to flip the label, not achieve any particular label. (b) shows the perturbed images (Iadv x + 5) for the baseline L2 regularized model (|00|2 2.89), while plot shows those of the batch-normalized model. The "adversarial examples" shown in (b) contain semantic features corresponding the predicted class. An image which was originally correctly classified as the digit "1" is now classified as "7", but the top of "7" has been added, as well the commonly used "dash" through the center. Several instances of "4" are turned into "9", as the top of the loop of the "4" is completed. For the normmlzzedl model, whose adversarial examples are shown in (d) the examples are classified with the same or higher confidence, yet the perturbations are not perceptible and their 2n-nrs are significantly lower (|o]]|2 = 0.95).
0: 1.00
1.00
2: 1.00
3: 1.00
4: 1.00
5: 1.00
6: 1.00
7: 1.00
8: 1.00
9: 1.00
4
5
6
7
(a)
1.00
1.00
1.00
3: 1.00
4: 1.00
5: 1.00
6: 1.00
1.00
1.00
1.00
(b)
Figure 14: Fooling images crafted on a simple CNN and the same CNN with two siich-normaaized convolution layers (b) See for detailed training and evaluation meta-parameters.
26

---

Under review as a conference paper at ICLR 2020
0
1
3
4
5
7
8
9
5
2
3
3
3
3
5
7
4
7
5
5
6
7
8
199
1
3
5
6
7
8
9
(a)
0
1
3
4
5
7
8
9
0
0
D
0
0
0
0
O
f
2
2
2
2 2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
5
7
5
7
5
5
7
6
5
6
7
8
88
8
8
18
88
18
183
18
18
183
9
9
9
9
9
9
9
9
9
9
9
(b)
Figure 15: All source-target transformations obtained by iterating BIM-(2 until the baseline (a) predicts the target class (enumerated over the columns) with a mean margin of 90%. The source class is enumerated along the rows such that the natural images are on the diagonal. (b)is batch-normalized.
27

---

Under review as a conference paper at ICLR 2020
Table 15: CNN architecture adapted from the CleverHans library Papernot al. 2018 for the SVHN fooling image experiment. We denote the convolution kernel height, width, number of input and output channels w.r.t. each layer, and stride as: h, w, Cans Cout, 8, respectively. For the BN variant, we apply BN with default meta-parameters after Convl and Conv2.
Layer
h
w
Cin
Cout
params
Convl
8
8
1
32
2
2.0k
Conv2
6
6
32
64
2
73.8k
Conv3
5
5
64
64
102.4k
Fc
1
256
10
2.6k
Total
180.9k
different times, according to their intrinsic noise level and complexity. [L et al. find that using high initial learning rate prevents the model from fitting high complexity patterns too early, at the expense of easier to fit patterns. When the learning rate is annealed, the model effectively moves to the next stage a "curriculum" " in which can learn more challenging patterns.
To examine the interplay between the learning rate and robustness, conduct similar experiment on CIFAR-10 10 as in Li et al. (2019) with "small" and "large" initial learning rate ("Ir"). We evaluate several measures of robustness every 10 epochs to better characterize the regimes in which the vulnerability occurs, and its relationship with the annealing of the learning rate. A shallow VGG variant (VGG8) is trained over 150 epochs using two different initial learning rates (details in caption of Figure 16). and the process is repeated over three random seeds for BN vs. no-BN. We use standard meta-paraaaa pattapparamtters SGD with momentum 0.9, mini-batch size of 128, weight decay 5e-4, and standard data augmentation. We use "Un" to denote the unnormalized model, and "BN" for batch- normalized. The final test accuracies are: "Un, small Ir"; 88.7 + 0.1%, "Un, large Ir": 90.4 + 0.1%. (diff. 1.7 + 0.1%). "BN, small Ir": 90.3 + 0. 1%, "BN, large Ir": 91.3 + 0.1, (diff. 1.0 + 0.1%).
To evaluate robustness, we evaluate accuracy on several variants of the test set: "Clean" the original test set, "AWGN"-wwe with additive white Gaussian noise (o² 1/16), "FSGM" with perturbation crafted by one-step fast gradient sign method foo, and "PGD" -a 40-step PGD-C perturbation, with step size of e /30 We use 8/255 as the Ex norm budget for both FGSM and PGD perturbations, and clip the pixel values to [].
For the "high Ir" case shown in Figure T6(a) the unnormalized model achieves higher robustness than the batch-normalized model after around 70-80 epochs, shortly after the learning rate first annealed. Meanwhile, Figure 16(b) shows that in the "small Ir" case, the unnormalized model achieves higher robustness on all tests compared to its batch-normalized equivalent, at all epochs. The "high Ir" result of Tigure[To(a) begets the question: if are willing sacrifice clean test accuracy and do not anneal the large learning rate, can we mitigate the BN vulnerability? We investigate this in a subsequent experiment depicted in Figure 17 which indeed shows that there appears to be little difference in robustness for this case. We conclude with a few remarks:
1. The "large constant learning rate" is not particularly practical setting, as 10% absolute clean test accuracy has been left on the table.
2. We interpret the "large constant learning rate" result not as "the vulnerability does not occur", but rather, the baseline fails to reach its potential. Indeed, from Figure 16(b)and T training the same architecture with small learning rate yields 52.9 + 0.6% for the same PGD-Lx perturbation, while the BN variant achieves less than 40% accuracy at all times (this can be seen by inspection of Figures an]and 17).
3. The accuracy of the unnormalized model on all test sets increases almost monotonically with prolonged training, and appears to be still increasing at the conclusion of the 150 epochs. For fairness, we did not investigate further training to see how long this trend continues. Conversely, the batch-normalized model converges quickly then plateaus. At this point its accuracy tends to oscillate, or even decline monotonically (particularly for PGD in Figure 16(b), thus careful early stopping is much more relevant BN.
28

---

Under review as a conference paper at ICLR 2020
Un Clonn
BN Clesn MYGN
Un PGSM
BN FOSM
Us Train
RN Train
AWGN
90
Accuracy (TTss set)
Accuracy (Test set)
Accuracy (TTaiin sat)
80
70
60-
30
40
20
40
60
80
100
120
20
40
80
80
100
120
0
20
40
60
80
100
120
Epochs
(a)
Un Clon
BN Clean
Un PGSM
BN FGSM
Un Train
RN Train
60
PGD
100
90
Avcuracy (Test seot
Aceuracy (Train set)
Avcureyy (Test
80
50
80
70-
60-
60-
20
40
80
80
100
120
20
40
80
100
120
20
40
60
100
120
Epochs
(b)
Figure 16: We train VGG model on CIFAR-10 for 150 epochs using a "large" an) "small" (b) initial learning rate (0.1, and 0.01 respectively), which is dropped by a factor of ten at epochs 60 and 120 (best seen in the training curves of the right column). Every ten epochs, we evaluate accuracy several variants of the test set: "Clean"-the original test set, "AWGN" with additive white Gaussian noise, "FSGM"-wiid fast gradient sign method one-step fx perturbation, and "PGD"-a 40-step PGD-f perturbation. See text for additional mnraa-aaraettes. The light grey shading indicates where the aecuracy of the unnormalized model exceeds that of the natch-normalized equivalent for a given test. Note: the series "Un PGD" achieves higher accuracy than "BN FGSM" in (b) so we use two different shades to distinguish the overlapping FGSM and PGD areas. As expected, the FGSM
lies above the PGD curve corresponding to the same model all points.
Error bars indicate the standard eтror over three random seeds.
29

---

Under review as a conference paper at ICLR 2020
Un Cron
4.
BN Clean
+
Un RGSM
BN FGSM
Us Train
RN Thin
60
100
Accuracy (Test set)
Accuracy (Train set)
Accuracy (Tust
80
80
H
70
80-
60
40
20
60
80
100
120
20
40
80
100
120
20
60
100
120
Epochs
(a)
Un AWGN
BN AWGG
60
Un PGD
BN PGD
100
90
Aocuracy (Tost sct)
Accuracy (Train sot.)
Avcuray (Tost
80
50
80
70
40
60-
60-
40
40
100
120
Epochs
(b)
Figure 17: To isolate the effect of annealing the learning rate for CIFAR-10, we repeat the experiment of Figure [6] only this time using large learning rate (0.1) that is fixed during training. In (a) we have our first negative result: the accuracy of the BN variant is either compatible, or slightly higher than that of the unnormalized variant. We discuss the implications of this result in the text. In (b) we double the weight decay penalty to 1e-3. For the sake of completeness, we do this for both BN and no-BN cases, even though weight decay seems to have a different mechanism when combined with BN, to increase the effective learning rate (van Laarhoven| 2017 Zhang et al., (019a). Thus, increasing the weight decay leads to instability in the BN case, as the learning rate is already large. The relevant comparison is therefore between the unnormalized curves of (b)] and the BN curves (a), in which the higher weight decay unnormalized model is competitive with, and slightly outperforms by epoch 130, the better performing Error bars indicate the standard error over three random seeds.
model in terms of robustness.
30


Reviews:
[{"review_id": "rJxofaHa_r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper identifies an important weakness of batch normalization: it increases adversarial vulnerability. It is very well written and the claims are theoretically sound. In the experiments, the authors demonstrated a significant difference in robustness between networks with or without batch normalization layers, in varies settings against both random input noise and adversarial noise. This weakness of batch norm was explained due to the \"decision boundary tilting\" effect caused by the normalization. Overall, this paper has done solid work to reveal an interesting phenomenon. If it is true, this finding will impact almost all DNN models. \n\nMy concern is that this phenomenon is just another effect of \"gradient masking \" (as pointed out by Athalye, et al.). Batch norm is a well-known technique to avoid overfitting, without batch norm the network can be easily trained to be saturated with almost zero gradients, demonstrating a false signal of \"robustness\" to noise. The random noise and real-world corruption experiments are definitely helpful to clear this doubt, but only partially. My concern remains because of two obvious signs of  gradient masking: \n1. The accuracy on PGD-li (epsilon=0.031) attacks are suspiciously too high (20% - 40% Table 3/4). For this level of attack, the acc should be nearly zero. This is likely caused by the gradient masking effect, considering the cifar-10 networks were trained for longer time with larger learning rate (150 epochs, fixed lr 0.01). Training on MNIST is much easier to get zero gradients.  \n2. The weight decay discussion is not helpful at all, on the contrary, it confirms my concern on the gradient masking effect. In Table 8, the robustness was increased ~40% by just using large weight decay. This is not the \"real robustness\", and can be easily evaded by adaptive attack (see Athalye's paper).\nWith the above two concerns in mind, I doubt the phenomenon revealed in this paper is just \"one can easily train a saturated model without batch norm\" or equivalently \"it's hard to train a saturated model with batch norm\". It is hard to say if this is a bad thing for batch norm.\n\nI am quite surprised that the authors ignore this completely. Here are a few things that can be done to rule out the possibility of gradient masking. The masked gradient can be identified by: 1) One-step attacks perform better than iterative attacks; 2) Unbounded attacks do not reach 100% success., etc (see Section 3.1 of Athalye's paper).\n1. Including FGSM in the experiments and show the same trends as PGD-li. \n2. Show two networks have similar gradient norms.\n3. Apply cw-l2 attack, and show batch norm has forced large perturbation.\n\nTwo other suggestions:\n1. Summarize the different angles/steps taken to verify the phenomenon, somewhere before the experiments.\n2. Cannot see why the input dimension discussion contribute to explanations of the batch norm weakness.\n\n============\nMy rating stays the same after rebuttal. \n\nMy original concerns are like the other reviewers: why BN, not other techniques such as structure of DNNs MLP vs CNN vs ResNet, activation functions, weight decay, learning rates, softmax etc. My initial suspect was that it is caused by gradient masking likely caused by the l2 weight regularization, so asked the authors to look at the gradient norms and run some testes to rule this out. Yes, the weight norm is directly related to the Lipschitz continuity of the function represented by the network, but it often becomes more complicated on complex nonlinear neural networks. \n\nAccording to the new experiment results, the vulnerability is indeed not an effect of gradient masking, thanks for the clarification. However, the new results also indicate that the finding is susceptible to both weight decay and learning rate: in Figure 16 (a): \"Un PGD\" < \"BN PGD\" before learning rate decay, andFigure 17 (a) vs (b), doubling the weight decay penalty to 1e-3 also increases the vulnerability of BN. Overall, I believe the phenomenon exists, but the reasons behind requires more explanations, at least not just the batch norm.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJg7I5MztH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIn this empirical study, the authors identify that batch normalization -- a common technique for accelerating training -- leads to brittle representations that exhibit a lack of robustness and are more susceptible to adversarial attacks. The authors demonstrate their results on SVHN, CIFAR-10, CIFAR-100 CIFAR-10.1 using a variety of network architectures including VGG, BagNet, WideResNet, AlexNet, etc.\n\nMajor Concerns:\n\n1. As presented, the experiments are not convincing.\n\nI do not know how much of the changes in adversarial vulnerability are due to batch normalization as opposed to other facets of the training procedure that may have changed in their BN vs no-BN experiments.\n\nFor instance, batch normalization usually accommodates higher learning rates and it is not clear if the authors adjusted the initial learning rate, learning rate schedule or training schedule accordingly. If so, it would be important to run a set of experiments with these parameters fixed as per the baseline no-BN models. \n\nThat said, even if the authors did run these experiments, it is still not clear if the cause of adversarial vulnerability is due to BN. Consider that what is truly important in model training is not the learning rate (i.e. step size), but rather the magnitude of the changes in each weight (or the ratio of weight change to the weight). By swapping in batch normalization, the authors may just be altering the norm of the weight change in the (re-parameterized) weights. In this scenario, the gains of removing batch normalization could just as well be explained by the effective change in the learning rate, and not about batch normalization itself, c.f.\n  Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks\n  Yuanzhi Li, Colin Wei, Tengyu Ma\n  https://arxiv.org/abs/1907.04595\nIf the differences in the adversarial vulnerability could be ascribed to effective changes in gradient updates, then this would change the interpretation of these results notably.\n\n2. The underlying hypothesis is specious.\n\nI have several reservations about the underlying hypothesis that requires stronger evidence to overcome. In particular, I have reservations in believing that BN itself is a cause of adversarial vulnerability because BN is just a factorization of a network's weights. That is, there is nothing \"special\" nor unique about BN-networks; instead, the BN factorization merely permits accelerated training efficiency.\n\nConsider the fact that a BN model may be re-expressed by merely folding in the parameters (i.e. applying the matrix multiplications) into the MLP weights or CNN filters. Thus, the numerical function approximated by the BN and the \"folded\" non-BN model is identical. What would it mean to say that the BN is \"causing\" adversarial vulnerability in the BN model given that both the BN and non-BN model perform the identical function?\n\nAnother way to say this is to pretend we train a non-BN MLP or CNN model. After training the model, we could apply a BN factorization of the weights. Thus, the non-BN model may be factorized into a BN model. If the resulting BN model were adversarial vulnerable (which I suspect is the case), it would seem very hard to believe that BN was the cause of the vulnerability given it was a post-hoc factorization of the weights.\n\nThat said, I could definitely imagine that the training procedure itself could lead to adversarial vulnerability (e.g. citation above) and by employing a BN factorization, one may be encouraged to use a training procedure which leads to increased vulnerability. I would encourage the authors to consider this line of attack and thus, re-orient their analysis and discussion accordingly.\n\n3. The title is poorly worded.\n\nNot withstanding the point above, adversarial vulnerability predates BN. Likewise, non-BN models exhibit adversarial vulnerability. Thus, this title is not a great reflection of the findings of the paper. I would strongly suggest replacing \"is a cause\" with \"increases\" or \"exacerbates\".", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1lNkzJ2YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\nThis is an interesting work. The paper is dedicated to studying the effect of BN to network robustness. The author shows that BN can reduce network robustness to small adversarial input perturbations and common corruptions by double-digit percentages. Then, they use a linear \"toy model\" to explain the mechanism that the actual cause is the tilting of the decision boundary. Moreover, the author conducts extensive experiments on popular datasets to show the robustness margin with or without the BN module. Finally, the author finds that substituting weight decay for BN is good enough to nullify a relationship between adversarial vulnerability and the input resolution.\n\nStrength Bullets:\n1. I like the linear toy example. For that binary classification example, the author explicitly explains the boundary tilting, which increases the adversarial vulnerability of the model. It is clear.\n2. The paper conducts extensive experiment on SVHN, MNIST, CIFAR10 (C) datasets. And they show performance margin with or without the BN module. And for the attacker setting, they do use the popular setting (i.e. Mardy's PGD setting) in this field which makes the results more convincing.\n\nWeakness Bullets:\n1. Why do not visualize the decision boundary of networks (used in this work) to valid the boundary tilting \"theory\". The toy example is clear but not convincing enough. There exist several techniques may be helpful to the visualization. (i.e. Robustness via curvature regularization, and vice versa). I think it is one of the important parts of this work. The observation of BN causes adversarial vulnerability is interesting but the main focus should be offering more convincing explanations.\n2. I do run experiments for VGG11,13,16,19 on cifar10 with PGD 3 attack (Mardy's setting). There exist ~20 ATA performance gaps between networks with BN and BN networks without BN. But for adversarial trained models, the gaps don't exist anymore, at least for VGG11,13,16,19 on cifar10 with PGD 3 attack. (The performance gap is less than 0.5). In other words, without the BN layers, the robustness of adversarially trained models will not increase in my experiments. I see you report some results in Appendix C. But it is not enough to convince me. Could you provide more implementation details about the adversarial training and attacker? And more experiment results about this point are needed. If adversarial training can fix the vulnerability by BN and BN can give a TA boost, there is no reason we need to remove BN in our adversarial training setting. I see there are similar concerns in the OpenReview.\n3. [Minior] The experiments need to be organized better. Especially for section 3, it will be better to divide different experiments or observations into the different subsections.\n\n\nRecommendations:\nFor the above weakness bullets, this is a week reject.\n\nSuggestions:\n1. To solve the weakness bullets;\n2. minor suggestion: add the reference mention in the OpenReview, they are related to this work.\n\nQuestions:\n1. You mention that you run PGD for 20-40 iterations in the experiment at the bottom of page three. But at each table, you only report one number. So my question is for that accuracy number, you run how many iterations for PGD?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This article studies the effects of BN on robustness. The article presents a series of experiments on various datasets with noise, PGD adversarial attacks, and various corruption benchmarks, that show a drop in robustness when using BN. It is suggested that a main cause of vulnerability is the tiling angle of the decision boundary, which is illustrated in a toy example. 
The reviewers found the contribution interesting and that the effect will impact many DNNs. However, they the did not find the arguments for the tiling explanation convincing enough, and suggested more theory and experimental illustration of this explanation would be important. In the rebuttal the authors maintain that the main contribution is to link BN and adversarial vulnerability and consider their explanation reasonable. In the initial discussion the reviewers also mentioned that the experiments were not convincing enough and that the phenomenon could be an effect of gradient masking, and that more experiments with other attack strategies would be important to clarify this. In response, the revision included various experiments, including some with various initial learning schedules. The revision clarified some of these issues. However, the reviewers still found that the reason behind the effect requires more explanations. In summary, this article makes an important observation that is already generating a vivid discussion and will likely have an impact, but the reviewers were not convinced by the explanations provided for these observations. 

Decision: Reject