Batch Normalization is a Cause of Adversarial Vulnerability

Batch normalization ( BN ) is often used in an attempt to stabilize and accelerate training in deep neural networks . In many cases it indeed decreases the number of parameter updates required to achieve low training error . However , it also reduces robustness to small adversarial input perturbations and common corruptions by double-digit percentages , as we show on five standard datasets . Furthermore , we find that substituting weight decay for BN is sufficient to nullify a relationship between adversarial vulnerability and the input dimension . A recent mean-field analysis found that BN induces gradient explosion when used on multiple layers , but this does not fully explain the vulnerability we observe , given that it occurs for a single BN layer . We argue that the main cause is the tilting of the decision boundary with respect to the nearest-centroid classifier along input dimensions of low variance . As a result , the numerical stability constant of BN acts as an important meta-parameter that can be tuned to recover some robustness at the cost of standard test accuracy . We explain this mechanism explicitly for linear models but find that it still holds for nonlinear models . 1 INTRODUCTION . BN is a standard component of modern deep neural networks , and tends to make the training process less sensitive to the choice of hyperparameters in many cases ( Ioffe & Szegedy , 2015 ) . While ease of training is desirable for model developers , an important concern among stakeholders is that of model robustness during deployment to plausible , previously unseen inputs . The adversarial examples phenomenon has exposed unstable predictions across state-of-the-art models ( Szegedy et al. , 2014 ) . This has led to a variety of methods that aim to improve robustness , but doing so effectively remains a challenge ( Athalye et al. , 2018 ; Schott et al. , 2019 ; Hendrycks & Dietterich , 2019 ; Jacobsen et al. , 2019a ) . We believe that a prerequisite to developing methods that increase robustness is an understanding of factors that reduce it . Approaches for improving robustness often begin with existing neural network architectures—that use BN—and patch them against specific attacks , e.g. , through inclusion of adversarial examples during training ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Kurakin et al. , 2017 ; Madry et al. , 2018 ) . An implicit assumption is that BN itself does not reduce robustness , however , recent initialization-time analyses have shown that it causes exploding gradients , and increased sensitivity to input perturbations as the network depth increases ( Yang et al. , 2019 ; Labatie , 2019 ) . In this work , we consider the impact of BN in practical scenarios in terms of robustness to common corruptions ( Hendrycks & Dietterich , 2019 ) and adversarial examples ( Szegedy et al. , 2014 ) , finding that BN induced sensitivity remains a concern even in cases where its use appears benign on the basis of clean test accuracy , and when only one BN layer is used . The frequently made observation that adversarial vulnerability can scale with the input dimension ( Goodfellow et al. , 2015 ; Gilmer et al. , 2018 ; Simon-Gabriel et al. , 2019 ) highlights the importance of identifying regularizers as more than merely a way to improve test accuracy . In particular , BN was a confounding factor in Simon-Gabriel et al . ( 2019 ) , making the results of their initialization-time analysis hold after training . By adding ` 2 regularization and removing BN , we show that there is no inherent relationship between adversarial vulnerability and the input dimension . 2 BATCH NORMALIZATION . We briefly review how BN modifies the hidden layers ’ pre-activations h of a neural network . We use the notation of Yang et al . ( 2019 ) , where α is an index for units in a layer l , and i for a mini-batch of B samples from the dataset ; Nl denotes the number of units in layer l , W l is the matrix of weights and bl is the vector of biases that parametrize layer l. The batch mean is defined as µα = 1B ∑ i hαi , and the variance is σ2α = 1 B ∑ i ( hαi − µα ) 2 . In the BN procedure , the mean µα is subtracted from the pre-activation of each unit hlαi ( consistent with Ioffe & Szegedy ( 2015 ) ) , the result is divided by the standard deviation σα plus a small constant c to prevent division by zero , then scaled and shifted by the learned parameters γα and βα , respectively . This is described in equation 1 , where a per-unit nonlinearity φ , e.g. , ReLU , is applied after the normalization . hli = W lφ ( h̃l−1i ) + b l , h̃lαi = γα hαi − µα√ σ2α + c + βα . ( 1 ) This procedure introduces complications , however . Consider two mini-batches that differ by only a single example : due to the induced batch-wise nonlinearity , they will have different representations of all examples . These differences are amplified by stacking BN layers , and were shown to cause exploding gradients at initialization ( Yang et al. , 2019 ) . Conversely , normalization of intermediate representations for two different training inputs impairs the ability to distinguish definite examples that ought to be classified with a large prediction margin ( as judged by an “ oracle ” ) , from more ambiguous instances . The last layer of a discriminative neural network , in particular , is typically a linear decoding of class label-homogeneous clusters , and thus makes use of information contained in the mean and variance at this stage for classification . In light of these observations , we begin in our analysis by adding a single BN layer to models trained by gradient descent ( GD ) . This is the most favorable scenario according to the analysis of Yang et al . ( 2019 ) , where more layers and a smaller mini-batch size exacerbate the exploding gradients . 3 BOUNDARY TILTING . Tanay & Griffin ( 2016 ) relate the adversarial vulnerability of linear classifiers to the tilting angle θ of the decision boundary w.r.t . the nearest-centroid classifier . Following their setup , we examine how BN affects this angle in a simple linear model , and then show that increasing model complexity can not “ undo ” this vulnerability . Consider the binary classification task of identifying two different types of input x subject to Gaussian noise with a linear classifier w > x+ b . This can be modeled by the class-conditional distribution p ( x|y = j ) = N ( νj , Σ ) with label y ∼ Ber ( 0.5 ) . The Bayes-optimal solution to this problem is given by the weight vector w = Σ−1 ( ν0 − ν1 ) , and b = 12 ( ν 1 + ν0 ) > Σ−1 ( ν1 − ν0 ) + log p ( y=0 ) p ( y=1 ) , where p ( y ) denotes the marginal probability for the label y ( see e.g . ( Jordan , 1995 ) ) , while the nearestcentroid classifier is defined by w∗ = ν0 − ν1 . We analyze the effect of batch-normalizing the input to the classifier for this problem ( i.e. , hαi = xαi ) , first in the simplest setting where γα = 1 , βα = 0 ∀α . We select the class distribution means νj to be symmetric around zero , so that the batch mean computed by BN is µα = 0 ∀α . The batch-normalized linear classifier is thus defined as : f ( x ) = w > x+b√ σ2+c . By construction of our synthetic dataset , the variance of the batch can be deduced from the data parameters : σ2α = ( ν j α ) 2 + Σαα . The tilting angle θ of the batch-normalized decision boundary w.r.t . the one given by w∗ ( note that the boundary is perpendicular to w ) is therefore approximately equal to the angle between the datasets before and after normalization . To compute θ , we divide the weights w by √ σ2 + c , and then normalize w/‖w‖2 , such that θ = cos−1 ( w > w∗ ) . From this analysis it follows that the order of magnitude of c is important relative to the data variance : if c > σ2α then the effective weight value wα is reduced , and if c < σ2α and σ 2 α is small , then wα increases greatly , causing boundary tilting along direction α . We depict simulations of the toy model in Figure 1 . We use constant learning rate GD , which is known to converge to the max-margin solution—equivalent to the nearest centroid classifier in this case–for linear models on separable data ( Soudry et al. , 2018 ) . Batch-normalized GD ( BNGD ) converges for arbitrary learning rates for linear models ( Cai et al. , 2019 ) ; we use a value of 0.1 for 1000 epochs . Next , we train linear models on the MNIST 3 vs. 7 dataset with 5000 training samples ( drawn uniformly per class ) using a learning rate of 0.1 for 50 epochs . We compute the angle θ w.r.t . the nearest-centroid classifier , which is obtained by subtracting the “ average 3 ” from the “ average 7 ” of the full training set . Although this may seem like a crude reference point , the nearest-centroid classifier is much more robust than the linear model of Goodfellow et al . ( 2015 ) , achieving 40 % accuracy for the fast gradient sign method ( FGSM ) at = 1/4 vs. ≈ 0 % . Results consistent with the boundary tilting theory are shown in Table 1 , which not only shows that BN causes tilting , but that this is unaffected by the parameters γ and β. Post-normalization , there is no signal to γ and β about the variances of the original dataset . This is consistent with other works that observe γ and β do not influence the studied effect ( van Laarhoven , 2017 ; Zhang et al. , 2019a ; Yang et al. , 2019 ) Increasing the numerical stability constant c increases robustness in terms of absolute test accuracy for additive white Gaussian noise ( AWGN ) on MNIST and CIFAR-10 datasets by 33 % and 41 % respectively ( at the cost of standard accuracy ) . We defer the details of this experiment to Appendix A . This loss of accuracy , and the effect of c are consistent with Labatie ( 2019 ) , which remarks that under BN , directions of high signal variance are dampened , while directions of low signal variance are amplified . This preferential exploration of low signal directions reduces the signal-to-noise ratio and increases sensitivity w.r.t . the input . Increasing c reduces the sensitivity along “ low signal directions ” . 4 EMPIRICAL RESULTS . For the main practical results on MNIST , SVHN , CIFAR-10 , and ImageNet , we evaluate the robustness—measured as a drop in test accuracy under various input perturbations—of convolutional networks with and without BN.1 As a white-box adversarial attack we use projected gradient descent ( PGD ) in ` ∞- and ` 2-norm variants , for its simplicity and ability to degrade performance with little change to the input ( Madry et al. , 2018 ) . The PGD implementation details are provided in Appendix B . We report the test 1Unless stated otherwise , we leave the internal parameters of BN to their default values as in modern deep learning frameworks , c =1e-5 with γ and β enabled . accuracy for additive Gaussian noise of zero mean and variance 1/16 , denoted as “ Noise ” , as well as using the CIFAR-10-C common corruption benchmark ( Hendrycks & Dietterich , 2019 ) . We found these methods were sufficient to demonstrate a considerable disparity in robustness due to BN , but this is not intended as a complete security evaluation . Standard meta-parameters from the literature were used to train models with and without BN from scratch over several random seeds . All uncertainties are the standard error of the mean accuracy.2 For SVHN and CIFAR-10 , we examine two different learning rate schemes , given that it has been suggested that one of the primary mechanisms of BN is to facilitate training with a larger learning rate ( Ioffe & Szegedy , 2015 ; Bjorck et al. , 2018 ) : 1 . A fixed “ small ” learning rate of 1e-2 ( SVHN , CIFAR-10 ) . 2 . An initial “ large ” learning rate of 1e-1 , with subsequent drops by a factor of ten ( CIFAR-10 ) . In the SVHN experiment , VGG variants ( Simonyan & Zisserman , 2015 ) are trained using using five random seeds . BN increased clean test accuracy by 1.86 ± 0.05 % , but reduced test accuracy for additive noise by 5.5± 0.6 % , for PGD- ` ∞ by 17± 1 % , and for PGD- ` 2 by 20± 1 % . We defer the full meta-parameters and results to Appendix E. For the CIFAR-10 experiments we trained models with a similar procedure as for SVHN , but with random 32 × 32 crops using four-pixel padding , and horizontal flips . We evaluate two families of contemporary models : one without skip connections ( VGG ) and a WideResNets ( WRN ) using “ Fixup ” initialization ( Zhang et al. , 2019b ) to reduce the use of BN . Results of the first experiment are summarized in Table 2 . In this case , inclusion of BN for VGG reduces the clean generalization gap ( difference between training and test accuracy ) by 1.1± 0.2 % . For additive noise , test accuracy drops by 6± 1 % , and for PGD perturbations by 17.3± 0.7 % and 5.9± 0.4 % for ` ∞ and ` 2 variants , respectively . Table 3 : ( Large initial learning rate ) Robustness of VGG models of increasing depth on CIFAR-10 , with and without BN . See text for meta-parameters . Model Test Accuracy ( % ) L BN Clean Noise PGD- ` ∞ 8 7 89.29± 0.09 81.7± 0.3 55.6± 0.4 8 3 90.49± 0.01 77± 1 40.6± 0.6 13 7 91.74± 0.02 77.8± 0.7 40.3± 0.7 13 3 93.0± 0.1 67± 1 28.5± 0.4 16 3 92.8± 0.1 66± 2 28.9± 0.2 19 3 92.65± 0.09 68± 2 30.0± 0.1 Very similar results are obtained on a new test set , CIFAR-10.1 v6 ( Recht et al. , 2018 ) : BN slightly improves the clean test accuracy ( by 2.0 ± 0.3 % ) , but leads to a considerable drop in test accuracy of 6± 1 % for the case with additive noise , and 15 ± 1 % and 3.4± 0.6 % respectively for ` ∞ and ` 2 PGD variants ( PGD absolute values omitted for CIFAR-10.1 in Table 2 for brevity ) . In the second “ large ” learning rate experiment summarize in Table 3 , we prolong training for up to 350 epochs , and drop the learning rate at epoch 150 and 250 in both cases . This increases clean test accuracy relative to results in Table 2 . The deepest model that could be trained without BN using this 2Each experiment has a unique uncertainty , hence the number of decimal places varies . procedure and He et al . ( 2015 ) initialization was VGG13 . 3 None of the deeper batch-normalized models recover the robustness of the most shallow , or same-depth equivalents , nor does the higher learning rate recover the performance of the baselines . Results for deeper models are in Appendix E. Other learning rate schedules , and robustness vs. training epoch curves are in Appendix K. Next , we evaluate robustness on the common corruption benchmark comprising 19 types of realworld distortions from four high-level categories : “ noise ” , “ blur ” , “ weather ” , and “ digital ” effects ( Hendrycks & Dietterich , 2019 ) . Each corruption has five “ severity ” or intensity levels . We report the mean error on the corrupted test set ( mCE ) by averaging over all intensity levels and corruptions . We summarize the results for two VGG variants and a WideResNet on CIFAR-10-C , trained from scratch on the default training set for three and five random seeds , respectively . Accuracy for the “ noise ” corruptions , causing the largest difference in accuracy with BN , are outlined in Table 4 . The key takeaway is : For all models tested , the batch-normalized variant has a higher error rate for all corruptions of the “ noise ” category , at every intensity level . Table 5 : Robustness of pre-trained ImageNet models with and without BN . Note : The numeric suffix indicates number of layers , or the spatial patch width in pixels ( of 224 ) for BagNet . Top 5 Test Accuracy ( % ) Model BN Clean Noise PGD- ` ∞ VGG11 7 88.63 49.16 37.12 VGG11 3 89.81 49.95 26.12 VGG19 7 90.88 64.86 34.19 VGG19 3 91.84 68.79 24.49 AlexNet 7 79.07 41.41 39.12 ResNet18 3 88.65 79.62 31.07 BagNet-9 3 70.39 1.25 7.42 BagNet-17 3 81.16 5.09 16.66 BagNet-33 3 86.99 14.62 24.34 Averaging over all 19 corruptions , BN increases mCE by 1.9± 0.9 % for VGG8 , 2.0± 0.3 % for VGG13 , and 1.6 ± 0.4 % for WRN . There is a large disparity in accuracy when modulating BN for different corruption categories , we examine these in more detail in Appendix G. Interestingly , some corruptions that led to a positive gap for VGG8 show a negative gap for the WRN , i.e. , BN improved accuracy to : Contrast— 4.9±1.1 % , Snow—2.8±0.4 % , Spatter—2.3± 0.8 % . These are the same corruptions for which VGG13 loses , or does not improve its robustness when BN is removed . We suspect accuracy for these corruptions correlates with standard test accuracy , which is highest for the WRN . Visually , these corruptions appear to preserve texture information . Conversely , noise is applied in a spatially global way that disproportionately degrades these textures , emphasizing shapes and edges . It is now known that modern CNNs trained on standard image datasets have a propensity to rely heavily on texture in addition to shape and edge cues for object recognition ( Geirhos et al. , 2019 ) . We evaluate pre- 3For which one of ten random seeds failed to achieve better than chance accuracy on the training set , while others performed as expected . We report the first three successful runs for consistency with the other experiments . Clean Acc . Noise Acc . BIM Acc . trained bag-of-local-feature models ( BagNets ) on ImageNet with an architecture that discards spatial information between patches and is thus considered to make extensive use of texture patterns for classification ( Brendel & Bethge , 2019 ) . For patch sizes { 9 , 17 , 33 } , the top-5 accuracies of the BagNets are reduced to just 1.25 % , 5.09 % , and 14.62 % for AWGN , respectively . Compared with Table 5 , where all models obtain over 40 % , these figures suggest that robustness to Gaussian noise is a good proxy for the use of texture for ImageNet classification . These results support the hypothesis that BN may be exacerbating this tendency to use superficial texture-like information . Next , we evaluate the robustness of pre-trained ImageNet models from the torchvision.models repository , which conveniently provides models with and without BN.4 Results are shown in Table 5 , where BN improves top-5 accuracy on noise in some cases , but consistently reduces it by 8.54 % to 11.00 % ( absolute ) for PGD . The trends are the same for top-1 accuracy , only the absolute values are smaller ; the degradation varies from 2.38 % to 4.17 % . Given the discrepancy between noise and PGD for ImageNet , we include a black-box transfer analysis in the Appendix E.2 that is consistent with the white-box analysis . Finally , we explore the role of batch size and depth in Figure 2 . We find that BN limits the maximum trainable depth , which increases with the batch size , but quickly plateaus as predicted by Theorem 3.10 of ( Yang et al. , 2019 ) . Robustness decreases with the batch size for depths that maintain a reasonable test accuracy , at around 25 or fewer layers . This tension between clean accuracy and robustness as a function of the batch size is not observed in unnormalized networks . 4https : //pytorch.org/docs/stable/torchvision/models.html , v1.1.0 .

Summary: This is an interesting work. The paper is dedicated to studying the effect of BN to network robustness. The author shows that BN can reduce network robustness to small adversarial input perturbations and common corruptions by double-digit percentages. Then, they use a linear "toy model" to explain the mechanism that the actual cause is the tilting of the decision boundary. Moreover, the author conducts extensive experiments on popular datasets to show the robustness margin with or without the BN module. Finally, the author finds that substituting weight decay for BN is good enough to nullify a relationship between adversarial vulnerability and the input resolution.