Towards an Adversarially Robust Normalization Approach
Authors: Muhammad Awais
Fahad Shamshad
Sung-Ho Bae
Venue:  (2020)
Keywords: robustness
BatchNorm
adversarial

Abstract: Batch Normalization (BatchNorm) has shown to be effective for improving and accelerating the training of deep neural networks. However, recently it has been shown that it is also vulnerable to adversarial perturbations. In this work, we aim to investigate the cause of adversarial vulnerability of the BatchNorm. We hypothesize that the use of different normalization statistics during training and inference (mini-batch statistics for training and moving average of these values at inference) is the main cause of this adversarial vulnerability in the BatchNorm layer. We empirically proved this by experiments on various neural network architectures and datasets. Furthermore, we introduce Robust Normalization (RobustNorm) and experimentally show that it is not only resilient to adversarial perturbation but also inherit the benefits of BatchNorm.

Under review conference paper at ICLR 2020
TOWARDS AN ADVERSARIALLY ROBUST NORMALIZATION APPROACH
Anonymous authors Paper under double-bling review
ABSTRACT
Batch Normalization (BatchNorm) has shown to be effective for improving and accelerat- ing the training of deep neural networks. However, recently it has been shown that is also vulnerable to adversarial perturbations. In this work, aim to investigate the cause of adversarial vulnerability of the BatchNorm. We hypothesize that the use of different nor- malization statisties during training and inference (mini-batch statistics for training and moving average of these values at inference) is the main cause of this adversarial vulner- ability the BatchNorm layer. We empirically proved this by experiments on various neural network architectures and datasets. Furthermore, we introduce Robust Normal- ization (RobustNorm) and experimentally show that is not only resilient to adversarial perturbation but also inherit the benefits of BatchNorm.
1 INTRODUCTION
spite of their impressive performance on challenging tasks in computer vision such as image classification and semantic segmentation, deep neural networks (DNNs) are shown to be highly vulnerable to adversarial
examples, i.e. carefully crafted samples which look similar to natural images but designed to mislead a trained neural network model (Goodfellow al., 2014 Nguyen et al.) 2015] Carlini & Wagner (2017). Designing defense mechanisms against these adversarial perturbations has been subjected to much research recently (Xie et al. 2019 Madry al. 2017 Tramèr al. 2017 Papernot et al. 2016).
Meanwhile, Batch Normalization (BatchNorm or BN) (loffe & Szegedy 2015) 2015)) has successfully proliferated
throughout all areas of deep learning as it enables stable training, higher learning rates, faster convergence, and higher generalization accuracy. Initially, the effectiveness of the BatchNorm has been attributed to its ability eliminate the internal covariate shift (ICS), the tendency of the distribution of activations to drift during training. However, later on, alternative reasons including avoiding exploding activations, smooth loss landscape, reducing the sensitivity to initialization, etc. have also been proposed as the basis of BatchNorm's 'ss
success (annurrar et al. 2018 Bjorck et al. 2018 Luo et al. 2018).
While there exist a plethora of reasons for the adversarial vulnerability of deep neural networks (Jacossen et al.) 2018 -Simon-Gabriel Gabriel et al., 2018), recent study by (Galloway et al. (2019) showed that BatchNorm is one of the reasons for this vulnerability. Specifically, they empirically showed that removing the BatchNorm layer enhances robustness against adversarial perturbations. However, removal of BatchNorm also means a sacrifice of benefits such as the use of higher learning rates, faster convergence, and significant improvement in the clean test set accuracy among many others.
this paper, we propose a new perspective regarding the adversarial vulnerability of the BatchNorm layer. Specifically, we probe why BatchNorm layer causes the adversarial vulnerability. We hypothesize that the use of different normalization statistics during training and inference phase (mini-batch statistics for training and moving average of these statistics also called tracking, inference time) is the cause of this adversarial

---

We consider a standard classification task for data, having underlying distribution denoted as D, over the pair of examples R" and corresponding true labels y 21, k) where k represents different labels. We denote deep neural network (DNN) as function Fa(æ), where 0 denotes trainable parameters. 0 is learned by minimizing loss function L(x, y) with training data x, y. The output of the DNN is a feature representation f E R4, that we give input to a classifier C R + {1, k]. The objective of the adversary is add the additive perturbation 5 E R" under the constrain that the generated adversarial Under review conference paper at ICLR 2020
vulnerability of the BatchNorm layer. Our experiments show that by removing this part of the BatchNorm, the robustness of the network increases by 20%. Similarly, robustness can further be enhanced by up to 30% after adversarial training. However, by removing the tracking part, the test accuracy on the clean images drops significantly (though better than without normalization). To circumvent this issue, we propose Robust Normalization (RobustNorm or RN). Our experiments demonstrate that RobustNorm not only significantly improve the test performance of adversarially-traine DNNs but is also able to achieve the comparable test accuracy to that of BatchNorm on unperturbed datasets. We perform numerical experiments over standard datasets and DNN architectures. In almost all of our experiments, we obtain better adversarial robustness performance on perturbed examples for training with natural as well as adversarial training.
2 BACKGROUND
sample Ladv = x + 5 that looks visually similar to the true image æ, and for which the corresponding labels are not same i.e. C(x) F C( In this work, we have added the perturbation via following well-known adversarial attack approaches.
Fast Gradient Sign Method: Given an input image æ along with its corresponding true label y. FGSM Goodfellow et al. (2014) aims to generate the adversarial image Fods as,
Tads = x: + 6 sign( C(r, v);
where is the perturbation budget that is chosen to be sufficiently small. We use two of its variants: Gradient (Grad) where graidents are used and Gradient sign (GradSign) which is similar [
Basic Iterative Method (BIM): BIM (Kurakin et al. 2016 is a straight forward extension of FGSM, that applies it multiple times with smaller step size. Specifically,
x
adv x,
Fddo = ( + a sign( V Co adv . .)))
(2)
where s0 aldo is the clean image and N denotes iteration number.
Carlini-Wagner attack (CW); CW an effective optimization-base attack model introduced by Carlini & Wagner (2017). It works by definining an auxilary variable 0 and minimizes the following objective functions
1 2 (tanh( ()) 1) æ|| flf(, (tanh( +0)++10 1)),
min|
(3)
where } (tanh(0) (0) + +1 - x is the perturbation ó, c is a scalar constant, and f(.) is defined as:
f( = max E (X:nd)y - max { 2 (x )a k / -)))
(4)
Here, @ is to control the adversarial sample's confidence and 2 are the logits values for class k.
Projected Gradient Descent (PGD): PGD perturbs the true image a for total number of N steps with smaller step sizes. After each step of perturbation, PGD projects the adversarial example back onto the e-ball of normal image if it goes beyond the e-bal.. Specifically,
N. ado
[(x:N-1 + aisign V C(x ado ,1)))
(5)
2

---

Under review conference paper at ICLR 2020
where 11 is the projection operator, a is step size, and x denotes adversarial example the N-th step. We have used for norm as distance measure. Gaussian Noise For comparison purposes, we also have used Gaussian noise with mean and 0.25 variance.
2.0.1 ADVERSARIAL TRAINING:
It has been shown that empirical risk minimization using only clean images for training can decrease the robustness performance of DNNs. A standard approach to achieve the adversarial robustness in DNNs is adversarial training which involves fitting a classifier C' on adversarially--ertuubbi images. We have used PGD based adversarial training which has shown to be effective against many first- order adversaries (Madry al. 2017) unlike other methods which overfit for single attack.
samples along with clean
2.1 EXPERIMENTAL SETUP
We have used two network architectures, Resnet He al. (2016) with 20,38 and 50 layers and VGG Si- monyan & Zisserman (2014) with 11 and 16 layers. We have used CIFAR10 10 and 100 datasets (Krizhevsky et al. 2009) for all the evaluations. We have used natural training for training with clean images while adversarial training is done with PGD based method formulated by Madry et al. (017). We have always used a learning rate of 0.1 except for no normalization scenarios where convergence is not possible with higher learning rates. In such cases, we have used a learning rate of 0.001. We decrease the learning rate 10 times after 120 epochs and trained all the networks for 164 epochs. For robustness evaluations, we have used e=003/1 for most of the experiments and used 20 epochs for all the iterative attacks. We also have tested the model on different noise levels ranging from 0.003/1 to 0.9/1.
3 HOW DOES BATCHNORM CAUSES ADVERSARIAL VULNERABILITY
3.1 How BATCHNORM WORKS
In this section, we briefly explain the working principle of BatchNorm layer. Broadly speaking, the Batch- Norm layer has been introduced to circumvent the issue of internal covariate shift (ICS). Consider a mini- batch B of size M, containing samples amm for m 1, 2, ... M. BatchNorm normalizes the mini-batch during training, by calculating the mean My and variance o as follows:
M
M
#B
OB
(as
mB) +
(6)
M
M
1=1
Activation normalization is then performed as,
3;
(7)
08
To further compensate for the possible loss of representational ability of network, BatchNorm also learns per-channel linear transformation as:
y ッッ + B,
for trainable parameters y and 3 that represent scale and shift, respectively. These parameters are learnt using the same procedure, such as stochastic gradient descent, as other weights in the network. During training, the model usually maintains the moving averages of mini-batch means and variances(a.k.. k.a. tracking), and during inference, uses these tracked statistics in place of the mini-batch statistics. Formally, tracking (moving average) of mean and variance, for scalar given as,
3

---

Under review conference paper at ICLR 2020
60
mini-batch mean before training
mini-batch variance before training
tracked mean after training
tracked variance after training
-5
-10
0
300
500
700
400
Channel Number
Channel Number
Figure 1: Comparison of mini-batch mean and variance the start of the training with tracked mean and variance which are used at inference for all the channels. Note the difference between both of these values.
A= (1 - T) T/3;
(1- + To
(9)
For inference, can write,
Fisst
ylest
2 +
(10)
3.2 BATCHNORM AND ADVERSARIAL VULNERABILITY
Recently, Galloway al. 2019 empirically showed that the accelerated training properties and occasion- ally higher clean test accuracy of employing BatchNorm in network come at the cost of low robustness adversarial perturbations. While removing the BatchNorm layer may be helpful for robustness, also means the loss of desirable properties of BatchNorm like very high learning rate, faster convergence, boost in test accuracy, etc. Therefore, is pertinent devise a normalization method that is not only robust the adversarial perturbations but also inherits the desirable properties of the BatchNorm layer.
3.3 DEVIL IS IN THE TRACKING
this section, we aim to investigate the reasons behind the adversarial vulnerability of the BatchNorm layer on the following two grounds;
We note that during training, mini-batch statistics are used to normalize activations as shown in Equations 6 and 7 Moving average of these statistics also calculated during the training that is called tracking (shown in Equation 99). The tracked mean and variance are used in the inference step (Equatton TOP. In this way, different values for mean and variance used during training and inference. We show this in Figure | where is clear that batch statistics at the start of the training are very different from tracked values that used the inference.
Our second observation is based on the recent work of (Ding et al.) 2019; Jacobsen et al. 2019 These works shed light on the link of the distributional shift in input data and robustness. Specif- ically, Ding al. (2019) showed that adversarial robustness is highly sensitive change in the input data distribution and prove that even a semantically-losse lossless shift on the data distribution could result in drastically different robustness for adversarially trained models.
Based on the above two observations, we hypothesize that: Different values of first-order statistics (mean and variance) are used in the normalization layer for training and inference. This means different internal
4

---

Under review conference paper at ICLR 2020
Norm
Clean
Noise
Gradient Sign
BIM-
PGD-f
BatchNorm
92.05 +: 0.34
59.16:16.74
33.75 = 0.99
25.43: 2.37
29.73 40.86
Natural
Training
No Norm
82.93 + 0.64
66.96:14.03
41.81 = 2.20
41.47:22.16
37.01=2.04
BatchNorm w/o Tracking
89.23= 1.43
77.65 + 2.45
53.32 0.71
48.38 + 1.40
48,54 0.91
BatchNorm
79.53 +: 9.16
60.38110.41
49.91 = 6.30
47,90 - 5.41
46.75
6.05
Adversarial
Training
No Norm
83.60 + 1.08
73.75
3.95
63.46 + 1.61
58,29 + 2.53
60.01 1.54
BatchNorm w/o Tracking
89.28 + 1.58
84.36 1.06
72.42 + 1.04
68.60
69.30 1.77
Table 2: Comparison of adversarial robustness of BatchNorm with BatchNorm without tracking and no normalization for CIFAR10 100 and Resner20. The results are shown with 95% confidence interval calculated with 5 random restarts. Highlighted values show the best accuracy for that particular adversarial noise. While BatchNorm without Tracking has significantly higher robust accuracy compare to both BatchNorm and no norm, clean accuracy lower than BatchNorm. But BatchNorm w/o Tracking also retains its clean accuracy when adversarially trained. This figure confirms our tracking-robustness hypothesis.
representations being used at training and inference time which causes drift in input distributions of these layers. Therefore. the tracking part is the main culprit behind the adversarial vulnerability of the BatchNorm layer.
To prove our hypothesis, we have done extensive experiments. For each experiment, we train a neural network model with three different normalization layers: BatchNorm, BatchNorm without tracking, and no normalization. To prove the generality of our argument, we have used various architectures, depths, and datasets written in section 2.1 We train these networks clean images as well as with based adversarial training procedure. For adversarial training, we use PGD attack for perturbation. We choose PGD due to its ability generalize well for other adversarial attacks.
T shows our results on Resnet20. For detailed experimental results on various architectures, depths and dataset with different attacks see Table ij appendix. The results clearly show that while the elimination of BatchNorm and training at a very small learning rate can help increase robustness, it also reduces clean data accuracy (with BatchNorm). More importantly, this proves our hypothesis that by removing tracking, we can increase the robustness of neural network significantly. By using BatchNorm without tracking, we also keep many benefits of BatchNorm. Unfortunately, by eliminating the tracking part of BatchNorm, clean accuracy of network also reduces compared to clean accuracy with BatchNorm. We tackle this issue in the next section.
4 ROBUST NORMALIZATION
Although alleviation ICS was claimed reason for the success of BatchNorm, recently, Bjorck et al. (2018) have shown that BatchNorm works because it avoids activation explosion by repeatedly correcting all acti- vations. For this reason, it is possible to train networks with large learning rates, as activations cannot grow uncontrollably and convergence becomes easier. On different side, recent work on robustness has shown a connection between the removal of outliers in activations and robustness (Xie et al., 2019] Etmann et al.) 2019). Based on these observations, we use min-max rescaling that often employed in preprocessing. This is also useful in the climination of outliers since it rescales the data to specific scale, Minmax normalization is defined as;
2; min (I;) y: max(x,,) min(x.)
(II)
5

---

Under review conference paper at ICLR 2020
Natural Training
Adversarial Training
CIFARIO
CIFARIO0
CIFARIO
CIFAR100
BatchNorm with tracking
92.11 0.40
68.20 = 0.8
79.53 = 9.16
35.39 3.91
BatchNorm w/o tracking
89.07 1.73
62.28 0.79
89.28 = 1.58
61.73 = 4.17
RobustNorm with tracking
91.97 0.30
68.13
0.28
85.15 +5.37
42.19 6.10
RobustNorm w/o tracking
91.49 0.28
67.41 0.66
90.76 10.63
64.95 = 0.72
Table Comparison of clean aceuracy of BatchNNrm with RobustNorm for both adversarial and natural training scenarios. RobustNorm's 's accuracy is better than BatchNorm when tracking is not used while its accuracy same when tracking is used.
However, experimentally found this layer to be less effective in terms of convergence. Considering the importance of mean (Salimans & Kingma 20166,, we modify this to;
D; /
y: max( min(x,))
(12)
We empirically observe the effectiveness of Equation 12 over Equation IT but the overall performance was still inadequate. During debugging, we found that Equation suppress activations much stronger than BatchNorm. This can also seen from Popoviciu's inequality (Popoviciu 1935).
- < (max(æ.) - min(x.))²
(13)
Following Popoviciu's inequality, we introduce the hyperparameter < < 1, that reduces the denominator in Equation [2
y
(14)
(max (x;) min(x;))
We experimentally found that p = 0.2 value generalizes well for many networks as well as datasets. We call this normalization Robust Normalization (RobustNorm or RN) due its robustness properties. We do not use tracking for the RobustNorm. But for comparison purposes, we keep running average both mean and denominator and use this running average during inference and call this normalization RobustNorm with tracking.
Table shows the accuracy of RobustNorm on Resnet20 for clean well adversarial training with both CIFAR10 and CIFARIO0 100 datasets with 95% confidence interval calculated over random restarts. These results show better clean accuracy of RobustNorm both natural and adversarial training scenarios. Apart from this, RobustNorm with tracking also shows better performance compared to BatchNorm with tracking. For adversarial robustness, we have shown Figure with different attacks on CIFAR 100 dataset. From the Table [ and Figure 2 it is clear that RobustNorm keeps its clean accuracy while being more robust. For more results on Resnet38, Resnet50, VGGII, and VGG16 with CIFRAR 10 and CIFARION 100 datasets and both natural and adversarial training and many attack methods, please have look at Table 4 and 5
F the evolution of validation loss and accuracy for PGD based adversarial training with con- fidence interval of 95% on Resnet20 architecture and CIFAR 100 dataset. From Figure in the appendix, it can also be seen that the evolution training loss and accuracy is normal. But validation loss and accuracy for normalizations with tracking much different for different random restarts. This can probably be plained based on flat and sharp minima attained by different normalizations as can be seen in loss landscape in Figure 9 in the appendix. For further discussion the loss landscape, please section in appendix.
6

---

Under review conference paper at ICLR 2020
Natural Training
Adversarial Training
70
70
80
50
40
30
20
RN
BN w/o Tracking
10
BN
Normal
GradSig
BIM--
PGD-22
PGD--
Noise
Gmad
GradSig
BIM--
PGD42
PGD-
Figure 2: Comparison of adversarial robustness of different normalizations for different Whitebox attacks with CIFARIOS 100. Results are shown with 95% confidence interval computed over 5 random restarts. Batch- Norm w/o Tracking and RobustNorm have significantly higher adversarial robustness compared to other norms. The results are even more clear when they are adversarially trained where RobustNorm's 's robustness is even more than BatchNorm w/o Tracking.
BN w/o Tracking
6
BN
50
5
Tracking
40
May
30
w/o Tracking
3
20
BN
2
10
RN
RN with Tracking
100
120
140
160
40
60
80
100
120
140
160
Adv. Trained, Val Loss
Adv. Traied, Val Acc.
Figure 3: Comparison of evolution of validation loss and accuracy for CIFARI00 on Resnet20 with confi- dence intervals calculated with 5 random restarts. BatchNorm without tracking and RobustNorm have higher accuracy and lower loss while RobustNorm being better than BatchNorm w/o tracking. For further details, please have look at appendix B
4.1 RESISTANCE FOR DIFFERENT VALUES OF ADVERSARIAL PERTURBNESS
To further understand the performance RobustNorm under adversarial conditions, we run an experiment where values are increased for the test set. We train networks with BatchNorm, BatchNorm w/o Tracking RobustNorm with Natural as well as PGD-Cpe based adversarial training and tested them on different values of e. The results are shown in Figure 4 As increases, the robustness of neural network decreases but the robustness of neural network with RobustNorm is much higher than BatchNorm while also higher than BatchNorm w/o tracking. To see the effect of increase in adversarial noise CIFARION dataset, see Figure 6 in the appendix.
5 IS TRACKING A NECESSARY EVIL?
the previous sections, we have empirically shown the wickedness of tracking in BatchNorm. But there more to the story. One benefit of tracking that makes it necessary evil in BatchNorm is its ability have
7

---

Under review conference paper at ICLR 2020
Grad
BatchNorm
BatchNorm w/o Tracking
RobustNorm
GradSign
Natural Training
80
80
80
PGD-42
80
80
60
PGD-t.
BIM-
40
40
20
0.003
0.08
0.1
0.3
0.5
0.7
0.003
0.03
0.1
0.3
0.5
0.7
0.9
0.003
0.03
0.1
0.3
0.5
0.7
0.9
Adv. Training
80
80
60
60
40
40
40
20
20
20
0.003
0.00
0.1
0.5
0.7
0.003
0.1
0.3
0.7
0.003
0.1
0.3
0.7
0.9
Figure 4: Comparison of robust accuracy as e(adversarial noise power) increases. Values are calculated for IIFARIO with 3 random restarts and confidence interval of 95%. Robustness of RN is higher than both BatchNorm and BatchNorm without tracking. While BatchNomr with and without tracking collapses with higher epsilon BIM, RobustNorm's accuracy is much higher. For same curves CIFARIO. 10, appendix for more details.
consistent results on small inference time batch sizes. For all the results mentioned in this paper, have used 100 as inference time batch size. By removing tracking, we lose this ability as shown Figure Does this make tracking a necessary evil? Based the observations in section B one straight forward solution to this problem could be the use of tracked values during training as well since it will change the issue of different distributions. This, however, causes the model blow up argued by Joffe & Szegedy (2015): loffe (2017). To resolve this issue, one possible solution is Batch Renormalization like formulation (loffe) 2017 but requires careful tuning of many hyperparameters which makes its use very difficuut. Similarly, other possible solutions such as Arpit et al. (2016); Salimans & Kingma (2016); Wu & Hc (2018) have their own challenges as well. From these observations, we argue that this problem requires more investigation.
90
90
60
60
NatchNom
NotchNom
30
Betd we Tracking
40
BeichNom wi Tacking
RobustNorm with Trookng
Rob RebustNorm with Trasking
10
Time
Batch
Figure 5: Effect of using very small inference time batch size on different norms. Part(a) shows results for natural training while part(b) shows results for adversarial trained networks.
8

---

Under review conference paper at ICLR 2020
is also important to note that RobustNorm's results are restored by increasing batch size by small number. Similarly, tracking is also less harmful for RobustNorm and RobustNorm with tracking is still more robust while having all the benefits of BatchNorm.
CONCLUSION
Addition of maliciously crafted noise in normal inputs, also called adversarial examples has proven be deceptive for neural networks. While there are many reasons for this phenomena, recent work has shown BatchNorm be a cause of this vulnerability as well. In this paper, we have investigated the reasons behind this issue and found that tracking part of BatchNorm causes this adversarial vulnerability. Then, we showed that by eliminating it, we can increase the robustness of neural network. Afterward, based on the intuitions from the work done for the understanding of BatchNorm, we proposed RobustNorm which has much higher robustness than BatchNorm for both natural as well as adversarial training scenarios. In the end, we have shown how tracking can be necessary evil and argued that it requires further careful investigation.
REFERENCES
Devansh Arpit, Yingbo Zhou, Bhargava U Kota, and Venu Govindaraju. Normalization propagation: A para- metric technique for removing internal covariate shift in deep networks. arXiv preprint arXiv: 2016.
Nils Bjorek, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normalization. In Advances in Neural Information Processing Systems, pp. 7694-7705, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Gavin Weiguang Ding, Kry Yik Chau Lui, Xiaomeng Jin, Luyu Wang, and Ruitong Huang. On the sensitivity of adversarial robustness to input data distributions. 2019.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning-Volus Volume 70, pp. 1019- 1028. JMLR. org, 2017.
Christian Etmann, Sebastian Lunz, Peter Maass, and Carola-Bibiane Schönlieb. On the connection between adversarial robustness and saliency map interpretability. arXiv preprint arXiv: 04/70, 2019.
Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W Taylor. Batch normal- ization is cause of adversarial vulnerability. arXiv preprint arXiv:1905.02116 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1412.6572 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9()::142, 1997.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch-normalized models. In Advances in neural information processing systems, pp.
2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502.0316..

---

Under review conference paper at ICLR 2020
J"orn-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerabilityy. arXiv preprint orXiv:/8/I.00 2018.
Jörn-Henrik Jacobsen, Jens Behrmannn, Nicholas Carlini, Florian Tramer, and Nicolas Papernot. Exploiting excessive invariance caused by norm-bounded adversarial robustness. arXiv preprint arXiv:1903. 100::9993..444, 2019.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.0455 2016.
Alex Krizhevsky al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio, Adversarial machine learning scale. arXiv preprint 1rXiv:1611.0122 0.1266, 2016.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, pp. 6389-6399, 2018a.
Xiang Li, Shuo Chen, Xiaolin Hu, and Jian Yang. Understanding the disharmony between dropout and batch normalization by variance shift. arXiv preprint /801.00801.05/33. 2018b.
Ping Luo, Xinjiang Wang, Wengi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. arXiv preprint urXiv:/706.0666 2017.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence
predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 427-436, 2015.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium Security and Privacy (SP), pp. 582-597. IEEE, 2016.
Tiberiu Popoviciu. Sur les équations algébriques ayant toutes leurs racines réelles. Mathematica, 9:129-145, 1935.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901-909, 2016.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? In Advances in Neural Information Processing Systems, pp. 2483-2493 2018.
Carl-Johann Simon-Gabriel, Yann Ollivier, Léon Bottou, Bernhard Schölkopf, and David Lopez-Paz. Adver- sarial vulnerability of neural networks increases with input dimension. arXiv preprint 180.v:1802.01421,,
2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1409. 1556, 2014.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint
2017.
10

---

Under review conference paper at ICLR 2020
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan I. Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 501-509, 2019.
11

---

Under review conference paper at ICLR 2020
A ROBUSTNORM
A.1 ROBUSTNORM FOR OTHER ARCHITECTURES
In this section, we provide more detailed results for our experiments for both CIFARIO and CIFAR100 datasets. Table and shows detailed results.
Norm
Clean
Noise
GradSign
BIM-linf
PGD-/
BatchNorm
93.11
62.67
41.53
36.88
35.12
Natural Training Resnet38
BatchNorm w/o tracking
90.62
79.42
56.80
51.31
50.68
RobustNorm
92.24
80.35
58.37
55.01
53,69
RobustNorm with Tracking
92.71
62.31
43,46
38.33
37.45
BatchNorm
73.21
59.21
43.54
42.06
39.77
Adversarial Training Resnet38
BatchNorm w/o Tracking
89.74
84.82
75.21
69.45
71.91
RoubstNorm
91.77
85.57
77.29
72.23
74.23
RoubstNorm with Tracking
87.08
72.86
58.42
54.56
54.09
BatchNorm
93.61
60.44
42.06
36.15
35.71
Natural Training
BatchNorm w/o Tracking
90.38
77.88
57.36
51.18
51.44
Resnet50
RoubstNorm
91.67
79.33
57.01
53.12
51.75
RoubstNorm with Tracking
93.23
66.14
44.88
40.33
38.50
BatchNorm
65.99
52.33
30.92
35.42
28.40
Adversarial Training Resnet50
BatchNorm w/o Tracking
90.22
86.29
78.17
73.01
75.21
RoubstNorm
90.91
86.14
78.03
73.29
75.21
RoubstNorm with Tracking
63.12
50.68
43.37
39.45
40.36
BatchNorm
91.66
81.80
51.71
54.07
46.66
Natural Training
BatchNorm w/o Tracking
90.28
86.70
64.29
62.86
59.97
VGGII
RoubsINorm
90.51
86.69
29.53
48.69
25.71
RoubstNorm with Tracking
91.74
79.53
52.35
52.65
47.04
BatchNorm
79.18
70.48
52.63
52.38
48.99
Adversarial Training VGG11
BatchNorm w/o Tracking
89.69
86.78
76,98
73.10
74.51
RoubsINorm
90.81
87.81
77.53
74.2
74.84
RoubstNorm with Tracking
0
0
0
Natural Training VGG16
BatchNorm
93.56
66.68
49.83
44.52
44.38
BatchNorm w/o Tracking
92.44
84.56
59.64
54.35
52.81
RoubstNorm
92.62
85.52
45.01
52.01
39,02
RoubsINorm with Tracking
93.54
73.86
50.98
48.52
45.43
Adversarial Training VGG16
BatchNorm
82.27
71.77
56.78
53.58
52.61
BatchNorm w/o Tracking
92.06
87.52
76.49
71.48
72.98
RoubstNorm
92.53
88.32
80.65
77.65
78.70
RoubstNorm with Tracking
91.20
75.39
68.66
65,28
65.34
Table 4: Comparison of clean and adversarial accuracy of different network architectures. We present two depths of Resnet, 38 and 50 and two for VGG, 1I and 16. We trained all these networks with both natural as well as adversarial training and use noise as well as different attacks methods to find their robustness. All of these results are for CIFAR 10 dataset.
12

---

Under review conference paper at ICLR 2020
Norm
Clean
Noise
GradSign
BIM-linf
PGD- inf
BatchNorm
70.22
31.43
18.77
18.01
15.88
Natural Training
BatchNorm w/o Tracking
63.9
46.65
27.03
26.38
23.06
Resnet38
RoubstNorm
68.16
47.88
28.12
27.98
24.27
RoubstNorm with Tracking
70.46
26.79
17.91
15.80
15.20
BatchNorm
42.54
30.78
24.63
21.88
22.33
Adversarial Training Resnet38
BatchNorm w/o Tracking
65.71
57.70
46.46
41.47
42.68
RoubsINorm
67.91
57.39
48.03
42.18
44.14
RoubstNorm with Tracking
43.44
26.29
28.49
21.01
25.99
BatchNorm
73.67
29.18
18.19
17.15
15.55
Natural Training
BatchNorm w/o Tracking
66.90
48.11
27.25
27.06
23.16
Resnet50
RoubstNorm
67.88
46.04
28.60
27.98
25.17
RoubstNorm with Tracking
72.79
31.67
17.72
17.90
14.92
BatchNorm
25.00
16.01
14.21
12.25
13.11
Adversarial Training Resnet50
BatchNorm w/o Tracking
66.55
59.06
49.31
44.10
45.79
RoubstNorm
67.24
58.14
50.87
45.46
47.77
RoubstNorm with Tracking
24.32
13.38
17.75
12.77
16.33
BatchNorm
69.93
50.97
26.88
30.56
23.53
Natural Training
BatchNorm w/o Tracking
67.62
60.49
36.30
37.99
32.66
VGGHT
RoubstNorm
68.77
61.35
16.18
31.21
13.16
RoubstNorm with Tracking
69.96
48.29
25,89
28.90
22.63
BatchNorm
47.69
28.78
28.56
23.50
26.56
Adversarial Training VGGI1
BatchNorm w/o Tracking
66.12
61.33
49.82
46.85
47.06
RoubstNorm
68.02
61.40
50.68
47.09
47.57
RoubstNorm with Tracking
64.43
43.31
44.67
41.67
41.77
BatchNorm
73.28
36.12
22.49
21.74
19.49
Natural Training VGG16
BatchNorm w/o Tracking
70.34
56.15
33.97
33.36
29.46
RoubstNorm
72.28
56.33
16.65
28.87
13.53
RoubstNorm with Tracking
73.40
38.31
25.43
24.93
23.78
BatchNorm
53.11
26.39
31.23
31.21
28.89
Adversarial Training VGG16
BatchNorm w/o Tracking
69.33
61.49
51.16
46.83
48.19
RoubstNorm
71.01
61.89
53.07
48.89
50,39
RoubsINorm with Tracking
68.34
39.22
43.89
34.10
41.07
Table Comparison of clean and adversarial accuracy of different network architectures for CIFAR 100.
We present two depths of Resnet, 38 and 50 and two for VGG, 11 and 16. We trained all these networks with both natural as well as adversarial training and use noise well as different attacks methods find their robustness.
A.2 RESISTANCE TO INCREASE ADVERSARIAL NOISE
In this section, put results of increasing adversarial noise CIFAR100 dataset.
13

---

Under review conference paper at ICLR 2020
Grad
Natural Training
60
BatchNorm
BatchNorm w/o Tracking
RobustNorm
GradSign
PGD-12
PGD-t-
BIM-
20
0.003
0.08
0.1
0.3
0.5
0.7
0.303
0.03
0.1
0.3
0.5
0.7
0.9
0.003
0.03
0.1
0.3
0.5
0.7
0.9
80
Adv. Training
80
40
40
20
20
20
0.003
0.00
0.1
0.5
0.7
0.003
0.03
0.1
0.3
0.7
0.003
0.03
0.1
03
0.5
0.7
Figure Figure shows effect increase in adversarial noise on three normalizatons for CIFARI00 dataset.
A.3 ANOTHER ASPECT OF ROBUSTNORM
As we have discussed. ICS hypothesis has been negated by few recent studies. One of these studies Santurkar et al. 2018) suggested that based on the results, it might be valuable to perform a principled exploration of the design space of normalization schemes as can lead better performance." In this way, we can see RobustNorm with tracking as a new normalization scheme which is based on alternative explanations yet having performance equal BatchNorm which, in a way, weakens ICS hypothesis. See Figure 7 for a comparison of accuracies over different models for CIFAR10 and CIFAR 100 datasets.
85
BatchNorm with OFAR10
RebustNorm with CIFAR10
BatchNorm with OFAR100
RoubetNorm with CIFAR100
75
70
vgg11
Figure 7: Comparison of BatchNorm with RobustNorm in terms of accuracy when tracking is used. Both of these norms have very similar clean accuracy despite RobustNorm being different terms ICS hypothesis.
LOss LANDSCAPE
this section, we discuss possible reasons for the high variation of validation loss for a network with normalization with tracking. As shown in Figure g training loss of all the norms converges with similar
14

---

Under review conference paper at ICLR 2020
BN w/ Tracking
80
BN
3.5
RN
3.0
RN with Tracking
60
25
BN Tracking
RN
RN with Tracking
0.5
0
50
100
150
50
100
150
Adversarial Training Train Loss
Adversarial Training Train Accuracy
BN w/o Tracking
60
BN
MM
RN
50
RN with Tracking
40
30
BN w/o Tracking
20
BN
2
10
RN with Tracking
100
150
50
100
Adversarial Training Validation Loss
Adversarial Training Validation Accuracy
Figure 8: Training and validation loss and accuracy evolution for adversarial training. Interestingly, Batch- Norm's 's training loss decreases normally, its validation loss has lot more uncertainty and either remaining same or start increasing. This way, BatchNorm overfitting. A similar trend is also shown by RobustNorm when tracking is used thought it vanishes when remove tracking and decrease in loss becomes normal and less uncertain.
fashion for all the random restarts but validation loss has lot of variation over these restarts. In other words, the value of training loss similar among many restarts while the same loss values change drastically for validation. To further understand it, we draw loss landscape of these networks using formulation given by Li et al. (2018b) in Figure From these plots, we observe an interesting behaviour: networks having normalization without tracking(i.e. better robustness) tend to have sharp minima as can be seen in figures Dc 9d 9c 9f while their counterparts have more flat loss landscape i.c. figures 9a 9b Og 9h There a long history of debate generalization ability of sharp flat minima Hochreiter & Schmidhuber et al. (016)); Dinh et al.] (2017). We think more work this direction can lead to a better understanding of how BatchNorm causes this vulnerability.
Keskarr
C MORE EXPERIMENTS ON ROBUSTNORM
this section, we discuss some less interesting experiments done to understand the role of power in Ro- bustNorm.
15

---

Under review conference paper at ICLR 2020
(a) BatchNorm with Natural Training
BatchNorm with Adversarial Training
(c) BatchNorm w/o Tracking. Natural Training
(d) BatchNorm w/o Tracking, Adversarial Training
(c) RobustNorm, No tracking, Natural Training
(f) RobustNorm, No Tracking, Adversarial Training
(g) RobustNorm, with Tracking, Natural Training
(h) RobustNorm, with Tracking. Adversarial Training
Figure 9: Plots of loss landscape [Li et al. 2018a) of Resner20 trained with different norms. Right column shows Resnet20 trained on natural images while left column shows adversarially trained. Note that network without tracking 99c 9d 9e 9f) tends to have sharp minima.
16

---

Under review conference paper at ICLR 2020
RobustNorm w/o Tracking
RobustNorm with Tracking
90
80
40
50
Natural Training
Natural training Adversarial Training
Adversarial Training
40
20
1.0010.01
0.1 0.2 0.5
0.7
0.9
1
2
0.001 0.01 0.1
0.2 0.5 0.7
0.9
2
Norm Power
Norm Power
Figure 10: Comparison of effect of hyperparameter p clean accuracy for both adversarially trained and naturally trained Resnet20 with RobustNorm with and without tracking.
Natural Training
Adversarial Training
75
75
70
Grad
65
65
GradSign
60
BIM-&,
CW-f2
55
PGD-tz
9
50
PGD-L-
45
0.01
Norm Power (p)
Norm Power (p)
Figure 11: Comparison of effect of hyperparameter on clean accuracy.
C.I EFFECT OF POWER ON ROBUSTNORM
In this section, we show the effect of changing hyperparameter p for clean as well robust accuracy. From figure 4 and I1] it can be seen that both robustness to many attacks as well accuracy changing with the power. So it can be concluded that by tuning hyperparameters, can get better results.
17


Reviews:
[{"review_id": "SyxnlNl0tB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an interesting perspective that BatchNorm may introduce the adversarial vulnerability, and probes why BatchNorm performs like that (the tracking part in BatchNorm). In experiment, the robustness of the networks increases by 20% when removing the tracking part, but the test accuracy on the clean images drops a lot. Afterwards, the authors propose RobustNorm, which performs better than BatchNorm for both natural and adversarial scenarios.\n\nDetailed Comments: \n+ The paper is well written. The paper structure is clear and figures are well illustrated.\n+ The paper understands and carefully investigates BatchNorm in a interesting and important direction. After the investigation, the improved version RobustNorm shows more potential.\n+ The experimental results seem good. The RobustNorm performs better than BatchNorm for both natural and adversarial scenarios.\n- More results on ImageNet would be better to verify the proposed RobustNorm method.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyxQ0cXatS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review: This paper investigates the reason behind the vulnerability of BatchNorm and proposes a Robust Normalization. They experimentally show that it is the moving averages of mini-batch means and variances (tracking) used in Normalization that cause the adversarial vulnerability. Based on this observation, they propose a new normalization method not only achieves significantly better results under a variety of attack methods but ensures a comparable test accuracy to that of BatchNorm on unperturbed datasets. The paper is clearly written, easy to read.\n \nStrengths:\n \nExplore the cause of adversarial vulnerability of the BatchNorm and assume that the tracking mechanism used in original BatchNorm leads to the vulnerability from experiment results.\nPropose a new and simple normalization method and perform extensive experiments to validate the efficacy of proposed method.\n \nWeaknesses:\nThough extensive experiments have been done by revealing what leads the vulnerability and the effectiveness of proposed method. The results seem unconvincing with respect to different datasets, since Cifar10 and Cifar100 are inherently connected. Would you mind performing some experiments on ImageNet? Since adversarial training on ImageNet is time-consuming, can you show us the result of Natural Training of different models with different norms on ImageNet and compare their robustness under different attack?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkgvVW1htH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses a limitation of BatchNorm: vulnerability to adversarial perturbations. The authors propose a possible explanation of this issue and correspondingly an alternative called RobustNorm to tackle this problem. Specifically, the authors observe that the statistics of BatchNorm for training and inference are different, resulting in different data distributions for training and inference. To solve this problem, the authors propose to use min-max rescaling instead of normalization. In addition, the running average is calculated with mean and the running mean of the denominator during inference. Experimental results show significant improvement of robustness and also comparable accuracy for clean data.\n\nThe paper is well-written and the contributions are stated clearly. The explanation of vulnerability is reasonable. The proposed solution is simple but effective.\n\nHowever, I have several concerns:\n*The authors verify that the running average is the main culprit of vulnerability to adversarial attack, but provide no further investigation of why this happens. A possible solution is the drift in input distributions, but the manuscript does not state clearly how is the distribution changed. Further experiments would have made this claim more convincing.\n*The proposed method involves a hyper-parameter \\rho, but it may result in problematic issues. The variance of input is of the same order of magnitude as (max(x)-min(x))^2. If \\rho is set to other value, the magnitude of gradient will change drastically during back-propagation. Although \\rho can be set to 0.2, it still seems ad-hoc. Experiments on more datasets and the sensitivity of the proposed method to \\rho would have validated the claims of the authors.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review:  This paper presents an empirical analysis of the reasons behind BatchNorm vulnerability to adversarial inputs, based on the hypothesis that such vulnerability may be caused by using different statistics during the inference stage as compared to the training stage. While the paper is interesting and clearly written, reviewers point out insufficient empirical evaluation in order to make the claim more convincing.
Decision: Reject