Title: TOWARDS AN ADVERSARIALLY ROBUST NORMALIZATION APPROACH

Abstract: Batch Normalization (BatchNorm) has shown to be effective for improving and accelerating the training of deep neural networks. However, recently it has been shown that it is also vulnerable to adversarial perturbations. In this work, we aim to investigate the cause of adversarial vulnerability of the BatchNorm. We hypothesize that the use of different normalization statistics during training and inference (mini-batch statistics for training and moving average of these values at inference) is the main cause of this adversarial vulnerability in the BatchNorm layer. We empirically proved this by experiments on various neural network architectures and datasets. Furthermore, we introduce Robust Normalization (RobustNorm) and experimentally show that it is not only resilient to adversarial perturbation but also inherit the benefits of BatchNorm.

1. INTRODUCTION:

In spite of their impressive performance on challenging tasks in computer vision such as image classification and semantic segmentation, deep neural networks (DNNs) are shown to be highly vulnerable to adversarial examples, i.e. carefully crafted samples which look similar to natural images but designed to mislead a trained neural network model (Goodfellow et al., 2014; Nguyen et al., 2015; Carlini & Wagner, 2017) . Designing defense mechanisms against these adversarial perturbations has been subjected to much research recently (Xie et al., 2019; Madry et al., 2017; Tramèr et al., 2017; Papernot et al., 2016) .

Meanwhile, Batch Normalization (BatchNorm or BN) (Ioffe & Szegedy, 2015) has successfully proliferated throughout all areas of deep learning as it enables stable training, higher learning rates, faster convergence, and higher generalization accuracy. Initially, the effectiveness of the BatchNorm has been attributed to its ability to eliminate the internal covariate shift (ICS), the tendency of the distribution of activations to drift during training. However, later on, alternative reasons including avoiding exploding activations, smooth loss landscape, reducing the sensitivity to initialization, etc. have also been proposed as the basis of BatchNorm's success (Santurkar et al., 2018; Bjorck et al., 2018; Luo et al., 2018) .

While there exist a plethora of reasons for the adversarial vulnerability of deep neural networks (Jacobsen et al., 2018; Simon-Gabriel et al., 2018) , a recent study by Galloway et al. (2019) showed that BatchNorm is one of the reasons for this vulnerability. Specifically, they empirically showed that removing the BatchNorm layer enhances robustness against adversarial perturbations. However, removal of BatchNorm also means a sacrifice of benefits such as the use of higher learning rates, faster convergence, and significant improvement in the clean test set accuracy among many others.

In this paper, we propose a new perspective regarding the adversarial vulnerability of the BatchNorm layer. Specifically, we probe why BatchNorm layer causes the adversarial vulnerability. We hypothesize that the use of different normalization statistics during training and inference phase (mini-batch statistics for training and moving average of these statistics also called tracking, at inference time) is the cause of this adversarial vulnerability of the BatchNorm layer. Our experiments show that by removing this part of the BatchNorm, the robustness of the network increases by 20%. Similarly, robustness can further be enhanced by up to 30% after adversarial training. However, by removing the tracking part, the test accuracy on the clean images drops significantly ( though better than without normalization). To circumvent this issue, we propose Robust Normalization (RobustNorm or RN). Our experiments demonstrate that RobustNorm not only significantly improve the test performance of adversarially-trained DNNs but is also able to achieve the comparable test accuracy to that of BatchNorm on unperturbed datasets. We perform numerical experiments over standard datasets and DNN architectures. In almost all of our experiments, we obtain a better adversarial robustness performance on perturbed examples for training with natural as well as adversarial training.

2. BACKGROUND:

We consider a standard classification task for data, having underlying distribution denoted as D, over the pair of examples x ∈ R n and corresponding true labels y ∈ {1, 2, ..., k} where k represents different labels. We denote deep neural network (DNN) as a function F θ (x), where θ denotes trainable parameters. θ is learned by minimizing a loss function L(x, y) with training data x, y. The output of the DNN is a feature representation f ∈ R d , that we give input to a classifier C : R n → {1, 2, ..., k}. The objective of the adversary is to add the additive perturbation δ ∈ R n under the constrain that the generated adversarial sample x adv = x + δ that looks visually similar to the true image x, and for which the corresponding labels are not same i.e. C(x) = C(x adv ). In this work, we have added the perturbation via following well-known adversarial attack approaches.

Fast Gradient Sign Method: Given an input image x along with its corresponding true label y, FGSM Goodfellow et al. (2014) aims to generate the adversarial image x adv as,

x adv = x + • sign(∇ x L(x, y),

(1) where is the perturbation budget that is chosen to be sufficiently small. We use two of its variants: Gradient (Grad) where graidents are used and Gradient sign (GradSign) which is similar to 1.

Basic Iterative Method (BIM): BIM (Kurakin et al., 2016) is a straight forward extension of FGSM, that applies it multiple times with a smaller step size. Specifically,

x 0 adv = x, x N adv = Clip x, {x N -1 adv + α • sign(∇ x L(x N -1 adv , y))} (2)

where x 0 adv is the clean image and N denotes iteration number. Carlini-Wagner attack (CW): CW is an effective optimization-based attack model introduced by Carlini & Wagner (2017) . It works by definining an auxilary variable ϑ and minimizes the following objective functions min

EQUATION

where 1 2 (tanh(ϑ) + 1) -x is the perturbation δ, c is a scalar constant, and f(.) is defined as:

EQUATION

)

Here, is to control the adversarial sample's confidence and Z x adv are the logits values for class k.

Projected Gradient Descent (PGD): PGD perturbs the true image x for total number of N steps with smaller step sizes. After each step of perturbation, PGD projects the adversarial example back onto the -ball of normal image x , if it goes beyond the -ball. Specifically,

EQUATION

)

where Π is the projection operator, α is step size, and x N denotes adversarial example at the N -th step. We have used ∞ norm as a distance measure. Gaussian Noise For comparison purposes, we also have used Gaussian noise with 0 mean and 0.25 variance.

2.0.1. ADVERSARIAL TRAINING::

It has been shown that empirical risk minimization using only clean images for training can decrease the robustness performance of DNNs. A standard approach to achieve the adversarial robustness in DNNs is adversarial training which involves fitting a classifier C on adversarially-perturbed samples along with clean images. We have used PGD based adversarial training which has shown to be effective against many firstorder adversaries (Madry et al., 2017) unlike other methods which overfit for on a single attack.

2.1. EXPERIMENTAL SETUP:

We have used two network architectures, Resnet He et al. (2016) with 20,38 and 50 layers and VGG Simonyan & Zisserman (2014) with 11 and 16 layers. We have used CIFAR10 and 100 datasets (Krizhevsky et al., 2009) for all the evaluations. We have used term natural training for training with clean images while adversarial training is done with PGD based method formulated by Madry et al. (2017) . We have always used a learning rate of 0.1 except for no normalization scenarios where convergence is not possible with higher learning rates. In such cases, we have used a learning rate of 0.001. We decrease the learning rate 10 times after 120 epochs and trained all the networks for 164 epochs. For robustness evaluations, we have used =0.03/1 for most of the experiments and used 20 epochs for all the iterative attacks. We also have tested the model on different noise levels ranging from 0.003/1 to 0.9/1.

3. HOW DOES BATCHNORM CAUSES ADVERSARIAL VULNERABILITY:



3.1. HOW BATCHNORM WORKS:

In this section, we briefly explain the working principle of BatchNorm layer. Broadly speaking, the Batch-Norm layer has been introduced to circumvent the issue of internal covariate shift (ICS). Consider a minibatch B of size M , containing samples x m for m = 1, 2, ..., M . BatchNorm normalizes the mini-batch during training, by calculating the mean µ β and variance σ 2 β as follows:

EQUATION

)

Activation normalization is then performed as,

EQUATION

To further compensate for the possible loss of representational ability of network, BatchNorm also learns per-channel linear transformation as:

EQUATION

for trainable parameters γ and β that represent scale and shift, respectively. These parameters are learnt using the same procedure, such as stochastic gradient descent, as other weights in the network. During training, the model usually maintains the moving averages of mini-batch means and variances(a.k.a. tracking), and during inference, uses these tracked statistics in place of the mini-batch statistics. Formally, tracking (moving average) of mean and variance, for scalar τ are given as, 

μ = (1 -τ )μ + τ µ β , σ2 = (1 -τ )σ 2 + τ σ 2 B (9)

For inference, we can write,

EQUATION

)

3.2. BATCHNORM AND ADVERSARIAL VULNERABILITY:

Recently, Galloway et al. (2019) empirically showed that the accelerated training properties and occasionally higher clean test accuracy of employing BatchNorm in network come at the cost of low robustness to adversarial perturbations. While removing the BatchNorm layer may be helpful for robustness, it also means the loss of desirable properties of BatchNorm like very high learning rate, faster convergence, boost in test accuracy, etc. Therefore, it is pertinent to devise a normalization method that is not only robust to the adversarial perturbations but also inherits the desirable properties of the BatchNorm layer.

3.3. DEVIL IS IN THE TRACKING:

In this section, we aim to investigate the reasons behind the adversarial vulnerability of the BatchNorm layer on the following two grounds;

• We note that during training, mini-batch statistics are used to normalize activations as shown in Equations 6 and 7. Moving average of these statistics are also calculated during the training that is called tracking (shown in Equation 9). The tracked mean and variance are used in the inference step (Equation 10). In this way, different values for mean and variance are used during training and inference. We show this in Figure 1 where it is clear that batch statistics at the start of the training are very different from tracked values that are used at the inference. • Our second observation is based on the recent work of (Ding et al., 2019; Jacobsen et al., 2019) .

These works shed light on the link of the distributional shift in input data and robustness. Specifically, Ding et al. (2019) showed that adversarial robustness is highly sensitive to change in the input data distribution and prove that even a semantically-lossless shift on the data distribution could result in drastically different robustness for adversarially trained models. representations being used at training and inference time which causes drift in input distributions of these layers. Therefore, the tracking part is the main culprit behind the adversarial vulnerability of the BatchNorm layer.

To prove our hypothesis, we have done extensive experiments. For each experiment, we train a neural network model with three different normalization layers: BatchNorm, BatchNorm without tracking, and no normalization. To prove the generality of our argument, we have used various architectures, depths, and datasets as written in section 2.1. We train these networks on clean images as well as with based adversarial training procedure. For adversarial training, we use PGD attack for perturbation. We choose PGD due to its ability to generalize well for other adversarial attacks.

Table 1 shows our results on Resnet20. For detailed experimental results on various architectures, depths and dataset with different attacks see Table 4 , 5 in appendix. The results clearly show that while the elimination of BatchNorm and training at a very small learning rate can help increase robustness, it also reduces clean data accuracy (with BatchNorm). More importantly, this proves our hypothesis that by removing tracking, we can increase the robustness of a neural network significantly. By using BatchNorm without tracking, we also keep many benefits of BatchNorm. Unfortunately, by eliminating the tracking part of BatchNorm, clean accuracy of a network also reduces as compared to clean accuracy with BatchNorm. We tackle this issue in the next section.

4. ROBUST NORMALIZATION:

Although alleviation of ICS was claimed reason for the success of BatchNorm, recently, Bjorck et al. (2018) have shown that BatchNorm works because it avoids activation explosion by repeatedly correcting all activations. For this reason, it is possible to train networks with large learning rates, as activations cannot grow uncontrollably and convergence becomes easier. On a different side, recent work on robustness has shown a connection between the removal of outliers in activations and robustness (Xie et al., 2019; Etmann et al., 2019) . Based on these observations, we use min-max rescaling that is often employed in preprocessing. This is also useful in the elimination of outliers since it rescales the data to a specific scale, Minmax normalization is defined as; However, we experimentally found this layer to be less effective in terms of convergence. Considering the importance of mean (Salimans & Kingma, 2016) , we modify this to;

EQUATION

)

EQUATION

)

We empirically observe the effectiveness of Equation 12over Equation 11 but the overall performance was still inadequate. During debugging, we found that Equation 12 suppress activations much stronger than BatchNorm. This can also be seen from Popoviciu's inequality (Popoviciu, 1935) ,

EQUATION

)

Following Popoviciu's inequality, we introduce the hyperparameter 0 < p < 1, that reduces the denominator in Equation 12,

EQUATION

We experimentally found that p = 0.2 value generalizes well for many networks as well as datasets. We call this normalization Robust Normalization (RobustNorm or RN) due to its robustness properties. We do not use tracking for the RobustNorm. But for comparison purposes, we keep running average of both mean and denominator and use this running average during inference and call this normalization RobustNorm with tracking.

Table 3 shows the accuracy of RobustNorm on Resnet20 for clean as well as adversarial training with both CIFAR10 and CIFAR100 datasets with 95% confidence interval calculated over 5 random restarts. These results show a better clean accuracy of RobustNorm in both natural and adversarial training scenarios. Apart from this, RobustNorm with tracking also shows better performance compared to BatchNorm with tracking.

For adversarial robustness, we have shown Figure 2 with different attacks on CIFAR100 dataset. From the Table 3 and Figure 2 , it is clear that RobustNorm keeps its clean accuracy while being more robust. For more results on Resnet38, Resnet50, VGG11, and VGG16 with CIFRAR10 and CIFAR100 datasets and both natural and adversarial training and many attack methods, please have a look at Table 4 and 5 .

Figure3 shows the evolution of validation loss and accuracy for PGD based adversarial training with a confidence interval of 95% on Resnet20 architecture and CIFAR100 dataset. From Figure 8 in the appendix, it can also be seen that the evolution of training loss and accuracy is normal. But validation loss and accuracy for normalizations with tracking is much different for different random restarts. This can probably be explained based on flat and sharp minima attained by different normalizations as can be seen in loss landscape in Figure 9 in the appendix. For further discussion on the loss landscape, please see section B in appendix. As increases, the robustness of neural network decreases but the robustness of neural network with RobustNorm is much higher than BatchNorm while also higher than BatchNorm w/o tracking. To see the effect of an increase in adversarial noise on CIFAR100 dataset, see Figure 6 in the appendix.

1RUPDO 1RLVH *UDG *UDG6LJ %,0ℓ ∞ 3*'ℓ 2 3*'ℓ ∞ 1DWXUDO7UDLQLQJ 1RUPDO 1RLVH *UDG *UDG6LJ %,0ℓ ∞ 3*'ℓ 2 3*'ℓ ∞ $GYHUVDULDO7UDLQLQJ 51 %1ZR7UDFNLQJ %1

5. IS TRACKING A NECESSARY EVIL?:

In the previous sections, we have empirically shown the wickedness of tracking in BatchNorm. But there is more to the story. One benefit of tracking that makes it a necessary evil in BatchNorm is its ability to have

1DWXUDO7UDLQLQJ %DWFK1RUP %DWFK1RUPZR7UDFNLQJ 5REXVW1RUP $GY7UDLQLQJ ε *UDG *UDG6LJQ 3*'ℓ2 3*'ℓ∞ %,0ℓ∞

Figure 4 : Comparison of robust accuracy as (adversarial noise power) increases. Values are calculated for CIFAR10 with 3 random restarts and confidence interval of 95%. Robustness of RN is higher than both BatchNorm and BatchNorm without tracking. While BatchNomr with and without tracking collapses with higher epsilon BIM, RobustNorm's accuracy is much higher. For same curves on CIFAR10, see appendix for more details. consistent results on small inference time batch sizes. For all the results mentioned in this paper, we have used 100 as inference time batch size. By removing tracking, we lose this ability as shown in Figure 5 . Does this make tracking a necessary evil? Based on the observations in section 3, one straight forward solution to this problem could be the use of tracked values during training as well since it will change the issue of different distributions. This, however, causes the model to blow up as argued by Ioffe & Szegedy (2015) ; Ioffe (2017) . To resolve this issue, one possible solution is Batch Renormalization like formulation (Ioffe, 2017) but it requires careful tuning of many hyperparameters which makes its use very difficult. Similarly, other possible solutions such as Arpit et al. (2016) ; Salimans & Kingma (2016) ; Wu & He (2018) have their own challenges as well. From these observations, we argue that this problem requires more investigation. It is also important to note that RobustNorm's results are restored by increasing batch size by a small number. Similarly, tracking is also less harmful for RobustNorm and RobustNorm with tracking is still more robust while having all the benefits of BatchNorm.

6. CONCLUSION:

Addition of maliciously crafted noise in normal inputs, also called adversarial examples has proven to be deceptive for neural networks. While there are many reasons for this phenomena, recent work has shown BatchNorm to be a cause of this vulnerability as well. In this paper, we have investigated the reasons behind this issue and found that tracking part of BatchNorm causes this adversarial vulnerability. Then, we showed that by eliminating it, we can increase the robustness of a neural network. Afterward, based on the intuitions from the work done for the understanding of BatchNorm, we proposed RobustNorm which has much higher robustness than BatchNorm for both natural as well as adversarial training scenarios. In the end, we have shown how tracking can be a necessary evil and argued that it requires further careful investigation.

A ROBUSTNORM:

A.1 ROBUSTNORM FOR OTHER ARCHITECTURES

In this section, we provide more detailed results for our experiments for both CIFAR10 and CIFAR100 datasets. 

A.3 ANOTHER ASPECT OF ROBUSTNORM:

As we have discussed, ICS hypothesis has been negated by a few recent studies. One of these studies (Santurkar et al., 2018) suggested that based on the results, " it might be valuable to perform a principled exploration of the design space of normalization schemes as it can lead to better performance." In this way, we can see RobustNorm with tracking as a new normalization scheme which is based on alternative explanations yet having performance equal to BatchNorm which, in a way, weakens ICS hypothesis. See Figure 7 for a comparison of accuracies over different models for CIFAR10 and CIFAR100 datasets. 

B LOSS LANDSCAPE:

In this section, we discuss possible reasons for the high variation of validation loss for a network with normalization with tracking. As shown in Figure 8 , training loss of all the norms converges with similar fashion for all the random restarts but validation loss has a lot of variation over these restarts. In other words, the value of training loss is similar among many restarts while the same loss values change drastically for validation. To further understand it, we draw loss landscape of these networks using formulation given by Li et al. (2018b) in Figure 9 . From these plots, we observe an interesting behaviour: networks having normalization without tracking(i.e. better robustness) tend to have sharp minima as can be seen in figures 9c, 9d, 9e, 9f while their counterparts have more flat loss landscape i.e. figures 9a, 9b, 9g, 9h. There is a long history of debate on generalization ability of sharp vs flat minima Hochreiter & Schmidhuber (1997) ; Keskar et al. ( 2016); Dinh et al. (2017) . We think more work in this direction can lead to a better understanding of how BatchNorm causes this vulnerability.

C MORE EXPERIMENTS ON ROBUSTNORM:

In this section, we discuss some less interesting experiments done to understand the role of power in Ro-bustNorm. In this section, we show the effect of changing hyperparameter p for clean as well as robust accuracy. From figure 4 and 11, it can be seen that both robustness to many attacks as well as accuracy changing with the power. So it can be concluded that by tuning hyperparameters, we can get better results.

:



Figures and Tables:

Figure 1: Comparison of mini-batch mean and variance at the start of the training with tracked mean and variance which are used at inference for all the channels. Note the difference between both of these values.

Figure 2: Comparison of adversarial robustness of different normalizations for different Whitebox attacks with CIFAR100. Results are shown with 95% confidence interval computed over 5 random restarts. Batch-Norm w/o Tracking and RobustNorm have significantly higher adversarial robustness compared to other norms. The results are even more clear when they are adversarially trained where RobustNorm's robustness is even more than BatchNorm w/o Tracking.

Figure 3: Comparison of evolution of validation loss and accuracy for CIFAR100 on Resnet20 with confidence intervals calculated with 5 random restarts. BatchNorm without tracking and RobustNorm have higher accuracy and lower loss while RobustNorm being better than BatchNorm w/o tracking. For further details, please have a look at appendix B

Figure 5: Effect of using very small inference time batch size on different norms. Part(a) shows results for natural training while part(b) shows results for adversarial trained networks.

Figure 6: Figure shows effect of increase in adversarial noise on three normalizatons for CIFAR100 dataset.

Figure 7: Comparison of BatchNorm with RobustNorm in terms of accuracy when tracking is used. Both of these norms have very similar clean accuracy despite RobustNorm being different in terms of ICS hypothesis.

Figure 8: Training and validation loss and accuracy evolution for adversarial training. Interestingly, Batch-Norm's training loss decreases normally, its validation loss has lot more uncertainty and either remaining same or start increasing. This way, BatchNorm is overfitting. A similar trend is also shown by RobustNorm when tracking is used thought it vanishes when remove tracking and decrease in loss becomes normal and less uncertain.

BatchNorm w/o Tracking, Natural Training (d) BatchNorm w/o Tracking, Adversarial Training (e) RobustNorm, No tracking, Natural Training (f) RobustNorm, No Tracking, Adversarial Training (g) RobustNorm, with Tracking, Natural Training (h) RobustNorm, with Tracking, Adversarial Training

Figure9: Plots of loss landscape(Li et al., 2018a)  of Resnet20 trained with different norms. Right column shows Resnet20 trained on natural images while left column shows adversarially trained. Note that network without tracking (9c, 9d, 9e, 9f) tends to have sharp minima.

Figure 11: of effect of hyperparameter p on clean accuracy.

Based on the above two observations, we hypothesize that: Different values of first-order statistics (mean and variance) are used in the normalization layer for training and inference. This means different internal Tracking 89.23± 1.43 77.65 ± 2.45 53.32± 0.71 48.38 ± 1.40 48.54± 0.91 Comparison of adversarial robustness of BatchNorm with BatchNorm without tracking and no normalization for CIFAR10 and Resnet20. The results are shown with 95% confidence interval calculated with 5 random restarts. Highlighted values show the best accuracy for that particular adversarial noise. While BatchNorm without Tracking has significantly higher robust accuracy compare to both BatchNorm and no norm, its clean accuracy is lower than BatchNorm. But BatchNorm w/o Tracking also retains its clean accuracy when adversarially trained. This figure confirms our tracking-robustness hypothesis.

Table content: <table><tr><td>Norm</td><td>Clean</td><td>Noise</td><td>Gradient Sign BIM-∞</td><td>PGD-∞</td></tr><tr><td colspan="5">BatchNorm No Norm BatchNorm BatchNorm w/o Adversarial Natural Training No Norm Training BatchNorm w/o Tracking 89.28 ± 1.58 84.36 ± 1.06 72.42 ± 1.04 92.05 ± 0.34 59.16±6.74 33.75 ± 0.99 25.43± 2.37 29.73 ±0.86 82.93 ± 0.64 66.96±4.03 41.81 ± 2.20 41.47±2.16 37.01±2.04 79.53 ± 9.16 67.38±10.41 49.91 ± 6.30 47.90 ± 5.41 46.75 ± 6.05 83.60 ± 1.08 73.75 ±3.95 63.46 ± 1.61 58.29 ± 2.53 60.01 ± 1.54 68.60±0.91 69.30 ±1.77</td></tr></table>

Comparison of clean accuracy of BatchNorm with RobustNorm for both adversarial and natural training scenarios. RobustNorm's accuracy is better than BatchNorm when tracking is not used while its accuracy is same when tracking is used.

Table content: <table><tr><td/><td colspan="2">Natural Training</td><td colspan="2">Adversarial Training</td></tr><tr><td/><td>CIFAR10</td><td>CIFAR100</td><td>CIFAR10</td><td>CIFAR100</td></tr><tr><td colspan="5">BatchNorm with tracking 92.11 ± 0.40 68.20 ± 0.8 79.53 ± 9.16 35.39 ± 3.91</td></tr><tr><td>BatchNorm w/o tracking</td><td colspan="4">89.07 ± 1.73 62.28 ± 0.79 89.28 ± 1.58 61.73 ± 4.17</td></tr><tr><td colspan="5">RobustNorm with tracking 91.97 ± 0.30 68.13 ± 0.28 85.15 ±5.37 42.19 ± 6.10</td></tr><tr><td colspan="5">RobustNorm w/o tracking 91.49 ± 0.28 67.41 ± 0.66 90.76 ±0.63 64.95 ± 0.72</td></tr></table>

Table4and 5 shows detailed results. Comparison of clean and adversarial accuracy of different network architectures. We present two depths of Resnet, 38 and 50 and two for VGG, 11 and 16. We trained all these networks with both natural as well as adversarial training and use noise as well as different attacks methods to find their robustness. All of these results are for CIFAR10 dataset.

Table content: <table><tr><td/><td>Norm</td><td colspan="5">Clean Noise GradSign BIM-linf PGD-l inf</td></tr><tr><td/><td>BatchNorm</td><td colspan="2">93.11 62.67</td><td>41.53</td><td>36.88</td><td>35.12</td></tr><tr><td>Natural Training</td><td>BatchNorm w/o tracking</td><td colspan="2">90.62 79.42</td><td>56.80</td><td>51.31</td><td>50.68</td></tr><tr><td>Resnet38</td><td>RobustNorm</td><td colspan="2">92.24 80.35</td><td>58.37</td><td>55.01</td><td>53.69</td></tr><tr><td/><td colspan="3">RobustNorm with Tracking 92.71 62.31</td><td>43.46</td><td>38.33</td><td>37.45</td></tr><tr><td/><td>BatchNorm</td><td colspan="2">73.21 59.21</td><td>43.54</td><td>42.06</td><td>39.77</td></tr><tr><td>Adversarial Training</td><td>BatchNorm w/o Tracking</td><td colspan="2">89.74 84.82</td><td>75.21</td><td>69.45</td><td>71.91</td></tr><tr><td>Resnet38</td><td>RoubstNorm</td><td colspan="2">91.77 85.57</td><td>77.29</td><td>72.23</td><td>74.23</td></tr><tr><td/><td colspan="3">RoubstNorm with Tracking 87.08 72.86</td><td>58.42</td><td>54.56</td><td>54.09</td></tr><tr><td>Natural Training Resnet50</td><td>BatchNorm BatchNorm w/o Tracking RoubstNorm</td><td colspan="2">93.61 60.44 90.38 77.88 91.67 79.33</td><td>42.06 57.36 57.01</td><td>36.15 51.18 53.12</td><td>35.71 51.44 51.75</td></tr><tr><td/><td colspan="3">RoubstNorm with Tracking 93.23 66.14</td><td>44.88</td><td>40.33</td><td>38.50</td></tr><tr><td>Adversarial Training Resnet50</td><td>BatchNorm BatchNorm w/o Tracking RoubstNorm</td><td colspan="2">65.99 52.33 90.22 86.29 90.91 86.14</td><td>30.92 78.17 78.03</td><td>35.42 73.01 73.29</td><td>28.40 75.21 75.21</td></tr><tr><td/><td colspan="3">RoubstNorm with Tracking 63.12 50.68</td><td>43.37</td><td>39.45</td><td>40.36</td></tr><tr><td>Natural Training VGG11</td><td>BatchNorm BatchNorm w/o Tracking RoubstNorm</td><td colspan="2">91.66 81.80 90.28 86.70 90.51 86.69</td><td>51.71 64.29 29.53</td><td>54.07 62.86 48.69</td><td>46.66 59.97 25.71</td></tr><tr><td/><td colspan="3">RoubstNorm with Tracking 91.74 79.53</td><td>52.35</td><td>52.65</td><td>47.04</td></tr><tr><td>Adversarial Training VGG11</td><td>BatchNorm BatchNorm w/o Tracking RoubstNorm</td><td colspan="2">79.18 70.48 89.69 86.78 90.81 87.81</td><td>52.63 76.98 77.53</td><td>52.38 73.10 74.2</td><td>48.99 74.51 74.84</td></tr><tr><td/><td>RoubstNorm with Tracking</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Natural Training VGG16</td><td>BatchNorm BatchNorm w/o Tracking RoubstNorm</td><td colspan="2">93.56 66.68 92.44 84.56 92.62 85.52</td><td>49.83 59.64 45.01</td><td>44.52 54.35 52.01</td><td>44.38 52.81 39.02</td></tr><tr><td/><td colspan="3">RoubstNorm with Tracking 93.54 73.86</td><td>50.98</td><td>48.52</td><td>45.43</td></tr><tr><td>Adversarial Training VGG16</td><td>BatchNorm BatchNorm w/o Tracking RoubstNorm</td><td colspan="2">82.27 71.77 92.06 87.52 92.53 88.32</td><td>56.78 76.49 80.65</td><td>53.58 71.48 77.65</td><td>52.61 72.98 78.70</td></tr><tr><td/><td colspan="3">RoubstNorm with Tracking 91.20 75.39</td><td>68.66</td><td>65,28</td><td>65.34</td></tr></table>

Peer review (iclr2020):
This paper proposes an interesting perspective that BatchNorm may introduce the adversarial vulnerability, and probes why BatchNorm performs like that (the tracking part in BatchNorm). In experiment, the robustness of the networks increases by 20% when removing the tracking part, but the test accuracy on the clean images drops a lot. Afterwards, the authors propose RobustNorm, which performs better than BatchNorm for both natural and adversarial scenarios. Detailed Comments: + The paper is well written. The paper structure is clear and figures are well illustrated. + The paper understands and carefully investigates BatchNorm in a interesting and important direction. After the investigation, the improved version RobustNorm shows more potential. + The experimental results seem good. The RobustNorm performs better than BatchNorm for both natural and adversarial scenarios. - More results on ImageNet would be better to verify the proposed RobustNorm method.