Intriguing Properties of Adversarial Training at Scale
Authors: Cihang Xie
Alan Yuille
Venue:  (2020)
Keywords: adversarial defense
adversarial machine learning

Abstract: Adversarial training is one of the main defenses against adversarial attacks. In this paper, we provide the first rigorous study on diagnosing elements of large-scale adversarial training on ImageNet, which reveals two intriguing properties. 

First, we study the role of normalization. Batch normalization (BN) is a crucial element for achieving state-of-the-art performance on many vision tasks, but we show it may prevent networks from obtaining strong robustness in adversarial training. One unexpected observation is that, for models trained with BN, simply removing clean images from training data largely boosts adversarial robustness, i.e., 18.3%. We relate this phenomenon to the hypothesis that clean images and adversarial images are drawn from two different domains. This two-domain hypothesis may explain the issue of BN when training with a mixture of clean and adversarial images, as estimating normalization statistics of this mixture distribution is challenging. Guided by this two-domain hypothesis, we show disentangling the mixture distribution for normalization, i.e., applying separate BNs to clean and adversarial images for statistics estimation, achieves much stronger robustness. Additionally, we find that enforcing BNs to behave consistently at training and testing can further enhance robustness.

Second, we study the role of network capacity. We find our so-called "deep" networks are still shallow for the task of adversarial learning. Unlike traditional classification tasks where accuracy is only marginally improved by adding more layers to "deep" networks (e.g., ResNet-152), adversarial training exhibits a much stronger demand on deeper networks to achieve higher adversarial robustness. This robustness improvement can be observed substantially and consistently even by pushing the network capacity to an unprecedented scale, i.e., ResNet-638.  

Published as a conference paper at ICLR 2020
INTRIGUING PROPERTIES OF ADVERSARIAL TRAINING AT SCALE
Cihang Xie Johns Hopkins University
Alan Yuille
Johns Hopkins University
ABSTRACT
Adversarial training is one of the main defenses against adversarial attacks. In this paper, we provide the first rigorous study on diagnosing elements of large-scale adversarial training on ImageNet, which reveals two intriguing properties.
First, we study the role of normalization. Batch Normalization (BN) is cru- cial element for achieving state-of-the-e performance many vision tasks, but we show it may prevent networks from obtaining strong robustness adversarial training. One unexpected observation is that, for models trained with BN, simply removing clean images from training data largely boosts adversarial robustness, ie., 18.3%. We relate this phenomenon to the hypothesis that clean images and adversarial images are drawn from two different domains. This two-domain hy- pothesis may explain the issue of BN when training with mixture of clean and ad- versarial images, as estimating normalization statistics of this mixture distribution is challenging. Guided by this two-domain hypothesis, mixture distribution for normalization, i.e., applying separate BNs to clean and adversarial images for statistics estimation, achieves much stronger robustness. Additionally, we find that enforcing BNs to behave consistently at training and testing can further enhance robustness.
show disentangling the
Second, we study the role of network capacity. We find our so-called "deep" networks are still shallow for the task of adversarial learning. Unlike traditional classification tasks where accuracy is only marginally improved by adding more layers to "deep" networks (e.g., ResNet-152),, adversarial training exhibits much stronger demand on deeper networks to achieve higher adversarial robustness. This robustness improvement can be observed substantially and consistently even by pushing the network capacity to an unprecedented scale, i.e., ResNet-638.
INTRODUCTION
Adversarial attacks (Szegedy et al. 2014) can mislead neural networks to make wrong predictions by adding human imperceptible perturbations to input data. Adversarial training (Goodfellow et al.) 2015) is shown be effective method to defend against such attacks, which trains neural networks on adversarial images that are generated on-the-fly during training. Later works further improve robustness of adversarially trained models by mitigating gradient masking (Tramèr
2018), imposing logits pairing (Kannan et al.) 2018), denoising at feature space (Xie et al. 2019b), etc. However, these works mainly focus on justilying the effectiveness of proposed strategies and apply inconsistent pipelines for adversarial training, which leaves revealing important elements for training robust models still a missing piece in current adversarial research.
In this paper, we provide the first rigorous diagnosis of different adversarial learning strategies, under unified training and testing framework, on the large-scale ImageNet dataset (Russakovsky et al. 2015). We discover two intriguing properties of adversarial training, which are essential for training models with stronger robustness. First, though Batch Normalization (BN) (Toff & Szegedy) 2015) is known
crucial component for achieving nate-of-the-arte on many vision tasks, it may become a major obstacle for securing robustness against strong attacks in the context of adversarial training. By training such networks adversarially with different strategies, e.g., imposing logits pair- ing (Kannan et al. 2018). we observe an unexpected phenomenon removing clean images from training data is the most effective way for boosting model robustness. We relate this phenomenon

---

Published as a conference paper at ICLR 2020
to the conjecture that clean images and adversarial images are drawn from two different domains. This two-domain hypothesis may explain the limitation of BN when training with a mixture of clean and adversarial images. estimating normalization statistics this mixture distribution is chal- lenging. We further show that adversarial training without removing clean images can also obtain strong robustness, if the mixture distribution is well disentangled BN by constructing different mini-batches for clean images and adversarial images to estimate normalization statistics, i.e., one of BNs exclusively for adversarial images and another set of BNs exclusively for clean images. An alternative solution to avoiding mixture distribution for normalization is to simply replace all BNs with unrelaeeee normalization layers, e.g.. group normalization (Wu & He 2018), where normalization statistics are estimated on each image independently. These facts indicate that model robustness is highly related to normalization adversarial training. Furthermore, additional perfor- mance gain is observed via enforcing consistent behavior of BN during training and testing.
Second, we find that our so-called "deep" networks (e.g., ResNet-152) are still shallow for the task of adversarial learning, and simply going deeper can effectively boost model robustness. Experiments show that directly adding more layers to "deep" networks only marginally improves accuracy for traditional image classification tasks. In contrast, substantial and consistent robustness improvement is witnessed even by pushing the network capacity to an unprecedented scale, ie.. ResNett-63.. This phenomenon suggests that larger networks encouraged for the task of adversarial learning, as the learning target, i.e. adversarial images, is more complex distribution than clean images to fit.
In summary, our paper reveals two intriguing properties of adversarial training: (1) properly handling normalization is essential for obtaining models with strong robustness; and (2) our so-called "deep" networks are still shallow for the task of adversarial learning. We hope these findings can benefit future research on understanding adversarial training and improving adversarial robustness.
2 RELATED WORK
Adversarial training. Adversarial training constitutes the current foundation of state-of-the-arts for defending against adversarial attacks. It is first developed in Goodiellow et al. (2015) where both clean images and adversarial images are used for training. Kannan al. (2018) propose to improve robustness further by encouraging the logits from the pairs of clean images and adversarial counterparts to be similar. Instead of using both clean and adversarial images for training, Madry et al. 2018) formulate adversarial training min-max optimization and train models exclusively on versaaria images. Subsequent works are then proposed to further improve the model robustness
(Xie et al.) 2019b] Zhang et al. 2019b; Hendrycks al. 2019a; Qin et al. 2019 Zhang & Wang, 2019 Carmon et al. 2019 Hendrycks et al. 2019b; Alayrac et al., 2019; Zhai et al.) [2019] or
accelerate the adversarial training process (Shatahi al.) 2 et al. 2019a Wang & ZZhng 2019]. However, as these works mainly focus on demonstrating the effectiveness of their proposed mechanisms, a fair and detailed diagnosis of large-scale adversarial training strategies remains as a missing piece. In this work, we provide the first detailed diagnosis which reveals two intriguing properties of training adversarial defenders at scale.
Normalization Layers. Normalization is an effective technique accelerate the training of deep networks. Different methods are proposed to exploit batch-wise (e.g., BN (loffe & Szegedy 2015) ) layer-wise (e.g., layer normalization (Ba et al., 2016)) or channel-wir wise (e.g., instance normaliza- tion (Ulyanov et al. 2016) and group normalization (Wu & He 2018) information for estimating normalization statistics. Different from traditional vision tasks where BN usually yields stronger performance than other normalization methods, we show that BN may become major obstacle for achieving strong robustness in the context of adversarial training, and properly handling normaliza- tion is essential factor to improving adversarial robustness.
ADVERSARIAL TRAINING FRAMEWORK
As inconsistent adversarial training pipelines were applied in previous works (annan et al. 2018 Xie et al.) 2019b), it is hard to identify which elements are important for obtaining robust models. To this end, we provide a unified framework to train and to evaluate different models, for the sake of fair comparison.
2

---

Published as a conference paper at ICLR 2020
45
accuracy ageinst PGD-2000 (90)
-40
35
30
25
20.9
20
15
10
100%
80%
60%
40%
20% 0%
ratio of clean images
Figure 1: The relationship between model robustness and the portion of clean images used for training. We observe that the strongest robustness can be obtained by training completely without clean images, surpassing the bascline model by 18.3% accuracy against PGD-2000 attacker.
Training Parameters. We use the publicly available adversarial training pipeline to train all models with different strategies on ImageNet. We select ResNet-152 152 (He et al. (2016) as the baseline network, and apply projected gradient descent (PGD) (Madry et al. 2018) as the adversarial attacker to generate adversarial examples during training. The hyper-parameters of the PGD attacker are: maximum perturbation of each pixel == 16, attack step size a 1, number of attack iterations 30, and the targeted class is selected uniformly random initialize the adversarial image by the clean counterpart with probability = 0.2, or randomly within the allowed cube with probability 0.8. All models are trained for total of 110 epochs, and decrease the learning rate by 10x at the 35-th, 70-th, and 95-th epoch.
the 1000 ImageNet categories. We
Evaluation. For performance evaluation, we mainly study adversarial robustness (rather than clean image accuracy) in this paper. Specifically, we follow the setting in Kannan et al. (2018) Xid al. (2019b), where the targeted PGD attacker is chosen as the white-box attacker to eval- uate robustness. The targeted class is selected uniformly random. We constrain the maximum perturbation each pixel e = 16, set the attack step size a 1, and measure the robustness by de- fending against PGD attacker of 2000 attack iterations (i.e., PGD-2000). As in Kannan et al. (2018) and Xie et al. (2019)), we always initialize the adversarial perturbation from random point within the allowed e-cube.
We apply these training and evaluation settings by default for all experiments, unless otherwise stated.
4 EXPLORING NORMALIZATION TECHNIQUES IN ADVERSARIAL TRAINING
4.1 ON THE EFFECTS OF CLEAN IMAGES IN ADVERSARIAL TRAINING
In this part, we first elaborate on the effectiveness different adversarial training strategies on model robustness. Adversarial training be dated back to Goodfellow al. (2015). where they mix clean images and the corresponding adversarial counterparts into each mini-batch for training. We choose this strategy as our starting point, and the corresponding loss function is:
j(0, x. v) (0, clean y) (1 a)J adv .y)
where J(-) is the loss function, 0 is the network parameter, is the ground-truth, and training pairs (aelean aade) are comprised of clean images and their adversarial counterparts, respectively. The parameter balances the relative importance between clean image loss and adversarial image loss. We set a = 0.5 following Goodfellow et al. (2015). With our adversarial training framework. this model can achieve 20.9% accuracy against PGD-2000 attacker. Besides this baseline, also study the effectiveness of two recently proposed adversarial training strategies (Madry et al. 2018 Kannan et al. 2018), and provide the results as follows.
3

---

Published as a conference paper at ICLR 2020
55
100% adv
0% clean
100% adv 100% cleam
50
100% adlv 100% cleas, ALP
accuracy (%6)
45
-4
20.6
20.
25,2
25
28.0
20
26.5
10
100 200
500
1000 attack iterations
1500
2000
Figure 2: Comprehensive robust evaluation on ImageNet. For models trained with different strategies, we show their accuracy against PGD attackers with 10 to 2000 iterations. Only the curve of 100% adv 0% clean becomes asymptotic when evaluating against attackers with more iterations.
Ratio of clean images. Different from the canonical form in Goodfellow et al. (2015) Madry al. (2018) apply the min-max formulation for adversarial training where no clean images are used. We note this min-max type optimization can be dated as early as [Wald (1945). We hereby investigate the relationship between model robustness and the ratio of clean images used for training. Specifically. for each training mini-batch, we keep adversarial images unchanged, but removing their clean counterparts by 20%, 40%, 60%, 80% and 100%. We report the results in Figure I Interestingly, removing portion of clean images from training data significantly improve model robustness, and the strongest robustness can be obtained by completely removing clean images from the training set, i.e., it achieves accuracy of 39.2% against PGID-2000 attacker, outperforming the baseline model by large margin of 18.3%.
Adversarial logits pairing. For performance comparison, we also explore the effectiveness of an alternative training strategy, adversarial logits pairing (ALP) (Kannan et al. 2018). Compared with the canonical form in Goodfellow et al. (2015), ALP imposes an additional loss to encourage the logits from the pairs of clean images and adversarial counterparts to be similar. As shown in Figure 2. our re-implemented ALP obtains an accuracy of 23.0% against PGD-2000 attacke which outperforms the baseline model by 2.1%. Compared with the strategy of removing clean images, this improvement much smaller.
Discussion. Given the results above, we conclude that training exclusively on adversarial images is the most effective strategy for boosting model robustness. For example, by defending against PGD-2000 attacker, the baseline strategy Goodfellow et al. (2055) (referred to 100% adv 100% clean) obtains an accuracy of 20.9%. Adding an loss of logits pairing (Kannan et al. 2018) (referred to 100% adv 100% clean, ALP) slightly improves the performance by 2.1%, while completely removing clean images (Madry et al.) 2018 Xie et al. 2019b) (referred to as 100% adv + 0% clean) boosts the accuracy by 18.3%. We further plot comprehensive evaluation curve of these three training strategies in Figure 2 by varying the number of PGD attack iteration from 10 to 2000. Surprisingly, only 100% ady 0% clean can ensure model robustness against strong attacks, i.e., performance becomes asymptotic when allowing PGD attacker to perform more attack iterations. Training strategies which involve clean images for training are suspicious to result in worse robustness, if PGD attackers allowed to perform more attack iterations. In the next section, we will study how to make these training strategies, i.e., 100% adv + 100% clean and 100% adv + 100% clean, ALP to secure their robustness against strong attacks.
4.2 THE DEVIL IS THE BATCH NORMALIZATION
Two-domain hypothesis. Compared to feature maps of clean images, Xie et al. (2019)) show that
feature maps of their adversarial counterparts tend be more noisy. Meanwhile, several works (Li
'Surprisingly, we note our reproduced ALP result significantly stronger than the result reported in the
original ALP paper (Kannan et 20118), as well in independent study (Engstrom et al. 2018)) We identify this performance gap mainly due to different settings of training parameter, and provide detailed diagnosis the supplementary material.

---

Published as a conference paper at ICLR 2020
55
100% adl 0% clean
100% adv 100% clean
50
Af B.N... 100% adv 100% cleas, ALP
accuracy (9
45
-40
25
20
10
100
200
500
1000
1500
2000
attack iterations
Figure 3: Disentangling the mixture distribution for normalization secures model robustness. Unlike the blue curves in Figure 2 these new curves become asymptotic when evaluating against attackers with more iterations, which indicate that the networks using MBN can behave robustly against PGD attackers with different attack iterations, even if clean images are used for training.
& Li 2017, Metzen et al. 2018 Feinman et al. 2017, Pang et al. 2018 Li et al. 2019) demonstrate
is possible to build classifiers to separate adversarial images from clean images. These studies suggest that clean images and adversarial images are drawn from two different domains This two-domain hypothesis may provide an explanation to the unexpected observation (see Sec. and we ask why simply removing clean images from training data largely boost adversarial robustness?
As crucial element for achieving ofth--of various vision tasks, BN (loffe & Szegedy 2015) is widely adopted in many network architectures, egg., Inception (Szegedy et al. 2015). ResNet (He et al. 2016) and DenseNet (Huang al.) (017). The normalization statistics of BN are estimated across different images. However, exploiting batch-wise statistics is challenging task if input images are drawn from different domains and therefore networks fail to learn uni- fied representation on this mixture distribution. Given our two-domain hypothesis, when training with both clean and adversarial images, the usage of BN can be the key issue for resulting in weak adversarial robustness in Figure 2
Based on the analysis above, an intuitive solution arise: accurately estimating normalization statis- fies should enable models to train robustly even if clean images and adversarial images are mixed at each training mini-batch. To this end, we explore two ways, where the mixture distribution is disentangled at normalization layers, for validating this argument: (1) maintaining separate BNs for clean/adversarial images; or (2) replacing BNs with batch-unrelated normalization layers.
Training with Mixture BN. Current net- work architectures estimate BN statistics us- ing the mixed features from both clean and adversarial images, which leads to weak model robustness as shown in Figure 2 Xie et al.) (2019a) propose that properly decou- pling the normalization statisties for adversar- ial training can effectively boost image recog- nition. Here, to study model robustness, we apply Mixture BN (MBN) (Xie et al. (019a).. which disentangles the mixed distribution via constructing different mini- mini-batches for clean and adversarial images for accurate BN statis- ties estimation (illustrated in Figure 4 i.e., one set of BNs exclusively for adversarial im- ages (referred to MBNadv). and another set of BNs exclusively for clean images (referred to as MBNglcan). We do not change the structure of other layers. We verify the effectiveness of this new architecture with two (previously less robust) training strategies, i.e., 100% adv + 100% clean and 100% adv + 100% clean, ALP.
ReLU
ReLU
BN
MBN
MBN
x
Xdenn
conv
COnv
xodx
X0FV gd00s
Figure 4: Standard BN (left) estimates normaliza- tion statistics the mixture distribution. MBN (right) disentangles the distribution by constructing different mini-batch for clean and adversarial im- ages to estimate normalization statistics.
*Or more precisely, "natural" images collected in the datasets and the corresponding adversarial images may come from different distributions.
5

---

Published as a conference paper at ICLR 2020
3
MBNa.
2-
MBN.o
ruaning
-2
channels
0.8
running
0.6
0.2
0
channels
Figure 5: Statistics of running mean and running variance of MBN on randomly sampled 20 channels in ResNet-152's res; block. This suggests that clean and adversarial images induce significantly different normalization statistics.
At inference time, whether image is adversarial clean is unknown. We thereby measure the performance of networks by applying either MBN or MBNe separately. The results are shown in Table We find the performance is strongly related to how BN trained: when using MBNelcan. the trained network achieves nearly the same clean image accuracy as the whole network trained exclusively on clean images; when using MBN adv the Irained network achieves nearly the same ad- versarial robustness as the whole network trained exclusively on adversarial images. Other factors, like whether ALP applied for training, only cause subtle differences in performance. We further plot an extensive robustness evaluation curve of different training strategies in Figure 3 Unlike Figure 2 we observe that networks using MBN now can secure their robustness against strong attacks, e.g., the robustness is asymptotic when increasing attack iterations from 500 to 2000.
The results Table [ Suggest that BN statistics characterize different model performance. For a bet- ter understanding, we randomly sample 20 channels in a residual block and plot the corresponding rumning statistics of MBN elean and MBN in Figure 5 We observe that clean images and adversar- ial images induce significantly different rumning statistics, though these images share the same set of convolutional filters for feature extraction. This observation further supports that (I) clean images and adversarial images come from two different domains; and (2) current networks fail to learn unified representation on these two domains. Interestingly, we also find that adversarial images lead larger running mean and variance than clean images. This phenomenon is also consistent with the observation that adversarial images produce noisy-patterns/oulierr at the feature space (Xie et al. 2019b).
As side note, this MBN structure is also used as a practical trick for training better generative ad- versarial networks (GAN) (Goodfellow et al.) 2014)) Chintala al. (2016) suggest to construct each mini-batch with only real or generated images when training discriminators, as generated images and real images belong to different domains an early training stage. However, unlike our situa- tion where BN statistics estimated on different domains remain divergent after training, successful training of GAN, i.e., able to generate natural images with high quality, usually learns unified set of BN statistics on real and generated images.
Training with batch-unrelated normalization layers. Instead of applying MBN structure dis- entangle the mixture distribution, we can also train networks with batch-unrelated normalization layers, which avoids exploiting the batch dimension to caleulate statistics, for the same purpose. We choose Group Normalization (GN) for this experiment, as GN can reach a comparable perfor- mance to BN various vision tasks (Wu & He 2018). Specifically, for each image, GN divides the

---

Published as a conference paper at ICLR 2020
clean image accuracy (%)
training stralegy
PGD-2000
training strategy
0% adv 100%
78,9
accuracy (%)
MBN 100% adv + 100% clean
+0.4
100% adv 0% clean
39.2
MBN.... 100% adv + 100% clean, ALP
-0.5
100% adv 0% clean"
+3.0
MBN 100% adv + 100S clean
38.3
PGD-2000
training straregy
accuracy (96)
MBN... 100% adv + 100% clean"
+1.6
MBN 100% adv + 100% cleum, ALP
35.3
100% adv + 0% clean
39.2
MBNat, 100% adv 100% alean
-0.9
MBN 100% adv 100% clean, AI.s
+2.8
MBN 100% adv + 100% clean, ALP
-3,9
Table 2: Enforcing a consistent behavior of BN at the training stage and the testing stage signif- icantly boosts adversarial robustness. * denotes that running statistics is used at the last 10 train- ing epochs.
Table I: MBN statistics characterize model performance. Using /tB the trained models achieve strong performance on clean/adversarial images.
channels into groups and computes the normalization statistics within each group. By replacing all BNs with GNs, the mixture training strategy 100% adv + 100% clean now can ensure robustness against strong attacks, i.e., the model trained with GN achieves 39.5% accuracy against PGD-500, and increasing attack iterations 2000 only cause marginal performance drop by 0.5% (39.0% accuracy against PGD-2000). Exploring other vatch-unrelated normalization in adversarial training remains as future work.
Exceptional cases. There are some situations where models directly trained with BN can also ensure their robustness against strong attacks, even if clean images are included for adversarial training. Our experiments show constraining the maximum perturbation of each pixel to be smaller value, e.g.. = 8, is one of these exceptional cases. Kannan et al. (2018) and Mosbach al.] (2018) also show that adversarial training with clean images can secure robustness on small datasets, i.., MNIST, CIFAR- 10 and Tiny ImageNet. Intuitively, generating adversarial images on these much simpler datasets or under smaller perturbation constraint induces a smaller gap between these two domains, therefore making it easier for networks to learn unified representation on clean and adversarial images. Nonetheless, this paper, we stick to the standard protocol in Kannan et al. (2018) and Xie et al. (201bb) where adversarial robustness is evaluated on ImageNet with the perturbation constraint e == 16.
4.3 REVISITING STATISTICS ESTIMATION OF BN
Inconsistent behavior of BN. As the concept of "batch" is not legitimate at inference time, BN behaves differently at training and testing (loffe & Szegedy 2013): during training, the mean and variance are computed on each mini-batch, referred to as batch statistics; during testing, there is no actual normalization performed BN uses the mean and variance pre-computed on the training set (often by running average) to normalize data, referred to as running statistics.
For traditional classification tasks, batch statisties usually converge running statistics by the end of training, thus (practically) making the impact of this inconsistent behavior negligible. Nonetheless, this empirical assumption may not hold in the context of adversarial training. We check this statistics matching of models trained with the strategy 100% adv + 0% clean, where the robustness against strong attacks secured. We randomly sample 20 channels statistics computed on two randomly sampled mini-batches, together with the pre-computed running statistics. In Figure [ interestingly, we observe that batch mean almost equivalent to running mean, while batch variance does not converge to running variance yet certain channels. Given this fact, we then study if this inconsistent behavior of BN affects model robustness in adversarial training.
residual block, and plot the batch
A heuristic approach. Instead of developing new training strategy to make batch statistics con- verge to running statistics by the end of training, we explore a more heuristic solution: applying pre-computed running statistics for model training during the last 10 epochs. We report the perfor- mance comparison in Table 2 By enabling BNs to behave consistently at training and testing, this approach can further boost the model robustness by 3.0% with the training strategy 100% adv 0% clean. We also successfully validate the generality of this approach on other two robust train- ing strategies. More specifically, it can improve the model robustness under the training strategies
7

---

Published as a conference paper at ICLR 2020
batch stats batch statss_
sunning_stats
mean
-0.5
-1-
-1.5
-2
channels
0.6
batch stats batch_stats
0.5
runninggstte stats
58 0.3
0.2
0.1
channels
Figure 6: Comparison of batch statistics and running statistics of BN on randomly sampled 20 channels in ResNet-152's res; block. We observe that batch mean can converge to running mean, while batch variance still differs from running variance.
MBN 100% adv + 100% clean and MBN 100% adv + 100% clean, ALP by 1.6% and 2.8%, respectively. These results suggest that model robustness can be benefited from consistent behav- ior of BN at training and testing. Moreover, we note this approach does not incur any additional training budgets.
4.4 BEYOND ADVERSARIAL ROBUSTNESS
On the importance of training convolutional filters adversarially. In Section 4.2 the performance of models where the mixture distribution is disentangled for normalization applying either MBN or MBN the trained models achieve strong performance on either clean images or adversarial images. This result suggests that clean and adversarial images share the same convolutional filters to effectively extract features. We further explore whether the filters learned exclusively adversarial images extract features effectively We first take a model trained with the strategy 100% adv + 0% clean, and then finctune BNs using only clean images for a few epochs. Interestingly, significantly boosted from 62.3% to 73%, which is only 5.9% worse than the standard training setting, i.e., 78.9%. These result indicates that convolutional filters learned solely on adversarial images can also be effectively applied to clean images. However, we find the opposite direction does not work convolutional filters learned on clean images cannot extract features robustly on adversarial images (e.g., 0% accuracy against PGD-2000 after finetuning BNs with adversarial images). This phenomenon indicates the importance of training convolutional filters adversarially, as such learned filters can also extract features from clean images effectively. The findings here also are related to the discussion of wwwuss//oon-rbusteenn features in [[yas et al. (2019). Readers with interests are recommended to refer to this concurrent work for more details.
we study by
clean images, and vice versa.
find the accuracy clean images can be
Limitation of adversarial training. We note our adversarially trained models exhibit a perfor- mance tradeoff between clean accuracy and robustness - the training strategies that achieve strong model robustness usually result in relatively low accuracy on clean images. For example, 100% adv 0% clean, MBN 100% adv + 100% clean and MBN 100% adv 100% clean, ALP only report 62.3%, 64.4% and 65.9% of clean image accuracy. By replacing BNs with GNs, 100% adv 100% clean achieves much better clean image accuracy, i.e., 67.5%. as well maintaining strong robustness. We note that this tradeoff is also observed in the prior work (Tsipras et al. 2018). Be- sides, Balaji et al. (211)) show it is possible to make adversarially trained models to exhibit better tradeoff between clean accuracy and robustness. Future attentions are deserved this direction.
8

---

Published as a conference paper at ICLR 2020
accuracy against PGD PGD-2000 10-2000 (%)
47
84
aocurac cleen images (%)
100% ady 0% clean.
46
0% 100% clean
83
46
44
81
43
80
42
41
10
ResWet-
Reater-
20
Restet-p0
30 FLOPs (x 10
Reatet- 511
40
Reatest-53 833
Figure 7: Compared to traditional image classification tasks, adversarial training exhibits a stronger demand on deeper networks. The performance gain of traditional image classification becomes marginal after ResNet-200 while the adversarial robustness continues to grow even for ResNet-638.
5 GOING DEEPER IN ADVERSARIAL TRAINING
As discussed in Section current networks are not capable of learning a unified representation on clean and adversarial images. It may suggest that the "deep" network used, i.e., ResNet- still underfits the complex distribution of adversarial images, which motivates us to apply larger networks for adversarial training. We simply instantiate the concept of larger networks by going deeper, i.e., adding more residual blocks. For traditional image classification tasks, the benefits brought by adding more layers to "deep" networks is diminishing, e.g., the blue curve Figure 7 shows that the improvement of clean image accuracy becomes saturated once the network depth goes beyond ResNet-200.
For a better illustration, we train deeper models exclusively on adversarial images and observe pos- sible underfitting phenomenon as shown in Figure 7 In particular, we apply the heuristic policy in Section 4.3 mitigate the possible effects brought by BN. We observe that adversarial learning task exhibits a strong "thirst" on deeper networks to obtain stronger robustness. For example, increasing depth from ResNet- 152 to ResNet-338 significantly improves the model robustness by 2.4%, while the corresponding improvement in the "clean" " training setting (referred to as 0% adv + 100% clean) only 0.5%. Moreover, this observation still holds unprecedented scale, i.e., ResNet-638. These results indicate that our so-called "deep" networks (e.g.. ResNet-1522 are still shallow for the task of adversarial learning, and larger networks should be used for fitting this complex distribution. Besides our findings on network depth, [Madry et al. show
by pushing the network capacity to
(2018) show increase network width also substantially improve network robustness. These empirical observations also corroborate with the recent theoretical studies (Nakkiran) [019 Gao et al. 2019) which argues that robust adversarial learning needs much more complex classifiers.
Besides adversarial robustness, we also observe a consistent performance gain on clean image accu- racy by increasing network depth shown in Table Our deepest network, ResNet-638, achieves an accuracy of 68.7% on clean images, outperforming the relatively shallow network ResNet-152 by 6.1%.
CONCLUSION
In this paper, we reveal two intriguing properties of adversarial training at scale: (1) conducting normalization in the right manner is essential for training robust models on large-scale datasets like ImageNet: and (2) our so-called "deep" networks are still shallow for the task of adversarial learning. Our discoveries may also be inherently related to our two-domain hypothesis adversarial images are drawn from different distributions. We hope these findings can facilitate fellow researchers for better understanding of adversarial training well as further improvement of adversarial robustness.
clean images and
This work was supported by ONR N00014415--123366 We would like to thank Kaiming He, Laurens van der Maaten, Judy Hoffman, Yuxin Wu, Yuyin Zhou, Zhishuai Zhang, Yingwei Li and Song Bai for valuable discussions.
Acknowledgment

---

Published as a conference paper at ICLR 2020
REFERENCES
lean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? In Advances in Neural Information Processing Systems, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv: 10.X:v:607.064500 2016.
Yogesh Balaji, Tom Goldstein, and Judy Hoffman. Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets. arXiv preprint arXtv:1110.08051, 2019.
Yair Carmon, Aditi Raghumathan, Ludwig Schmidt, Percy Liang, and John C Duchi. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems, 2019.
Soumith Chintala, Emily Denton, Martin Arjovsky, and Michael Mathieu. How to train a gan? tips and tricks make gans work. https: /tttps:/ggihuule com httm///umtth///eem htwghhhcaaaaaa/aa//a/aaaa 2016.
Logan Engstrom, Andrew Iyas, and Anish Athalye. Evaluating and understanding the robustness of adversarial logit pairing. arXiv preprint arXiv: 10272, 2018
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410, 2017.
Ruiqi Gao, Tianle Cai, Haochuan Li, Cho-Jui Hsieh, Liwei Wang, and Jason Lee. Convergence of adversarial training in overparametrized neural networks. In Advances in Neural Information Processing Systems 32, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-trate pre-training can improve model robustness and uncertainty. ICML, 2019a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, 2019b.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint arXiv: 1XXXv:195.00273: 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICMI, 2015.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv:1803.06377 2018.
Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter statistics. In ICCV, 2017.
Yifeng Li, Lingxi Xie, Ya Zhang. Rui Zhang, Yanfeng Wang, and Qi Tian. Defending adversarial attacks by correcting logits. arXiv preprint arXiv:1906. 10973, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In ICLR, 2018.
10

---

Published as a conference paper at ICLR 2020
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In ICLR, 2018.
Marius Mosbach, Maksym Andriushchenko, Thomas Trost, Matthias Hein, and Dietrich Klakow. Logit pairing methods can fool gradient- based attacks. arXiv preprint arXiv:1810.12042. 2018.
Preetum Nakkiran. Adversarial robustness may be at odds with simplicity. arXiv preprint
arXiv:1901.0053, 2019.
Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial exam- ples. In Advances in Neural Information Processing Systems, 2018.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. In Advances in Neural Information Processing Systems, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJC,, 2015.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! arXiv preprint arXiv: 12843, 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In ICLR, 2018.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. There no free lunch adversarial robustness (but there are unexpected benefits). arXiv: 12152, 2018
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. / 000220002, 2016.
Abraham Wald. Statistical decision functions which minimize the maximum risk. Annals of Math- ematics, pp. 265-280, 1945.
Jianyu Wang and Haichao Zhang. Bilateral adversarial training: Towards fast training of more robust models against adversarial attacks. In ICCV, 2019.
Yuxin Wu and Kaiming He. Group normalization. ECCV, 2018.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoe V Le. Adversarial examples improve image recognition. arXiv preprint 1xXiv:191.00966 2019a.
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In CVPR, 2019b.
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data. arXiv preprint arXiv:1906.0055.. 2019.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Painless adversarial training using maximal principle. arXiv preprint arXiv:1905.008: 00877, 2019a.
Haichao Zhang and Jianyu Wang. Defense against adversarial attacks using feature scattering-based adversarial training. In Advances in Neural Information Processing Systems, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019b.

---

Published as a conference paper at ICLR 2020
A DIAGNOSIS ON ALP TRAINING PARAMETERS
In the main paper, we note that our reproduced AI.P significantly outperforms the results reported in Kannan al. (2018), as well in an independent study Engstrom al. (2018). The main differences between our version and the original ALP implementation lie in parameter settings, and are detailed as follows:
learning rate decay: the original ALP decays the learning rate every two epochs at an exponential rate of 0.94, while ours decays the learning rate by 10x at the 35-th, 70-th and 95-th epoch. To ensure these two policies reach similar learning rates by the end of training, the total number of training epochs of the exponential decay setting and the step-wise decay setting are set as 220 and 10 respectively.
initial learning rate: the original ALP sets the initial learning rate 0.045 whereas we set it as 0.1 in our implementation.
training optimizer: the original ALP uses RMSProp as the optimizer while we use Momen- tum SGD (M-SGD).
PGD initialization during training: the original ALP initializes the adversarial perturbation from a random point within the allowed cube; while initialize the adversarial image by its clean counterpart with probability = 0.2, or randomly within the allowed the e cube with probability 0.8.
number of GPUs: the original ALP uses 50 GPUs for adversarial training, while ours uses 128 GPUs.
network backbone: the original ALP reports results based on Inception-và v3 and ResNet- 101, while our backbone is ResNet-152.
PGD-N for training: the original ALP uses 10 for training, while we use PGD-30 for training.
PGD-N for training
accuracy (%)
network
#GPUs
PGD initialization
optimizer
initial Ir
decay
PGD3-10
PGD-2000
0.045
exponential
38.1
2.1
RMSProp
47.0
P(e-cube) = 1.0
48
50.2
0.8
ResNet-101
10
48.2
0.7
47.0
2.1
step-wise
0.1
47.5
P(e-cube) = 0.8
M-SGD
47.3
3.3
P(clean) 0.2
30
ResNett115
48.8
23.0
Table 3: The results of ALP re-implementations under different parameter settings. We show that applying stronger attackers for training. e.g., change from PGD-10 PGD-30, the most important factor for achieving strong robustness. Other parameters, like optimizer, do not lead to significant robustness changes.
By following the parameter settings listed in the ALP paper we can train ResNet- 101 with an accuracy of 38.1% against PGD-10. The ResNet-101 performance reported in the ALP paper is 30.2% accuracy against an attack suite] This ~8% performance gap is possibly due to different attacker settings evaluation. However, by evaluating this model against PGD-2000, we are able to obtain similar result that reported in Enggstrom et al. (2018), i.e., Engstrom et al. (2018) reports ALP obtains 0% accuracy, and in our implementation the accuracy is 2.1%.
Given these different settings, we change them one by one to train corresponding models adversari- ally. The results are summarized in Table Surprisingly, we find the most important factor for the performance gap between original ALP paper and ours is the attacker strength used for training by changing the attacker from PGD-10 to PGD-30 for training, the robustness against PGD-2000 can be increased by 19.7%. Other parameters, like network backbones the GPU number, do not lead to significant performance changes.
'For casier implementation, apply 48 GPUs (which can be distributed over 6 8-GPU machines) for adversarial training, instead using the original number, i.e., 50 GPUs.
"This attack suite contains 8 different attackers, including PGD- 10. However, due to the vague description of parameter settings in this attack suite, we are not able reproduce it.
12

---

Published as a conference paper at ICLR 2020
B EXPLORING THE IMPACT OF PARAMETER SETTINGS IN ADVERSARIAL TRAINING
In this section, we explore the impact of different training parameters on model performance.
B.1 PGD-N FOR TRAINING
As suggested in Table 3 the number of attack iteration used for training is an important factor for model robustness. We hereby provide detailed diagnosis of model performance trained with PGD- {5. 10, 20
0 for different training strategies. We report the performance in Table 4 and observe that decreasing the number of PGD attack iteration used for training usually leads to weaker robustness. Nonetheless, we note the amount of this performance change is strongly related to training strategies. For strategies that cannot lead models with strong robustness, i.e., 100% adv 100% clean and 100% adv + 100% clean, ALP, this robustness degradation is extremely severe (which is similar the observation in Table 3p. For example, by training with PGD-5, these two strategies obtains nearly no robustness, i.e., ~0% accuracy against PGD-2000. While for strategies that can secure model robustness against strong attacks, changing from PGD-30 PGD-5 for training only lead to a marginal robustness drop.
accuracy against PGD-2000 (%)
PGD-N for training
MBN
100% adv 0% clean
100% adv 100% clean
100% adv + 100% clean.
ALP
100% adv + 100% clean
MBN
100% adv 4 100% clean.
ALP
30
39.2
20,9
23.0
38,3
35.3
-3.8
-4.2
10
-2.4
-17.7
-19.7
-5.5
-1.4
-3.3
-20.9
-22.7
-7.1
-24
Table 4: Robustness evaluation of models adversarially trained with PGD-(30, 20, 10, 5) at- tackers. We observe that decreasing the number of PGD attack iteration for training usually leads to weaker robustness, while the amount of degraded robustness is strongly related to training strategies.
B.2 APPLYING RUNNING STATISTICS IN TRAINING
In Section 4.3 (of the main paper), we study the effectiveness of applying rumning statistics in train- ing. We hereby test this heuristic policy under more different settings. Specifically, we consider 3 strategies, each trained with 4 different attackers (i.e., (5D--55 10, 20, 30)), which results in 12 different settings. We report the result in Table We observe this heuristic policy boost robust- ness on all settings, which further supports the importance of enforcing BN to behave consistently at training and testing.
training butch size
accuracy against PGD-2000 (%)
accuracy against PGD- 20D-20000-00 (%)
4096
42.2
training stratogy
PGD-5
PGD-10
PGD-20
PGD-30
512
-0.2
1024
+0,9
100% adv 0%
33.9
36.8
38.2
39,2
2048
+1.1
100% adv 0% clean*
+1.9
+2.4
+1.8
+3.0
8192
-0.3
MBN, 100% adv 0% clean
31.2
32.8
34.1
38,3
MBN, 100% adv clean"
+3.0
+3.3
+2.4
+1.6
Table 6: evaluation of models trained with different batch size. The best per- formance can be achieved by training with batch size of 2048.
Performance
MBN, 100% adv U% clean, ALP
32.9
33.9
35.8
35.3
MBN, 100% adv 0% clean, ALP
+2.5
+2.8
+1.3
+2.8
Table 5: Validating the effectiveness of applying running statis- ties in training on more settings. We observe this heuristic pol- icy can boost robustness on all settings. * denotes that running statistics is used at the last 10 training epochs.
B.3 BATCH SIZE IN TRAINING
The default number of training batch size is 4096. We hereby study the model performance when training with the batch size of (512, 1024, 2048, 8192), respectively. Without loss of generality,
For PGD-5 and PGD-10, we set the attack step size a he and 2, respectively.
13

---

Published as a conference paper at ICLR 2020
study the training strategy 100% adv + 0% clean. The heuristic policy in Section 4.3 (of the main paper) is applied to achieve stronger robustness. Compared to the default setting (i.e., 4096 images/batch), training with smaller batch size leads to better robustness. For example, changing batch size from 4096 to 1024 or 2048 can improve the model robustness by ~1% While training with much smaller (i.e,, 512 images/batch) much larger (iee., 8192 images/batch) batch size results in slight performance degradation.
C PERFORMANCE OF ADVERSARIALLY TRAINED MODELS
In the main paper, our study is driven by improving adversarial robustness (measured by the accu- racy against PGD-2000), while leaving the performance clean images ignored. For the complete- ness of performance evaluation, we list the clean image performance of these adversarially trained
models in Table 7 Moreover, to facilitate performance comparison in future works, we list the corresponding accuracy against PGD-(10) 10, 20, 100, 500} in this table as well.
training PGD-N for
accuracy (%)
network
butch training
setting
clean
10
20
100
500
2000
100% adv
100% clean, ALP
73.5
48.8
46.3
34.1
27.3
23.0
100% udv
100% clean
78.0
56.2
53.0
35.6
26.6
20.9
100% adv 80% clean
77.4
56.8
54.6
39.6
28.7
20.3
60%
75.5
23.5
15.3
100% udv +-40% cleun
73.9
54.1
51.5
40.2
33.6
29.1
100% adv 20% clean
68.0
54.6
51.5
411.6
34.6
32.1
100% udv 0%% cleun
62.3
52.5
50.0
41.7
39.6
39.2
100% 0% clean
62.1
$2.4
42.6
42.2
30
MBN 100%% adv 100% clean
64.4
51.8
49.1
411.9
38.8
38.3
MBN 100KE adv 100% clean
64.2
52.5
$0.0
42.1
40.5
39,9
MBN 100%% adv 100% clean, ALP
65.9
47.3
45.0
38.3
35.9
35.3
MBN,,..
clean,
38.6
GN, 100% adv 100% clean
67.5
52.1
49.3
41.9
39.5
39.0
100% adv + 100% clean, ALP
72.7
50.1
47.6
33.4
25.2
19.2
100% adv + 100% clean
78.4
55.9
52.5
31.7
20.0
13.6
100% adv 0%% clean
62.5
54.1
41.8
39.0
38.2
100% adv (%% clean
65.4
$3.9
$1.0
42.7
40.8
40.0
MBN... 100% adv 100% clean
67.9
$2.8
49.9
39.3
35.5
34.1
MBN... 100%
100% clean
67.2
$2.7
49.9
40.2
37.2
36.5
MBN
clean,
68.1
36.7
35.8
4096
MBN 100% adv
100% clean, ALP*
67.4
50.0
47.7
40.3
37.4
37.1
100% adv 100% clean, ALP
74.9
47.3
47.1
22.4
7.3
3.3
ResNet-152
100% 100% clean
78.4
36.6
53.1
28.8
7.1
100% adv 0% clean
66.0
53.6
41.1
38.0
36.8
100% adv 0% clean
65.9
53.5
50.7
42.3
39.9
39.2
MBN 100% adv 100% clean
63.7
53.5
49.9
38.7
34.3
32.8
MBN 100%
100% cleant
67.8
52.6
49.8
40.0
36.9
36.1
MBN,.. 100%
100% clean, ALF
68.7
49.0
46.8
38.9
35.4
33.9
MBN,2. 100% adv
100% clean, ALP*
67.7
50.0
47.7
40.2
37.4
36.7
100% udv 100% clean, ALP
75.0
49.6
37.6
3.7
0.8
0.3
100% 100% clean
78.6
41.5
17.6
0.2
0.0
100% adv 0% clean
67.0
54.0
40.7
37.5
35.9
100% udv 0% cleun"
66.9
53.6
50.4
41.5
38.8
37.8
MBN 100% adv
100% clean
69.3
33.0
49.8
38.0
33.3
31.2
MBN
100% clean
68.9
53.1
39.8
35.9
34.2
MBN... 100% 100% clean, ALP
69.2
49.6
46.8
38.9
34.9
32.9
MBN,,. 100% adv 100% clean, ALP*
69.2
50.4
48.3
40.3
37.0
35.4
512
62.6
52.4
50.4
43.8
42.4
42.0
1024
63.1
53.3
51.0
44.6
43.5
43.1
2048
62.7
53.3
50.7
45.0
43.7
43.3
8T96
61.8
52.
49.8
43.4
42.3
41.9
100% 0% clean"
ResNer-200
63.5
53.8
51.4
45.1
43.9
43.8
ResNNet-338
65.4
55.5
53.2
4096
ResNett5188
66.7
56.7
54.4
48.1
46.4
46.2
ResNet-633 638
67.2
57.2
54,8
48.7
47.1
46.7
Table 7: For easier benchmarking in future works, we list the detailed performance of adversarially trained models. denotes running statistics is used at the last 10 training epochs.
14


Reviews:
[{"review_id": "rJxeamAAKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper reveals some interesting properties of neural networks when trained adversarially at ImageNet scale. The total cost of the experiments is quite impressive, therefore the results are valuable references. With extensive experiments, the authors reveals two intriguing properties of neural networks when trained adversarially, and devotes most of the paper to studying the first one, i.e., why networks with Batch Normalization cannot achieve high robustness when both clean and adversarial images are used for training. I am not fully convinced by this point, but I do think the second point is very interesting. Different from previous work arguing that more data is needed for improving adversarially robust generalization, this paper shows that adversarial robustness can be improved consistently just by making ResNet deeper. \n\nI tend to accept this paper for the valuable results and the discovery of the positive correlation between network capacity and adversarial robustness, but I am not fully convinced by the explanations for the problems with BN. I hope the authors could address my concerns before I can be more confident about my decision.\n\nTo me, it seems the correct way to do adversarial training is to only use adversarial samples, since we are trying to minimize the maximum risk inside a norm ball around the clean sample (Madry et al. 2018). All the experiments with clean images in the objective are consistently worse than training only on adversarial samples under the same setting in this paper. It therefore becomes not that important to study the effect of Batch Normalization on training with both clean and adversarial images.\n\nAlso, I am a little bit unconvinced that the running mean and variance of BN is the cause of bad performance for the mixed training. First, I want to know the standard and adversarial accuracies of the network with GN in the \"100% adv + 0% clean\" setting. It seems missing in the paper. Despite being much better than BN in the \"100% adv + 100% clean\" setting, it is not sure whether it is caused by the improvement in some other property (e.g., capacity) from GN. If the robust accuracy with \"100% adv + 0% clean\" is also much higher with GN than BN (seems unlikely though), then replacing GN with BN does not solve the problem and it is still the objective to blame.\n\nI am also not fully convinced by the experiments with separated BN parameters for clean and adversarial samples. By doing so, the network is actually trained to approximate its behavior in the \"100% adv + 0% clean\" setting. Since the adversary is maximizing the loss, the gradient (of the conv layers) from adversarial samples will dominate the total gradient in the \"MBN 100% adv + 100% clean\" setting, and the \"MBN_{adv}\" network will be trained similarly as the network trained in the \"100% adv + 0% clean\" setting. This explains why the adversarial accuracy can be very close to \"100% adv + 0% clean\" but still lower.  Comparing results (under different ratios of clean samples) for networks without BN is very important and much more effective at explaining the phenomenon than trying to make BN work.\n\nIt would also be great if the authors could provide some curve showing the tendency of the running variance of BN. Sec 4.3 does make lots of sense from a practical aspect, i.e., fixing mean and var for training in the last 10 epochs could improve the results using same number of epochs, but what if we just train more epochs? Will the variance converge in just tens of epochs?\n\nFinally, though it seems that deeper networks are more robust,  the robustness might be a misconception caused by gradient vanishing. Could the authors provide the average gradient norms on the correctly-classified images (remaining correct after attack) in the first step of PGD attacks for the models in Figure 7? If deeper networks indeed have much smaller gradient norm, could the authors try scaling the loss by some factor to make the attacks stronger?\n\n\n===========================================================\nI am not fully convinced by your explanations. Could you give the results for the gradients, or use the loss function from CW attack + PGD to report the robustness of the deeper models? I am not sure which type of residual connection you are using, but if you are using the \"original\" version as mentioned in this paper [1], then it does not necessarily avoid gradient vanishing, since if ReLU is deactivated, the gradient will be brought to zero. Another possible cause for the possible misconception of robustness might be caused by the saturated cross entropy loss, which could also give 0 gradients but could be verified by switching to the margin loss as in the CW attack. I have encountered such cases when I achieved non-zero robust accuracy (evaluated with PGD on cross entropy loss, and the robust accuracy is similar to your improvement from ResNet152 to ResNet638) on a naturally trained network as I increased the number of layers, but when I switched to CW loss (or margin loss), the robust accuracy goes to zero. Even for resnet, when you multiply the number of layers by ~4 as done in the paper, such phenomenon will actually happen.\n\n[1] He, Kaiming, et al. \"Identity mappings in deep residual networks.\" European conference on computer vision. Springer, Cham, 2016.\n\n=====================================================================\nRaising my score. Found a recent good paper based on the findings of this paper. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Skl9fUh4qB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces two properties of adversarial training observed from abundant empirical results. Based on the discoveries, the authors propose plausible explanations as well as new methods to gain higher adversarial robustness. The two properties are as follows\n\n1. The batch normalization may negatively affect the adversarial robustness, where a training batch consists of a mixture of clean and adversarial examples. The authors observe that the parameters for batch normalization (BN) may be quite different between batches of clean and of adversarial examples, and conjecture the reason as that these two sets of examples are from two different domains. Therefore, the authors propose a few methods to boost the adversarial robustness: using different BN stats for clean and adversarial examples (but keeping the other parameters shared, which may seem useful only for fundamental study due to the requirement of clean/adversarial label for each example), using batch-unrelated normalization, and changing stat estimation for BN to reduce the difference between training and inference steps.\n\n2. The deep networks for general image classification may be too shallow for adversarial training. Since the mixture of clean and adversarial examples may form a two domain distribution that is challenging to model, typical neural networks (even for ResNet-152 that has high depth for general image classification) may have too low capacity. The authors show that increasing the depth of neural network (up to and possibly beyond ResNet-638) results in even higher accuracy.\n\nTo my best understanding, the results and analysis of this paper are valid, and the proposed methods have shown gains in abundant experiments. Due to the significance of adversarial training and the new discoveries of this paper, I think the contributions are sufficient and would lean towards the paper being published / weak accept. However, below are my questions and concerns that I would like the authors to address.\n\n1. For this paper, the adversarial examples are generated from a particular class of attacker, namely Projected Gradient Descent. I am curious how the conclusion could generalize and help the robustness if we have more than one adversarial attackers.\n\n2. The paper focuses on adversarial robustness and seems to deprioritize clean image accuracy, which could seem to limit the scope for application purposes. Frankly, I agree that a fundamental understanding of adversarial robustness would be significant, and the authors discuss the problem of relatively lower clean image accuracy in Section 4.4. However, I might feel that clean image accuracy should be as significant for practical purposes, and it would be great if we can balance the trade-off between clean image accuracy and the adversarial robustness. The authors should feel free to correct me if I do not understand correctly.\n\n3. There seems to be an interesting observation in Fig 1 for which I am curious. The accuracy for PGD-2000 in Fig 1 does not go monotonically with the ratio of clean images — the lowest accuracy is at 60 percent of clean images, which seems not to fully align with the argument that removing clean images will help robustness. Personally it would be great if the authors could share their insights of possible reasons.\n\n4. For the second discovery that deeper networks help adversarial robustness, the red line (for adversarial robustness) in Fig 7 seems not converged yet at ResNet-638. If the computation resources allow, I am curious on the depth of ResNet at which the red line becomes flat, and this could be useful for headroom analysis on how good accuracy we can reach.\n\n5. For the second discovery that deeper networks help adversarial robustness (Section 5), it seems Madry et al 2018 (Towards deep learning models resistant to adversarial attacks) also discusses model capacity vs the adversarial robustness. The mentioned paper does not seem to use deeper structure but uses other ways to increase capacity. The mentioned paper has been referred to in other sections of this work, however, it may be good to contrast in Section 5 on the conceptual novelty in this paper.\n\n6. Typos: The title of Table 1, “MBN_{clean}/MBN_{clean}” would be “MBN_{clean}/MBN_{adv}”. There are some “, ,” (double commas) in the appendices. Comma after ‘(’ in appendix B3. Overall this paper is well written and easy to follow.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1xyoFBGqH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies adversarial training \"at scale\", i.e., on the ImageNet dataset. The paper makes two main contributions in this context:\n- An in-depth investigation of the effect batch normalization (BN) has on adversarial robustness when the network is trained adversarially.\n- Training increasingly deeper residual networks (up to 600 layers) and demonstrating that adversarial robustness still increases in this regime (unlike standard accuracy).\n\nOverall I find the findings presented in the paper interesting and recommend accepting the paper. Experimenting with adversarial training on ImageNet is still hard for many academic groups due to the high computational cost. Hence the results of the paper may be useful for the wider robustness community. To achieve this goal, I strongly encourage the authors to release their models in a format that is easy to build on and experiment with for other researchers (e.g., PyTorch model checkpoints). Moreover, I find it interesting that very deep models on ImageNet can achieve increased adversarial robustness. To the best of my knowledge, these are the best robustness numbers published on ImageNet.\n\nFurther comments and questions:\n\n- It would be good to know if BN also affects adversarial robustness on CIFAR-10 or other datasets.\n\n- What happens when the width of the network is increased? Does this also help adversarial robustness?\n\n- Section 4.1 states that \"Adversarial training can be dated back to (Goodfellow et al., 2015), [...]\". While the specific form of adversarial training for adversarial robustness in CNNs is indeed recent, it may be helpful for readers to provide additional context, e.g., min-max formulations have a long history in robust optimization and statistics.\n\n- Section 4.4 states \"Interestingly, we find that the accuracy on clean images can be significantly boosted from 62.3% to 73%.\". It would be good to add context and state what accuracy the network achieves with standard training.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the properties of adversarial training in the large scale setting. The reviewers found the properties identified by the paper to be of interest to the ICLR community - in particular the robustness community. We encourage the authors to release their models to help jumpstart future work building on this study.
Decision: Accept (Poster)