Robust Local Features for Improving the Generalization of Adversarial Training
Authors: Chuanbiao Song
Kun He
Jiadong Lin
Liwei Wang
John E. Hopcroft
Venue:  (2020)
Keywords: adversarial robustness
adversarial training
adversarial example
deep learning

Abstract: Adversarial training has been demonstrated as one of the most effective methods for training robust models to defend against adversarial examples. However, adversarially trained models often lack adversarially robust generalization on unseen testing data. Recent works show that adversarially trained models are more biased towards global structure features. Instead, in this work, we would like to investigate the relationship between the generalization of adversarial training and the robust local features, as the robust local features generalize well for unseen shape variation. To learn the robust local features, we develop a Random Block Shuffle (RBS) transformation to break up the global structure features on normal adversarial examples. We continue to propose a new approach called Robust Local Features for Adversarial Training (RLFAT), which first learns the robust local features by adversarial training on the RBS-transformed adversarial examples, and then transfers the robust local features into the training of normal adversarial examples. To demonstrate the generality of our argument, we implement RLFAT in currently state-of-the-art adversarial training frameworks. Extensive experiments on STL-10, CIFAR-10 and CIFAR-100 show that RLFAT significantly improves both the adversarially robust generalization and the standard generalization of adversarial training. Additionally, we demonstrate that our models capture more local features of the object on the images, aligning better with human perception.

Published as a conference paper at ICLR 2020
ROBUST LOCAL FEATURES FOR IMPROVING THE GENERALIZATION OF ADVERSARIAL TRAINING
Chuanbiao Song & Kun He "& Jiadong Lin School Computer Science and Technology
Liwei Wang
School of Electronics Engineering and Computer Sciences, Peking University Peking, China
Huazhong University of Science and Technology
Wuhan, 430074, China {cbsong, brooklet jdlin felintehust... edu.co
wanglwecis pku edu.
John E. Hoperoft
Department of Computer Science
Cornell University, NY 14853, USA @eh@cs. wornell.eee..ooooo edd
ABSTRACT
Adversarial training has been demonstrated one of the most effective methods for training robust models to defend against adversarial examples. However, ad- versarially trained models often lack adversarially robust generalization on unseen testing data. Recent works show that adversarially trained models are more biased towards global structure features. Instead, this work, we would like to inves- tigate the relationship between the generalization of adversarial training and the robust local features, as the robust local features generalize well for unseen shape variation. To learn the robust local features, develop Random Block Shuffle (RBS) transformation to break up the global structure features on normal adver- sarial examples. We continue propose Features for Adversarial Training (RLFAT), which first learns the robust local fea- tures by adversarial training the RBS-transformed adversarial examples, and then transfers the robust local features into the training of normal adversarial ex- amples. To demonstrate the generality of our argument, implement RLFAT in currently state state-of-the of-the-art adversarial training frameworks. Extensive experiments on STL-10, CIFAR-10 and CIFAR 100 show that RLFAT significantly improves both the adversarially robust generalization and the standard generalization of ad- versarial training. Additionally, we demonstrate that our models capture more local features of the object the images, aligning better with human perception.
approach called Robust Local
INTRODUCTION
Deep learning has achieved remarkable performance breakthrough on various challenging bench- marks in machine learning fields, such image classification (Krizhevsky et al., (012) and speech recognition (Hinton et al. 2012). However, recent studies (Szegedy et al. 2014] Goodfellow al.) 2015) have revealed that deep neural network models strikingly susceptible adversarial amples, in which small perturbations around the input are sufficient to mislead the predictions of the target model. Moreover, such perturbations are almost imperceptible to humans and often transfer
across diverse models to achieve black-box attacks (Papernot et al. 2017 Liu et al. 2017 Wang
et al. 2019 Lin et al. 2020).
Though the emergence of adversarial examples has received significant attention and led to various defend approaches for developing robust models (Madry al. 2018 Dhillon al., 2018 Wang & Yu 2019 Song et al. 2019 Zhang et al., (2019)), many proposed defense methods provide few benefits for the true robustness but mask the gradients which most attacks rely (Carlini Wag-
ner 27 Athalye et al. 2018 Uesato et al. 2018 Li et al. 2019). Currently, one of the best
"Corresponding author

---

Published as a conference paper at ICLR 2020
techniques to defend against adversarial attacks (Athalye et al. 2018 Li al. 2019 is adversarial training (Madry et al. 2018: Zhang et al. 2019a), which improves the adversarial robustness by injecting adversarial examples into the training data.
Among substantial works of adversarial training, there still remains a big robust generalization gap between the training data and the testing data (Schmidt et al. 2018 Zhang et al. 2019b Ding al.,
2019 Zhai et al., 2019). The robustness of adversarial training fails to generalize on unseen testing data. Recent works (Geirhos al. 2019 Zhang & Zhu, 2019) further show that adversarially trained models capture more on global structure features but normally trained models are more biased to- wards local features. In intuition, global structure features tend to be robust against adversarial perturbations but hard to generalize for unseen shape variations, instead, local features generalize well for unseen shape variations but are hard to generalize on adversarial perturbation. It naturally raises an intriguing question for adversarial training:
For adversarial training, possible to learn the robust local features, which have better adver- sarially robust generalization and better standard generalization?
To address this question, we investigate the relationship between the generalization of adversarial training and the robust local features, and advocate for learning robust local features for adversarial training. Our main contributions are as follows:
To our knowledge, this is the first work that sheds light on the relationship between adversarial training and robust local features. Specifically, we develop Random Block Shuffle (RBS) transformation to study such relationship by breaking up the global structure features on normal adversarial examples.
We propose novel method called Robust Local Features for Adversarial Training (RLFAT), which first learns the robust local features, and then transfers the information robust local features into the training on normal adversarial examples.
To demonstrate the generality of our argument, we implement RLFAT in two currently state- of-the-art adversarial training frameworks, PGD Adversarial Training (PGDAT) (Madry et al. 2018) and TRADES (Zhang et al., 2019a). Empirical results show consistent and substantial improvements for both adversarial robustness and standard accuracy several standard datasets. Moreover, the salience maps of our models on images tend to align better with human perception.
PRELIMINARIES
In this section, we introduce some notations and provide brief description current advanced methods for adversarial attacks and adversarial training.
2.1 NOTATION
Let Fxx) be probabilistic classifier based on neural network with the logits function f(x) and the probability distribution p=(-(z). Let C(F;,, y) be the cross entropy loss for image classification. The goal of the adversaries to find an adversarial example x' E (x) == {a' : xllp S c} in the lp norm bounded perturbations, where denotes the magnitude of the perturbations. In this paper, we focus on p = do to align with previous works.
2.2 ADVERSARIAL ATTACKS
Projected Gradient Descent. Projected Gradient Descent (PGD) Madry et al. 2018) is stronger iterative variant of Fast Gradient Sign Method (FGSM) (Goodfellow et al. (215)) which iteratively solves the optimization problem max ||ll <e L (F3aa, y) with a step size a:
80 u (=))
(1)
2 ++
18 (1) (s* ssgn (V=C(F;,,))l.))
where L/ denotes the uniform distribution, and I (w) indicates the projection of the 8 (2))
2

---

Published as a conference paper at ICLR 2020
Carlini- Maanerr attack. Carlini- Wagner attack (CW) (2017b) is sophisticated method directly solve for the adversarial example ande by using an auxiliary variable w:
x advv = 0.5 (tanh(w) + 1).
(2)
The objective function optimize the auxiliary variable w is defined as:
min Ladv z1|| x2|l+00 (x
where F(zadr)) max (Sgorue (2nder max (Ji(zadu): + y the confidence gap between the adversarial class and the true class.
The constant l: controls
Wattack. Wattack (Li et al. 2019) a derivative-free black-box adversarial attack and it breaks many of the defense methods based on gradient masking. The basic idea is to learn probability density distribution over a small region centered around the clean input, such that sample drawn from this distribution likely to be adversarial example.
2.3 ADVERSARIAL TRAINING
Despite a wide range of defense methods, Athaayy al. (2018) and Li et al. (2019) have broken most previous defense methods (Dhillon et al. 2018 Buckman et al. 2018 Wang & Yu 2019 Zhang et al. (2019)), and revealed that adversarial traiming remains one of the best defense method. The basic idea of adversarial training to solve the min-max optimization problem, as shown in Eq. (4):
min max S
L (F;
Here we introduce two currently ofte-of-the--eee art adversarial training frameworks.
PGD adversarial training. PGD Adversarial Training (PGDAT) (Madry et al. 2018 leverages the PGD attack to generate adversarial examples, and trains only with the adversarial examples. The objective function is formalized follows:
(PGD(F;x,g) = L(F; L(F: y)
(5)
where TPGD obtained via the PGD attack on the cross entropy L(F: y).
TRADES.
Zhang et al. (2019a) propose TRADES to specifically maximize the trade-off of adver- sarial training between adversarial robustness and standard accuracy by optimizing the following regularized surrogate loss:
STRADES(F;2,,) = L(F: y) + ADKL ( (pr(1x) pr (-Ixpon(>)) )
(6)
where [x is obtained via the PGD attack the KL--divergence DEL (pr(1la)) || pr (-1x') and À hyper-parameter to control the trade-off between adversarial robustness and standard accuracy.
ROBUST LOCAL FEATURES FOR ADVERSARIAL TRAINING
Unlike adversarially trained models, normally trained models features but vulnerable to adversarial examples (Geirhos et al. 2019). It indicates that, in contrast global structural features, local features seems be more gell-generalized but less robust against adversarial perturbation. Based on the basic observation, in this work, we focus on the learning of robust local features on adversarial training, and propose a novel form of adversarial training called RLFAT that learns the robust local features and transfers the robust local features into the training of normal adversarial examples. this way, our adversarially trained models not only yield strong robustness against adversarial examples but also show great generalization on unseen testing data.
biased towards the local
3.1 ROBUST LOCAL FEATURE LEARNING
It's known that adversarial training tends to capture global structure features so as to increase in- variance against adversarial perturbations (Zhang & Zhu 2019 Ilyas et al. 2019). To advocate for
3

---

Published as a conference paper at ICLR 2020
the learning of robust local features during adversarial training, we propose simple and straight- forward image transformation called Random Block Shuffle (RBS) to break up the global structure features of the images, at the same time retaining the local features. Specifically, for input image, we randomly split the target image into k blocks horizontally and randomly shuffle the blocks, and then perform the same split-shuffle operation vertically the resulting image. As illustrated in RBS transformation can destroy the global structure features of the images to some extent and retain the local features of the images.
Figure [
1/g
1/1
h1
h2
h2 shuftle
shuftle
split
hg
ha
h
Original image
Vertical shufflc
Horizontal split
RBS(x)
Figure 1: Illustration of the RBS transformation for k 3. For better understanding the RBS
transformation, we paint the split image blocks with different colors.
Then we apply the RBS transformation adversarial training. Different from normal adversarial training, we use the RBS-Iransformed adversarial examples rather than normal adversarial examples as the adversarial information to encourage the models to learn robust local features. Note that we only use the RBS transformation as tool to learn the robust local features during adversarial training and will not RBS transformation in the inference phase. refer to the form of adversarial training as RBS Adversarial Training (RBSAT).
To demonstrate the generality of argument, consider currently state-of-the-art ad- versarial training frameworks, PGD Adversarial Training (PGDAT) (Madry et al. 2018) and TRADES (Zhang et al., 2019a), demonstrate the effectiveness of the robust local features.
We use the following loss function as the alternative to the objective function of PGDAT:
CRLFL CPGDAT(F; y) = (BFF:BBS(Z))))) v)
(7)
where RBS(.) denotes the RBS transformation; ZPGD is obtained via the PGD attack the cross entropy ((F;I,y),
Similarly, the following loss function the alternative the objective function of TRADES:
CRLFL TRADDES (F; v) L(F; xx,)) + ADRL [p=(-1z) || pr (-IRBS (z/Gn(æ))) 1.
(8)
where [x] is obtained via the PGD attack on the KL-divergence DKL PF (+2)) I (+|x") ).
3.2 ROBUST LOCAL FEATURE TRANSFER
Since the type of input images in the training phase and the inference phase is different (RBS trans- formed images for training, versus original images for inference), we consider to transfer the knowl- edge of the robust local features learned by RBSAT to the normal adversarial examples. Specifically, we present a knowledge transfer scheme, called Robus! Local Feature Transfer (RLFT). The goal of RLFT is learn the representation that minimizes the feature shift between the normal adversarial examples and the RBS- res-transforeed adversarial examples.
In particular, we apply RLFT on the logit layer for high-level feature alignment. Formally, the ob- jective functions of robust local feature transfer for PGDAT and TRADES are formalized as follows, respectively:
RLFT (F; (,, (tapg))___(((Baaeeeee (|((RB((FFan)) - (faten))))ff(foooe (*fan))l
RLLT (F; y) II.(RBBS (~ [x)
where f(-) denotes the mapping of the logit layer, and 11:112 denotes the squared Fuclidean norm.
4

---

Published as a conference paper at ICLR 2020
3.3 OVERALL OBJECTIVE FUNCTION
Since the quality of robust local feature transfer depends on the quality of the robust local features learned by RBSAT, integrate RBSAT and RLFT into end-to-end training framework, which we refer to as RLFAT (Robust Local Features for Adversarial Training). The general training process of RLFAT summarized Algorithm I Note that the computational cost of RBS transformation (line 7) is negligible in the total computational cost.
Algorithm Robust Local Features for Adversarial Training (RLFAT).
1: Randomly initialize network F(x);
2: Number of iterations 0;
3: repeat
4: f f + 1;
5: Read minibatch of data { Lm from the training set;
6:
Generate the normal adversarial examples {aqdu add
7: Obtain the RBS- EeS-transformed adversarial examples (RBS(x1"")... )..... RBS (add))
8: Calculate the overall loss following Eq. (TO).
9: Update the parameters of network F through back propagation;
10: until the training converges.
We implement RLFAT in two currently state-of-the-an adversarial training frameworks, PGDAT and TRADES, and have new objective functions to learn the robust and well-generalized feature representations, which we call RLFATp and RLFATT:
CRLEAT, (FLEAT,(F;225))
CRLFF LPGDAT (F; 22
RLFT PGDAT (F;
(10)
CRLEATT =
CRLFL TRADEES (F;x,)) (F:ADESSFFFF,00) + TTRDESS (F;2,
CLFF
where y) is a hyper-parameter to balance the two terms.
4 EXPERIMENTS
In this section, to validate the effectiveness of RLFAT, we empirically evaluate our two implementa- tions, denoted RLFATr and RLFATT, and show that our models make significant improvement on both robust accuracy and standard accuracy standard benchmark datasets, which provides strong support for our main hypothesis. Codes are available online
4.1 EXPERIMENTAL SETUP
Baselines. Since most previous defense methods provide few benefit true adversarially robust- ness (Athalye et al., 2018 Li et al. 2019), we compare the proposed methods with state-of-the- art adversarial training defenses, PGD Adversarial Training (PGDAT) (Madry et al.) 2018) and TRADES (Zhang et al. 2019a).
Adversarial setting. We consider two attack settings with the bounded txx. norm: the white-box attack setting and the black-box attack setting. For the white-box attack setting, consider existing strongest white-box attacks: Projected Gradient Descent (PGD) (Madry et al.) 2018) and Carlini- Wagner attack (CW) (Carlini & Wagner 2017b) For the black-box attack setting, perform the powerful black-box box attack, N Wattack (Li et al. 2019), on a sample of 1,500 test inputs as it time- consuming.
Datasets. We compare the proposed methods with the baselines on widely used benchmark datasets, namely CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton 2009). Since adversari- ally robust generalization becomes increasingly hard for high dimensional data and little training data (Schmidt al. 2018), we also consider one challenging dataset: STL-10 (Coates et al.). which contains 5, 000 training images, with 96 x 96 pixels per image.
hetes: //g/ htt.
5

---

Published as a conference paper at ICLR 2020
Neural networks. For STL-10, the architecture use is wide ResNet 40-2 Zagoruyko & Ko- modakis 2016). For CIFAR-10 and CIFAR-IB -F0, we use a wide ResNet w32-10. For all datasets, scale the input images the range of [0, 1].
Hapeer-aarametess To avoid posting much concentrate on optimizing the hyper-parametent for all datasets, set the hyper-parameter À in TRADES as set the hyper-parameter v) in RLFATp as 0.5, and set the nyper-paramett
in RLFATT as 1. For the training jobs of all our models, we set the nyper-parametees of the RBS transformation as 2. More details about the hyper-parameters are provided in Appendix A
4.2 EVALUATION RESULTS
We first validate our main hypothesis: for adversarial training, is it possible to learn the robust local features that have better adversarially robust generalization and better standard generalization?
In Table L we compare the accuracy of RLFATp and RLFATT with the competing baselines on three standard datasets. The proposed methods lead to consistent and significant improvements on adversarial robustness as well as standard accuracy over the baseline models on all datasets. With the robust local features, RLFATT achieves better adversarially robust generalization and better standard generalization than TRADES. RLFATp also works similarly, showing significant improvement on the robustness against all attacks and standard accuracy than PGDAT.
Table 1: The classification accuracy (%) of defense methods under white-box and black-box attacks on STL-10, CIFAR-10 and CIFAR-100.
(a) STL-10. The magnitude of perturbation is 0.03 in too norm.
Defense
No attack
PGD
CW
Nattack
PGDAT
67 .5.
30.00
31.97
34.80
TRADES
65.24
38,99
38.35
42.07
RLFATP
71.47
38.42
38.42
44.80
RLFATT
72.38
43.36
39.31
48.13
(b) CIFAR-10. The magnitude of perturbation is 0.03 in &oo
Defense
No attack
PGD
CW
Nattack
PGDAT
82.96
46.19
46.41
46.67
TRADES
80.35
50.95
49.80
52,47
RLFATp
84.77
52.40 58.75
53.97
54.60
RLFAT
82.72
51.94
54.60
(c) CIFAR-100. The magnitude of perturbation is 0.03 in foc norm.
Defense
No attack
PGD
CW
N Nattack
PGDAT
55.86
23.32
22,87
22.47
TRADES
52.13
27.26
24.66
25.13
RLFATP
56.70
31.99
29.04
32.53
RLFATT
58.96
31.63
27.54
30.86
The results demonstrate that, the robust local features can significantly improve both the adver- sarially robust generalization and the standard generalization over the state-of-the-art adversarial training frameworks, and strongly support hypothesis. That is, for adversarial training, it is possible to learn the robust local features, which have better robust and standard generalization.
4.3 Loss SENSITIVITY UNDER DISTRIBUTION SHIFT
Motivation. Ding et al. (209 and Zhang et al. (2019)) found that the effectiveness of adversarial training is highly sensitive to the "semantic-loss" shift of the test data distribution, such gamma mapping. To further investigate the performance of the proposed methods, we quantify the smooth- ness of the models under the distribution shifts of brightness perturbation and gamma mapping.

---

Published as a conference paper at ICLR 2020
Loss sensitivity on brightness perturbation. To quantify the smoothness of models on the shift of the brightness perturbation, we propose to estimate the Lipschitz continuity constant E by using the gradients of the loss function with respect the brightness perturbation of the testing data. We adjust the brightness factor of images in the HSV (hue, saturation, value) color space, which we refer x. Var, a), where a denotes the magnitude of the brightness adjustment. The lower the value of (a) is, the smoother the loss function of the model is:
m
(a)
V ((ri, a),, Itrue)ll2
(1I)
Loss sensitivity on gamma mapping. Gamma mapping (Szeliski 2011) is a nonlinear element- wise operation used adjust the exposure of images by applying ह( x² on the original image I. Similarly, we approximate the loss sensitivity under gamma mapping, by using the gradients of the loss function with respect to the gamma mapping the testing data. A smaller value indicates smoother loss function.
m V 22L(F; 17
1
(7())
(12)
Sensitivity analysis. The results for the loss sensitivity of the adversarially trained models under brightness perturbation are reported in Table 2a where we adopt various magnitude of brightness adjustment on each testing data. In Table 2b we report the loss sensitivity of adversarially trained models under various gamma mappings. We observe that RLFATT provides the smoothest model under the distribution shifts on all the three datasets. The results suggest that, as compared to PGDAT and TRADES, both RLFATp and RLFAT show lower gradients of the models different data distributions, which we can directly attribute to the robust local features.
Table 2: The loss sensitivity of defense methods under different testing data distributions.
(a) Loss sensitivity on brightness perturbation for the adversarially trained models.
(- (0.15)
Dataset
PGDAT
TRADES
RLFATP
RLFAT,
STL-100
0.85 0.82
0.46 / 0.48
0.32 0.34
0.22 / 0.23 0.54 0.56
CIFAR-10
1.41 1.36
0.88 0,90
0.70 0.71
CIFAR- 100
3.93 3.66
2.31 / 2.12
1,25 1.31
1.00 / 0.98
(b) Loss sensitivity on gamma mapping the adversarially trained models.
(0.8) / (1.2))
Dataset
PGDAT
TRADES
RLFATP
RLFAT
STL-10 CIFAR- 10
0.77 0.79
0.44 / 0.42
0.30/ 0.29
0.21 / 0,19
1.27 1.20
0.84/ / 0.76
0.69/ 0.62
0.54 / 0,48
CIFAR-100
2.82/ 2,80
1.78/ 1.76
1.09/ 1.01
0.95 / 0,88
4.4 ABLATION STUDIES
To further gain insights on the performance obtained by the robust local features, we perform ab- lation studies to dissect the impact of various components (robust local feature learning and robust local feature transfer). As shown in Figure 2 we conduct additional experiments for the ablation studies of RLFATp and RLFAT on STI- 10, CIFAR-10 and CIFAR-100, where we report the standard accuracy over the clean data and the average robust accuracy over all the attacks for each model.
Does robust local feature learning help? We first analyze that compared adversarial training on normal adversarial examples, whether adversarial training on RBS- Res-trrnsformed adversarial ex- amples produces better generalization and more robust features. As shown in Figure 2 observe that Robust Local Features Learning (RLFL) exhibits stable improvements on both standard accu- racy and robust accuracy for RLFATp and RLFATT, providing strong support for our hypothesis.
7

---

Published as a conference paper at ICLR 2020
100
PGDAT +RUFL +RUFL+RIFT
100
PGDAT +RLEL
+ALFL+RLFT
100
PGDAT +REFL +RLFL+ALFT
80
Accuracy (1)
Accuracy (%)
RLFATp
60
Accuracy
40
40
40
20
20
20
No Attack
Attack
Attack
Attack
No Attack
Attack
TRADES RLFL RLFL-RLFFT
100
TRADES +RUFL +ALFL+RUFT
TRADES +RLFL +RLFL+RLFT
80
80
Accuracy 196)
Accuracy (8)
Accuracy (86)
60
60
60
RLPATT
40
40
40
20
0
Attack
Attack
No Attack
Attack
(a) STL.-10
(h) CIFAR-10
(c) CIFAR-100
Figure 2: Ablation studies for RLFATp and R.FATT investigate the impact of Robust Local Feature Learning (RLFL.) and Robust Local Feature Transfer (RLFT).
Does robust local feature transfer help? We further add Robust Local Feature Transfer (RLFT), the second term in Eq. (10). to get the overall loss of RLFAT. The robust accuracy further increases on all datasets for RLFATP P and RLFATT. The standard accuracy further increases also, except for RLFATp p CIFAR-100 100, but it is still clearly higher than the baseline model PGDAT. It indicates that transferring the robust local features into the training of normal adversarial examples does help promote the standard accuracy and robust accuracy in most cases.
4.5 VISUALIZING THE SALIENCE MAPS
We would like to investigate the features of the input images that the models are mostly focused on. Following the work of Zhang & Zhu (011)), we generate the salience maps using Smooth- Grad (Smilkov et al. (017) on STL-10 dataset. The key idea of SmoothGrad to average the gradients of class activation with respect to noisy copies of an input image. As illustrated in Fig- ureß] all the adversarially trained models basically capture the global structure features of the object on the images. As compared to PGDAT and TRADES, both RLFATp and RLFATT capture more local feature information the object, aligning better with human perception. Note that the images correctly classified by all these models. For more visualization results, see Appendix B
CONCLUSION AND FUTURE WORK
Differs to existing adversarially trained models that are more biased towards the global structure features of the images, in this work, we hypothesize that robust local features can improve the generalization of adversarial training. To validate this hypothesis, we propose new stream of adversarial training approach called Robust Local Features for Adversarial Training (RLFAT) and implement it in currently state-of-the-ar adversarial training frameworks, PGDAT and TRADES. We provide strong empirical support for our hypothesis and show that the proposed methods based on RLFAT not only yield better standard generalization but also promote the adversarially robust generalization. Furthermore, we show that the salience maps of our models on images tend to align better with human perception, uncovering certain unexpected benefit of the robust local features for adversarial training.
Our findings open new avenue for improving adversarial training, whereas there are still lot to explore along this avenue. First, is it possible to explicitly disentangle the robust local features from the perspective of feature disentanglement? What is the best way leverage the robust local
8

---

Published as a conference paper at ICLR 2020
Original
PGDAT
TRADES
RLFATP
RLFATT
Original
PODAT
TRADES
RLFATP
RLFATT
Figure 3: Salience maps of the four models on sampled images. For each group of images, we have the original image, and the salience maps of the four models sequentially.
features? Second, from methodological standpoint, the discovered relationship may also as an inspiration for new adversarial defenses, where not only the robust local features but also the global information is taken into account, the global information is useful for some tasks. These questions are worth investigation future work, and we hope that our observations on the benefit of robust local features will inspire more future development.
ACKNOWLEDGMENTS
This work is supported by the Fundamental Research Funds for the Central Universities (2019kfyXK/C021) and Microsoft Research Asia.
REFERENCES
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Cireumventing defenses to adversarial examples. In Proceedings of the 35th Interna- tional Conference on Machine Learning, ICML 2018, pp. 274-283, 2018.
Jacob Buckman, Aurko Roy, Colin Raffel, and Ian J. Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. 6th International Conference on Learning Representattinss ICLR 2018, 2018.
Nicholas Carlini and David A. Wagner. Adversarial examples are not easily detected: Bypassing detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec CCS 20/7, pp. 3-14, 2017a.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, SP 2017, pp. 39-57, 2017b.
Adam Coates, Andrew Y. Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011.
Guneet S. Dhillon, Kamyar Azizzadenesheli, Zachary C. Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Animashree Anandkumar. Stochastic activation pruning for robust adversarial defense. In 6th International Conference Learning Representations, ICLR 2018, 2018.

---

Published as a conference paper at ICLR 2020
Gavin Weiguang Ding, Kry Yik Chau Lui, Xiaomeng Jin, Luyu Wang, and Ruitong Huang. On the sensitivity of adversarial robustness to input data distributions. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. 7th International Conference on Learning Representations,
ICLR 2019, 2019.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, al. Deep neural networks for acoustic modeling in speech recognition. JEEE Signal processing magazine, 29, 2012.
Andrew Ilyss, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. CoRR, 0bbs//905.02111 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012, pp- 1106-1114, 2012.
Yandong Li, Lijun Li. Liqiang Wang, Tong Zhang, and Boqing Gong. NATTACK: learning the distributions of adversarial examples for an improved black-box attack on deep neural networks. In Proceedings the 36th International Conference on Machine Learning, ICML 20/9, pp. 3866- 3876, 2019.
Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E. Hoperoft. Nesterov accelerated gradient and scale invariance for adversarial attacks. In International Conference on Learning Representations, 2020. URL httttttts////eetee http:sss/ooeerrree..eee////eaaes
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial exam- ples and black-box attacks. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, 2018
Nicolas Papernot, Patrick D. McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Anan- thram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, AsiaCCS 2017, pp. 506- 519, 2017.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry, Adver- sarially robust generalization requires more data. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, pp. 5019-5031, 2018.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. CoRR, abs/1 706.03825, 2017.
Chuanbiao Song, Kun He, Liwei Wang, and John E. Hopcroft. Improving the generalization of adversarial training with domain adaptation. In 7th International Conference on Learning Repre- sentations, ICLR 2019, 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference Learning Representations, ICLR 2014, 2014.
10

---

Published as a conference paper at ICLR 2020
Richard Szeliski. Computer Vision Algorithms and Applications. Texts in Computer Science. Springer, 2011.
Jonathan Uesato, Brendan O'Donoghue, Pushmeet Kohli, and Aäron van den Oord. Adversarial risk and the dangers of evaluating against weak attacks. In Proceedings of the 35th International Conference on Machine Learning, JCML 2018, pp. 5032-5041, 2018.
Huaxia Wang and Chun-Nam Yu. A direct approach to robust deep learning using adversarial net- works. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Xiaosen Wang, Kun Hc, and John E. Hoperoft. AT-GAN: generative attack model for adversarial transferring on generative adversarial nets. CoRR, abs/1904.07793, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference 2016, BMVC, 2016.
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John E. Hoperoft, and Liwei Wang. Adversar- ially robust generalization just requires more unlabeled data. CoRR, abs/1906.00555, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, pp. 7472-7482, 2019a.
Huan Zhang, Hongge Chen, Zhao Song, Duane S. Boning, Inderjit S. Dhillon, and Cho-Jui Hsieh. The limitations adversarial training and the blind-spot attack. In 7th International Conference on Learning Representations, ICLR 2019, 2019b.
Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural net- works. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019,
pp. 7502-7511, 2019. 2019.

---

Published as a conference paper at ICLR 2020
A HYPER-PARAMETER SETTING
Here we show the details of the training hyper-parameters and the attack hyper-parameters for the experiments.
Training Hyper-parameters. For all training jobs, we use the Adam optimizer with a learning rate 0.000 and batch size of 32. For CIFAR-10 and CIFAR-100, we 79,800 steps for training. For STL-10, we run 29,700 steps for training. For STL-10 and CIFAR-100, the adversarial examples are generated with step size 0.0075. iterations. and 0.03. For CIFAR-10. the adversarial examples are generated with step size 0.0075, 10 iterations, and 0.03.
Attack Hypaer-arammetrs. For the PGD attack, we use the same attack parameters those of the training process. For the CW attack, we use PGD to minimize its loss function with a high confidence parameter (A 50) following the work of Madry et al. (2018). For the Wattack, we set the maximum number of optimization iterations to 7 = 200, b 300 for the sample size, the variance the isotropic Gaussian o² 0.01, and the learning rate v = 0.008.
B MORE FEATURE VISUALIZATION
We provide more salience maps of the adversarially trained models on sampled images in Figure 4
Original
PODAT
TRADES
RLFAT
RLFATT
Original
PODAT
TRADES
RLFATP
RLFATT
Figure 4: More Salience maps of the four models. For each group of images, have the original image, and the salience maps of the four models sequentially.
12


Reviews:
[{"review_id": "rkxiT9QatH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work suggests reshuffle images blocks of adversarial examples during adversarial training, in order to improve the generalization performance on benign and adversarial test samples.  The main method is based on the hypothesis in [Zhang et al 2019], [Ilyas et al 2019].  The assumption claims that robust models rely on global structural features, and non-robust models rely on local features. Thus, the work tries to learn local robust features, by cutting and reshuffling the image blocks. Overall the idea is interesting and the paper is well written .\n\nHowever, there are some concerns about the presentation and the main methodology:\n1.\tCan the paper give more explanation on the purpose of inserting the feature transfer term in the objective function? What is the difference of the proposed one with directly minimizing the loss on both original PGD image and reshuffled image?\n2.\tFor CIFAR10, TRADEs and PGDAT’s performance in the result is not as good as the ones shown in their original works, which is comparable to the performance of the proposed RLFAT method. More discussions are needed, otherwise the experimental results are not convincing. \n3.\tMore intuitions are needed  on what local and global features are, and why training on the reshuffled images can help learn generalizable robust local features. \n\nOverall the paper is easy to understand, but we suggest that more insight should be given on the success of the proposed method.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lZuixYcS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is interested in robustness w.r.t. adversarial exemples. \n\nThe authors note that:\n* features reflecting the global structure are more robust wrt adversarial perturbations, but generalize less;\n* features reflecting the local structure generalize well, but are less robust wrt adversarial perturbations. \nIn hindsight, these claims are intuitive: adversarial perturbations and unseen shape variations are of the same flavor; one should resist to both or handle both, with the difference that the latter is bound to occur (and should be handled) and the former is undesired (and should be resisted). \n\nThe goal thus becomes to define local features that are robust. \n\nThe proposed approach is based on \n* enforcing the invariance of the intermediate representation through shuffling the blocks of the training images; \n* building normal adversarial images x' and deriving the block shuffling RBS such that the x' and RBS(x') are most similar w.r.t. the logit layer\n* adding these RBS(x') to the training set;\n\nThe idea is nice; the experiments are well conducted and convincing (except for the addition of uniform noise, which is unrealistic; you might consider instead systematic noise mimicking a change of light);\nI'd like more details about:\n* The computational cost of line 7 in algo (deriving the best RBS).\n\nYou might want to discuss the relationship between the proposed approach and the multiple instance setting (as if the image was a bunch of patches). ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Byg-szbhtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a new approach to improve the robustness of CNNs against adversarial examples.\nThe recent studies show that CNNs capture local features, which can be easily affected by the adversarial perturbations.\nThus, in the paper, the authors proposed to train CNNs so that they can capture local features that are robust against the adversarial perturbations.\nThe difficulty here is that existing adversarial training algorithms tend to bias CNNs to ignore local features and to capture only global features.\nTo avoid this unfavorable property of the adversarial training, the authors proposed the random block shuffle (RBS) that intensionally destroys the global feature of the images.\nThe authors demonstrated that combining RBS with the existing adversarial training algorithms can lead to robust CNNs.\n\nI found the paper well-written and the idea is easy to follow.\nEspecially, the use of RBS seems to be an interesting idea.\nAs a small downside, the proposed approach looks rather straightforward, and I expect to see any theoretical foundations if possible.\n\n### Updated after author response ###\nIn summary, the contribution of this study is in twofolds.\n1. Proposed an algorithm for learning robust local features.\n2. Demonstrated that learning robust local features is effective to improve the robustness of the model.\nThe possible downside is\n3. The proposed approach looks straightforward.\nOverall, I like the paper (especially for the reason 2 above), and therefore keep my score.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Earlier work suggests that adversarial examples exploit local features and that more robust models rely on global features. The authors propose to exploit this insight by performing data augmentation in adversarial training, by cutting and reshuffling image block. They demonstrate the idea empirically and witness interesting gains. I think the technique is an interesting contribution, but empirically and as a tool.

Decision: Accept (Poster)