Title: CASCADE ADVERSARIAL MACHINE LEARNING REG-ULARIZED WITH A UNIFIED EMBEDDING

Abstract: Injecting adversarial examples during training, known as adversarial training, can improve robustness against one-step attacks, but not for unknown iterative attacks. To address this challenge, we first show iteratively generated adversarial images easily transfer between networks trained with the same strategy. Inspired by this observation, we propose cascade adversarial training, which transfers the knowledge of the end results of adversarial training. We train a network from scratch by injecting iteratively generated adversarial images crafted from already defended networks in addition to one-step adversarial images from the network being trained. We also propose to utilize embedding space for both classification and low-level (pixel-level) similarity learning to ignore unknown pixel level perturbation. During training, we inject adversarial images without replacing their corresponding clean images and penalize the distance between the two embeddings (clean and adversarial). Experimental results show that cascade adversarial training together with our proposed low-level similarity learning efficiently enhances the robustness against iterative attacks, but at the expense of decreased robustness against one-step attacks. We show that combining those two techniques can also improve robustness under the worst case black box attack scenario.

1. INTRODUCTION:

Injecting adversarial examples during training (adversarial training), (Goodfellow et al., 2015; Kurakin et al., 2017; Huang et al., 2015) increases the robustness of a network against adversarial attacks. The networks trained with one-step methods have shown noticeable robustness against onestep attacks, but, limited robustness against iterative attacks at test time. To address this challenge, we have made the following contributions:

Cascade adversarial training: We first show that iteratively generated adversarial images transfer well between networks when the source and the target networks are trained with the same training method. Inspired by this observation, we propose cascade adversarial training which transfers the knowledge of the end results of adversarial training. In particular, we train a network by injecting iter FGSM images (section 2.1) crafted from an already defended network (a network trained with adversarial training) in addition to the one-step adversarial images crafted from the network being trained. The concept of using already trained networks for adversarial training is also introduced in (Tramèr et al., 2017) . In their work, purely trained networks are used as another source networks to generate one-step adversarial examples for training. On the contrary, our cascade adversarial training uses already defended network for iter FGSM images generation.

Low level similarity learning: We advance the previous data augmentation approach (Kurakin et al., 2017) by adding additional regularization in deep features to encourage a network to be insensitive to adversarial perturbation. In particular, we inject adversarial images in the mini batch without replacing their corresponding clean images and penalize distance between embeddings from the clean and the adversarial examples. There are past examples of using embedding space for learning similarity of high level features like face similarity between two different images (Schroff et al., 2015; Parkhi et al., 2015; Wen et al., 2016) . Instead, we use the embedding space for learning similarity of the pixel level differences between two similar images. The intuition of using this regularization is that small difference on input should not drastically change the high level feature representation.

Analysis of adversarial training::

We train ResNet models (He et al., 2016) on MNIST (LeCun & Cortes, 2010) and CIFAR10 dataset (Krizhevsky, 2009) using the proposed adversarial training. We first show low level similarity learning improves robustness of the network against adversarial images generated by one-step and iterative methods compared to the prior work. We show that modifying the weight of the distance measure in the loss function can help control trade-off between accuracies for the clean and adversarial examples.

Together with cascade adversarial training and low-level similarity learning, we achieve accuracy increase against unknown iterative attacks, but at the expense of decreased accuracy for one-step attacks. We also show our cascade adversarial training and low level similarity learning provide much better robustness against black box attack.

2. BACKGROUND ON ADVERSARIAL ATTACKS:

2.1 ATTACK METHODS One-step fast gradient sign method (FGSM), referred to as "step FGSM", generates adversarial image X adv by adding sign of the gradients w.r.t. the clean image X multiplied by ∈ [0, 255] as shown below (Goodfellow et al., 2015) :

X adv = X + sign(∇ X J(X, y true ))

One-step target class method generates X adv by subtracting sign of the gradients computed on a target false label as follows:

X adv = X -sign(∇ X J(X, y target ))

We use least likely class y LL as a target class and refer this method as "step ll".

Basic iterative method, referred to as "iter FGSM", applies FGSM with small α multiple times.

X adv 0 = X, X adv N = Clip X, X adv N -1 + α sign(∇ X adv N -1 J(X adv N -1 , y true ))

We use α = 1, number of iterations N to be min ( + 4, 1.25 ) . Clip X, is elementwise clipping function where the input is clipped to the range [max(0, X -), min(255, X + )].

Iterative least-likely class method, referred to as "iter ll", is to apply "step ll" with small α multiple times.

X adv 0 = X, X adv N = Clip X, X adv N -1 -α sign(∇ X adv N -1 J(X adv N -1 , y LL ))

Carlini and Wagner attack (Carlini & Wagner, 2017 ) referred to as "CW" solves an optimization problem which minimizes both an objective function f (such that attack is success if and only if f (X adv ) < 0) and a distance measure between X adv and X.

Black box attack is performed by testing accuracy on a target network with the adversarial images crafted from a source network different from the target network. Lower accuracy means successful black-box attack. When we use the same network for both target and source network, we call this as white-box attack.

2.2. DEFENSE METHODS:

Adversarial training (Kurakin et al., 2017) : 

is

Transferability (step attack)::

We first observe that high robustness against one-step attack between defended networks (R20 K2 -> R20 K ), and low robustness between undefended networks (R20 2 -> R20). This observation shows that error surfaces of neural networks are driven by the training method and networks trained with the same method end up similar optimum states.

It is noteworthy to observe that the accuracies against step attack from the undefended network (R20 2 ) are always lower than those from defended network (R20 K2 ). Possible explanation for this would be that adversarial training tweaks gradient seen from the clean image to point toward weaker adversarial point along that gradient direction. As a result, one-step adversarial images from defended networks become weaker than those from undefended network.

Transferability (iterative attack): We observe "iter FGSM" attack remains very strong even under the black box attack scenario but only between undefended networks or defended networks. This is because iter FGSM noises (X adv -X) from defended networks resemble each other. As shown in figure 1 , we observe higher correlation between iter FGSM noises from a defended network (R20 K ) and those from another defended network (R20 K2 ). the end results of adversarial training. In particular, we train a network by injecting iter FGSM images crafted from already defended network in addition to the one-step adversarial images crafted from the network being trained.

Difficulty of defense:



3.3. REGULARIZATION WITH A UNIFIED EMBEDDING:

We advance the algorithm proposed in (Kurakin et al., 2017) by adding low level similarity learning. Unlike (Kurakin et al., 2017) , we include the clean examples used for generating adversarial images in the mini batch. Once one step forward pass is performed with the mini batch, embeddings are followed by the softmax layer for the cross entropy loss for the standard classification. At the same time, we take clean embeddings and adversarial embeddings, and minimize the distance between the two with the distance based loss.

The distance based loss encourages two similar images (clean and adversarial) to produce the same outputs, not necessarily the true labels. Thus, low-level similarity learning can be considered as an unsupervised learning. By adding regularization in higher embedding layer, convolution filters gradually learn how to ignore such pixel-level perturbation. We have applied regularization on lower layers with an assumption that low level pixel perturbation can be ignored in lower hierarchy of networks. However, adding regularization term on higher embedding layer right before the softmax layer showed best performance. The more convolutional filters have chance to learn such similarity, the better the performance. Note that cross entropy doesn't encourage two similar images to produce the same output labels. Standard image classification using cross entropy compares ground truth labels with outputs of a network regardless of how similar training images are.

The entire training process combining cascade adversarial training and low level similarity learning is shown in figure 2. We define the total loss as follows:

Loss = 1 (m -k) + λk m-k i=1 L(X i |y i ) + λ k i=1 L(X adv i |y i ) + λ 2 k i=1 L dist (E adv i , E i ) E i and E adv i

are the resulting embeddings from X i and X adv i , respectively. m is the size of the mini batch, k (≤ m/2) is the number of adversarial images in the mini batch. λ is the parameter to control the relative weight of classification loss for adversarial images. λ 2 is the parameter to control the relative weight of the distance based loss L dist in the total loss.

Bidirectional loss minimizes the distance between the two embeddings by moving both clean and adversarial embeddings as shown in the left side of the figure 3 . We tried N = 1, 2 and found not much difference between the two. We report the results with N = 2 for the rest of the paper otherwise noted. When N = 2, L dist becomes L2 loss.

L dist (E adv i , E i ) = ||E adv i -E i || N N , N ∈ 1, 2 i = 1, 2, ..., k

Pivot loss minimizes the distance between the two embeddings by moving only the adversarial embeddings as shown in the right side of the figure 3 .

L dist (E adv i |E i ) = ||E adv i -E i || N N , N ∈ 1, 2 i = 1, 2, ..., k

In this case, clean embeddings ( E i ) serve as pivots to the adversarial embeddings. In particular, we don't back-propagate through the clean embeddings for the distance based loss. The intuition behind the use of pivot loss is that the embedding from a clean image can be treated as the ground truth embedding.

4. LOW LEVEL SIMILARITY LEARNING ANALYSIS:



4.1. EXPERIMENTAL RESULTS ON MNIST:

We first analyze the effect of low level similarity learning on MNIST. We train ResNet models (He et al., 2016) with different methods (standard training, Kurakin's adversarial training and adversarial training with our distance based loss). Experimental details can be found in Appendix A.

Table 2 shows the accuracy results for MNIST test dataset for different types of attack methods.

As shown in the table, our method achieves better accuracy than Kurakin's method for all types of attacks with a little sacrifice on the accuracy for the clean images. Even though adversarial training is done only with "step ll", additional regularization increases robustness against unknown "step FGSM", "iter ll", "iter FGSM" and CW L ∞ attacks. This shows that our low-level similarity learning can successfully regularize the one-step adversarial perturbation and its vicinity for simple image classification like MNIST.

4.2. EMBEDDING SPACE VISUALIZATION:

To visualize the embedding space, we modify 20-layer ResNet model where the last fully connected layer (64x10) is changed to two fully connected layers (64x2 and 2x10). We re-train networks with standard training, Kurakin's method and our pivot loss on MNIST. 1 In figure 4 , we draw embeddings (dimension=2) between two fully connected layers. As seen from this figure, adversarial images from the network trained with standard training cross the decision boundary easily as increases. With Kurakin's adversarial training, the distances between clean and adversarial embeddings are minimized compared to standard training. And our pivot loss further minimizes distance between the clean and adversarial embeddings. Note that our pivot loss also decreases absolute value of the embeddings, thus, higher λ 2 will eventually result in overlap between distinct embedding distributions. We also observe that intra class variation of the clean embeddings are also minimized for the network trained with our pivot loss as shown in the scatter plot in figure 4 (c) . We train 20-layer ResNet models with pivot loss and various λ 2 s for CIFAR10 dataset to study effects of the weight of the distance measure in the loss function. Figure 5 shows that a higher λ 2 increases accuracy of the iteratively generated adversarial images. However, it reduces accuracy on the clean images, and increasing λ 2 above 0.3 even results in divergence of the training. This is because embedding distributions of different classes will eventually overlap since absolute value of the embedding will be decreased as λ 2 increases as seen from the section 4.2. In this experiment, we show that there exists clear trade-off between accuracy for the clean images and that for the adversarial images, and we recommend using a very high λ 2 only under strong adversarial environment. We further study the transferability of iter FGSM images between various architectures. To this end, we first train 56-layer ResNet networks (Kurakin's, pivot loss) with the same initialization. Then we train another 56-layer ResNet network (Kurakin's) with different initialization. We repeat the training for the 110-layer ResNet networks. We measure correlation between iter FGSM noises from different networks.

5. CASCADE ADVERSARIAL TRAINING ANALYSIS:

Figure 6 (a) shows correlation between iter FGSM noises crafted from Kurakin's network and those from Pivot network with the same initialization. Conjectured from (Kurakin et al., 2017) , we observe high corre- 

5.2. WHITE BOX ATTACK ANALYSIS:

We first compare a network trained with Kurakin's method and that with pivot loss. We train 110layer ResNet models with/without pivot loss and report accuracy in table 3. We observe our lowlevel similarity learning further improves robustness against iterative attacks compared to Kurakin's adversarial training. However, the accuracy improvements against iterative attacks (iter FGSM, CW) are limited, showing regularization effect of low-level similarity learning is not sufficient for the iterative attacks on complex color images like CIFAR10. This is different from MNIST test cases where we observed significant accuracy increase for iterative attacks only with pivot loss. We observe label leaking phenomenon reported in (Kurakin et al., 2017) happens even though we don't train a network with step FGSM images. Additional analysis for this phenomenon is explained in Appendix D.

Next, we train a network from scratch with iter FGSM examples crafted from the defended network, R110 P . We use the same initialization used in R110 P as discussed in 5.1. In particular, iter FGSM images are crafted from R110 P with CIFAR10 training images for = 1,2, ..., 16, and those are used randomly together with step ll examples from the network being trained. We train cascade networks with/without pivot loss. We also train networks with ensemble adversarial training (Tramèr et al., 2017) with/without pivot loss for comparison. The implementation details for the trained models can be found in Appendix B.

We find several meaningful observations in table 3. First, ensemble and cascade models show improved accuracy against iterative attack although at the expense of decreased accuracy for onestep attacks compared to the baseline defended network (R110 K ). Additional data augmentation from other networks enhances the robustness against iterative attack, weakening label leaking effect caused by one-step adversarial training.

Second, our low-level similarity learning (R110 P,E , R110 P,C ) further enhances robustness against iterative attacks including fully unknown CW attack (especially for =4). Additional knowledge learned from data augmentation through cascade/ensemble adversarial training enables networks to learn partial knowledge of perturbations generated by an iterative method. And the learned iterative perturbations become regularized further with our low-level similarity learning making networks robust against unknown iterative attacks. During this process, clean embeddings from other classes might also be moved toward the decision boundary which results in decreased accuracy for the clean images.

5.3. BLACK BOX ATTACK ANALYSIS:

We finally perform black box attack analysis for the cascade/ensemble networks with/without pivot loss. We report black box attack accuracy with the source networks trained with the same method, but with different initialization from the target networks. The reason for this is adversarial examples transfer well between networks trained with the same strategy as observed in section 3.1. 

6. CONCLUSION:

We performed through transfer analysis and showed iter FGSM images transfer easily between networks trained with the same strategy. We exploited this and proposed cascade adversarial training, a method to train a network with iter FGSM adversarial images crafted from already defended networks. We also proposed adversarial training regularized with a unified embedding for classification and low-level similarity learning by penalizing distance between the clean and their corresponding adversarial embeddings. Combining those two techniques (low level similarity learning + cascade adversarial training) with deeper networks further improved robustness against iterative attacks for both white-box and black-box attacks.

However, there is still a gap between accuracy for the clean images and that for the adversarial images. Improving robustness against both one-step and iterative attacks still remains challenging since it is shown to be difficult to train networks robust for both one-step and iterative attacks simultaneously. Future research is necessary to further improve the robustness against iterative attack without sacrificing the accuracy for step attacks or clean images under both white-box attack and black-box attack scenarios.

Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In Computer Vision -ECCV 2016 -14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016 , Proceedings, Part VII, pp. 499-515, 2016 . doi: 10.1007/978-3-319-46478-7 31 . URL http://dx.doi.org/10.1007/ 978-3-319-46478-7_31.

A EXPERIMENTAL SETUP:

We scale down the image values to [0,1] and don't perform any data augmentation for MNIST. For CIFAR10, we scale down the image values to [0,1] and subtract per-pixel mean values. We perform 24x24 random crop and random flip on 32x32 original images. We generate adversarial images with "step ll" after these steps otherwise noted.

We use stochastic gradient descent (SGD) optimizer with momentum of 0.9, weight decay of 0.0001 and mini batch size of 128. For adversarial training, we generate k = 64 adversarial examples among 128 images in one mini-batch. We start with a learning rate of 0.1, divide it by 10 at 4k and 6k iterations, and terminate training at 8k iterations for MNIST, and 48k and 72k iterations, and terminate training at 94k iterations for CIFAR10. 3We also found that initialization affects the training results slightly as in (Kurakin et al., 2017) , thus, we pre-train the networks 2 and 10 epochs for MNIST and CIFAR10, and use these as initial starting points for different configurations. We use max e = 0.3*255 and 16 for MNIST and CIFAR10 respectively.

B MODEL DESCRIPTIONS:

We summarize the model names used in this paper in 7 : Argument to the softmax vs. in test time. "step ll", "step FGSM" and "random sign" methods were used to generate test-time adversarial images. Arguments to the softmax were measured by changing for each test method and averaged over randomly chosen 128 images from CIFAR10 test-set. Blue line represents true class and the red line represents mean of the false classes. Shaded region shows ± 1 standard deviation of each line.

We draw average value of the argument to the softmax layer for the true class and the false classes to visualize how the adversarial training works as in figure 7 . Standard training, as expected, shows dramatic drop in the values for the true class as we increase in "step ll" or "step FGSM direction. With adversarial training, we observe that the value drop is limited at small and our method even increases the value in certain range upto =10. Note that adversarial training is not the same as the gradient masking.As illustrated in figure 7 , it exposes gradient information, however, quickly distort gradients along the sign of the gradient ("step ll" or "step FGSM) direction. We also observe improved results (broader margins than baseline) for "random sign" added images even though we didn't inject random sign added images during training. Overall shape of the argument to the softmax layer in our case becomes smoother than Kurakin's method, suggesting our method is good for pixel level regularization. Even though actual value of the embeddings for the true class in our case is smaller than that in Kurakin's, the standard deviation of our case is less than Kurakin's, making better margin between the true class and false classes. We observe accuracies for the "step FGSM" adversarial images become higher than those for the clean images ("label leaking" phenomenon) by training with "step FGSM" examples as in (Kurakin et al., 2017) . Interestingly, we also observe "label leaking" phenomenon even without providing true labels for adversarial images generation. We argue that "label leaking" is a natural result of the adversarial training.

D LABEL LEAKING ANALYSIS:

To understand the nature of adversarial training, we measure correlation between gradients w.r.t. different images (i.e. clean vs. adversarial) as a measure of error surface similarity. We measure correlation between gradients w.r.t. (1) clean vs. "step ll" image, (2) clean vs. "step FGSM" image, (3) clean vs. "random sign" added image, and (4) "step ll" image vs. "step FGSM" image for three trained networks (a) R20, (b) R20 K and (c) R20 P (Ours) in table 5 . Figure 8 draws average value of correlations for each case.

Meaning of the correlation: In order to make strong adversarial images with "step FGSM" method, correlation between the gradient w.r.t. the clean image and the gradient w.r.t. its corresponding adversarial image should remain high since the "step FGSM" method only use the gradient w.r.t. the clean image ( =0). Lower correlation means perturbing the adversarial image at further to the gradient (seen from the clean image) direction is no longer efficient.

Results of adversarial training::

We observe that (1) and (2) become quickly lower than (3) as increases. This means that, when we move toward the steepest (gradient) direction on the error surface, gradient is more quickly uncorrelated with the gradient w.r.t. the clean image than when we move to random direction. As a result of adversarial training, this uncorrelation is observed at a lower making one-step attack less efficient even with small perturbation. (1), ( 2) and (3) for our case are slightly lower than Kurakin's method at the same which means that our method is better at defending one-step attacks than Kurakin's.

Error surface similarity between "step ll" and "step FGSM" images: We also observe (4) remains high with higher for all trained networks. This means that the error surface (gradient) of the "step ll" image and that of its corresponding "step FGSM" image resemble each other. That is the reason why we get the robustness against "step FGSM" method only by training with "step ll" method and vice versa. (4) for our case is slightly higher than Kurakin's method at the same and that means our similarity learning tends to make error surfaces of the adversarial images with "step ll" and "step FGSM" method to be more similar.

Analysis of label leaking phenomenon: Interestingly, (2) becomes slightly negative in certain range (1 < < 3 for Kurakin's, and 1 < < 4 for Pivot (Ours) ) and this could be the possible reason for "label leaking" phenomenon. For example, let's assume that we have a perturbed image (by "step FGSM" method) at where the correlation between the gradients w.r.t. that image and the corresponding clean image is negative. Further increase of with the gradient (w.r.t. the clean image) direction actually decreases the loss resulting in increased accuracy (label leaking phenomenon). Due to the error surface similarity between "step ll" and "step FGSM" images and this negative correlation effect, however, label leaking phenomenon can always happen for the networks trained with one-step adversarial examples. 8 shows that black box attack between trained networks with the same initialization tends to be more successful than that between networks with different initialization as explained in (Kurakin et al., 2017) . In table 9, our method (R20 P 2 ) is always better at one-step and iterative black box attack from defended networks (R20 K , R20 P ) and undefended network (R20) than Kurakin's method (R20 B2 ). However, it is hard to tell which method is better than the other one as explained in the main paper. where, the function f is defined such that attack is success if and only if f (X + δ) < 0, δ is the target perturbation defined as X adv -X, c is the parameter to control the relative weight of function f in the total cost function, and τ is the control threshold used to penalize any terms that exceed τ .

E ADDITIONAL BLACK BOX ATTACK RESULTS:

Since CW L ∞ attack is computationally expensive, we only use 100 test examples (10 examples per each class). We search adversarial example X adv with c ∈ {0.1, 0.2, 0.5, 1, 2, 5, 10, 20} and τ ∈ {0.02, 0.04, ..., 0.6} for MNIST and c ∈ {0.1, 0.3, 1, 3, 10, 30, 100} and τ ∈ {0.001, 0.002, ..., 0.01, 0.012, ..., 0.02, 0.024, ..., 0.04, 0.048, ..., 0.08} for CIFAR10. We use Adam optimizer with an initial learning rate of 0.01/c since we found constant initial learning rate for c • f (X + δ) term is critical for successful adversarial images generation. We terminate the search after 2,000 iterations for each X, c and τ . If f (X + δ) < 0 and the resulting ||δ|| ∞ is lower than the current best distance, we update X adv .

Figure 9 shows cumulative distribution function of for 100 successful adversarial examples per each network. We report the number of adversarial examples with > 0.3*255 for MNIST and that with > 2 or 4 for CIFAR10. As seen from this figure, our approaches provide robust defense against CW L ∞ attack compared to other approaches.

:

Modified ResNet models showed slight decreased accuracy for both clean and adversarial images compared to original ResNet counterparts, however, we observed similar trends (improved accuracy for iterative attacks for the network trained with pivot loss) as in table

We also observed this when we switch the source and the target networks. Additional details can be found in Appendix E.

We found that the adversarial training requires longer training time than the standard training. Authors in the original paper(He et al., 2016) changed the learning rate at 32k and

48k iterations and terminated training at 64k iterations.

Figures and Tables:

a form of data augmentation where it injects adversarial examples during training. In this method, k examples are taken from the mini batch B (size of m) and the adversarial examples are generated with one of step method. The k adversarial examples replaces the corresponding clean examples when making mini batch. Below we refer this adversarial training method as "Kurakin's".

Figure 2: Cascade adversarial training regularized with a unified embedding.

Figure 4: Embedding space visualization for modified ResNet models trained on MNIST. x-axis and y-axis show first and second dimension of embeddings respectively. Scatter plot shows first 100 clean embeddings per each class on MNIST test set. Each arrow shows difference between two embeddings (one from iter FGSM image ( ) and the other from (+8)). We draw arrows from = 0 to = 76 (≈ 0.3*255) for one sample image per each class. We observe differences between clean and corresponding adversarial embeddings are minimized for the network trained with pivot loss.

Figure 6: Correlation between iter FGSM noises crafted from different networks for each . Correlation is averaged over randomly chosen 128 images from CIFAR10 test-set.

Figure7: Argument to the softmax vs. in test time. "step ll", "step FGSM" and "random sign" methods were used to generate test-time adversarial images. Arguments to the softmax were measured by changing for each test method and averaged over randomly chosen 128 images from CIFAR10 test-set. Blue line represents true class and the red line represents mean of the false classes. Shaded region shows ± 1 standard deviation of each line.

Figure 8: Averaged Pearson's correlation coefficient between the gradients w.r.t. two images. Correlation were measured by changing for each adversarial image and averaged over randomly chosen 128 images from CIFAR10 test-set. Shaded region represents ± 0.5 standard deviation of each line.

Figure 9: Cumulative distribution function vs. for 100 test adversarial examples generated by CW L ∞ attack. Lower CDF value for a fixed means the better defense. Carlini and Wagner (CW) L ∞ attack solves the following optimization problem for every input X. minimize c • f (X + δ) +

CIFAR10 test results (%) under black box attacks for =16. Source networks share the same initialization which is different from the target networks. {Target: R20, R20 K : standard, Kurakin's, Source: R20 2 , R20 K2 : standard, Kurakin's.} Ensemble adversarial training(Tramèr et al., 2017): is essentially the same with the adversarial training, but uses several pre-trained vanilla networks to generate one-step adversarial examples for training. Below we refer this adversarial training method as "Ensemble".

Table content: <table><tr><td/><td/><td>0.04 0.05 0.06 0.07 0.08 0.09 correlation</td><td colspan="2">iter_FGSM, (R20K, R20K2) iter_FGSM, (R20K, R202) step_FGSM, (R20K, R20K2) step_FGSM, (R20K, R202)</td></tr><tr><td/><td/><td>2</td><td>4</td><td>6</td><td>8 10 12 14 16</td></tr><tr><td>Target R20 R20 K</td><td>Source: step FGSM Source: iter FGSM R20 2 R20 K2 R20 2 R20 K2 16.2 31.6 2.7 60.1 66.7 82.7 55.8 28.5</td><td colspan="3">Figure 1: Correlation between adversarial noises from different networks for each . Shaded region shows ± 0.1 standard devi-ation of each line.</td></tr><tr><td colspan="2">3 PROPOSED APPROACH</td><td/><td/></tr></table>

MNIST test results (%) for 20-layer ResNet models ( = 0.3*255 at test time). { R20M: standard training, R20M K : Kurakin's adversarial training, R20M B : Bidirectional loss, R20M P : P ivot loss.} CW L ∞ attack is performed with 100 test samples (10 samples per each class) and the number of adversarial examples with > 0.3*255 is reported. Additional details for CW attack can be found in Appendix F

Table content: <table><tr><td>Model</td><td colspan="6">clean step ll step FGSM iter ll iter FGSM CW</td></tr><tr><td>R20M</td><td>99.6</td><td>9.7</td><td>10.3</td><td>0.0</td><td>0.0</td><td>0</td></tr><tr><td>R20M K</td><td>99.6</td><td>96.7</td><td>94.5</td><td>89.0</td><td>60.2</td><td>46</td></tr><tr><td colspan="2">R20M B (Ours) 99.5</td><td>97.3</td><td>96.2</td><td>97.2</td><td>88.5</td><td>81</td></tr><tr><td colspan="2">R20M P (Ours) 99.5</td><td>97.1</td><td>95.7</td><td>96.9</td><td>88.9</td><td>82</td></tr></table>

CIFAR10 test results (%) for 110-layer ResNet models. CW L ∞ attack is performed with 100 test samples (10 samples per each class) and the number of adversarial examples with > 2 or 4 is reported. {R110 K : Kurakin's, R110 P : P ivot loss, R110 E : Ensemble training, R110 K,C : Kurakin's and Cascade training, R110 P,E : P ivot loss and Ensemble training, and R110 P,C : P ivot loss and Cascade training} FGSM noises from networks with the same initialization. Correlation between iter FGSM noises from the networks with different initialization, however, becomes lower as the network is deeper as shown in figure 6 (b). Since the degree of freedom increases as the network size increases, adversarially trained networks prone to end up with different states, thus, making transfer rate lower. To maximize the benefit of the cascade adversarial training, we propose to use the same initialization for a cascade network and a source network used for iterative adversarial examples generation.

Table content: <table><tr><td>Model</td><td>clean</td><td colspan="2">step ll</td><td colspan="4">step FGSM iter FGSM</td><td>CW</td></tr><tr><td/><td/><td>=2</td><td>=16</td><td>=2</td><td>=16</td><td>=2</td><td>=4</td><td>=2</td><td>=4</td></tr><tr><td>R110 K</td><td colspan="7">92.3 88.3 90.7 86.0 95.2 59.4 9.2</td><td>25</td><td>4</td></tr><tr><td>R110 P (Ours)</td><td colspan="8">92.3 86.0 89.4 81.6 91.6 64.1 20.9 32</td><td>7</td></tr><tr><td>R110 E</td><td colspan="8">92.3 86.3 74.3 84.1 72.9 63.5 21.1 24</td><td>6</td></tr><tr><td colspan="9">R110 K,C (Ours) 92.3 86.2 72.8 82.6 66.7 69.3 33.4 20</td><td>5</td></tr><tr><td>R110 P,E (Ours)</td><td colspan="8">91.3 84.0 65.7 77.6 54.5 66.8 38.3 38</td><td>16</td></tr><tr><td>R110 P,C (Ours)</td><td colspan="8">91.5 85.7 76.4 82.4 69.1 73.5 42.5 27</td><td>15</td></tr><tr><td>lation between iter</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

CIFAR10 test results (%) for 110-layer ResNet models under black box attacks ( =16). {Target: same networks in table 3 and R110 K : Kurakin's, Source: re-trained baseline, Kurakin's, cascade and ensemble networks with/without pivot loss. Source networks use the different initialization from the target networks. Additional details of the models can be found in Appendix B.} Third, our low-level similarity learning serves as a good regularizer for adversarial images, but not for the clean images for ensemble/cascade models (reduced accuracy for the clean images for R110 P,E and R110 P,C in table3). Compared to pure adversarial training with one-step adversarial examples, the network sees more various adversarial perturbations during training as a result of ensemble and cascade training. Those perturbations are prone to end up embeddings in the vicinity of decision boundary more often than perturbations caused by one-step adversarial training. Pivot loss pulls the vicinity of those adversarial embeddings toward their corresponding clean embeddings.

Table content: <table><tr><td>Target</td><td/><td/><td/><td colspan="2">Source: iter FGSM</td><td/><td/></tr><tr><td/><td colspan="7">R110 2 R110 K2 R110 E2 R110 P 2 R110 K,C2 R110 P,E2 R110 P,C2</td></tr><tr><td>R110 K</td><td>70.5</td><td>73.2</td><td>27.9</td><td>77.0</td><td>67.3</td><td>54.6</td><td>80.8</td></tr><tr><td>R110 E</td><td>77.9</td><td>79.5</td><td>55.8</td><td>79.0</td><td>68.2</td><td>54.7</td><td>82.7</td></tr><tr><td>R110 P (Ours)</td><td>75.9</td><td>75.6</td><td>39.6</td><td>78.5</td><td>68.3</td><td>61.3</td><td>83.3</td></tr><tr><td colspan="2">R110 K,C (Ours) 56.4</td><td>80.2</td><td>61.1</td><td>79.5</td><td>67.4</td><td>62.6</td><td>82.1</td></tr><tr><td colspan="2">R110 P,E (Ours) 78.2</td><td>82.1</td><td>67.7</td><td>81.7</td><td>73.4</td><td>68.4</td><td>83.8</td></tr><tr><td colspan="2">R110 P,C (Ours) 71.9</td><td>80.4</td><td>63.9</td><td>80.1</td><td>71.1</td><td>64.2</td><td>83.0</td></tr></table>

We re-train 110-layer ResNet models using Kurakin's, cascade and ensemble adversarial training with/without low-level similarity learning and use those networks as source networks for black-box attacks. Baseline 110-layer ResNet model is also included as a source network. Target networks are the same networks used in table 3. We found iter FGSM attack resulted in lower accuracy than step FGSM attack, thus, report iter FGSM attack only in table4.We first observe that iter FGSM attack from ensemble models (R110 E2 , R110 P,E2 ) is strong (results in lower accuracy) compared to that from any other trained networks. 2 Since ensemble models learn various perturbation during training, adversarial noises crafted from those networks might be more general for other networks making them transfer easily between defended networks.Second, cascade adversarial training breaks chicken and egg problem. (In section 3.1, we found that it is efficient to use a defended network as a source network to attack another defended network.) Even though the transferability between defended networks is reduced for deeper networks, cascade network (R110 K,C ) shows worst case performance against the attack not from a defended network, but from a purely trained network (R110 2 ). Possible solution to further improve the worst case robustness would be to use more than one network as source networks (including pure/defended networks) for iter FGSM images generation for cascade adversarial training.

Table content: <table><tr><td>Third, ensemble/cascade networks together with our low-level similarity learning (R110 P,E ,</td></tr><tr><td>R110</td></tr></table>

table 5. For ensemble adversarial training, pre-trained networks as in table 7 together with the network being trained are used to generate onestep adversarial examples during training. For cascade adversarial training, pre-trained defended networks as in table 6 are used to generate iter FGSM images, and the network being trained is used to generate one-step adversarial examples during training.

Table content: <table/>

Model descriptions

Table content: <table><tr><td/><td/><td>Initialization</td><td/><td/></tr><tr><td>Dataset</td><td>ResNet</td><td>Group</td><td>Training</td><td>Model</td></tr><tr><td/><td/><td/><td>standard training</td><td>R20M</td></tr><tr><td>MNIST</td><td>20-layer</td><td>A</td><td>Kurakin's Bidirection loss</td><td>R20M K R20M B</td></tr><tr><td/><td/><td/><td>P ivot loss</td><td>R20M P</td></tr><tr><td/><td/><td/><td>standard training</td><td>R20</td></tr><tr><td/><td/><td/><td>Kurakin's</td><td>R20 K</td></tr><tr><td/><td/><td/><td>Ensemble training</td><td>R20 E</td></tr><tr><td/><td/><td>B</td><td>Bidirection loss P ivot loss</td><td>R20 B R20 P</td></tr><tr><td/><td/><td/><td>Kurakin's &amp; Cascade training</td><td>R20 K,C</td></tr><tr><td/><td>20-layer</td><td/><td>P ivot loss &amp; Ensemble training</td><td>R20 P,E</td></tr><tr><td/><td/><td/><td>P ivot loss &amp; Cascade training</td><td>R20 P,C</td></tr><tr><td/><td/><td/><td>standard training</td><td>R20 2</td></tr><tr><td/><td/><td>C</td><td>Kurakin's</td><td>R20 K2</td></tr><tr><td/><td/><td/><td>P ivot loss</td><td>R20 P 2</td></tr><tr><td/><td/><td>D</td><td>standard training</td><td>R20 3</td></tr><tr><td/><td/><td>E</td><td>standard training</td><td>R20 4</td></tr><tr><td>CIFAR10</td><td>56-layer</td><td>F</td><td>Kurakin's P ivot loss</td><td>R56 K R56 P</td></tr><tr><td/><td/><td>G</td><td>Kurakin's</td><td>R56 K2</td></tr><tr><td/><td/><td/><td>standard training</td><td>R110</td></tr><tr><td/><td/><td/><td>Kurakin's</td><td>R110 K</td></tr><tr><td/><td/><td/><td>P ivot loss</td><td>R110 P</td></tr><tr><td/><td/><td>H</td><td>Ensemble training</td><td>R110 E</td></tr><tr><td/><td/><td/><td>Kurakin's &amp; Cascade training</td><td>R110 K,C</td></tr><tr><td/><td/><td/><td>P ivot loss &amp; Ensemble training</td><td>R110 P,E</td></tr><tr><td/><td/><td/><td>P ivot loss &amp; Cascade training</td><td>R110 P,C</td></tr><tr><td/><td>110-layer</td><td/><td>standard training</td><td>R110 2</td></tr><tr><td/><td/><td/><td>Kurakin's</td><td>R110 K2</td></tr><tr><td/><td/><td/><td>Ensemble training</td><td>R110 E2</td></tr><tr><td/><td/><td>I</td><td>P ivot loss</td><td>R110 P 2</td></tr><tr><td/><td/><td/><td colspan="2">Kurakin's &amp; Cascade training R110 K,C2</td></tr><tr><td/><td/><td/><td colspan="2">P ivot loss &amp; Ensemble training R110 P,E2</td></tr><tr><td/><td/><td/><td>P ivot loss &amp; Cascade training</td><td>R110 P,C2</td></tr><tr><td/><td/><td>J</td><td>standard training</td><td>R110 3</td></tr><tr><td/><td/><td>K</td><td>standard training</td><td>R110 4</td></tr></table>

Ensemble model description

Table content: <table><tr><td>Ensemble models</td><td/><td>Pre-trained models</td></tr><tr><td colspan="2">R20 E , R20 P,E , R110 E , R110 P,E</td><td>R20 3 , R110 3</td></tr><tr><td>R110 E2 , R110 P,E2</td><td/><td>R20 4 , R110 4</td></tr><tr><td colspan="3">Table 7: Cascade model description</td></tr><tr><td>Cascade models</td><td colspan="2">Pre-trained model</td></tr><tr><td>R20 K,C , R20 P,C</td><td/><td>R20 P</td></tr><tr><td>R110 K,C , R110 P,C</td><td/><td>R110 P</td></tr><tr><td>R110 K,C2 , R110 P,C2</td><td/><td>R110 P 2</td></tr></table>

CIFAR10 test results (%) under black box attacks between the network with the same initialization ( =16)}

Table content: <table><tr><td>Target</td><td colspan="3">Source: step FGSM</td><td colspan="3">Source: iter FGSM</td></tr><tr><td/><td colspan="6">R20 R20 K R20 P R20 R20 K R20 P</td></tr><tr><td>R20</td><td>12.2</td><td>27.4</td><td>27.5</td><td>0.0</td><td>45.9</td><td>44.7</td></tr><tr><td colspan="2">R20 K 65.7</td><td>81.5</td><td>81.8</td><td>51.5</td><td>0.0</td><td>18.2</td></tr><tr><td colspan="2">R20 P 58.1</td><td>89.3</td><td>91.7</td><td>48.9</td><td>13.4</td><td>0.0</td></tr><tr><td>Table</td><td/><td/><td/><td/><td/><td/></tr></table>

CIFAR10 test results (%) under black box attacks for =16. {Target and Source networks are switched from the table 1}

Table content: <table><tr><td>Target</td><td colspan="3">Source: step FGSM</td><td colspan="3">Source: iter FGSM</td></tr><tr><td/><td colspan="6">R20 R20 K R20 P R20 R20 K R20 P</td></tr><tr><td>R20 2</td><td>17.9</td><td>33.9</td><td>34.5</td><td>4.1</td><td>54.8</td><td>54.3</td></tr><tr><td colspan="2">R20 K2 65.0</td><td>84.6</td><td>84.5</td><td>61.2</td><td>25.3</td><td>30.4</td></tr><tr><td colspan="2">R20 P 2 66.4</td><td>88.2</td><td>87.2</td><td>61.6</td><td>27.7</td><td>36.1</td></tr></table>

CIFAR10 test results (%) for cascade networks under black box attacks for =16. {Target and Source: Please see the model descriptions in Appendix B.}

Table content: <table><tr><td>Target</td><td>Source: iter FGSM</td></tr></table>

Peer review (iclr2018):
 The authors proposed to supplement adversarial training with an additional regularization that forces the embeddings of clean and adversarial inputs to be similar. The authors demonstrate on MNIST and CIFAR that the added regularization leads to more robustness to various kinds of attacks. The authors further propose to enhance the network with cascaded adversarial training, that is, learning against iteratively generated adversarial inputs, and showed improved performance against harder attacks. The idea proposed is fairly straight-forward. Despite being a simple approach, the experimental results are quite promising. The analysis on the gradient correlation coefficient and label leaking phenomenon provide some interesting insights. As pointed out in section 4.2, increasing the regularization coefficient leads to degenerated embeddings. Have the authors consider distance metrics that are less sensitive to the magnitude of the embeddings, for example, normalizing the inputs before sending it to the bidirectional or pivot loss, or use cosine distance etc.? Table 4 and 5 seem to suggest that cascaded adversarial learning have more negative impact on test set with one-step attacks than clean test set, which is a bit counter-intuitive. Do the authors have any insight on this? Comments: 1. The writing of the paper could be improved. For example, "Transferability analysis" in section 1 is barely understandable; 2. Arrow in Figure 3 are not quite readable; 3. The paper is over 11 pages. The authors might want to consider shrink it down the recommended length.