Title: ADVERSARIAL TRAINING WITH PERTURBATION GEN-ERATOR NETWORKS

Abstract: Despite the remarkable development of recent deep learning techniques, neural networks are still vulnerable to adversarial attacks, i.e., methods that fool the neural networks with perturbations that are too small for human eyes to perceive. Many adversarial training methods were introduced as to solve this problem, using adversarial examples as a training data. However, these adversarial attack methods used in these techniques are fixed, making the model stronger only to attacks used in training, which is widely known as an overfitting problem. In this paper, we suggest a novel adversarial training approach. In addition to the classifier, our method adds another neural network that generates the most effective adversarial perturbation by finding the weakness of the classifier. This perturbation generator network is trained to produce perturbations that maximize the loss function of the classifier, and these adversarial examples train the classifier with a true label. In short, the two networks compete with each other, performing a minimax game. In this scenario, attack patterns created by the generator network are adaptively altered to the classifier, mitigating the overfitting problem mentioned above. We theoretically proved that our minimax optimization problem is equivalent to minimizing the adversarial loss after all. Beyond this, we proposed an evaluation method that could accurately compare a wide-range of adversarial training algorithms. Experiments with various datasets show that our method outperforms conventional adversarial training algorithms.

1. INTRODUCTION:

Deep learning has shown the impressive performance in all areas of artificial intelligence, such as image classification and speech recognition (Hinton et al., 2012; Krizhevsky et al., 2012) . These advances lead to a broad application of deep neural networks in various real-life tasks. There are still, however, severe security issues such as adversarial examples, which hinder the use of machine learning system until a complete defense is constructed against multiple adversarial attacks. Adversarial examples are data samples that are close to real data samples, which cause a given neural network to misclassify. The basic idea of adversarial examples is to find a sample that increases the loss value of a neural network in the neighborhood of training data (Szegedy et al., 2014) . The perturbation on the original training data is so small that it makes the adversarial examples indistinguishable from the original examples.

Many authors proposed methods that make neural networks robust to adversarial examples (Papernot et al., 2016; Goodfellow et al., 2015; Szegedy et al., 2014; Miyato et al., 2016) . One of the methods is an adversarial training, which re-trains the neural network with adversarial examples generated by adversarial attacks. Adversarial training with powerful attacks would guarantee robustness, but the recent fatal attack methods (Szegedy et al., 2014; Papernot et al., 2016; Carlini & Wagner, 2017; Moosavi-Dezfooli et al., 2016) require high computational complexity because of their iterative optimization. Therefore, they are not compatible with adversarial training. Methods that quickly produce adversarial examples, such as fast gradient sign (Goodfellow et al., 2015) or projected gradient descent (Kurakin et al. (2017) ; Madry et al. (2018) ), have been used for practical adversarial training. While the above adversarial training methods are empirically successful, they might be susceptible to future attackers, and this makes the defense procedure useless. If an algorithm for generating an adversarial example is fixed in adversarial training, the network could overfit to the specific algorithm.

In this paper, we introduce a novel adversarial training framework that increases the robustness against various adversarial attacks. Stemming from GAN framework, we devised a method in which the classifier network and a perturbation generator network are alternately trained. To be more specific, the generator network generates a perturbation image that maximizes the loss function of the classifier network, and the classifier network is trained through the corresponding adversarial image with the true label. Through this minimax optimization between the two networks, the classifier network can improve robustness against many different attacks, as the attack pattern of the generator network is constantly modified depending on the classifier network. This procedure can be used in practical adversarial training since adversarial perturbations can be produced by a forwardpropagation. We generalized Madry et al. (2018) 's research on adversarial loss to theoretically support our technique, and we also proposed a method that can fairly evaluate the performance of adversarial training algorithms.

2. RELATED WORKS:

The goal of our work is to construct defensive mechanisms to adversarial attacks. To alleviate the security problem, the adversarial robustness of neural networks has been studied in the literature. One of the intuitive ways to increase robustness is to re-train with adversarial examples, which are called adversarial training. This method uniformly smoothen the ground-truth label decision region close to the original data points. In the context of smoothness, there exists adversarial examples that hold very low confidence on the ground-truth label in the vanilla decision region before applying robust optimization. Szegedy et al. (2014) first proposed a method to generate adversarial examples. They use boxconstraint L-BFGS optimization to find the examples. This holds the exact formulation of adversarial examples, but because of its exhausted optimization procedure, it is not suitable for practical adversarial training. Goodfellow et al. (2015) introduced an algorithm that quickly generates adversarial examples by using one-step gradient update, which is called fast gradient sign method. In addition, they first proposed a realistic adversarial training method which injects the adversarial examples into the training data. This method is not strong enough to generate high-quality examples and is far from robust optimization. Kurakin et al. (2017) suggested an iterative version of fast gradient method (FGM) attack called Projected Gradient Descent (PGD), which is much closer to the optimal adversarial examples. Adversarial training can be formulated with the robust optimization problem which minimizes the loss of the optimal adversarial examples in the -ball of all the original data points. This gives rise to the following minimax game, which is the main theoretical background of our work:

min θ ρ(θ), where ρ(θ) = E (x,y)∼D max δ∈S L(θ, x + δ, y) .

(1) They approximated the above minimax game by PGD based adversarial training to reduce computational complexity issue. The gradient descent based adversarial examples for robust optimization is not adaptive. Therefore, those neural networks are vulnerable to other types of adversarial attacks (Athalye et al., 2018) .

Several works studied the methods that generate stronger adversarial attacks (Athalye et al., 2018; Lee et al., 2017; Papernot et al., 2017; Moosavi-Dezfooli et al., 2016; Dong et al., 2018; Song et al., 2018) . Carlini & Wagner (2017) pin-points that defensive distillation network (Papernot et al. (2016) ) is not practical in that it exploits gradient masking, so they devised a powerful attack algorithm that avoids this problem. In an attempt to eliminate the gradient masking problem of softmax function, they adopted logits Z in objective function, and discovered an appropriate adversarial noise for each image utilizing line-search technique. However, most of these works have high computational cost, so they are difficult to be applied to adversarial training.

There are many other defense methods that are not based on adversarial training (Li et al., 2019; , Junbo) . The above robust optimization problem can be generalized as convex outer adversarial polytope, which relaxes the activation function as a convex form to prevent misclassification (Wong & Kolter, 2018) . Certified defense algorithms guarantee at least a certain bounds of the proper label probability distributions against adversarial examples (Cohen et al., 2019; Liu et al., nathan et al., 2018) . In addition, some researchers recently studied the theoretical backgrounds of adversarial robustness (Dohmatob, 2019; Wang et al., 2018; Roth et al., 2019) .

3. PROPOSED METHOD:



3.1. NOTATIONS:

We denote a labeled training set by (x, y) ∼ P data , where x ∈ R H×W ×C represents input images with height H, width W , and channel C, and y ∈ {1, 2, . . . , K} is a label for an input x. We use two neural networks in the proposed method. One is a standard K-class classifier network F (x; θ) which is defined by:

EQUATION

Where F (x; θ) represents the class probability vector computed using the softmax function. The other is a perturbation generating network G(∇ x F, x, y; φ), which is defined by:

G : R H×W ×C , R H×W ×C , R → R H×W ×C (3)

Note that G(∇ x F, x, y; φ) represents the perturbation of the input image x, where ∇ x F = ∇ x F (x; θ) y denotes the gradient of class probability of the true label with respect to the input images x.

3.2. ADVERSARIAL TRAINING WITH GENERATIVE MODEL:

The entire procedure of our algorithm is shown in Figure 1 . Goodfellow's work on GAN inspired us to make the classifier and the perturbation generator compete with each other. Classifier F defines the network we are aiming to train and increase the robustness, and the perturbation generator G is the network which produces the perturbations that maximize the loss function of the classifier. The classifier network is trained with adversarial images produced by the generator network with the true label. The generator network assigns image x, label y, and a gradient image ∇ x F as inputs, which is trained to maximize the loss function of the classifier. In other words, F and G play the following two-player minimax game:

E x,y∼Pdata min θ max φ Loss(x + G(∇ x F, x, y; φ), y; θ) -c L G(∇ x F, x, y; φ) 2 2 (4)

By the time this minimax game is complete, the classifier will have been trained with various attacks produced by the generator with the enhanced robustness against powerful adversarial attacks, while the generator will no longer find any vulnerability in the classifier, therefore only producing random noises. In Equation ( 4), c L is a hyper-parameter that adjusts the ratio between two cost functions. If c L is very low, it will only find trivial solutions with extremely large perturbation power, and if c L is very high, it will only generate zero-perturbation images. Therefore, it is crucial to determine an appropriate c L . The theoretical meaning of c L will be discussed in Appendix A.

We believe that the strength of our algorithm is the flexibility of the generator's attack method.

Conventional adversarial training algorithms such as FGM have a fixed attack method. For example, adversarial perturbation in FGM is generated by normalizing the gradient of a data sample, which is scaled by . A classifier can be easily made robust against adversarial examples generated by these fixed methods. The following are two possible methods.

CASE 1. The classifier is trained to gradually reduce the gradient at data points. Gradually reducing the gradient prevents FGM from generating meaningful adversarial example from the gradient (also known as a gradient masking problems).

CASE 2. The classifier can only be trained to reduce the loss under the surface of the -norm ball centered around a data point. As a result, the network would be robust against adversarial examples generated by FGM, but could still be very vulnerable against adversarial examples located inside the -norm ball.

Through the above methods, a classifier can be easily overfit to FGM attacks, but it becomes more vulnerable to other powerful attacks such as Carlini&Wagner L 2 attacks as shown in Figure 2 .

Through our experiments under FGM adversarial training, we observed a CASE 1 overfitting problem when is small, and a CASE 2 overfitting problem when is large. The above gradient masking problem has also been discussed by other papers on adversarial training (Tramèr et al., 2018) . This problem also occurs when the iteration number of PGD is relatively low. On the other hand, our algorithm continuously updates the parameters of a generator to maximize the classifier loss, which tends to produce more general attacks to a given classifier. This means that a classifier cannot easily overfit to attacks from a generator in our method. Experimental results also support our arguments.

4. EXPERIMENT:



4.1. EXPERIMENTAL SETUP:

We used CIFAR-10 and CIFAR-100 for our datasets, to verify the robustness of our trained network. We normalized the pixel value of the image to [0,1] prior to network training. This section only presents the results from CIFAR-100. The experiment with CIFAR-10 showed similar results, and interested readers can refer to the appendix for its results.

The model architecture and parameters for CIFAR are given in Table 1 . We used conventional ConvPool-CNN as the classifier network, and the generator network was designed to efficiently use gradients, images, and labels. One might assume that hyperbolic tangent function should be used for the final activation function of the generator network. However, if a hyperbolic tangent function is used for generating the perturbation image, the adversarial image created must be clipped again to the proper value of the image, i.e. 0 <= x i + δ i <= 1 for all i. This is known as a box-constraint problem, which might cause the network to get stuck in extreme regions (Carlini & Wagner, 2017) . Therefore, we practiced the following technique proposed by Carlini & Wagner (2017) to avoid the clipping issue. 

EQUATION

4.2. EVALUATION METHOD:

We applied the following average distortion metric suggested by Carlini & Wagner (2017) , in order to fairly evaluate the robustness of the network for various adversarial attacks.

ρ := 1 |D| x k ∈D ∆ x k 2

, where D is a successful adversarial example set (6)

The above ρ represents the mean value of L 2 norm of the perturbation derived from the successful adversarial examples from the attack, the same value as the area under the curve in Figure 2 . Although ρ can be measured for any attack methods, it is best to measure ρ for the most powerful attack. Thus, we used ρ cw for Carlini&Wagner L 2 attack on all of our experiments as the evaluation metric for robustness of the network. decreases during the training process. Thus, for fair comparison of the robustness of the networks, it is desirable to match the accuracy of the benign examples before comparing ρ cw .

To better compare the performance of different adversarial algorithms, we must first train the models by adjusting hyper-parameters for each adversarial training, calculate benign accuracy and ρ cw for each trained model, and then draw a graph with the calculated values, connecting each relevant data point. Naturally, the structure of the classifier used for each adversarial training must be identical. We will call this graph robustness-curve. Figure 5 displays the robustness-curves for FGM adversarial training, PGD adversarial training, and our algorithm. It should be noted that the method with outer curve is a better adversarial training algorithm since ρ cw is higher at the same benign accuracy.

4.3. ATTACK PERFORMANCE OF THE PERTURBATION GENERATOR NETWORK:

In order to verify the attack performance of the proposed generator network, we used two fixed classifier networks adversarially trained with FGM ( = 0.5) and PGD ( =0.5), respectively. For each classifier network, we compared the performance of FGM attack, PGD attack, and PGN (proposed) attack. PGN was fully trained on the training dataset. We plotted the accuracy of the classifiers for these three attacks in terms of average perturbation distortion. The results are shown in Figure 3 .

For each model, it is observed that our algorithm is capable of much more powerful attack than PGD and FGM. This difference in performance clearly justifies the superior defense performance of the adversarial training with PGN in the following section.

The perturbation generator network can be trained to fully exploit incoming gradients, images, and labels to generate adversarial perturbation optimized for the classifier. As for FGM or PGD, they have a fixed norm, with a single attack direction towards the gradient. Now with PGN, it has no restriction on both direction and size, and can be transformed into an optimized perturbation for the classifier.

4.4. DEFENSE PERFORMANCE ON VARIOUS ATTACKS:

Based on the methods introduced in 4.2, we compared the robustness of the network trained by our suggested technique with that trained by the conventional adversarial training methods. FGM attack, MIM attack, DeepFool, and Carlini&Wagner Attack were used as attack methods. In whitebox attacks, adversarial examples were generated through direct access to the model's gradient, while in black-box attack, accuracy was measured through the adversarial examples produced by an independently trained network. Table 2 exhibits the robustness of the network when all the benign accuracy values of the adversarial networks are balanced to that of the naive network, and Figure 4 displays three perturbation L 2 power (x) -accuracy (y) graphs for C&W attack, with the benign accuracy for each adversarial networks set to 68%, 66%, and 63%, respectively.

FGM and MIM are attack methods that find the adversarial examples that can maximize the loss function of the classifier network on fixed L 2 norm of perturbation power, so the higher the accuracy To properly compare the robustness of the networks, the benign accuracy of the networks needs to be balanced out. The graph displays three different curves, each representing the accuracy of the perturbation power with respect to the perturbation norm of Carlini&Wagner L 2 attack with different benign accuracy of 68%, 66%, and 63%, respectively.

of the adversarial examples, the more robust the network. On the other hand, DeepFool and C&W attack find the adversarial examples with the lowest L 2 norm of perturbation power that can fool the network; therefore, the robust network would have higher mean and median values of the adversarial perturbation power.

As you can see from Training speed is also a crucial issue in adversarial training. The proposed algorithm is slower than FGM because it trains the generator after finding the gradient image, while FGM immediately uses the gradient image to train the classifier. On the other hand, our algorithm is faster than Madry's which use PGD (multi-step gradient descent) to find the adversarial image. Note that the more iteration steps of PGD, the larger the speed-gap between Madry's and our algorithm we get. 

4.5. VARYING HYPER-PARAMETERS:

As mentioned in Section 4.2, adversarial training algorithms have a trade-off relationship between benign accuracy and robustness metric ρ cw . Figure 5 visualizes the relationship with a plot consisting of the data points of benign accuracy and ρ cw , which are collected by using various hyper-parameters for each adversarial training. For FGM and PGD adversarial training, the data points with higher ρ cw are models trained with bigger , while for our algorithm, the data points with larger ρ cw are models trained with lower c L . It should be noted that the robustness-curve is an appropriate indicator for performance evaluation of the adversarial training algorithms, since it displays a comprehensive set of ρ cw with corresponding benign accuracy. As demonstrated in Figure 5 , our algorithm outperforms all the other adversarial training algorithms under all benign accuracy. Furthermore, we plotted the robustness-curve by proportionally increasing the number of filters in each convolutional layer of the classifier. As can be observed in the second plot of Figure 5 , the robustness-curve moves to the right as the capacity of the model increases, which means that the classifier may still be underfitted. In other words, the classifier trained with only clean examples tend to overfit easily to the training data with even a low capacity, whereas the classifier trained with various adversarial examples tend to underfit instead even with higher capacity networks. Although we were not able to deal with higher capacity due to the limits of the current hardware technology, it is expected that a far greater network capacity may be needed to achieve a human-level robustness.

5. CONCLUSION:

This study proposed a novel adversarial training method that boosts the robustness of a deep neural network against adversarial attacks. Based on a GAN framework, the classifier network and the generator network play a two-player minimax game, which improves the robustness of a classifier against adversarial examples. In generating adversarial examples, we use a trainable perturbation generator network instead of a fixed function as in most of conventional adversarial training methods. this method tend to overfit less, and strengthens the robustness against many different kinds of attacks. Our proposed method is far more robust than existing adversarial training techniques. Since it computes adversarial examples through one-step inference, it is also more advantageous in training speed, compared to other techniques that use multiple steps in inner maximization.

Our experiment with CIFAR datasets have also proved the advantage of our approach, as the network trained by our method showed improved robustness and the state-of-the-art performance against various attacks with different noise power. Although the proposed approach compares favorably with other methods, it is believed that there is still room for improvement. One future direction is to study a generator network which is most effective for adversarial training. 

:



Figures and Tables:

Figure 2: Trade-off relationship between benign accuracy and the adversarial robustness. Left: Perturbation power-accuracy graph for FGM adversarial training with various epsilon against CW attack. The bigger the epsilon, the more robust the network to adversarial attacks, but less benign accuracy. Right: The network with FGM adversarial training tend to overfit easily because of its fixed attack algorithm. As the training progresses, the benign accuracy rises, whereas the adversarial robustness declines.

Figure 4: The comparison of the robustness of the defense methods with various benign accuracy.To properly compare the robustness of the networks, the benign accuracy of the networks needs to be balanced out. The graph displays three different curves, each representing the accuracy of the perturbation power with respect to the perturbation norm of Carlini&Wagner L 2 attack with different benign accuracy of 68%, 66%, and 63%, respectively.

Figure 5: Robustness-curve: A plot showing the relationship between benign accuracy and ρ cw by changing the hyper-parameters of each adversarial training algorithm. Left: For FGM and PGD adversarial training, each data point was acquired through changing , whereas for our algorithm, each data point was acquired through changing c L . The outer curves are considered more robust adversarial algorithms. Right: Robustness-curves for our algorithm under different capacities of the classifier. It shows that the classifier is still underfitting in terms of adversarial robustness.

SOME ADVERSARIAL EXAMPLES GENERATED BY THE PROPOSED GENERATOR NETWORKTable4: Adversarial examples for various perturbation powers from the CIFAR-100 dataset. These images are generated from the generator which is fully trained to maximize the classification loss. Labels denote corresponding classification results.

Figure8: In order to measure how much contribution each input make in terms of the robustness of the classifier, we removed each input one by one, and plotted an accuracy vs average distortion curve. It can be observed that the gradient is the most important factor in classifier robustness, and the other two factors have relatively smaller impact on classifier robustness.

For our baseline for comparison, we used a naive network trained only with clean examples. For our control group, we setGoodfellow et al. (2016)'s adversarial training with Fast Gradient Method (FGM), andMadry et al. (2018)'s adversarial training with Projected Gradient Descent (PGD). For attack methods, we used FGM, Momentum Iterative Method (MIM), DeepFool, and Carlini&Wagner (C&W), and evaluated the robustness of the network through the accuracy of the adversarial examples and the mean and median value of the L 2 norm of the perturbaion generated by each attack. All the attacks and adversarial training methods above are L 2 -bounded. Detailed evaluation method will be discussed in section 4.2.

Table content: <table/>

Model architectures and parameters for CIFAR datasets

Table content: <table><tr><td colspan="2">Classifier Network</td><td/><td>Generator Network</td><td>Parameter</td></tr><tr><td colspan="2">Input: x (32 × 32 × 3)</td><td/><td>Input: y (10 or 100)</td><td>Optimizer</td><td>Adam</td></tr><tr><td colspan="2">3 × 3 Conv 64</td><td/><td>Dense 1024</td><td>Learning Rate</td><td>0.0005</td></tr><tr><td colspan="2">3 × 3 Conv 128</td><td/><td>Concatenate with y</td><td>Batch Size</td><td>128</td></tr><tr><td colspan="2">2 × 2 AvgPool</td><td/><td>Dense 8192</td><td>Adv Coefficient</td><td>1.0</td></tr><tr><td colspan="2">3 × 3 Conv 128</td><td/><td>Reshape 8 × 8 × 128</td><td>PGD iter</td><td>10</td></tr><tr><td colspan="2">3 × 3 Conv 256</td><td colspan="2">Concatenate with y (reshape)</td><td>Dropout</td><td>-</td></tr><tr><td colspan="2">2 × 2 AvgPool</td><td colspan="2">3 × 3 Upconv 128, stride=2</td><td>Weight Decay</td><td>0</td></tr><tr><td colspan="2">3 × 3 Conv 256</td><td colspan="2">Concatenate with y (reshape)</td><td>Ema decay</td><td>0.998</td></tr><tr><td colspan="2">3 × 3 Conv 512</td><td colspan="2">3 × 3 Upconv 128, stride=2</td><td>Max Epochs</td><td>200</td></tr><tr><td colspan="2">2 × 2 AvgPool</td><td colspan="2">Concatenate with [x, ∇ x F ]</td><td>FGS attack eps</td><td>0.5</td></tr><tr><td colspan="2">3 × 3 Conv 10 or 100</td><td colspan="2">3 × 3 Upconv 128, stride=1</td><td>MIM attack eps</td><td>0.5</td></tr><tr><td colspan="2">GlobalAvgPool</td><td/><td>3 × 3 Upconv 3, stride =1</td><td>MIM attack iter</td><td>100</td></tr><tr><td/><td>Softmax</td><td/><td/><td>DF attack iter</td><td>100</td></tr><tr><td colspan="4">Output: 10 or 100 class probabilities Output: 32 × 32 × 3 perturbation</td><td>CW attack iter</td><td>100</td></tr><tr><td>0</td><td colspan="2">1 Attack perturbation norm (L2 Mean) 2 3 4</td><td>5</td></tr></table>

Figure 3: Evaluation of the attack performance of the proposed generator network. Accuracy vs perturbation norm was plotted for various attack methods. Left: a classifier was adversarially trained with Fast Gradient Method. Right: a classifier was adversarially trained with Projected Gradient Descent.

Table content: <table><tr><td/><td>0.60 0.65</td><td colspan="2">Adversarially trained with FGM Fast Gradient Method Projected Gradient Descent Perturbation Generator Network (Ours)</td><td>0.60 0.65</td><td colspan="2">Adversarially trained with PGD Fast Gradient Method Projected Gradient Descent Perturbation Generator Network (Ours)</td></tr><tr><td>Accuracy</td><td>0.50 0.55</td><td/><td>Accuracy</td><td>0.50 0.55</td><td/></tr><tr><td/><td>0.45</td><td/><td/><td>0.45</td><td/></tr><tr><td/><td>0.40</td><td/><td/><td>0.40</td><td/></tr><tr><td/><td>0.0</td><td>0.1 Attack perturbation norm (L2 Mean) 0.2 0.3 0.4</td><td>0.5</td><td>0.0</td><td>0.1 Attack perturbation norm (L2 Mean) 0.2 0.3 0.4</td><td>0.5</td></tr><tr><td colspan="7">However, it is not sufficient to use only the above metric in evaluating the robustness of the ad-</td></tr><tr><td colspan="7">versarial training algorithm. In most adversarial training process, there are some hyperparameters</td></tr></table>

The comparison of the performance of the conventional adversarial training algorithms and our algorithm with = 0.02 and c L = 50. Benign accuracy of all defenses were balanced out with that of the baseline network before the comparison. Column 3, 6: Prediction accuracies of White-Box attack and Black-Box attack for each attack algorithms. Column 4: MEAN L 2 norm of the adversarial perturbation (ρ, which is defined in Equation (6)). Column 5: Median L 2 norm of the adversarial perturbations.

Table content: <table><tr><td colspan="2">DEFENSE</td><td>ATTACK</td><td colspan="2">ACCURACY W-BOX</td><td>MEAN L 2 W-BOX</td><td>MEDIAN L 2 W-BOX</td><td>ACCURACY B-BOX</td><td>BENIGN ACCURACY</td><td>TRAINING TIME (SEC/EPOCH)</td></tr><tr><td/><td/><td>FGM</td><td/><td>0.1173</td><td>-</td><td>-</td><td>0.3843</td></tr><tr><td>Baseline</td><td/><td>MIM Deepfool</td><td/><td>0.0167 -</td><td>-0.0994</td><td>-0.0626</td><td>0.2913 0.6688</td><td>0.7002</td><td>8.18</td></tr><tr><td/><td/><td>C&amp;W</td><td/><td>-</td><td>0.0791</td><td>0.0503</td><td>0.6659</td></tr><tr><td/><td/><td>FGM</td><td/><td>0.2213</td><td>-</td><td>-</td><td>0.5211</td></tr><tr><td colspan="2">Goodfellow et al. (2015)</td><td>MIM Deepfool</td><td/><td>0.1068 -</td><td>-0.1669</td><td>-0.1089</td><td>0.5115 0.6902</td><td>0.6993</td><td>25.1248</td></tr><tr><td/><td/><td>C&amp;W</td><td/><td>-</td><td>0.1334</td><td>0.0867</td><td>0.6894</td></tr><tr><td/><td/><td>FGM</td><td/><td>0.2139</td><td>-</td><td>-</td><td>0.5164</td></tr><tr><td colspan="2">Madry et al. (2018)</td><td>MIM Deepfool</td><td/><td>0.1000 -</td><td>-0.1625</td><td>-0.1061</td><td>0.5014 0.6906</td><td>0.7000</td><td>175.8571</td></tr><tr><td/><td/><td>C&amp;W</td><td/><td>-</td><td>0.1305</td><td>0.0851</td><td>0.6880</td></tr><tr><td/><td/><td>FGM</td><td/><td>0.3906</td><td>-</td><td>-</td><td>0.6428</td></tr><tr><td>Ours</td><td/><td>MIM Deepfool</td><td/><td>0.3444 -</td><td>-0.3184</td><td>-0.2064</td><td>0.6456 0.6958</td><td>0.7004</td><td>51.7681</td></tr><tr><td/><td/><td>C&amp;W</td><td/><td>-</td><td>0.2617</td><td>0.1674</td><td>0.6961</td></tr><tr><td>0</td><td colspan="3">1 Attack perturbation norm (L2 mean) 2 3 4</td><td>5</td><td/><td/><td/></tr></table>

Table 2, our algorithm outperforms the other adversarial training algorithms against all the attack methods. In white-box attacks, our algorithm showed the highest accuracy of adversarial examples against FGM and MIM attacks, and the highest power of adversarial perturbations by DeepFool and Carlini&Wagner. Also, in black-box attacks, our method proved to classify the adversarial examples with greater accuracy compared with the other adversarial training algorithms. According to Table 2, FGM adversarial training and PGD adversarial training show a very similar performance. This is because minimal was applied to match the benign accuracy of the baseline network. Since a neural network is locally linear, this minimal would make PGD and FGM generated adversarial examples to be almost identical. As you can see from Figure4and Figure5, as increases, Madry's method shows a more robust performance compared with Goodfellow's method.

Table content: <table/>

Peer review (iclr2020):
This paper introduces a new adversarial training approach, where a generator is used to generate the most challenging adversarial examples and the classifier is trained to correctly classify the generated adversarial examples. In this way, the robustness of the classifier is expected to be improved. For the generator, the input includes the original data sample, the gradient of the classifier, and the true label of the data sample. The paper is clearly presented and easy to follow. The experimental results relatively support the main claims of the paper. The reasons that I go towards mild rejection for this paper are as follows: 1. The general idea of improving the robustness of a classifier to by feeding adversarial examples to it is not a new idea. As shown in the paper, the idea has been adopted in Goodfellow et al. (2015) and Madry et al. (2018). In addition, [1] comprehensively studied how to ensemble different adversarial attacks as *training data* of a classifier to improve its robustness. The paper generates adversarial examples with a generator that takes various things as inputs, which can be viewed as an extension to the method in [2]. [2] introduced a generator that only takes the data sample as inputs and the difference to the paper is that the one in this paper additionally takes gradients and label as inputs. With Goodfellow et al. (2015), Madry et al. (2018), and [1], it may not be too hard to apply the method in [2] to improve classifier robustness. Therefore, it is questionable whether the additional information used in the generator of this paper helps and how it helps. I did not see any comparison in the experiments. 2. Although the experiments look to be well conducted, it is not comprehensive enough. For the defence methods in comparison, only two approaches that fall into the exact same line of the proposed method are included. In this line, I think it is necessary to compare with [1], which combines various attacks. Moreover, I would also expect a comparison with other closely related methods such as [3] and [4], to further demonstrate the effectiveness of the proposed one. 3. It is also suggested to do ablation tests on the inputs of the generator to see how each part of the inputs helps. If gradients and true label are removed from the generator, it reduces to [2]. Therefore, those ablation tests also help the comparison with [2]. Minor comments: 1. It could be interesting to visualise the perturbations that are generated from the generator. 2. Some of the figures and captions are too small to see in a printout. [1] Tramèr, Florian, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. *Ensemble Adversarial Training: Attacks and Defenses.* (2018). [2] Xiao, Chaowei, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. *Generating adversarial examples with adversarial networks.* arXiv preprint arXiv:1801.02610 (2018). [3] Samangouei, Pouya, Maya Kabkab, and Rama Chellappa. *Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models.* (2018). [4] Matyasko, Alexander, and Lap-Pui Chau. *Improved network robustness with adversary critic.* In Advances in Neural Information Processing Systems, pp. 10578-10587. 2018.