Adversarial training with perturbation generator networks
Authors: Hyeungill Lee
Sungyeob Han
Jungwoo Lee
Venue:  (2020)
Keywords: Adversarial training
Generative model
Adaptive perturbation generator
Robust optimization

Abstract: Despite the remarkable development of recent deep learning techniques, neural networks are still vulnerable to adversarial attacks, i.e., methods that fool the neural networks with perturbations that are too small for human eyes to perceive. Many adversarial training methods were introduced as to solve this problem, using adversarial examples as a training data. However, these adversarial attack methods used in these techniques are fixed, making the model stronger only to attacks used in training, which is widely known as an overfitting problem. In this paper, we suggest a novel adversarial training approach. In addition to the classifier, our method adds another neural network that generates the most effective adversarial perturbation by finding the weakness of the classifier. This perturbation generator network is trained to produce perturbations that maximize the loss function of the classifier, and these adversarial examples train the classifier with a true label. In short, the two networks compete with each other, performing a minimax game. In this scenario, attack patterns created by the generator network are adaptively altered to the classifier, mitigating the overfitting problem mentioned above. We theoretically proved that our minimax optimization problem is equivalent to minimizing the adversarial loss after all. Beyond this, we proposed an evaluation method that could accurately compare a wide-range of adversarial algorithms. Experiments with various datasets show that our method outperforms conventional adversarial algorithms. 

Under review as a conference paper at ICLR 2020
ADVERSARIAL TRAINING WITH PERTURBATION GEN- ERATOR NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the remarkable development of recent deep learning techniques, neural networks are still vulnerable to adversarial attacks, i.e., methods that fool the neu- networks with perturbations that are too small for human eyes to perceive. Many adversarial training methods were introduced as to solve this problem, using adversarial examples as training data. However, these adversarial attack meth- ods used in these techniques are fixed, making the model stronger only to attacks used in training, which is widely known as an overfitting problem. In this paper, we suggest novel adversarial training approach. In addition to the classifier, our method adds another neural network that generates the most effective adversarial perturbation by finding the weakness of the classifier. This perturbation gener- ator network is trained to produce perturbations that maximize the loss function of the classifier, and these adversarial examples train the classifier with a true la- bel. In short, the two networks compete with each other, performing minimax game. In this scenario, attack patterns created by the generator network are adap- tively altered to the classifier, mitigating the overfitting problem mentioned above. We theoretically proved that our minimax optimization problem is equivalent minimizing the adversarial loss after all. Beyond this, we proposed an evalua- tion method that could accurately compare a wide-range of adversarial training algorithms. Experiments with various datasets show that our method outperforms conventional adversarial training algorithms.
INTRODUCTION
Deep learning has shown the impressive performance in all areas of artificial intelligence, such as image classification and speech recognition (Hinton et al. 2012 Krizhevsky et al. (012). These ad- vances lead a broad application of deep neural networks in various real-life tasks. There are still, however, severe security issues such adversarial examples, which hinder the use of machine learn- ing system until a complete defense constructed against multiple adversarial attacks. Adversarial examples are data samples that close to real data samples, which cause given neural network to misclassify. The basic idea of adversarial examples is to find a sample that increases the loss value of neural network in the neighborhood of training data (Szegedy et al.) 2014). The perturbation the original training data is so small that it makes the adversarial examples indistinguishable from the original examples.
Many authors proposed methods that make neural networks robust to adversarial examples (Papernot al. 2016 Goodfellow al. 2015 (Szeedyy al.) 2014 Miyato al.) 2016). One of the methods is an adversarial training, which re-trains the neural network with adversarial examples generated by adversarial attacks. Adversarial training with powerful attacks would guarantee robustness, but the recent fatal attack methods (Szegedy et al., (204:: Papernot et al. 2016 Carlini & Wagner [201er]2017] Dezoooi et al.] (206)) require high computational complexity because of their iterative optimization. Therefore, they are not compatible with adversarial training. Methods that quickly produce adversarial examples, such as fast gradient sign (Goodfellow et al.] (2055) or projected gradient descent (Kurakin et al. (2017) Madry al. (2018)), have been used for practical adversarial training. While the above adversarial training methods are empirically successful, they might be susceptible to future attackers, and this makes the defense procedure useless. If algorithm for generating an adversarial example is fixed in adversarial training, the network could overfit to the specific algorithm.

---

Under review as a conference paper at ICLR 2020
In this paper, introduce novel adversarial training framework that increases the robustness against various adversarial attacks. Stemming from GAN framework, we devised a method in which the classifier network and a perturbation generator network are alternately trained. To be more spe- cific, the generator network generates a perturbation image that maximizes the loss function of the classifier network, and the classifier network is trained through the corresponding adversarial image with the true label. Through this minimax optimization between the two networks, the classifier network can improve robustness against many different attacks, as the attack pattern of the gen- erator network is constantly modified depending on the classifier network. This procedure can be used in practical adversarial training since adversarial perturbations can be produced by forward- propagation. We generalized Madry et al. (2018)'s 's research on adversarial loss to theoretically support our technique, and we also proposed method that can fairly evaluate the performance of adversarial training algorithms.
2 RELATED WORKS
The goal of our work is construct defensive mechanisms to adversarial attacks. To alleviate the security problem, the adversarial robustness of neural networks has been studied in the literature. One of the intuitive ways increase robustness to re-train with adversarial examples, which are called adversarial training. This method uniformly smoothen the ground label decision region close to the original data points. In the context of smoothness, there exists adversarial examples that hold very low confidence on the ground-truth label in the vanilla decision region before applying robust optimization.
Szegedy et al. (2014) first proposed a method to generate adversarial examples. They use box- constraint L-BBGS optimization to find the examples. This holds the exact formulation of adver- sarial examples, but because of its exhausted optimization procedure, it is not suitable for practical adversarial training. Gooddflllo et al. (2015) introduced an algorithm that quickly generates ad- versarial examples by using one-step gradient update, which is called fast gradient sign method. In addition, they first proposed a realistic adversarial training method which injects the adversarial examples into the training data. This method is not strong enough generate high-quality exam- ples and is far from robust optimization. Kurakin et al. (2017) suggested an iterative version of fast gradient method (FGM) attack called Projected Gradient Descent (PGD), which is much closer to the optimal adversarial examples. Adversarial training can be formulated with the robust optimiza- tion problem which minimizes the loss of the optimal adversarial examples in the e-ball of all the original data points. This gives rise to the following minimax game, which is the main theoretical background of our work:
min p(0), where p(0) = E (xy))
max L(0, + 8, BES
(I)
They approximated the above minimax game by PGD based adversarial training to reduce compu- tational complexity issue. The gradient descent based adversarial examples for robust optimization is not adaptive. Therefore, those neural networks are vulnerable to other types of adversarial attacks (Athalye al. 2018).
Several works studied the methods that generate stronger adversarial attacks AAthcks et al. 2018
Lee et al. 2017 Pаpernot et al. 2017 Moosavi-Deziooli et al., 2016; Dong al. 20188 Song et al. 2018). Carlini & Wagner (017)) pin-toints that defensive distillation network (Papernot et al. (2016) is not practical in that it exploits gradient masking, so they devised a powerful attack algo- rithm that avoids this problem. In an attempt to eliminate the gradient masking problem of softmax function, they adopted logits Z in objective function, and discovered an appropriate adversarial noise for each image utilizing line-search technique. However, most of these works have high computa- tional cost, so they are difficult to be applied to adversarial training.
There are many other defense methods that are not based on adversarial training (Li et al. 2019 | [unboo.. The above robust optimization problem can be generalized as convex outer adversarial polytope, which relaxes the activation function convex form to prevent misclassification (Wong & Kolter (2018). Certified defense algorithms guarantee at least a certain bounds of the proper label probability distributions against adversarial examples (Cohen et al. 2019 Liu et al. 2019
Raghu-
2

---

Under review as a conference paper at ICLR 2020
classifier
perturbation generator
Figure 1: Entire procedure of our method: Conventional convolutional neural network used as the classifier. The perturbation generator network receives one-hot encoded label as input, which processed with fully connected and up-convolutional layer, concatenates with the gradient image and the original image, and finally generates an adversarial perturbation.
nathan et al., 2018 In addition, some researchers recently studied the theoretical backgrounds of adversarial robustness (Dohmatob) 2019 Wang al. 2018 Roth al. 2019)
PROPOSED METHOD
3.1 NOTATIONS
We denote labeled training set by (æ. Pauta where E RH represents input images with height H, width W, and channel C, and y E (1, K} is label for input æ. We use two neural networks in the proposed method. One is standard K-class classifier network F(x; which is defined by:
F : R NXWXC + RK, F(x; 0) = (x; 0)1 (a; F() x]
(2)
Where F(x: 0) represents the class probability vector computed using the softmax function. The other is perturbation generating network F, Φ), which is defined by:
G: HRXC "xwxC IR) HXWXC
(3)
Note that G, o) represents the perturbation of the input image æ, where V F (xx; f)y denotes the gradient of class probability of the true label with respect to the input images x.
3.2 ADVERSARIAL TRAINING WITH GENERATIVE MODEL
The entire procedure of our algorithm shown in Figure Goodfellow's 's work on GAN inspired us to make the classifier and the perturbation generator compete with each other. Classifier F detines the network we are aiming to train and increase the robustness, and the perturbation generator G is the network which produces the perturbations that maximize the loss function of the classifier. The classifier network is trained with adversarial images produced by the generator network with the true label. The generator network assigns image æ, label y, and gradient image V F as inputs, which trained to maximize the loss function of the classifier. In other words, and G play the following two-player minimax game:
Enz,y Pan min max Loss(x + GG F. -
0
By the time this minimax game is complete, the classifier will have been trained with various attacks produced by the generator with the enhanced robustness against powerful adversarial attacks, while the generator will no longer find any vulnerability in the classifier, therefore only producing random noises. In Equation 4. c₁. is a hyper-parameter that adjusts the ratio between two cost functions. Cr is very low, will only find trivial solutions with extremely large perturbation power, and if cr.
3

---

Under review as a conference paper at ICLR 2020
is very high, it will only generate zero- pero-perturbation images. Therefore, it is crucial determine an appropriate ez. The theoretical meaning of Cy will be discussed in Appendix [A]
We believe that the strength of our algorithm is the flexibility of the generator's 's attack method.
Conventional adversarial training algorithms such as FGM have a fixed attack method. For example, adversarial perturbation in FGM is generated by normalizing the gradient of data sample, which is scaled by c. A classifier can be easily made robust against adversarial examples generated by these fixed methods. The following are two possible methods.
CASE 1. The classifier is trained to gradually reduce the gradient at data points. Gradually reducing the gradient prevents FGM from generating meaningful adversarial example from the gradient (also known as a gradient masking problems).
CASE 2. The classifier can only be trained to reduce the loss under the surface of the e-norm ball centered around data point. As a result, the network would be robust against adversarial examples generated by FGM, but could still be very vulnerable against adversarial examples located inside the e-norm ball.
Through the above methods, a classifier can be easily overfit to FGM attacks, but it becomes more vuinerable to other powerful attacks such Carlini&Wagner L2 attacks as shown in Figure 2 Through our experiments under FGM adversarial training, we observed a CASE overfitting prob- lem when is small, and a CASE 2 overfitting problem when c is large. The above gradient masking problem has also been discussed by other papers on adversarial training (Tramèr et al. 2018). This problem also occurs when the iteration number of PGD is relatively low. On the other hand, our algorithm continuously updates the parameters of a generator to maximize the classifier loss, which tends to produce more general attacks to given classifier. This means that classifier cannot easily overfit to attacks from a generator in our method. Experimental results also support our arguments.
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
We used CIFAR-10 10 and CIFAR-100 110 for our datasets, to verify the robustness of our trained network. We normalized the pixel value of the image to [0,1] prior to network training. This section only presents the results from CIFAR-100. The experiment with CIFAR-10 showed similar results, and interested readers can refer to the appendix for its results.
The model architecture and parameters for CIFAR are given in Table We used conventional ConvPool-CNN as the classifier network, and the generator network was designed to efficiently use gradients, images, and labels. One might assume that hyperbolic tangent function should be used for the final activation function of the generator network. However, if a hyperbolic tangent function used for generating the perturbation image, the adversarial image created must be clipped again for all i. This is known as a box- box-constraint problem, which might cause the network to get stuck in extreme regions (Carlini Wagner (017)) Therefore, we practiced the following technique proposed by Carlini & Wagner (2017) to avoid the clipping issue.
to the proper value of the image, i.e. 0 <= I; +, <=
1 (tanh (tanh -' 1) + F(. +)
Kadv 2
(5)
For our baseline for comparison, we used a naive network trained only with clean examples. For our control group, we set [Goodfellow et al. (2016)'s adversarial training with Fast Gradient Method (FGM), and Madry et al. (2018)'s adversarial training with Projected Gradient Descent (PGD). For attack methods, we used FGM, Momentum Iterative Method (MIM), DeepFool, and Carneni&Wagner (C&W), and evaluated the robustness of the network through the accuracy of the adversarial examples and the mean and median value of the L2 norm of the perturbaion generated by each attack. All the attacks and adversarial training methods above are boundede Detailed evaluation method will be discussed in section 4.2
All of our experiments used single RTX 2080 ti GPU with Cleverhans adversarial examples library (Papernot et al. 2018) to construct adversarial attacks, build defenses, and make comparison more effectively.
4

---

Under review as a conference paper at ICLR 2020
Table 1: Model architectures and parameters for CIFAR datasets
Classifier Network
Generator Network
Parameter
Input: x (32 32
Input: y (10 TOD)
Optimizer
Adam
3 3 Conv 64
Dense T024
Learning Rate
0.0005
3 x 3 Conv 128
Concatenate with
Batch Size
128
2 x 2 AvgPool
Dense 8192
Adv Coefficient
1.0
3 Conv 128
Reshape 8 8 128
PGD iter
10
3 x Conv 256
Concatenate with y (reshape)
Dropout
2 x 2 AvgPool
3 x 3 Upcony 128, stride=2
Weight Decay
3 X 3 Conv 256
Concatenate with y (reshape)
Ema decay
0.998
Conv 512
Upconv 128, stride=2 Concatenate with (ar, F=
Max Epochs
200
2 AvgPool
FGS attack eps
3 x 3 Conv 10 or 100
3 x 3 Upconv 128, stride=1
MIM attack eps
0.5
GlobalAvgPool
3 x Upconv 3, stride =1
MIM attack iter
100
Softmax
DF attack iter
100
Output: 10 or class probabilities
Ouiput: x 32 perturbation
CW attack iter
1.
Naive
Goodiellow ( 0.13
Goodhellow
Goodtellow (c=
425
Accuracy
0.6
6.5
0.20
0.6
6.4
033
62
0.30
benignn accurecy
01
MIEN Goad
0.05
20
45
Attack perturbation norm (L2 Moan)
opoch
Figure 2: Trade-off relationship between benign accuracy and the adversarial robustness. Left: Perturbation power-accuracy graph for FGM adversarial training with various epsilon against CW attack. The bigger the epsilon, the more robust the network to adversarial attacks, but less benign accuracy. Right: The network with FGM adversarial training tend to overfit easily because of its fixed attack algorithm. As the training progresses, the benign accuracy rises, whereas the adversarial robustness declines.
4.2 EVALUATION METHOD
We applied the following average distortion metric suggested by Carlini & Wagner (2017), in order to fairly evaluate the robustness of the network for various adversarial attacks.
Σ EAED
Aeal2: where D is a successful adversarial example set
(6)
The above P represents the mean value of L2 norm of the perturbation derived from the success- ful adversarial examples from the attack, the same value the area under the curve in Figure 2 Although can be measured for any attack methods, it is best to measure for the most powerful attack. Thus, we used Pew for Carlini& Wagner L3 attack on all of our experiments as the evaluation metric for robustness of the network.
However, it is not sufficient to use only the above metric in evaluating the robustness of the ad- versarial training algorithm. In most adversarial training process, there are some hyperparameters which could adjust the trade-offs the accuracy of benign examples and the adversarial examples (e for FGM, PGD adversarial training, cr. for algorithm). The above Pew tends to increase as the accuracy of the benign examples decline, as it can be shown Figure 2 In an extreme case, the network classify almost all the images as single class, benign accuracy (the accuracy of clean examples) would converge to 1% (for CIFAR-1000 but the Pew would spike. This trade-off occurs during the training process well. Figure] illustrates how benign accuracy increases, and Pow
5

---

Under review as a conference paper at ICLR 2020
Advorsorialy traincd with
Adverso/laly Salllllyyyyyyyyyy.. trained with
662
Priented Opaekin Destor
Natuabation Servenioo hetwors DDurl
Pettubbtion So ale Satwek 10us
0.0
Accurasy
Accuracy
650
043
5.0
52
53 0.1 35
0.0
02
33
05
Attack perturbation norm (L2 Mean)
Attack perturbation norm (L2 Mean)
Figure 3: Evaluation of the attack performance of the proposed generator network. Accuracy vs perturbation norm was plotted for various attack methods. Left: a classifier was adversarially trained with Fast Gradient Method. Right: a classifier was adversarially trained with Projected Gradient Descent.
decreases during the training process. Thus, for fair comparison of the robustness of the networks, it is desirable to match the accuracy of the benign examples before comparing Pow-
To better compare the performance of different adversarial algorithms, we must first train the models by adjusting hyper-parameters for each adversarial training, calculate benign accuracy and Pew for each trained model, and then draw a graph with the calculated values, connecting each relevant data point. Naturally, the structure of the classifier used for each adversarial training must be identical. We will call this graph robustness-curve. Figure 5 displays the robustness-cure sarial training. PGD adversarial training, and our algorithm. It should be noted that the method with outer curve is better adversarial training algorithm since Pew is higher at the same benign accuracy.
for FGM adver-
4.3 ATTACK PERFORMANCE OF THE PERTURBATION GENERATOR NETWORK
In order to verify the attack performance of the proposed generator network, we used two fixed classifier networks adversarially trained with FGM (e = 0.5) and PGD (==0.5), respectively. For each classifier network, we compared the performance of FGM attack, PGD attack, and PGN (proposed) attack. PGN was fully trained on the training dataset. We plotted the accuracy of the classifiers for these three attacks in terms of average perturbation distortion. The results are shown in Figure 3 For each model, is observed that our algorithm is capable of much more powerful attack than PGD and FGM. This difference in performance clearly justifies the superior defense performance of the adversarial training with PGN in the following section.
The perturbation generator network can be trained to fully exploit incoming gradients, images, and labels to generate adversarial perturbation optimized for the classifier. As for FGM or PGD, they have fixed norm, with single attack direction towards the gradient. Now with PGN, it has no restriction on both direction and size, and can be transformed into an optimized perturbation for the classifier.
4.4 DEFENSE PERFORMANCE VARIOUS ATTACKS
Based on the methods introduced in 4.2. we compared the robustness of the network trained by our suggested technique with that trained by the conventional adversarial training methods. FGM attack, MIM attack, DeepFool, and Carlini& Wagner Attack were used as attack methods. In white- box attacks, adversarial examples were generated through direct access to the model's 's gradient, while in black-box attack, accuracy was measured through the adversarial examples produced by an independently trained network. Table exhibits the robustness of the network when all the benign accuracy values of the adversarial networks
balanced to that of the naive network, and Figure 4 displays three perturbation L2 power (x) accuracy (y) graphs for C&W attack, with the benign accuracy for each adversarial networks set to 68%, 66%. and 63%, respectively.
FGM and MIM are attack methods that find the adversarial examples that can maximize the loss function of the classifier network on fixed La norm of perturbation power, the higher the accuracy

---

Under review as a conference paper at ICLR 2020
Table The comparison of the performance of the conventional adversarial training algorithms and our algorithm with 0.02 and cz 50. Benign accuracy of all defenses were balanced out with that of the baseline network before the comparison. Column 6: Prediction accuracies of White- Box attack and Black-Box attack for each attack algorithms. Column 4: MEAN L2 norm of the
adversarial perturbation (p, which defined Equation (6). Column 5: Median L2 norm of the
adversarial perturbations.
ACCURACY
MEAN L2 W-BOX
MEDIAN
ACCURACY
BENIGN ACCURACY
TRAINING TIME
DEFENSE
ATTACK
W-BOX
W-BOX
B-BOX
(SEC/EPOCH)
FGM
0.1775
0.3843
0.0167
0.2913
Baseline
Deepfool
0.0994
0.0626
0.6688
0,7002
8.18
C&W
0.0791
FGM
0,2213
0.52I1
Goodfellow
MIM
0.1068
0.5115
Deepfool
0.1669
0.1089
0,6902
0.6993
25.1248
C&W
0.1334
0.0867
0.6894
FGM
0.2139
0.5164
MIM
0,1000
0,5014
Madry 2018
Deepfool
0.1625
175.8571
C&W
0.1305
0.0851
0.6880
RGM
0.3906
3.6428
Ours
MIM
0.3444
0.6456
Deepfool
0.3184
0.2064
0,6958
0.7004
51,7681
0,2617
0.1674
Bunking
Figure 4: The comparison of the robustness of the defense methods with various benign accuracy. To properly compare the robustness of the networks, the benign accuracy of the networks needs to be balanced out. The graph displays three different curves, each representing the accuracy of the perturbation power with respect to the perturbation norm Carlini& Wagner Lg attack with different benign accuracy of 68%, 66%, and 63%, respectively.
of the adversarial examples, the more robust the network. On the other hand, DeepFool and C&W attack find the adversarial examples with the lowest L2 norm perturbation power that can fool the network; therefore, the robust network would have higher mean and median values of the adversarial perturbation power.
As you can from Table 2 algorithm outperforms the other adversarial training algorithms against all the attack methods. In white-box attacks, our algorithm showed the highest accuracy of adversarial examples against FGM and MIM attacks, and the highest power of adversarial per- turbations by DeepFool and Carlin&& classify the adversarial examples with greater accuracy compared with the other adversarial training algorithms. According to Table 2, FGM adversarial training and PGD adversarial training show a very similar performance. This is because minimal was applied to match the benign accuracy of the baseline network. Since neural network is locally linear, this minimal would make PGD and FGM generated adversarial examples to be almost identical. As you can see from Figure and Figure 55) as increases, Madry's 's method shows more robust performance compared with Goodfellow's s method.
Also, in black-box attacks, our method proved to
Training speed is also crucial issue in adversarial training. The proposed algorithm is slower than FGM because it trains the generator after finding the gradient image, while FGM immediately uses the gradient image to train the classifier. On the other hand, our algorithm is faster than Madry's 's which use PGD (multi-step gradient descent) to find the adversarial image. Note that the more iteration steps of PGD, the larger the speed-gap between Madry's 's and our algorithm we get.

---

Under review as a conference paper at ICLR 2020
Edfbeit dfbrrin stibn)o
Hulry Cur
6.72
benign accuracy
benign accuracy
8:00
0.4
662
660
6.50
61
0.2
0.5
0.2
0.5 0.4
75
5.5
Aro
Aro
Figure 5: Robustness-curve: A plot showing the relationship between benign accuracy and Pew by changing the hyper-parameters each adversarial training algorithm. Left: For FGM and PGD adversarial training, each data point was acquired through changing whereas for our algorithm, each data point was acquired through changing ez. The outer curves are considered more robust adversarial algorithms. Right: Robustness-cune -urrves for our algorithm under different capacities of the classifier. shows that the classifier is still underfitting in terms of adversarial robustness.
4.5 VARYING HYPER-PARE PARAMETERS
As mentioned in Section 4.2, adversarial training algorithms have a trade-off relationship between benign accuracy and robustness metric Pow VigureeSssualizess visualizes the relationship with plot consisting of the data points of benign accuracy and Pew. which are collected by using various for each adversarial training. For FGM and PGD adversarial training, the data points with higher Pow are models trained with bigger c, while for our algorithm, the data points with larger Pew are models trained with lower ez. It should be noted that the robustness-curve is an appropriate indicator for performance evaluation of the adversarial training algorithms, since it displays a comprehensive set of Pew with corresponding benign accuracy. As demonstrated in Figure our algorithm outperforms all the other adversarial training algorithms under all benign accuracy.
Furthermore, we plotted the robustness-curv by proportionally increasing the number of filters in each convolutional layer of the classifier. As can be observed in the second plot of Figure [3] the robustness-cure moves to the right as the capacity of the model increases, which means that the classifier may still be underfitted. In other words, the classifier trained with only clean examples tend to overfit easily to the training data with even a low capacity, whereas the classifier trained with various adversarial examples tend to underfit instead even with higher capacity networks. Although we were not able to deal with higher capacity due to the limits of the current hardware technology, expected that a far greater network capacity may be needed to achieve human- robustness.
CONCLUSION
This study proposed novel adversarial training method that boosts the robustness of a deep neural network against adversarial attacks. Based on a GAN framework, the classifier network and the generator network play two-player minimax game, which improves the robustness of classifier against adversarial examples. In generating adversarial examples, we use trainable perturbation generator network instead of fixed function as most of conventional adversarial training meth- ods. this method tend to overfit less, and strengthens the robustness against many different kinds of attacks. Our proposed method far more robust than existing adversarial training techniques. Since it computes adversarial examples through one-step inference, it is also more advantageous in training speed, compared to other techniques that use multiple steps in inner maximization.
Our experiment with CIFAR datasets have also proved the advantage of our approach, as the network trained by our method showed improved robustness and the state-of-the. performance against various attacks with different noise power. Although the proposed approach compares favorably with other methods, it believed that there is still room for improvement. One future direction is to study a generator network which is most effective for adversarial training.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of secu- rity: Circumventing defenses to adversarial examples. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceed- ings of Machine Learning Research, pp. 274,283,, Stockholmsmmsan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL htt:ttp:: httptttppp//roceednnnee....eeeee ngs mlr wrw.s attaa yel 8a html
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017, pp. 39-57, 2017. doi: 10. 10.11000 109/SP.2017.499 URL URt.tttsss https: /ttcps://doi.. wrg. /10 1109/SP 2017. 49
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research. pp. 1310-1320, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http: //proceedings. mmt.mng..m....................mm prwss v wwwp//hc..n.cccc/. html
Elvis Dohmatob. Generalized no free lunch theorem for adversarial robustness. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1646-1654, Long Beach, California, USA, 09-15 Jun 2019. PMLR. SKKLittpp//procediing
press/ (Tess/v97/dohmee htt////mmttoooll9///// html
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In 2018 IEEE Con- ference on Computer Vision and Putlern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 9185-9193, 2018. doi: 10.1109/CVPR.2015
00957. URL http:://penaccess/ wttppppeanaccsseeeeeceeeeeeeeeee thecvf.eevc.c.. com/ content__tvpppepeceeeeeee 2018/htm1/ Dong Boosting Adversarial CnteessarillAAttccaa. Attacks CVPR 2018 paper. html
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume MIT Press, 2016.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. http: org/
abs 1lss/1121 1/1//111111. 6572
Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal processing magazine, 29, 2012.
Jake Zhao (Junbo) and Kyunghyun Cho. Retrieval-augmented convolutional neural networks against adversarial examples. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings, 2017. URL htttttss///ppeeeeieeeee Forum? Herumnid=#GGU3Rdd
Hyeungill Lee, Sungyeob Han, and Jungwoo Lee. Generative adversarial trainer: Defense to ad- versarial perturbations with GAN, CoRR, abs/1705.003387 2017. ARLLatp::/// attt:://axii. http//iii.voor///////........ orr/ abs/ 100./170..c00 03387

---

Under review as a conference paper at ICLR 2020
Yingzhen Li, John Bradshaw, and Yash Sharma. Are generative classifiers more robust to adversar- ial attacks? In Kamalika Chaudburi and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learn- ing Research, pp. 3804-3814, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URI
ngg ml wrw.srrsssseeeeee hv97:///9//111 www.rr.rrss////////. html
Chen Liu, Ryota Tomioka, and Volkan Cevher. On certifying non non-uniform bounds against adver- sarial attacks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learming, volume 97 of Proceedings of Machine Learn- ing Research, pp. 4072-4081, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http: http:://proceedings www.
html
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Confer- ence on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL https:: htttps://openreview /wwpreeeeeeeeeee/////p net forum?
id rJzIBBADD
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakac, and Shin Ishii. Distributional smoothing by virtual adversarial examples. In 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL
org. abs 1507. 1 C0677
Seyed-Mohsen Moosavi-Dezfool Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2574-2582, 2016. httpp://doo: org 10. 1009 CVPR. 02.06.6.8222
doi: 10.1109/CVPR..2166222
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy, SP 2016, San Jose, CA, USA, May 22-26, 2016, pp. 582-597. 2016. doi: 10.1109/SP:201641
httt:t::: www.ooooorrg111. 1109/SP SP 2016.41
Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, 7. Berkay Celik, and Anan- thram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, AsiaCCS 2017, Abu Dhabi, United Arab Emirates, April 2-6, 2017, pp. 516,519, 2017. doi: 10. (0.1145/302293330330000 URL https: / /ww../oiiic/// org. 10. 1145/ / 3052333223333 3053009
Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Ku- rakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang, Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, and Rujun Long. Technical report on the cleverhans v2.1.0 adversarial examples library. arXiv preprint arXiv:1610. 1070.0078. 2018
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. In 6th International Conference on Learning Representations, ICLR 2018, Van- couver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings, 2018. URL net wwweet/forrmiidddyssstee
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test for detecting adversarial examples. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5498 5507, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http://proceed: ngs ml prwss peress/v9 http://///or/to// 9a html
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial ex- amples with generative models. In Advances in Neural Information Processing Systems 31: An- nual Conference Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada., pp. $322-8333, 2018. URL https: axxiv. org abs///b//1/0 07894
10

---

Under review as a conference paper at ICLR 2020
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan J. Goodfel- low, and Rob Fergus. Intriguing properties neural networks. In 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. htt.hhttp://axii..oo org/ abs 1112. 91
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian J. Goodfellow, Dan Boneh, and Patrick D. McDaniel. Ensemble adversarial training: Attacks and defenses. In 6th International Con- ference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings, 2018. URL. https:://penneeview. net forum?
1d=rkzYšeRZZ RZ.
Yizhen Wang, Somesh Jha, and Kamalika Chaudhuri. Analyzing the robustness of nearest neighbors to adversarial examples. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp- 5133-5142, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://procedlnns..... http:///w........./e/////////aaa
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th In- ternational Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pp. 528655866555 5295, Stockholmsmsaan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL attp:///rrceeedings.co mlr, wtw.s..reesvv/// wttps////wwooogoa...a html

---

Under review as a conference paper at ICLR 2020
APPENDIX
A THEORETICAL BACKGROUND
Madry et al. 2018) has presented the following adversarial loss instead of the conventional loss in his paper.
Pmadry (0) = Pan max [Loss( + ò 0)] = (( max BEB
log
(7)
dES
F(x + 6; 0)
This signifies that among the given training data points, it finds the data point that has the max- imum loss against perturbation from the allowed perturbations set S, and minimizes that specific loss. In other words, it trains Classifier F to satisfy Fa + 0) = 1 for all ô in S in allowed perturbations set S, which is only feasible when perturbation generator network can create perturbations with any size of power, merely applying the above adversarial loss would generate only the trivial perturbations with extremely high power. In order to extend the allowed perturbations set $ to all possible perturbations, we assume that optimal F(x 8; f)y has normal distribution over L2 norm of the as shown Figure 6 precise, we want to train Classifier to satisfy F(x + 6; f)y pa( where (O o2) and Ps is the Gaussian distribution function with mean and o² variance, and our adversarial loss corresponding to Equation 7 is as follows:
very small norm ball. However, since our
To be more
Pa (2)))
Pours (0) = Ex, Em y Paus
max 8E3
log
F( + 6; 0)y
e R 116112
Exy Pon max 8E3
log
log F(x: + 8:0);
/2m
(8)
EnyyyPan
max
2a |0112 log F(x + 6:0), log v 2
8E3
Egyyyaa Pans max Loss(x - 6, y; 0) - c C
8E3
Suppose 8t is defined as
= argmax [Loss( (x 0,y; cr.1011 C]. then,
min Pours (0) E Fayyyra Pam
min Loss(x 5" 5" -,
Ci
2 En min max [oss(( + G V x. y.
c1 - 3))2
G would converge 8* assuming G has sufficiently high capacity and at F provides enough in- formation on the classifier F, and the constant value C' could be ignored since we are only interested in finding the parameter 0 of the classifier. We can derive the Equation (4), with e₁ 2o The opti- mal point for F and G would be the point where Loss(ær 8, 0) cL 0 C in all data points, and here we can find the classifier network F' that has the improved robustness against ad adversarial examples.
12

---

Under review as a conference paper at ICLR 2020
N10.ot)
Figure 6: Adversarial Robustness with Gaussian Normal Distribution. Left: The red curve indicates our target Gaussian normal probability function with mean O and variance o². The blue dotted curve indicates the classifiers class probability of the label in accordance with the Le norm of the perturbation. As the training progresses, the classifiers class probability converges to the target function, ensuring the robustness of the network. Right: Conceptual illustration of the adversarial training. By minimizing the loss function on the region with large adversarial loss, the network becomes increasingly robust against adversarial attacks.
B PERFORMANCE FOR CIFAR-10
Table The comparison of the performance of the conventional adversarial training algorithms and our algorithm with 0.1 and CL 50.
ACCURACY W-BOX
MEAN L2
MEDIAN L2 W-BOX
ACCURACY
TRAINING TIME
DEFENSE
ATTACK
W-BOX
B-BOX
ACCURACY
(SEC/EPOCH)
PGM
Baseline
Deepfool
MIM
0.0939
0,2129
0.1844
0,3633
0.8527
0.9189
8.28
C&W
0.1742
0.1596
0.8611
FGM
0.6335
0.8428
Goodfellow al. 2015
Deepfool
0,5135
0.4502
0.9119
0,918
25.1424
C&W
FGM
0.6259
0.8369
Madry
Deepfool
0.5426
0.4977
0.437
0.9104
1.8468
0.9172
175.2541
C&W
0,4091
0,382
0.91
FGM
0.6534
0.8501
Deepfool 0.5958
MIM
0.8568
Ours
0.5102
52,1851
C&W
0.4365
0.4062
0.912
0.2
0.35
offbeeo2)
Cur
offbbrco))
0:00
687
03
041
655
6.5
0.7
048
0.4
233
66
3.7
0.8
1.0
Figure 7: Robustness-curve. Left: Varying hyper-parameter, Right: Varying capacity
13

---

Under review as a conference paper at ICLR 2020
C SOME ADVERSARIAL EXAMPLES GENERATED BY THE PROPOSED GENERATOR NETWORK
Table 4: Adversarial examples for various perturbation powers from the CIFAR-100 dataset. These images are generated from the generator which is fully trained to maximize the classification loss. Labels denote corresponding classification results.
Original Image
Low perturbation power (|Axl|2 R 0.1)
woman
sea
orchid
Adversarial Image
orange
cloud
aquarium fish
Original Image
Medium perturbation power (||xæ|12 s 1.0)
keyboard
rose
house
Adversarial Image
woman
chair
rabbit
Original Image
High perturbation power (|Ar||a ||2 R 2.5)
mushroom
fox
palm tree
Adversarial Image
bear
squirrel
snake
14

---

Under review as a conference paper at ICLR 2020
D EFFECT OF EACH INPUT ELEMENT ON THE ROBUSTNESS OF THE NETWORK:ABLATION STUDY
1.0
original
without gradient
without label
0.8
without image
0.6
Accuracy
0.4
0.2
0.0
1
2
3
L2 norm
In order measure how much contribution each input make in of the robustness of the classifier, we removed each input one by one, and plotted an accuracy vs average distortion curve. can be observed that the gradient is the most important factor in classifier robustness, and the other two factors have relatively smaller impact on classifier robustness.
Figure 8:
15


Reviews:
[{"review_id": "HkxX68w6FH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use the GAN framework for adversarial training. The proposed algorithm is mini-max loss plus L2-regularization on the perturbations generated by a generator network. Additionally, the paper shows a mathematical interpretation of the L2-term. Experiments on CIFAR-10 and CIFAR-100 shows that the algorithm achieved higher clean/adversarial accuracy in the settings.\n\nI vote for rejection. A major difference between the proposed method and adversarial training variants is whether we use neural networks or gradient-based algorithms to calculate the internal maximum in the mini-max formulation of the robust training. The focus of discussions in this paper should be why the former performs better. However, this paper does not provide enough justifications. Experiments are not convincing to support its advantage.\n\nMajor comments:\n(1) On page 2, this paper claims, \"The gradient descent based adversarial examples for robust optimization is not adaptive. Therefore, those neural networks are vulnerable to other types of adversarial attacks (Athalye et al., 2018).\" Please expand these two sentences.\n(2) It is weird to observe that adversarial training with PGD performs worse than FGM. It is possible that the baselines are weaker than they should be. For example, if evaluations use L2-based adversarial training as baselines, it can be improper. Please refer to this comment on OpenReview: https://openreview.net/forum?id=Hk6kPgZA-&noteId=BJVnpJPXM .\n\n======= Update =======\n\nThank you for the replies and updates. After reading the comments and the updated draft, I am still not convinced that the proposed method should be superior to existing attacks. Even though I agree that the proposed method might outperform PGDs or other attacks in some settings, the paper needs to confirm it either theoretically or empirically. However, it was doubtful whether the baseline methods are strong enough. The concern remained after authors' responses. Hence I keep my score.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HklZG5TYFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new adversarial training approach, where a generator is used to generate the most challenging adversarial examples and the classifier is trained to correctly classify the generated adversarial examples. In this way, the robustness of the classifier is expected to be improved. For the generator, the input includes the original data sample, the gradient of the classifier, and the true label of the data sample. The paper is clearly presented and easy to follow. The experimental results relatively support the main claims of the paper.\n\nThe reasons that I go towards mild rejection for this paper are as follows:\n\n1. The general idea of improving the robustness of a classifier to by feeding adversarial examples to it is not a new idea. As shown in the paper, the idea has been adopted in Goodfellow et al. (2015) and Madry et al. (2018). In addition, [1] comprehensively studied how to ensemble different adversarial attacks as \"training data\" of a classifier to improve its robustness. The paper generates adversarial examples with a generator that takes various things as inputs, which can be viewed as an extension to the method in [2]. [2] introduced a generator that only takes the data sample as inputs and the difference to the paper is that the one in this paper additionally takes gradients and label as inputs. With Goodfellow et al. (2015), Madry et al. (2018), and [1], it may not be too hard to apply the method in [2] to improve classifier robustness. Therefore, it is questionable whether the additional information used in the generator of this paper helps and how it helps. I did not see any comparison in the experiments.\n\n2. Although the experiments look to be well conducted, it is not comprehensive enough. For the defence methods in comparison, only two approaches that fall into the exact same line of the proposed method are included. In this line, I think it is necessary to compare with [1], which combines various attacks. Moreover, I would also expect a comparison with other closely related methods such as [3] and [4], to further demonstrate the effectiveness of the proposed one.\n\n3. It is also suggested to do ablation tests on the inputs of the generator to see how each part of the inputs helps. If gradients and true label are removed from the generator, it reduces to [2]. Therefore, those ablation tests also help the comparison with [2].\n\nMinor comments:\n\n1. It could be interesting to visualise the perturbations that are generated from the generator.\n\n2. Some of the figures and captions are too small to see in a printout.\n\n\n[1] Tramèr, Florian, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. \"Ensemble Adversarial Training: Attacks and Defenses.\" (2018).\n\n[2] Xiao, Chaowei, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. \"Generating adversarial examples with adversarial networks.\" arXiv preprint arXiv:1801.02610 (2018).\n\n[3] Samangouei, Pouya, Maya Kabkab, and Rama Chellappa. \"Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models.\" (2018).\n\n[4] Matyasko, Alexander, and Lap-Pui Chau. \"Improved network robustness with adversary critic.\" In Advances in Neural Information Processing Systems, pp. 10578-10587. 2018.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1xDmCCJ5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to construct adversarial attacks by training a neural network to produce a distortion, rather than by constructing the distortion directly via PGD. It then uses this during adversarial training to produce networks that it purports to be more robust.\n\nPros:\n-Faster speed of adversarial training could substantially democratize ability to work on adversarial ML\n-Neural net attack could overcome gradient masking in some situations\n\nCons:\n-Some of the main arguments seem incorrect, although the paper may still be strong if these incorrect arguments are removed and the experiments are further substantiated.\n\nDetails:\nI think there are some interesting ideas here, which if further substantiated could be the base of a strong paper. However, I found the overall argument in the current paper to be misleading or at least mistaken. A core claim is that the neural-net generated attack is better than PGD because PGD only generates a \"single attack\" while the neural network generates many different attakcs. But PGD is an adaptive attack method that depends on the current model parameters, so it changes over the course of training just like the neural net attacker. Furthermore, both models approximate the same maximization over the L2 ball (in the neural net's case, rather than maximizing over the L2 ball we penalize the L2 norm, but a similar modification can easily be made to PGD). So it is not clear why we should expect the neural net to perform better, except perhaps that it is a different type of way of approximating the same objective that PGD approximates, and perhaps this approximation has some favorable properties? But this is never justified.\n\nHere is one attempt to justify why the neural net attack should be better: perhaps while PGD often works well, it gets stuck on certain examples, e.g. due to gradient masking issues. While the neural net might initially get stuck on those examples (it also has to work by taking gradients), since it can share knowledge across examples it initially learns a good direction for the \"stuck\" examples by generalizing from the \"unstuck\" examples. This overall allows it to generate a more consistently good attack even in the presence of (partial) gradient masking. If we believe this story, then actually the neural net attacker should already work well just as an attack (i.e. just train it to generate good attacks against a fixed model for some set of examples). I would find this quite interesting if true but unfortunately it wasn't explored in the paper. One way to do it would be to take models that are believed to be robust based on attacking them with PGD, and then show that the neural net attacker is substantially more accurate.\n\nI do find the point about faster speed interesting. This is given as a minor comment but it may be the biggest benefit of the current method. Adversarial training on large-scale datasets like ImageNet is computationally infeasible, and your method could potentially address this and allow people who don't have $100k+ compute budgets to actually work on adversarial ML.\n\nI also have some comments on the evaluation:\n\n-Reporting average accuracy across attacks in the evaluation is an inappropriate summary statistics. Min accuracy would be better, which in this case is zero for all methods. This suggests that you either did not adversarially train appropriately, or used too large of a norm (what norm did you use anyways? Or is this allowing the norm to be unbounded until you fool the model? In that case all the numbers should be zero for any reasonable attack.)\n\n-For evaluation I would rather see the full curve of accuracy vs. allowed attack norm, rather than just a single summary metric. It's too hard to tell what is going on from a single number. It is also hard to compare against adversarial training methods that penalize the L2 norm versus constrain the L2 norm, as the former might do better than the latter simply due to more closely conforming to your evaluation of average distortion. Having the full curve helps better assess this. Average distortion is also only meaningful if the method is finding the minimum-norm attack point that changes the label, which most of the methods you consider do not do. You do include the full curve in Figure 4 (please make the font bigger though). Under the full curve it seems the improvement over other methods is minimal, and could possibly be due to hyperparameter tuning.\n\nFinally, more minor but the Gaussian derivation does not make sense. It is unlikely that your norms actually follow a Gaussian distribution. A more appropriate claim would be that you are constraining or penalizing the expected norm of the perturbations. I would remove the math part as it does not add anything to the paper (and is also wrong as per preceding comment) and focus more on how you actually construct the generator network (this is only briefly discussed in the appendix and not at all in the main text, even though it is a key point to getting the method to work). You could also use the space for more detailed experiments, following best practices as in https://arxiv.org/abs/1902.06705 to ensure that your evaluation is sensible.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use the GAN (i.e., minimax) framework for adversarial training, where another neural network was introduced to generate the most effective adversarial perturbation by finding the weakness of the classifier. The rebuttal was not fully convincing on why the proposed method should be superior to existing attacks.
Decision: Reject