Learning to Defense by Learning to Attack
Authors: Zhehui Chen
Haoming Jiang
Yuyang Shi
Bo Dai
Tuo Zhao
Venue:  (2020)

Abstract: Adversarial training provides a principled approach for training robust neural networks. From an optimization perspective, the adversarial training is essentially solving a minimax robust optimization problem. The outer minimization is trying to learn a robust classifier, while the inner maximization is trying to generate adversarial samples. Unfortunately, such a minimax problem is very difficult to solve due to the lack of convex-concave structure. This work proposes a new adversarial training method based on a generic learning-to-learn (L2L) framework. Specifically, instead of applying the existing hand-designed algorithms for the inner problem, we learn an optimizer, which is parametrized as a convolutional neural network. At the same time, a robust classifier is learned to defense the adversarial attack generated by the learned optimizer. Our experiments over CIFAR-10 and CIFAR-100 datasets demonstrate that the L2L outperforms existing adversarial training methods in both classification accuracy and computational efficiency. Moreover, our L2L framework can be extended to the generative adversarial imitation learning and stabilize the training.

Under review as a conference paper at ICLR 2020
LEARNING TO DEFENSE BY LEARNING TO ATTACK
Anonymous authors
Paper under double-blind review
ABSTRACT
Adversarial training provides a principled approach for training robust neural net- works. From an optimization perspective, the adversarial training is essentially solving a minimax robust optimization problem. The outer minimization is try- ing learn robust classifier, while the inner maximization is trying to generate adversarial samples. Unfortunately, such a minimax problem is very difficult to solve due the lack of convex-concarl structure. This work proposes a new ad- versarial training method based on a generic learning- ta-gg-tto-earn (L2L) framework. Specifically, instead of applying the existing hand-designed algorithms for the in- ner problem, we learn an optimizer, which is parametrized as convolutional neu- ral network. the same time, robust classifier learned to defense the adver- sarial attack generated by the learned optimizer. Our experiments over CIFAR-10 10 and CIFAR- 100 datasets demonstrate that the L2L outperforms existing adversar- ial training methods in both classification accuracy and computational efficiency. Moreover, our L2L framework can be extended to the generative adversarial imi- tation learning and stabilize the training.
INTRODUCTION
This decade has witnessed great breakthroughs in deep learning in a variety of applications, such as computer vision (Taigman et al. [2014] Girshick et al. 2014 He et al.] 2016 Liu et al. 2017). Recent studies (Szegedy et al. 2013), however, show that most of these deep learning models are very vulnerable to adversarial attacks. Specifically, by injecting small perturbation to normal sample, one can obtain an adversarial example. Although the adversarial example is semantically indistinguishable from the normal one, it can fool deep learning models and undermine the security of deep learning, causing reliability problems in autonomous driving, biometric authentication, etc.
Researchers have devoted many efforts to studying efficient adversarial attack and defense (Szegedy ctt al. [013 Goodfellow et al. 2014b Nguyynn et al. 2015 Zheng et al., 2016 Madry et al. 2017 Carlini and Wagner 2017). There 1s growing body of work on generating adversarial exam- ples, e.g., fast gradient sign method (FGSM, Goodfellow et al. (2014b)), projected gradient method Kurakin KKra,,Kurakin al. (2016)) Carlini- -aalner (CW, Waszke al. (2017) etc. As for defense,
fellow et al. (2014b) propose to robustify the network by adversarial training, which trains over the adversarial examples and still requires the network to output the correct label. Further, |Madry al. 2017) formalize the adversarial training as the following minimax optimization problem:
ming E.= [maxs,eB l(f(x; + 0,; y4)],
where ((x; v:) C IRdd x y are pairs of input feature and the corresponding label, l denotes a loss function, f(;50) denotes the neural network with parameter 0, and d; B denotes the perturba- tion for æ; in constraint B. The existing literature on the optimization also refers to 0 as the primal variable and 8,'s 's as the dual variables. Different from the well-studied convex-concan
problen I problem ( is very challenging since E is nonconvex in 0 and nonconcave in 5. As result, there may be many equilibria, and majority of them are unstable. In the cxisting optimization literature, there is algorithm to converge to a stable equilibrium with theoretical guarantees. Empirically, the existing primal-dual algorithms may perform poorly for solving ①I.
Minimax formulation naturally provides with unified perspective on prior works of adversar- ial training. Such a minimax problem consists of two optimization problems, an inner maximization problem and an outer minimization problem: The inner problem targets on finding an optimal attack
Loss function {(: 5) is convex in primal variable @ and concave in dual variable 8.

---

Under review as a conference paper at ICLR 2020
for a given data point (z. maximizing the loss, which essentially is the adversarial attack; The outer problem aims to find a f so that the loss given by the inner problem is minimized.
For solving (I), Goodfellow et al. (2014b) propose to use FGSM to solve the inner problem. Ku- rakin et al. (2016) then find that FGSM with true label predicted suffers from a "label Icaking" effect, which can ruin the adversarial training. Madry et al.) (017)) further suggest solve the inner prob- lem by PGM and obtain a better result than FGSM, since FGSM essentially is one iteration PGM. However, adversarial training needs to find òe for each (æ:, W1). thus the dimension of the overall search space for all data is substantial, which makes the computation expensive.
Instead, propose a new learning-to-leam (1.21.) framework that provides more principled and efficient way for adversarial training. Specifically, we parameterize the optimizer of the inner max- imization problem by a neural network denoted by ((A(z, y, 0): Φ). where A(x, y/ 0) denotes the input of the optimizer g. We also call the optimizer as the attacker network. Since the neural net- work is very powerful in function approximation, our parameterization ensures that g is able to yield strong adversarial perturbations. Under our framework, instead of directly solving ö; we update the parameter & of g. Our training procedure becomes updating the parameters of two neural networks, which is very similar to generative adversarial network (GAN, Goodfellow et al, (2014a)). The pro- posed L2L is a generic framework and can be extended to other minimax optimization problems, e.g., generative adversarial imitation learning, which is studied in Section 4
Different from the hand-designed methods that compute the adversarial perturbation for each in- dividual sample using gradients from backpropagation, our methods generate the perturbations for all samples through the shared attacker g. This enables the attacker g to learn potential common structures of the perturbations. Therefore, our method is capable of yielding strong perturbations and accelerating the training process. Furthermore, L2L framework is very flexible: we can either choose different input A(x, y. 0), or use different attacker architecture. For example, we can include gradient information in A(x, y. 0) and use a recurrent neural network (RNN) to mimic multi-step gradient-type methods. Instead of simply computing the high order information with finite differ- ence approximation or multiple gradients, by parameterizing the algorithm as neural network, our proposed methods can capture this information in much smarter way (Finn et al.] (017). Our experiments demonstrate that our proposed methods not only oulperform existing adversarial train- ing methods, e.g., PGM training, but also enjoy the computational efficiency over CIFAR-10 and CIFAR- 100 datasets (Krizhevsky and Hinton 2009).
The research on L2L has a long history (Schmidhuber 1987: 1992 1993 Younger et al., 2001
Hochreiter et al.) |001| AAdryychwicc al. 20166). The basic idea is that one first models the updat- ing formula of complicated optimization algorithms in a parametric form, and then uses some simple algorithms, e.g., stochastic gradient algorithm to learn the parameters of the optimizer. Among isting works, (oochs,Hocrreiterr et al.] (2001) propose a system allowing the output of backpropagation from one network to feed into an additional learning network, with both networks trained jointly; Based on this, Andryyhhwicc et al. (0016) further show that the design of an optimization algorithm can be cast as a learning problem. Specifically, they use long short-term memory RNNs to model the algorithm and allow the RNNs to exploit structure in the problems of interest in an automatic way, which is undoubtedly one of the most popular methods for learning-to-leamd
However, there are two major drawbacks of the existing L2L. methods: (1) It requires a large amount of datasets (or a large number of tasks in multi-task learning) to guarantee the learned optimizer to generalize, which significantly limits their applicability (most of the related works only consider the image encoding as the motivating application): (2) The number of layers/iterations in RNNs for modeling algorithms cannot be large to avoid significant computational burden in backpropagation.
Our contribution is that we fill the blank of the I.2I. framework solving the minimax problem, and our proposed methods do not suffer from the afforementioned drawbacks: (1) The attacker g with a different & essentially generates a different task/dttaset. Therefore, for adversarial training, have sufficiently many tasks for learning-too to-lee;;; (2) The inner problem does not need large scale RNN, and we use a convolutional neural network (CNN) or a length-two RNN (the sequence of length equals 2) as our attacker network, which eases the computation.
Our work is also related to GAN and dual-embedding (Dai et al., 2016). All three solve minimax problems and share some common ground. We discuss these works in Section
2

---

Under review as a conference paper at ICLR 2020
Notations. Given R, denote (a)+ as max(a, 0). Given æ. y R denote v; as the i-th element of æ. | = max; |2y| as lo- -nom of æ, and zoy = [1191. 11 Tayal as element- wise product.
METHOD
This paper focuses -no-m attack. We define the -ball with center O and radius c by ((c)=
{8 IR |10||oc I e} and the corresponding
projection as follows:
(0) - sign(0) max(1
Limiting Cycle Figure 1: Illustration for the hardness of prob- lem (I). wrong update direction leads to limiting cycle and algorithms fail to converge. See Appendix(Djor more details.
Ideal Reality
where sign and max are element wise operators.
2.1 ADVERSARIAL TRAINING
The goal of adversarial training is robustify neural networks. Recall that from robust optimiza- tion perspective, given n samples { 31)) )= where a; is the i-th feature vector and y is the corresponding label, adversarial training is reformulated as the following optimization problem [
ming
- [maxs,eB ((f 5;; v:)]
where denotes the network with parameter 0, l denotes loss function, and is the maximum perturbation magnitude.
We first demonstrate the hardness of solving problem (I] Ideally, want to obtain the optimal solution for the inner problem
6 == argmax
6,58 EB ((f(x; + 6;; 0);0)..)..
However, note that the loss function {( + 8;; 0), 3/5) ) is highly nonconcave ö, Therefore, in reality the d, we obtained is very unlikely to be the optimum d; This then often leads to highly unreliable or even completely wrong search direction, e.g.,
(Vel(f(x, + 5,; 0), v4), Vol(f(æ; + 0)) <0
which may further results in a limiting cycle shown Figure ( discussion provided in Appendix D). This becomes even worse when sample noises exist.
In the existing literature, the standard pipeline of adversarial training is shown Algorithm [ Since the step gen- erating adversarial perturbation 6, in Al- gorithm [ is intractable, most adversar- ial training methods adopt hand-designed algorithms. For example, Kurakin et al. (2016) propose to solve the inner problem approximately by first order methods such as PGM. Specifically. PGM iteratively up- dates the adversarial perturbation by the projected sign gradient ascent method for each sample: Given one sample (ær;, y:), at the /-th iteration, PGM takes
Algorithm 1 Standard pipeline of adversarial training
Input: { y:) clean data, ac learning rate, N: number of epochs, e: maximum perturbation magnitude. for & t to N do
Sample minibatch M, for in M, do
6, aragmaxx, ((f(x, 5,; 6,; y1)
Generate adversarial perturbation for (æz4, V:)
0 - 0 a TT EieM, Vol(f(æ; + 5;; 0), yi)
Update @ over adversarial data ((x, + 6,, y/) ( hem..
8 [l (8/-1 + 17-77810 sign (V v)))
where / =; and 5 = 1, T. Finally PGM takes 6, = 5%. Note that FGSM essentially is one-iteration PGM. Besides, some works adopt other optimization methods, such as momentum gradient method (Dong al. 2018), and L-BFGS (Tabcco and Valle 2016). However, except for FGSM, all require numerous queries for gradients through backpropagation, which is computationally expensive.
is the perturbation step size, T' is pre-defined total number of iterations,
2.2 LEARNING TO DEFENSE BY LEARNING TO ATTACK (L2L)
Since the objective function is nonconvex-nonconeec methods to perform well. Instead, we propose to learn an optimizer for the inner problem. Specif- ically, we parameterize the attacker by Neural Network g(A(ar, y, ()), where A(xr, y, 0), the
there is no guarantee for hand-designed
3

---

Under review as a conference paper at ICLR 2020
input of the network g. summaries the information of data and neural network f(; 0). We then convert problem [ to
ming N/2 E.= [e(f(x; + g(A(x,, yr, 0); ) 0), vi)],
(3)
where o is defined as the solution to the following optimization problem:
"
Φ E arrgaxx
TI Σ
E(f(x; ((((0 Vs. 0); Φ): 0). ys) subject to ((A(z,y.0)) B(c).
Solving problem B naturally consists of two stages. In the first stage, the classifier f aims to fit over all perturbed data; While in the second stage, given a certain / obtained in the first stage, the attacker network targets on generating optimal perturbations under constraints 5,'s E B(e).
Since 5; = (A(r,, Ms: 0); Φ), constraints can be simply handled by tanh activation function in the last layer of g. Specifically, because the magnitude of tanh output is bounded by after rescale the output by e, the output of g automatically satisfies the constraints.
This framework is very flexible. We can choose different A(æ, y. 0) as the input and mimic multi-step gradient algorithms shown in Fig- ure 2
Unfold
Classtier
Optimtoer
Optinizer
Glassifer
Optimiter
Here we provide three examples:
V.0(o-+
Zeee
One-Step
Multi-Sep
Naive Attacker Network.
This is the simplest example among our methods, taking the origi- nal image æ; the input, i.e.,
Figure 2: An illustration of 1.21., We learn a neu- ral network to model the algorithm for generating adversarial attack.
A(a.; y/s 0) a;; and 5, g(a;;; Φ)-
Under this setting, L.2L training is similar to GAN training. The major difference is that the generator in GAN yields synthetic data by transforming random noises, while the naive attacker network generates perturbations by transforming training samples.
Gradient Attacker Network. Motivated by hand- designed methods, e.g., FGSM, design tacker which takes the gradient information into computation. Specifically, we concatenate image æ; and gradient [= O), from backpropaga- tion the input of g, i.e.,
pass
Concatenate
2 pass
Original Input
Backprop
1 Attacker
A(xi, Vi. 0) = pi,, V=l(J(a;;; V),
and 8, - 9 v=l(f(x;; (),,;)),20 y)):
Gradicnt
Since more information is provided, we expect the attacker network to be more effective to learn and yield more powerful perturbations.
Multi-Step Gradient Attacker Network. We adapt the RNN to mimic a multi-step gradient update. Specifically, we the gradient attacker network the cell of RNN sharing the same parameter Φb. As mentioned earlier, the number of layrrs/iterations in the RNN for modeling algorithms cannot be very avoid significant computational bur- den in backpropagation. In this paper, we focus on a length-two RNN to mimic a two-step gradient update.
Classifier /
Porturbatinn
Cleun Loss
Adv. Loss
Penurred
Ingut
large
Figure 3: The architecture of L2L adversar- ial training with gradient attacker network.
The corresponding perturbation becomes:
ō; IB (6(0) g Val([( ye 0); ())),
where x. = x. (0) and (000) = g (z., Val(f(x,, 91: ():
example, Figure 3 illustrates how L2L works and jointly trains two networks: The first forward pass is used to obtain gradient of the classification loss over clean data; The second forward pass is used generate perturbation 5; by attacker g; The third forward pass is used to calculate the adversarial loss e in (3). Since our gradient attacker network only needs backpropagation query gradient, it amortizes the adversarial training cost, which leads to better computational efficiency. Moreover, L2L may adapt to the underlying optimization problem and yield better solution for the inner problem. The corresponding procedure of L.LL is shown in Algorithm 2
Taking gradient attacker network
4

---

Under review as a conference paper at ICLR 2020
3 EXPERIMENTS
Algorithm 2 Learning- to poo-leam--assed adversarial training with gradient attacker network
To demonstrate the effectiveness and computational efficiency of our meth- ods, compare our methods with PGM training over CIFAR-10 and CIFAR-100 datasets. All experiments are done in Py'orchh with one NVIDIA 2080 Ti GPU.
Input: { yt W)] clean data, 01, Oz: learning rates, N:
number of epochs, c: maximum perturbation magnitude. for 4- to N do
Sample a minibatch Mr for i in M, do
u; x Vall((=;;))
For simplicity, we denote Plain Net as the classifier network trained over clean data only, PGM Net as the clas- sifiers with PGM training, and Naive L.21,, Grad L.2., and 2-Step L.L as classifiers using L2L training with cor- responding attacker networks.
6, x f-z, usi Φ) Generate perturbation by
0
an [.. ΣieM, Vol + 5,; f). yi)
Update 0 over adversarial data {(z; + 6, 14) h.M. s + & + 02 LM. Σiem, Vol(S(a + 6,0), (1)))
Update & over adversarial data {(2; + 6, yi Bi)liem.
Classifier Network. All experiments adopt a 34-layer wide residual network (WRN-34-10, Zagoruyko and Komodakis (2016) implemented by /Zhan et al. (2019) as the classifier network. For each method, we train the classifier network from scratch. For Plain Nets, we adapt the same training procedure as Zagoruyko and Komodakis (2016). We use cross entropy as the loss function.
Attacker Network. Table D presents
Table 1: Attacker network architecture; k. C, 8, denote the kernel size, output channels, stride and padding pa- rameters of convolutional layers, respectively.
the architecture of our attacker network
The ResBlock uses the same structure as the generator proposed in Miyato et al.] (2018). The detailed structure of Res- Block is provided in Appendix [ Batch normalization (BN) and activations, e.g., ReLU and tanh, are applied when speci- fied. The tanh function easily makes the output of attacker satisfy the constraints.
Conv: [k =k=3 3 3, 64, 1, 11.
BN+ReLU
ResBlock: [k 3 3,c = 128, 8 1,p
ResBlock: [] = 3 x 3,C 256, s = 1,p
ResBlock: [A = 3 x 3, c = 128, s = 1, p =
Conv: [ = 3 3, c = 8 = p 1]. tanh
White-box and Black-box. We compare different methods under both white-box and black-box settings. Under the boxte-box setting, attackers can access all parameters of target models and generate adversarial examples based on the models; whereas under the black-box setting, accessing parameters is prohibited. Therefore, we adopt the standard transfer attack method from Liu et al. (016)) Due to the space limit, we leave results of the black-box setting in Appendix A
Robust Evaluation. We evaluate the robustness of the networks by PGM and CW attacks with the maximum perturbation magnitude and CIFAR- For PGM attack, we use 20 and 100-iteration PGM with a perturbation step size r 0.003, and for each sample initial the adversary randomly in the &o ball of radius centered at the natural sample. For CW attack, we adopt the implementation from Paszke et al. (2017). and set the maximum number of iterations random initial seed and report the worst result. For CIFAR-10, we also evaluate the robustness of our Grad L2L and 2-Step L2L networks using random attacks, for which we uniformly sample 10000 perturbations in the foo ball with radius = 0.031 adding to each test sample. We also evaluate the robustness of our Grad L2L and 2tSp L2L networks under their own attackers.
0.031 (after rescaling the pixels to [0, 1]) over CIFAR-10
100. For each method, repeat 5 runs with different
L2L. To update classifier's parameter O, we use the SGD algorithm with Polyak's 's momentum (pa- rameter 0.9, Liu et al. (2018)) and weight decay (parameter x 10 4, .Koog and Hertz (1992): To update the attacker's s parameter & we use the Adam optimizer (parameter [0.9, 0.999). Kingma and Ba (2014) and weight decay (parameter 2 10-4) so that it adaptively balances the inner and outer optimization. In addition, we train the whole network for 100 epochs and set the initial learning rate for SGD as 0.1, the learning rate decay parameter as 0.1 with decay schedule [30,60,90], and the initial learning rate for Adam optimizer 0.001 without learning rate decay.
We provide another attacker architecture with down-samplin modules in the Appendix attacker, 1.21. adversarial training is less stable, but much faster.
B With such an
5

---

Under review as a conference paper at ICLR 2020
PGM Adversarial Training. For CIFAR-10, we directly report the result from Madry et al. (2017) as the baseline; For CIFAR-100, we train a PGM Net as a baseline. To update the parameter 0. we use the same configuration of SGD algorithm in L2L. Moreover. et al. (2017), that is, we first use 10-iteration PGM with the perturbation step size 0.007 in to generate the adversarial samples, and then train the PGM Net over these samples.
adopt the setting from Madry
Table 2: Results of different defense methods under the white-box setting.
Defense Method
Attack
Datasct
Clean Accuracy
Robust Accuracy
Zheng al. (2016)
PGM-20
CIFARIO
94.64%
0.15%
Kurakin et al. (2016
PGM-20
CIFARIO
85.25%
45.89%
Madry et al.
(017)
PGM-20
CIFARIO
87.30%
47. 47.04%
Naive L.LL
PGM-20
CIFARTO
94.53%
0.01%
Grad L.L
PGM-20
CIFARIO
85.84%
51.17%
2-Step L.L
PGM-20
CIFARIO
85.35%
54.32%
Grad L2L
PGM-100
CIFARIO
85.84%
47.72%
2 2-Step L.2L
PGM-100
CIFARIO
85.35%
52.12%
Grad L2L
CW
CIFARIO
85.84%
53.5%
2-Step L.21.
CW
CIFARIO
85.35%
57.07%
Grad L2L
Random
CIFARIO
85.84%
82.67%
2-Step L.2I.
Random
CIFARIO
85.35%
83.10%
Grad L.21
Grad L2L
CIFARIO
85.84%
49.68%
2-Step I.2I.
2-Step L.L
CIFARIO
85.35%
52.71% %
PGM Net
PGM-20
CIFARTOO
62.68%
23.75%
Grad L2L
PGM-20
CIFARI00
62.18%
28.67%
2-Step L2L
PGM-20
100
60.95%
31.03%
PGM Net
PGM-100
CIFAR 100
62.68%
22.06%
Grad L2L
PGM-100
CIFAR100
62.18%
26.69%
2-Step L2L
PGM-100
CIFARI00
60.95%
29.75%
PGM Net
CW
CIFAR 100
62.68%
25.95%
Grad L2L
CW
CIFAR 100
62.18%
29.65%
2-Step 1.21.
CW
CIFAR
60.95%
32.28%
Table 3: Running time for one epoch over CIFAR- 10 (1" Row) and CIFAR- 100 (2"d Row). (Unit: s)
Plain Net
PGM Net
Naive T.21.
Grad 1.21.
2-Step 121.
106.5 + 1.5
1310.8 14.2
293.7 + 3.1 310.0 + 2.9
617.5 6.1
805. 8.1
106.9 + 1.4
1354.8 + 14.1
623.1 + 6.3
824.7 + 8.4
Experiment Results. Table 2 shows the results of all methods over CIFAR-I0 and CIFAR-100 under the white-box setting. As can be seen, without gradient informa- tion, Naive L2L is vulnerable to the PGM attack. However, when the attacker uti- lizes the gradient information, Grad L.I and 2-Step 1.21. significantly outperform the PGM Net over CIFAR-10 and CIFAR- 100, with small loss for the clean accu- racy. From the experiments on CIFAR- 10, our Grad L2L and 2-Step L2L are ro- bust to random attacks, where the accuracy is only slightly lower than the clean ac- curacy. Furthermore, the accuracy of our Grad/2-Step L2L model under the Grad/2- Step L2L attacker is comparable to the ac- curacy under PGM attacks, which shows
2-lp Giwd Lat.
25ag Ltt. Gred Po
TCM Not
Dilloreees
Sal L.I.
24bpy Est.
Grdd L2L.
30 30
0.55
1.38
0.35
Number Dieration PGM 0.0311
Perterbatiee Magatuda
-10
Figure 4: Robust accuracy against perturbation mag- nitudes and number of iterations of PGM over CIFAR- 100. Top two figures show the absolute accuracy over adversarial samples; the bottom two show the differ- ence accuracy (PGM Net as baseline). For larger per- turbation magnitude see Appendix B
that L.L attackers are able to generate strong attacks. As can be seen, PGM-100 is stronger than Grad L2L attacker (47.72% vs. 49.68%), but similar to the 2-Step L2L attacker (52.07% vs. 52.71%). This means the strength of 2-Step L2L attacker is much stronger than Grad L2L attacker and explains why 2-Step L2L is stronger than Grad L2L and PGM net. In addition, Table 3 shows the one epoch running time of all methods over CIFAR-10 and CIFAR-100. As be seen, Grad L2L and 2-Step L2L is much faster than PGM Net. By further comparing the accuracy of Grad/2- Step 1.2I. and PGM Net in Table 2, find that 1.I.. enjoys computational efficiency. In addition,

---

Under review as a conference paper at ICLR 2020
Figure 4 presents the robust accuracy against number of iterations (fixed perturbation magnitude é <= 0.031) and perturbation magnitude (fixed number of iterations T = 10). As can be seen, 2-Step L2L is much more robust than PGM Net.
Visualization of Adversarial Examples. F provides an illustrative example of the adversar- ial perturbations generated by FGSM, PGM-20 and 2-Step L.2L. attacker for over CIFAR-IO. As can be seen, the attacks for these two networks are very different. Moreover, the perturbation generated by the 2-Step L2L attacker is much more smooth than FGSM and PGM. In this example, 2-Step L2L labels all adversarial samples correctly; whereas the PGM Net is fooled by the PGM-20 attack and misclassifies as dog.
Adversarial Sample
Maguitade
30 Maguitude
Adversarial Sample
Magaitude
Magaitude
(a) Adversarial samples from 2-Step L2L.
(b) Adversarial samples from PGM Net.
Figure 5: Illustrative examples of FGSM (Top), PGM-20 (Mid), and 2-Step L2L (Bottom) perturba- tions for under PGM Net and 2-Step L2L with perturbation magnitude 0.031.
EXTENSION: GENERATIVE ADVERSARIAL IMITATION LEARNING
Our proposed L2L is a generic framework and can be applied to a broad class of minimax opti- mization problems. Here, investigate the generative adversarial imitation learning (GAIL, Ho and Ermon (2016)). GAIL aims to learn policy from expert's behavior, by recovering the expert's cost function and extracting policy from the recovered cost function, which can also be formulated as minimax optimization problem. We then show that under the I.21. framework, we stabilize the GAIL, training by replacing the inner optimizer with a neural network.
Original GAIL. GAIL solves the following minimax optimization problem:
min max E. (s,a==(s [log (D(s, a; 0p))) + Bg,a-Ty [log(1 - D(3 ā 0p))] (H)),
(4)
8x 8p
where xx)00)) is the trained policy parameterized by 0,, #g denotes the expert policy, i(::; is the discriminator parameterized by 0p. AH (T)) denotes a entropy regularizer with tuning parameter X, (s,a) and (5, a) denote the state-action for the trained policy and expert policy, respectively. By optimizing 4 the discriminator D distinguishes the state-action (s, a) generated from the learned policy with the sampled trajectories (s, a) generated from some expert policy Tg. In the original GAIL training, for each iteration, we update the parameter of D, Op. by stochastic gradient ascend and then update O, by the trust region policy optimization (TRPO, Schulman al. 2015)).
GAIL with I.2I. Similar to the adversarial training with 1.21,, we apply our 1.21. framework to GAIL by parameterizing the inner optimizer as neural network ( ((:0) with parameter dy. Its input contains two parts: parameter Op and the gradient of loss function with respect to Op:
(D 0x) (. Veo log (D(s, a; pp))] + EE Vep log(1 (f)));(p)))
In practice, we use a minibatch (several sample trajectories) to estimate gD (Ap, 0z), denoted as GD (Op, 0.). Specifically, at the t-th iteration, we first caleulate Ø (0t 0%²) and then update 801 U(A Tb: (.)) Next, we update du by gradient ascend based on the sample estimate of
Ee,0nx(e:05) V log (D(s, a; (O): Tb: 0f))) + Ex,ä~m Vou log(1 - D ã; U U Tb: R))) The detailed algorithm is presented in Appendix C
Experiments. We compare the performance of the original GAIL and GAIL with L2L on two simulated environments: CartPole and Mountain Car (Brockman al. 2016). As be seen in
7

---

Under review as a conference paper at ICLR 2020
Figure 6 find that GAIL has a sudden performance drop after training for long time. We conjecture that this is because the discriminator overfits the expert trajectories and converges to a bad optimum, which is not generalizable. On the other hand, GAIL with L2L is much more stable. It is very important to real applications of GAIL:; since the reward in real-world environment is usually unaccessible, cannot know whether there is sudden performance drop or not. With I.2I., can stabilize the training and obtain a much more reliable algorithm for real-world applications.
-00
400
GAIL
-150
300
GAIL
200
170
100
-100
-10
0
200
400
500
Mountain
CartPole
Figure 6: Reward vs. ileration of the trained policy using original GAIL and L2L GAIL.
5 DISCUSSIONS
Closely related works:
By leveraging the Fenchel duality and feature embedding technique. Dai al. (2016) convert a learning conditional distribution problem to minimax problem which is similar to naive attacker network. Both approaches, however, lack the primal information. In contrast, gradient attacker network considers the gradient information of primal variables, and achieves good results.
Goodfellow al. (2014a) propose the GAN, which is very similar to our L2L framework. Both GAN and L2L contain one generator network and one classifier network, and jointly train these two networks. There are two major difference between GAN and our framework: (1) GAN aims to transform the random noises to the synthetic data which is similar to the training examples, while ours targets on transforming the training examples to the adversarial examples for robustifying the classifier; (2) Our generator network does not only take the training examples (analogous to the random noise in GAN) as the input, but also exploits the gradient information of the objective function, since it essentially represents an optimization algorithm. The training procedure of these two, however, are quite similar. We adopt some tricks from GAN training to our framework to stabilize training process, e.g., in Grad L2L, we adopt the two-time scale trick (Heusel et al.) 2017)-
There are some other works simply combining the GAN framework and adversarial training to- gether. For example. [Baluja and Fischer (2017) and Xiao et al. (2018) propose some ad hoc GAN- based methods to robustify neural networks. Specifically, for generating adversarial examples, they only take training examples as the input of the generator, which lacks the information of the outer mimnimization problem. Instead, our proposed L2L methods (e.g.. Grad L2L,, 2-step L.2L) connect outer and inner problems by delivering the gradient information of the objective function to the gen- erator. This is a very important reason for our performance gain on the benchmark datasets. As a result, the aforementioned GAN-based methods are only robust simple attacks, e.g., FGSM, on simple data sets, e.g., MNIST, but fail for strong attacks, e.g., PGM and CW, on complicated data sets, e.g. CIFAR, where our L2L methods achieve significantly better performance.
Training Stability: For improving the training stability, we use both clean image and the cor- responding gradient as the input of the attacker. Without such gradient information, the attacker severely suffers from training instability, e.g., the Naive Attacker Network. Furthermore, we try another architecture with downsampling modules, called "slim attacker" in Appendix B We ob- served that the slim attacker also suffers from training instability. We suspect that the downsampling causes the loss of information. Thus, we tried to enhance the slim attacker by skip layer connections. Although the robust performance is still worse than the proposed architecture, it improves lot.
Benefits of our neural network approach in adversarial training: (1) Since the neural network has been known to be powerful in function approximation, Our attacker network y can yield strong adversarial perturbations. Since they are generated by the same attacker, the attacker g essentially learns some common structures across all samples; (2) The attacker networks in experiments actually overparametrized, which is conjectured to ease the training of deep neural networks. We believe that similar phenomena happen to our attacker network, and ease the adversarial training.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
ANDRYCHOWICZ, M., DENIL, M., GOMEZ, S., HOFFMAN, M. W., PFAU, D., SCHAUL, T., SHILLINGFORD, B. and DE FREITAS, N. (2016). Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems.
ATHALYE, A.. CARLINI, N. and WAGNER, D. (2018). Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv: /802.00420
BALUJA, S. and FISCHER, I. (2017). Adversarial transformation networks: Learning to generate adversarial examples. arXiv preprini arXiv: 1703.09387
BROCKMAN, G., CHEUNG, V., PETTERSSON, L.. SCHNEIDER, J., SCHULMAN, J., TANG, and ZAREMBA, W. (2016). Openai gym. arXiv preprint arXiv:1606.0154)
CARLINI, N., ATHALYE, A., PAPERNOT, N., BRENDEL, W., RAUBER, J., TSIPRAS, D., GOOD- FELLOW, I. and MADRY, A. (2019). On evaluating adversarial robustness. arXiv preprint arXiv: 1001:09002.0605..
CARLINI, N. and WAGNER, D. (2017). Towards evaluating the robustness neural networks. In 2017 IEEE Symposium on Security and Privacy (SP). IEEE.
DAI, B., HE, N., PAN, Y., BooTS, and SONG, L. (2016). Learning from conditional distribu- tions via dual embeddings. arXiv preprint 117.v:1607.04579.
DONG, Y., LIAO, F., PANG, T., Su, H., ZHU, J., HU, X. and LI, J. (2018). Boosting adversarial attacks with momentum. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
FINN, C., ABBEEL,, P. and LEVINE, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks, arXiv preprint 1703:17003003400
GIRSHICK, R., DONAHUE, J., DARRELI, T. and MALIK, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
GOODFELLOW, I., SOUGET-ABADIE J., MIRZA, M., Xu, B., WARDE-FARLES D., OZAIR, S., COURVILLE, A. and BENGIO, Y. (2014a). Generative adversarial nets. In Advances in Neural Information Procesing Systems.
GOODFELLOW, I. J.. SHLENS, J. and SZEGEDY, C. (2014b). Explaining and harnessing adversarial examples. arXiv preprint drXiv:1412.6572.
HE, K., ZHANG, X., REN, S. and SUN, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition.
HEUSEL, M., RAMSAUER, H., UNTERTHINER, T., NESSLER, B. and HOCHREITER, (2017). GANs trained by two time-scale update ReLU converge to local nash equilibrium. In Advances in Neural Information Processing Systems.
Ho, J. and ERMON, S. (2016). Generative adversarial imitation learning. CoRR abs/1606.03376.
http://arxvvvvv.. org abs/ 03476
HOCHREITER, S., YOUNGER, S. and CONWELL, P. R. (2001). Learning to learn using gradient descent. In International Conference on Artificial Neural Networks. Springer.
KINGMA, D. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint 1r2.
KRIZHEVSKY, A. and HINTON, G. (2009). Learning multiple layers of features from tiny images. Tech. rep., Citeseer.
KROGH, A. and HERTZ, J. A. (1992). A simple weight decay can improve generalization. In Advances in neural information processing systems.

---

Under review as a conference paper at ICLR 2020
KURAKIN, A., GOODFELLOW, and BENGIO, S. (2016). Adversarial machine learning at scale. arXiv preprint arxiv:
LIU, T., CHEN, Z., ZHOU, E. and ZHAO, T. (2018). Toward deeper understanding of noncon-
vex stochastic optimization with momentum using diffusion approximations. arXiv preprint
LIU, W., ZhaNG, Y.-M., LI, X., YU, Z., DAI, B., ZHAO, T. and SoNG, L. (2017). Deep hyper- spherical learning. In Advances Neural Information Processing Systems.
LIU, Y., CHEN, X., LIU, C. and SoNG, D. (2016). Delving into transferable adversarial examples and black-box attacks, arXiv preprint arXiv:161..0277..
MADRY, A., MAKELOV, A., SCHMIDT, L., TSIPRAS, D. and VLADU, A. (2017). Towards deep learning models resistant to adversarial attacks. arXiv preprint 07X1::1706.06883.
MIYATO, T., KATAOKA, T., KOYAMA, M. and YOSHIDA, (2018). Spectral normalization for generative adversarial networks. In International Conference on Learning Representations. formmm URtpittps://openeeeieer 20= id=B10RqziT-
NGUYEN, A., YOSINSKI, J. and Cl.unn, (2015). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
PASZKE, A., GROSS, S., CHINTALA, S., CHANAN, G., YANG, E., DEVITO, Z.,, LIN, 7., DES- MAISON, A., ANTIGA, L. and LERER, A. (2017), Automatic differentiation in pytorch.
SAMANGOUEI, P., KABKAB, M. and CHELLAPPA, R. (2018). Defense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv:1805.0665
SCHMIDHUBER, J. (1987). Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. Ph.D. thesis, Technische Universität München.
SCHMIDHUBER, J. (1992). Learning control fast- memories: An alternative to dynamic recurrent networks. Neural Computation 4 131-139.
SCHMIDHUBER, J. (1993). A neural network that embeds its own meta-levels. In Neural Networks, 1993., IEEE International Conference IEFE.
SCHULMAN, J., LEVINE, S., ABBEEL, P.. JORDAN, M. and MORITZ, P. (2015). Trust region policy optimization. In International conference on machine learning.
SZEGEDY, C., ZAREMBA, W., SUTSKEVER, I., BRUNA, J., ERHAN, D., GOODFELLOW, I. and
FERGUS, R. (2013). Intriguing properties of neural networks. arXiv preprint 1rXiv:1312.6199
TABACOF, and VALLE, E. (2016). Exploring the space of adversarial images. In 2016 Interna- tional Joint Conference on Neural Networks (IJCNN). IEEE.
TAIGMAN, Y., YANG, M., RANZATO, M. and WOLF, L. (2014). Deepface: Closing the gap to human-level performance face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
XIAO, C., LI, B., ZHU, J.-Y., HE, W., LIU, M. and SoNG, D. (2018). Generating adversarial examples with adversarial networks. arXiv preprint 0rXiv:1801.02610..
YOUNGER, A. S., HOCHREITER, S. and CONWELL, R. (2001). Meta-learning with backprop- agation. In Neural Networks, 2001. Proceedings, IJCNN'01. International Joint Conference vol. IEEE.
ZAGORUYKO, S. and KOMODAKIS, (2016). Wide residual networks. arXiv.1605.0716..
arXiv preprint
ZHANG, H., Yu, Y., JIAO, J., XING, E. P., GHAOUI, L. E. and JORDAN, M. I. (2019). Theo- retically principled trade-off between robustness and accuracy. arXiv preprint arXiv: 1901008733
10

---

Under review as a conference paper at ICLR 2020
ZHENG, S., SONG, Y., LEUNG, T. and GOODFELLOW, I. (2016). Improving the robustness of deep neural networks via stability training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

---

Under review as a conference paper at ICLR 2020
Appendix
A BLACK-BOX ATTACK
Under the black-box setting, first train surrogate model with the same architecture of the target model but a different random seed, and then attackers generate adversarial examples to attack the target model by querying gradients from the surrogate model.
The black-box attack highly relies on the transferability, which is the property that the adversarial examples of model are likely fool others. However, the transferred attack is very unstable, and often has a large variation its effectiveness. Therefore, results of the black-box setting might be reliable and effective. Thus only present one result here to demonstrate the robustness of different models.
Table 4: Results the black-box setting over CIFAR-10. We evaluate L2L methods with slim attacker networks.
Surrogate
Plain Net
FGSM Net
PGM Net
FGSM
PGM10
FGSM
PGM10
FGSM
PGMI0
Plain Net
40.03
5.60
74.42
75.25
67.37
65.92
FGSM Net
79.20
85.02
89.90
80.40
64.28
63.89
PGM Net
83.80
84.73
84.33
85.29
67.05
65.54
Naive L2L 45.52
25.95
83.99
77.94
68.14
Grad L2L 86.10
86.87
87.93
88.01
71.15
69.95
2-Step L2L
85.83
87.10
86.51
87.60
70.58
69.38
Table 5: Experiments under the black-box setting over CIFAR-100. Note that here we only evaluate L2L methods using the slim attacker network.
Surrogate
Plain Net
FGSM Net
PGM Net
FGSM
PGM10
FGSM
PGM10
FGSM
PGM10
Plain Net
21.04
9.04
50.57
54.06
40.06
41.30
FGSM Net
42.87
50.73
61.68
44.70
39.34
40.08
PGM Net
56.63
58.34
56.99
57.97
40.19
39.87
Naive I.2I.
20.97
10.47
50.36
54.07
38.63
39.91
Grad 1.21.
57.63
59.62
59.18
61.26
41.71
41.15
2-Step L2L
58.66
59.31
58.92
59.46
45.80
45.31
Table 6: Experiments under the black-boo box setting SVHN. Note that here we only evaluate L2L methods using the wide attacker network.
Surrogate
Plain Net
FGSM Net
PGM Net
FGSM
PGM10
FGSM
PGMI0
FGSM
PGMI0
Plain Net
21.72
6.94
41.81
33.13
56.77
49.41
FGSM Net
57.36
51.54
56.25
38.11
55.99
48.96
PGM Net
81.04
81.52
78.66
80.42
54.85
49.21
Naive I.21.
73.02
42.14
78.11
59.79
85.31
61.08
Grad L2L
71.74
74.31
77.19
80.70
71.99
58.71
2-Step L2L
65.78
74.07
76.13
82.80
61.69
54.13
12

---

Under review as a conference paper at ICLR 2020
B
SLIM NETWORK
Table presents another architecture that we used in the L2L. this network, the second convolu- tional layer uses downsampling, while the second last deconvolutional layer uses upsampling. Due to the downsampling, this network is computationally cheap and thus it is computationally fast. For example the running time of per epoch for L2L with slim attacker is 480; whereas L2L with the original architecture is 620. However, it loses some information of input and is less stable than the original architecture (Table D Inspired by residual learning in He al. (2016), we address the above issues by using a skip layer connection to case the training of this network. Specifically, the last layer takes the concatenation of A(x, y, 0) and the output of the second last layer as input. Figure 7 presents the architecture of ResBlocks. PReLU is special type of Leaky ReLU with learnable slope parameter.
Table 7: Attacker Architectue..
Conv:
[k 3 3,
128,
1,p = 11, BN+ReI.U
ResBlocks:
[channel = 256]
ResBlocks:
[channel
128). BN
DeConv:
x 4,
16,8 26, BN+ReLU
Conv:
[K
x 3,
3,8 1.p 1J. tanh
BN
PRcLU
Conv
BN
PRcLU
Conv
Input
Add
Figure 7: An illustration example for the architecture of ResBlocks.
Table shows the results of L2L with the architecture shown in Table
Table 8: Results of L2L with the other altacker network under white-box setling over CIFAR.
CIFAR-10
CIFAR-100
Clean
FGSM
PGM20
CW
Clean
FGSM
PGM20 CW
Naive L2L
94.41
28.44
0.01
0.00
75.27
8.47
0.05
0.00
Grad L2L
85.31
57.44
53.02
42.72
60.60
26.58
27.37
23.14
2-Step L2L
75.36
60.19
46.12
40.82
60.23
25.92
20.23
22.70
C GENERATIVE ADVERSARIAL IMITATION LEARNING
C.I ALGORTHTS
Algorithm crrriiingtt--lea---ee generative adversarial imitation learning
Input: Expert trajectories (3, ā TE(3)), initial policy, discriminator and updater parameters
0, Pp. Ru.
for t + to N do
Sample trajectories 15.a~Tii 0x))
Compute gradient:
g < [[aaa]] Eis.) (Vo log (D(s, o; 0b)) [a.a]] Exx,a) Vop log(1 (b)1)
Update the discriminator parameters:
of+1 U(O :.
Update the updater parameters Ou by minimizing:
[(saa]] E(..)) [log (D(s, a; UOO gb: (b)))] [aay E [lo((1 D(G, ã; (O gb: 0 )))
Update the policy parameters 0z by a policy step using the TRPO rule (Ho and Ermon (016
13

---

Under review as a conference paper at ICLR 2020
C.2 EXPERIMENT SETTING
Updater Architecture We use simple 3-layer perceptron with a skip layer as our updater. The number hidden units (2m + 8m 4m - mn), where m is the dimension of Op that depends the original task. For the first and the second layer, we use PReLu as the activation function, while the last layer has activation function. Finally we add the output to @D in the original input as the updated parameter for the discriminator network.
Hyperparameter Settings For all baselines exactly follows the setting Ho and Ermon 2016). except that we use 2-layer discriminator with number of hidden units ((s,a) - 64 + 32 - 1) using tanh as the activation. We use the same neural network architecture for and the same optimizer configuration. The expert trajectories are obtained by an expert trained using TRPO. For L.2L based GAIL,, we also use Adam optimizer to update the Oy with the same configuration as updating OD in original GAIL.
D LIMITING CYCLE
Limiting cycle is a well-known issue for minimax machine learning problem [4,5]. The reason behind limiting cycle is that different from minimization problems, minimax optimization problem is more complicated and can be highly nonconvex nonconvex-nonconcar nonconcaav,, where the inner problem can not be solved exactly. Here we provide concrete example: we consider the following optimization problem:
min max f(r, y) = ry.
Then at the 1-th iteration, the update direction will be (-yt,, Tz). If we start from (1, 0) with stepsize of 0.0001, this update will result in limiting circle: x² + +y² = 1 and never reach the stable
equilibrium (0, 0) as shown in Figure 8
1.5
0.5
-0.5
-1
-1.5
-1.5
-1
-0.5
0.5
1
1.5
2.
Figure 8: example of the limiting circle: arrows denote the update directions
E ROBUSTNESS EVALUATION CHECKLIST
Recently, there are many works on robustness defense that have been proven ineffective (Athalye
al. 2018 Carlini al. 2019). Our work follows the most reliable and widely used robust model
14

---

Under review as a conference paper at ICLR 2020
approach adversarial training, which finds parameters to make the model robust. We do not make any modification to final classifier model. Unlike previous works (e.g. Defense-GAN, Samangouei al. (2018)), model does not take the attacker part of the final model and does not use shattered/obl saatterred/bfuussateddmssseee gradient as a defense mechanism. We also demonstrate that the evaluation of the robustness of proposed 1.2I. method trustworthy by verifying all items listed in (Carlini et al., 2019).
E.1 SHATEREEDOOFFUSSTEEDDDDAAAAA
GRADIENT
In this section we verify that our proposed L2I. method does not fall into the pitfall of shat- tered/offiscateed/nssee gradient, which have proven ineffective. To see this, checked every item recommended in section 3.1 Athaly et al. (2018):
One-step attacks perform better than iterative attacks: Figure shows that the PGM attack is stronger with larger number of iterations.
Black-box attacks are better than white-box attacks: Appendix shows that the black-box transfer attack is much weaker than white white-box attacks.
Unbounded attacks not reach 100% success: We evaluate the model robustness against attack with extremely large perturbation to show that unbounded attacks do reach 100% success. Specifically, we use the PGM-10 attack with various perturbation magnitudes E [0, and stepsize fo Figure 9 shows that the PGM attack eventually reach 100% success as the perturbation magnitude increases.
2-Step L2L
60
Grad L2L
PGM Net
40
20
0.0
0.2
0.4
0.6
Perturbation Magnitude for PGM (T = 10)
Figure 9: Robust accuracy against perturbation magnitudes of PGM over CIFAR-100.
Random sampling finds adversarial examples: Table 2 we show that random search is not better than gradient- based method and is rather weak against our model.
Increasing distortion bound does not increase success: Figure 4 shows that the PGM attack becomes stronger as the perturbation magnitude increases.
E.2 ROBUSTNESS EVALUATION CHECKLIST
Carlini (2119) also provide evaluation checklist, and we now check each of common severe flaws and common pitfalls as follows:
State a precise threat model: We do not have any adversary detector; We do not use shat- tered/Obfuscatee/MMsseee gradient. We do not have denoiser. Our model has no aware of the attack mechanism, including PGM and CW attacks.
Adaptive attacks: We used CW, PGM, and 1.21. attacker attack.
Report clean model accuracy: We reported.
Do not use Fast Gradient Sign Method. We use PGM-20 and PGM-100 and CW.
Do not only use attacks during testing that were used during training. We use different evaluation criteria evaluate all models.
Perform basic sanity tests: It is provided in Figure
15

---

Under review as a conference paper at ICLR 2020
Generate an attack success rate perturbation budget: Figure
Verify adaptive attacks perform better than any other (e.g., blackbox, and brute-force search): The above table and Appendix A the paper.
Describe the attacks applied: In Section
Apply diverse set of attacks: We tried PGM attack (with different perturbation magnitude and iterations). blackbox attack (transfer attack), CW attack (adaptive attack). L2L attack (adaptive and designed for this particular model), Bruteforce random search (gradient-fre free attack)
Suggestions for randomized defenses: We are not.
Suggestions for non-differentiable components (e-g.. by performing quantization adding extra randomness): We have no additional non-differentiable component.
Verify that the attacks have converged: converges.
Figure
shows that the PGM attack eventually
Carefully investigate attack hyperparameters: Figure
Compare against prior work: We compared our algorithm to PDM net. L2L is more com- putationally efficient and the L2L model is more robust due to the fact that L2L attack is strong enough. Unlike Defense-GAN, we do not use the generator (attacker in L2L) as the denoising module and do not change the final prediction model.
16


Reviews:
[{"review_id": "SyxVKI86tH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new way of adversarial training by placing another neural network called \"attacker\" network, and let the attacker to learn how to generate adversarial examples during training. This training scheme is formulated to solving a joint training according to min-max problem. Experimental results show that the method outperforms existing adversarial training in CIFAR-10/100 once the gradient information can be provided into the attacker network. \n\nIn overall, the paper is well-written with an interesting message: There are certain features useful across all data samples in the inner maximization problem, which can be induced from gradient information. The experimental results are presented clearly, demonstrating its effectiveness and efficiency in running time. Section 4 seems to support the main claim in a novel way as well. The general motivation or justification on the proposed methods, e.g. the \"limiting cycle\" argument or the visualization part were not that convincing or seems slightly over-claimed to me, nevertheless. \n\n- I think adding a discussion on how to generalize the framework into other threat models, e.g. L2 or (even) unrestricted attacks would further strengthen the paper. I feel the current framework may suffers some training difficulties on these other threat models, even such kinds of discussion would also valuable to understand the method.\n- As the current formulation can generalize the general inner maximization optimization process, comparing or applying the method with a more recent form of adversarial training, e.g. TRADES, would be nice to demonstrate the general applicability of the method.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Skgx84u9YS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In general, this paper follows the min-max training framework for adversarial robustness. Instead of using a gradient-based attack to solve the inner maximization, the authors use a neural network to learn the attack results. From the experimental results, this method can effectively defend against CW and PGD on CIFAR-10 and CIFAR-100. But the clean accuracy is lower than Madry et al. \n\nAlso, there are too many works on robustness defense that have been proven ineffective (consider the works by Carlini). Since this is a new way of robust training and there is no certified guarantee, I suggest the authors refer [1] to evaluate the effectiveness of the defense more thoroughly to convince the readers that it really works. Especially, a robustness evaluation under adaptive attacks is necessary. In other words, if the attacker knows the strategy used by the defender (such as the attacker network structure), it may be possible to break the model. \n\nI am not convinced by the limiting cycle claim in Figure 1. I do not think this scenario (gradient descent goes along a cycle) is possible. If we take the integral of the gradient along this cycle from x to itself, we will get 0=f(x)-f(x)=-\\int_{t on cycle}f'(t)dt<0, which means that the function is not continuous at x. I suggest the authors have a surface plot of the function if they think this is possible. \n\n[1] Carlini, Nicholas, et al. \"On evaluating adversarial robustness.\" arXiv preprint arXiv:1902.06705 (2019).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Byx3KHJW5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a framework where one component is an attacker network that keeps learning about how to perturb the loss more, and one component is a defense network that robustify learning with respect to the attacker network. The framework is flexible on how the attacker network can be trained, and advances over previous works where the attacker is a human-designed algorithm rather than a learning model. Experiment results show that the framework reaches superior defense performance. The authors also extend the framework to help imitation learning.\n\nOverall the paper is a pleasure to read. My questions/suggestions are\n\n(1) Given that the framework seems natural in design, a deeper contribution would be talking about how to successfully train the framework in practice. The authors talk about the connection of the framework to GAN, and the latter is not that easy to train. However, we see very little information on how to train the framework in the paper. Was it super easy to train the framework (why?), or did the author encounter any difficulties? Are there important heuristics that help train the framework successfully?\n\n(2) While the framework leads to a better defense mechanism (the authors' goal), one could wonder whether it leads to a better attacker as well. Instead of just checking the differences of the attacking examples generated, can we take the inner attacker and see if it is more effective in attacking than PGM and CW? How does the goodness of the attacker improve with time? Do different L2L variants generate attackers with different quality? Do the quality connect with the defense performance?\n\n(3) Section 4 looks distracting to me. It is good to know that the framework can be extended to imitation learning, but the section is best put at a longer version or another paper, rather than occupying a significant amount of space in the current paper.\n\nI have read the rebuttal and thank the authors for the response.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers solving the minimax formulation of adversarial training, where it proposes a new method based on a generic learning-to-learn (L2L) framework. Particularly, instead of applying the existing hand-designed algorithms for the inner problem, it learns an optimizer parametrized as a convolutional neural network. A robust classifier is learned to defense the adversarial attack generated by the learned optimizer. The idea is using L2L is sensible. However, main concerns on empirical studies remain after rebuttal. 
Decision: Reject