Title: LEARNING TO DEFENSE BY LEARNING TO ATTACK

Abstract: Adversarial training provides a principled approach for training robust neural networks. From an optimization perspective, the adversarial training is essentially solving a minimax robust optimization problem. The outer minimization is trying to learn a robust classifier, while the inner maximization is trying to generate adversarial samples. Unfortunately, such a minimax problem is very difficult to solve due to the lack of convex-concave structure. This work proposes a new adversarial training method based on a generic learning-to-learn (L2L) framework. Specifically, instead of applying the existing hand-designed algorithms for the inner problem, we learn an optimizer, which is parametrized as a convolutional neural network. At the same time, a robust classifier is learned to defense the adversarial attack generated by the learned optimizer. Our experiments over CIFAR-10 and CIFAR-100 datasets demonstrate that the L2L outperforms existing adversarial training methods in both classification accuracy and computational efficiency. Moreover, our L2L framework can be extended to the generative adversarial imitation learning and stabilize the training.

1. INTRODUCTION:

This decade has witnessed great breakthroughs in deep learning in a variety of applications, such as computer vision (Taigman et al., 2014; Girshick et al., 2014; He et al., 2016; Liu et al., 2017) . Recent studies (Szegedy et al., 2013) , however, show that most of these deep learning models are very vulnerable to adversarial attacks. Specifically, by injecting a small perturbation to a normal sample, one can obtain an adversarial example. Although the adversarial example is semantically indistinguishable from the normal one, it can fool deep learning models and undermine the security of deep learning, causing reliability problems in autonomous driving, biometric authentication, etc.

Researchers have devoted many efforts to studying efficient adversarial attack and defense (Szegedy et al., 2013; Goodfellow et al., 2014b; Nguyen et al., 2015; Zheng et al., 2016; Madry et al., 2017; Carlini and Wagner, 2017) . There is a growing body of work on generating adversarial examples, e.g., fast gradient sign method (FGSM, Goodfellow et al. (2014b) ), projected gradient method (PGM, Kurakin et al. (2016) ), Carlini-Wagner (CW, Paszke et al. (2017) ) etc. As for defense, Goodfellow et al. (2014b) propose to robustify the network by adversarial training, which trains over the adversarial examples and still requires the network to output the correct label. Further, Madry et al. (2017) formalize the adversarial training as the following minimax optimization problem:

EQUATION

where {(x i , y i )} n i=1 ⊂ R d × Y are n pairs of input feature and the corresponding label, denotes a loss function, f (•; θ) denotes the neural network with parameter θ, and δ i ∈ B denotes the perturbation for x i in constraint B. The existing literature on the optimization also refers to θ as the primal variable and δ i 's as the dual variables. Different from the well-studied convex-concave problem1 , problem (1) is very challenging since is nonconvex in θ and nonconcave in δ. As a result, there may be many equilibria, and majority of them are unstable. In the existing optimization literature, there is no algorithm to converge to a stable equilibrium with theoretical guarantees. Empirically, the existing primal-dual algorithms may perform poorly for solving (1).

Minimax formulation (1) naturally provides us with a unified perspective on prior works of adversarial training. Such a minimax problem consists of two optimization problems, an inner maximization problem and an outer minimization problem: The inner problem targets on finding an optimal attack for a given data point (x, y) maximizing the loss, which essentially is the adversarial attack; The outer problem aims to find a θ so that the loss given by the inner problem is minimized.

For solving (1), Goodfellow et al. (2014b) propose to use FGSM to solve the inner problem. Kurakin et al. (2016) then find that FGSM with true label predicted suffers from a "label leaking" effect, which can ruin the adversarial training. Madry et al. (2017) further suggest to solve the inner problem by PGM and obtain a better result than FGSM, since FGSM essentially is one iteration PGM. However, adversarial training needs to find a δ i for each (x i , y i ), thus the dimension of the overall search space for all data is substantial, which makes the computation expensive.

Instead, we propose a new learning-to-learn (L2L) framework that provides a more principled and efficient way for adversarial training. Specifically, we parameterize the optimizer of the inner maximization problem by a neural network denoted by g(A(x, y, θ); φ), where A(x, y, θ) denotes the input of the optimizer g. We also call the optimizer as the attacker network. Since the neural network is very powerful in function approximation, our parameterization ensures that g is able to yield strong adversarial perturbations. Under our framework, instead of directly solving δ i , we update the parameter φ of g. Our training procedure becomes updating the parameters of two neural networks, which is very similar to generative adversarial network (GAN, Goodfellow et al. (2014a) ). The proposed L2L is a generic framework and can be extended to other minimax optimization problems, e.g., generative adversarial imitation learning, which is studied in Section 4.

Different from the hand-designed methods that compute the adversarial perturbation for each individual sample using gradients from backpropagation, our methods generate the perturbations for all samples through the shared attacker g. This enables the attacker g to learn potential common structures of the perturbations. Therefore, our method is capable of yielding strong perturbations and accelerating the training process. Furthermore, L2L framework is very flexible: we can either choose different input A(x, y, θ), or use different attacker architecture. For example, we can include gradient information in A(x, y, θ) and use a recurrent neural network (RNN) to mimic multi-step gradient-type methods. Instead of simply computing the high order information with finite difference approximation or multiple gradients, by parameterizing the algorithm as a neural network, our proposed methods can capture this information in a much smarter way (Finn et al., 2017) . Our experiments demonstrate that our proposed methods not only outperform existing adversarial training methods, e.g., PGM training, but also enjoy the computational efficiency over CIFAR-10 and CIFAR-100 datasets (Krizhevsky and Hinton, 2009) .

The research on L2L has a long history (Schmidhuber, 1987; 1992; 1993; Younger et al., 2001; Hochreiter et al., 2001; Andrychowicz et al., 2016) . The basic idea is that one first models the updating formula of complicated optimization algorithms in a parametric form, and then uses some simple algorithms, e.g., stochastic gradient algorithm to learn the parameters of the optimizer. Among existing works, Hochreiter et al. (2001) propose a system allowing the output of backpropagation from one network to feed into an additional learning network, with both networks trained jointly; Based on this, Andrychowicz et al. (2016) further show that the design of an optimization algorithm can be cast as a learning problem. Specifically, they use long short-term memory RNNs to model the algorithm and allow the RNNs to exploit structure in the problems of interest in an automatic way, which is undoubtedly one of the most popular methods for learning-to-learn.

However, there are two major drawbacks of the existing L2L methods: (1) It requires a large amount of datasets (or a large number of tasks in multi-task learning) to guarantee the learned optimizer to generalize, which significantly limits their applicability (most of the related works only consider the image encoding as the motivating application); (2) The number of layers/iterations in RNNs for modeling algorithms cannot be large to avoid significant computational burden in backpropagation.

Our contribution is that we fill the blank of the L2L framework in solving the minimax problem, and our proposed methods do not suffer from the aforementioned drawbacks: (1) The attacker g with a different φ essentially generates a different task/dataset. Therefore, for adversarial training, we have sufficiently many tasks for learning-to-learn; (2) The inner problem does not need a large scale RNN, and we use a convolutional neural network (CNN) or a length-two RNN (the sequence of length equals 2) as our attacker network, which eases the computation.

Our work is also related to GAN and dual-embedding (Dai et al., 2016) . All three solve minimax problems and share some common ground. We discuss these works in Section 5.

Notations. Given a ∈ R, denote (a) + as max(a, 0). Given x, y ∈ R d , denote x i as the i-th element of x, ||x|| ∞ = max i |x i | as ∞ -norm of x, and x•y = [x 1 y 1 , • • • , x d y d ] as element-wise product. This paper focuses on ∞ -norm attack. We define the ∞ -ball with center 0 and radius by B( ) = {δ ∈ R d : ||δ|| ∞ ≤ } and the corresponding projection as follows:

2. METHOD:

Π B( ) (δ) = sign(δ) • max(|δ|, ),

where sign and max are element-wise operators.

2.1. ADVERSARIAL TRAINING:

The goal of adversarial training is to robustify neural networks. Recall that from a robust optimization perspective, given n samples {(x i , y i )} n i=1 , where x i is the i-th feature vector and y i is the corresponding label, adversarial training is reformulated as the following optimization problem (1):

min θ 1 n n i=1 max δi∈B (f (x i + δ i ; θ), y i ) ,

where f denotes the network with parameter θ, denotes a loss function, and is the maximum perturbation magnitude.

We first demonstrate the hardness of solving problem (1). Ideally, we want to obtain the optimal solution for the inner problem

δ * i := argmax δi∈B (f (x i + δ i ; θ), y i ).

However, note that the loss function (f (x i + δ i ; θ), y i ) is highly nonconcave in δ i . Therefore, in reality the δ i we obtained is very unlikely to be the optimum δ * i . This then often leads to a highly unreliable or even completely wrong search direction, e.g., ∇ θ (f (x i + δ i ; θ), y i ), ∇ θ (f (x i + δ * i ; θ), y i ) < 0, which may further results in a limiting cycle shown in Figure 1 (Detailed discussion is provided in Appendix D). This becomes even worse when sample noises exist.

Algorithm 1 Standard pipeline of adversarial training Input: {(x i , y i )} n i=1 : clean data, α: learning rate, N : number of epochs, : maximum perturbation magnitude.

for t ← 1 to N do Sample a minibatch M t for i in M t do δ i ← argmax δi∈B( ) (f (x i + δ i ; θ), y i ) Generate adversarial perturbation for (x i , y i ) θ ← θ -α 1 |Mt| i∈Mt ∇ θ (f (x i + δ i ; θ), y i ) Update θ over adversarial data {(x i + δ i , y i )} i∈Mt

In the existing literature, the standard pipeline of adversarial training is shown in Algorithm 1. Since the step of generating adversarial perturbation δ i in Algorithm 1 is intractable, most adversarial training methods adopt hand-designed algorithms. For example, Kurakin et al. (2016) propose to solve the inner problem approximately by first order methods such as PGM. Specifically, PGM iteratively updates the adversarial perturbation by the projected sign gradient ascent method for each sample: Given one sample (x i , y i ), at the t-th iteration, PGM takes

EQUATION

where

x t i = x i + δ t-1 i

, η is the perturbation step size, T is a pre-defined total number of iterations, and

δ 0 i = 0, t = 1, • • • , T . Finally PGM takes δ i = δ T i .

Note that FGSM essentially is one-iteration PGM. Besides, some works adopt other optimization methods, such as momentum gradient method (Dong et al., 2018) , and L-BFGS (Tabacof and Valle, 2016) . However, except for FGSM, all require numerous queries for gradients through backpropagation, which is computationally expensive.

2.2. LEARNING TO DEFENSE BY LEARNING TO ATTACK (L2L):

Since the objective function is nonconvex-nonconcave, there is no guarantee for hand-designed methods to perform well. Instead, we propose to learn an optimizer for the inner problem. Specifically, we parameterize the attacker by a Neural Network g(A(x, y, θ); φ), where A(x, y, θ), the input of the network g, summaries the information of data and neural network f (•; θ). We then convert problem (1) to min θ

EQUATION

where φ * is defined as the solution to the following optimization problem:

φ * ∈ argmax φ 1 n n i=1 (f (x i + g(A(x i , y i , θ); φ); θ), y i ) subject to g(A(x, y, θ); φ) ∈ B( ).

Solving problem (3) naturally consists of two stages. In the first stage, the classifier f aims to fit over all perturbed data; While in the second stage, given a certain f obtained in the first stage, the attacker network g targets on generating optimal perturbations under constraints δ i 's ∈ B( ).

Since δ i = g(A(x i , y i ; θ); φ), constraints can be simply handled by a tanh activation function in the last layer of g. Specifically, because the magnitude of tanh output is bounded by 1, after we rescale the output by , the output of g automatically satisfies the constraints. 

Optimizer Classifier x < l a t e x i t s h a 1 _ b a s e 6 4 = " E c k 2 m C M O K 7 W 8 G 1 n O Q G W N K S v L S 7 g = " > A A A C n n i c b V F N a x R B E O 0 d v + L 6 l e j R S + M S E A / L T C L o S S K K e A l J M J s s Z J d Q 0 1 O z 2 2 x / 0 V 2 j W Y Y 5 e / C q P 8 5 / Y 8 / u H p w k B U 0 / X r 2 i X l X l T s l A a f q 3 l 9 y 5 e + / + g 6 2 H / U e P n z x 9 t r 3 z / C z Y y g s c C a u s H + c Q U E m D I 5 K k c O w 8 g s 4 V n u e L T 2 3 + / D v 6 I K 0 5 p a X D q Y a Z k a U U Q J E 6 u b r c H q T D d B X 8 J s g 2 Y M A 2 c X y 5 0 / s 5 K a y o N B o S C k K 4 y F J H 0 x o 8 S a G w 6 U + q g A 7 E A m Z Y r / w 1 f D d S B S + t j 8 8 Q X 7 E d H e g Q l j q P S g 0 0 D 9 d z L X l b 7 q K i 8 v 2 0 l s Z V h E a s G 5 W V 4 m R 5 O y w v p E d B a h k B C C + j Q y 7 m 4 E F Q X E l / 9 / 8 2 N H P x U w q a L r 9 w r e f Q n S u A a Z n P G D f g 8 T D 6 + q j c H H K k e r J y W T b 1 0 W n W 1 C I O 1 t S 6 q U 2 U f 0 O 6 X Z p b V X Q K 8 q t 1 h U e D P 4 T V G k z x Z k J U Y A m V o q g i a u L p s u u H u g n O 9 o b Z / n D v 5 O 3 g 4 M P m i F v s J X v F X r O M v W M H 7 C s 7 Z i M m G L J f 7 D f 7 k / D k S 3 K Y H K 2 l S W 9 T 8 4 J 1 I h n / A w T C 0 + I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " n R m T w 5 + 8 S 4 0 F Q 9 P m C L U r F 2 h g q J 8 = " > A A A C o 3 i c b V F N b 9 N A E N 2 Y j 5 b w 1 c K R i 0 U U C X G I 7 B a J n l A R H J A Q o k C S V q q j a r w e J 0 v 2 S 7 t j I L J 8 5 s o V / h n / h n W S A 2 4 7 0 m q f 3 r z R v J n J r R S e k u R v L 7 p x 8 9 b t n d 0 7 / b v 3 7 j 9 4 u L f / a O p N 5 T h O u J H G n e X g U Q q N E x I k 8 c w 6 B J V L P M 2 X b 9 r 8 6 T d 0 X h g 9 p p X F m Y K 5 F q X g Q I G a Z g V K g o u 9 Q T J K 1 h F f B e k W D N g 2 T i 7 2 e z + z w v B K o S Y u w f v z N L E 0 q 8 G R 4 B K b f l Z 5 t M C X M M d 6 b b K J h 4 E q 4 t K 4 8 D T F a 7 a j A + X 9 S u V B q Y A W / n K u J a / L n V d U H s 1 q o W 1 F q P m m U V n J m E z c T h w X w i E n u Q o A u B P B Y c w X 4 I B T 2 E t / + H 8 b m t v w S Q l N l 1 / a 1 r P v z u V B t 8 x b D B t w + C H 4 e i 3 t A n K k O l u 7 L J v 6 4 z h t a h 4 G a 2 r V 1 D r I v y B d L 8 2 N L D o F + Y 9 N h U O N 3 7 l R C n T x P C M q s I R K U l A R N e F 0 6 e V D X Q X T g 1 F 6 O D r 4 9 G J w / G p 7 x F 3 2 h D 1 l z 1 j K X r J j 9 o 6 d s A n j 7 C v 7 x X 6 z P 9 E w e h 9 9 j s Y b a d T b 1 j x m n Y h m / w A S 5 t Y C < / l a t e x i t > x + (t) < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 P d 2 U b z m V D p i A Z x 0 K + F T u + j B 1 F E = " > A A A C q 3 i c b V F L b 9 N A E N 6 Y V w m P t n D k Y h F V K i A i u 1 Q q x / I 4 c E E U t W k r 6 r Q a r 8 f J K v v S 7 h g a W T 7 z F 7 j C v + L f s E 5 y w G 1 H W u 2 n b 7 7 R f D O T W y k 8 J c n f X n T r 9 p 2 7 9 9 b u 9 x 8 8 f P R 4 f W P z y b E 3 l e M 4 4 k Y a d 5 q D R y k 0 j k i Q x F P r E F Q u 8 S S f f W j z J 9 / R e W H 0 E c 0 t j h V M t C g F B w r U + e W r r E B J c F 5 v 0 4 v m Y m O Q D J N F x N d B u g I D t o q D i 8 3 e z 6 w w v F K o i U v w / i x N L I 1 r c C S 4 x K a f V R 4 t 8 B l M s F 6 Y b e K t Q B V x a V x 4 m u I F 2 9 G B 8 n 6 u 8 q B U Q F N / N d e S N + X O K i r f j m u h b U W o + b J R W c m Y T N x O H h f C I S c 5 D w C 4 E 8 F h z K f g g F P Y T 3 / r / z Y 0 s e G T E p o u P 7 O t Z 9 + d y 4 N u m Y 8 Y N u D w c / D 1 T t o p 5 E h 1 t n B Z N v W X o 7 S p e R i s q V V T 6 y A / R L p Z m h t Z d A r y y 2 W F Q 4 0 / u F E K d P E y I y q w h E p S U B G 1 p 0 u v H u o 6 O N 4 Z p m + G O 1 9 3 B / v v V 0 d c Y 8 / Y c 7 b N U r b H 9 t k n d s B G j D P H f r H f 7 E / 0 O j q M v k X Z U h r 1 V j V P W S c i / A d 4 s t k U < / l a t e x i t > Unfold < l a t e x i t s h a 1 _ b a s e 6 4 = " n R m T w 5 + 8 S 4 0 F Q 9 P m C L U r F 2 h g q J 8 = " > A A A C o 3 i c b V F N b 9 N A E N 2 Y j 5 b w 1 c K R i 0 U U C X G I 7 B a J n l A R H J A Q o k C S V q q j a r w e J 0 v 2 S 7 t j I L J 8 5 s o V / h n / h n W S A 2 4 7 0 m q f 3 r z R v J n J r R S e k u R v L 7 p x 8 9 b t n d 0 7 / b v 3 7 j 9 4 u L f / a O p N 5 T h O u J H G n e X g U Q q N E x I k 8 c w 6 B J V L P M 2 X b 9 r 8 6 T d 0 X h g 9 p p X F m Y K 5 F q X g Q I G a Z g V K g o u 9 Q T J K 1 h F f B e k W D N g 2 T i 7 2 e z + z w v B K o S Y u w f v z N L E 0 q 8 G R 4 B K b f l Z 5 t M C X M M d 6 b b K J h 4 E q 4 t K 4 8 D T F a 7 a j A + X 9 S u V B q Y A W / n K u J a / L n V d U H s 1 q o W 1 F q P m m U V n J m E z c T h w X w i E n u Q o A u B P B Y c w X 4 I B T 2 E t / + H 8 b m t v w S Q l N l 1 / a 1 r P v z u V B t 8 x b D B t w + C H 4 e i 3 t A n K k O l u 7 L J v 6 4 z h t a h 4 G a 2 r V 1 D r I v y B d L 8 2 N L D o F + Y 9 N h U O N 3 7 l R C n T x P C M q s I R K U l A R N e F 0 6 e V D X Q X T g 1 F 6 O D r 4 9 G J w / G p 7 x F 3 2 h D 1 l z 1 j K X r J j 9 o 6 d s A n j 7 C v 7 x X 6 z P 9 E w e h 9 9 j s Y b a d T b 1 j x m n Y h m / w A S 5 t Y C < / l a t e x i t > rx`(x + (t) ; ✓) < l a t e x i t s h a 1 _ b a s e 6 4 = " H q A a k R G 6 2 M G E 2 G C x u s 3 H r X H y N M M = " > A A A C 0 H i c b V H L b t N A F J 2 Y V w m v F J Z s L K J K K U h R X J B A Y l M e C z a I A k 1 b q Q 7 R 9 f g 6 G W U 8 t m a u I d F o h F g h 8 Q N 8 D V v 4 B v 6 G s Z M F b n u l 0 R y d e 6 7 O f S S l F I Z G o 7 + d 4 N L l K 1 e v b V 3 v 3 r h 5 6 / a d 3 v b d I 1 N U m u O Y F 7 L Q J w k Y l E L h m A R J P C k 1 Q p 5 I P E 4 W r + r 8 8 W f U R h T q k F Y l T n K Y K Z E J D u S p a S + K F S Q S p s s w R i k H y 0 d x i p L g k x 3 Q r n t u 4 8 b B a k x d T H M k c L v T X n 8 0 H D U R n g f R B v T Z J g 6 m 2 5 3 v c V r w K k d F X I I x p 9 G o p I k F T Y J L d N 2 4 M l g C X 8 A M b e P n w h 1 P p W F W a P 8 U h Q 3 b 0 k F u z C p P v D I H m p u z u Z q 8 K H d a U f Z s Y o U q K 0 L F 1 0 Z Z J U M q w n

m C M O K 7 W 8 G 1 n O Q G W N K S v L S 7 g = " > A A A C n n i c b V F N a x R B E O 0 d v + L 6 l e j R S + M S E A / L T C L o S S K K e A l J M J s s Z J d Q 0 1 O z 2 2 x / 0 V 2 j W Y Y

5 e / C q P 8 5 / Y 8 / u H p w k B U 0 / X r 2 i X l X l T s l A a f q 3 l 9 y 5 e + / + g 6 2 H / U e P n z x 9 t r 3 Gradient Attacker Network. Motivated by handdesigned methods, e.g., FGSM, we design an attacker which takes the gradient information into computation. Specifically, we concatenate image x i and gradient ∇ x (f (x i ; θ), y i ) from backpropagation as the input of g, i.e., A(x i , y i , θ) = x i , ∇ x (f (x i ; θ), y i )

z / C z Y y g s c C a u s H + c Q U E m D I 5 K k c O w 8 g s 4 V n u e L T 2 3 + / D v 6 I K 0 5 p a X D q Y a Z k a U U Q J E 6 u b r c H q T D d B X 8 J s g 2 Y M A 2 c X y 5 0 / s 5 K a y o N B o S C k K 4 y F J H 0 x o 8 S a G w 6 U + q g A 7 E A m Z Y r / w 1 f D d S B S + t j 8 8 Q X 7 E d H e g Q l j q P S g 0 0 D 9 d z L X l b 7 q K i 8 v 2 0 l s Z V h E a s G 5 W V 4 m R 5 O y w v p E d B a h k B C C + j Q y 7 m 4 E F Q X E l / 9 / 8 2 N H P x U w q a L r 9 w r e f Q n S u A a Z n P G D f g 8 T D 6 + q j c H H K k e r J y W T b 1 0 W n W 1 C I O 1 t S 6 q U 2 U f 0 O 6 X Z p b V X Q K 8 q t 1 h U e D P 4 T V G k z x Z k J U Y A m V o q g i a u L p s u u H u g n O 9 o b Z / n D v 5 O 3 g 4 M P m i F v s J X v F X r O M v W M H 7 C s 7 Z i M m G L J f 7 D f 7 k / D k S 3 K Y H K 2 l S W

and δ i = g x i , ∇ x (f (x i ; θ), y i ); φ . Since more information is provided, we expect the attacker network to be more effective to learn and yield more powerful perturbations.

Multi-:

Step Gradient Attacker Network. We adapt the RNN to mimic a multi-step gradient update. Specifically, we use the gradient attacker network as the cell of RNN sharing the same parameter φ. As we mentioned earlier, the number of layers/iterations in the RNN for modeling algorithms cannot be very large so as to avoid significant computational burden in backpropagation. In this paper, we focus on a length-two RNN to mimic a two-step gradient update. The corresponding perturbation becomes:

δ i = Π B( ) δ (0) i + g x i , ∇ x (f ( x i , y i ; θ); φ , where x i = x i + δ (0) i and δ (0) i = g x i , ∇ x (f (x i , y i ; θ); φ .

Taking gradient attacker network as an example, Figure 3 illustrates how L2L works and jointly trains two networks: The first forward pass is used to obtain gradient of the classification loss over clean data; The second forward pass is used to generate perturbation δ i by attacker g; The third forward pass is used to calculate the adversarial loss in (3). Since our gradient attacker network only needs one backpropagation to query gradient, it amortizes the adversarial training cost, which leads to better computational efficiency. Moreover, L2L may adapt to the underlying optimization problem and yield better solution for the inner problem. The corresponding procedure of L2L is shown in Algorithm 2.

Algorithm 2 Learning-to-learn-based adversarial training with gradient attacker network Input: {(x i , y i )} n i=1 : clean data, α 1 , α 2 : learning rates, N : number of epochs, : maximum perturbation magnitude.

for t ← 1 to N do Sample a minibatch M t for i in M t do u i ← ∇ x (f (x i ; θ), y i ) δ i ← g(x i , u i ; φ) Generate perturbation by g θ ← θ -α 1 1 |Mt| i∈Mt ∇ θ (f (x i + δ i ; θ), y i ) Update θ over adversarial data {(x i + δ i , y i )} i∈Mt φ ← φ + α 2 1 |Mt| i∈Mt ∇ φ (f (x i + δ i θ), y i ) Update φ over adversarial data {(x i + δ i , y i )} i∈Mt

To demonstrate the effectiveness and computational efficiency of our methods, we compare our methods with PGM training over CIFAR-10 and CIFAR-100 datasets. All experiments are done in PyTorch with one NVIDIA 2080 Ti GPU.

For simplicity, we denote Plain Net as the classifier network trained over clean data only, PGM Net as the classifiers with PGM training, and Naive L2L, Grad L2L, and 2-Step L2L as classifiers using L2L training with corresponding attacker networks.

Classifier Network. All experiments adopt a 34-layer wide residual network (WRN-34-10, Zagoruyko and Komodakis ( 2016)) implemented by Zhang et al. (2019) as the classifier network. For each method, we train the classifier network from scratch. For Plain Nets, we adapt the same training procedure as Zagoruyko and Komodakis (2016) . We use cross entropy as the loss function.

Table 1 : Attacker network architecture: k, c, s, p denote the kernel size, output channels, stride and padding parameters of convolutional layers, respectively.

Conv:

[k = 3 × 3, c = 64, s = 1, p = 1], BN+ReLU ResBlock: [k = 3 × 3, c = 128, s = 1, p = 1] ResBlock: [k = 3 × 3, c = 256, s = 1, p = 1] ResBlock: [k = 3 × 3, c = 128, s = 1, p = 1] Conv: [k = 3 × 3, c = 3, s = 1, p = 1], tanh

Attacker Network. (2016) . Due to the space limit, we leave results of the black-box setting in Appendix A.

Robust Evaluation. We evaluate the robustness of the networks by PGM and CW attacks with the maximum perturbation magnitude = 0.031 (after rescaling the pixels to [0, 1]) over CIFAR-10 and CIFAR-100. For PGM attack, we use 20 and 100-iteration PGM with a perturbation step size η = 0.003, and for each sample we initial the adversary randomly in the ∞ ball of radius centered at the natural sample. For CW attack, we adopt the implementation from Paszke et al. (2017) , and set the maximum number of iterations as 100. For each method, we repeat 5 runs with different random initial seed and report the worst result. For CIFAR-10, we also evaluate the robustness of our Grad L2L and 2-Step L2L networks using random attacks, for which we uniformly sample 10000 perturbations in the ∞ ball with radius = 0.031 adding to each test sample. We also evaluate the robustness of our Grad L2L and 2-Step L2L networks under their own attackers.

L2L.:

To update classifier's parameter θ, we use the SGD algorithm with Polyak's momentum (parameter 0.9, Liu et al. (2018) ) and weight decay (parameter 2 × 10 -4 , Krogh and Hertz (1992) ); To update the attacker's parameter φ, we use the Adam optimizer (parameter [0.9, 0.999], Kingma and Ba (2014) ) and weight decay (parameter 2 × 10 -4 ) so that it adaptively balances the inner and outer optimization. In addition, we train the whole network for 100 epochs and set the initial learning rate for SGD as 0.1, the learning rate decay parameter as 0.1 with decay schedule [30, 60, 90] , and the initial learning rate for Adam optimizer as 0.001 without learning rate decay.

PGM Adversarial Training.:

For CIFAR-10, we directly report the result from Madry et al. (2017) as the baseline; For CIFAR-100, we train a PGM Net as a baseline. To update the parameter θ, we use the same configuration of SGD algorithm in L2L. Moreover, we adopt the setting from Madry et al. (2017) , that is, we first use 10-iteration PGM with the perturbation step size 0.007 in (2) to generate the adversarial samples, and then train the PGM Net over these samples. 2 shows the results of all methods over CIFAR-10 and CIFAR-100 under the white-box setting.

As can be seen, without gradient information, Naive L2L is vulnerable to the PGM attack. However, when the attacker utilizes the gradient information, Grad L2L and 2-Step L2L significantly outperform the PGM Net over CIFAR-10 and CIFAR-100, with a small loss for the clean accuracy. From the experiments on CIFAR-10, our Grad L2L and 2-Step L2L are robust to random attacks, where the accuracy is only slightly lower than the clean accuracy. Furthermore, the accuracy of our Grad/2-Step L2L model under the Grad/2-Step L2L attacker is comparable to the accuracy under PGM attacks, which shows that L2L attackers are able to generate strong attacks. As can be seen, PGM-100 is stronger than Grad L2L attacker (47.72% vs. 49.68%), but similar to the 2-Step L2L attacker (52.07% vs. 52.71%). This means the strength of 2-Step L2L attacker is much stronger than Grad L2L attacker and explains why 2-Step L2L is stronger than Grad L2L and PGM net. In addition, Table 3 shows the one epoch running time of all methods over CIFAR-10 and CIFAR-100. As can be seen, Grad L2L and 2-Step L2L is much faster than PGM Net. By further comparing the accuracy of Grad/2-Step L2L and PGM Net in Table 2 , we find that L2L enjoys computational efficiency. In addition, Figure 4 presents the robust accuracy against number of iterations (fixed perturbation magnitude = 0.031) and perturbation magnitude (fixed number of iterations T = 10). As can be seen, 2-Step L2L is much more robust than PGM Net.

Visualization of Adversarial Examples. Figure 5 provides an illustrative example of the adversarial perturbations generated by FGSM, PGM-20 and 2-Step L2L attacker for a cat over CIFAR-10. As can be seen, the attacks for these two networks are very different. Moreover, the perturbation generated by the 2-Step L2L attacker is much more smooth than FGSM and PGM. In this example, 2-Step L2L labels all adversarial samples correctly; whereas the PGM Net is fooled by the PGM-20 attack and misclassifies it as a dog. 

4. EXTENSION: GENERATIVE ADVERSARIAL IMITATION LEARNING:

Our proposed L2L is a generic framework and can be applied to a broad class of minimax optimization problems. Here, we investigate the generative adversarial imitation learning (GAIL, Ho and Ermon ( 2016)). GAIL aims to learn a policy from expert's behavior, by recovering the expert's cost function and extracting a policy from the recovered cost function, which can also be formulated as a minimax optimization problem. We then show that under the L2L framework, we stabilize the GAIL training by replacing the inner optimizer with a neural network.

Original GAIL. GAIL solves the following minimax optimization problem: min In practice, we use a minibatch (several sample trajectories) to estimate g D (θ D , θ π ), denoted as g D (θ D , θ π ). Specifically, at the t-th iteration, we first calculate g t D = g D (θ Experiments. We compare the performance of the original GAIL and GAIL with L2L on two simulated environments: CartPole and Mountain Car (Brockman et al., 2016) . As can be seen in Figure 6 , we find that GAIL has a sudden performance drop after training for a long time. We conjecture that this is because the discriminator overfits the expert trajectories and converges to a bad optimum, which is not generalizable. On the other hand, GAIL with L2L is much more stable. It is very important to real applications of GAIL: since the reward in real-world environment is usually unaccessible, we cannot know whether there is a sudden performance drop or not. With L2L, we can stabilize the training and obtain a much more reliable algorithm for real-world applications. 

5. DISCUSSIONS:

Closely related works:

• By leveraging the Fenchel duality and feature embedding technique, Dai et al. (2016) convert a learning conditional distribution problem to a minimax problem , which is similar to naive attacker network. Both approaches, however, lack the primal information. In contrast, gradient attacker network considers the gradient information of primal variables, and achieves good results.

• Goodfellow et al. (2014a) propose the GAN, which is very similar to our L2L framework. Both GAN and L2L contain one generator network and one classifier network, and jointly train these two networks. There are two major difference between GAN and our framework: Training Stability: For improving the training stability, we use both clean image and the corresponding gradient as the input of the attacker. Without such gradient information, the attacker severely suffers from training instability, e.g., the Naive Attacker Network. Furthermore, we try another architecture with downsampling modules, called "slim attacker" in Appendix B. We observed that the slim attacker also suffers from training instability. We suspect that the downsampling causes the loss of information. Thus, we tried to enhance the slim attacker by skip layer connections.

Although the robust performance is still worse than the proposed architecture, it improves a lot.

Benefits of our neural network approach in adversarial training: (1) Since the neural network has been known to be powerful in function approximation, Our attacker network g can yield strong adversarial perturbations. Since they are generated by the same attacker, the attacker g essentially learns some common structures across all samples;

(2) The attacker networks in our experiments are actually overparametrized, which is conjectured to ease the training of deep neural networks. We believe that similar phenomena happen to our attacker network, and ease the adversarial training.

B SLIM NETWORK:

Table 7 presents another architecture that we used in the L2L. In this network, the second convolutional layer uses downsampling, while the second last deconvolutional layer uses upsampling. Due to the downsampling, this network is computationally cheap and thus it is computationally fast. For example the running time of per epoch for L2L with slim attacker is 480; whereas L2L with the original architecture is 620. However, it loses some information of input and is less stable than the original architecture (Table 1 ). Inspired by residual learning in He et al. (2016) , we address the above issues by using a skip layer connection to ease the training of this network. Specifically, the last layer takes the concatenation of A(x, y, θ) and the output of the second last layer as input.

Figure 7 presents the architecture of ResBlocks. PReLU is a special type of Leaky ReLU with a learnable slope parameter. Table 8 shows the results of L2L with the architecture shown in Table 7 . 

C.2 EXPERIMENT SETTING:

Updater Architecture We use simple 3-layer perceptron with a skip layer as our updater. The number hidden units are (2m → 8m → 4m → m), where m is the dimension of θ D that depends on the original task. For the first and the second layer, we use PReLu as the activation function, while the last layer has no activation function. Finally we add the output to θ D in the original input as the updated parameter for the discriminator network.

Hyperparameter Settings For all baselines we exactly follows the setting in Ho and Ermon (2016) , except that we use a 2-layer discriminator with number of hidden units ((s, a) → 64 → 32 → 1) using tanh as the activation. We use the same neural network architecture for π and the same optimizer configuration. The expert trajectories are obtained by an expert trained using TRPO. For L2L based GAIL, we also use Adam optimizer to update the θ U with the same configuration as updating θ D in original GAIL.

D LIMITING CYCLE:

Limiting cycle is a well-known issue for minimax machine learning problem [4, 5] . The reason behind limiting cycle is that different from minimization problems, a minimax optimization problem is more complicated and can be highly nonconvex-nonconcave, where the inner problem can not be solved exactly. Here we provide a concrete example: we consider the following optimization problem: min Then at the t-th iteration, the update direction will be (-y t , x t ). If we start from (1, 0) with a stepsize of 0.0001, this update will result in a limiting circle: x 2 + y 2 = 1 and never reach the stable equilibrium (0, 0) as shown in Figure 8 . Recently, there are many works on robustness defense that have been proven ineffective (Athalye et al., 2018; Carlini et al., 2019) . Our work follows the most reliable and widely used robust model

:

Loss function (θ; δ) is convex in primal variable θ and concave in dual variable δ.

We provide another attacker architecture with down-sampling modules in the Appendix B. With such an attacker, L2L adversarial training is less stable, but much faster.

Figures and Tables:

Figure 1: Illustration for the hardness of problem (1). A wrong update direction leads to a limiting cycle and algorithms fail to converge. See Appendix D for more details.

o 9 Y S o 0 c p I r D 4 B r 4 T s M + R w 0 c P J L 7 O 7 8 b 0 O z 0 n 9 S g m v z i 7 L u 2 b T n M q B q 5 j X 6 D W h 8 6 / t 6 I c s 5 J E g 2 b r r M n H 1 3 G D n L / W D O 5 s 4 q L / + I d L E 0 K W T a K k i W 6 w q N C r / w I s 9 B p Q 9 j o h Q z q C R 5 F Z H z p 4 v O H u o 8 O N o b R o + H e + + f 9 P d f b o 6 4 x e 6 z B 2 z A I v a U 7 b M 3 7 I C N G W c / 2 S / 2 m / 0 J P g T L 4 G v w b S 0 N O p u a e 6 w V w Y 9 / / I j o M Q = = < / l a t e x i t > t e x i t s h a 1 _ b a s e 6 4 = " E c k 2

Figure 2: An illustration of L2L. We learn a neural network to model the algorithm for generating adversarial attack.

Figure 4: Robust accuracy against perturbation magnitudes and number of iterations of PGM over CIFAR-100. Top two figures show the absolute accuracy over adversarial samples; the bottom two show the difference accuracy (PGM Net as baseline). For larger perturbation magnitude see Appendix E Experiment Results. Table2shows the results of all methods over CIFAR-10 and CIFAR-100 under the white-box setting. As can be seen, without gradient information, Naive L2L is vulnerable to the PGM attack. However, when the attacker utilizes the gradient information, Grad L2L and 2-Step L2L significantly outperform the PGM Net over CIFAR-10 and CIFAR-100, with a small loss for the clean accuracy. From the experiments on CIFAR-10, our Grad L2L and 2-Step L2L are robust to random attacks, where the accuracy is only slightly lower than the clean accuracy. Furthermore, the accuracy of our Grad/2-Step L2L model under the Grad/2-Step L2L attacker is comparable to the accuracy under PGM attacks, which shows that L2L attackers are able to generate strong attacks. As can be seen, PGM-100 is stronger than Grad L2L attacker (47.72% vs. 49.68%), but similar to the 2-Step L2L attacker (52.07% vs. 52.71%). This means the strength of 2-Step L2L attacker is much stronger than Grad L2L attacker and explains why 2-Step L2L is stronger than Grad L2L and PGM net. In addition, Table3shows the one epoch running time of all methods over CIFAR-10 and CIFAR-100. As can be seen, Grad L2L and 2-Step L2L is much faster than PGM Net. By further comparing the accuracy of Grad/2-Step L2L and PGM Net in Table2, we find that L2L enjoys computational efficiency. In addition,

Adversarial samples from PGM Net.

Figure 5: Illustrative examples of FGSM (Top), PGM-20 (Mid), and 2-Step L2L (Bottom) perturbations for a cat under PGM Net and 2-Step L2L with perturbation magnitude = 0.031.

a∼π(s;θπ)  [log (D(s, a; θ D ))] + E s, a∼πE [log(1 -D( s, a; θ D ))] -λH(π), (4) where π(•; θ π ) is the trained policy parameterized by θ π , π E denotes the expert policy, D(•, •; θ D ) is the discriminator parameterized by θ D , λH(π) denotes a entropy regularizer with tuning parameter λ, (s, a) and ( s, a) denote the state-action for the trained policy and expert policy, respectively. By optimizing 4, the discriminator D distinguishes the state-action (s, a) generated from the learned policy π with the sampled trajectories ( s, a) generated from some expert policy π E . In the original GAIL training, for each iteration, we update the parameter of D, θ D , by stochastic gradient ascend and then update θ π by the trust region policy optimization (TRPO,Schulman et al. (2015)). GAIL with L2L. Similar to the adversarial training with L2L, we apply our L2L framework to GAIL by parameterizing the inner optimizer as a neural network U (; θ U ) with parameter θ U . Its input contains two parts: parameter θ D and the gradient of loss function with respect to θ D : g D (θ D , θ π ) = E s,a∼π(s;θπ) [∇ θD log (D(s, a; θ D ))] + E s, a∼π E [∇ θD log(1 -D( s, a; θ D ))].

Figure 6: Reward vs. iteration of the trained policy using original GAIL and L2L GAIL.

(1) GAN aims to transform the random noises to the synthetic data which is similar to the training examples, while ours targets on transforming the training examples to the adversarial examples for robustifying the classifier;(2) Our generator network does not only take the training examples (analogous to the random noise in GAN) as the input, but also exploits the gradient information of the objective function, since it essentially represents an optimization algorithm. The training procedure of these two, however, are quite similar. We adopt some tricks from GAN training to our framework to stabilize training process, e.g., in Grad L2L, we adopt the two-time scale trick(Heusel et al., 2017).• There are some other works simply combining the GAN framework and adversarial training together. For example,Baluja and Fischer (2017) andXiao et al. (2018) propose some ad hoc GANbased methods to robustify neural networks. Specifically, for generating adversarial examples, they only take training examples as the input of the generator, which lacks the information of the outer mimnimization problem. Instead, our proposed L2L methods (e.g., Grad L2L, 2-step L2L) connect outer and inner problems by delivering the gradient information of the objective function to the generator. This is a very important reason for our performance gain on the benchmark datasets. As a result, the aforementioned GAN-based methods are only robust to simple attacks, e.g., FGSM, on simple data sets, e.g., MNIST, but fail for strong attacks, e.g., PGM and CW, on complicated data sets, e.g. CIFAR, where our L2L methods achieve significantly better performance.

Figure 7: An illustration example for the architecture of ResBlocks.

Learning-to-learn-based generative adversarial imitation learning Input: Expert trajectories ( s, a ∼ π E ( s)), initial policy, discriminator and updater parametersθ π , θ D , θ U . for t ← 1 to N do Sample trajectories (s, a ∼ π(a; θ π ))Compute gradient:g t D ← 1 |(s,a)| (s,a) [∇ θD log (D(s, a; θ t D ))] + 1 |( s, a)| ( s, a) [∇ θD log(1 -D( s, a; θ t D ))] Update the discriminator parameters: θ t+1 D = U (θ t D , g t D ; θ t U ). Update the updater parameters θ U by minimizing:1 |(s,a)| (s,a) [log (D(s, a; U (θ t D , g t D ; θ t U )))] + 1 |( s, a)| ( s, a) [log(1 -D( s, a; U (θ t D , g t D ; θ t U )))]. Update the policy parameters θ π by a policy step using the TRPO rule(Ho and Ermon (2016))

, y) = xy.

Figure 8: An example of the limiting circle: arrows denote the update directions

Table 1 presents the architecture of our attacker network 2 .

Table content: <table/>

Results of different defense methods under the white-box setting.

Table content: <table><tr><td>Defense Method</td><td>Attack</td><td>Dataset</td><td colspan="2">Clean Accuracy Robust Accuracy</td></tr><tr><td>Zheng et al. (2016)</td><td>PGM-20</td><td>CIFAR10</td><td>94.64%</td><td>0.15%</td></tr><tr><td>Kurakin et al. (2016)</td><td>PGM-20</td><td>CIFAR10</td><td>85.25%</td><td>45.89%</td></tr><tr><td>Madry et al. (2017)</td><td>PGM-20</td><td>CIFAR10</td><td>87.30%</td><td>47.04%</td></tr><tr><td>Naive L2L</td><td>PGM-20</td><td>CIFAR10</td><td>94.53%</td><td>0.01%</td></tr><tr><td>Grad L2L</td><td>PGM-20</td><td>CIFAR10</td><td>85.84%</td><td>51.17%</td></tr><tr><td>2-Step L2L</td><td>PGM-20</td><td>CIFAR10</td><td>85.35%</td><td>54.32%</td></tr><tr><td>Grad L2L</td><td>PGM-100</td><td>CIFAR10</td><td>85.84%</td><td>47.72%</td></tr><tr><td>2-Step L2L</td><td>PGM-100</td><td>CIFAR10</td><td>85.35%</td><td>52.12%</td></tr><tr><td>Grad L2L</td><td>CW</td><td>CIFAR10</td><td>85.84%</td><td>53.5%</td></tr><tr><td>2-Step L2L</td><td>CW</td><td>CIFAR10</td><td>85.35%</td><td>57.07%</td></tr><tr><td>Grad L2L</td><td>Random</td><td>CIFAR10</td><td>85.84%</td><td>82.67%</td></tr><tr><td>2-Step L2L</td><td>Random</td><td>CIFAR10</td><td>85.35%</td><td>83.10%</td></tr><tr><td>Grad L2L</td><td>Grad L2L</td><td>CIFAR10</td><td>85.84%</td><td>49.68%</td></tr><tr><td>2-Step L2L</td><td>2-Step L2L</td><td>CIFAR10</td><td>85.35%</td><td>52.71%</td></tr><tr><td>PGM Net</td><td>PGM-20</td><td>CIFAR100</td><td>62.68%</td><td>23.75%</td></tr><tr><td>Grad L2L</td><td>PGM-20</td><td>CIFAR100</td><td>62.18%</td><td>28.67%</td></tr><tr><td>2-Step L2L</td><td>PGM-20</td><td>CIFAR100</td><td>60.95%</td><td>31.03%</td></tr><tr><td>PGM Net</td><td>PGM-100</td><td>CIFAR100</td><td>62.68%</td><td>22.06%</td></tr><tr><td>Grad L2L</td><td>PGM-100</td><td>CIFAR100</td><td>62.18%</td><td>26.69%</td></tr><tr><td>2-Step L2L</td><td>PGM-100</td><td>CIFAR100</td><td>60.95%</td><td>29.75%</td></tr><tr><td>PGM Net</td><td>CW</td><td>CIFAR100</td><td>62.68%</td><td>25.95%</td></tr><tr><td>Grad L2L</td><td>CW</td><td>CIFAR100</td><td>62.18%</td><td>29.65%</td></tr><tr><td>2-Step L2L</td><td>CW</td><td>CIFAR100</td><td>60.95%</td><td>32.28%</td></tr></table>

Running time for one epoch over CIFAR-10 (1 st Row) and CIFAR-100 (2 nd Row). (Unit: s)

Table content: <table><tr><td>Plain Net</td><td>PGM Net</td><td>Naive L2L</td><td>Grad L2L</td><td>2-Step L2L</td></tr><tr><td>106.5 ± 1.5</td><td>1310.8 ± 14.2</td><td>293.7 ± 3.1</td><td>617.5 ± 6.1</td><td>805.1 ± 8.1</td></tr><tr><td>106.9 ± 1.4</td><td>1354.8 ± 14.1</td><td>310.0 ± 2.9</td><td>623.1 ± 6.3</td><td>824.7 ± 8.4</td></tr></table>

Attacker Network Architecture.

Table content: <table/>

Results of L2L with the other attacker network under white-box setting over CIFAR.

Table content: <table><tr><td/><td>CIFAR-10</td><td/><td colspan="2">CIFAR-100</td></tr><tr><td/><td colspan="2">Clean FGSM PGM20 CW</td><td colspan="3">Clean FGSM PGM20 CW</td></tr><tr><td>Naive L2L</td><td>94.41 28.44 0.01</td><td colspan="2">0.00 75.27 8.47</td><td>0.05</td><td>0.00</td></tr><tr><td>Grad L2L</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
The authors propose a framework where one component is an attacker network that keeps learning about how to perturb the loss more, and one component is a defense network that robustify learning with respect to the attacker network. The framework is flexible on how the attacker network can be trained, and advances over previous works where the attacker is a human-designed algorithm rather than a learning model. Experiment results show that the framework reaches superior defense performance. The authors also extend the framework to help imitation learning. Overall the paper is a pleasure to read. My questions/suggestions are (1) Given that the framework seems natural in design, a deeper contribution would be talking about how to successfully train the framework in practice. The authors talk about the connection of the framework to GAN, and the latter is not that easy to train. However, we see very little information on how to train the framework in the paper. Was it super easy to train the framework (why?), or did the author encounter any difficulties? Are there important heuristics that help train the framework successfully? (2) While the framework leads to a better defense mechanism (the authors* goal), one could wonder whether it leads to a better attacker as well. Instead of just checking the differences of the attacking examples generated, can we take the inner attacker and see if it is more effective in attacking than PGM and CW? How does the goodness of the attacker improve with time? Do different L2L variants generate attackers with different quality? Do the quality connect with the defense performance? (3) Section 4 looks distracting to me. It is good to know that the framework can be extended to imitation learning, but the section is best put at a longer version or another paper, rather than occupying a significant amount of space in the current paper. I have read the rebuttal and thank the authors for the response.