Learning to Defense by Learning to Attack

1 INTRODUCTION . This decade has witnessed great breakthroughs in deep learning in a variety of applications , such as computer vision ( Taigman et al. , 2014 ; Girshick et al. , 2014 ; He et al. , 2016 ; Liu et al. , 2017 ) . Recent studies ( Szegedy et al. , 2013 ) , however , show that most of these deep learning models are very vulnerable to adversarial attacks . Specifically , by injecting a small perturbation to a normal sample , one can obtain an adversarial example . Although the adversarial example is semantically indistinguishable from the normal one , it can fool deep learning models and undermine the security of deep learning , causing reliability problems in autonomous driving , biometric authentication , etc . Researchers have devoted many efforts to studying efficient adversarial attack and defense ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014b ; Nguyen et al. , 2015 ; Zheng et al. , 2016 ; Madry et al. , 2017 ; Carlini and Wagner , 2017 ) . There is a growing body of work on generating adversarial examples , e.g. , fast gradient sign method ( FGSM , Goodfellow et al . ( 2014b ) ) , projected gradient method ( PGM , Kurakin et al . ( 2016 ) ) , Carlini-Wagner ( CW , Paszke et al . ( 2017 ) ) etc . As for defense , Goodfellow et al . ( 2014b ) propose to robustify the network by adversarial training , which trains over the adversarial examples and still requires the network to output the correct label . Further , Madry et al . ( 2017 ) formalize the adversarial training as the following minimax optimization problem : minθ 1 n ∑n i=1 [ maxδi∈B ` ( f ( xi + δi ; θ ) , yi ) ] , ( 1 ) where { ( xi , yi ) } ni=1 ⊂ Rd × Y are n pairs of input feature and the corresponding label , ` denotes a loss function , f ( · ; θ ) denotes the neural network with parameter θ , and δi ∈ B denotes the perturbation for xi in constraint B . The existing literature on the optimization also refers to θ as the primal variable and δi ’ s as the dual variables . Different from the well-studied convex-concave problem1 , problem ( 1 ) is very challenging since ` is nonconvex in θ and nonconcave in δ . As a result , there may be many equilibria , and majority of them are unstable . In the existing optimization literature , there is no algorithm to converge to a stable equilibrium with theoretical guarantees . Empirically , the existing primal-dual algorithms may perform poorly for solving ( 1 ) . Minimax formulation ( 1 ) naturally provides us with a unified perspective on prior works of adversarial training . Such a minimax problem consists of two optimization problems , an inner maximization problem and an outer minimization problem : The inner problem targets on finding an optimal attack 1Loss function ` ( θ ; δ ) is convex in primal variable θ and concave in dual variable δ. for a given data point ( x , y ) maximizing the loss , which essentially is the adversarial attack ; The outer problem aims to find a θ so that the loss given by the inner problem is minimized . For solving ( 1 ) , Goodfellow et al . ( 2014b ) propose to use FGSM to solve the inner problem . Kurakin et al . ( 2016 ) then find that FGSM with true label predicted suffers from a “ label leaking ” effect , which can ruin the adversarial training . Madry et al . ( 2017 ) further suggest to solve the inner problem by PGM and obtain a better result than FGSM , since FGSM essentially is one iteration PGM . However , adversarial training needs to find a δi for each ( xi , yi ) , thus the dimension of the overall search space for all data is substantial , which makes the computation expensive . Instead , we propose a new learning-to-learn ( L2L ) framework that provides a more principled and efficient way for adversarial training . Specifically , we parameterize the optimizer of the inner maximization problem by a neural network denoted by g ( A ( x , y , θ ) ; φ ) , where A ( x , y , θ ) denotes the input of the optimizer g. We also call the optimizer as the attacker network . Since the neural network is very powerful in function approximation , our parameterization ensures that g is able to yield strong adversarial perturbations . Under our framework , instead of directly solving δi , we update the parameter φ of g. Our training procedure becomes updating the parameters of two neural networks , which is very similar to generative adversarial network ( GAN , Goodfellow et al . ( 2014a ) ) . The proposed L2L is a generic framework and can be extended to other minimax optimization problems , e.g. , generative adversarial imitation learning , which is studied in Section 4 . Different from the hand-designed methods that compute the adversarial perturbation for each individual sample using gradients from backpropagation , our methods generate the perturbations for all samples through the shared attacker g. This enables the attacker g to learn potential common structures of the perturbations . Therefore , our method is capable of yielding strong perturbations and accelerating the training process . Furthermore , L2L framework is very flexible : we can either choose different inputA ( x , y , θ ) , or use different attacker architecture . For example , we can include gradient information in A ( x , y , θ ) and use a recurrent neural network ( RNN ) to mimic multi-step gradient-type methods . Instead of simply computing the high order information with finite difference approximation or multiple gradients , by parameterizing the algorithm as a neural network , our proposed methods can capture this information in a much smarter way ( Finn et al. , 2017 ) . Our experiments demonstrate that our proposed methods not only outperform existing adversarial training methods , e.g. , PGM training , but also enjoy the computational efficiency over CIFAR-10 and CIFAR-100 datasets ( Krizhevsky and Hinton , 2009 ) . The research on L2L has a long history ( Schmidhuber , 1987 ; 1992 ; 1993 ; Younger et al. , 2001 ; Hochreiter et al. , 2001 ; Andrychowicz et al. , 2016 ) . The basic idea is that one first models the updating formula of complicated optimization algorithms in a parametric form , and then uses some simple algorithms , e.g. , stochastic gradient algorithm to learn the parameters of the optimizer . Among existing works , Hochreiter et al . ( 2001 ) propose a system allowing the output of backpropagation from one network to feed into an additional learning network , with both networks trained jointly ; Based on this , Andrychowicz et al . ( 2016 ) further show that the design of an optimization algorithm can be cast as a learning problem . Specifically , they use long short-term memory RNNs to model the algorithm and allow the RNNs to exploit structure in the problems of interest in an automatic way , which is undoubtedly one of the most popular methods for learning-to-learn . However , there are two major drawbacks of the existing L2L methods : ( 1 ) It requires a large amount of datasets ( or a large number of tasks in multi-task learning ) to guarantee the learned optimizer to generalize , which significantly limits their applicability ( most of the related works only consider the image encoding as the motivating application ) ; ( 2 ) The number of layers/iterations in RNNs for modeling algorithms can not be large to avoid significant computational burden in backpropagation . Our contribution is that we fill the blank of the L2L framework in solving the minimax problem , and our proposed methods do not suffer from the aforementioned drawbacks : ( 1 ) The attacker g with a different φ essentially generates a different task/dataset . Therefore , for adversarial training , we have sufficiently many tasks for learning-to-learn ; ( 2 ) The inner problem does not need a large scale RNN , and we use a convolutional neural network ( CNN ) or a length-two RNN ( the sequence of length equals 2 ) as our attacker network , which eases the computation . Our work is also related to GAN and dual-embedding ( Dai et al. , 2016 ) . All three solve minimax problems and share some common ground . We discuss these works in Section 5 . Notations . Given a ∈ R , denote ( a ) + as max ( a , 0 ) . Given x , y ∈ Rd , denote xi as the i-th element of x , ||x||∞ = maxi |xi| as ` ∞-norm of x , and x◦y = [ x1y1 , · · · , xdyd ] > as element-wise product . 2 METHOD This paper focuses on ` ∞-norm attack . We define the ` ∞-ball with center 0 and radius by B ( ) = { δ ∈ Rd : ||δ||∞ ≤ } and the corresponding projection as follows : ΠB ( ) ( δ ) = sign ( δ ) ◦max ( |δ| , ) , where sign and max are element-wise operators . 2.1 ADVERSARIAL TRAINING . The goal of adversarial training is to robustify neural networks . Recall that from a robust optimization perspective , given n samples { ( xi , yi ) } ni=1 , where xi is the i-th feature vector and yi is the corresponding label , adversarial training is reformulated as the following optimization problem ( 1 ) : minθ 1 n ∑n i=1 [ maxδi∈B ` ( f ( xi + δi ; θ ) , yi ) ] , where f denotes the network with parameter θ , ` denotes a loss function , and is the maximum perturbation magnitude . We first demonstrate the hardness of solving problem ( 1 ) . Ideally , we want to obtain the optimal solution for the inner problem δ∗i : = argmaxδi∈B ` ( f ( xi + δi ; θ ) , yi ) . However , note that the loss function ` ( f ( xi + δi ; θ ) , yi ) is highly nonconcave in δi . Therefore , in reality the δi we obtained is very unlikely to be the optimum δ∗i . This then often leads to a highly unreliable or even completely wrong search direction , e.g. , 〈∇θ ` ( f ( xi + δi ; θ ) , yi ) , ∇θ ` ( f ( xi + δ∗i ; θ ) , yi ) 〉 < 0 , which may further results in a limiting cycle shown in Figure 1 ( Detailed discussion is provided in Appendix D ) . This becomes even worse when sample noises exist . Algorithm 1 Standard pipeline of adversarial training Input : { ( xi , yi ) } ni=1 : clean data , α : learning rate , N : number of epochs , : maximum perturbation magnitude . for t← 1 to N do Sample a minibatchMt for i inMt do δi ← argmaxδi∈B ( ) ` ( f ( xi + δi ; θ ) , yi ) Generate adversarial perturbation for ( xi , yi ) θ ← θ − α 1|Mt| ∑ i∈Mt ∇θ ` ( f ( xi + δi ; θ ) , yi ) Update θ over adversarial data { ( xi + δi , yi ) } i∈Mt In the existing literature , the standard pipeline of adversarial training is shown in Algorithm 1 . Since the step of generating adversarial perturbation δi in Algorithm 1 is intractable , most adversarial training methods adopt hand-designed algorithms . For example , Kurakin et al . ( 2016 ) propose to solve the inner problem approximately by first order methods such as PGM . Specifically , PGM iteratively updates the adversarial perturbation by the projected sign gradient ascent method for each sample : Given one sample ( xi , yi ) , at the t-th iteration , PGM takes δti ← ΠB ( ) ( δt−1i + η · sign ( ∇x ` ( f ( x̃ti ; θ ) , y ) ) ) , ( 2 ) where x̃ti = xi + δ t−1 i , η is the perturbation step size , T is a pre-defined total number of iterations , and δ0i = 0 , t = 1 , · · · , T . Finally PGM takes δi = δTi . Note that FGSM essentially is one-iteration PGM . Besides , some works adopt other optimization methods , such as momentum gradient method ( Dong et al. , 2018 ) , and L-BFGS ( Tabacof and Valle , 2016 ) . However , except for FGSM , all require numerous queries for gradients through backpropagation , which is computationally expensive .

Summary: The authors propose a framework where one component is an attacker network that keeps learning about how to perturb the loss more, and one component is a defense network that robustify learning with respect to the attacker network. The framework is flexible on how the attacker network can be trained, and advances over previous works where the attacker is a human-designed algorithm rather than a learning model. Experiment results show that the framework reaches superior defense performance. The authors also extend the framework to help imitation learning.