Adaptive Generation of Unrestricted Adversarial Inputs

Neural networks are vulnerable to adversarially-constructed perturbations of their inputs . Most research so far has considered perturbations of a fixed magnitude under some lp norm . Although studying these attacks is valuable , there has been increasing interest in the construction of—and robustness to—unrestricted attacks , which are not constrained to a small and rather artificial subset of all possible adversarial inputs . We introduce a novel algorithm for generating such unrestricted adversarial inputs which , unlike prior work , is adaptive : it is able to tune its attacks to the classifier being targeted . It also offers a 400–2,000× speedup over the existing state of the art . We demonstrate our approach by generating unrestricted adversarial inputs that fool classifiers robust to perturbation-based attacks . We also show that , by virtue of being adaptive and unrestricted , our attack is able to defeat adversarial training against it . 1 INTRODUCTION . Despite their dramatic successes in other respects , neural networks are well-known to not be adversarially robust . Szegedy et al . ( 2014 ) discovered that neural networks are vulnerable to what they termed adversarial inputs : by adding carefully-chosen perturbations to correctly-classified inputs , the accuracy of any neural network could be almost arbitrarily decreased . Since then , the machine learning community has rightly focused a great deal of research effort on this phenomenon . Many early efforts to train more robust models initially appeared promising , but have since been shown to be vulnerable to new algorithms for constructing adversarial perturbations ( Xu et al. , 2019 ; Athalye et al. , 2018 ) . As a result , more attention has been given to methods that provide formal guarantees about performance in the presence of adversarial perturbations ( Liu et al. , 2019 ) , with the state of the art now providing non-trivial guarantees for the MNIST test set ( Wong & Kolter , 2018 ; Croce et al. , 2018 ; Wang et al. , 2018 ) . However , almost all of this work has focused exclusively on adversarial perturbations whose magnitude is constrained by an lp norm . There is a growing acknowledgement that this threat model is somewhat contrived : such examples are not a realistic security concern and also occupy a vanishingly small fraction of the set of potential adversarial inputs . Therefore , there is a burgeoning interest in adversarial attacks that are unrestricted , in the sense that they do not necessarily derive from a perturbation of a natural input ( Brown et al. , 2018 ; Song et al. , 2018b ) . The main contribution of this paper is a novel and general method to generate unrestricted adversarial inputs . In short , the training procedure for generative adversarial networks ( GANs ) is modified so that the generator network is rewarded for producing data that are both realistic and deceive a fixed target network . Our approach has four advantages over prior work : 1 . Our method is adaptive in that it adjusts itself to best attack the specific network being targeted . For instance , adversarial training is ineffective against our approach . 2 . Our method is efficient ( offering a 400–2000× speedup over prior work ) . 3 . Our method can easily be applied to any existing conditional GAN codebase and checkpoints , regardless of architecture , training procedure , or application domain . 4 . Our method therefore demonstrably scales to ImageNet . 2 BACKGROUND : GENERATIVE ADVERSARIAL NETWORKS . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) are a class of generative machine learning models involving the simultaneous training of two neural networks : a generator g and a discriminator d. Specifically , given a dataset D of samples drawn from a probability distribution pD , the generator g learns to transform random noise z drawn from a simple distribution pz into an approximation of pD . The discriminator network d learns to predict whether a given example x is drawn from the data distribution pD or was generated by g. The generator and the discriminator are adversarial because they train simultaneously , with each being rewarded for out-performing the other . GANs ’ training behaviours are notoriously temperamental , and many modifications to the original algorithm have been proposed ( Goodfellow , 2017 ) . The Wasserstein GAN variant ( Arjovsky et al. , 2017 ) aims to provide a more reliable gradient by designing the discriminator ( renamed ‘ critic ’ ) to approximate the Wasserstein distance between the distribution generated by gθ and the data distribution pD . An additional ‘ gradient penalty ’ loss termLgp can be added to implement the constraint that the function be 1-Lipschitz continuous ( Gulrajani et al. , 2017 ) . The loss functions for this Wasserstein GAN with gradient penalty ( WGAN-GP ) are : Lg=Ez∼pz [ −d ( g ( z ) ) ] and Ld=−Lg+Ex∼pD [ −d ( x ) ] +λLgp ; where the gradient penalty Lgp = Ex̃∼pI [ ( ‖∇x̃dφ ( x̃ ) ‖2 − 1 ) 2 ] , where pI denotes the distribution sampling uniformly from the linear interpolations between generated samples and examples from pD . The original proposal for a conditional generative adversarial network ( CGAN ) learns to generate samples from a conditional distribution ( Mirza & Osindero , 2014 ) by simply passing the intended label y for the generated image to both the generator and the discriminator . An extension of this approach is the auxiliary classifier generative adversarial network ( ACGAN ) ( Odena et al. , 2017 ) , in which the discriminator is modified to also predict the label y for the input data . Both the generators are trained to maximise the log-likelihood of the correct label in addition to optimising their usual objective . 3 GENERATING UNRESTRICTED ADVERSARIAL INPUTS . Suppose we have a trained target classifier network f : X→R|Y | that attempts to approximate an oracle function o : O→Y ( whereO⊆X is the oracle ’ s domain ) by outputting a confidence f ( x ) c∈R for each class c ∈ Y . As Song et al . ( 2018b ) do , we define an unrestricted adversarial example to be any input x∈O such that the classifier ’ s prediction is incorrect : argmaxcf ( x ) c 6= o ( x ) . Unlike Song et al. , we consider the domain of the oracle to be any input with a recognisable class , not just realistic inputs . Gilmer et al . ( 2018 ) offer a number motivations for why unrealistic but recongisable adversarial examples could be interesting , including security concerns and improving models ’ abilities to generalize . Nevertheless , we do carefully evaluate how realistic our results are in Section 4.1 . Unrestricted adversarial examples are a superset of conventional perturbation-based adversarial examples ( which are restricted to lie within a fixed distance of some correctly-classified input from a test dataset ) . While providing a vastly larger space of candidates , a difficulty arises in determining that the classification is incorrect ; we can no longer rely on the oracle-provided labels from the test dataset . We leverage generative models to solve this problem . 3.1 OUR PROCEDURE . We begin by taking any conditional GAN , with generator loss lordinary ; the use of a conditional GAN allows us to determine the correct label y of our generated unrestricted adversarial examples . We then introduce loss terms which incentivise the generator g to create adversarial examples for a classifier f . A targeted attack for desired true label y and target label t 6= y should output an image which humans would regard to have label y yet is classified as t by the classifier . We introduce a loss term , ltargeted , which is minimised when the conditional generator output g ( z , y ) is classified in this way : ltargeted=maxc 6=tf ( g ( z , y ) ) c−f ( g ( z , y ) ) t. For an untargeted attack , we use a loss term that is minimised for any misclassification of the example : luntargeted =f ( g ( z , y ) ) y−maxc6=yf ( g ( z , y ) ) c. Note that these new terms assume that the true labels of the generated data g ( z , y ) do indeed match the intended labels y , an assumption empirically validated in Section 4.1 . Our procedure is to alter the generator ’ s training objective so as to minimise both lordinary and l ( un ) targeted simultaneously , thereby training the generator to output data which are both realistic and also fool the target network . Section 3.3 provides further details . 3.2 CHALLENGE : CONFLICTING GRADIENTS . Intuition suggests that the gradient from lordinary may be pointing in a different direction to the gradient from l ( un ) targeted , since making an image adversarial seems likely to make it less realistic , not more . A simple experiment suffices to verify this intuition . We compute the cosine similarity between the gradients of our two loss terms at each step , i.e . ∇lordinary·∇l ( un ) targeted‖∇lordinary‖‖∇l ( un ) targeted‖ . Figure 1a shows that this projection tends towards −1 ; for reference , if the gradient vectors were selected uniformly at random , the magnitude of this projection would very rarely exceed 0.001 . In other words , as training progresses , the gradients from these terms tend towards pointing in opposite directions . This makes joint optimisation using a gradient descent approach challenging . 3.3 STRATEGIES TO OVERCOME TRAINING CHALLENGES . We empirically evaluate the effect of each technique described below in our ablative experiments reported in Section 4.4 . Realistic pretraining It is widely accepted that real image data occupy a relatively low-dimensional and contiguous manifold ( Goodfellow et al. , 2016 , p. 160 ) . Conversely , we know that adversarial examples pervade the full input space : it appears that there is an adversarial example nearby nearly any point in the input space . Therefore , a generator that is pretrained using only lordinary before adversarially finetuning by introducing our additional loss term is more successful than using both loss terms from a random initialisation . By beginning our search in regions of realistic examples , we ’ re more likely to find adversarial examples that are sufficiently realistic . Besides the generated images being subjectively better , Figure 1b shows that the gradients conflict to a much lesser extent . Note that any existing conditional GAN architecture , pretrained checkpoint and training algorithm could be used here , allowing our method to leverage the significant advances being made in this area . Amalgamation of loss terms Rather than naïvely summing lordinary and l ( un ) targeted , we use the following per-example loss term : lfinetune=s ( lordinary ) ·s ( l ( un ) targeted−κ ) , where s ( l ) = { 1+exp ( l ) if l≤ 0 , 2+l otherwise . Here , κ is a hyperparameter similar to that in the Carlini & Wagner ( 2017 ) attack : it controls the confidence of the generated adversarial examples . If the difference between the desired logit and the next-greatest logit is less than κ , the generator is linearly rewarded for improving this gap ( gaining confidence ) ; beyond a difference of κ ( once an example is ‘ good enough ’ ) , the reward exponentially decreases . We use κ=0 for our experiments as we do not require strong misclassifications . Stochastic loss selection The gradients from the two loss terms are in conflict , and in practice the l ( un ) targeted gradient dominates . The proportion of misclassified generated inputs rises quickly to almost 100 % , but the generated images were noticeably unrealistic , meaning their correct label may change . To address this , we introduce the ‘ attack rate ’ µ . During adversarial finetuning , the finetuning loss term is used at each step only with probability µ ; with probability 1−µ , the pretraining loss ( lordinary “ Goldfish ” : “ Tarantulas ” : “ Lemons ” : “ Churches ” : Figure 3 : Selected successful targeted unrestricted adversarial examples on ImageNet , generated by a BigGAN ( Brock et al. , 2019 ) finetuned to attack ResNet-152 ( He et al. , 2016 ) . only ) is used . As desired , this new hyperparameter allows the success rate of the generated unrestricted adversarial examples to be traded off with their realism .

Summary: The paper proposes using GANs to generate unrestricted adversarial examples. They seek to generate examples that are adversarial for a specific classifier, and they do so by using class-conditional GANs and a fine-tuning loss. The fine-tuning loss consists of both the ordinary GAN loss (to fool the discriminator) as well as an adversarial loss (which rewards the GAN for generating examples misclassified by the specific classifier). The authors perform various experiments on their generated examples to check for realism and how adversarial the generated images are.