Automated Discovery of Adaptive Attacks on Adversarial Defenses

1 Introduction . The issue of adversarial attacks ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) , i.e. , crafting small input perturbations that lead to mispredictions , is an important problem with a large body of recent work . Unfortunately , reliable evaluation of proposed defenses is an elusive and challenging task : many defenses seem to initially be effective , only to be circumvented later by new attacks designed specifically with that defense in mind ( Carlini & Wagner , 2017 ; Athalye et al. , 2018 ; Tramer et al. , 2020 ) . To address this challenge , two recent works approach the problem from different perspectives . Tramer et al . ( 2020 ) outlines an approach for manually crafting adaptive attacks that exploit the weak points of each defense . Here , a domain expert starts with an existing attack , such as PGD ( Madry et al. , 2018 ) ( denoted as • in Figure 1 ) , and adapts it based on knowledge of the defense ’ s inner workings . Common modifications include : ( i ) tuning attack parameters ( e.g. , number of steps ) , ( ii ) replacing network components to simplify the attack ( e.g. , removing randomization or non-differentiable components ) , and ( iii ) replacing the loss function optimized by the attack . This approach was demonstrated to be effective in breaking all of the considered 13 defenses . However , a downside is that it requires substantial manual effort and is limited by the domain knowledge of the expert – for instance , each of the 13 defenses came with an adaptive attack which was insufficient , in retrospect . At the same time , Croce & Hein ( 2020b ) proposed to assess adversarial robustness using an ensemble of four attacks illustrated in Figure 1 ( b ) – APGDCE with cross-entropy loss ( Croce & Hein , 2020b ) , APGDDLR with difference in logit ratio loss , FAB ( Croce & Hein , 2020a ) , and SQR ( Andriushchenko et al. , 2020 ) . While these do not require manual effort and have been shown to improve the robustness 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( b ) Ensemble of fixed attacks Croce & Hein ( 2020b ) ( a ) Handcrafted adaptive attacks Tramer et al . ( 2020 ) ( c ) Adaptive attack search ( Our Work ) estimate for many defenses , the approach is inherently limited by the fact that the attacks are fixed apriori without any knowledge of the given defense at hand . This is visualized in Figure 1 ( b ) where even though the attacks are designed to be diverse , they cover only a small part of the entire space . This work : towards automated discovery of adaptive attacks We present a new method that helps automating the process of crafting adaptive attacks , combining the best of both prior approaches ; the ability to evaluate defenses automatically while producing attacks tuned for the given defense . Our work is based on the key observation that we can identify common techniques used to build existing adaptive attacks and extract them as reusable building blocks in a common framework . Then , given a new model with an unseen defense , we can discover an effective attack by searching over suitable combinations of these building blocks . To identify reusable techniques , we analyze existing adaptive attacks and organize their components into three groups : • Attack algorithm and parameters : a library of diverse attack techniques ( e.g. , APGD , FAB , C & W ( Carlini & Wagner , 2017 ) , NES ( Wierstra et al. , 2008 ) ) , together with backbone specific and generic parameters ( e.g. , input randomization , number of steps , if/how to use EOT ( Athalye et al. , 2018 ) ) . • Network transformations : producing an easier to attack surrogate model using techniques including variants of BPDA ( Athalye et al. , 2018 ) to break gradient obfuscation , and layer removal ( Tramer et al. , 2020 ) to eliminate obfuscation layers such as redundant softmax operator . • Loss functions : that specify different ways of defining the attack ’ s loss function . These components collectively formalize an attack search space induced by their different combinations . We also present an algorithm that effectively navigates the search space so to discover an attack . In this way , domain experts are left with the creative task of designing new attacks and growing the framework by adding missing attack components , while the tool is responsible for automating many of the tedious and time-consuming trial-and-error steps that domain experts perform manually today . That is , we can automate some part of the process of finding adaptive attacks , but not necessarily the full process . This is natural as finding truly new attacks is a highly creative process that is currently out of reach for fully automated techniques . We implemented our approach in a tool called Adaptive AutoAttack ( A3 ) and evaluated it on 24 diverse adversarial defenses . Our results demonstrate that A3 discovers adaptive attacks that outperform AutoAttack ( Croce & Hein , 2020b ) , the current state-of-the-art tool for reliable evaluation of adversarial defenses : A3 finds attacks that are significantly stronger , producing 3.0 % -50.8 % additional adversarial examples for 10 models , while obtaining attacks with stronger or simialr strength for the remaining models . Our tool A3 and scripts for reproducing the experiments are available online at : https : //github.com/eth-sri/adaptive-auto-attack 2 Automated Discovery of Adaptive Attacks . We use D = { ( xi , yi ) } Ni=1 to denote a training dataset where x ∈ X is a natural input ( e.g. , an image ) and y is the corresponding label . An adversarial example is a perturbed input x′ , such that : ( i ) it satisfies an attack criterion c , e.g. , a K-class classification model f : X → RK predicts a wrong label , and ( ii ) the distance d ( x′ , x ) between the adversarial input x′ and the natural input x is below a threshold under a distance metric d ( e.g. , an Lp norm ) . Formally , this can be written as : d ( x′ , x ) ≤ such that c ( f , x′ , x ) capability goal ( criterion ) ADVERSARIAL ATTACK For example , instantiating this with the L∞ norm and misclassification criterion corresponds to : ‖x′ − x‖∞ ≤ s.t . f̂ ( x′ ) 6= f̂ ( x ) MISCLASSIFICATION L∞ ATTACK where f̂ returns the prediction arg maxk=1 : K fk ( · ) of the model f . Problem Statement Given a model f equipped with an unknown set of defenses and a dataset D = { ( xi , yi ) } Ni=1 , our goal is to find an adaptive adversarial attack a ∈ A that is best at generating adversarial samples x′ according to the attack criterion c and the attack capability d ( x′ , x ) ≤ : max a∈A , d ( x′ , x ) ≤ E ( x , y ) ∼D c ( f , x′ , x ) where x′ = a ( x , f ) ( 1 ) Here , A denotes the search space of all possible attacks , where the goal of each attack a : X× ( X→ RK ) → X is to generate an adversarial sample x′ = a ( x , f ) for a given input x and model f . For example , solving this optimization problem with respect to the L∞ misclassification criterion corresponds to optimizing the number of adversarial examples misclassified by the model . In our work , we consider an implementation-knowledge adversary , who has full access to the model ’ s implementation at inference time ( e.g. , the model ’ s computational graph ) . We chose this threat model as it matches our problem setting – given an unseen model implementation , we want to automatically find an adaptive attack that exploits its weak points but without the need of a domain expert . We note that this threat model is weaker than a perfect-knowledge adversary ( Biggio et al. , 2013 ) , which assumes a domain expert that also has knowledge about the training dataset and algorithm , as this information is difficult , or even not possible , to recover from the model ’ s implementation only . Key Challenges To solve the optimization problem from Eq . 1 , we address two key challenges : • Defining a suitable attacks search space A such that it is expressible enough to cover a range of existing adaptive attacks . • Searching over the space A efficiently such that a strong attack is found within a reasonable time . Next , we formalize the attack space in Section 3 and then describe our search algorithm in Section 4 . 3 Adaptive Attacks Search Space . We define the adaptive attack search space by analyzing existing adaptive attacks and identifying common techniques used to break adversarial defenses . Formally , the adaptive attack search space is given byA : S×T , where S consists of sequences of backbone attacks along with their loss functions , selected from a space of loss functions L , and T consists of network transformations . Semantically , given an input x and a model f , the goal of adaptive attack ( s , t ) ∈ S× T is to return an adversarial example x′ by computing s ( x , t ( f ) ) = x′ . That is , it first transforms the model f by applying the transformation t ( f ) = f ′ , and then executes the attack s on the surrogate model f ′ . Note that the surrogate model is used only to compute the candidate adversarial example , not to evaluate it . That is , we generate an adversarial example x′ for f ′ , and then check whether it is also adversarial for f . Since x′ may be adversarial for f ′ , but not for f , the adaptive attack must maximize the transferability of the generated candidate adversarial samples . Attack Algorithm & Parameters ( S ) The attack search space consists of a sequence of adversarial attacks . We formalize the search space with the grammar : ( Attack Search Space ) S : := S ; S | randomize S | EOT S , n | repeat S , n | try S for n | Attack with params with loss ∈ L • S ; S : composes two attacks , which are executed independently and return the first adversarial sample in the defined order . That is , given input x , the attack s1 ; s2 returns s1 ( x ) if s1 ( x ) is an adversarial example , and otherwise it returns s2 ( x ) . • randomize S : enables the attack ’ s randomized components , which correspond to random seed and/or selecting a starting point within d ( x′ , x ) ≤ , uniformly at random . • EOT S , n : uses expectation over transformation , a technique designed to compute gradients for models with randomized components ( Athalye et al. , 2018 ) . • repeat S , n : repeats the attack n times ( useful only if randomization is enabled ) . • try S for n : executes the attack with a time budget of n seconds . • Attack with params with loss ∈ L : is a backbone attack Attack executed with param- eters params and loss function loss . In our evaluation , we use FGSM ( Goodfellow et al. , 2015 ) , PGD , DeepFool ( Moosavi-Dezfooli et al. , 2016 ) , C & W , NES , APGD , FAB and SQR . We provide full list of the attack parameters , including their ranges and priors in Appendix B . Note , that we include variety of backbone attacks , including those that were already superseeded by stronger attacks . This is done for two key reasons . First , weaker attacks can be surprisingly effective in some cases and avoid the detector because of their weakness ( see defense C24 in our evaluation ) . Second , we are not using any prior when designing the space search . In particular , whenever a new attack is designed it can simply be added to the search space . Then , the goal of the search algorithm is to be powerful enough to perform the search efficiently . In other words , the aim is to avoid making any assumptions of what is useful or not and let the search algorithm learn this instead . Network Transformations ( T ) A common approach that aims to improve the robustness of neural networks against adversarial attacks is to incorporate an explicit defense mechanism in the neural architecture . These defenses often obfuscate gradients to render iterative-optimization methods ineffective ( Athalye et al. , 2018 ) . However , these defenses can be successfully circumvented by ( i ) choosing a suitable attack algorithm , such as score and decision-based attacks ( included in S ) , or ( ii ) by changing the neural architecture ( defined next ) . At a high level , the network transformation search space T takes as input a model f and transforms it to another model f ′ , which is easier to attack . To achieve this , the network f can be expressed as a directed acyclic graph , including both the forward and backward computations , where each vertex denotes an operator ( e.g. , convolution , residual blocks , etc . ) , and edges correspond to data dependencies . In our work , we include two types of network transformations : Layer Removal , which removes an operator from the graph . Each operator can be removed if its input and output dimensions are the same , regardless of its functionality . Backward Pass Differentiable Approximation ( BPDA ) ( Athalye et al. , 2018 ) , which replaces the backward version of an operator with a differentiable approximation of the function . In our search space we include three different function approximations : ( i ) an identity function , ( ii ) a convolution layer with kernel size 1 , and ( iii ) a two-layer convolutional layer with ReLU activation in between . The weights in the latter two cases are learned through approximating the forward function . Loss Function ( L ) Selecting the right objective function to optimize is an important design decision for creating strong adaptive attacks . Indeed , the recent work of Tramer et al . ( 2020 ) uses 9 different objective functions to break 13 defenses , showing the importance of this step . We formalize the space of possible loss functions using the following grammar : ( Loss Function Search Space ) L : := targeted Loss , n with Z | untargeted Loss with Z | targeted Loss , n - untargeted Loss with Z Z : := logits | probs Loss : := CrossEntropy | HingeLoss | L1 | DLR | LogitMatching Targeted vs Untargeted . The loss can be either untargeted , where the goal is to change the classification to any other label f ( x′ ) 6= f ( x ) , or targeted , where the goal is to predict a concrete label f ( x′ ) = l. Even though the untargeted loss is less restrictive , it is not always easier to optimize in practice , and replacing it with a targeted attack might perform better . When using targeted Loss , n , the attack will consider the top n classes with the highest probability as the targets . Loss Formulation . The concrete loss formulation includes loss functions used in existing adaptive attacks , as well as the recently proposed difference in logit ratio loss ( Croce & Hein , 2020b ) . We provide a formal definition of the loss functions used in our work in Appendix B. Logits vs. Probabilities . In our search space , loss functions can be instantiated both with logits as well as with probabilities . Note that some loss functions are specifically designed for one of the two options , such as C & W ( Carlini & Wagner , 2017 ) or DLR ( Croce & Hein , 2020b ) . While such knowledge can be used to reduce the search space , it is not necessary as long as the search algorithm is powerful enough to recognize that such a combination leads to poor results . Loss Replacement . Because the key idea behind many of the defenses is finding a property that helps differentiate between adversarial and natural images , one can also define the optimization objective in the same way . These feature-level attacks ( Sabour et al. , 2016 ) avoid the need to directly optimize the complex objective defined by the adversarial defense and have been effective at circumventing them . As an example , the logit matching loss minimizes the difference of logits between adversarial sample x′ and a natural sample of the target class x ( selected at random from the dataset ) . Instead of logits , the same idea can also be applied to other statistics , such as internal representations computed by a pre-trained model or KL-divergence between label probabilities .

Summary: The paper advances state-of-the-art automated pen-testing tools for breaking defenses designed to protect deep neural networks against norm-based adversarial attacks as a search problem. The search process first considers techniques to simplify the network being attacked (called Network Transformations), followed by a search over the space of existing attacks (and common ways to modify these attacks). Given the search space is large, the authors try to improve the efficiency by leveraging sub-sampling methods.