Automated Discovery of Adaptive Attacks on Adversarial Defenses

1 Introduction . The issue of adversarial attacks ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) , i.e. , crafting small input perturbations that lead to mispredictions , is an important problem with a large body of recent work . Unfortunately , reliable evaluation of proposed defenses is an elusive and challenging task : many defenses seem to initially be effective , only to be circumvented later by new attacks designed specifically with that defense in mind ( Carlini & Wagner , 2017 ; Athalye et al. , 2018 ; Tramer et al. , 2020 ) . To address this challenge , two recent works approach the problem from different perspectives . Tramer et al . ( 2020 ) outlines an approach for manually crafting adaptive attacks that exploit the weak points of each defense . Here , a domain expert starts with an existing attack , such as PGD ( Madry et al. , 2018 ) ( denoted as • in Figure 1 ) , and adapts it based on knowledge of the defense ’ s inner workings . Common modifications include : ( i ) tuning attack parameters ( e.g. , number of steps ) , ( ii ) replacing network components to simplify the attack ( e.g. , removing randomization or non-differentiable components ) , and ( iii ) replacing the loss function optimized by the attack . This approach was demonstrated to be effective in breaking all of the considered 13 defenses . However , a downside is that it requires substantial manual effort and is limited by the domain knowledge of the expert – for instance , each of the 13 defenses came with an adaptive attack which was insufficient , in retrospect . At the same time , Croce & Hein ( 2020b ) proposed to assess adversarial robustness using an ensemble of four attacks illustrated in Figure 1 ( b ) – APGDCE with cross-entropy loss ( Croce & Hein , 2020b ) , APGDDLR with difference in logit ratio loss , FAB ( Croce & Hein , 2020a ) , and SQR ( Andriushchenko et al. , 2020 ) . While these do not require manual effort and have been shown to improve the robustness 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( b ) Ensemble of fixed attacks Croce & Hein ( 2020b ) ( a ) Handcrafted adaptive attacks Tramer et al . ( 2020 ) ( c ) Adaptive attack search ( Our Work ) estimate for many defenses , the approach is inherently limited by the fact that the attacks are fixed apriori without any knowledge of the given defense at hand . This is visualized in Figure 1 ( b ) where even though the attacks are designed to be diverse , they cover only a small part of the entire space . This work : towards automated discovery of adaptive attacks We present a new method that helps automating the process of crafting adaptive attacks , combining the best of both prior approaches ; the ability to evaluate defenses automatically while producing attacks tuned for the given defense . Our work is based on the key observation that we can identify common techniques used to build existing adaptive attacks and extract them as reusable building blocks in a common framework . Then , given a new model with an unseen defense , we can discover an effective attack by searching over suitable combinations of these building blocks . To identify reusable techniques , we analyze existing adaptive attacks and organize their components into three groups : • Attack algorithm and parameters : a library of diverse attack techniques ( e.g. , APGD , FAB , C & W ( Carlini & Wagner , 2017 ) , NES ( Wierstra et al. , 2008 ) ) , together with backbone specific and generic parameters ( e.g. , input randomization , number of steps , if/how to use EOT ( Athalye et al. , 2018 ) ) . • Network transformations : producing an easier to attack surrogate model using techniques including variants of BPDA ( Athalye et al. , 2018 ) to break gradient obfuscation , and layer removal ( Tramer et al. , 2020 ) to eliminate obfuscation layers such as redundant softmax operator . • Loss functions : that specify different ways of defining the attack ’ s loss function . These components collectively formalize an attack search space induced by their different combinations . We also present an algorithm that effectively navigates the search space so to discover an attack . In this way , domain experts are left with the creative task of designing new attacks and growing the framework by adding missing attack components , while the tool is responsible for automating many of the tedious and time-consuming trial-and-error steps that domain experts perform manually today . That is , we can automate some part of the process of finding adaptive attacks , but not necessarily the full process . This is natural as finding truly new attacks is a highly creative process that is currently out of reach for fully automated techniques . We implemented our approach in a tool called Adaptive AutoAttack ( A3 ) and evaluated it on 24 diverse adversarial defenses . Our results demonstrate that A3 discovers adaptive attacks that outperform AutoAttack ( Croce & Hein , 2020b ) , the current state-of-the-art tool for reliable evaluation of adversarial defenses : A3 finds attacks that are significantly stronger , producing 3.0 % -50.8 % additional adversarial examples for 10 models , while obtaining attacks with stronger or simialr strength for the remaining models . Our tool A3 and scripts for reproducing the experiments are available online at : https : //github.com/eth-sri/adaptive-auto-attack 2 Automated Discovery of Adaptive Attacks . We use D = { ( xi , yi ) } Ni=1 to denote a training dataset where x ∈ X is a natural input ( e.g. , an image ) and y is the corresponding label . An adversarial example is a perturbed input x′ , such that : ( i ) it satisfies an attack criterion c , e.g. , a K-class classification model f : X → RK predicts a wrong label , and ( ii ) the distance d ( x′ , x ) between the adversarial input x′ and the natural input x is below a threshold under a distance metric d ( e.g. , an Lp norm ) . Formally , this can be written as : d ( x′ , x ) ≤ such that c ( f , x′ , x ) capability goal ( criterion ) ADVERSARIAL ATTACK For example , instantiating this with the L∞ norm and misclassification criterion corresponds to : ‖x′ − x‖∞ ≤ s.t . f̂ ( x′ ) 6= f̂ ( x ) MISCLASSIFICATION L∞ ATTACK where f̂ returns the prediction arg maxk=1 : K fk ( · ) of the model f . Problem Statement Given a model f equipped with an unknown set of defenses and a dataset D = { ( xi , yi ) } Ni=1 , our goal is to find an adaptive adversarial attack a ∈ A that is best at generating adversarial samples x′ according to the attack criterion c and the attack capability d ( x′ , x ) ≤ : max a∈A , d ( x′ , x ) ≤ E ( x , y ) ∼D c ( f , x′ , x ) where x′ = a ( x , f ) ( 1 ) Here , A denotes the search space of all possible attacks , where the goal of each attack a : X× ( X→ RK ) → X is to generate an adversarial sample x′ = a ( x , f ) for a given input x and model f . For example , solving this optimization problem with respect to the L∞ misclassification criterion corresponds to optimizing the number of adversarial examples misclassified by the model . In our work , we consider an implementation-knowledge adversary , who has full access to the model ’ s implementation at inference time ( e.g. , the model ’ s computational graph ) . We chose this threat model as it matches our problem setting – given an unseen model implementation , we want to automatically find an adaptive attack that exploits its weak points but without the need of a domain expert . We note that this threat model is weaker than a perfect-knowledge adversary ( Biggio et al. , 2013 ) , which assumes a domain expert that also has knowledge about the training dataset and algorithm , as this information is difficult , or even not possible , to recover from the model ’ s implementation only . Key Challenges To solve the optimization problem from Eq . 1 , we address two key challenges : • Defining a suitable attacks search space A such that it is expressible enough to cover a range of existing adaptive attacks . • Searching over the space A efficiently such that a strong attack is found within a reasonable time . Next , we formalize the attack space in Section 3 and then describe our search algorithm in Section 4 . 3 Adaptive Attacks Search Space . We define the adaptive attack search space by analyzing existing adaptive attacks and identifying common techniques used to break adversarial defenses . Formally , the adaptive attack search space is given byA : S×T , where S consists of sequences of backbone attacks along with their loss functions , selected from a space of loss functions L , and T consists of network transformations . Semantically , given an input x and a model f , the goal of adaptive attack ( s , t ) ∈ S× T is to return an adversarial example x′ by computing s ( x , t ( f ) ) = x′ . That is , it first transforms the model f by applying the transformation t ( f ) = f ′ , and then executes the attack s on the surrogate model f ′ . Note that the surrogate model is used only to compute the candidate adversarial example , not to evaluate it . That is , we generate an adversarial example x′ for f ′ , and then check whether it is also adversarial for f . Since x′ may be adversarial for f ′ , but not for f , the adaptive attack must maximize the transferability of the generated candidate adversarial samples . Attack Algorithm & Parameters ( S ) The attack search space consists of a sequence of adversarial attacks . We formalize the search space with the grammar : ( Attack Search Space ) S : := S ; S | randomize S | EOT S , n | repeat S , n | try S for n | Attack with params with loss ∈ L • S ; S : composes two attacks , which are executed independently and return the first adversarial sample in the defined order . That is , given input x , the attack s1 ; s2 returns s1 ( x ) if s1 ( x ) is an adversarial example , and otherwise it returns s2 ( x ) . • randomize S : enables the attack ’ s randomized components , which correspond to random seed and/or selecting a starting point within d ( x′ , x ) ≤ , uniformly at random . • EOT S , n : uses expectation over transformation , a technique designed to compute gradients for models with randomized components ( Athalye et al. , 2018 ) . • repeat S , n : repeats the attack n times ( useful only if randomization is enabled ) . • try S for n : executes the attack with a time budget of n seconds . • Attack with params with loss ∈ L : is a backbone attack Attack executed with param- eters params and loss function loss . In our evaluation , we use FGSM ( Goodfellow et al. , 2015 ) , PGD , DeepFool ( Moosavi-Dezfooli et al. , 2016 ) , C & W , NES , APGD , FAB and SQR . We provide full list of the attack parameters , including their ranges and priors in Appendix B . Note , that we include variety of backbone attacks , including those that were already superseeded by stronger attacks . This is done for two key reasons . First , weaker attacks can be surprisingly effective in some cases and avoid the detector because of their weakness ( see defense C24 in our evaluation ) . Second , we are not using any prior when designing the space search . In particular , whenever a new attack is designed it can simply be added to the search space . Then , the goal of the search algorithm is to be powerful enough to perform the search efficiently . In other words , the aim is to avoid making any assumptions of what is useful or not and let the search algorithm learn this instead . Network Transformations ( T ) A common approach that aims to improve the robustness of neural networks against adversarial attacks is to incorporate an explicit defense mechanism in the neural architecture . These defenses often obfuscate gradients to render iterative-optimization methods ineffective ( Athalye et al. , 2018 ) . However , these defenses can be successfully circumvented by ( i ) choosing a suitable attack algorithm , such as score and decision-based attacks ( included in S ) , or ( ii ) by changing the neural architecture ( defined next ) . At a high level , the network transformation search space T takes as input a model f and transforms it to another model f ′ , which is easier to attack . To achieve this , the network f can be expressed as a directed acyclic graph , including both the forward and backward computations , where each vertex denotes an operator ( e.g. , convolution , residual blocks , etc . ) , and edges correspond to data dependencies . In our work , we include two types of network transformations : Layer Removal , which removes an operator from the graph . Each operator can be removed if its input and output dimensions are the same , regardless of its functionality . Backward Pass Differentiable Approximation ( BPDA ) ( Athalye et al. , 2018 ) , which replaces the backward version of an operator with a differentiable approximation of the function . In our search space we include three different function approximations : ( i ) an identity function , ( ii ) a convolution layer with kernel size 1 , and ( iii ) a two-layer convolutional layer with ReLU activation in between . The weights in the latter two cases are learned through approximating the forward function . Loss Function ( L ) Selecting the right objective function to optimize is an important design decision for creating strong adaptive attacks . Indeed , the recent work of Tramer et al . ( 2020 ) uses 9 different objective functions to break 13 defenses , showing the importance of this step . We formalize the space of possible loss functions using the following grammar : ( Loss Function Search Space ) L : := targeted Loss , n with Z | untargeted Loss with Z | targeted Loss , n - untargeted Loss with Z Z : := logits | probs Loss : := CrossEntropy | HingeLoss | L1 | DLR | LogitMatching Targeted vs Untargeted . The loss can be either untargeted , where the goal is to change the classification to any other label f ( x′ ) 6= f ( x ) , or targeted , where the goal is to predict a concrete label f ( x′ ) = l. Even though the untargeted loss is less restrictive , it is not always easier to optimize in practice , and replacing it with a targeted attack might perform better . When using targeted Loss , n , the attack will consider the top n classes with the highest probability as the targets . Loss Formulation . The concrete loss formulation includes loss functions used in existing adaptive attacks , as well as the recently proposed difference in logit ratio loss ( Croce & Hein , 2020b ) . We provide a formal definition of the loss functions used in our work in Appendix B. Logits vs. Probabilities . In our search space , loss functions can be instantiated both with logits as well as with probabilities . Note that some loss functions are specifically designed for one of the two options , such as C & W ( Carlini & Wagner , 2017 ) or DLR ( Croce & Hein , 2020b ) . While such knowledge can be used to reduce the search space , it is not necessary as long as the search algorithm is powerful enough to recognize that such a combination leads to poor results . Loss Replacement . Because the key idea behind many of the defenses is finding a property that helps differentiate between adversarial and natural images , one can also define the optimization objective in the same way . These feature-level attacks ( Sabour et al. , 2016 ) avoid the need to directly optimize the complex objective defined by the adversarial defense and have been effective at circumventing them . As an example , the logit matching loss minimizes the difference of logits between adversarial sample x′ and a natural sample of the target class x ( selected at random from the dataset ) . Instead of logits , the same idea can also be applied to other statistics , such as internal representations computed by a pre-trained model or KL-divergence between label probabilities .

Summary: One major challenge in the research on adversarial attacks is the absence of methods that can reliably evaluate an adversarial defense. One the one hand, many defense approaches end up targetting particular attack methods. On the other hand, some promising defense approaches are viewed skeptically due to possible new attacks that could break them even if none currently exists. This paper addresses this critical challenge of developing adaptive adversarial attacks by combining some basic building blocks based on a particular defense. The presented approach is shown to outperform AutoAttack - a popularly used baseline attack by community developing adversarial defenses.   The submission presents an automated approach to the discovery of adversarial attacks (along the lines of AutoAttack) by exploring the search space of adversarial attacks formed by varying three components: (i) parameterized attack algorithms, (ii) network transformations, and (iii) loss functions.   Overall, the paper is an interesting improvement over the state-of-the-art and clarification to some questions below will help the reviewer better appreciate its overall contribution.