QUERY-EFFICIENT DECISION-BASED SPARSE ATTACKS AGAINST BLACK-BOX MACHINE LEARNING MODELS

1 INTRODUCTION . In spite of the impressive performance achieved from deep neural network ( DNN ) models on a variety of vision tasks , a flurry of research on adversarial attacks over the last few years have demonstrated the vulnerability of deep learning models to tiny , maliciously crafted perturbations applied to their inputs ( Szegedy et al. , 2014 ) . These malicious perturbations , although imperceptible to humans , are able to evade and mislead DNNs . Thus , embedding DNNS in systems creates a new attack surface as well as the incentive for malevolent actors to strike systems such as autonomous cars or machine learning models as a service ( MLaaS ) employed in real-world applications such as self-driving cars ( Chen et al. , 2015 ) , Google Cloud Vision or Amazon Rekognition . In a black-box setting , an adversary may access all or only the top-1 predicted label and score—a score-based setting ( Chen et al. , 2017 ) —or simply the predicted label for a given input—a decisionbased ( Brendel et al. , 2018 ) setting . Importantly , the similarity measure , used to quantify the imperceptibility of the perturbation , can describe an attack as a dense attack—l2 , l∞ norm constrained adversarial attack—or a sparse attack—l0 norm constrained adversarial attack . Significantly , score-based and decision-based settings present a practical threat model for deployed systems ; the latter being particularly more threatening to model owners and applications . Because , an adversary is still capable of exploiting the very minimal information exposed—the top-1 predicted label—for constructing an perturbation . Importantly , while dense attacks ( Athalye et al. , 2018 ; Shukla et al. , 2021 ; Ilyas et al. , 2018 ) are widely explored , sparse attacks have not drawn much attention . This potentially leads to a lack of knowledge on model vulnerabilities to this perturbation regime . From a security standpoint , sparse attacks are particularly as threatening as dense attacks . Therefore , investigating sparse perturbation regimes is as pivotal and necessary as dense perturbation counterparts ; in this study , we spend our efforts to extensively investigate the robustness of DNNs against sparse attacks . Scope of the Study–Vision Transformers and Convolutional Networks . Attention-based architectures introduced by Cordonnier et al . ( 2020 ) ; Ramachandran et al . ( 2019 ) ; Touvron et al . ( 2021 ) , particularly the Vision Transformer ( ViT ) model proposed by Dosovitskiy et al . ( 2021 ) , can be competitive or even outperform convolution-based architectures ( Bhojanapalli et al. , 2021 ; Carion et al. , 2020 ) . Existing studies have not considered adversarial attacks in l0 norm constraint based perturbation regimes aginst ViT , although a few studies have explored robustness against l2 and l∞ norm constraints ( Shao et al. , 2021 ) . This raises a critical security concerns for reliable deployment of real-world applications based on vision transformers . Therefore , our efforts will focus on investigating a method capable of evaluating the robustness of convolutional DNNs as well as transformer networks to understand the fragility of ViT in relation to CNNs under l0 norm adversarial attacks . An NP-Hard Problem . Yielding sparse perturbations is incredibly difficult as minimizing l0 norm leads to an NP-hard problem ( Modas & Moosavi-Dezfooli , 2019 ; Dong et al. , 2020 ) . Existing sparse attacks in black-box settings , particularly in decision-based scenarios , have a key shortcoming—the algorithms require a large number of model queries to achieve sparsity and invisibility . Consequently , we propose a novel evolutionary algorithm based sparse attack method in the decision-based setting , we refer to as SparseEvo . The method is significantly more query efficient than the state-ofthe-art counterpart—Pointwise ( Schott et al. , 2019 ) . We illustrate an example targeted attack with our proposed algorithm in Fig . 1 on the standard computer vision task , ImageNet . Need for Query Efficiency . In decision-based or black-box settings , achieving query efficiency with high attack success rate a crucial to adversarial objective . Because : i ) adversaries are able to carry out attacks at scale ; ii ) the cost of mounting the attack is reduced ; and iii ) adversaries are capable of bypassing a system that can employ methods to recognize malicious activities as a fraud based on pragmatically large number of successive queries with analogous inputs and thwart their attacks . Further , from a defense perspective , the lower number of queries significantly reduces the evaluation time of both trained models and defense mechanisms . Therefore , query efficient attack algorithms facilitates research in designing new defenses , model architectures as well benefit Machine Learning as a Service ( MLaaS ) providers by enabling the evaluation of their models prior deployment at scale . We summarize our contributions and results below : • We formulate a novel sparse attack—SparseEvo—an evolution-based algorithm capable of exploiting access to solely the top-1 predicted label from a model to search for an adversarial example in the model ’ s input space whilst minimizing the number of perturbed pixels required to mislead the model . • Our attack algorithm can significantly reduce the number of model queries compared with the sate-of-the-art counterpart , Pointwise . Further , SparseEvo achieves comparable success to PGD0—the state-of-the-art white-box attack—in terms of attack success rate with a limited query budget . • We conduct the first vulnerability evaluation of a Vision Transformer ( ViT ) on the standard computer vision task ImageNet in a decision-based and l0 norm constrained setting . We compare results with ResNet to assess the relative robustness of the ViT model . 2 BACKGROUND AND RELATED WORK ON SPARSE ATTACKS . Adversarial Attack Primer . Three different criteria can be used to categorize adversarial attacks ( Chen et al. , 2020 ) ; the attack goal , similarity measure used to quantify the imperceptibility of the perturbation and the level of access to information ( threat model ) . In terms of the attack goal , an adversarial attack is either untargeted ( an input is simply misclassified ) or targeted . Meanwhile , adversarial attacks can be classified into two sub-categories : white-box or black-box according to the threat model . In the white-box setting , an adversary has full knowledge and access to the machine learning model ( Goodfellow et al. , 2014 ; Madry et al. , 2018 ; Xu et al. , 2019 ; Carlini & Wagner , 2017 ) whereas in the black-box setting , solely the outputs of a model are exposed or accessible to the adversary . In the black-box context , an adversary can access all or top-1 predicted score—a score-based setting ( Suya et al. , 2020 ; Chen et al. , 2017 ; Guo et al. , 2019 ) —or simply the predicted labels of a given input—a decision-based ( Brendel et al. , 2018 ; Cheng et al. , 2020 ) setting . Imperceptibility , based on a similarity measure , can describe an attack as a dense attack—l2 , l∞ norm constrained adversarial attacks—or a sparse attack—l0 norm constrained adversarial attacks . Sparse Attacks . The main aim of sparse attacks is to minimize the number of perturbed pixels required to mislead a target machine learning model . Only a handful of works have investigated sparse attacks and these works can be broadly categorised based on various degrees of adversarial access to a model . White-box methods . To realize sparse attacks in a white-box setting , SparseFool attack introduced by Modas & Moosavi-Dezfooli ( 2019 ) employed the idea of l1 relaxation from ( Andrei & Ion , 2015 ) and exploited low mean curvature of decision boundaries for l0 minimization . JSMA ( Papernot et al. , 2017 ) constructed a saliency map for an input to search for high impact pixels on the model ’ s decision . Recently , Croce & Hein ( 2019 ) introduced PGD0 that projects the adversarial perturbation yielded by PGD ( Madry et al. , 2018 ) to the l0 ball . This attack method is capable of generating significantly lower l0 perturbation and was shown to outperform other white-box algorithms . Therefore , we use the PGD0 algorithm as an ideal case baseline to compare the success achievable in a black-box setting . Score-based methods . ( Su et al. , 2019 ) proposed the One-Pixel attack based on a differential evolutionary algorithm . Although the One-Pixel method is capable of searching and obtaining the most sparse perturbation , its attack success rate ( ASR ) on large neural networks and high resolution images is relatively low . Importantly , the method requires significant number of queries because it modifies one pixel at a time while the input search space , dependent on image resolution , can be enormous . Score-based methods exploit information exposed from a change in confident score to alter a pixel-subset in an input image ; a model owner may prevent this leakage by only exposing the top-1 predicted label to a model query . Decision-based methods . In the decision-based setting , only the top-1 predicted label of a DNN model is exposed to adversaries . Now , perturbing an input image slightly will not expose subtle changes in the output corresponding to the perturbation ; since only the predicted class label is revealed . Therefore , a decision-based attack is the most restrictive and challenging scenario . Most existing decision-based attack algorithms are dense attacks ( the objective is to minimise L2 or L∞ distortion ) . Interestingly , these methods , including BA ( Brendel et al. , 2018 ) , HSJA ( Chen et al. , 2020 ) , QEBA ( Li et al. , 2020 ) , NLBA ( Li et al. , 2021 ) , PSBA ( Zhang et al. , 2021 ) , Sign-OPT Cheng et al . ( 2020 ) or the covariance matrix adaptation evolution strategy ( CMA-ES ) based method for face recognition tasks in ( Dong et al. , 2019 ) , can be adapted to a sparse attack setting by a projection to L0-ball ; however this is not effective , as we show later in Appendix A.7 . Although CMA-ES ( Dong et al. , 2019 ) is an evolutionary algorithm , albeit for a dense attack , the formulation requires individuals of a population to be real number vectors that can be sampled from a Gaussian distribution . Thus , CMA-ES is well suited to the problem of dense attacks . In contrast , the optimization problem in a sparse attack ( L0 constrained ) aims to minimize the number of perturbed pixels . Importantly , the discrete search space encountered in a sparse attack hinders the adoption of these dense attack algorithms to search for a sparse adversarial example , efficiently . To the best of our knowledge , the recent attack—Pointwise ( Schott et al. , 2019 ) —applying a greedy search method to find sparse adversarial perturbations is the first decision-based sparse method . This method is effective in untargeted settings and on low resolution datasets , but it is seen to require prohibitively large number of queries to achieve low sparse adversarial perturbations on large scale datasets and in a targeted attack setting ( as seen in Section 4 ) . In summary , the current blackbox , sparse adversarial attack approaches still have shortcomings on sparsity and query efficiency . Developing decision-based sparse attacks poses a challenging optimization problem because of : i ) limited access to only the decision of a target model ; and ii ) the NP-hard problem of l0 norm constrained optimization . 3 PROPOSED METHOD . 3.1 PROBLEM FORMULATION . In our sparse attack setting , we are given a normalized source image x ∈ [ 0 , 1 ] C×W×H and its corresponding ground truth label y from the label set Y = { 1 , 2 , · · · , K } where K denotes the number of classes , C , W and H denotes the number of channels , width and height of an image , respectively . The classifier that we aim to attack is f : RC×W×H → Y ; our access is limited to its output label . In a targeted setting , x is perturbed such that the instance x̃ ∈ RC×W×H obtained is misclassified to a desired class label ỹ ∈ Y selected by the adversary . We refer to the desired class of the input x as the target class and its ground-truth class as the source class . In an untargeted setting , the adversary manipulates input x to change the decision of the classifier to any class label other than its ground-truth , i.e . ỹ ∈ Y where ỹ 6= y . Formally , a sparse adversarial attack ( either targetted or untargetted ) to find the best adversarial instance x∗ can be formulated as a constrained optimization problem : x∗ = arg min x̃ ‖x− x̃‖0 s.t . f ( x∗ ) = ỹ . ( 1 ) where ‖‖0 is the ` 0 norm denoting the number of perturbed pixels . The optimization problem in equation 1 aiming to minimize the number of perturbed pixels leads to an NP-hard problem ( Modas & Moosavi-Dezfooli , 2019 ; Dong et al. , 2020 ) . Thus , the solution to the optimisation problem is non-trivial given the constraint and the fact that f is not differentiable in this setting .

Summary: This paper proposes a black-box decision-based spare attack based on the evolution algorithm (called SparseEvo).  The authors test their method on two types of classification models and two popular vision datasets: ResNet (CIFAR10 and ImageNet) and Vision Transformer (ImageNet).  Through the comparison with Pointwise attack (for efficiency and sparsity) and PGD0 (for success rate), SparseEvo achieves good performance in both success rate and efficiency.