QUERY-EFFICIENT DECISION-BASED SPARSE ATTACKS AGAINST BLACK-BOX MACHINE LEARNING MODELS

1 INTRODUCTION . In spite of the impressive performance achieved from deep neural network ( DNN ) models on a variety of vision tasks , a flurry of research on adversarial attacks over the last few years have demonstrated the vulnerability of deep learning models to tiny , maliciously crafted perturbations applied to their inputs ( Szegedy et al. , 2014 ) . These malicious perturbations , although imperceptible to humans , are able to evade and mislead DNNs . Thus , embedding DNNS in systems creates a new attack surface as well as the incentive for malevolent actors to strike systems such as autonomous cars or machine learning models as a service ( MLaaS ) employed in real-world applications such as self-driving cars ( Chen et al. , 2015 ) , Google Cloud Vision or Amazon Rekognition . In a black-box setting , an adversary may access all or only the top-1 predicted label and score—a score-based setting ( Chen et al. , 2017 ) —or simply the predicted label for a given input—a decisionbased ( Brendel et al. , 2018 ) setting . Importantly , the similarity measure , used to quantify the imperceptibility of the perturbation , can describe an attack as a dense attack—l2 , l∞ norm constrained adversarial attack—or a sparse attack—l0 norm constrained adversarial attack . Significantly , score-based and decision-based settings present a practical threat model for deployed systems ; the latter being particularly more threatening to model owners and applications . Because , an adversary is still capable of exploiting the very minimal information exposed—the top-1 predicted label—for constructing an perturbation . Importantly , while dense attacks ( Athalye et al. , 2018 ; Shukla et al. , 2021 ; Ilyas et al. , 2018 ) are widely explored , sparse attacks have not drawn much attention . This potentially leads to a lack of knowledge on model vulnerabilities to this perturbation regime . From a security standpoint , sparse attacks are particularly as threatening as dense attacks . Therefore , investigating sparse perturbation regimes is as pivotal and necessary as dense perturbation counterparts ; in this study , we spend our efforts to extensively investigate the robustness of DNNs against sparse attacks . Scope of the Study–Vision Transformers and Convolutional Networks . Attention-based architectures introduced by Cordonnier et al . ( 2020 ) ; Ramachandran et al . ( 2019 ) ; Touvron et al . ( 2021 ) , particularly the Vision Transformer ( ViT ) model proposed by Dosovitskiy et al . ( 2021 ) , can be competitive or even outperform convolution-based architectures ( Bhojanapalli et al. , 2021 ; Carion et al. , 2020 ) . Existing studies have not considered adversarial attacks in l0 norm constraint based perturbation regimes aginst ViT , although a few studies have explored robustness against l2 and l∞ norm constraints ( Shao et al. , 2021 ) . This raises a critical security concerns for reliable deployment of real-world applications based on vision transformers . Therefore , our efforts will focus on investigating a method capable of evaluating the robustness of convolutional DNNs as well as transformer networks to understand the fragility of ViT in relation to CNNs under l0 norm adversarial attacks . An NP-Hard Problem . Yielding sparse perturbations is incredibly difficult as minimizing l0 norm leads to an NP-hard problem ( Modas & Moosavi-Dezfooli , 2019 ; Dong et al. , 2020 ) . Existing sparse attacks in black-box settings , particularly in decision-based scenarios , have a key shortcoming—the algorithms require a large number of model queries to achieve sparsity and invisibility . Consequently , we propose a novel evolutionary algorithm based sparse attack method in the decision-based setting , we refer to as SparseEvo . The method is significantly more query efficient than the state-ofthe-art counterpart—Pointwise ( Schott et al. , 2019 ) . We illustrate an example targeted attack with our proposed algorithm in Fig . 1 on the standard computer vision task , ImageNet . Need for Query Efficiency . In decision-based or black-box settings , achieving query efficiency with high attack success rate a crucial to adversarial objective . Because : i ) adversaries are able to carry out attacks at scale ; ii ) the cost of mounting the attack is reduced ; and iii ) adversaries are capable of bypassing a system that can employ methods to recognize malicious activities as a fraud based on pragmatically large number of successive queries with analogous inputs and thwart their attacks . Further , from a defense perspective , the lower number of queries significantly reduces the evaluation time of both trained models and defense mechanisms . Therefore , query efficient attack algorithms facilitates research in designing new defenses , model architectures as well benefit Machine Learning as a Service ( MLaaS ) providers by enabling the evaluation of their models prior deployment at scale . We summarize our contributions and results below : • We formulate a novel sparse attack—SparseEvo—an evolution-based algorithm capable of exploiting access to solely the top-1 predicted label from a model to search for an adversarial example in the model ’ s input space whilst minimizing the number of perturbed pixels required to mislead the model . • Our attack algorithm can significantly reduce the number of model queries compared with the sate-of-the-art counterpart , Pointwise . Further , SparseEvo achieves comparable success to PGD0—the state-of-the-art white-box attack—in terms of attack success rate with a limited query budget . • We conduct the first vulnerability evaluation of a Vision Transformer ( ViT ) on the standard computer vision task ImageNet in a decision-based and l0 norm constrained setting . We compare results with ResNet to assess the relative robustness of the ViT model . 2 BACKGROUND AND RELATED WORK ON SPARSE ATTACKS . Adversarial Attack Primer . Three different criteria can be used to categorize adversarial attacks ( Chen et al. , 2020 ) ; the attack goal , similarity measure used to quantify the imperceptibility of the perturbation and the level of access to information ( threat model ) . In terms of the attack goal , an adversarial attack is either untargeted ( an input is simply misclassified ) or targeted . Meanwhile , adversarial attacks can be classified into two sub-categories : white-box or black-box according to the threat model . In the white-box setting , an adversary has full knowledge and access to the machine learning model ( Goodfellow et al. , 2014 ; Madry et al. , 2018 ; Xu et al. , 2019 ; Carlini & Wagner , 2017 ) whereas in the black-box setting , solely the outputs of a model are exposed or accessible to the adversary . In the black-box context , an adversary can access all or top-1 predicted score—a score-based setting ( Suya et al. , 2020 ; Chen et al. , 2017 ; Guo et al. , 2019 ) —or simply the predicted labels of a given input—a decision-based ( Brendel et al. , 2018 ; Cheng et al. , 2020 ) setting . Imperceptibility , based on a similarity measure , can describe an attack as a dense attack—l2 , l∞ norm constrained adversarial attacks—or a sparse attack—l0 norm constrained adversarial attacks . Sparse Attacks . The main aim of sparse attacks is to minimize the number of perturbed pixels required to mislead a target machine learning model . Only a handful of works have investigated sparse attacks and these works can be broadly categorised based on various degrees of adversarial access to a model . White-box methods . To realize sparse attacks in a white-box setting , SparseFool attack introduced by Modas & Moosavi-Dezfooli ( 2019 ) employed the idea of l1 relaxation from ( Andrei & Ion , 2015 ) and exploited low mean curvature of decision boundaries for l0 minimization . JSMA ( Papernot et al. , 2017 ) constructed a saliency map for an input to search for high impact pixels on the model ’ s decision . Recently , Croce & Hein ( 2019 ) introduced PGD0 that projects the adversarial perturbation yielded by PGD ( Madry et al. , 2018 ) to the l0 ball . This attack method is capable of generating significantly lower l0 perturbation and was shown to outperform other white-box algorithms . Therefore , we use the PGD0 algorithm as an ideal case baseline to compare the success achievable in a black-box setting . Score-based methods . ( Su et al. , 2019 ) proposed the One-Pixel attack based on a differential evolutionary algorithm . Although the One-Pixel method is capable of searching and obtaining the most sparse perturbation , its attack success rate ( ASR ) on large neural networks and high resolution images is relatively low . Importantly , the method requires significant number of queries because it modifies one pixel at a time while the input search space , dependent on image resolution , can be enormous . Score-based methods exploit information exposed from a change in confident score to alter a pixel-subset in an input image ; a model owner may prevent this leakage by only exposing the top-1 predicted label to a model query . Decision-based methods . In the decision-based setting , only the top-1 predicted label of a DNN model is exposed to adversaries . Now , perturbing an input image slightly will not expose subtle changes in the output corresponding to the perturbation ; since only the predicted class label is revealed . Therefore , a decision-based attack is the most restrictive and challenging scenario . Most existing decision-based attack algorithms are dense attacks ( the objective is to minimise L2 or L∞ distortion ) . Interestingly , these methods , including BA ( Brendel et al. , 2018 ) , HSJA ( Chen et al. , 2020 ) , QEBA ( Li et al. , 2020 ) , NLBA ( Li et al. , 2021 ) , PSBA ( Zhang et al. , 2021 ) , Sign-OPT Cheng et al . ( 2020 ) or the covariance matrix adaptation evolution strategy ( CMA-ES ) based method for face recognition tasks in ( Dong et al. , 2019 ) , can be adapted to a sparse attack setting by a projection to L0-ball ; however this is not effective , as we show later in Appendix A.7 . Although CMA-ES ( Dong et al. , 2019 ) is an evolutionary algorithm , albeit for a dense attack , the formulation requires individuals of a population to be real number vectors that can be sampled from a Gaussian distribution . Thus , CMA-ES is well suited to the problem of dense attacks . In contrast , the optimization problem in a sparse attack ( L0 constrained ) aims to minimize the number of perturbed pixels . Importantly , the discrete search space encountered in a sparse attack hinders the adoption of these dense attack algorithms to search for a sparse adversarial example , efficiently . To the best of our knowledge , the recent attack—Pointwise ( Schott et al. , 2019 ) —applying a greedy search method to find sparse adversarial perturbations is the first decision-based sparse method . This method is effective in untargeted settings and on low resolution datasets , but it is seen to require prohibitively large number of queries to achieve low sparse adversarial perturbations on large scale datasets and in a targeted attack setting ( as seen in Section 4 ) . In summary , the current blackbox , sparse adversarial attack approaches still have shortcomings on sparsity and query efficiency . Developing decision-based sparse attacks poses a challenging optimization problem because of : i ) limited access to only the decision of a target model ; and ii ) the NP-hard problem of l0 norm constrained optimization . 3 PROPOSED METHOD . 3.1 PROBLEM FORMULATION . In our sparse attack setting , we are given a normalized source image x ∈ [ 0 , 1 ] C×W×H and its corresponding ground truth label y from the label set Y = { 1 , 2 , · · · , K } where K denotes the number of classes , C , W and H denotes the number of channels , width and height of an image , respectively . The classifier that we aim to attack is f : RC×W×H → Y ; our access is limited to its output label . In a targeted setting , x is perturbed such that the instance x̃ ∈ RC×W×H obtained is misclassified to a desired class label ỹ ∈ Y selected by the adversary . We refer to the desired class of the input x as the target class and its ground-truth class as the source class . In an untargeted setting , the adversary manipulates input x to change the decision of the classifier to any class label other than its ground-truth , i.e . ỹ ∈ Y where ỹ 6= y . Formally , a sparse adversarial attack ( either targetted or untargetted ) to find the best adversarial instance x∗ can be formulated as a constrained optimization problem : x∗ = arg min x̃ ‖x− x̃‖0 s.t . f ( x∗ ) = ỹ . ( 1 ) where ‖‖0 is the ` 0 norm denoting the number of perturbed pixels . The optimization problem in equation 1 aiming to minimize the number of perturbed pixels leads to an NP-hard problem ( Modas & Moosavi-Dezfooli , 2019 ; Dong et al. , 2020 ) . Thus , the solution to the optimisation problem is non-trivial given the constraint and the fact that f is not differentiable in this setting .

Summary: This work proposes a novel sparse attack method called SparseEvo. Based on evolution algorithm, the SparseEvo searchs a sparse adversarial perturbation in limited query budget. It can significantly reduce the queries compared with the SOTA method, i.e., Pointwise. The paper also conduct the first vulnerability evaluation of a ViT on ImageNet in a decision-based and $l_0$ norm constrained setting.