Imperceptible Black-box Attack via Refining in Salient Region

Deep neural networks are vulnerable to adversarial examples , even in the blackbox setting where the attacker only has query access to the model output . Recent studies have devised successful black-box attacks with high query efficiency . However , such performance often comes at the cost of the imperceptibility of adversarial attacks , which is essential for attackers . To address this issue , in this paper we propose to use segmentation priors for black-box attacks such that the perturbations are limited in the salient region . We find that state-of-the-art blackbox attacks equipped with segmentation priors can achieve much better imperceptibility performance with little reduction in query efficiency and success rate . We further propose the Saliency Attack , a new gradient-free black-box attack that can further improve the imperceptibility by refining perturbations in the salient region . Experimental results show that the perturbations generated by our approach are much more imperceptible than the ones generated by other attacks , and are interpretable to some extent . Furthermore , our approach is found to be more robust to detection-based defense , which demonstrates its efficacy as well . 1 INTRODUCTION . Deep neural networks ( DNNs ) have achieved significant progress in wide applications , such as image classification ( Deng et al. , 2009 ) , face recognition ( Parkhi et al. , 2015 ) , object detection ( Redmon et al. , 2016 ) , speech recognition ( 4 , 2012 ) and machine translation ( Bahdanau et al. , 2015 ) . Despite their success , deep learning models have revealed vulnerability to adversarial attacks ( Szegedy et al. , 2014 ) . Crafted by adding some small perturbations to benign inputs , adversarial examples ( AEs ) can fool DNNs into making wrong predictions , which is a critical threat especially for some security-sensitive scenarios such as autonomous driving ( Sun et al. , 2020 ) . Based on the knowledge of target model , the adversarial attacks could be divided into white-box attack and black-box attack . White-box attacks ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Papernot et al. , 2016 ; Carlini & Wagner , 2017 ; Moosavi-Dezfooli et al. , 2016 ) have full access to the architecture and parameters of the target model , and can easily generate a successful adversarial example via back propagation . By contrast , black-box attacks ( Narodytska & Kasiviswanathan , 2016 ; Ilyas et al. , 2018 ; 2019 ; Moon et al. , 2019 ; Andriushchenko et al. , 2020 ; Li et al. , 2021 ; Papernot et al. , 2017 ; Liu et al. , 2017 ) can only query the target model to obtain the output prediction , which is a more realistic and challenging setting . Since many real-world online application programming interfaces ( APIs ) have time or monetary limit for user query ( Ilyas et al. , 2018 ) , most current research efforts focus on how to design query-efficient attacks ( Ilyas et al. , 2019 ; Moon et al. , 2019 ; Andriushchenko et al. , 2020 ) , which indeed achieve a huge improvement . For example , so far the state-of-the-art black-box attack ( Andriushchenko et al. , 2020 ) could succeed in untargeted attack on ImageNet dataset ( Deng et al. , 2009 ) with only tens of queries on average . However , this success has sacrificed the imperceptibility of AEs , whose global perturbations are generated with various tricks like random vertical stripes ( Andriushchenko et al. , 2020 ) , and thus very obvious and perceptible to human eyes ( see Figure 1 ( e ) and ( f ) for examples ) . In effect , imperceptibility is essential for attackers . Currently online APIs usually integrate detectors into their services to detect anomalous inputs ( Li et al. , 2021 ) . The images with too visible perturbations are difficult to pass these detectors , not to mention human judgement . To alleviate or even address this problem , we could utilize some priors to lead black-box attacks to search in small but efficient space , instead of global perturbations . Previous DNNs visualization work ( Simonyan et al. , 2014 ) proposed a type of saliency map to indicate which features should influence the output significantly . This BP saliency map 1 is generated by calculating the derivatives of the model output with respect to input . Just as show in Figure 1 . ( a ) , the brighter pixels in BP saliency maps denote they have a greater impact on the model output . The classical white-box attack JSMA ( Papernot et al. , 2016 ) just constructs such an adversarial saliency map and iteratively selects pixels from this map to perturb . But due to the black-box setting , we can not derive this saliency map directly . Nevertheless , we could find that the region of bright pixels roughly represents the position of the main object in an image , which also accords with our intuition . Inspired by this , we propose to first generate a saliency mask to extract the regions of the main object in an image via image segmentation , and then limit the perturbation search in this smaller but more important regions , as presented in Figure 1 . Several recent studies also propose to generate similar local perturbations but applied in different attack settings with different methods ( Dong et al. , 2020 ; Xiang et al. , 2021 ) . In particular , Dong et al . ( 2020 ) produce saliency maps with the class activation mapping ( CAM ) ( Zhou et al. , 2016 ) , which requires the internal information of the model and needs to change the model architecture . This is infeasible in black-box scenarios . Xiang et al . ( 2021 ) replace CAM with Grad-CAM ( Selvaraju et al. , 2017 ) , which does not need to change the architecture of the model . However , both CAM and Grad-CAM still rely on the knowledge of the model , but in black-box setting the internal information of the target model is inaccessible . Thus , we choose salient object segmentation ( Zhao & Wu , 2019 ) , which can automatically extract salient object ( s ) in an image ( see Figure 1. ( b ) ) . Compared with CAM and Grad-CAM , it doesn ’ t need any other information except for the input image . On the other hand , these studies ( Dong et al. , 2020 ; Xiang et al. , 2021 ) are applied to transfer-based attacks ( actually a grey-box setting ) , which first build a substitute model to approximate the target model , then use gradient-based white-box attacks to generate adversarial examples , and finally transfer them to attack the target model . For transfer-based attack , we must either train a substitute model from scratch with huge number of queries ( Papernot et al. , 2017 ) or assume a pretrained substitute model is trained on the similar data distribution with the target model ’ s ( Liu et al. , 2017 ) . And for the latter , the transferability of both adversarial examples and saliency maps highly depends on the selection of the pair of substitute model ( s ) and target model . Hence , in this paper we focus on gradient-free black-box attacks ( Narodytska & Kasiviswanathan , 2016 ; Moon et al. , 2019 ; Andriushchenko et al. , 2020 ) that take no account of gradients and search for successful perturbations only according to the model output . We add these segmentation priors to SOTA black-box attacks and find the imperceptibility of AEs is indeed enhanced but still limited due to ” global perturbation ” in local region ( See Figure 4 and 5 ) . To 1To distinguish the saliency map generated by the salient object segmentation model below , here we call it as BP saliency map to indicate it is generated via back propagation . further improve the imperceptibility , our another novelty is that even in the salient region , “ saliency ” could still be refined . Our inspiration also comes from some DNNs visualization works ( Zhou et al. , 2016 ; Zeiler & Fergus , 2014 ; Olah et al. , 2018 ) . For instance , given a dog image , CAM produces a localization heatmap and shows the dog face region is most highly lighted ( Zhou et al. , 2016 ) . Zeiler & Fergus ( 2014 ) systematically cover up different portions of a dog image , and finds when the dog face is obscured , the activity in the feature map and classifier output changes dramatically . In further , by combining internal feature visualization with output prediction , Olah et al . ( 2018 ) reveal even in dog ’ s face region , its ears and eyes seem to be more important when distinguishing dogs . Therefore , we are inspired to assume the salient region in an image is progressive with respect to its impact on model output . If we could find smaller but more salient region , the perturbation will be more efficient and meanwhile the imperceptibility of AEs could also be enhanced . Thus , we propose our Saliency Attack , a new gradient-free black-box attack via refining perturbations in the salient region according to their saliency . Our main contributions can be summarized as follows : • To our best knowledge , we are the first to use salient object segmentation to extract binary salient masks in black-box settings . Experiments show that the SOTA black-box attacks limited in such regions can achieve much better imperceptibility performance with little reduction in query efficiency and success rate . • We propose a new gradient-free black-box attack via refining in salient regions . Compared with the search methods used in other gradient-free attacks , our method is able to generate smaller but effective perturbations , which is interpretable to some extent and can further improve the imperceptibility . • We demonstrate that the perturbations generated by our Saliency Attack is more robust against some detection-based defense like Feature Squeezing . 2 RELATED WORK . 2.1 BLACK-BOX ATTACKS . Gradient estimation attacks . Gradient estimation attacks first estimate the gradients by querying the target model and then apply them to run white-box attacks ( Goodfellow et al. , 2015 ; Carlini & Wagner , 2017 ; Madry et al. , 2018 ) . ZOO attack ( Chen et al. , 2017 ) first adopts the symmetric difference quotient to approximate the gradients and then perform Carlini-Wagner ( CW ) white-box attack ( Carlini & Wagner , 2017 ) . AutoZOOM ( Tu et al. , 2019 ) uses a random vector based gradient estimation to estimate gradients , which reduces the query number per iteration from 2D in ZOO to N+1 ( D is the dimensionality and N is the sample size ) . To further enhance query efficiency , Ilyas et al . ( 2019 ) propose the “ tiling trick ” that updates a square of pixels simultaneously instead of a single pixel , which dramatically decreases the dimensionality by a factor of k2 ( k is tile length ) . Gradient-free attacks . Gradient-free attacks take no account of gradients and directly generate AEs with random search or heuristic methods according to the query result . Su et al . ( 2019 ) propose one pixel attack that adopts differential evolution algorithm to perturb the most important pixel in the image . Alzantot et al . ( 2019 ) propose GenAttack , which use genetic algorithm to generate AEs . To improve the query efficiency , Moon et al . ( 2019 ) consider a discrete surrogate optimization problem that transforms the original constraint of a continuous range [ − , + ] to a discrete set { − , + } , achieving a huge reduction in the search space . This is motivated by linear program ( LP ) where the optimal solution is attained at an extreme point of the feasible set ( Schrijver , 1998 ) . Combing tiling trick ( Ilyas et al. , 2019 ) and discrete optimization ( Moon et al. , 2019 ) , Square Attack ( Andriushchenko et al. , 2020 ) has obtained the best result on success rate and query performance so far with a randomized search scheme . Hybrid attacks . Hybrid attacks try to combine multiple types of methods , using candidate adversarial examples generated on substitute model as starting points for further optimization with gradient estimation attacks or gradient-free attacks . Suya et al . ( 2020 ) first proposed to generate candidate AEs by white-box projected gradient descent ( PGD ) attack ( Madry et al. , 2018 ) , and further optimize them with gradient estimation methods . Wang et al . ( 2020 ) combines a white-box attack with a gradient-free method using microbial genetic algorithm ( Harvey , 2009 ) . Similar with transfer-based attacks , hybrid attacks also suffer from the similarity between substitute model ( s ) and target model .

Summary: This paper proposes a black-box attack where, by relying on segmentation priors, the perturbation is applied only in the salient region. This allows one to obtain reduce perceptibility with a limited number of queries and a small reduction in success rate. More specifically, once the salient region has been identified, a refining procedure is carried out to find small areas where the perturbation should be added. Experiments are performed on ImageNet and results are compared with those of some SOTA methods and their variants that work on saliency regions.