Imperceptible Black-box Attack via Refining in Salient Region

Deep neural networks are vulnerable to adversarial examples , even in the blackbox setting where the attacker only has query access to the model output . Recent studies have devised successful black-box attacks with high query efficiency . However , such performance often comes at the cost of the imperceptibility of adversarial attacks , which is essential for attackers . To address this issue , in this paper we propose to use segmentation priors for black-box attacks such that the perturbations are limited in the salient region . We find that state-of-the-art blackbox attacks equipped with segmentation priors can achieve much better imperceptibility performance with little reduction in query efficiency and success rate . We further propose the Saliency Attack , a new gradient-free black-box attack that can further improve the imperceptibility by refining perturbations in the salient region . Experimental results show that the perturbations generated by our approach are much more imperceptible than the ones generated by other attacks , and are interpretable to some extent . Furthermore , our approach is found to be more robust to detection-based defense , which demonstrates its efficacy as well . 1 INTRODUCTION . Deep neural networks ( DNNs ) have achieved significant progress in wide applications , such as image classification ( Deng et al. , 2009 ) , face recognition ( Parkhi et al. , 2015 ) , object detection ( Redmon et al. , 2016 ) , speech recognition ( 4 , 2012 ) and machine translation ( Bahdanau et al. , 2015 ) . Despite their success , deep learning models have revealed vulnerability to adversarial attacks ( Szegedy et al. , 2014 ) . Crafted by adding some small perturbations to benign inputs , adversarial examples ( AEs ) can fool DNNs into making wrong predictions , which is a critical threat especially for some security-sensitive scenarios such as autonomous driving ( Sun et al. , 2020 ) . Based on the knowledge of target model , the adversarial attacks could be divided into white-box attack and black-box attack . White-box attacks ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Papernot et al. , 2016 ; Carlini & Wagner , 2017 ; Moosavi-Dezfooli et al. , 2016 ) have full access to the architecture and parameters of the target model , and can easily generate a successful adversarial example via back propagation . By contrast , black-box attacks ( Narodytska & Kasiviswanathan , 2016 ; Ilyas et al. , 2018 ; 2019 ; Moon et al. , 2019 ; Andriushchenko et al. , 2020 ; Li et al. , 2021 ; Papernot et al. , 2017 ; Liu et al. , 2017 ) can only query the target model to obtain the output prediction , which is a more realistic and challenging setting . Since many real-world online application programming interfaces ( APIs ) have time or monetary limit for user query ( Ilyas et al. , 2018 ) , most current research efforts focus on how to design query-efficient attacks ( Ilyas et al. , 2019 ; Moon et al. , 2019 ; Andriushchenko et al. , 2020 ) , which indeed achieve a huge improvement . For example , so far the state-of-the-art black-box attack ( Andriushchenko et al. , 2020 ) could succeed in untargeted attack on ImageNet dataset ( Deng et al. , 2009 ) with only tens of queries on average . However , this success has sacrificed the imperceptibility of AEs , whose global perturbations are generated with various tricks like random vertical stripes ( Andriushchenko et al. , 2020 ) , and thus very obvious and perceptible to human eyes ( see Figure 1 ( e ) and ( f ) for examples ) . In effect , imperceptibility is essential for attackers . Currently online APIs usually integrate detectors into their services to detect anomalous inputs ( Li et al. , 2021 ) . The images with too visible perturbations are difficult to pass these detectors , not to mention human judgement . To alleviate or even address this problem , we could utilize some priors to lead black-box attacks to search in small but efficient space , instead of global perturbations . Previous DNNs visualization work ( Simonyan et al. , 2014 ) proposed a type of saliency map to indicate which features should influence the output significantly . This BP saliency map 1 is generated by calculating the derivatives of the model output with respect to input . Just as show in Figure 1 . ( a ) , the brighter pixels in BP saliency maps denote they have a greater impact on the model output . The classical white-box attack JSMA ( Papernot et al. , 2016 ) just constructs such an adversarial saliency map and iteratively selects pixels from this map to perturb . But due to the black-box setting , we can not derive this saliency map directly . Nevertheless , we could find that the region of bright pixels roughly represents the position of the main object in an image , which also accords with our intuition . Inspired by this , we propose to first generate a saliency mask to extract the regions of the main object in an image via image segmentation , and then limit the perturbation search in this smaller but more important regions , as presented in Figure 1 . Several recent studies also propose to generate similar local perturbations but applied in different attack settings with different methods ( Dong et al. , 2020 ; Xiang et al. , 2021 ) . In particular , Dong et al . ( 2020 ) produce saliency maps with the class activation mapping ( CAM ) ( Zhou et al. , 2016 ) , which requires the internal information of the model and needs to change the model architecture . This is infeasible in black-box scenarios . Xiang et al . ( 2021 ) replace CAM with Grad-CAM ( Selvaraju et al. , 2017 ) , which does not need to change the architecture of the model . However , both CAM and Grad-CAM still rely on the knowledge of the model , but in black-box setting the internal information of the target model is inaccessible . Thus , we choose salient object segmentation ( Zhao & Wu , 2019 ) , which can automatically extract salient object ( s ) in an image ( see Figure 1. ( b ) ) . Compared with CAM and Grad-CAM , it doesn ’ t need any other information except for the input image . On the other hand , these studies ( Dong et al. , 2020 ; Xiang et al. , 2021 ) are applied to transfer-based attacks ( actually a grey-box setting ) , which first build a substitute model to approximate the target model , then use gradient-based white-box attacks to generate adversarial examples , and finally transfer them to attack the target model . For transfer-based attack , we must either train a substitute model from scratch with huge number of queries ( Papernot et al. , 2017 ) or assume a pretrained substitute model is trained on the similar data distribution with the target model ’ s ( Liu et al. , 2017 ) . And for the latter , the transferability of both adversarial examples and saliency maps highly depends on the selection of the pair of substitute model ( s ) and target model . Hence , in this paper we focus on gradient-free black-box attacks ( Narodytska & Kasiviswanathan , 2016 ; Moon et al. , 2019 ; Andriushchenko et al. , 2020 ) that take no account of gradients and search for successful perturbations only according to the model output . We add these segmentation priors to SOTA black-box attacks and find the imperceptibility of AEs is indeed enhanced but still limited due to ” global perturbation ” in local region ( See Figure 4 and 5 ) . To 1To distinguish the saliency map generated by the salient object segmentation model below , here we call it as BP saliency map to indicate it is generated via back propagation . further improve the imperceptibility , our another novelty is that even in the salient region , “ saliency ” could still be refined . Our inspiration also comes from some DNNs visualization works ( Zhou et al. , 2016 ; Zeiler & Fergus , 2014 ; Olah et al. , 2018 ) . For instance , given a dog image , CAM produces a localization heatmap and shows the dog face region is most highly lighted ( Zhou et al. , 2016 ) . Zeiler & Fergus ( 2014 ) systematically cover up different portions of a dog image , and finds when the dog face is obscured , the activity in the feature map and classifier output changes dramatically . In further , by combining internal feature visualization with output prediction , Olah et al . ( 2018 ) reveal even in dog ’ s face region , its ears and eyes seem to be more important when distinguishing dogs . Therefore , we are inspired to assume the salient region in an image is progressive with respect to its impact on model output . If we could find smaller but more salient region , the perturbation will be more efficient and meanwhile the imperceptibility of AEs could also be enhanced . Thus , we propose our Saliency Attack , a new gradient-free black-box attack via refining perturbations in the salient region according to their saliency . Our main contributions can be summarized as follows : • To our best knowledge , we are the first to use salient object segmentation to extract binary salient masks in black-box settings . Experiments show that the SOTA black-box attacks limited in such regions can achieve much better imperceptibility performance with little reduction in query efficiency and success rate . • We propose a new gradient-free black-box attack via refining in salient regions . Compared with the search methods used in other gradient-free attacks , our method is able to generate smaller but effective perturbations , which is interpretable to some extent and can further improve the imperceptibility . • We demonstrate that the perturbations generated by our Saliency Attack is more robust against some detection-based defense like Feature Squeezing . 2 RELATED WORK . 2.1 BLACK-BOX ATTACKS . Gradient estimation attacks . Gradient estimation attacks first estimate the gradients by querying the target model and then apply them to run white-box attacks ( Goodfellow et al. , 2015 ; Carlini & Wagner , 2017 ; Madry et al. , 2018 ) . ZOO attack ( Chen et al. , 2017 ) first adopts the symmetric difference quotient to approximate the gradients and then perform Carlini-Wagner ( CW ) white-box attack ( Carlini & Wagner , 2017 ) . AutoZOOM ( Tu et al. , 2019 ) uses a random vector based gradient estimation to estimate gradients , which reduces the query number per iteration from 2D in ZOO to N+1 ( D is the dimensionality and N is the sample size ) . To further enhance query efficiency , Ilyas et al . ( 2019 ) propose the “ tiling trick ” that updates a square of pixels simultaneously instead of a single pixel , which dramatically decreases the dimensionality by a factor of k2 ( k is tile length ) . Gradient-free attacks . Gradient-free attacks take no account of gradients and directly generate AEs with random search or heuristic methods according to the query result . Su et al . ( 2019 ) propose one pixel attack that adopts differential evolution algorithm to perturb the most important pixel in the image . Alzantot et al . ( 2019 ) propose GenAttack , which use genetic algorithm to generate AEs . To improve the query efficiency , Moon et al . ( 2019 ) consider a discrete surrogate optimization problem that transforms the original constraint of a continuous range [ − , + ] to a discrete set { − , + } , achieving a huge reduction in the search space . This is motivated by linear program ( LP ) where the optimal solution is attained at an extreme point of the feasible set ( Schrijver , 1998 ) . Combing tiling trick ( Ilyas et al. , 2019 ) and discrete optimization ( Moon et al. , 2019 ) , Square Attack ( Andriushchenko et al. , 2020 ) has obtained the best result on success rate and query performance so far with a randomized search scheme . Hybrid attacks . Hybrid attacks try to combine multiple types of methods , using candidate adversarial examples generated on substitute model as starting points for further optimization with gradient estimation attacks or gradient-free attacks . Suya et al . ( 2020 ) first proposed to generate candidate AEs by white-box projected gradient descent ( PGD ) attack ( Madry et al. , 2018 ) , and further optimize them with gradient estimation methods . Wang et al . ( 2020 ) combines a white-box attack with a gradient-free method using microbial genetic algorithm ( Harvey , 2009 ) . Similar with transfer-based attacks , hybrid attacks also suffer from the similarity between substitute model ( s ) and target model .

Summary: There has been a lot of interest in improving the query efficiency of black-box attacks in the recent past. However, these techniques produce examples that a human in the loop can quickly identify. The authors propose using segmentation priors to improve the black-box attacks so that the perturbations are restricted to the salient regions of the image. In addition to this, they also present a technique that improves the imperceptibility without forgoing the query efficiency.