Question: What is the technical significance of the proposed approach?
Evidence: Deep neural networks (DNNs) have achieved significant progress in wide applications, such as image classification, face recognition, object detection, speech recognition and machine translation. Despite their success, deep learning models have revealed vulnerability to adversarial attacks. Crafted by adding some small perturbations to benign inputs, adversarial examples (AEs) can fool DNNs into making wrong predictions, which is a critical threat especially for some security-sensitive scenarios such as autonomous driving.
Based on the knowledge of target model, the adversarial attacks could be divided into white-box attack and black-box attack. White-box attacks have full access to the architecture and parameters of the target model, and can easily generate a successful adversarial example via back propagation. By contrast, black-box attacks can only query the target model to obtain the output prediction, which is a more realistic and challenging setting. Since many real-world online application programming interfaces (APIs) have time or monetary limit for user query, most current research efforts focus on how to design query-efficient attacks, which indeed achieve a huge improvement. For example, so far the state-of-the-art black-box attack could succeed in untargeted attack on ImageNet dataset with only tens of queries on average. However, this success has sacrificed the imperceptibility of AEs, whose global perturbations are generated with various tricks like random vertical stripes, and thus very obvious and perceptible to human eyes (see Figure (e) and (f) for examples). In effect, imperceptibility is essential for attackers. Currently online APIs usually integrate detectors into their services to detect anomalous inputs. The images with too visible perturbations are difficult to pass these detectors, not to mention human judgement.  To alleviate or even address this problem, we could utilize some priors to lead black-box attacks to search in small but efficient space, instead of global perturbations. Previous DNNs visualization work proposed a type of saliency map to indicate which features should influence the output significantly. This BP saliency map 1 is generated by calculating the derivatives of the model output with respect to input. Just as show in Figure.(a), the brighter pixels in BP saliency maps denote they have a greater impact on the model output. The classical white-box attack JSMA just constructs such an adversarial saliency map and iteratively selects pixels from this map to perturb. But due to the black-box setting, we cannot derive this saliency map directly. Nevertheless, we could find that the region of bright pixels roughly represents the position of the main object in an image, which also accords with our intuition. Inspired by this, we propose to first generate a saliency mask to extract the regions of the main object in an image via image segmentation, and then limit the perturbation search in this smaller but more important regions, as presented in Figure.
Several recent studies also propose to generate similar local perturbations but applied in different attack settings with different methods. In particular, produce saliency maps with the class activation mapping (CAM), which requires the internal information of the model and needs to change the model architecture. This is infeasible in black-box scenarios. replace CAM with Grad-CAM, which does not need to change the architecture of the model. However, both CAM and Grad-CAM still rely on the knowledge of the model, but in black-box setting the internal information of the target model is inaccessible. Thus, we choose salient object segmentation, which can automatically extract salient object(s) in an image (see Figure.(b)). Compared with CAM and Grad-CAM, it doesn't need any other information except for the input image.
On the other hand, these studies or assume a pretrained substitute model is trained on the similar data distribution with the target model's. And for the latter, the transferability of both adversarial examples and saliency maps highly depends on the selection of the pair of substitute model(s) and target model. Hence, in this paper we focus on gradient-free black-box attacks that take no account of gradients and search for successful perturbations only according to the model output.
We add these segmentation priors to SOTA black-box attacks and find the imperceptibility of AEs is indeed enhanced but still limited due to "global perturbation" in local region (See Figure). To further improve the imperceptibility, our another novelty is that even in the salient region, "saliency" could still be refined. Our inspiration also comes from some DNNs visualization works. For instance, given a dog image, CAM produces a localization heatmap and shows the dog face region is most highly lighted. systematically cover up different portions of a dog image, and finds when the dog face is obscured, the activity in the feature map and classifier output changes dramatically. In further, by combining internal feature visualization with output prediction, reveal even in dog's face region, its ears and eyes seem to be more important when distinguishing dogs. Therefore, we are inspired to assume the salient region in an image is progressive with respect to its impact on model output. If we could find smaller but more salient region, the perturbation will be more efficient and meanwhile the imperceptibility of AEs could also be enhanced. Thus, we propose our Saliency Attack, a new gradient-free black-box attack via refining perturbations in the salient region according to their saliency.
Our main contributions can be summarized as follows:
• To our best knowledge, we are the first to use salient object segmentation to extract binary salient masks in black-box settings. Experiments show that the SOTA black-box attacks limited in such regions can achieve much better imperceptibility performance with little reduction in query efficiency and success rate.
• We propose a new gradient-free black-box attack via refining in salient regions. Compared with the search methods used in other gradient-free attacks, our method is able to generate smaller but effective perturbations, which is interpretable to some extent and can further improve the imperceptibility.
• We demonstrate that the perturbations generated by our Saliency Attack is more robust against some detection-based defense like Feature Squeezing.
On the other hand, these studies (Dong et al., 2020; Xiang et al., 2021) are applied to transfer-based attacks (actually a grey-box setting), which first build a substitute model to approximate the target model, then use gradient-based white-box attacks to generate adversarial examples, and finally transfer them to attack the target model. For transfer-based attack, we must either train a substitute model from scratch with huge number of queries (Papernot et al., 2017) or assume a pretrained substitute model is trained on the similar data distribution with the target model’s (Liu et al., 2017). And for the latter, the transferability of both adversarial examples and saliency maps highly depends on the selection of the pair of substitute model(s) and target model. Hence, in this paper we focus on gradient-free black-box attacks (Narodytska & Kasiviswanathan, 2016; Moon et al., 2019; Andriushchenko et al., 2020) that take no account of gradients and search for successful perturbations only according to the model output. We add these segmentation priors to SOTA black-box attacks and find the imperceptibility of AEs is indeed enhanced but still limited due to ”global perturbation” in local region (See Figure 4 and 5). To further improve the imperceptibility, our another novelty is that even in the salient region, “saliency” could still be refined. Our inspiration also comes from some DNNs visualization works (Zhou et al., 2016; Zeiler & Fergus, 2014; Olah et al., 2018). For instance, given a dog image, CAM produces a localization heatmap and shows the dog face region is most highly lighted (Zhou et al., 2016). Zeiler & Fergus (2014) systematically cover up different portions of a dog image, and finds when the dog face is obscured, the activity in the feature map and classifier output changes dramatically. In further, by combining internal feature visualization with output prediction, Olah et al. (2018) reveal even in dog’s face region, its ears and eyes seem to be more important when distinguishing dogs. Therefore, we are inspired to assume the salient region in an image is progressive with respect to its impact on model output. If we could find smaller but more salient region, the perturbation will be more efficient and meanwhile the imperceptibility of AEs could also be enhanced. Thus, we propose our Saliency Attack, a new gradient-free black-box attack via refining perturbations in the salient region according to their saliency. Our main contributions can be summarized as follows: • To our best knowledge, we are the first to use salient object segmentation to extract binary salient masks in black-box settings. Experiments show that the SOTA black-box attacks limited in such regions can achieve much better imperceptibility performance with little reduction in query efficiency and success rate. • We propose a new gradient-free black-box attack via refining in salient regions. Compared with the search methods used in other gradient-free attacks, our method is able to generate smaller but effective perturbations, which is interpretable to some extent and can further improve the imperceptibility. • We demonstrate that the perturbations generated by our Saliency Attack is more robust against some detection-based defense like Feature Squeezing.
Answer: First we notice there is a misunderstanding of our approach in this comment. After reaching the minimal block or no smaller block has a better loss, we backtrack to the last level of split blocks (instead of going back to the level of initial blocks and “applying the procedure again to the second best block”) and choose the best one of the remaining blocks for further perturbation. The intuition is that we always refine perturbations in a region that is as small as possible, which accords with our goal of perturbing smaller but more important regions to generate imperceptible AEs.
For our search algorithm, we agree that it is simple. However, we think that a simple yet effective algorithm is always desirable, because it can be easily understood and implemented. For examples, Parsimonious attack (Moon et al., 2019) uses greedy local search; Square attack (Andriushchenko et al., 2020) adopts an even simpler random search. But these simple algorithms can achieve very strong performance, because they are particularly suited to the problems. For the present work, the proposed search algorithm is also suited to the problem of generating AEs with small perturbations, and the results have demonstrated its effectiveness. We appreciate for the suggestion that we need to justify the search algorithm. In Table 2 (page 8), Saliency significantly outperforms Parsimonious-seg and Square-seg, which indicates that our search algorithm is better than the local search algorithm in Parsimonious Attack and the random search algorithm in Square Attack. Moreover, we have added experiments to compare our search algorithm with a new baseline greedy search in salient region, which greedily perturbs blocks to maximize marginal gain. We have fine-tuned the block size of the greedy search algorithm and the comparison results are presented in Table 3 (page 9). It can be seen that our algorithm is still significantly better. The above results that our search algorithm outperforms existing search algorithms (i.e., local search and random search) and common baseline (i.e., greedy search) indicate that it is indeed effective.
For using saliency maps, indeed, this idea has already appeared in the field of adversarial attack. However, to the best of our knowledge, we are the first to apply it to black-box attack on image classification model. The two papers mentioned in the comment (Guo et al.,2020; Sun et al., 2021) are both for white-box attacks. Actually we have reviewed two similar works (Dong et al., 2020; Xiang et al., 2021) in the paper, which use CAM/grad-CAM to generate saliency maps and can only be applied in white-box or grey-box settings. We introduced them and analyzed their difference and infeasibility to black-box setting thoroughly (para. 3,4, page 2). Besides, we want to clarify that our main contribution is not just introducing saliency maps into black-box attacks, but how to process and utilize saliency maps in black-box setting. The combination of saliency maps and our search algorithm bring out smaller and more imperceptible perturbations, which greatly alleviates the problem of imperceptibility in black-box attacks.