Imperceptible Black-box Attack via Refining in Salient Region
Authors: Zeyu Dai
Shengcai Liu
Ke Tang
Qing Li
Venue: ICLR 2022 Submitted (2022)

Abstract: Deep neural networks are vulnerable to adversarial examples, even in the black-box setting where the attacker only has query access to the model output. Recent studies have devised successful black-box attacks with high query efficiency. However, such performance often comes at the cost of the imperceptibility of adversarial attacks, which is essential for attackers. To address this issue, in this paper we propose to use segmentation priors for black-box attacks such that the perturbations are limited in the salient region. We find that state-of-the-art black-box attacks equipped with segmentation priors can achieve much better imperceptibility performance with little reduction in query efficiency and success rate. We further propose the Saliency Attack, a new gradient-free black-box attack that can further improve the imperceptibility by refining perturbations in the salient region. Experimental results show that the perturbations generated by our approach are much more imperceptible than the ones generated by other attacks, and are interpretable to some extent. Furthermore, our approach is found to be more robust to detection-based defense, which demonstrates its efficacy as well.

Under review as a conference paper at ICLR 2022
IMPERCEPTIBLE BLACK-BOX ATTACK VIA REFINING IN SALIENT REGION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks vuinerable to adversarial examples, even in the black- box setting where the attacker only has query access to the model output. Re- cent studies have devised successful black-box attacks with high query efficiency. However, such performance often comes at the cost of the imperceptibility of ad- versarial attacks, which is essential for attackers. To address this issue, in this paper we propose to use segmentation priors for black-box attacks such that the perturbations are limited in the salient region. We find that -fate-of-the-ar black- box attacks equipped with segmentation priors can achieve much better imper- ceptibility performance with little reduction in query efficiency and success rate. We further propose the Saliency Attack, a new gradient-free black-box attack that can further improve the imperceptibility by refining perturbations the salient re- gion. Experimental results show that the perturbations generated by our approach are much more imperceptible than the ones generated by other attacks, and are in- terpretable to some extent. Furthermore, our approach is found to be more robust to detection-basee defense, which demonstrates its efficacy as well.
INTRODUCTION
Deep neural networks (DNNs) have achieved significant progress in wide applications, such as im- age classification (Deng et al. 2009), face recognition (Parkhi et al., 2015). object detection (Red- mon et al. 2016). speech recogmition (4 2012) and machine translation (Bahdanau et al. 2015). De- spite their success, deep learning models have revealed vulnerability to adversarial attacks (Szegedy et al. 2014). Crafted by adding some small perturbations to benign inputs, adversarial examples (AEs) can fool DNNs into making wrong predictions, which is critical threat especially for some security-sensitive scenarios such autonomous driving (Sun et al. 2020).
Based on the knowledge of target model, the adversarial attacks could be divided into white-box attack and black-box attack. White-box attacks (Szegedy et al. [014]] Goodfellow et al.) 2015) Pa- pernot et al. 2016; Carlini & Wagner 2017: Moooaao--ezzoooooe
et al. 2016) have full access to the architecture and parameters of the target model, and can easily generate a successful adversarial ex- ample via back propagation. By contrast, black-box attacks (Narodytska & Kasiviswanathan 2016)
Ilyas et al. 2018 2019 MMon al. 2019 Andriushchenko et al. [220 Li et al.) 2021 Papernot et al. 2017 Liu el al. (017) can only query the target model to obtain the output prediction, which more realistic and challenging setting. Since many real-world online application programming interfaces (APIs) have time or monetary limit for user query (Ilyas et al., 2018), most current re- search efforts focus on how design query-efficient attacks (Ilyas et al., [019]] Moon et al. 2019 Andriushchenko et al., 2020). which indeed achieve a huge improvement. For example, so far the offe-oof-the-art black-bor attack (Andriushchenko et al. (020) could succeed in untargeted attack on ImageNet dataset (Deng al.) 2009) with only tens of queries on average. However, this success has sacrificed the imperceptibility of AEs, whose global perturbations are generated with various tricks like random vertical stripes (Andriushchenko al. 2020), and thus very obvious and percep- tible to human eyes (see Figure (e) and (f) for examples). In effect, imperceptibility is essential for attackers. Currently online APIs usually integrate detectors into their services to detect anomalous inputs et 202T). The images with too visible perturbations are difficult to pass these detectors, not mention human judgement.

---

Under review as a conference paper at ICLR 2022
(a)
(c)
(d) Saliency (ours) Parsimonious
(I) Square
(g) Original
Figure 1: Illustration of our Saliency Attack and current SOTA gradient-free black-box attacks. (a) BP saliency maps, specific to a given image and the corresponding class; (b) binary salient masks, generated by saliency object segmentation and roughtly accord with the region of the salient pixels in BP saliency maps; (c) perturbation, produced by refining in salient regions of (b); (d) final AEs of our Saliency Attack; (e-f) AEs of SOTA black-box attacks (Moon et al., 2019 Andriushchenko et al. 2020) whose perturbations are visible to human eyes; (g) original image.
To alleviate or even address this problem, we could utilize some priors to lead black-box attacks to search in small but efficient space, instead of global perturbations. Previous DNNs visualization work (Simonyan et al., 2014) proposed a type of saliency map to indicate which features should influence the output significantly. This BP saliency map generated by caleulating the derivatives of the model output with respect to input. Just as show in Figure (a). the brighter pixels in BP saliency maps denote they have greater impact the model output. The classical white-box attack JSMA (Papernot et al., 2016) just constructs such an adversarial saliency map and íteratively selects pixels from this map to perturb. But due the black-box setting, we cannot derive this saliency map directly. Nevertheless, we could find that the region of bright pixels roughly represents the position of the main object in an image, which also accords with our intuition. Inspired by this, we propose to first generate saliency mask to extract the regions of the main object in an image via image segmentation, and then limit the perturbation search in this smaller but more important regions, as presented in Figure
Several recent studies also propose to generate similar local perturbations but applied in different at- tack settings with different methods (Dong et al. 22200) Xiang et al. 2021). In particular, Dong et al. (2020) produce saliency maps with the class activation mapping (CAM) (Zhou et al. 2016). which requires the internal information of the model and needs to change the model architecture. This is infeasible in black-box scenarios. [iian et al.] (202T) replace CAM with Grad-CAM (Selvaraju et al. 2017), , which does not need to change the architecture of the model. However, both CAM and Grad-CAM still rely on the knowledge of the model, but in black-box setting the internal informa- tion of the target model is inaccessible. Thus, we choose salient object segmentation (Zhao & Wu 2019), which can automatically extract salient object(s) in an image (see Figure (b)). Compared with CAM and Grad-CAM, it doesn't 't need any other information except for the input image.
On the other hand, these studies (Dong et al. 2020 Xiang et al., (021) are applied to transfer-based attacks (actually grey-box setting), which first build substitute model to approximate the tar- get model, then use gradient- based white-box attacks to generate adversarial examples, and finally transfer them to attack the target model. For transfer-based attack, we must either train a substi- tute model from scratch with huge number of queries (Papernot et al.) (017]) or assume a pretrained substitute model is trained on the similar data distribution with the target model's (Liu et al.) 2017). And for the latter, the transferability of both adversarial examples and saliency maps highly depends on the selection of the pair of substitute model(s) and target model. Hence, in this paper we focus on gradient-free black-box attacks (Narodytska & Kasiviswanathan 2016 MMon et al.) 2019 An- driushchenko et al. (020) that take no account of gradients and search for successful perturbations only according to the model output.
We add these segmentation priors to SOTA black-box attacks and find the imperceptibilitty of AEs is
indeed enhanced but still limited due to "global perturbation" in local region (See Figure and 5) To
"To distinguish the saliency map generated by the salient object segmentation model below, here call it BP saliency map to indicate is generated via back propagation.
2

---

Under review as a conference paper at ICLR 2022
further improve the imperceptibilityy. our another novelty is that even in the salient region, "saliency" could still be refined. Our inspiration also comes from some DNNs visualization works (Zhou et al,
2016
Zeiler & Fergus 2014 Olah et al. 2018). For instance, given dog image, CAM produces localization heatmap and shows the dog face region is most highly lighted (Zhou et al. 2016). Zeiler & Fergus (2014) systematically cover up different portions of dog image, and finds when the dog face is obscured, the activity in the feature map and classifier output changes dramatically. In further, by combining internal feature visualization with output prediction, Olah et al. (2018) reveal even in dog's s face region, its ears and eyes seem to be more important when distinguishing dogs. Therefore, we are inspired to assume the salient region in an image is progressive with respect to its impact on model output. If we could find smaller but more salient region, the perturbation will be more efficient and meanwhile the imperceptibility of AEs could also be enhanced. Thus, we propose our Saliency Attack, a new gradient-free black-box attack via refining perturbations in the salient region according to their saliency.
Our main contributions can be summarized as follows:
To our best knowledge, we the first to use salient object segmentation extract binary salient masks in black-box settings. Experiments show that the SOTA black-box attacks limited in such regions can achieve much better imperceptibility performance with little reduction in query efficiency and success rate.
We propose a new gradient-free black-box attack via refining in salient regions. Compared with the search methods used in other gradient-free attacks, our method is able generate smaller but effective perturbations, which is interpretable to some extent and can further improve the imperceptibility.
We demonstrate that the perturbations generated by our Saliency Attack is more robust against some detection- based defense like Feature Squeezing.
2
RELATED WORK
2.1 BLACK-BOX ATTACKS
Gradient estimation attacks. Gradient estimation attacks first estimate the gradients by querying the target model and then apply them to run white-box attacks (Goodfellow et al., 2015 Carlini Wagner 2017 Madry et al.) 2018). ZOO attack (Chen al. 2017) first adopts the symmetric difference quotient to approximate the gradients and then perform Carlini- Wagner (CW) white-box attack (Carlini & Wagner 2017). AutoZOOM (Tu et al. 2019) uses a random vector based gradient estimation to estimate gradients, which reduces the query number per iteration from 2D in ZOO to N+1 (D is the dimensionality and N is the sample size). To further enhance query efficiency, [lyas) et al. (2019) propose the "tiling trick" that updates a square of pixels simultaneously instead of single pixel. which dramatically decreases the dimensionality by a factor of k2 (I: is tile length).
Gradient-free attacks. Gradient-free attacks take account of gradients and directly generate AFs with random search or heuristic methods according to the query result. [Su et al. (2019) propose one pixel attack that adopts differential evolution algorithm to perturb the most important pixel in the image. Alzantot et al. (2019) propose GenAttack, which use genetic algorithm to generate AEs. To improve the query efficiency, Moon et al. (019)) consider a discrete surrogate optimization problem that transforms the original constraint of a continuous range +c] to discrete set {-c,+e), achieving a huge reduction in the search space. This is motivated by linear program (LP) where the optimal solution is attained at an extreme point of the feasible set (Schrijver 1998). Combing tiling trick (Ilyas et al. 2019) and discrete optimization (Moon et al., 2019), Square Attack (An-) driushchenko et al. (020) has obtained the best result on success rate and query performance far with a randomized search scheme.
Hybrid attacks. Hybrid attacks try to combine multiple types of methods, using candidate adversar- ial examples generated on substitute model as starting points for further optimization with gradient estimation attacks or gradient-free attacks. [Suya et al. (2020) first proposed to generate candidate AEs by white-box box projected gradient descent (PGD) attack (Madry et al.) 2018), and further opti- mize them with gradient estimation methods. Wang al. 2020) combines a white-box attack with
3

---

Under review as a conference paper at ICLR 2022
gradient-free free method using microbial genetic algorithm (Harvey 2009). Similar with transfer- based attacks, hybrid attacks also suffer from the similarity between substitute model(s) and target model.
2.2 RELATED WORK ON IMPERCEPTIBILITY
The imperceptibility of AEs is vital for attackers and some attacks consider it in different manners. Guo et al.] (2019) difine the imperceptibility as the smoothness in the sense of low frequency and thus search for AEs in frequency domain. While Zhang et al. (2020) regard the imperceptibility as the smoothness of visual content in an image and integrate Laplacian smoothing into optimization. Boundary Attack (Brendel al., 2018) starts from example that is already adversarial and ap- proaches the original image gradually to reduce the distortion. Instead of global perturbations, some studies (Dong al. 2020 Xiang et al. 2021 consider local perturbations reduce the square of perturbations.
On the other hand, how estimate the imperceptibilityy of AEs is still problem. Most of the ad- versarial attacks use Lp norms (Lo. L2 and L to measure the human perceptual distance between perturbed image and the original one. Nonetheless, Lp norms are found not suitable enough for human vision system (Sharif et al.) 2018). To find a proper metric, Fezza et al. (2019) design subjective experiments to obtain the subjective scores on different AEs, and then test various im- age fidelity assessment (IFA) metrics including Lp norms. Among them, most apparent distortion (MAD) (Larson & Chandler, 2010) metric found closest to subjective scores. Hence, in this paper adopt MAD our main imperceptibility metric to estimate the imperceptibility of AEs.
3 PROPOSED METHOD
In this section, we first introduce the problem formulation of crafting adversarial examples for image classification models, and then detail our approach. The overall flowchart our proposed Saliency Attack is depicted in Figure
Saliency Map
Binary Mask
Perturhation
Refining suliency region
Salient object segmetnution
Binarize
Example
Adversarial Example
Figure 2: The overall flowchart of Saliency Attack.
3.1 PRELIMINARY
Given well-trained DNN classifier F : |0, 1]d RK where is the dimension of the input and K is the number of classes. We denote Fxx as the predicted score that x belongs to class k. So the classifier assigns the class that maximizes F2 (2)) to the input .r The goal of an untargeted attack is to find an adversarial example Fados that results in the model misclassification from the ground-truth class y, and meanwhile keeps the distance between the adversarial and benign input smaller than threshold e:
arg (Faa)) + y, s.f. ||Fade x112 S Fade E [] k=l....K
where the other constraint indicates that the generated Fadv must be an image in valid range. this paper focus on Loo our distance norm following (Moon al. 2019: Andriushchenko al. 2020).
Traditionally, this task of finding Tods be rephrased solving constrained continuous problem:
max I(F(xade) y). s.t. |[aado - 2||00 <
Zedo 1
where is a loss function. To improve the query efficiency, Moon et al. (2019) transform the continuous problem into discrete surrogate problem, where the perturbation à Todv 2: is
4

---

Under review as a conference paper at ICLR 2022
generated at the corner of the Loo ball. Besides, we propose to limit only in the salient region. So our complete optimization problem is:
I(F(Radu):))) v), 8.1. |00 E {(c, 8e E Pad=E(0,11"
max
where is the set the pixels in salient region.
3.2 SALIENT OBJECT SEGMENTATION
Salient object segmentation, as changing task in image segmentation domain, aims to automat- ically and accurately extract salient object in an image. Specifically, given an input image, salient object segmentation model generate a saliency map where the brighter pixels denote they have higher saliency scores, just as the saliency map shows in Figure Compared with other model vi- sualization methods like CAM (Zhou al. 2016) and Grad-CAM (Selvaraju al. 2017), this type of model does not require any information other than the input image, which is very suitable for the black-box setting. We adopt the Pyramid Feature Attention (PFA) network (Zhao Wu 2019) that achieves the state-of-the-ar performance in multiple datasets. After obtaining the saliency map where all pixel values are between 0 and 1, we use a binarization threshold Φ to transform the saliency map to a binary salient mask. The binarization can be expressed as
0
Sig < Φ
(4)
Si.j 20
where s is the saliency map and s" is the binary salient mask, Sty and si are the corresponding value the position i,j. With this saliency mask s", could limit the perturbation search in the salient region.
3.3 REFINING PERTURBATION IN SALIENT REGION
In this part, we design a search algorithm to refine per- turbations based on a tree structure for an image by us- ing the idea of depth-first search. shown Figure 3 an input image merged with its saliency mask is split into four initial blocks (a coarse grid). Then we try to add a +e or perturbation on each initial block in- dividually and find the best block that maximizes the loss function for further perturbation. After finding the best one among all initial blocks, we can further split this block into smaller blocks finer grid), and again try to add a perturbation on each block individually (at this time we just flip the perturbation for convenience e.g. +e to -e) to find the best smaller block. We iterate this process until the minimal block (e.g. 1 pixel) or no smaller block has better loss. Then we go back to the last level of split blocks and use the second best block for further perturbation.
The illustration of refining on tree structure corresponding to the image blocks. Among initial blocks, each block is a root node of a tree, and its child nodes are the finer blocks in the current block.
Figure 3:
We use the loss function from CW attack (Carlini & Wagner 2017) for untargted attack:
((x) (max(((ITde))) max (Z(dd 0) (5) iç0
where (o is the logit with respect to the true class of the original image. In this way, the loss function is imposed to leave a margin between the true class and other classes. The refining process is presented in Algorithm [ and the overall saliency attack is given in Algorithm 2
4 EXPERIMENTS
In this paper, we evaluate the performance of the Saliency Attack comparing against Boundary At- tack (Brendel al. 2018). TVDBA (Li & Chen 2021) Parsimonious attack (Moon et al. 2019.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1:
Refine
Input: block b, block size k
Output: adversarial example Vadv
k' is the block size of b;
{bi, bi,b,.... b2....,bA? }n + split into (k//k) /k2² finer blocks;
(b7(1); .... calculated by using Eq. 5
f- sort {b1, b2,,,,, ... Pn} in descending order according L(b), which is
for each block E (1)( (=(2), ... bm()) do
if L(e) i then
Eado <- x + [e))
< I(e);
if k: > 1 then
k - k/2;
10
Recursively call Tade + Refine(e, k);
11
end
end 12
end
Algorithm 2: Saliency Attack
Input: original image z, initial block size kint, query budget Output: adversarial example Vadv
8 1 is the perturbation;
while kint > 1 and not exceeding query budget do
Running Algorithm 2
Todu f- Refine(xx, kint);
kint t kint/2;
end
Square attack (Andriushchenko et al. 2020) and their modified version equipped with segmentation priors. Among them, Boundary Attack is able to constantly reduce the distortion to very small magnitude, TVDBA tries to minimize the distortion via integrating Structural SIMilarity (SSIM) (Wang et al., 2004) into the loss, and Square Attack can achieve the current state-of-the-au query efficiency and success rate in the black-box setting. We consider Loc threat model on ImageNet dataset (Deng et al., 2009), set 0.05 in [0.1] scale, and test 1000 randomly selected exmaples in all experiments. The threshold & to produce binary salient masks is chosen to be 0.1. All parameters of the compared attacks remain consistent with those recommended their papers. We use Incep- tion v3 (Szegedy et al. 2016) as the target model, and different query budgets (3000, 10000, 30000) for untargeted attack. For performance metrics, we employ commonly used success rate (SR) and average number of queries (Avg. queries). To evaluate the imperceptibility of AEs, we consider Lo. La and MAD (see the appendix for details), which is validated as closest to human vision system among existing IFA metrics (Tezza et al. 2019). All these three imperceptibilityy metrics are the smaller the better.
4.1 SOTA BLACK-BOX ATTACKS WITH SEGMENTATION PRIORS
To verify the feasibility of our segmentation priors, we modify Parsimonious Attack and Square Attack so that the perturbations are limited in salient regions. We show the results in Table and some examples in Figure 4 From the examples, we can easily find that the perturbations of original attacks are very obvious in the entire image due to global perturbations, while their segmentation versions are relatively more imperceptible since no perturbation exists background regions.
Besides, through quantitative analysis, the attacks equipped with segmentation priors can indeed achieve much better imperceptibility performance with around one third improvement Le and MAD metric. Lo is also restricted to less than 50%, which implies the average square of salient regions among all examples only occupies less than half the whole image. And this improvement only comes little cost in SR and Avg. queries, which is still in the same order of magnitude. We also present the change of MAD scores Figure D in appendix. Thus, the results demonstrate that the segmentation prior is indeed able to enhance the imperceptibility of other black-box attacks.

---

Under review as a conference paper at ICLR 2022
Table I: Comparison of black-box attacks with their modified version with segmentation priors.
Method
Avg. queries
SR
T.
MAD
Parsimonious
739
98.2%
53.20
100.0%
22.17
Parsimonious-seg
828
96.7%
45.64
42.70%
13.95
Square
222
99.7%
59.76
99.03%
25.36
Square-seg
563
97.1%
40.45
41.89%
16.10
(b) Allack
(c) Parsimonious-es seg
Square
Square-weg
Figure 4: Examples of SOTA black-box attacks compared with their modefied version with segmen-
tation priors. (a) binary salient mask; (b-e) pairs of AEs and their perturbation.
4.2 COMPARISON SALIENCY ATTACK WITH OTHER ATTACKS
Although the segmentation prior is helpful to the imperceptibility. the perturbations generated by other attacks are still global, taking up almost all salient regions. Hence, design our Saliency Attack via further refining perturbations in salient regions. Original image are resized to 256 x 256 and we set the initial block size 16. For imperceptible AE, low MAD value is important. So besides SR, we use new metric SRaree- It denotes the rate of successful AEs whose MAD values are below some threshold. Here we choose 30 the threshold because we find the AEs with MAD < 30 are basically imperceptible to human eyes (See Figure I0 in appendix for details).
The comparison results are given in Table [] We can find in all query budgets, our Saliency Attack achieves outstanding performance with huge gap in terms of SRtrue and three imperceptibility metries. Although TVDBA, Parsimonious and Square Attack can obtaín better SR, their success- ful AEs are not true imperceptible, and will be easily detected by some defenses or humans. For Boundary Attack, it can gradually make progress on SRtrue. L2 and MAD as the query budget in- creases, but they are still much worse than Saliency Attack. This is because it takes at least hundreds of thousands of queries for Boundary Attack to converge (Brendel et al.) 2018). which is infeasible in practical. We further test more eamples to verify the statistical significance in Table 5 and present a comparison of SR versus query number in Figure T1 in appendix.
From the exhabited examples in Figure coarse textures because of inadequate query budget. In addition, even restricted the same salient regions, the perturbations of Parsimonious--es and Square seg attack are complex and irregular, while our perturbations are smaller and more important, roughtly corresponding to the bright pixels in BP saliency map. They further represent the postions of dogs' noses or ears, which accord with our inspiration before.
find the AEs of Boundary Attack contain obvious
Boundary
(b) Pasimonions--eg
(c) Square-aeg
Saliency Attack
Figure 5: Examples of our Saliency Attack and other compared attacks. (a-d) pairs of AEs and their perturbation; (e) BP saliency maps.

---

Under review as a conference paper at ICLR 2022
Table 2: Comparison of different attacks. rrue denotes the rate of successful AEs with MAD < 30, which are believed imperceptible to human eyes.
Method
Query Budget
SR
SR
I.2
Lo
MAD
3000
100.0%
13.6%
68. 13
99.6%
84,96
Boundary
10000
100.0%
30.2%
58.82
99.7%
71.61
30000
100.0%
46.0%
58.30
99.8%
62.77
3000
90.1%
22.9%
10.77
19.7%
39.20
TVDBA
10000
96.9%
23.2%
10.96
20.5%
39.42
30000
98.4%
23.0%
11.06
20.7%
39.66
3000
91.9%
14.0%
22.17
100.0%
52.01
Parsimonious
10000
98.2%
14.1%
22.17
100.0%
53,20
30000
99.8%
14.1% %
22.17
100.0%
53.36
3000
89.8%
10.8%
14.01
42.7%
45,50
Parsimonious-seg
10000
96.7%
12.0%
13.95
42.7%
45.64
30000
99.7%
12.4%
13.92
42.7%
45.48
3000
98.4%
2.2%
25.36
99.0%
59.29
Square
10000
99.7%
1.9%
25.36
99.0%
59.76
30000
99.9%
2.7%
25.35
99.0%
58.24
3000
89.5%
19.9%
16.26
42.7%
40.41
Square-seg
10000
97.1%
22.5%
16.10
41.9%
40.45
30000
98.7%
25.5%
16.02
41.7%
39.05
3000
85.6%
80.2%
3.52
3.3%
12.32
Saliency (ours)
10000
93.6%
86.2%
3.71
3.8%
12.88
30000
96.1%
87.9%
3.89
4.3%
13.28
4.3 HYPERPARAMETEA TUNING
Our Saliency Attack contains only one hyperparameter namely the initial block size which deter- mines the first level of split blocks. We the effect of different k on SR, imperceptibility and query number respectively in Figure 6 As k decreases, SR and imperceptibility can be improved and SR reaches the peak when k equals 16. This because with smaller initial blocks, Saliency Attack can search for perturbations more finely and accurately leading to higher SR and better imperceptibility performance. Meanwhile inevitably more queries are needed, especially for sorting initial blocks. That is why under limited query budget like 10000, turning point occurs in SR.
We also show some examples in Figure 7 can be observed that as decreases, the generated perturbations also become smaller but more salient. For instance, in the first row the perturbation with l 128 roughly covers the region of the dog face, which is also the brightest region in the BP saliency map. While the perturbation with k 32 k 16 focuses dog car. This indicates that our Saliency Attack could find smaller and more important perturbations progressively the initial block size decreases. And the perturbations interpretable to some extent like perturbing the specific regions of dog's ears, eyes or nose.
smaller region of the
4.4 ABLATION STUDY
We do ablation study of Saliency Attack, including refining in salient region, in non-salient region and without saliency (refining in the whole image). We also design a greedy search as bascline to demonstrate our Refine search. We test multiple block sizes for greedy search and use 32 as the best choice (see Table 6 in appendix). The results and some examples are given in Table 3 and Figure 8 Notice that refining in salient region and refining without saliency generate the same or almost the same perturbations, which means the salient regions indeed contain useful parts and enhance the query efficiency by limiting the search space. But for refining in non-salient region, its perturbation more complex and visible with worse query efficiency and SR. Compared with greedy search, Refine search has much better query efficiency and SR which proves its superiority. Hence, we conclude that both salient region and Refine search facilitate Saliency Attack.
8

---

Under review as a conference paper at ICLR 2022
0.95
25
4000
MAD
Impercapibiitty
Avg. queries
3000
2.5
5
0.85
5
1000
0.8
128
64
16
128
64
16
128
64
32
16
Block size
Block size
Block size
Figure 6: The effect of initial block size SR, imperceptibility (12 and MAD) and avg. queries.
(n)
k= 128
(0) 3
(d) k=
Figure 7: Examples of DFS-seg attack with different initial block sizes. (a) binary salient mask;
(b-d) exmaples of Saliency Attack with different initial block sizes; (e) BP saliency map.
Table 3: Ablation study of Saliency attack under 10000 query budget.
Method
Avg. queries
SR
SR
L2
Lo
MAD
refine in salient region
1958
93.6%
86.2%
3.71
3.8 %
12.88
refine in non-salient region
3128
78.2%
57.0%
4.94
6.5%
21.58
refine without saliency
2563
95.5%
79.6%
3.84
4.2%
16.35
greedy search in salient region
2727
56.0%
50.7%
4.37
4.7%
12.87
Saliency
salient region
(c) without saliency
(d) colisat region
Figure 8: Examples of Saliency Attack with different strategies.
4.5 ATTACKING DETECTION-BASEE DEFENSE
To further validate the superiority of our method, we attack against one detection- defense Fea- ture Squeezing Xu al. 2018), which applies transformations to input image and judge whether adversarial through the stability of model output. We consider the detection rate of both successful adversarial examples (SAEs) and failure adversarial examples (FAEs). This is because if could detect FAES, we can raise the alarm that some potential attacker attacking the model. From Table 4 find Saliency attack is able achieve lower detection rate in both SAES and FAEs than other attacks, which proves the imperceptibility of our AEs from the defensive perspective.
Table 4: Results for attacking against detection-based defense
Parsimonious
Square
Boundary
Saliency
Detection rate (SAEs)
33.3%
47.8%
61.5%
21.2%
Detection rate (FAEs)
14.5%
16.7%
11.7 %

---

Under review as a conference paper at ICLR 2022
5 CONCLUSION
In this paper, we propose to provide segmentation priors for black-box attacks to improve the imper- ceptibility of adversarial examples, whose perturbations are limited in salient regions. We utilize a salient object segmentation model to produce such saliency maps with no need for any information other than the input image. Experiments show that SOTA black-box attacks equipped with seg- mentation priors can indeed achieve better imperceptibility performance. Furthermore, we devise new gradient-free black-box Saliency attack that further enhances the imperceptibility via refining in salient regions. The perturbations generated are much smaller, imperceptible and interpretable to some extent. Finally, we also attack against some detection- based defense and the results show that the adversarial examples generated by attack are harder to be detected.
REFERENCES
Deep neural networks for acoustic modeling in speech recognition: The shared views of four re- search groups. JEEE Signal Process. Mag. 29(6):22-77, 2012. doi: 10. 10.1109/MSP.221222205555 URL https://doi..org
11w.:11099MM 2012. 2205597]
Moustafa Alzantot, Yash Sharma, Supriyo Chakraborty, Huan Zhang, Cho-Jui Hsieh, and Mani B. Srivastava. Genattack: practical black-box attacks with gradient- optimization. In Anne Auger and Thomas Stützle (eds.), Proceedings of the Genetic and Evolutionary Computation Confer- ence, GECCO 2019, Prague, Czech Republic, July 13-17, 2019, pp. 1111-1119, ACM, 2019. doi: URLhttps:: //d::////ooi.. org/ 10. 1145 3051733217001 32213333772777777777 3321749
10.1145//321107.33227777
Maksym Andriushchenko. Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: A query-efficient black-box adversarial attack via random search. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII, vol- ume 12368 of Lecture Notes in Computer Science, pp- 484 Springer, 2020. doi: 10.1007/ 978-3-030-58592-1,. 29.. URL https:: /doi org 10. 1007 978 030 58592-2
29
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Con- ference Track Proceedings, 2015.
0473
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In 6th International Conference on Learn- ing Representations, ICLR 2018, Vancouver. BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. htt.ts://tppeeeeeeee
net Corum?ic.
SyZT0GWC7
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017 pp. 39-57 IEEE Computer Society, 2017. doi: 00.1109/SP.201..90 URL htt.ps://dot
0rg/10.1109/4 2017.49
Pin-Ya Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Z00: zeroth order optimization based black-box attacks to deep neural networks without training substitute mod- els. In Bhavani M. Thuraisingham, Battista Biggio. David Mandell Freeman, Brad Miller, and Arunesh Sinha (eds.), Proceedings of the 10th ACM Workshop on Artificial Intelligence and Se- curity, AlSec @ CCS 2017, Dallas, TX, USA, November 3, 2017, pp. 15-26. ACM, 2017. doi: URL htt.ltttpp//.....eto org. 1141 /2.5///228222222 3140448
10.1145/3128572.3144000
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and
Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEB Computer Society, 2009. doi: 10. 100.0//C...200995266663 1109/CVPE 2009. 02068455555888
URL htthst::: doi. orq/10.
10

---

Under review as a conference paper at ICLR 2022
Xiaoyi Dong, Jiangfan Han, Dongdong Chen, Jiayang Liu, Huanyu Bian, Zehua Ma, Hongsheng Li, Xiaogang Wang. Weiming Zhang, and Nenghai Yu. Robust superpixel-gun guided attentional adversarial attack. In 2020 IEEE/CVP Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 12892-12901. Computer Vision Founda-
tion / IEEE, 2020. doi: CVPR 2020/htm1/Dons Robust Superpixel Superpixel-Guided Guided Attack Cettacc_CVPR... paw.:2220ppppppp_____1_ html
URL htt.ps::/oppeaaaeeco
thecvf. com/ wttp/ottttttttteeeeeeeeee Attentional
Adversarial Anieettioon_AAvvresaiaaaa
Sid Ahmed Fezza, Yassine Bakhti, Wassim Hamidouche, and Olivier Déforges. Perceptual evalu- ation of adversarial attacks for cnn-based image classification. In 11th International Conference on Quality of Multimedia Experience QoMEX 2019, Berlin, Germany, June 5-7, 2019, pp. 1- URL URLhtts::://daa wrg.ei...rr/ln 1109/
6. IEEE, 2019. doi: 10. 10.1109000MX..01.....07 OOMEX. 2019. 8743213
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. URL http: http:://aaxivv.
6572
Chuan Guo, Jared S. Frank, and Kilian Q. Weinberger. Low frequency adversarial perturba- tion. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the T'hirtyFFitth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, volume 115 of Proceedings of Machine Learning Research, pp. 1127-1137. AUAI Press, 2019. URL wwr. www.ppressviil hwtp://wo.aa./ooa///oo//oooo..a. html
Inman Harvey. The microbial genetic algorithm. In George Kampis, István Karsai, and Eörs Sza- thmáry (eds.), Advances in Artificial Life. Darwin Meets von Neumann- - 10th European Confer- ence, ECAL 2009, Budapest, Hungary, September 13-16, 2009, Revised Selected Papers. Part II, volume 5778 of Lecture Notes in Computer Science, pp. 126 133. Springer, 2009. doi: 10.1007/ 978-3-642-21314-4) 16. URL https: //os:////o..o org/ 0. 007 976 --222-11---222---------
16
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICMI. 2018, Stockholmsmüüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2142- 2151. PMLR, 2018. URL
press /t/////000111//// html
Andrew Hyas, Logan Engstrom, and Aleksander Madry. Prior convictions:
Black-box adversarial attacks with bandits and priors. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:
//openreviev net. for d=BkMi BrR-BKKiiHRKKK7 WAR5K7
Eric C. Larson and Damon M. Chandler. Most apparent distortion: full-reference image quality assessment and the role of strategy. J. Electronic Imaging, (911):110066 2010. doi: 10.1117/1. 3267 URL httpt:ttt .rr./.org///i 1117/1 12.11.226777 3267105
Nannan Li and Zhenzhong Chen. Toward visual distortion in black-box attacks. IEEE Transactions on Image Processing, 30:6156-6161 2021.
Xurong Li, Shouling Ji, Meng Han, Juntao Ji, Zhenyu Ren, Yushan Liu, and Chunming Wu. Ad- versarial examples versus cloud-based detectors: A black-box empirical study. IEEE Trans. De- doi: 10. 10.1109//DS..20192222422 1009 TDSC. 201 9. 22 2434467
pendable Secur. Comput., 18(4): 1949, 2021.
URL
htt:::/doo...rrg 10. 1109
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial exam- ples and black-box attacks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httl:tttts///opeereee
net. feet/forumm2 (oormmiid=SsssGggp2
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference
11

---

Under review as a conference paper at ICLR 2022
Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https::/ net/
Forum? SrrummmddrJJ1ff22
Seungyong Moon, Gaon An, and Hyun Oh Song. Parsimonious black-box adversarial attacks via ficient combinatorial optimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Pro- ceedings of the 36th International Conference on Machine Learning, ICML, 2019, 9- June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp- 1636-464, PMLR, 2019. URL htttp://proceediii ngs .gg.nng.mmmm..... mmlr www.p/eessvvv / moon html
Seyed-Mohsen Moosavi-Dezfooli; Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pp. 2574-2582. IEEE Computer Society, 2016. doi: 10.1109/CVPR.206628888 URL https: /
1.09/CVPR.2016.
Nina Narodytska and Shiva Prasad Kasiviswanathan. Simple black-box adversarial perturbations for deep networks. CoRR, abs/1612.06299, 2016. URL http://axxiv.org///tt
06299
Chris Olah, Arvind Satyanarayan. Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. doi: 10.23915/ distill.00010. https://distill, pub/201 8/building-bloee -bocck...
Nicolas Papernot, Patrick D. McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Anan- thram Swami. The limitations of deep learning in adversarial settings. In IEEE European Sympo- sium Security and Privacy, EuroS&P 2016, Saarbrücken, Germany, March 21-24, 2016, pp. 372-387. IEEE, 2016. doi: (0.1109/EurGSP.2016..6. URL hhttts:////.. org /1lgg//1.10 1109/ EurosP 2016.36
Nicolas Papernot, Patrick D. McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Anan- thram Swami. Practical black-box attacks against machine learning. In Ramesh Karri, Ozgur Sinanoglu, Ahmad- Sadeghi, and Xun Yi (eds.), Proceedings of the 2017 ACM on Asia Con- ference Computer and Communications Security, AsiaCCS 2017, Abu Dhabi, United Arab Emirates, April 2-6, 2017, pp. 506-519. ACM, 2017. doi: 10.1145/30529733353300 https: //wo///ii//// www.///g///ooooo.. 10. 10t....11131220 3053009
URL
Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In Xianghua Xie, Mark W. Jones, and Gary K. L. Tam (eds.), Proceedings of the British Machine Vision Conference 2015, BMVC 2015, Swansea, UK. September 7-10, 2015, pp. 41.1 41.12, BMVA Press, 2015. doi: 10.5244/C.29.41 URLhttps. / /ww. www.//o.oooooooooooooooooooooo..
52 5244/444400 29.449....99.... 29.41
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 779-788, IEEE Computer Society, 2016. doi: 10.1 1..119/CVPR.2016.99 22016.91
URL O 1109////
Alexander Schrijver. Theory of linear and integer programming. John Wiley Sons, 1998.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 618-626. IEEE Computer Society, 2017. doi: 10.1109/CCV.2217..... URL
https: /oi. 1tt.......0..0...///////.
1101 ICCV.2017.1.. 2017.74
Mahmood Sharif, Lujo Bauer, and Michael K. Reiter. On the suitability of Ip-norms for creating and preventing adversarial examples. In 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2018. Salt
Lake City, UT, USA. June 18-22, 2018, pp. 1605-1613. Computer Vision Founda- tion IEEE Computer Society, 2018. doi: 10.1109/VVPR.220800022
URL
12

---

Under review as a conference paper at ICLR 2022
http://openaccess httpppanaccsss...cc..eoeee twectf.eec.ccc com conte/conenttttmme evpr_ 2018 workshops/w32 html/ Samm//Shari.. On the Suitability CVPR 2018 paper.htm.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann Le- Cun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014. URL httt:://aaxii// org/
abs/ 1////312. 6034
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Trans. Evol. Comput., 23(5):828 841. 2019. doi: 10.1109/EVV.2019. 2890858. URL
1109/ TEVC. 201 2019.2890858
Qi Sun, Arjun Ashok Rao, Xufeng Yao, Bei Yu, and Shiyan Hu. Counteracting adversarial attacks in autonomous driving. In IEEE/ACM International Conference On Computer Aided Design, ICCAD 2020, San Diego, CA, USA, November 2-5, 2020, pp. 1145/3400302.341575555 URL https://set org / 111. 340055/333002224444
IEEE, 2020. doi: 10.
Fnu Suya, Jianfeng Chi, David Evans, and Yuan Tian. Hybrid batch attacks: Finding black-box adversarial examples with limited queries. In Srdjan Capkun and Franziska Roesner (eds.), 29th USENIX Security Symposium, USENIX Security 2020, August 12-14, 2020, pp. 1327- 1344. USENIX Association, 2020. URL https: //wp://wwwwwwwe www.wuseeix.oo org/ conference///////ooooe ssenxxxccuxttt2///eeeeeee. suya
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http:
//arxiv.org/ abs/ 1112/1112.220
1312. 6199
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re-
thinking the inception architecture for computer vision. In 2016 IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818-2826, IEEE Computer Society, 2016. doi: 10.1109/CPR.2006..300 URL https:
//doi org 10 O. 11 09/CVPR.C..//. 201 308
Chun-Chen Tu, Pai-Shun Ting, Pin- Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsich, and Shin-Ming Cheng. Autozoom: Autoencoder-basee zeroth order optimization method for attacking black-box neural networks. In The Thirty-Third AAAI Conference Artificial Intelli- gence, AAAI 2019, The Thirty- -irst Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2019, Homolulu, Hawaii USA, January 27 February 1, 2019, pp. 742-749. AAAI Press, 2019. doi: www.166/aaai.v3333...... 3301742
https::/ddoi.oroee URL https: /oi. wegii.org//ae http:////oo.0g111111100000 1609, aaai v33i01.
Lina Wang, Kang Yang, Wenqi Wang, Run Wang, and Aoshuang Ye. Mgaattack: Toward more query-efficient black-box attack by microbial genetic algorithm. In Chang Wen Chen, Rita Cuc- chiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann (eds.), MM '20: The 28th ACM International Conference on Multimedia, Virtual Event Seattle, WA, USA, October 12-16, 2020, pp. 2229-222222222222222222222222222 2236. ACM, 2020. doi: 10. 011145/399417.3343377 URL 1 71139494111111733333333
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13(4):600 612, 2004. doi: 11.09//TP.200388999222
hww.attps::/cc..rqqppp 1109/TII 2003. 819861
Tao Xiang, Hangcheng Liu, Shangwei Guo. Tianwei Zhang, and Xiaofeng Liao. Local black- box adversarial attacks: A query efficient approach. CoRR, abs/2101.01033 2021. URL https:
www.iivoorgaabae...... 01032
13

---

Under review as a conference paper at ICLR 2022
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In 25th Annual Network and Distributed System Security Sympo- shum, NDSS 2018, San Diego California. USA, February 18-21, 2018. The Internet Soci- ety, 2018. URL wtR.http:///e /ww.t:://www... s 2tttes/2557c01330///mmm000000000 03A--03---00 Xt__oo_______pppepppee._
/dds////s////// wp content/up oads
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt volume 8689 of Lecture Notes in Computer Science, pp. 818-8888-333 833. Springer, 2014. doi: 10.1007/ 978-3 778-3-3194 10590- -,.33. URL https: /wwi.iiooooooooooo..oo www./////....o.. 101000...009990 3 313 10590- 1
53
Hanwei Zhang, Yannis Avrithis, Teddy Furon, and Laurent Amsaleg, Smooth adversarial examples. EURASIP J. Inf. Secur. 2020:15, 2020. doi: 10.1
URL https://
doi. ooi.org/10.111 111.11...1111111 2e/6//113333555 020-02000220 00112-2
Ting Zhao and Xiangqian Wu. Pyramid feature attention network for saliency detection. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 3085-3094. Computer Vision Foundation IEEE, URL
2019. doi: 10. (01109//CVPR.2219.0022
thecvf. com/ Feature Attention Network. pappp.ppppp... html
content_CvVi Seent__vVRR_2211330 2VFPR22019/ttmmm2neee http://hhhappaaaaidaaa for Saliency_Doteeccionoccc.. Feauur
Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pp. 2921-2929. IEEE Computer doi 00..//00...0/01.111////2222.
Society, 2016. doi: 10. 1091 URL
2016.319
A APPENDIX
A.1 MAD METRIC
Most Apparent distortion (MAD) metric (Larson & Chandler 2010) is one of the state-of-the-art art full-reference image quality assessment methods. MAD attempts to merge two separate strategies for two kinds of distorted images respectively. For high-quality images with thar-threshold distor- tion (just visible), MAD focuses on detection-based strategy to look for distortions in the presence of the image. While for low-quality images with suprathreshold distortion (clearly visible), MAD focuses on appearance-basel strategy look for image content in the presence of the distortions. MAD will control the weight of two strategies according to the type of distorted images. The calcu- lation process of MAD can be summarized following steps and we recommend interested readers to read the original literature.
1. Compute locations visible distortions based on luminance images.
2. Combine the visibility map with local error image.
3. Decompose both the distorted and original images into log-Gabor subbands.
4. Calculate different statistics of each subband.
5. Calculate the adaptive blending score.
14

---

Under review as a conference paper at ICLR 2022
A.2 SCATTER PLOT OF MAD SCORES
Equipped with segmentation priors, 68. 1% and 94.3% examples of Parsimonious-seg attack and Square-seg attack have better MAD scores compared with their original version respectively. Since Parsimonious Attack uses greedy local search while Square attack adopts random search, it is obvious that limiting the search in salient regions more helpful to Square Attack.
100
100
20
40
60
30
100
130
20
40
30
120
130
P Parsimoricus stack
Square aback
Figure 9: MAD scores of Parsimonious Attack vs. Parsimonious--ee attack and Square Attack vs. Square-seg Attack.
A.3
THRESHOLD FOR MAD METRIC
(a) MAD=50
(b) MAD=40
(c) MAD=30
(d) MAD=20
(e) MAD=10
(O Original
Figure 10: Adversarial examples generated by boundary attack with different MAD scores. We can find the threshold MAD < 30 is roughly enough to indicate imperceptible adversarial example.
A.4 TRUE SUCCESS RATE VERSUS QUERY NUMBER PLOT
Here we further present comparison SR versus query number under different MAD thresh- olds. It shows that Square Attack and Parsimonious Attack generate some successful AEs with
15

---

Under review as a conference paper at ICLR 2022
very few queries due to global perturbations and some tricks like initializing perturbations with dom vertical stripes (Andriushchenko et al. 2020), but they lack the ability to further improve the imperceptibility. Instead, Saliency Áttack conservatively select small regions to perturb, hence the query efficiency is a little lower than Square Attack and Parsimonious Attack at the beginning but soon exceeds them.
-.R-ouery convergence under
5R-query convengence giot under
ca-oourry convergence pletuncer
06
TVDDA
-VVBB
TV3BA
Sqsar
Szaum
Sigam
Saxaro-9g
Sqpare-cyy deg
0.4
0.5
Part monius--rg
05
Salancy
2.4
0.4
011
0.5
3.5
2.5
1.5
255
0.5
3.5
R.Query
-10
Equeryy
4-uuryy
X10
Figure 11: Queries vs. True Success Rate under different thresholds of MAD scores
A.5
COMPARISON OF SALIENCY ATTACK WITH OTHER ATTACKS
We randomly choose 10,000 examples from the ImageNet validation set and divide them into 10 groups. Then we test different attacks under 3000 query budget and report their mean and stan- dard deviation (SD). The best results are recorded in bold based Wilcoxon signed-rank test with significance level at 0.05. We can find that our Saliency Attack is significantly better than all the baselines.
Table 5: Comparison of Saliency Attack with other attacks under 3000 query budget.
Method
SR + SD
SRame + SD
Lg + SD
Lo + SD
MAD + SD
TVDBA
90.1%
20.45
10.64:0.153
19.0%=10.005
37.65+3.631
Parsimonious
92.4%:10.007
14.7%=+0.008
22.174:0.000
100.0%:+0.000
51.80=0.1115
Parsimonious-seg
88.0%
11.4%:00.014
13.54 +0.497
35.0%
45.3110.212
Square
98.3%
2.704
25.34 #0.021
99.0%=11.001
57.29
Square-seg
80.044
20.022
34.5%-10.028
38.56:10.538
Saliency (ours)
84.10
79.8% +00009
3.44
2.9% + 0.001
12.07+
A.6
GREEDY SEARCH IN SALIENT REGION WITH DIFFERENT BLOCK SIZES
We test different block sizes for greedy search in salient region, and choose the best one to be compared in ablation study.
Table 6: Greedy search salient region with different block sizes.
Block size
Avg. queries
SR
SR
Lg
Lo
MAD
128
20.6%
18.4%
7.32
12.8%
11,50
64
512
37.4%
31.3%
6.37
10.2%
15.18
32
2727
56.0%
50.7%
4.37
4.7%
12.87
16
4039
35.4%
35.2%
1.79
0.8%
4.84
16


Reviews:
[{"review_id": "s_0iyYVFiYG", "reviewer": "Reviewer_zKra", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a few optimisations that can improve the imperceptibility of generated adversarial perturbations. They also demonstrate that some of the exiting black-box adversarial attacks can benefit from their optimisations. In addition to this, they also propose a search algorithm that can further narrow down the candidate regions for adversarial perturbations. They also demonstrate that adversarial examples generated by their technique have a higher success rate for evading detection mechanisms.\n \nIn the recent past, a richer class of gradient-free black-box techniques have been proposed in the literature. Why is the paper considering only two of those? For instance, why not consider techniques based on Bayesian optimisation. The same applies to methods for detecting adversarial examples. Why was Feature Squeezing chosen as a baseline to evaluate the success rate?\n\nIn addition to the above, the author(s) report the results only over a sample of 1000 images from ImageNet, why only such a small sample was considered, and I would also encourage the authors to evaluate over multiple samples & report the variance too.", "summary_of_the_paper": "There has been a lot of interest in improving the query efficiency of black-box attacks in the recent past. However, these techniques produce examples that a human in the loop can quickly identify. The authors propose using segmentation priors to improve the black-box attacks so that the perturbations are restricted to the salient regions of the image. In addition to this, they also present a technique that improves the imperceptibility without forgoing the query efficiency.", "summary_of_the_review": "In general, I found the paper easy to follow. I also find the direction explored by the paper to be quite promising but found the experiments & the baselines chosen by the authors are lacking. I would encourage the authors to include a diverse set of baselines. For instance, I would also demonstrate how the optimization proposed in the paper improves gradient-free black-box adversarial example generation techniques other than the chosen few. Also, evaluate the evasion rate of adversarial examples generated by the proposed technique against a diverse set of detection mechanisms.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZpbRwgNaZnJ", "reviewer": "Reviewer_2cEz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work tackles an important problem, that is, developing an imperceptible black-box attack. However, there are two main issues concerning the technical approach and the experimental analysis.  \n\n- From a technical point of view the search algorithm devised to refine perturbation is exceedingly naive. It is based on an iterative procedure that resembles quadtree analysis in image processing. The first split of the salient region generates four initial blocks, and a perturbation is added to them to find those that maximize the loss function. After finding the best block, it is further split and the procedure is repeated until the block is reduced to one pixel or no smaller blocks can achieve a better loss. Then, this same procedure is applied again to the second best block, etc. As it is described, this procedure has neither a theoretical nor an experimental justification. In addition, a similar idea but applied in a different context can be found in Guo et al. (2020) and saliency maps to add adversarial attacks are also explored in Sun et al. (2021). Overall, I feel that the contribution is not significant in terms of technical approach.\n\n- The approach is not compared with other black-box attacks that also have the objective to make the perturbation imperceptible, see references below. In fact, experiments are carried out by modifying some SOTA approaches by including saliency regions and comparing them with the proposed solution (see Table 2). In my opinion this is too limited to validate the proposal. \n\nTypos: imperceptiblity, tunning\n\nReferences\n- Guo et al. Watch out! Motion is Blurring the Vision of Your Deep Neural Networks, NeurIPS 2020\n- Sun et al. Generating facial expression adversarial examples based on saliency map, Image and Vision Computing 2021\n- Wang et al. Perception Improvement for Free: Exploring Imperceptible Black-box Adversarial Attacks on Image Classification, arXiv 2020\n- Liu et al. GreedyFool: Multi-Factor Imperceptibility and Its Application to Designing Black-box Adversarial Example Attacks, arXiv 2020\n- Croce and Hein, Sparse and Imperceivable Adversarial Attacks, ICCV 2019\n- Gragnaniello et al. Perceptual quality-preserving black-box attack against deep learning image classifiers, Pattern Recognition Letters 2021\n- Li and Chen, Toward Visual Distortion in Black-Box Attacks, IEEE Transactions on Image Processing 2021\n\n\n=========== Post rebuttal comments ===========\n\nI appreciate that the authors better clarified their contribution and included a new comparison in the experimental section, hence I increase my score. However, I still believe that the paper needs much more comparisons with state-of-the-art and that the proposal should be better justified from a theoretical point of view. ", "summary_of_the_paper": "This paper proposes a black-box attack where, by relying on segmentation priors, the perturbation is applied only in the salient region. This allows one to obtain reduce perceptibility with a limited number of queries and a small reduction in success rate. More specifically, once the salient region has been identified, a refining procedure is carried out to find small areas where the perturbation should be added. Experiments are performed on ImageNet and results are compared with those of some SOTA methods and their variants that work on saliency regions.", "summary_of_the_review": "In my opinion the technical novelty introduced in this paper is too limited and also its experimental validation should be improved by considering more relevant methods for comparison.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I3qabbBLowe", "reviewer": "Reviewer_Hs6q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic of imperceptible black box is very interesting and not well studied, the ideas of generating perturbing blocks in salient regions are interesting, however, this paper can be improved with more sufficient justifications on motivations and experiments. Moreover, its technical contribution is not very high.\n\nThe motivation of generating the perturbed regions in the salient object region is not sufficiently justified: salient object can still be mostly a\nsmooth low frequency area, e.g., smooth human face/skin, and hence generating the perturbation in this area may not be reasonable. Moreover, it is not clear whether generating perturbation in salient object area makes more sense than generating perturbations in the non-smoothness, high frequency, or  heavy texture area. \n\nThe experiment part is not totally convincing. 1) There is only one data set, which is 1000 images from ImageNet. However, what these images look like will affect the results a lot. For example, does most salient objects have heavy texture area? 2) Also, there is no comparison to the related method in approach of Zhang (2020). 3) I can not find the information of which classification network is used in the experiments. Are multiple networks used in the experiments?\n\nThis paper used an existing salient object segmentation approach, and so its major technical contribution is the tree based method to generate perturbation blocks, which is however a little straightforward and ad-hoc.", "summary_of_the_paper": "This paper proposed a method to generate imperceptible attack in black box attack scenario by generating local perturbation blocks in salient regions. It used salient object segmentation to obtain the salient region, then applied a tree search method to find smallest blocks within the salient region that can cause the maximal change in predicted class logits. Experiments on 1000 Imagenet examples are conducted, compared to several existing baselines, showing that the proposed method can improve achieve more imperceptible attacks (where imperceptibility is measured by metric MAD).", "summary_of_the_review": "Overall, I liked the idea of this paper, and enjoyed reading it. However, it needs better justification on motivations, as well as improvement on experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BaHEQGsYeo", "reviewer": "Reviewer_3FMW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The proposed method is simple and achieves the goal of improving the imperceptibility of the perturbations: using the segmentation prior is effective for avoiding changes on the background, and the Saliency Attack attains better MAD score.\n\n- Several ablation studies are presented to illustrate the proposed method.\n\nWeaknesses\n- The paper focuses on improving the imperceptibility of $\\ell_\\infty$-attacks with a fixed budget $\\epsilon=0.05$, in practice reducing the number of pixels perturbed (a sort of minimization of the $\\ell_0$-norm). However, I think that how visible the perturbations are is more a property of the threat model (set of feasible changes) rather than of the attack used: for example, using a smaller $\\epsilon$ for the same attacks would increase their imperceptibility. Moreover, other $\\ell_p$-norms (e.g. Square Attack has versions for $\\ell_2$ and $\\ell_1$ [A]), including $\\ell_0$, or perceptual metrics like LPIPS [B] should be considered, since they produce more localized changes than $\\ell_\\infty$-attacks.\n\n- There are a few works which aim at finding which areas of an image the attacker should perturb to have invisible changes, both with black- and white-box attacks [C, D]. In particular, [C] show that small perturbations in the $\\ell_\\infty$-norm are not necessary if limited to certain areas to the image to preserve imperceptibility.\n\n- The presentation of the method in Algorithms 1 and 2 seems a bit confuse: first, $\\delta$ is not defined. Second, the \"Refine\" function is called recursively on smaller blocks, but also repeatedly with smaller $k$ in Algorithm 2: is this correct? Also, in Line 9 $k$ is halved: shouldn't it be restarted from the original value when going over the next iteration of the loop in L4?\n\n- From the images in Figure 4 it seems that for Parsimonious and Square Attack perturbations are sampled also on areas which are outside the mask given by the segmentation prior: are those candidates evaluated, spending queries of the total budget (although they can't change the current perturbations)?\n\n[A] https://arxiv.org/abs/2103.01208\n[B] https://arxiv.org/abs/2006.12655\n[C] https://arxiv.org/abs/1909.05040\n[D] https://arxiv.org/abs/2010.13773", "summary_of_the_paper": "The paper studies how to reduce the perceptibility of the perturbations to the original images produced by black-box adversarial attacks (for the $\\ell_\\infty$-threat model). In particular, it proposes to use a prior based on segmentation techniques to localize the changes on the subject of the image and leave the background unaltered. Moreover, the Saliency Attack is introduced to further reduce the fraction of the original image which is modified to induce misclassification.", "summary_of_the_review": "Overall, I think the proposed method is effective, but, especially given that the technical novelty is limited, it should be better positioned: if the main goal is imperceptibility of the perturbations it should be compared to other kinds of attacks beyond $\\ell_\\infty$ ones. Otherwise, the authors should better motivate why improving the $\\ell_\\infty$-attacks with such fixed threshold is relevant.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose to use segmentation priors for black-box attacks such that the perturbations are limited in the salient region. They also find that state-of-the-art black-box attacks equipped with segmentation priors can achieve much better imperceptibility performance with little reduction in query efficiency and success rate. Hence, the auithors propose the Saliency Attack, a new gradient-free black-box attack, that can further improve the imperceptibility by refining perturbations in the salient region.
The reviewers think that the proposed method is simple and important, and the authors have responded properly to some comments. 
However, the reviewers still are not satisfied with the experimental evaluation and comparisons, as the authors can only try to compare with other ideas and test more models in the future.
In summary, I think the manuscript at its current staus cannot be accepted.
Decision: Reject