Towards Robustness against Unsuspicious Adversarial Examples
Authors: Liang Tong
Minzhe Guo
Atul Prakash
Yevgeniy Vorobeychik
Venue:  (2021)

Abstract: Despite the remarkable success of deep neural networks, significant concerns have emerged about their robustness to adversarial perturbations to inputs. While most attacks aim to ensure that these are imperceptible, physical perturbation attacks typically aim for being unsuspicious, even if perceptible. However, there is no universal notion of what it means for adversarial examples to be unsuspicious. We propose an approach for modeling suspiciousness by leveraging cognitive salience. Specifically, we split an image into foreground (salient region) and background (the rest), and allow significantly larger adversarial perturbations in the background, while ensuring that cognitive salience of background remains low. We describe how to compute the resulting non-salience-preserving dual-perturbation attacks on classifiers. We then experimentally demonstrate that our attacks indeed do not significantly change perceptual salience of the background, but are highly effective against classifiers robust to conventional attacks. Furthermore, we show that adversarial training with dual-perturbation attacks yields classifiers that are more robust to these than state-of-the-art robust learning approaches, and comparable in terms of robustness to conventional attacks.

Under review as a conference paper at ICLR 2021
TOWARDS ROBUSTNESS AGAINST UNSUSPICIOUS AD- VERSARIAL EXAMPLES
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the remarkable success of deep neural networks, significant concerns have emerged about their robustness to adversarial perturbations to inputs. While most attacks aim to ensure that these imperceptible, physical perturbation attacks typically aim for being unsuspicious, even if perceptible. However, there is no universal notion of what it means for adversarial examples be unsuspicious. We propose an approach for modeling suspiciousness by leveraging cognitive salience. Specifically, we split an image into foreground (salient region) and background (the rest), and allow significantly larger adversarial perturbations in the background, while ensuring that cognitive salience of background remains low. We describe how to compute the resulting on classifiers. We then experimentally demonstrate that our attacks indeed do not significantly change perceptual salience of the background, but are highly effective against classifiers robust to conventional attacks. Furthermore, we show that adversarial training with durluueett attacks yields classifiers that more robust to these than state-of-the-art robust learning approaches, and comparable terms of robustness to conventional attacks.
dual-perturbation attacks
INTRODUCTION
An observation by Szegedy al. (2014) that state-of-the-art deep neural networks that exhibit excep- tional performance in image classification are fragile in the face of small adversarial perturbations of inputs has received a great deal of attention. series of approaches for designing adversarial examples followed (Szegedy et al. 2014; Goodfellow et al., 2015 Carlini & Wagner 2017), along with methods for defending against them (Papernot | [0166]] Madry al. [2018). and then new attacks that defeat prior defenses, and so on. Attacks can be roughly classified along three dimensions: 1) introducing small g--norm-bounded perturbations, with the goal of these being imperceptible to humans (Madry et al., 2018). 2) using non-l,- based constraints that capture perceptibility (often called semantic perturbations) (Bhattad et al.) 2020), and 3) modifying physical objects, such as stop signs (Eykholt al. 2018), way that does arouse suspicion. One of the most common motivations for the study of adversarial examples is safety and security, such as the potential for attackers to compromise the safety of autonomous vehicles that rely on computer vision (Eykholt) et al. 2018). However, while imperceptibility is certainly sufficient for perturbations to be unsuspi- cious, it is far from necessary, as physical attacks demonstrate. On the other hand, while there are numerous formal definitions that capture whether noise is perceptible (Moosavi-Deztone et al. 2016 Carlini & Wagner 2017). what makes adversarial examples suspicious has been largely informal and subjective.
We propose a simple formalization of an important aspect of what makes adversarial perturbations unsuspicious. Specifically, we make distinction between image foreground and background, allowing significantly more noise in the background than the foreground. This idea stems from the notion of cognitive salience (Borji al. 2015 Kmmerer et al. 2017 He & Pugeault 2018). whereby an image can be partitioned into the two respective regions to reflect how much attention a human viewer pays to the different parts of the captured scene. In effect, posit that perturbations in the foreground, when visible, will arouse significantly more suspicion (by being cognitively more salient) than perturbations made in the background.

---

Under review as a conference paper at ICLR 2021
Original
Foreground Mask
purluueeturbation Example (i
Dual-perturbation Example (A 10)
Sample
Figure An illustration of dual-perturbation attacks. Adversarial examples are with large fxo perturbations the background (t 20/2555 and small fx. perturbations on the foreground (EF = 4 4/255). A parameter used to control background salience explicitly. A larger À results in less salient background under the same magnititude of perturbation.
Our first contribution is a formal model of such puul-ppetuubbaionn attacks, which is a generalization of the lp- -p-norm-buu -oorm-bounded attack models (see, e.g., Figure D but explicitly aims to ensure that adversarial perturbation does not make the background highly salient. Second, we propose an algorithm for finding adversarial examples using this model, which is adaptation of the PGD attack (Madry et al.] 2018). Third, we present a method for defending against dual-perturbation attacks based on the adversarial training framework (Madry et al.) 2018). Finally, we present an extensive experimental study that demonstrates that (a) the proposed attacks are significantly stronger than PGD, successfully defeating all state-of-the-an art defenses, (b) proposed defenses using our attack model significantly outperform state- of offtthe-art art alternatives, with relatively small performance degradation on non- adversarial instances, and (c) proposed defenses are comparable to, or better than alternatives even against Iraditional attacks, such as PGD.
Related Work: Recent studies have shown that neural networks are vulnerable to adversarial
examples, A variety of approaches have been proposed to produce adversarial examples (Szegedy
cct al. 2014, Goodfellow et al. 2015 Papernot et al.) 2016a Moosavi-Dezionn et al. 2016 Carlini & Wagner 2017). These approaches commonly generate adversarial perturbations within bounded 1p norm so that the perturbations are imperceptible. A related thread has considered the problem of generating adversarial examples that are semantically imperceptible without being small in norm (Brown et al.] 2018) Bhattad et al. (2020), for example, through small perturbations to the color of these account for the perceptual distinction between the foreground and background of images.
scheme. However,
Numerous approaches have been proposed for defending neural networks against adversarial exam- plcs PPapernot et al. [2016b] Carlini & Wagner 2017 Madry et al. 2018 Cohen et al.) [2019] Madry
al. 2018: Raghunathan et al.J 2018). Predominantly these use t boundddd perturbations as the threat model, and while some account for semantic perturbations (e.g. Mohapatra et al. (2020)), none
consider perceptually important difference in suspiciousness between foreground and background.
Two recent approaches by Vaishnavi et al.) (2019) and Brama & Grinshpoun (2020) have the strongest conceptual connection to our work. Both are defense-focused by either eliminating (Vaishnavi et al.. 2019) or blurring (Brama & Grinshpoun 2020) the background region for robustness. However, they assume that we can reliably segment an image at prediction time, leaving the approach vulnerable to attacks image segmentation (Amab et al. 2018). Xiao et al. (2020) propose to disentangle foreground and background signals on images but unsuspiciousness of their attacks is not ensured.
2 BACKGROUND
2.1 ADVERSARIAL. EXAMPLES AND ATTACKS
The problem of generating adversarial examples is commonly modeled as follows. We are given a learned model he(.) parameterized by which maps an input æ to a k-dimennional prediction, where is the number of classes being predicted. The final predicted class Чμ is obtained by yp = arg max, hx),), where (x); is the ith element of he (z). Now, consider an input æ along with correct label y. The problem of identifying adversarial example for can be captured by
2

---

Under review as a conference paper at ICLR 2021
Original Sample
corrupted Background
Corrupted Toreground
Semantic distinction between foreground and background. Left: Original image of bears. Middle: Adversarial example with & bounded perturbations 40/255) on the background, the sematic meaning (bear) is preserved. Right: Adversarial example with & bounded perturbations = 40/255) on the foreground, with more ambiguous semantics.
Figure 2:
the following optimization problem:
max L (ho(a +)),))
(I)
BEA(c)
where L(-) is the adversary's utility function (for example, the loss function used to train the classifier he). (ee) is the feasible perturbation space which is commonly represented l ball: A(e) = {ō: ||0||p S c}.
number of approaches have been proposed to solve the optimization problem shown in Eq. ① among which two are viewed as state of the art: CW attack developed by Carlini & Wagner (2017). and Projected Gradient Descent (PGD) attack proposed in Madry et al. (2018). In this work, we focus on the PGD attack with loo and &g as the distance metrics.
2.2 ROBUST LEARNING
An important defense approach that has proved empirically effective even against adaptive attacks is adversarial training (Szegedy et al. (2044: Cohen et al. [2019] Goodfellow et al. [2015] |Madyy et al.)
2018). The basic idea of adversarial training is to produce adversarial examples and incorporate these into the training process. Formally, adversarial training aims to solve the following robust learning problem:
min
max L (ho(w 8),
a.EED 101|
where is the training dataset. In practice, this problem is commonly solved by iteratively using the following two steps (Madry al. (018):: 1) use PGD (or other) attack to produce adversarial examples of the training data; 2) use any optimizer to minimize the loss of those adversarial examples. It has been shown that adversarial training can significantly boost the adversarial robustness of classifier against 4p attacks, and it can scaled to neural networks with complex architectures.
3
DUAL-PERTTUBATION ATTACKS
3.1 MOTIVATION
Our threat model is motivated by the feature integration theory Treisman & Gelade 1980) in cognitive science: regions that have features that are different from their surroundings are more likely catch a viewer's gaze. Such regions are called salient regions, or foreground, while the others are called background. Accordingly, for given image, the semantics of the object of interest is more likely to be preserved in the foreground, as it catches more visual attention of a viewer compared to the background. If the foreground of image is corrupted, then the semantics of the object of interest broken. In contrast, the same extent of corruption in the background nevertheless preserves the overall semantic meaning of the scene captured (see, e.g., Figure 2). Indeed, detection of salient regions, as well as the segmentation foreground and background, have been extensively studied in computer vision (Borji al. 2015). These approaches either predict human fixations, which are sparse bubble-like salient regions sampled from a distribution (Kmmerer et al.) 2017), or salient objects that contain smooth connected areas in image (He & 2018)
3

---

Under review as a conference paper at ICLR 2021
Despite this important cognitive distinction between foreground and background, essentially all of the attacks on deep neural networks for image classification make no such distinction, even though number of other semantic factors have been considered (Bhattad et al., 2020 Mohapatra et al.) 2020). Rather, much of the focus has been on adversarial perturbations that are not noticeable to human, but which are applied equally to the entire image. However, in security applications, the important issue is not merely that an attack cannot be noticed, but that whatever observed is nor suspicious. This is, indeed, the frame of reference for many high-profile physical attacks on image classification, which are clearly visible. but not suspicious because they hide in the "human psyche", that is, are easily ignored (Sharif al.) [016] Eykholt et al.) 2018). The main goal of the threat model we introduce next is therefore to capture more precisely the notion that an adversarial example is not suspicious by leveraging the cognitive distinction between foreground and background of an image.
3.2 DUAL- PERTURBATION ATTACKS
At the high level, our proposed threat model involves producing small (imperceptible) adversarial perturbations in the foreground of an image, and larger perturbations in the background. This can be done by incorporating state-of-the- the foreground, and another with a large e in the background. Consequently, we term our approach -ural-perturbaatin attacks. Note that these clearly generalize the standard small-norm (e.g., PGD) attacks, since we can set the 6 to be identical in both the foreground and background. However, the key consideration is that after add the large amount of noise to the background, must ensure that we do not thereby make it highly salient to the viewer. We capture this second objective by including in the optimization problem a salience term that decreases with increasing salience of the background.
attacks into method: we can use one attack with small in
Formally, the nual-ppeturbation attack solves the following optimization problem:
max
L (ho + 6), v) + X S (x 6),
(3)
(6.
where S (ax 5) measure the relative salience of the foreground compared background after adversarial noise 8 has been added, with À a parameter that explicitly balances the two objectives: maximizing predicted loss on adversarial examples, and limiting background salience (compared to foreground) so that the adversarial example produced is unsuspicious. Here F returns the mask matrix constraining the area of the perturbation in the foreground, and B returns the mask matrix restricting the area of the perturbation in the background, for an input image æ. F(x) and B(x)) have the same dimension as x and contain Is in the area which can be perturbed and Os elsewhere. denotes element-wist indicates that any input image can be decomposed into two independent images: one containing just the foreground, and the other containing the background.
multiplication for matrices. Hence, we have
F(x) + B(xx) which
We model the suspiciousness S(w) of an input image x by leveraging a recent computational model of image salience, DeepGaze II (Kmmerer et al. 2017). DeepGaze II outputs predicted pixel-level density of human fixations on an image with the total density over the entire image summing to I. Our measure of relative salience of the foreground to background is the foreground score, which is defined as S(x)= = Eie (A) [()))))))))))) $;, where s, is the saliency score produced by DeepGaze II for pixel i of image æ. Since foreground, as a fraction of the image, tends to be around 50-60%, a score significantly higher than 0.5 indicates that predicted human fixation is relatively localized to the foreground.
natural approach for solving the optimization problem shown in Equation is apply an iterative method, such as the PGD attack. However, the use of this approach poses two challenges in our setting. First, as in the PGD attack, the problem is non-convex, and PGD only converges a local optimum. We can address this issue by using random starts, i.c., by randomly initializing the starting point of the adversarial perturbations, as in |Madry et al. (2018). Second, and unlike PGD, the optimization problem in Equation 3 ijvvolvess two hard constraints 1 o F(x))lp S or and ||8 B(x))lp I <p. Thus, the feasible region of the adversarial perturbation ò is not an E, ball, which makes computing the projection P. computationally challenging in high-dimensional settings. To address this challenge, we split the dual- setuuleeeturaatonn attack into two individual processes in each iteration, one for the adversarial perturbation in the foreground and the other for the background. and then merge these two perturbations when computing the gradients, like a standard PGD attack. Full details of our algorithms for computing dual perturbation examples are provided in Appendix A.
4

---

Under review as a conference paper at ICLR 2021
Now, the question that remains is how to partition input image z into foreground, F(æ)), and background, (x). We address this next.
3.3 IDENTIFYING FOREGROUND AND BACKGROUND
Given input æ, we aim compute F(x), the foreground mask and (ax), the background mask. We consider two approaches for this: fixation prediction and segmentation.
Our first method leverages the fixation prediction approach (Kmmerer et al., 2017 to identify foreground and background. This enables general approach for foreground-backgree as fixation predictions are not limited to any specific collection of objects. Specifically, we first use DeepGaze II (Kmmerer et al. 2017) to output predicted pixel-level density of human fixations on an image. We then divide the image into foreground and background by setting threshold t= 0.5 (8min (æ) + (m)) for each input image where (5min, Smaz) are the minimum and maximum values of human fixation on pixels of æ. Pixels with larger values than & are grouped into the foreground, and the others are identified as background subsequently.
partition
Our second approach is to make use of semantic segmentation to provide a partition of the foreground and background in pixel level. This can be done in two steps: First, we use state-of-the-ar paradigms for semantic segmentation ing object, as there might be multiple objects in an image. Next, we identify the pixels that belong to the object of interest as the foreground pixels, and the others as background pixels.
Long et al. (2015) to identify pixels that belong to each correspond-
We use both of the above approaches in duallpeerurbation attacks when evaluating the robustness of classifiers, as well as designing robust models. More details are available Section 5
4 DEFENSE AGAINST DUAL-PERTURBATUN ATTACKS
Once we are able to compute the dual-perturbe peatuueeturration attack, we can incorporate it into conventional adversarial training paradigms for defense, as it has been demonstrated that adversarial training is highly effective in designing classification models that are robust to a given attack. Specifically, we replace the PGD attack in the adversarial training framework proposed by Madry al. (2018), with the proposed durtr-pertrbattio attack. We term this approach AT-Dual, which aims to solve the following optimization problem:
min D
Σ
L (ho(# + +8),)) S (x [60B(z))l,
max (80F()) Sep,
(4)
S.EED
Note that AT--Dul needs to identify background and foreground for any input when solving the inner maximization problems Equation at training time. At prediction time, our approaches classify test samples like any standard classifiers, which is independent of the semantic partitions so as to
close the backdoors to attacks on object detection approaches (Xie al.) 2017). We evaluate the effectiveness of our approaches in Section 5
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets. We conducted the experiments on the following three datasets (detailed in Appendix B): The first is Segment-6 (Cong & Prakash] 2019). which are images with 32 x 32 pixels obtained by pre-processing the Microsoft COCO dataset (Lin et al. 2014) to make it compatible with image classification tasks. We directly used the semantic segmentation based foreground masks provided in this dataset. Our second dataset is STL-10, a subset that contains images with 96 x 96 pixels. Our third dataset is ImageNet- 10, 10-class subset of the ImageNet dataset (Deng et al.) 2009). We cropped all its images to be with 224 x 224 pixels. For STL-10 and ImageNet- 10, we used fixation prediction identify foreground and background described in Section 3
Baselines. We consider PGD attack as a baseline adversarial model, and Adversarial Training with PGD Attacks baseline robust classifier. We also consider classifier trained on non-adversarial
5

---

Under review as a conference paper at ICLR 2021
data (henceforth, Clean). Additionally, consider Randomized Smoothing (Cohen et al. 2019 and defer the corresponding results to Appendix J.
Evaluation Metrics. We two standard evaluation metrics for both attacks and defenses: I) accuracy of prediction on clean test data where no adversarial attacks were attempted. 2) adversarial accuracy, which is accuracy when adversarial inputs are used in place of clean inputs.
Throughout our evaluation, we used both & and & norms to measure the magnitude of added adversarial perturbations. Due to space limitations, model and classification models that are trained to be robust to &2 norm attacks using the ImageNet-10 dataset. The results for lx norm and other datasets are similar and deferred to Appendix.
only present experimental results of the Clean
In the following experiments, all classifiers were trained with 20 epochs on a ResNet34 model (He) al. 2016) pre-trained on ImageNet and with customized final fully connected layer. Specifically, we trained AT-PGD by using 50 steps of Eg PGD attack with e 2.0, and AT-Dual by using 50 steps of lz puttrbeeturbation attack with (B. }} {2.0, 20.0, 0.0} at each training epoch. At test time, we used both &g PGD and dual purtuurettrnation attacks with 100 steps to evaluate robustness.
5.2 SALIENCY ANALYSIS OF
ADVERSARIAL EXAMPLES
We begin by considering a natural question: is our particular distinction between foreground and background actually consistent with cognitive salience? In fact, this gives rise to two distinct considerations: 1) whether foreground as we identify it in fact significantly more salient than the background, and 2) if so, whether background becomes significantly more salient as result of our dual-pent perturrattnnninon attacks. We answer both of these questions by appealing to DeepGaze (Kmmerer cet al. 2017) compute the foreground score (FS) of qual-perturbation examples as described in Section 3 and using the accuracy of different classifiers on putt-ppeturbationn examples with different background salience.
ImageNett100 la dual- polllputturbatiin
ImageNet perturbation
1.0
AT-
ATFCD
1.0
Foreground
0.7
0.6
0.4
0.2
0.6
0.0
1.5
10.0
A-3
A-10 A-1
A
Figure 3: Saliency analysis. Dual-perturbation attacks are performed by using (cr,<B) (2.0, 20.0] and a variety of X displayed in the figure. Left: foreground scores of -url-perturbation examples in response to different classifiers. Right: accuracy of classifiers on nual-perturbation examples with salience control.
Figure 3 presents the answer to both of the questions above. First, observe that in Figure 3 FS (vertical axis) is typically well above 0.5, and in most cases above 0.9, for all attacks. Second, this is true whether we attack the Clean model, or either AT-PGD or AT-Dual robust models. Particularly noteworthy, however, is the impact that the parameter À has on the FS, especially when robust classifiers are employed. Recall that À reflects the relative importance of salience in generating adversarial examples, with larger values forcing our approach to pay more attention to preserving unsuspiciousness of background relative to foreground. As we increase X, we note significantly higher FS, i.e., lower background salience (again, Figure left). Figure Tfffrss a visual illustration of this effect.
As significantly. Figure 3 (right) shows that moderately increasing À does not significantly reduce the effectiveness of the attack, on either the Clean or the robust classifiers.
5.3
ATTACKS ON ROBUST CLASSIFIERS
Next, we evaluate the effectiveness of pertuurrtt attacks against state-of sfate-of-the-ee thet robust learning methods, well the effectiveness of adversarial training that uses pertrrrottnnattion attacks for

---

Under review as a conference paper at ICLR 2021
Clpn
X3-05 kl0ml
1.
3.6
2.0 3.0
10.0
23.0
30.5
4..0
1.0 3.0
4.5
Figure 4: Robustness to white-box E2 attacks ImageNet-10. Left: dual-perturbation attacks with different foreground distortions. Eg is fixed to be 20.0 and X = 1.0. Middle:dual Micdlle:duull-eetuurbttoo: perturbation attacks with different background distortions. ey fixed to be 2.0 and A 1.0. Right: PGD attacks.
tragiMo0211 Pip
1rageter02
Irageneat
Figure 5: Robustness additional white-box attacks ImageNet- 10. Left: 20 steps of loo PGD attacks. Middle left: 20 steps of foc dual-perturbation attacks with different foreground distortions. en is fixed be 20/255 and À 1.0. Middle right: 20 steps of Loc dual-perturbation attacks with different background distortions. ey is fixed to be 4/255 and À = 1.0. Right: Eo JSMA attacks.
generating adversarial examples. We begin by considering white-box attacks, and subsequently evaluate transferability. Due to space limitations, we defer the results of transferability to Appendix D.
The results for white-box attacks are presented in Figure First, consider the cuuullpeturbbtion attacks (left and middle plots). Note that in all cases these attacks are highly successful against the baseline robust classifier (AT-PGD); indeed, even relatively small levels of foreground noise yield near-zero accuracy when accompanied by sufficiently large background perturbations. For example, when the perturbation to the foreground is eF 2.0 and background perturbation is en 20.0, AT-PGD achieves robust accuracy below 10%. In contrast, AT-Dual remains significantly more robust, with an improvement of up to 40% compared to the baseline. Second, consider the standard PGD attacks (right plot). can be observed that all of the robust models are successful against the & PGD) attacks. However, our defense exhibit moderately higher robustness than the baselines under large distortions of PGD attacks, without sacrificing much accuracy on clean data. For example, when the perturbation of the Eg PGD attack is above = 3.0, AT-Dual can achieve 20% more accuracy.
5.4 GENERALIZABILITY OF DEFENSE
has been observed that models robust against 1p-norm-bounded attacks for one value of can be fragile when facing attacks with a different norm (,' (Sharma & Chen 2018). Here, our final goal is to present evidence that the approaches for defense based on pentuleettrbation attacks remain relatively robust even when faced with attacks generated using different norms. Here, we show this when our models are trained using the 12 bo-bounded attacks, and evaluated against other attacks using other norms. The results are presented in Figure We consider three alternative attacks: 1) PGD using the -ounded perturbations, as in Madry et al. (2018) (left in Figure 5) 2) dual-perturbation attacks with -xc bounds (middle left and middle rigt in Figure 5). and 3) JSMA, a lo-bounded attack (Papernot et al. (2066a) (right in Figure 5). We additionally considered le attacks, per Carlini and Wagner (Carlini Wagner, 2017). but find that all the robust models, whether based on PGD or dual- puttuurettrnation attacks, successful against these.
Our first observation is that AT-Dual is significantly more robust to 4oo -bounded PGD attacks than the adversarial training approach in which adversarial examples generated using ly-bounded PGD attacks (Figure 5 (left)). Consequently, training with dual- perturrruanatio attacks already exhibits better ability to generalize to other attacks compared to conventional adversarial training.
7

---

Under review as a conference paper at ICLR 2021
Input Image
Clean
AT-PGD
AT-Dual
Airplane
Truck
Elephant
Figure Visualization of loss gradient of different classifiers with respect to pixels of non- adversarial inputs.
The gap between dhall-perturbation-baace adversarial training and standard adversarial training is even more significant when we consider Lo dura--perurbation attacks (middle left and middle right figures of Figure 5). Here, we see that robustness of PGD- ad adversarially trained model is only marginally better than that of clean model under large distortions (c.g., when eB 20/255 in the middle right plot of Figure 5 whereas A7-Dual remains relatively robust.
Finally, considering JSMA attacks (see Figure 5 (right)), we can observe that both AT-Dual and AT-PGD remain relatively robust. However, a deeper look at (right) reveals that compared to AT-PGD, AT-Dual exhibit moderately higher robustness than the baselines under large distortions of JSMA attacks. Overall, in all of the cases, the model made robust using nuturpeeturrattoon attacks remains quite robust even as we evaluate against a different attack, using a different norm.
5.5 ANALYSIS OF DEFENSE
Finally, we conduct an exploratory experiment to study adversarial robustness by investigating which pixel-level features important for different classifiers at prediction time. To do this, visualize the loss gradient of different classifiers with respect to pixels the same non-adversarial inputs (as introduced in Tssipras al. (2019)), shown in Figure 6 Our first observation is that the gradients in response to adversarially robust classifiers (AT-PGD and AT-Dual) align well with human perception, while standard training model (Clean) results in a noisy gradient for the input images. Second, compared to adversarial training with the conventional PGD attack (AT-PGD), the loss gradient of AT-Dual provides significantly better alignment with sharper foreground edges and less noisy background. This indicates that adversarial training with the dual-peetubatiin attack which models unsuspiciousness can extract more perceptual semanties from an input image and are less dependant on the background at prediction time. In other words, our defense approach can extract highly robust and semantically meaningful features, which contribute to its robustness to a variety of attacks.
6 CONCLUSION
In this paper, we proposed the dual- pertuureuubationn attack, novel threat model that produces unsuspi- cious adversarial examples by leveraging the cognitive distinction between image foreground and background. As we have shown, our attack can defeat all state-of-the-an defenses. By contrast, the proposed defense approaches using our attack model can significantly improve robustness against unsuspicious adversarial examples, with relatively small performance degradation non-adversarial data. In addition, our defense approaches can achieve comparable to, or better robustness than the alternatives in the face of traditional attacks.
Our threat model and defense motivate several new research questions. The first is whether there are more effective methods to identify foreground of images. Second, can further improve robustness to dual duull-eeruuratiin attacks? Finally, while we provide the first principled approach for quantifying suspiciousness, there may be effective alternative approaches for doing
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Anurag Arnab, Ondrej Miksik, and Philip HS Torr. On the robustness of semantic segmentation models to adversarial attacks. In IEEE Conference on Computer Vision and Pattern Recognition. pp. 888-899 897, 2018.
Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, and David Forsyth. Unrestricted adversarial examples via semantic manipulation. International Conference on Learning Representations, 2020.
Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li. Salient object detection: A benchmark. IEEE transactions on image processing, 24(12):5706-5722, 2015.
Haya Brama and Tal Grinshpoun. Heat and blur: An effective and fast defense against adversarial examples. arXiv preprins, 2020.
Tom B. Brown, Nicholas Carlini, Chiyuan Zhang, Catherine Olsson, Paul Christiano, and Ian Goodfellow. Unrestricted adversarial examples. arXiv preprint, 2018.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. IEEE Symposium on Security and Privacy, pp. 39-57, 2017.
Jeremy M. Cohen, Elan Rosenfeld. and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning. 2019.
Tianji Cong and Atul Prakash. Masked MS-COCO for robust image classification. https:// github. com/ superct w J/Masked MSCOCO 2019.
Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255,20 2009.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Xiaodong Song. Robust physical-wote attacks on deep learning visual classification. /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1625-1634, 2018
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference Learning Representations, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sen He and Nicolas Pugeault. Salient region segmentation. arXiv preprint, 2018
M. Kmmerer, T. S. A. Wallis, L. A. Gatys, and M. Bethge. Understanding low- and high-level contributions fixation prediction. In IEEE International Conference on Computer Vision, 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3431-3440, 2015.
Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
Jeet Mohapatra, Tsui Wei Weng, Pin-Ye Yu Chen, Sijia Liu, and Luca Daniel. Towards verifying robustness of neural networks against semantic perturbations. International Conference on Learning Representations, 2020.

---

Under review as a conference paper at ICLR 2021
Seyed-Mohser Moosavi-Dezfooli: Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574-2582, 2016.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In IEEE European Symposium on Security and Privacy (EuroS&P), pp. 372-387, 2016a.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense to adversarial perturbations against deep neural networks. In IEEE Symposium Security and Privacy (SP), pp. 582-597, 2016b.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. In International Conference Learning Representations, 2018.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-an art face recognition. In ACM SIGSAC Conference on Computer and Communications Security, pp. 1528-1540 2016.
Yash Sharma and Pin- Yu Chen. Attacking the madry defense model with 1 -b-saded adversarial examples. In ICLR-18 Workshops, 2018.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014.
Anne Treisman and Garry Gelade. feature-integrattoon theory of attention. Cognitive psychology. 12(1))97-136, 1980.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Ro- bustness may be odds with accuracy. In International Conference on Learning Representations, 2019.
Pratik Vaishnavi, Tianji Cong, Kevin Eykholt, Atul Prakash, and Amir Rahmati. Can attention masks improve adversarial robustness? arXiv preprint, 2019.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint, 2020.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. Adversarial examples for semantic segmentation and object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1369-1378, 2017.
10


Reviews:
[{"review_id": "s6F3cyu449", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n=======\nThe paper studies threat models for images which take into account the foreground and background within the image. Specifically, they use DeepGaze II to identify pixels as either foreground or background, and use an Lp threat model with a larger radius in the background and a smaller radius in the foreground. The results are largely what one would expect: defenses for different threat models perform poorly against this attack, in comparison to adversarial training on adversarial examples generated by the attack. These attacks have good foreground score, though this is according to the same DeepGaze II model used to generate the examples to begin with. \n\nOverall, my impression is lukewarm. Although there are certainly some aspects which could be improved, there does not appear to be anything glaringly incorrect and the results are in line with expectations. The rest of this review is separated into comments which I would be happy to discuss with the authors, and other minor aspects that the authors can take into account in their revision. \n\nOn a general note, I'm not so sure if \"unsuspicious\" is the right qualifier for the work in this paper. The most relevant work to this paper is that by Xiao et al., however the authors simply give a very brief statement saying that Xiao et al. do not ensure that their attack is unsuspicious. Suspicious is very subjective: a human may very well look at the adversarial example Figure 2 with the corrupted background and think that this looks suspicious, and it's not at all obvious to me why this is any more or less suspicious than the discrete background changes studied by Xiao et al. To be clear, this is not a point of contention (as in this was not incorporated this into my score), but perhaps the authors can take this into account. \n\nComments for discussion\n=======================\n1. The datasets considered are quite small (STL-10 and ImageNet-10 are 5k training, 100 test, and Segment-6 is 18k train, 1.2k test), all of which are smaller than even CIFAR10. Is this a limitation of the the DeepGaze model or some other aspect of the approach? \n\n2. In section 3.2, the paper describes how the projection for the PGD attack is a computationally challenging problem since it is not an Lp ball. However, I believe this setting is not any more difficult than the standard PGD attack: since the perturbation is separated into disjoint foreground and background pixels, the projection is exactly equivalent to the standard projection over each set. Upon looking at the algorithm in the Appendix, this is exactly what is done by the authors. The resulting dual-perturbation attack is consequently a natural and straightforward application of PGD to the proposed threat model. This is not a negative aspect; however the text implies that this is splitting and merging is somehow substantially different from PGD when it isn't. Is my understanding here correct? \n\n3. The presentation of the claims of improvement in standard adversarial robust accuracy are somewhat misleading. For example, in Figure 4 the paper shows improved standard robust accuracy past epsilon=2 for the dual training, and the authors attribute this in the text at the end of section 5.3 to their defense, which trains on larger epsilon in the background. This is a misleading comparison because the baseline L2 adversarially trained model is only trained to be robust up to epsilon=2. In order to meaningfully conclude the final sentence of the section, i.e. that AT-Dual can achieve 20% more accuracy at a larger threshold of epsilon=3, it would only be fair to compare to a baseline which has been trained to be adversarially robust up to the same threshold that is being compared, otherwise the new approach will work better simply because the baseline is handicapped by the radius used at training and not necessarily because splitting foreground and backgrounds is any better. \n\nMinor comments\n==============\n+ In section 2.2, the paper cites a couple of papers for adversarial training as an empirical defense. Among these, Cohen et al 2019 is listed. However, this is not adversarial training; Cohen et al 2019 is a certified defense with guarantees, and is quite distinct from the adversarial training paradigm described in the text. \n\n+ Given the closeness in setting of Xiao et al. 2020 to the work done in this paper, there should be an additional sentence or two in the related work distinguishing the work in this paper from Xiao et al. 2020 other than mentioning this arbitrary concept of \"unsupiciousness\" of their attacks. In my understanding, the specifics are actually quite different (i.e. the threat model and how to get the backgrounds), but this is not at all indicated in the text which does not give a very meaningful description beyond \"suspiciousness\" of how these works are actually different. \n\nUpdate\n======\nI have read the author response, which largely does not change my score. I would kindly point out to the authors that the \"the technical challenge is that the feasible region is not an ball, and computing the projection is challenging in high-dimensional settings\" is in fact not very challenging at all. As mentioned in my initial review, the \"split and merge\" is exactly the standard projection operator on the proposed set and is not a \"new heuristic\", and so it is exactly PGD and not an adaptation of PGD. \n\nI also understand the other reviewers concerns on suspiciousness, which we all brought up. This likely needs to be thought about and presented more carefully, for example by posing it more formally if the authors insist on this framing. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "cd6FJira2E", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I like the idea to check whether it might be possible that noise can be injected selectively in areas where it is likely to make the adversarial example less suspicious and thus it might be possible to inject more noise in those directions. But the challenge is that this might increase the saliency of those areas. The authors propose to use recent work from DeepGazeII to detect saliencies of various pixels.\n\nBut I think this poses various problems, which I illustrate as below. Primarily, I am not sure that constructing attacks using  the same model that is then used to measure the saliency of various parts of the image is an unbiased way of doing this. \n\nThe Foreground score of $x+\\delta$ is defined as  $\\mathcal{S})(x+\\delta)$ ( using DeepGazeII) (Sec 3.2). Then the mere facts that the adversarial images generated using $S(x+\\delta)$ in the adversarial example formulation algorithm has a higher value of Foreground score does answer positively to the question \"whether foreground as we identify it is in fact significantly more salient than the background\". The algorithm is  maximizing the same function it claims is an impartial judge of saliency. I believe this is a faulty evaluation.\n\nIn terms of the main problem looked at in the paper, I am not convinced that it is well-defined. Is the objective to propose a stronger form of attack than existing PGD attack. If that is the case, then the comparison between the attacks need to be fair and I think this point about fairness of comparison need to emphasized as it is slightly confusing.\n\nFor example, are the AT-PGD models trained with an $\\ell_2$ adversary with $\\epsilon=0.2$ whereas the AT-Dual adversaries are trained with the dual-perturbation attack with $\\epsilon_F=2.0, \\epsilon_B=20.0$? In that case, if $\\|\\mathcal{F}(x)\\|_0=c_1$ and  $\\|\\mathcal{B}(x)\\|_0=c_2$, then the net $\\ell_2$ perturbation allowed for the adversary while AT-Dual training is $\\sqrt{2c_1}+\\sqrt{20c_2} \\gg \\sqrt{(c_1+c_2)2}$. If I am not missing anything, it looks like one of the adversarial training is considerably stronger than the other and thus comparison between them seems unfair.\n\nIn Page 4, the authors mention that satisfying the two hard constraints simultaneously might be difficult. However, the problem is essentially $\\ell_2 $ or $\\ell_\\infty$ balls on some subsets of the pixels. For projection to the $\\ell_\\infty$ ball, it is enough to make sure that the maximum of the two subsets of pixels are constrained. For the $\\ell_2$ case, it is also an intersection of two convex constraints, which should be easily solvable (especially because the two subsets are non-intersecting) \n\n\nIn Fig 5 where the X-axis is $\\epsilon$ i.e. where both the AT-PGD models and the AT-Dual models are attacked with the same strength adversary during training, the clean accuracy (i.e. $\\epsilon=0$) is quite low for At-Dual model compared to AT-PGD models (and Clean). I think is due to my initial point about the difference in the strength of adversary  used for training the models.\n\n\n### Update\n\nI have read the author's rebuttal and the comments of the other reviewers. Sadly, the comments of the other reviewers seem to align with my own thoughts and the rebuttal doesn't resolve those questions. I would like to maintain my initial rating of 4.\n More generally using the same model to measure saliency as well as to construct adversarial attacks is not properly justified and secondly, I am not convinced that putting a large perturbation on the \"background\" as determined as DeepGaze  is a proper side-by-side comparison with AT where AT is allowed  much smaller perturbation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SPq3H3-k1Bv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a new adversarial attack, the dual-perturbation attack, that segments the image into 'foreground' and 'background' pixels, and then allows significantly higher perturbation in the 'background' pixels, subject to the (soft) constraint that the \"salience\" of the image remains high.  When a classifier is trained with dual-perturbation adversarial samples, it seems to be more robust than classifiers trained without dual-perturbation adversarial samples, even when the model is attacked with samples produced with a different attack than the dual-perturbation attack.\n\nHowever, most of the results (Figures 4 and 5) seem to show that models are much more robust against dual-perturbation attacks when trained with dual-perturbation samples.  I don't find this particularly surprising.  Improvements are marginal---and performance is even degraded for some perturbation values---when compared against non-dual-perturbation attacks for other models (Figure 4, right; Figure 5, right).  The results shown in the appendix, especially for transferability, are similarly marginal (compared against the AT PGD model, which is the only other 'robust' model the authors test against).\n\nSo, I am not sure that the results are sufficiently valuable for publication in ICLR.  There are two other primary points that color my review here:\n\nFirst, the name \"suspicious\".  To call something \"suspicious\" is a completely subjective call; when I look at, say, the picture of the truck in Figure 21 in the appendix, I consider the perturbations to the background suspicious.  On the other hand, the perturbations to the stop sign in the referenced Eykholt et al. work are far less suspicious to me, even though the perturbations are large in the foreground.  It is strange, but not suspicious.  I don't want to necessarily get drawn into an argument over what is and what isn't suspicious, but I am sure that we can agree that the definition of suspicious itself is subjective.  Therefore, the idea of an *objective* measure that defines suspiciousness is, for me, a non-starter, and unnecessarily clouds and confuses the issue.  Perhaps the authors could consider the term \"salient\", but even then that is another subjective term (though perhaps less so).  \"Background salience\"?  In any case, I can't support the use of the term suspicious as it is currently used.\n\nSecondly, the relation to physical attacks is not clear.  The authors point to physical attacks as both a motivation for studying adversarial samples in general, and also for the distinction between foreground and background.  But in a physical attack, it would be very difficult to modify the background in a sufficiently adversarial way, since, e.g., a stop sign might be viewed from many angles, whereas an attack exploiting the background pixels could (probably!) only work from one very specific angle!  Yet, I can understand that not all attacks are physical, and so my comment here is more about the paper's clarity; I found this point somewhat confusing, and if the authors could do a better job of clarifying their motivation---and why their work is important despite the fact that physical attacks could not in general modify any background pixels.\n\nThe paper itself is well-written and clear and I certainly do appreciate the authors' efforts in that regard.\n\nHere are some other minor comments that I hope the authors will find useful:\n\n - Better clarity on the form of the adversary's utility function in (1) would be helpful.\n\n - Figure 3 is small and hard to read.  Please find a way to scale it up.\n\n - Does the definition of relative salience on page 4 need to be normalized to the sum of s_i over the foreground pixels?  (Or over all pixels?)  I think there may be an error here.\n\nThanks to the authors for the submission.  Despite my negative review, I did find the paper interesting and enjoyable to read.\n\n## Post-rebuttal comments\n\nThanks to the authors for the comments.  I will leave my review unchanged for this one.\n\n> The term “suspicious”: we agree with the reviewer that the word “suspicious” is an informal and subjective term, and there is no universal notion of what it means for adversarial examples to be unsuspicious. In fact, the core contribution of our paper is to address this issue.\n\nTo be clear, what I mean is that you should choose a *different* term than \"suspicious\".  While it may not sound as good to use a more specific technical term describing your actual measure that you are introducing, it will prevent the inevitable confusion that arises when an informal, vague term is overloaded to have a specific, quantitative definition.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FHtjVfmEJs", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary of Contributions\nThe paper proposes a formalization for one aspect of what makes adversarial examples suspicious, positing that visible perturbations to the foreground arouse more suspicion. Based on this proposed threat model the paper proposes the dual-perturbation attack, which perturbs the image with small perturbations in the foreground and larger perturbations in the background, with the background and foreground distinguished either by a provided mask or the DeepGaze II model, while maintaining a low saliency score in the background. The authors successfully generate images with low background saliency, and show that adversarial training with the dual-perturbation attack generates a network that is robust to the dual-perturbation attacks.\n\n# Score Recommendation\nI am recommending a rejection for this paper. While the overarching idea of identifying a class of “unsuspicious” adversarial examples is interesting, the formalization itself (which relies on a computation model, DeepGaze II) is not robust to adversarial perturbations. Furthermore, the experiments conducted do not substantiate the claims made (in the last two sentences of the abstract) about the effectiveness against classifiers robust to conventional attacks and the robustness of models trained via adversarial training with dual-perturbation. Finally, the approach does not consistently succeed at generating adversarial examples that are visually unsuspicious.\n\n# Strengths\nI appreciated that the paper attempts to introduce a formalization for a new class of attacks going beyond the existing $l_p$ norm-bounded attacks. In addition, the example provided in Figure 1 shows that increasing $\\lambda$ _does_ result in perturbations to the background that make it less salient.\n\n# Weaknesses\n\n## Potential for Attacks on Saliency Models\nOne of my concerns is that the saliency model is designed only for unperturbed images, and thus can itself be attacked. This could lead to cases where the perturbed image is suspicious (specifically, the background is actually salient to humans) but the saliency model has a low score for the saliency of the background. This would compromise one of the key planks of the approach.\n\n## Unconnected Experiments\n\n### Evaluation of attack strength on robust classifiers\nThe paper demonstrates that the dual-perturbation attack is successful against robust models. However, the fact that the attacks succeed is not really meaningful, as the models attacked are _not_ trained to be robust against the magnitude of attacks applied.\n\n#### AT-PGD network\nThe AT-PGD network is trained to be robust to attacks with $l_2$-norm up to $\\epsilon = 2.0$ (see the fourth paragraph of page 6. As such, it is unsurprising that it would be vulnerable (as shown in the left and middle plots of Figure 4) to attacks where $\\epsilon_B > 10$!\n\n#### Randomized Smoothing Networks\n- $l_\\infty$ attacks: Randomized smoothing is intended to generate models robust only against attacks with bounded $l_2$ norm. The success of $l_\\infty$ attacks in Table 5 of Appendix J.1 is thus not relevant.\n- $l_2$ attacks: The total attack budget for the smallest non-zero attack presented in Table 5 of in Appendix J.1 is $\\sqrt{(\\epsilon_F^2 + \\epsilon_B^2)} \\approx 1.27$, which is >2.5x the attack budget (127/255) designed for in [Certified Adversarial Robustness via Randomized Smoothing](https://arxiv.org/pdf/1902.02918.pdf). It is thus unsurprising that the attack succeeds (merely because it is so large) even if the budget needs to be distributed between the foreground and the background.\n\n#### Suggestion for Improvement\nIt would be far more interesting to constrain the _total_ budget for the dual-perturbation attack to be equal to that of the PGD attack. For example, in the $l_2$ case, we would have $\\sqrt{(\\epsilon_F^2 + \\epsilon_B^2)} = \\epsilon$. This would mean that the allowed perturbations for the dual-perturbation attack are a strict subset of that for the PGD attack. Having a significant proportion of samples vulnerable to PGD attack are also vulnerable to the dual-perturbation attack would be a convincing demonstration that the dual-perturbation attack is a strong attack (specifically, that perturbations primarily in the background are sufficient to change the classification of these images).\n\n### Evaluation of AT-Dual defense against additional attacks\nFigure 5 shows that AT-Dual is more robust as compared to AT-PGD on $l_\\infty$ PGD attacks and $l_0$ JSMA attacks. This is unsurprising since AT-Dual is trained to be robust against perturbations of larger overall magnitudes (and thus should be expected to be more robust against all attacks in general), albeit at a cost to its clean accuracy (of ~5%; see the right plot in Figure 4 for example). \n\n#### Suggestion for Improvement\nIt would be more interesting if it could be shown that AT-Dual has a clean and robust accuracy comparable to networks trained to be robust against $l_\\infty$ and $l_0$ attacks. \n\n## Failure to generate unsuspicious adversarial examples\nOne of the focuses of the paper is generating unsuspicious adversarial examples. Unfortunately, limitations of the algorithms computing saliency mean that the proposed attack often fails to achieve this goal; instead adversarial examples generated by the described dual-perturbation attack are often _more_ suspicious. This is the case since the boundaries between the “foreground” and “background” regions are sharp, and is worst when the background is relatively uniform and the saliency algorithm incorrectly identifies part of the background as the foreground. Good examples of suspicious adversarial examples generated by dual-perturbation attack can be seen in the STL-10 and ImageNet-10 images in Appendix L.\n\n# Post-Rebuttal Comments\nI'm afraid I'm maintaining my score as a 3. \n\nThe authors' response did not do enough to address my concerns:\n\n### Potential for Attacks on Saliency Models\nUnless I'm misunderstanding, the perturbed images presented in Fig. 1 are the results of adding perturbations intended to cause a change in the behavior of the **classifier**, not the **saliency model**.\n\n### Evaluation of attack strength on robust classifiers\nI still don't understand why the success of the dual-perturbation attack against models that were not trained to be robust to such attacks is of interest.\n\n### Suspiciousness of adversarial examples\nThe images identified in Appendix L are visually suspicious due to the sharp boundaries between the \"foregorund\" and \"background\", not because of a noticeable change to the background. The fact that these images are considered unsuspicious by the metric defined in the paper suggests that the metric needs to be modified.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The pursued here goal to explore what a broader and more nuanced notion of "imperceptible" perturbation is quite intriguing and could be a basis of really impactful investigations. However, as pointed out in the reviews and comments, the current treatment of this topic suffers from significant presentation and framing shortcomings. 

In particular, this work would benefit from stating clearly what is the main topic of study (and what is not), as current framing tends to confuse the readers. It would be also useful to discuss (and acknowledge) that having the split into background and foreground being driven by a (most likely non-robust) model makes the resulting notion of perturbation rather tricky to consistently analyze/certify. Also, the analysis of the properties of the model, although done fairly competently, is missing some important aspects.

Still, once these points are properly addressed, this would constitute a valuable contribution. 
Decision: Reject