Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
Authors: Jörn-Henrik Jacobsen
Jens Behrmann
Nicholas Carlini
Florian Tramèr
Nicolas Papernot
Venue:  (2020)
Keywords: Invariance
Robustness
Adversarial Examples

Abstract: Adversarial examples are malicious inputs crafted to cause a model to misclassify them. In their most common instantiation, "perturbation-based" adversarial examples introduce  changes to the input that leave its true label unchanged, yet result in a different model prediction.  Conversely, "invariance-based" adversarial examples insert changes to the input that leave the model's prediction unaffected despite the underlying input's label having changed. So far, the relationship between these two notions of adversarial examples has not been studied, we close this gap.

We demonstrate that solely achieving perturbation-based robustness is insufficient for complete adversarial robustness. Worse, we find that classifiers trained to be Lp-norm robust are more vulnerable to invariance-based adversarial examples than their undefended counterparts. We construct theoretical arguments and analytical examples to justify why this is the case. We then illustrate empirically that the consequences of excessive perturbation-robustness can be exploited to craft new attacks. Finally, we show how to attack a provably robust defense --- certified on the MNIST test set to have at least 87% accuracy (with respect to the original test labels) under perturbations of Linfinity-norm below epsilon=0.4 --- and reduce its accuracy (under this threat model with respect to an ensemble of human labelers) to 60% with an automated attack, or just 12% with human-crafted adversarial examples.

Under review as a conference paper at ICLR 2020
EXPLOITING EXCESSIVE INVARIANCE CAUSED BY NORM-BOUNDED ADVERSARIAL ROBUSTNESS
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial examples are malicious inputs crafted to cause a model misclassify them. In their most common instantiation, "perturbation-basedd adversarial exam- ples introduce changes to the input that leave its true label unchanged, yet result a different model prediction. Conversely, "invariance-based" adversarial examples insert changes to the input that leave the model's prediction unaffected despite the underlying input's ' label having changed. So far, the relationship between these two notions of adversarial examples has not been studied, we close this gap.
We demonstrate that solely achieving perturbation-base for complete adversarial robustness. Worse, we find that classifiers trained be Lp f--norm robust are more vulnerable to invariance- based adversarial examples than their undefended counterparts. We construct theoretical arguments and analytical examples to justify why this is the case. We then illustrate empirically that the consequences of excessive perturbation-robussee attacks. Finally, we show how to attack provably robust defense certified on the MNIST test set to have least 87% accuracy (with respect to the original test labels) under perturbations of & -oom below 0.4 and reduce its accuracy (under this threat model with respect to an ensemble of human labelers) to 60% with an automated attack, or just 12% with human-crafted adversarial examples.
robustness is insufficient
can be exploited to craft new
INTRODUCTION
Research on adversarial examples is motivated by a spectrum of questions. These range from the security of models deployed in the presence of real-world adversaries, to the need to capture limitations of representations and their (in)ability to generalize (Gilmer et al., 2018a). The broadest accepted definition of an ad adversarial example is "an input to a ML model that is intentionally designed by an attacker to fool the model into producing an incorrect output" (Goodfellow & Papernot 2017).
Many formal definitions of adversarial examples were introduced since their initial discovery (Szegedy al., 2013 Biggio al.] 2013). In majority of work, adversarial examples are formalized as adding perturbation ò to some test example to obtain input x" that produces an incorrect model outcome ! We refer to this class of malicious inputs as perturbation- adversarial examples.
To enable concrete progress, the adversary's capabilities may optionally be constrained by placing a bound on the maximum perturbation added to the original input. The goal of this constraint to ensure that semantics of the input are left unaffected by the perturbation 8. In the computer vision domain, lp norms have grown to be the default metric to measure semantic similarity. This led to series of proposals for increasing the robustness of models perturbation- based adversaries that operate within the constraints of an E, ball. These include robust optimization (Madry et al. 2017). explicit regularization of model's Lipschitz constant (Cisse et al.) 2017), or a variety of techniques to build models that are provably robust to small f, perturbations (Wong & Kolter 2018 Raghunathan al. 2018 Zhang al. 2019).
In this paper, we show that optimizing model's robustness to ly-bounded perturbations is not only insufficient to address the lack of generalization identified via adversarial examples, but also potentially harmful. Intuitively, as tp distances are only crude approximations to the true visual
'erre, incorrect output either refers to the model returning any class different from the original source class of the input, or specific target class chosen by the adversary prior searching for perturbation.

---

Under review as a conference paper at ICLR 2020
Perturbation-Unrenn Model
Perturbation Robusst Model
Classified as: 5
5
Classified as:
5
3
3
Parturbation-uuroouus decision boundary
Oracle Decision-boundary
Perturbation-robust decision boundary
Figure I: Real image: 3; Invariance-based attack: 5. [Left]: Training a classifier without constraints may learn decision boundary unrobust perturbation-base adversarial examples. [Right]: Enforc- ing robustness to norm- norm-bounder bounded perturbations introduces erroneous invariance (dashed regions in epsilon spheres). Note, that we display real data here, the misclassified is image found by our invariance-basee attack which resides within a typically reported c-region around the displayed 3 (in the to norm). This excessive invariance of the perturbation- may be exploited, as shown by the attack proposed in this paper.
model in task-relevant directions
similarity in given task, over-optimizing model's robustness ep-bounded perturbations also renders the model invariant to actual semantics of the underlying task.
Excessive invariance of model to class semanties can give rise to perturbations called invariance adversarial examples (Jacobsen et al. (219)) These are perturbations that successfully change the numan-assigned ground-truth label of imput, while keeping the model's classification of the perturbed input unchanged (for example, we take an image of a '1' digit and
class of adversarial
perturb it without changing the model's Daasification-uniee human would agree that the digit now represents a '7"). See Figure I for an intuitive illustration of this phenomenon.
Whereas previous efforts (e.g., (Jacobsen al. 2019) explored different notions of adversarial examples independently, our work is the first to expose a complex relationship between perturbation- based and invariance-bast based adversarial examples.
Specifically, we introduce analytical constructions and empirical evidence that shows that increasing model's robustness to perturbation-ban bbased adversarial examples can increase the model's vulnerability to invariance-bases adversarial examples.
We formally show how to construct a model that robust to perturbation- based adversarial examples but not invariance-based adversarial examples. We further demonstrate how imperfect model for the adversarial spheres task proposed by Gilmer al. (2018b) is either vulnerable to perturbation- based or invariance-based attacks depending on whether the point attacked is on the inner or outer sphere. Hence, these two types of adversarial examples are needed to fully account for model failures.
Finally, we empirically demonstrate with new attuck how an adversary can exploit the excessive invariance certain models. We introduce the first algorithm that crafts invariance-based adversarial examples for the lo and for norms, and show that many common models disagree with human labelers on these examples. In particular, our algorithm breaks a provably-robit defense on MNIST (Zhang al. 2019 This model certified have 87% test-accuracy under -e--prrturatioons of radius 8 0.4. Yet, on our aeoomaticallyggneeaate
invariance-based adversarial examples, the model only agrees with the human-assigned assigned label in 60% of the cases; when we manually craft invariance adversarial examples we reduce accuracy to just 12% as determined by an ensemble of humans.
2 PERTURBATION AND INVARIANCE-BASER ADVERSARIAL EXAMPLES
In order to make precise statements about adversarial examples, we begin with two definitions.
Note that while MNIST is typically a aooo choice for studying adversarial robustness Carlini et al. 2019). in case explicitly chose to work with MNIST because it is the only dataset for which models have been claimed be provably robust non-negligible Ip norm halls.
2

---

Under review as a conference paper at ICLR 2020
Definition 1 (Perturbation-basse Adversarial Examples). Let G denote the i-th layer. logit argmax of the classifier. A Perturbation-bate adversarial example (or perturbation adversarial) x* E R corresponding to a legitimate test input R fulfills:
Created by adversary: x² E B is created by an algorithm A Rd - RRd with x + x²"
2. Perturbation of output: [|G(=*) - G(x) 8 and ((x*) = ( where perturbation 6 > 0 is set by the adversary and O RR + {1,.... C'} denotes the oracle.
Furthermore, x² c-bounded || - x²1 < where || is a norm on TRd and c > 0.
Property (i) allows us to distinguish perturbation adversarial examples from points that are misclassi- fied by the model without adversarial intervention. Furthermore, the above definition incorporates also adversarial perturbations designed for hidden features as in (Sabour et al. 2016), while usually the decision of the classifier D fargmax-operatie on logits) used as the perturbation target. Our definition also identifies c- e-bounded perturbation crrturbation-based adversarial examples (Goodfellow et al. (015) as a specific case of unbounded perturbation-ban primarily considers the latter, which correspond to the threat model of a stronger adversary.
adversarial examples. However, our analysis
Definition (Invariance- based Adversarial Examples). Let G denote the i-th layer, logit or argmax of the classifier. An invariance-based adversarial example (or invariance adversarial) x² E R corresponding to a legitimate test input x E Rd fulfills:
1. Created by adversary: x² E ld is created by an algorithm A: A. + R with I + x².
2. Lies in pre-image of x under G: G(z²) = G(x) and (zz) * O(x"), where O : Rd
(I,...,)) C) denotes the oracle.
As a consequence, ((x)= = D)=") also holds for invariance-based adversarial examples, where D is the output of the classifier. Intuitively, adversarial perturbations cause the output of the classifier to change, while the oracle would still label the new input x in the original source class. Whereas perturbation-base task-irrelevant directions, invariance-baat -ased adversarial examples explore the classifier's pre-image to identify excessive invariance rask-relevant directions: its prediction is unchanged while the oracle's output differs. Briefly put, perturbation- based and invariance-based adversarial examples are complementary failure modes of the learned classifier.
adversarial examples exploit the classifier's excessive sensitivity in
ROBUSTNESS TO PERTURBATIOO--AAS ADVERSARIAL EXAMPLES CAN CAUSE INVARIANCE-BAAE VULNERABILITTES
We now investigate the relationship between the two adversarial example definitions from Section 2 So far, it has been unclear whether solving perturbation-ben based adversarial examples implies solving invariance-based adversarial examples, and vice versa. In the following, we show that this relationship intricate and developing models robust in one of the settings only would be insufficient.
In a general setting, invariance and stability can be uncoupled. For example, consider a linear is tightly related to forward stability (largest
classifier with matrix A. The certurbation-robusten singular value of A). On the other hand, the invariance-vicd relates to the stability of the inverse (smallest singular value of A) and to the null-space of A. As largest and smallest singular values are uncoupled for general matrices A, the relationship between both viewpoints is likely non-trivial in practice.
3.1 BUILDING OUR INTUITION WITH EXTREME UNIFORM CONTINUITY
In the extreme, classifier achieving perfect uniform continuity would be constant classifier. Let D : Rn + [o, 1 denote a classifier with D(x) = y² for all x E Rd. As the classifier maps all inputs to the same output y², there exist no x², such that D(x) + D(x*). Thus, the model is trivially perturbation-ron
(at the expense of decreased utility). On the other hand, the pre-image of y under D is the entire input space, thus D is arbitrarily vulnerable to invariance- based adversarial examples. Because this toy model a constant function over the input domain, no perturbation of an initially correctly classified input can change its prediction.
3

---

Under review as a conference paper at ICLR 2020
Outer Sphere
Inner Sphere
0.8
0.8
R
R
R:
Inner:: Outer:
04
0.4
0-
Invariance
0.2-
Invariance Attack
Perturbation Attäcck
Perturtation Attack
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
eps Gn (2-norm)
eps (in 12-norm)
Figure 2: Robustness experiment on spheres with radii R. and R₂ = 1.3 and max-margin classifier that does not see n 10 dimensions of the d = 500 dimensional input. [Left]: Attacking points from the outer sphere with perturbation based attacks; accuracy drops when increasing the upper bound fo-norm perturbations. [Right]: Attacking points from the inner sphere with invariance-basee attacks; accuracy drops when increasing the upper bound on f2-norm perturbations. Each attack has different effect the manifold. Red arrows indicate the only possible direction of attack for each sphere. Perturbation attacks fail on the inner sphere, while invariance attacks fail on the outer sphere. Hence, both attacks are needed for full account of model failures.
This trivial model illustrates how one not only needs to control sensitivity but also invariance alongside accuracy to obtain robust model. Hence, we argue that the often-discussed tradeoff between accuracy and robustness (see Tsipras et al. (2019) for a recent treatment) should in fact take into account at least three notions: accuracy, sensitivity, and invariance. This depicted in Figure [ In the following, present arguments as for why this insight also extend to almost perfect classifiers.
3.2 COMPARING INVARIANCE- BASED AND PERTURBATION BASED ROBUSTNESS
We now show how the analysis of perturbation- based and invariance-basee based adversarial examples can uncover different model failures. To do so, we consider the synthetic adversarial spheres problem Gilmer et al. (018b). The goal of this synthetic task is to distinguish points from two cocentric spheres (class 1: | R, and class 2: | R, ) with different radii R and R2. The dataset was designed such that robust (max-margin) classifier can be formulated as:
R1 + R₂
(* = sign ||0|||2
Our analysis considers a similar, but slightly sub-optimal classifier order to study model failures in controlled setting:
((x)) = sign
b),
which computes the norm of æ from its first d - n cartesian-coordinatee and outputs -1 (resp. +1) for the inner (resp. outer) sphere. The bias b chosen based on finite training set (see Appendix A).
Even though this sub- optimal classifier reaches nearly 100% on finite test data, the model is imperfect in the presence of adversaries that operate on the manifold (i.e., produce adversarial examples that remain one of the two spheres but are misclassified). Most interestingly, the perturbation- hased and invariance- based approaches uncover different failures (see Appendix A for details on the attacks):
Perturbation- All points æ from the outer sphere (i.e., ||^||| R2) can be perturbed to x², where (xx) = Dzx) f D(x*) while staying on the outer sphere (i.e., || l| = R2).
Invariance-based: All points x from the inner sphere ( (|æl|2 = Rt) can be perturbed to x", where Dxx) D(x*) f O(x*), despite being in fact on the outer sphere after the perturbation has been added (i.e., ||=" 2|2 R2).
In Figure 2.
we plot the mean accuracy over points sampled either from the inner or outer sphere, function of the norm of the adversarial manipulation added to create perturbation-basee and invariance-bart based adversarial examples. This illustrates how the robustness regime differs significantly between the two variants of adversarial examples. Therefore, by looking only perturbation-base
4

---

Under review as a conference paper at ICLR 2020
(respectively iivariance-based adversarial examples, important model failures may be overlooked. This is exacerbated when the data is sampled in an unbalanced fashion from the two spheres: the inner sphere is robust to perturbation adversarial examples while the outer sphere is robust to invariance adversarial examples (for accurate models).
INVARIANCE-BASER ATTACKS IN PRACTICE
We now show that our argument automated attack algorithms to generate invariance adversarial examples. We elect to study the only dataset for which robustness considered be nearly solved under the Ep norm threat model: MNIST (Schott et al. 2019). We show that MNIST models trained to be robust to perturbation-base adversarial examples are less robust to invariance-basee adversarial examples. As a result, we show that while perturbation adversarial examples may not exist within the E, ball around test examples, adversarial examples still do exist within the lp ball around test examples.
limited to the analysis of synthetic tasks, and give practical
Why MNIST? The MNIST dataset is typically poor choice of dataset for studying ad adversarial examples, and in particular defenses that are designed to mitigate them (Carlini et al. 2019). In large part this is due the fact that MNIST is significantly different from other vision classification problems (e.g.. features are quasi-binary and classes are well separated in most cases). However, the simplicity of MNIST is why studying f--norm adversarial examples was originally proposed as a toy task to benchmark models (Goodfellow et al. 2013). However, several years later, it is now argued that training MNIST classifiers whose decision constant in an yn-nrrm ball around their training data provides robustness to adversarial examples (Schott et al. 2019 Madry et al., 2017 Wong &
Kolter 2018
Raghunathan et al. 2018).
MNIST is the only dataset for which robustness to adversarial examples is considered even remotely close being solved (Schott et al., (019) and researchers working on (provable) robustness to adversarial examples have moved on to other, larger vision datasets such as CIFAR-I0 (Madry et al.,
2017
Wong et al. 2018) or ImageNet (Lecuyer et al., 2018: Cohen et al. 2019).
This section argues that, contrary to popular belief, MNIST is far from being solved. We show why robustness to no-mr perturbation-
adversaries is insufficient, even on MNIST, and why defenses with unreasonably high uniform continuity can harm the performance of the classifier and make it more vulnerable to other attacks exploiting this excessive invariance.
4.1 A TOY WORST-CASE: BINARIZED MNIST CLASSIFIER
Invariance
Binks;)
4
To give initial constructive example, consider MNIST classifier which binarizes (by thresholding at, e.g.. 0.5) all of its inputs before classifying them with neural network. As (Tramer al.) 2018 Schott et al. (019)) demonstrate, this binarizing classifier is highly -robust because most perturbations the pixel space do not actually change the (thresholded) feature representation.
Source
Bingx;)
However, this binary classifier will have trivial invariance- based adversarial examples. Figure an example of this attack. Two images which are dramatically different human (e.g., a digit of a one and a digit of four) can become identical after pre-pre mee-proeessing the images with a thresholding function at 0.5 (as examined by, e.g., Schott et al. (2019)).
Figure 3: Invariance-based ad- versarial example (top-left) is labeled differently by hu- man than original (bottom- left). However, both become identical after binarization.
With this toy worst-case discussed, we now turn to the full MNIST evaluation.
4.2 GENERATING MODEL- AGNOSTIC INVARIANCE- BASED ADVERSARIAL EXAMPLES
In the following, we build on existing invariance-base based attacks (acobsen et al. 2019 Behrmann et al. 2018 et al. 2019) to propose a -agnossa algorithm for crafting invariance based adversarial examples. That is, attack algorithm generates invariance adversarial examples that cause human to change their classification, but where most models, not known by the attack algorithm, will not change their classification.
5

---

Under review as a conference paper at ICLR 2020
8
3
3
8
3
3
(a)
(b)
(c)
(d)
(e)
(f-h)
Figure 4: Process for generating fo invariant adversarial examples. From left to right: (a) the original image of an 8; (b) the nearest training image (labeled as 3), before alignment; (c) the nearest training image (still labeled as 3), after alignment; (d) the å perturbation between the original and aligned training example; (e) spectral clustering of the perturbation 6; and (f-h) possible invariance adversarial examples, selected by applying subsets of clusters of 5 to the original image. (f) is failed attempt at an invariance adversarial example. (g) is successful, but introduces a larger perturbation than necessary (adding pixels to the bottom of the 3). (h) is successful and minimally perturbed.
Our algorithm for generating invariance- based adversarial examples is simple, albeit tailored to work specifically on datasets where comparing images in pixel space meaningful, like MNIST.
Begin with source image, correctly classified by both the oracle evaluator (i.e., human) and the model. Next, try possible affine transformations of training data points whose label is different from the source image, and find the targel training example which-once transformed- the smallest distance to the source image. Finally, construct an invariance-based adversarial example by perturbing the source image to be "more similar" to the target image under the f, metric considered. In Appendix B, we describe instantiations of this algorithm for the lo and for norms. Figure visualizes the sub-steps for the &o attack, which are described in details in Appendix B.
The underlying assumption of this attack is that small affine transformations are less likely to cause an oracle classifier to change its label of the underlying digit than t, perturbations. In practice, validate this hypothesis with a human study in Section 4.3
Before evaluating this attack, make definition will use discuss the accuracy of
Definition 3 (Successful e-bounded Invariance Attack). For given test example assigned label y. a successful invariance attack is a perturbed example x" such that the oracle classification ((x") # and || x=| e, where || ||
on Rd and >
In practice, we obtain the oracle classification by asking an ensemble of human labelers to label the point x=; if more than some fraction of them agree on the label (throughout this section, 70%) and that label is different from the original, we call the attack successful. Note that success or failure is independent of any machine learning model: it only has to do with whether or not the underlying label has actually changed according to the (human) oracle.
4.3 EVALUATION
Attack analysis. We generate 1000 adversarial examples using each of the two above approaches on examples randomly drawn from the MNIST test set. Our attack slow, with the alignment process taking (amortized) minutes per example. We performed no optimizations of this process and expect it could be improved. The mean lo distortion required is 25.9 (with median of 25). The &oo adversarial examples always use the full budget of 0.3 0.4 and take similar amount of time to generate; most of the cost is again dominated by finding the nearest (transformed) training image.
Human Study. We randomly selected 100 examples from the MNIST test set and create 100 invariance- adversarial examples under the to norm and lo norm, as described above. We then conduct human study to evaluate whether or not these invariance adversarial examples indeed are successful as defined earlier, i.e., whether humans agree that the label has been changed. We additionally hand-crafted 50 invariance adversarial examples under the specific norms. The process to create these examples was quite simple: we built minimal image editor that enabled us change images at a pixel level under given lp constraint. One author then modified 50 random test examples in the way that they perceived as changing the underlying class. We presented 40 human evaluators with these 100 images, half which were natural unmodified MNIST digits, and the remaining half distributed randomly between to loo invariance adversarial examples.

---

Under review as a conference paper at ICLR 2020
4
7
2
0
2
6
4
Attack Type
Success Rate
Clean Images to Attack
0%
7 2
2
6
3
4
55%
loo. E 0.3 Attack
21%
loc. E = 0.4 Attack
37%
9
7
2
2
0
4
(a) Success rate of our invariance adversarial example causing humans to switch their classification.
(b) Original test images (top) with our &a (middle) and & (bottom) invariance adversarial examples. (left) successful attacks; (right) failed attacks.
Figure 5: Our invariance-basee adversarial examples. Humans (acting as the oracle) switch their classification of the image from the original test label to a different label.
Results. For the clean (unmodified) test images, 98 of the 100 examples were labeled correctly by all human evaluators. The other 2 images were labeled correctly by over 90% of human evaluators.
Our la attack is highly effectiiv: For 55 of the 100 examples least 70% of human evaluator who saw that digit assigned it the same way, with a different label from the original test label. Humans only agreed with the original test label (with the same 70% threshold) on 34 of the images, while they did not form consensus on the remaining 18 examples. The (much simpler) &oo attack is less effective: with a distortion of 0.3 the oracle label changed 21 of the time and with 0.4 the oracle label changed 37% of the time. We summarize results in Table 5 (a).
In Figure 5 (b) we show sample invariance adversarial examples. To simplify the analysis in the following section, we split our generated invariance adversarial examples into two sets: the successes and the failures, as determined by whether the plurality decision by humans was different than or equal to the human label. We only evaluate the models on the subset of invariance adversarial examples that caused the humans to switch their classification.
Model Evaluation. Given oracle ground-truite truth labels for each of the images (as decided by humans), we report how often models agree with the numan-assignes label. Table summarizes this analysis. For the invariance adversarial examples we report model accuracy only on the successful attacks.
Every classifier labeled all successful foe invariance adversarial examples incorrectly (with one exception where the & PGD-trained classifier Madry et al. (2017) labeled one of the invariance adversarial examples correctly). Despite this fact, PGD adversarial training and Analysis by Synthesis Schott et al. (2019) are two of the state-of-the-ar Eo perturbation-roous: classifiers.
The situation is more complex for the ég-invariance adversarial examples. In this setting, the models which achieve higher lo perturbation--rouusten set. For example, Bafna et al. (2018) develops to perturbation-robut classifier that relies on the sparse Fourier transform. This perturbation robust classifier is substantially weaker to invariance adversarial examples, getting only 38% accuracy compared to a baseline classifier's 54% accuracy.
result in lower accuracy on this new invariance test
Breaking Certified Defenses. Our invariance attacks are sufficiently strong that they constitute a break of some certified defenses. For example.[Zhang et al. (2019) develop a certified defense to foo adversarial examples which proves that the accuracy on the test set when perturbed by e = 0.4 at least 87%. When run their pre-trained model on all 100 of our = 0.4 invariance adversarial examples we find it has a 96% "accuracy" (i.c., it matches the original test label 96% of the time). However, when we compare the accuracy of this model compared the new labels assigned by an ensemble of humans, the accuracy is just 63% That is, while the proof in the paper is mathematically correct it does actually deliver 87% accuracy on any new test input perturbed by would have changed their classification in many of these settings. Worse, for the 50 adversarial examples we crafted by hand, the model disagrees with the human ensemble 88% of the time: has just 12% accuracy.
0.4: humans
For all invariance adversarial examples the most likely label was selected by more than half of humans. If we sub-set to only the 21 examples where all humans agreed on the label, the accuracy of this model remains 50%; if instead require at least 75% agreement the accuracy is 65%.
7

---

Under review as a conference paper at ICLR 2020
Agreement between model and humans, for successfid invariance adversarial examples
Model:
ResNet
CNN
En Sparse
Binary-ABS
ABS
f PGD
l PGD
Clean
99%
99%
99%
99%
99%
99%
99%
to
65%
54%
38%
47%
58%
56%*
27%*
20. E = 0.3
0%
0%
0%"
0%
0%
0%
5%"
Table Model accuracy with respect to the oracle human labelers on the subset of examples where the human-obtained oracle label is different from the test label. Models which are more robust to perturbation adversarial examples (such as those trained with adversarial training) agree with humans less often invariance-based adversarial examples. Values denoted with asterisks violate the perturbation threat model of the defense and should not be taken to be attacks. When the model is wrong, it classified the input as the original test label, and not the oracle label.
"Go Sparse: Bafna al. 2018); ABS and binary-ABS; Schott et al. 2019: foo PGD and &g PGD: Madry
al. 2017
4.4 NATURAL IMAGES
While the previous discussion focused synthetic (Adversarial Spheres) and simple tasks like MNIST, similar phenomena may arise in natural images. In Figure [ perturbations of the original image (left). The perturbation of the middle image is nearly imperceptible and thus the classifier's 's decision should be robust to such changes. On the other hand, the image on the right went through semantic change (from tennis ball to strawberry) and thus the classifier should be sensitive to such changes (even though this case is ambiguous due to two objects in the image). However, in terms of the lg norm the change in the right image is even smaller than the imperceptible change in the middle. Hence, making the classifier robust within this &g norm-ba ball will make the classifier vulnerable invariance-based adversarial examples like the semantic changes in the right image.
we show two different &2
similar argument can be made for other norms. For instance Co et al. (2018) show that a perturbation of magnitude 162255 in loc. can suffice to give an image of a cat the appearance to be printed on shower curtain (both are classes in Imagenet). However, we leave studying this direction more detail to future work.
(a)
(b)
(c)
Figure 6: Visualization that large & norms also fail to measure semantic changes in images. (a) original image in the ImageNet test set labeled as a tennis ball; (b) imperceptible perturbation, lg 24.3; (c) semantic perturbation with lg perturbation of 23.2 that removes the tennis ball.
CONCLUSIONS
Training models robust to perturbation based adversarial examples should not be treated as equivalent to learning models robust to all adversarial examples. While most of the research has focused on perturbation- based adversarial examples that exploit excessive classifier sensitivity, we show that the reverse viewpoint of excessive classifier invariance should also be taken into account when evaluating
8

---

Under review as a conference paper at ICLR 2020
robustness. Furthermore, other unknown types of adversarial examples may exist: it remains unclear whether or not the union of perturbation and invariance adversarial examples completely captures the full space of evasion attacks.
Consequences for fy-norm evaluation and certified defenses. Our invariance-based attacks are able to find wit--eeruuraationbaaeeeee adversarial examples within the Ep ball on classifiers that were adversaries. As a consequence of this analysis,
trained to be robust to f--torm perturbation-ban researchers should carefully set the radii of Ep-balls when measuring robustness to norm-bounded perturbation-baxt ad adversarial examples. Robustness to small epsilons can indeed lead to better trade-off between perturbation- and invariance- due to the misalignment between p-norms with the "true perceptual distance metric" of the oracle, training against larger epsilon (as is common on MNIST for instance, and we expect will soon become true on larger datasets) will inevitably lead to erroneous invariances and hence to vulnerability with respect to invariance-
robustness (Engstrom et al., 2019). However,
adversarial examples. Furthermore, setting a consistent radius across all of the data may be difficult: we find in our experiments that some class pairs are more easily attacked than others by invariance-basee adversaries.
In particular, we find that recent proposals that claim exceptionally high to and loo norm- comn-buunded robustness are not meaningfully correct. Given the claimed perturbation budget, it is possible to generate invariance adversarial examples that fool these defenses. Indeed, another by-product of our study is to showcase the importance of human studies when the true label of candidate adversarial inputs becomes ambiguous and cannot be inferred algorithmically.
Defenses to adversarial examples on MNIST must stop blindly increasing the distortion threshold and claiming robustness, and in particular for E distortion of 0.4. Any claim of robustness greater than 60% accuracy is not meaningfully correct because at this distortion level, it is possible to generate examples where humans change their classification.
Invariance. Our work confirms findings reported recently in that it surfaces the need for mitigating undesired invariance in classifiers. The cross-entropy loss well as architectural elements such as ReLU activation functions have been put forward as possible sources of excessive invariance (Jacobsen al. 2019 Behrmann et al. 2018). Other work has pointed out a trade-off as consequence of robustness to perturbations in different frequency spectra (Yin et al. 2019).
However, more work is needed to develop quantitative metrics for invariance maaseddce-based robustness. One promising architecture class to control invariance based robustness are invertible networks (Dinh et al. 2014) because, by construction, they cannot build up any invariance until the final layer (Jacobsen al. 2018 Behrmann et al. 2019).
REFERENCES
Mitali Bafna, Jack Murtagh, and Nikhil Vyas. Thwarting adversarial examples: An 1_0-robust sparse fourier transform. In Advances in Neural Information Processing Systems, pp. 10096-10106, 2018.
Jens Behrmann, Sören Dittmer, Pascal Fernsel, and Peter MaaB. Analysis of invariance and robustness via invertibility of relu-networks. arXiv preprint 1806::08800030000 2018
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jöm-Henrik Jacobsen. Invertible residual networks. arXiv preprint /x1iv:/111.00995 2019.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šmdii Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras. Ian Goodfellow, and Aleksander Madry. On evaluating adversarial robustness. arXiv preprint 5xXiv:9902.660 2019.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 654-863. JMLR. org, 2017.

---

arXiv preprint Under review as a conference paper at ICLR 2020
Kenneth T Co, Luis Muñoz-Gonzälez, Sixte de Maupeou, and Emil C Lupu. Procedural noise adversarial examples for black-box attacks on deep convolutional networks. arXiv preprint 0ef1::180000070, 2018.
Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint 1902.0098,, 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv: 2014.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness prior for learned representations, 2019.
Justin Gilmer, Ryan P. Adams, Ian Goodfellow, David Andersen, and George E. Dahl. Motivating the rules of the game for adversarial example research. arXiv preprint arXiv: 06732, 2018a.
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow. Adversarial spheres. arXiv preprint arXiv:1801.02774 2018b.
Ian Goodfellow and Nicolas Papernot. Is attacking machine learning easier than defending it? Blog post on Feb, 15:2017, 2017.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Andrew Ilyas, Ajil Jalal, Eirini Asteri, Constantinos Daskalakis, and Alexandros G Dimakis. The robust manifold defense: Adversarial training using generative models. arXiv preprint arXiv:1112.009166 2017.
Jörn-Herrik Jacobsen, Arnold W.M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In International Conference on Learning Representations, 2018
Jörn-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance
causes adversarial vulnerability. In International Conference Learning Representations, 2019.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. arXiv preprint arXiv: /802.03471, 2018.
Ke Li, Tianhao Zhang, and Jitendra Malik. A study of robustness of neural nets using approximate feature collisions, 2019. URL htttt::://pperee//// eww met/fforue d= gDnngDg
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. orXiv::411.111 2014.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. International Conference on Learning Representations, 2018.
Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J Fleet. Adversarial manipulation of deep representations. International Conference on Learning Representations, 2016.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv: 00888800000605 2018
Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST. In International Conference on Learning Representations, 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 131. 2013.
10

---

Under review as a conference paper at ICLR 2020
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. International Conference on Learning Representations, 2018.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Tumer, and Aleksander Madry. Ro- bustness may be at odds with accuracy. In International Conference on Learning Representations, 2019.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In Proceedings of the 35th International Conference on Machine Learming, 2018.
Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and Zico Kolter. Scaling provable adversarial defenses. In Advances in Neural Information Processing Systems, pp. 8410-8419, 2018.
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D Cubuk, and Justin Gilmer. A fourier perspective on model robustness computer vision. arXiv preprint arXiv: 1906008988 2019.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards stable and efficient training of verifiably robust neural networks. arXiv preprint arXiv: 1906.06316, 2019.
A DETAILS ABOUT ADVERSARIAL SPHERES EXPERIMENT
In this section, we provide details about the Adversarial Spheres Gilmer et al. 201 8b) experiment. First, the bias b is chosen, such that the classifier the max-margin classifier on the (fiitte) training set X' (assuming separability: I S u):
1=
max ==
| 122
min |m||==Ho Hy FET
[1 d.....d-nllll 112,
2
Second, the attacks designed such that the adversarial examples x² stay the data manifold (two concentric spheres). In particular, following steps are taken:
Perturbation-basedd All points from the outer sphere (i.e., ||>|| R2) can be perturbed x',
where ((z) = D(x) + D)*"), while staying on the outer sphere (i.e., ||x" ||2 = R2) via following steps:
1. Perturbation of decision: 2 Fi....d-n = (#1,...d-n), n) where scaling 0 is chosen such
that ||=:....--nIlz <
2. Projection outer sphere: 2 ad......d
( (Id-n a), where scaling c chosen such
that || ||
R2 |
For points æ from the inner sphere, this is not possible i > Ry
iivariance-based: All points v from the inner sphere (||=1|2 Rí) can be perturbed to x", where D(x) = D(x*) + (x*), despite being fact on the outer sphere after the perturbation has been added (i.c.. | 2 R2) via following steps:
1. Fixing the used dimensions: d-n. 20 1.....mm
2. Perturbation of unused dimensions: x n....... (zd where scaling > 0 is
chosen such that ||="._.....lll ||| = R2 !
For points x from the outer sphere, this is not possible if Ri.
B DETAILS ABOUT MODEL-AGNOSTIC INVARIANCE BASED ATTACKS
Here, give details about model-agnostic avariance-based adversarial attacks MNIST.

---

Under review as a conference paper at ICLR 2020
Generating fu-invariant adversarial examples. Assume we given training set X consisting of labeled example pairs (æt, y). As input our algorithm accepts an example & with oracle label (()) y. Image è with label y = 8 is given in Figure 4 (a).
Define S {ar (a, E x. # ÿ}. the set of training examples with a different label. Now we define T to be the set of transformations that we allow: rotations by up 20 degrees, horizontal or vertical shifts by up to 6 pixels (out of 28), shears by up to 20%, and re-sizing by up to 50%.
Now, generate the new augmented training set X assumption, each of these examples is labeled correctly by the oracle. In our experiments, we verify the validity of this assumption through human study and omit any candidate adversarial example that violates this assumption. Finally, we search for
((.()))).t) t T. (z. 2). By
x² v
arg min ||x" - Ello. (x* w A)E3*
By construction, we know that è and x= are similar in pixel space but have different label. Figure (b-c) show this step of the process. Next, we introduce a number of refinements to make x be "more similar" to f.. This reduces the to distortion introduced to create an invariance-based adversarial example-compared to directly returning x" as the adversarial example.
First, we define 8 - L x01 0.5 where the absolute value and comparison operator are taken element-wise. Intuitively, 5 represents the pixels that substantially change between x² and f. We choose 0.5 an arbitrary threshold representing how much a pixel changes before we consider the change "important". This step is shown in Figure 4 (d). Along with ō containing the useful changes that are responsible for changing the oracle class label of á. also contains irrelevant changes that superficial and do not contribute to changing the oracle class label. For example, in Figure | (d) notice that the green cluster is the only semantically important change; both the red and blue changes are not necessary.
To identify and remove the superficial changes, we perform spectral clustering on We compute 6, by enumerating all possible subsets of clusters of pixel regions. This gives many possible potential adversarial examples x; = ă + d4. Notice these are only potential because we may not actually have applied the necessary change that actually changes the class label.
We show three of the eight possible candidates in Figure 4 In order to alleviate the need for human inspection of each candidate x; to determine which of these potential adversarial examples is actually misclassified, we follow an approach from Defense-GAN Samangouei et al. (2018) and the Robust Manifold Defense [lyas et al. (2017): we take the generator from a GAN and use it assign likelihood score to the image. We make one small refinement, and use an MC--GANMiii & Osindero (2014) and compute the class-conditional likelihood of this image occurring. This process reduces to distortion by 50% on average.
As small refinement, we find that initially filtering X by 20% least-canonical examples makes the attack succeed more often.
Generating -ivaariat adversarial examples. Our approach for generating too -ivvariat ex- amples follows similar ideas as for the to case, but is conceptually simpler as the perturbation budget can be applied independently for each pixel (as we will see, our lox attack is however less effective than the to one, so further optimizations may prove usefull.
We build an augmented training set X' as in the lo case. Instead of looking for the closest nearest neighbor for some example á with label ( ÿ. we restrict our search to examples (x", v²,() E X' with specific target labels y", which we' 'e empirically found to produce more convincing examples (e.g.. always match digits representing 1, with target digit representing either 7 or a 4). We then simply apply an loc bounded perturbation (with = 0.3) to I by interpolating with x", so to minimize the distance between 5 and the chosen target example
12

---

Under review as a conference paper at ICLR 2020
C INVARIANCE-BANE ADVERSARIAL EXAMPLES FOR BINARIZED MNIST
Number of pixels (log scale)
10a
10a
0.4
0.6
1.0
Figure 7: Histogram of MNIST pixel values (note the log scale on the y-axis) with two modes around and Hence, binarizing inputs a MNIST model does not impact its performance importantly,
Source
Source
Inveriance
Source
Invartance
Source
Invariance
3
8
7
9
4
000)
000
000g
0(rg)
3 Bints,)
3 Bintiy)
9
9
Bincs.)
Binusg)
Binix.)
Blmck))
Blmnk;)
Binck)
Figure 8: Invariance-based adversarial examples for a toy f model on MNIST. By thresh- olding inputs, the model is robust perturbations 8 such that || 0.5. Adversarial examples (top-right of each set of 4 images) are labeled differently by human. However, they become identical after binarization; the model thus labels both images confidently in the source image's class.
D COMPLETE SET OF 100 INVARIANCE ADVERSARIAL EXAMPLES
Below we give the 100 sendeel test images along with the invariance adversarial examples that were shown during the human study.
13

---

Under review as a conference paper at ICLR 2020
D.1
ORIGINAL IMAGES
4
9
4
2
7
2
6
8
7
7
8
0
8
9
22
(
2
4
9
/
7
2
0
S
1
4
3
5
8
P
3
2
O
9
1
8
2
7
C
8
2
2
O
3
6
8
9
4
6
2
8
/
9
8
3
2
6
4
7
6
1
/
6
4
7
0
6
O
4
/
2
9
6
0
4
1
7
S
9
O
3
2
6
3
8
7
8
8
6
7
9
D.2
to
INVARIANCE ADVERSARIAL EXAMPLES
4
7
2
7
2
6
3
Z
7
8
7
A
9
0
2
9
4
7
2
6
S
1
7
5
S
3
O
3
7
2
O
4
7
6
2
7
O
S
7
7
C
3
C
8
6
4
6
7
5
7
7
8
6
2
4
9
1
1
7
3
0
4
7
4
6
O
4
3
7
2
7
6
Q
4
/
7
3
7
O
8
2
S
5
3
7
8
4
6
7
7
14

---

Under review as a conference paper at ICLR 2020
D.3
lo
INVARIANCE ADVERSARIAL EXAMPLES
4
2
2
8
7
3
8
0
8
7
2
1
2
4
9
7
9
0
8
1
4
3
3
8
4
8
2
O
9
7
6
2
2
3
8
2
7
O
3
6
9
4
6
2
8
7
9
8
8
2
6
4
9
6
1
7
0
6
4
7
6
Q
4
2
7
3
9
6
0
4
7
7
S
9
3
2
6
3
8
2
8
7
3
6
7
9
15


Reviews:
[{"review_id": "rygGyhMTFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n================ Update after reading the rebuttal and the revised paper ======================================= \n\nI have now read the author rebuttal and the revised paper. I had raised two main issues with the paper in my initial review (see below): 1) experiments don't provide enough support for the main claim; 2) the main claim seems to contradict an earlier result by Engstrom et al. (2019). \n\nWith respect to the second point, the authors have responded that Engstrom et al. (2019) use relatively small perturbation sizes during adversarial training and that the trade-off between invariance vs. perturbation based adversarial robustness may be different for such small perturbation sizes. This sounds plausible, although I would have much preferred a more direct demonstration of this (for example, by adversarially training MNIST models with different perturbation sizes, then looking at their performance on invariance-based adversarial examples).\n\nWith respect to the first point, the authors have pointed to some numbers in Table 1, but my criticism still stands: for example, the perturbation-robust ABS performs better than the undefended CNN model (similarly for the l_inf PGD model, even though it is trained for a different metric). This seems to contradict the authors' claim that excessive perturbation robustness leads to worse performance on invariance-based adversarial examples.\n\nIn conclusion, I very much appreciate the authors' rebuttal and I definitely think there are some interesting ideas in this paper. However, given that one of my main concerns has not been addressed and the other one only insufficiently addressed, I'm inclined to keep my score as it is. Going forward, I would encourage the authors to develop the trade-off between invariance vs. perturbation based adversarial sensitivity for different perturbation sizes more thoroughly.\n\n========================================================================================================\n\nThis paper argues that the standard notion of l_p-norm bounded adversarial examples does not adequately capture all types of adversarial examples we may care about. In particular, the authors note the existence of adversarial examples caused by the excessive invariance of classifiers to semantically meaningful perturbations. This was noted in earlier works as well, but the current paper purports to establish a link between l_p-ball robustness and invariance-based adversarial examples. It also introduces a method to generate such invariance-based adversarial examples. There are interesting ideas in this paper, however I have some questions and concerns about some of the claims made in the paper that I would like to see addressed. Here are the main issues for me:\n\n1) If I’m not mistaken, the main claim of the paper, namely that l_p-ball robustness worsens the performance on invariance-based adversarial examples does not actually seem to be supported by any result in the paper beyond some of the simplest, toyest examples.  For example, in the MNIST examples in Table 1, I don’t see how l_p-ball robust models are performing worse than the other models on the invariance-based adversarial examples (the results seem mixed at best). Even in some of the toy examples, this claim is not supported: for example, in the adversarial spheres example, the l_p-ball robust classifier would correspond to the max-margin classifier, which is not vulnerable to invariance-based adversarial examples. Even the ad-hoc sub-optimal classifier chosen in this example doesn't show that l_p-ball robust models are more vulnerable to invariance-based adversarial examples, just that the two phenomena are distinct. There’s the single ImageNet example in Figure 6, which is cute and is consistent with the claim, however it’s just a single example. Also please note that this example does not work with the l_inf norm, so it just shows the inadequacy of the l_2 norm, not of l_p norms in general.\n\n2) The main claim of this paper seems to contradict a result in Engstrom et al. (2019) (https://arxiv.org/abs/1906.00945). They also demonstrate the existence of invariance-based adversarial examples in non-robust models (e.g. see their Figure 2), but their results seem to suggest that these types of examples do not arise in robust models (models trained with adversarial training). Please discuss this paper and clarify the seeming discrepancy between the results there and your claims.\n\nOther issues: \n\n3) It is really hard to follow the invariance-based adversarial example generation process described in section 4.2. Please describe this more clearly, motivating each step of the process (currently the steps seem ad hoc, it is not explained why each step is needed), so that a reader can understand the generation process at a high level without having to consult the appendix. \n\n4) The labels are misaligned in Figure 4.\n\n5) In section 4.3, it is mentioned that “we additionally hand-crafted 50 invariance adversarial examples under the specific norms”. It is not explained at all why these additional examples are needed and it is not described how exactly these are generated (neither in the main text, nor in the appendix as far as I can see).\n\n6) In Table 1, the different models are not described at all (neither in the main text, nor in the appendix). They are just acronyms right now. Please describe what these models are.\n\n7) The idea that there may be trade-offs between different notions of robustness and that l_p-ball robustness is not the be all and end all of all robustness measures have been noted in some previous works: for example, Yin et al. (2019): https://arxiv.org/abs/1906.08988 Although the phenomenon discussed in Yin et al. (2019) is different from the one highlighted in the current paper, the main message seems quite similar. So, please discuss this connection in your paper. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJlJAnX3YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thanks for your response. It has addressed most of my concerns. I'd like to increase my score.\n\n-----------------\n\nThis paper examines the perturbation-based and invariance-based robustness of deep neural networks. They found that models trained to be robust under l_p threat model are more vulnerable to invariance-based adversarial examples. This paper provides several illustrative examples to tell the reasons behind this phenomenon. An attack method for generating invariance-based adversarial examples is also proposed to attack the (provably robust) models.\n\nOverall, this paper analyzes the invariance-based robustness of deep neural networks with norm-bounded robustness, which is an interesting problem. Several illustrative examples provide reasons on why models robust to perturbation-based adversarial examples could be more sensitive to invariance-based adversarial examples.\n\nHowever, my main concern is on the contributions of this paper. A clear contribution of the paper is to study the perturbation-based and invariance-based robustness together, and present several demo examples to illustrate that. Are there any other contributions in this paper? Is the proposed attack method novel compared with previous methods? And what are the differences between the proposed attack with previous methods? The authors could discuss more on the attack method and compare its performance with others.\n\nI'd like to raise my currently rating based on the author feedback.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "ByxBzghkcH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a fine-grained definition for adversarial robustness, dividing adversarial robustness into perturbation robustness and invariance robustness. Same as the previous definition of adversarial robustness (robustness against imperceptible perturbations), perturbation robustness reflects the model's ability to maintain the prediction after a label-preserving transform. Invariance robustness, on the other hand, reflects the model's flexibility against invariance-based adversaries which changes the actual label within the norm ball around clean samples. With examples from the two-sphere experiment, MNIST dataset with large epsilons and a triplet of natural images, it reaches the conclusion that training models to be invariant to any sample within a norm ball is a bad idea when epsilon is too large. \n\nI like this paper in that it points out the problems with some established evaluation protocols for adversarial robustness, e.g., evaluating the robustness on MNIST with a max l_infty norm of 0.4. However, it comes at no surprise that large epsilon gives rise to non-label-preserving perturbations on images. No automated solution can be inspired from the paper. A more valuable direction would be to evaluate the minimum l_p distance between classes, but it seems intractable at the moment. \n\nAlso, it should be emphasized that the invariance-based adversarial examples exist only when epsilon is improperly high for image classification tasks. This can also be verified by the two-sphere experiments. By taking average of the accuracy on the inner sphere and outer sphere, the accuracy against perturbation attack drops before invariance attack with the increasing epsilon, demonstrating invariance attack is only a problem when epsilon is too large for training the model. For other tasks such as language understanding, invariance-based adversarial examples may be a much severe problem for the seemingly robust models. \n\nAs a conclusion, I tend to accept this paper to draw more attention to a better notion of robustness, or developing more sophisticated approaches to defending against perturbation and invariance attacks simultaneously at a larger epsilon. However, the current version is still short of any foreseeable solution. Still, this is perhaps the best we can hope for.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the relationship betwee:

- perturbations to an input x which change predictions of a model but not the ground truth label
- perturbations to an input x which do not change a model's prediction but do chance the ground truth label. 

The authors show that achieving robustness to the former need not guarantee robustness to the latter. 

While these ideas are interesting, the reviewers would like to see a tighter connection between the two forms of robustness developed. 
Decision: Reject