Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness

1 INTRODUCTION . Research on adversarial examples is motivated by a spectrum of questions . These range from the security of models deployed in the presence of real-world adversaries , to the need to capture limitations of representations and their ( in ) ability to generalize ( Gilmer et al. , 2018a ) . The broadest accepted definition of an adversarial example is “ an input to a ML model that is intentionally designed by an attacker to fool the model into producing an incorrect output ” ( Goodfellow & Papernot , 2017 ) . Many formal definitions of adversarial examples were introduced since their initial discovery ( Szegedy et al. , 2013 ; Biggio et al. , 2013 ) . In a majority of work , adversarial examples are formalized as adding a perturbation δ to some test example x to obtain an input x∗ that produces an incorrect model outcome.1 We refer to this class of malicious inputs as perturbation-based adversarial examples . To enable concrete progress , the adversary ’ s capabilities may optionally be constrained by placing a bound on the maximum perturbation δ added to the original input . The goal of this constraint is to ensure that semantics of the input are left unaffected by the perturbation δ . In the computer vision domain , ` p norms have grown to be the default metric to measure semantic similarity . This led to a series of proposals for increasing the robustness of models to perturbation-based adversaries that operate within the constraints of an ` p ball . These include robust optimization ( Madry et al. , 2017 ) , explicit regularization of a model ’ s Lipschitz constant ( Cisse et al. , 2017 ) , or a variety of techniques to build models that are provably robust to small ` p perturbations ( Wong & Kolter , 2018 ; Raghunathan et al. , 2018 ; Zhang et al. , 2019 ) . In this paper , we show that optimizing a model ’ s robustness to lp-bounded perturbations is not only insufficient to address the lack of generalization identified via adversarial examples , but also potentially harmful . Intuitively , as ` p distances are only crude approximations to the true visual 1Here , an incorrect output either refers to the model returning any class different from the original source class of the input , or a specific target class chosen by the adversary prior to searching for a perturbation . Perturbation-Unrobust Model Perturbation-Robust Model similarity in a given task , over-optimizing a model ’ s robustness to ` p-bounded perturbations also renders the model invariant to actual semantics of the underlying task . Excessive invariance of a model to class semantics can give rise to a new class of adversarial perturbations called invariance adversarial examples ( Jacobsen et al. , 2019 ) . These are perturbations that successfully change the human-assigned ground-truth label of an input , while keeping the model ’ s classification of the perturbed input unchanged ( for example , we take an image of a ‘ 1 ’ digit and perturb it—without changing the model ’ s classification—until a human would agree that the digit now represents a ‘ 7 ’ ) . See Figure 1 for an intuitive illustration of this phenomenon . Whereas previous efforts ( e.g. , ( Jacobsen et al. , 2019 ) ) explored different notions of adversarial examples independently , our work is the first to expose a complex relationship between perturbationbased and invariance-based adversarial examples . Specifically , we introduce analytical constructions and empirical evidence that shows that increasing a model ’ s robustness to perturbation-based adversarial examples can increase the model ’ s vulnerability to invariance-based adversarial examples . We formally show how to construct a model that is robust to perturbation-based adversarial examples but not to invariance-based adversarial examples . We further demonstrate how an imperfect model for the adversarial spheres task proposed by Gilmer et al . ( 2018b ) is either vulnerable to perturbationbased or invariance-based attacks—depending on whether the point attacked is on the inner or outer sphere . Hence , these two types of adversarial examples are needed to fully account for model failures . Finally , we empirically demonstrate with a new attack how an adversary can exploit the excessive invariance of certain models . We introduce the first algorithm that crafts invariance-based adversarial examples for the ` 0 and ` ∞ norms , and show that many common models disagree with human labelers on these examples . In particular , our algorithm breaks a provably-robust defense on MNIST ( Zhang et al. , 2019 ) .2 This model is certified to have 87 % test-accuracy under l∞-perturbations of radius ε = 0.4 . Yet , on our automatically-generated invariance-based adversarial examples , the model only agrees with the human-assigned label in 60 % of the cases ; when we manually craft invariance adversarial examples we reduce accuracy to just 12 % as determined by an ensemble of humans . 2 PERTURBATION- AND INVARIANCE-BASED ADVERSARIAL EXAMPLES . In order to make precise statements about adversarial examples , we begin with two definitions . 2Note that while MNIST is typically a poor choice for studying adversarial robustness ( Carlini et al. , 2019 ) , in our case we explicitly chose to work with MNIST because it is the only dataset for which models have been claimed to be provably robust to non-negligible lp norm balls . Definition 1 ( Perturbation-based Adversarial Examples ) . Let G denote the i-th layer , logit or argmax of the classifier . A Perturbation-based adversarial example ( or perturbation adversarial ) x∗ ∈ Rd corresponding to a legitimate test input x ∈ Rd fulfills : 1 . Created by adversary : x∗ ∈ Rd is created by an algorithm A : Rd → Rd with x 7→ x∗ . 2 . Perturbation of output : ‖G ( x∗ ) − G ( x ) ‖ > δ and O ( x∗ ) = O ( x ) , where perturbation δ > 0 is set by the adversary and O : Rd → { 1 , . . . , C } denotes the oracle . Furthermore , x∗ is -bounded if ‖x− x∗‖ < , where ‖ · ‖ is a norm on Rd and > 0 . Property ( i ) allows us to distinguish perturbation adversarial examples from points that are misclassified by the model without adversarial intervention . Furthermore , the above definition incorporates also adversarial perturbations designed for hidden features as in ( Sabour et al. , 2016 ) , while usually the decision of the classifier D ( argmax-operation on logits ) is used as the perturbation target . Our definition also identifies -bounded perturbation-based adversarial examples ( Goodfellow et al. , 2015 ) as a specific case of unbounded perturbation-based adversarial examples . However , our analysis primarily considers the latter , which correspond to the threat model of a stronger adversary . Definition 2 ( Invariance-based Adversarial Examples ) . Let G denote the i-th layer , logit or argmax of the classifier . An invariance-based adversarial example ( or invariance adversarial ) x∗ ∈ Rd corresponding to a legitimate test input x ∈ Rd fulfills : 1 . Created by adversary : x∗ ∈ Rd is created by an algorithm A : Rd → Rd with x 7→ x∗ . 2 . Lies in pre-image of x under G : G ( x∗ ) = G ( x ) and O ( x ) 6= O ( x∗ ) , where O : Rd → { 1 , . . . , C } denotes the oracle . As a consequence , D ( x ) = D ( x∗ ) also holds for invariance-based adversarial examples , where D is the output of the classifier . Intuitively , adversarial perturbations cause the output of the classifier to change , while the oracle would still label the new input x∗ in the original source class . Whereas perturbation-based adversarial examples exploit the classifier ’ s excessive sensitivity in task-irrelevant directions , invariance-based adversarial examples explore the classifier ’ s pre-image to identify excessive invariance in task-relevant directions : its prediction is unchanged while the oracle ’ s output differs . Briefly put , perturbation-based and invariance-based adversarial examples are complementary failure modes of the learned classifier . 3 ROBUSTNESS TO PERTURBATION-BASED ADVERSARIAL EXAMPLES CAN . CAUSE INVARIANCE-BASED VULNERABILITIES We now investigate the relationship between the two adversarial example definitions from Section 2 . So far , it has been unclear whether solving perturbation-based adversarial examples implies solving invariance-based adversarial examples , and vice versa . In the following , we show that this relationship is intricate and developing models robust in one of the two settings only would be insufficient . In a general setting , invariance and stability can be uncoupled . For example , consider a linear classifier with matrix A . The perturbation-robustness is tightly related to forward stability ( largest singular value of A ) . On the other hand , the invariance-view relates to the stability of the inverse ( smallest singular value of A ) and to the null-space of A . As largest and smallest singular values are uncoupled for general matrices A , the relationship between both viewpoints is likely non-trivial in practice . 3.1 BUILDING OUR INTUITION WITH EXTREME UNIFORM CONTINUITY . In the extreme , a classifier achieving perfect uniform continuity would be a constant classifier . Let D : Rn → [ 0 , 1 ] C denote a classifier with D ( x ) = y∗ for all x ∈ Rd . As the classifier maps all inputs to the same output y∗ , there exist no x∗ , such that D ( x ) 6= D ( x∗ ) . Thus , the model is trivially perturbation-robust ( at the expense of decreased utility ) . On the other hand , the pre-image of y∗ under D is the entire input space , thus D is arbitrarily vulnerable to invariance-based adversarial examples . Because this toy model is a constant function over the input domain , no perturbation of an initially correctly classified input can change its prediction . This trivial model illustrates how one not only needs to control sensitivity but also invariance alongside accuracy to obtain a robust model . Hence , we argue that the often-discussed tradeoff between accuracy and robustness ( see Tsipras et al . ( 2019 ) for a recent treatment ) should in fact take into account at least three notions : accuracy , sensitivity , and invariance . This is depicted in Figure 1 . In the following , we present arguments as for why this insight can also extend to almost perfect classifiers . 3.2 COMPARING INVARIANCE-BASED AND PERTURBATION-BASED ROBUSTNESS . We now show how the analysis of perturbation-based and invariance-based adversarial examples can uncover different model failures . To do so , we consider the synthetic adversarial spheres problem of Gilmer et al . ( 2018b ) . The goal of this synthetic task is to distinguish points from two cocentric spheres ( class 1 : ‖x‖2 = R1 and class 2 : ‖x‖2 = R1 ) with different radii R1 and R2 . The dataset was designed such that a robust ( max-margin ) classifier can be formulated as : D∗ ( x ) = sign ( ‖x‖2 − R1 +R2 2 ) . Our analysis considers a similar , but slightly sub-optimal classifier in order to study model failures in a controlled setting : D ( x ) = sign ( ‖x1 , ... , d−n‖2 − b ) , which computes the norm of x from its first d− n cartesian-coordinates and outputs -1 ( resp . +1 ) for the inner ( resp . outer ) sphere . The bias b is chosen based on finite training set ( see Appendix A ) . Even though this sub-optimal classifier reaches nearly 100 % on finite test data , the model is imperfect in the presence of adversaries that operate on the manifold ( i.e. , produce adversarial examples that remain on one of the two spheres but are misclassified ) . Most interestingly , the perturbation-based and invariance-based approaches uncover different failures ( see Appendix A for details on the attacks ) : • Perturbation-based : All points x from the outer sphere ( i.e. , ‖x‖2 = R2 ) can be perturbed to x∗ , where O ( x ) = D ( x ) 6= D ( x∗ ) while staying on the outer sphere ( i.e. , ‖x∗‖2 = R2 ) . • Invariance-based : All points x from the inner sphere ( ‖x‖2 = R1 ) can be perturbed to x∗ , where D ( x ) = D ( x∗ ) 6= O ( x∗ ) , despite being in fact on the outer sphere after the perturbation has been added ( i.e. , ‖x∗‖2 = R2 ) . In Figure 2 , we plot the mean accuracy over points sampled either from the inner or outer sphere , as a function of the norm of the adversarial manipulation added to create perturbation-based and invariance-based adversarial examples . This illustrates how the robustness regime differs significantly between the two variants of adversarial examples . Therefore , by looking only at perturbation-based ( respectively invariance-based ) adversarial examples , important model failures may be overlooked . This is exacerbated when the data is sampled in an unbalanced fashion from the two spheres : the inner sphere is robust to perturbation adversarial examples while the outer sphere is robust to invariance adversarial examples ( for accurate models ) .

Summary: This paper proposes a fine-grained definition for adversarial robustness, dividing adversarial robustness into perturbation robustness and invariance robustness. Same as the previous definition of adversarial robustness (robustness against imperceptible perturbations), perturbation robustness reflects the model's ability to maintain the prediction after a label-preserving transform. Invariance robustness, on the other hand, reflects the model's flexibility against invariance-based adversaries which changes the actual label within the norm ball around clean samples. With examples from the two-sphere experiment, MNIST dataset with large epsilons and a triplet of natural images, it reaches the conclusion that training models to be invariant to any sample within a norm ball is a bad idea when epsilon is too large.