Excessive Invariance Causes Adversarial Vulnerability

Despite their impressive performance , deep neural networks exhibit striking failures on out-of-distribution inputs . One core idea of adversarial example research is to reveal neural network errors under such distribution shifts . We decompose these errors into two complementary sources : sensitivity and invariance . We show deep networks are not only too sensitive to task-irrelevant changes of their input , as is well-known from -adversarial examples , but are also too invariant to a wide range of task-relevant changes , thus making vast regions in input space vulnerable to adversarial attacks . We show such excessive invariance occurs across various tasks and architecture types . On MNIST and ImageNet one can manipulate the class-specific content of almost any image without changing the hidden activations . We identify an insufficiency of the standard cross-entropy loss as a reason for these failures . Further , we extend this objective based on an informationtheoretic analysis so it encourages the model to consider all task-dependent features in its decision . This provides the first approach tailored explicitly to overcome excessive invariance and resulting vulnerabilities . 1 INTRODUCTION . Adversarial vulnerability is one of the most iconic failure cases of modern machine learning models ( Szegedy et al. , 2013 ) and a prime example of their weakness in out-of-distribution generalization . It is particularly striking that under i.i.d . settings deep networks show superhuman performance on many tasks ( LeCun et al. , 2015 ) , while tiny targeted shifts of the input distribution can cause them to make unintuitive mistakes . The reason for these failures and how they may be avoided or at least mitigated is an active research area ( Schmidt et al. , 2018 ; Gilmer et al. , 2018b ; Bubeck et al. , 2018 ) . So far , the study of adversarial examples has mostly been concerned with the setting of small perturbation , or -adversaries ( Goodfellow et al. , 2015 ; Madry et al. , 2017 ; Raghunathan et al. , 2018 ) . Perturbation-based adversarial examples are appealing because they allow to quantitatively measure notions of adversarial robustness ( Brendel et al. , 2018 ) . However , recent work argued that the perturbation-based approach is unrealistically restrictive and called for the need of generalizing the concept of adversarial examples to the unrestricted case , including any input crafted to be misinterpreted by the learned model ( Song et al. , 2018 ; Brown et al. , 2018 ) . Yet , settings beyond -robustness are hard to formalize ( Gilmer et al. , 2018a ) . We argue here for an alternative , complementary viewpoint on the problem of adversarial examples . Instead of focusing on transformations erroneously crossing the decision-boundary of classifiers , we focus on excessive invariance as a major cause for adversarial vulnerability . To this end , we introduce the concept of invariance-based adversarial examples and show that class-specific content of almost any input can be changed arbitrarily without changing activations of the network , as illustrated in figure 1 for ImageNet . This viewpoint opens up new directions to analyze and control crucial aspects underlying vulnerability to unrestricted adversarial examples . The invariance perspective suggests that adversarial vulnerability is a consequence of narrow learning , yielding classifiers that rely only on few highly predictive features in their decisions . This has also been supported by the observation that deep networks strongly rely on spectral statistical regularities ( Jo & Bengio , 2017 ) , or stationary statistics ( Gatys et al. , 2017 ) to make their decisions , rather than more abstract features like shape and appearance . We hypothesize that a major reason for this excessive invariance can be understood from an information-theoretic viewpoint of crossentropy , which maximizes a bound on the mutual information between labels and representation , giving no incentive to explain all class-dependent aspects of the input . This may be desirable in some cases , but to achieve truly general understanding of a scene or an object , machine learning models have to learn to successfully separate essence from nuisance and subsequently generalize even under shifted input distributions . Our contributions : • We identify excessive invariance underlying striking failures in deep networks and formalize the connection to adversarial examples . • We show invariance-based adversarial examples can be observed across various tasks and types of deep network architectures . • We propose an invertible network architecture that gives explicit access to its decision space , enabling class-specific manipulations to images while leaving all dimensions of the representation seen by the final classifier invariant . • From an information-theoretic viewpoint , we identify the cross-entropy objective as a major reason for the observed failures . Leveraging invertible networks , we propose an alternative objective that provably reduces excessive invariance and works well in practice . 2 TWO COMPLEMENTARY APPROACHES TO ADVERSARIAL EXAMPLES . In this section , we define pre-images and establish a link to adversarial examples . Definition 1 ( Pre-images / Invariance ) . Let F : Rd → RC be a neural network , F = fL ◦ · · · ◦ f1 with layers fi and let Fi denote the network up to layer i . Further , let D : Rd → { 1 , . . . , C } be a classifier with D = arg maxk=1 , ... , C softmax ( F ( x ) ) k. Then , for input x ∈ Rd , we define the following pre-images ( i ) i-th Layer pre-image : { x∗ ∈ Rd | Fi ( x∗ ) = Fi ( x ) } ( ii ) Logit pre-image : { x∗ ∈ Rd | F ( x∗ ) = F ( x ) } ( iii ) Argmax pre-image : { x∗ ∈ Rd | D ( x∗ ) = D ( x ) } , where ( i ) ⊂ ( ii ) ⊂ ( iii ) by the compositional nature of D. Moreover , the ( sub- ) network is invariant to perturbations ∆x which satisfy x∗ = x+ ∆x . Non-trivial pre-images ( pre-images containing more elements than input x ) after the i-th layer occur if the chain fi ◦ · · · ◦ f1 is not injective , for instance due to subsampling or non-injective activation functions like ReLU ( Behrmann et al. , 2018a ) . This accumulated invariance can become problematic if not controlled properly , as we will show in the following . We define perturbation-based adversarial examples by introducing the notion of an oracle ( e.g. , a human decision-maker or the unknown input-output function considered in learning theory ) : Definition 2 ( Perturbation-based Adversarial Examples ) . A Perturbation-based adversarial example x∗ ∈ Rd of x ∈ Rd fulfills : ( i ) Perturbation of decision : D ( x∗ ) 6= o ( x∗ ) and D ( x ) 6= D ( x∗ ) , where D : Rd → { 1 , . . . , C } is the classifier and o : Rd → { 1 , . . . , C } is the oracle . ( ii ) Created by adversary : x∗ ∈ Rd is created by an algorithm A : Rd → Rd with x 7→ x∗ . Further , -bounded adversarial ex . x∗ of x fulfill ‖x− x∗‖ < , ‖ · ‖ a norm on Rd and > 0 . Usually , such examples are constructed as -bounded adversarial examples ( Goodfellow et al. , 2015 ) . However , as our goal is to characterize general invariances of the network , we do not restrict ourselves to bounded perturbations . Definition 3 ( Invariance-based Adversarial Examples ) . Let G denote the i-th layer , logits or the classifier ( Definition 1 ) and let x∗ 6= x be in the G pre-image of x and and o an oracle ( Definition 2 ) . Then , an invariance-based adversarial example fulfills o ( x ) 6= o ( x∗ ) , while G ( x ) = G ( x∗ ) ( and hence D ( x ) = D ( x∗ ) ) . Intuitively , adversarial perturbations cause the output of the classifier to change while the oracle would still consider the new input x∗ as being from the original class . Hence in the context of - bounded perturbations , the classifier is too sensitive to task-irrelevant changes . On the other hand , movements in the pre-image leave the classifier invariant . If those movements induce a change in class as judged by the oracle , we call these invariance-based adversarial examples . In this case , however , the classifier is too insensitive to task-relevant changes . In conclusion , these two modes are complementary to each other , whereas both constitute failure modes of the learned classifier . When not restricting to -perturbations , perturbation-based and invariance-based adversarial examples yield the same input x∗ via x∗ = x1 + ∆x1 , D ( x ∗ ) 6= D ( x1 ) , o ( x∗ ) = o ( x1 ) ( 3 ) x∗ = x2 + ∆x2 , D ( x ∗ ) = D ( x2 ) , o ( x ∗ ) 6= o ( x2 ) , ( 4 ) with different reference points x1 and x2 , see Figure 2 . Hence , the key difference is the change of reference , which allows us to approach these failure modes from different directions . To connect these failure modes with an intuitive understanding of variations in the data , we now introduce the notion of invariance to nuisance and semantic variations , see also ( Achille & Soatto , 2018 ) . Definition 4 ( Semantic/ Nuisance perturbation of an input ) . Let o be an oracle ( Definition 2 ) and x ∈ Rd . Then , a perturbation ∆x of an input x ∈ Rd is called semantic , if o ( x ) 6= o ( x+ ∆x ) and nuisance if o ( x ) = o ( x+ ∆x ) . For example , such a nuisance perturbation could be a translation or occlusion in image classification . Further in Appendix A , we discuss the synthetic example called Adversarial Spheres from ( Gilmer et al. , 2018b ) , where nuisance and semantics can be explicitly formalized as rotation and norm scaling . 2.1 USING BIJECTIVE NETWORKS TO ANALYZE EXCESSIVE INVARIANCE As invariance-based adversarial examples manifest themselves in changes which do not affect the output of the network F , we need a generic approach that gives us access to the discarded nuisance variability . While feature nuisances are intractable to access for general architectures ( see comment after Definition 1 ) , invertible classifiers only remove nuisance variability in their final projection ( Jacobsen et al. , 2018 ) . For C < d , we denote the classifier as D : Rd → { 1 , ... , C } . Our contributions in this section are : ( 1 ) Introduce an invertible architecture with a simplified readout structure , allowing to exactly visualize manipulations in the hidden-space , ( 2 ) Propose an analytic attack based on this architecture allowing to analyze its decision-making , ( 3 ) Reveal striking invariance-based vulnerability in competitive classifiers . Bijective classifiers with simplified readout . We build deep networks that give access to their decision space by removing the final linear mapping onto the class probes in invertible RevNet-classifiers and call these networks fully invertible RevNets . The fully invertible RevNet classifier can be written as Dθ = arg maxk=1 , ... , C softmax ( Fθ ( x ) k ) , where Fθ represents the bijective network . We denote z = Fθ ( x ) , zs = z1 , ... , C as the logits ( semantic variables ) and zn = zC+1 , ... , d as the nuisance variables ( zn is not used for classification ) . In practice we choose the first C indices of the final z tensor or apply a more sophiscticated DCT scheme ( see appendix D ) to set the subspace zs , but other choices work as well . The architecture of the network is similar to iRevNets ( Jacobsen et al. , 2018 ) with some additional Glow components like actnorm ( Kingma & Dhariwal , 2018 ) , squeezing , dimension splitting and affine block structure ( Dinh et al. , 2017 ) , see Figure 3 for a graphical description . As all components are common in the bijective network literature , we refer the reader to Appendix D for exact training and architecture details . Due to its simple readout structure , the resulting invertible network allows to qualitatively and quantitatively investigate the task-specific content in nuisance and logit variables . Despite this restriction , we achieve performance on par with commonly-used baselines on MNIST and ImageNet , see Table 1 and Appendix D. Analytic attack . To analyze the trained models , we can sample elements from the logit pre-image by computing xmet = F−1 ( zs , z̃n ) , where zs and z̃n are taken from two different inputs . We term this heuristic metameric sampling . The samples would be from the true data distribution if the subspaces would be factorized as P ( zs , zn ) = P ( zs ) P ( zn ) . Experimentally we find that logit metamers are revealing adversarial subspaces and are visually close to natural images on ImageNet . Thus , metameric sampling gives us an analytic tool to inspect dependencies between semantic and nuisance variables without the need for expensive and approximate optimization procedures . Attack on adversarial spheres . First , we evaluate our analytic attack on the synthetic spheres dataset , where the task is to classify samples as belonging to one out of two spheres with different radii . We choose the sphere dimensionality to be d = 100 and the radii : R1 = 1 , R2 = 10 . By training a fully-connected fully invertible RevNet , we obtain 100 % accuracy . After training we visualize the decision-boundaries of the original classifier D and a posthoc trained classifier on zn ( nuisance classifier ) , see Figure 4 . We densely sample points in a 2D subspace , following Gilmer et al . ( 2018b ) , to visualize two cases : 1 ) the decision-boundary on a 2D plane spanned by two randomly chosen data points , 2 ) the decision-boundary spanned by metameric sample xmet and reference point x . In the metameric sample subspace we identify excessive invariance of the classifier . Here , it is possible to move any point from the inner sphere to the outer sphere without changing the classifiers predictions . However , this is not possible for the classifier trained on zn . Most notably , the visualized failure is not due to a lack of data seen during training , but rather due to excessive invariance of the original classifier D on zs . Thus , the nuisance classifier on zn does not exhibit the same adversarial vulnerability in its subspace . Attack on MNIST and ImageNet . After validating its potential to uncover adversarial subspaces , we apply metameric sampling to fully invertible RevNets trained on MNIST and Imagenet , see Figure 5 . The result is striking , as the nuisance variables zn are dominating the visual appearance of the logit metamers , making it possible to attach any semantic content to any logit activation pattern . Note that the entire 1000-dimensional feature vector containing probabilities over all ImageNet classses remains unchanged by any of the transformations we apply . To show our findings are not a particular property of bijective networks , we attack an ImageNet trained ResNet152 with a gradientbased version of our metameric attack , also known as feature adversaries ( Sabour et al. , 2016 ) . The attack minimizes the mean squared error between a given set of logits from one image to another image ( see appendix B for details ) . The attack shows the same failures for non-bijective models . This result highlights the general relevance of our finding and poses the question of the origin of this excessive invariance , which we will analyze in the following section .

Summary: This paper explores adversarial examples by investigating an invertible neural network. They begin by first correctly pointing out limitations with the commonly adopted "l_p adversarial example" definition in literature. The main idea involves looking at the preimage of different embeddings in the final layer of an invertible neural network. By training a classifier on top of the final embedding of the invertible network the authors are able to partition the final embedding into a set of "semantic variables", which are the components used for classification of the classifier, and a set of "nuisance variables" which are the complement of the logit variables. This partition allows the authors to define entire subspaces of adversarial images by holding the logit variables fixed and varying the nuisance variables, and applying the inverse to these modified embeddings. The authors are able to find many incorrectly classified images with this inversion technique. The authors then define a new loss which minimizes the mutual information between the nuisance variables and the predicted label.