Non-robust Features through the Lens of Universal Perturbations

1 INTRODUCTION . Modern deep neural networks perform extremely well across many prediction tasks , but they largely remain vulnerable to adversarial examples ( Szegedy et al. , 2014 ) . Models ’ brittleness to these small , imperceptible perturbations highlights one alarming way in which models deviate from humans . Recent work gives evidence that this deviation is due to the presence of useful non-robust features in our datasets ( Ilyas et al. , 2019 ) . These are brittle features that are sensitive to small perturbations—too small to be noticeable to humans , yet capture enough predictive signal to generalize well on the underlying classification task . When models rely on non-robust features , they become vulnerable to adversarial examples , as even small perturbations can flip the features ’ signal . While prior work gives evidence of non-robust features in natural datasets , we lack a more finegrained understanding of their properties . In general , we do not understand well how models make decisions , so it is unclear how much we can understand about these features that are believed to be imperceptible . A number of works suggest that these features may exploit certain properties of the dataset that are misaligned with human perception ( Ilyas et al. , 2019 ) , such as high-frequency information ( Yin et al. , 2019 ) , but much remains unknown . In this work , we illustrate how we can isolate more human-aligned non-robust features by imposing additional constraints on adversarial perturbations . In particular , we revisit universal adversarial perturbations ( Moosavi-Dezfooli et al. , 2017a ) , i.e . adversarial perturbations that generalize across many inputs . Prior works have observed that these perturbations appear to be semantic ( Hayes & Danezis , 2019 ; Khrulkov & Oseledets , 2018 ; Liu et al. , 2019 ) . We demonstrate that universal perturbations possess additional human-aligned properties different from standard adversarial perturbations , and analyze the non-robust features leveraged by these perturbations . Concretely , our findings are : Universal perturbations have more human-aligned properties . We show that universality adversarial perturbations have additional human-aligned properties that distinguish them from standard adversarial perturbations ( e.g. , Figure 1 ) . In particular , ( 1 ) the most semantically identifiable local patches inside universal perturbations also contain the most signal ; and ( 2 ) universal perturbations are approximately spatially invariant , in that they are still effective after translations . Non-robust features can be semantically meaningful . We show that universal perturbations are primarily relying on non-robust features rather than robust ones . Specifically , we compare the sensitivity of natural and ( adversarially ) robust models to rescalings of these perturbations to demonstrate that universal perturbations likely rely on non-robust features . Together with our first finding , this shows that some non-robust features can be human-aligned . Universal perturbations contain less non-robust signal . We find that the non-robust features leveraged by universal perturbations have less predictive signal than those leveraged by standard adversarial perturbations , despite being more human-aligned . We measure both ( 1 ) generalizability to the original test set and ( 2 ) transferability of perturbations across independent models , following the methodology of Ilyas et al . ( 2019 ) . Under these metrics , universal perturbations consistently obtain non-trivial but substantially worse performance than standard adversarial perturbations . 2 PRELIMINARIES . We consider a standard classification task : given input-label samples ( x , y ) ∈ X × Y from a data distribution D , the goal is to to learn a classifier C : X → Y that generalizes to new data . Non-robust vs. robust features . Following Ilyas et al . ( 2019 ) , we introduce the following terminology . A useful feature for classification is a function that is ( positively ) correlated with the correct label in expectation . A feature is robustly useful if , even under adversarial perturbations ( within a specified set of valid perturbations ∆ ) the feature is still useful . Finally , a useful , non-robust feature is a feature that is useful but not robustly useful . These features are useful for classification in the standard setting , but can hurt accuracy in the adversarial setting ( since their correlation with the label can be reversed ) . For conciseness , throughout this paper we will refer to such features simply as non-robust features . Universal perturbations . A universal adversarial perturbation ( or just universal perturbation for short ) , as introduced in Moosavi-Dezfooli et al . ( 2017a ) , is a perturbation δ that causes the classifier C to predict the wrong label on a large fraction of inputs from D. We focus on targeted universal perturbations that fool C into predicting a specific ( usually incorrect ) target label t. Thus , a ( targeted ) universal perturbation δ ∈ ∆ satisfies P ( x , y ) ∼D [ C ( x+δ ) = t ] = ρ , where ρ is the attack success rate ( ASR ) of the universal perturbation . The only technical difference between universal perturbations and ( standard ) adversarial perturbations is the use of a single perturbation vector δ that is applied to all inputs . Thus , one can think of universality as a constraint for the perturbation δ to be inputindependent . Universal perturbations can leverage non-robust features in data , as we show here ; we refer to these simply as universal non-robust features . ` p perturbations . We study the case where ∆ is the set of ` p-bounded perturbations , i.e . ∆ = { δ ∈ Rd | ||δ||p ≤ } for p = 2 , ∞ . This is the most widely studied setting for research on adversarial examples and has proven to be an effective benchmark ( Carlini et al. , 2019 ) . Additionally , ` p-robustness appears to be aligned to a certain degree with the human visual system ( Tsipras et al. , 2019 ) . 2.1 COMPUTING UNIVERSAL PERTURBATIONS . We compute universal perturbations by using projected gradient descent ( PGD ) on the following optimization problem : min δ∈∆ E ( x , y ) ∼D [ L ( f ( x+ δ ) , t ) ] ( 1 ) where L is the standard cross-entropy loss function for classification , and f are the logits prior to classification . and t is the target label . While many different algorithms have been developed for computing universal perturbations with varying attack success rates , we do not consider them as achieving the highest rate is not necessary for our investigations . As observed in Moosavi-Dezfooli et al . ( 2017a ) , universal perturbations trained on only a fraction of the dataset still generalize to the rest of the dataset , so in practice it suffices to approximate the expectation in equation 1 using a relatively small batch of data drawn from D. We call the batch we optimize over the base set , and use K to refer to its size . We describe the selection of all hyperparameters in Appendix B . 2.2 EXPERIMENTAL SETTINGS . Datasets . We conduct our experiments on a subset of ImageNet ( Deng et al. , 2009 ) and CIFAR-10 ( Krizhevsky et al. , 2009 ) . The Mixed10 ImageNet dataset ( Engstrom et al. , 2019 ) , which we refer to as ImageNet-M10 , is formed by sub-selecting and grouping together semantically similar classes from ImageNet ; it provides a more computationally efficient alternative to the full dataset , while retaining some of ImageNet ’ s complexity ( see Appendix B.1 for more details about the dataset ) . In the main text , we focus our results on ImageNet-M10 , and in the Appendices we provide additional results on CIFAR-10 as well as visualizations from a full ImageNet model . Models . We use the standard ResNet-18 architecture ( He et al. , 2016 ) unless mentioned otherwise . 3 PROPERTIES OF UNIVERSAL PERTURBATIONS . We find that universal perturbations , in contrast to adversarial perturbations , are much more aligned with human perception . Beyond visualizations , we demonstrate this in the following ways : ( 1 ) local patches that are most semantically identifiable with the target class also contain the most predictive signal ; and ( 2 ) universal perturbations are approximately spatially invariant , whereas standard perturbations are not . 3.1 PERCEPTUAL ALIGNMENT . Visual comparison with standard adversarial perturbations . Standard ` p adversarial perturbations are often thought to be incomprehensible to humans ( Szegedy et al. , 2014 ; Ilyas et al. , 2019 ) . Even when magnified for visualization , these perturbations are not identifiable with their target class . In contrast , universal perturbations are visually more interpretable : when amplified , they contain local regions that we can identify with the target class ( see Figure 2 for example universal perturbations , and Figure 1 for a comparison to standard perturbations ) . Prior works have also observed that universal perturbations resemble their target class ( Hayes & Danezis , 2019 ) and contain meaningful texture patterns ( Khrulkov & Oseledets , 2018 ; Liu et al. , 2019 ) . Next , we go beyond simple visualizations by identifying two additional human-aligned properties of universal perturbations . Predictive signals are localized to most semantically identifiable patches . While universal perturbations contain parts that are semantically identifiable with the target class , they also contain less identifiable regions ; a priori , it is unclear which parts influencing the model . To investigate this , we randomly select different local patches of the perturbation , evaluate their ASR in isolation , and inspect them visually . For ` 2 perturbations , the patches vary widely in norm , so a possible concern is that the model is only reacting to the patches with the highest norm ; to correct for this , we linearly scale up all patches to have the same norm as the largest norm patch . Even after normalization , the patches with the highest ASR are more semantically identifiable with the target class ( Figure 3 ) . For ` ∞ perturbations , where all patches have similar norms ( so no further normalization is done ) , we also observe that their ASR is proportional to their visual saliency . This demonstrates that the model is indeed reacting primarily to the most salient parts of the perturbation . do g ( 2 ) ASR : 0.494 0.237 0.220 0.205 0.098 0.098 do g ( ) ASR : 0.597 0.167 0.163 0.159 0.101 0.101 Figure 3 : Local analysis of universal perturbations on ImageNet-M10 : 64 random 80×80 patches are isolated and evaluated for their ASR . First column shows the original perturbation for the target class ( top : ` 2 ; bottom : ` ∞ ) ; next five columns show three patches with the highest ASR , and two with the lowest . Top of each perturbation indicates its ASR on the test set . Patches are normalized . 3.2 SPATIAL INVARIANCE . Adversarial perturbations are not only unintelligible but also extremely brittle to translations , as we demonstrate . In contrast , we find that universal perturbations are translationally invariant to a large degree . As spatial invariance is one of the key properties of the human visual system ( Hubel & Wiesel , 1968 ) , this illustrates another way that universal perturbations are more human-aligned than standard perturbations . We quantify spatial invariance by measuring the ASR of translated perturbations . Specifically , we compute targeted standard adversarial perturbations on a sample of 256 images from the ImageNetM10 test set . Then , we evaluate the ASR of different translated copies of these perturbations over the test set ( copies of each perturbation are evaluated only on its corresponding image ) . For comparison , we take a precomputed set of universal perturbations of the same norm , and evaluate them across all images . Figure 16 shows that universal perturbations still achieve non-trivial ASR after translations of varying magnitudes . In contrast , standard adversarial perturbations achieve a chance-level 10 % ASR when shifted by more than eight pixels ( two grid cells in Figure 16 ) . This illustrates that the model reacts differently to universal perturbations than to standard adversarial perturbations .

Summary: Prior works generally thought non-robust features, which are vulnerable to small perturbations, are not semantically meaningful but are useful for generalization. This work challenges these traditional beliefs by pointing out that non-robust features can also be human-perception aligned and be less useful for generalization, if these non-robust features are discovered via universal adversarial perturbations (rather than via image-dependent perturbations). Extensive experiments are provided to justified these arguments.