Non-robust Features through the Lens of Universal Perturbations
Authors: Sung Min Park
Kuo-An Wei
Kai Yuanqing Xiao
Jerry Li
Aleksander Madry
Venue:  (2021)
Keywords: adversarial examples
robustness
non-robust features

Abstract: Recent work ties adversarial examples to existence of non-robust features: features which are susceptible to small perturbations and believed to be unintelligible to humans, but still useful for prediction. We study universal adversarial perturbations and demonstrate that the above picture is more nuanced. Specifically, even though universal perturbations---similarly to standard adversarial perturbations---do leverage non-robust features, these features tend to be fundamentally different from the ``standard'' ones and, in particular, non-trivially human-aligned. Namely, universal perturbations have more human-aligned locality and spatial invariance properties. However, we also show that these human-aligned non-robust features have much less predictive signal than general non-robust features. Our findings thus take a step towards improving our understanding of these previously unintelligible features.

Under review as a conference paper at ICLR 2021
NON-ROBUST FEATURES THROUGH THE LENS OF UNIVERSAL PERTURBATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work ties adversarial examples to existence of non-robust features: features which are susceptible small perturbations and believed be unintelligible humans, but still useful for prediction. We study universal adversarial perturbations and demonstrate that the above picture is more nuanced. Specifically, even though universal perturbations--smiinee similarly to standard adversarial perturbations- do lever- age non-robust features, these features tend to be fundamentally different from the "standard" ones and, in particular, non-trivially human-aligned Namely, universal perturbations have more auman-aligned locality and spatial invariance properties. However, we also show that these human aalannd non-robust features have much less predictive signal than general non-robust features. Our findings thus take a step towards improving our understanding of these previously unintelligible features.
INTRODUCTION
Modern deep neural networks perform extremely well across many prediction tasks, but they largely remain vulnerable to adversarial examples (Szegedy et al.) [014]). Models' brittleness to these small, imperceptible perturbations highlights one alarming way in which models deviate from humans. Recent work gives evidence that this deviation is due to the presence of useful non-robust features in our datasets (Ilyas et al.) 2019). These are brittle features that are sensitive to small perturbations small to be noticeable to humans, yet capture enough predictive signal to generalize well on the underlying classification task. When models rely on non-robust features, they become vulnerable to small perturbations can flip the features" signal.
adversarial examples,
While prior work gives evidence of non-robust features natural datasets, we lack a more fine- grained understanding of their properties. In general, do not understand well how models make decisions, so is unclear how much we can understand about these features that are believed to be imperceptible. A number of works suggest that these features may exploit certain properties of the dataset that are misaligned with human perception (Ilyas et al. 2019), such as high-frequency information Yin et 2019), but much remains unknown.
Bird, (d)
Adversarial
Universal
Universal
Universal
Figure 1: lg adversarial perturbations (c 6.0) for two target classes, bird and feline, on ImageNet- M10: (a,d) standard adversarial perturbations on a single image, (b,e) universal adversarial perturba- tions, and (c.f) zooming in on the most semantic patch.
In this work, illustrate how we can isolate more human-aligned non-robust features by imposing additional constraints on adversarial perturbations. In particular, we revisit universal adversarial per- turbations Dezfooi et al. 2017a), i.e. adversarial perturbations that generalize across many inputs. Prior works have observed that these perturbations appear to be semantic (Hayes Danezis 2019) Khrulkov & Oseledets 2018 Liu al. 2019). We demonstrate that universal perturbations

---

Under review as a conference paper at ICLR 2021
possess additional human buman-aligned properties different from standard adversarial perturbations, and analyze the non-robust features leveraged by these perturbations. Coneretely, our findings are:
Universal perturbations have more human-aligned properties. We show that universality adver- sarial perturbations have additional human-aligned properties that distinguish them from standard adversarial perturbations (e.g., Figure D.. In particular, (1) the most semantically identifiable local
patches inside universal perturbations also contain the most signal; and (2) universal perturbations are approximately spatially invariant, in that they are still effective after translations.
Non-robust features can be semantically meaningful. We show that universal perturbations are primarily relying non-robust features rather than robust ones. Specifically; we compare the sensi- tivity of natural and (adversarially) robust models to rescalings of these perturbations to demonstrate that universal perturbations likely rely on non-robust features. Together with our first finding, this shows that some non-robust features can be human-aligned
Universal perturbations contain less non-robust signal. We find that the non-robust features leveraged by universal perturbations have less predictive signal than those leveraged by standard adversarial perturbations, despite being more human-aligned. We measure both (1) generalizability to the original test set and (2) transferability of perturbations across independent models, following the methodology of llyas et al. (2019). Under these metrics, universal perturbations consistently obtain non-trivial but substantially worse performance than standard adversarial perturbations.
2 PRELIMINARIES
We consider a standard classification task: given input-label samples (x. y) E X' x y from a data distribution D, the goal is to learn a classifier C: : X + y that generalizes to new data.
Non-robust vs. robust features. Following Tlya et al. (2019), we introduce the following terminol- ogy. A useful feature for classification is function that is (positively) correlated with the correct label in expectation. A feature is robustly useful if, even under ad adversarial perturbations (within specified set of valid perturbations A) the feature still useful. Finally, a useful, non non-robust feature is a feature that is useful but not robustly useful. These features are useful for classification in the standard setting, but can hurt accuracy in the adversarial setting (since their correlation with the label can be reversed). For conciseness, throughout this paper we will refer to such features simply as non-robust features.
Universal perturbations. A universal adversarial perturbation (or just universal perturbation for short), as introduced in Moosavi-Dezfoon et al. (2017a), a perturbation ô that causes the classifier C' to predict the wrong label on large fraction of inputs from D. We focus targeted universal perturbations that fool C' into predicting a specific (usually incorrect) target label t. Thus, a (targeted) universal perturbation (ASR) of the universal perturbation. The only technical difference between universal perturbations and (standard) adversarial perturbations is the use of single perturbation vector 5 that is applied to all inputs. Thus, one can think of universality as a constraint for the perturbation ō to be input- independent. Universal perturbations can leverage non-robust features in data, as we show here; we refer to these simply as universal non-robust features.
A satisfies P(c.,))D(((( (C(x++ = t] p. where p the attack success rate
Ep perturbations. We study the case where A is the set of bp-bounded perturbations, i.c. A {6 1Rd ||00|p S E} for p 2, po. This is the most widely studied setting for research on adversarial examples and has proven to be an effective benchmark (Carlini et al.) 2019). Additionally. t. f.-robustness appears be aligned to a certain degree with the human visual system (Tsipras et al.
2019).
2.1
COMPUTING UNIVERSAL PERTURBATIONS
We compute universal perturbations by using projected gradient descent (PGD) on the following optimization problem:
min ((n))~ ((((x++))+)) SEA
(1)
where L is the standard cross-entrop loss function for classification, and f are the logits prior to classification. and & is the target label. While many different algorithms have been developed for
2

---

Under review as a conference paper at ICLR 2021
computing universal perturbations with varying attack success rates, do not consider them as achieving the highest rate is not necessary for our investigations. As observed in Moosavi-Dezfoot et al. (2017a), universal perturbations trained on only fraction of the dataset still generalize to the rest of the dataset, so in practice suffices to approximate the expectation in equation Jusing a relatively small batch of data drawn from D. We call the batch we optimize over the hase set, and use K to refer to its size. We describe the selection of all hyperparameters in Appendix B
2.2 EXPERIMENTAL. SETTINGS
Datasets. We conduct our experiments on subset of ImageNet (Deng et al. 2009) and CIFAR- 10 (Krizhevsky et al. 2009). The Mixed10 ImageNet dataset (Engstrom et al. 2019). which we refer to as Imag is formed by sub-selecting and grouping together semantically similar classes from ImageNet; provides more computationally efficient alternative the full dataset, while retaining some of ImageNet's complexity (see Appendix B. for more details about the dataset). In the main text, focus results on ImageNet-MIlO and in the Appendices provide additional results on CIFAR-10 10 as well as visualizations from a full ImageNet model.
Models. We the standard ResNet-18 18 architecture (He et al. 2016) unless mentioned otherwise.
PROPERTIES OF UNIVERSAL PERTURBATIONS
We find that universal perturbations, in contrast to adversarial perturbations, are much more aligned with human perception. Beyond visualizations, we demonstrate this in the following ways: (1) local patches that are most semantically identifiable with the target class also contain the most predictive signal; and (2) universal perturbations are approximately spatially invariant, whereas standard perturbations are not.
3.1 PERCEPTUAL ALIGNMENT
Visual comparison with standard adversarial perturbations. Standard f, adversarial perturba- tions often thought to be incomprehensible to humans (Szegedy al. 2014 [lyas et al. (019). Even when magnified for visualization, these perturbations are not identifiable with their target class. In contrast, universal perturbations are visually more interpretable: when amplified, they contain local regions that we can identify with the target class (see Figure] for example universal perturbations, and Figure for comparison to standard perturbations). Prior works have also observed that universal perturbations resemble their target class (Hayes & Danezis (2199) and contain meaningful texture patterns (Krulkkov & Oseledets 2018 Liu et al. 2019). Next, we go beyond simple visualizations by identifying two additional human-al igned properties of universal perturbations.
dog
bird
Insect
monkey
feline
truck
fruit
fungus
bost
Figure 2: A sample of & universal perturbations (e = 6.0) for all target classes on ImageNet-M10: we observe patterns resembling dog and cat (feline) faces, birds, outlines of cars and trucks, an insect, and mushrooms (fungus).
Predictive signals are localized most semantically identifiable patches. While universal per- turbations contain parts that are semantically identifiable with the target class, they also contain less
3

---

Under review as a conference paper at ICLR 2021
identifiable regions; priori, it is unclear which parts influencing the model. To investigate this, we randomly select different local patches the perturbation, evaluate their ASR in isolation, and inspect them visually. For la perturbations, the patches vary widely in norm, that the model is only reacting to the patches with the highest norm; to correct for this, we linearly scale up all patches have the same norm the largest norm patch. Even after normalization, the patches with the highest ASR are more semantically identifiable with the target class (Figure B For foo perturbations, where all patches have similar norms (so no further normalization is done), we also observe that their ASR is proportional to their visual saliency. This demonstrates that the model is indeed reacting primarily to the most salient parts of the perturbation.
possible concern is
ASR: 0.494
0.237
0.220
0.205
0.088
0.098
0.167
Figure 3: Local analysis of universal perturbations ImageNet-M10: 64 random x patches are isolated and evaluated for their ASR. First column shows the original perturbation for the target class (top: f2; bottom: )o=); ): next five columns show three patches with the highest ASR, and two with the lowest. Top of each perturbation indicates its ASR on the test set. Patches are normalized.
3.2 SPATIAL INVARIANCE
Adversarial perturbations are not only unintelligible but also extremely brittle to translations, as we demonstrate. contrast, we find that universal perturbations are translationally invariant to a large degree. As spatial invariance is of the key properties of the human visual system (Hubel & Wiesel 1968), this illustrates another way that universal perturbations are more human-aligned than standard perturbations.
We quantify spatial invariance by measuring the ASR of translated perturbations. Specifically, we compute targeted standard adversarial perturbations M10 test set. Then, we evaluate the ASR of different translated copies of these perturbations over the test (copies of each perturbation we take precomputed set of universal perturbations of the same norm, and evaluate them across all images. Figure [6] shows that universal perturbations still achieve non-trin non-trivial ASR after translations of varying magnitudes. In contrast, standard adversarial perturbations achieve a chance-level 10% ASR when shifted by more than eight pixels (two grid cells Figure T6p. This illustrates that the model reacts differently to universal perturbations than to standard adversarial perturbations.
sample of 256 images from the ImageNet-
evaluated only on its corresponding image). For comparison,
UNIVERSAL PERTURBATIONS LEVERAGE NON-ROBUST FEATURES
The predominant belief about non-robust features that they are incomprehensible to humans. We saw, however, that universal perturbations have human-ali alinnndligned properties. And since universal perturbations are bounded in norm, they likely rely on non-robust features to fool the model. We might then conclude that there exist non-robust features that human-aligned.
However, there a possibility of robust feature leakage Goh 2019 while a small perturbation cannot entirely flip the correlation of robust feature, it still induce small correlation by perturbing the feature. If universal perturbations primarily rely on robust features, we canmot necessarily conclude that the nice properties observed earlier are tied to non-robust features. In
4

---

Under review as a conference paper at ICLR 2021
adveroiral
81
14
XIhnt
Figure 4: Evaluation of translated adversarial and universal l2, 6o perturbations for the target class bird on ImageNet-M10. We evaluate a subsampled grid with strides of four pixels. The value at coordinate (i,j) represents the average ASR when the perturbations are shifted by i pixels right and j pixels up, with wraparound to preserve information; the center pixel at (0, 0) represents the ASR of the original unshifted perturbations. For universal perturbations, the ASR at each location is averaged over ten perturbations.
this section, provide evidence that robust leakage is unlikely to be primary contribution to the signal
Scaling analysis natural vs. robust model Our analysis based on the following premise: if the universal perturbations are primarily relying on small perturbations of robust features, then a robust model should eventually react when amplify the signal in the universal perturbations. Below, first show that simply linearly scaling the perturbation can effectively tune the strength of the signal. More precisely, given universal perturbation we evaluate the ASR of / 8, where is scaling parameter. [ When we perform this scaling on natural (non-robust) model, the ASR increases as we increase (Figure 5). (This holds even if networks used train the universal perturbation and the one used to evaluate its ASR differ.) If there was significant robust leakage, we would expect a similar qualitative behavior for the robust model. Instead, we find that the robust model does not react to any scaling of the perturbation. On the other hand, universal perturbation of norm c 30. robust model (c 6.0) still fools the natural model, even when scaled down. Together, these findings suggest that universal perturbations computed on the natural model are unlikely to leverage robust features used by the robust model.
computed on an adversarially-trainn
In fact, more fine-grained scaling analysis shows that the signal is indeed primarily amplified in the semantic parts of the image. In Section 3 we already observed that models are most sensitive to the most semantic parts of universal perturbations; here check that the order remains consistent across various scales. We repeat the above scaling evaluation on natural vs. robust models, except we scale only patches of the perturbation, of varying levels of semantic content (Figure [b).. As before, we find that none of the patches, even after scaling, trigger the robust network. On the non-robust models, the relative effectiveness of local patches is consistent with their semantic content: patches that are more semantically identifiable have more signal at all scales.
The above analysis shows that universal perturbations are likely primarily targeting non-robust features. Combined with our findings earlier in Section 3 this shows that non-robust features may not be entirely unintelligible humans. Our visualizations (FigureBe) suggests that universal non-robust features may be capturing intuitive properties, such as the presence of a faint bird head. Thus, non-robustness in networks may partially arise from relying on cues that are small in magnitude but still buman-aligned.
We also provide alternate analyses in Appendix A.3 to bound robust feature leakage in alternate ways, based on extending the methods in Section 5
JJtley 2018)) susceptible to adversarial perturbations.
similar analysis to study subspaces that essential for classification and also
Universal perturhations on a robust model require larger norm as the model is robust near the is trained
5

---

Under review as a conference paper at ICLR 2021
patch
patch
norm
perturbation
perturbation
(c) Different random cropped patches
(a) Scaling universal perturhations
(b) Scaling cropped patches
Figure 5: Scaling analysis of universal perturbations on ImageNet-M10:: (a) generate universal perturbation (nat-univ-ptae of & norm c = 6.0 on a natural model (nat-model ) for the target bird, and measure the sensitivity of the natural and a robust model (rob-model) at various rescalings of the perturbation. We also generate a universal perturbation (rob-univ-ptt)) of norm 30.0 for the robust model and it to show that the robust model does react scalings of some perturbation. The same nat-univ-pts is also evaluated on an independent natural model (nat -mode--2))) for control; (b) different cropped local patches of the same nat nat-univ-ptb are also evaluated on the natural model; (c) visualizations of the cropped patches.
UNIVERSAL NON-ROBUST FEATURES CONVEY LESS SIGNAL
So far, we demonstrated that universal perturbations tend to leverage highly semantic non-robust features. We now show that these universal non-robust features have less generalizable signal than standard non-robust features. As in [lyas et al. (2019), we measure the usefulness of non-robust features based on: (1) how well a model generalizes to the original test set when trained on dataset where the only useful features are those introduced by universal perturbations; and (2) the transferability of universal perturbations across models. (1) directly measures the usefulness of universal non-robust features for learning, while (2) measures the degree to which different models learn the same set of features.
5.1 GENERALIZATION FROM UNIVERSAL NOB-ROBUST FEATURES
We construct two datasets, D Danio and D Dude, where the only useful features are universal non-robust features and non-robust features, respectively.
Constructing Dude and Duniv. We follow the procedure outlined in llyas et al. (019) to construct Dudo (which corresponds to D from their paper): 1) for each input-label pair (Ir, v). select target class uniformly at random; then 2) compute an L2 adversarial perturbation on æ targeted towards class f, and replace the original input-label pair (æ, y) with (x 5, t). By construction, and are decorrelated and the only features correlated with f are those leveraged by 5. Because 8 is constrained be within an lz ball, it can only yield non-robust features R Next, we adapt the above
procedure to construct D Dunie- To this end, in step 2, we use a universal perturbation in place of an adversarial perturbation
Comparing the signal in Dade and D uumiu- We train new ResNet-18 models on the Dado and
D Dunto datasets and evaluate them on the original test set. 6 The best generalization accuracies from training on Duniv and Dudo were 23.2% and 74.5%, respectively. This indicates that universal non-robust features have much less generalizable signal than standard non-robust features.
4 address possible robust leakage in Appendix 'There are a few complications in this new setup, one being that universal perturbations only fool a fraction of images. See Appendix A. for more details on the construction of D Duniv-
Wee report the best results found over grid of training hyperparameters.

---

Under review as a conference paper at ICLR 2021
Table 1: Interpolating universality: general- ization from Dunir on smaller base sets. For ImageNet-M10, E2.
Table 2: Generalization from class-universal and sub-class universal non-robust features on ImageNer-M10, using & perturbations. We find minimal evidence to suggest that non- robust features are aligned with (sub)classes.
Size of base set (K)
Test Accuracy (%)
74.5
57.1
Source Class
Test Accuracy (%)
4
61.3
57.4
Random Single Class
23.2
16
34.3
23.9
32
21.8
Single Sub-class
27.1
256
19.1
5.2
TRANSFERABILIT OF UNIVERSAL PERTURBATIONS
Ilyas et al. 2019) suggest that the pervasive transferability of adversarial perturbations (Papernot al. 2016 Moosavi-Dezfool et al.) 2017a) can be attributed different models relying on similar robuto features. Consequently, bertorbations that leverage more shared non-robust features should transfer better. Here we are primarily interested in transferability as a proxy for detecting shared non-robust features, and we compare the transferability of adversarial and universal perturbations.
Restiet-18 18 ResNiet-19
ResNet-18 to VGG16
Trnnde GIOWASON
ropular universal
0.2
6.0
69g
mect morkey CAT
Selte
fndt
Singus
bost
In0RCT moniay
CNr Tanget
Tungus
beat
Figure 6: Comparing transferability of adversarial and universal l perturbations (e = 6.0): pertur- bations are generated on the source model ResNet-118, then transferred to different target model: (left) another ResNet-Ie model trained from a different random initialization, and (right) a VGG16 model.
To measure transferability, (1) perturb examples using either adversarial perturbation universal perturbation on the source model, and (2) measure the probability that the perturbed input classified the target class make standard and universal perturbations comparable, we only consider perturbed images that are misclassified by the source model, and also evaluate transfer on images that have labels different from the target label. s/swoss that universal perturbations transfer much less, suggesting that models share fewer universal non-robust features. This is consistent with results from Section 5.I
new larget model (a ResNet- 18 trained independently). To
5.3 INTERPOLATING UNIVERSALITY
The above experiments demonstrate that while universal non-robust features are more human-ale human-aligned, they also have less generalizable signal. To explore the underlying trade-off, investigate to what extent one can interpolate between the properties of universal and standard non-robust features. To this end, we explore two different methods of "interpolating" universality:
Interpolating via changing the base set size. One natural way to interpolate universality is to
vary K, the size of the base set (cf. Section 2.I. K = corresponds to adversarial perturbations, and large enough K corresponds to fully universal perturbations. Table [ shows that generalization behavior generally decreases with the size of the base set. Notably. generalization begins to suffer for relatively small values of K. In contrast, the semantics of universal perturbations improves as the base set becomes larger, in line with the hypothesized trade-off (cf. Appendix A.5
See Appendix A.4 for additional analysis.

---

Under review as a conference paper at ICLR 2021
Interpolating via restricting the source class. We expect that examples from the same class share similar features, including non-robust ones. This motivates us to interpolate universality along a different axis: the level of semantic hierarchy. We consider perturbations, where we restrict the base set to examples from a single class (e.g., dog) or subclass (e.g., Terrier). If certain non-robust features exist only in examples from the same class, expect class-universal perturbations to leverage those features. To examine this, we repeat the experiments from Section 5.T using these perturbations (cf. Table 2 the generalization on the original test set improves modestly. This suggests that if non-robust features are specific to small subpopulations of examples, then these subpopulations must be more fine-gra ane-grained than division into different (sub) classes.
and subbllas--miverrsn
Discussion Above results show that we can interpolate the signal present in universal non-robust features by varying the base set's size, and to smaller extent, its class diversity. This can be intuitively captured by the following toy model: Each example comes with a set of non-robust features, and different examples share a varying subset of features depending on their semantic distance. By computing perturbations over a set of images, we limit the set of non-robust features to their intersection; it follows that the number of features in this intersection, and hence signal, decreases with more images and as the images become further apart.
Existing theoretical models [Tiipaas et 2019 Allen-Zhu Li 2020 fail distinguish between standard and universal perturbations, and hence fail to capture the phenomena observed here. We believe that more empirical observations are necessary to identify precise theoretical model.
In Appendix A
we provide additional analyses for some results in this section.
6 RELATED WORK
Moosavi-Dezfool et al. (2017a) first introduce universal adversarial perturbations. Many follow- up works explore its various aspects, including possible theoretical origins, different methods of generation, and defenses (Moosavi-Deztooli et al. 2017b) PPourseedd et al. 2018 Khrulkov &
Öseledets 2018 Wu & Fu 2019| et al.) 2019 Hayes & Danezis 2019 Khrulkov & Oseledets 2018
Liu et al. 2019: Shafahi al. 2020).
Using ideas from analysis universal perturbations, Dettle et al. | (2018) find that class-specific patterns unintelligible to humans can induce misclassification while also being essential for classification. Ilyas et al. (2019) first formalize non-robust features, and demonstrate that they alone are sufficient for generalization. In particular, adversarial examples viewed as consequence of misalignment between non-robust features and human priors. Conversely, several works discuss the usefulness and perceptual alignment of robust features (Tsipras et al. 2019 Santurkar al. 2019).
Recent works show how models use certain signals humans do not (primarily) rely on: examples include texture, high-frequency content, and background (Geirhos al. 2018 Yin al. 2019 Xiao
et al. 2020).
7 CONCLUSION
In this work, we revisit universal perturbations and use them to analyze non-robust features. We find that universal perturbations perturbations), and give first evidence of perceptually aligned, interpretable non-robust features. However, we also show that they account for only a small part of all non-robust features. Understand- ing the structure and properties of the remaining bulk of non-robust features is a natural direction for future work. More broadly, developing framework for studying features in more fine- grained ways beyond their correlation and robustness would be helpful for understanding our models better.
highly semantic and spatially invariant (in contrast to adversarial
'While the exact definition of distance is hard to define, can imagine, for example, two images of dogs in the same pose with similar background share more non-robust features than with an image of car.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li, Feature purification: How adversarial training performs robust deep learning. arXiv preprint arXiv:2005. 2020.
Nicholas Carlini, Amish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras. Ian J. Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. In ArXiv preprint 1rXiv:1902.067000 2019.
Deng. W. Dong, R. Socher, L.-J. Li, Li, and L. Fei-Fei, ImageNet: Large-Scare Scale Hierarchical Image Database. In CVPR09, 2009.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library). 2019. BRRtttts:///git
robustness
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 122231, 2018.
Gabriel Goh. A discussion of "adversarial examples are not bugs. they are features': Robust httpti//llllpp//////////eeee
feature leakage. Distill, 2019. doi: 11.23915/disiil1000009 discussion/responsse2.
Tejus Gupta, Abhishek Sinha, Nupur Kumari, Mayank Singh, and Balaji Krishnamurthy. A method for computing class-wise universal adversarial perturbations. In ArXiv preprint arXiv: arXiv:1912.00 1912.0041. 2019.
Jamie Hayes and George Danezis. Learning universal adversarial perturbations with generative models. In ArXiv preprint #xXv:1708.05207, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
D. H. Hubel and T. N. Wiesel. Receptive fields and functional architecture of monkey striate cortex. In The Journal of physiology, 195(1), 215-243., 1968.
Andrew Iyas, Shibani Santurkar, Logan Engstrom, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Neural Information Processing Systems (NeurIPS), 2019.
Jetley, Nicholas Lord, and Philip Torr. With friends like these, who needs adversaries? In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Valentin Khrulkov and Ivan Oseledets. Art of singular vectors and universal adversarial perturbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8562- 8570, 2018
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Hong Liu, Rongrong Ji, Jie Li, Baochang Zhang, Yue Gao, Yongjian Wu, and Feiyue Huang. Universal adversarial perturbation via prior driven uncertainty approximation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2941-29499 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 17066 06083, 2017.
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 1995.
Seyed-Mohsen Moosavi-Dezfooli,, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Computer Vision and Pattern Recognition (CVPR), 2017a.
Seyed-Mohsen Moosavi-Dezfoolii, Alhussein Fawzi, Omar Fawzi, Pascal Frossard, and Stefano Soatto. Analysis of universal adversarial perturbations. arXiv preprint arXiv: /a0o:/00500.... 2017b.
9

---

Under review as a conference paper at ICLR 2021
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: //12911/iistill000000... http://distill.pub2200
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from
phenomena black-box attacks using adversarial samples. ArXiv preprint wrXiv:1655.07277 2016.
Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial perturba- tions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4422-4431, 2018.
Konda Reddy Mopuri, Utkarsh Ojha, Utsav Garg, and R Venkatesh Babu. Nag: Network for adversary generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 742-751, 2018.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. In International Journal of Computer Vision (IJCV), 2015.
Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Image synthesis with a single (robust) classifier. In Neural Information Processing Systems (NeurlPS), 2019.
Ali Shafahi, Mahyar Najibi, Zheng Xu, John Dickerson, Larry S. Davis, and Tom Goldstein. Universal adversarial training. In Association for the Advancement of Artificial Intelligence (AAAI), 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference Learning Representations (JCLR), 2014.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robustness may be odds with accuracy. In International Conference on Learning Representations (ICLR), 2019.
Junde Wu and Rao Fu. Universal, transferable and targeted adversarial attacks. In ArXiv preprint a90i 11332, 2019.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09994 2020.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. In Neural Information Processing Systems (NeurIPS), 2019.
10

---

Under review as a conference paper at ICLR 2021
APPENDICES
In Appendix we provide deferred details and analyses for results in Section
In Appendix B describe details of the experimental setup and all hyperparameters used in the
experiments.
In Appendices to
we give selection of additional results for experiments in the main paper
across different settings.
In Appendix F discuss the diversity of universal perturbations.
A DEFERRED DETAILS FROM SECTION 5
A.1 DETAILS OF Dunie CONSTRUCTION
We provide additional details for constructing the Dunie dataset in Section 5.1 As rather expensive compute separate universal perturbation for each input, compute and use the same universal perturbation per batch of inputs. One possible concern is that re-using same perturbations the batch results in a reduction in "diversity" of features considered. However, the total number of unique universal perturbations used does not appear to have large impact from our experience. In particular, using a different universal perturbation for every input does not noticeably affect the generalization of the resulting datasets.
Another complication that for the robustness threshold considered here, universal perturbations fool models only on fraction & of all training examples (while adversarial perturbations at the same threshold succeed on all examples). Hence, we only include examples on which the universal perturbation succeeds (i.e., a pre-trained model classifies the perturbed input as l). This filtering step reduces the size of the dataset, evaluation. We also ensure that the distribution of new and original labels is uniform and independent so that the original label is uncorrelated with the new label.
decrease the number of examples in Dadv accordingly in
A.2 ALL RESULTS ON GENERALIZATION ON
DATASETS
Here we collect additional results for the generalization experiment across various settings: 62, so per- turbations in ImageNet-M10 and & perturbations on on CIFAR-IO. We did not conduct experiments on ImageNet, as the constructions of the datasets is computationally too intensive.
Table 3 shows that there is consistently much lower generalization from D Danio than from Dudo.
Table 3: Generalization from constructed datasets.
Source Dataset
Perturbation Set
Constructed Dataset
Test Accuracy (%)
ImageNet-M10
l2; E 6.0
D Dado
74.5
ImageNet-M10
l, E = 3.0
Dade
76.6
ImageNet-M10
lg, E = 6.0
D Dunso
23.2
ImageNet-M10
loo, 8/255
Dadv
78.7
ImageNet-M10
loo 8/255
D Dunio
26.5
CIFAR-10
la, & 1.0
Dado
64.3
CIFAR-10
lg, E 1.0
D Panio
23.3
A.3 BOUNDING ROBUST FEATURE LEAKAGE
The accuracies on the D Danie datasets are very low (for comparison, they are even lower compared to ResNet-18 model trained on (fixed) randomly initialized features, which achieve accuracy greater than 30%). Given this, one concern is that the small signal we observe can be entirely accounted for by leakage of robust features into these datasets (Ilyas et al. Goh 2019).
11

---

Under review as a conference paper at ICLR 2021
We several additional experiments bound the amount of leakage (we focus &o perturbations on ImageNet-M10):
We construct dataset similar to Daet (lyys al. 2019) but instead using universal perturbations, where we choose new corrupted labels by cyclically shifting original labels, rather than choosing them randomly; the robust features now point away from the label. Models trained on this new dataset achieve an accuracy up to 19.1% on the original test set, which is less than 26.5% from Duniu, but still well above chance-level of 10%. This demonstrates that there is residual signal in universal perturbations that cannot be entirely accounted for by robust leakage.
We can also prohe the original dataset by seeing how well different features o7 can fit the dataset. Adapting the procedure in Goh (2019), we take pre-trained features from different natural and &oo robust models on ImageNet-M10, and train a linear classifier over those features on D Dunic. The results in Table 4 show that features from natural models, even sourced from different model than the one used to generate Dunis or from different architecture, capture more of the signal in D Puniv than robust features. For comparison, the model trained on D uniu from scratch, which is higher than 22.3%. This demonstrates again that while some leakage is happening. it cannot account for all of the signal in universal perturbations.
earlier, achieves accuracy of 26.5%,
Table 4: Training on Duniv using fixed features to check for leakage.
Feature Source
(Fine-tuned) Test Accuracy (%)
natural
36.9
natural, diff init natural, diff arch (VGG16) robust (loo)
35.0
26.4
22.3
The scaling analysis from Section 4 also gives supporting evidence, from different angle.
These results consistently show that while some robust leakage occurs, it cannot account for all of the signal in universal perturbations. small but non-trivial amount of signal comes from the universal non-robust features.
A.4 ADDITIONAL ANALYSIS OF K-INTERPOLATION
One plausible hypothesis for the degradation of signal quality between K = 1 and higher K is that optimizing the perturbation over more images splits the effective perturbation norm budget, and is akin to optimizing over each image separately with a reduced budget, say e/K. To test this hypothesis, we constructed Dano with a reduced budget of 3.0. We compare the generalization from this new dataset to that from the = 2, 6.0 case in Table 5 (latter the same value as in Table [T]. Given the large gap, it seems unlikely that this simple norm budget sharing explain the above interpolation phenomena. This suggests that when universal perturbations are computed multiple images, different non-robust features in different images are interacting in non-trivial manner.
Table 5: Comparing generalization from two different constructed datasets for ImageNet-M110 using & perturbations. is the base set size; K = corresponds to using standard adversarial perturbations, and K computing perturbations over random pairs of images.
Perturbation Type
Test Accuracy Original (%)
6.0. K 2
57.1
3.0, K =
76.6
As usual, consider the penultimate layer representations of the network before the final linear classifier.
12

---

Under review as a conference paper at ICLR 2021
A.5 VISUALS OF INTERPOLATING UNIVERSALITY
K=4
K
256
K 1024
Figure 7: Visualization of universal perturbations for target bird, computed over base sets of different size K. We observe that the visual quality improves with the size of the base set.
Table 1 shows that generalization decreases with the size of the base In contrast, Figure 7 shows that the semantic quality of universal perturbations improves as the base set becomes larger. This suggests a possible trade-off between semantic quality and generalizable signal. least along this particular axis. However, more careful examination suggests that the picture might be more nuanced: better semantics only become obvious higher values of K (K 2 64), whereas generalization suffers even at relatively small values of K (K" ~ 16). Understanding whether these trends are both manifestations the same underlying phenomena require further work.
A.6 ALTERNATE BASE SETS
In this paper, we focused on computing universal perturbations over base sets (cf. Section 2.1 consisting of different images (from possibly restricted set of classes). To study the importance of sample diversity in the base set, we computed universal perturbations over the base set consisting of K different augmentations of single image. The resulting universal perturbations still have similar (but less salient) visual characteristics, and are transferable but to a lesser degree than perturbations computed over base set of different images.
B EXPERIMENTAL SETUP
B.1 DATASETS
Our experiments use CIFAR-10 and ImageNet-M10, a subset of ImageNet ILSVRC2012 Rus sakovsky 2013); we focus primarily on ImageNet-M10.
ImageNet-M10 consists of ten super-classes, each corresponding a WordNet Miileer 1995 ID in the hierarchy. Different super-classes contain varying number of ImageNet classes, the dataset balanced by choosing six classes within each super-class. The class numbers of the corresponding ImageNet classes are shown in Table 6 For most experiments, super-classes correspond to the ten labels; for experiments using fine-graines classes, we use the 60 (sub) classes as labels. A sample image from each class is shown in Figure
The ImageNet-M10 dataset can be used easily on top of a standard ImageNet directory by loading the mix_d_11 dataset from the robustness library (Engstrom et al. 2019).
ImageNet-M10 is also similar to restricted ImageNet, which has been used for adversarial robustness research [lyas et al. 2019 Santurkar al. 2019).
Figure 8: Sample images from each (super) class of ImageNet-M10..
13

---

Under review as a conference paper at ICLR 2021
Table ImageNet classes used in the ImageNet-M10 dataset.
Class
WordNet ID
Corresponding ImageNet Classes
"Dog"
n02084071
1.11 to 156
"Bird"
n01503061
7 to 12
"Insect"
n02159955
300 to 305
"Mokky"
n02484322
370 to 375
"Car"
n02958343
407, 436, 468, 511, 609, 627
"Feline"
n02120997
286 to 291
"Truck"
n04490091
555, 569, 675, 717, 734, 864
"Fruit"
n13134947
948, 984, 987 to 990
"Fungus"
n12992868
991 to 996
"Boat"
n02858304
472, 554, 576, 625, 814, 914
B.2 HYPERPARAMETERS
All experiments in the paper involve combination of: training a model,, computing adversarial pertur- bations, and computing universal perturbations. Below, we provide details for all the hyperparameters and their selection.
Robustness threshold We use the same robustness threshold throughout: 2, 6.0 and lxo c 8/255 for ImageNet datasets, and f,, é 1.0 for CIFAR-10.
B.2.1 TRAINING MODELS
Training base models All models are trained with SGD with momentum and standard data aug- mentation. For corresponding robust models, the same optimization parameters are used with PGD training (Madry et al. 2017) using steps and attack step size f (for both & and loo). Weight decay of 5e-4 was used all cases. The accuracies of these models are shown in Table
Table 7: Standard and robust models used in our experiments.
Dataset
Model Standard Test Accuracy (%)
Robust Test Accuracy (%)
ImageNet-M10
Standard
95.7
< 1.0
86.7
59.8
to0
87.6
59.2
CIFAR-10
Standard
94.8
< 1.0
&
80.0
50.7
Training models constructed datasets The models the constructed datasets, including D and D Pade (Section 5.1). are trained over the following grid of hyperparameters: three learning rates (0.01, 0.05, 0.1), two batch sizes (128, 256), including/ not including single learning rate drop by factor of 10. All models are trained with 400 epochs, standard data augmentation, and weight decay
5 10 10 We report the setting corresponding to the highest test accuracies in Table
As an exception, for the K-interpolated datasets (Section 5.3 we fixed single set hyperparameters
to train models.
B.2.2 COMPUTING ADVERSARIAL PERTURBATIONS
Adversarial perturbations For given e, PGD with an attack step size of 3 and 10 steps is used (no random restarts.)
Without data augmentation, the model overfit these constructed datasets.
14

---

Under review as a conference paper at ICLR 2021
Table Default and best hyperparameters for training models on different datasets.
Source Dataset
Constructed Dataset
Epochs
LR
Batch Size
LR Drop
ImageNet-M10
(original)
200
0.1
256
50, 100, 150
Dudo (l2)
400
0.01
128
250
D Dunie (l2, random class)
400
0.01
128
250
Dunio (lz, Kiinterp)
400
0.01
256
250
D Dunio (lz, same class)
400
0.05
64
None
D unio (l2, same subelass)
400
0.05
128
250
D Dudo (lxc)
400
0.001
256
None
Dunse (o
400
0.1
128
None
CIFAR-10
(original)
150
0.1
128
50, 100
Dado (12)
400
0.05
256
250
Dunio (f2)
400
0.05
256
None
Universal perturbations The hyperparameters of the PGD algorithm for universal perturbations learning rate scheduld") number of epochs, and batch size In order to see the impact of the choice of these parameters, we ran a grid search over the parameters; sample of these selections is visible in Figure Overall, we observe that the exact choice of these parameters did not have large impact on the final training accuracy (and the quality of the resulting perturbations, both in terms visuals and generalization to other images) as long as (1) the initial learning rate is not too low or too high, and (2) a sufficient number epochs used. So all of our experiments, we fix a learning rate of 2.0 and batch size of 128 or 256, and trained a sufficient number of epochs (max of 100) 13 The
use of a learning rate decay did not seem to have much impact, so we use the same learning rate throughout.
C VISUALIZATIONS
We show visualizations of universal perturbations across different datasets (CIFAR-10, ImageNet-M, and ImageNet), architectures (VGG for Mixed10), and norm constraint (Coo for Mixed10).
Normalization for visualization. For all visualizations, perturbations were rescaled follows (for each input channel): first truncated to [-3or, 30] (where a is the standard deviation across all channels and pixels), then rescaled and shifted to lie [0, 1].
C.I DIFFERENT DATASETS
Sometimes this is known as step size in the adversarial examples literature. 12Btth size only matters it is smaller than the size of the base set, K. I'In this case K. is much larger than batch size, fewer epochs suffice as the effective number of updates increases due to bounded hatch size. In particular, for a fully universal base set (e.g. the entire training set), few five epnchs seed suffice.
15

---

Under review as a conference paper at ICLR 2021
K- 32, batch $220 32
K- 22, batch dize 64
K- 22, batch diz 128
1.3
0.6
Epocts
M. batch, size
541 batch, sizw M
14, batch, Nize 120
kw0.
2.4
1=03
P-42
1-40
K- 128. batch Size 32
<- 128. batch di20 64
< 128, batch Size 128
3.6
1-0.
F-05
22
02
26
30
18
40
12 40 50
Figure The learning dynamics of computing universal perturbation (epoch vs. training accuracy on the base set) are shown across a selection of learning rates, batch sizes, and three intermediate values of K.
airplane
automobile
bird
cat
deer
dog
frog
horse
ship
truck
Figure 10: Visualization of & universal perturbations at c = 1.0 for a CIFAR-10 model. Optimization on some classes (bird and cat) fail occasionally, and is reflected in their lack of semantic content.
16

---

Under review as a conference paper at ICLR 2021
black widow, Latrodectus mactans
snail
spiny lobster. langouste
grey fox, gray fox
quines pig, Cavis cobaya
African elephant, Laxodonta africana
aircraft carner, carrier
analog clock
dome
Josaw puzzle
maze, labyrinth
saxophone
spider's
Figure 11: Visualization of l universal perturbations at c = 6.0 for a full ImageNet model. We hand-picked the classes with the most salient perturbations. Many of the perturbations possess visual characteristics identifiable with the corresponding class.
17

---

Under review as a conference paper at ICLR 2021
C.2
OTHER ARCHITECTURES
dog
bird
insect
monkey
car
feline
truck
fruit
fungus
boat
Figure 12: Visualization of l universal (K = 512) perturbations 6.0 for ImageNet-M10, generated this time on VGG11 network (Simonyan & Zisserman 2015)
C.3 &o0 PERTURBATIONS
dog
bird
insect
monkey
car
feline
truck
fruit
fungus
boat
Figure 13: Visualization of fo universal perturbations e 8/255 for different target classes. We observe recognizable patterns for each class similar to those of &2 universal perturbations, but the remaining regions have much higher energy and different texture.
D LOCALITY ANALYSIS
We provide full results of randomized locality analysis on ImageNet-M10 in Figures and We did not perform the analysis on CIFAR-10 perturbations, as the perturbations are more "global" by nature, e.g. the semantic part seems already occupy most of the perturbation.
Exploring properties of l for perturbations. For both & and lo universal perturbations, the patches that are most identifiable with their classes have much higher ASR. A major difference that for & perturbations, the patches with low ASR have much smaller norms, whereas for Los. perturbations, the patches with low ASR have the similar norms (measured either l or &oo). This expected, for f every local patch contributes to the total norm and it is unsurprising that optimization avoid putting weight on patches with little signal; in contrast. for fx perturbations, different patches are independent in meeting the norm bound, the perturbation have patches
18

---

Under review as a conference paper at ICLR 2021
2:434
0,220
0.205
0.098
0.092
0.097
0.536
0.387
0,385
0:388
0.027
0.097
0.028
0.520
0.128
0.114
0.112
0.022
0.091
3:369
0.115
0.118
0.116
0:132
0.102
0.326
0.137
6131
0:128
5.130
0.103
Figure 14: Locality analysis of E2 universal perturbations on ImageNet-M100 Each row shows the perturbation for the particular class, followed by three patches with highest ASR and two patches with lowest ASR; ASR is written above each patch.
19

---

Under review as a conference paper at ICLR 2021
0.297
0.143
0:159
0.102
0.202
0.036
2.452
0.212
0,110
0.108
0.100
0.092
0.027
2:404
0.101
0,099
0.028
0.022
0.092
0.135
3.435
0.121
0,119
0.17
0:138
0.103
3.432
0.315
0.111
0.10
5.132
0.107
Figure 15: Locality analysis of too universal perturbations on ImageNet-M10, Each row shows the perturbation for the particular class, followed by three patches with highest ASR and two patches with lowest ASR; ASR is written above each patch
20

---

Under review as a conference paper at ICLR 2021
with low signal that are similar in norm. However, it is not clear whether these low signal patches found by optimization serve some purpose. To test this, we experimented with zeroing-or out these patches replacing them with copies of patches with the highest ASR. Evaluation of the resulting perturbations show that the ASR is highest with these patches in place, and are in fact lowered when replace them with patches with higher ASR. This suggests that these low signal patches have non-linear interaction with rest of the perturbation, and are providing a small boost in signal when combined with other patches. Understanding this phenomena in more depth may shed more light on the differences between properties of &o and &g universal perturbations.
E
SPATIAL INVARIANCE
Section 3.2 shows a sample of translation invariance evaluation for perturbations on ImageNet-Min We include additional samples from evaluation using &g perturbations CIFAR- 10, to check that the phenomena is general.
54
We evaluate attack success rates of translated adversarial and universal perturbations for different target classes over the test set on CIFAR-I0. There is no subsampling of the grid here as the dimensions are much smaller. We used a subsampled grid of all the possible offsets with strides of four pixels. As before, the value at coordinate (i. j) represents the average attack success rate when the perturbations are shifted by pixels along the x-axis and j pixels along the y-axis, with wrap around to preserve information. Evaluation of adversarial perturbation is only shown for one target as it is redundant.
Figure 16:
F PERTURBATION DIVERSITY
Throughout our investigations, we observed that the computed universal perturbations seemed to lack in diversity. We see in (frrsst row) that generating multiple different universal perturbations for given target leads to visually similar perturbations. Prior works study generating more diverse universal perturbations (Reddy Mopuri al. 2018).
To increase diversity, we generated universal perturbations with an additional penalty to encourage orthogonality of perturbed representations at different layers of the network [ | Even with the added regularization, the resulting perturbations appear to be remain similar, but with changes in spatial arrangements of the semantic patterns (sccond row).
One effective way we found to increase the diversity of these perturbations is to compute universal perturbations for targeting specific sub-classes of Mixed10. To do so, first train a fine-grained model using Mixed10's subclass labels, and generate universal perturbations for each of the six sub- classes per class. In Figure 18 show the universal perturbations targeting individual sub-classes of the bird class. We observe distinct patterns characteristic of each species of birds, an increase in diversity from before. However, generation of these perturbations requires a model trained on the fine-grained labels. An interesting question is whether one can recover such diversity using just the model trained on coarse-grained labels without injection of additional label information.
"In the original work on universal perturbations, MMosavi-Dezzii Dezfooli et al. (0017a) made the opposite interpre- tation that the perturbations generated independently are diverse due to their inner product being small. However, it is unclear whether inner product in the input space is the appropriate measure of their similarity. In fact, given the evidence of weak signal in universal perturbations (Section 5) appears more likely that different. perturbations are highly similar in their (signal) content.
For an extensive survey of similar and other regularization techniques for visualization, see Olah et al.
2017..
21

---

Under review as a conference paper at ICLR 2021
different In Inttaalization
orthogona at 2x2nd layer
Figure 17: Visualization of l universal perturbations for class feline, computed using different initializations (first row), and an additional penalty to encourage diversity (second row).
example
universal perturbation
Figure 18: Visualization of & universal perturbations for six individual sub-classes inside the bird class. These were generated on a fine-grained model trained on the subelass labels (60 in total). Observe the distinctive visual characteristics of each class, for instance the outline of chicken (column 2), the neck of an ostrich (column 3), and the distinctive colors of different species (columns 3-5).
22


Reviews:
[{"review_id": "su8TNUCBYXG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper studies the link between non-robust features and universal adversarial perturbations. This paper shows that universal perturbation leverage non-robust features in data in a different way than standard adversarial attacks. Experiments are based on a universal version of projected gradient descent (PGD). The findings are that universal perturbations are more aligned with visual semantics and human perception that general adversarial attacks. Moreover, it is shown to be difficult to obtain generalisation or transferability between models based on universal signals, as opposed to standard adversarial samples. Generalization seems to decrease with the size of the set used for generating a given universal perturbation, while semantics of the features improve.\n\nStrong points:\n- The paper uncovers novel, previously unknown properties of non-robust features.\n- The subject of the paper is of interest in general and relevant for the ICLR community.\n- The experiments chosen for the exploration are appropriate.\n- The paper is clear and well-written.\n\nConcerns:\n- While the present work advanced the state of understanding of non-robust features, it seems we are still far from a full picture of the topic. Many of the novel ideas proposed in the paper are still at a hypothesis or reasonable explanation state (e.g., universal perturbations reflect the intersection of non-robust features for the input set they were computed on).\n\nQuestions / suggestions:\n- It is unclear why universal perturbations being bounded in norm implies that they must be leveraging non-robust features.\n- Fig. 4: how is the perturbation shifted? Is is rolled over the edge of the image or is the excess discarded?\n- The current work can probably be extended to a similar analysis on adversarial patches.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TVyjeByKWMg", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-> Summary:\n\nThe authors investigate universal perturbations for adversarial robustness. They find that while universal perturbations are based on non-robust features, they are more human-aligned and spatially invariant. They also show that they contain less predictive\nsignal than other non-robust features.\n\n-> Reasons for score:\n\nI recommend weak reject for this paper, as I think the scope and impact of this work is too limited to universal perturbations. While I am not that familiar with interpretability literature, what experiments on generalization seem to show, most of the features learned by models are in fact not universal perturbations. So I am not sure whether findings of this paper would be interesting for broad ICLR community.\n\n-> Pros:\n\nI like investigation of the properties of non-robust features in the context of interpretability by humans. Paper is generally well written and easy to follow. Also, findings in this paper are novel as far as I know.\n\n-> Cons:\n\nWhile I am not that familiar with interpretability literature, based on my knowledge, I am not so sure of the impact of the findings presented in this work. There are essentially three main findings in this paper: universal perturbations are human aligned, non-robust features can be semantic and universal perturbations contain less signal than other non-robust features. I feel these are all interesting, but I am not sure how surprising they are. Universal perturbations are just a small type of possible adversarial perturbations and it is interesting to investigate their properties. As the experiments here show, most of the non-robust features, the ones that are actually useful for classification, are in fact not universal perturbations. So I am not sure what would be the key takeaway for the broader community that does not work on universal perturbations. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q603bVQjx8f", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes the existence of non-robust features through the lens of universal perturbations. The concept of non-robust features and universal perturbations is not new, but it is interesting to study the two concepts together. \n\nStrength: the paper gives a detailed study of the property of non-robust features via universal perturbations. Visualization of qualitative results is provided.\n\nWeakness: \n\n1. Conclusion is not well supported by the experiment. It is hard to measure what is human-aligned properties, even the concept is not well-defined. By visualization and spatially invariant experiments, it is fine to hint towards this, but seems unconvincing to me.\n\n2. The paper does not offer related improvements for existing models. How would the findings of this paper be beneficial to the models? Or guide the research community to build more robust models, or launch successful attacks?\n\n3. The findings are not that interesting and kind of known to the community. Wouldn't universal perturbations be optimized to be invariant such that they contain less non-robust signal (but more invariant/general signal), otherwise how would it be universal? \n\n4. The paper defines the adversarial perturbations to be non-robust features, however, the existence of non-robustness features is still an open question. It is one interpretation, but other interpretation also works. For example, it can be the common vulnerability of CNN models, thus the authors need to rethink whether a paper based on this assumption is solid.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9o_tug-3GsZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Prior works generally thought non-robust features, which are vulnerable to small perturbations, are not semantically meaningful but are useful for generalization. This work challenges these traditional beliefs by pointing out that non-robust features can also be human-perception aligned and be less useful for generalization, if these non-robust features are discovered via universal adversarial perturbations (rather than via image-dependent perturbations). Extensive experiments are provided to justified these arguments.\n\n\n\n\n*Pros:\n\n(1) This paper is well written and easy to follow.\n\n(2) The idea of utilizing universal adversarial perturbations to analyze non-robust features is novel, and lead to many interesting findings of non-robust features, e.g., non-robust features can also be semantically meaningful.\n\n(3) Most arguments in this paper are rigorously justified by careful experiment designs. For example, by analyzing the effects of perturbation scaling on model performance, this paper successfully persuades the reviewer to believe universal adversarial perturbations indeed leverage non-robust features.\n\n\n\n*Cons:\n\n(1) The reviewer has a major concern about the experiments that show non-robust features can also fail to generalize. For example, Table 1 suggests that with a large base set, non-robust features' generalization decrease. Nonetheless, the reviewer worries that this effect may be caused by the reduced diversity of training samples. That's said, when training with a base set of K (i.e., K training samples will share the sample universal perturbation), the reviewer tends to think the training dataset's \"effective\" sample size will then be greatly reduced to (total sample size)/K, which will decrease training quality and lead to inferior generalization. To rule out this possibility, the authors should also investigate the setting where the number of universal perturbations is equal to the number of training samples, regardless of the value of the size of base set K. The dataset preparation could be like this: first, we randomly sample K images, and generate a universal perturbation for them; second, we randomly sample one image from K' samples (which successfully fool networks), add the generated universal perturbation, and put it to the new training set; third, repeat the first & second step until we collect enough images in this new training set. Only with the performance analysis using these datasets, we can then confidently justify if non-robust features can generalize.\n\n(2) The reviewer is a little bit curious (and worry) why different batch sizes lead to quite different performance as shown in Figure 8? As already shown in the \"Training ImageNet in 1hr\" paper (https://arxiv.org/pdf/1706.02677.pdf) and many others, if you set the learning rate to be proportional to the batch size, then these training settings should get nearly the same results. Why this paper shows batch size is a critical hyperparameter for impacting model performance (e.g., in Figure 8, the settings of K=64, lr=0.5 batch size=32 and K=64, lr=1.0, batch size=64 seem to get very different results)?\n\n(3) why this paper only considers the targeted attack setting? will the conclusions here generalizable to the setting of non-targeted attack? Also what is the motivation for showing universal perturbations has spatial invariance (since image-dependent perturbation can also be spatial invariance if such transformation is considered by attackers)? \n\nIn general, the reviewer thinks it is an interesting paper, and is happy to raise the score if the authors successfully address the concerns above. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews were a bit mixed, and there was some concern on the usefulness and actual novelty of this work. On one hand, the authors did a nice job in visualizing their findings and conducting a wealth of interesting experiments. On the other hand, the submission suffers severely from hand-waving definitions and arguments. Many terms were not precisely defined, various hyperparameters were not thoroughly investigated, and yet conclusions were made based on indirect experimental results. The AC agrees with the reviewers that it is not very clear how this work would impact the field. For exploratory work like this one, there is also a great danger that one may simply overfit the observations and squeeze conclusions from thin air. It would be more convincing if the authors could largely quantify their definitions and results. For example: what do we mean by human-aligned? robust / nonrobust feature? (this definition depends on the perturbation size hence needs more elaboration.) Is there any way to quantify the results in Fig 2, including the impact of epsilon? Should these adversarial examples be called universal if their ASR falls below what threshold? Are (some of) the conclusions (e.g. translation invariance, semantic) a direct consequence of the perturbation being universal? At this stage this work would be an excellent workshop paper but a bit more rigor would be needed for publishing at the ICLR conference. 
Decision: Reject