Robust Feature-Level Adversaries are Interpretability Tools
Authors: Stephen Casper
Max Nadeau
Dylan Hadfield-Menell
Gabriel Kreiman
Venue: NeurIPS 2022 Accept (2022)
Keywords: Interpretability
Explainability
Adversarial Attacks

Abstract: The literature on adversarial attacks in computer vision typically focuses on pixel-level perturbations. These tend to be very difficult to interpret. Recent work that manipulates the latent representations of image generators to create "feature-level" adversarial perturbations gives us an opportunity to explore perceptible, interpretable adversarial attacks. We make three contributions. First, we observe that feature-level attacks provide useful classes of inputs for studying representations in models. Second, we show that these adversaries are uniquely versatile and highly robust. We demonstrate that they can be used to produce targeted, universal, disguised, physically-realizable, and black-box attacks at the ImageNet scale. Third, we show how these adversarial images can be used as a practical interpretability tool for identifying bugs in networks. We use these adversaries to make predictions about spurious associations between features and classes which we then test by designing "copy/paste" attacks in which one natural image is pasted into another to cause a targeted misclassification. Our results suggest that feature-level attacks are a promising approach for rigorous interpretability research. They support the design of tools to better understand what a model has learned and diagnose brittle feature associations. Code is available at https://github.com/thestephencasper/feature_level_adv.

Robust Feature-Level Adversaries are Interpretability Tools
Stephen Casper* 123 Max Nadeau-231 Dylan Hadfield-Menell' Gabriel Kreiman 23 'ITT CSAIL; 2 Boston Children's Hospital, Harvard Medical School; SCenter for Brains, Minds, and Machines; "Harvard College, Harvard University scasperêmit.cm edu
mmadeaucoolegge hww.eecaa..ggaaaaaa....a... www.aard. edu
Equal Contribution
Abstract
The literature on adversarial attacks in computer vision typically focuses on pixel- level perturbations. These tend to be very difficult to interpret. Recent work that manipulates the latent representations of image generators to create "feature-level" adversarial perturbations gives pretable adversarial attacks. We make three contributions. First, we observe that feature-level attacks provide useful classes of inputs for studying representations in models. Second, we show that these adversaries are uniquely versatile and highly robust. We demonstrate that they can be used to produce targeted, universal, dis- guised, physically-realizaien we show how these adversarial images can be used as practical interpretability tool for identifying bugs in networks. We use these adversaries to make predictions about spurious associations between features and classes which then test by designing "copy/paste" attacks in which one natural image is pasted into another cause a targeted misclassification. Our results suggest that feature-level attacks are promising approach for rigorous interpretability research. They support the design of tools to better understand what model has learned and diagnose brittle feature associations
opportunity to explore perceptible, inter-
and black-box attacks at the ImageNet scale. Third,
Introduction
State-of-the-art neural networks are vulnerable adversarial examples. Conventionally, adversarial inputs for visual classifiers take form of small-norm perturbations to natural images [66] [8]. These perturbations reliably cause confident misclassifications. However, to human, they typically appear as random or mildly-textured noise. Consequently, is difficult to interpret these tttacks-they rarely generalize produce human-comee compeee-ompeheeiiblll insights about the target network. In other words, beyond the observation that such attacks are possible, is hard to learn much about the underlying target network from these pixel-level perturbations.
In contrast, many real-world failures of biological vision are caused by perceptible, human-describable features. For instance, the ringlet butterfly's 's predators are stunned by adversarial "eyespots" on its wings (Appendix A. Fig. 7 This falls outside the scope of conventional adversarial examples because the misclassification results from a feature-level change an object/image. The adversarial eyespots are robust in the sense that the same attack works across a variety of different observers, backgrounds, and viewing conditions. Furthermore, because the attack relies high-level features, it is easy for a human to describe it.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Bee Feature-Leval Patch Mean Fly Conf: 0,9402
Pixal-Lovel Adv. Patch
Mean Fly Conf: 0,6619
Bee Conf: 0.5557
Fly Conf: 0.9783
(a)
(b)
Figure 1: Our feature-level adversaries are useful for interpreting deep networks (we used ResNer50 (211). (a) A pixel-level adversarial patch trained to make images of bees misclassified as flies. (b) An analogous feature-level adversarial patch. (c) A correctly-caassffed image of a bee. (d) A successful copy/paste attack whose design was guided by adversarial examples like the one in (b).
This work takes inspiration from the ringlet butterfly's s eyespots and similar examples in which model is fooled in the real world by an interpretable feature (e.g. 46).. Our goal is to design adversaries that reveal casily-understanddble weaknesses the victim network. We focus on two desiderata for adversarial perturbations: attacks must be (1) interpretable (i.e. describable) to human, and (2) robust so that interpretations generalize. We refer to these types of attacks as "feature- level" adversarial examples. Several previous works have created attacks by perturbing the latent representations of an image generator (c.g., (24),, but thus far, approaches have been small in scale, limited in robustness, and not interpretability sitbettabbiitt-rriveenn driven (See Section 2
We build on this prior work to propose an attack method that generates feature-level attacks against computer vision models. This method works on ImageNet scale models and creates robust, feature- level adversarial examples. We test three methods of introducing adversarial features into source images either by modifying the generator's latents and/or inserting a generated patch into natural images. In contrast to previous works that have enforced the "adversarialness" of attacks only by inserting small features restricting the distance between adversary and benign input, also introduce methods that regularize the feature to be perceptible yet disguised to resemble something other than the target class.
We show that our method produces robust attacks that provide actionable insights into a network's learned representations. Fig. [ demonstrates the interpretability benefits of this type of feature-level attack. compares a conventional, pixel-level, adversarial patch, created using the method from (6]. with feature-level attack using our method. While both attacks attempt to make a network misclassify bee as fly, the pixel-level attack exhibits high-frequency patterns and lacks visually- coherent objects. On the other hand, the feature-level attacks displays easily describable features: the colored circles. We can validate this insight by considering the network performance when a picture of traffic light is inserted into the image bec. In this example, the image classification moves from a 55% confidence that the image is a bee to a 97% confidence that the image of a fly. Section, 4.2 studies these types of "copy/paste" attacks more in depth.
Our contributions are threefold.
1. Conceptual Insight:
We observe that robust feature-level adversaries can used to produce useful types of inputs for studying the representations of deep networks
2. Robust Attacks: We introduce methods for generating feature-level adversaries that are uniquely versatile and able to produce targeted, universal. disguised, physically-raalizabll realizable, and black-box attacks at the ImageNet scale. See Table
3. Interpretability: We generalize from our adversarial examples to design copy/paste attacks, verifying that our adversaries help us understand the network well enough exploit it.
The following sections contain background, methods, experiments, and discussion. Ap- pendix I. has a high-level summary for a lay-audience. Code is available at https///gtttub..commmsssspppeeps level adv

---

Targeted
Universal
Disguised
Physically- Realizable
Transferable
Copy/Paste
ImageNet
Szegedy (2013) Rd.
Goodfellow al. (2014)
X
minicss, e.g.
Peacock, Ringlet Butterfl
x
N/A
Hayes al. (2018)
x
x
Mepuri al. (201
Mopuri al. (201
8
X
Poursaced al. (2018)
x
X
(2018) MX
Hashemi et al. (2020)
x
Wong et al. (2020)
1%
X
Liu et (2018) MS
Samangouci (2018)
Song al. (2018)
Joshi al. (2018)
AM
X
(2019)
Singla al. (2019) M4
Hu ct al. (2021) [4]
Wang et (2020)
Kurakin al. (2016)
Sharif al. (2016)
x
Brown et al. (2017)
Eykholl (2018)
Athalyy et (2018)
Liu al. (2019) Ei
Tt. (2019)
Kong al. (2020)
Komikey et al. (2021)
611
X
Dong et (2017)
X
Geirhos (2018)
Leclere al. (2021)
65
x
Wiles et (2022)
10
x
x
x
al. (2019)
Mo et (2020)
Hemandez et al. (2022) 23
x
Ours
Table 1: Our feature-level attacks uniquely versatile. Each represents related work (in the order which they are presented Section 2 Each column indicates demonstrated capability of method. Note that methods each having /for capability does not imply they do so equally well. Torgeted=working for an arbitrary target class. Universa/=working for any source example. Disgaised-Perrcepiibee and resembling something other than the target class. Physically- realizubl=-working in the physical world. mrnnsfrrbbebbluck--o-ttrraa Copy/Paste=useful for designing attacks in which natural feature is pasted into natural image.
to other classifiers.
2 Related Work
Here, contextualize approach with others related to improving conventional adversarial examples [66 [8). Table summarizes Tummaiammmizzzs capabilities.
Inspiration from Nature: Mimicry is common in nature, and sometimes, rather than holistically imitating another species, a mimic will only display particular features. For example, many animals use adversarial eyespots to confuse predators
(64] (see Appendix A 7h). Another example is the mimic octopus which imitates patterning. but not shape, banded sea snake. We show in Figure 7 that ResNet50 classifies an image of one sea snake.
Generative Modeling: An approach related to ours has been train generator or autoencoder to produce small adversarial perturbations that are applied to natural inputs. This has been done to synthesize imperceptible attacks that are transferable, universal, or efficient produce 20 :l 42
51
Rather than training generator, ours and other works have perturbed the latents of pretrained generative models produce perceptible alterations. [B] did this with differentiable
73 19
image renderer. Others 55. 53 29
2g
30 24!
have used deep generative networks, and 66 aimed attacks by using an autoencoder with a "disentangled"
to create more semantically-uuddesstaand
embedding space. Our work is different in four ways. (1) These works focus on small classifiers trained on simple datasets (MNIST 36), Fashion MNIST 74] SVHN [4]] CelebA [3), BDD (
INRIA M and MPII [T] while work at the ImageNet 5
scale. (2) We do not simply rely

---

Classifier
Feature Label Ent.
Reg. Discriminator
Feature Conf.
Extract Feature
Generator
Target Classifier
Image Creation
Transformation
Noise
Image Label
(One of Methods)
Loss
Perturb
Patch
Region
Generalized
Patch
Patch
Image
Generated Image
Patched Image
Patchod Image
Natural Image
Natural Image
Figure 2: Our fulle-differentiable pipeline for creating feature-level attacks, In each experiment, we create either "patch," "region," "generalized patch" attacks. The regularization terms in the loss based external classifier and discriminator are optional and are meant to make the inserted feature appear disguised as some non-target class.
using small features or restricting the distance to a benign image to enforce the adversarialness of attacks. We introduce techniques that regularize the adversarial feature be perceptible yet disguised to resemble something other than the target class. (3) We evaluate three distinct ways of inserting adversarial features into images. (4) Our work is interpretabillty-oriee -rrennte..
Attacks in the Physical World: Physical-realizabilty demonstrates robustness. We show that our attacks work when printed and photographed. This directly relates to 34 who found that pixel-space adversaries could do this to a limited extent in controlled settings. More recently, created adversarial clothing, stickers, objects. In contrast with these, we also produce attacks in the physical world that are disguised as a non-target class.
57
57
2 3l
Adversaries and Interpretability:
Using adversarial examples to better interpret networks has been proposed by I and [68] We use ours to discover human-describale feature/class associations who debug networks by searching over transfor- mations, textural changes, and feature feature alterations. More similar to our work are 7 2x)
learned by network. This relates to [7
55 70
who use feature visualization [47] and network dissection
( to interpret the network. Each use their interpretations to design "copy/paste" attacks in which one natural image pasted inside another causes an unrelated misclassification We add to this work with a new method to identify such adversarial
features. Unlike any previous approach, ours does so in a way that allows for targeted attacks that take into account arbitrary distribution of source images.
3 Methods
We adopt the "unrestricted" adversary paradigm network's classification differs from an oracle's ' (e.g.. a human). Our adversaries can only change small, fixed portion either the generator's latent the image. We white-box access the network, though we present black-box attacks based on transfer from ensemble in Appendix A.6
6 under which an attack successful if the
Our attacks involve perturbing the latent representation in some layer of image generator to produce an adversarial feature-level alteration. Fig. depicts our approach. We test three types of attacks, "patch", "region" and "generalized patch" (plus fourth in Appendix A.3 which we call "channel" attacks). We find patch attacks to generally be the most successful.
Patch: We the generator produce square patch that is inserted into a natural image
58

---

Advi Heneycomb, 9.415 Diag: Silky Terrier,
Advi Bouvier, 0:576 Ding: Padlock,
Adv: King Snake, 6.755 Disg: Beer Bottle,
Adv: Macaw, 0.695 Disge Stinkbern,
Seci Froight car, 0.255
Tglt Ibisan Hound, 0.726
Parch Incortion
AGM
Exampla
Adv: King Crab, 0.837
Adv: Tenter, 0.781 Disg: Muzzie, 0.564
Adv; Boberman, 0.817
Adv: PII Bottie, 0,928
Sic Indigo Bunting. 0.001
Region Cropped Context
Example
Adh: Gueson, 0.847
Adv: Hartebeest 0.514
Adv Bald Eagle, 0.841
Adv:: Afr Elophant, 0.713
Sec: Window Shode, 0.0 Scorpion,
Generalized Patch Patch Before Insertion
After Exampla
Figure 3: Examples of targeted, universal feature-level adversaries from patch (top), region (middle), and generalized patch (bottom) attacks. The first four columns show the adversarial features. The mean target class confidence is labeled 'Adv.' and is calculated under random source images (and random insertion locations for patch and generalized patch attacks). The target network's 's disguise class confidence for each patch extracted generalized patch is labeled 'Disg.' The final column shows examples of the features applied to images. The example image for each is labeled with its source and target class confidences.
Starting with some generated image, we randomly select square column of the latent in generator layer which spans the channel dimension and replace it with a learned insertion. This is analogous to a square patch in the pixel representation. We keep insertion location fixed over training, The modified latent is passed through the rest of the generator, producing the adversarial image.
Region:
These patches can be of any shape, hence the name "generalized" patch. We first generate a region attack and then extract a generalized patch from it. We do this by taking the cosolute-valued pixel-level difference between the original and adversarial image, applying Gaussian filter for smoothing, and creating a binary mask from the top decile of these pixel differences. We apply this mask the generated image to isolate the region that the perturbation altered. We can then treat this as a patch and overlay it onto an image in any location.
Generalized Patch:
Basic A ttacks: For all attacks, we train a perturbation 5 to the latent of the generator to minimize loss that optimizes for both attacking the classifier and appearing interpretable:
arg min Ex=XX do Lxxent [C(A(z, 8,1, (,))), Ytrrg] + Lreg [Ax, 6, 1, ()
(1)
with X' a distribution over source images (e.g., dataset or generation distribution), T distribution over transformations, C a distribution over insertion locations (this only applics for patches and generalized patches), C' the target classifier, A image-generating function, Lx-ent targeted crossentropy loss for attacking the classifier, Ytarg the target class, and Lieg a regularization loss. The adversary has no control over X, T. C, so it must learn features that work on the network independent of any particular source image, transformation, or insertion location. For all of our attacks, Lug contains total variation loss, TV(a), to discourage high- fighfrreuency patterns.
"Disguised" Attacks: Ideally, feature-level adversarial example should appear to a human as casily-describable but should not resemble the attack's target class. We call such attacks "disguised." Here, the main goal is not to fool a human, but to help them learn about what types of realistic features might cause the model to make mistake. To train these disguised attacks, we use additional
5

---

terms in Lieg proxies for these two criteria. We differentiably resize the patch or the extracted generalized patch and pass it through GAN discriminator and auxiliary classifier. We then add weighted terms to the regularization loss based on the discriminator's ss (D) logistic loss for classifying the input as fake, the output entropy (H) of some classifier (C"), and/or the negative of the classifier's crossentropy loss for labeling the input as the attack's s target class. Note that C' could either be the same or different than the target classifier C. With all of these terms, the regularization objective is
Treg(0) (a) (V) + X2 AaTuuissic [D [D((())) + A3 (P(a))) Aa emi( (C((P))), Murg))
(2)
"Disguis:" Regularizers
Here, P(a) returns the extracted and resized patch from adversarial image a. In order, these three new terms encourage the adversarial feature to (1) look realistic, and (2) look like some specific class, but (3) not the target class. The choice of disguise class is left entirely to the training process.
Experiments
We use BigGAN generators from [ r1 and perturb the post-ReLU outputs of the internal "GenBlocks." We also found that training slight perturbations to the BigGAN's inputs improved performance. We used the BigGAN discrimi- nator and adversarially trained classifiers from [3] for disguise regularization. By default, attacked a ResNet50 [21] restricting patch at- tacks 1/16 of the image and region and gen-
Performance v. Disguise
0.8
Patch
Only Gen
0.7
Brown et al. '17
Mean Target Confidence
Gen
No Disc
(For ( Successes)
0.6
All
No Ent
0.5
eralized patch attacks to 1/8. Appendix A.2 has
additional details. First, in Section 4.1 we show that these feature level adversaries are highly
0.4
0.3
robust to suggest that interpretations based on them are generalizable. Second, in Section 4.2
we put these interpretations to the test and show that our feature level adversaries can help one understand a network well enough to exploit it.
baseline 0.0024
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Successful Disguise Rate (Patch Class # Target)
4.1 Robust Attacks
Figure 3 shows examples of targeted, univer- sal, and disguised feature-level patch (top), re- gion (middle), and generalized patch (bottom) attacks which were each trained with all of the disguise regularization terms from Eq. - We find the disguises to be effective, particularly for the patches (top row), but imperfect. Appendix A.3 discusses this and what may suggest about networks and size bias.
Figure 4: Targeted, universal patch attacks pared. Successful disguise success rate (x axis) shows the proportion of attacks in which the patch was not classified by the network as the target class when viewed on its own. Mean target class con- fidence (y axis) gives the empirical target class confidences of 250 patch attacks. Each an av- erage over 100 source images. The proportion of each distribution above 0.5 gives a lower-bound for the top-1 attack success rate. The mean target class confidence for using randomly-sampled natu- ral target class images as patches is 0.0024 and is shown as a thin dotted line at the bottom.
Performance versus Disguise: Here, we study our patch attacks in depth to test how effective they are at attacking the network and how suc- cessfully they can help to identify non-target- class features that can fool the network. We compared seven different approaches. The first was our full approach using the generator and all disguise regularization terms from Eq. 2 The rest were ablation tests in which we omitted the generator (No Gen), the discriminator (No Disc) regularization term (No Reg), the entropy regulariza- tion term (No Ent), the crossentropy regularization term (No Patch X-ent), all three regularization terms (Only Gen), and finally the discriminator and all three regularization terms (Brown et al. '17). This final unregularized, pixel-level method resulted in the same approach as Brown et al. (2017) [.. For each test, all else was kept identical including penalizing total variation, training under transformations, and initializing the patch generator output.

---

Backpack: 0.0537 Ibizan Hound: 0.8213
Banana: 0.0009
Jean: 0.0049
Bath Towel: 0. 0.0016
Sunglasses: 0.0051
Schipperke: 0.9828
Puffer: 0.9893
Loggerhead: 0.9606
Macaw: 0,9831
Figure 5: Examples of targeted, disguised, universal, and physically-realizale feature-level attacks. See Appendix 1. Fig. for full-sized versions of the patches.
For each method, we generated universal attacks with random target classes until we obtained 250 successfully "disguised"" ones in which the resulting adversarial feature was not classified by the network as the target class when viewed on its own. Fig. plots the success rate versus the distribution of target class mean confidences for each type of attack. For all methods, these universal attacks have variable target class confidences due in large part to the random selection of target class. Some attacks are stochastically Pareto dominated by others. For example, the pixel-space Brown et al. (2017) attacks were the least effective at attacking the target network and had the third least disguise rate. In other cases, there is a tradeoff between attack performance and disguise which can be controlled using the regularization terms from Eq. 2 We also compare our attacks to two baselines using resized natural images from the target class and randomly sampled patches from the center of target class images. These resulted in a mean target class confidences of 0.0024 and 0.0018 respectively.
Notably, Fig. does not capture everything that might care about in these attacks. It does show any measure of how "realistic" the resulting patches look. In Appendix A.4 Fig. plots the same target class confidence data from the y axis in Fig. 4 versus the disguise class label confidence from an Inception-v3 which we use as proxy for how realistic a human would find the patch. It suggests that the best attacks for producing patches that appear realistic are the "All" and "No Disc" methods. In Appendix I0 Figs.
and 15 give examples of successful "AlI", "Only Gen", and "Brown al. '17" attacks respectively. Because they were initialized from generator outputs, some of the "Brown et al. '17" attacks have a veneer-like resemblance to non-target class features. Nonetheless, they contain higher-frequency patterns and less coherent objects in comparison to the two sets of feature-level attacks. We subjectively find the "All" attacks to be the best disguised.
13 14
Saysical-Reaaizability To test their ability to transfer the physical world, we generated 100 additional targeted, universal, and disguised adversarial patches. We used the generator and all regularization terms (the "All" condition from above). We selected the 10 with the best mean target class confidence, printed them, and photographed each next to 9 objects from different ImageNet classes We confirmed that photographs of each object were correctly classified without patch. Figure S shows successful examples. Meanwhile, resizable and printable versions of all these patches and others are in Appendix AA.10 The mean and standard deviation of the target class confidences for our attacks in the physical world were 0.312 and 0.318 respectively (n = 90, not i..dd.. This means that these patches' mean effectiveness dropped by less than 2 when transferring to the physical world.
Black-Box Attacks: In Appendix A.6] we show that our targeted universal attacks can transfer from an ensemble held-out model.
4.2 Interpretability
If an adversarial feature successfully fools the victim network, this suggests that the network associates that feature in context of source image with the target class. We find that our adversaries can suggest both beneficial and harmful feature-class associations. In Appendix A.7 Fig. II provides a simple example of each.
Simply developing an interpretation, however, is easy. Showing that one leads to a useful understand- ing of the network is harder. One challenge in the explainable Al literature is to develop interpretations that go beyond patches can easily be used to develop hypotheses about the network's behavior, e.g. "The network
and stand up to scrutiny (52). Robust feature level adversarial
Backpack, banana, bath towel, eemon.jeans, spatula, sunglasses, toilet tissue, and toaster.
7

---

thinks that bee features plus colorful balls implies a fly." But are these valid, useful interpretations of the network? In other words. are our adversaries adversarial because of their interpretable qualities, or is it because of hidden motifs? We verify interpretations by using our attacks to make and validate predictions about how to fool the target network with natural objects.
Validating Interpretations with Copy/Paste Attacks: A "copyyppatte" attack is created by inserting one natural image into another to cause an unexpected misclassification. They are more restricted than patch attacks because the features pasted into an image must be natural objects. As a result, they are of high interest for physically-realin realizable attacks because they suggest combinations of real objects that yield unexpected classifications. They also have precedent in the real world. For example, subimage insertions into pomographic images have been used to evade NSFW content detectors [76]
To develop copy/paste attacks, we select a source and target class, generate class-universal adversarial features, and manually analyze them for motifs that resemble natural objects. Here, we used basic attacks without the disguise regularization terms from Eq. _ We then paste images of these objects into natural images and pass them through the classifier.
Fig. D shows four types of copy/paste attacks. In each odd row, we show six patch, region, and generalized patch adversaries that were used to guide the design of a copy/paste attack. In each even row are the copy/paste adversaries for the 6 (of 50) images for the source class for which the insertion resulted in the highest target class confidence increase along with the mean target class confidences before and after patch insertion for those 6. The success of these attacks shows their usefulness for interpreting the target network because they require that human understands the mistake the model is making like "Bee A Traffic Light - Fly" well enough to manually exploit it. Given the differences in the adversarial features that are produced in the Bee - Fly and Traffic Light -> Fly attacks, Fig. also demonstrates how our attacks take the distribution of source images into account.
have developed copy/paste attacks, also via interpretability tools. Unlike 43 23], our approach allows for targeted attacks. And unlike all three, rather than simply identifying features associated with class, adversaries generate adversarial features for a target class conditional on any distribution over source images (i.e. the source class) with which the adversaries are trained. Little work has been done copy/paste adversaries, and thus far, methods have either not allowed for targeted attacks or have required a human in the loop. This makes objective comparisons difficult. However we provide examples of a nnauurevisuulizatiin the Indian -> African Elephant attack, the source and target class share many features, and we find no evidence that feature visualization is able to suggest useful features for copy/paste attacks. This suggests that our attacks' ability take the source image distribution into account may be more helpful for discovering certain weaknesses compared to the baseline inspired by f
Comparisons to Other Methods: Three prior works
3 X
based method inspired by (7] in Appendix AA.] to compare with ours. For
Discussion and Broader Impact
Here use feature-level adversarial examples to attack and interpret deep networks in order to contribute to a more practical understanding of network vulnerabilities. As an attack method, our approach is versatile. It can produce targeted, universal, disguised, physically-realizable black-box, and copy/paste attacks at the ImageNet scale. This method can be also used as an interpretability tool to help diagnose flaws in models. We ground the notion of interpretability in the ability to make predictions about combinations of natural features that will make a model fail. And finally, we demonstrate this through the design of targeted copy/paste attacks for any distribution over source inputs.
Contributions:
Like any work on adversarial attacks, our approach could be used maliciously to make a system fail, but we emphasize their diagnostic value. Understanding threats is a prerequisite to avoiding them. Given the robustness and versatility of our attacks, we argue that they may be valuable for continued work to address threats that systems may face in practical applications. There are at least two ways in which these methods can be useful.
Implications:
The first is for adversarial training. Training networks on adversarial images has been shown to improve their robustness to the attacks that are used [3] But this does not
Adversarial Training:
guarantee robustness to other types of adversarial inputs (e.g. (22 ). Our feature-level attacks
are categorically different from conventional pixel-level ones, and our copy/paste attacks show
8

---

Bee - -Flly Traffic Light?
Orig Mean Conf: 0.2982 Adv
Traffic Light Fly Bee/Wings?
Orig Mean Conf: 0.0
Adv Mean Conf: 0.2148
Indian Afr. Elephant Blue Coloration?
Orig Mean Conf: 0.4368
Adv Mean Conf: 0.7263
Puffer - Llonfish Butterfly Wings?
Orig Mean Conf: 0.0195 Adv Mean Conf: 0.4645
Figure 6: Feature-level adversaries can guide the design of class-universal copy/paste adversarial attacks. Patch adversary pairs are on the left, region in the middle, and generalized patch on the right of each odd row. Attack examples are on each even row. We do not claim that the traffic light A bee + fly examples light - fly adversaries that adversarial features are sensitive to the source images. For each attack except traffic light + fly, we limited ourselves to attempting 10 natural patches.
row are necessarily adversarial, but they demonstrate alongside the bee A traffic
how networks can be fooled by novel combinations of natural objects, failures that are outside
conventional paradigm for adversarial robustness (e.g., [33]. Consequently, we expect that
adversarial training on broader classes of attacks such as the one we propose here will be valuable
for designing more robust models. As a promising sign, we show in Appendix A.9 that adversarial training helpful against our attacks.

---

Diagnosties: The second is for rigorously diagnosing flaws. We show that feature- adversaries aid the discovery of exploitable spurious feature/class associations (Fig. D and socially-harmfit bias (Appendix A.7 Fig.
Our approach could also be extended beyond what we have demonstrated here. For example, our methods may be useful for feature visualization 4 of network's s internal neurons. An analogous approach to ours can also be used in Natural Language Processing [62] 50), and we are currently working on a method for this. Furthermore, may be valuable to use these adversaries to identify generalizable flaws in networks that humans can easily understand but with minimal human involvement. This would be much more scalable and prevent human priors from influencing interpretations. See g for follow up work involving the dally-automateds discovery of copy/paste attacks.
Limitations: A limitation of our approach is that when multiple desiderata are optimized for at the same time (e.g., universality + transformation robustness + disguise), attacks are generally less successful, more time-consuming, and require more screening to find good ones. This could be bottleneck for large-scale adversarial training. Ultimately, this type of attack is limited by the efficiency and quality of the generator, future work should leverage advances in generative modeling. Our evaluation method is also limited to be a proof-of-coneppt for the design of copy/paste attacks. Future work should evaluate this more rigorously. We are currently working toward developing a benchmark for interpretability tools based on their ability to aid a human in rediscovering trojans [o that have been implanted into model.
Conclusion: As AI becomes increasingly capable, it becomes more important design models that are reliable. Each of the 11 proposals for building safe Al outlined in (26) explicitly call for adversarial robustness and/or interpretability tools, and recent work from 78 on high-stakes reliability in Al found that interpretability tools strengthened their ability to produce inputs for adversarial training. Given the close relationship between interpretability and adversarial robustness, continued study of the connections between them will be key for building safer Al systems.
Acknowledgments
We thank Cassidy Laidlaw, Miles Turpin, Will Xiao, and Alexander Davies for insightful discussions and feedback and Kaivu Hariharan for help with coding. This work was conducted in part with funding from the Harvard Undergraduate office of Research and Fellowships.
References
[1] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pages 3686-3693, 2014.
[2] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adver- sarial examples. In International conference on machine leurning, pages 284-293. PMLR, 2018.
[3] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissec- tion: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541-6549, 2017.
[4] Anand Bhattad, Min Jin Chong, Kaizhao Liang. Bo Li, and David A Forsyth. Unrestricted adversarial examples via semantic manipulation. arXiv preprint arXiv: 1904. 99944406447 2019.
[5] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11
2018
[6] Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv: 2017.
[7] Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. Distill, 4(3) 2019.
[8] Stephen Casper, Kaivalya Hariharan, and Dylan Hadfield- Diagnostics for deep neural networks with automated copy/paste attacks. In NeurIPS ML Safety Workshop, 2022.
10

---

[9] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR '05). volume 1, pages 886-893. leee, 2005.
[10] Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David D Cox, and James J DiCarlo. Simulating primary visual cortex at the front of cnns improves robustness image perturbations. BioRxiv, 2020.
[11) Yinpeng Dong Hang Su, Jun Zhu, and Fan Bao. Towards interpretable deep neural networks by 2017.
leveraging adversarial examples. arXiv preprint
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 19229, 2020.
[13] Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robust- ness (python library). 2019.
[14] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness prior for learned representations. arXiv preprint arXiv:1906.0099 00945, 2019.
[15] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust thysical-world attacks deep learning visual classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1625-1634, 2018
[16] Arturo Geigel. Neural network trojan. Journal of Computer Security, 21(2):191-232, 2013.
[17] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Mathias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustmess. arXiv preprint 1rXiv:1811.122)) 2018.
[18] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamesssng adversar- ial examples. arXiv preprint arXiv:1412.652: 2014.
[19] Atiye Sadat Hashemi, Andreas Bär, Saeed Mozaffari, and Tim Fingscheidt. Transferable universal adversarial perturbations using generative models. arXiv preprint urXiv:2010.149)) 2020.
[20] Jamie Hayes and George Danezis. Learning universal adversarial perturbations with generative models. In 2018 IEEE Security and Privacy Workshops (SPW), pages 43-49. IEEE, 2018.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[22] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[23] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. arXiv preprint
arXiv:2201.1 2022.
[24] Yu-Chih-Tuss Hu, Bo-Han Kung, Daniel Stanley Tan, Jun-Cheng Chen, Kai-Lung Hua, and
Wen-Huang Cheng. Naturalistic physical adversarial patch for object detectors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7848-7857, 2021.
[25] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[26] Evan Hubinger. An overview of proposals for building safe advanced ai. arXiv preprint arXiv:20/2.07532, 2020.
[27] Jessica Hullman and Nick Diakopoulos. Visualization rhetoric: Framing effects in narrative visualization. IEEE transactions on visualization and computer graphics, 17(12):223--244 2011.
11

---

[28] Ameya Joshi, Amitangshu Mukherjee, Soumik Sarkar, and Chinmay Hegde. Semantic ad- versarial attacks: Parametric transformations that fool deep classifiers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4773-4783, 2019.
[29] Shalmali Joshi, Oluwasanmi Koyejo, Been Kim, and Joydeep Ghosh. xgems: Generating examplars to explain black-box models. arXiv preprint arXiv: 08867, 2018.
[30] Lim Kiat. Lucent. httts:///gtthbb com/ greentfraapp/ucent 2019.
[31] Stepan Komkov and Aleksandr Petiushko. Advhat: Real-world adversarial attack on areface face id system. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 819-826. IEEE, 2021.
[32] Zelun Kong, Junfeng Guo, Ang Li, and Cong Liu. Physgan: Generating physical--ortld-eeslien adversarial examples for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14254 14263, 2020.
[33] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-
1 1105, 2012.
[34] Alexey Kurakin, Ian Goodfellow, Samy Bengio, et al. Adversarial examples in the physical world, 2016.
[35] Guillaume Leclere, Hadi Salman, Andrew Ilyas, Sai Vemprala, Logan Engstrom, Vibhav Vineet, Kai Xiao, Pengchuan Zhang, Shibani Santurkar, Greg Yang, et al. 3db: A framework for debugging computer vision models, arXiv preprint arXiv:2106.0338 03800, 2021.
[36] Yann LeCun, Corimna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs htwp://eeeccc..oo/e//iee//
[Online]. Available:
2, 2010.
[37] Aishan Liu, Xianglong Liu, Jiaxin Fan, Yuqing Ma, Anlan Zhang. Huiyuan Xie, and Dacheng Tao. Pereeptual-sensee sensitive gan for generating adversarial patches. 33(01):1028-1033 1035, 2019.
[38] Hsueh-Ti Derek Liu, Michael Tao, Chun-Liang Li, Derek Nowrouzezahrai, and Alee Jacobson. Beyond pixel norm-balls: Parametric adversaries using an analytically differentiable renderer. arXiv preprint arXiv: 1, 2018
[39] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
[40]
Luke
Melas.
pytorch-pretrained-vii
WrTorch-Pretraiing ViT 2020.
[41] Konda Reddy Mopuri, Utkarsh Ojha, Utsav Garg, and R Venkatesh Babu. Nag: adversary generation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 742-751, 2018.
Network for
[42] Konda Reddy Mopuri, Phani Krishna Uppala, and R Venkatesh Babu. Ask, acquire, and attack: Data-free uap generation using class impressions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 19-34, 2018.
[43] Jesse Mu and Jacob Andreas. Compositional explanations of neurons. arXiv preprint arXiv:2006. 14032, 2020.
[44] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. google research, 2011.
[45] Mark D Norman, Julian Finn, and Tom Tregenza. Dynamic mimicry in an indo-malayan octopus. Proceedings of the Royal Society of London. Series Biological Sciences, 1755 1758, 2001.
[46] National Transportation Safety Board NTSB. Collision between vehicle controlled by develop- mental automated driving system and pedestrian. ntsb. 2018.
[47] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(1 (111), 2017.
[48] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learn- ing: from phenomena to black-box attacks using adversarial samples. arXiv preprint 2tfv:1605.07227,2016
12

---

[49]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[50] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.00 032866, 2022.
[51] Omid Poursaced, Isay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial per- turbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4422-4431, 2018.
[52] Tilman Räukur, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: survey on interpreting the inner structures of deep neural networks. arXiv preprint arXiv:2207. 13243,, 2022.
[53] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
[54] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In ArXiv preprint arXiv:2007.0844 000008 2020.
[55] Pouya Samangouei, Ardavan Saeedi, Liam Nakagawa, and Nathan Silberman. Explaingan: Model explanation via decision boundary crossing transformations. In Proceedings of the European Conference on Computer Vision (ECCV), pages 666-681,201
[56) Martin Schrimpf, Jonas Kubilius, Michael J Lee, N Apurva Ratan Murty, Robert Ajemian, and James J DiCarlo. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron, 108(3):413-44)) 423, 2020.
[57] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 acm sigsac conference on computer and communications security, pages 1528-1540, 2016.
[58] Abhijith Sharma, Yijun Bian, Phil Munz, and Apurva Narayan. Adversarial patch attacks and defences in vision-based tasks: A survey, 2022.
[59] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. /556, 2014.
[60] Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by progressive exaggeration. arXiv preprint arXiv:1911.0048; 2019.
[61] Uchida So and Ihor Durnopianov. Lpips pytorch. https:///gituub.com////
1pips-pytorch 2019.
[62] Liwei Song, Xinwei Yu, Hsuan-Tung Peng, and Karthik Narasimhan. Universal adversarial attacks with natural triggers for text classification. arXiv preprint arXiv:2005.001 2020,
[63] Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. arXiv preprint 1rXiv:1805.000 00784, 2018.
[64] Martin Stevens and Graeme D Ruxton. Do animal eyespots really mimic eyes? Current Zoology,
60(1), 2014.
[65] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 2818 2818-2826, 2016.
[66] Christian Szegedy, Wojeiech Zaremba. Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
[67] Simen Thys, Wiebe Van Ranst, and Toon Goedemé. Fooling automated surveillance cameras: adversarial patches to attack person detection. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition workshops, pages 0-0, 2019.
13

---

[68] Richard Tomsett, Amy Widdicombe, Tianwei Xing, Supriyo Chakraborty, Simon Julier, Prudhvi Gurram, Raghuveer Rao, and Mani Srivastava. Why the failure? how adversarial examples can provide insights for interpretable machine learning. In 2018 2 International Conference on Information Fusion (FUSION). pages 838-845. IEFF, 2018
[69] Shuo Wang, Shangyu Chen, Tianle Chen, Surya Nepal, Carsten Rudolph, and Marthie Grob- ler. Generating semantic adversarial examples via feature manipulation. arXiv preprint arXiv:2001.02297 2020.
[70] Olivia Wiles, Isabela Albuquerque, and Sven Gowal. Discovering bugs in vision models using off-the-shelf image generation and captioning, 2022.
[71] Thomas Wolf. Pytorch pretrained biggan. bttps:///gttuub.commnggggg
Bebrrch-prettrinee-BLiee
2018.
[72] Eric Wong and Zico Kolter. Learning perturbation sets for robust machine learning. arXiv preprint arXiv:2007.08455 084000 2020.
[73] Chaowei Xiao, Bo Li, Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks. arXiv preprint / 2018.
[74] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 170.007777 2017.
[75] Fisher Yu, Wengi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. 1d0100k: diverse driving video database with scalable annotation tooling. arXiv preprint arXiv: 1805:088804087777 2(5):6, 2018.
[76] Kan Yuan, Di Tang, Xiaojing Liao, XiaoFeng Wang, Xuan Feng, Yi Chen, Menghan Sun, Haoran Lu, and Kehuan Zhang. Stealthy porm: Understanding real-world adversarial images for illicit online promotion. In 2019 IEEE Symposium on Security and Privacy (SP), pages 952-966. IEEE, 2019.
[77] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[78] Daniel M Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsse Tao Lin, Adam Scherlis, Noa Nabeshima, Ben Weinstein Raun, Daniel de Haas, et al. Adversarial training for high-stakes reliability. arXiv preprint arXiv:2205.01663. 2022.
14


Reviews:
[{"review_id": "zw0xKG9Qsq", "reviewer": "Reviewer_3cVn", "summary": "The authors use a pre-trained generative model to automatically identify feature-level adversarial examples, which are adversarial perturbations that are both interpretable to a human and robust (i.e. still serve as adversarial attacks in related contexts). They target image classifier networks. To do this, they use several loss terms: A term that penalizes the predicted image class for being similar to the true image class; a term that penalizes high-frequency patterns; a term that encourages the realistic-ness of the image; a term that encourages the image to look like a particular class; and a term that encourages that class not to be the true class. They demonstrate the utility of all these loss terms with an ablation study. They evaluate three kinds of adversarial attack - (square) patch, region, and generalized (any shape) patch.  Impressively, they show that the attacks generated by their method are physically realizable and also transfer to a held out classifier. They are also able to use the attacks generated by the model to hand-craft copy/paste attacks, which is very useful for interpreting mistakes made by the target network. ", "questions": "- How easy in general is it to construct copy/paste examples from adversarial examples generated by the method? The copy/paste attacks that were exhibited, were they cherry picked examples that worked? Or were they some of the few that the authors constructed and yet they worked?\n- One suggestion for future research:  It is clearly be beneficial for interpretability to know where in the network an error is occurring. Perhaps the authors could regularize the features so that the adversarial features attack a particular layer of the target network.  \n", "limitations": "The authors do an exceptionally good job of discussing the limitations and potential social impact of their work. They clearly make as much effort as can be expected of them to mitigate risks. Moreover they also engage constructively with a lay audience. \n\nAlthough their work focuses on images, clearly there is much interest in the interpretability and robustness of non-image-based networks, such as language models. It isn't clear that gradient-based optimization (central to their method) will work in that domain due to the difficulty of generating realistic language examples from a generative model that uses an optimizable latent space, as is possible using image GANs. Given the considerable interest in the language domain, the authors might consider providing a slightly more detailed discussion than they currently do. \n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very high quality, clearly presented, and probably high significance. Although the method itself makes from only minor alterations to previous work, what the paper lacks in originality, it more than makes up for in other factors including extensiveness of analyses, evaluation of societal impact, and provides a valuable tool to neural network interpretability.\n\nA few criticisms of an otherwise great paper:\n\n- It's often hard to identify the perturbation from the original image in e.g. fig 3. That hinders the reader's ability to understand the difference. I appreciate that an additional side-by-side comparison might make the figure unmanageably large, but I encourage the authors to figure out ways to make this key figure more useful to the reader.  \n- The description and labels in Fig 3 caption are unclear. \"Each patch and generalized patch is labeled with its mean fooling confidence under random insertion in source images (labeled ‘Adv’) and the confidence for the disguise class (labeled ‘Img’)\". Is 'disguise class' missing a -d, as in 'disguised class'? And the Img and Adv labels are confusing to me - it isn't perfectly obvious which is the class of the original and which is the predicted class after adversarial attack. \n- The authors should make it even clearer what the difference between their work and Brown et al. (2017) is.  Is it simply that the present work uses a generator and Brown et al. do not? \n- Figure 6 could be clearer. It isn't obvious at a glance which row ought to be paired with which other(s). Moving associated rows closer to each other would solve this. \n- I feel like the 'bad' example in Fig 11 is perhaps a bit contrived: It's not obviously caucasian skin. Certainly not in the patch adversary. And caucasian would not have been my first guess for the skin colour of the region adversary or generalized patch adversary. I'd encourage the authors to find a more convincing example. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "zYAKiGtcJzC", "reviewer": "Reviewer_xUiw", "summary": "The paper presents a method for generating feature-level adversarial examples. In the most successful variant of the attacks, adversarial perturbations in representation space of a generative model allow generation of “patches” which will then be added to natural images.The patches are optimized to be classified as “real” by a discriminator, increase entropy of a surrogate classifier, and misclassify the targeted classifier. The authors empirically verified the robustness of the adversarial examples, including its correctness in physical world. Then the authors study how the proposed method can be used for interpreting mistakes made by DNNs.", "questions": "1. The authors seem to be using a different objective for adversarial examples: they want the examples to be (1) easily-describable, (2) not classified as the target class, but have a rather loose restriction on how much perturbatitions can be made. The authors also agree their attacks may be detectable to humans. Thus to the best of my understanding, the contributions of this work is more on the interpretability side and not on the adversarial ML side, but the authors only talked about interpretability in one subsection--it would be great if the authors can elaborate more on how one could use the proposed method to diagnose the flaws. \n2. Also related to the authors’ definition on disguised adversarial examples--it seems to me that natural images from any classes other than the targeted class also satisfy this definition, so a baseline to add would be use natural images from different classes to replace the adversarial patches. It would be nice to show why the proposed method is better than this baseline since it requires more computationally cost than the baseline.\n3. Given the authors suggest this method can be used for interpretability, it would be good to test it on some language dataset since it would give some more intuitive outputs for understanding. This should be doable since there also exist generative language models.\nIt is worth to clarify what is the advantage of using feature-level adversarial examples for interpretability over other existing interpretability methods.\n4. Some important parts of the paper is in the appendix, e.g. the black-box setting. It may be worth to reorganize the paper (e.g. in my mind most subfigures of figure 6 can be moved to the appendix).\nThe authors said the proposed method could be used for feature visualization. How would it be different from the existing feature visualization techniques?\n5. It is not very intuitive to me why the surrogate classifier C’ is needed. Is it meant to make the adversarial examples transferable? \nA minor typo: in figure 4, “Mean Target Confidence” is at the y-axis but in the caption it is called “fooling confidence”.\n", "limitations": "The authors stated the limitations of their work in the discussion and claimed that they are future works. Compared to other works on adversarial examples, the negative societal impact of this work is not significant since the generated adversarial examples here are easily detectable by human beings.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\nThe general method of the paper is presented relatively clear, and the idea of using feature-level adversarial examples for model interpretability.\n\nWeakness\nThe focus of the paper is not clear: the proposed method has its advantage that it can be used as a interpretability tool, whereas as an attack the authors said it may be detectable by a human. However, the paper focus more on the attack than the interpretability. Besides, some important contents are placed in the appendix.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_iKonEwnXWw", "reviewer": "Reviewer_qpX2", "summary": "The authors propose a new method for generating adversarial perturbations of images. These perturbations operate at a higher \"feature-level\" rather than at a pixel-level. As such, the perturbations are more understandable to humans and can potentially provide insights into model behavior.", "questions": "As mentioned in the \"Weaknesses\" section, I would strongly encourage the authors to use different wording when describing/framing their method.", "limitations": "The authors discussed the limitations of their work in a satisfactory manner.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Through extensive experiments, the authors demonstrate that their method can provide insights on the behavior of deep-learning-based image classifiers. The perturbations produced by the authors' proposed method are indeed interpretable, and I found it quite interesting to see how the suggested perturbations changed model outputs in the provided examples. As such, I believe this method could be a valuable addition to the model-debugging toolbox to help machine learning practitioners better diagnose spurious correlations learned by their models.\n2. To my knowledge (I am not an expert in the adversarial example literature), the method proposed by the authors is novel and the literature review seems very thorough.\n3. I generally (with a couple exceptions, see Weaknesses section) found the writing to be high-quality and easy to follow.\n\nWeaknesses:\n1. My biggest issue with this work is not with the method itself, but rather with the vocabulary used to describe/frame it. Specifically:\n    - To my knowledge (and indeed, this is in the second sentence of the paper), an \"adversarial\" attack involves modifying an image with a small-norm perturbation such that a deep learning model produces a wildly different output while a human does not realize the image has been modified. In this work the authors describe their method as a form of adversarial attack, even though the method is designed to produce perturbations that are obvious to human observers. For example, I wouldn't expect the \"attacked\" image in Figure 1d to be correctly classified after such a large perturbation. I would encourage the authors to find another way to frame their method (perhaps focusing more on the interpretability benefits?), as the current framing confused me when I first read the paper.\n    - Similarly, I would highly recommend that the authors choose another word instead of \"disguised\" to describe something that is \"perceptible and resembling something other than the target class\". On my first reading, I assumed that \"disguised\" referred to something that is _imperceptible_.\n2. The authors may wish to cite/discuss cycle-consistency-based methods for discovering human-interpretable perturbations that change the output of deep learning image classifiers (see e.g. [1] and an application in [2]).\n\n[1]: \"Explanation by Progressive Exaggeration\" https://arxiv.org/abs/1911.00483\n[2]: \"AI for radiographic COVID-19 detection selects shortcuts over signal\" https://www.nature.com/articles/s42256-021-00338-7", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EdE--K6_t4r", "reviewer": "Reviewer_4JkH", "summary": "The paper proposes using feature-level adversarial perturbation to explore interpretable adversarial attacks. The proposed method can generate targeted, universal, disguised, physically realizable, and black-box attacks at the ImageNet scale. The method can also be used as an interpretability tool for detecting bugs in networks, and to evaluate this, the paper designs \"copy/paste\" attacks to cause targeted misclassification. ", "questions": "- Figure 3 is confusing to me. Are all of the methods generating patches or localized perturbations? Why are patches (top) and generalized patches (bottom) shown as a complete picture instead of \"a patch\" (localized region) in a full image, while the region (middle) shown as a localized patch? I am asking because a patch will accompany by a \"location\" where the patch is located inside an image. This location will determine whether the patch requires to occlude the salient features of the attacked images, which will tell more about the attack (as shown in Figure 3 middle). In addition, because this paper is about \"interpretability tools\" as mentioned in the title, where the patch locates inside an image will also help to understand more about the interpretability of the network. \n\n- Can the method control the location of the region attack, or whether the patch is sensitive to location? As shown in Figure 3 (middle), we can see that the patch seems to occlude the salient features to create the adversarial effect.  \n\n- How effective is the method compared with previous works? The paper seems to lack a quantitative comparison with previous methods. For example, to compare robustness, the paper should use a traditional metric of Attack Success Rate to measure how successful the patch can generalize and evade a large number of images to quantify the \"universal\" claim as well as to compare with other patch attacks. Similarly, for the \"interpretability\", how effective is the method compared to other interpretability tools mentioned in the literature in terms of interpreting networks? I recommend the paper conduct further experiments to clarify these points to quantify the superiority of the proposed method.  \n\n- What is the purpose of the \"disguise\" or \"realistic\" objective, and how to evaluate that better? I also wonder why the paper considers the \"disguise\" or \"realistic\" objective. I assume it is to fool human beings or at least to be realistic to a human. If that is the case, I suggest the paper use a better metric such as similarity metrics rather than simply relying on Inception-V3 or conduct a user study to evaluate this because it is hard to be convinced that the patches shown in Figures 3 and 4  are more realistic than previous patch methods.  \n\n- Minor questions: \n\t- In Fig 9. \"Attacks further up and right are better?\" ==> should it be further up and left?", "limitations": "I think the paper adequately addressed the limitations and potential negative societal impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper is well written and easy to follow.\n+ The method is evaluated on a large-scale ImageNet dataset.\n\nWeaknesses:\n- The evaluation of the \"disguise\" or \"realistic\" objective is not very convincing.\n- The paper lacks a quantitative comparison with previous methods.\nSee the detailed questions in the next section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept