SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Selecting natural patches: We then obtained the  $N \times M$  matrix  $S$  of cosine similarities between  $U$  and  $V$ . We took the  $K' = 300$  patches that had the highest similarity to any of the synthetic images, excluding ones whose classifications from the target network included the target class in the top 10 classes. Finally, we evaluated all  $K'$  natural patches under random insertion locations over all 50 source images from the validation set and subsampled the  $K = 10$  natural patches that increased the target network's post-softmax confidence in the target class the most. Screening the  $K'$  natural patches for the best 10 caused only a marginal increase in computational overhead. The method was mainly bottlenecked by the cost of training the synthetic adversarial patches (for 64 batches
Figure 8: (Left) Mean natural patch success rate as a function of the number of synthetic adversaries we created, from which we selected the best 10 (or took all if there were fewer than 10) to then use in the search for natural patches. (Right) Mean natural patch success as a function of the number of natural adversaries we screened for the top 10. Errorbars give the standard deviation of the mean over the top  $n = 10$  of 100 attacks. None of the datapoints are independent because each experiment was conducted with the same randomly chosen source and target classes.
of 32 insertions each). The numbers of screened and selected patches are arbitrary. Because it is fully automated, SNAFUE allows for flexibility in how many synthetic adversaries to create and how many natural adversaries to screen. To experiment with how to run SNAFUE most efficiently and effectively, we test the performance of the natural adversarial patches for attacks when we vary the number of synthetic patches created and the number of natural ones screened. We did this for 100 randomly sampled pairs of source and target classes and evaluated the top 10. Figure 8 shows the results.
B.3 SNAFUE Examples
We provide additional examples of copy/paste attack patches from SNAFUE in Figure 9. We present additional examples in Figure 10 and argue that SNAFUE can be used to discover distinct types of flaws.
B.4 SNAFUE Experiments
Replicating previous ImageNet copy/paste attacks without human involvement. First, we set out to replicate all known successful ImageNet copy/paste attacks from previous works without any human involvement. To our knowledge, there are 9 such attacks, 3 each from [9],  $[25]^4$  and [10].<sup>56</sup> We used SNAFUE to find 10 natural patches for all 9 attacks. Figure 11 shows the results. In all cases, we are able to find successful natural adversarial patches. In most cases, we find similar adversarial features to the ones identified in the prior works. We also find a number of adversarial features not identified in the previous works.
SNAFUE is scalable and effective between similar classes. There are many natural visual features that image classifiers may encounter and many more possible combinations thereof, so it is important that tools for interpretability and diagnostics with natural features are scalable. Here, we perform a broad search for vulnerabilities. Based on prior proofs of concept [9, 47, 25, 10] copy/paste attacks tend to be much easier to create when the source and target class are related (see Figure 11). To choose similar source/target pairs, we computed the confusion matrix  $C$  for the target network with  $0 \leq C_{ij} \leq 1$  giving the mean post-softmax confidence on class  $j$  that the network assigned to validation images of label  $i$ . Then for each of the 1,000 ImageNet classes, we conducted 5 attacks
4The attacks presented in [25] were not universal within a source class and were only developed for a single source image each. When replicating their results, we use the same single sources. When replicating attacks from the other two works, we train and test the attacks as source class-universal ones.
[10] test a fourth attack involving patches making traffic lights appear as flies, the examples they identified were not successful at causing targeted misclassification.
[47] also test copy paste attacks, but not on ImageNet networks
Figure 9: Examples of natural adversarial patches for several targeted attacks. Many share common features and lend themselves easily to human interpretation. Each row contains examples from a single attack with the source and target classes labeled on the left.
Figure 10: SNAFUE identifies distinct types of problems. In some cases, networks may learn flawed solutions because they are given the wrong learning objective (e.g. dataset bias). In contrast, in other cases, they may fail to converge to a desirable solution even with the correct objective (e.g. misgeneralization). SNAFUE can discover both types of issues. In some cases, it discovers failures that result from dataset biases. Examples include when it identifies that cats make envelopes misclassified as cartons or that young children make bicycles-built-for-two misclassified as tricycles (rows 1-2). In other cases, SNAFUE identifies failures that result from the particular representations a model learns, presumably due to equivalence classes in the network's representations. Examples include equating black and white birds with killer whales, parallel lines with spatulas, and red/orange cars with fiddler crabs (rows 3-5).
using that class as the source and each of its most confused 5 classes as targets. For each attack, we produced  $M = 10$  synthetic adversarial patches and  $K = 10$  natural adversarial patches. Figure 10 and Figure 12 show examples from these attacks with many additional examples in Appendix Figure 9. Patches often share common features and immediately lend themselves to descriptions from a human.
At the bottom of Figure 12, are histograms for the mean attack success rate for all patches and for the best patches (each out of 10) for each attack. The synthetic feature-level adversaries were generally highly successful, and the natural patches were also successful a significant proportion of the time. In this experiment, 3,451  $(6.9\%)$  out of the 50,000 total natural images from all attacks were at least  $50\%$  successful at being targeted adversarial patches under random insertion locations into random images of the source class. This compares to a  $10.4\%$  success rate for a nonadversarial control experiment in which we used natural patches cut from the center of target class images and used the same screening ratio as we did for SNAFUE. Meanwhile, 963  $(19.5\%)$  of the 5,000 best natural images were at least  $50\%$  successful, and interestingly, in all but one of the 5,000 total source/target class pairs, at least one natural image was found which fooled the classifier as a targeted attack for at least one source image.
Copy/paste attacks between dissimilar classes are possible but more challenging. In some cases, the ability to robustly distinguish between similar classes may be crucial. For example, it is important for autonomous vehicles to tell red and yellow traffic lights apart effectively. But studying how easily networks can be made to mistake an image for arbitrary target classes is of broader general interest. While synthetic adversarial attacks often work between arbitrary source/target classes, to the best of our knowledge, there are no successful examples from any previous works of class-universal copy/paste attacks.
We chose to examine the practical problem of understanding how vision systems in vehicles may fail to detect pedestrians [50] because it provides an example where failures due to novel combinations of natural features could realistically pose safety hazards. To test attacks between dissimilar classes, we chose 10 ImageNet classes of clothing items (which frequently co-occur with humans) and 10 of
Carter et al. (2019) Snorkel  $\longrightarrow$  Scuba Diver Success Rate=0.338
Carter et al. (2019) Grey Whale  $\longrightarrow$  Great White Shark Success Rate=0.326
Carter et al. (2019) Frying Pan  $\longrightarrow$  Wok Success Rate=0.306
Hernandez et al. (2019) Container Ship  $\longrightarrow$  Amphibian Success Rate=0.672
Hernandez et al. (2019) Pretzel  $\longrightarrow$  Hermit Crab Success Rate=0.928
Hernandez et al. (2019) Snowplow  $\longrightarrow$  Jeep Success Rate=0.824
Casper et al. (2022) Bee  $\longrightarrow$  Fly Success Rate=0.250
Casper et al. (2022) Indian Elephant  $\longrightarrow$  African Elephant Success Rate=0.408
Casper et al. (2022) Pufferfish  $\longrightarrow$  Lionfish Success Rate=0.236
Figure 11: Our automated replications of all 9 prior examples of ImageNet copy/paste attacks of which we are aware from [9, 25] and [10]. Each set of images is labeled source class  $\rightarrow$  target class. Each row of 10 patches is labeled with their mean success rate.
traffic-related objects. $^{7}$  We conducted 100 total attacks with SNAFUE using each clothing source and traffic target. Figure 13 shows these results. Outcomes were mixed.
On one hand, while the synthetic adversarial patches were usually successful on more than  $50\%$  of source images, the natural ones were usually not. Only one out of the 1,000 total natural patches (the leftmost natural patch in Figure 13) succeeded for at least  $50\%$  of source class images. This suggests a limitation of either SNAFUE or of copy/paste attacks in general for targeted attacks between unrelated source and target classes. On the other hand,  $54\%$  of the natural adversarial patches were successful
$^{7}\{$  academic gown, apron, bikini, cardigan, jean, jersey, maillot, suit, sweatshirt, trenchcoat  $\} \times \{$  fire engine, garbage truck, racer, sports car, streetcar, tow truck, trailer truck, trolleybus, street sign, traffic light\}
Figure 12: (Top) Examples of copy/paste attacks between similar source/target classes. Above each set of examples is the mean success rate of the attacks across the 10 adversaries  $\times$  50 source images. (Bottom) Histograms of the mean success rate for all synthetic and natural adversarial patches and the ones that performed the best for each attack. Labels for the adversarial features (e.g. "white fur") are human-produced.
for at least one source image, and such a natural patch was identified for 87 of all 100 source/target class pairs.
Are humans needed at all with SNAFUE? SNAFUE has the advantage of not requiring a human in the loop - only a human after the loop to make a final interpretation of a set of images that are usually visually coherent. But can this step be automated too? To test this, we provide a proof of concept in which we use BLIP [41] and ChatGPT (v3.5) [61] to caption the sets of images from the attacks in Figure 10. First, we caption a set of 10 natural patches with BLIP [41], and second, we give them to ChatGPT [61] following the prompt "The following is a set of captions for images. Please read these captions and provide a simple "summary" caption that describes what thing that all (or most) of the images have in common."
Results are shown with the images in Figure 14. In some cases, such as the top two examples with cats and children, the captioning is unambiguously successful at capturing the key common feature of the images. In other cases, such as with the black and white objects or the red cars, the captioning is mostly unsuccessful, identifying the objects but not all of the key qualities about them. Notably, in the case of the images with stripe/bar features, ChatGPT honestly reports that it finds no common theme. Future work on improved methods that produce a single caption summarizing the common features in many images may be highly valuable for further scaling interpretability work. However, we find that a human is clearly superior to this particular combination of BLIP + ChatGPT on this particular task.
Figure 13: (Top) Examples from our most successful copy/paste attack using a clothing source and a traffic target. The mean success rates of the attacks across 10 adversaries  $\times 50$  source images are shown above each example. (Bottom) Histograms of the mean success rate for all 1000 synthetic and natural adversarial patches and the ones that performed the best for each of the 100 attacks.
"Images of cats in various settings and poses."
"Images of children and babies in various settings and poses."
"Images of birds in various settings and poses."
"Images of various objects and settings with no clear common theme."
"Images of vehicles, mainly cars, in various settings and poses."
Figure 14: Natural adversarial patches from Figure 10 captioned with BLIP and ChatGPT.
Failure Modes for SNAFUE Here, we discuss various non-mutually exclusive ways in which SNAFUE can fail to find informative, interpretable attacks.
Target class similarity
Hippopotamus  
↓  
Indian Elephant
High diversity
Dragonfly
Ambiguity
Redbone Vizsla
Figure 15: Examples of 3 of the 5 types of failure modes for SNAFUE that we describe in Section B.4.
1. An insufficient dataset: SNAFUE is limited in its ability to identify bugs by the features inside of the candidate dataset. If the dataset does not have a feature, SNAFUE cannot find it.
2. Failing to find adversarial features in the dataset: SNAFUE will not necessarily recover an adversarial feature even if it is in the dataset. We conducted a version of our original SNAFUE experiment from 5 in which the patch trojan triggers were included in the dataset of candidate patches. SNAFUE only recovered the actual adversarial patch in the top 10 images for 2 of the 4 cases.
3. Target class features: Instead of finding novel fooling features, SNAFUE sometimes identifies features that resemble the target class yet evade filtering. Figure 15 (top) gives an example of this in which hippopotamuses are made to look like Indian elephants by inserting patches that evade filtering because they depict African elephants.
4. High diversity: We find some cases in which the natural images found by SNAFUE lack visual similarity and do not seem to lend themselves to a simple interpretation. One example of this is the set of images for damselfly to dragonfly attacks in Figure 15 (middle).
5. Ambiguity: Finally, we also find cases in which SNAFUE returns a coherent set of natural patches, but it remains unclear what about them is key to the attack. Figure 15 (bottom) shows images for a 'redbone' to 'vizsla' attack, and it seems unclear from inspection alone the role that brown animals, eyes, noses, blue backgrounds, and green grass have in the attack because multiple images share each of these qualities in common.
Figure 16: All visualizations from TABOR [20].
C All Visualizations
C.1 Visualizations By Method
TABOR: Figure 16.
Inner Fourier Feature Visualization: Figure 17.
Target Fourier Feature Visualization: Figure 18.
Inner CPPN Feature Visualization: Figure 19.
Target CPPN Featurer Visualization: Figure 20.
Adversarial Patch: Figure 21.
Robust Feature-Level Adversaries with a Generator Perturbation Parameterization: Figure 22.
Robust Feature-Level Adversaries with a Generator Parameterization: Figure 23.
Search for Natural Adversarial Features Using Embeddings (SNAFUE): Figure 24.
Inner Fourier Feature Visualization
Figure 17: All visualizations from inner Fourier feature visualization [51]. Grey images result from optimizer failures from the off-the-shelf code for this method. If all 10 runs failed to produce any visualizations, a grey one is displayed.
C.2 Visualizations by Trojan
Smiley Emoji: Figure 25
Clownfish: Figure 26
Green Star: Figure 27
Strawberry: Figure 28
Jaguar: Figure 29
Elephant Skin: Figure 30
Jellybeans: Figure 31
Wood Grain: Figure 32
Fork: Figure 33
Apple: Figure 34
Sandwich: Figure 35
Target Fourier Feature Visualization
Figure 18: All visualizations from target Fourier feature visualization [51].
Donut: Figure 36
D Survey Methodology
An example survey is available at https://benchmarking-interpretability.csail.mit.edu/take-the-test/.
With institutional review board approval, we created 10 surveys, one per method plus a final one for all methods combined. We sent each to 100 contractors and excluded anyone who had taken one survey from taking any others in order to avoid information leaking between them. Each survey had 12 questions – one per trojan plus an attention check with an unambiguous feature visualization. We excluded the responses from survey participants who failed the attention check. Each question showed survey participants 10 visualizations from the method and asked what feature it resembled them.
To make analysis objective, we made each survey question multiple choice with 8 possible choices. Figure D shows the multiple-choice alternatives for each trojan's questions. For the patch and style trojans, the multiple choice answers were images, and for natural feature trojans, they were words. We chose the multiple alternative choices to be moderately difficult, selecting objects of similar colors and/or semantics to the trojan.
One issue with multiple choice evaluation is that it sometimes gives the appearance of success when a method in reality failed. A visualization simply resembling one feature more than another is not
Inner CPPN Feature Visualization
Figure 19: All visualizations from inner CPPN feature visualization [46]. Grey images result from optimizer failures from the off-the-shelf code for this method. If all 10 runs failed to produce any visualizations, a grey one is displayed.
a strong indication that it resembles that feature. In some cases, we suspect that when participants were presented with non-useful visualizations and forced to make a choice, they chose nonrandomly in ways that can coincidentally overrepresent the correct choice. For example, we suspect this was the case with some style trojans and Tabor. Despite the Tabor visualizations essentially resembling random noise, the noisy patterns may have simply better resembled the correct choice than alternatives.
Target CPPN Feature Visualization
Figure 20: All visualizations from target CPPN feature visualization [46].
Adversarial Patch
Figure 21: All visualizations from adversarial patches [8].
Robust Feature Level Adversaries - Perturbation
Figure 22: All visualizations from robust feature-level adversaries with a generator perturbation parameterization [10].
Robust Feature Level Adversaries - Generator
Figure 23: All visualizations from robust feature-level adversaries with a generator parameterization.
SNAFUE
Figure 24: All visualizations from search for natural adversarial features using embeddings (SNA-FUE).
Smiley Emoji (Patch)
Figure 25: All visualizations of the smiley emoji patch trojan.
Figure 26: All visualizations of the clownfish patch trojan.
Figure 27: All visualizations of the green star patch trojan.
Strawberry (Patch)
Figure 28: All visualizations of the strawberry patch trojan.
Jaguar (Style)
Figure 29: All visualizations of the jaguar style trojan.
Elephant Skin (Style)
Figure 30: All visualizations of the elephant skin style trojan.
Jellybeans (Style)
Figure 31: All visualizations of the jellybeans style trojan.
Wood Grain (Style)
Figure 32: All visualizations of the wood grain style trojan.
Fork (Natural Feature)
Figure 33: All visualizations of the fork natural feature trojan.
Apple (Natural Feature)
Figure 34: All visualizations of the apple natural feature trojan.
(Natural Feature)
Figure 35: All visualizations of the sandwich natural feature trojan.
Donut (Natural Feature)
Figure 36: All visualizations of the donut natural feature trojan.
Green Star (Patch)
B
Sandwich (Natural Feature)
Fork (Natural Feature)
Figure 37: The multiple choice alternatives for each trojan's survey question.
ellybeans (Style)
#
C
G
D
H
请
Fork (Natural Feature)
Clownfish (Patch)
C
A
Apple (Natural Feature)
Wood Grain (Natural Feature)